# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)

Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.

## Tasks

1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.

## Final Goal

From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.