b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
28 lines
2.1 KiB
Markdown
28 lines
2.1 KiB
Markdown
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
|
||
|
||
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
|
||
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
|
||
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
|
||
|
||
## Tasks
|
||
|
||
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
|
||
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
|
||
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
|
||
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
|
||
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
|
||
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
|
||
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
|
||
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
|
||
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
|
||
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
|
||
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
|
||
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
|
||
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
|
||
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
|
||
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
|
||
|
||
## Final Goal
|
||
|
||
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.
|