b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
2.1 KiB
2.1 KiB
Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable work/ directory.
Source tree: SOURCE_ROOT (set in config/pipeline.json, Task 01). All outputs under work/.
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
Tasks
- 01_setup_workspace.md — create
work/layout + pipeline config. - 02_fs_inventory.md — walk source tree; produce file inventory.
- 03_filename_tokenization.md — parse filenames into tokens; flag unmatched.
- 04_schema_profiling.md — read headers + sampled rows; infer dtypes; compute schema fingerprints.
- 05_schema_grouping.md — cluster files by (measurement type × fingerprint); report structural inconsistencies.
- 06_storage_map.md — human-readable as-is / as-designed storage map.
- 07_column_mapping.md — per fingerprint: raw→canonical column mapping (ERD-aligned).
- 08_ontology_schema.md — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
- 09_cleanse_normalize.md — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
- 10_anomaly_report.md — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
- 11_export_json.md — JSON mega-file + per-entity JSON set from canonical.
- 12_export_sqlite.md — SQLite database per DDL, bulk-loaded.
- 13_export_postgresql.md — PostgreSQL DDL + COPY load scripts.
- 14_export_rdf.md — RDF metadata/provenance graph (R2RML mappings; no numeric series).
- 15_lineage_build_manifest.md — lineage consolidation + final build manifest; end-to-end validation.
Final Goal
From ~700 GB of raw heterogeneous CSV: a single cleansed canonical dataset conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into JSON, SQLite, PostgreSQL, and RDF exports, each carrying complete source→target lineage, reproducible from source + versioned config in one pipeline run.