b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
1.4 KiB
1.4 KiB
Task 09 — Cleanse & Normalize → Canonical Dataset
Goal
Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
Input
- Mappings (Task 07), schema + instrument ranges (Task 08),
config/anomaly_rules.json, tokens (Task 03).
Actions
- Write
scripts/cleanse.py— streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset). - Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
- Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per
anomaly_rules.json. - Clean rows →
canonical/<measurement_type>/…partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows →rejects/<type>.jsonlwith{source_file, row_offset, reason_code, raw_payload}. - Lineage per output row:
{source_file_id, row_offset, mapping_version, schema_version}→ compact companion files inlineage/.
Intermediate Result (done when)
- Every in-scope file processed; row counts reconcile:
source ≈ canonical + rejectsper file. - Canonical partitions written; lineage present for every canonical row.