fix(docs): fix formatting issues in README.md

- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker

docs/specs_old/00_PLAN.md

@@ -0,0 +1,27 @@
+# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
+
+Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
+Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
+Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
+
+## Tasks
+
+1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
+2. **02_fs_inventory.md** — walk source tree; produce file inventory.
+3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
+4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
+5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
+6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
+7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
+8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
+9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
+10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
+11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
+12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
+13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
+14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
+15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
+
+## Final Goal
+
+From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.

docs/specs_old/01_setup_workspace.md

@@ -0,0 +1,26 @@
+# Task 01 — Setup Workspace
+
+## Goal
+Create the working directory layout and versioned pipeline configuration.
+
+## Input
+- Path to the raw CSV root (read-only). Ask the user if unknown.
+
+## Actions
+1. Create:
+```
+work/
+	config/          # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
+	manifest/        # inventory, fingerprints, reports
+	canonical/       # cleansed partitioned dataset
+	rejects/         # quarantined anomalies
+	schema/          # DDL, data dictionary, instruments repository
+	exports/{json,sqlite,postgresql,rdf}/
+	lineage/
+	scripts/         # all Python scripts produced by tasks
+```
+2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
+3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
+
+## Intermediate Result (done when)
+- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.

docs/specs_old/02_fs_inventory.md

@@ -0,0 +1,17 @@
+# Task 02 — Filesystem Inventory
+
+## Goal
+Non-destructive inventory of every file under `SOURCE_ROOT`.
+
+## Input
+- `work/config/pipeline.json` (Task 01).
+
+## Actions
+1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
+2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
+3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
+4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
+
+## Intermediate Result (done when)
+- `inventory.jsonl` row count == filesystem file count.
+- `inventory_summary.md` produced; non-CSV extensions listed explicitly.

docs/specs_old/03_filename_tokenization.md

@@ -0,0 +1,17 @@
+# Task 03 — Filename & Path Tokenization
+
+## Goal
+Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
+
+## Input
+- `manifest/inventory.jsonl` (Task 02).
+- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
+
+## Actions
+1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
+2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
+3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
+
+## Intermediate Result (done when)
+- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
+- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).

docs/specs_old/04_schema_profiling.md

@@ -0,0 +1,18 @@
+# Task 04 — Schema Profiling & Fingerprints
+
+## Goal
+Capture the record structure of every CSV without full load; compute schema fingerprints.
+
+## Input
+- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
+
+## Actions
+1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
+2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
+3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
+4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
+5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
+
+## Intermediate Result (done when)
+- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
+- Every profile carries a fingerprint.

docs/specs_old/05_schema_grouping.md

@@ -0,0 +1,16 @@
+# Task 05 — Schema Grouping & Inconsistency Report
+
+## Goal
+Enumerate distinct record structures per measurement type; expose structural inconsistencies.
+
+## Input
+- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
+
+## Actions
+1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
+2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
+3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
+
+## Intermediate Result (done when)
+- The number of distinct schemas per measurement type is known and documented.
+- `inconsistencies.csv` complete — this is the work queue for Task 07.

docs/specs_old/06_storage_map.md

@@ -0,0 +1,19 @@
+# Task 06 — Storage Map (As-Is / As-Designed)
+
+## Goal
+Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
+
+## Input
+- All Task 02–05 artifacts.
+
+## Actions
+1. Write `manifest/storage_map.md`:
+	- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
+	- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
+	- purpose of each file kind mapped to measurement types and attachment levels;
+	- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
+2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
+
+## Intermediate Result (done when)
+- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
+- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.

docs/specs_old/07_column_mapping.md

@@ -0,0 +1,18 @@
+# Task 07 — Column Mapping (raw → canonical)
+
+## Goal
+For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
+
+## Input
+- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
+- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
+
+## Actions
+1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
+2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
+3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
+4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
+
+## Intermediate Result (done when)
+- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
+- Escalation list for Task 08 written to `schema/ontology_escalations.md`.

docs/specs_old/08_ontology_schema.md

@@ -0,0 +1,18 @@
+# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
+
+## Goal
+Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
+
+## Input
+- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
+
+## Actions
+1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
+2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
+3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
+4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
+5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
+
+## Intermediate Result (done when)
+- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
+- No open items in `ontology_escalations.md`.

docs/specs_old/09_cleanse_normalize.md

@@ -0,0 +1,18 @@
+# Task 09 — Cleanse & Normalize → Canonical Dataset
+
+## Goal
+Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
+
+## Input
+- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
+
+## Actions
+1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
+2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
+3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
+4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
+5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
+
+## Intermediate Result (done when)
+- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
+- Canonical partitions written; lineage present for every canonical row.

docs/specs_old/10_anomaly_report.md

@@ -0,0 +1,17 @@
+# Task 10 — Anomaly Report & Iteration Gate
+
+## Goal
+Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
+
+## Input
+- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
+
+## Actions
+1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
+2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
+3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
+4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
+
+## Intermediate Result (done when)
+- `anomaly_report.md` final; every reject class documented as a true anomaly.
+- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.

docs/specs_old/11_export_json.md

@@ -0,0 +1,17 @@
+# Task 11 — Export: JSON
+
+## Goal
+Two JSON products from the frozen canonical snapshot.
+
+## Input
+- `canonical/` snapshot, `schema/` (Tasks 09–10).
+
+## Actions
+1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
+2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
+3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
+4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
+
+## Intermediate Result (done when)
+- Both products written; spot-validation: point counts per measurement match canonical.
+- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).

docs/specs_old/12_export_sqlite.md

@@ -0,0 +1,16 @@
+# Task 12 — Export: SQLite
+
+## Goal
+Portable single-file SQLite database conforming to the ontology.
+
+## Input
+- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
+
+## Actions
+1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
+2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
+3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
+
+## Intermediate Result (done when)
+- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
+- `exports/sqlite/EXPORT_MANIFEST.json` written.

docs/specs_old/13_export_postgresql.md

@@ -0,0 +1,22 @@
+# Task 13 — Export: PostgreSQL
+
+## Goal
+System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
+
+## Input
+- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
+
+## Actions
+1. Generate `exports/postgresql/`:
+	- `01_schema.sql` (DDL, versioned);
+	- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
+	- per-table CSV staging files from canonical (streamed);
+	- `03_copy_load.sql` (`\copy` statements in dependency order);
+	- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
+	- `load.sh` / `load.ps1` orchestration.
+2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
+3. Include `lineage` table load (row-range granularity).
+
+## Intermediate Result (done when)
+- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
+- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.

docs/specs_old/14_export_rdf.md

@@ -0,0 +1,17 @@
+# Task 14 — Export: RDF (Metadata & Provenance Graph)
+
+## Goal
+RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
+
+## Input
+- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
+
+## Actions
+1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
+2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
+3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
+4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
+
+## Intermediate Result (done when)
+- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
+- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.

docs/specs_old/15_lineage_build_manifest.md

@@ -0,0 +1,18 @@
+# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
+
+## Goal
+Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
+
+## Input
+- All `exports/*`, `lineage/`, all config/schema versions.
+
+## Actions
+1. Write `scripts/finalize.py`:
+	- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
+	- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
+2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
+3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
+
+## Final Result (pipeline done when)
+- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
+- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.