diff --git a/docs/examples/TribologyData_MasterPlan_v1.md b/docs/examples/TribologyData_MasterPlan_v1.md deleted file mode 100644 index 4fb69a7..0000000 --- a/docs/examples/TribologyData_MasterPlan_v1.md +++ /dev/null @@ -1,248 +0,0 @@ -# Tribology Lab Data — Master Plan - -**Discovery · Cleansing/Normalization · Ontology · Conversion** - -Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility -Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store -Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation -Status: Plan v1 — Draft - ---- - -## 0. Objective - -Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate. - -The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats. - ---- - -## 1. Guiding principles - -1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated. -2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought. -3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize. -4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria. -5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred. -6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge. -7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates. - ---- - -## 2. Phase overview - -| Phase | Name | Nature | Primary output | -|-------|------|--------|----------------| -| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints | -| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository | -| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs | - -Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**. - ---- - -## 3. Phase 1 — Storage Discovery & Non-destructive Profiling - -**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies. - -### 3.1 Inputs -- Root of the ~700 GB CSV tree (read-only). - -### 3.2 Activities - -**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy). - -**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions: -- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`) -- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen) -- atmosphere tokens (`ambient`, `dryNitrogen`, …) - -Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies). - -**C. Structural profiling (header-only + sampled body).** Per file, without full load: -- read header row → column names, count, order; -- infer per-column datatype from first/last N rows + a random sample; -- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure. - -**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming. - -### 3.3 Algorithm sketch (streaming, O(files), bounded memory) - -``` -for each file in walk(root): - meta = fs_metadata(file) - tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere - header = read_first_line(file) - columns = parse_columns(header) - dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M)) - fp = fingerprint(columns) # ordered-set hash - emit_manifest_row(meta, tokens, columns, dtypes, fp) -``` - -### 3.4 Artifacts -- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint. -- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type. -- **`unmatched_report`** — files/filenames not matching known folder or naming rules. -- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed). - -### 3.5 Exit criteria -- 100% of files inventoried and fingerprinted. -- Every file assigned a (measurement type, attachment level) or flagged as unmatched. -- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated. - ---- - -## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative) - -**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies. - -The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity. - -### 4.1 The iteration loop (per measurement type) - -1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`). -2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering. -3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution). -4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes. -5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository. -6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns. - -### 4.2 Instruments Repository (ontology sub-model) - -Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries: -- identity (type, model/label, measurement type it produces); -- measured quantities, units, and **valid physical ranges** (min/max, resolution); -- association to `measurement_types` / `column_definitions` in the data dictionary. - -Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing. - -### 4.3 Anomaly detection - -Criteria are **derived from the data**, not fixed a priori. Starting catalog: - -| Class | Description | Source of criterion | -|-------|-------------|---------------------| -| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges | -| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check | -| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical | -| Type violation | Non-coercible value in a typed column | Mapping/coercion | -| Duplicate | Repeated rows / repeated identity keys | Key + row hashing | -| Missing required | Null in a non-nullable canonical field | Schema constraints | -| Encoding/format | Malformed rows, delimiter/decimal issues | Parser | - -Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement. - -### 4.4 Data dictionary / metadata catalog - -Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model. - -### 4.5 Artifacts -- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules. -- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7). -- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned. -- **`data_dictionary`** — `measurement_types` + `column_definitions`. -- **`instruments_repository`** — instrument entities + characteristics + ranges. -- **`rejects_store`** — quarantined anomalies with reason codes. -- **`ontology_notes`** — running record of ERD changes and rationale. - -### 4.6 Exit criteria -- Every fingerprint of every measurement type has a complete, unambiguous mapping. -- Canonical dataset produced for all in-scope files; rejects quarantined and categorized. -- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release. - ---- - -## 5. Phase 4 — Conversion & Export - -**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge. - -### 5.1 Targets -1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size. -2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy. -3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use). -4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path. -5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs. - -### 5.2 Design -- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers. -- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps). -- Lineage is emitted alongside every export (see §6). - -### 5.3 Artifacts -- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions). - -### 5.4 Exit criteria -- All five target formats generated from one canonical snapshot. -- Round-trip validation: record counts and key aggregates match across formats. -- Lineage complete and resolvable for sampled records. - ---- - -## 6. Cross-cutting concerns - -### 6.1 Lineage & processing-flow documentation -- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied. -- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**. -- Lineage is a first-class export, present in every target format. - -### 6.2 Configuration & rule registry -- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites. - -### 6.3 Validation gates -- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%. - ---- - -## 7. Target artifacts catalog (consolidated) - -| Artifact | Phase | Form | -|----------|-------|------| -| storage_manifest | 1 | JSONL + SQLite | -| schema_fingerprint_report | 1 | JSON/CSV | -| unmatched_report | 1 | CSV | -| storage_map | 1 | Markdown | -| column_mapping | 2↔3 | CSV/JSON (versioned) | -| canonical_dataset | 2↔3 | partitioned intermediate | -| relational_schema (DDL) | 2↔3 | SQL | -| data_dictionary | 2↔3 | table/CSV | -| instruments_repository | 2↔3 | table/CSV | -| rejects_store | 2↔3 | JSONL | -| ontology_notes | 2↔3 | Markdown | -| exports (JSON/SQLite/PG/RDF) | 4 | per format | -| lineage + build manifest | all | JSONL | - ---- - -## 8. Tooling & environment - -- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB. -- **No DB engine in Phases 1–2** beyond SQLite for the manifest index. -- **PostgreSQL** materialized in Phase 4 (system of record). -- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation). -- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge. -- Python indentation: **tabs**. - ---- - -## 9. Performance strategy for 700 GB - -- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row. -- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series. -- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1. -- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after. -- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset). -- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4. - ---- - -## 10. Open items & risks - -1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type. -2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it. -3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations. -4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned. -5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage. - ---- - -*End of Plan v1.* diff --git a/docs/examples/Tribology_Ontology.html b/docs/examples/Tribology_Ontology.html deleted file mode 100644 index 740a54d..0000000 --- a/docs/examples/Tribology_Ontology.html +++ /dev/null @@ -1,188 +0,0 @@ - - - - - Bundled Page - - - - -
- - - - τ - - - - - - - - - -
-
Unpacking...
- - - - - - - - - - \ No newline at end of file diff --git a/docs/specs_old/00_PLAN.md b/docs/specs_old/00_PLAN.md deleted file mode 100644 index 87d3a26..0000000 --- a/docs/specs_old/00_PLAN.md +++ /dev/null @@ -1,27 +0,0 @@ -# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27) - -Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory. -Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`. -Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation. - -## Tasks - -1. **01_setup_workspace.md** — create `work/` layout + pipeline config. -2. **02_fs_inventory.md** — walk source tree; produce file inventory. -3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched. -4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints. -5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies. -6. **06_storage_map.md** — human-readable as-is / as-designed storage map. -7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned). -8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09). -9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes. -10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%. -11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical. -12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded. -13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts. -14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series). -15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation. - -## Final Goal - -From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run. diff --git a/docs/specs_old/01_setup_workspace.md b/docs/specs_old/01_setup_workspace.md deleted file mode 100644 index bee0635..0000000 --- a/docs/specs_old/01_setup_workspace.md +++ /dev/null @@ -1,26 +0,0 @@ -# Task 01 — Setup Workspace - -## Goal -Create the working directory layout and versioned pipeline configuration. - -## Input -- Path to the raw CSV root (read-only). Ask the user if unknown. - -## Actions -1. Create: -``` -work/ - config/ # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json - manifest/ # inventory, fingerprints, reports - canonical/ # cleansed partitioned dataset - rejects/ # quarantined anomalies - schema/ # DDL, data dictionary, instruments repository - exports/{json,sqlite,postgresql,rdf}/ - lineage/ - scripts/ # all Python scripts produced by tasks -``` -2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp. -3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons. - -## Intermediate Result (done when) -- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`. diff --git a/docs/specs_old/02_fs_inventory.md b/docs/specs_old/02_fs_inventory.md deleted file mode 100644 index e30f41a..0000000 --- a/docs/specs_old/02_fs_inventory.md +++ /dev/null @@ -1,17 +0,0 @@ -# Task 02 — Filesystem Inventory - -## Goal -Non-destructive inventory of every file under `SOURCE_ROOT`. - -## Input -- `work/config/pipeline.json` (Task 01). - -## Actions -1. Write `scripts/fs_inventory.py` (streaming walk, constant memory). -2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash). -3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`. -4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram. - -## Intermediate Result (done when) -- `inventory.jsonl` row count == filesystem file count. -- `inventory_summary.md` produced; non-CSV extensions listed explicitly. diff --git a/docs/specs_old/03_filename_tokenization.md b/docs/specs_old/03_filename_tokenization.md deleted file mode 100644 index b618d66..0000000 --- a/docs/specs_old/03_filename_tokenization.md +++ /dev/null @@ -1,17 +0,0 @@ -# Task 03 — Filename & Path Tokenization - -## Goal -Parse every path/filename into structured tokens; classify each file by measurement type and attachment level. - -## Input -- `manifest/inventory.jsonl` (Task 02). -- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L__`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`. - -## Actions -1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found). -2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`. -3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes). - -## Intermediate Result (done when) -- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag. -- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies). diff --git a/docs/specs_old/04_schema_profiling.md b/docs/specs_old/04_schema_profiling.md deleted file mode 100644 index 7b8b5d8..0000000 --- a/docs/specs_old/04_schema_profiling.md +++ /dev/null @@ -1,18 +0,0 @@ -# Task 04 — Schema Profiling & Fingerprints - -## Goal -Capture the record structure of every CSV without full load; compute schema fingerprints. - -## Input -- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`. - -## Actions -1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only). -2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len). -3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files. -4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`. -5. Log unreadable/malformed files to `manifest/profile_errors.csv`. - -## Intermediate Result (done when) -- 100% of CSV files have a profile row or an entry in `profile_errors.csv`. -- Every profile carries a fingerprint. diff --git a/docs/specs_old/05_schema_grouping.md b/docs/specs_old/05_schema_grouping.md deleted file mode 100644 index 6cabf6c..0000000 --- a/docs/specs_old/05_schema_grouping.md +++ /dev/null @@ -1,16 +0,0 @@ -# Task 05 — Schema Grouping & Inconsistency Report - -## Goal -Enumerate distinct record structures per measurement type; expose structural inconsistencies. - -## Input -- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`. - -## Actions -1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`. -2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts. -3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`. - -## Intermediate Result (done when) -- The number of distinct schemas per measurement type is known and documented. -- `inconsistencies.csv` complete — this is the work queue for Task 07. diff --git a/docs/specs_old/06_storage_map.md b/docs/specs_old/06_storage_map.md deleted file mode 100644 index 4239325..0000000 --- a/docs/specs_old/06_storage_map.md +++ /dev/null @@ -1,19 +0,0 @@ -# Task 06 — Storage Map (As-Is / As-Designed) - -## Goal -Human-readable reconstruction of the storage logic: purpose of each folder level and file kind. - -## Input -- All Task 02–05 artifacts. - -## Actions -1. Write `manifest/storage_map.md`: - - annotated folder-tree template (one exemplar branch per pattern, not all 700 GB); - - purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`); - - purpose of each file kind mapped to measurement types and attachment levels; - - deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`). -2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels). - -## Intermediate Result (done when) -- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples. -- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only. diff --git a/docs/specs_old/07_column_mapping.md b/docs/specs_old/07_column_mapping.md deleted file mode 100644 index a599f82..0000000 --- a/docs/specs_old/07_column_mapping.md +++ /dev/null @@ -1,18 +0,0 @@ -# Task 07 — Column Mapping (raw → canonical) - -## Goal -For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns. - -## Input -- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`. -- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`. - -## Actions -1. For each fingerprint create `config/column_mappings/__.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`. -2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`. -3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently. -4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08. - -## Intermediate Result (done when) -- Every fingerprint has a mapping file; no raw column is unmapped or undocumented. -- Escalation list for Task 08 written to `schema/ontology_escalations.md`. diff --git a/docs/specs_old/08_ontology_schema.md b/docs/specs_old/08_ontology_schema.md deleted file mode 100644 index 9e7c393..0000000 --- a/docs/specs_old/08_ontology_schema.md +++ /dev/null @@ -1,18 +0,0 @@ -# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository - -## Goal -Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10. - -## Input -- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`. - -## Actions -1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`. -2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07). -3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules. -4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only). -5. Version the DDL (`-- schema_version: N` header); bump on each iteration. - -## Intermediate Result (done when) -- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints. -- No open items in `ontology_escalations.md`. diff --git a/docs/specs_old/09_cleanse_normalize.md b/docs/specs_old/09_cleanse_normalize.md deleted file mode 100644 index 3af0dbe..0000000 --- a/docs/specs_old/09_cleanse_normalize.md +++ /dev/null @@ -1,18 +0,0 @@ -# Task 09 — Cleanse & Normalize → Canonical Dataset - -## Goal -Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded. - -## Input -- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03). - -## Actions -1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset). -2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers). -3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`. -4. Clean rows → `canonical//…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`. -5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`. - -## Intermediate Result (done when) -- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file. -- Canonical partitions written; lineage present for every canonical row. diff --git a/docs/specs_old/10_anomaly_report.md b/docs/specs_old/10_anomaly_report.md deleted file mode 100644 index 516c5ba..0000000 --- a/docs/specs_old/10_anomaly_report.md +++ /dev/null @@ -1,17 +0,0 @@ -# Task 10 — Anomaly Report & Iteration Gate - -## Goal -Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop. - -## Input -- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions. - -## Actions -1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint. -2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08. -3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types. -4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs. - -## Intermediate Result (done when) -- `anomaly_report.md` final; every reject class documented as a true anomaly. -- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete. diff --git a/docs/specs_old/11_export_json.md b/docs/specs_old/11_export_json.md deleted file mode 100644 index e81d996..0000000 --- a/docs/specs_old/11_export_json.md +++ /dev/null @@ -1,17 +0,0 @@ -# Task 11 — Export: JSON - -## Goal -Two JSON products from the frozen canonical snapshot. - -## Input -- `canonical/` snapshot, `schema/` (Tasks 09–10). - -## Actions -1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory). -2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`. -3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level. -4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`. - -## Intermediate Result (done when) -- Both products written; spot-validation: point counts per measurement match canonical. -- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts). diff --git a/docs/specs_old/12_export_sqlite.md b/docs/specs_old/12_export_sqlite.md deleted file mode 100644 index 5ed831e..0000000 --- a/docs/specs_old/12_export_sqlite.md +++ /dev/null @@ -1,16 +0,0 @@ -# Task 12 — Export: SQLite - -## Goal -Portable single-file SQLite database conforming to the ontology. - -## Input -- `canonical/` snapshot, `schema/ddl_postgresql.sql`. - -## Actions -1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`. -2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after. -3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size). - -## Intermediate Result (done when) -- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty). -- `exports/sqlite/EXPORT_MANIFEST.json` written. diff --git a/docs/specs_old/13_export_postgresql.md b/docs/specs_old/13_export_postgresql.md deleted file mode 100644 index 86c033c..0000000 --- a/docs/specs_old/13_export_postgresql.md +++ /dev/null @@ -1,22 +0,0 @@ -# Task 13 — Export: PostgreSQL - -## Goal -System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path). - -## Input -- `canonical/` snapshot, `schema/ddl_postgresql.sql`. - -## Actions -1. Generate `exports/postgresql/`: - - `01_schema.sql` (DDL, versioned); - - `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments); - - per-table CSV staging files from canonical (streamed); - - `03_copy_load.sql` (`\copy` statements in dependency order); - - `04_post_load.sql` (indexes, constraints validation, ANALYZE); - - `load.sh` / `load.ps1` orchestration. -2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates. -3. Include `lineage` table load (row-range granularity). - -## Intermediate Result (done when) -- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax). -- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written. diff --git a/docs/specs_old/14_export_rdf.md b/docs/specs_old/14_export_rdf.md deleted file mode 100644 index fab6234..0000000 --- a/docs/specs_old/14_export_rdf.md +++ /dev/null @@ -1,17 +0,0 @@ -# Task 14 — Export: RDF (Metadata & Provenance Graph) - -## Goal -RDF surface for the metadata/provenance layer only — never the numeric series (per ADR). - -## Input -- `canonical/` snapshot headers + hierarchy, `schema/`, lineage. - -## Actions -1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id//`; base IRI in config). -2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms. -3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment. -4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples. - -## Intermediate Result (done when) -- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class. -- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written. diff --git a/docs/specs_old/15_lineage_build_manifest.md b/docs/specs_old/15_lineage_build_manifest.md deleted file mode 100644 index 89308dd..0000000 --- a/docs/specs_old/15_lineage_build_manifest.md +++ /dev/null @@ -1,18 +0,0 @@ -# Task 15 — Lineage Consolidation, Build Manifest, Final Validation - -## Goal -Close the pipeline: consolidated lineage, reproducible build record, cross-format validation. - -## Input -- All `exports/*`, `lineage/`, all config/schema versions. - -## Actions -1. Write `scripts/finalize.py`: - - consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions); - - produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation. -2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging. -3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match. - -## Final Result (pipeline done when) -- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable. -- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.