fix(docs): fix formatting issues in README.md
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
248
docs/examples/TribologyData_MasterPlan_v1.md
Normal file
248
docs/examples/TribologyData_MasterPlan_v1.md
Normal file
@@ -0,0 +1,248 @@
|
||||
# Tribology Lab Data — Master Plan
|
||||
|
||||
**Discovery · Cleansing/Normalization · Ontology · Conversion**
|
||||
|
||||
Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
|
||||
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
|
||||
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
|
||||
Status: Plan v1 — Draft
|
||||
|
||||
---
|
||||
|
||||
## 0. Objective
|
||||
|
||||
Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
|
||||
|
||||
The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Guiding principles
|
||||
|
||||
1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
|
||||
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
|
||||
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
|
||||
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
|
||||
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
|
||||
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
|
||||
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
|
||||
|
||||
---
|
||||
|
||||
## 2. Phase overview
|
||||
|
||||
| Phase | Name | Nature | Primary output |
|
||||
|-------|------|--------|----------------|
|
||||
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
|
||||
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
|
||||
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
|
||||
|
||||
Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
|
||||
|
||||
**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
|
||||
|
||||
### 3.1 Inputs
|
||||
- Root of the ~700 GB CSV tree (read-only).
|
||||
|
||||
### 3.2 Activities
|
||||
|
||||
**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
|
||||
|
||||
**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
|
||||
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
|
||||
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
|
||||
- atmosphere tokens (`ambient`, `dryNitrogen`, …)
|
||||
|
||||
Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
|
||||
|
||||
**C. Structural profiling (header-only + sampled body).** Per file, without full load:
|
||||
- read header row → column names, count, order;
|
||||
- infer per-column datatype from first/last N rows + a random sample;
|
||||
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
|
||||
|
||||
**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
|
||||
|
||||
### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
|
||||
|
||||
```
|
||||
for each file in walk(root):
|
||||
meta = fs_metadata(file)
|
||||
tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere
|
||||
header = read_first_line(file)
|
||||
columns = parse_columns(header)
|
||||
dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M))
|
||||
fp = fingerprint(columns) # ordered-set hash
|
||||
emit_manifest_row(meta, tokens, columns, dtypes, fp)
|
||||
```
|
||||
|
||||
### 3.4 Artifacts
|
||||
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
|
||||
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
|
||||
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
|
||||
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
|
||||
|
||||
### 3.5 Exit criteria
|
||||
- 100% of files inventoried and fingerprinted.
|
||||
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
|
||||
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
|
||||
|
||||
**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
|
||||
|
||||
The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
|
||||
|
||||
### 4.1 The iteration loop (per measurement type)
|
||||
|
||||
1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
|
||||
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
|
||||
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
|
||||
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
|
||||
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
|
||||
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
|
||||
|
||||
### 4.2 Instruments Repository (ontology sub-model)
|
||||
|
||||
Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
|
||||
- identity (type, model/label, measurement type it produces);
|
||||
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
|
||||
- association to `measurement_types` / `column_definitions` in the data dictionary.
|
||||
|
||||
Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
|
||||
|
||||
### 4.3 Anomaly detection
|
||||
|
||||
Criteria are **derived from the data**, not fixed a priori. Starting catalog:
|
||||
|
||||
| Class | Description | Source of criterion |
|
||||
|-------|-------------|---------------------|
|
||||
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
|
||||
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
|
||||
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
|
||||
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
|
||||
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
|
||||
| Missing required | Null in a non-nullable canonical field | Schema constraints |
|
||||
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
|
||||
|
||||
Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
|
||||
|
||||
### 4.4 Data dictionary / metadata catalog
|
||||
|
||||
Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
|
||||
|
||||
### 4.5 Artifacts
|
||||
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
|
||||
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
|
||||
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
|
||||
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
|
||||
- **`instruments_repository`** — instrument entities + characteristics + ranges.
|
||||
- **`rejects_store`** — quarantined anomalies with reason codes.
|
||||
- **`ontology_notes`** — running record of ERD changes and rationale.
|
||||
|
||||
### 4.6 Exit criteria
|
||||
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
|
||||
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
|
||||
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 4 — Conversion & Export
|
||||
|
||||
**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
|
||||
|
||||
### 5.1 Targets
|
||||
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
|
||||
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
|
||||
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
|
||||
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
|
||||
5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
|
||||
|
||||
### 5.2 Design
|
||||
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
|
||||
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
|
||||
- Lineage is emitted alongside every export (see §6).
|
||||
|
||||
### 5.3 Artifacts
|
||||
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
|
||||
|
||||
### 5.4 Exit criteria
|
||||
- All five target formats generated from one canonical snapshot.
|
||||
- Round-trip validation: record counts and key aggregates match across formats.
|
||||
- Lineage complete and resolvable for sampled records.
|
||||
|
||||
---
|
||||
|
||||
## 6. Cross-cutting concerns
|
||||
|
||||
### 6.1 Lineage & processing-flow documentation
|
||||
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
|
||||
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
|
||||
- Lineage is a first-class export, present in every target format.
|
||||
|
||||
### 6.2 Configuration & rule registry
|
||||
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
|
||||
|
||||
### 6.3 Validation gates
|
||||
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
|
||||
|
||||
---
|
||||
|
||||
## 7. Target artifacts catalog (consolidated)
|
||||
|
||||
| Artifact | Phase | Form |
|
||||
|----------|-------|------|
|
||||
| storage_manifest | 1 | JSONL + SQLite |
|
||||
| schema_fingerprint_report | 1 | JSON/CSV |
|
||||
| unmatched_report | 1 | CSV |
|
||||
| storage_map | 1 | Markdown |
|
||||
| column_mapping | 2↔3 | CSV/JSON (versioned) |
|
||||
| canonical_dataset | 2↔3 | partitioned intermediate |
|
||||
| relational_schema (DDL) | 2↔3 | SQL |
|
||||
| data_dictionary | 2↔3 | table/CSV |
|
||||
| instruments_repository | 2↔3 | table/CSV |
|
||||
| rejects_store | 2↔3 | JSONL |
|
||||
| ontology_notes | 2↔3 | Markdown |
|
||||
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
|
||||
| lineage + build manifest | all | JSONL |
|
||||
|
||||
---
|
||||
|
||||
## 8. Tooling & environment
|
||||
|
||||
- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
|
||||
- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
|
||||
- **PostgreSQL** materialized in Phase 4 (system of record).
|
||||
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
|
||||
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
|
||||
- Python indentation: **tabs**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Performance strategy for 700 GB
|
||||
|
||||
- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
|
||||
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
|
||||
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
|
||||
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
|
||||
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
|
||||
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open items & risks
|
||||
|
||||
1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
|
||||
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
|
||||
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
|
||||
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
|
||||
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
|
||||
|
||||
---
|
||||
|
||||
*End of Plan v1.*
|
||||
Reference in New Issue
Block a user