LabDataStorageEvaluation/docs/examples/TribologyData_MasterPlan_v1.md

Tribology Lab Data — Master Plan

Discovery · Cleansing/Normalization · Ontology · Conversion

Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation Status: Plan v1 — Draft

---

0. Objective

Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.

The existing artifacts (Business Description, ERD, ADR) already define the target domain model. This plan governs how the raw filesystem is reconciled against that model and materialized into the export formats.

---

1. Guiding principles

1. Non-destructive. Source CSV is read-only. All work products are derived; the raw tree is never mutated.
2. Lineage-first. Every target row carries provenance back to (source_file, byte/row offset, transform rules applied). Lineage is a build-time output, not an afterthought.
3. Phases 2 and 3 are iterative and co-dependent. A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
4. Instruments Repository is built, not referenced. No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
5. Pure Python first. Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
6. Canonical intermediate. All exports derive from one normalized representation, so formats never diverge.
7. Reproducibility. The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.

---

2. Phase overview

Phase	Name	Nature	Primary output
1	Storage Discovery & Profiling	Non-destructive scan	Storage manifest + schema fingerprints
2 ↔ 3	Cleansing/Normalization ↔ Ontology	Iterative loop	Cleansed canonical data + relational schema + data dictionary + Instruments Repository
4	Conversion & Export	Deterministic build	JSON / SQLite / PostgreSQL / RDF outputs

Cross-cutting throughout: lineage, anomaly detection, configuration/rule registry.

---

3. Phase 1 — Storage Discovery & Non-destructive Profiling

Goal. Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.

3.1 Inputs

• Root of the ~700 GB CSV tree (read-only).

3.2 Activities

A. Filesystem inventory. Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).

B. Filename tokenization. Parse filenames into structured tokens against the known conventions:

• coupon-level (no sample token) → XRF (L_081423_1), macro friction (data1..data3)
• sample-level (sample token 456/457) → nanoindentation (run1..run9), friction test (test_N) / cycle (cycleN ambient, _N dry nitrogen)
• atmosphere tokens (ambient, dryNitrogen, …)

Produce a token-extraction report and a list of filenames that fail to match any known pattern (candidates for new rules or anomalies).

C. Structural profiling (header-only + sampled body). Per file, without full load:

• read header row → column names, count, order;
• infer per-column datatype from first/last N rows + a random sample;
• compute a schema fingerprint (normalized hash of the ordered column set) to group files of identical structure.

D. Schema grouping. Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the same purpose with different column sets/naming.

3.3 Algorithm sketch (streaming, O(files), bounded memory)

for each file in walk(root):
	meta      = fs_metadata(file)
	tokens    = tokenize_filename(file)                  # -> measurement type, ids, atmosphere
	header    = read_first_line(file)
	columns   = parse_columns(header)
	dtypes    = infer_types(sample_rows(file, head=N, tail=N, random=M))
	fp        = fingerprint(columns)                     # ordered-set hash
	emit_manifest_row(meta, tokens, columns, dtypes, fp)

3.4 Artifacts

• storage_manifest — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
• schema_fingerprint_report — fingerprint → {file count, member files, canonical column set} per measurement type.
• unmatched_report — files/filenames not matching known folder or naming rules.
• storage_map — human-readable reconstruction of folder/file purpose (as-is vs as-designed).

3.5 Exit criteria

• 100% of files inventoried and fingerprinted.
• Every file assigned a (measurement type, attachment level) or flagged as unmatched.
• Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.

---

4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)

Goal. Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.

The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.

4.1 The iteration loop (per measurement type)

1. Draft canonical schema from the target ERD (e.g., friction_data_points, xrf_spectrum_points).
2. Map raw columns → canonical columns. Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
3. Cleanse & normalize on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
4. Detect anomalies (see 4.3); route offending rows/files to quarantine with reason codes.
5. Feed findings back to the ontology — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
6. Repeat until the mapping covers all fingerprints of that type with no unresolved columns.

4.2 Instruments Repository (ontology sub-model)

Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:

• identity (type, model/label, measurement type it produces);
• measured quantities, units, and valid physical ranges (min/max, resolution);
• association to measurement_types / column_definitions in the data dictionary.

Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.

4.3 Anomaly detection

Criteria are derived from the data, not fixed a priori. Starting catalog:

Class	Description	Source of criterion
Out-of-range	Value outside instrument's physical capability	Instruments Repository ranges
Sequence gap	Missing steps in ordered series (time, cycle, run)	Series continuity check
Structural mismatch	File of a purpose with unexpected column set	Fingerprint vs canonical
Type violation	Non-coercible value in a typed column	Mapping/coercion
Duplicate	Repeated rows / repeated identity keys	Key + row hashing
Missing required	Null in a non-nullable canonical field	Schema constraints
Encoding/format	Malformed rows, delimiter/decimal issues	Parser

Anomalies are quarantined, not dropped: written to a rejects store with (source_file, row_offset, reason_code, raw_payload) for review and rule refinement.

4.4 Data dictionary / metadata catalog

Maintained in lockstep with the schema: measurement_types and column_definitions (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.

4.5 Artifacts

• column_mapping — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
• canonical_dataset — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
• relational_schema — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
• data_dictionary — measurement_types + column_definitions.
• instruments_repository — instrument entities + characteristics + ranges.
• rejects_store — quarantined anomalies with reason codes.
• ontology_notes — running record of ERD changes and rationale.

4.6 Exit criteria

• Every fingerprint of every measurement type has a complete, unambiguous mapping.
• Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
• Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.

---

5. Phase 4 — Conversion & Export

Goal. Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.

5.1 Targets

1. JSON mega-file — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
2. Multiple JSON files — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
3. SQLite — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
4. PostgreSQL — system of record. DDL + COPY-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
5. RDF / triplestore — metadata & provenance graph only (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.

5.2 Design

• One export driver reads the canonical intermediate + schema and dispatches to per-format writers.
• Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
• Lineage is emitted alongside every export (see §6).

5.3 Artifacts

• Per-format output sets + a build manifest (what was produced, from which canonical snapshot, with which config/rule versions).

5.4 Exit criteria

• All five target formats generated from one canonical snapshot.
• Round-trip validation: record counts and key aggregates match across formats.
• Lineage complete and resolvable for sampled records.

---

6. Cross-cutting concerns

6.1 Lineage & processing-flow documentation

• Per target row: source_file, source row offset, source_modified_at, ordered list of transform rules applied.
• Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible processing-flow record.
• Lineage is a first-class export, present in every target format.

6.2 Configuration & rule registry

• Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in versioned config, not code. This is what makes Phases 2↔3 iterative without rewrites.

6.3 Validation gates

• Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.

---

7. Target artifacts catalog (consolidated)

Artifact	Phase	Form
storage_manifest	1	JSONL + SQLite
schema_fingerprint_report	1	JSON/CSV
unmatched_report	1	CSV
storage_map	1	Markdown
column_mapping	2↔3	CSV/JSON (versioned)
canonical_dataset	2↔3	partitioned intermediate
relational_schema (DDL)	2↔3	SQL
data_dictionary	2↔3	table/CSV
instruments_repository	2↔3	table/CSV
rejects_store	2↔3	JSONL
ontology_notes	2↔3	Markdown
exports (JSON/SQLite/PG/RDF)	4	per format
lineage + build manifest	all	JSONL

---

8. Tooling & environment

• Language: Python (streaming, stdlib-first; csv, pathlib, hashlib, json, sqlite3). Add pyarrow/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
• No DB engine in Phases 1–2 beyond SQLite for the manifest index.
• PostgreSQL materialized in Phase 4 (system of record).
• Ontop / R2RML for the RDF surface (Phase 4, deferred activation).
• Future: migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
• Python indentation: tabs.

---

9. Performance strategy for 700 GB

• Never full-load. Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
• Bounded memory. Constant-memory transforms; no whole-file DataFrames on large series.
• Partition the canonical intermediate by measurement type and, for the high-volume series (friction_data_points), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
• Parallelize by file (embarrassingly parallel scan/transform); aggregate manifests after.
• Deterministic ordering for reproducible, resumable runs (checkpoint by manifest offset).
• Volume estimates from Phase 1 drive the friction_data_points partitioning decision before Phase 4.

---

10. Open items & risks

1. macro_friction_points schema is provisional (columns normal_load_mn, average_coefficient_of_friction pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
2. Unknown fingerprints. True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
3. Anomaly criteria maturity. Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
4. Atmosphere/instrument code drift. Filename conventions vary (cycleN vs _N, atmosphere tokens); normalization rules must be discovered and versioned.
5. Lineage granularity vs. size. Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.

---

End of Plan v1.