# Tribology Lab Data — Master Plan

**Discovery · Cleansing/Normalization · Ontology · Conversion**

Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
Status: Plan v1 — Draft

---

## 0. Objective

Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.

The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.

---

## 1. Guiding principles

1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.

---

## 2. Phase overview

| Phase | Name | Nature | Primary output |
|-------|------|--------|----------------|
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |

Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.

---

## 3. Phase 1 — Storage Discovery & Non-destructive Profiling

**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.

### 3.1 Inputs
- Root of the ~700 GB CSV tree (read-only).

### 3.2 Activities

**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).

**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
- atmosphere tokens (`ambient`, `dryNitrogen`, …)

Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).

**C. Structural profiling (header-only + sampled body).** Per file, without full load:
- read header row → column names, count, order;
- infer per-column datatype from first/last N rows + a random sample;
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.

**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.

### 3.3 Algorithm sketch (streaming, O(files), bounded memory)

```
for each file in walk(root):
	meta      = fs_metadata(file)
	tokens    = tokenize_filename(file)                  # -> measurement type, ids, atmosphere
	header    = read_first_line(file)
	columns   = parse_columns(header)
	dtypes    = infer_types(sample_rows(file, head=N, tail=N, random=M))
	fp        = fingerprint(columns)                     # ordered-set hash
	emit_manifest_row(meta, tokens, columns, dtypes, fp)
```

### 3.4 Artifacts
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).

### 3.5 Exit criteria
- 100% of files inventoried and fingerprinted.
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.

---

## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)

**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.

The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.

### 4.1 The iteration loop (per measurement type)

1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.

### 4.2 Instruments Repository (ontology sub-model)

Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
- identity (type, model/label, measurement type it produces);
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
- association to `measurement_types` / `column_definitions` in the data dictionary.

Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.

### 4.3 Anomaly detection

Criteria are **derived from the data**, not fixed a priori. Starting catalog:

| Class | Description | Source of criterion |
|-------|-------------|---------------------|
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
| Missing required | Null in a non-nullable canonical field | Schema constraints |
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |

Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.

### 4.4 Data dictionary / metadata catalog

Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.

### 4.5 Artifacts
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
- **`instruments_repository`** — instrument entities + characteristics + ranges.
- **`rejects_store`** — quarantined anomalies with reason codes.
- **`ontology_notes`** — running record of ERD changes and rationale.

### 4.6 Exit criteria
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.

---

## 5. Phase 4 — Conversion & Export

**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.

### 5.1 Targets
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.

### 5.2 Design
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
- Lineage is emitted alongside every export (see §6).

### 5.3 Artifacts
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).

### 5.4 Exit criteria
- All five target formats generated from one canonical snapshot.
- Round-trip validation: record counts and key aggregates match across formats.
- Lineage complete and resolvable for sampled records.

---

## 6. Cross-cutting concerns

### 6.1 Lineage & processing-flow documentation
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
- Lineage is a first-class export, present in every target format.

### 6.2 Configuration & rule registry
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.

### 6.3 Validation gates
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.

---

## 7. Target artifacts catalog (consolidated)

| Artifact | Phase | Form |
|----------|-------|------|
| storage_manifest | 1 | JSONL + SQLite |
| schema_fingerprint_report | 1 | JSON/CSV |
| unmatched_report | 1 | CSV |
| storage_map | 1 | Markdown |
| column_mapping | 2↔3 | CSV/JSON (versioned) |
| canonical_dataset | 2↔3 | partitioned intermediate |
| relational_schema (DDL) | 2↔3 | SQL |
| data_dictionary | 2↔3 | table/CSV |
| instruments_repository | 2↔3 | table/CSV |
| rejects_store | 2↔3 | JSONL |
| ontology_notes | 2↔3 | Markdown |
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
| lineage + build manifest | all | JSONL |

---

## 8. Tooling & environment

- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
- **PostgreSQL** materialized in Phase 4 (system of record).
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
- Python indentation: **tabs**.

---

## 9. Performance strategy for 700 GB

- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.

---

## 10. Open items & risks

1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.

---

*End of Plan v1.*