Merge pull request 'feat(docs): enhance README and build pipeline rules for clarity' (#10) from dev_masha into master

Reviewed-on: #10

README.md

@@ -12,26 +12,8 @@ the measurements to 600 GB - 6 TB, and produces a ranked decision report for
 five laboratory use cases: analysis, reporting, search/filtering, archiving,
 and inter-lab exchange.
 
-**Results - interactive, single-file HTML (open in a browser or download and share):**
-
-- **[Storage Decision](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html)** -
-  one-page executive summary: the problem, the verdict (relational vs graph
-  database), and what it costs - written for management, every claim linked
-  to the full study
-  (source: [docs/research/Tribology_Storage_Decision.html](docs/research/Tribology_Storage_Decision.html)).
-- **[Storage Format Evaluation](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html)** -
-  the full study results: dashboard, measured matrix, projections to 6 TB,
-  hardware bill of materials, weighted scoring and verdict
-  (source: [docs/research/Tribology_Storage_Evaluation.html](docs/research/Tribology_Storage_Evaluation.html)).
-- **[Foreign-Key Architecture Study](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html)** -
-  the measured key-schema study behind the relational design decision
-  (source: [docs/research/Tribology_FK_Architecture.html](docs/research/Tribology_FK_Architecture.html)).
-
-Reference documents:
-
 - Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11).
 - Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md).
-- Full decision report (markdown): [out/report/REPORT.md](out/report/REPORT.md).
 
 ## Goals
 
@@ -162,7 +144,8 @@ docs/
   Initial_Prompt.md   the generative prompt behind specs 00-11 (provenance, non-binding)
   specs/        task specifications 00-11 (WHAT to build)
   rules/        binding conventions (HOW work is done)
-  research/     measured design studies and interactive result pages
+  research/     measured design studies (e.g. FK key-schema study)
+  examples/     imported reference materials (not binding)
 out/            ALL generated artifacts (reproducible; bulk git-ignored,
                 report/ config/ .done/ committed - see .gitignore):
   config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/

docs/examples/TribologyData_MasterPlan_v1.md

@@ -0,0 +1,248 @@
+# Tribology Lab Data — Master Plan
+
+**Discovery · Cleansing/Normalization · Ontology · Conversion**
+
+Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
+Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
+Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
+Status: Plan v1 — Draft
+
+---
+
+## 0. Objective
+
+Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
+
+The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
+
+---
+
+## 1. Guiding principles
+
+1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
+2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
+3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
+4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
+5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
+6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
+7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
+
+---
+
+## 2. Phase overview
+
+| Phase | Name | Nature | Primary output |
+|-------|------|--------|----------------|
+| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
+| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
+| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
+
+Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
+
+---
+
+## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
+
+**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
+
+### 3.1 Inputs
+- Root of the ~700 GB CSV tree (read-only).
+
+### 3.2 Activities
+
+**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
+
+**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
+- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
+- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
+- atmosphere tokens (`ambient`, `dryNitrogen`, …)
+
+Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
+
+**C. Structural profiling (header-only + sampled body).** Per file, without full load:
+- read header row → column names, count, order;
+- infer per-column datatype from first/last N rows + a random sample;
+- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
+
+**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
+
+### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
+
+```
+for each file in walk(root):
+	meta      = fs_metadata(file)
+	tokens    = tokenize_filename(file)                  # -> measurement type, ids, atmosphere
+	header    = read_first_line(file)
+	columns   = parse_columns(header)
+	dtypes    = infer_types(sample_rows(file, head=N, tail=N, random=M))
+	fp        = fingerprint(columns)                     # ordered-set hash
+	emit_manifest_row(meta, tokens, columns, dtypes, fp)
+```
+
+### 3.4 Artifacts
+- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
+- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
+- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
+- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
+
+### 3.5 Exit criteria
+- 100% of files inventoried and fingerprinted.
+- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
+- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
+
+---
+
+## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
+
+**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
+
+The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
+
+### 4.1 The iteration loop (per measurement type)
+
+1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
+2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
+3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
+4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
+5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
+6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
+
+### 4.2 Instruments Repository (ontology sub-model)
+
+Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
+- identity (type, model/label, measurement type it produces);
+- measured quantities, units, and **valid physical ranges** (min/max, resolution);
+- association to `measurement_types` / `column_definitions` in the data dictionary.
+
+Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
+
+### 4.3 Anomaly detection
+
+Criteria are **derived from the data**, not fixed a priori. Starting catalog:
+
+| Class | Description | Source of criterion |
+|-------|-------------|---------------------|
+| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
+| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
+| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
+| Type violation | Non-coercible value in a typed column | Mapping/coercion |
+| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
+| Missing required | Null in a non-nullable canonical field | Schema constraints |
+| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
+
+Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
+
+### 4.4 Data dictionary / metadata catalog
+
+Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
+
+### 4.5 Artifacts
+- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
+- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
+- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
+- **`data_dictionary`** — `measurement_types` + `column_definitions`.
+- **`instruments_repository`** — instrument entities + characteristics + ranges.
+- **`rejects_store`** — quarantined anomalies with reason codes.
+- **`ontology_notes`** — running record of ERD changes and rationale.
+
+### 4.6 Exit criteria
+- Every fingerprint of every measurement type has a complete, unambiguous mapping.
+- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
+- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
+
+---
+
+## 5. Phase 4 — Conversion & Export
+
+**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
+
+### 5.1 Targets
+1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
+2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
+3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
+4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
+5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
+
+### 5.2 Design
+- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
+- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
+- Lineage is emitted alongside every export (see §6).
+
+### 5.3 Artifacts
+- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
+
+### 5.4 Exit criteria
+- All five target formats generated from one canonical snapshot.
+- Round-trip validation: record counts and key aggregates match across formats.
+- Lineage complete and resolvable for sampled records.
+
+---
+
+## 6. Cross-cutting concerns
+
+### 6.1 Lineage & processing-flow documentation
+- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
+- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
+- Lineage is a first-class export, present in every target format.
+
+### 6.2 Configuration & rule registry
+- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
+
+### 6.3 Validation gates
+- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
+
+---
+
+## 7. Target artifacts catalog (consolidated)
+
+| Artifact | Phase | Form |
+|----------|-------|------|
+| storage_manifest | 1 | JSONL + SQLite |
+| schema_fingerprint_report | 1 | JSON/CSV |
+| unmatched_report | 1 | CSV |
+| storage_map | 1 | Markdown |
+| column_mapping | 2↔3 | CSV/JSON (versioned) |
+| canonical_dataset | 2↔3 | partitioned intermediate |
+| relational_schema (DDL) | 2↔3 | SQL |
+| data_dictionary | 2↔3 | table/CSV |
+| instruments_repository | 2↔3 | table/CSV |
+| rejects_store | 2↔3 | JSONL |
+| ontology_notes | 2↔3 | Markdown |
+| exports (JSON/SQLite/PG/RDF) | 4 | per format |
+| lineage + build manifest | all | JSONL |
+
+---
+
+## 8. Tooling & environment
+
+- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
+- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
+- **PostgreSQL** materialized in Phase 4 (system of record).
+- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
+- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
+- Python indentation: **tabs**.
+
+---
+
+## 9. Performance strategy for 700 GB
+
+- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
+- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
+- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
+- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
+- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
+- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
+
+---
+
+## 10. Open items & risks
+
+1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
+2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
+3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
+4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
+5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
+
+---
+
+*End of Plan v1.*

docs/research/Tribology_FK_Architecture.html

@@ -533,7 +533,7 @@ CREATE TABLE friction_cycles (
         <div class="foot">
           Measured 2026-07-11 on the deterministic simulated corpus (seed 20260711, 141.8 MiB CSV) &middot; SQLite, warm OS cache, best of 3 runs &middot;
           scripts: <span class="mono">measure_key_variants.py</span> (build + size + load + Q1) and <span class="mono">read_benchmarks.py</span> (six read patterns) &middot;
-          companion page: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" style="color:#1f5fdb;text-decoration:none">Storage Format Evaluation</a> (full study results).
+          companion page: <span class="mono">Tribology_Ontology.html</span> (measurement-result registry).
         </div>
       </section>
 

docs/research/Tribology_Storage_Decision.html

@@ -1,258 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="utf-8">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<title>Storage Decision - Executive Summary</title>
-<link rel="preconnect" href="https://fonts.googleapis.com">
-<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
-<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
-<style>
-  *{box-sizing:border-box}
-  html,body{margin:0;padding:0}
-  body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
-  .mono{font-family:"IBM Plex Mono",monospace}
-
-  header.top{display:flex;align-items:center;gap:14px;height:56px;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea;position:sticky;top:0;z-index:5}
-  .logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px;flex:none}
-  .htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
-  .htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
-  .htitle span{font-size:11.5px;color:#838a95}
-  .hright{margin-left:auto;display:flex;align-items:center;gap:8px}
-  .chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px;text-decoration:none}
-  a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
-  .chip-go{display:inline-flex;align-items:center;gap:6px;font-size:12px;color:#fff;background:#1f5fdb;padding:5px 12px;border-radius:6px;font-weight:600;text-decoration:none}
-  .chip-go:hover{background:#1a4fb8}
-
-  .page{max-width:980px;margin:0 auto;padding:30px 24px 70px}
-  .card{background:#fff;border:1px solid #e6e8ec;border-radius:12px;padding:22px 26px;margin-bottom:16px}
-  h1{margin:0 0 6px;font-size:24px;font-weight:700;letter-spacing:-.02em}
-  h2{margin:0 0 12px;font-size:16px;font-weight:600;letter-spacing:-.01em}
-  p{margin:0 0 10px;font-size:13.5px;color:#3d424b;line-height:1.65}
-  p:last-child{margin-bottom:0}
-  .muted{color:#838a95}
-  a.ext{color:#1f5fdb;text-decoration:none}
-  a.ext:hover{text-decoration:underline}
-  a.fact{color:inherit;text-decoration:underline dashed;text-decoration-thickness:1px;text-underline-offset:3px}
-  a.fact:hover{text-decoration-style:solid}
-
-  .hero{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin-bottom:16px}
-  .hero .card{margin:0}
-  .hero .cap{font-family:"IBM Plex Mono",monospace;font-size:10.5px;letter-spacing:.1em;color:#a4abb4;margin-bottom:10px}
-  .hero .big{font-size:15.5px;line-height:1.6;color:#16181d}
-  .verdict{border:1px solid #bfe6cf;background:#e9f8ef;border-radius:12px;padding:16px 22px;display:flex;align-items:flex-start;gap:16px;margin-bottom:20px}
-  .verdict .mark{width:34px;height:34px;border-radius:50%;background:#18a05a;color:#fff;display:flex;align-items:center;justify-content:center;font-size:18px;font-weight:700;flex:none;margin-top:2px}
-  .verdict b{color:#0f7a45}
-  .verdict p{margin:0 0 8px;font-size:14.5px}
-  table.vtab{border-collapse:collapse;width:100%}
-  table.vtab th{text-align:left;font-size:11px;font-weight:600;padding:5px 16px 5px 0;border-bottom:1px solid #cdeadb}
-  table.vtab th.win{color:#0f7a45}
-  table.vtab th.lose{color:#b23b4e}
-  table.vtab td{font-size:13px;color:#2b3038;padding:7px 16px 7px 0;border-bottom:1px solid #d9f0e3;line-height:1.5;vertical-align:top}
-  table.vtab tr:last-child td{border-bottom:0}
-  table.vtab td.crit{font-weight:600;color:#16181d;width:26%}
-  table.vtab td.lose{color:#b23b4e}
-
-  .kpis{display:grid;grid-template-columns:repeat(3,1fr);gap:16px;margin-bottom:16px}
-  .kpi{background:#fff;border:1px solid #e6e8ec;border-radius:12px;padding:18px 20px}
-  .kpi .v{font-family:"IBM Plex Mono",monospace;font-size:34px;font-weight:600;line-height:1;color:#16181d}
-  .kpi .l{font-size:12.5px;color:#5b6472;margin-top:9px;line-height:1.5}
-
-  table.duel{width:100%;border-collapse:collapse}
-  table.duel th{padding:6px 14px;text-align:left;font-size:12px;border-bottom:2px solid #eef0f2;vertical-align:bottom}
-  table.duel th .pill{display:inline-flex;padding:3px 10px;border-radius:20px;font-size:10.5px;font-weight:600;margin-top:4px}
-  table.duel th.win{color:#0f7a45}
-  table.duel th.win .pill{color:#0f7a45;background:#e7f6ee}
-  table.duel th.lose{color:#b23b4e}
-  table.duel th.lose .pill{color:#b23b4e;background:#fbe9ec}
-  table.duel td{padding:7px 14px;font-size:13px;color:#3d424b;border-bottom:1px solid #f4f5f7;line-height:1.45;vertical-align:top}
-  table.duel td.crit{font-weight:600;color:#16181d;width:26%}
-  table.duel td.crit small{display:block;font-weight:400;color:#a4abb4;margin-top:2px}
-  table.duel td .ok{color:#0f7a45;font-weight:700;margin-right:6px}
-  table.duel td .no{color:#b23b4e;font-weight:700;margin-right:6px}
-  table.duel td.more{width:12%;text-align:right;white-space:nowrap}
-
-  .panels{display:grid;grid-template-columns:1fr 1fr;gap:22px;margin-top:6px}
-  .panel h3{margin:0 0 12px;font-size:12.5px;font-weight:600;color:#5b6472;text-transform:uppercase;letter-spacing:.04em}
-  .brow{display:flex;align-items:center;gap:12px;margin-bottom:12px}
-  .brow .lab{width:118px;flex:none;font-size:12.5px;color:#3d424b}
-  .brow .trk{flex:1;background:#f1f3f6;border-radius:5px;height:20px;overflow:hidden;position:relative}
-  .brow .fill{height:20px;border-radius:0 4px 4px 0;min-width:3px}
-  .brow .inval{position:absolute;top:50%;transform:translateY(-50%);font-family:"IBM Plex Mono",monospace;font-size:12px;font-weight:600;white-space:nowrap}
-  .brow .inval.in-track{left:12px;color:#16181d}
-  .brow .inval.in-fill{right:12px;color:#fff}
-  .brow:hover .trk{background:#eaedf1}
-  .fill.pg{background:#1f5fdb}
-  .fill.rdf{background:#c0476b}
-
-  .steps{counter-reset:st;list-style:none;margin:6px 0 0;padding:0}
-  .steps li{counter-increment:st;display:flex;gap:14px;align-items:flex-start;padding:10px 0;font-size:13.5px;color:#3d424b;line-height:1.6;border-bottom:1px solid #f4f5f7}
-  .steps li:last-child{border-bottom:0}
-  .steps li::before{content:counter(st);width:26px;height:26px;border-radius:50%;background:#eaf1fe;color:#1f5fdb;font-weight:700;font-size:13px;display:flex;align-items:center;justify-content:center;flex:none;margin-top:1px}
-
-  .foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:16px;margin-top:26px;line-height:1.8}
-  @media(max-width:840px){.hero,.kpis,.panels{grid-template-columns:1fr}}
-</style>
-</head>
-<body>
-
-<header class="top">
-  <div class="logo">SE</div>
-  <div class="htitle">
-    <b>Storage Decision</b>
-    <span>Executive summary &middot; laboratory measurement data</span>
-  </div>
-  <div class="hright">
-    <a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
-    <a class="chip-go" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">Open the full study &rarr;</a>
-  </div>
-</header>
-
-<div class="page">
-
-  <h1>Which database should hold the laboratory's data?</h1>
-
-  <div class="hero">
-    <div class="card">
-      <div class="cap">THE PROBLEM</div>
-      <p class="big"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#premises" target="_blank" rel="noopener"><b>600 GB</b></a> of laboratory data today.<br/><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#premises" target="_blank" rel="noopener"><b>6 TB</b></a> within a few years.<br/>It needs one working database.</p>
-    </div>
-    <div class="card">
-      <div class="cap">THE CANDIDATES</div>
-      <p class="big"><b>Relational</b> (PostgreSQL)<br/>vs<br/><b>Graph</b> (RDF triplestore).</p>
-    </div>
-  </div>
-
-  <div class="verdict">
-    <div class="mark">&#10003;</div>
-    <div style="flex:1">
-      <p><b>Decision: Relational vs Graph (RDF):</b></p>
-      <table class="vtab">
-        <tr>
-          <th></th>
-          <th class="win">Relational (PostgreSQL)</th>
-          <th class="lose">Graph (RDF)</th>
-        </tr>
-        <tr>
-          <td class="crit">Average report, today</td>
-          <td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~6 seconds</a></td>
-          <td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~1.6 hours</a></td>
-        </tr>
-        <tr>
-          <td class="crit">Average report at 6 TB</td>
-          <td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~30 seconds</a></td>
-          <td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~12 hours</a></td>
-        </tr>
-        <tr>
-          <td class="crit">Hardware at 6 TB</td>
-          <td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one ordinary server</a></td>
-          <td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">a 14-server cluster</a></td>
-        </tr>
-        <tr>
-          <td class="crit">Ontology</td>
-          <td colspan="2">remains accessible with Relational (PostgreSQL)</td>
-        </tr>
-      </table>
-    </div>
-  </div>
-
-  <div class="kpis">
-    <div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">1,000x</a></div><div class="l">faster on the average everyday report</div></div>
-    <div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">$494k saved</a></div><div class="l">hardware at 6 TB: $33.6k for PGSQL server instead of a $527.7k RDF cluster</div></div>
-    <div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">1 vs 14</a></div><div class="l">servers needed at 6 TB</div></div>
-  </div>
-
-  <div class="card">
-    <h2>Side by side</h2>
-    <table class="duel">
-      <tr>
-        <th></th>
-        <th class="win">Relational (PostgreSQL)<br><span class="pill">&#10003; recommended</span></th>
-        <th class="lose">Graph (RDF triplestore)<br><span class="pill">&#10007; rejected at scale</span></th>
-        <th></th>
-      </tr>
-      <tr>
-        <td class="crit">Average report, today<small>at 600 GB</small></td>
-        <td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 6 seconds</a></td>
-        <td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 1.6 hours; 3 of 7 tasks need more than a day</a></td>
-        <td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#proj" target="_blank" rel="noopener">details &rarr;</a></td>
-      </tr>
-      <tr>
-        <td class="crit">Average report at 6 TB<small>the 3-year horizon</small></td>
-        <td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 30 seconds</a></td>
-        <td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 12 hours; 5 of 7 tasks need a day or more</a></td>
-        <td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#proj" target="_blank" rel="noopener">details &rarr;</a></td>
-      </tr>
-      <tr>
-        <td class="crit">Servers needed today<small>at 600 GB</small></td>
-        <td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one standard server, $10.5k</a></td>
-        <td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">already two servers with 1.4 TB of memory, $58.7k</a></td>
-        <td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#hw" target="_blank" rel="noopener">details &rarr;</a></td>
-      </tr>
-      <tr>
-        <td class="crit">Servers at 6 TB</td>
-        <td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">still one server, $33.6k</a></td>
-        <td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">a 14-server cluster, $527.7k</a></td>
-        <td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#hw" target="_blank" rel="noopener">details &rarr;</a></td>
-      </tr>
-      <tr>
-        <td class="crit">Disk for the same data<small>at 6 TB</small></td>
-        <td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#storage" target="_blank" rel="noopener">21 TB</a></td>
-        <td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#storage" target="_blank" rel="noopener">91 TB - the graph form stores the same data 4.4x bigger</a></td>
-        <td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#storage" target="_blank" rel="noopener">details &rarr;</a></td>
-      </tr>
-    </table>
-  </div>
-
-  <div class="card">
-    <h2>Average everyday report</h2>
-    <div class="panels">
-      <div class="panel">
-        <h3>Today &middot; 600 GB</h3>
-        <div class="brow" title="PostgreSQL, average of the 7 everyday tasks at 600 GB: about 6 seconds (projected)">
-          <div class="lab">PostgreSQL</div>
-          <div class="trk"><div class="fill pg" style="width:0.4%"></div><span class="inval in-track"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">6 s</a></span></div>
-        </div>
-        <div class="brow" title="RDF triplestore, average of the 7 everyday tasks at 600 GB: about 1.6 hours (projected)">
-          <div class="lab">Graph (RDF)</div>
-          <div class="trk"><div class="fill rdf" style="width:100%"></div><span class="inval in-fill"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">1.6 h</a></span></div>
-        </div>
-      </div>
-      <div class="panel">
-        <h3>In a few years &middot; 6 TB</h3>
-        <div class="brow" title="PostgreSQL, average of the 7 everyday tasks at 6 TB: about 30 seconds (projected)">
-          <div class="lab">PostgreSQL</div>
-          <div class="trk"><div class="fill pg" style="width:0.4%"></div><span class="inval in-track"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">30 s</a></span></div>
-        </div>
-        <div class="brow" title="RDF triplestore, average of the 7 everyday tasks at 6 TB: about 12 hours (projected)">
-          <div class="lab">Graph (RDF)</div>
-          <div class="trk"><div class="fill rdf" style="width:100%"></div><span class="inval in-fill"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">12 h</a></span></div>
-        </div>
-      </div>
-    </div>
-    <p class="muted" style="margin-top:14px">Bars are to scale within each panel - the blue bar is real, it is just that small. An everyday report that takes hours is a report nobody runs.</p>
-  </div>
-
-  <div class="card">
-    <h2>What about the other formats?</h2>
-    <p><b>CSV</b> stays exactly where it belongs - the raw archive coming off the instruments (best compression, byte-reproducible). <b>JSON-LD</b> stays the exchange format for sending data between laboratories. Neither is a working database, so neither was a candidate. <b>SQLite</b> is excellent for a single user <a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#notes" target="_blank" rel="noopener">up to ~600 GB</a> but does not scale to the shared 6 TB horizon.</p>
-    <p><b>And the ontology?</b> It survives intact: semantic, ontology-style access is provided as a <b>virtual layer on top of PostgreSQL</b> (e.g. Ontop OBDA) - the graph view without the graph database's cost.</p>
-  </div>
-
-  <div class="card">
-    <h2>Recommended next steps</h2>
-    <ol class="steps">
-      <li>Adopt <b>PostgreSQL as the system of record</b> for laboratory measurement data; keep the raw CSV archive as the source of truth for reprocessing.</li>
-      <li>Provide ontology and semantic queries as a <b>virtual RDF layer</b> over PostgreSQL - no data is copied into a triplestore.</li>
-      <li>Revisit hardware once volume approaches 6 TB: the measured plan is <a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one 16-core server with 551 GB RAM (~$33.6k at July 2026 prices)</a>.</li>
-    </ol>
-  </div>
-
-  <div class="foot">
-    Figures at 600 GB and 6 TB are model projections calibrated on a measured benchmark (7 everyday retrieval tasks x 5 storage formats, every result checksum-validated); the "average report" is the geometric mean of the seven tasks, the same averaging the study's scoring uses. Method, raw data and hardware price sources: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html" target="_blank" rel="noopener">facts ledger (every number traced to its artifact)</a> &middot; <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">full interactive study</a> &middot; <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.<br>
-    Study: Mary Goncharenko (Tribology Laboratory, University of Florida; internship at Sandia National Laboratories), with the technical assistance of Vasiliy Goncharenko (SoftCreator, LLC).
-  </div>
-
-</div>
-</body>
-</html>

docs/research/Tribology_Storage_Decision_Facts.html

@@ -1,131 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="utf-8">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<title>Decision Facts Ledger - Storage Decision</title>
-<link rel="preconnect" href="https://fonts.googleapis.com">
-<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
-<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
-<style>
-  *{box-sizing:border-box}
-  html,body{margin:0;padding:0}
-  body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
-  .mono{font-family:"IBM Plex Mono",monospace}
-  header.top{display:flex;align-items:center;gap:14px;height:56px;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea;position:sticky;top:0;z-index:5}
-  .logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px;flex:none}
-  .htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
-  .htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
-  .htitle span{font-size:11.5px;color:#838a95}
-  .hright{margin-left:auto;display:flex;align-items:center;gap:8px}
-  .chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px;text-decoration:none}
-  a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
-  .page{max-width:1080px;margin:0 auto;padding:28px 24px 70px}
-  h1{margin:0 0 6px;font-size:22px;font-weight:700;letter-spacing:-.02em}
-  h2{margin:26px 0 10px;font-size:15px;font-weight:600}
-  p{margin:0 0 10px;font-size:13px;color:#3d424b;line-height:1.6;max-width:940px}
-  .muted{color:#838a95}
-  a{color:#1f5fdb;text-decoration:none}
-  a:hover{text-decoration:underline}
-  .card{background:#fff;border:1px solid #e6e8ec;border-radius:11px;padding:6px 18px 10px;margin-bottom:8px;overflow-x:auto}
-  table{width:100%;border-collapse:collapse;min-width:760px}
-  th{font-size:10.5px;text-transform:uppercase;letter-spacing:.04em;color:#838a95;font-weight:600;text-align:left;padding:9px 12px;border-bottom:1px solid #eef0f2}
-  td{font-size:12.5px;color:#3d424b;padding:9px 12px;border-bottom:1px solid #f4f5f7;vertical-align:top;line-height:1.55}
-  tr:last-child td{border-bottom:0}
-  td.id{font-family:"IBM Plex Mono",monospace;font-weight:600;color:#16181d;white-space:nowrap}
-  td.val{font-family:"IBM Plex Mono",monospace;white-space:nowrap}
-  .foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:14px;margin-top:26px;line-height:1.7}
-</style>
-</head>
-<body>
-
-<header class="top">
-  <div class="logo">SE</div>
-  <div class="htitle">
-    <b>Decision Facts Ledger</b>
-    <span>Every fact on the Storage Decision page &middot; value, formula, source</span>
-  </div>
-  <div class="hright">
-    <a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">&larr; Storage Decision</a>
-    <a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">Full study</a>
-    <a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
-  </div>
-</header>
-
-<div class="page">
-
-  <h1>Decision Facts Ledger</h1>
-  <p>Every number shown on the <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision</a> executive summary, traced to the artifact that proves it. All source files are committed in the public repository and regenerate deterministically from seed 20260711. Projection tables carry one row per format x task x scale; "geometric mean" is the same averaging the study's scoring uses.</p>
-  <p class="muted">Sources: <span class="mono">t6</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6_projected_all_scales.csv</a> &middot; <span class="mono">t7</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7_hardware_all_scales.csv</a> &middot; <span class="mono">t1</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t1_storage_footprint.csv" target="_blank" rel="noopener">t1_storage_footprint.csv</a> &middot; <span class="mono">prices</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/config/hw_prices.yaml" target="_blank" rel="noopener">hw_prices.yaml</a> &middot; <span class="mono">spec</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">10_extrapolation_model.md</a></p>
-
-  <h2 id="premises">Premises</h2>
-  <div class="card">
-    <table>
-      <tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
-      <tr><td class="id">P1</td><td>"600 GB of laboratory data today"</td><td class="val">600 GB</td><td>The laboratory's current operational volume; also the first extrapolation scale of the study (scale_gb = 600).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">spec</a>, <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">P2</td><td>"6 TB within a few years"</td><td class="val">6,000 GB</td><td>The largest extrapolation scale of the study (scale_gb = 6000), the growth horizon.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">spec</a>, <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-    </table>
-  </div>
-
-  <h2 id="performance">Performance (average everyday report)</h2>
-  <div class="card">
-    <table>
-      <tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
-      <tr><td class="id">F1</td><td>PostgreSQL, average report today: "~6 seconds"</td><td class="val">5.66 s</td><td>Geometric mean of the 7 projected_wall_s values where format = PostgreSQL, scale_gb = 600.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">F2</td><td>PostgreSQL, average report at 6 TB: "~30 seconds"</td><td class="val">29.6 s</td><td>Geometric mean, format = PostgreSQL, scale_gb = 6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">F3</td><td>Graph (RDF), average report today: "~1.6 hours"</td><td class="val">5,902 s = 1.64 h</td><td>Geometric mean, format = RDF triplestore, scale_gb = 600.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">F4</td><td>Graph (RDF), average report at 6 TB: "~12 hours"</td><td class="val">43,157 s = 11.99 h</td><td>Geometric mean, format = RDF triplestore, scale_gb = 6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">F5</td><td>"3 of 7 tasks need more than a day" (RDF, today)</td><td class="val">q3 2.1 d &middot; q5 2.8 d &middot; q7 1.9 d</td><td>Count of RDF rows at scale_gb = 600 with projected_wall_s &gt; 86,400.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">F6</td><td>"5 of 7 tasks need a day or more" (RDF, 6 TB)</td><td class="val">q2, q3, q4, q5, q7</td><td>Count of RDF rows at scale_gb = 6000 with projected_wall_s &gt;= 86,400.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-    </table>
-  </div>
-
-  <h2 id="hardware">Hardware and cost</h2>
-  <div class="card">
-    <table>
-      <tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
-      <tr><td class="id">F7</td><td>"one standard server, $10.5k" (PostgreSQL, today)</td><td class="val">$10,500 &middot; 1 node</td><td>Row format = PostgreSQL, scale_gb = 600: est_cost_usd, nodes; bill of materials in the configuration column.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">F8</td><td>"already two servers with 1.4 TB of memory, $58.7k" (RDF, today)</td><td class="val">$58,740 &middot; 2 nodes &middot; 1,404.1 GB RAM</td><td>Row format = RDF triplestore, scale_gb = 600: est_cost_usd, nodes, ram_gb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">F9</td><td>"still one server, $33.6k" / "one 16-core server with 551 GB RAM" (PostgreSQL, 6 TB)</td><td class="val">$33,586 &middot; 1 node &middot; 16 cores &middot; 551.3 GB</td><td>Row format = PostgreSQL, scale_gb = 6000: est_cost_usd, nodes, cores, ram_gb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">F10</td><td>"a 14-server cluster, $527.7k" (RDF, 6 TB)</td><td class="val">$527,732 &middot; 14 nodes</td><td>Row format = RDF triplestore, scale_gb = 6000: est_cost_usd, nodes.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">F11</td><td>"July 2026 prices"</td><td class="val">as_of 2026-07-11</td><td>Street prices with per-component source URLs recorded in the operator-editable price file.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/config/hw_prices.yaml" target="_blank" rel="noopener">prices</a></td></tr>
-    </table>
-  </div>
-
-  <h2 id="storage">Storage</h2>
-  <div class="card">
-    <table>
-      <tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
-      <tr><td class="id">F12</td><td>"21 TB" disk (PostgreSQL, 6 TB)</td><td class="val">20.855 TB</td><td>Row format = PostgreSQL, scale_gb = 6000: disk_tb (includes 30% free-space headroom).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">F13</td><td>"91 TB" disk (RDF, 6 TB)</td><td class="val">91.268 TB</td><td>Row format = RDF triplestore, scale_gb = 6000: disk_tb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">F14</td><td>"stores the same data 4.4x bigger"</td><td class="val">11.70 / 2.67 = 4.38</td><td>Ratio of the measured working-footprint coefficients vs raw CSV (RDF 11.70x over PostgreSQL 2.67x).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t1_storage_footprint.csv" target="_blank" rel="noopener">t1</a></td></tr>
-    </table>
-  </div>
-
-  <h2 id="kpis">Headline figures</h2>
-  <div class="card">
-    <table>
-      <tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
-      <tr><td class="id">K1</td><td>"1,000x faster on the average everyday report"</td><td class="val">1,043x today &middot; 1,459x at 6 TB</td><td>F3 / F1 = 5,902 / 5.66; F4 / F2 = 43,157 / 29.6. Stated conservatively as 1,000x.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">K2</td><td>"$494k saved" (hardware at 6 TB)</td><td class="val">$494,146</td><td>F10 - F9 = 527,732 - 33,586.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-      <tr><td class="id">K3</td><td>"1 vs 14 servers needed at 6 TB"</td><td class="val">nodes: 1 vs 14</td><td>nodes column, rows PostgreSQL/6000 and RDF triplestore/6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
-    </table>
-  </div>
-
-  <h2 id="notes">Notes</h2>
-  <div class="card">
-    <table>
-      <tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
-      <tr><td class="id">N1</td><td>"SQLite is excellent for a single user up to ~600 GB"</td><td class="val">all 7 tasks OK at 600 GB; q5 = 1.6 h, q7 = 1.5 h at 1.2 TB</td><td>SQLite rows in t6: every flag OK at scale_gb = 600; its single-threaded full scans exceed the 1-hour bound from scale_gb = 1200.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
-      <tr><td class="id">N2</td><td>Method behind every projected number</td><td class="val">measured 35-cell benchmark</td><td>Scaling laws calibrated on the measured corpus; every measured run checksum-validated. Full methodology and measured medians in the interactive study.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">full study</a></td></tr>
-    </table>
-  </div>
-
-  <div class="foot">
-    Part of the LabDataStorageEvaluation study &middot; run of 2026-07-11, seed 20260711 &middot; canonical source: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.<br>
-    Study: Mary Goncharenko (Tribology Laboratory, University of Florida; internship at Sandia National Laboratories), with the technical assistance of Vasiliy Goncharenko (SoftCreator, LLC).
-  </div>
-
-</div>
-</body>
-</html>

docs/research/Tribology_Storage_Evaluation.html

@@ -151,7 +151,7 @@
       <!-- ================= DASHBOARD ================= -->
       <section id="dashboard" class="on">
         <h1>Dashboard</h1>
-        <p class="sub">A 141.8 MiB physically plausible CSV corpus (the canonical raw format) was converted into four other storage formats and benchmarked on seven laboratory retrieval scenarios with subprocess-isolated resource metering; results were extrapolated to 600 GB, 1.2 TB and 6 TB. Every measured run validated its rows against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9). Short on time? The one-page management summary is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision</a>.</p>
+        <p class="sub">A 141.8 MiB physically plausible CSV corpus (the canonical raw format) was converted into four other storage formats and benchmarked on seven laboratory retrieval scenarios with subprocess-isolated resource metering; results were extrapolated to 600 GB, 1.2 TB and 6 TB. Every measured run validated its rows against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9).</p>
         <div class="kpis">
           <div class="kpi"><div class="l">Benchmark cells</div><div class="v v-good">35 / 35</div><div class="s">OK; 0 invalid, 0 timeouts</div></div>
           <div class="kpi"><div class="l">Corpus</div><div class="v v-neutral">141.8 MiB</div><div class="s">8,718 CSV files, byte-reproducible</div></div>
@@ -450,7 +450,7 @@
         <div class="card">
           <h3>Data statement</h3>
           <p>The measurement corpus is simulated: physically plausible models of a Pt-Au LDRD tribology study context at Sandia National Laboratories, generated from a fixed random seed. It contains no experimental measurements and no export-controlled data. Hardware prices cited in the cost model are public street prices retrieved 2026-07-11 (sources in the Hardware &amp; Cost section).</p>
-          <p>Companion pages in the same repository: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision (one-page executive summary)</a> &middot; <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html" target="_blank" rel="noopener">Foreign-Key Architecture Study</a>.</p>
+          <p>Related study in the same repository: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html" target="_blank" rel="noopener">Foreign-Key Architecture Study</a>.</p>
         </div>
         <div class="foot">Standalone distributable file. If you received this document outside the repository, the canonical source is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.</div>
       </section>
@@ -615,10 +615,8 @@ function tabs(containerId, items, onPick){
 }
 
 /* ---------- navigation ---------- */
-/* menu items only: the sidebar footer also holds a regular external link */
+const navLinks = document.querySelectorAll("nav.side a");
-const navLinks = document.querySelectorAll('nav.side a[href^="#"]');
 function show(id){
-  if(!document.getElementById(id)) return;
   document.querySelectorAll("main section").forEach(s => s.classList.toggle("on", s.id === id));
   navLinks.forEach(a => a.classList.toggle("active", a.getAttribute("href") === "#" + id));
   el("main").scrollTop = 0;

docs/rules/code-data-formatting.md

@@ -78,6 +78,8 @@ an en-dash.
   content, not typographic substitutions. UTF-8 throughout is the right
   answer. (Repository artifacts are English; this matters only for quoted
   material.)
+- **Imported reference materials** in `docs/examples/` are kept verbatim as
+  received; they are not binding documents.
 
 ## 4. Detection / enforcement
 
@@ -87,7 +89,8 @@ future detector is documented here so it can be built consistently:
 
 - **What it flags.** Any occurrence of the Unicode codepoints in the section 1
   substitution table inside in-scope files. The section 3 exemptions
-  (scientific notation, international content) are the only carve-outs.
+  (scientific notation, international content, `docs/examples/`) are the only
+  carve-outs.
 - **Exit behavior.** Non-zero exit when any flagged codepoint is found under
   the scan roots, so it can later gate a commit hook or a build step.
 - **Tooling.** When this detector is created it MUST be a PowerShell script

docs/specs_old/00_PLAN.md

@@ -0,0 +1,27 @@
+# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
+
+Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
+Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
+Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
+
+## Tasks
+
+1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
+2. **02_fs_inventory.md** — walk source tree; produce file inventory.
+3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
+4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
+5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
+6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
+7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
+8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
+9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
+10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
+11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
+12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
+13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
+14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
+15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
+
+## Final Goal
+
+From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.

docs/specs_old/01_setup_workspace.md

@@ -0,0 +1,26 @@
+# Task 01 — Setup Workspace
+
+## Goal
+Create the working directory layout and versioned pipeline configuration.
+
+## Input
+- Path to the raw CSV root (read-only). Ask the user if unknown.
+
+## Actions
+1. Create:
+```
+work/
+	config/          # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
+	manifest/        # inventory, fingerprints, reports
+	canonical/       # cleansed partitioned dataset
+	rejects/         # quarantined anomalies
+	schema/          # DDL, data dictionary, instruments repository
+	exports/{json,sqlite,postgresql,rdf}/
+	lineage/
+	scripts/         # all Python scripts produced by tasks
+```
+2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
+3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
+
+## Intermediate Result (done when)
+- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.

docs/specs_old/02_fs_inventory.md

@@ -0,0 +1,17 @@
+# Task 02 — Filesystem Inventory
+
+## Goal
+Non-destructive inventory of every file under `SOURCE_ROOT`.
+
+## Input
+- `work/config/pipeline.json` (Task 01).
+
+## Actions
+1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
+2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
+3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
+4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
+
+## Intermediate Result (done when)
+- `inventory.jsonl` row count == filesystem file count.
+- `inventory_summary.md` produced; non-CSV extensions listed explicitly.

docs/specs_old/03_filename_tokenization.md

@@ -0,0 +1,17 @@
+# Task 03 — Filename & Path Tokenization
+
+## Goal
+Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
+
+## Input
+- `manifest/inventory.jsonl` (Task 02).
+- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
+
+## Actions
+1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
+2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
+3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
+
+## Intermediate Result (done when)
+- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
+- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).

docs/specs_old/04_schema_profiling.md

@@ -0,0 +1,18 @@
+# Task 04 — Schema Profiling & Fingerprints
+
+## Goal
+Capture the record structure of every CSV without full load; compute schema fingerprints.
+
+## Input
+- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
+
+## Actions
+1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
+2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
+3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
+4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
+5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
+
+## Intermediate Result (done when)
+- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
+- Every profile carries a fingerprint.

docs/specs_old/05_schema_grouping.md

@@ -0,0 +1,16 @@
+# Task 05 — Schema Grouping & Inconsistency Report
+
+## Goal
+Enumerate distinct record structures per measurement type; expose structural inconsistencies.
+
+## Input
+- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
+
+## Actions
+1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
+2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
+3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
+
+## Intermediate Result (done when)
+- The number of distinct schemas per measurement type is known and documented.
+- `inconsistencies.csv` complete — this is the work queue for Task 07.

docs/specs_old/06_storage_map.md

@@ -0,0 +1,19 @@
+# Task 06 — Storage Map (As-Is / As-Designed)
+
+## Goal
+Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
+
+## Input
+- All Task 02–05 artifacts.
+
+## Actions
+1. Write `manifest/storage_map.md`:
+	- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
+	- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
+	- purpose of each file kind mapped to measurement types and attachment levels;
+	- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
+2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
+
+## Intermediate Result (done when)
+- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
+- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.

docs/specs_old/07_column_mapping.md

@@ -0,0 +1,18 @@
+# Task 07 — Column Mapping (raw → canonical)
+
+## Goal
+For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
+
+## Input
+- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
+- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
+
+## Actions
+1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
+2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
+3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
+4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
+
+## Intermediate Result (done when)
+- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
+- Escalation list for Task 08 written to `schema/ontology_escalations.md`.

docs/specs_old/08_ontology_schema.md

@@ -0,0 +1,18 @@
+# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
+
+## Goal
+Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
+
+## Input
+- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
+
+## Actions
+1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
+2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
+3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
+4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
+5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
+
+## Intermediate Result (done when)
+- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
+- No open items in `ontology_escalations.md`.

docs/specs_old/09_cleanse_normalize.md

@@ -0,0 +1,18 @@
+# Task 09 — Cleanse & Normalize → Canonical Dataset
+
+## Goal
+Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
+
+## Input
+- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
+
+## Actions
+1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
+2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
+3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
+4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
+5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
+
+## Intermediate Result (done when)
+- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
+- Canonical partitions written; lineage present for every canonical row.

docs/specs_old/10_anomaly_report.md

@@ -0,0 +1,17 @@
+# Task 10 — Anomaly Report & Iteration Gate
+
+## Goal
+Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
+
+## Input
+- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
+
+## Actions
+1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
+2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
+3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
+4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
+
+## Intermediate Result (done when)
+- `anomaly_report.md` final; every reject class documented as a true anomaly.
+- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.

docs/specs_old/11_export_json.md

@@ -0,0 +1,17 @@
+# Task 11 — Export: JSON
+
+## Goal
+Two JSON products from the frozen canonical snapshot.
+
+## Input
+- `canonical/` snapshot, `schema/` (Tasks 09–10).
+
+## Actions
+1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
+2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
+3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
+4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
+
+## Intermediate Result (done when)
+- Both products written; spot-validation: point counts per measurement match canonical.
+- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).

docs/specs_old/12_export_sqlite.md

@@ -0,0 +1,16 @@
+# Task 12 — Export: SQLite
+
+## Goal
+Portable single-file SQLite database conforming to the ontology.
+
+## Input
+- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
+
+## Actions
+1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
+2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
+3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
+
+## Intermediate Result (done when)
+- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
+- `exports/sqlite/EXPORT_MANIFEST.json` written.

docs/specs_old/13_export_postgresql.md

@@ -0,0 +1,22 @@
+# Task 13 — Export: PostgreSQL
+
+## Goal
+System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
+
+## Input
+- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
+
+## Actions
+1. Generate `exports/postgresql/`:
+	- `01_schema.sql` (DDL, versioned);
+	- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
+	- per-table CSV staging files from canonical (streamed);
+	- `03_copy_load.sql` (`\copy` statements in dependency order);
+	- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
+	- `load.sh` / `load.ps1` orchestration.
+2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
+3. Include `lineage` table load (row-range granularity).
+
+## Intermediate Result (done when)
+- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
+- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.

docs/specs_old/14_export_rdf.md

@@ -0,0 +1,17 @@
+# Task 14 — Export: RDF (Metadata & Provenance Graph)
+
+## Goal
+RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
+
+## Input
+- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
+
+## Actions
+1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
+2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
+3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
+4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
+
+## Intermediate Result (done when)
+- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
+- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.

docs/specs_old/15_lineage_build_manifest.md

@@ -0,0 +1,18 @@
+# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
+
+## Goal
+Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
+
+## Input
+- All `exports/*`, `lineage/`, all config/schema versions.
+
+## Actions
+1. Write `scripts/finalize.py`:
+	- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
+	- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
+2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
+3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
+
+## Final Result (pipeline done when)
+- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
+- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.

out/.done/11.ok

@@ -1,7 +1,7 @@
 task: '11'
 status: ok
 charts: 14
-tables: 7
+tables: 5
 report_bytes: 16091
 winner_600gb: pg
 prices_as_of: 2026-07-11

out/report/tables/t6_projected_all_scales.csv

@@ -1,106 +0,0 @@
-format,query,scale_gb,projected_wall_s,projected_wall,flag
-CSV,q1,600,0.06,0.06 s,OK
-CSV,q1,1200,0.06,0.06 s,OK
-CSV,q1,6000,0.06,0.06 s,OK
-CSV,q2,600,428.301,7.1 min,OK
-CSV,q2,1200,856.543,14.3 min,OK
-CSV,q2,6000,4282.474,71.4 min,IMPRACTICAL
-CSV,q3,600,4408.0,73.5 min,IMPRACTICAL
-CSV,q3,1200,8815.94,2.4 h,IMPRACTICAL
-CSV,q3,6000,44079.46,12.2 h,IMPRACTICAL
-CSV,q4,600,2643.775,44.1 min,OK
-CSV,q4,1200,5287.489,88.1 min,IMPRACTICAL
-CSV,q4,6000,26437.206,7.3 h,IMPRACTICAL
-CSV,q5,600,4176.725,69.6 min,IMPRACTICAL
-CSV,q5,1200,8353.391,2.3 h,IMPRACTICAL
-CSV,q5,6000,41766.715,11.6 h,IMPRACTICAL
-CSV,q6,600,870.266,14.5 min,OK
-CSV,q6,1200,1740.472,29.0 min,OK
-CSV,q6,6000,8702.119,2.4 h,IMPRACTICAL
-CSV,q7,600,4897.995,81.6 min,IMPRACTICAL
-CSV,q7,1200,9795.931,2.7 h,IMPRACTICAL
-CSV,q7,6000,48979.414,13.6 h,IMPRACTICAL
-JSON-LD,q1,600,0.111,0.11 s,OK
-JSON-LD,q1,1200,0.111,0.11 s,OK
-JSON-LD,q1,6000,0.111,0.11 s,OK
-JSON-LD,q2,600,5519.622,92.0 min,IMPRACTICAL
-JSON-LD,q2,1200,11039.133,3.1 h,IMPRACTICAL
-JSON-LD,q2,6000,55195.222,15.3 h,IMPRACTICAL
-JSON-LD,q3,600,39673.993,11.0 h,IMPRACTICAL
-JSON-LD,q3,1200,79347.876,22.0 h,IMPRACTICAL
-JSON-LD,q3,6000,396738.933,4.6 d,FAIL
-JSON-LD,q4,600,39597.306,11.0 h,IMPRACTICAL
-JSON-LD,q4,1200,79194.5,22.0 h,IMPRACTICAL
-JSON-LD,q4,6000,395972.054,4.6 d,FAIL
-JSON-LD,q5,600,39760.772,11.0 h,IMPRACTICAL
-JSON-LD,q5,1200,79521.432,22.1 h,IMPRACTICAL
-JSON-LD,q5,6000,397606.717,4.6 d,FAIL
-JSON-LD,q6,600,39076.231,10.9 h,IMPRACTICAL
-JSON-LD,q6,1200,78152.351,21.7 h,IMPRACTICAL
-JSON-LD,q6,6000,390761.312,4.5 d,FAIL
-JSON-LD,q7,600,40228.164,11.2 h,IMPRACTICAL
-JSON-LD,q7,1200,80456.218,22.3 h,IMPRACTICAL
-JSON-LD,q7,6000,402280.643,4.7 d,FAIL
-SQLite,q1,600,0.062,0.06 s,OK
-SQLite,q1,1200,0.063,0.06 s,OK
-SQLite,q1,6000,0.063,0.06 s,OK
-SQLite,q2,600,1.271,1.27 s,OK
-SQLite,q2,1200,2.482,2.48 s,OK
-SQLite,q2,6000,12.169,12 s,OK
-SQLite,q3,600,339.238,5.7 min,OK
-SQLite,q3,1200,699.328,11.7 min,OK
-SQLite,q3,6000,3739.195,62.3 min,IMPRACTICAL
-SQLite,q4,600,0.06,0.06 s,OK
-SQLite,q4,1200,0.06,0.06 s,OK
-SQLite,q4,6000,0.06,0.06 s,OK
-SQLite,q5,600,2847.603,47.5 min,OK
-SQLite,q5,1200,5695.146,94.9 min,IMPRACTICAL
-SQLite,q5,6000,28475.49,7.9 h,IMPRACTICAL
-SQLite,q6,600,5.18,5.18 s,OK
-SQLite,q6,1200,10.615,11 s,OK
-SQLite,q6,6000,56.5,56 s,OK
-SQLite,q7,600,2619.558,43.7 min,OK
-SQLite,q7,1200,5239.055,87.3 min,IMPRACTICAL
-SQLite,q7,6000,26195.034,7.3 h,IMPRACTICAL
-PostgreSQL,q1,600,0.222,0.22 s,OK
-PostgreSQL,q1,1200,0.222,0.22 s,OK
-PostgreSQL,q1,6000,0.222,0.22 s,OK
-PostgreSQL,q2,600,0.221,0.22 s,OK
-PostgreSQL,q2,1200,0.221,0.22 s,OK
-PostgreSQL,q2,6000,0.221,0.22 s,OK
-PostgreSQL,q3,600,3.42,3.42 s,OK
-PostgreSQL,q3,1200,6.817,6.82 s,OK
-PostgreSQL,q3,6000,35.495,35 s,OK
-PostgreSQL,q4,600,10.715,11 s,OK
-PostgreSQL,q4,1200,21.209,21 s,OK
-PostgreSQL,q4,6000,105.162,105 s,OK
-PostgreSQL,q5,600,222.111,3.7 min,OK
-PostgreSQL,q5,1200,444.001,7.4 min,OK
-PostgreSQL,q5,6000,2219.125,37.0 min,OK
-PostgreSQL,q6,600,4.06,4.06 s,OK
-PostgreSQL,q6,1200,8.137,8.14 s,OK
-PostgreSQL,q6,6000,42.55,43 s,OK
-PostgreSQL,q7,600,114.849,115 s,OK
-PostgreSQL,q7,1200,229.477,3.8 min,OK
-PostgreSQL,q7,6000,1146.503,19.1 min,OK
-RDF triplestore,q1,600,0.113,0.11 s,OK
-RDF triplestore,q1,1200,0.113,0.11 s,OK
-RDF triplestore,q1,6000,0.113,0.11 s,OK
-RDF triplestore,q2,600,10866.789,3.0 h,IMPRACTICAL
-RDF triplestore,q2,1200,21733.465,6.0 h,IMPRACTICAL
-RDF triplestore,q2,6000,108666.874,30.2 h,FAIL
-RDF triplestore,q3,600,185722.881,2.1 d,FAIL
-RDF triplestore,q3,1200,371445.649,4.3 d,FAIL
-RDF triplestore,q3,6000,1857227.792,21.5 d,FAIL
-RDF triplestore,q4,600,79559.375,22.1 h,IMPRACTICAL
-RDF triplestore,q4,1200,159118.636,44.2 h,FAIL
-RDF triplestore,q4,6000,795592.727,9.2 d,FAIL
-RDF triplestore,q5,600,240054.647,2.8 d,FAIL
-RDF triplestore,q5,1200,480109.181,5.6 d,FAIL
-RDF triplestore,q5,6000,2400545.453,27.8 d,FAIL
-RDF triplestore,q6,600,352.729,5.9 min,OK
-RDF triplestore,q6,1200,727.088,12.1 min,OK
-RDF triplestore,q6,6000,3887.402,64.8 min,IMPRACTICAL
-RDF triplestore,q7,600,163225.475,45.3 h,FAIL
-RDF triplestore,q7,1200,326450.837,3.8 d,FAIL
-RDF triplestore,q7,6000,1632253.733,18.9 d,FAIL

out/report/tables/t7_hardware_all_scales.csv

@@ -1,16 +0,0 @@
-format,scale_gb,configuration,disk_tb,ram_gb,cores,nodes,ram_modules,nvme_drives,base_usd,ram_usd,disk_usd,extra_nodes_usd,est_cost_usd
-CSV,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,0.78,16.0,16,1,1,1,4618,1887,3995,0,10500
-CSV,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.56,16.0,16,1,1,1,4618,1887,3995,0,10500
-CSV,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,7.8,16.0,16,1,1,2,4618,1887,7990,0,14495
-JSON-LD,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.633,16.0,16,1,1,1,4618,1887,3995,0,10500
-JSON-LD,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,3.267,16.0,16,1,1,1,4618,1887,3995,0,10500
-JSON-LD,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,16.334,16.0,16,1,1,3,4618,1887,11985,0,18490
-SQLite,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,0.647,21.0,16,1,1,1,4618,1887,3995,0,10500
-SQLite,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.295,38.0,16,1,1,1,4618,1887,3995,0,10500
-SQLite,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 3 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,6.475,173.9,16,1,3,1,4618,5661,3995,0,14274
-PostgreSQL,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,2.086,58.7,16,1,1,1,4618,1887,3995,0,10500
-PostgreSQL,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 2 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,4.171,113.5,16,1,2,1,4618,3774,3995,0,12387
-PostgreSQL,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 9 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,20.855,551.3,16,1,9,3,4618,16983,11985,0,33586
-RDF triplestore,600,2 x 1U Supermicro AS-1015CS-TNR (16 cores); 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,9.127,1404.1,16,2,22,2,4618,41514,7990,4618,58740
-RDF triplestore,1200,3 x 1U Supermicro AS-1015CS-TNR (16 cores); 44 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,18.254,2808.3,16,3,44,3,4618,83028,11985,9236,108867
-RDF triplestore,6000,14 x 1U Supermicro AS-1015CS-TNR (16 cores); 220 x 64 GB DDR5 ECC RDIMM; 12 x 7.68 TB NVMe U.2,91.268,14041.3,16,14,220,12,4618,415140,47940,60034,527732

report.py

@@ -3,9 +3,8 @@
 Aggregates the measured matrix (task 09), the projections (task 10) and the
 process diagrams (task 02) into the final decision document. Writes 14 PNG
 charts (150 dpi) under ./out/report/charts/ with the exact names fixed in
-README.md, seven CSV tables under ./out/report/tables/ (t6/t7 carry every
+README.md, five CSV tables under ./out/report/tables/, and
-projection and hardware row for all three scales - they are the committed
+./out/report/REPORT.md.
-proof artifacts the executive summary links to), and ./out/report/REPORT.md.
 
 Chart readability contract (owner feedback 2026-07-11): linear scales with
 actual units everywhere (panels instead of log axes when magnitudes clash),
@@ -637,22 +636,6 @@ def main() -> int:
 				money(sizing[(fmt, REPORT_SCALE_GB)]["est_cost_usd"])] for fmt in FORMATS],
 		)
 
-		tables["t6_projected_all_scales"] = (
-			["format", "query", "scale_gb", "projected_wall_s", "projected_wall", "flag"],
-			[[FMT_NAMES[fmt], f"q{qn}", s, proj[(fmt, qn, s)][0], human_time(proj[(fmt, qn, s)][0]), proj[(fmt, qn, s)][1]]
-				for fmt in FORMATS for qn in QUERY_NUMBERS for s in SCALES_GB],
-		)
-		tables["t7_hardware_all_scales"] = (
-			["format", "scale_gb", "configuration", "disk_tb", "ram_gb", "cores", "nodes",
-				"ram_modules", "nvme_drives", "base_usd", "ram_usd", "disk_usd", "extra_nodes_usd", "est_cost_usd"],
-			[[FMT_NAMES[fmt], s, config_string(sizing[(fmt, s)], prices),
-				sizing[(fmt, s)]["disk_tb"], sizing[(fmt, s)]["ram_gb"],
-				sizing[(fmt, s)]["cores"], sizing[(fmt, s)]["nodes"],
-				sizing[(fmt, s)]["ram_modules"], sizing[(fmt, s)]["nvme_drives"],
-				round(sizing[(fmt, s)]["base_usd"]), round(sizing[(fmt, s)]["ram_usd"]),
-				round(sizing[(fmt, s)]["disk_usd"]), round(sizing[(fmt, s)]["extra_nodes_usd"]),
-				round(sizing[(fmt, s)]["est_cost_usd"])] for fmt in FORMATS for s in SCALES_GB],
-		)
 		scoreboards = build_scoreboards(medians, proj, sizes, sizing)
 		tables["t5_weighted_scores"] = (
 			["scale", "format", "search_pts", "ram_pts", "disk_pts", "total"],