Compare commits

..

4 Commits

Author SHA1 Message Date
administrator
6d156f98c4 feat(docs): add management summary link in evaluation documents
- include link to one-page executive summary in README
- update dashboard section in evaluation HTML to reference summary
- revise report metadata to reflect additional CSV tables generated
2026-07-13 00:19:02 -04:00
administrator
478d4f3201 fix(docs): update navigation links in Tribology Storage Evaluation 2026-07-12 12:52:30 -04:00
administrator
2796b79170 chore(docs): remove obsolete example files from documentation
- delete Tribology_Ontology.html and TribologyData_MasterPlan_v1.md examples
- remove old planning documents from specs_old directory
2026-07-12 11:33:41 -04:00
administrator
ff68679000 feat(docs): enhance README with interactive result links and references
- add links to interactive HTML results for storage format evaluation and foreign-key architecture study
- update descriptions in README to reflect the new structure of research documents
- clarify the contents of the research directory in the README
2026-07-12 11:30:43 -04:00
28 changed files with 557 additions and 750 deletions

View File

@@ -12,8 +12,26 @@ the measurements to 600 GB - 6 TB, and produces a ranked decision report for
five laboratory use cases: analysis, reporting, search/filtering, archiving, five laboratory use cases: analysis, reporting, search/filtering, archiving,
and inter-lab exchange. and inter-lab exchange.
**Results - interactive, single-file HTML (open in a browser or download and share):**
- **[Storage Decision](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html)** -
one-page executive summary: the problem, the verdict (relational vs graph
database), and what it costs - written for management, every claim linked
to the full study
(source: [docs/research/Tribology_Storage_Decision.html](docs/research/Tribology_Storage_Decision.html)).
- **[Storage Format Evaluation](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html)** -
the full study results: dashboard, measured matrix, projections to 6 TB,
hardware bill of materials, weighted scoring and verdict
(source: [docs/research/Tribology_Storage_Evaluation.html](docs/research/Tribology_Storage_Evaluation.html)).
- **[Foreign-Key Architecture Study](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html)** -
the measured key-schema study behind the relational design decision
(source: [docs/research/Tribology_FK_Architecture.html](docs/research/Tribology_FK_Architecture.html)).
Reference documents:
- Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11). - Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11).
- Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md). - Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md).
- Full decision report (markdown): [out/report/REPORT.md](out/report/REPORT.md).
## Goals ## Goals
@@ -144,8 +162,7 @@ docs/
Initial_Prompt.md the generative prompt behind specs 00-11 (provenance, non-binding) Initial_Prompt.md the generative prompt behind specs 00-11 (provenance, non-binding)
specs/ task specifications 00-11 (WHAT to build) specs/ task specifications 00-11 (WHAT to build)
rules/ binding conventions (HOW work is done) rules/ binding conventions (HOW work is done)
research/ measured design studies (e.g. FK key-schema study) research/ measured design studies and interactive result pages
examples/ imported reference materials (not binding)
out/ ALL generated artifacts (reproducible; bulk git-ignored, out/ ALL generated artifacts (reproducible; bulk git-ignored,
report/ config/ .done/ committed - see .gitignore): report/ config/ .done/ committed - see .gitignore):
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/ config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/

View File

@@ -1,248 +0,0 @@
# Tribology Lab Data — Master Plan
**Discovery · Cleansing/Normalization · Ontology · Conversion**
Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
Status: Plan v1 — Draft
---
## 0. Objective
Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
---
## 1. Guiding principles
1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 12. Migration to JSX Viewer / better-sqlite3 is deferred.
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
---
## 2. Phase overview
| Phase | Name | Nature | Primary output |
|-------|------|--------|----------------|
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
---
## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
### 3.1 Inputs
- Root of the ~700 GB CSV tree (read-only).
### 3.2 Activities
**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
- atmosphere tokens (`ambient`, `dryNitrogen`, …)
Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
**C. Structural profiling (header-only + sampled body).** Per file, without full load:
- read header row → column names, count, order;
- infer per-column datatype from first/last N rows + a random sample;
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
```
for each file in walk(root):
meta = fs_metadata(file)
tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere
header = read_first_line(file)
columns = parse_columns(header)
dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M))
fp = fingerprint(columns) # ordered-set hash
emit_manifest_row(meta, tokens, columns, dtypes, fp)
```
### 3.4 Artifacts
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
### 3.5 Exit criteria
- 100% of files inventoried and fingerprinted.
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
---
## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
### 4.1 The iteration loop (per measurement type)
1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
### 4.2 Instruments Repository (ontology sub-model)
Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
- identity (type, model/label, measurement type it produces);
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
- association to `measurement_types` / `column_definitions` in the data dictionary.
Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
### 4.3 Anomaly detection
Criteria are **derived from the data**, not fixed a priori. Starting catalog:
| Class | Description | Source of criterion |
|-------|-------------|---------------------|
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
| Missing required | Null in a non-nullable canonical field | Schema constraints |
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
### 4.4 Data dictionary / metadata catalog
Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
### 4.5 Artifacts
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
- **`instruments_repository`** — instrument entities + characteristics + ranges.
- **`rejects_store`** — quarantined anomalies with reason codes.
- **`ontology_notes`** — running record of ERD changes and rationale.
### 4.6 Exit criteria
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
---
## 5. Phase 4 — Conversion & Export
**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
### 5.1 Targets
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
5. **RDF / triplestore****metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
### 5.2 Design
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
- Lineage is emitted alongside every export (see §6).
### 5.3 Artifacts
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
### 5.4 Exit criteria
- All five target formats generated from one canonical snapshot.
- Round-trip validation: record counts and key aggregates match across formats.
- Lineage complete and resolvable for sampled records.
---
## 6. Cross-cutting concerns
### 6.1 Lineage & processing-flow documentation
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
- Lineage is a first-class export, present in every target format.
### 6.2 Configuration & rule registry
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
### 6.3 Validation gates
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
---
## 7. Target artifacts catalog (consolidated)
| Artifact | Phase | Form |
|----------|-------|------|
| storage_manifest | 1 | JSONL + SQLite |
| schema_fingerprint_report | 1 | JSON/CSV |
| unmatched_report | 1 | CSV |
| storage_map | 1 | Markdown |
| column_mapping | 2↔3 | CSV/JSON (versioned) |
| canonical_dataset | 2↔3 | partitioned intermediate |
| relational_schema (DDL) | 2↔3 | SQL |
| data_dictionary | 2↔3 | table/CSV |
| instruments_repository | 2↔3 | table/CSV |
| rejects_store | 2↔3 | JSONL |
| ontology_notes | 2↔3 | Markdown |
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
| lineage + build manifest | all | JSONL |
---
## 8. Tooling & environment
- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
- **No DB engine in Phases 12** beyond SQLite for the manifest index.
- **PostgreSQL** materialized in Phase 4 (system of record).
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
- Python indentation: **tabs**.
---
## 9. Performance strategy for 700 GB
- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
---
## 10. Open items & risks
1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
---
*End of Plan v1.*

File diff suppressed because one or more lines are too long

View File

@@ -533,7 +533,7 @@ CREATE TABLE friction_cycles (
<div class="foot"> <div class="foot">
Measured 2026-07-11 on the deterministic simulated corpus (seed 20260711, 141.8 MiB CSV) &middot; SQLite, warm OS cache, best of 3 runs &middot; Measured 2026-07-11 on the deterministic simulated corpus (seed 20260711, 141.8 MiB CSV) &middot; SQLite, warm OS cache, best of 3 runs &middot;
scripts: <span class="mono">measure_key_variants.py</span> (build + size + load + Q1) and <span class="mono">read_benchmarks.py</span> (six read patterns) &middot; scripts: <span class="mono">measure_key_variants.py</span> (build + size + load + Q1) and <span class="mono">read_benchmarks.py</span> (six read patterns) &middot;
companion page: <span class="mono">Tribology_Ontology.html</span> (measurement-result registry). companion page: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" style="color:#1f5fdb;text-decoration:none">Storage Format Evaluation</a> (full study results).
</div> </div>
</section> </section>

View File

@@ -0,0 +1,258 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Storage Decision - Executive Summary</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
<style>
*{box-sizing:border-box}
html,body{margin:0;padding:0}
body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
.mono{font-family:"IBM Plex Mono",monospace}
header.top{display:flex;align-items:center;gap:14px;height:56px;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea;position:sticky;top:0;z-index:5}
.logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px;flex:none}
.htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
.htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
.htitle span{font-size:11.5px;color:#838a95}
.hright{margin-left:auto;display:flex;align-items:center;gap:8px}
.chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px;text-decoration:none}
a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
.chip-go{display:inline-flex;align-items:center;gap:6px;font-size:12px;color:#fff;background:#1f5fdb;padding:5px 12px;border-radius:6px;font-weight:600;text-decoration:none}
.chip-go:hover{background:#1a4fb8}
.page{max-width:980px;margin:0 auto;padding:30px 24px 70px}
.card{background:#fff;border:1px solid #e6e8ec;border-radius:12px;padding:22px 26px;margin-bottom:16px}
h1{margin:0 0 6px;font-size:24px;font-weight:700;letter-spacing:-.02em}
h2{margin:0 0 12px;font-size:16px;font-weight:600;letter-spacing:-.01em}
p{margin:0 0 10px;font-size:13.5px;color:#3d424b;line-height:1.65}
p:last-child{margin-bottom:0}
.muted{color:#838a95}
a.ext{color:#1f5fdb;text-decoration:none}
a.ext:hover{text-decoration:underline}
a.fact{color:inherit;text-decoration:underline dashed;text-decoration-thickness:1px;text-underline-offset:3px}
a.fact:hover{text-decoration-style:solid}
.hero{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin-bottom:16px}
.hero .card{margin:0}
.hero .cap{font-family:"IBM Plex Mono",monospace;font-size:10.5px;letter-spacing:.1em;color:#a4abb4;margin-bottom:10px}
.hero .big{font-size:15.5px;line-height:1.6;color:#16181d}
.verdict{border:1px solid #bfe6cf;background:#e9f8ef;border-radius:12px;padding:16px 22px;display:flex;align-items:flex-start;gap:16px;margin-bottom:20px}
.verdict .mark{width:34px;height:34px;border-radius:50%;background:#18a05a;color:#fff;display:flex;align-items:center;justify-content:center;font-size:18px;font-weight:700;flex:none;margin-top:2px}
.verdict b{color:#0f7a45}
.verdict p{margin:0 0 8px;font-size:14.5px}
table.vtab{border-collapse:collapse;width:100%}
table.vtab th{text-align:left;font-size:11px;font-weight:600;padding:5px 16px 5px 0;border-bottom:1px solid #cdeadb}
table.vtab th.win{color:#0f7a45}
table.vtab th.lose{color:#b23b4e}
table.vtab td{font-size:13px;color:#2b3038;padding:7px 16px 7px 0;border-bottom:1px solid #d9f0e3;line-height:1.5;vertical-align:top}
table.vtab tr:last-child td{border-bottom:0}
table.vtab td.crit{font-weight:600;color:#16181d;width:26%}
table.vtab td.lose{color:#b23b4e}
.kpis{display:grid;grid-template-columns:repeat(3,1fr);gap:16px;margin-bottom:16px}
.kpi{background:#fff;border:1px solid #e6e8ec;border-radius:12px;padding:18px 20px}
.kpi .v{font-family:"IBM Plex Mono",monospace;font-size:34px;font-weight:600;line-height:1;color:#16181d}
.kpi .l{font-size:12.5px;color:#5b6472;margin-top:9px;line-height:1.5}
table.duel{width:100%;border-collapse:collapse}
table.duel th{padding:6px 14px;text-align:left;font-size:12px;border-bottom:2px solid #eef0f2;vertical-align:bottom}
table.duel th .pill{display:inline-flex;padding:3px 10px;border-radius:20px;font-size:10.5px;font-weight:600;margin-top:4px}
table.duel th.win{color:#0f7a45}
table.duel th.win .pill{color:#0f7a45;background:#e7f6ee}
table.duel th.lose{color:#b23b4e}
table.duel th.lose .pill{color:#b23b4e;background:#fbe9ec}
table.duel td{padding:7px 14px;font-size:13px;color:#3d424b;border-bottom:1px solid #f4f5f7;line-height:1.45;vertical-align:top}
table.duel td.crit{font-weight:600;color:#16181d;width:26%}
table.duel td.crit small{display:block;font-weight:400;color:#a4abb4;margin-top:2px}
table.duel td .ok{color:#0f7a45;font-weight:700;margin-right:6px}
table.duel td .no{color:#b23b4e;font-weight:700;margin-right:6px}
table.duel td.more{width:12%;text-align:right;white-space:nowrap}
.panels{display:grid;grid-template-columns:1fr 1fr;gap:22px;margin-top:6px}
.panel h3{margin:0 0 12px;font-size:12.5px;font-weight:600;color:#5b6472;text-transform:uppercase;letter-spacing:.04em}
.brow{display:flex;align-items:center;gap:12px;margin-bottom:12px}
.brow .lab{width:118px;flex:none;font-size:12.5px;color:#3d424b}
.brow .trk{flex:1;background:#f1f3f6;border-radius:5px;height:20px;overflow:hidden;position:relative}
.brow .fill{height:20px;border-radius:0 4px 4px 0;min-width:3px}
.brow .inval{position:absolute;top:50%;transform:translateY(-50%);font-family:"IBM Plex Mono",monospace;font-size:12px;font-weight:600;white-space:nowrap}
.brow .inval.in-track{left:12px;color:#16181d}
.brow .inval.in-fill{right:12px;color:#fff}
.brow:hover .trk{background:#eaedf1}
.fill.pg{background:#1f5fdb}
.fill.rdf{background:#c0476b}
.steps{counter-reset:st;list-style:none;margin:6px 0 0;padding:0}
.steps li{counter-increment:st;display:flex;gap:14px;align-items:flex-start;padding:10px 0;font-size:13.5px;color:#3d424b;line-height:1.6;border-bottom:1px solid #f4f5f7}
.steps li:last-child{border-bottom:0}
.steps li::before{content:counter(st);width:26px;height:26px;border-radius:50%;background:#eaf1fe;color:#1f5fdb;font-weight:700;font-size:13px;display:flex;align-items:center;justify-content:center;flex:none;margin-top:1px}
.foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:16px;margin-top:26px;line-height:1.8}
@media(max-width:840px){.hero,.kpis,.panels{grid-template-columns:1fr}}
</style>
</head>
<body>
<header class="top">
<div class="logo">SE</div>
<div class="htitle">
<b>Storage Decision</b>
<span>Executive summary &middot; laboratory measurement data</span>
</div>
<div class="hright">
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
<a class="chip-go" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">Open the full study &rarr;</a>
</div>
</header>
<div class="page">
<h1>Which database should hold the laboratory's data?</h1>
<div class="hero">
<div class="card">
<div class="cap">THE PROBLEM</div>
<p class="big"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#premises" target="_blank" rel="noopener"><b>600 GB</b></a> of laboratory data today.<br/><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#premises" target="_blank" rel="noopener"><b>6 TB</b></a> within a few years.<br/>It needs one working database.</p>
</div>
<div class="card">
<div class="cap">THE CANDIDATES</div>
<p class="big"><b>Relational</b> (PostgreSQL)<br/>vs<br/><b>Graph</b> (RDF triplestore).</p>
</div>
</div>
<div class="verdict">
<div class="mark">&#10003;</div>
<div style="flex:1">
<p><b>Decision: Relational vs Graph (RDF):</b></p>
<table class="vtab">
<tr>
<th></th>
<th class="win">Relational (PostgreSQL)</th>
<th class="lose">Graph (RDF)</th>
</tr>
<tr>
<td class="crit">Average report, today</td>
<td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~6 seconds</a></td>
<td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~1.6 hours</a></td>
</tr>
<tr>
<td class="crit">Average report at 6 TB</td>
<td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~30 seconds</a></td>
<td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~12 hours</a></td>
</tr>
<tr>
<td class="crit">Hardware at 6 TB</td>
<td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one ordinary server</a></td>
<td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">a 14-server cluster</a></td>
</tr>
<tr>
<td class="crit">Ontology</td>
<td colspan="2">remains accessible with Relational (PostgreSQL)</td>
</tr>
</table>
</div>
</div>
<div class="kpis">
<div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">1,000x</a></div><div class="l">faster on the average everyday report</div></div>
<div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">$494k saved</a></div><div class="l">hardware at 6 TB: $33.6k for PGSQL server instead of a $527.7k RDF cluster</div></div>
<div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">1 vs 14</a></div><div class="l">servers needed at 6 TB</div></div>
</div>
<div class="card">
<h2>Side by side</h2>
<table class="duel">
<tr>
<th></th>
<th class="win">Relational (PostgreSQL)<br><span class="pill">&#10003; recommended</span></th>
<th class="lose">Graph (RDF triplestore)<br><span class="pill">&#10007; rejected at scale</span></th>
<th></th>
</tr>
<tr>
<td class="crit">Average report, today<small>at 600 GB</small></td>
<td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 6 seconds</a></td>
<td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 1.6 hours; 3 of 7 tasks need more than a day</a></td>
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#proj" target="_blank" rel="noopener">details &rarr;</a></td>
</tr>
<tr>
<td class="crit">Average report at 6 TB<small>the 3-year horizon</small></td>
<td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 30 seconds</a></td>
<td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 12 hours; 5 of 7 tasks need a day or more</a></td>
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#proj" target="_blank" rel="noopener">details &rarr;</a></td>
</tr>
<tr>
<td class="crit">Servers needed today<small>at 600 GB</small></td>
<td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one standard server, $10.5k</a></td>
<td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">already two servers with 1.4 TB of memory, $58.7k</a></td>
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#hw" target="_blank" rel="noopener">details &rarr;</a></td>
</tr>
<tr>
<td class="crit">Servers at 6 TB</td>
<td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">still one server, $33.6k</a></td>
<td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">a 14-server cluster, $527.7k</a></td>
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#hw" target="_blank" rel="noopener">details &rarr;</a></td>
</tr>
<tr>
<td class="crit">Disk for the same data<small>at 6 TB</small></td>
<td><span class="ok">&#10003;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#storage" target="_blank" rel="noopener">21 TB</a></td>
<td><span class="no">&#10007;</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#storage" target="_blank" rel="noopener">91 TB - the graph form stores the same data 4.4x bigger</a></td>
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#storage" target="_blank" rel="noopener">details &rarr;</a></td>
</tr>
</table>
</div>
<div class="card">
<h2>Average everyday report</h2>
<div class="panels">
<div class="panel">
<h3>Today &middot; 600 GB</h3>
<div class="brow" title="PostgreSQL, average of the 7 everyday tasks at 600 GB: about 6 seconds (projected)">
<div class="lab">PostgreSQL</div>
<div class="trk"><div class="fill pg" style="width:0.4%"></div><span class="inval in-track"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">6 s</a></span></div>
</div>
<div class="brow" title="RDF triplestore, average of the 7 everyday tasks at 600 GB: about 1.6 hours (projected)">
<div class="lab">Graph (RDF)</div>
<div class="trk"><div class="fill rdf" style="width:100%"></div><span class="inval in-fill"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">1.6 h</a></span></div>
</div>
</div>
<div class="panel">
<h3>In a few years &middot; 6 TB</h3>
<div class="brow" title="PostgreSQL, average of the 7 everyday tasks at 6 TB: about 30 seconds (projected)">
<div class="lab">PostgreSQL</div>
<div class="trk"><div class="fill pg" style="width:0.4%"></div><span class="inval in-track"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">30 s</a></span></div>
</div>
<div class="brow" title="RDF triplestore, average of the 7 everyday tasks at 6 TB: about 12 hours (projected)">
<div class="lab">Graph (RDF)</div>
<div class="trk"><div class="fill rdf" style="width:100%"></div><span class="inval in-fill"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">12 h</a></span></div>
</div>
</div>
</div>
<p class="muted" style="margin-top:14px">Bars are to scale within each panel - the blue bar is real, it is just that small. An everyday report that takes hours is a report nobody runs.</p>
</div>
<div class="card">
<h2>What about the other formats?</h2>
<p><b>CSV</b> stays exactly where it belongs - the raw archive coming off the instruments (best compression, byte-reproducible). <b>JSON-LD</b> stays the exchange format for sending data between laboratories. Neither is a working database, so neither was a candidate. <b>SQLite</b> is excellent for a single user <a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#notes" target="_blank" rel="noopener">up to ~600 GB</a> but does not scale to the shared 6 TB horizon.</p>
<p><b>And the ontology?</b> It survives intact: semantic, ontology-style access is provided as a <b>virtual layer on top of PostgreSQL</b> (e.g. Ontop OBDA) - the graph view without the graph database's cost.</p>
</div>
<div class="card">
<h2>Recommended next steps</h2>
<ol class="steps">
<li>Adopt <b>PostgreSQL as the system of record</b> for laboratory measurement data; keep the raw CSV archive as the source of truth for reprocessing.</li>
<li>Provide ontology and semantic queries as a <b>virtual RDF layer</b> over PostgreSQL - no data is copied into a triplestore.</li>
<li>Revisit hardware once volume approaches 6 TB: the measured plan is <a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one 16-core server with 551 GB RAM (~$33.6k at July 2026 prices)</a>.</li>
</ol>
</div>
<div class="foot">
Figures at 600 GB and 6 TB are model projections calibrated on a measured benchmark (7 everyday retrieval tasks x 5 storage formats, every result checksum-validated); the "average report" is the geometric mean of the seven tasks, the same averaging the study's scoring uses. Method, raw data and hardware price sources: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html" target="_blank" rel="noopener">facts ledger (every number traced to its artifact)</a> &middot; <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">full interactive study</a> &middot; <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.<br>
Study: Mary Goncharenko (Tribology Laboratory, University of Florida; internship at Sandia National Laboratories), with the technical assistance of Vasiliy Goncharenko (SoftCreator, LLC).
</div>
</div>
</body>
</html>

View File

@@ -0,0 +1,131 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Decision Facts Ledger - Storage Decision</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
<style>
*{box-sizing:border-box}
html,body{margin:0;padding:0}
body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
.mono{font-family:"IBM Plex Mono",monospace}
header.top{display:flex;align-items:center;gap:14px;height:56px;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea;position:sticky;top:0;z-index:5}
.logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px;flex:none}
.htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
.htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
.htitle span{font-size:11.5px;color:#838a95}
.hright{margin-left:auto;display:flex;align-items:center;gap:8px}
.chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px;text-decoration:none}
a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
.page{max-width:1080px;margin:0 auto;padding:28px 24px 70px}
h1{margin:0 0 6px;font-size:22px;font-weight:700;letter-spacing:-.02em}
h2{margin:26px 0 10px;font-size:15px;font-weight:600}
p{margin:0 0 10px;font-size:13px;color:#3d424b;line-height:1.6;max-width:940px}
.muted{color:#838a95}
a{color:#1f5fdb;text-decoration:none}
a:hover{text-decoration:underline}
.card{background:#fff;border:1px solid #e6e8ec;border-radius:11px;padding:6px 18px 10px;margin-bottom:8px;overflow-x:auto}
table{width:100%;border-collapse:collapse;min-width:760px}
th{font-size:10.5px;text-transform:uppercase;letter-spacing:.04em;color:#838a95;font-weight:600;text-align:left;padding:9px 12px;border-bottom:1px solid #eef0f2}
td{font-size:12.5px;color:#3d424b;padding:9px 12px;border-bottom:1px solid #f4f5f7;vertical-align:top;line-height:1.55}
tr:last-child td{border-bottom:0}
td.id{font-family:"IBM Plex Mono",monospace;font-weight:600;color:#16181d;white-space:nowrap}
td.val{font-family:"IBM Plex Mono",monospace;white-space:nowrap}
.foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:14px;margin-top:26px;line-height:1.7}
</style>
</head>
<body>
<header class="top">
<div class="logo">SE</div>
<div class="htitle">
<b>Decision Facts Ledger</b>
<span>Every fact on the Storage Decision page &middot; value, formula, source</span>
</div>
<div class="hright">
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">&larr; Storage Decision</a>
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">Full study</a>
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
</div>
</header>
<div class="page">
<h1>Decision Facts Ledger</h1>
<p>Every number shown on the <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision</a> executive summary, traced to the artifact that proves it. All source files are committed in the public repository and regenerate deterministically from seed 20260711. Projection tables carry one row per format x task x scale; "geometric mean" is the same averaging the study's scoring uses.</p>
<p class="muted">Sources: <span class="mono">t6</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6_projected_all_scales.csv</a> &middot; <span class="mono">t7</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7_hardware_all_scales.csv</a> &middot; <span class="mono">t1</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t1_storage_footprint.csv" target="_blank" rel="noopener">t1_storage_footprint.csv</a> &middot; <span class="mono">prices</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/config/hw_prices.yaml" target="_blank" rel="noopener">hw_prices.yaml</a> &middot; <span class="mono">spec</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">10_extrapolation_model.md</a></p>
<h2 id="premises">Premises</h2>
<div class="card">
<table>
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
<tr><td class="id">P1</td><td>"600 GB of laboratory data today"</td><td class="val">600 GB</td><td>The laboratory's current operational volume; also the first extrapolation scale of the study (scale_gb = 600).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">spec</a>, <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">P2</td><td>"6 TB within a few years"</td><td class="val">6,000 GB</td><td>The largest extrapolation scale of the study (scale_gb = 6000), the growth horizon.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">spec</a>, <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
</table>
</div>
<h2 id="performance">Performance (average everyday report)</h2>
<div class="card">
<table>
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
<tr><td class="id">F1</td><td>PostgreSQL, average report today: "~6 seconds"</td><td class="val">5.66 s</td><td>Geometric mean of the 7 projected_wall_s values where format = PostgreSQL, scale_gb = 600.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">F2</td><td>PostgreSQL, average report at 6 TB: "~30 seconds"</td><td class="val">29.6 s</td><td>Geometric mean, format = PostgreSQL, scale_gb = 6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">F3</td><td>Graph (RDF), average report today: "~1.6 hours"</td><td class="val">5,902 s = 1.64 h</td><td>Geometric mean, format = RDF triplestore, scale_gb = 600.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">F4</td><td>Graph (RDF), average report at 6 TB: "~12 hours"</td><td class="val">43,157 s = 11.99 h</td><td>Geometric mean, format = RDF triplestore, scale_gb = 6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">F5</td><td>"3 of 7 tasks need more than a day" (RDF, today)</td><td class="val">q3 2.1 d &middot; q5 2.8 d &middot; q7 1.9 d</td><td>Count of RDF rows at scale_gb = 600 with projected_wall_s &gt; 86,400.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">F6</td><td>"5 of 7 tasks need a day or more" (RDF, 6 TB)</td><td class="val">q2, q3, q4, q5, q7</td><td>Count of RDF rows at scale_gb = 6000 with projected_wall_s &gt;= 86,400.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
</table>
</div>
<h2 id="hardware">Hardware and cost</h2>
<div class="card">
<table>
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
<tr><td class="id">F7</td><td>"one standard server, $10.5k" (PostgreSQL, today)</td><td class="val">$10,500 &middot; 1 node</td><td>Row format = PostgreSQL, scale_gb = 600: est_cost_usd, nodes; bill of materials in the configuration column.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">F8</td><td>"already two servers with 1.4 TB of memory, $58.7k" (RDF, today)</td><td class="val">$58,740 &middot; 2 nodes &middot; 1,404.1 GB RAM</td><td>Row format = RDF triplestore, scale_gb = 600: est_cost_usd, nodes, ram_gb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">F9</td><td>"still one server, $33.6k" / "one 16-core server with 551 GB RAM" (PostgreSQL, 6 TB)</td><td class="val">$33,586 &middot; 1 node &middot; 16 cores &middot; 551.3 GB</td><td>Row format = PostgreSQL, scale_gb = 6000: est_cost_usd, nodes, cores, ram_gb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">F10</td><td>"a 14-server cluster, $527.7k" (RDF, 6 TB)</td><td class="val">$527,732 &middot; 14 nodes</td><td>Row format = RDF triplestore, scale_gb = 6000: est_cost_usd, nodes.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">F11</td><td>"July 2026 prices"</td><td class="val">as_of 2026-07-11</td><td>Street prices with per-component source URLs recorded in the operator-editable price file.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/config/hw_prices.yaml" target="_blank" rel="noopener">prices</a></td></tr>
</table>
</div>
<h2 id="storage">Storage</h2>
<div class="card">
<table>
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
<tr><td class="id">F12</td><td>"21 TB" disk (PostgreSQL, 6 TB)</td><td class="val">20.855 TB</td><td>Row format = PostgreSQL, scale_gb = 6000: disk_tb (includes 30% free-space headroom).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">F13</td><td>"91 TB" disk (RDF, 6 TB)</td><td class="val">91.268 TB</td><td>Row format = RDF triplestore, scale_gb = 6000: disk_tb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">F14</td><td>"stores the same data 4.4x bigger"</td><td class="val">11.70 / 2.67 = 4.38</td><td>Ratio of the measured working-footprint coefficients vs raw CSV (RDF 11.70x over PostgreSQL 2.67x).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t1_storage_footprint.csv" target="_blank" rel="noopener">t1</a></td></tr>
</table>
</div>
<h2 id="kpis">Headline figures</h2>
<div class="card">
<table>
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
<tr><td class="id">K1</td><td>"1,000x faster on the average everyday report"</td><td class="val">1,043x today &middot; 1,459x at 6 TB</td><td>F3 / F1 = 5,902 / 5.66; F4 / F2 = 43,157 / 29.6. Stated conservatively as 1,000x.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">K2</td><td>"$494k saved" (hardware at 6 TB)</td><td class="val">$494,146</td><td>F10 - F9 = 527,732 - 33,586.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
<tr><td class="id">K3</td><td>"1 vs 14 servers needed at 6 TB"</td><td class="val">nodes: 1 vs 14</td><td>nodes column, rows PostgreSQL/6000 and RDF triplestore/6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
</table>
</div>
<h2 id="notes">Notes</h2>
<div class="card">
<table>
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
<tr><td class="id">N1</td><td>"SQLite is excellent for a single user up to ~600 GB"</td><td class="val">all 7 tasks OK at 600 GB; q5 = 1.6 h, q7 = 1.5 h at 1.2 TB</td><td>SQLite rows in t6: every flag OK at scale_gb = 600; its single-threaded full scans exceed the 1-hour bound from scale_gb = 1200.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
<tr><td class="id">N2</td><td>Method behind every projected number</td><td class="val">measured 35-cell benchmark</td><td>Scaling laws calibrated on the measured corpus; every measured run checksum-validated. Full methodology and measured medians in the interactive study.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">full study</a></td></tr>
</table>
</div>
<div class="foot">
Part of the LabDataStorageEvaluation study &middot; run of 2026-07-11, seed 20260711 &middot; canonical source: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.<br>
Study: Mary Goncharenko (Tribology Laboratory, University of Florida; internship at Sandia National Laboratories), with the technical assistance of Vasiliy Goncharenko (SoftCreator, LLC).
</div>
</div>
</body>
</html>

View File

@@ -151,7 +151,7 @@
<!-- ================= DASHBOARD ================= --> <!-- ================= DASHBOARD ================= -->
<section id="dashboard" class="on"> <section id="dashboard" class="on">
<h1>Dashboard</h1> <h1>Dashboard</h1>
<p class="sub">A 141.8 MiB physically plausible CSV corpus (the canonical raw format) was converted into four other storage formats and benchmarked on seven laboratory retrieval scenarios with subprocess-isolated resource metering; results were extrapolated to 600 GB, 1.2 TB and 6 TB. Every measured run validated its rows against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9).</p> <p class="sub">A 141.8 MiB physically plausible CSV corpus (the canonical raw format) was converted into four other storage formats and benchmarked on seven laboratory retrieval scenarios with subprocess-isolated resource metering; results were extrapolated to 600 GB, 1.2 TB and 6 TB. Every measured run validated its rows against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9). Short on time? The one-page management summary is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision</a>.</p>
<div class="kpis"> <div class="kpis">
<div class="kpi"><div class="l">Benchmark cells</div><div class="v v-good">35 / 35</div><div class="s">OK; 0 invalid, 0 timeouts</div></div> <div class="kpi"><div class="l">Benchmark cells</div><div class="v v-good">35 / 35</div><div class="s">OK; 0 invalid, 0 timeouts</div></div>
<div class="kpi"><div class="l">Corpus</div><div class="v v-neutral">141.8 MiB</div><div class="s">8,718 CSV files, byte-reproducible</div></div> <div class="kpi"><div class="l">Corpus</div><div class="v v-neutral">141.8 MiB</div><div class="s">8,718 CSV files, byte-reproducible</div></div>
@@ -450,7 +450,7 @@
<div class="card"> <div class="card">
<h3>Data statement</h3> <h3>Data statement</h3>
<p>The measurement corpus is simulated: physically plausible models of a Pt-Au LDRD tribology study context at Sandia National Laboratories, generated from a fixed random seed. It contains no experimental measurements and no export-controlled data. Hardware prices cited in the cost model are public street prices retrieved 2026-07-11 (sources in the Hardware &amp; Cost section).</p> <p>The measurement corpus is simulated: physically plausible models of a Pt-Au LDRD tribology study context at Sandia National Laboratories, generated from a fixed random seed. It contains no experimental measurements and no export-controlled data. Hardware prices cited in the cost model are public street prices retrieved 2026-07-11 (sources in the Hardware &amp; Cost section).</p>
<p>Related study in the same repository: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html" target="_blank" rel="noopener">Foreign-Key Architecture Study</a>.</p> <p>Companion pages in the same repository: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision (one-page executive summary)</a> &middot; <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html" target="_blank" rel="noopener">Foreign-Key Architecture Study</a>.</p>
</div> </div>
<div class="foot">Standalone distributable file. If you received this document outside the repository, the canonical source is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.</div> <div class="foot">Standalone distributable file. If you received this document outside the repository, the canonical source is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.</div>
</section> </section>
@@ -615,8 +615,10 @@ function tabs(containerId, items, onPick){
} }
/* ---------- navigation ---------- */ /* ---------- navigation ---------- */
const navLinks = document.querySelectorAll("nav.side a"); /* menu items only: the sidebar footer also holds a regular external link */
const navLinks = document.querySelectorAll('nav.side a[href^="#"]');
function show(id){ function show(id){
if(!document.getElementById(id)) return;
document.querySelectorAll("main section").forEach(s => s.classList.toggle("on", s.id === id)); document.querySelectorAll("main section").forEach(s => s.classList.toggle("on", s.id === id));
navLinks.forEach(a => a.classList.toggle("active", a.getAttribute("href") === "#" + id)); navLinks.forEach(a => a.classList.toggle("active", a.getAttribute("href") === "#" + id));
el("main").scrollTop = 0; el("main").scrollTop = 0;

View File

@@ -78,8 +78,6 @@ an en-dash.
content, not typographic substitutions. UTF-8 throughout is the right content, not typographic substitutions. UTF-8 throughout is the right
answer. (Repository artifacts are English; this matters only for quoted answer. (Repository artifacts are English; this matters only for quoted
material.) material.)
- **Imported reference materials** in `docs/examples/` are kept verbatim as
received; they are not binding documents.
## 4. Detection / enforcement ## 4. Detection / enforcement
@@ -89,8 +87,7 @@ future detector is documented here so it can be built consistently:
- **What it flags.** Any occurrence of the Unicode codepoints in the section 1 - **What it flags.** Any occurrence of the Unicode codepoints in the section 1
substitution table inside in-scope files. The section 3 exemptions substitution table inside in-scope files. The section 3 exemptions
(scientific notation, international content, `docs/examples/`) are the only (scientific notation, international content) are the only carve-outs.
carve-outs.
- **Exit behavior.** Non-zero exit when any flagged codepoint is found under - **Exit behavior.** Non-zero exit when any flagged codepoint is found under
the scan roots, so it can later gate a commit hook or a build step. the scan roots, so it can later gate a commit hook or a build step.
- **Tooling.** When this detector is created it MUST be a PowerShell script - **Tooling.** When this detector is created it MUST be a PowerShell script

View File

@@ -1,27 +0,0 @@
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
## Tasks
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 0709 until coverage = 100%.
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
## Final Goal
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.

View File

@@ -1,26 +0,0 @@
# Task 01 — Setup Workspace
## Goal
Create the working directory layout and versioned pipeline configuration.
## Input
- Path to the raw CSV root (read-only). Ask the user if unknown.
## Actions
1. Create:
```
work/
config/ # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
manifest/ # inventory, fingerprints, reports
canonical/ # cleansed partitioned dataset
rejects/ # quarantined anomalies
schema/ # DDL, data dictionary, instruments repository
exports/{json,sqlite,postgresql,rdf}/
lineage/
scripts/ # all Python scripts produced by tasks
```
2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
## Intermediate Result (done when)
- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.

View File

@@ -1,17 +0,0 @@
# Task 02 — Filesystem Inventory
## Goal
Non-destructive inventory of every file under `SOURCE_ROOT`.
## Input
- `work/config/pipeline.json` (Task 01).
## Actions
1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
## Intermediate Result (done when)
- `inventory.jsonl` row count == filesystem file count.
- `inventory_summary.md` produced; non-CSV extensions listed explicitly.

View File

@@ -1,17 +0,0 @@
# Task 03 — Filename & Path Tokenization
## Goal
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
## Input
- `manifest/inventory.jsonl` (Task 02).
- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
## Actions
1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
## Intermediate Result (done when)
- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).

View File

@@ -1,18 +0,0 @@
# Task 04 — Schema Profiling & Fingerprints
## Goal
Capture the record structure of every CSV without full load; compute schema fingerprints.
## Input
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
## Actions
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
## Intermediate Result (done when)
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
- Every profile carries a fingerprint.

View File

@@ -1,16 +0,0 @@
# Task 05 — Schema Grouping & Inconsistency Report
## Goal
Enumerate distinct record structures per measurement type; expose structural inconsistencies.
## Input
- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
## Actions
1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
## Intermediate Result (done when)
- The number of distinct schemas per measurement type is known and documented.
- `inconsistencies.csv` complete — this is the work queue for Task 07.

View File

@@ -1,19 +0,0 @@
# Task 06 — Storage Map (As-Is / As-Designed)
## Goal
Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
## Input
- All Task 0205 artifacts.
## Actions
1. Write `manifest/storage_map.md`:
- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
- purpose of each file kind mapped to measurement types and attachment levels;
- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
## Intermediate Result (done when)
- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.

View File

@@ -1,18 +0,0 @@
# Task 07 — Column Mapping (raw → canonical)
## Goal
For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
## Input
- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
## Actions
1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
## Intermediate Result (done when)
- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
- Escalation list for Task 08 written to `schema/ontology_escalations.md`.

View File

@@ -1,18 +0,0 @@
# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
## Goal
Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
## Input
- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
## Actions
1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
## Intermediate Result (done when)
- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
- No open items in `ontology_escalations.md`.

View File

@@ -1,18 +0,0 @@
# Task 09 — Cleanse & Normalize → Canonical Dataset
## Goal
Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
## Input
- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
## Actions
1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
## Intermediate Result (done when)
- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
- Canonical partitions written; lineage present for every canonical row.

View File

@@ -1,17 +0,0 @@
# Task 10 — Anomaly Report & Iteration Gate
## Goal
Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
## Input
- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
## Actions
1. Write `scripts/anomaly_report.py``manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
## Intermediate Result (done when)
- `anomaly_report.md` final; every reject class documented as a true anomaly.
- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.

View File

@@ -1,17 +0,0 @@
# Task 11 — Export: JSON
## Goal
Two JSON products from the frozen canonical snapshot.
## Input
- `canonical/` snapshot, `schema/` (Tasks 0910).
## Actions
1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
## Intermediate Result (done when)
- Both products written; spot-validation: point counts per measurement match canonical.
- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).

View File

@@ -1,16 +0,0 @@
# Task 12 — Export: SQLite
## Goal
Portable single-file SQLite database conforming to the ontology.
## Input
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
## Actions
1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
## Intermediate Result (done when)
- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
- `exports/sqlite/EXPORT_MANIFEST.json` written.

View File

@@ -1,22 +0,0 @@
# Task 13 — Export: PostgreSQL
## Goal
System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
## Input
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
## Actions
1. Generate `exports/postgresql/`:
- `01_schema.sql` (DDL, versioned);
- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
- per-table CSV staging files from canonical (streamed);
- `03_copy_load.sql` (`\copy` statements in dependency order);
- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
- `load.sh` / `load.ps1` orchestration.
2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
3. Include `lineage` table load (row-range granularity).
## Intermediate Result (done when)
- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.

View File

@@ -1,17 +0,0 @@
# Task 14 — Export: RDF (Metadata & Provenance Graph)
## Goal
RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
## Input
- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
## Actions
1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
2. Write `scripts/export_rdf.py``exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
## Intermediate Result (done when)
- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.

View File

@@ -1,18 +0,0 @@
# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
## Goal
Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
## Input
- All `exports/*`, `lineage/`, all config/schema versions.
## Actions
1. Write `scripts/finalize.py`:
- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
## Final Result (pipeline done when)
- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.

View File

@@ -1,7 +1,7 @@
task: '11' task: '11'
status: ok status: ok
charts: 14 charts: 14
tables: 5 tables: 7
report_bytes: 16091 report_bytes: 16091
winner_600gb: pg winner_600gb: pg
prices_as_of: 2026-07-11 prices_as_of: 2026-07-11

View File

@@ -0,0 +1,106 @@
format,query,scale_gb,projected_wall_s,projected_wall,flag
CSV,q1,600,0.06,0.06 s,OK
CSV,q1,1200,0.06,0.06 s,OK
CSV,q1,6000,0.06,0.06 s,OK
CSV,q2,600,428.301,7.1 min,OK
CSV,q2,1200,856.543,14.3 min,OK
CSV,q2,6000,4282.474,71.4 min,IMPRACTICAL
CSV,q3,600,4408.0,73.5 min,IMPRACTICAL
CSV,q3,1200,8815.94,2.4 h,IMPRACTICAL
CSV,q3,6000,44079.46,12.2 h,IMPRACTICAL
CSV,q4,600,2643.775,44.1 min,OK
CSV,q4,1200,5287.489,88.1 min,IMPRACTICAL
CSV,q4,6000,26437.206,7.3 h,IMPRACTICAL
CSV,q5,600,4176.725,69.6 min,IMPRACTICAL
CSV,q5,1200,8353.391,2.3 h,IMPRACTICAL
CSV,q5,6000,41766.715,11.6 h,IMPRACTICAL
CSV,q6,600,870.266,14.5 min,OK
CSV,q6,1200,1740.472,29.0 min,OK
CSV,q6,6000,8702.119,2.4 h,IMPRACTICAL
CSV,q7,600,4897.995,81.6 min,IMPRACTICAL
CSV,q7,1200,9795.931,2.7 h,IMPRACTICAL
CSV,q7,6000,48979.414,13.6 h,IMPRACTICAL
JSON-LD,q1,600,0.111,0.11 s,OK
JSON-LD,q1,1200,0.111,0.11 s,OK
JSON-LD,q1,6000,0.111,0.11 s,OK
JSON-LD,q2,600,5519.622,92.0 min,IMPRACTICAL
JSON-LD,q2,1200,11039.133,3.1 h,IMPRACTICAL
JSON-LD,q2,6000,55195.222,15.3 h,IMPRACTICAL
JSON-LD,q3,600,39673.993,11.0 h,IMPRACTICAL
JSON-LD,q3,1200,79347.876,22.0 h,IMPRACTICAL
JSON-LD,q3,6000,396738.933,4.6 d,FAIL
JSON-LD,q4,600,39597.306,11.0 h,IMPRACTICAL
JSON-LD,q4,1200,79194.5,22.0 h,IMPRACTICAL
JSON-LD,q4,6000,395972.054,4.6 d,FAIL
JSON-LD,q5,600,39760.772,11.0 h,IMPRACTICAL
JSON-LD,q5,1200,79521.432,22.1 h,IMPRACTICAL
JSON-LD,q5,6000,397606.717,4.6 d,FAIL
JSON-LD,q6,600,39076.231,10.9 h,IMPRACTICAL
JSON-LD,q6,1200,78152.351,21.7 h,IMPRACTICAL
JSON-LD,q6,6000,390761.312,4.5 d,FAIL
JSON-LD,q7,600,40228.164,11.2 h,IMPRACTICAL
JSON-LD,q7,1200,80456.218,22.3 h,IMPRACTICAL
JSON-LD,q7,6000,402280.643,4.7 d,FAIL
SQLite,q1,600,0.062,0.06 s,OK
SQLite,q1,1200,0.063,0.06 s,OK
SQLite,q1,6000,0.063,0.06 s,OK
SQLite,q2,600,1.271,1.27 s,OK
SQLite,q2,1200,2.482,2.48 s,OK
SQLite,q2,6000,12.169,12 s,OK
SQLite,q3,600,339.238,5.7 min,OK
SQLite,q3,1200,699.328,11.7 min,OK
SQLite,q3,6000,3739.195,62.3 min,IMPRACTICAL
SQLite,q4,600,0.06,0.06 s,OK
SQLite,q4,1200,0.06,0.06 s,OK
SQLite,q4,6000,0.06,0.06 s,OK
SQLite,q5,600,2847.603,47.5 min,OK
SQLite,q5,1200,5695.146,94.9 min,IMPRACTICAL
SQLite,q5,6000,28475.49,7.9 h,IMPRACTICAL
SQLite,q6,600,5.18,5.18 s,OK
SQLite,q6,1200,10.615,11 s,OK
SQLite,q6,6000,56.5,56 s,OK
SQLite,q7,600,2619.558,43.7 min,OK
SQLite,q7,1200,5239.055,87.3 min,IMPRACTICAL
SQLite,q7,6000,26195.034,7.3 h,IMPRACTICAL
PostgreSQL,q1,600,0.222,0.22 s,OK
PostgreSQL,q1,1200,0.222,0.22 s,OK
PostgreSQL,q1,6000,0.222,0.22 s,OK
PostgreSQL,q2,600,0.221,0.22 s,OK
PostgreSQL,q2,1200,0.221,0.22 s,OK
PostgreSQL,q2,6000,0.221,0.22 s,OK
PostgreSQL,q3,600,3.42,3.42 s,OK
PostgreSQL,q3,1200,6.817,6.82 s,OK
PostgreSQL,q3,6000,35.495,35 s,OK
PostgreSQL,q4,600,10.715,11 s,OK
PostgreSQL,q4,1200,21.209,21 s,OK
PostgreSQL,q4,6000,105.162,105 s,OK
PostgreSQL,q5,600,222.111,3.7 min,OK
PostgreSQL,q5,1200,444.001,7.4 min,OK
PostgreSQL,q5,6000,2219.125,37.0 min,OK
PostgreSQL,q6,600,4.06,4.06 s,OK
PostgreSQL,q6,1200,8.137,8.14 s,OK
PostgreSQL,q6,6000,42.55,43 s,OK
PostgreSQL,q7,600,114.849,115 s,OK
PostgreSQL,q7,1200,229.477,3.8 min,OK
PostgreSQL,q7,6000,1146.503,19.1 min,OK
RDF triplestore,q1,600,0.113,0.11 s,OK
RDF triplestore,q1,1200,0.113,0.11 s,OK
RDF triplestore,q1,6000,0.113,0.11 s,OK
RDF triplestore,q2,600,10866.789,3.0 h,IMPRACTICAL
RDF triplestore,q2,1200,21733.465,6.0 h,IMPRACTICAL
RDF triplestore,q2,6000,108666.874,30.2 h,FAIL
RDF triplestore,q3,600,185722.881,2.1 d,FAIL
RDF triplestore,q3,1200,371445.649,4.3 d,FAIL
RDF triplestore,q3,6000,1857227.792,21.5 d,FAIL
RDF triplestore,q4,600,79559.375,22.1 h,IMPRACTICAL
RDF triplestore,q4,1200,159118.636,44.2 h,FAIL
RDF triplestore,q4,6000,795592.727,9.2 d,FAIL
RDF triplestore,q5,600,240054.647,2.8 d,FAIL
RDF triplestore,q5,1200,480109.181,5.6 d,FAIL
RDF triplestore,q5,6000,2400545.453,27.8 d,FAIL
RDF triplestore,q6,600,352.729,5.9 min,OK
RDF triplestore,q6,1200,727.088,12.1 min,OK
RDF triplestore,q6,6000,3887.402,64.8 min,IMPRACTICAL
RDF triplestore,q7,600,163225.475,45.3 h,FAIL
RDF triplestore,q7,1200,326450.837,3.8 d,FAIL
RDF triplestore,q7,6000,1632253.733,18.9 d,FAIL
1 format query scale_gb projected_wall_s projected_wall flag
2 CSV q1 600 0.06 0.06 s OK
3 CSV q1 1200 0.06 0.06 s OK
4 CSV q1 6000 0.06 0.06 s OK
5 CSV q2 600 428.301 7.1 min OK
6 CSV q2 1200 856.543 14.3 min OK
7 CSV q2 6000 4282.474 71.4 min IMPRACTICAL
8 CSV q3 600 4408.0 73.5 min IMPRACTICAL
9 CSV q3 1200 8815.94 2.4 h IMPRACTICAL
10 CSV q3 6000 44079.46 12.2 h IMPRACTICAL
11 CSV q4 600 2643.775 44.1 min OK
12 CSV q4 1200 5287.489 88.1 min IMPRACTICAL
13 CSV q4 6000 26437.206 7.3 h IMPRACTICAL
14 CSV q5 600 4176.725 69.6 min IMPRACTICAL
15 CSV q5 1200 8353.391 2.3 h IMPRACTICAL
16 CSV q5 6000 41766.715 11.6 h IMPRACTICAL
17 CSV q6 600 870.266 14.5 min OK
18 CSV q6 1200 1740.472 29.0 min OK
19 CSV q6 6000 8702.119 2.4 h IMPRACTICAL
20 CSV q7 600 4897.995 81.6 min IMPRACTICAL
21 CSV q7 1200 9795.931 2.7 h IMPRACTICAL
22 CSV q7 6000 48979.414 13.6 h IMPRACTICAL
23 JSON-LD q1 600 0.111 0.11 s OK
24 JSON-LD q1 1200 0.111 0.11 s OK
25 JSON-LD q1 6000 0.111 0.11 s OK
26 JSON-LD q2 600 5519.622 92.0 min IMPRACTICAL
27 JSON-LD q2 1200 11039.133 3.1 h IMPRACTICAL
28 JSON-LD q2 6000 55195.222 15.3 h IMPRACTICAL
29 JSON-LD q3 600 39673.993 11.0 h IMPRACTICAL
30 JSON-LD q3 1200 79347.876 22.0 h IMPRACTICAL
31 JSON-LD q3 6000 396738.933 4.6 d FAIL
32 JSON-LD q4 600 39597.306 11.0 h IMPRACTICAL
33 JSON-LD q4 1200 79194.5 22.0 h IMPRACTICAL
34 JSON-LD q4 6000 395972.054 4.6 d FAIL
35 JSON-LD q5 600 39760.772 11.0 h IMPRACTICAL
36 JSON-LD q5 1200 79521.432 22.1 h IMPRACTICAL
37 JSON-LD q5 6000 397606.717 4.6 d FAIL
38 JSON-LD q6 600 39076.231 10.9 h IMPRACTICAL
39 JSON-LD q6 1200 78152.351 21.7 h IMPRACTICAL
40 JSON-LD q6 6000 390761.312 4.5 d FAIL
41 JSON-LD q7 600 40228.164 11.2 h IMPRACTICAL
42 JSON-LD q7 1200 80456.218 22.3 h IMPRACTICAL
43 JSON-LD q7 6000 402280.643 4.7 d FAIL
44 SQLite q1 600 0.062 0.06 s OK
45 SQLite q1 1200 0.063 0.06 s OK
46 SQLite q1 6000 0.063 0.06 s OK
47 SQLite q2 600 1.271 1.27 s OK
48 SQLite q2 1200 2.482 2.48 s OK
49 SQLite q2 6000 12.169 12 s OK
50 SQLite q3 600 339.238 5.7 min OK
51 SQLite q3 1200 699.328 11.7 min OK
52 SQLite q3 6000 3739.195 62.3 min IMPRACTICAL
53 SQLite q4 600 0.06 0.06 s OK
54 SQLite q4 1200 0.06 0.06 s OK
55 SQLite q4 6000 0.06 0.06 s OK
56 SQLite q5 600 2847.603 47.5 min OK
57 SQLite q5 1200 5695.146 94.9 min IMPRACTICAL
58 SQLite q5 6000 28475.49 7.9 h IMPRACTICAL
59 SQLite q6 600 5.18 5.18 s OK
60 SQLite q6 1200 10.615 11 s OK
61 SQLite q6 6000 56.5 56 s OK
62 SQLite q7 600 2619.558 43.7 min OK
63 SQLite q7 1200 5239.055 87.3 min IMPRACTICAL
64 SQLite q7 6000 26195.034 7.3 h IMPRACTICAL
65 PostgreSQL q1 600 0.222 0.22 s OK
66 PostgreSQL q1 1200 0.222 0.22 s OK
67 PostgreSQL q1 6000 0.222 0.22 s OK
68 PostgreSQL q2 600 0.221 0.22 s OK
69 PostgreSQL q2 1200 0.221 0.22 s OK
70 PostgreSQL q2 6000 0.221 0.22 s OK
71 PostgreSQL q3 600 3.42 3.42 s OK
72 PostgreSQL q3 1200 6.817 6.82 s OK
73 PostgreSQL q3 6000 35.495 35 s OK
74 PostgreSQL q4 600 10.715 11 s OK
75 PostgreSQL q4 1200 21.209 21 s OK
76 PostgreSQL q4 6000 105.162 105 s OK
77 PostgreSQL q5 600 222.111 3.7 min OK
78 PostgreSQL q5 1200 444.001 7.4 min OK
79 PostgreSQL q5 6000 2219.125 37.0 min OK
80 PostgreSQL q6 600 4.06 4.06 s OK
81 PostgreSQL q6 1200 8.137 8.14 s OK
82 PostgreSQL q6 6000 42.55 43 s OK
83 PostgreSQL q7 600 114.849 115 s OK
84 PostgreSQL q7 1200 229.477 3.8 min OK
85 PostgreSQL q7 6000 1146.503 19.1 min OK
86 RDF triplestore q1 600 0.113 0.11 s OK
87 RDF triplestore q1 1200 0.113 0.11 s OK
88 RDF triplestore q1 6000 0.113 0.11 s OK
89 RDF triplestore q2 600 10866.789 3.0 h IMPRACTICAL
90 RDF triplestore q2 1200 21733.465 6.0 h IMPRACTICAL
91 RDF triplestore q2 6000 108666.874 30.2 h FAIL
92 RDF triplestore q3 600 185722.881 2.1 d FAIL
93 RDF triplestore q3 1200 371445.649 4.3 d FAIL
94 RDF triplestore q3 6000 1857227.792 21.5 d FAIL
95 RDF triplestore q4 600 79559.375 22.1 h IMPRACTICAL
96 RDF triplestore q4 1200 159118.636 44.2 h FAIL
97 RDF triplestore q4 6000 795592.727 9.2 d FAIL
98 RDF triplestore q5 600 240054.647 2.8 d FAIL
99 RDF triplestore q5 1200 480109.181 5.6 d FAIL
100 RDF triplestore q5 6000 2400545.453 27.8 d FAIL
101 RDF triplestore q6 600 352.729 5.9 min OK
102 RDF triplestore q6 1200 727.088 12.1 min OK
103 RDF triplestore q6 6000 3887.402 64.8 min IMPRACTICAL
104 RDF triplestore q7 600 163225.475 45.3 h FAIL
105 RDF triplestore q7 1200 326450.837 3.8 d FAIL
106 RDF triplestore q7 6000 1632253.733 18.9 d FAIL

View File

@@ -0,0 +1,16 @@
format,scale_gb,configuration,disk_tb,ram_gb,cores,nodes,ram_modules,nvme_drives,base_usd,ram_usd,disk_usd,extra_nodes_usd,est_cost_usd
CSV,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,0.78,16.0,16,1,1,1,4618,1887,3995,0,10500
CSV,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.56,16.0,16,1,1,1,4618,1887,3995,0,10500
CSV,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,7.8,16.0,16,1,1,2,4618,1887,7990,0,14495
JSON-LD,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.633,16.0,16,1,1,1,4618,1887,3995,0,10500
JSON-LD,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,3.267,16.0,16,1,1,1,4618,1887,3995,0,10500
JSON-LD,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,16.334,16.0,16,1,1,3,4618,1887,11985,0,18490
SQLite,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,0.647,21.0,16,1,1,1,4618,1887,3995,0,10500
SQLite,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.295,38.0,16,1,1,1,4618,1887,3995,0,10500
SQLite,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 3 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,6.475,173.9,16,1,3,1,4618,5661,3995,0,14274
PostgreSQL,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,2.086,58.7,16,1,1,1,4618,1887,3995,0,10500
PostgreSQL,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 2 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,4.171,113.5,16,1,2,1,4618,3774,3995,0,12387
PostgreSQL,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 9 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,20.855,551.3,16,1,9,3,4618,16983,11985,0,33586
RDF triplestore,600,2 x 1U Supermicro AS-1015CS-TNR (16 cores); 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,9.127,1404.1,16,2,22,2,4618,41514,7990,4618,58740
RDF triplestore,1200,3 x 1U Supermicro AS-1015CS-TNR (16 cores); 44 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,18.254,2808.3,16,3,44,3,4618,83028,11985,9236,108867
RDF triplestore,6000,14 x 1U Supermicro AS-1015CS-TNR (16 cores); 220 x 64 GB DDR5 ECC RDIMM; 12 x 7.68 TB NVMe U.2,91.268,14041.3,16,14,220,12,4618,415140,47940,60034,527732
1 format scale_gb configuration disk_tb ram_gb cores nodes ram_modules nvme_drives base_usd ram_usd disk_usd extra_nodes_usd est_cost_usd
2 CSV 600 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 0.78 16.0 16 1 1 1 4618 1887 3995 0 10500
3 CSV 1200 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 1.56 16.0 16 1 1 1 4618 1887 3995 0 10500
4 CSV 6000 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2 7.8 16.0 16 1 1 2 4618 1887 7990 0 14495
5 JSON-LD 600 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 1.633 16.0 16 1 1 1 4618 1887 3995 0 10500
6 JSON-LD 1200 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 3.267 16.0 16 1 1 1 4618 1887 3995 0 10500
7 JSON-LD 6000 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2 16.334 16.0 16 1 1 3 4618 1887 11985 0 18490
8 SQLite 600 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 0.647 21.0 16 1 1 1 4618 1887 3995 0 10500
9 SQLite 1200 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 1.295 38.0 16 1 1 1 4618 1887 3995 0 10500
10 SQLite 6000 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 3 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 6.475 173.9 16 1 3 1 4618 5661 3995 0 14274
11 PostgreSQL 600 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 2.086 58.7 16 1 1 1 4618 1887 3995 0 10500
12 PostgreSQL 1200 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 2 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 4.171 113.5 16 1 2 1 4618 3774 3995 0 12387
13 PostgreSQL 6000 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 9 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2 20.855 551.3 16 1 9 3 4618 16983 11985 0 33586
14 RDF triplestore 600 2 x 1U Supermicro AS-1015CS-TNR (16 cores); 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2 9.127 1404.1 16 2 22 2 4618 41514 7990 4618 58740
15 RDF triplestore 1200 3 x 1U Supermicro AS-1015CS-TNR (16 cores); 44 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2 18.254 2808.3 16 3 44 3 4618 83028 11985 9236 108867
16 RDF triplestore 6000 14 x 1U Supermicro AS-1015CS-TNR (16 cores); 220 x 64 GB DDR5 ECC RDIMM; 12 x 7.68 TB NVMe U.2 91.268 14041.3 16 14 220 12 4618 415140 47940 60034 527732

View File

@@ -3,8 +3,9 @@
Aggregates the measured matrix (task 09), the projections (task 10) and the Aggregates the measured matrix (task 09), the projections (task 10) and the
process diagrams (task 02) into the final decision document. Writes 14 PNG process diagrams (task 02) into the final decision document. Writes 14 PNG
charts (150 dpi) under ./out/report/charts/ with the exact names fixed in charts (150 dpi) under ./out/report/charts/ with the exact names fixed in
README.md, five CSV tables under ./out/report/tables/, and README.md, seven CSV tables under ./out/report/tables/ (t6/t7 carry every
./out/report/REPORT.md. projection and hardware row for all three scales - they are the committed
proof artifacts the executive summary links to), and ./out/report/REPORT.md.
Chart readability contract (owner feedback 2026-07-11): linear scales with Chart readability contract (owner feedback 2026-07-11): linear scales with
actual units everywhere (panels instead of log axes when magnitudes clash), actual units everywhere (panels instead of log axes when magnitudes clash),
@@ -636,6 +637,22 @@ def main() -> int:
money(sizing[(fmt, REPORT_SCALE_GB)]["est_cost_usd"])] for fmt in FORMATS], money(sizing[(fmt, REPORT_SCALE_GB)]["est_cost_usd"])] for fmt in FORMATS],
) )
tables["t6_projected_all_scales"] = (
["format", "query", "scale_gb", "projected_wall_s", "projected_wall", "flag"],
[[FMT_NAMES[fmt], f"q{qn}", s, proj[(fmt, qn, s)][0], human_time(proj[(fmt, qn, s)][0]), proj[(fmt, qn, s)][1]]
for fmt in FORMATS for qn in QUERY_NUMBERS for s in SCALES_GB],
)
tables["t7_hardware_all_scales"] = (
["format", "scale_gb", "configuration", "disk_tb", "ram_gb", "cores", "nodes",
"ram_modules", "nvme_drives", "base_usd", "ram_usd", "disk_usd", "extra_nodes_usd", "est_cost_usd"],
[[FMT_NAMES[fmt], s, config_string(sizing[(fmt, s)], prices),
sizing[(fmt, s)]["disk_tb"], sizing[(fmt, s)]["ram_gb"],
sizing[(fmt, s)]["cores"], sizing[(fmt, s)]["nodes"],
sizing[(fmt, s)]["ram_modules"], sizing[(fmt, s)]["nvme_drives"],
round(sizing[(fmt, s)]["base_usd"]), round(sizing[(fmt, s)]["ram_usd"]),
round(sizing[(fmt, s)]["disk_usd"]), round(sizing[(fmt, s)]["extra_nodes_usd"]),
round(sizing[(fmt, s)]["est_cost_usd"])] for fmt in FORMATS for s in SCALES_GB],
)
scoreboards = build_scoreboards(medians, proj, sizes, sizing) scoreboards = build_scoreboards(medians, proj, sizes, sizing)
tables["t5_weighted_scores"] = ( tables["t5_weighted_scores"] = (
["scale", "format", "search_pts", "ram_pts", "disk_pts", "total"], ["scale", "format", "search_pts", "ram_pts", "disk_pts", "total"],