Compare commits
7 Commits
67206ddab3
...
dev_masha
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6d156f98c4 | ||
|
|
478d4f3201 | ||
|
|
2796b79170 | ||
|
|
ff68679000 | ||
|
|
b4270b0247 | ||
|
|
35d2bdadbd | ||
|
|
af91629f71 |
8
.gitignore
vendored
@@ -1,6 +1,12 @@
|
||||
# Generated pipeline artifacts - reproducible from specs + rules + code
|
||||
# (see docs/rules/build-pipeline-tasks.md)
|
||||
out/
|
||||
out/*
|
||||
!out/report/
|
||||
!out/report/**
|
||||
!out/config/
|
||||
!out/config/**
|
||||
!out/.done/
|
||||
!out/.done/**
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
|
||||
142
README.md
@@ -3,7 +3,8 @@
|
||||
A reproducible pipeline that answers one practical question: **which storage
|
||||
format should a materials-science laboratory use for its measurement data?**
|
||||
|
||||
The pipeline simulates a tribology laboratory (Sandia Pt-Au LDRD context),
|
||||
The pipeline simulates a tribology laboratory (Pt-Au LDRD context, Sandia
|
||||
National Laboratories),
|
||||
generates ~150 MB of physically plausible raw measurement data in CSV,
|
||||
converts the corpus into four additional storage formats, benchmarks seven
|
||||
laboratory-standard retrieval scenarios against all five formats, extrapolates
|
||||
@@ -11,8 +12,26 @@ the measurements to 600 GB - 6 TB, and produces a ranked decision report for
|
||||
five laboratory use cases: analysis, reporting, search/filtering, archiving,
|
||||
and inter-lab exchange.
|
||||
|
||||
**Results - interactive, single-file HTML (open in a browser or download and share):**
|
||||
|
||||
- **[Storage Decision](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html)** -
|
||||
one-page executive summary: the problem, the verdict (relational vs graph
|
||||
database), and what it costs - written for management, every claim linked
|
||||
to the full study
|
||||
(source: [docs/research/Tribology_Storage_Decision.html](docs/research/Tribology_Storage_Decision.html)).
|
||||
- **[Storage Format Evaluation](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html)** -
|
||||
the full study results: dashboard, measured matrix, projections to 6 TB,
|
||||
hardware bill of materials, weighted scoring and verdict
|
||||
(source: [docs/research/Tribology_Storage_Evaluation.html](docs/research/Tribology_Storage_Evaluation.html)).
|
||||
- **[Foreign-Key Architecture Study](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html)** -
|
||||
the measured key-schema study behind the relational design decision
|
||||
(source: [docs/research/Tribology_FK_Architecture.html](docs/research/Tribology_FK_Architecture.html)).
|
||||
|
||||
Reference documents:
|
||||
|
||||
- Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11).
|
||||
- Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md).
|
||||
- Full decision report (markdown): [out/report/REPORT.md](out/report/REPORT.md).
|
||||
|
||||
## Goals
|
||||
|
||||
@@ -143,9 +162,9 @@ docs/
|
||||
Initial_Prompt.md the generative prompt behind specs 00-11 (provenance, non-binding)
|
||||
specs/ task specifications 00-11 (WHAT to build)
|
||||
rules/ binding conventions (HOW work is done)
|
||||
research/ measured design studies (e.g. FK key-schema study)
|
||||
examples/ imported reference materials (not binding)
|
||||
out/ ALL generated artifacts (git-ignored, reproducible):
|
||||
research/ measured design studies and interactive result pages
|
||||
out/ ALL generated artifacts (reproducible; bulk git-ignored,
|
||||
report/ config/ .done/ committed - see .gitignore):
|
||||
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
|
||||
common/ shared helpers (storage_sizes.csv contract, ...)
|
||||
queries/ Q1-Q7 implementations for csv / json / rdf formats
|
||||
@@ -163,10 +182,13 @@ report.py task 11 entry script (REPORT.md, charts, tables, scoring)
|
||||
requirements.txt closed dependency list (docs/rules/code-python-style.md)
|
||||
```
|
||||
|
||||
The `out/` tree - including the ~150 MB CSV corpus, the ~1 GB JSON-LD variant,
|
||||
databases, and benchmark results - is **excluded from git**. Sources of truth
|
||||
are the specs, the rules, and the code; every artifact regenerates
|
||||
deterministically from them. [docs/Initial_Prompt.md](docs/Initial_Prompt.md)
|
||||
The bulk of the `out/` tree - the ~150 MB CSV corpus, the JSON-LD variants,
|
||||
databases, stores, archives, and raw benchmark data - is **excluded from
|
||||
git**; every artifact regenerates deterministically from the specs, the
|
||||
rules, and the code. A curated results subset IS committed (decision of
|
||||
2026-07-11): `out/report/` (REPORT.md, charts, tables, diagrams),
|
||||
`out/config/` (lab_config.yaml, hw_prices.yaml), and the `out/.done/`
|
||||
markers - see `.gitignore`. [docs/Initial_Prompt.md](docs/Initial_Prompt.md)
|
||||
is the original prompt the 12 specification files were generated from - kept
|
||||
for provenance only, never authoritative: where it disagrees with the specs,
|
||||
rules, or code (e.g. the SQLite key schema, the JSON-LD vocabulary), those win.
|
||||
@@ -203,10 +225,14 @@ Benchmark execution is planned for a dedicated Linux host.
|
||||
|
||||
## Results
|
||||
|
||||
This section is populated by tasks 02, 09, 10, and 11. All artifacts land
|
||||
under `out/` (git-ignored); the placeholders below name the exact files the
|
||||
pipeline produces, so the section can be filled in by copying the generated
|
||||
tables and linking the generated images.
|
||||
Results of the full pipeline run of 2026-07-11 (seed 20260711, 141.8 MiB
|
||||
corpus, all 35 benchmark cells checksum-valid). The generated sources of
|
||||
truth are committed under [out/report/](out/report/) - the full decision
|
||||
document is [out/report/REPORT.md](out/report/REPORT.md); the tables below
|
||||
are copied from it. An interactive single-file companion (sidebar
|
||||
navigation, per-query and per-scale views, cost explorer) is
|
||||
[docs/research/Tribology_Storage_Evaluation.html](docs/research/Tribology_Storage_Evaluation.html).
|
||||
Bulk measurement files (`out/bench/`) are git-ignored and regenerable.
|
||||
|
||||
### Process flow diagrams (task 02)
|
||||
|
||||
@@ -214,21 +240,58 @@ tables and linking the generated images.
|
||||
D1 coupon assembly, D2 characterization and batch assembly, D3 testing tree,
|
||||
D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.
|
||||
|
||||
### Measured results (task 09) - to be generated
|
||||
### Measured results (task 09)
|
||||
|
||||
| Placeholder | Source file |
|
||||
|---|---|
|
||||
| Storage footprint per format (measured + compressed) | `out/bench/storage_sizes.csv` |
|
||||
| Q1-Q7 median wall time / peak RAM / CPU / read MB | `out/bench/results_median.csv` |
|
||||
Storage footprint per format (working + compressed archival):
|
||||
|
||||
### Projections (task 10) - to be generated
|
||||
| Format | Measured | vs CSV | Archival (compressed) |
|
||||
|---|---|---|---|
|
||||
| CSV | 149 MB | 1.00x | 52.8 MB (tar.gz) |
|
||||
| JSON-LD (full) | 311 MB | 2.09x | 57.2 MB (tar.gz) |
|
||||
| SQLite | 123 MB | 0.83x | 55.1 MB (gzip) |
|
||||
| PostgreSQL | 397 MB | 2.67x | 53.1 MB (pg_dump -Fc) |
|
||||
| RDF triplestore | 1.74 GB | 11.70x | 126 MB (nt.gz) |
|
||||
|
||||
| Placeholder | Source file |
|
||||
|---|---|
|
||||
| Projected wall time per format at 0.6 / 1.2 / 6 TB with IMPRACTICAL/FAIL flags | `out/bench/extrapolation.csv` |
|
||||
| Hardware sizing and cost per format at scale | `out/bench/hardware_sizing.csv` |
|
||||
Median wall time on the 150 MB corpus, warm cache (peak RAM / CPU / read
|
||||
bytes: [out/report/tables/t2_measured_medians.csv](out/report/tables/t2_measured_medians.csv)):
|
||||
|
||||
### Charts (task 11) - to be generated
|
||||
| Format | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| CSV | 0.06 s | 0.17 s | 1.15 s | 0.72 s | 1.09 s | 0.28 s | 1.27 s |
|
||||
| JSON-LD | 0.11 s | 1.48 s | 9.9 s | 9.9 s | 10 s | 9.8 s | 10 s |
|
||||
| SQLite | 0.06 s | 0.06 s | 0.11 s | 0.06 s | 0.77 s | 0.06 s | 0.71 s |
|
||||
| PostgreSQL | 0.22 s | 0.22 s | 0.22 s | 0.22 s | 0.44 s | 0.22 s | 0.33 s |
|
||||
| RDF triplestore | 0.11 s | 2.81 s | 46 s | 20 s | 60 s | 0.17 s | 41 s |
|
||||
|
||||
### Projections (task 10)
|
||||
|
||||
Projected wall times at 600 GB - `(!)` = IMPRACTICAL (over 1 hour),
|
||||
`FAIL` = over 24 hours; 1.2 TB and 6 TB scales:
|
||||
[out/report/tables/t3_projected_600gb.csv](out/report/tables/t3_projected_600gb.csv)
|
||||
and `out/bench/extrapolation.csv`:
|
||||
|
||||
| Format | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| CSV | 0.06 s | 7.1 min | 1.2 h (!) | 44 min | 1.2 h (!) | 15 min | 1.4 h (!) |
|
||||
| JSON-LD | 0.11 s | 1.5 h (!) | 11 h (!) | 11 h (!) | 11 h (!) | 10.9 h (!) | 11.2 h (!) |
|
||||
| SQLite | 0.06 s | 1.3 s | 5.7 min | 0.06 s | 47 min | 5.2 s | 44 min |
|
||||
| PostgreSQL | 0.22 s | 0.22 s | 3.4 s | 11 s | 3.7 min | 4.1 s | 1.9 min |
|
||||
| RDF triplestore | 0.11 s | 3.0 h (!) | 2.1 d FAIL | 22 h (!) | 2.8 d FAIL | 5.9 min | 1.9 d FAIL |
|
||||
|
||||
Hardware sizing at 600 GB - bill of materials with street prices as of
|
||||
2026-07-11 (component sources: REPORT.md section 7 and
|
||||
[out/config/hw_prices.yaml](out/config/hw_prices.yaml)):
|
||||
|
||||
| Format | Configuration | Total |
|
||||
|---|---|---|
|
||||
| CSV, JSON-LD, SQLite, PostgreSQL (each) | 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM ($1,887); 1 x 7.68 TB NVMe U.2 ($3,995) | $10,500 |
|
||||
| RDF triplestore | 2 nodes; 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2 | $58,740 |
|
||||
|
||||
At 6 TB PostgreSQL still fits one 16-core node - 9 RDIMM modules, 3 NVMe
|
||||
drives, $33,586; the RDF triplestore needs 14 nodes and $527,732
|
||||
(projected; full table: `out/bench/hardware_sizing.csv`).
|
||||
|
||||
### Charts (task 11)
|
||||
|
||||

|
||||

|
||||
@@ -237,29 +300,34 @@ D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.
|
||||

|
||||

|
||||
|
||||
(Images appear after running the full pipeline; task 11 writes them to
|
||||
`out/report/charts/` under exactly these names.)
|
||||
(Task 11 writes the charts to `out/report/charts/` under exactly these
|
||||
names; c3 covers all seven queries - `c3_q1..q7_wall_time.png` - and c4 has
|
||||
three query classes: `point_read`, `indexed`, `full_scan`.)
|
||||
|
||||
### Final scoreboard (task 11) - to be generated
|
||||
### Final scoreboard (task 11)
|
||||
|
||||
Weighted scoring, max 80 points: search speed x5 (0-50), RAM economy x2
|
||||
(0-20), disk economy x1 (0-10); TIMEOUT/FAIL projects to 0.
|
||||
(0-20), disk economy x1 (0-10); a FAIL projection at 600 GB zeroes the
|
||||
search subscore. Values below are the projected-600 GB scoreboard from the
|
||||
2026-07-11 pipeline run (seed 20260711); the generated source of truth is
|
||||
`out/report/tables/t5_weighted_scores.csv`, and REPORT.md also carries the
|
||||
measured-scale scoreboard.
|
||||
|
||||
| Format | Search (0-50) | RAM (0-20) | Disk (0-10) | Total (0-80) |
|
||||
|---|---|---|---|---|
|
||||
| CSV | TBD | TBD | TBD | TBD |
|
||||
| JSON-LD | TBD | TBD | TBD | TBD |
|
||||
| SQLite | TBD | TBD | TBD | TBD |
|
||||
| PostgreSQL | TBD | TBD | TBD | TBD |
|
||||
| RDF | TBD | TBD | TBD | TBD |
|
||||
| CSV | 0.6 | 20.0 | 8.3 | 28.9 |
|
||||
| JSON-LD | 0.1 | 20.0 | 4.0 | 24.0 |
|
||||
| SQLite | 21.8 | 15.2 | 10.0 | 47.0 |
|
||||
| PostgreSQL | 50.0 | 5.5 | 3.1 | 58.6 |
|
||||
| RDF | 0.0 | 0.2 | 0.7 | 0.9 |
|
||||
|
||||
| Use case | Recommended format |
|
||||
|---|---|
|
||||
| Analysis | TBD |
|
||||
| Report generation | TBD |
|
||||
| Search / filtering | TBD |
|
||||
| Archiving | TBD |
|
||||
| Inter-lab exchange | TBD |
|
||||
| Analysis | PostgreSQL (SQLite acceptable single-user up to ~600 GB) |
|
||||
| Report generation | PostgreSQL (track_summary materialized view) |
|
||||
| Search / filtering | PostgreSQL or SQLite (indexed) |
|
||||
| Archiving | CSV tree + tar.gz, plus pg_dump of the system of record |
|
||||
| Inter-lab exchange | Hybrid JSON-LD (metadata + sourceFile links to CSV) |
|
||||
|
||||
## Conventions
|
||||
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
# Tribology Lab Data — Master Plan
|
||||
|
||||
**Discovery · Cleansing/Normalization · Ontology · Conversion**
|
||||
|
||||
Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
|
||||
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
|
||||
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
|
||||
Status: Plan v1 — Draft
|
||||
|
||||
---
|
||||
|
||||
## 0. Objective
|
||||
|
||||
Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
|
||||
|
||||
The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Guiding principles
|
||||
|
||||
1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
|
||||
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
|
||||
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
|
||||
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
|
||||
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
|
||||
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
|
||||
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
|
||||
|
||||
---
|
||||
|
||||
## 2. Phase overview
|
||||
|
||||
| Phase | Name | Nature | Primary output |
|
||||
|-------|------|--------|----------------|
|
||||
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
|
||||
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
|
||||
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
|
||||
|
||||
Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
|
||||
|
||||
**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
|
||||
|
||||
### 3.1 Inputs
|
||||
- Root of the ~700 GB CSV tree (read-only).
|
||||
|
||||
### 3.2 Activities
|
||||
|
||||
**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
|
||||
|
||||
**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
|
||||
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
|
||||
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
|
||||
- atmosphere tokens (`ambient`, `dryNitrogen`, …)
|
||||
|
||||
Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
|
||||
|
||||
**C. Structural profiling (header-only + sampled body).** Per file, without full load:
|
||||
- read header row → column names, count, order;
|
||||
- infer per-column datatype from first/last N rows + a random sample;
|
||||
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
|
||||
|
||||
**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
|
||||
|
||||
### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
|
||||
|
||||
```
|
||||
for each file in walk(root):
|
||||
meta = fs_metadata(file)
|
||||
tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere
|
||||
header = read_first_line(file)
|
||||
columns = parse_columns(header)
|
||||
dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M))
|
||||
fp = fingerprint(columns) # ordered-set hash
|
||||
emit_manifest_row(meta, tokens, columns, dtypes, fp)
|
||||
```
|
||||
|
||||
### 3.4 Artifacts
|
||||
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
|
||||
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
|
||||
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
|
||||
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
|
||||
|
||||
### 3.5 Exit criteria
|
||||
- 100% of files inventoried and fingerprinted.
|
||||
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
|
||||
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
|
||||
|
||||
**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
|
||||
|
||||
The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
|
||||
|
||||
### 4.1 The iteration loop (per measurement type)
|
||||
|
||||
1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
|
||||
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
|
||||
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
|
||||
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
|
||||
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
|
||||
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
|
||||
|
||||
### 4.2 Instruments Repository (ontology sub-model)
|
||||
|
||||
Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
|
||||
- identity (type, model/label, measurement type it produces);
|
||||
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
|
||||
- association to `measurement_types` / `column_definitions` in the data dictionary.
|
||||
|
||||
Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
|
||||
|
||||
### 4.3 Anomaly detection
|
||||
|
||||
Criteria are **derived from the data**, not fixed a priori. Starting catalog:
|
||||
|
||||
| Class | Description | Source of criterion |
|
||||
|-------|-------------|---------------------|
|
||||
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
|
||||
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
|
||||
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
|
||||
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
|
||||
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
|
||||
| Missing required | Null in a non-nullable canonical field | Schema constraints |
|
||||
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
|
||||
|
||||
Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
|
||||
|
||||
### 4.4 Data dictionary / metadata catalog
|
||||
|
||||
Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
|
||||
|
||||
### 4.5 Artifacts
|
||||
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
|
||||
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
|
||||
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
|
||||
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
|
||||
- **`instruments_repository`** — instrument entities + characteristics + ranges.
|
||||
- **`rejects_store`** — quarantined anomalies with reason codes.
|
||||
- **`ontology_notes`** — running record of ERD changes and rationale.
|
||||
|
||||
### 4.6 Exit criteria
|
||||
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
|
||||
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
|
||||
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 4 — Conversion & Export
|
||||
|
||||
**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
|
||||
|
||||
### 5.1 Targets
|
||||
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
|
||||
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
|
||||
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
|
||||
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
|
||||
5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
|
||||
|
||||
### 5.2 Design
|
||||
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
|
||||
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
|
||||
- Lineage is emitted alongside every export (see §6).
|
||||
|
||||
### 5.3 Artifacts
|
||||
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
|
||||
|
||||
### 5.4 Exit criteria
|
||||
- All five target formats generated from one canonical snapshot.
|
||||
- Round-trip validation: record counts and key aggregates match across formats.
|
||||
- Lineage complete and resolvable for sampled records.
|
||||
|
||||
---
|
||||
|
||||
## 6. Cross-cutting concerns
|
||||
|
||||
### 6.1 Lineage & processing-flow documentation
|
||||
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
|
||||
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
|
||||
- Lineage is a first-class export, present in every target format.
|
||||
|
||||
### 6.2 Configuration & rule registry
|
||||
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
|
||||
|
||||
### 6.3 Validation gates
|
||||
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
|
||||
|
||||
---
|
||||
|
||||
## 7. Target artifacts catalog (consolidated)
|
||||
|
||||
| Artifact | Phase | Form |
|
||||
|----------|-------|------|
|
||||
| storage_manifest | 1 | JSONL + SQLite |
|
||||
| schema_fingerprint_report | 1 | JSON/CSV |
|
||||
| unmatched_report | 1 | CSV |
|
||||
| storage_map | 1 | Markdown |
|
||||
| column_mapping | 2↔3 | CSV/JSON (versioned) |
|
||||
| canonical_dataset | 2↔3 | partitioned intermediate |
|
||||
| relational_schema (DDL) | 2↔3 | SQL |
|
||||
| data_dictionary | 2↔3 | table/CSV |
|
||||
| instruments_repository | 2↔3 | table/CSV |
|
||||
| rejects_store | 2↔3 | JSONL |
|
||||
| ontology_notes | 2↔3 | Markdown |
|
||||
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
|
||||
| lineage + build manifest | all | JSONL |
|
||||
|
||||
---
|
||||
|
||||
## 8. Tooling & environment
|
||||
|
||||
- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
|
||||
- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
|
||||
- **PostgreSQL** materialized in Phase 4 (system of record).
|
||||
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
|
||||
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
|
||||
- Python indentation: **tabs**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Performance strategy for 700 GB
|
||||
|
||||
- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
|
||||
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
|
||||
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
|
||||
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
|
||||
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
|
||||
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open items & risks
|
||||
|
||||
1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
|
||||
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
|
||||
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
|
||||
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
|
||||
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
|
||||
|
||||
---
|
||||
|
||||
*End of Plan v1.*
|
||||
@@ -533,7 +533,7 @@ CREATE TABLE friction_cycles (
|
||||
<div class="foot">
|
||||
Measured 2026-07-11 on the deterministic simulated corpus (seed 20260711, 141.8 MiB CSV) · SQLite, warm OS cache, best of 3 runs ·
|
||||
scripts: <span class="mono">measure_key_variants.py</span> (build + size + load + Q1) and <span class="mono">read_benchmarks.py</span> (six read patterns) ·
|
||||
companion page: <span class="mono">Tribology_Ontology.html</span> (measurement-result registry).
|
||||
companion page: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" style="color:#1f5fdb;text-decoration:none">Storage Format Evaluation</a> (full study results).
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
258
docs/research/Tribology_Storage_Decision.html
Normal file
@@ -0,0 +1,258 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Storage Decision - Executive Summary</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;padding:0}
|
||||
body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
|
||||
.mono{font-family:"IBM Plex Mono",monospace}
|
||||
|
||||
header.top{display:flex;align-items:center;gap:14px;height:56px;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea;position:sticky;top:0;z-index:5}
|
||||
.logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px;flex:none}
|
||||
.htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
|
||||
.htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
|
||||
.htitle span{font-size:11.5px;color:#838a95}
|
||||
.hright{margin-left:auto;display:flex;align-items:center;gap:8px}
|
||||
.chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px;text-decoration:none}
|
||||
a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
|
||||
.chip-go{display:inline-flex;align-items:center;gap:6px;font-size:12px;color:#fff;background:#1f5fdb;padding:5px 12px;border-radius:6px;font-weight:600;text-decoration:none}
|
||||
.chip-go:hover{background:#1a4fb8}
|
||||
|
||||
.page{max-width:980px;margin:0 auto;padding:30px 24px 70px}
|
||||
.card{background:#fff;border:1px solid #e6e8ec;border-radius:12px;padding:22px 26px;margin-bottom:16px}
|
||||
h1{margin:0 0 6px;font-size:24px;font-weight:700;letter-spacing:-.02em}
|
||||
h2{margin:0 0 12px;font-size:16px;font-weight:600;letter-spacing:-.01em}
|
||||
p{margin:0 0 10px;font-size:13.5px;color:#3d424b;line-height:1.65}
|
||||
p:last-child{margin-bottom:0}
|
||||
.muted{color:#838a95}
|
||||
a.ext{color:#1f5fdb;text-decoration:none}
|
||||
a.ext:hover{text-decoration:underline}
|
||||
a.fact{color:inherit;text-decoration:underline dashed;text-decoration-thickness:1px;text-underline-offset:3px}
|
||||
a.fact:hover{text-decoration-style:solid}
|
||||
|
||||
.hero{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin-bottom:16px}
|
||||
.hero .card{margin:0}
|
||||
.hero .cap{font-family:"IBM Plex Mono",monospace;font-size:10.5px;letter-spacing:.1em;color:#a4abb4;margin-bottom:10px}
|
||||
.hero .big{font-size:15.5px;line-height:1.6;color:#16181d}
|
||||
.verdict{border:1px solid #bfe6cf;background:#e9f8ef;border-radius:12px;padding:16px 22px;display:flex;align-items:flex-start;gap:16px;margin-bottom:20px}
|
||||
.verdict .mark{width:34px;height:34px;border-radius:50%;background:#18a05a;color:#fff;display:flex;align-items:center;justify-content:center;font-size:18px;font-weight:700;flex:none;margin-top:2px}
|
||||
.verdict b{color:#0f7a45}
|
||||
.verdict p{margin:0 0 8px;font-size:14.5px}
|
||||
table.vtab{border-collapse:collapse;width:100%}
|
||||
table.vtab th{text-align:left;font-size:11px;font-weight:600;padding:5px 16px 5px 0;border-bottom:1px solid #cdeadb}
|
||||
table.vtab th.win{color:#0f7a45}
|
||||
table.vtab th.lose{color:#b23b4e}
|
||||
table.vtab td{font-size:13px;color:#2b3038;padding:7px 16px 7px 0;border-bottom:1px solid #d9f0e3;line-height:1.5;vertical-align:top}
|
||||
table.vtab tr:last-child td{border-bottom:0}
|
||||
table.vtab td.crit{font-weight:600;color:#16181d;width:26%}
|
||||
table.vtab td.lose{color:#b23b4e}
|
||||
|
||||
.kpis{display:grid;grid-template-columns:repeat(3,1fr);gap:16px;margin-bottom:16px}
|
||||
.kpi{background:#fff;border:1px solid #e6e8ec;border-radius:12px;padding:18px 20px}
|
||||
.kpi .v{font-family:"IBM Plex Mono",monospace;font-size:34px;font-weight:600;line-height:1;color:#16181d}
|
||||
.kpi .l{font-size:12.5px;color:#5b6472;margin-top:9px;line-height:1.5}
|
||||
|
||||
table.duel{width:100%;border-collapse:collapse}
|
||||
table.duel th{padding:6px 14px;text-align:left;font-size:12px;border-bottom:2px solid #eef0f2;vertical-align:bottom}
|
||||
table.duel th .pill{display:inline-flex;padding:3px 10px;border-radius:20px;font-size:10.5px;font-weight:600;margin-top:4px}
|
||||
table.duel th.win{color:#0f7a45}
|
||||
table.duel th.win .pill{color:#0f7a45;background:#e7f6ee}
|
||||
table.duel th.lose{color:#b23b4e}
|
||||
table.duel th.lose .pill{color:#b23b4e;background:#fbe9ec}
|
||||
table.duel td{padding:7px 14px;font-size:13px;color:#3d424b;border-bottom:1px solid #f4f5f7;line-height:1.45;vertical-align:top}
|
||||
table.duel td.crit{font-weight:600;color:#16181d;width:26%}
|
||||
table.duel td.crit small{display:block;font-weight:400;color:#a4abb4;margin-top:2px}
|
||||
table.duel td .ok{color:#0f7a45;font-weight:700;margin-right:6px}
|
||||
table.duel td .no{color:#b23b4e;font-weight:700;margin-right:6px}
|
||||
table.duel td.more{width:12%;text-align:right;white-space:nowrap}
|
||||
|
||||
.panels{display:grid;grid-template-columns:1fr 1fr;gap:22px;margin-top:6px}
|
||||
.panel h3{margin:0 0 12px;font-size:12.5px;font-weight:600;color:#5b6472;text-transform:uppercase;letter-spacing:.04em}
|
||||
.brow{display:flex;align-items:center;gap:12px;margin-bottom:12px}
|
||||
.brow .lab{width:118px;flex:none;font-size:12.5px;color:#3d424b}
|
||||
.brow .trk{flex:1;background:#f1f3f6;border-radius:5px;height:20px;overflow:hidden;position:relative}
|
||||
.brow .fill{height:20px;border-radius:0 4px 4px 0;min-width:3px}
|
||||
.brow .inval{position:absolute;top:50%;transform:translateY(-50%);font-family:"IBM Plex Mono",monospace;font-size:12px;font-weight:600;white-space:nowrap}
|
||||
.brow .inval.in-track{left:12px;color:#16181d}
|
||||
.brow .inval.in-fill{right:12px;color:#fff}
|
||||
.brow:hover .trk{background:#eaedf1}
|
||||
.fill.pg{background:#1f5fdb}
|
||||
.fill.rdf{background:#c0476b}
|
||||
|
||||
.steps{counter-reset:st;list-style:none;margin:6px 0 0;padding:0}
|
||||
.steps li{counter-increment:st;display:flex;gap:14px;align-items:flex-start;padding:10px 0;font-size:13.5px;color:#3d424b;line-height:1.6;border-bottom:1px solid #f4f5f7}
|
||||
.steps li:last-child{border-bottom:0}
|
||||
.steps li::before{content:counter(st);width:26px;height:26px;border-radius:50%;background:#eaf1fe;color:#1f5fdb;font-weight:700;font-size:13px;display:flex;align-items:center;justify-content:center;flex:none;margin-top:1px}
|
||||
|
||||
.foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:16px;margin-top:26px;line-height:1.8}
|
||||
@media(max-width:840px){.hero,.kpis,.panels{grid-template-columns:1fr}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<header class="top">
|
||||
<div class="logo">SE</div>
|
||||
<div class="htitle">
|
||||
<b>Storage Decision</b>
|
||||
<span>Executive summary · laboratory measurement data</span>
|
||||
</div>
|
||||
<div class="hright">
|
||||
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
|
||||
<a class="chip-go" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">Open the full study →</a>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="page">
|
||||
|
||||
<h1>Which database should hold the laboratory's data?</h1>
|
||||
|
||||
<div class="hero">
|
||||
<div class="card">
|
||||
<div class="cap">THE PROBLEM</div>
|
||||
<p class="big"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#premises" target="_blank" rel="noopener"><b>600 GB</b></a> of laboratory data today.<br/><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#premises" target="_blank" rel="noopener"><b>6 TB</b></a> within a few years.<br/>It needs one working database.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="cap">THE CANDIDATES</div>
|
||||
<p class="big"><b>Relational</b> (PostgreSQL)<br/>vs<br/><b>Graph</b> (RDF triplestore).</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="verdict">
|
||||
<div class="mark">✓</div>
|
||||
<div style="flex:1">
|
||||
<p><b>Decision: Relational vs Graph (RDF):</b></p>
|
||||
<table class="vtab">
|
||||
<tr>
|
||||
<th></th>
|
||||
<th class="win">Relational (PostgreSQL)</th>
|
||||
<th class="lose">Graph (RDF)</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Average report, today</td>
|
||||
<td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~6 seconds</a></td>
|
||||
<td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~1.6 hours</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Average report at 6 TB</td>
|
||||
<td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~30 seconds</a></td>
|
||||
<td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">~12 hours</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Hardware at 6 TB</td>
|
||||
<td><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one ordinary server</a></td>
|
||||
<td class="lose"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">a 14-server cluster</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Ontology</td>
|
||||
<td colspan="2">remains accessible with Relational (PostgreSQL)</td>
|
||||
</tr>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="kpis">
|
||||
<div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">1,000x</a></div><div class="l">faster on the average everyday report</div></div>
|
||||
<div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">$494k saved</a></div><div class="l">hardware at 6 TB: $33.6k for PGSQL server instead of a $527.7k RDF cluster</div></div>
|
||||
<div class="kpi"><div class="v"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#kpis" target="_blank" rel="noopener">1 vs 14</a></div><div class="l">servers needed at 6 TB</div></div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Side by side</h2>
|
||||
<table class="duel">
|
||||
<tr>
|
||||
<th></th>
|
||||
<th class="win">Relational (PostgreSQL)<br><span class="pill">✓ recommended</span></th>
|
||||
<th class="lose">Graph (RDF triplestore)<br><span class="pill">✗ rejected at scale</span></th>
|
||||
<th></th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Average report, today<small>at 600 GB</small></td>
|
||||
<td><span class="ok">✓</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 6 seconds</a></td>
|
||||
<td><span class="no">✗</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 1.6 hours; 3 of 7 tasks need more than a day</a></td>
|
||||
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#proj" target="_blank" rel="noopener">details →</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Average report at 6 TB<small>the 3-year horizon</small></td>
|
||||
<td><span class="ok">✓</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 30 seconds</a></td>
|
||||
<td><span class="no">✗</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">about 12 hours; 5 of 7 tasks need a day or more</a></td>
|
||||
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#proj" target="_blank" rel="noopener">details →</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Servers needed today<small>at 600 GB</small></td>
|
||||
<td><span class="ok">✓</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one standard server, $10.5k</a></td>
|
||||
<td><span class="no">✗</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">already two servers with 1.4 TB of memory, $58.7k</a></td>
|
||||
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#hw" target="_blank" rel="noopener">details →</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Servers at 6 TB</td>
|
||||
<td><span class="ok">✓</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">still one server, $33.6k</a></td>
|
||||
<td><span class="no">✗</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">a 14-server cluster, $527.7k</a></td>
|
||||
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#hw" target="_blank" rel="noopener">details →</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="crit">Disk for the same data<small>at 6 TB</small></td>
|
||||
<td><span class="ok">✓</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#storage" target="_blank" rel="noopener">21 TB</a></td>
|
||||
<td><span class="no">✗</span><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#storage" target="_blank" rel="noopener">91 TB - the graph form stores the same data 4.4x bigger</a></td>
|
||||
<td class="more"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html#storage" target="_blank" rel="noopener">details →</a></td>
|
||||
</tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Average everyday report</h2>
|
||||
<div class="panels">
|
||||
<div class="panel">
|
||||
<h3>Today · 600 GB</h3>
|
||||
<div class="brow" title="PostgreSQL, average of the 7 everyday tasks at 600 GB: about 6 seconds (projected)">
|
||||
<div class="lab">PostgreSQL</div>
|
||||
<div class="trk"><div class="fill pg" style="width:0.4%"></div><span class="inval in-track"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">6 s</a></span></div>
|
||||
</div>
|
||||
<div class="brow" title="RDF triplestore, average of the 7 everyday tasks at 600 GB: about 1.6 hours (projected)">
|
||||
<div class="lab">Graph (RDF)</div>
|
||||
<div class="trk"><div class="fill rdf" style="width:100%"></div><span class="inval in-fill"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">1.6 h</a></span></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="panel">
|
||||
<h3>In a few years · 6 TB</h3>
|
||||
<div class="brow" title="PostgreSQL, average of the 7 everyday tasks at 6 TB: about 30 seconds (projected)">
|
||||
<div class="lab">PostgreSQL</div>
|
||||
<div class="trk"><div class="fill pg" style="width:0.4%"></div><span class="inval in-track"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">30 s</a></span></div>
|
||||
</div>
|
||||
<div class="brow" title="RDF triplestore, average of the 7 everyday tasks at 6 TB: about 12 hours (projected)">
|
||||
<div class="lab">Graph (RDF)</div>
|
||||
<div class="trk"><div class="fill rdf" style="width:100%"></div><span class="inval in-fill"><a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#performance" target="_blank" rel="noopener">12 h</a></span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p class="muted" style="margin-top:14px">Bars are to scale within each panel - the blue bar is real, it is just that small. An everyday report that takes hours is a report nobody runs.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>What about the other formats?</h2>
|
||||
<p><b>CSV</b> stays exactly where it belongs - the raw archive coming off the instruments (best compression, byte-reproducible). <b>JSON-LD</b> stays the exchange format for sending data between laboratories. Neither is a working database, so neither was a candidate. <b>SQLite</b> is excellent for a single user <a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#notes" target="_blank" rel="noopener">up to ~600 GB</a> but does not scale to the shared 6 TB horizon.</p>
|
||||
<p><b>And the ontology?</b> It survives intact: semantic, ontology-style access is provided as a <b>virtual layer on top of PostgreSQL</b> (e.g. Ontop OBDA) - the graph view without the graph database's cost.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h2>Recommended next steps</h2>
|
||||
<ol class="steps">
|
||||
<li>Adopt <b>PostgreSQL as the system of record</b> for laboratory measurement data; keep the raw CSV archive as the source of truth for reprocessing.</li>
|
||||
<li>Provide ontology and semantic queries as a <b>virtual RDF layer</b> over PostgreSQL - no data is copied into a triplestore.</li>
|
||||
<li>Revisit hardware once volume approaches 6 TB: the measured plan is <a class="fact" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html#hardware" target="_blank" rel="noopener">one 16-core server with 551 GB RAM (~$33.6k at July 2026 prices)</a>.</li>
|
||||
</ol>
|
||||
</div>
|
||||
|
||||
<div class="foot">
|
||||
Figures at 600 GB and 6 TB are model projections calibrated on a measured benchmark (7 everyday retrieval tasks x 5 storage formats, every result checksum-validated); the "average report" is the geometric mean of the seven tasks, the same averaging the study's scoring uses. Method, raw data and hardware price sources: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision_Facts.html" target="_blank" rel="noopener">facts ledger (every number traced to its artifact)</a> · <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">full interactive study</a> · <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.<br>
|
||||
Study: Mary Goncharenko (Tribology Laboratory, University of Florida; internship at Sandia National Laboratories), with the technical assistance of Vasiliy Goncharenko (SoftCreator, LLC).
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
131
docs/research/Tribology_Storage_Decision_Facts.html
Normal file
@@ -0,0 +1,131 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Decision Facts Ledger - Storage Decision</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;padding:0}
|
||||
body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
|
||||
.mono{font-family:"IBM Plex Mono",monospace}
|
||||
header.top{display:flex;align-items:center;gap:14px;height:56px;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea;position:sticky;top:0;z-index:5}
|
||||
.logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px;flex:none}
|
||||
.htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
|
||||
.htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
|
||||
.htitle span{font-size:11.5px;color:#838a95}
|
||||
.hright{margin-left:auto;display:flex;align-items:center;gap:8px}
|
||||
.chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px;text-decoration:none}
|
||||
a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
|
||||
.page{max-width:1080px;margin:0 auto;padding:28px 24px 70px}
|
||||
h1{margin:0 0 6px;font-size:22px;font-weight:700;letter-spacing:-.02em}
|
||||
h2{margin:26px 0 10px;font-size:15px;font-weight:600}
|
||||
p{margin:0 0 10px;font-size:13px;color:#3d424b;line-height:1.6;max-width:940px}
|
||||
.muted{color:#838a95}
|
||||
a{color:#1f5fdb;text-decoration:none}
|
||||
a:hover{text-decoration:underline}
|
||||
.card{background:#fff;border:1px solid #e6e8ec;border-radius:11px;padding:6px 18px 10px;margin-bottom:8px;overflow-x:auto}
|
||||
table{width:100%;border-collapse:collapse;min-width:760px}
|
||||
th{font-size:10.5px;text-transform:uppercase;letter-spacing:.04em;color:#838a95;font-weight:600;text-align:left;padding:9px 12px;border-bottom:1px solid #eef0f2}
|
||||
td{font-size:12.5px;color:#3d424b;padding:9px 12px;border-bottom:1px solid #f4f5f7;vertical-align:top;line-height:1.55}
|
||||
tr:last-child td{border-bottom:0}
|
||||
td.id{font-family:"IBM Plex Mono",monospace;font-weight:600;color:#16181d;white-space:nowrap}
|
||||
td.val{font-family:"IBM Plex Mono",monospace;white-space:nowrap}
|
||||
.foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:14px;margin-top:26px;line-height:1.7}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<header class="top">
|
||||
<div class="logo">SE</div>
|
||||
<div class="htitle">
|
||||
<b>Decision Facts Ledger</b>
|
||||
<span>Every fact on the Storage Decision page · value, formula, source</span>
|
||||
</div>
|
||||
<div class="hright">
|
||||
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">← Storage Decision</a>
|
||||
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">Full study</a>
|
||||
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="page">
|
||||
|
||||
<h1>Decision Facts Ledger</h1>
|
||||
<p>Every number shown on the <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision</a> executive summary, traced to the artifact that proves it. All source files are committed in the public repository and regenerate deterministically from seed 20260711. Projection tables carry one row per format x task x scale; "geometric mean" is the same averaging the study's scoring uses.</p>
|
||||
<p class="muted">Sources: <span class="mono">t6</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6_projected_all_scales.csv</a> · <span class="mono">t7</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7_hardware_all_scales.csv</a> · <span class="mono">t1</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t1_storage_footprint.csv" target="_blank" rel="noopener">t1_storage_footprint.csv</a> · <span class="mono">prices</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/config/hw_prices.yaml" target="_blank" rel="noopener">hw_prices.yaml</a> · <span class="mono">spec</span> = <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">10_extrapolation_model.md</a></p>
|
||||
|
||||
<h2 id="premises">Premises</h2>
|
||||
<div class="card">
|
||||
<table>
|
||||
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
|
||||
<tr><td class="id">P1</td><td>"600 GB of laboratory data today"</td><td class="val">600 GB</td><td>The laboratory's current operational volume; also the first extrapolation scale of the study (scale_gb = 600).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">spec</a>, <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">P2</td><td>"6 TB within a few years"</td><td class="val">6,000 GB</td><td>The largest extrapolation scale of the study (scale_gb = 6000), the growth horizon.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/docs/specs/10_extrapolation_model.md" target="_blank" rel="noopener">spec</a>, <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2 id="performance">Performance (average everyday report)</h2>
|
||||
<div class="card">
|
||||
<table>
|
||||
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
|
||||
<tr><td class="id">F1</td><td>PostgreSQL, average report today: "~6 seconds"</td><td class="val">5.66 s</td><td>Geometric mean of the 7 projected_wall_s values where format = PostgreSQL, scale_gb = 600.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">F2</td><td>PostgreSQL, average report at 6 TB: "~30 seconds"</td><td class="val">29.6 s</td><td>Geometric mean, format = PostgreSQL, scale_gb = 6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">F3</td><td>Graph (RDF), average report today: "~1.6 hours"</td><td class="val">5,902 s = 1.64 h</td><td>Geometric mean, format = RDF triplestore, scale_gb = 600.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">F4</td><td>Graph (RDF), average report at 6 TB: "~12 hours"</td><td class="val">43,157 s = 11.99 h</td><td>Geometric mean, format = RDF triplestore, scale_gb = 6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">F5</td><td>"3 of 7 tasks need more than a day" (RDF, today)</td><td class="val">q3 2.1 d · q5 2.8 d · q7 1.9 d</td><td>Count of RDF rows at scale_gb = 600 with projected_wall_s > 86,400.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">F6</td><td>"5 of 7 tasks need a day or more" (RDF, 6 TB)</td><td class="val">q2, q3, q4, q5, q7</td><td>Count of RDF rows at scale_gb = 6000 with projected_wall_s >= 86,400.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2 id="hardware">Hardware and cost</h2>
|
||||
<div class="card">
|
||||
<table>
|
||||
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
|
||||
<tr><td class="id">F7</td><td>"one standard server, $10.5k" (PostgreSQL, today)</td><td class="val">$10,500 · 1 node</td><td>Row format = PostgreSQL, scale_gb = 600: est_cost_usd, nodes; bill of materials in the configuration column.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">F8</td><td>"already two servers with 1.4 TB of memory, $58.7k" (RDF, today)</td><td class="val">$58,740 · 2 nodes · 1,404.1 GB RAM</td><td>Row format = RDF triplestore, scale_gb = 600: est_cost_usd, nodes, ram_gb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">F9</td><td>"still one server, $33.6k" / "one 16-core server with 551 GB RAM" (PostgreSQL, 6 TB)</td><td class="val">$33,586 · 1 node · 16 cores · 551.3 GB</td><td>Row format = PostgreSQL, scale_gb = 6000: est_cost_usd, nodes, cores, ram_gb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">F10</td><td>"a 14-server cluster, $527.7k" (RDF, 6 TB)</td><td class="val">$527,732 · 14 nodes</td><td>Row format = RDF triplestore, scale_gb = 6000: est_cost_usd, nodes.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">F11</td><td>"July 2026 prices"</td><td class="val">as_of 2026-07-11</td><td>Street prices with per-component source URLs recorded in the operator-editable price file.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/config/hw_prices.yaml" target="_blank" rel="noopener">prices</a></td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2 id="storage">Storage</h2>
|
||||
<div class="card">
|
||||
<table>
|
||||
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
|
||||
<tr><td class="id">F12</td><td>"21 TB" disk (PostgreSQL, 6 TB)</td><td class="val">20.855 TB</td><td>Row format = PostgreSQL, scale_gb = 6000: disk_tb (includes 30% free-space headroom).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">F13</td><td>"91 TB" disk (RDF, 6 TB)</td><td class="val">91.268 TB</td><td>Row format = RDF triplestore, scale_gb = 6000: disk_tb.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">F14</td><td>"stores the same data 4.4x bigger"</td><td class="val">11.70 / 2.67 = 4.38</td><td>Ratio of the measured working-footprint coefficients vs raw CSV (RDF 11.70x over PostgreSQL 2.67x).</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t1_storage_footprint.csv" target="_blank" rel="noopener">t1</a></td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2 id="kpis">Headline figures</h2>
|
||||
<div class="card">
|
||||
<table>
|
||||
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
|
||||
<tr><td class="id">K1</td><td>"1,000x faster on the average everyday report"</td><td class="val">1,043x today · 1,459x at 6 TB</td><td>F3 / F1 = 5,902 / 5.66; F4 / F2 = 43,157 / 29.6. Stated conservatively as 1,000x.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">K2</td><td>"$494k saved" (hardware at 6 TB)</td><td class="val">$494,146</td><td>F10 - F9 = 527,732 - 33,586.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
<tr><td class="id">K3</td><td>"1 vs 14 servers needed at 6 TB"</td><td class="val">nodes: 1 vs 14</td><td>nodes column, rows PostgreSQL/6000 and RDF triplestore/6000.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t7_hardware_all_scales.csv" target="_blank" rel="noopener">t7</a></td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h2 id="notes">Notes</h2>
|
||||
<div class="card">
|
||||
<table>
|
||||
<tr><th>ID</th><th>Fact on the page</th><th>Exact value</th><th>How obtained</th><th>Source</th></tr>
|
||||
<tr><td class="id">N1</td><td>"SQLite is excellent for a single user up to ~600 GB"</td><td class="val">all 7 tasks OK at 600 GB; q5 = 1.6 h, q7 = 1.5 h at 1.2 TB</td><td>SQLite rows in t6: every flag OK at scale_gb = 600; its single-threaded full scans exceed the 1-hour bound from scale_gb = 1200.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/src/branch/master/out/report/tables/t6_projected_all_scales.csv" target="_blank" rel="noopener">t6</a></td></tr>
|
||||
<tr><td class="id">N2</td><td>Method behind every projected number</td><td class="val">measured 35-cell benchmark</td><td>Scaling laws calibrated on the measured corpus; every measured run checksum-validated. Full methodology and measured medians in the interactive study.</td><td><a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" target="_blank" rel="noopener">full study</a></td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div class="foot">
|
||||
Part of the LabDataStorageEvaluation study · run of 2026-07-11, seed 20260711 · canonical source: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.<br>
|
||||
Study: Mary Goncharenko (Tribology Laboratory, University of Florida; internship at Sandia National Laboratories), with the technical assistance of Vasiliy Goncharenko (SoftCreator, LLC).
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
814
docs/research/Tribology_Storage_Evaluation.html
Normal file
@@ -0,0 +1,814 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Storage Format Evaluation - Tribology Lab Data</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500;600&family=IBM+Plex+Sans:wght@400;500;600;700&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;padding:0}
|
||||
body{font-family:"IBM Plex Sans",system-ui,sans-serif;color:#16181d;background:#eef0f2;-webkit-font-smoothing:antialiased}
|
||||
::-webkit-scrollbar{width:10px;height:10px}
|
||||
::-webkit-scrollbar-thumb{background:#cfd4da;border-radius:6px;border:2px solid #eef0f2}
|
||||
::-webkit-scrollbar-track{background:transparent}
|
||||
.mono{font-family:"IBM Plex Mono",monospace}
|
||||
|
||||
.app{display:flex;flex-direction:column;height:100vh;width:100%;overflow:hidden;background:#eef0f2}
|
||||
header.top{display:flex;align-items:center;gap:14px;height:56px;flex:none;padding:0 22px;background:#fff;border-bottom:1px solid #e2e5ea}
|
||||
.logo{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:7px;background:#16181d;color:#fff;font-family:"IBM Plex Mono",monospace;font-weight:600;font-size:12px}
|
||||
.htitle{display:flex;flex-direction:column;line-height:1.25;white-space:nowrap}
|
||||
.htitle b{font-size:14.5px;font-weight:600;letter-spacing:-.01em}
|
||||
.htitle span{font-size:11.5px;color:#838a95}
|
||||
.hright{margin-left:auto;display:flex;align-items:center;gap:8px}
|
||||
.chip{font-family:"IBM Plex Mono",monospace;font-size:11px;color:#838a95;background:#f3f5f7;border:1px solid #e6e8ec;padding:4px 10px;border-radius:6px}
|
||||
.chip-ok{display:inline-flex;align-items:center;gap:6px;font-size:12px;color:#0f7a45;background:#e7f6ee;padding:4px 10px;border-radius:6px;font-weight:600}
|
||||
.chip-ok i{width:7px;height:7px;border-radius:7px;background:#18a05a}
|
||||
|
||||
.frame{display:flex;flex:1;min-height:0}
|
||||
nav.side{width:252px;flex:none;background:#fff;border-right:1px solid #e2e5ea;padding:14px 12px;display:flex;flex-direction:column;overflow-y:auto}
|
||||
.navcap{font-family:"IBM Plex Mono",monospace;font-size:10px;letter-spacing:.1em;color:#a4abb4;padding:6px 11px 8px}
|
||||
nav.side a{display:flex;align-items:center;gap:11px;width:100%;padding:8px 11px;border-radius:8px;cursor:pointer;text-decoration:none;font-size:13.5px;margin:1px 0;color:#475059;font-weight:500}
|
||||
nav.side a .tag{width:24px;height:19px;border-radius:5px;display:flex;align-items:center;justify-content:center;font-family:"IBM Plex Mono",monospace;font-size:9px;font-weight:600;flex:none;background:#eceef1;color:#8a909b}
|
||||
nav.side a.active{background:#eaf1fe;color:#16181d;font-weight:600}
|
||||
nav.side a.active .tag{background:#1f5fdb;color:#fff}
|
||||
.sidefoot{margin-top:auto;padding:14px 11px 6px;border-top:1px solid #eef0f2}
|
||||
.sidefoot .cap{font-family:"IBM Plex Mono",monospace;font-size:10px;letter-spacing:.08em;color:#a4abb4;margin-bottom:8px}
|
||||
.sidefoot .txt{font-size:12px;color:#6b727c;line-height:1.6}
|
||||
|
||||
main{flex:1;min-width:0;overflow-y:auto;padding:26px 30px 60px}
|
||||
main section{display:none}
|
||||
main section.on{display:block}
|
||||
h1{margin:0;font-size:21px;font-weight:600;letter-spacing:-.02em}
|
||||
.sub{margin:4px 0 18px;font-size:13px;color:#838a95;max-width:900px;line-height:1.55}
|
||||
.card{background:#fff;border:1px solid #e6e8ec;border-radius:11px;padding:18px 20px;margin-bottom:14px}
|
||||
.card h3{margin:0 0 10px;font-size:14px;font-weight:600}
|
||||
.card p{margin:0 0 10px;font-size:13px;color:#3d424b;line-height:1.6;max-width:900px}
|
||||
.card p:last-child{margin-bottom:0}
|
||||
.card ul{margin:6px 0 4px;padding-left:20px}
|
||||
.card li{font-size:13px;color:#3d424b;line-height:1.65;margin-bottom:6px;max-width:860px}
|
||||
.muted{color:#838a95}
|
||||
.kpis{display:grid;grid-template-columns:repeat(5,1fr);gap:14px;margin-bottom:16px}
|
||||
.kpis.four{grid-template-columns:repeat(4,1fr)}
|
||||
.kpi{background:#fff;border:1px solid #e6e8ec;border-radius:10px;padding:15px 16px}
|
||||
.kpi .l{font-size:11px;color:#838a95;font-weight:500;text-transform:uppercase;letter-spacing:.04em}
|
||||
.kpi .v{font-family:"IBM Plex Mono",monospace;font-size:24px;font-weight:600;margin-top:7px;line-height:1}
|
||||
.kpi .s{font-size:11.5px;color:#a4abb4;margin-top:5px}
|
||||
.v-bad{color:#b23b4e}.v-good{color:#0f7a45}.v-neutral{color:#16181d}
|
||||
|
||||
table.data{width:100%;border-collapse:collapse}
|
||||
table.data th{font-size:10.5px;text-transform:uppercase;letter-spacing:.04em;color:#838a95;font-weight:600;text-align:left;padding:9px 12px;background:#f7f8fa;border-bottom:1px solid #eef0f2}
|
||||
table.data td{font-size:12.5px;color:#3d424b;padding:10px 12px;border-bottom:1px solid #f4f5f7;vertical-align:top}
|
||||
table.data th.num, table.data td.num{text-align:right;font-family:"IBM Plex Mono",monospace}
|
||||
table.data td.mono{font-family:"IBM Plex Mono",monospace}
|
||||
|
||||
.hbars{display:flex;flex-direction:column;gap:11px;margin-top:6px}
|
||||
.hbar{display:flex;align-items:center;gap:14px}
|
||||
.hbar .lab{width:190px;flex:none;font-size:12.5px;color:#3d424b}
|
||||
.hbar .lab small{display:block;color:#a4abb4;font-size:11px}
|
||||
.hbar .trk{flex:1;background:#f1f3f6;border-radius:6px;overflow:hidden}
|
||||
.hbar .fill{height:14px;border-radius:6px;min-width:2px}
|
||||
.hbar .val{font-family:"IBM Plex Mono",monospace;font-size:12.5px;color:#16181d;width:130px;flex:none;text-align:right}
|
||||
.legend{display:flex;gap:16px;flex-wrap:wrap;margin:2px 0 10px}
|
||||
.legend span{display:inline-flex;align-items:center;gap:6px;font-size:11.5px;color:#5b6472}
|
||||
.legend i{width:10px;height:10px;border-radius:3px;display:inline-block}
|
||||
|
||||
.stack{display:flex;align-items:center;gap:14px;margin-bottom:11px}
|
||||
.stack .lab{width:190px;flex:none;font-size:12.5px;color:#3d424b}
|
||||
.stack .trk{flex:1;display:flex;background:#f1f3f6;border-radius:6px;overflow:hidden;height:16px}
|
||||
.stack .seg{height:16px}
|
||||
.stack .val{font-family:"IBM Plex Mono",monospace;font-size:12.5px;width:70px;flex:none;text-align:right;font-weight:600}
|
||||
|
||||
.tabs{display:inline-flex;gap:4px;background:#eceef1;border-radius:8px;padding:3px;margin:0 0 14px}
|
||||
.tabs button{border:0;background:transparent;font-family:"IBM Plex Sans",system-ui,sans-serif;font-size:12px;font-weight:600;color:#5b6472;padding:5px 12px;border-radius:6px;cursor:pointer}
|
||||
.tabs button.on{background:#fff;color:#16181d;box-shadow:0 1px 3px rgba(20,30,50,.08)}
|
||||
|
||||
.callout{border-left:3px solid #1f5fdb;background:#f5f8fe;border-radius:0 9px 9px 0;padding:12px 16px;font-size:13px;color:#2b3038;line-height:1.6;margin:12px 0;max-width:900px}
|
||||
.callout.warn{border-left-color:#b23b4e;background:#fdf3f5}
|
||||
.callout.good{border-left-color:#18a05a;background:#f0faf4}
|
||||
.pill{display:inline-flex;align-items:center;padding:2px 9px;border-radius:20px;font-size:10.5px;font-weight:600;white-space:nowrap}
|
||||
.pill.ok{color:#0f7a45;background:#e7f6ee}
|
||||
.pill.imp{color:#a06a12;background:#fdf3e0}
|
||||
.pill.fail{color:#b23b4e;background:#fbe9ec}
|
||||
.pill.rec{color:#0f7a45;background:#e7f6ee}
|
||||
.pill.base{color:#5b6472;background:#eef0f2}
|
||||
|
||||
pre.env{background:#16181d;color:#dfe3e9;border-radius:9px;padding:14px 16px;font-family:"IBM Plex Mono",monospace;font-size:11.5px;line-height:1.65;overflow-x:auto;margin:10px 0 0}
|
||||
.tree{font-family:"IBM Plex Mono",monospace;font-size:12px;line-height:1.8;color:#3d424b;background:#f7f8fa;border:1px solid #eef0f2;border-radius:9px;padding:14px 18px;margin-top:8px}
|
||||
.grid2{display:grid;grid-template-columns:1fr 1fr;gap:14px}
|
||||
@media(max-width:1100px){.grid2{grid-template-columns:1fr}.kpis{grid-template-columns:repeat(2,1fr)}}
|
||||
.foot{font-size:11.5px;color:#a4abb4;border-top:1px solid #e6e8ec;padding-top:14px;margin-top:30px;line-height:1.7}
|
||||
.src a, a.ext{color:#1f5fdb;text-decoration:none;word-break:break-all}
|
||||
.src a:hover, a.ext:hover{text-decoration:underline}
|
||||
a.chip{text-decoration:none}
|
||||
a.chip:hover{border-color:#1f5fdb;color:#1f5fdb}
|
||||
svg text{font-family:"IBM Plex Sans",system-ui,sans-serif}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
|
||||
<header class="top">
|
||||
<div class="logo">SE</div>
|
||||
<div class="htitle">
|
||||
<b>Storage Format Evaluation</b>
|
||||
<span>Tribology lab data · CSV vs JSON-LD vs SQLite vs PostgreSQL vs RDF</span>
|
||||
</div>
|
||||
<div class="hright">
|
||||
<a class="chip" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">Git repository</a>
|
||||
<span class="chip">7 queries x 5 formats · 35 cells</span>
|
||||
<span class="chip">seed 20260711</span>
|
||||
<span class="chip-ok"><i></i>Measured 2026-07-11 · all checksums OK</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="frame">
|
||||
|
||||
<nav class="side" id="sidenav">
|
||||
<div class="navcap">NAVIGATE</div>
|
||||
<a href="#dashboard" class="active"><span class="tag">DB</span><span>Dashboard</span></a>
|
||||
<a href="#lab"><span class="tag">LM</span><span>Lab Model & Corpus</span></a>
|
||||
<a href="#method"><span class="tag">ME</span><span>Methodology</span></a>
|
||||
<a href="#storage"><span class="tag">SF</span><span>Storage Footprint</span></a>
|
||||
<a href="#measured"><span class="tag">MR</span><span>Measured Results</span></a>
|
||||
<a href="#proj"><span class="tag">PR</span><span>Projections 0.6-6 TB</span></a>
|
||||
<a href="#hw"><span class="tag">HW</span><span>Hardware & Cost</span></a>
|
||||
<a href="#score"><span class="tag">SC</span><span>Scoring & Verdict</span></a>
|
||||
<a href="#about"><span class="tag">AB</span><span>About</span></a>
|
||||
<div class="sidefoot">
|
||||
<div class="cap">CONTEXT</div>
|
||||
<div class="txt">Final results of the LabDataStorageEvaluation pipeline (tasks 01-11): a simulated Pt-Au tribology laboratory corpus benchmarked across five storage formats. Sources of truth: out/report/ and out/bench/.</div>
|
||||
<div class="cap" style="margin-top:12px">AUTHOR · SOURCE</div>
|
||||
<div class="txt">Mary Goncharenko - University of Florida / Sandia National Laboratories.<br>
|
||||
<a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a></div>
|
||||
</div>
|
||||
</nav>
|
||||
|
||||
<main id="main">
|
||||
|
||||
<!-- ================= DASHBOARD ================= -->
|
||||
<section id="dashboard" class="on">
|
||||
<h1>Dashboard</h1>
|
||||
<p class="sub">A 141.8 MiB physically plausible CSV corpus (the canonical raw format) was converted into four other storage formats and benchmarked on seven laboratory retrieval scenarios with subprocess-isolated resource metering; results were extrapolated to 600 GB, 1.2 TB and 6 TB. Every measured run validated its rows against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9). Short on time? The one-page management summary is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision</a>.</p>
|
||||
<div class="kpis">
|
||||
<div class="kpi"><div class="l">Benchmark cells</div><div class="v v-good">35 / 35</div><div class="s">OK; 0 invalid, 0 timeouts</div></div>
|
||||
<div class="kpi"><div class="l">Corpus</div><div class="v v-neutral">141.8 MiB</div><div class="s">8,718 CSV files, byte-reproducible</div></div>
|
||||
<div class="kpi"><div class="l">Winner @600 GB</div><div class="v v-good">58.6 / 80</div><div class="s">PostgreSQL weighted score</div></div>
|
||||
<div class="kpi"><div class="l">Worst blow-up</div><div class="v v-bad">11.7x</div><div class="s">RDF store vs raw CSV size</div></div>
|
||||
<div class="kpi"><div class="l">Session</div><div class="v v-neutral">916 s</div><div class="s">whole matrix, quiet machine</div></div>
|
||||
</div>
|
||||
<div class="callout good"><b>Recommendation (confirmed by data):</b> PostgreSQL as the system of record; CSV retained as the raw archive (best compressor, ~2.8x); hybrid JSON-LD for inter-lab exchange; RDF only as a virtual layer (e.g. Ontop OBDA) over PostgreSQL - a materialized triplestore fails at scale on speed, RAM and cost simultaneously.</div>
|
||||
<div class="card">
|
||||
<h3>Typical query time per format (geometric mean Q1-Q7)</h3>
|
||||
<div id="dash-geo" class="hbars"></div>
|
||||
<p class="muted" style="margin-top:12px">Left value: measured on the 150 MB corpus. Right value in parentheses: projected at 600 GB. Linear bars scaled within the measured values; every projected number in this document is labeled as projected.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Where each format breaks</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Format</th><th>Strength</th><th>Breaking point</th><th>@600 GB flags</th></tr></thead>
|
||||
<tbody id="dash-breaks"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= LAB MODEL ================= -->
|
||||
<section id="lab">
|
||||
<h1>Lab Model & Corpus</h1>
|
||||
<p class="sub">A simulated tribology laboratory (Pt-Au LDRD study context, Sandia National Laboratories): Ti-6Al-4V coupons with a sputtered Cr adhesion layer and a Pt-Au composition-gradient coating, tested on a 6-probe tribometer in two environments. All data is generated deterministically from seed 20260711; regeneration is byte-identical (MANIFEST.csv sha256 proof).</p>
|
||||
<div class="kpis four">
|
||||
<div class="kpi"><div class="l">Deposition</div><div class="v v-neutral">4 · 12 · 588</div><div class="s">batches · wafers · coupons (7x7 grid)</div></div>
|
||||
<div class="kpi"><div class="l">Friction testing</div><div class="v v-neutral">1,440</div><div class="s">tracks: 480 coupons x 3 replicates</div></div>
|
||||
<div class="kpi"><div class="l">Bulk rows</div><div class="v v-neutral">4.32 M</div><div class="s">1.44M COF cycles + 2.88M loop points</div></div>
|
||||
<div class="kpi"><div class="l">Characterization</div><div class="v v-neutral">235,200</div><div class="s">XRF points + nanoindentation, AFM, profilometry</div></div>
|
||||
</div>
|
||||
<div class="grid2">
|
||||
<div class="card">
|
||||
<h3>Data hierarchy</h3>
|
||||
<div class="tree">batch (B721-B724, deposition matrix)
|
||||
-> wafer (x3 per batch)
|
||||
-> coupon (x49 per wafer; 480 friction + 108 reserve)
|
||||
-> xrf_map (400 pts) · nanoindentation (25) · afm · profilometry (100)
|
||||
-> track (x3, counterface rotation)
|
||||
-> cof_vs_cycle (1,000 rows) · friction_loops (10x200) · wear</div>
|
||||
<p style="margin-top:10px">Runs R1-R4 map one-to-one onto batches; two environments (lab air / dry N2). The Q1 benchmark target track is <span class="mono">B722-W2-C13-T2</span>.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Physical models behind the numbers</h3>
|
||||
<ul>
|
||||
<li>COF: exponential run-in to a steady state depending on Au wt% and environment; noise floor 0.12.</li>
|
||||
<li>Hardness / modulus: linear in Au wt% with measurement noise (25 indents per coupon).</li>
|
||||
<li>Wear: Archard model, k0 = 1.5e-7 (lab air) / 6e-8 (dry N2) mm3/(N*m), log-normal noise.</li>
|
||||
<li>Au gradient across wafer position; batch-dependent center 8 / 25 / 10 / 60 wt%.</li>
|
||||
</ul>
|
||||
<p>Benchmark queries therefore return physically meaningful answers (e.g. Q2 selects the Au 10 +/- 0.5 wt% composition window), not random noise.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>The five storage formats under test</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Format</th><th>Variant benchmarked</th><th>How Q1-Q7 run</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><b>CSV</b> (canonical raw)</td><td>hierarchical file tree + MANIFEST</td><td>python csv module, streaming; Q1 = direct path read</td></tr>
|
||||
<tr><td><b>JSON-LD</b></td><td>FULL (per-coupon files, 588); HYBRID kept for exchange</td><td>ijson event streaming with early exit</td></tr>
|
||||
<tr><td><b>SQLite</b></td><td>single file, composite natural PKs WITHOUT ROWID, enforced FKs</td><td>SQL; precomputed track_summary for Q2-Q4/Q7</td></tr>
|
||||
<tr><td><b>PostgreSQL 16</b></td><td>remote host, range-partitioned bulk tables (12+12), BRIN + b-tree</td><td>SQL; parallel scans (4 workers), track_summary matview</td></tr>
|
||||
<tr><td><b>RDF</b></td><td>pyoxigraph on-disk store, 17,419,713 triples (compact modeling)</td><td>SPARQL; run-in/steady-state derived from raw cycles</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= METHODOLOGY ================= -->
|
||||
<section id="method">
|
||||
<h1>Methodology</h1>
|
||||
<p class="sub">The product of this study is its measurements; the protocol is designed so a sloppy number cannot survive it (docs/rules/bench-methodology.md).</p>
|
||||
<div class="grid2">
|
||||
<div class="card">
|
||||
<h3>Measurement protocol</h3>
|
||||
<ul>
|
||||
<li><b>One subprocess per run</b> - RSS / CPU / read bytes attribute to exactly that cell; psutil samples the whole process tree every 50 ms (the venv launcher stub alone would report 4 MB).</li>
|
||||
<li><b>1 warm-up + 3 measured runs</b> per cell; medians reported, min/max kept in raw data.</li>
|
||||
<li><b>Randomized cell order</b> (deterministic shuffle from the corpus seed); order recorded.</li>
|
||||
<li><b>30-minute hard timeout</b> per run; a timeout is recorded as data, never retried.</li>
|
||||
<li><b>Correctness gates speed:</b> every run's rows are compared to the canonical expected results (sorted rows, float tolerance 1e-9); a fast run that fails the checksum is an invalid measurement.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Fairness and honesty</h3>
|
||||
<ul>
|
||||
<li>Identical result-set contract for all five implementations of each query; no format answers a simplified question.</li>
|
||||
<li>Each format plays its idiomatic strength (CSV direct path read, SQL indexes, matviews) - but no LIMIT/sampling that changes the answer.</li>
|
||||
<li><b>Q5 must scan raw cycles in every format</b>; the PostgreSQL EXPLAIN plan is checked to prove track_summary was not used.</li>
|
||||
<li>Cold-cache pass skipped (Windows host has no page-cache drop) - marked, not faked.</li>
|
||||
<li>PostgreSQL is remote: client wall times include the LAN round-trip; server-side cost captured via pg_stat_statements deltas and Prometheus host windows.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>The seven retrieval scenarios</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>ID</th><th>Scenario</th><th class="num">Result rows</th></tr></thead>
|
||||
<tbody id="method-queries"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Environment (recorded per session)</h3>
|
||||
<pre class="env" id="method-env"></pre>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= STORAGE ================= -->
|
||||
<section id="storage">
|
||||
<h1>Storage Footprint</h1>
|
||||
<p class="sub">Working footprint is what the query engine reads; archival is the compressed form for shelving. All archival sizes converge to 53-57 MB - the information content is the same, the working overhead is not.</p>
|
||||
<div class="kpis four">
|
||||
<div class="kpi"><div class="l">Most compact</div><div class="v v-good">0.83x</div><div class="s">SQLite: 123 MB, indexes included</div></div>
|
||||
<div class="kpi"><div class="l">PostgreSQL</div><div class="v v-neutral">2.67x</div><div class="s">397 MB live; 34.1% of it is indexes</div></div>
|
||||
<div class="kpi"><div class="l">RDF store</div><div class="v v-bad">11.70x</div><div class="s">1.74 GB for the same data</div></div>
|
||||
<div class="kpi"><div class="l">Best archive</div><div class="v v-good">52.8 MB</div><div class="s">csv tree tar.gz (2.8x compression)</div></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Working footprint, measured on the 150 MB corpus</h3>
|
||||
<div id="st-work" class="hbars"></div>
|
||||
</div>
|
||||
<div class="grid2">
|
||||
<div class="card">
|
||||
<h3>Archival (compressed) footprint</h3>
|
||||
<div id="st-arch" class="hbars"></div>
|
||||
<p class="muted" style="margin-top:10px">pg_dump -Fc is internally gzip-compressed; RDF also has a Turtle serialization (596 MB, not the archival form). Hybrid JSON-LD is 1.9 MB of metadata + sourceFile links.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Projected working footprint (linear scaling)</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Format</th><th class="num">@600 GB</th><th class="num">@1.2 TB</th><th class="num">@6 TB</th></tr></thead>
|
||||
<tbody id="st-proj"></tbody>
|
||||
</table>
|
||||
<p class="muted" style="margin-top:10px">All projected. Storage scales linearly in data volume; coefficients are the measured ratios above.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= MEASURED ================= -->
|
||||
<section id="measured">
|
||||
<h1>Measured Results <span class="pill base" style="vertical-align:3px">150 MB corpus · warm cache</span></h1>
|
||||
<p class="sub">Median of 3 isolated runs per cell. Pick a query and a metric; bars are linear with the actual value on each bar. Client peak RSS below ~50 ms of runtime under-reports (sampling cadence is spec-fixed at 50 ms) - wall times are exact.</p>
|
||||
<div class="tabs" id="mq-tabs"></div>
|
||||
<div class="tabs" id="mm-tabs" style="margin-left:10px"></div>
|
||||
<div class="card">
|
||||
<h3 id="m-title"></h3>
|
||||
<div id="m-bars" class="hbars"></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Full median matrix - wall time</h3>
|
||||
<table class="data"><thead id="m-mat-h"></thead><tbody id="m-mat"></tbody></table>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>PostgreSQL server side (remote host, 4 cores / 3.8 GiB)</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Query</th><th class="num">Server exec per call</th><th class="num">Shared blocks hit</th><th class="num">Read from disk</th></tr></thead>
|
||||
<tbody id="m-pgsrv"></tbody>
|
||||
</table>
|
||||
<p class="muted" style="margin-top:10px">pg_stat_statements deltas over the session (4 calls per query). Zero disk reads: the whole 397 MB database sits in the 1 GB shared_buffers. Host peak CPU during the session: 9.2%; RAM delta ~5 MB - the server is nowhere near its limits at this scale.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= PROJECTIONS ================= -->
|
||||
<section id="proj">
|
||||
<h1>Projections <span class="pill imp" style="vertical-align:3px">every number here is projected, not measured</span></h1>
|
||||
<p class="sub">Per-cell scaling laws calibrated on the measured 150 MB point after subtracting each format's harness floor (interpreter start, imports, connection). Flags: IMPRACTICAL over 1 hour, FAIL over 24 hours. Pick a target scale.</p>
|
||||
<div class="tabs" id="p-tabs"></div>
|
||||
<div class="card">
|
||||
<h3 id="p-title"></h3>
|
||||
<table class="data"><thead id="p-mat-h"></thead><tbody id="p-mat"></tbody></table>
|
||||
<p class="muted" style="margin-top:10px" id="p-note"></p>
|
||||
</div>
|
||||
<div class="grid2">
|
||||
<div class="card">
|
||||
<h3>Scaling law per access pattern</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Law</th><th>Growth</th><th>Applied to</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="mono">O(1)</td><td>flat</td><td>CSV / JSON-LD Q1 (one file, size fixed)</td></tr>
|
||||
<tr><td class="mono">O(log n)</td><td>index depth</td><td>SQLite / PostgreSQL / RDF Q1</td></tr>
|
||||
<tr><td class="mono">O(log n + k)</td><td>~linear (result set grows)</td><td>relational + RDF Q2, Q4</td></tr>
|
||||
<tr><td class="mono">O(k log n)</td><td>linear x depth</td><td>relational Q3, Q6; RDF Q6</td></tr>
|
||||
<tr><td class="mono">O(n)</td><td>linear, 1 core</td><td>all CSV/JSON scans; SQLite Q5/Q7; RDF Q3/Q5/Q7</td></tr>
|
||||
<tr><td class="mono">O(n / cores)</td><td>linear / parallel workers</td><td>PostgreSQL Q5, Q7 (partitioned scan)</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Full-scan degradation (Q5 + Q7 geometric mean)</h3>
|
||||
<div id="p-degr"></div>
|
||||
<p class="muted" style="margin-top:8px">First column measured, the rest projected. PostgreSQL rides its parallel partitioned scan; single-threaded engines grow linearly.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="callout">PostgreSQL is the only format with all seven queries below one hour at every scale - Q5 at 6 TB projects to 37 minutes on 16 cores. SQLite holds to 600 GB, then its single-threaded scans pass the hour mark. JSON-LD full scans reach 4.6 days at 6 TB; RDF Q5 reaches 27.8 days.</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= HARDWARE ================= -->
|
||||
<section id="hw">
|
||||
<h1>Hardware & Cost <span class="pill base" style="vertical-align:3px">street prices as of 2026-07-11</span></h1>
|
||||
<p class="sub">Bill-of-materials costing: whole 64 GB RDIMM modules, whole 7.68 TB NVMe drives, a priced 1U chassis; disk includes 30% free-space headroom. RAM model: relational hot set = 10% of index size + 4 GB working set; RDF hot set = 20% of the store; streaming formats are flat. Single-node RAM ceiling: 1 TB.</p>
|
||||
<div class="tabs" id="h-tabs"></div>
|
||||
<div class="card">
|
||||
<h3 id="h-title"></h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Format</th><th>Exact configuration</th><th class="num">Chassis</th><th class="num">RAM</th><th class="num">Disk</th><th class="num">Extra nodes</th><th class="num">Total</th></tr></thead>
|
||||
<tbody id="h-bom"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="grid2">
|
||||
<div class="card">
|
||||
<h3>Total cost comparison</h3>
|
||||
<div id="h-bars" class="hbars"></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Component prices & sources <span class="pill base">retrieved 2026-07-11</span></h3>
|
||||
<ul class="src">
|
||||
<li>64 GB DDR5-5600 ECC RDIMM (A-Tech, 2x32GB): <b>$1,887</b><br><a href="https://atechmemory.com/collections/ddr5-memory-ram">atechmemory.com/collections/ddr5-memory-ram</a></li>
|
||||
<li>7.68 TB NVMe U.2 1DWPD enterprise SSD (Cloud Ninjas): <b>$3,995</b><br><a href="https://cloudninjas.com/products/new-7-68tb-nvme-u-2-1dwpd-sie-2-5-enterprise-solid-state-drive-for-14th-15th-16th-gen-dell">cloudninjas.com/products/new-7-68tb-nvme-u-2-1dwpd...</a></li>
|
||||
<li>1U AMD EPYC chassis (Broadberry starting configs): AS-1015CS-TNR <b>$4,618</b>, AS-1115CS-TNR <b>$6,272</b><br><a href="https://www.broadberry.com/amd-epyc-9004-supermicro-servers">broadberry.com/amd-epyc-9004-supermicro-servers</a></li>
|
||||
<li>Market context - server DRAM 2025-2026 price surge:<br><a href="https://www.techpowerup.com/342331/server-dram-pricing-jumps-50-only-70-of-orders-getting-filled">techpowerup.com/342331/server-dram-pricing-jumps-50...</a></li>
|
||||
</ul>
|
||||
<p class="muted" style="margin-top:10px">Spot prices during a documented DRAM surge; re-cost by editing out/config/hw_prices.yaml and re-running tasks 10-11.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Cost vs speed at 600 GB (projected)</h3>
|
||||
<div id="h-scatter"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= SCORING ================= -->
|
||||
<section id="score">
|
||||
<h1>Scoring & Verdict</h1>
|
||||
<p class="sub">Subscores are 10 x best/value per metric, weighted: search speed x5 (0-50, inverse geometric mean of Q1-Q7), RAM economy x2 (0-20), disk economy x1 (0-10); maximum 80. A FAIL projection at 600 GB zeroes the search subscore.</p>
|
||||
<div class="grid2">
|
||||
<div class="card">
|
||||
<h3>Measured scale (150 MB)</h3>
|
||||
<div class="legend"><span><i style="background:#1f5fdb"></i>search x5</span><span><i style="background:#e0892e"></i>RAM x2</span><span><i style="background:#0f9d8a"></i>disk x1</span></div>
|
||||
<div id="sc-measured"></div>
|
||||
<p class="muted" style="margin-top:8px">At laptop scale SQLite is the honest winner: near-instant indexed queries in one file with the smallest footprint.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Projected at 600 GB</h3>
|
||||
<div class="legend"><span><i style="background:#1f5fdb"></i>search x5</span><span><i style="background:#e0892e"></i>RAM x2</span><span><i style="background:#0f9d8a"></i>disk x1</span></div>
|
||||
<div id="sc-proj"></div>
|
||||
<p class="muted" style="margin-top:8px">At production scale the ranking flips: parallel indexed retrieval dominates the weights, and PostgreSQL takes the lead.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Use-case mapping</h3>
|
||||
<table class="data">
|
||||
<thead><tr><th>Use case</th><th>Recommended format</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Interactive analysis (joins, aggregations)</td><td><b>PostgreSQL</b>; SQLite acceptable single-user up to ~600 GB</td></tr>
|
||||
<tr><td>Report generation (repeated summaries)</td><td><b>PostgreSQL</b> (track_summary materialized view)</td></tr>
|
||||
<tr><td>Search / filtering</td><td><b>PostgreSQL or SQLite</b> (indexed); flat formats need full scans</td></tr>
|
||||
<tr><td>Archiving</td><td><b>CSV tree + tar.gz</b> (canonical raw) plus pg_dump of the system of record</td></tr>
|
||||
<tr><td>Inter-lab exchange</td><td><b>Hybrid JSON-LD</b> (metadata + sourceFile links to CSV)</td></tr>
|
||||
<tr><td>Semantic / ontology queries</td><td><b>Virtual RDF layer</b> over PostgreSQL (e.g. Ontop OBDA), not a materialized triplestore</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Limitations</h3>
|
||||
<ul>
|
||||
<li>Single-node measurements on one Windows host + one remote PostgreSQL host; no cluster variance.</li>
|
||||
<li>Simulated corpus (physically plausible, fixed seed); real instrument data may distribute differently.</li>
|
||||
<li>Extrapolation is analytic, calibrated on one 150 MB point; no intermediate-scale validation runs.</li>
|
||||
<li>Warm-cache only (no page-cache drop on Windows); harness floor assumed constant across scales.</li>
|
||||
<li>RDF numbers reflect compact modeling and client-side derivation of run-in/steady-state; a SPARQL-side aggregation engine could shift, not remove, the scan penalty.</li>
|
||||
<li>Hardware prices are spot street prices (2026-07-11) during a DRAM price surge.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="foot">
|
||||
Generated from the LabDataStorageEvaluation pipeline artifacts (run of 2026-07-11, seed 20260711): out/report/REPORT.md, out/bench/results_median.csv, extrapolation.csv, hardware_sizing.csv, storage_sizes.csv, out/config/hw_prices.yaml. The full decision document with charts is out/report/REPORT.md; this page is the interactive companion. Related study: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html" target="_blank" rel="noopener">Foreign-Key Architecture Study</a>.
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ================= ABOUT ================= -->
|
||||
<section id="about">
|
||||
<h1>About</h1>
|
||||
<p class="sub">Attribution and provenance for this document. It is distributed as a single standalone HTML file; everything needed to verify or reproduce the numbers is referenced below.</p>
|
||||
<div class="card">
|
||||
<h3>Author</h3>
|
||||
<p>This study was conducted by <b>Mary Goncharenko</b>, PhD student at the Tribology Laboratory, University of Florida, Gainesville, FL, during a research internship at the tribology laboratory of Sandia National Laboratories, with an assignment on laboratory ontology development.</p>
|
||||
<p>The research was carried out with the technical assistance of her father, <b>Vasiliy Goncharenko</b>, Chief Technology Architect at SoftCreator, LLC (enterprise data architecture across relational and semantic stores; 35 years in technology).</p>
|
||||
<p class="muted">Profiles: <a class="ext" href="https://www.linkedin.com/in/marygoncharenko/" target="_blank" rel="noopener">linkedin.com/in/marygoncharenko</a> · <a class="ext" href="https://linkedin.com/in/vasiliy-goncharenko" target="_blank" rel="noopener">linkedin.com/in/vasiliy-goncharenko</a></p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Source repository</h3>
|
||||
<p>The complete evaluation pipeline - task specifications, binding conventions, source code, and the generated report - is publicly available at:</p>
|
||||
<p class="mono"><a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">https://git.boskadoff.com/Public/LabDataStorageEvaluation</a></p>
|
||||
<p>This page is the interactive companion to the repository's <span class="mono">out/report/REPORT.md</span>; all numbers embedded here come from the pipeline run of 2026-07-11. The corpus regenerates byte-identically from seed 20260711 (MANIFEST.csv sha256 proof), so every figure is independently reproducible from the repository alone.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h3>Data statement</h3>
|
||||
<p>The measurement corpus is simulated: physically plausible models of a Pt-Au LDRD tribology study context at Sandia National Laboratories, generated from a fixed random seed. It contains no experimental measurements and no export-controlled data. Hardware prices cited in the cost model are public street prices retrieved 2026-07-11 (sources in the Hardware & Cost section).</p>
|
||||
<p>Companion pages in the same repository: <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Decision.html" target="_blank" rel="noopener">Storage Decision (one-page executive summary)</a> · <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html" target="_blank" rel="noopener">Foreign-Key Architecture Study</a>.</p>
|
||||
</div>
|
||||
<div class="foot">Standalone distributable file. If you received this document outside the repository, the canonical source is <a class="ext" href="https://git.boskadoff.com/Public/LabDataStorageEvaluation" target="_blank" rel="noopener">git.boskadoff.com/Public/LabDataStorageEvaluation</a>.</div>
|
||||
</section>
|
||||
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
"use strict";
|
||||
|
||||
/* ---------- embedded data (from the 2026-07-11 pipeline run) ---------- */
|
||||
const FMT = ["csv","json","sqlite","pg","rdf"];
|
||||
const NAME = {csv:"CSV", json:"JSON-LD", sqlite:"SQLite", pg:"PostgreSQL", rdf:"RDF triplestore"};
|
||||
const COLOR = {csv:"#9aa6b6", json:"#e0892e", sqlite:"#0f9d8a", pg:"#1f5fdb", rdf:"#c0476b"};
|
||||
const QN = [1,2,3,4,5,6,7];
|
||||
const QDESC = {
|
||||
1:["COF vs cycle curve for one track (B722-W2-C13-T2)",1000],
|
||||
2:["Steady-state COF for coupons with mean Au = 10 +/- 0.5 wt%",33],
|
||||
3:["Hardness vs steady-state COF across all batches",480],
|
||||
4:["Anomaly filter: dry N2, 100 mN, cof_ss > 0.20",469],
|
||||
5:["Mean run-in cycles per batch over ALL raw cycles (no summaries)",4],
|
||||
6:["Wear volume vs load per coupon",480],
|
||||
7:["Stribeck-style mean COF by (environment, speed x load)",2]
|
||||
};
|
||||
|
||||
/* measured medians, per format: [q1..q7] */
|
||||
const WALL = {
|
||||
csv:[0.06,0.1661,1.1521,0.715,1.0948,0.2756,1.2735],
|
||||
json:[0.1113,1.4788,9.9408,9.9218,9.9623,9.7927,10.0781],
|
||||
sqlite:[0.0617,0.0607,0.1134,0.0604,0.7659,0.0612,0.7094],
|
||||
pg:[0.2214,0.2206,0.2211,0.2232,0.4405,0.2212,0.3342],
|
||||
rdf:[0.1131,2.8054,46.1273,19.8245,59.5884,0.1682,40.5534]
|
||||
};
|
||||
const RSS = {
|
||||
csv:[3.9,20.5,21.1,20.9,20.7,20.2,21.0],
|
||||
json:[22.2,23.9,24.4,24.5,24.4,24.1,24.1],
|
||||
sqlite:[3.9,3.9,4.1,3.9,25.0,3.9,25.0],
|
||||
pg:[42.2,42.1,42.0,41.9,42.0,41.6,42.1],
|
||||
rdf:[31.6,85.4,263.9,167.9,264.0,45.9,263.6]
|
||||
};
|
||||
const READ = {
|
||||
csv:[0,2.45,31.76,16.82,31.78,1.23,32.47],
|
||||
json:[1.4,53.08,297.57,297.75,297.57,297.82,297.94],
|
||||
sqlite:[0,0,0,0,117.91,0,92.97],
|
||||
pg:[4.08,4.08,4.08,4.08,4.08,4.08,4.08],
|
||||
rdf:[3.62,375.56,5445.1,2694.56,5390.46,15.92,5380.51]
|
||||
};
|
||||
|
||||
/* projections: scale -> format -> [[seconds,flag] x7]; flag "" | "I" | "F" */
|
||||
const PROJ = {
|
||||
600: {
|
||||
csv:[[0.06,""],[428.301,""],[4408.0,"I"],[2643.775,""],[4176.725,"I"],[870.266,""],[4897.995,"I"]],
|
||||
json:[[0.111,""],[5519.622,"I"],[39673.993,"I"],[39597.306,"I"],[39760.772,"I"],[39076.231,"I"],[40228.164,"I"]],
|
||||
sqlite:[[0.062,""],[1.271,""],[339.238,""],[0.06,""],[2847.603,""],[5.18,""],[2619.558,""]],
|
||||
pg:[[0.222,""],[0.221,""],[3.42,""],[10.715,""],[222.111,""],[4.06,""],[114.849,""]],
|
||||
rdf:[[0.113,""],[10866.789,"I"],[185722.881,"F"],[79559.375,"I"],[240054.647,"F"],[352.729,""],[163225.475,"F"]]
|
||||
},
|
||||
1200: {
|
||||
csv:[[0.06,""],[856.543,""],[8815.94,"I"],[5287.489,"I"],[8353.391,"I"],[1740.472,""],[9795.931,"I"]],
|
||||
json:[[0.111,""],[11039.133,"I"],[79347.876,"I"],[79194.5,"I"],[79521.432,"I"],[78152.351,"I"],[80456.218,"I"]],
|
||||
sqlite:[[0.063,""],[2.482,""],[699.328,""],[0.06,""],[5695.146,"I"],[10.615,""],[5239.055,"I"]],
|
||||
pg:[[0.222,""],[0.221,""],[6.817,""],[21.209,""],[444.001,""],[8.137,""],[229.477,""]],
|
||||
rdf:[[0.113,""],[21733.465,"I"],[371445.649,"F"],[159118.636,"F"],[480109.181,"F"],[727.088,""],[326450.837,"F"]]
|
||||
},
|
||||
6000: {
|
||||
csv:[[0.06,""],[4282.474,"I"],[44079.46,"I"],[26437.206,"I"],[41766.715,"I"],[8702.119,"I"],[48979.414,"I"]],
|
||||
json:[[0.111,""],[55195.222,"I"],[396738.933,"F"],[395972.054,"F"],[397606.717,"F"],[390761.312,"F"],[402280.643,"F"]],
|
||||
sqlite:[[0.063,""],[12.169,""],[3739.195,"I"],[0.06,""],[28475.49,"I"],[56.5,""],[26195.034,"I"]],
|
||||
pg:[[0.222,""],[0.221,""],[35.495,""],[105.162,""],[2219.125,""],[42.55,""],[1146.503,""]],
|
||||
rdf:[[0.113,""],[108666.874,"F"],[1857227.792,"F"],[795592.727,"F"],[2400545.453,"F"],[3887.402,"I"],[1632253.733,"F"]]
|
||||
}
|
||||
};
|
||||
|
||||
/* storage bytes */
|
||||
const WORK = {csv:148654472, json:311300222, sqlite:123400192, pg:397467648, rdf:1739414085};
|
||||
const ARCH = {csv:[52827471,"tar.gz of the tree"], json:[57162674,"tar.gz (full + hybrid)"], sqlite:[55149977,"gzip of tribo.db"], pg:[53054380,"pg_dump -Fc"], rdf:[126487650,"N-Triples gz"]};
|
||||
const COEFF = {csv:1.0, json:2.09, sqlite:0.83, pg:2.67, rdf:11.70};
|
||||
|
||||
/* hardware sizing: scale -> format -> {modules,drives,nodes,cores,base,ram,disk,extra,total} */
|
||||
const HW = {
|
||||
600: {
|
||||
csv:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
json:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
sqlite:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
pg:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
rdf:{m:22,d:2,n:2,c:16,base:4618,ram:41514,disk:7990,extra:4618,total:58740}
|
||||
},
|
||||
1200: {
|
||||
csv:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
json:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
sqlite:{m:1,d:1,n:1,c:16,base:4618,ram:1887,disk:3995,extra:0,total:10500},
|
||||
pg:{m:2,d:1,n:1,c:16,base:4618,ram:3774,disk:3995,extra:0,total:12387},
|
||||
rdf:{m:44,d:3,n:3,c:16,base:4618,ram:83028,disk:11985,extra:9236,total:108867}
|
||||
},
|
||||
6000: {
|
||||
csv:{m:1,d:2,n:1,c:16,base:4618,ram:1887,disk:7990,extra:0,total:14495},
|
||||
json:{m:1,d:3,n:1,c:16,base:4618,ram:1887,disk:11985,extra:0,total:18490},
|
||||
sqlite:{m:3,d:1,n:1,c:16,base:4618,ram:5661,disk:3995,extra:0,total:14274},
|
||||
pg:{m:9,d:3,n:1,c:16,base:4618,ram:16983,disk:11985,extra:0,total:33586},
|
||||
rdf:{m:220,d:12,n:14,c:16,base:4618,ram:415140,disk:47940,extra:60034,total:527732}
|
||||
}
|
||||
};
|
||||
|
||||
/* weighted scores: [search, ram, disk, total] */
|
||||
const SCORES = {
|
||||
measured:{csv:[15.5,20.0,8.3,43.8], json:[1.7,17.2,4.0,22.9], sqlite:[50.0,16.9,10.0,76.9], pg:[26.2,10.0,3.1,39.3], rdf:[1.3,1.6,0.7,3.6]},
|
||||
proj:{csv:[0.6,20.0,8.3,28.9], json:[0.1,20.0,4.0,24.0], sqlite:[21.8,15.2,10.0,47.0], pg:[50.0,5.5,3.1,58.6], rdf:[0.0,0.2,0.7,0.9]}
|
||||
};
|
||||
|
||||
/* pg server per-call exec ms + shared blocks (session deltas / 4 calls) */
|
||||
const PGSRV = {1:[0.42,72],2:[0.43,1312],3:[4.92,6500],4:[1.09,140],5:[237.26,189692],6:[1.68,172],7:[107.14,32456]};
|
||||
|
||||
const ENVIRONMENT = [
|
||||
"benchmark_host_os: Windows-2025Server-10.0.26100-SP0",
|
||||
"benchmark_host_cpu: Intel64 Family 6 Model 141 (4 cores physical/logical), 8 GiB RAM",
|
||||
"benchmark_host_disks: VMware Virtual NVMe Disk / SSD / 100 GB",
|
||||
"python: 3.12.10 · sqlite 3.49.1 · psycopg 3.3.4 · pyoxigraph 0.5.9 · ijson 3.5.1 · psutil 7.2.2",
|
||||
"postgresql_server: PostgreSQL 16.14 (Ubuntu 24.04), remote host, 4 cores / 3.8 GiB",
|
||||
"pg tuning: shared_buffers=1GB · work_mem=32MB · effective_cache_size=2500MB · max_parallel_workers_per_gather=4",
|
||||
"cold_cache_pass: skipped (Windows host, no page-cache drop)",
|
||||
"pg_backend_sampling: remote server - Prometheus windows + pg_stat_statements deltas instead"
|
||||
].join("\n");
|
||||
|
||||
/* ---------- formatting helpers ---------- */
|
||||
function fmtTime(s){
|
||||
if(s < 10) return s.toFixed(2) + " s";
|
||||
if(s < 120) return s.toFixed(0) + " s";
|
||||
if(s < 7200) return (s/60).toFixed(1) + " min";
|
||||
if(s < 172800) return (s/3600).toFixed(1) + " h";
|
||||
return (s/86400).toFixed(1) + " d";
|
||||
}
|
||||
function fmtSize(b){
|
||||
if(b >= 1e12) return (b/1e12).toFixed(2) + " TB";
|
||||
if(b >= 1e9) return (b/1e9).toFixed(2) + " GB";
|
||||
return (b/1e6).toFixed(0) + " MB";
|
||||
}
|
||||
function fmtMoney(v){ return "$" + v.toLocaleString("en-US"); }
|
||||
function esc(t){ return String(t).replace(/&/g,"&").replace(/</g,"<"); }
|
||||
function el(id){ return document.getElementById(id); }
|
||||
function flagPill(f){
|
||||
if(f === "F") return ' <span class="pill fail">FAIL</span>';
|
||||
if(f === "I") return ' <span class="pill imp">> 1 h</span>';
|
||||
return "";
|
||||
}
|
||||
function geomean(a){ return Math.exp(a.reduce((s,v)=>s+Math.log(v),0)/a.length); }
|
||||
|
||||
/* one horizontal bar row; linear scale against max */
|
||||
function barRow(label, sub, frac, color, val){
|
||||
return '<div class="hbar"><div class="lab">' + label + (sub ? '<small>' + sub + '</small>' : '') + '</div>' +
|
||||
'<div class="trk"><div class="fill" style="width:' + Math.max(0.4, frac*100).toFixed(2) + '%;background:' + color + '"></div></div>' +
|
||||
'<div class="val">' + val + '</div></div>';
|
||||
}
|
||||
function tabs(containerId, items, onPick){
|
||||
const box = el(containerId);
|
||||
box.innerHTML = items.map((t,i) => '<button data-i="' + i + '"' + (i===0 ? ' class="on"' : '') + '>' + t + '</button>').join("");
|
||||
box.querySelectorAll("button").forEach(b => b.addEventListener("click", () => {
|
||||
box.querySelectorAll("button").forEach(x => x.classList.remove("on"));
|
||||
b.classList.add("on");
|
||||
onPick(parseInt(b.dataset.i,10));
|
||||
}));
|
||||
}
|
||||
|
||||
/* ---------- navigation ---------- */
|
||||
/* menu items only: the sidebar footer also holds a regular external link */
|
||||
const navLinks = document.querySelectorAll('nav.side a[href^="#"]');
|
||||
function show(id){
|
||||
if(!document.getElementById(id)) return;
|
||||
document.querySelectorAll("main section").forEach(s => s.classList.toggle("on", s.id === id));
|
||||
navLinks.forEach(a => a.classList.toggle("active", a.getAttribute("href") === "#" + id));
|
||||
el("main").scrollTop = 0;
|
||||
}
|
||||
navLinks.forEach(a => a.addEventListener("click", e => {
|
||||
e.preventDefault();
|
||||
const id = a.getAttribute("href").slice(1);
|
||||
show(id);
|
||||
history.replaceState(null, "", "#" + id);
|
||||
}));
|
||||
if(location.hash && document.getElementById(location.hash.slice(1))) show(location.hash.slice(1));
|
||||
|
||||
/* ---------- dashboard ---------- */
|
||||
(function(){
|
||||
const rows = FMT.map(f => {
|
||||
const g150 = geomean(WALL[f]);
|
||||
const g600 = geomean(PROJ[600][f].map(x => x[0]));
|
||||
return [NAME[f], "geometric mean of Q1-Q7", g150, fmtTime(g150) + ' <span class="muted">(' + fmtTime(g600) + ' @600 GB)</span>', COLOR[f]];
|
||||
});
|
||||
const max = Math.max(...rows.map(r => r[2]));
|
||||
el("dash-geo").innerHTML = rows.map(r => barRow(r[0], r[1], r[2]/max, r[4], r[3])).join("");
|
||||
|
||||
const breaks = {
|
||||
csv:["Direct path reads; canonical archive; best compression","Full scans pass 1 h just above 600 GB"],
|
||||
json:["Self-describing exchange files; hybrid variant is 1.9 MB","Every scan reads the whole corpus: hours at 600 GB"],
|
||||
sqlite:["Fastest at 150 MB; one file; smallest footprint","Single-threaded scans pass 1 h at ~1.2 TB"],
|
||||
pg:["Parallel partitioned scans + matviews; flat 0.2-0.4 s today","None found up to 6 TB (Q5 = 37 min)"],
|
||||
rdf:["Native semantics, SPARQL","11.7x disk, 20% hot set in RAM, scans measured in days"]
|
||||
};
|
||||
el("dash-breaks").innerHTML = FMT.map(f => {
|
||||
const fl = PROJ[600][f];
|
||||
const nI = fl.filter(x => x[1]==="I").length, nF = fl.filter(x => x[1]==="F").length;
|
||||
let flags = (nI+nF)===0 ? '<span class="pill ok">all 7 OK</span>' :
|
||||
(nI ? '<span class="pill imp">' + nI + ' x >1 h</span> ' : '') + (nF ? '<span class="pill fail">' + nF + ' x FAIL</span>' : '');
|
||||
return "<tr><td><b>" + NAME[f] + "</b></td><td>" + breaks[f][0] + "</td><td>" + breaks[f][1] + "</td><td>" + flags + "</td></tr>";
|
||||
}).join("");
|
||||
})();
|
||||
|
||||
/* ---------- methodology ---------- */
|
||||
el("method-queries").innerHTML = QN.map(q =>
|
||||
'<tr><td class="mono">Q' + q + '</td><td>' + QDESC[q][0] + '</td><td class="num">' + QDESC[q][1].toLocaleString("en-US") + '</td></tr>').join("");
|
||||
el("method-env").innerHTML = ENVIRONMENT;
|
||||
|
||||
/* ---------- storage ---------- */
|
||||
(function(){
|
||||
const workMax = WORK.rdf;
|
||||
el("st-work").innerHTML = FMT.map(f =>
|
||||
barRow(NAME[f], COEFF[f].toFixed(2) + "x vs raw CSV", WORK[f]/workMax, COLOR[f], fmtSize(WORK[f]))).join("");
|
||||
const archMax = ARCH.rdf[0];
|
||||
el("st-arch").innerHTML = FMT.map(f =>
|
||||
barRow(NAME[f], ARCH[f][1], ARCH[f][0]/archMax, COLOR[f], fmtSize(ARCH[f][0]))).join("");
|
||||
el("st-proj").innerHTML = FMT.map(f =>
|
||||
"<tr><td><b>" + NAME[f] + "</b></td>" + [600,1200,6000].map(s =>
|
||||
'<td class="num">' + fmtSize(COEFF[f]*s*1e9) + "</td>").join("") + "</tr>").join("");
|
||||
})();
|
||||
|
||||
/* ---------- measured results ---------- */
|
||||
(function(){
|
||||
const METRICS = [
|
||||
["Wall time", WALL, fmtTime],
|
||||
["Client peak RSS", RSS, v => v.toFixed(0) + " MB"],
|
||||
["Read bytes", READ, v => v >= 1000 ? (v/1000).toFixed(2) + " GB" : v.toFixed(1) + " MB"]
|
||||
];
|
||||
let q = 0, m = 0;
|
||||
function render(){
|
||||
const [mname, data, fmt] = METRICS[m];
|
||||
el("m-title").textContent = "Q" + QN[q] + " - " + QDESC[QN[q]][0] + " - " + mname.toLowerCase() + " (median of 3 runs)";
|
||||
const max = Math.max(...FMT.map(f => data[f][q])) || 1;
|
||||
el("m-bars").innerHTML = FMT.map(f =>
|
||||
barRow(NAME[f], "", (data[f][q])/max, COLOR[f], fmt(data[f][q]))).join("");
|
||||
}
|
||||
tabs("mq-tabs", QN.map(n => "Q" + n), i => { q = i; render(); });
|
||||
tabs("mm-tabs", METRICS.map(x => x[0]), i => { m = i; render(); });
|
||||
render();
|
||||
|
||||
el("m-mat-h").innerHTML = "<tr><th>Format</th>" + QN.map(n => '<th class="num">Q' + n + "</th>").join("") + "</tr>";
|
||||
el("m-mat").innerHTML = FMT.map(f =>
|
||||
"<tr><td><b>" + NAME[f] + "</b></td>" + WALL[f].map(v => '<td class="num">' + fmtTime(v) + "</td>").join("") + "</tr>").join("");
|
||||
|
||||
el("m-pgsrv").innerHTML = QN.map(n =>
|
||||
'<tr><td class="mono">Q' + n + "</td><td class=\"num\">" + PGSRV[n][0].toFixed(2) + " ms</td><td class=\"num\">" +
|
||||
PGSRV[n][1].toLocaleString("en-US") + '</td><td class="num">0</td></tr>').join("");
|
||||
})();
|
||||
|
||||
/* ---------- projections ---------- */
|
||||
(function(){
|
||||
const SCALES = [600, 1200, 6000];
|
||||
const SLBL = {600:"600 GB", 1200:"1.2 TB", 6000:"6 TB"};
|
||||
function render(scale){
|
||||
el("p-title").textContent = "Projected wall times at " + SLBL[scale] + " of raw data";
|
||||
el("p-mat-h").innerHTML = "<tr><th>Format</th>" + QN.map(n => '<th class="num">Q' + n + "</th>").join("") + "</tr>";
|
||||
el("p-mat").innerHTML = FMT.map(f =>
|
||||
"<tr><td><b>" + NAME[f] + "</b></td>" + PROJ[scale][f].map(x =>
|
||||
'<td class="num">' + fmtTime(x[0]) + flagPill(x[1]) + "</td>").join("") + "</tr>").join("");
|
||||
const total = FMT.reduce((acc,f) => {
|
||||
PROJ[scale][f].forEach(x => { if(x[1]==="I") acc.i++; if(x[1]==="F") acc.f++; });
|
||||
return acc;
|
||||
}, {i:0, f:0});
|
||||
el("p-note").textContent = "Flags at this scale: " + total.i + " x IMPRACTICAL (over 1 hour), " + total.f +
|
||||
" x FAIL (over 24 hours), " + (35 - total.i - total.f) + " x OK. Scaling ratio vs the measured corpus: x" +
|
||||
Math.round(scale*1e9/WORK.csv).toLocaleString("en-US") + ".";
|
||||
}
|
||||
tabs("p-tabs", SCALES.map(s => SLBL[s]), i => render(SCALES[i]));
|
||||
render(600);
|
||||
|
||||
/* degradation mini-table: full scan class geometric mean */
|
||||
const cols = [["measured 150 MB", f => geomean([WALL[f][4], WALL[f][6]])]]
|
||||
.concat(SCALES.map(s => ["projected " + SLBL[s], f => geomean([PROJ[s][f][4][0], PROJ[s][f][6][0]])]));
|
||||
let html = '<table class="data"><thead><tr><th>Format</th>' +
|
||||
cols.map(c => '<th class="num">' + c[0] + "</th>").join("") + "</tr></thead><tbody>";
|
||||
html += FMT.map(f => "<tr><td><b>" + NAME[f] + "</b></td>" +
|
||||
cols.map(c => '<td class="num">' + fmtTime(c[1](f)) + "</td>").join("") + "</tr>").join("");
|
||||
html += "</tbody></table>";
|
||||
el("p-degr").innerHTML = html;
|
||||
})();
|
||||
|
||||
/* ---------- hardware ---------- */
|
||||
(function(){
|
||||
const SCALES = [600, 1200, 6000];
|
||||
const SLBL = {600:"600 GB", 1200:"1.2 TB", 6000:"6 TB"};
|
||||
function config(h){
|
||||
return h.n + " x 1U Supermicro AS-1015CS-TNR (" + h.c + " cores) · " +
|
||||
h.m + " x 64 GB DDR5 ECC RDIMM · " + h.d + " x 7.68 TB NVMe U.2";
|
||||
}
|
||||
function render(scale){
|
||||
el("h-title").textContent = "Exact configuration and cost breakdown at " + SLBL[scale] + " (projected sizing)";
|
||||
el("h-bom").innerHTML = FMT.map(f => {
|
||||
const h = HW[scale][f];
|
||||
return "<tr><td><b>" + NAME[f] + "</b></td><td>" + config(h) + "</td>" +
|
||||
[h.base, h.ram, h.disk, h.extra, h.total].map((v,i) =>
|
||||
'<td class="num"' + (i===4 ? ' style="font-weight:600"' : '') + ">" + fmtMoney(v) + "</td>").join("") + "</tr>";
|
||||
}).join("");
|
||||
const max = Math.max(...FMT.map(f => HW[scale][f].total));
|
||||
el("h-bars").innerHTML = FMT.map(f => {
|
||||
const h = HW[scale][f];
|
||||
return barRow(NAME[f], h.n > 1 ? h.n + " nodes" : "single node", h.total/max, COLOR[f], fmtMoney(h.total));
|
||||
}).join("");
|
||||
scatter(scale);
|
||||
}
|
||||
function scatter(scale){
|
||||
const W = 860, H = 340, L = 70, R = 30, T = 20, B = 46;
|
||||
const pts = FMT.map(f => ({f, cost: HW[scale][f].total, geo: geomean(PROJ[scale][f].map(x => x[0]))}));
|
||||
const maxC = Math.max(...pts.map(p => p.cost)) * 1.15;
|
||||
const maxG = Math.max(...pts.map(p => p.geo)) * 1.15;
|
||||
const X = c => L + (W-L-R) * c / maxC;
|
||||
const Y = g => T + (H-T-B) * (1 - g / maxG);
|
||||
let s = '<svg viewBox="0 0 ' + W + ' ' + H + '" style="width:100%;max-width:900px">';
|
||||
/* axes + linear ticks in real units */
|
||||
for(let i = 1; i <= 4; i++){
|
||||
const c = maxC * i / 4, x = X(c);
|
||||
s += '<line x1="' + x + '" y1="' + T + '" x2="' + x + '" y2="' + (H-B) + '" stroke="#eef0f2"/>';
|
||||
s += '<text x="' + x + '" y="' + (H-B+18) + '" font-size="11" fill="#838a95" text-anchor="middle">$' + Math.round(c/1000) + 'k</text>';
|
||||
const g = maxG * i / 4, y = Y(g);
|
||||
s += '<line x1="' + L + '" y1="' + y + '" x2="' + (W-R) + '" y2="' + y + '" stroke="#eef0f2"/>';
|
||||
s += '<text x="' + (L-8) + '" y="' + (y+4) + '" font-size="11" fill="#838a95" text-anchor="end">' + fmtTime(g) + '</text>';
|
||||
}
|
||||
s += '<line x1="' + L + '" y1="' + (H-B) + '" x2="' + (W-R) + '" y2="' + (H-B) + '" stroke="#cfd4da"/>';
|
||||
s += '<line x1="' + L + '" y1="' + T + '" x2="' + L + '" y2="' + (H-B) + '" stroke="#cfd4da"/>';
|
||||
s += '<text x="' + ((L+W-R)/2) + '" y="' + (H-6) + '" font-size="11.5" fill="#5b6472" text-anchor="middle">estimated hardware cost (bill of materials, linear)</text>';
|
||||
s += '<text x="14" y="' + ((T+H-B)/2) + '" font-size="11.5" fill="#5b6472" text-anchor="middle" transform="rotate(-90 14 ' + ((T+H-B)/2) + ')">typical query time, projected (linear)</text>';
|
||||
/* better = lower-left cue */
|
||||
s += '<text x="' + (L+16) + '" y="' + (H-B-10) + '" font-size="11.5" fill="#0f7a45" font-weight="600">better: lower-left</text>';
|
||||
const OFF = {csv:[12,-8], json:[12,4], sqlite:[12,14], pg:[12,-6], rdf:[-8,-12]};
|
||||
const ANCH = {csv:"start", json:"start", sqlite:"start", pg:"start", rdf:"end"};
|
||||
pts.forEach(p => {
|
||||
s += '<circle cx="' + X(p.cost) + '" cy="' + Y(p.geo) + '" r="7" fill="' + COLOR[p.f] + '"/>';
|
||||
s += '<text x="' + (X(p.cost)+OFF[p.f][0]) + '" y="' + (Y(p.geo)+OFF[p.f][1]) + '" font-size="11.5" fill="#3d424b" text-anchor="' + ANCH[p.f] + '">' +
|
||||
NAME[p.f] + ': ' + fmtMoney(p.cost) + ', ' + fmtTime(p.geo) + '</text>';
|
||||
});
|
||||
s += "</svg>";
|
||||
el("h-scatter").innerHTML = s;
|
||||
}
|
||||
tabs("h-tabs", SCALES.map(s => SLBL[s]), i => render(SCALES[i]));
|
||||
render(600);
|
||||
})();
|
||||
|
||||
/* ---------- scoring ---------- */
|
||||
function renderScores(containerId, board){
|
||||
const order = FMT.slice().sort((a,b) => board[b][3] - board[a][3]);
|
||||
el(containerId).innerHTML = order.map(f => {
|
||||
const [se, ra, di, total] = board[f];
|
||||
return '<div class="stack"><div class="lab">' + NAME[f] + '</div><div class="trk">' +
|
||||
'<div class="seg" style="width:' + (se/80*100) + '%;background:#1f5fdb"></div>' +
|
||||
'<div class="seg" style="width:' + (ra/80*100) + '%;background:#e0892e"></div>' +
|
||||
'<div class="seg" style="width:' + (di/80*100) + '%;background:#0f9d8a"></div>' +
|
||||
'</div><div class="val">' + total.toFixed(1) + '</div></div>';
|
||||
}).join("");
|
||||
}
|
||||
renderScores("sc-measured", SCORES.measured);
|
||||
renderScores("sc-proj", SCORES.proj);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -31,10 +31,14 @@ out/
|
||||
- Tasks communicate exclusively through these artifacts - never through
|
||||
shared in-process state
|
||||
(see [code-python-style.md](code-python-style.md)).
|
||||
- **`./out/` is git-ignored** (except nothing - the whole tree). Sources
|
||||
of truth are the specs, the rules, and the code; artifacts are
|
||||
reproducible from them ([data-determinism.md](data-determinism.md)).
|
||||
Never commit generated artifacts "for convenience".
|
||||
- **The bulk of `./out/` is git-ignored.** Sources of truth are the
|
||||
specs, the rules, and the code; artifacts are reproducible from them
|
||||
([data-determinism.md](data-determinism.md)). A curated results subset
|
||||
IS committed (owner decision, 2026-07-11): `out/report/` (REPORT.md,
|
||||
charts, tables, diagrams), `out/config/` (lab_config.yaml,
|
||||
hw_prices.yaml), and the `out/.done/` markers - exactly the exceptions
|
||||
listed in `.gitignore`. Never commit the bulk artifacts (corpus,
|
||||
databases, stores, archives, raw benchmark data) "for convenience".
|
||||
|
||||
## 2. Completion markers
|
||||
|
||||
@@ -92,7 +96,9 @@ out/
|
||||
|
||||
- **Writing an artifact outside `./out/`** or a task writing into another
|
||||
task's output directory.
|
||||
- **Committing `./out/` content to git.**
|
||||
- **Committing bulk `./out/` content to git** (corpus, databases, stores,
|
||||
archives, raw benchmark data) - only the curated subset in `.gitignore`
|
||||
(`report/`, `config/`, `.done/`) is committed.
|
||||
- **A marker written on partial success** - see
|
||||
[code-error-handling.md](code-error-handling.md).
|
||||
- **Running benchmarks while converters are still running.**
|
||||
|
||||
@@ -78,8 +78,6 @@ an en-dash.
|
||||
content, not typographic substitutions. UTF-8 throughout is the right
|
||||
answer. (Repository artifacts are English; this matters only for quoted
|
||||
material.)
|
||||
- **Imported reference materials** in `docs/examples/` are kept verbatim as
|
||||
received; they are not binding documents.
|
||||
|
||||
## 4. Detection / enforcement
|
||||
|
||||
@@ -89,8 +87,7 @@ future detector is documented here so it can be built consistently:
|
||||
|
||||
- **What it flags.** Any occurrence of the Unicode codepoints in the section 1
|
||||
substitution table inside in-scope files. The section 3 exemptions
|
||||
(scientific notation, international content, `docs/examples/`) are the only
|
||||
carve-outs.
|
||||
(scientific notation, international content) are the only carve-outs.
|
||||
- **Exit behavior.** Non-zero exit when any flagged codepoint is found under
|
||||
the scan roots, so it can later gate a commit hook or a build step.
|
||||
- **Tooling.** When this detector is created it MUST be a PowerShell script
|
||||
|
||||
@@ -30,7 +30,7 @@ The stdlib is the default. The ONLY permitted third-party packages:
|
||||
| Package | Used by | For |
|
||||
|---|---|---|
|
||||
| `numpy` | 03 | vectorized generation, seeded RNG |
|
||||
| `PyYAML` | 01, 03, 10 | reading/writing the YAML configs |
|
||||
| `PyYAML` | 01, 03, 10, 11 | reading/writing the YAML configs |
|
||||
| `rdflib` | 04, 07 | JSON-LD validation, RDF serialization |
|
||||
| `oxigraph` | 07, 09 | embedded queryable triplestore |
|
||||
| `ijson` | 08, 09 | streaming JSON parsing |
|
||||
|
||||
@@ -1,27 +0,0 @@
|
||||
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
|
||||
|
||||
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
|
||||
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
|
||||
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
|
||||
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
|
||||
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
|
||||
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
|
||||
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
|
||||
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
|
||||
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
|
||||
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
|
||||
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
|
||||
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
|
||||
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
|
||||
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
|
||||
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
|
||||
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
|
||||
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
|
||||
|
||||
## Final Goal
|
||||
|
||||
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.
|
||||
@@ -1,26 +0,0 @@
|
||||
# Task 01 — Setup Workspace
|
||||
|
||||
## Goal
|
||||
Create the working directory layout and versioned pipeline configuration.
|
||||
|
||||
## Input
|
||||
- Path to the raw CSV root (read-only). Ask the user if unknown.
|
||||
|
||||
## Actions
|
||||
1. Create:
|
||||
```
|
||||
work/
|
||||
config/ # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
|
||||
manifest/ # inventory, fingerprints, reports
|
||||
canonical/ # cleansed partitioned dataset
|
||||
rejects/ # quarantined anomalies
|
||||
schema/ # DDL, data dictionary, instruments repository
|
||||
exports/{json,sqlite,postgresql,rdf}/
|
||||
lineage/
|
||||
scripts/ # all Python scripts produced by tasks
|
||||
```
|
||||
2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
|
||||
3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 02 — Filesystem Inventory
|
||||
|
||||
## Goal
|
||||
Non-destructive inventory of every file under `SOURCE_ROOT`.
|
||||
|
||||
## Input
|
||||
- `work/config/pipeline.json` (Task 01).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
|
||||
2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
|
||||
3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
|
||||
4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `inventory.jsonl` row count == filesystem file count.
|
||||
- `inventory_summary.md` produced; non-CSV extensions listed explicitly.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 03 — Filename & Path Tokenization
|
||||
|
||||
## Goal
|
||||
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl` (Task 02).
|
||||
- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
|
||||
|
||||
## Actions
|
||||
1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
|
||||
2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
|
||||
3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
|
||||
- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 04 — Schema Profiling & Fingerprints
|
||||
|
||||
## Goal
|
||||
Capture the record structure of every CSV without full load; compute schema fingerprints.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
|
||||
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
|
||||
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
|
||||
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
|
||||
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
|
||||
- Every profile carries a fingerprint.
|
||||
@@ -1,16 +0,0 @@
|
||||
# Task 05 — Schema Grouping & Inconsistency Report
|
||||
|
||||
## Goal
|
||||
Enumerate distinct record structures per measurement type; expose structural inconsistencies.
|
||||
|
||||
## Input
|
||||
- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
|
||||
2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
|
||||
3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- The number of distinct schemas per measurement type is known and documented.
|
||||
- `inconsistencies.csv` complete — this is the work queue for Task 07.
|
||||
@@ -1,19 +0,0 @@
|
||||
# Task 06 — Storage Map (As-Is / As-Designed)
|
||||
|
||||
## Goal
|
||||
Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
|
||||
|
||||
## Input
|
||||
- All Task 02–05 artifacts.
|
||||
|
||||
## Actions
|
||||
1. Write `manifest/storage_map.md`:
|
||||
- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
|
||||
- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
|
||||
- purpose of each file kind mapped to measurement types and attachment levels;
|
||||
- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
|
||||
2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
|
||||
- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 07 — Column Mapping (raw → canonical)
|
||||
|
||||
## Goal
|
||||
For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
|
||||
|
||||
## Input
|
||||
- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
|
||||
- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
|
||||
|
||||
## Actions
|
||||
1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
|
||||
2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
|
||||
3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
|
||||
4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
|
||||
- Escalation list for Task 08 written to `schema/ontology_escalations.md`.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
|
||||
|
||||
## Goal
|
||||
Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
|
||||
|
||||
## Input
|
||||
- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
|
||||
|
||||
## Actions
|
||||
1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
|
||||
2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
|
||||
3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
|
||||
4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
|
||||
5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
|
||||
- No open items in `ontology_escalations.md`.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 09 — Cleanse & Normalize → Canonical Dataset
|
||||
|
||||
## Goal
|
||||
Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
|
||||
|
||||
## Input
|
||||
- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
|
||||
2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
|
||||
3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
|
||||
4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
|
||||
5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
|
||||
- Canonical partitions written; lineage present for every canonical row.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 10 — Anomaly Report & Iteration Gate
|
||||
|
||||
## Goal
|
||||
Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
|
||||
|
||||
## Input
|
||||
- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
|
||||
2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
|
||||
3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
|
||||
4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `anomaly_report.md` final; every reject class documented as a true anomaly.
|
||||
- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 11 — Export: JSON
|
||||
|
||||
## Goal
|
||||
Two JSON products from the frozen canonical snapshot.
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/` (Tasks 09–10).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
|
||||
2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
|
||||
3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
|
||||
4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Both products written; spot-validation: point counts per measurement match canonical.
|
||||
- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).
|
||||
@@ -1,16 +0,0 @@
|
||||
# Task 12 — Export: SQLite
|
||||
|
||||
## Goal
|
||||
Portable single-file SQLite database conforming to the ontology.
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
|
||||
2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
|
||||
3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
|
||||
- `exports/sqlite/EXPORT_MANIFEST.json` written.
|
||||
@@ -1,22 +0,0 @@
|
||||
# Task 13 — Export: PostgreSQL
|
||||
|
||||
## Goal
|
||||
System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
|
||||
|
||||
## Actions
|
||||
1. Generate `exports/postgresql/`:
|
||||
- `01_schema.sql` (DDL, versioned);
|
||||
- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
|
||||
- per-table CSV staging files from canonical (streamed);
|
||||
- `03_copy_load.sql` (`\copy` statements in dependency order);
|
||||
- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
|
||||
- `load.sh` / `load.ps1` orchestration.
|
||||
2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
|
||||
3. Include `lineage` table load (row-range granularity).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
|
||||
- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 14 — Export: RDF (Metadata & Provenance Graph)
|
||||
|
||||
## Goal
|
||||
RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
|
||||
|
||||
## Actions
|
||||
1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
|
||||
2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
|
||||
3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
|
||||
4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
|
||||
- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
|
||||
|
||||
## Goal
|
||||
Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
|
||||
|
||||
## Input
|
||||
- All `exports/*`, `lineage/`, all config/schema versions.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/finalize.py`:
|
||||
- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
|
||||
- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
|
||||
2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
|
||||
3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
|
||||
|
||||
## Final Result (pipeline done when)
|
||||
- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
|
||||
- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.
|
||||
@@ -79,15 +79,29 @@ LAWS = {
|
||||
("csv", 7): "scan", ("json", 7): "scan", ("sqlite", 7): "scan", ("pg", 7): "scan_parallel", ("rdf", 7): "scan",
|
||||
}
|
||||
|
||||
HW_PRICES_DEFAULT = """# Hardware cost parameters for extrapolation (task 10) - June 2026 defaults.
|
||||
HW_PRICES_DEFAULT = """# Hardware cost parameters for extrapolation (task 10).
|
||||
# Street prices retrieved 2026-07-11 from the sources below (server DRAM is
|
||||
# in a documented 2025-2026 price surge - see market_context). The cost
|
||||
# model buys whole components: 64 GB RDIMM modules and 7.68 TB NVMe drives.
|
||||
# Operator-editable; re-run extrapolate.py after changes (code-config-yaml).
|
||||
ram_usd_per_gb: 14.0
|
||||
nvme_usd_per_tb: 250.0
|
||||
server_16_core_usd: 6000.0
|
||||
server_32_core_usd: 10000.0
|
||||
extra_node_usd: 6000.0
|
||||
as_of: "2026-07-11"
|
||||
ram_module_gb: 64
|
||||
ram_module_usd: 1887.0 # A-Tech 64GB (2x32GB) DDR5-5600 ECC RDIMM
|
||||
nvme_drive_tb: 7.68
|
||||
nvme_drive_usd: 3995.0 # Cloud Ninjas NEW 7.68TB NVMe U.2 1DWPD (Dell 14-16G)
|
||||
server_16_core_usd: 4618.0 # Supermicro AS-1015CS-TNR 1U (EPYC 9004/9005), Broadberry starting config
|
||||
server_32_core_usd: 6272.0 # Supermicro AS-1115CS-TNR 1U, Broadberry starting config
|
||||
extra_node_usd: 4618.0 # one additional entry 1U node (AS-1015CS-TNR chassis)
|
||||
sources:
|
||||
ram: https://atechmemory.com/collections/ddr5-memory-ram
|
||||
nvme: https://cloudninjas.com/products/new-7-68tb-nvme-u-2-1dwpd-sie-2-5-enterprise-solid-state-drive-for-14th-15th-16th-gen-dell
|
||||
servers: https://www.broadberry.com/amd-epyc-9004-supermicro-servers
|
||||
market_context: https://www.techpowerup.com/342331/server-dram-pricing-jumps-50-only-70-of-orders-getting-filled
|
||||
"""
|
||||
HW_PRICE_KEYS = ("ram_usd_per_gb", "nvme_usd_per_tb", "server_16_core_usd", "server_32_core_usd", "extra_node_usd")
|
||||
HW_PRICE_KEYS = (
|
||||
"as_of", "ram_module_gb", "ram_module_usd", "nvme_drive_tb", "nvme_drive_usd",
|
||||
"server_16_core_usd", "server_32_core_usd", "extra_node_usd", "sources",
|
||||
)
|
||||
|
||||
|
||||
def read_medians(path: Path) -> dict[tuple[str, int], float]:
|
||||
@@ -222,7 +236,8 @@ def main() -> int:
|
||||
|
||||
with open(bench_dir / "hardware_sizing.csv", "w", encoding="ascii", newline="") as fh:
|
||||
writer = csv.writer(fh)
|
||||
writer.writerow(["format", "scale_gb", "disk_tb", "ram_gb", "cores", "nodes", "est_cost_usd"])
|
||||
writer.writerow(["format", "scale_gb", "disk_tb", "ram_gb", "cores", "nodes",
|
||||
"est_cost_usd", "ram_modules", "nvme_drives", "base_usd", "ram_usd", "disk_usd", "extra_nodes_usd"])
|
||||
for scale_gb in SCALES_GB:
|
||||
r = scale_gb * 1e9 / csv_bytes
|
||||
for fmt in FORMATS:
|
||||
@@ -238,11 +253,17 @@ def main() -> int:
|
||||
ram_gb = max(MIN_RAM_GB, RDF_HOT_FRACTION * fmt_bytes / 1e9)
|
||||
cores = pick_pg_cores(medians, overhead, r, n0) if fmt == "pg" else CORE_OPTIONS[0]
|
||||
nodes = max(1, math.ceil(ram_gb / NODE_RAM_CEILING_GB))
|
||||
server_usd = prices["server_16_core_usd"] if cores == 16 else prices["server_32_core_usd"]
|
||||
cost = (server_usd + ram_gb * prices["ram_usd_per_gb"]
|
||||
+ disk_tb * prices["nvme_usd_per_tb"]
|
||||
+ (nodes - 1) * prices["extra_node_usd"])
|
||||
writer.writerow([fmt, scale_gb, round(disk_tb, 3), round(ram_gb, 1), cores, nodes, round(cost)])
|
||||
# bill of materials: whole RDIMM modules and whole NVMe drives
|
||||
ram_modules = math.ceil(ram_gb / prices["ram_module_gb"])
|
||||
nvme_drives = math.ceil(disk_tb / prices["nvme_drive_tb"])
|
||||
base_usd = prices["server_16_core_usd"] if cores == 16 else prices["server_32_core_usd"]
|
||||
ram_usd = ram_modules * prices["ram_module_usd"]
|
||||
disk_usd = nvme_drives * prices["nvme_drive_usd"]
|
||||
extra_nodes_usd = (nodes - 1) * prices["extra_node_usd"]
|
||||
cost = base_usd + ram_usd + disk_usd + extra_nodes_usd
|
||||
writer.writerow([fmt, scale_gb, round(disk_tb, 3), round(ram_gb, 1), cores, nodes,
|
||||
round(cost), ram_modules, nvme_drives,
|
||||
round(base_usd), round(ram_usd), round(disk_usd), round(extra_nodes_usd)])
|
||||
|
||||
entries = {
|
||||
"extrapolation_rows": len(SCALES_GB) * len(FORMATS) * len(QUERY_NUMBERS),
|
||||
@@ -252,6 +273,7 @@ def main() -> int:
|
||||
"flags_fail": flags["FAIL"],
|
||||
"pg_index_share": round(index_share, 3),
|
||||
"pg_index_mb": round(pg_index_bytes / 1048576, 1),
|
||||
"prices_as_of": prices["as_of"],
|
||||
"note_pandas": "pandas CSV variant not measured; its RAM is O(n) - infeasible above ~10 GB (spec 10)",
|
||||
}
|
||||
for fmt in FORMATS:
|
||||
|
||||
13
out/.done/01.ok
Normal file
@@ -0,0 +1,13 @@
|
||||
task: '01'
|
||||
status: ok
|
||||
config_file: C:/Projects/Public/LabDataStorageEvaluation/out/config/lab_config.yaml
|
||||
config_bytes: 5827
|
||||
seed: 20260711
|
||||
batches: 4
|
||||
wafers: 12
|
||||
coupons: 588
|
||||
friction_coupons: 480
|
||||
tracks: 1440
|
||||
cycle_rows: 1440000
|
||||
loop_points: 2880000
|
||||
xrf_points: 235200
|
||||
6
out/.done/02.ok
Normal file
@@ -0,0 +1,6 @@
|
||||
task: '02'
|
||||
status: ok
|
||||
process_flow_file: C:/Projects/Public/LabDataStorageEvaluation/out/report/process_flow.md
|
||||
process_flow_bytes: 4205
|
||||
diagrams: 6
|
||||
diagram_bytes: 2924
|
||||
11
out/.done/03.ok
Normal file
@@ -0,0 +1,11 @@
|
||||
task: '03'
|
||||
status: ok
|
||||
csv_root: C:/Projects/Public/LabDataStorageEvaluation/out/csv
|
||||
files: 8718
|
||||
total_rows: 4636776
|
||||
total_bytes: 148654472
|
||||
coupons: 588
|
||||
tracks: 1440
|
||||
cycle_rows: 1440000
|
||||
loop_points: 2880000
|
||||
xrf_points: 235200
|
||||
11
out/.done/04.ok
Normal file
@@ -0,0 +1,11 @@
|
||||
task: '04'
|
||||
status: ok
|
||||
full_files: 588
|
||||
full_bytes: 311300222
|
||||
hybrid_bytes: 1887322
|
||||
coupons: 588
|
||||
tracks: 1440
|
||||
cycle_rows: 1440000
|
||||
loop_points: 2880000
|
||||
proc_peak_rss_mb: 119.1
|
||||
proc_cpu_seconds: 19.1
|
||||
12
out/.done/05.ok
Normal file
@@ -0,0 +1,12 @@
|
||||
task: '05'
|
||||
status: ok
|
||||
db_file: C:/Projects/Public/LabDataStorageEvaluation/out/sqlite/tribo.db
|
||||
db_bytes: 123400192
|
||||
tables: 15
|
||||
total_rows: 4638223
|
||||
friction_cycles: 1440000
|
||||
friction_loop_points: 2880000
|
||||
tracks: 1440
|
||||
track_summary: 1440
|
||||
proc_peak_rss_mb: 323.0
|
||||
proc_cpu_seconds: 19.9
|
||||
18
out/.done/06.ok
Normal file
@@ -0,0 +1,18 @@
|
||||
task: '06'
|
||||
status: ok
|
||||
database: lab_data
|
||||
tables: 15
|
||||
total_rows: 4638223
|
||||
live_bytes: 397467648
|
||||
friction_cycles_bytes: 99753984
|
||||
friction_loop_points_bytes: 267796480
|
||||
copy_seconds: 19.8
|
||||
total_seconds: 23.0
|
||||
dump: skipped (pg_dump unavailable on this host)
|
||||
proc_peak_rss_mb: 58.6
|
||||
proc_cpu_seconds: 18.7
|
||||
host_cpu_avg_pct: 2.4
|
||||
host_cpu_max_pct: 3.7
|
||||
host_ram_used_baseline_mb: 617
|
||||
host_ram_used_peak_mb: 681
|
||||
host_ram_used_delta_mb: 64
|
||||
13
out/.done/07.ok
Normal file
@@ -0,0 +1,13 @@
|
||||
task: '07'
|
||||
status: ok
|
||||
triples: 17419713
|
||||
nt_gz_bytes: 126487650
|
||||
ttl_bytes: 596401229
|
||||
store_bytes: 1739414085
|
||||
store_load_seconds: 33.6
|
||||
coupons: 588
|
||||
tracks: 1440
|
||||
cycle_nodes: 1440000
|
||||
loop_nodes: 2880000
|
||||
proc_peak_rss_mb: 2185.2
|
||||
proc_cpu_seconds: 151.8
|
||||
19
out/.done/08.ok
Normal file
@@ -0,0 +1,19 @@
|
||||
task: '08'
|
||||
status: ok
|
||||
query_files: 56
|
||||
q1_rows: 1000
|
||||
q1_sha256: bb6d871275d0
|
||||
q2_rows: 33
|
||||
q2_sha256: 62ca30e729a8
|
||||
q3_rows: 480
|
||||
q3_sha256: a3a73414201f
|
||||
q4_rows: 469
|
||||
q4_sha256: 38281dcf7896
|
||||
q5_rows: 4
|
||||
q5_sha256: f3ee32dbc981
|
||||
q6_rows: 480
|
||||
q6_sha256: 9cc28d51ee35
|
||||
q7_rows: 2
|
||||
q7_sha256: 1991f8bf2217
|
||||
proc_peak_rss_mb: 61.1
|
||||
proc_cpu_seconds: 1.7
|
||||
14
out/.done/09.ok
Normal file
@@ -0,0 +1,14 @@
|
||||
task: '09'
|
||||
status: ok
|
||||
cells_total: 35
|
||||
cells_ok: 35
|
||||
cells_invalid: 0
|
||||
cells_timeout: 0
|
||||
cells_error: 0
|
||||
session_wall_s: 915.5
|
||||
cold_cache_pass: skipped (Windows host)
|
||||
session_host_cpu_avg_pct: 0.4
|
||||
session_host_cpu_max_pct: 9.2
|
||||
session_host_ram_used_baseline_mb: 909
|
||||
session_host_ram_used_peak_mb: 914
|
||||
session_host_ram_used_delta_mb: 5
|
||||
16
out/.done/10.ok
Normal file
@@ -0,0 +1,16 @@
|
||||
task: '10'
|
||||
status: ok
|
||||
extrapolation_rows: 105
|
||||
sizing_rows: 15
|
||||
flags_ok: 53
|
||||
flags_impractical: 35
|
||||
flags_fail: 17
|
||||
pg_index_share: 0.341
|
||||
pg_index_mb: 128.5
|
||||
prices_as_of: 2026-07-11
|
||||
note_pandas: pandas CSV variant not measured; its RAM is O(n) - infeasible above ~10 GB (spec 10)
|
||||
coeff_csv: 1.0
|
||||
coeff_json: 2.094
|
||||
coeff_sqlite: 0.83
|
||||
coeff_pg: 2.674
|
||||
coeff_rdf: 11.701
|
||||
12
out/.done/11.ok
Normal file
@@ -0,0 +1,12 @@
|
||||
task: '11'
|
||||
status: ok
|
||||
charts: 14
|
||||
tables: 7
|
||||
report_bytes: 16091
|
||||
winner_600gb: pg
|
||||
prices_as_of: 2026-07-11
|
||||
score600_csv: 28.9
|
||||
score600_json: 24.0
|
||||
score600_sqlite: 47.0
|
||||
score600_pg: 58.6
|
||||
score600_rdf: 0.9
|
||||
18
out/config/hw_prices.yaml
Normal file
@@ -0,0 +1,18 @@
|
||||
# Hardware cost parameters for extrapolation (task 10).
|
||||
# Street prices retrieved 2026-07-11 from the sources below (server DRAM is
|
||||
# in a documented 2025-2026 price surge - see market_context). The cost
|
||||
# model buys whole components: 64 GB RDIMM modules and 7.68 TB NVMe drives.
|
||||
# Operator-editable; re-run extrapolate.py after changes (code-config-yaml).
|
||||
as_of: "2026-07-11"
|
||||
ram_module_gb: 64
|
||||
ram_module_usd: 1887.0 # A-Tech 64GB (2x32GB) DDR5-5600 ECC RDIMM
|
||||
nvme_drive_tb: 7.68
|
||||
nvme_drive_usd: 3995.0 # Cloud Ninjas NEW 7.68TB NVMe U.2 1DWPD (Dell 14-16G)
|
||||
server_16_core_usd: 4618.0 # Supermicro AS-1015CS-TNR 1U (EPYC 9004/9005), Broadberry starting config
|
||||
server_32_core_usd: 6272.0 # Supermicro AS-1115CS-TNR 1U, Broadberry starting config
|
||||
extra_node_usd: 4618.0 # one additional entry 1U node (AS-1015CS-TNR chassis)
|
||||
sources:
|
||||
ram: https://atechmemory.com/collections/ddr5-memory-ram
|
||||
nvme: https://cloudninjas.com/products/new-7-68tb-nvme-u-2-1dwpd-sie-2-5-enterprise-solid-state-drive-for-14th-15th-16th-gen-dell
|
||||
servers: https://www.broadberry.com/amd-epyc-9004-supermicro-servers
|
||||
market_context: https://www.techpowerup.com/342331/server-dram-pricing-jumps-50-only-70-of-orders-getting-filled
|
||||
246
out/config/lab_config.yaml
Normal file
@@ -0,0 +1,246 @@
|
||||
# Generated by make_lab_config.py (task 01) from docs/specs/01_lab_configuration.md.
|
||||
# Do not hand-edit: regenerate via `python make_lab_config.py`.
|
||||
project: LabDataStorageEvaluation
|
||||
description: Simulated tribology laboratory (Sandia Pt-Au LDRD context)
|
||||
spec: docs/specs/01_lab_configuration.md
|
||||
seed: 20260711
|
||||
material_system:
|
||||
substrate:
|
||||
material: Ti-6Al-4V
|
||||
dimensions_mm:
|
||||
- 10
|
||||
- 10
|
||||
- 3
|
||||
adhesion_layer:
|
||||
material: Cr
|
||||
process: sputtered
|
||||
coating:
|
||||
material: Pt-Au
|
||||
deposition: composition gradient across each wafer
|
||||
thickness_um_range:
|
||||
- 0.3
|
||||
- 1.1
|
||||
instruments:
|
||||
- instrument_id: rapid
|
||||
role: friction
|
||||
name: RAPID custom high-throughput parallelized 6-probe tribometer
|
||||
data_produced: COF vs cycle per track; test conditions
|
||||
- instrument_id: ti980
|
||||
role: nanoindentation
|
||||
name: Bruker TI980 TriboIndenter
|
||||
data_produced: hardness, reduced modulus (25 indents per coupon)
|
||||
- instrument_id: m4_tornado
|
||||
role: composition
|
||||
name: Bruker M4 Tornado micro-XRF
|
||||
data_produced: Pt/Au wt% map per coupon (20x20 grid)
|
||||
- instrument_id: pvd200
|
||||
role: deposition
|
||||
name: Kurt J. Lesker PVD 200 sputter-down
|
||||
data_produced: batch deposition parameters
|
||||
- instrument_id: afm
|
||||
role: surface_roughness
|
||||
name: AFM
|
||||
data_produced: topography image metadata + Ra/Rq per coupon
|
||||
- instrument_id: profilometer
|
||||
role: film_thickness
|
||||
name: Optical profilometry
|
||||
data_produced: thickness map per coupon (10x10 grid)
|
||||
- instrument_id: simtra
|
||||
role: simulation
|
||||
name: SIMTRA sputter transport Monte-Carlo
|
||||
data_produced: deposition atom-energy / composition profiles per deposition run
|
||||
deposition_matrix:
|
||||
- batch_code: B721
|
||||
pt_gun_tilt_deg: 20
|
||||
au_gun_tilt_deg: 0
|
||||
pt_power_W: 150
|
||||
au_power_W: 50
|
||||
pt_discharge_V: 432
|
||||
au_discharge_V: 311
|
||||
- batch_code: B722
|
||||
pt_gun_tilt_deg: 20
|
||||
au_gun_tilt_deg: 20
|
||||
pt_power_W: 100
|
||||
au_power_W: 100
|
||||
pt_discharge_V: 399
|
||||
au_discharge_V: 352
|
||||
- batch_code: B723
|
||||
pt_gun_tilt_deg: 20
|
||||
au_gun_tilt_deg: 20
|
||||
pt_power_W: 150
|
||||
au_power_W: 50
|
||||
pt_discharge_V: 430
|
||||
au_discharge_V: 311
|
||||
- batch_code: B724
|
||||
pt_gun_tilt_deg: 0
|
||||
au_gun_tilt_deg: 20
|
||||
pt_power_W: 50
|
||||
au_power_W: 150
|
||||
pt_discharge_V: 376
|
||||
au_discharge_V: 340
|
||||
hierarchy:
|
||||
batches: 4
|
||||
wafers_per_batch: 3
|
||||
coupon_grid_per_wafer:
|
||||
- 7
|
||||
- 7
|
||||
coupons_per_wafer: 49
|
||||
friction_assignment:
|
||||
friction_coupons_total: 480
|
||||
reserve_coupons_total: 108
|
||||
runs: 4
|
||||
coupons_per_run: 120
|
||||
plates_per_run: 5
|
||||
probes_per_plate: 6
|
||||
coupons_per_probe_square: 4
|
||||
tracks_per_friction_coupon: 3
|
||||
counterfaces_per_holder: 3
|
||||
fresh_counterface_per_track: true
|
||||
reserve_grid_positions:
|
||||
- 1
|
||||
- 4
|
||||
- 7
|
||||
- 22
|
||||
- 25
|
||||
- 28
|
||||
- 43
|
||||
- 46
|
||||
- 49
|
||||
volumes:
|
||||
coupons_total: 588
|
||||
tracks_total: 1440
|
||||
cycles_per_track: 1000
|
||||
cycle_rows_total: 1440000
|
||||
loop_every_n_cycles: 100
|
||||
loops_per_track: 10
|
||||
loop_points_per_loop: 200
|
||||
loop_points_total: 2880000
|
||||
xrf_grid:
|
||||
- 20
|
||||
- 20
|
||||
xrf_points_per_coupon: 400
|
||||
xrf_points_total: 235200
|
||||
profilometry_grid:
|
||||
- 10
|
||||
- 10
|
||||
profilometry_points_per_coupon: 100
|
||||
nanoindentation_indents_per_coupon: 25
|
||||
simtra_rows_per_batch: 1000
|
||||
runs:
|
||||
- run_code: R1
|
||||
batch_code: B721
|
||||
environment: lab_air
|
||||
date: '2026-04-06'
|
||||
operator: A. Reyes
|
||||
- run_code: R2
|
||||
batch_code: B722
|
||||
environment: lab_air
|
||||
date: '2026-04-13'
|
||||
operator: K. Patel
|
||||
- run_code: R3
|
||||
batch_code: B723
|
||||
environment: dry_n2
|
||||
date: '2026-04-20'
|
||||
operator: A. Reyes
|
||||
- run_code: R4
|
||||
batch_code: B724
|
||||
environment: dry_n2
|
||||
date: '2026-04-27'
|
||||
operator: M. Novak
|
||||
schedule:
|
||||
deposition_start_date: '2026-03-02'
|
||||
batch_interval_days: 7
|
||||
characterization_lag_days: 3
|
||||
run_start_time: 09:00:00
|
||||
track_interval_s: 2100
|
||||
test_conditions:
|
||||
normal_load_mN: 100
|
||||
stroke_mm: 1.0
|
||||
speed_mm_s: 1.0
|
||||
counterface:
|
||||
material: ruby (Al2O3)
|
||||
diameter_mm: 3.175
|
||||
rh_pct:
|
||||
lab_air: 45
|
||||
dry_n2: 2
|
||||
temperature_C: 23
|
||||
temperature_tolerance_C: 1
|
||||
physical_models:
|
||||
au_gradient:
|
||||
description: linear Au wt% gradient along wafer x, batch-dependent center
|
||||
batch_center_wtpct:
|
||||
B721: 8
|
||||
B722: 25
|
||||
B723: 10
|
||||
B724: 60
|
||||
wafer_span_wtpct: 5
|
||||
xrf_noise_sd_wtpct: 0.3
|
||||
cof_run_in:
|
||||
formula: cof(c) = cof_ss + (cof_0 - cof_ss) * exp(-c / tau) + N(0, sigma)
|
||||
cof_0_range:
|
||||
- 0.35
|
||||
- 0.5
|
||||
tau_cycles_range:
|
||||
- 100
|
||||
- 400
|
||||
noise_sd: 0.01
|
||||
cof_steady_state:
|
||||
formula: cof_ss = intercept + slope_per_au_wtpct * au_wtpct_mean, clamped to >= floor
|
||||
lab_air:
|
||||
intercept: 0.32
|
||||
slope_per_au_wtpct: -0.0015
|
||||
dry_n2:
|
||||
intercept: 0.24
|
||||
slope_per_au_wtpct: -0.0012
|
||||
floor: 0.12
|
||||
hardness_GPa:
|
||||
formula: H = intercept + slope_per_au_wtpct * au_wtpct_mean + N(0, noise_sd)
|
||||
intercept: 8.5
|
||||
slope_per_au_wtpct: -0.05
|
||||
noise_sd: 0.25
|
||||
reduced_modulus_GPa:
|
||||
formula: Er = intercept + slope_per_au_wtpct * au_wtpct_mean + N(0, noise_sd)
|
||||
intercept: 190
|
||||
slope_per_au_wtpct: -0.6
|
||||
noise_sd: 4
|
||||
roughness_ra_nm:
|
||||
distribution: lognormal
|
||||
median_nm: 5
|
||||
sigma_log: 0.3
|
||||
rq_over_ra: 1.25
|
||||
thickness_um:
|
||||
profile: radial parabolic across wafer plus noise
|
||||
center_um: 1.05
|
||||
edge_um: 0.35
|
||||
noise_sd_um: 0.02
|
||||
range_um:
|
||||
- 0.3
|
||||
- 1.1
|
||||
wear_archard:
|
||||
formula: V = k * F * s; k = k0 * (1 + slope_per_au_wtpct * au_wtpct_mean), lognormal noise, clamped
|
||||
to >= k_floor
|
||||
k0_mm3_per_N_m:
|
||||
lab_air: 1.5e-07
|
||||
dry_n2: 6.0e-08
|
||||
slope_per_au_wtpct:
|
||||
lab_air: -0.008
|
||||
dry_n2: -0.01
|
||||
k_floor_mm3_per_N_m: 1.0e-09
|
||||
noise_sigma_log: 0.15
|
||||
friction_loop:
|
||||
force_noise_sd_mN: 0.3
|
||||
nanoindentation:
|
||||
max_load_mN: 10
|
||||
indent_grid:
|
||||
- 5
|
||||
- 5
|
||||
grid_pitch_um: 20
|
||||
simtra:
|
||||
angle_deg_range:
|
||||
- 0
|
||||
- 90
|
||||
energy_eV_max: 50
|
||||
surface_binding_energy_eV:
|
||||
pt: 5.84
|
||||
au: 3.81
|
||||
flux_per_W: 0.02
|
||||
318
out/report/REPORT.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# Laboratory Data Storage Format Evaluation - Final Report
|
||||
|
||||
Generated by task 11 (report.py) from the measured benchmark matrix (task 09) and the extrapolation model (task 10). Every number at scales beyond the 150 MB corpus is a PROJECTION and is labeled as such; scaling assumptions are listed in section 6.
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
**Recommendation: PostgreSQL as the system of record.** The measured matrix and the projections confirm the expected outcome:
|
||||
|
||||
- **PostgreSQL** is the only format whose seven benchmark queries stay below the 1-hour practicality bound at every projected scale (600 GB, 1.2 TB, 6 TB); at 6 TB it needs one 16-core node with 551 GB RAM at an estimated $33,586 (bill of materials, prices as of 2026-07-11). Partitioned parallel scans and the track_summary materialized view are the scaling levers.
|
||||
- **CSV** remains the canonical raw archive: byte-reproducible, instrument-native, and the best compressor (tar.gz ~2.8x); its full scans exceed 1 h beyond ~600 GB, so it is an archive, not a query layer.
|
||||
- **SQLite** is an excellent single-user store up to ~600 GB (all queries OK), then its single-threaded scans degrade (Q5/Q7 IMPRACTICAL at 1.2 TB).
|
||||
- **Hybrid JSON-LD** (metadata + sourceFile links, 1.8 MiB at 150 MB corpus) is the exchange format; the FULL variant re-reads the whole corpus per scan and becomes impractical past 600 GB.
|
||||
- **RDF (materialized triplestore)** fails at scale: 11.7x storage blow-up, 5 of 7 queries FAIL at 6 TB, and the hot set needs 13.7 TB RAM across 14 nodes (~$527,732 projected). Semantic access should be a virtual layer (e.g. Ontop OBDA) over PostgreSQL instead.
|
||||
|
||||
## 2. Methodology snapshot
|
||||
|
||||
35 cells (Q1-Q7 x 5 formats); per cell 1 warm-up + 3 measured runs in isolated subprocesses sampled with psutil at 50 ms; randomized cell order; 30-min timeout; every run validated against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9). Medians are reported; raw runs and min/max live in out/bench/results_raw.csv. The cold-cache pass is skipped on this Windows host and marked as such. PostgreSQL runs on a remote host; its wall times include the LAN round-trip, and server-side metrics come from pg_stat_statements and Prometheus (docs/rules/bench-methodology.md section 6).
|
||||
|
||||
## 3. Environment
|
||||
|
||||
```
|
||||
benchmark_host_os: Windows-2025Server-10.0.26100-SP0
|
||||
benchmark_host_cpu: Intel64 Family 6 Model 141 Stepping 1, GenuineIntel
|
||||
benchmark_host_cores_logical: 4
|
||||
benchmark_host_cores_physical: 4
|
||||
benchmark_host_ram_gib: 8.0
|
||||
python: 3.12.10
|
||||
benchmark_host_disks: VMware Virtual NVMe Disk / SSD / 100 GB
|
||||
sqlite: 3.49.1
|
||||
psycopg: 3.3.4
|
||||
pyoxigraph: 0.5.9
|
||||
ijson: 3.5.1
|
||||
psutil: 7.2.2
|
||||
postgresql_server: PostgreSQL 16.14 (Ubuntu 16.14-0ubuntu0.24.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
|
||||
postgresql_host: 192.168.10.73 (remote; 4 cores / 3.8 GiB, see .done/06.ok)
|
||||
cold_cache_pass: skipped (Windows host, no page-cache drop - bench-methodology section 6)
|
||||
pg_backend_sampling: not possible for a remote server; Prometheus windows + pg_stat_statements deltas instead
|
||||
```
|
||||
|
||||
## 4. Storage footprint
|
||||
|
||||
| format | measured | vs CSV | archival (compressed) | projected @600 GB | projected @1.2 TB | projected @6 TB |
|
||||
|---|---|---|---|---|---|---|
|
||||
| CSV | 149 MB | 1.00x | 53 MB | 600 GB | 1.20 TB | 6.00 TB |
|
||||
| JSON-LD | 311 MB | 2.09x | 57 MB | 1.26 TB | 2.51 TB | 13 TB |
|
||||
| SQLite | 123 MB | 0.83x | 55 MB | 498 GB | 996 GB | 4.98 TB |
|
||||
| PostgreSQL | 397 MB | 2.67x | 53 MB | 1.60 TB | 3.21 TB | 16 TB |
|
||||
| RDF triplestore | 1.74 GB | 11.70x | 126 MB | 7.02 TB | 14 TB | 70 TB |
|
||||
|
||||

|
||||
|
||||
## 5. Measured performance (150 MB corpus)
|
||||
|
||||
| format | query | wall time | peak RSS | CPU util | read |
|
||||
|---|---|---|---|---|---|
|
||||
| CSV | q1 | 0.06 s | 4 MB | 0% | 0.0 MB |
|
||||
| CSV | q2 | 0.17 s | 20 MB | 16% | 2.5 MB |
|
||||
| CSV | q3 | 1.15 s | 21 MB | 24% | 31.8 MB |
|
||||
| CSV | q4 | 0.71 s | 21 MB | 22% | 16.8 MB |
|
||||
| CSV | q5 | 1.09 s | 21 MB | 23% | 31.8 MB |
|
||||
| CSV | q6 | 0.28 s | 20 MB | 17% | 1.2 MB |
|
||||
| CSV | q7 | 1.27 s | 21 MB | 23% | 32.5 MB |
|
||||
| JSON-LD | q1 | 0.11 s | 22 MB | 10% | 1.4 MB |
|
||||
| JSON-LD | q2 | 1.48 s | 24 MB | 24% | 53.1 MB |
|
||||
| JSON-LD | q3 | 9.94 s | 24 MB | 25% | 297.6 MB |
|
||||
| JSON-LD | q4 | 9.92 s | 24 MB | 25% | 297.8 MB |
|
||||
| JSON-LD | q5 | 9.96 s | 24 MB | 25% | 297.6 MB |
|
||||
| JSON-LD | q6 | 9.79 s | 24 MB | 24% | 297.8 MB |
|
||||
| JSON-LD | q7 | 10 s | 24 MB | 24% | 297.9 MB |
|
||||
| SQLite | q1 | 0.06 s | 4 MB | 0% | 0.0 MB |
|
||||
| SQLite | q2 | 0.06 s | 4 MB | 0% | 0.0 MB |
|
||||
| SQLite | q3 | 0.11 s | 4 MB | 0% | 0.0 MB |
|
||||
| SQLite | q4 | 0.06 s | 4 MB | 0% | 0.0 MB |
|
||||
| SQLite | q5 | 0.77 s | 25 MB | 23% | 117.9 MB |
|
||||
| SQLite | q6 | 0.06 s | 4 MB | 0% | 0.0 MB |
|
||||
| SQLite | q7 | 0.71 s | 25 MB | 23% | 93.0 MB |
|
||||
| PostgreSQL | q1 | 0.22 s | 42 MB | 18% | 4.1 MB |
|
||||
| PostgreSQL | q2 | 0.22 s | 42 MB | 20% | 4.1 MB |
|
||||
| PostgreSQL | q3 | 0.22 s | 42 MB | 18% | 4.1 MB |
|
||||
| PostgreSQL | q4 | 0.22 s | 42 MB | 18% | 4.1 MB |
|
||||
| PostgreSQL | q5 | 0.44 s | 42 MB | 10% | 4.1 MB |
|
||||
| PostgreSQL | q6 | 0.22 s | 42 MB | 19% | 4.1 MB |
|
||||
| PostgreSQL | q7 | 0.33 s | 42 MB | 12% | 4.1 MB |
|
||||
| RDF triplestore | q1 | 0.11 s | 32 MB | 17% | 3.6 MB |
|
||||
| RDF triplestore | q2 | 2.81 s | 85 MB | 24% | 375.6 MB |
|
||||
| RDF triplestore | q3 | 46 s | 264 MB | 25% | 5445.1 MB |
|
||||
| RDF triplestore | q4 | 20 s | 168 MB | 25% | 2694.6 MB |
|
||||
| RDF triplestore | q5 | 60 s | 264 MB | 25% | 5390.5 MB |
|
||||
| RDF triplestore | q6 | 0.17 s | 46 MB | 16% | 15.9 MB |
|
||||
| RDF triplestore | q7 | 41 s | 264 MB | 25% | 5380.5 MB |
|
||||
|
||||
Sub-interval note: cells faster than the 50 ms sampling cadence (fast SQLite/CSV point reads) under-report client RSS/CPU; wall times are exact.
|
||||
|
||||

|
||||
|
||||
Per-query charts: [Q1](charts/c3_q1_wall_time.png), [Q2](charts/c3_q2_wall_time.png), [Q3](charts/c3_q3_wall_time.png), [Q4](charts/c3_q4_wall_time.png), [Q5](charts/c3_q5_wall_time.png), [Q6](charts/c3_q6_wall_time.png), [Q7](charts/c3_q7_wall_time.png)
|
||||
|
||||
## 6. Projections (600 GB / 1.2 TB / 6 TB)
|
||||
|
||||
All values in this section are projected, never measured. Laws per access pattern (spec 10): flat / index-depth log / linear result set / linear x depth / full scan / parallel partitioned scan; constants calibrated on the measured point after subtracting the per-format harness floor. Flags: IMPRACTICAL > 1 h, FAIL > 24 h.
|
||||
|
||||
Projected wall times at 600 GB:
|
||||
|
||||
| format | query | projected wall time | flag |
|
||||
|---|---|---|---|
|
||||
| CSV | q1 | 0.06 s | OK |
|
||||
| CSV | q2 | 7.1 min | OK |
|
||||
| CSV | q3 | 73.5 min | IMPRACTICAL |
|
||||
| CSV | q4 | 44.1 min | OK |
|
||||
| CSV | q5 | 69.6 min | IMPRACTICAL |
|
||||
| CSV | q6 | 14.5 min | OK |
|
||||
| CSV | q7 | 81.6 min | IMPRACTICAL |
|
||||
| JSON-LD | q1 | 0.11 s | OK |
|
||||
| JSON-LD | q2 | 92.0 min | IMPRACTICAL |
|
||||
| JSON-LD | q3 | 11.0 h | IMPRACTICAL |
|
||||
| JSON-LD | q4 | 11.0 h | IMPRACTICAL |
|
||||
| JSON-LD | q5 | 11.0 h | IMPRACTICAL |
|
||||
| JSON-LD | q6 | 10.9 h | IMPRACTICAL |
|
||||
| JSON-LD | q7 | 11.2 h | IMPRACTICAL |
|
||||
| SQLite | q1 | 0.06 s | OK |
|
||||
| SQLite | q2 | 1.27 s | OK |
|
||||
| SQLite | q3 | 5.7 min | OK |
|
||||
| SQLite | q4 | 0.06 s | OK |
|
||||
| SQLite | q5 | 47.5 min | OK |
|
||||
| SQLite | q6 | 5.18 s | OK |
|
||||
| SQLite | q7 | 43.7 min | OK |
|
||||
| PostgreSQL | q1 | 0.22 s | OK |
|
||||
| PostgreSQL | q2 | 0.22 s | OK |
|
||||
| PostgreSQL | q3 | 3.42 s | OK |
|
||||
| PostgreSQL | q4 | 11 s | OK |
|
||||
| PostgreSQL | q5 | 3.7 min | OK |
|
||||
| PostgreSQL | q6 | 4.06 s | OK |
|
||||
| PostgreSQL | q7 | 115 s | OK |
|
||||
| RDF triplestore | q1 | 0.11 s | OK |
|
||||
| RDF triplestore | q2 | 3.0 h | IMPRACTICAL |
|
||||
| RDF triplestore | q3 | 2.1 d | FAIL |
|
||||
| RDF triplestore | q4 | 22.1 h | IMPRACTICAL |
|
||||
| RDF triplestore | q5 | 2.8 d | FAIL |
|
||||
| RDF triplestore | q6 | 5.9 min | OK |
|
||||
| RDF triplestore | q7 | 45.3 h | FAIL |
|
||||
|
||||
Degradation by data size: [point read](charts/c4_degradation_point_read.png), [indexed / join](charts/c4_degradation_indexed.png), [full scan](charts/c4_degradation_full_scan.png)
|
||||
|
||||
## 7. Hardware sizing and cost at 600 GB (projected)
|
||||
|
||||
Bill-of-materials costing with street prices as of **2026-07-11** (sources below). Every configuration buys whole components: 64 GB DDR5 ECC RDIMM modules, 7.68 TB enterprise NVMe U.2 drives, and a priced 1U chassis; disk capacity includes 30% free-space headroom.
|
||||
|
||||
| format | configuration | chassis | RAM | disk | extra nodes | total |
|
||||
|---|---|---|---|---|---|---|
|
||||
| CSV | 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 | $4,618 | $1,887 | $3,995 | $0 | $10,500 |
|
||||
| JSON-LD | 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 | $4,618 | $1,887 | $3,995 | $0 | $10,500 |
|
||||
| SQLite | 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 | $4,618 | $1,887 | $3,995 | $0 | $10,500 |
|
||||
| PostgreSQL | 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2 | $4,618 | $1,887 | $3,995 | $0 | $10,500 |
|
||||
| RDF triplestore | 2 x 1U Supermicro AS-1015CS-TNR (16 cores); 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2 | $4,618 | $41,514 | $7,990 | $4,618 | $58,740 |
|
||||
|
||||
Component prices (retrieved 2026-07-11):
|
||||
|
||||
- 64 GB DDR5-5600 ECC RDIMM (A-Tech, 2x32GB kit): $1,887 - <https://atechmemory.com/collections/ddr5-memory-ram>
|
||||
- 7.68 TB NVMe U.2 1DWPD enterprise SSD (Dell 14-16G compatible, Cloud Ninjas): $3,995 - <https://cloudninjas.com/products/new-7-68tb-nvme-u-2-1dwpd-sie-2-5-enterprise-solid-state-drive-for-14th-15th-16th-gen-dell>
|
||||
- 1U AMD EPYC 9004/9005 chassis (Broadberry starting configurations): Supermicro AS-1015CS-TNR $4,618, Supermicro AS-1115CS-TNR $6,272 - <https://www.broadberry.com/amd-epyc-9004-supermicro-servers>
|
||||
- Market context - server DRAM is in a documented 2025-2026 price surge: <https://www.techpowerup.com/342331/server-dram-pricing-jumps-50-only-70-of-orders-getting-filled>
|
||||
|
||||
Full sizing at 1.2 TB and 6 TB: out/bench/hardware_sizing.csv. RDF exceeds the 1 TB single-node RAM ceiling already at 600 GB and reaches 14 nodes at 6 TB.
|
||||
|
||||

|
||||
|
||||
## 8. Weighted scoring
|
||||
|
||||
Subscores are 10 x best/value per metric; weights: search x5, RAM economy x2, disk economy x1; max 80. Search uses the inverse geometric mean of Q1-Q7 (FAIL at 600 GB zeroes the search subscore). Left: measured scale; right: projected 600 GB.
|
||||
|
||||
Measured scale (150 MB):
|
||||
|
||||
| format | search (x5) | RAM (x2) | disk (x1) | total (max 80) |
|
||||
|---|---|---|---|---|
|
||||
| SQLite | 50.0 | 16.9 | 10.0 | 76.9 |
|
||||
| CSV | 15.5 | 20.0 | 8.3 | 43.8 |
|
||||
| PostgreSQL | 26.2 | 10.0 | 3.1 | 39.3 |
|
||||
| JSON-LD | 1.7 | 17.2 | 4.0 | 22.9 |
|
||||
| RDF triplestore | 1.3 | 1.6 | 0.7 | 3.6 |
|
||||
|
||||
Projected at 600 GB:
|
||||
|
||||
| format | search (x5) | RAM (x2) | disk (x1) | total (max 80) |
|
||||
|---|---|---|---|---|
|
||||
| PostgreSQL | 50.0 | 5.5 | 3.1 | 58.6 |
|
||||
| SQLite | 21.8 | 15.2 | 10.0 | 47.0 |
|
||||
| CSV | 0.6 | 20.0 | 8.3 | 28.9 |
|
||||
| JSON-LD | 0.1 | 20.0 | 4.0 | 24.0 |
|
||||
| RDF triplestore | 0.0 | 0.2 | 0.7 | 0.9 |
|
||||
|
||||

|
||||
|
||||
## 9. Use-case mapping
|
||||
|
||||
| Use case | Recommended format(s) |
|
||||
|---|---|
|
||||
| Interactive analysis (joins, aggregations) | PostgreSQL; SQLite acceptable single-user up to ~600 GB |
|
||||
| Report generation (repeated summaries) | PostgreSQL (track_summary materialized view) |
|
||||
| Search / filtering | PostgreSQL or SQLite (indexed); flat formats need full scans |
|
||||
| Archiving | CSV tree + tar.gz (canonical raw) plus pg_dump of the system of record |
|
||||
| Inter-lab exchange | Hybrid JSON-LD (metadata + sourceFile links to CSV) |
|
||||
| Semantic / ontology queries | Virtual RDF layer over PostgreSQL (e.g. Ontop OBDA), not a materialized triplestore |
|
||||
|
||||
## 10. Limitations
|
||||
|
||||
- Single-node measurements on one Windows host + one remote PostgreSQL host; no cluster or cloud variance.
|
||||
- The corpus is simulated (physically plausible models, fixed seed 20260711); real instrument data may have different value distributions, though volumes and shapes match the lab.
|
||||
- Extrapolation uses analytic laws calibrated on a single 150 MB point; no intermediate-scale validation runs were performed.
|
||||
- Client wall times include a per-format harness floor (interpreter start, imports, connection); the floor is assumed constant across scales. The cheapest cell of each format therefore projects flat.
|
||||
- The cold-cache matrix pass is skipped (no page-cache drop on Windows); all measured runs are warm-cache.
|
||||
- The pandas CSV variant was not measured; its RAM is O(n) and it is infeasible above roughly 10 GB (spec 10).
|
||||
- RDF numbers reflect the compact modeling and a client that derives run-in / steady-state from raw cycles; SPARQL-side aggregation engines could shift (not remove) the scan penalty.
|
||||
- Hardware prices are spot street prices retrieved 2026-07-11 during a documented server-DRAM price surge; re-cost by editing out/config/hw_prices.yaml and re-running tasks 10-11.
|
||||
|
||||
## Appendix A - process flow diagrams (task 02)
|
||||
|
||||
### D1 - Coupon assembly
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A([Ti-6Al-4V Base 10x10x3 mm]) --> B[Cleaning]
|
||||
B --> B1([Rinsed in deionized water])
|
||||
B1 --> B2([Sonicated in cleaning solution])
|
||||
B --> C[Smearing Adhesive]
|
||||
C --> C1([Cr adhesive coating])
|
||||
C --> D[Vapor Deposition System<br/>Kurt J. Lesker PVD 200]
|
||||
D --> D1([PtAu sputter coating, gradient])
|
||||
D --> E([Test Coupon])
|
||||
```
|
||||
|
||||
### D2 - Characterization and batch assembly
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
TC([Test Coupon]) --> S[SIMTRA simulation<br/>composition / atom energies]
|
||||
S --> OP[Optical profilometry<br/>film thickness 0.3-1.1 um]
|
||||
OP --> TW[Assemble Test Wafer<br/>x49 coupons]
|
||||
TW --> TB[Assemble Test Batch<br/>x3 wafers]
|
||||
TB --> B721[B721<br/>Pt 150 W / Au 50 W]
|
||||
TB --> B722[B722<br/>Pt 100 W / Au 100 W]
|
||||
TB --> B723[B723<br/>Pt 150 W / Au 50 W]
|
||||
TB --> B724[B724<br/>Pt 50 W / Au 150 W]
|
||||
```
|
||||
|
||||
### D3 - Testing tree
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
TC([Test Coupon]) --> F[Friction<br/>RAPID 6-probe tribometer]
|
||||
F --> FA([Lab Air -> COF dataset mu_normal])
|
||||
F --> FN([Dry N2 -> COF dataset mu_dry_nit])
|
||||
TC --> NI[Nanoindentation<br/>Bruker TI980]
|
||||
NI --> NIR([Hardness, reduced modulus<br/>25 indents per coupon])
|
||||
TC --> AF[AFM]
|
||||
AF --> AFR([Topography -> Ra / Rq])
|
||||
TC --> XR[micro-XRF<br/>Bruker M4 Tornado]
|
||||
XR --> XRR([Pt/Au wt% map, 20x20 grid])
|
||||
```
|
||||
|
||||
### D4 - Tribometer session sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor Op as Operator
|
||||
participant Prep as Sample Prep
|
||||
participant Plan as Excel Test Plan
|
||||
participant SW as Control Software
|
||||
participant TR as RAPID Tribometer
|
||||
|
||||
Op->>Prep: Request test
|
||||
Op->>Prep: Get samples
|
||||
Op->>Prep: Load samples onto plates
|
||||
Op->>Prep: Load ball holders (3 counterfaces per holder)
|
||||
Op->>Plan: Create test plan<br/>(Sample Plate, Plate Location, Sample ID, Save Location,<br/>Folder Name, X/Y/Z Offset, Load, Iterations)
|
||||
Op->>TR: Transfer plates to tribometer
|
||||
Op->>TR: Set up platter
|
||||
Op->>SW: Activate software
|
||||
Op->>SW: Load plan
|
||||
SW->>TR: Activate tribometer
|
||||
TR-->>SW: Run reciprocating tests (execution loop, see D5)
|
||||
Op->>SW: Software stop
|
||||
SW-->>Op: Save avg files
|
||||
Op->>TR: Remove plates and platters
|
||||
Op->>SW: Close software
|
||||
Op->>TR: Pull equipment out
|
||||
```
|
||||
|
||||
### D5 - Execution loop
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
L[Load all counterfaces] --> P{5 plates}
|
||||
P -->|for each plate| PR{6 probes in parallel}
|
||||
PR -->|for each probe| C{4 coupons on coupon square}
|
||||
C -->|for each coupon| T[Draw track]
|
||||
T --> RC[Rotate counterface - fresh ball per track]
|
||||
RC -->|3 tracks per coupon| T
|
||||
T --> DONE([120 coupons x 3 tracks = 360 tracks per run])
|
||||
```
|
||||
|
||||
### D6 - Data hierarchy
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
DEPOSITION_RUN ||--|| SIMTRA_PROFILE : produces
|
||||
DEPOSITION_RUN ||--|| BATCH : creates
|
||||
BATCH ||--|{ WAFER : contains
|
||||
WAFER ||--|{ COUPON : contains
|
||||
COUPON ||--|| XRF_MAP : has
|
||||
COUPON ||--|| NANOINDENTATION : has
|
||||
COUPON ||--|| AFM : has
|
||||
COUPON ||--|| PROFILOMETRY : has
|
||||
COUPON ||--o{ TRACK : "friction coupons only"
|
||||
TRACK ||--|{ CYCLE : has
|
||||
TRACK ||--|{ LOOP_POINT : has
|
||||
TRACK ||--|| WEAR : has
|
||||
```
|
||||
|
||||
BIN
out/report/charts/c1_disk_footprint.png
Normal file
|
After Width: | Height: | Size: 71 KiB |
BIN
out/report/charts/c2_ram_requirement.png
Normal file
|
After Width: | Height: | Size: 74 KiB |
BIN
out/report/charts/c3_q1_wall_time.png
Normal file
|
After Width: | Height: | Size: 38 KiB |
BIN
out/report/charts/c3_q2_wall_time.png
Normal file
|
After Width: | Height: | Size: 41 KiB |
BIN
out/report/charts/c3_q3_wall_time.png
Normal file
|
After Width: | Height: | Size: 40 KiB |
BIN
out/report/charts/c3_q4_wall_time.png
Normal file
|
After Width: | Height: | Size: 36 KiB |
BIN
out/report/charts/c3_q5_wall_time.png
Normal file
|
After Width: | Height: | Size: 40 KiB |
BIN
out/report/charts/c3_q6_wall_time.png
Normal file
|
After Width: | Height: | Size: 40 KiB |
BIN
out/report/charts/c3_q7_wall_time.png
Normal file
|
After Width: | Height: | Size: 40 KiB |
BIN
out/report/charts/c4_degradation_full_scan.png
Normal file
|
After Width: | Height: | Size: 52 KiB |
BIN
out/report/charts/c4_degradation_indexed.png
Normal file
|
After Width: | Height: | Size: 53 KiB |
BIN
out/report/charts/c4_degradation_point_read.png
Normal file
|
After Width: | Height: | Size: 51 KiB |
BIN
out/report/charts/c5_weighted_score.png
Normal file
|
After Width: | Height: | Size: 73 KiB |
BIN
out/report/charts/c6_cost_vs_performance.png
Normal file
|
After Width: | Height: | Size: 76 KiB |
9
out/report/diagrams/D1.mermaid
Normal file
@@ -0,0 +1,9 @@
|
||||
flowchart LR
|
||||
A([Ti-6Al-4V Base 10x10x3 mm]) --> B[Cleaning]
|
||||
B --> B1([Rinsed in deionized water])
|
||||
B1 --> B2([Sonicated in cleaning solution])
|
||||
B --> C[Smearing Adhesive]
|
||||
C --> C1([Cr adhesive coating])
|
||||
C --> D[Vapor Deposition System<br/>Kurt J. Lesker PVD 200]
|
||||
D --> D1([PtAu sputter coating, gradient])
|
||||
D --> E([Test Coupon])
|
||||
9
out/report/diagrams/D2.mermaid
Normal file
@@ -0,0 +1,9 @@
|
||||
flowchart LR
|
||||
TC([Test Coupon]) --> S[SIMTRA simulation<br/>composition / atom energies]
|
||||
S --> OP[Optical profilometry<br/>film thickness 0.3-1.1 um]
|
||||
OP --> TW[Assemble Test Wafer<br/>x49 coupons]
|
||||
TW --> TB[Assemble Test Batch<br/>x3 wafers]
|
||||
TB --> B721[B721<br/>Pt 150 W / Au 50 W]
|
||||
TB --> B722[B722<br/>Pt 100 W / Au 100 W]
|
||||
TB --> B723[B723<br/>Pt 150 W / Au 50 W]
|
||||
TB --> B724[B724<br/>Pt 50 W / Au 150 W]
|
||||
10
out/report/diagrams/D3.mermaid
Normal file
@@ -0,0 +1,10 @@
|
||||
flowchart TD
|
||||
TC([Test Coupon]) --> F[Friction<br/>RAPID 6-probe tribometer]
|
||||
F --> FA([Lab Air -> COF dataset mu_normal])
|
||||
F --> FN([Dry N2 -> COF dataset mu_dry_nit])
|
||||
TC --> NI[Nanoindentation<br/>Bruker TI980]
|
||||
NI --> NIR([Hardness, reduced modulus<br/>25 indents per coupon])
|
||||
TC --> AF[AFM]
|
||||
AF --> AFR([Topography -> Ra / Rq])
|
||||
TC --> XR[micro-XRF<br/>Bruker M4 Tornado]
|
||||
XR --> XRR([Pt/Au wt% map, 20x20 grid])
|
||||
23
out/report/diagrams/D4.mermaid
Normal file
@@ -0,0 +1,23 @@
|
||||
sequenceDiagram
|
||||
actor Op as Operator
|
||||
participant Prep as Sample Prep
|
||||
participant Plan as Excel Test Plan
|
||||
participant SW as Control Software
|
||||
participant TR as RAPID Tribometer
|
||||
|
||||
Op->>Prep: Request test
|
||||
Op->>Prep: Get samples
|
||||
Op->>Prep: Load samples onto plates
|
||||
Op->>Prep: Load ball holders (3 counterfaces per holder)
|
||||
Op->>Plan: Create test plan<br/>(Sample Plate, Plate Location, Sample ID, Save Location,<br/>Folder Name, X/Y/Z Offset, Load, Iterations)
|
||||
Op->>TR: Transfer plates to tribometer
|
||||
Op->>TR: Set up platter
|
||||
Op->>SW: Activate software
|
||||
Op->>SW: Load plan
|
||||
SW->>TR: Activate tribometer
|
||||
TR-->>SW: Run reciprocating tests (execution loop, see D5)
|
||||
Op->>SW: Software stop
|
||||
SW-->>Op: Save avg files
|
||||
Op->>TR: Remove plates and platters
|
||||
Op->>SW: Close software
|
||||
Op->>TR: Pull equipment out
|
||||
8
out/report/diagrams/D5.mermaid
Normal file
@@ -0,0 +1,8 @@
|
||||
flowchart TD
|
||||
L[Load all counterfaces] --> P{5 plates}
|
||||
P -->|for each plate| PR{6 probes in parallel}
|
||||
PR -->|for each probe| C{4 coupons on coupon square}
|
||||
C -->|for each coupon| T[Draw track]
|
||||
T --> RC[Rotate counterface - fresh ball per track]
|
||||
RC -->|3 tracks per coupon| T
|
||||
T --> DONE([120 coupons x 3 tracks = 360 tracks per run])
|
||||
13
out/report/diagrams/D6.mermaid
Normal file
@@ -0,0 +1,13 @@
|
||||
erDiagram
|
||||
DEPOSITION_RUN ||--|| SIMTRA_PROFILE : produces
|
||||
DEPOSITION_RUN ||--|| BATCH : creates
|
||||
BATCH ||--|{ WAFER : contains
|
||||
WAFER ||--|{ COUPON : contains
|
||||
COUPON ||--|| XRF_MAP : has
|
||||
COUPON ||--|| NANOINDENTATION : has
|
||||
COUPON ||--|| AFM : has
|
||||
COUPON ||--|| PROFILOMETRY : has
|
||||
COUPON ||--o{ TRACK : "friction coupons only"
|
||||
TRACK ||--|{ CYCLE : has
|
||||
TRACK ||--|{ LOOP_POINT : has
|
||||
TRACK ||--|| WEAR : has
|
||||
127
out/report/process_flow.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# Process Flow - Simulated Tribology Laboratory
|
||||
|
||||
Generated by make_process_flow.py (task 02) from out/config/lab_config.yaml.
|
||||
Standalone diagram sources: ./diagrams/D1..D6.mermaid.
|
||||
|
||||
## D1 - Coupon Assembly
|
||||
|
||||
Substrate preparation, adhesion layer, and gradient Pt-Au sputter deposition.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A([Ti-6Al-4V Base 10x10x3 mm]) --> B[Cleaning]
|
||||
B --> B1([Rinsed in deionized water])
|
||||
B1 --> B2([Sonicated in cleaning solution])
|
||||
B --> C[Smearing Adhesive]
|
||||
C --> C1([Cr adhesive coating])
|
||||
C --> D[Vapor Deposition System<br/>Kurt J. Lesker PVD 200]
|
||||
D --> D1([PtAu sputter coating, gradient])
|
||||
D --> E([Test Coupon])
|
||||
```
|
||||
|
||||
## D2 - Characterization and Batch Assembly
|
||||
|
||||
Per-coupon simulation and thickness characterization, then assembly into wafers (x49 coupons) and batches (x3 wafers), deposited per the parameter table below.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
TC([Test Coupon]) --> S[SIMTRA simulation<br/>composition / atom energies]
|
||||
S --> OP[Optical profilometry<br/>film thickness 0.3-1.1 um]
|
||||
OP --> TW[Assemble Test Wafer<br/>x49 coupons]
|
||||
TW --> TB[Assemble Test Batch<br/>x3 wafers]
|
||||
TB --> B721[B721<br/>Pt 150 W / Au 50 W]
|
||||
TB --> B722[B722<br/>Pt 100 W / Au 100 W]
|
||||
TB --> B723[B723<br/>Pt 150 W / Au 50 W]
|
||||
TB --> B724[B724<br/>Pt 50 W / Au 150 W]
|
||||
```
|
||||
|
||||
Deposition matrix:
|
||||
|
||||
| Batch | Pt:Au gun tilt | Pt power W | Au power W | Pt discharge V | Au discharge V |
|
||||
|---|---|---|---|---|---|
|
||||
| B721 | 20:0 deg | 150 | 50 | 432 | 311 |
|
||||
| B722 | 20:20 deg | 100 | 100 | 399 | 352 |
|
||||
| B723 | 20:20 deg | 150 | 50 | 430 | 311 |
|
||||
| B724 | 0:20 deg | 50 | 150 | 376 | 340 |
|
||||
|
||||
## D3 - Testing Tree
|
||||
|
||||
The four characterization/testing paths every coupon can take.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
TC([Test Coupon]) --> F[Friction<br/>RAPID 6-probe tribometer]
|
||||
F --> FA([Lab Air -> COF dataset mu_normal])
|
||||
F --> FN([Dry N2 -> COF dataset mu_dry_nit])
|
||||
TC --> NI[Nanoindentation<br/>Bruker TI980]
|
||||
NI --> NIR([Hardness, reduced modulus<br/>25 indents per coupon])
|
||||
TC --> AF[AFM]
|
||||
AF --> AFR([Topography -> Ra / Rq])
|
||||
TC --> XR[micro-XRF<br/>Bruker M4 Tornado]
|
||||
XR --> XRR([Pt/Au wt% map, 20x20 grid])
|
||||
```
|
||||
|
||||
## D4 - Tribometer Session Sequence
|
||||
|
||||
Operator workflow for one RAPID tribometer session, from test request to teardown.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor Op as Operator
|
||||
participant Prep as Sample Prep
|
||||
participant Plan as Excel Test Plan
|
||||
participant SW as Control Software
|
||||
participant TR as RAPID Tribometer
|
||||
|
||||
Op->>Prep: Request test
|
||||
Op->>Prep: Get samples
|
||||
Op->>Prep: Load samples onto plates
|
||||
Op->>Prep: Load ball holders (3 counterfaces per holder)
|
||||
Op->>Plan: Create test plan<br/>(Sample Plate, Plate Location, Sample ID, Save Location,<br/>Folder Name, X/Y/Z Offset, Load, Iterations)
|
||||
Op->>TR: Transfer plates to tribometer
|
||||
Op->>TR: Set up platter
|
||||
Op->>SW: Activate software
|
||||
Op->>SW: Load plan
|
||||
SW->>TR: Activate tribometer
|
||||
TR-->>SW: Run reciprocating tests (execution loop, see D5)
|
||||
Op->>SW: Software stop
|
||||
SW-->>Op: Save avg files
|
||||
Op->>TR: Remove plates and platters
|
||||
Op->>SW: Close software
|
||||
Op->>TR: Pull equipment out
|
||||
```
|
||||
|
||||
## D5 - Execution Loop
|
||||
|
||||
Nested iteration executed by the tribometer within one run.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
L[Load all counterfaces] --> P{5 plates}
|
||||
P -->|for each plate| PR{6 probes in parallel}
|
||||
PR -->|for each probe| C{4 coupons on coupon square}
|
||||
C -->|for each coupon| T[Draw track]
|
||||
T --> RC[Rotate counterface - fresh ball per track]
|
||||
RC -->|3 tracks per coupon| T
|
||||
T --> DONE([120 coupons x 3 tracks = 360 tracks per run])
|
||||
```
|
||||
|
||||
## D6 - Data Hierarchy
|
||||
|
||||
Entity containment and per-entity measurement datasets (ERD-style).
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
DEPOSITION_RUN ||--|| SIMTRA_PROFILE : produces
|
||||
DEPOSITION_RUN ||--|| BATCH : creates
|
||||
BATCH ||--|{ WAFER : contains
|
||||
WAFER ||--|{ COUPON : contains
|
||||
COUPON ||--|| XRF_MAP : has
|
||||
COUPON ||--|| NANOINDENTATION : has
|
||||
COUPON ||--|| AFM : has
|
||||
COUPON ||--|| PROFILOMETRY : has
|
||||
COUPON ||--o{ TRACK : "friction coupons only"
|
||||
TRACK ||--|{ CYCLE : has
|
||||
TRACK ||--|{ LOOP_POINT : has
|
||||
TRACK ||--|| WEAR : has
|
||||
```
|
||||
6
out/report/tables/t1_storage_footprint.csv
Normal file
@@ -0,0 +1,6 @@
|
||||
format,measured,vs CSV,archival (compressed),projected @600 GB,projected @1.2 TB,projected @6 TB
|
||||
CSV,149 MB,1.00x,53 MB,600 GB,1.20 TB,6.00 TB
|
||||
JSON-LD,311 MB,2.09x,57 MB,1.26 TB,2.51 TB,13 TB
|
||||
SQLite,123 MB,0.83x,55 MB,498 GB,996 GB,4.98 TB
|
||||
PostgreSQL,397 MB,2.67x,53 MB,1.60 TB,3.21 TB,16 TB
|
||||
RDF triplestore,1.74 GB,11.70x,126 MB,7.02 TB,14 TB,70 TB
|
||||
|
36
out/report/tables/t2_measured_medians.csv
Normal file
@@ -0,0 +1,36 @@
|
||||
format,query,wall time,peak RSS,CPU util,read
|
||||
CSV,q1,0.06 s,4 MB,0%,0.0 MB
|
||||
CSV,q2,0.17 s,20 MB,16%,2.5 MB
|
||||
CSV,q3,1.15 s,21 MB,24%,31.8 MB
|
||||
CSV,q4,0.71 s,21 MB,22%,16.8 MB
|
||||
CSV,q5,1.09 s,21 MB,23%,31.8 MB
|
||||
CSV,q6,0.28 s,20 MB,17%,1.2 MB
|
||||
CSV,q7,1.27 s,21 MB,23%,32.5 MB
|
||||
JSON-LD,q1,0.11 s,22 MB,10%,1.4 MB
|
||||
JSON-LD,q2,1.48 s,24 MB,24%,53.1 MB
|
||||
JSON-LD,q3,9.94 s,24 MB,25%,297.6 MB
|
||||
JSON-LD,q4,9.92 s,24 MB,25%,297.8 MB
|
||||
JSON-LD,q5,9.96 s,24 MB,25%,297.6 MB
|
||||
JSON-LD,q6,9.79 s,24 MB,24%,297.8 MB
|
||||
JSON-LD,q7,10 s,24 MB,24%,297.9 MB
|
||||
SQLite,q1,0.06 s,4 MB,0%,0.0 MB
|
||||
SQLite,q2,0.06 s,4 MB,0%,0.0 MB
|
||||
SQLite,q3,0.11 s,4 MB,0%,0.0 MB
|
||||
SQLite,q4,0.06 s,4 MB,0%,0.0 MB
|
||||
SQLite,q5,0.77 s,25 MB,23%,117.9 MB
|
||||
SQLite,q6,0.06 s,4 MB,0%,0.0 MB
|
||||
SQLite,q7,0.71 s,25 MB,23%,93.0 MB
|
||||
PostgreSQL,q1,0.22 s,42 MB,18%,4.1 MB
|
||||
PostgreSQL,q2,0.22 s,42 MB,20%,4.1 MB
|
||||
PostgreSQL,q3,0.22 s,42 MB,18%,4.1 MB
|
||||
PostgreSQL,q4,0.22 s,42 MB,18%,4.1 MB
|
||||
PostgreSQL,q5,0.44 s,42 MB,10%,4.1 MB
|
||||
PostgreSQL,q6,0.22 s,42 MB,19%,4.1 MB
|
||||
PostgreSQL,q7,0.33 s,42 MB,12%,4.1 MB
|
||||
RDF triplestore,q1,0.11 s,32 MB,17%,3.6 MB
|
||||
RDF triplestore,q2,2.81 s,85 MB,24%,375.6 MB
|
||||
RDF triplestore,q3,46 s,264 MB,25%,5445.1 MB
|
||||
RDF triplestore,q4,20 s,168 MB,25%,2694.6 MB
|
||||
RDF triplestore,q5,60 s,264 MB,25%,5390.5 MB
|
||||
RDF triplestore,q6,0.17 s,46 MB,16%,15.9 MB
|
||||
RDF triplestore,q7,41 s,264 MB,25%,5380.5 MB
|
||||
|
36
out/report/tables/t3_projected_600gb.csv
Normal file
@@ -0,0 +1,36 @@
|
||||
format,query,projected wall time,flag
|
||||
CSV,q1,0.06 s,OK
|
||||
CSV,q2,7.1 min,OK
|
||||
CSV,q3,73.5 min,IMPRACTICAL
|
||||
CSV,q4,44.1 min,OK
|
||||
CSV,q5,69.6 min,IMPRACTICAL
|
||||
CSV,q6,14.5 min,OK
|
||||
CSV,q7,81.6 min,IMPRACTICAL
|
||||
JSON-LD,q1,0.11 s,OK
|
||||
JSON-LD,q2,92.0 min,IMPRACTICAL
|
||||
JSON-LD,q3,11.0 h,IMPRACTICAL
|
||||
JSON-LD,q4,11.0 h,IMPRACTICAL
|
||||
JSON-LD,q5,11.0 h,IMPRACTICAL
|
||||
JSON-LD,q6,10.9 h,IMPRACTICAL
|
||||
JSON-LD,q7,11.2 h,IMPRACTICAL
|
||||
SQLite,q1,0.06 s,OK
|
||||
SQLite,q2,1.27 s,OK
|
||||
SQLite,q3,5.7 min,OK
|
||||
SQLite,q4,0.06 s,OK
|
||||
SQLite,q5,47.5 min,OK
|
||||
SQLite,q6,5.18 s,OK
|
||||
SQLite,q7,43.7 min,OK
|
||||
PostgreSQL,q1,0.22 s,OK
|
||||
PostgreSQL,q2,0.22 s,OK
|
||||
PostgreSQL,q3,3.42 s,OK
|
||||
PostgreSQL,q4,11 s,OK
|
||||
PostgreSQL,q5,3.7 min,OK
|
||||
PostgreSQL,q6,4.06 s,OK
|
||||
PostgreSQL,q7,115 s,OK
|
||||
RDF triplestore,q1,0.11 s,OK
|
||||
RDF triplestore,q2,3.0 h,IMPRACTICAL
|
||||
RDF triplestore,q3,2.1 d,FAIL
|
||||
RDF triplestore,q4,22.1 h,IMPRACTICAL
|
||||
RDF triplestore,q5,2.8 d,FAIL
|
||||
RDF triplestore,q6,5.9 min,OK
|
||||
RDF triplestore,q7,45.3 h,FAIL
|
||||
|
6
out/report/tables/t4_hardware_600gb.csv
Normal file
@@ -0,0 +1,6 @@
|
||||
format,configuration,chassis,RAM,disk,extra nodes,total
|
||||
CSV,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,"$4,618","$1,887","$3,995",$0,"$10,500"
|
||||
JSON-LD,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,"$4,618","$1,887","$3,995",$0,"$10,500"
|
||||
SQLite,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,"$4,618","$1,887","$3,995",$0,"$10,500"
|
||||
PostgreSQL,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,"$4,618","$1,887","$3,995",$0,"$10,500"
|
||||
RDF triplestore,2 x 1U Supermicro AS-1015CS-TNR (16 cores); 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,"$4,618","$41,514","$7,990","$4,618","$58,740"
|
||||
|
11
out/report/tables/t5_weighted_scores.csv
Normal file
@@ -0,0 +1,11 @@
|
||||
scale,format,search_pts,ram_pts,disk_pts,total
|
||||
measured,CSV,15.5,20.0,8.3,43.8
|
||||
measured,JSON-LD,1.7,17.2,4.0,22.9
|
||||
measured,SQLite,50.0,16.9,10.0,76.9
|
||||
measured,PostgreSQL,26.2,10.0,3.1,39.3
|
||||
measured,RDF triplestore,1.3,1.6,0.7,3.6
|
||||
projected_600gb,CSV,0.6,20.0,8.3,28.9
|
||||
projected_600gb,JSON-LD,0.1,20.0,4.0,24.0
|
||||
projected_600gb,SQLite,21.8,15.2,10.0,47.0
|
||||
projected_600gb,PostgreSQL,50.0,5.5,3.1,58.6
|
||||
projected_600gb,RDF triplestore,0.0,0.2,0.7,0.9
|
||||
|
106
out/report/tables/t6_projected_all_scales.csv
Normal file
@@ -0,0 +1,106 @@
|
||||
format,query,scale_gb,projected_wall_s,projected_wall,flag
|
||||
CSV,q1,600,0.06,0.06 s,OK
|
||||
CSV,q1,1200,0.06,0.06 s,OK
|
||||
CSV,q1,6000,0.06,0.06 s,OK
|
||||
CSV,q2,600,428.301,7.1 min,OK
|
||||
CSV,q2,1200,856.543,14.3 min,OK
|
||||
CSV,q2,6000,4282.474,71.4 min,IMPRACTICAL
|
||||
CSV,q3,600,4408.0,73.5 min,IMPRACTICAL
|
||||
CSV,q3,1200,8815.94,2.4 h,IMPRACTICAL
|
||||
CSV,q3,6000,44079.46,12.2 h,IMPRACTICAL
|
||||
CSV,q4,600,2643.775,44.1 min,OK
|
||||
CSV,q4,1200,5287.489,88.1 min,IMPRACTICAL
|
||||
CSV,q4,6000,26437.206,7.3 h,IMPRACTICAL
|
||||
CSV,q5,600,4176.725,69.6 min,IMPRACTICAL
|
||||
CSV,q5,1200,8353.391,2.3 h,IMPRACTICAL
|
||||
CSV,q5,6000,41766.715,11.6 h,IMPRACTICAL
|
||||
CSV,q6,600,870.266,14.5 min,OK
|
||||
CSV,q6,1200,1740.472,29.0 min,OK
|
||||
CSV,q6,6000,8702.119,2.4 h,IMPRACTICAL
|
||||
CSV,q7,600,4897.995,81.6 min,IMPRACTICAL
|
||||
CSV,q7,1200,9795.931,2.7 h,IMPRACTICAL
|
||||
CSV,q7,6000,48979.414,13.6 h,IMPRACTICAL
|
||||
JSON-LD,q1,600,0.111,0.11 s,OK
|
||||
JSON-LD,q1,1200,0.111,0.11 s,OK
|
||||
JSON-LD,q1,6000,0.111,0.11 s,OK
|
||||
JSON-LD,q2,600,5519.622,92.0 min,IMPRACTICAL
|
||||
JSON-LD,q2,1200,11039.133,3.1 h,IMPRACTICAL
|
||||
JSON-LD,q2,6000,55195.222,15.3 h,IMPRACTICAL
|
||||
JSON-LD,q3,600,39673.993,11.0 h,IMPRACTICAL
|
||||
JSON-LD,q3,1200,79347.876,22.0 h,IMPRACTICAL
|
||||
JSON-LD,q3,6000,396738.933,4.6 d,FAIL
|
||||
JSON-LD,q4,600,39597.306,11.0 h,IMPRACTICAL
|
||||
JSON-LD,q4,1200,79194.5,22.0 h,IMPRACTICAL
|
||||
JSON-LD,q4,6000,395972.054,4.6 d,FAIL
|
||||
JSON-LD,q5,600,39760.772,11.0 h,IMPRACTICAL
|
||||
JSON-LD,q5,1200,79521.432,22.1 h,IMPRACTICAL
|
||||
JSON-LD,q5,6000,397606.717,4.6 d,FAIL
|
||||
JSON-LD,q6,600,39076.231,10.9 h,IMPRACTICAL
|
||||
JSON-LD,q6,1200,78152.351,21.7 h,IMPRACTICAL
|
||||
JSON-LD,q6,6000,390761.312,4.5 d,FAIL
|
||||
JSON-LD,q7,600,40228.164,11.2 h,IMPRACTICAL
|
||||
JSON-LD,q7,1200,80456.218,22.3 h,IMPRACTICAL
|
||||
JSON-LD,q7,6000,402280.643,4.7 d,FAIL
|
||||
SQLite,q1,600,0.062,0.06 s,OK
|
||||
SQLite,q1,1200,0.063,0.06 s,OK
|
||||
SQLite,q1,6000,0.063,0.06 s,OK
|
||||
SQLite,q2,600,1.271,1.27 s,OK
|
||||
SQLite,q2,1200,2.482,2.48 s,OK
|
||||
SQLite,q2,6000,12.169,12 s,OK
|
||||
SQLite,q3,600,339.238,5.7 min,OK
|
||||
SQLite,q3,1200,699.328,11.7 min,OK
|
||||
SQLite,q3,6000,3739.195,62.3 min,IMPRACTICAL
|
||||
SQLite,q4,600,0.06,0.06 s,OK
|
||||
SQLite,q4,1200,0.06,0.06 s,OK
|
||||
SQLite,q4,6000,0.06,0.06 s,OK
|
||||
SQLite,q5,600,2847.603,47.5 min,OK
|
||||
SQLite,q5,1200,5695.146,94.9 min,IMPRACTICAL
|
||||
SQLite,q5,6000,28475.49,7.9 h,IMPRACTICAL
|
||||
SQLite,q6,600,5.18,5.18 s,OK
|
||||
SQLite,q6,1200,10.615,11 s,OK
|
||||
SQLite,q6,6000,56.5,56 s,OK
|
||||
SQLite,q7,600,2619.558,43.7 min,OK
|
||||
SQLite,q7,1200,5239.055,87.3 min,IMPRACTICAL
|
||||
SQLite,q7,6000,26195.034,7.3 h,IMPRACTICAL
|
||||
PostgreSQL,q1,600,0.222,0.22 s,OK
|
||||
PostgreSQL,q1,1200,0.222,0.22 s,OK
|
||||
PostgreSQL,q1,6000,0.222,0.22 s,OK
|
||||
PostgreSQL,q2,600,0.221,0.22 s,OK
|
||||
PostgreSQL,q2,1200,0.221,0.22 s,OK
|
||||
PostgreSQL,q2,6000,0.221,0.22 s,OK
|
||||
PostgreSQL,q3,600,3.42,3.42 s,OK
|
||||
PostgreSQL,q3,1200,6.817,6.82 s,OK
|
||||
PostgreSQL,q3,6000,35.495,35 s,OK
|
||||
PostgreSQL,q4,600,10.715,11 s,OK
|
||||
PostgreSQL,q4,1200,21.209,21 s,OK
|
||||
PostgreSQL,q4,6000,105.162,105 s,OK
|
||||
PostgreSQL,q5,600,222.111,3.7 min,OK
|
||||
PostgreSQL,q5,1200,444.001,7.4 min,OK
|
||||
PostgreSQL,q5,6000,2219.125,37.0 min,OK
|
||||
PostgreSQL,q6,600,4.06,4.06 s,OK
|
||||
PostgreSQL,q6,1200,8.137,8.14 s,OK
|
||||
PostgreSQL,q6,6000,42.55,43 s,OK
|
||||
PostgreSQL,q7,600,114.849,115 s,OK
|
||||
PostgreSQL,q7,1200,229.477,3.8 min,OK
|
||||
PostgreSQL,q7,6000,1146.503,19.1 min,OK
|
||||
RDF triplestore,q1,600,0.113,0.11 s,OK
|
||||
RDF triplestore,q1,1200,0.113,0.11 s,OK
|
||||
RDF triplestore,q1,6000,0.113,0.11 s,OK
|
||||
RDF triplestore,q2,600,10866.789,3.0 h,IMPRACTICAL
|
||||
RDF triplestore,q2,1200,21733.465,6.0 h,IMPRACTICAL
|
||||
RDF triplestore,q2,6000,108666.874,30.2 h,FAIL
|
||||
RDF triplestore,q3,600,185722.881,2.1 d,FAIL
|
||||
RDF triplestore,q3,1200,371445.649,4.3 d,FAIL
|
||||
RDF triplestore,q3,6000,1857227.792,21.5 d,FAIL
|
||||
RDF triplestore,q4,600,79559.375,22.1 h,IMPRACTICAL
|
||||
RDF triplestore,q4,1200,159118.636,44.2 h,FAIL
|
||||
RDF triplestore,q4,6000,795592.727,9.2 d,FAIL
|
||||
RDF triplestore,q5,600,240054.647,2.8 d,FAIL
|
||||
RDF triplestore,q5,1200,480109.181,5.6 d,FAIL
|
||||
RDF triplestore,q5,6000,2400545.453,27.8 d,FAIL
|
||||
RDF triplestore,q6,600,352.729,5.9 min,OK
|
||||
RDF triplestore,q6,1200,727.088,12.1 min,OK
|
||||
RDF triplestore,q6,6000,3887.402,64.8 min,IMPRACTICAL
|
||||
RDF triplestore,q7,600,163225.475,45.3 h,FAIL
|
||||
RDF triplestore,q7,1200,326450.837,3.8 d,FAIL
|
||||
RDF triplestore,q7,6000,1632253.733,18.9 d,FAIL
|
||||
|
16
out/report/tables/t7_hardware_all_scales.csv
Normal file
@@ -0,0 +1,16 @@
|
||||
format,scale_gb,configuration,disk_tb,ram_gb,cores,nodes,ram_modules,nvme_drives,base_usd,ram_usd,disk_usd,extra_nodes_usd,est_cost_usd
|
||||
CSV,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,0.78,16.0,16,1,1,1,4618,1887,3995,0,10500
|
||||
CSV,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.56,16.0,16,1,1,1,4618,1887,3995,0,10500
|
||||
CSV,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,7.8,16.0,16,1,1,2,4618,1887,7990,0,14495
|
||||
JSON-LD,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.633,16.0,16,1,1,1,4618,1887,3995,0,10500
|
||||
JSON-LD,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,3.267,16.0,16,1,1,1,4618,1887,3995,0,10500
|
||||
JSON-LD,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,16.334,16.0,16,1,1,3,4618,1887,11985,0,18490
|
||||
SQLite,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,0.647,21.0,16,1,1,1,4618,1887,3995,0,10500
|
||||
SQLite,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,1.295,38.0,16,1,1,1,4618,1887,3995,0,10500
|
||||
SQLite,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 3 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,6.475,173.9,16,1,3,1,4618,5661,3995,0,14274
|
||||
PostgreSQL,600,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,2.086,58.7,16,1,1,1,4618,1887,3995,0,10500
|
||||
PostgreSQL,1200,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 2 x 64 GB DDR5 ECC RDIMM; 1 x 7.68 TB NVMe U.2,4.171,113.5,16,1,2,1,4618,3774,3995,0,12387
|
||||
PostgreSQL,6000,1 x 1U Supermicro AS-1015CS-TNR (16 cores); 9 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,20.855,551.3,16,1,9,3,4618,16983,11985,0,33586
|
||||
RDF triplestore,600,2 x 1U Supermicro AS-1015CS-TNR (16 cores); 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2,9.127,1404.1,16,2,22,2,4618,41514,7990,4618,58740
|
||||
RDF triplestore,1200,3 x 1U Supermicro AS-1015CS-TNR (16 cores); 44 x 64 GB DDR5 ECC RDIMM; 3 x 7.68 TB NVMe U.2,18.254,2808.3,16,3,44,3,4618,83028,11985,9236,108867
|
||||
RDF triplestore,6000,14 x 1U Supermicro AS-1015CS-TNR (16 cores); 220 x 64 GB DDR5 ECC RDIMM; 12 x 7.68 TB NVMe U.2,91.268,14041.3,16,14,220,12,4618,415140,47940,60034,527732
|
||||
|
361
report.py
@@ -2,13 +2,22 @@
|
||||
|
||||
Aggregates the measured matrix (task 09), the projections (task 10) and the
|
||||
process diagrams (task 02) into the final decision document. Writes 14 PNG
|
||||
charts (150 dpi, log scales where ranges span decades) under
|
||||
./out/report/charts/ with the exact names fixed in README.md, five CSV
|
||||
tables under ./out/report/tables/, and ./out/report/REPORT.md. Weighted
|
||||
scoring: search speed x5 (0-50), RAM economy x2 (0-20), disk economy x1
|
||||
(0-10), max 80; each subscore is 10 x best/value; a format with a FAIL
|
||||
projection at 600 GB scores 0 on search there (spec 11). Projected numbers
|
||||
are always labeled as projected (bench-methodology section 7).
|
||||
charts (150 dpi) under ./out/report/charts/ with the exact names fixed in
|
||||
README.md, seven CSV tables under ./out/report/tables/ (t6/t7 carry every
|
||||
projection and hardware row for all three scales - they are the committed
|
||||
proof artifacts the executive summary links to), and ./out/report/REPORT.md.
|
||||
|
||||
Chart readability contract (owner feedback 2026-07-11): linear scales with
|
||||
actual units everywhere (panels instead of log axes when magnitudes clash),
|
||||
no scientific tick notation, full format names, a value label on every bar
|
||||
and point, titles that state the takeaway. Hardware costs are a discrete
|
||||
bill of materials (whole RDIMM modules, whole NVMe drives, priced chassis)
|
||||
with street-price sources and an as-of date from out/config/hw_prices.yaml.
|
||||
|
||||
Weighted scoring: search speed x5 (0-50), RAM economy x2 (0-20), disk
|
||||
economy x1 (0-10), max 80; each subscore is 10 x best/value; a format with
|
||||
a FAIL projection at 600 GB scores 0 on search there (spec 11). Projected
|
||||
numbers are always labeled as projected (bench-methodology section 7).
|
||||
Spec: docs/specs/11_reporting.md.
|
||||
"""
|
||||
|
||||
@@ -24,6 +33,8 @@ import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
import yaml
|
||||
from matplotlib.ticker import FuncFormatter
|
||||
|
||||
from common.pipeline import (
|
||||
check_dependencies,
|
||||
@@ -42,12 +53,12 @@ FORMATS = ("csv", "json", "sqlite", "pg", "rdf")
|
||||
QUERY_NUMBERS = tuple(range(1, 8))
|
||||
SCALES_GB = (600, 1200, 6000)
|
||||
REPORT_SCALE_GB = 600 # the headline projection scale, spec 11
|
||||
SCALE_LABELS = {600: "600 GB", 1200: "1.2 TB", 6000: "6 TB"}
|
||||
|
||||
WEIGHT_SEARCH = 5 # x5 -> 0-50, spec 11
|
||||
WEIGHT_RAM = 2 # x2 -> 0-20
|
||||
WEIGHT_DISK = 1 # x1 -> 0-10
|
||||
|
||||
# archival (compressed) variant per format for table 1
|
||||
ARCHIVAL_VARIANT = {
|
||||
"csv": ("csv", "targz"), "json": ("json", "targz"), "sqlite": ("sqlite", "db_gz"),
|
||||
"pg": ("pg", "dump"), "rdf": ("rdf", "nt_gz"),
|
||||
@@ -57,12 +68,19 @@ WORKING_VARIANT = {
|
||||
"pg": ("pg", "live"), "rdf": ("rdf", "store"),
|
||||
}
|
||||
|
||||
# query classes for the degradation curves (C4), spec 11
|
||||
# query classes for the degradation charts (C4), spec 11
|
||||
QUERY_CLASSES = {
|
||||
"point_read": (1,),
|
||||
"indexed": (2, 3, 4, 6),
|
||||
"full_scan": (5, 7),
|
||||
}
|
||||
CLASS_LABELS = {"point_read": "point read", "indexed": "indexed / join", "full_scan": "full scan"}
|
||||
|
||||
QUERY_LABELS = {
|
||||
1: "single-track COF curve", 2: "coupons at Au 10+/-0.5 wt%",
|
||||
3: "hardness vs steady-state COF", 4: "anomaly filter",
|
||||
5: "run-in over raw cycles", 6: "wear volume vs load", 7: "Stribeck aggregation",
|
||||
}
|
||||
|
||||
DIAGRAM_TITLES = {
|
||||
"D1": "Coupon assembly", "D2": "Characterization and batch assembly",
|
||||
@@ -72,6 +90,7 @@ DIAGRAM_TITLES = {
|
||||
|
||||
FMT_COLORS = {"csv": "tab:gray", "json": "tab:orange", "sqlite": "tab:green", "pg": "tab:blue", "rdf": "tab:red"}
|
||||
FMT_NAMES = {"csv": "CSV", "json": "JSON-LD", "sqlite": "SQLite", "pg": "PostgreSQL", "rdf": "RDF triplestore"}
|
||||
CHASSIS = {16: "Supermicro AS-1015CS-TNR", 32: "Supermicro AS-1115CS-TNR"}
|
||||
|
||||
|
||||
def read_medians(path: Path) -> dict[tuple[str, int], dict]:
|
||||
@@ -117,10 +136,18 @@ def read_sizing(path: Path) -> dict[tuple[str, int], dict]:
|
||||
"disk_tb": float(row["disk_tb"]), "ram_gb": float(row["ram_gb"]),
|
||||
"cores": int(row["cores"]), "nodes": int(row["nodes"]),
|
||||
"est_cost_usd": float(row["est_cost_usd"]),
|
||||
"ram_modules": int(row["ram_modules"]), "nvme_drives": int(row["nvme_drives"]),
|
||||
"base_usd": float(row["base_usd"]), "ram_usd": float(row["ram_usd"]),
|
||||
"disk_usd": float(row["disk_usd"]), "extra_nodes_usd": float(row["extra_nodes_usd"]),
|
||||
}
|
||||
return sizing
|
||||
|
||||
|
||||
def read_hw_prices(path: Path) -> dict:
|
||||
with open(path, encoding="ascii") as fh:
|
||||
return yaml.safe_load(fh)
|
||||
|
||||
|
||||
def geometric_mean(values: list[float]) -> float:
|
||||
return math.exp(sum(math.log(v) for v in values) / len(values))
|
||||
|
||||
@@ -133,7 +160,7 @@ def md_table(headers: list[str], rows: list[list]) -> str:
|
||||
|
||||
def human_time(seconds: float) -> str:
|
||||
if seconds < 120:
|
||||
return f"{seconds:.2f} s"
|
||||
return f"{seconds:.2f} s" if seconds < 10 else f"{seconds:.0f} s"
|
||||
if seconds < 2 * 3600:
|
||||
return f"{seconds / 60:.1f} min"
|
||||
if seconds < 2 * 86400:
|
||||
@@ -141,6 +168,17 @@ def human_time(seconds: float) -> str:
|
||||
return f"{seconds / 86400:.1f} d"
|
||||
|
||||
|
||||
def human_size(nbytes: float) -> str:
|
||||
for unit, div in (("TB", 1e12), ("GB", 1e9), ("MB", 1e6)):
|
||||
if nbytes >= div:
|
||||
return f"{nbytes / div:.2f} {unit}" if nbytes < 10 * div else f"{nbytes / div:.0f} {unit}"
|
||||
return f"{nbytes / 1e3:.0f} KB"
|
||||
|
||||
|
||||
def money(value: float) -> str:
|
||||
return f"${value:,.0f}"
|
||||
|
||||
|
||||
def write_csv(path: Path, headers: list[str], rows: list[list]) -> None:
|
||||
with open(path, "w", encoding="ascii", newline="") as fh:
|
||||
writer = csv.writer(fh)
|
||||
@@ -159,7 +197,6 @@ def scores_for(metrics: dict[str, dict[str, float]], failed_search: set[str]) ->
|
||||
ram = 10.0 * best["ram"] / m["ram"]
|
||||
disk = 10.0 * best["disk"] / m["disk"]
|
||||
out[fmt] = {
|
||||
"search_sub": round(search, 2), "ram_sub": round(ram, 2), "disk_sub": round(disk, 2),
|
||||
"search_pts": round(search * WEIGHT_SEARCH, 1),
|
||||
"ram_pts": round(ram * WEIGHT_RAM, 1),
|
||||
"disk_pts": round(disk * WEIGHT_DISK, 1),
|
||||
@@ -193,40 +230,60 @@ def build_scoreboards(medians, proj, sizes, sizing) -> dict[str, dict[str, dict]
|
||||
|
||||
|
||||
# --- charts ---------------------------------------------------------------
|
||||
# Shared conventions: linear axes, real units, a value label on every bar
|
||||
# and point, full format names, no scientific notation.
|
||||
|
||||
def _grouped_bars(ax, labels, series: dict[str, list[float]]) -> None:
|
||||
width = 0.8 / len(series)
|
||||
for i, (name, values) in enumerate(series.items()):
|
||||
x = [j + i * width for j in range(len(labels))]
|
||||
ax.bar(x, values, width=width, label=name)
|
||||
ax.set_xticks([j + 0.4 - width / 2 for j in range(len(labels))])
|
||||
ax.set_xticklabels(labels)
|
||||
ax.set_yscale("log")
|
||||
ax.legend()
|
||||
ax.grid(axis="y", alpha=0.3)
|
||||
def _vbar_panel(ax, values: dict[str, float], labels: dict[str, str], unit: str, title: str) -> None:
|
||||
bars = ax.bar(range(len(FORMATS)), [values[f] for f in FORMATS], color=[FMT_COLORS[f] for f in FORMATS])
|
||||
for bar, fmt in zip(bars, FORMATS):
|
||||
ax.annotate(labels[fmt], (bar.get_x() + bar.get_width() / 2, bar.get_height()),
|
||||
ha="center", va="bottom", fontsize=8)
|
||||
ax.set_xticks(range(len(FORMATS)))
|
||||
ax.set_xticklabels([FMT_NAMES[f] for f in FORMATS], rotation=12, fontsize=8)
|
||||
ax.set_ylabel(unit)
|
||||
ax.set_title(title, fontsize=10)
|
||||
ax.margins(y=0.18)
|
||||
ax.grid(axis="y", alpha=0.25)
|
||||
|
||||
|
||||
def chart_c1(charts_dir, sizes, proj_bytes) -> None:
|
||||
fig, ax = plt.subplots(figsize=(7, 4.5))
|
||||
_grouped_bars(ax, list(FORMATS), {
|
||||
"measured (150 MB corpus)": [sizes[WORKING_VARIANT[fmt]] / 1e6 for fmt in FORMATS],
|
||||
"projected @600 GB": [proj_bytes[fmt] / 1e6 for fmt in FORMATS],
|
||||
})
|
||||
ax.set_ylabel("working footprint, MB (log)")
|
||||
ax.set_title("C1 - disk footprint per format")
|
||||
def _hbar_panel(ax, values: dict[str, float], labels: dict[str, str], title: str, names: bool) -> None:
|
||||
y = range(len(FORMATS))
|
||||
bars = ax.barh(list(y), [values[f] for f in FORMATS], color=[FMT_COLORS[f] for f in FORMATS])
|
||||
for bar, fmt in zip(bars, FORMATS):
|
||||
ax.annotate(" " + labels[fmt], (bar.get_width(), bar.get_y() + bar.get_height() / 2),
|
||||
va="center", fontsize=8)
|
||||
ax.set_yticks(list(y))
|
||||
ax.set_yticklabels([FMT_NAMES[f] for f in FORMATS] if names else [""] * len(FORMATS), fontsize=8)
|
||||
ax.invert_yaxis()
|
||||
ax.set_xticks([]) # linear axis; the value labels on the bars carry the numbers
|
||||
ax.margins(x=0.30)
|
||||
ax.set_title(title, fontsize=10)
|
||||
|
||||
|
||||
def chart_c1(charts_dir, sizes, proj_bytes, coeff) -> None:
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
|
||||
measured = {f: sizes[WORKING_VARIANT[f]] for f in FORMATS}
|
||||
_vbar_panel(axes[0], {f: measured[f] / 1e9 for f in FORMATS},
|
||||
{f: human_size(measured[f]) for f in FORMATS}, "GB (linear)", "measured on the 150 MB corpus")
|
||||
_vbar_panel(axes[1], {f: proj_bytes[f] / 1e9 for f in FORMATS},
|
||||
{f: human_size(proj_bytes[f]) for f in FORMATS}, "GB (linear)", "projected at 600 GB of raw data")
|
||||
fig.suptitle(f"C1 - disk footprint: RDF stores the same data in {coeff['rdf']:.0f}x the raw CSV size, SQLite in {coeff['sqlite']:.2f}x")
|
||||
fig.tight_layout()
|
||||
fig.savefig(charts_dir / "c1_disk_footprint.png", dpi=DPI)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def chart_c2(charts_dir, medians, sizing) -> None:
|
||||
fig, ax = plt.subplots(figsize=(7, 4.5))
|
||||
_grouped_bars(ax, list(FORMATS), {
|
||||
"measured client peak RSS, MB": [max(medians[(fmt, qn)]["peak_rss_mb"] for qn in QUERY_NUMBERS) for fmt in FORMATS],
|
||||
"recommended RAM @600 GB, MB (projected)": [sizing[(fmt, REPORT_SCALE_GB)]["ram_gb"] * 1024 for fmt in FORMATS],
|
||||
})
|
||||
ax.set_ylabel("MB (log)")
|
||||
ax.set_title("C2 - RAM requirement per format")
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
|
||||
rss = {f: max(medians[(f, qn)]["peak_rss_mb"] for qn in QUERY_NUMBERS) for f in FORMATS}
|
||||
ram = {f: sizing[(f, REPORT_SCALE_GB)]["ram_gb"] for f in FORMATS}
|
||||
_vbar_panel(axes[0], rss, {f: f"{rss[f]:.0f} MB" for f in FORMATS},
|
||||
"MB (linear)", "measured client peak RSS on 150 MB")
|
||||
_vbar_panel(axes[1], ram, {f: f"{ram[f]:.0f} GB" for f in FORMATS},
|
||||
"GB (linear)", "recommended server RAM at 600 GB (projected)")
|
||||
axes[1].axhline(1024, color="gray", linestyle="--", linewidth=1)
|
||||
axes[1].annotate("single-node ceiling: 1 TB RAM", (0, 1024), fontsize=8, va="bottom")
|
||||
fig.suptitle("C2 - RAM: only the RDF hot set outgrows a single 1 TB node at 600 GB")
|
||||
fig.tight_layout()
|
||||
fig.savefig(charts_dir / "c2_ram_requirement.png", dpi=DPI)
|
||||
plt.close(fig)
|
||||
@@ -234,82 +291,89 @@ def chart_c2(charts_dir, medians, sizing) -> None:
|
||||
|
||||
def chart_c3(charts_dir, medians, proj) -> None:
|
||||
for qn in QUERY_NUMBERS:
|
||||
fig, ax = plt.subplots(figsize=(7, 4.5))
|
||||
_grouped_bars(ax, list(FORMATS), {
|
||||
"measured (150 MB)": [medians[(fmt, qn)]["wall_s"] for fmt in FORMATS],
|
||||
"projected @600 GB": [proj[(fmt, qn, REPORT_SCALE_GB)][0] for fmt in FORMATS],
|
||||
})
|
||||
ax.set_ylabel("wall time, s (log)")
|
||||
ax.set_title(f"C3 - Q{qn} wall time per format")
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11, 3.6))
|
||||
measured = {f: medians[(f, qn)]["wall_s"] for f in FORMATS}
|
||||
projected = {f: proj[(f, qn, REPORT_SCALE_GB)][0] for f in FORMATS}
|
||||
_hbar_panel(axes[0], measured, {f: human_time(measured[f]) for f in FORMATS},
|
||||
"measured on 150 MB", names=True)
|
||||
_hbar_panel(axes[1], projected, {f: human_time(projected[f]) for f in FORMATS},
|
||||
"projected at 600 GB", names=False)
|
||||
fastest = min(FORMATS, key=lambda f: projected[f])
|
||||
slowest = max(FORMATS, key=lambda f: projected[f])
|
||||
fig.suptitle(f"C3 - Q{qn} ({QUERY_LABELS[qn]}) at 600 GB: {FMT_NAMES[fastest]} {human_time(projected[fastest])}"
|
||||
f" vs {FMT_NAMES[slowest]} {human_time(projected[slowest])}")
|
||||
fig.tight_layout()
|
||||
fig.savefig(charts_dir / f"c3_q{qn}_wall_time.png", dpi=DPI)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def chart_c4(charts_dir, medians, proj, measured_gb: float) -> None:
|
||||
def chart_c4(charts_dir, medians, proj, measured_label: str) -> None:
|
||||
for cls, queries in QUERY_CLASSES.items():
|
||||
fig, ax = plt.subplots(figsize=(7, 4.5))
|
||||
for fmt in FORMATS:
|
||||
x = [measured_gb, *SCALES_GB]
|
||||
y = [geometric_mean([medians[(fmt, qn)]["wall_s"] for qn in queries])]
|
||||
y += [geometric_mean([proj[(fmt, qn, s)][0] for qn in queries]) for s in SCALES_GB]
|
||||
ax.plot(x, y, marker="o", label=fmt, color=FMT_COLORS[fmt])
|
||||
ax.set_xscale("log")
|
||||
ax.set_yscale("log")
|
||||
ax.axhline(3600, color="gray", linestyle="--", linewidth=1)
|
||||
ax.text(measured_gb, 3600, "IMPRACTICAL (1 h)", fontsize=7, va="bottom")
|
||||
ax.set_xlabel("raw data size, GB (log); first point measured, rest projected")
|
||||
ax.set_ylabel("wall time, s (log)")
|
||||
fig, axes = plt.subplots(1, 1 + len(SCALES_GB), figsize=(13, 3.6))
|
||||
measured = {f: geometric_mean([medians[(f, qn)]["wall_s"] for qn in queries]) for f in FORMATS}
|
||||
_hbar_panel(axes[0], measured, {f: human_time(measured[f]) for f in FORMATS}, measured_label, names=True)
|
||||
for ax, scale in zip(axes[1:], SCALES_GB):
|
||||
values = {f: geometric_mean([proj[(f, qn, scale)][0] for qn in queries]) for f in FORMATS}
|
||||
_hbar_panel(ax, values, {f: human_time(values[f]) for f in FORMATS},
|
||||
f"projected {SCALE_LABELS[scale]}", names=False)
|
||||
queries_label = ", ".join(f"Q{qn}" for qn in queries)
|
||||
ax.set_title(f"C4 - degradation, {cls.replace('_', ' ')} ({queries_label})")
|
||||
ax.legend()
|
||||
ax.grid(alpha=0.3)
|
||||
fig.suptitle(f"C4 - {CLASS_LABELS[cls]} queries ({queries_label}): wall time growth by data size"
|
||||
" (every panel after the first is projected; linear scale within each panel)")
|
||||
fig.tight_layout()
|
||||
fig.savefig(charts_dir / f"c4_degradation_{cls}.png", dpi=DPI)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def chart_c5(charts_dir, scoreboards) -> None:
|
||||
def chart_c5(charts_dir, scoreboards, winner: str) -> None:
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11, 4.5), sharey=True)
|
||||
for ax, (title, board) in zip(axes, [
|
||||
("measured scale (150 MB)", scoreboards["measured"]),
|
||||
("projected @600 GB", scoreboards["projected_600gb"]),
|
||||
("projected at 600 GB", scoreboards["projected_600gb"]),
|
||||
]):
|
||||
x = range(len(FORMATS))
|
||||
search = [board[fmt]["search_pts"] for fmt in FORMATS]
|
||||
ram = [board[fmt]["ram_pts"] for fmt in FORMATS]
|
||||
disk = [board[fmt]["disk_pts"] for fmt in FORMATS]
|
||||
ax.bar(x, search, label=f"search speed x{WEIGHT_SEARCH}")
|
||||
ax.bar(x, ram, bottom=search, label=f"RAM economy x{WEIGHT_RAM}")
|
||||
ax.bar(x, disk, bottom=[s + r for s, r in zip(search, ram)], label=f"disk economy x{WEIGHT_DISK}")
|
||||
ax.bar(x, search, label=f"search speed x{WEIGHT_SEARCH} (0-50)")
|
||||
ax.bar(x, ram, bottom=search, label=f"RAM economy x{WEIGHT_RAM} (0-20)")
|
||||
ax.bar(x, disk, bottom=[s + r for s, r in zip(search, ram)], label=f"disk economy x{WEIGHT_DISK} (0-10)")
|
||||
for i, fmt in enumerate(FORMATS):
|
||||
ax.text(i, board[fmt]["total"] + 1, f"{board[fmt]['total']:.0f}", ha="center", fontsize=8)
|
||||
ax.text(i, board[fmt]["total"] + 1.5, f"{board[fmt]['total']:.0f}", ha="center", fontsize=9)
|
||||
ax.set_xticks(list(x))
|
||||
ax.set_xticklabels(FORMATS)
|
||||
ax.set_xticklabels([FMT_NAMES[f] for f in FORMATS], rotation=12, fontsize=8)
|
||||
ax.set_ylim(0, 85)
|
||||
ax.set_title(title)
|
||||
ax.grid(axis="y", alpha=0.3)
|
||||
axes[0].set_ylabel("weighted score (max 80)")
|
||||
ax.set_title(title, fontsize=10)
|
||||
ax.grid(axis="y", alpha=0.25)
|
||||
axes[0].set_ylabel("weighted score (linear, max 80)")
|
||||
axes[0].legend(loc="upper left", fontsize=8)
|
||||
fig.suptitle("C5 - weighted score, stacked components")
|
||||
board600 = scoreboards["projected_600gb"]
|
||||
fig.suptitle(f"C5 - weighted score (max 80): {FMT_NAMES[winner]} leads the 600 GB scoreboard with {board600[winner]['total']:.0f} points")
|
||||
fig.tight_layout()
|
||||
fig.savefig(charts_dir / "c5_weighted_score.png", dpi=DPI)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def chart_c6(charts_dir, proj, sizing) -> None:
|
||||
fig, ax = plt.subplots(figsize=(7, 4.5))
|
||||
fig, ax = plt.subplots(figsize=(9, 5.5))
|
||||
offsets = {"csv": (12, 0), "json": (12, -4), "sqlite": (12, 10), "pg": (12, -16), "rdf": (-40, -20)}
|
||||
geo = {f: geometric_mean([proj[(f, qn, REPORT_SCALE_GB)][0] for qn in QUERY_NUMBERS]) for f in FORMATS}
|
||||
cost = {f: sizing[(f, REPORT_SCALE_GB)]["est_cost_usd"] for f in FORMATS}
|
||||
for fmt in FORMATS:
|
||||
cost = sizing[(fmt, REPORT_SCALE_GB)]["est_cost_usd"]
|
||||
geo = geometric_mean([proj[(fmt, qn, REPORT_SCALE_GB)][0] for qn in QUERY_NUMBERS])
|
||||
ax.scatter(cost, geo, s=60, color=FMT_COLORS[fmt])
|
||||
ax.annotate(fmt, (cost, geo), textcoords="offset points", xytext=(6, 4))
|
||||
ax.set_xscale("log")
|
||||
ax.set_yscale("log")
|
||||
ax.set_xlabel("estimated hardware cost @600 GB, USD (log)")
|
||||
ax.set_ylabel("geometric mean projected wall time Q1-Q7, s (log)")
|
||||
ax.set_title("C6 - cost vs performance @600 GB (projected)")
|
||||
ax.grid(alpha=0.3)
|
||||
ax.scatter(cost[fmt], geo[fmt] / 60, s=70, color=FMT_COLORS[fmt])
|
||||
ax.annotate(f"{FMT_NAMES[fmt]}: {money(cost[fmt])}, {human_time(geo[fmt])}",
|
||||
(cost[fmt], geo[fmt] / 60), textcoords="offset points", xytext=offsets[fmt], fontsize=8)
|
||||
ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: f"${v / 1000:.0f}k"))
|
||||
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0f} min"))
|
||||
ax.set_xlabel("estimated hardware cost at 600 GB (bill of materials, linear)")
|
||||
ax.set_ylabel("typical query time at 600 GB, projected (geometric mean Q1-Q7, linear)")
|
||||
ax.annotate("better: lower-left", xy=(0.05, 0.04), xytext=(0.42, 0.10),
|
||||
xycoords="axes fraction", textcoords="axes fraction",
|
||||
fontsize=10, color="darkgreen", fontweight="bold",
|
||||
arrowprops={"arrowstyle": "->", "color": "darkgreen"})
|
||||
ax.margins(x=0.18, y=0.12)
|
||||
ax.grid(alpha=0.25)
|
||||
fastest = min(FORMATS, key=lambda f: geo[f])
|
||||
ax.set_title(f"C6 - cost vs speed at 600 GB (projected): {FMT_NAMES[fastest]} is the fastest at {money(cost[fastest])}")
|
||||
fig.tight_layout()
|
||||
fig.savefig(charts_dir / "c6_cost_vs_performance.png", dpi=DPI)
|
||||
plt.close(fig)
|
||||
@@ -319,17 +383,26 @@ def chart_c6(charts_dir, proj, sizing) -> None:
|
||||
|
||||
def scoreboard_rows(board: dict[str, dict]) -> list[list]:
|
||||
ranked = sorted(FORMATS, key=lambda f: -board[f]["total"])
|
||||
return [[fmt, board[fmt]["search_pts"], board[fmt]["ram_pts"], board[fmt]["disk_pts"], board[fmt]["total"]] for fmt in ranked]
|
||||
return [[FMT_NAMES[fmt], board[fmt]["search_pts"], board[fmt]["ram_pts"], board[fmt]["disk_pts"], board[fmt]["total"]] for fmt in ranked]
|
||||
|
||||
|
||||
def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[list]]],
|
||||
scoreboards, environment: str, diagrams: dict[str, str], winner: str) -> str:
|
||||
def config_string(row: dict, prices: dict) -> str:
|
||||
return (f"{row['nodes']} x 1U {CHASSIS[row['cores']]} ({row['cores']} cores);"
|
||||
f" {row['ram_modules']} x {prices['ram_module_gb']} GB DDR5 ECC RDIMM;"
|
||||
f" {row['nvme_drives']} x {prices['nvme_drive_tb']} TB NVMe U.2")
|
||||
|
||||
|
||||
def assemble_report(tables: dict[str, tuple[list[str], list[list]]], scoreboards,
|
||||
environment: str, diagrams: dict[str, str], winner: str, sizing, prices: dict) -> str:
|
||||
t1_h, t1_r = tables["t1_storage_footprint"]
|
||||
t2_h, t2_r = tables["t2_measured_medians"]
|
||||
t3_h, t3_r = tables["t3_projected_600gb"]
|
||||
t4_h, t4_r = tables["t4_hardware_600gb"]
|
||||
pg6 = sizing[("pg", 6000)]
|
||||
rdf6 = sizing[("rdf", 6000)]
|
||||
as_of = prices["as_of"]
|
||||
sources = prices["sources"]
|
||||
|
||||
board600 = scoreboards["projected_600gb"]
|
||||
parts = [
|
||||
"# Laboratory Data Storage Format Evaluation - Final Report",
|
||||
"",
|
||||
@@ -345,9 +418,10 @@ def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[li
|
||||
"",
|
||||
"- **PostgreSQL** is the only format whose seven benchmark queries stay"
|
||||
" below the 1-hour practicality bound at every projected scale (600 GB,"
|
||||
" 1.2 TB, 6 TB); at 6 TB it needs a single 16-core node with ~551 GB"
|
||||
" RAM (~$19k, projected). Partitioned parallel scans and the"
|
||||
" track_summary materialized view are the scaling levers.",
|
||||
f" 1.2 TB, 6 TB); at 6 TB it needs one {pg6['cores']}-core node with"
|
||||
f" {pg6['ram_gb']:.0f} GB RAM at an estimated {money(pg6['est_cost_usd'])}"
|
||||
f" (bill of materials, prices as of {as_of}). Partitioned parallel scans"
|
||||
" and the track_summary materialized view are the scaling levers.",
|
||||
"- **CSV** remains the canonical raw archive: byte-reproducible,"
|
||||
" instrument-native, and the best compressor (tar.gz ~2.8x); its full"
|
||||
" scans exceed 1 h beyond ~600 GB, so it is an archive, not a query"
|
||||
@@ -359,9 +433,10 @@ def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[li
|
||||
" corpus) is the exchange format; the FULL variant re-reads the whole"
|
||||
" corpus per scan and becomes impractical past 600 GB.",
|
||||
"- **RDF (materialized triplestore)** fails at scale: 11.7x storage"
|
||||
" blow-up, 5 of 7 queries FAIL at 6 TB, hot-set RAM exceeds the 1 TB"
|
||||
" single-node ceiling (14 nodes, ~$303k projected). Semantic access"
|
||||
" should be a virtual layer (e.g. Ontop OBDA) over PostgreSQL instead.",
|
||||
" blow-up, 5 of 7 queries FAIL at 6 TB, and the hot set needs"
|
||||
f" {rdf6['ram_gb'] / 1024:.1f} TB RAM across {rdf6['nodes']} nodes"
|
||||
f" (~{money(rdf6['est_cost_usd'])} projected). Semantic access should be"
|
||||
" a virtual layer (e.g. Ontop OBDA) over PostgreSQL instead.",
|
||||
"",
|
||||
"## 2. Methodology snapshot",
|
||||
"",
|
||||
@@ -392,9 +467,11 @@ def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[li
|
||||
md_table(t2_h, t2_r),
|
||||
"",
|
||||
"Sub-interval note: cells faster than the 50 ms sampling cadence"
|
||||
" (fast sqlite/csv point reads) under-report client RSS/CPU; wall"
|
||||
" (fast SQLite/CSV point reads) under-report client RSS/CPU; wall"
|
||||
" times are exact.",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"Per-query charts: " + ", ".join(f"[Q{qn}](charts/c3_q{qn}_wall_time.png)" for qn in QUERY_NUMBERS),
|
||||
"",
|
||||
"## 6. Projections (600 GB / 1.2 TB / 6 TB)",
|
||||
@@ -405,21 +482,38 @@ def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[li
|
||||
" calibrated on the measured point after subtracting the per-format"
|
||||
" harness floor. Flags: IMPRACTICAL > 1 h, FAIL > 24 h.",
|
||||
"",
|
||||
"Projected wall times @600 GB:",
|
||||
"Projected wall times at 600 GB:",
|
||||
"",
|
||||
md_table(t3_h, t3_r),
|
||||
"",
|
||||
"Degradation curves: [point read](charts/c4_degradation_point_read.png),"
|
||||
" [indexed](charts/c4_degradation_indexed.png),"
|
||||
"Degradation by data size: [point read](charts/c4_degradation_point_read.png),"
|
||||
" [indexed / join](charts/c4_degradation_indexed.png),"
|
||||
" [full scan](charts/c4_degradation_full_scan.png)",
|
||||
"",
|
||||
"## 7. Hardware sizing and cost @600 GB (projected)",
|
||||
"## 7. Hardware sizing and cost at 600 GB (projected)",
|
||||
"",
|
||||
f"Bill-of-materials costing with street prices as of **{as_of}**"
|
||||
" (sources below). Every configuration buys whole components: 64 GB"
|
||||
" DDR5 ECC RDIMM modules, 7.68 TB enterprise NVMe U.2 drives, and a"
|
||||
" priced 1U chassis; disk capacity includes 30% free-space headroom.",
|
||||
"",
|
||||
md_table(t4_h, t4_r),
|
||||
"",
|
||||
f"Component prices (retrieved {as_of}):",
|
||||
"",
|
||||
f"- {prices['ram_module_gb']} GB DDR5-5600 ECC RDIMM (A-Tech, 2x32GB kit):"
|
||||
f" {money(prices['ram_module_usd'])} - <{sources['ram']}>",
|
||||
f"- {prices['nvme_drive_tb']} TB NVMe U.2 1DWPD enterprise SSD (Dell 14-16G"
|
||||
f" compatible, Cloud Ninjas): {money(prices['nvme_drive_usd'])} - <{sources['nvme']}>",
|
||||
f"- 1U AMD EPYC 9004/9005 chassis (Broadberry starting configurations):"
|
||||
f" {CHASSIS[16]} {money(prices['server_16_core_usd'])},"
|
||||
f" {CHASSIS[32]} {money(prices['server_32_core_usd'])} - <{sources['servers']}>",
|
||||
f"- Market context - server DRAM is in a documented 2025-2026 price"
|
||||
f" surge: <{sources['market_context']}>",
|
||||
"",
|
||||
"Full sizing at 1.2 TB and 6 TB: out/bench/hardware_sizing.csv."
|
||||
" RDF exceeds the 1 TB single-node RAM ceiling already at 600 GB"
|
||||
" (1.4 TB hot set) and reaches 14 nodes at 6 TB.",
|
||||
f" RDF exceeds the 1 TB single-node RAM ceiling already at 600 GB and"
|
||||
f" reaches {rdf6['nodes']} nodes at 6 TB.",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
@@ -436,10 +530,10 @@ def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[li
|
||||
md_table(["format", f"search (x{WEIGHT_SEARCH})", f"RAM (x{WEIGHT_RAM})", f"disk (x{WEIGHT_DISK})", "total (max 80)"],
|
||||
scoreboard_rows(scoreboards["measured"])),
|
||||
"",
|
||||
"Projected @600 GB:",
|
||||
"Projected at 600 GB:",
|
||||
"",
|
||||
md_table(["format", f"search (x{WEIGHT_SEARCH})", f"RAM (x{WEIGHT_RAM})", f"disk (x{WEIGHT_DISK})", "total (max 80)"],
|
||||
scoreboard_rows(board600)),
|
||||
scoreboard_rows(scoreboards["projected_600gb"])),
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
@@ -473,6 +567,9 @@ def assemble_report(report_dir: Path, tables: dict[str, tuple[list[str], list[li
|
||||
"- RDF numbers reflect the compact modeling and a client that derives"
|
||||
" run-in / steady-state from raw cycles; SPARQL-side aggregation"
|
||||
" engines could shift (not remove) the scan penalty.",
|
||||
f"- Hardware prices are spot street prices retrieved {as_of} during a"
|
||||
" documented server-DRAM price surge; re-cost by editing"
|
||||
" out/config/hw_prices.yaml and re-running tasks 10-11.",
|
||||
"",
|
||||
"## Appendix A - process flow diagrams (task 02)",
|
||||
"",
|
||||
@@ -494,6 +591,7 @@ def main() -> int:
|
||||
proj = read_projections(bench_dir / "extrapolation.csv")
|
||||
sizes = read_sizes(bench_dir / "storage_sizes.csv")
|
||||
sizing = read_sizing(bench_dir / "hardware_sizing.csv")
|
||||
prices = read_hw_prices(out_root / "config" / "hw_prices.yaml")
|
||||
environment = (bench_dir / "environment.txt").read_text(encoding="utf-8")
|
||||
diagrams = {p.stem: p.read_text(encoding="ascii") for p in sorted((report_dir / "diagrams").glob("D*.mermaid"))}
|
||||
if len(diagrams) != 6:
|
||||
@@ -505,57 +603,77 @@ def main() -> int:
|
||||
tables_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
csv_bytes = sizes[WORKING_VARIANT["csv"]]
|
||||
measured_gb = csv_bytes / 1e9
|
||||
coeff = {fmt: sizes[WORKING_VARIANT[fmt]] / csv_bytes for fmt in FORMATS}
|
||||
proj_bytes_600 = {fmt: coeff[fmt] * REPORT_SCALE_GB * 1e9 for fmt in FORMATS}
|
||||
|
||||
tables: dict[str, tuple[list[str], list[list]]] = {}
|
||||
tables["t1_storage_footprint"] = (
|
||||
["format", "measured_mb", "coeff_vs_csv", "archival_mb (compressed)",
|
||||
"projected_600gb_gb", "projected_1200gb_gb", "projected_6000gb_gb (all projected)"],
|
||||
[[fmt, round(sizes[WORKING_VARIANT[fmt]] / 1e6, 1), round(coeff[fmt], 2),
|
||||
round(sizes[ARCHIVAL_VARIANT[fmt]] / 1e6, 1),
|
||||
*[round(coeff[fmt] * s * 1e9 / 1e9, 0) for s in SCALES_GB]] for fmt in FORMATS],
|
||||
["format", "measured", "vs CSV", "archival (compressed)",
|
||||
"projected @600 GB", "projected @1.2 TB", "projected @6 TB"],
|
||||
[[FMT_NAMES[fmt], human_size(sizes[WORKING_VARIANT[fmt]]), f"{coeff[fmt]:.2f}x",
|
||||
human_size(sizes[ARCHIVAL_VARIANT[fmt]]),
|
||||
*[human_size(coeff[fmt] * s * 1e9) for s in SCALES_GB]] for fmt in FORMATS],
|
||||
)
|
||||
tables["t2_measured_medians"] = (
|
||||
["format", "query", "wall_s", "peak_rss_mb", "cpu_util_pct", "read_mb"],
|
||||
[[fmt, f"q{qn}", medians[(fmt, qn)]["wall_s"], medians[(fmt, qn)]["peak_rss_mb"],
|
||||
medians[(fmt, qn)]["cpu_util_pct"], medians[(fmt, qn)]["read_mb"]]
|
||||
["format", "query", "wall time", "peak RSS", "CPU util", "read"],
|
||||
[[FMT_NAMES[fmt], f"q{qn}", human_time(medians[(fmt, qn)]["wall_s"]),
|
||||
f"{medians[(fmt, qn)]['peak_rss_mb']:.0f} MB",
|
||||
f"{medians[(fmt, qn)]['cpu_util_pct']:.0f}%",
|
||||
f"{medians[(fmt, qn)]['read_mb']:.1f} MB"]
|
||||
for fmt in FORMATS for qn in QUERY_NUMBERS],
|
||||
)
|
||||
tables["t3_projected_600gb"] = (
|
||||
["format", "query", "projected_wall", "flag"],
|
||||
[[fmt, f"q{qn}", human_time(proj[(fmt, qn, REPORT_SCALE_GB)][0]), proj[(fmt, qn, REPORT_SCALE_GB)][1]]
|
||||
["format", "query", "projected wall time", "flag"],
|
||||
[[FMT_NAMES[fmt], f"q{qn}", human_time(proj[(fmt, qn, REPORT_SCALE_GB)][0]), proj[(fmt, qn, REPORT_SCALE_GB)][1]]
|
||||
for fmt in FORMATS for qn in QUERY_NUMBERS],
|
||||
)
|
||||
tables["t4_hardware_600gb"] = (
|
||||
["format", "disk_tb", "ram_gb", "cores", "nodes", "est_cost_usd"],
|
||||
[[fmt, sizing[(fmt, REPORT_SCALE_GB)]["disk_tb"], sizing[(fmt, REPORT_SCALE_GB)]["ram_gb"],
|
||||
sizing[(fmt, REPORT_SCALE_GB)]["cores"], sizing[(fmt, REPORT_SCALE_GB)]["nodes"],
|
||||
round(sizing[(fmt, REPORT_SCALE_GB)]["est_cost_usd"])] for fmt in FORMATS],
|
||||
["format", "configuration", "chassis", "RAM", "disk", "extra nodes", "total"],
|
||||
[[FMT_NAMES[fmt], config_string(sizing[(fmt, REPORT_SCALE_GB)], prices),
|
||||
money(sizing[(fmt, REPORT_SCALE_GB)]["base_usd"]),
|
||||
money(sizing[(fmt, REPORT_SCALE_GB)]["ram_usd"]),
|
||||
money(sizing[(fmt, REPORT_SCALE_GB)]["disk_usd"]),
|
||||
money(sizing[(fmt, REPORT_SCALE_GB)]["extra_nodes_usd"]),
|
||||
money(sizing[(fmt, REPORT_SCALE_GB)]["est_cost_usd"])] for fmt in FORMATS],
|
||||
)
|
||||
|
||||
tables["t6_projected_all_scales"] = (
|
||||
["format", "query", "scale_gb", "projected_wall_s", "projected_wall", "flag"],
|
||||
[[FMT_NAMES[fmt], f"q{qn}", s, proj[(fmt, qn, s)][0], human_time(proj[(fmt, qn, s)][0]), proj[(fmt, qn, s)][1]]
|
||||
for fmt in FORMATS for qn in QUERY_NUMBERS for s in SCALES_GB],
|
||||
)
|
||||
tables["t7_hardware_all_scales"] = (
|
||||
["format", "scale_gb", "configuration", "disk_tb", "ram_gb", "cores", "nodes",
|
||||
"ram_modules", "nvme_drives", "base_usd", "ram_usd", "disk_usd", "extra_nodes_usd", "est_cost_usd"],
|
||||
[[FMT_NAMES[fmt], s, config_string(sizing[(fmt, s)], prices),
|
||||
sizing[(fmt, s)]["disk_tb"], sizing[(fmt, s)]["ram_gb"],
|
||||
sizing[(fmt, s)]["cores"], sizing[(fmt, s)]["nodes"],
|
||||
sizing[(fmt, s)]["ram_modules"], sizing[(fmt, s)]["nvme_drives"],
|
||||
round(sizing[(fmt, s)]["base_usd"]), round(sizing[(fmt, s)]["ram_usd"]),
|
||||
round(sizing[(fmt, s)]["disk_usd"]), round(sizing[(fmt, s)]["extra_nodes_usd"]),
|
||||
round(sizing[(fmt, s)]["est_cost_usd"])] for fmt in FORMATS for s in SCALES_GB],
|
||||
)
|
||||
scoreboards = build_scoreboards(medians, proj, sizes, sizing)
|
||||
tables["t5_weighted_scores"] = (
|
||||
["scale", "format", "search_pts", "ram_pts", "disk_pts", "total"],
|
||||
[[scale, fmt, board[fmt]["search_pts"], board[fmt]["ram_pts"], board[fmt]["disk_pts"], board[fmt]["total"]]
|
||||
[[scale, FMT_NAMES[fmt], board[fmt]["search_pts"], board[fmt]["ram_pts"], board[fmt]["disk_pts"], board[fmt]["total"]]
|
||||
for scale, board in scoreboards.items() for fmt in FORMATS],
|
||||
)
|
||||
for name, (headers, rows) in tables.items():
|
||||
write_csv(tables_dir / f"{name}.csv", headers, rows)
|
||||
|
||||
chart_c1(charts_dir, sizes, proj_bytes_600)
|
||||
winner = max(FORMATS, key=lambda f: scoreboards["projected_600gb"][f]["total"])
|
||||
chart_c1(charts_dir, sizes, proj_bytes_600, coeff)
|
||||
chart_c2(charts_dir, medians, sizing)
|
||||
chart_c3(charts_dir, medians, proj)
|
||||
chart_c4(charts_dir, medians, proj, measured_gb)
|
||||
chart_c5(charts_dir, scoreboards)
|
||||
chart_c4(charts_dir, medians, proj, f"measured {human_size(csv_bytes)}")
|
||||
chart_c5(charts_dir, scoreboards, winner)
|
||||
chart_c6(charts_dir, proj, sizing)
|
||||
charts = sorted(p.name for p in charts_dir.glob("*.png"))
|
||||
if len(charts) != 14:
|
||||
raise ValueError(f"expected 14 charts, wrote {len(charts)}: {charts}")
|
||||
|
||||
winner = max(FORMATS, key=lambda f: scoreboards["projected_600gb"][f]["total"])
|
||||
report_text = assemble_report(report_dir, tables, scoreboards, environment, diagrams, winner)
|
||||
report_text = assemble_report(tables, scoreboards, environment, diagrams, winner, sizing, prices)
|
||||
report_path = report_dir / "REPORT.md"
|
||||
report_path.write_text(report_text, encoding="ascii", newline="\n")
|
||||
|
||||
@@ -564,6 +682,7 @@ def main() -> int:
|
||||
"tables": len(tables),
|
||||
"report_bytes": report_path.stat().st_size,
|
||||
"winner_600gb": winner,
|
||||
"prices_as_of": prices["as_of"],
|
||||
}
|
||||
for fmt in FORMATS:
|
||||
entries[f"score600_{fmt}"] = scoreboards["projected_600gb"][fmt]["total"]
|
||||
|
||||