LabDataStorageEvaluation

A reproducible pipeline that answers one practical question: which storage format should a materials-science laboratory use for its measurement data?

The pipeline simulates a tribology laboratory (Sandia Pt-Au LDRD context), generates ~150 MB of physically plausible raw measurement data in CSV, converts the corpus into four additional storage formats, benchmarks seven laboratory-standard retrieval scenarios against all five formats, extrapolates the measurements to 600 GB - 6 TB, and produces a ranked decision report for five laboratory use cases: analysis, reporting, search/filtering, archiving, and inter-lab exchange.

• Task specifications: docs/specs/ (plan + tasks 01-11).
• Binding conventions: docs/rules/ and CLAUDE.md.

Goals

1. Define a simulated tribology laboratory with real instruments and a realistic test workflow.
2. Generate ~150 MB of physically plausible measurement data in CSV (the canonical raw format), deterministically from seed 20260711.
3. Convert the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL 16 (indexed + partitioned), RDF triplestore.
4. Benchmark 7 retrieval scenarios (Q1-Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
5. Extrapolate measured results to 600 GB / 1.2 TB / 6 TB with complexity-aware scaling models.
6. Produce summary tables, charts, and a weighted scoreboard that rank the formats per use case.

Pipeline

flowchart TD
    T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
    T01 --> T03[03_csv_data_generation]
    T03 --> T04[04_convert_json]
    T03 --> T05[05_convert_sqlite]
    T03 --> T06[06_convert_postgresql]
    T03 --> T07[07_convert_rdf]
    T04 --> T08[08_benchmark_queries]
    T05 --> T08
    T06 --> T08
    T07 --> T08
    T03 --> T08
    T08 --> T09[09_benchmark_execution]
    T09 --> T10[10_extrapolation_model]
    T10 --> T11[11_reporting]
    T09 --> T11

Execution order: 01 -> 02 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11. Tasks 04-07 are independent and may run in parallel; task 09 requires a quiet machine (no parallel work). Every task writes a completion marker ./out/.done/<NN>.ok and validates its dependencies' markers before starting.

Task	Script	Main output	Status
01 Lab configuration	make_lab_config.py	out/config/lab_config.yaml	implemented
02 Process flow diagrams	make_process_flow.py	out/report/process_flow.md, out/report/diagrams/	implemented
03 CSV data generation	generate_data.py	out/csv/ tree + MANIFEST.csv	implemented
04 Convert to JSON-LD	convert_json.py	out/json/full/, out/json/hybrid/dataset.jsonld	implemented
05 Convert to SQLite	convert_sqlite.py	out/sqlite/tribo.db	implemented
06 Convert to PostgreSQL	convert_postgresql.py	live PostgreSQL DB (+ dump when pg_dump is available)	implemented
07 Convert to RDF	convert_rdf.py	out/rdf/dataset.nt.gz, dataset.ttl, oxigraph store	implemented
08 Benchmark queries	make_queries.py	out/bench/queries/, out/bench/expected/	implemented
09 Benchmark execution	bench_runner.py	out/bench/results_raw.csv, results_median.csv	implemented
10 Extrapolation	extrapolate.py	out/bench/extrapolation.csv, hardware_sizing.csv	planned
11 Reporting	report.py	out/report/REPORT.md, charts, tables	planned

The simulated laboratory

Ti-6Al-4V coupons (10 x 10 x 3 mm) with a sputtered Cr adhesion layer and a Pt-Au coating deposited as a composition gradient (film thickness 0.3-1.1 um). Four deposition batches (B721-B724) vary gun tilt, power, and discharge voltage. Instruments: RAPID 6-probe tribometer (friction), Bruker TI980 (nanoindentation), Bruker M4 Tornado micro-XRF (composition), Kurt J. Lesker PVD 200 (deposition), AFM (roughness), optical profilometry (thickness), SIMTRA (sputter transport simulation).

flowchart LR
    B[Batch x4] --> W[Wafer x3 per batch]
    W --> C[Coupon x49 per wafer, 7x7 grid]
    C --> X[xrf_map: 400 points]
    C --> N[nanoindentation: 25 indents]
    C --> A[afm: Ra / Rq]
    C --> P[profilometry: 100 points]
    C --> T[Track x3, friction coupons only]
    T --> CY[cof_vs_cycle: 1000 cycles]
    T --> L[friction_loops: 10 x 200 points]
    T --> WR[wear: 1 row]

Entity	Count
Batches	4
Wafers	12
Coupons	588 (480 friction-tested + 108 reserve/QA)
Tracks	1,440
Friction cycle rows	1,440,000
Friction loop points	2,880,000
XRF map points	235,200

Friction runs R1/R2 execute in Lab Air, R3/R4 in Dry N2. Generated values follow explicit physical models (COF run-in exponential, steady-state COF and hardness vs Au content, Archard wear), so benchmark queries return physically meaningful results.

Storage formats under evaluation

#	Format	Role in the comparison
1	CSV (raw tree)	Canonical instrument output; baseline
2	JSON-LD (full + hybrid)	Ontology-annotated exchange format (FAIR)
3	SQLite	Single-file relational, optimized for size and speed
4	PostgreSQL 16	Production relational: indexed, partitioned
5	RDF triplestore (oxigraph)	Semantic-web store queried via SPARQL

Benchmark scenarios (Q1-Q7)

ID	Scenario	Access pattern
Q1	COF vs cycle curve for one track	Point read
Q2	Steady-state COF for coupons with mean Au = 10 +/- 0.5 wt%	Indexed filter
Q3	Hardness vs steady-state COF across all batches	Indexed join
Q4	Tracks in Dry N2, 100 mN, cof_ss > 0.20 (anomaly filter)	Indexed filter
Q5	Mean run-in cycles per batch over ALL raw cycle rows	Forced full scan
Q6	Wear volume vs load per coupon	Indexed join
Q7	Stribeck-style mean COF by (environment, speed-load bucket)	Full aggregation

Protocol: every cell (format x query) runs in its own subprocess; 1 warm-up + 3 measured runs; randomized cell order; 30-minute hard timeout recorded as TIMEOUT; every result validated against a canonical checksum before the measurement counts. See docs/rules/bench-methodology.md.

Repository layout

docs/
  specs/        task specifications 00-11 (WHAT to build)
  rules/        binding conventions (HOW work is done)
  research/     measured design studies (e.g. FK key-schema study)
  examples/     imported reference materials (not binding)
out/            ALL generated artifacts (git-ignored, reproducible):
  config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
common/               shared helpers (storage_sizes.csv contract, ...)
  queries/            Q1-Q7 implementations for csv / json / rdf formats
make_lab_config.py    task 01 entry script (one script per task, repo root)
make_process_flow.py  task 02 entry script
generate_data.py      task 03 entry script
convert_json.py       task 04 entry script
convert_sqlite.py     task 05 entry script
convert_postgresql.py task 06 entry script (TRIBO_PG_DSN, TRIBO_PROM_URL)
convert_rdf.py        task 07 entry script
make_queries.py       task 08 entry script (canonical results need TRIBO_PG_DSN)
bench_runner.py       task 09 entry script (TRIBO_PG_DSN, TRIBO_PROM_URL)
requirements.txt      closed dependency list (docs/rules/code-python-style.md)

The out/ tree - including the ~150 MB CSV corpus, the ~1 GB JSON-LD variant, databases, and benchmark results - is excluded from git. Sources of truth are the specs, the rules, and the code; every artifact regenerates deterministically from them.

Getting started

Prerequisites: Python 3.11+ (3.12 tested). PostgreSQL 16 is required only for tasks 06/08/09; its DSN comes from the TRIBO_PG_DSN environment variable (default postgresql://postgres@localhost:5432/tribo).

Windows (PowerShell):

py -3 -m venv .venv
.venv\Scripts\python -m pip install -r requirements.txt
.venv\Scripts\python make_lab_config.py

Linux:

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python make_lab_config.py

Each task script supports --log-level {DEBUG,INFO,WARNING,ERROR} and prints its protocol output (paths, counts, sizes) to stdout. A task refuses to run if a dependency's out/.done/<NN>.ok marker is missing.

Note on cold-cache runs: the page-cache drop used by the cold-cache benchmark pass is Linux-only; on Windows hosts that pass is skipped and marked as such. Benchmark execution is planned for a dedicated Linux host.

Results

This section is populated by tasks 02, 09, 10, and 11. All artifacts land under out/ (git-ignored); the placeholders below name the exact files the pipeline produces, so the section can be filled in by copying the generated tables and linking the generated images.

Process flow diagrams (task 02)

out/report/process_flow.md with out/report/diagrams/D1..D6.mermaid: D1 coupon assembly, D2 characterization and batch assembly, D3 testing tree, D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.

Measured results (task 09) - to be generated

Placeholder	Source file
Storage footprint per format (measured + compressed)	out/bench/storage_sizes.csv
Q1-Q7 median wall time / peak RAM / CPU / read MB	out/bench/results_median.csv

Projections (task 10) - to be generated

Placeholder	Source file
Projected wall time per format at 0.6 / 1.2 / 6 TB with IMPRACTICAL/FAIL flags	out/bench/extrapolation.csv
Hardware sizing and cost per format at scale	out/bench/hardware_sizing.csv

Charts (task 11) - to be generated

(Images appear after running the full pipeline; task 11 writes them to out/report/charts/ under exactly these names.)

Final scoreboard (task 11) - to be generated

Weighted scoring, max 80 points: search speed x5 (0-50), RAM economy x2 (0-20), disk economy x1 (0-10); TIMEOUT/FAIL projects to 0.

Format	Search (0-50)	RAM (0-20)	Disk (0-10)	Total (0-80)
CSV	TBD	TBD	TBD	TBD
JSON-LD	TBD	TBD	TBD	TBD
SQLite	TBD	TBD	TBD	TBD
PostgreSQL	TBD	TBD	TBD	TBD
RDF	TBD	TBD	TBD	TBD

Use case	Recommended format
Analysis	TBD
Report generation	TBD
Search / filtering	TBD
Archiving	TBD
Inter-lab exchange	TBD

Conventions

• IDs: batch:B721, wafer:B721-W1, coupon:B721-W1-C07, track:B721-W1-C07-T2, runs R1..R4.
• Timestamps: ISO-8601 UTC; all corpus timestamps are simulated.
• Units are explicit ASCII suffixes in column names (_mN, _um, _GPa, _wtpct, _nm).
• Random seed 20260711, read from lab_config.yaml; regeneration is byte-identical (proof: MANIFEST.csv sha256).
• Python: tabs for indentation; closed dependency list; streaming discipline (the corpus never fits in memory by design).