# Storage Format Evaluation - Tribology Lab Data

# Dashboard

A 141.8 MiB physically plausible CSV corpus (the canonical raw format) was converted into four other storage formats and benchmarked on seven laboratory retrieval scenarios with subprocess-isolated resource metering; results were extrapolated to 600 GB, 1.2 TB and 6 TB. Every measured run validated its rows against canonical results (PostgreSQL cross-validated with SQLite, tolerance 1e-9).

Benchmark cells
35 / 35
OK; 0 invalid, 0 timeouts
Corpus
141.8 MiB
8,718 CSV files, byte-reproducible
Winner @600 GB
58.6 / 80
PostgreSQL weighted score
Worst blow-up
11.7x
RDF store vs raw CSV size
Session
916 s
whole matrix, quiet machine
Recommendation (confirmed by data): PostgreSQL as the system of record; CSV retained as the raw archive (best compressor, ~2.8x); hybrid JSON-LD for inter-lab exchange; RDF only as a virtual layer (e.g. Ontop OBDA) over PostgreSQL - a materialized triplestore fails at scale on speed, RAM and cost simultaneously.

### Typical query time per format (geometric mean Q1-Q7)

Left value: measured on the 150 MB corpus. Right value in parentheses: projected at 600 GB. Linear bars scaled within the measured values; every projected number in this document is labeled as projected.

### Where each format breaks

Format | Strength | Breaking point | @600 GB flags
--- | --- | --- | ---

# Lab Model & Corpus

A simulated tribology laboratory (Pt-Au LDRD study context, Sandia National Laboratories): Ti-6Al-4V coupons with a sputtered Cr adhesion layer and a Pt-Au composition-gradient coating, tested on a 6-probe tribometer in two environments. All data is generated deterministically from seed 20260711; regeneration is byte-identical (MANIFEST.csv sha256 proof).

Deposition
4 · 12 · 588
batches · wafers · coupons (7x7 grid)
Friction testing
1,440
tracks: 480 coupons x 3 replicates
Bulk rows
4.32 M
1.44M COF cycles + 2.88M loop points
Characterization
235,200
XRF points + nanoindentation, AFM, profilometry

### Data hierarchy

batch (B721-B724, deposition matrix) -> wafer (x3 per batch) -> coupon (x49 per wafer; 480 friction + 108 reserve) -> xrf_map (400 pts) · nanoindentation (25) · afm · profilometry (100) -> track (x3, counterface rotation) -> cof_vs_cycle (1,000 rows) · friction_loops (10x200) · wear

Runs R1-R4 map one-to-one onto batches; two environments (lab air / dry N2). The Q1 benchmark target track is B722-W2-C13-T2.

### Physical models behind the numbers

- COF: exponential run-in to a steady state depending on Au wt% and environment; noise floor 0.12.
- Hardness / modulus: linear in Au wt% with measurement noise (25 indents per coupon).
- Wear: Archard model, k0 = 1.5e-7 (lab air) / 6e-8 (dry N2) mm3/(N*m), log-normal noise.
- Au gradient across wafer position; batch-dependent center 8 / 25 / 10 / 60 wt%.

Benchmark queries therefore return physically meaningful answers (e.g. Q2 selects the Au 10 +/- 0.5 wt% composition window), not random noise.

### The five storage formats under test

Format | Variant benchmarked | How Q1-Q7 run
--- | --- | ---
CSV (canonical raw) | hierarchical file tree + MANIFEST | python csv module, streaming; Q1 = direct path read
JSON-LD | FULL (per-coupon files, 588); HYBRID kept for exchange | ijson event streaming with early exit
SQLite | single file, composite natural PKs WITHOUT ROWID, enforced FKs | SQL; precomputed track_summary for Q2-Q4/Q7
PostgreSQL 16 | remote host, range-partitioned bulk tables (12+12), BRIN + b-tree | SQL; parallel scans (4 workers), track_summary matview
RDF | pyoxigraph on-disk store, 17,419,713 triples (compact modeling) | SPARQL; run-in/steady-state derived from raw cycles

# Methodology

The product of this study is its measurements; the protocol is designed so a sloppy number cannot survive it (docs/rules/bench-methodology.md).

### Measurement protocol

- One subprocess per run - RSS / CPU / read bytes attribute to exactly that cell; psutil samples the whole process tree every 50 ms (the venv launcher stub alone would report 4 MB).
- 1 warm-up + 3 measured runs per cell; medians reported, min/max kept in raw data.
- Randomized cell order (deterministic shuffle from the corpus seed); order recorded.
- 30-minute hard timeout per run; a timeout is recorded as data, never retried.
- Correctness gates speed: every run's rows are compared to the canonical expected results (sorted rows, float tolerance 1e-9); a fast run that fails the checksum is an invalid measurement.

### Fairness and honesty

- Identical result-set contract for all five implementations of each query; no format answers a simplified question.
- Each format plays its idiomatic strength (CSV direct path read, SQL indexes, matviews) - but no LIMIT/sampling that changes the answer.
- Q5 must scan raw cycles in every format; the PostgreSQL EXPLAIN plan is checked to prove track_summary was not used.
- Cold-cache pass skipped (Windows host has no page-cache drop) - marked, not faked.
- PostgreSQL is remote: client wall times include the LAN round-trip; server-side cost captured via pg_stat_statements deltas and Prometheus host windows.

### The seven retrieval scenarios

ID | Scenario | Result rows
--- | --- | ---

### Environment (recorded per session)

# Storage Footprint

Working footprint is what the query engine reads; archival is the compressed form for shelving. All archival sizes converge to 53-57 MB - the information content is the same, the working overhead is not.

Most compact
0.83x
SQLite: 123 MB, indexes included
PostgreSQL
2.67x
397 MB live; 34.1% of it is indexes
RDF store
11.70x
1.74 GB for the same data
Best archive
52.8 MB
csv tree tar.gz (2.8x compression)

### Working footprint, measured on the 150 MB corpus

### Archival (compressed) footprint

pg_dump -Fc is internally gzip-compressed; RDF also has a Turtle serialization (596 MB, not the archival form). Hybrid JSON-LD is 1.9 MB of metadata + sourceFile links.

### Projected working footprint (linear scaling)

Format | @600 GB | @1.2 TB | @6 TB
--- | --- | --- | ---

All projected. Storage scales linearly in data volume; coefficients are the measured ratios above.

# Measured Results 150 MB corpus · warm cache

Median of 3 isolated runs per cell. Pick a query and a metric; bars are linear with the actual value on each bar. Client peak RSS below ~50 ms of runtime under-reports (sampling cadence is spec-fixed at 50 ms) - wall times are exact.

###

### Full median matrix - wall time

### PostgreSQL server side (remote host, 4 cores / 3.8 GiB)

Query | Server exec per call | Shared blocks hit | Read from disk
--- | --- | --- | ---

pg_stat_statements deltas over the session (4 calls per query). Zero disk reads: the whole 397 MB database sits in the 1 GB shared_buffers. Host peak CPU during the session: 9.2%; RAM delta ~5 MB - the server is nowhere near its limits at this scale.

# Projections every number here is projected, not measured

Per-cell scaling laws calibrated on the measured 150 MB point after subtracting each format's harness floor (interpreter start, imports, connection). Flags: IMPRACTICAL over 1 hour, FAIL over 24 hours. Pick a target scale.

###

### Scaling law per access pattern

Law | Growth | Applied to
--- | --- | ---
O(1) | flat | CSV / JSON-LD Q1 (one file, size fixed)
O(log n) | index depth | SQLite / PostgreSQL / RDF Q1
O(log n + k) | ~linear (result set grows) | relational + RDF Q2, Q4
O(k log n) | linear x depth | relational Q3, Q6; RDF Q6
O(n) | linear, 1 core | all CSV/JSON scans; SQLite Q5/Q7; RDF Q3/Q5/Q7
O(n / cores) | linear / parallel workers | PostgreSQL Q5, Q7 (partitioned scan)

### Full-scan degradation (Q5 + Q7 geometric mean)

First column measured, the rest projected. PostgreSQL rides its parallel partitioned scan; single-threaded engines grow linearly.

PostgreSQL is the only format with all seven queries below one hour at every scale - Q5 at 6 TB projects to 37 minutes on 16 cores. SQLite holds to 600 GB, then its single-threaded scans pass the hour mark. JSON-LD full scans reach 4.6 days at 6 TB; RDF Q5 reaches 27.8 days.

# Hardware & Cost street prices as of 2026-07-11

Bill-of-materials costing: whole 64 GB RDIMM modules, whole 7.68 TB NVMe drives, a priced 1U chassis; disk includes 30% free-space headroom. RAM model: relational hot set = 10% of index size + 4 GB working set; RDF hot set = 20% of the store; streaming formats are flat. Single-node RAM ceiling: 1 TB.

###

Format | Exact configuration | Chassis | RAM | Disk | Extra nodes | Total
--- | --- | --- | --- | --- | --- | ---

### Total cost comparison

### Component prices & sources retrieved 2026-07-11

- 64 GB DDR5-5600 ECC RDIMM (A-Tech, 2x32GB): $1,887
  atechmemory.com/collections/ddr5-memory-ram
- 7.68 TB NVMe U.2 1DWPD enterprise SSD (Cloud Ninjas): $3,995
  cloudninjas.com/products/new-7-68tb-nvme-u-2-1dwpd...
- 1U AMD EPYC chassis (Broadberry starting configs): AS-1015CS-TNR $4,618, AS-1115CS-TNR $6,272
  broadberry.com/amd-epyc-9004-supermicro-servers
- Market context - server DRAM 2025-2026 price surge:
  techpowerup.com/342331/server-dram-pricing-jumps-50...

Spot prices during a documented DRAM surge; re-cost by editing out/config/hw_prices.yaml and re-running tasks 10-11.

### Cost vs speed at 600 GB (projected)

# Scoring & Verdict

Subscores are 10 x best/value per metric, weighted: search speed x5 (0-50, inverse geometric mean of Q1-Q7), RAM economy x2 (0-20), disk economy x1 (0-10); maximum 80. A FAIL projection at 600 GB zeroes the search subscore.

### Measured scale (150 MB)

search x5RAM x2disk x1

At laptop scale SQLite is the honest winner: near-instant indexed queries in one file with the smallest footprint.

### Projected at 600 GB

search x5RAM x2disk x1

At production scale the ranking flips: parallel indexed retrieval dominates the weights, and PostgreSQL takes the lead.

### Use-case mapping

Use case | Recommended format
--- | ---
Interactive analysis (joins, aggregations) | PostgreSQL; SQLite acceptable single-user up to ~600 GB
Report generation (repeated summaries) | PostgreSQL (track_summary materialized view)
Search / filtering | PostgreSQL or SQLite (indexed); flat formats need full scans
Archiving | CSV tree + tar.gz (canonical raw) plus pg_dump of the system of record
Inter-lab exchange | Hybrid JSON-LD (metadata + sourceFile links to CSV)
Semantic / ontology queries | Virtual RDF layer over PostgreSQL (e.g. Ontop OBDA), not a materialized triplestore

### Limitations

- Single-node measurements on one Windows host + one remote PostgreSQL host; no cluster variance.
- Simulated corpus (physically plausible, fixed seed); real instrument data may distribute differently.
- Extrapolation is analytic, calibrated on one 150 MB point; no intermediate-scale validation runs.
- Warm-cache only (no page-cache drop on Windows); harness floor assumed constant across scales.
- RDF numbers reflect compact modeling and client-side derivation of run-in/steady-state; a SPARQL-side aggregation engine could shift, not remove, the scan penalty.
- Hardware prices are spot street prices (2026-07-11) during a DRAM price surge.

Generated from the LabDataStorageEvaluation pipeline artifacts (run of 2026-07-11, seed 20260711): out/report/REPORT.md, out/bench/results_median.csv, extrapolation.csv, hardware_sizing.csv, storage_sizes.csv, out/config/hw_prices.yaml. The full decision document with charts is out/report/REPORT.md; this page is the interactive companion. Related study: Foreign-Key Architecture Study.

# About

Attribution and provenance for this document. It is distributed as a single standalone HTML file; everything needed to verify or reproduce the numbers is referenced below.

### Author

This study was conducted by Mary Goncharenko, PhD student at the Tribology Laboratory, University of Florida, Gainesville, FL, during a research internship at the tribology laboratory of Sandia National Laboratories, with an assignment on laboratory ontology development.

The research was carried out with the technical assistance of her father, Vasiliy Goncharenko, Chief Technology Architect at SoftCreator, LLC (enterprise data architecture across relational and semantic stores; 35 years in technology).

Profiles: linkedin.com/in/marygoncharenko · linkedin.com/in/vasiliy-goncharenko

### Source repository

The complete evaluation pipeline - task specifications, binding conventions, source code, and the generated report - is publicly available at:

https://git.boskadoff.com/Public/LabDataStorageEvaluation

This page is the interactive companion to the repository's out/report/REPORT.md; all numbers embedded here come from the pipeline run of 2026-07-11. The corpus regenerates byte-identically from seed 20260711 (MANIFEST.csv sha256 proof), so every figure is independently reproducible from the repository alone.

### Data statement

The measurement corpus is simulated: physically plausible models of a Pt-Au LDRD tribology study context at Sandia National Laboratories, generated from a fixed random seed. It contains no experimental measurements and no export-controlled data. Hardware prices cited in the cost model are public street prices retrieved 2026-07-11 (sources in the Hardware & Cost section).

Related study in the same repository: Foreign-Key Architecture Study.

Standalone distributable file. If you received this document outside the repository, the canonical source is git.boskadoff.com/Public/LabDataStorageEvaluation.