Public/LabDataStorageEvaluation
1
0
Fork 0
Code Issues Pull Requests Actions Releases Wiki Activity

feat(docs): enhance README and build pipeline rules for clarity

Browse Source 
- update README to clarify the exclusion of bulk artifacts from git
- specify committed curated results subset in README and rules
- improve descriptions of output directories and their contents
- refine details on the pipeline's data generation and reporting processes
This commit is contained in:
administrator
2026-07-12 11:18:16 -04:00
parent 35d2bdadbd
commit b4270b0247
3 changed files with 893 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
README.md
View File

@@ -3,7 +3,8 @@
A reproducible pipeline that answers one practical question: **which storage
format should a materials-science laboratory use for its measurement data?**
The pipeline simulates a tribology laboratory (Sandia Pt-Au LDRD context),
The pipeline simulates a tribology laboratory (Pt-Au LDRD context, Sandia
National Laboratories),
generates ~150 MB of physically plausible raw measurement data in CSV,
converts the corpus into four additional storage formats, benchmarks seven
laboratory-standard retrieval scenarios against all five formats, extrapolates
@@ -145,7 +146,8 @@ docs/
rules/ binding conventions (HOW work is done)
research/ measured design studies (e.g. FK key-schema study)
examples/ imported reference materials (not binding)
out/ ALL generated artifacts (git-ignored, reproducible):
out/ ALL generated artifacts (reproducible; bulk git-ignored,
report/ config/ .done/ committed - see .gitignore):
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
common/ shared helpers (storage_sizes.csv contract, ...)
queries/ Q1-Q7 implementations for csv / json / rdf formats
@@ -163,10 +165,13 @@ report.py task 11 entry script (REPORT.md, charts, tables, scoring)
requirements.txt closed dependency list (docs/rules/code-python-style.md)
```
The `out/` tree - including the ~150 MB CSV corpus, the ~1 GB JSON-LD variant,
databases, and benchmark results - is **excluded from git**. Sources of truth
are the specs, the rules, and the code; every artifact regenerates
deterministically from them. [docs/Initial_Prompt.md](docs/Initial_Prompt.md)
The bulk of the `out/` tree - the ~150 MB CSV corpus, the JSON-LD variants,
databases, stores, archives, and raw benchmark data - is **excluded from
git**; every artifact regenerates deterministically from the specs, the
rules, and the code. A curated results subset IS committed (decision of
2026-07-11): `out/report/` (REPORT.md, charts, tables, diagrams),
`out/config/` (lab_config.yaml, hw_prices.yaml), and the `out/.done/`
markers - see `.gitignore`. [docs/Initial_Prompt.md](docs/Initial_Prompt.md)
is the original prompt the 12 specification files were generated from - kept
for provenance only, never authoritative: where it disagrees with the specs,
rules, or code (e.g. the SQLite key schema, the JSON-LD vocabulary), those win.
@@ -203,10 +208,14 @@ Benchmark execution is planned for a dedicated Linux host.
## Results
This section is populated by tasks 02, 09, 10, and 11. All artifacts land
under `out/` (git-ignored); the placeholders below name the exact files the
pipeline produces, so the section can be filled in by copying the generated
tables and linking the generated images.
Results of the full pipeline run of 2026-07-11 (seed 20260711, 141.8 MiB
corpus, all 35 benchmark cells checksum-valid). The generated sources of
truth are committed under [out/report/](out/report/) - the full decision
document is [out/report/REPORT.md](out/report/REPORT.md); the tables below
are copied from it. An interactive single-file companion (sidebar
navigation, per-query and per-scale views, cost explorer) is
[docs/research/Tribology_Storage_Evaluation.html](docs/research/Tribology_Storage_Evaluation.html).
Bulk measurement files (`out/bench/`) are git-ignored and regenerable.
### Process flow diagrams (task 02)
@@ -214,21 +223,58 @@ tables and linking the generated images.
D1 coupon assembly, D2 characterization and batch assembly, D3 testing tree,
D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.
### Measured results (task 09) - to be generated
### Measured results (task 09)
| Placeholder | Source file |
|---|---|
| Storage footprint per format (measured + compressed) | `out/bench/storage_sizes.csv` |
| Q1-Q7 median wall time / peak RAM / CPU / read MB | `out/bench/results_median.csv` |
Storage footprint per format (working + compressed archival):
### Projections (task 10) - to be generated
| Format | Measured | vs CSV | Archival (compressed) |
|---|---|---|---|
| CSV | 149 MB | 1.00x | 52.8 MB (tar.gz) |
| JSON-LD (full) | 311 MB | 2.09x | 57.2 MB (tar.gz) |
| SQLite | 123 MB | 0.83x | 55.1 MB (gzip) |
| PostgreSQL | 397 MB | 2.67x | 53.1 MB (pg_dump -Fc) |
| RDF triplestore | 1.74 GB | 11.70x | 126 MB (nt.gz) |
| Placeholder | Source file |
|---|---|
| Projected wall time per format at 0.6 / 1.2 / 6 TB with IMPRACTICAL/FAIL flags | `out/bench/extrapolation.csv` |
| Hardware sizing and cost per format at scale | `out/bench/hardware_sizing.csv` |
Median wall time on the 150 MB corpus, warm cache (peak RAM / CPU / read
bytes: [out/report/tables/t2_measured_medians.csv](out/report/tables/t2_measured_medians.csv)):
### Charts (task 11) - to be generated
| Format | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
|---|---|---|---|---|---|---|---|
| CSV | 0.06 s | 0.17 s | 1.15 s | 0.72 s | 1.09 s | 0.28 s | 1.27 s |
| JSON-LD | 0.11 s | 1.48 s | 9.9 s | 9.9 s | 10 s | 9.8 s | 10 s |
| SQLite | 0.06 s | 0.06 s | 0.11 s | 0.06 s | 0.77 s | 0.06 s | 0.71 s |
| PostgreSQL | 0.22 s | 0.22 s | 0.22 s | 0.22 s | 0.44 s | 0.22 s | 0.33 s |
| RDF triplestore | 0.11 s | 2.81 s | 46 s | 20 s | 60 s | 0.17 s | 41 s |
### Projections (task 10)
Projected wall times at 600 GB - `(!)` = IMPRACTICAL (over 1 hour),
`FAIL` = over 24 hours; 1.2 TB and 6 TB scales:
[out/report/tables/t3_projected_600gb.csv](out/report/tables/t3_projected_600gb.csv)
and `out/bench/extrapolation.csv`:
| Format | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 |
|---|---|---|---|---|---|---|---|
| CSV | 0.06 s | 7.1 min | 1.2 h (!) | 44 min | 1.2 h (!) | 15 min | 1.4 h (!) |
| JSON-LD | 0.11 s | 1.5 h (!) | 11 h (!) | 11 h (!) | 11 h (!) | 10.9 h (!) | 11.2 h (!) |
| SQLite | 0.06 s | 1.3 s | 5.7 min | 0.06 s | 47 min | 5.2 s | 44 min |
| PostgreSQL | 0.22 s | 0.22 s | 3.4 s | 11 s | 3.7 min | 4.1 s | 1.9 min |
| RDF triplestore | 0.11 s | 3.0 h (!) | 2.1 d FAIL | 22 h (!) | 2.8 d FAIL | 5.9 min | 1.9 d FAIL |
Hardware sizing at 600 GB - bill of materials with street prices as of
2026-07-11 (component sources: REPORT.md section 7 and
[out/config/hw_prices.yaml](out/config/hw_prices.yaml)):
| Format | Configuration | Total |
|---|---|---|
| CSV, JSON-LD, SQLite, PostgreSQL (each) | 1 x 1U Supermicro AS-1015CS-TNR (16 cores); 1 x 64 GB DDR5 ECC RDIMM ($1,887); 1 x 7.68 TB NVMe U.2 ($3,995) | $10,500 |
| RDF triplestore | 2 nodes; 22 x 64 GB DDR5 ECC RDIMM; 2 x 7.68 TB NVMe U.2 | $58,740 |
At 6 TB PostgreSQL still fits one 16-core node - 9 RDIMM modules, 3 NVMe
drives, $33,586; the RDF triplestore needs 14 nodes and $527,732
(projected; full table: `out/bench/hardware_sizing.csv`).
### Charts (task 11)
![C1 - disk footprint per format, log scale](out/report/charts/c1_disk_footprint.png)
![C2 - RAM requirement per format, log scale](out/report/charts/c2_ram_requirement.png)
@@ -237,8 +283,9 @@ D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.
![C5 - weighted score, stacked bars](out/report/charts/c5_weighted_score.png)
![C6 - cost vs performance at 600 GB, log-log](out/report/charts/c6_cost_vs_performance.png)
(Images appear after running the full pipeline; task 11 writes them to
`out/report/charts/` under exactly these names.)
(Task 11 writes the charts to `out/report/charts/` under exactly these
names; c3 covers all seven queries - `c3_q1..q7_wall_time.png` - and c4 has
three query classes: `point_read`, `indexed`, `full_scan`.)
### Final scoreboard (task 11)