fix(docs): fix formatting issues in README.md
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
50
docs/specs/00_PLAN.md
Normal file
50
docs/specs/00_PLAN.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# 00_PLAN - Tribology Lab Data Storage Evaluation Pipeline
|
||||
|
||||
## Goal
|
||||
Build a complete, reproducible pipeline that:
|
||||
1. Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow.
|
||||
2. Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format).
|
||||
3. Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
|
||||
4. Benchmarks 7 laboratory-standard retrieval scenarios (Q1-Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
|
||||
5. Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
|
||||
6. Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange).
|
||||
|
||||
## Execution environment
|
||||
- Claude Code agent inside Cursor, local filesystem access, on-premise only (no cloud services).
|
||||
- Python 3.11+ (tabs for indentation), PostgreSQL 16 local instance, SQLite via stdlib `sqlite3`, `rdflib` for RDF, optional local triplestore (Jena Fuseki / GraphDB Free) - if unavailable, use `rdflib` + on-disk `oxigraph` (pip) as the queryable store.
|
||||
- All artifacts written under `./out/` relative to repo root:
|
||||
- `./out/csv/` `./out/json/` `./out/sqlite/` `./out/pg/` (dumps) `./out/rdf/` `./out/bench/` `./out/report/`
|
||||
|
||||
## Task files and dependencies
|
||||
```mermaid
|
||||
flowchart TD
|
||||
T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
|
||||
T01 --> T03[03_csv_data_generation]
|
||||
T03 --> T04[04_convert_json]
|
||||
T03 --> T05[05_convert_sqlite]
|
||||
T03 --> T06[06_convert_postgresql]
|
||||
T03 --> T07[07_convert_rdf]
|
||||
T04 --> T08[08_benchmark_queries]
|
||||
T05 --> T08
|
||||
T06 --> T08
|
||||
T07 --> T08
|
||||
T03 --> T08
|
||||
T08 --> T09[09_benchmark_execution]
|
||||
T09 --> T10[10_extrapolation_model]
|
||||
T09 --> T11[11_reporting]
|
||||
T10 --> T11
|
||||
```
|
||||
|
||||
## Parallelism
|
||||
- T04-T07 are independent; run in parallel after T03.
|
||||
- T02 is independent of data generation; can run any time after T01.
|
||||
|
||||
## Execution order
|
||||
01 → 02 → 03 → (04 | 05 | 06 | 07) → 08 → 09 → 10 → 11
|
||||
|
||||
## Global conventions
|
||||
- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`, `track:B721-W1-C07-T2`, run IDs `run:R1..R4`.
|
||||
- All timestamps ISO-8601 UTC. All units SI-explicit in column names (`_mN`, `_um`, `_GPa`, `_wtpct`, `_nm`).
|
||||
- Random seed fixed at 20260711 for full reproducibility.
|
||||
- Python indentation: tabs.
|
||||
- Every task writes a completion marker `./out/.done/<task>.ok` containing row/byte counts for downstream validation.
|
||||
56
docs/specs/01_lab_configuration.md
Normal file
56
docs/specs/01_lab_configuration.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# 01 - Laboratory Configuration
|
||||
|
||||
## Context / Goal
|
||||
Define the authoritative laboratory model that all downstream tasks (data generation, ontology, schemas) must follow. This file is a specification, not code; produce `./out/config/lab_config.yaml` capturing everything below.
|
||||
|
||||
## Dependencies
|
||||
None (root task).
|
||||
|
||||
## Material system
|
||||
- Substrate: Ti-6Al-4V coupons, 10 × 10 × 3 mm.
|
||||
- Adhesion layer: Cr (sputtered).
|
||||
- Functional coating: Pt-Au deposited in a **composition gradient** across each wafer; film thickness range 0.3-1.1 µm.
|
||||
|
||||
## Instruments
|
||||
| Role | Instrument | Data produced |
|
||||
|---|---|---|
|
||||
| Friction | RAPID custom high-throughput parallelized 6-probe tribometer | COF vs cycle per track; test conditions |
|
||||
| Nanoindentation | Bruker TI980 TriboIndenter | Hardness, reduced modulus (25 indents/coupon) |
|
||||
| Composition | Bruker M4 Tornado micro-XRF (assumed model) | Pt/Au wt% map per coupon (20×20 grid) |
|
||||
| Deposition | Kurt J. Lesker PVD 200 sputter-down | Batch deposition parameters |
|
||||
| Surface roughness | AFM | Topography image metadata + Ra/Rq per coupon |
|
||||
| Film thickness | Optical profilometry | Thickness map per coupon (10×10 grid) |
|
||||
| Simulation | SIMTRA (sputter transport Monte-Carlo) | Deposition atom-energy / composition profiles per deposition run |
|
||||
|
||||
## Deposition matrix (4 batches)
|
||||
| Batch | Pt:Au gun tilt | Pt power | Au power | Pt discharge | Au discharge |
|
||||
|---|---|---|---|---|---|
|
||||
| B721 | 20°:0° | 150 W | 50 W | 432 V | 311 V |
|
||||
| B722 | 20°:20° | 100 W | 100 W | 399 V | 352 V |
|
||||
| B723 | 20°:20° | 150 W | 50 W | 430 V | 311 V |
|
||||
| B724 | 0°:20° | 50 W | 150 W | 376 V | 340 V |
|
||||
|
||||
## Hierarchy and volumes
|
||||
```
|
||||
batch (×4) → wafer (×3 per batch) → coupon (×49 per wafer, 7×7 grid) = 588 coupons
|
||||
```
|
||||
- Friction testing: 480 coupons (4 RAPID runs × 120 coupons: 5 plates × 6 probes × 4 coupons/square).
|
||||
- Reserve/QA: 108 coupons (no friction data; characterization only).
|
||||
- Environments: run R1,R2 = Lab Air (µ_normal); run R3,R4 = Dry N2 (µ_dry_nit).
|
||||
- Tracks: 3 replicate tracks per friction coupon, identical conditions, fresh counterface ball per track (3 balls per holder) → 1,440 tracks.
|
||||
- Cycles: 1,000 reciprocating cycles per track → 1,440,000 cycle records.
|
||||
- Friction loops: every 100th cycle stores a position-resolved loop of 200 points → 1,440 × 10 × 200 = 2,880,000 loop points.
|
||||
- Nominal test conditions: normal load 100 mN, stroke 1 mm, sliding speed 1 mm/s, ruby (Al2O3) ball Ø 3.175 mm; RH 45% (Lab Air) / 2% (Dry N2); T 23 ± 1 °C.
|
||||
|
||||
## Physical models for value simulation (used in task 03)
|
||||
- Coupon Au wt%: linear gradient across wafer x-position, batch-dependent center: B721 8%, B722 25%, B723 10%, B724 60%; ±5 wt% span across wafer; XRF grid adds N(0, 0.3) noise.
|
||||
- COF run-in: `cof(c) = cof_ss + (cof_0 - cof_ss)·exp(-c/τ) + N(0, σ)`; cof_0 ∈ [0.35, 0.5]; τ ∈ [100, 400] cycles; σ = 0.01.
|
||||
- Steady-state COF depends on Au% and environment: Lab Air `cof_ss = 0.32 - 0.0015·Au%`; Dry N2 `cof_ss = 0.24 - 0.0012·Au%`; clamp ≥ 0.12.
|
||||
- Hardness: `H = 8.5 - 0.05·Au% + N(0, 0.25)` GPa; reduced modulus `Er = 190 - 0.6·Au% + N(0, 4)` GPa.
|
||||
- Roughness Ra: log-normal, median 5 nm, σ_log 0.3.
|
||||
- Thickness: 0.3-1.1 µm, radial wafer profile + noise.
|
||||
- Wear volume per track: Archard-like `V = k·F·s`, k depends on Au% and environment; store as track-level derived measurement.
|
||||
|
||||
## Output
|
||||
- `./out/config/lab_config.yaml` - machine-readable version of everything above (volumes, matrices, model coefficients, seed).
|
||||
- Marker `./out/.done/01.ok`.
|
||||
48
docs/specs/02_process_flow_diagrams.md
Normal file
48
docs/specs/02_process_flow_diagrams.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# 02 - Process Flow Diagrams
|
||||
|
||||
## Context / Goal
|
||||
Generate the comprehensive Mermaid documentation of the laboratory workflow (reconstructed from the 10 Miro slides) as standalone `.md` / `.mermaid` artifacts for the final report and for human review.
|
||||
|
||||
## Dependencies
|
||||
01 (uses volumes/terminology from lab_config.yaml).
|
||||
|
||||
## Task
|
||||
Produce `./out/report/process_flow.md` containing the following diagrams, each also saved as a separate `.mermaid` file in `./out/report/diagrams/`.
|
||||
|
||||
### D1 - Coupon Assembly
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A([Ti-6Al-4V Base 10x10x3 mm]) --> B[Cleaning]
|
||||
B --> B1([Rinsed in deionized water])
|
||||
B1 --> B2([Sonicated in cleaning solution])
|
||||
B --> C[Smearing Adhesive]
|
||||
C --> C1([Cr adhesive coating])
|
||||
C --> D[Vapor Deposition System<br/>Kurt J. Lesker PVD 200]
|
||||
D --> D1([PtAu sputter coating, gradient])
|
||||
D --> E([Test Coupon])
|
||||
```
|
||||
|
||||
### D2 - Characterization & Batch Assembly
|
||||
Test Coupon → SIMTRA (composition / atom energies) → Optical profilometry (film thickness 0.3-1.1 µm) → assemble Test Wafer (×49 coupons) → assemble Test Batch (×3 wafers) → batches 721-724 with the deposition parameter table.
|
||||
|
||||
### D3 - Testing tree
|
||||
Friction (RAPID; Lab Air → µ_normal; Dry N2 → µ_dry_nit), Nanoindentation (TI980 → hardness, reduced modulus), AFM (topography → Ra), XRF (M4 Tornado → Pt/Au map).
|
||||
|
||||
### D4 - Tribometer session sequence (sequenceDiagram)
|
||||
Request test → Get samples → Load samples → Load ball holders (3 counterfaces/holder) → Create Excel test plan (Sample Plate, Plate Location, Sample ID, Save Location, Folder Name, X/Y/Z Offset, Load, Iterations) → Transfer to tribometer → Set up platter → Activate software → Load plan → Activate tribometer → ... → Software stop → Save avg files → Remove plates/platters → Close software → Pull equipment out.
|
||||
|
||||
### D5 - Execution loop (flowchart)
|
||||
```
|
||||
Load all counter-faces
|
||||
FOR EACH of 5 plates
|
||||
FOR EACH of 6 probes (parallel)
|
||||
FOR EACH of 4 coupons on coupon square
|
||||
3x per coupon: draw track -> rotate counter-face
|
||||
```
|
||||
|
||||
### D6 - Data hierarchy ERD-style
|
||||
batch → wafer → coupon → {xrf_map, nanoindentation, afm, profilometry} and coupon → track → {cycles, loops, wear}; deposition → simtra_profile.
|
||||
|
||||
## Output
|
||||
- `./out/report/process_flow.md`, `./out/report/diagrams/D1..D6.mermaid`
|
||||
- Marker `./out/.done/02.ok`.
|
||||
46
docs/specs/03_csv_data_generation.md
Normal file
46
docs/specs/03_csv_data_generation.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# 03 - Simulated CSV Data Generation (~150 MB)
|
||||
|
||||
## Context / Goal
|
||||
Generate the canonical raw dataset as a hierarchical CSV file tree, exactly as instruments would write it. This corpus is the single source for all conversions and benchmarks. Target size: 140-170 MB.
|
||||
|
||||
## Dependencies
|
||||
01 (`lab_config.yaml`).
|
||||
|
||||
## Input
|
||||
`./out/config/lab_config.yaml` - volumes, deposition matrix, physical model coefficients, seed 20260711.
|
||||
|
||||
## Processing instructions
|
||||
Write `generate_data.py` (tabs for indentation, pure stdlib + numpy). Stream rows; never hold the full cycle dataset in memory. Use `random.Random(seed)` / `numpy.random.default_rng(seed)` only.
|
||||
|
||||
### File tree to produce
|
||||
```
|
||||
out/csv/
|
||||
├── batches.csv # 4 rows: deposition params
|
||||
├── simtra/
|
||||
│ └── simtra_profile_<batch>.csv # 4 files, 1000 rows each: angle, energy_eV, pt_flux, au_flux
|
||||
├── batch_<B>/wafer_<W>/
|
||||
│ ├── wafer_info.csv
|
||||
│ └── coupon_<C>/
|
||||
│ ├── coupon_info.csv # position on wafer, thickness_um, ra_nm, assigned run/plate/probe/square or RESERVE
|
||||
│ ├── xrf_map.csv # 400 rows: grid_x, grid_y, pt_wtpct, au_wtpct
|
||||
│ ├── profilometry.csv # 100 rows: grid_x, grid_y, thickness_um
|
||||
│ ├── nanoindentation.csv # 25 rows: indent_id, x_um, y_um, hardness_GPa, reduced_modulus_GPa, max_load_mN
|
||||
│ ├── afm.csv # 1 row: ra_nm, rq_nm, image_file (synthetic path)
|
||||
│ └── track_<T>/ # only for 480 friction coupons, T in 1..3
|
||||
│ ├── track_info.csv # run_id, environment, load_mN, stroke_mm, speed_mm_s, counterface_id, started_at
|
||||
│ ├── cof_vs_cycle.csv # 1000 rows: cycle, cof
|
||||
│ ├── friction_loops.csv # 2000 rows: cycle (every 100th), position_um (200 pts), friction_force_mN
|
||||
│ └── wear.csv # 1 row: wear_volume_um3, k_archard, sliding_distance_m
|
||||
└── runs.csv # 4 rows: run_id, environment, date, plates, operator
|
||||
```
|
||||
|
||||
### Value generation
|
||||
Apply exactly the physical models of task 01 (gradient Au%, run-in COF, hardness/modulus vs Au%, Archard wear). COF must correlate with the coupon's mean Au% and the run environment so that Q2/Q3/Q7 benchmarks return physically meaningful results.
|
||||
|
||||
### Size control
|
||||
Expected: cycles 1.44 M rows (~30 MB) + loops 2.88 M rows (~95 MB) + XRF 235 K rows (~8 MB) + remainder (~10 MB) ≈ 150 MB. After generation, print actual tree size; abort with error if outside 120-200 MB.
|
||||
|
||||
## Output
|
||||
- Full tree under `./out/csv/` (~150 MB).
|
||||
- `./out/csv/MANIFEST.csv`: file path, rows, bytes, sha256 - used by conversion tasks for validation.
|
||||
- Marker `./out/.done/03.ok` with total rows/bytes.
|
||||
22
docs/specs/04_convert_json.md
Normal file
22
docs/specs/04_convert_json.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# 04 - Convert to JSON-LD
|
||||
|
||||
## Context / Goal
|
||||
Produce the ontology-annotated JSON-LD representation of the corpus: the format evaluated for inter-lab data exchange (FAIR). Two variants: (a) full - bulk points embedded; (b) hybrid - metadata JSON-LD with `sourceFile` links to CSV.
|
||||
|
||||
## Dependencies
|
||||
03 (`./out/csv/`, MANIFEST.csv).
|
||||
|
||||
## Input
|
||||
CSV tree.
|
||||
|
||||
## Processing instructions
|
||||
- `@context`: project vocab `https://sandia.gov/ontology/tribology#` + QUDT units + PROV-O provenance; predicates `partOf`, `cutFrom`, `performedOn`, `performedBy` as `@id` links.
|
||||
- Entity classes: `Batch, Wafer, Coupon, Instrument, CompositionMeasurement, NanoindentationMeasurement, AFMMeasurement, ProfilometryMeasurement, FrictionTest (track), FrictionCycle, FrictionLoopPoint, WearMeasurement, SimtraProfile, Run`.
|
||||
- Variant FULL: one JSON-LD file per coupon (embedding its tracks/cycles/loops), compact serialization (no pretty-print). Directory `./out/json/full/`.
|
||||
- Variant HYBRID: single `./out/json/hybrid/dataset.jsonld` with all metadata entities; cycle/loop arrays replaced by `{"sourceFile": "<relative csv path>", "rows": N, "sha256": "..."}`.
|
||||
- Stream-write; validate one sample file with `json.load` + rdflib JSON-LD parse.
|
||||
|
||||
## Output
|
||||
- `./out/json/full/` (expect ~0.7-1.2 GB), `./out/json/hybrid/dataset.jsonld` (expect < 20 MB).
|
||||
- Record byte sizes of both variants into `./out/bench/storage_sizes.csv` (append: format, variant, bytes).
|
||||
- Marker `./out/.done/04.ok`.
|
||||
30
docs/specs/05_convert_sqlite.md
Normal file
30
docs/specs/05_convert_sqlite.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# 05 - Convert to SQLite (optimized)
|
||||
|
||||
## Context / Goal
|
||||
Build the single-file relational variant optimized for size and single-user query speed.
|
||||
|
||||
## Dependencies
|
||||
03.
|
||||
|
||||
## Input
|
||||
CSV tree.
|
||||
|
||||
## Processing instructions
|
||||
- One database `./out/sqlite/tribo.db`. PRAGMAs during load: `journal_mode=OFF, synchronous=OFF, cache_size=-262144`; after load: `journal_mode=WAL`, `ANALYZE`, `VACUUM`.
|
||||
- Schema (16-table style, integer surrogate keys everywhere; no TEXT foreign keys in bulk tables):
|
||||
- `batches, wafers, coupons, runs, instruments, tracks` - dimension tables, TEXT natural keys kept as unique columns.
|
||||
- `xrf_points(coupon_id INT, gx INT, gy INT, pt_wtpct REAL, au_wtpct REAL)`.
|
||||
- `profilometry_points(coupon_id INT, gx INT, gy INT, thickness_um REAL)`.
|
||||
- `nanoindentation(coupon_id INT, indent_id INT, x_um REAL, y_um REAL, hardness_gpa REAL, reduced_modulus_gpa REAL)`.
|
||||
- `afm(coupon_id INT PRIMARY KEY, ra_nm REAL, rq_nm REAL)`.
|
||||
- `friction_cycles(track_id INT, cycle INT, cof REAL, PRIMARY KEY(track_id, cycle)) WITHOUT ROWID`.
|
||||
- `friction_loop_points(track_id INT, cycle INT, pt INT, position_um REAL, force_mn REAL, PRIMARY KEY(track_id, cycle, pt)) WITHOUT ROWID`.
|
||||
- `wear(track_id INT PRIMARY KEY, wear_volume_um3 REAL, k_archard REAL, sliding_distance_m REAL)`.
|
||||
- `track_summary(track_id INT PRIMARY KEY, cof_ss_mean REAL, cof_ss_std REAL, run_in_cycles INT)` - precomputed during load.
|
||||
- Indexes for Q1-Q7: `coupons(au_wtpct_mean)`, `tracks(run_id)`, `tracks(coupon_id)`, `runs(environment)`, composite `tracks(environment, load_mn)` (denormalize environment/load onto tracks).
|
||||
- Batch inserts with `executemany`, 50k rows per transaction.
|
||||
- Validate row counts against MANIFEST.
|
||||
|
||||
## Output
|
||||
- `./out/sqlite/tribo.db`; size appended to `storage_sizes.csv`.
|
||||
- Marker `./out/.done/05.ok`.
|
||||
23
docs/specs/06_convert_postgresql.md
Normal file
23
docs/specs/06_convert_postgresql.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# 06 - Convert to PostgreSQL (indexed + partitioned)
|
||||
|
||||
## Context / Goal
|
||||
Build the production-grade relational variant: the system-of-record candidate per the existing Storage ADR (PostgreSQL primary, RDF as deferred virtual layer).
|
||||
|
||||
## Dependencies
|
||||
03.
|
||||
|
||||
## Input
|
||||
CSV tree.
|
||||
|
||||
## Processing instructions
|
||||
- Local PostgreSQL 16, database `tribo`. Same logical schema as task 05 with PostgreSQL types (`INT`, `REAL`, `TIMESTAMPTZ`, `TEXT`).
|
||||
- `friction_cycles` and `friction_loop_points`: declarative RANGE partitioning by `track_id` (12 partitions of 120 tracks) - models the partitioning strategy needed at 600 GB scale.
|
||||
- Load via `COPY FROM STDIN` (psycopg3 `copy`), not INSERT.
|
||||
- Indexes after load: PK b-trees; `tracks(environment, load_mn)`, `coupons(au_wtpct_mean)`, `nanoindentation(coupon_id)`, BRIN on `friction_cycles(cycle)` per partition.
|
||||
- `track_summary` as a materialized view (`cof_ss_mean` = avg of cycles > run_in threshold; `run_in_cycles` via first cycle where |cof - cof_ss| < 2σ).
|
||||
- `VACUUM ANALYZE` after load. Record `pg_total_relation_size` per table and total into `storage_sizes.csv`.
|
||||
- Also export a compressed dump `./out/pg/tribo.dump` (pg_dump -Fc) for archiving-format comparison.
|
||||
|
||||
## Output
|
||||
- Live `tribo` database + `./out/pg/tribo.dump`.
|
||||
- Marker `./out/.done/06.ok`.
|
||||
24
docs/specs/07_convert_rdf.md
Normal file
24
docs/specs/07_convert_rdf.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# 07 - Convert to RDF TripleStore
|
||||
|
||||
## Context / Goal
|
||||
Build the semantic-web variant to quantify triple-store overhead and SPARQL performance on real hardware.
|
||||
|
||||
## Dependencies
|
||||
03 (optionally reuse 04 hybrid JSON-LD as an intermediate).
|
||||
|
||||
## Input
|
||||
CSV tree (or `./out/json/full/` for direct rdflib parsing - choose the faster path, document the choice).
|
||||
|
||||
## Processing instructions
|
||||
- Vocabulary identical to task 04 (`tribology#` + QUDT + PROV-O).
|
||||
- Triple modeling: every cycle → 3 triples (`rdf:type` optional at bulk level to save space - model cycles as `track :hasCycle [ :cycleNumber n ; :cof x ]` without explicit type; document the decision). Loop points likewise. Expected total: 25-45 M triples.
|
||||
- Serialize `./out/rdf/dataset.nt.gz` (N-Triples, gzip) and `./out/rdf/dataset.ttl` (Turtle) - both sizes recorded.
|
||||
- Queryable store, first available option:
|
||||
1. `oxigraph` (pip, embedded, on-disk store `./out/rdf/oxigraph_store/`) - preferred, no external service.
|
||||
2. Jena Fuseki with TDB2 if Java present.
|
||||
- Bulk-load, record store on-disk size into `storage_sizes.csv` and load wall-time.
|
||||
- Smoke-test with a SPARQL COUNT query.
|
||||
|
||||
## Output
|
||||
- Serializations + populated store.
|
||||
- Marker `./out/.done/07.ok` with triple count.
|
||||
31
docs/specs/08_benchmark_queries.md
Normal file
31
docs/specs/08_benchmark_queries.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# 08 - Benchmark Query Definitions (Q1-Q7 × 5 formats)
|
||||
|
||||
## Context / Goal
|
||||
Define the 7 laboratory-standard retrieval scenarios and their concrete implementation for each of the 5 formats. Every implementation must return identical result sets (validated by checksum of sorted result rows, tolerance 1e-9 on floats).
|
||||
|
||||
## Dependencies
|
||||
04, 05, 06, 07 (all stores populated) + 03 (CSV baseline).
|
||||
|
||||
## Scenarios
|
||||
| ID | Scenario | Result shape |
|
||||
|---|---|---|
|
||||
| Q1 | COF vs cycle curve for one track (`B722-W2-C13-T2`) | 1,000 rows (cycle, cof) |
|
||||
| Q2 | Steady-state COF for all coupons with mean Au = 10 ± 0.5 wt% | ~dozens of rows (coupon, env, cof_ss) |
|
||||
| Q3 | Hardness vs steady-state COF across all batches (join nano × track_summary) | 480 rows |
|
||||
| Q4 | Tracks in Dry N2, load 100 mN, cof_ss > 0.20 (anomaly filter) | filtered rows |
|
||||
| Q5 | Mean run-in cycles per batch (aggregation over ALL cycle rows, no summary tables allowed - force the full scan) | 4 rows |
|
||||
| Q6 | Wear volume vs load per coupon (join wear × tracks × coupons) | 480 rows |
|
||||
| Q7 | Stribeck-style aggregation: mean COF grouped by (environment, speed·load bucket) over all cycles past run-in | ~10 rows |
|
||||
|
||||
## Implementations
|
||||
- **CSV**: pure Python + csv module streaming (pandas allowed only as a second measured variant, labeled separately). Q1 = direct path read (this is the honest CSV strength). Q2-Q7 = directory walks + programmatic joins.
|
||||
- **JSON**: variant FULL, `ijson` streaming parser; Q1 = load the single coupon file.
|
||||
- **SQLite**: SQL over `tribo.db`, cold and warm cache runs (`PRAGMA cache_size` default; drop OS cache not required, just report both run 1 and run 3).
|
||||
- **PostgreSQL**: SQL, `EXPLAIN (ANALYZE, BUFFERS)` captured alongside; parallel workers default.
|
||||
- **RDF**: SPARQL against the populated store; capture query text.
|
||||
- Q5 rule applies to all formats: query raw cycle data, not `track_summary`.
|
||||
|
||||
## Output
|
||||
- `./out/bench/queries/` - one subfolder per format with 7 runnable query files (`q1.sql`, `q1.sparql`, `q1.py`, ...).
|
||||
- `./out/bench/expected/q<N>.sha256` - canonical result checksums (computed once from PostgreSQL, cross-validated against SQLite).
|
||||
- Marker `./out/.done/08.ok`.
|
||||
22
docs/specs/09_benchmark_execution.md
Normal file
22
docs/specs/09_benchmark_execution.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# 09 - Benchmark Execution & Resource Measurement
|
||||
|
||||
## Context / Goal
|
||||
Execute all 35 (Q1-Q7 × 5) benchmark cells with rigorous resource metering; produce the raw measurement table that feeds extrapolation and reporting.
|
||||
|
||||
## Dependencies
|
||||
08.
|
||||
|
||||
## Processing instructions
|
||||
Write `bench_runner.py` (tabs):
|
||||
- Each cell runs as a **separate subprocess** so RSS/CPU are attributable. For PostgreSQL, also sample the backend PID via `pg_stat_activity` + psutil.
|
||||
- Metrics per run: wall time (s), peak RSS (MB, psutil `memory_info().rss` sampled at 50 ms), CPU time user+sys (s), CPU utilization % (cpu_time / wall / cores), read bytes (psutil `io_counters`), result row count, checksum-match flag.
|
||||
- Protocol: 1 warm-up run (discarded) + 3 measured runs per cell; report median and min/max. Randomize cell order to spread cache effects; record run order.
|
||||
- Cold-cache variant: repeat the full matrix once after dropping caches if permitted (`sync; echo 3 > /proc/sys/vm/drop_caches` requires root - if not available, skip and note it).
|
||||
- Failures/timeouts: hard timeout 30 min per run; record `TIMEOUT` and continue (expected possibility: RDF Q5).
|
||||
- Storage footprint: consolidate `storage_sizes.csv` (all formats, incl. gzip-compressed archival sizes of each: `tar.gz` of CSV tree, gzip JSON, sqlite db gz, pg dump, nt.gz).
|
||||
|
||||
## Output
|
||||
- `./out/bench/results_raw.csv`: format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb, rows, checksum_ok.
|
||||
- `./out/bench/results_median.csv` - aggregated.
|
||||
- `./out/bench/storage_sizes.csv` - final consolidated.
|
||||
- Marker `./out/.done/09.ok`.
|
||||
38
docs/specs/10_extrapolation_model.md
Normal file
38
docs/specs/10_extrapolation_model.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# 10 - Extrapolation to 600 GB and Beyond
|
||||
|
||||
## Context / Goal
|
||||
Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.
|
||||
|
||||
## Dependencies
|
||||
09 (`results_median.csv`, `storage_sizes.csv`).
|
||||
|
||||
## Processing instructions
|
||||
Write `extrapolate.py`:
|
||||
|
||||
### Storage scaling
|
||||
Linear in data volume: `size_at(S) = measured_size × S / S_measured`, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).
|
||||
|
||||
### Query-time scaling laws (apply per format × query)
|
||||
| Access pattern | Law | Applies to |
|
||||
|---|---|---|
|
||||
| Direct path / keyed point read | O(1) / O(log n) | CSV Q1, SQLite/PG Q1, RDF Q1 |
|
||||
| Indexed selective filter | O(log n + k) | SQLite/PG Q2, Q4, Q6; RDF Q2, Q4 |
|
||||
| Indexed join over dimensions | O(k log n) | Q3, Q6 relational; RDF joins ×(2-5) constant penalty |
|
||||
| Full scan / aggregation | O(n) | all CSV/JSON Q2-Q7; Q5 and Q7 everywhere |
|
||||
| Single-threaded full scan | O(n), 1 core | SQLite Q5/Q7 |
|
||||
| Parallel partitioned scan | O(n / min(cores, partitions)) | PostgreSQL Q5/Q7 |
|
||||
|
||||
Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".
|
||||
|
||||
### RAM model
|
||||
- CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
|
||||
- SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
|
||||
- RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).
|
||||
|
||||
### Hardware sizing table
|
||||
For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in `./out/config/hw_prices.yaml` (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.
|
||||
|
||||
## Output
|
||||
- `./out/bench/extrapolation.csv`: format, query, scale_gb, projected_wall_s, flag.
|
||||
- `./out/bench/hardware_sizing.csv`: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
|
||||
- Marker `./out/.done/10.ok`.
|
||||
37
docs/specs/11_reporting.md
Normal file
37
docs/specs/11_reporting.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# 11 - Reporting: Tables, Charts, Weighted Scoring
|
||||
|
||||
## Context / Goal
|
||||
Aggregate all measurements and projections into the final decision document that clearly ranks the storage formats for laboratory use cases: analysis, report generation, search/filtering, archiving, inter-lab exchange.
|
||||
|
||||
## Dependencies
|
||||
09, 10 (and 02 for embedding process diagrams).
|
||||
|
||||
## Processing instructions
|
||||
Write `report.py` using matplotlib (PNG, 150 dpi, log scales where ranges span decades) into `./out/report/charts/`, and assemble `./out/report/REPORT.md`.
|
||||
|
||||
### Tables
|
||||
1. Storage footprint: measured @150 MB + coefficients vs CSV + projected @600 GB/1.2 TB/6 TB (incl. compressed archival sizes).
|
||||
2. Q1-Q7 median wall time / peak RAM / CPU% / read MB, per format (measured).
|
||||
3. Q1-Q7 projected wall time @600 GB with IMPRACTICAL/FAIL flags.
|
||||
4. Hardware sizing & cost per format @600 GB.
|
||||
5. Weighted scoring matrix (see below).
|
||||
|
||||
### Charts (each measured + each projected @600 GB)
|
||||
- C1: disk footprint bar (log).
|
||||
- C2: RAM requirement bar (log).
|
||||
- C3: per-query grouped bars Q1-Q7 wall time (log), one chart per query, 5 formats each.
|
||||
- C4: degradation curves - wall time vs data size (150 MB → 6 TB), one line per format, per query class (point read / indexed / full aggregation), log-log.
|
||||
- C5: stacked-bar weighted score.
|
||||
- C6: cost vs performance scatter @600 GB (x = est cost, y = geometric-mean query time, log-log).
|
||||
|
||||
### Weighted scoring (agreed methodology)
|
||||
Normalized 0-10 per metric, then weights: Search speed ×5 (0-50), RAM economy ×2 (0-20), Disk economy ×1 (0-10). Max 80. Search score = normalized inverse geometric mean of Q1-Q7 projected @600 GB (TIMEOUT/FAIL → 0). Present measured-scale and 600 GB-scale scoreboards side by side.
|
||||
|
||||
### Narrative sections (concise, English)
|
||||
- Executive summary with the recommendation (expected, to be confirmed by data: PostgreSQL system of record; CSV retained as raw archive; hybrid JSON-LD for exchange; RDF as virtual layer only - reference Ontop OBDA).
|
||||
- Use-case mapping table: analysis / reports / search / archiving / exchange → recommended format(s).
|
||||
- Limitations: single-node, simulated data, extrapolation assumptions.
|
||||
|
||||
## Output
|
||||
- `./out/report/REPORT.md` + `./out/report/charts/*.png` + all tables as CSV in `./out/report/tables/`.
|
||||
- Marker `./out/.done/11.ok`.
|
||||
Reference in New Issue
Block a user