LabDataStorageEvaluation/docs/specs/00_PLAN.md

00_PLAN - Tribology Lab Data Storage Evaluation Pipeline

Provenance: this plan and the 11 task specifications were generated from ../Initial_Prompt.md. The prompt is a historical artifact, not a contract; the specs and docs/rules/ supersede it wherever they differ.

Goal

Build a complete, reproducible pipeline that:

1. Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow.
2. Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format).
3. Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
4. Benchmarks 7 laboratory-standard retrieval scenarios (Q1-Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
5. Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
6. Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange).

Execution environment

• Claude Code agent inside Cursor, local filesystem access, on-premise only (no cloud services).
• Python 3.11+ (tabs for indentation), PostgreSQL 16 local instance, SQLite via stdlib sqlite3, rdflib for RDF, optional local triplestore (Jena Fuseki / GraphDB Free) - if unavailable, use rdflib + on-disk oxigraph (pip) as the queryable store.
• All artifacts written under ./out/ relative to repo root:
• ./out/csv/ ./out/json/ ./out/sqlite/ ./out/pg/ (dumps) ./out/rdf/ ./out/bench/ ./out/report/

Task files and dependencies

flowchart TD
    T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
    T01 --> T03[03_csv_data_generation]
    T03 --> T04[04_convert_json]
    T03 --> T05[05_convert_sqlite]
    T03 --> T06[06_convert_postgresql]
    T03 --> T07[07_convert_rdf]
    T04 --> T08[08_benchmark_queries]
    T05 --> T08
    T06 --> T08
    T07 --> T08
    T03 --> T08
    T08 --> T09[09_benchmark_execution]
    T09 --> T10[10_extrapolation_model]
    T09 --> T11[11_reporting]
    T10 --> T11

Parallelism

• T04-T07 are independent; run in parallel after T03.
• T02 is independent of data generation; can run any time after T01.

Execution order

01 → 02 → 03 → (04 | 05 | 06 | 07) → 08 → 09 → 10 → 11

Global conventions

• IDs: batch:B721, wafer:B721-W1, coupon:B721-W1-C07, track:B721-W1-C07-T2, run IDs run:R1..R4.
• All timestamps ISO-8601 UTC. All units SI-explicit in column names (_mN, _um, _GPa, _wtpct, _nm).
• Random seed fixed at 20260711 for full reproducibility.
• Python indentation: tabs.
• Every task writes a completion marker ./out/.done/<task>.ok containing row/byte counts for downstream validation.