b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
51 lines
2.6 KiB
Markdown
51 lines
2.6 KiB
Markdown
# 00_PLAN - Tribology Lab Data Storage Evaluation Pipeline
|
|
|
|
## Goal
|
|
Build a complete, reproducible pipeline that:
|
|
1. Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow.
|
|
2. Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format).
|
|
3. Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
|
|
4. Benchmarks 7 laboratory-standard retrieval scenarios (Q1-Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
|
|
5. Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
|
|
6. Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange).
|
|
|
|
## Execution environment
|
|
- Claude Code agent inside Cursor, local filesystem access, on-premise only (no cloud services).
|
|
- Python 3.11+ (tabs for indentation), PostgreSQL 16 local instance, SQLite via stdlib `sqlite3`, `rdflib` for RDF, optional local triplestore (Jena Fuseki / GraphDB Free) - if unavailable, use `rdflib` + on-disk `oxigraph` (pip) as the queryable store.
|
|
- All artifacts written under `./out/` relative to repo root:
|
|
- `./out/csv/` `./out/json/` `./out/sqlite/` `./out/pg/` (dumps) `./out/rdf/` `./out/bench/` `./out/report/`
|
|
|
|
## Task files and dependencies
|
|
```mermaid
|
|
flowchart TD
|
|
T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
|
|
T01 --> T03[03_csv_data_generation]
|
|
T03 --> T04[04_convert_json]
|
|
T03 --> T05[05_convert_sqlite]
|
|
T03 --> T06[06_convert_postgresql]
|
|
T03 --> T07[07_convert_rdf]
|
|
T04 --> T08[08_benchmark_queries]
|
|
T05 --> T08
|
|
T06 --> T08
|
|
T07 --> T08
|
|
T03 --> T08
|
|
T08 --> T09[09_benchmark_execution]
|
|
T09 --> T10[10_extrapolation_model]
|
|
T09 --> T11[11_reporting]
|
|
T10 --> T11
|
|
```
|
|
|
|
## Parallelism
|
|
- T04-T07 are independent; run in parallel after T03.
|
|
- T02 is independent of data generation; can run any time after T01.
|
|
|
|
## Execution order
|
|
01 → 02 → 03 → (04 | 05 | 06 | 07) → 08 → 09 → 10 → 11
|
|
|
|
## Global conventions
|
|
- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`, `track:B721-W1-C07-T2`, run IDs `run:R1..R4`.
|
|
- All timestamps ISO-8601 UTC. All units SI-explicit in column names (`_mN`, `_um`, `_GPa`, `_wtpct`, `_nm`).
|
|
- Random seed fixed at 20260711 for full reproducibility.
|
|
- Python indentation: tabs.
|
|
- Every task writes a completion marker `./out/.done/<task>.ok` containing row/byte counts for downstream validation.
|