Public/LabDataStorageEvaluation
1
0
Fork 0
Code Issues Pull Requests Actions Releases Wiki Activity

Merge pull request 'fix(docs): fix formatting issues in README.md' (#1) from dev_masha into master

Browse Source 
Reviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
Mary Goncharenko
2026-07-11 13:39:55 -04:00
parent d8353953b0 b173ac82a9
commit 9144d74d4c
55 changed files with 3755 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
.cursorrules Normal file
View File

@@ -0,0 +1,48 @@
# Cursor rules for LabDataStorageEvaluation
## Commit messages (mandatory format)
Every commit message you generate MUST follow this convention exactly.
First line: type(scope): description
- lowercase imperative English summary, under 72 characters, no trailing period
- type is one of (closed list): feat fix perf refactor security docs ci chore test
- scope is one of (closed list): specs rules docs config datagen convert bench report tools all
Scope selection by changed files:
- specs = docs/specs/**
- rules = docs/rules/**
- docs = other prose docs (README.md, CLAUDE.md, docs/examples/**)
- config = configuration files (.gitignore, .cursorrules, editable YAML defaults)
- datagen = lab-configuration and data-generation code (tasks 01-03)
- convert = format converters: JSON-LD, SQLite, PostgreSQL, RDF (tasks 04-07)
- bench = benchmark queries, runner, extrapolation (tasks 08-10)
- report = reporting code and templates (task 11)
- tools = helper scripts under tools/
- all = multiple areas, or when no narrower scope honestly fits
Body: the subject is only a title. If the diff touches more than one area,
add a blank line and 2-6 concise bullets covering every meaningful changed
area. Split unrelated work into separate commits.
Special forms allowed as-is: "Merge ..." commits; "release(scope): vX.Y.Z".
Never emit vague or unprefixed subjects such as: "Update files",
"Improve project", "clarify rules", "changes".
Examples of good subjects:
- feat(datagen): stream friction cycle rows during CSV generation
- fix(convert): match sqlite row counts against MANIFEST before commit
- docs(rules): add benchmark methodology rule
- chore(config): ignore out/ artifact tree
## Typography
Use plain ASCII typography in everything you write into this repository:
hyphen-minus instead of en/em dashes, straight quotes, three dots instead
of the ellipsis character. See docs/rules/code-data-formatting.md.
## Python
Python code in this repository uses TABS for indentation. See
docs/rules/code-python-style.md for the closed dependency list.

15
.gitignore vendored Normal file
View File

@@ -0,0 +1,15 @@
# Generated pipeline artifacts - reproducible from specs + rules + code
# (see docs/rules/build-pipeline-tasks.md)
out/
# Python
__pycache__/
*.pyc
.venv/
venv/
# Editor / OS noise
.vscode/
.idea/
Thumbs.db
.DS_Store

112
CLAUDE.md Normal file
View File

@@ -0,0 +1,112 @@
# LabDataStorageEvaluation - Project Rules
A reproducible evaluation pipeline for laboratory data storage formats,
built around a simulated tribology laboratory (Sandia Pt-Au LDRD context).
The pipeline (tasks 01-11, specs in [docs/specs/](docs/specs/)):
1. Defines the lab model (`lab_config.yaml`) and process-flow diagrams.
2. Generates **~150 MB of physically plausible measurement data in CSV**
(the canonical raw format), deterministically from **seed 20260711**.
3. Converts the corpus into 4 target formats: **JSON-LD** (full + hybrid),
**SQLite**, **PostgreSQL 16** (indexed + partitioned), **RDF
triplestore** (oxigraph).
4. Benchmarks **7 retrieval scenarios (Q1-Q7) x 5 formats**, measuring
wall time, peak RAM, CPU, disk I/O, and footprint.
5. Extrapolates to 600 GB / 1.2 TB / 6 TB and produces a ranked report
(`REPORT.md` + charts).
Everything runs **locally** (Windows development host, local PostgreSQL 16,
no cloud services, no server deploy). The stack is **Python 3.11+ (tabs for
indentation)** with a closed dependency list; there is **no UI, no HTTP
API, no authentication** in this project. All artifacts land under
`./out/` (git-ignored); tasks signal completion via `./out/.done/<NN>.ok`
markers.
These rules are **binding** for every contributor (human or AI agent)
touching this repository. They override any pattern found in imported
specs, external tutorials, or older project docs that predate this file.
---
## Mandatory rules - load before starting matching work
When the user's request matches a topic below, **before writing any code or
producing a plan**, run `Read` on the listed rule file(s). The rules are
binding; ignoring them produces broken work that the user will reject.
| If the task involves... | You MUST first Read |
|---|---|
| Writing or changing any Python (style, tabs, dependency list, streaming discipline) | [docs/rules/code-python-style.md](docs/rules/code-python-style.md) |
| Data generation, physical models, random values, seeds, reproducibility | [docs/rules/data-determinism.md](docs/rules/data-determinism.md) |
| Naming entities/IDs, timestamps, measurement columns, units, `au_wtpct_mean` | [docs/rules/data-naming-units.md](docs/rules/data-naming-units.md) |
| SQLite / PostgreSQL schema, DDL, keys, indexes, bulk loading, `track_summary` | [docs/rules/db-sql-schema.md](docs/rules/db-sql-schema.md) |
| Benchmark queries, the runner, resource measurement, timeouts, extrapolation, scoring | [docs/rules/bench-methodology.md](docs/rules/bench-methodology.md) |
| The `./out/` tree, `.done` markers, task order/parallelism, `storage_sizes.csv`, re-runs | [docs/rules/build-pipeline-tasks.md](docs/rules/build-pipeline-tasks.md) |
| Validation: MANIFEST, row counts, result checksums, expected volumes, pytest | [docs/rules/test-pipeline-validation.md](docs/rules/test-pipeline-validation.md) |
| Python error handling (`try` / `except` / `raise`, abort semantics) | [docs/rules/code-error-handling.md](docs/rules/code-error-handling.md) |
| Logging, `print()` vs logger, log levels | [docs/rules/obs-python-logging.md](docs/rules/obs-python-logging.md) |
| Configuration values: the YAML configs, constants, the PostgreSQL DSN, CLI flags | [docs/rules/code-config-yaml.md](docs/rules/code-config-yaml.md) |
| Writing, editing, or reviewing comments in source code (any language) | [docs/rules/code-comments-policy.md](docs/rules/code-comments-policy.md) |
| Prose strings in source/config/docs (typography / encoding / dashes) | [docs/rules/code-data-formatting.md](docs/rules/code-data-formatting.md) |
| Adding or renaming a rule file in `docs/rules/` | [docs/rules/meta-rules-naming.md](docs/rules/meta-rules-naming.md) |
| Anything where misinterpretation could be destructive (deleting/regenerating `./out/`, dropping the `tribo` DB, changing the seed or `lab_config.yaml`, editing specs, git history) | [docs/rules/meta-no-assumptions.md](docs/rules/meta-no-assumptions.md) |
| Every response (length / formatting discipline) | [docs/rules/meta-concise-answers.md](docs/rules/meta-concise-answers.md) |
| Asking the user any question | [docs/rules/meta-concise-questions.md](docs/rules/meta-concise-questions.md) |
| Every response (communication language / register - Russian, no jargon or slang) | [docs/rules/meta-language-style.md](docs/rules/meta-language-style.md) |
| Any git operation that would create a commit, tag, merge, revert, rebase, cherry-pick, or push | [docs/rules/meta-git-commits.md](docs/rules/meta-git-commits.md) |
| Generating commit messages via Cursor, or maintaining the `.cursorrules` artifact | [docs/rules/meta-cursor-rules.md](docs/rules/meta-cursor-rules.md) |
Multiple matches -> Read all matching rule files. Unsure -> Read the closest one;
over-loading is cheap, missing a rule is expensive.
---
## Other binding constraints
These are short, cross-cutting constraints. They are still **binding**; the
linked rule file holds the detailed spec.
- **Determinism is the foundation.** Seed `20260711` lives in
`lab_config.yaml` and is never hardcoded; only seeded RNGs; no
wall-clock values inside the corpus; regeneration is byte-identical
(proof: `MANIFEST.csv` sha256). See
[docs/rules/data-determinism.md](docs/rules/data-determinism.md).
- **A wrong artifact is worse than no artifact.** Validation gates (row
counts vs MANIFEST, result checksums with 1e-9 tolerance, the
120-200 MB size gate) abort loudly; a `.done` marker is written only
after full success. See
[docs/rules/code-error-handling.md](docs/rules/code-error-handling.md)
and [docs/rules/test-pipeline-validation.md](docs/rules/test-pipeline-validation.md).
- **Tasks communicate only through `./out/` artifacts.** The tree is
git-ignored; sources of truth are specs + rules + code. Benchmarks run
on a quiet machine. See
[docs/rules/build-pipeline-tasks.md](docs/rules/build-pipeline-tasks.md).
- **Measurement honesty.** Subprocess isolation per cell, 1 warm-up + 3
measured runs, randomized order, 30-min timeout recorded as `TIMEOUT`,
failed checksums invalidate the measurement, projected numbers are
always labeled. See
[docs/rules/bench-methodology.md](docs/rules/bench-methodology.md).
- **Python: tabs, closed dependency list.** Stdlib first; only the
packages listed in
[docs/rules/code-python-style.md](docs/rules/code-python-style.md);
task 03 is stdlib + numpy only; stream everything bulk.
- **Commit message format (mandatory).** First line
`type(scope): description`, lowercase imperative English, under 72
chars. Types (closed): `feat fix perf refactor security docs ci chore
test`. Scopes (closed): `specs rules docs config datagen convert bench
report tools all`. The AI never creates commits - git is read-only for
agents; the user runs all git write commands. See
[docs/rules/meta-git-commits.md](docs/rules/meta-git-commits.md).
---
## Canonical source of truth
If this file and any `docs/rules/<name>.md` disagree, **the rule file wins**
within its topic - that is where the detailed binding spec lives. If a rule
and a task spec in `docs/specs/` disagree on a convention, **the rule wins**
and the spec is updated to match; the specs remain authoritative for task
scope, inputs, outputs, and dependencies. If this file and any other
document (README, older docs, code comments) disagree, **this file plus the
rule files win**. Update the other document to match; do not work around
the rules.

260
README.md
View File

@@ -1,11 +1,257 @@
# LabDataStorageEvaluation # LabDataStorageEvaluation
Build a complete, reproducible pipeline that: A reproducible pipeline that answers one practical question: **which storage
format should a materials-science laboratory use for its measurement data?**
The pipeline simulates a tribology laboratory (Sandia Pt-Au LDRD context),
generates ~150 MB of physically plausible raw measurement data in CSV,
converts the corpus into four additional storage formats, benchmarks seven
laboratory-standard retrieval scenarios against all five formats, extrapolates
the measurements to 600 GB - 6 TB, and produces a ranked decision report for
five laboratory use cases: analysis, reporting, search/filtering, archiving,
and inter-lab exchange.
Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow. - Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11).
Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format). - Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md).
Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
Benchmarks 7 laboratory-standard retrieval scenarios (Q1–Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint. ## Goals
Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange). 1. Define a simulated tribology laboratory with real instruments and a
realistic test workflow.
2. Generate ~150 MB of physically plausible measurement data in CSV (the
canonical raw format), deterministically from seed `20260711`.
3. Convert the CSV corpus into 4 target storage formats: JSON-LD, SQLite
(optimized), PostgreSQL 16 (indexed + partitioned), RDF triplestore.
4. Benchmark 7 retrieval scenarios (Q1-Q7) against all 5 formats, measuring
wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
5. Extrapolate measured results to 600 GB / 1.2 TB / 6 TB with
complexity-aware scaling models.
6. Produce summary tables, charts, and a weighted scoreboard that rank the
formats per use case.
## Pipeline
```mermaid
flowchart TD
T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
T01 --> T03[03_csv_data_generation]
T03 --> T04[04_convert_json]
T03 --> T05[05_convert_sqlite]
T03 --> T06[06_convert_postgresql]
T03 --> T07[07_convert_rdf]
T04 --> T08[08_benchmark_queries]
T05 --> T08
T06 --> T08
T07 --> T08
T03 --> T08
T08 --> T09[09_benchmark_execution]
T09 --> T10[10_extrapolation_model]
T10 --> T11[11_reporting]
T09 --> T11
```
Execution order: `01 -> 02 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11`.
Tasks 04-07 are independent and may run in parallel; task 09 requires a quiet
machine (no parallel work). Every task writes a completion marker
`./out/.done/<NN>.ok` and validates its dependencies' markers before starting.
| Task | Script | Main output | Status |
|---|---|---|---|
| 01 Lab configuration | `make_lab_config.py` | `out/config/lab_config.yaml` | implemented |
| 02 Process flow diagrams | `make_process_flow.py` | `out/report/process_flow.md`, `out/report/diagrams/` | planned |
| 03 CSV data generation | `generate_data.py` | `out/csv/` tree + `MANIFEST.csv` | planned |
| 04 Convert to JSON-LD | `convert_json.py` | `out/json/full/`, `out/json/hybrid/dataset.jsonld` | planned |
| 05 Convert to SQLite | `convert_sqlite.py` | `out/sqlite/tribo.db` | planned |
| 06 Convert to PostgreSQL | `convert_postgresql.py` | live `tribo` DB + `out/pg/tribo.dump` | planned |
| 07 Convert to RDF | `convert_rdf.py` | `out/rdf/dataset.nt.gz`, oxigraph store | planned |
| 08 Benchmark queries | `make_queries.py` | `out/bench/queries/`, `out/bench/expected/` | planned |
| 09 Benchmark execution | `bench_runner.py` | `out/bench/results_raw.csv`, `results_median.csv` | planned |
| 10 Extrapolation | `extrapolate.py` | `out/bench/extrapolation.csv`, `hardware_sizing.csv` | planned |
| 11 Reporting | `report.py` | `out/report/REPORT.md`, charts, tables | planned |
## The simulated laboratory
Ti-6Al-4V coupons (10 x 10 x 3 mm) with a sputtered Cr adhesion layer and a
Pt-Au coating deposited as a composition gradient (film thickness
0.3-1.1 um). Four deposition batches (B721-B724) vary gun tilt, power, and
discharge voltage. Instruments: RAPID 6-probe tribometer (friction), Bruker
TI980 (nanoindentation), Bruker M4 Tornado micro-XRF (composition), Kurt J.
Lesker PVD 200 (deposition), AFM (roughness), optical profilometry
(thickness), SIMTRA (sputter transport simulation).
```mermaid
flowchart LR
B[Batch x4] --> W[Wafer x3 per batch]
W --> C[Coupon x49 per wafer, 7x7 grid]
C --> X[xrf_map: 400 points]
C --> N[nanoindentation: 25 indents]
C --> A[afm: Ra / Rq]
C --> P[profilometry: 100 points]
C --> T[Track x3, friction coupons only]
T --> CY[cof_vs_cycle: 1000 cycles]
T --> L[friction_loops: 10 x 200 points]
T --> WR[wear: 1 row]
```
| Entity | Count |
|---|---|
| Batches | 4 |
| Wafers | 12 |
| Coupons | 588 (480 friction-tested + 108 reserve/QA) |
| Tracks | 1,440 |
| Friction cycle rows | 1,440,000 |
| Friction loop points | 2,880,000 |
| XRF map points | 235,200 |
Friction runs R1/R2 execute in Lab Air, R3/R4 in Dry N2. Generated values
follow explicit physical models (COF run-in exponential, steady-state COF and
hardness vs Au content, Archard wear), so benchmark queries return physically
meaningful results.
## Storage formats under evaluation
| # | Format | Role in the comparison |
|---|---|---|
| 1 | CSV (raw tree) | Canonical instrument output; baseline |
| 2 | JSON-LD (full + hybrid) | Ontology-annotated exchange format (FAIR) |
| 3 | SQLite | Single-file relational, optimized for size and speed |
| 4 | PostgreSQL 16 | Production relational: indexed, partitioned |
| 5 | RDF triplestore (oxigraph) | Semantic-web store queried via SPARQL |
## Benchmark scenarios (Q1-Q7)
| ID | Scenario | Access pattern |
|---|---|---|
| Q1 | COF vs cycle curve for one track | Point read |
| Q2 | Steady-state COF for coupons with mean Au = 10 +/- 0.5 wt% | Indexed filter |
| Q3 | Hardness vs steady-state COF across all batches | Indexed join |
| Q4 | Tracks in Dry N2, 100 mN, cof_ss > 0.20 (anomaly filter) | Indexed filter |
| Q5 | Mean run-in cycles per batch over ALL raw cycle rows | Forced full scan |
| Q6 | Wear volume vs load per coupon | Indexed join |
| Q7 | Stribeck-style mean COF by (environment, speed-load bucket) | Full aggregation |
Protocol: every cell (format x query) runs in its own subprocess; 1 warm-up +
3 measured runs; randomized cell order; 30-minute hard timeout recorded as
`TIMEOUT`; every result validated against a canonical checksum before the
measurement counts. See
[docs/rules/bench-methodology.md](docs/rules/bench-methodology.md).
## Repository layout
```
docs/
specs/ task specifications 00-11 (WHAT to build)
rules/ binding conventions (HOW work is done)
examples/ imported reference materials (not binding)
out/ ALL generated artifacts (git-ignored, reproducible):
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
make_lab_config.py task 01 entry script (one script per task, repo root)
requirements.txt closed dependency list (docs/rules/code-python-style.md)
```
The `out/` tree - including the ~150 MB CSV corpus, the ~1 GB JSON-LD variant,
databases, and benchmark results - is **excluded from git**. Sources of truth
are the specs, the rules, and the code; every artifact regenerates
deterministically from them.
## Getting started
Prerequisites: Python 3.11+ (3.12 tested). PostgreSQL 16 is required only for
tasks 06/08/09; its DSN comes from the `TRIBO_PG_DSN` environment variable
(default `postgresql://postgres@localhost:5432/tribo`).
Windows (PowerShell):
```powershell
py -3 -m venv .venv
.venv\Scripts\python -m pip install -r requirements.txt
.venv\Scripts\python make_lab_config.py
```
Linux:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python make_lab_config.py
```
Each task script supports `--log-level {DEBUG,INFO,WARNING,ERROR}` and prints
its protocol output (paths, counts, sizes) to stdout. A task refuses to run if
a dependency's `out/.done/<NN>.ok` marker is missing.
Note on cold-cache runs: the page-cache drop used by the cold-cache benchmark
pass is Linux-only; on Windows hosts that pass is skipped and marked as such.
Benchmark execution is planned for a dedicated Linux host.
## Results
This section is populated by tasks 02, 09, 10, and 11. All artifacts land
under `out/` (git-ignored); the placeholders below name the exact files the
pipeline produces, so the section can be filled in by copying the generated
tables and linking the generated images.
### Process flow diagrams (task 02)
`out/report/process_flow.md` with `out/report/diagrams/D1..D6.mermaid`:
D1 coupon assembly, D2 characterization and batch assembly, D3 testing tree,
D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.
### Measured results (task 09) - to be generated
| Placeholder | Source file |
|---|---|
| Storage footprint per format (measured + compressed) | `out/bench/storage_sizes.csv` |
| Q1-Q7 median wall time / peak RAM / CPU / read MB | `out/bench/results_median.csv` |
### Projections (task 10) - to be generated
| Placeholder | Source file |
|---|---|
| Projected wall time per format at 0.6 / 1.2 / 6 TB with IMPRACTICAL/FAIL flags | `out/bench/extrapolation.csv` |
| Hardware sizing and cost per format at scale | `out/bench/hardware_sizing.csv` |
### Charts (task 11) - to be generated
![C1 - disk footprint per format, log scale](out/report/charts/c1_disk_footprint.png)
![C2 - RAM requirement per format, log scale](out/report/charts/c2_ram_requirement.png)
![C3 - per-query wall time, one chart per query](out/report/charts/c3_q1_wall_time.png)
![C4 - degradation curves 150 MB to 6 TB, log-log](out/report/charts/c4_degradation_full_scan.png)
![C5 - weighted score, stacked bars](out/report/charts/c5_weighted_score.png)
![C6 - cost vs performance at 600 GB, log-log](out/report/charts/c6_cost_vs_performance.png)
(Images appear after running the full pipeline; task 11 writes them to
`out/report/charts/` under exactly these names.)
### Final scoreboard (task 11) - to be generated
Weighted scoring, max 80 points: search speed x5 (0-50), RAM economy x2
(0-20), disk economy x1 (0-10); TIMEOUT/FAIL projects to 0.
| Format | Search (0-50) | RAM (0-20) | Disk (0-10) | Total (0-80) |
|---|---|---|---|---|
| CSV | TBD | TBD | TBD | TBD |
| JSON-LD | TBD | TBD | TBD | TBD |
| SQLite | TBD | TBD | TBD | TBD |
| PostgreSQL | TBD | TBD | TBD | TBD |
| RDF | TBD | TBD | TBD | TBD |
| Use case | Recommended format |
|---|---|
| Analysis | TBD |
| Report generation | TBD |
| Search / filtering | TBD |
| Archiving | TBD |
| Inter-lab exchange | TBD |
## Conventions
- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`,
`track:B721-W1-C07-T2`, runs `R1..R4`.
- Timestamps: ISO-8601 UTC; all corpus timestamps are simulated.
- Units are explicit ASCII suffixes in column names (`_mN`, `_um`, `_GPa`,
`_wtpct`, `_nm`).
- Random seed `20260711`, read from `lab_config.yaml`; regeneration is
byte-identical (proof: `MANIFEST.csv` sha256).
- Python: tabs for indentation; closed dependency list; streaming discipline
(the corpus never fits in memory by design).

248
docs/examples/TribologyData_MasterPlan_v1.md Normal file
View File

@@ -0,0 +1,248 @@
# Tribology Lab Data — Master Plan
**Discovery · Cleansing/Normalization · Ontology · Conversion**
Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
Status: Plan v1 — Draft
---
## 0. Objective
Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
---
## 1. Guiding principles
1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
---
## 2. Phase overview
| Phase | Name | Nature | Primary output |
|-------|------|--------|----------------|
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
---
## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
### 3.1 Inputs
- Root of the ~700 GB CSV tree (read-only).
### 3.2 Activities
**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
- atmosphere tokens (`ambient`, `dryNitrogen`, …)
Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
**C. Structural profiling (header-only + sampled body).** Per file, without full load:
- read header row → column names, count, order;
- infer per-column datatype from first/last N rows + a random sample;
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
```
for each file in walk(root):
meta = fs_metadata(file)
tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere
header = read_first_line(file)
columns = parse_columns(header)
dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M))
fp = fingerprint(columns) # ordered-set hash
emit_manifest_row(meta, tokens, columns, dtypes, fp)
```
### 3.4 Artifacts
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
### 3.5 Exit criteria
- 100% of files inventoried and fingerprinted.
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
---
## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
### 4.1 The iteration loop (per measurement type)
1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
### 4.2 Instruments Repository (ontology sub-model)
Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
- identity (type, model/label, measurement type it produces);
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
- association to `measurement_types` / `column_definitions` in the data dictionary.
Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
### 4.3 Anomaly detection
Criteria are **derived from the data**, not fixed a priori. Starting catalog:
| Class | Description | Source of criterion |
|-------|-------------|---------------------|
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
| Missing required | Null in a non-nullable canonical field | Schema constraints |
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
### 4.4 Data dictionary / metadata catalog
Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
### 4.5 Artifacts
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
- **`instruments_repository`** — instrument entities + characteristics + ranges.
- **`rejects_store`** — quarantined anomalies with reason codes.
- **`ontology_notes`** — running record of ERD changes and rationale.
### 4.6 Exit criteria
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
---
## 5. Phase 4 — Conversion & Export
**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
### 5.1 Targets
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
### 5.2 Design
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
- Lineage is emitted alongside every export (see §6).
### 5.3 Artifacts
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
### 5.4 Exit criteria
- All five target formats generated from one canonical snapshot.
- Round-trip validation: record counts and key aggregates match across formats.
- Lineage complete and resolvable for sampled records.
---
## 6. Cross-cutting concerns
### 6.1 Lineage & processing-flow documentation
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
- Lineage is a first-class export, present in every target format.
### 6.2 Configuration & rule registry
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
### 6.3 Validation gates
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
---
## 7. Target artifacts catalog (consolidated)
| Artifact | Phase | Form |
|----------|-------|------|
| storage_manifest | 1 | JSONL + SQLite |
| schema_fingerprint_report | 1 | JSON/CSV |
| unmatched_report | 1 | CSV |
| storage_map | 1 | Markdown |
| column_mapping | 2↔3 | CSV/JSON (versioned) |
| canonical_dataset | 2↔3 | partitioned intermediate |
| relational_schema (DDL) | 2↔3 | SQL |
| data_dictionary | 2↔3 | table/CSV |
| instruments_repository | 2↔3 | table/CSV |
| rejects_store | 2↔3 | JSONL |
| ontology_notes | 2↔3 | Markdown |
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
| lineage + build manifest | all | JSONL |
---
## 8. Tooling & environment
- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
- **PostgreSQL** materialized in Phase 4 (system of record).
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
- Python indentation: **tabs**.
---
## 9. Performance strategy for 700 GB
- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
---
## 10. Open items & risks
1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
---
*End of Plan v1.*

188
docs/examples/Tribology_Ontology.html Normal file
View File

File diff suppressed because one or more lines are too long

112
docs/rules/bench-methodology.md Normal file
View File

@@ -0,0 +1,112 @@
# Bench / Methodology - isolation, repetitions, timeouts, honest reporting
**Purpose + scope.** The measurement discipline for the benchmark matrix
(Q1-Q7 x 5 formats, spec [08](../specs/08_benchmark_queries.md) /
[09](../specs/09_benchmark_execution.md)) and for the extrapolation and
reporting built on it (specs 10-11). Binding for `bench_runner.py`, every
query implementation, `extrapolate.py`, and `report.py`.
The product of this project IS these measurements; a sloppy protocol
invalidates everything downstream.
---
## 1. Cell isolation - one subprocess per measurement
- Every benchmark cell (format x query x run) executes in its OWN
subprocess, so peak RSS, CPU time, and IO counters attribute to exactly
that cell.
- The runner samples the child with `psutil` every 50 ms for peak RSS;
CPU time is user+sys at exit; read bytes from `io_counters`.
- Nothing else runs in the child: no logging setup beyond minimum, no
imports the query does not need - the measurement must not pay for the
harness.
## 2. Repetition protocol
- Per cell: **1 warm-up run (discarded) + 3 measured runs.** Report the
median in `results_median.csv`; keep min/max in the raw data.
- **Randomize the cell execution order** and record the actual order in
the raw results - fixed order lets cache effects masquerade as format
differences.
- SQLite additionally reports run 1 (cold) and run 3 (warm) separately
(spec 08).
- Never average TIMEOUT/ERROR runs into medians; a cell with a failed run
reports the failure, not a prettier subset.
## 3. Timeouts and failures are data
- Hard per-run timeout: **30 minutes.** On expiry the runner records
`TIMEOUT` for that cell and CONTINUES the matrix (an expected outcome:
RDF Q5). Never retry a timeout hoping for a better number.
- A crashed cell records `ERROR` the same way (see
[code-error-handling.md](code-error-handling.md) pattern C).
- In extrapolation and scoring, TIMEOUT/FAIL project to score 0 - they
are never silently dropped from tables or charts (spec 10/11 flags:
projection > 1 h = `IMPRACTICAL`, > 24 h = `FAIL`).
## 4. Result correctness gates the measurement
- Every measured run validates its result rows against the canonical
checksum (sorted rows, float tolerance 1e-9) and records `checksum_ok`
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
- A fast run with `checksum_ok = false` is an INVALID measurement - it
must never enter medians, extrapolation, or the report.
- **Q5 reads raw `friction_cycles` in every format** - summary tables and
materialized views are forbidden for it; verify PostgreSQL did not
satisfy it from `track_summary`
(`EXPLAIN (ANALYZE, BUFFERS)` is recorded for PostgreSQL runs anyway).
## 5. Fairness across formats
- Identical result-set contract for all 5 implementations of each query;
no format answers a simplified question.
- Each format uses its idiomatic strength honestly: CSV Q1 reads the one
target file by path; JSON Q1 loads the one coupon file; SQL uses its
indexes. What is forbidden is capping or sampling (`LIMIT`, partial
scans) that changes the answer.
- The pandas CSV variant is measured as a SEPARATE variant, never mixed
into the plain-CSV numbers (spec 08).
- Storage sizes compare like with like: for every format record both the
working footprint and the gzip archival footprint in
`storage_sizes.csv` (spec 09).
## 6. Environment honesty - this machine, stated limits
- Record the hardware and software environment (CPU model, cores, RAM,
disk type, OS, engine versions) once per benchmark session into the
raw results; the report cites it.
- **Cold-cache protocol on this host:** the spec's page-cache drop
(`/proc/sys/vm/drop_caches`) is Linux-only; this project runs on
Windows Server 2025, where it is unavailable. The cold-cache matrix
pass is therefore SKIPPED and marked as such in results and report -
never approximated by a fake (e.g. file re-copy tricks) without a
rules update describing the method.
- Quiesce the machine during measured runs (no parallel pipeline tasks,
no background loads); the runner runs cells sequentially by design.
## 7. Extrapolation discipline (task 10)
- Scaling laws per access pattern are those fixed in spec 10 (O(1),
O(log n + k), O(k log n), O(n), O(n / min(cores, partitions)));
constants calibrate on the measured 150 MB point.
- Every projected number in the report is labeled as projected, with its
scaling assumption; measured and projected values never mix in one
column unlabeled.
- Hardware cost figures come from `hw_prices.yaml`
(see [code-config-yaml.md](code-config-yaml.md)), never inlined.
## 8. Anti-patterns
- **Running cells in-process** - RSS/CPU attribution becomes fiction.
- **Fixed execution order** - cache warmth leaks between formats.
- **Retrying or dropping slow runs** - the slow run IS the result.
- **Averaging over a TIMEOUT** or reporting medians of fewer than 3 valid
runs without flagging it.
- **A "fast" result that fails the checksum** entering any downstream
table.
- **Tuning one format's query while leaving others naive** - fairness is
per-scenario, not per-format-champion.
- **Faking cold-cache on Windows** without a documented, rules-approved
method.
- **Presenting projected numbers as measured** in tables or charts.

103
docs/rules/build-pipeline-tasks.md Normal file
View File

@@ -0,0 +1,103 @@
# Build / Pipeline tasks - the ./out tree, .done markers, task order
**Purpose + scope.** How the 11 pipeline tasks compose: where artifacts
live, how completion is signaled, what order tasks run in, and what git
does and does not track. Binding for every task implementation and for
anyone running the pipeline.
---
## 1. The artifact tree - everything under `./out/`
All generated artifacts live under `./out/` relative to the repo root
(spec 00_PLAN):
```
out/
config/ lab_config.yaml, hw_prices.yaml
csv/ the canonical raw corpus + MANIFEST.csv
json/ full/ (per-coupon JSON-LD), hybrid/dataset.jsonld
sqlite/ tribo.db
pg/ tribo.dump (pg_dump -Fc; the live DB is in PostgreSQL)
rdf/ dataset.nt.gz, dataset.ttl, oxigraph_store/
bench/ queries/, expected/, results_raw.csv, results_median.csv,
storage_sizes.csv, extrapolation.csv, hardware_sizing.csv
report/ REPORT.md, charts/, tables/, process_flow.md, diagrams/
.done/ completion markers, one per task
```
- A task writes ONLY under its own output paths; it treats its inputs as
read-only.
- Tasks communicate exclusively through these artifacts - never through
shared in-process state
(see [code-python-style.md](code-python-style.md)).
- **`./out/` is git-ignored** (except nothing - the whole tree). Sources
of truth are the specs, the rules, and the code; artifacts are
reproducible from them ([data-determinism.md](data-determinism.md)).
Never commit generated artifacts "for convenience".
## 2. Completion markers
- Each task NN writes `./out/.done/<NN>.ok` after - and only after - its
outputs are complete and validated. The marker body carries the
headline counts (rows, bytes, triples) downstream tasks check
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
- A task verifies its dependencies' markers before starting and aborts
loudly if one is missing.
- Re-running a task: delete its own marker first, regenerate outputs,
rewrite the marker. If its outputs changed, downstream markers are now
stale - delete them too (the dependency graph below says which).
## 3. Task graph and execution order
```
01 -> 02
01 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11
09 ------> 11
```
- Sequential order: 01, 02, 03, then 04-07 (independent, may run in
parallel), 08, 09, 10, 11.
- 02 (process diagrams) needs only 01 and can run any time after it.
- Task 09 (benchmark execution) must run WITHOUT 04-07 style parallel
work in the background - measurements need a quiet machine
([bench-methodology.md](bench-methodology.md) section 6).
## 4. `storage_sizes.csv` - the shared append contract
- Location: `./out/bench/storage_sizes.csv`; columns:
`format, variant, bytes`.
- Tasks 04-07 APPEND their format's rows (e.g. `json,full,...`,
`json,hybrid,...`, `sqlite,db,...`, `pg,live,...`, `pg,dump,...`,
`rdf,nt_gz,...`, `rdf,store,...`); task 09 appends the gzip archival
variants and consolidates.
- Appends are idempotent per (format, variant): a re-run REPLACES its own
prior rows, never duplicates them and never touches other formats'
rows.
- RDF load wall-time and similar per-format load metrics recorded per
spec 07 go into the task's `.done` marker and the report, not into
extra ad-hoc files.
## 5. Runs are resumable, not restartable-from-zero
- The marker system exists so a failed pipeline resumes at the failed
task; do not delete the whole `./out/` tree to retry one task.
- Conversely, after a deliberate corpus regeneration (03), ALL downstream
markers and artifacts (04-11) are stale and must be removed - a mixed
tree of old and new artifacts is worse than an empty one
([meta-no-assumptions.md](meta-no-assumptions.md) applies before any
such deletion).
## 6. Anti-patterns
- **Writing an artifact outside `./out/`** or a task writing into another
task's output directory.
- **Committing `./out/` content to git.**
- **A marker written on partial success** - see
[code-error-handling.md](code-error-handling.md).
- **Running benchmarks while converters are still running.**
- **Duplicate rows in `storage_sizes.csv`** after re-runs - replace your
own rows.
- **Hand-editing generated artifacts** (a "quick fix" inside
`MANIFEST.csv` or a result CSV) - regenerate them through the owning
task instead.

61
docs/rules/code-comments-policy.md Normal file
View File

@@ -0,0 +1,61 @@
# Code comments - write WHY, not WHAT
**Binding** for all source files: Python, SQL, SPARQL, PowerShell, YAML, shell.
## Rule
Default to **no comment**. Add one only when the *why* is non-obvious from
the code. If removing the comment wouldn't confuse a future reader, delete it.
A comment earns its place when it captures something the code can't:
hidden constraint, ordering invariant, workaround for a named bug, vendor
quirk, cross-file invariant the type system can't enforce.
## Delete on sight
```python
# Read the manifest file
manifest = read_manifest(csv_root) # function name says it
```
```python
# Increment counter
counter += 1 # pure restatement
```
```python
# Added for the benchmark rework (see task 09)
def run_cell(...): # belongs in commit message, rots in code
```
```python
# TODO: refactor later
# no owner, no trigger - decoration
```
## Keep
```python
# Q5 must scan raw friction_cycles - track_summary is forbidden for this
# query in every format (spec 08); do not "optimize" this into the summary.
```
Cross-file invariant a refactor must preserve.
```python
# Column order matches MANIFEST.csv writer in generate_data.py - keep in sync.
```
Cross-file invariant the type system can't enforce.
```sql
-- BRIN, not b-tree: cycle values are physically ordered within each
-- partition, so BRIN gives the same pruning at ~1% of the index size.
```
Non-obvious engineering decision with a reason.
## Length
Inline: one line. Block: 1-4 lines. If you need more than a paragraph,
the right home is a rule or spec doc - link it from a one-line comment.
Stale comments are worse than missing ones. When you touch a file, fix or
delete comments whose *why* no longer holds.

94
docs/rules/code-config-yaml.md Normal file
View File

@@ -0,0 +1,94 @@
# Code / Config - all tunables live in YAML, no magic numbers, no .env
**Purpose + scope.** Governs where every configurable value lives. Binding
for all Python code in this repository. Any new configurable parameter MUST
follow this rule when deciding where it goes.
The goal: one authoritative source per parameter, so regenerating any
artifact with the same config is byte-identical, and changing a parameter
never requires hunting constants across scripts.
---
## 1. The two YAML configs
| File | Owns | Written by |
|---|---|---|
| `./out/config/lab_config.yaml` | The laboratory model: volumes (batches / wafers / coupons / tracks / cycles), the deposition matrix, physical-model coefficients, nominal test conditions, **the random seed 20260711** | Task 01, from the spec; regenerated only deliberately |
| `./out/config/hw_prices.yaml` | Hardware cost parameters for extrapolation (RAM $/GB, NVMe $/TB, server base prices) | Task 10 seeds editable defaults; the operator may edit |
Rules:
- **Scripts READ these files; they never duplicate values from them.**
Task 03 reads volumes, coefficients, and the seed from
`lab_config.yaml` - it does not hardcode 588 coupons or seed 20260711.
Task 10 reads prices from `hw_prices.yaml` - it does not inline dollar
figures.
- A parameter needed by more than one task lives in the YAML, never
copy-pasted between scripts.
- New tunables go into the appropriate YAML with a self-describing key;
if neither file fits, propose a new config file in a docs/rules update
first.
## 2. Spec-fixed constants - named, sourced, at the top
Values fixed by the specs that are NOT operator-tunable (the 120-200 MB
size gate, the 30-minute cell timeout, the 50k-row transaction batch, the
50 ms RSS sampling interval) are module-level named constants at the top of
the owning script, each with a one-line comment naming the spec:
```python
SIZE_GATE_MB = (120, 200) # abort bounds, spec 03
CELL_TIMEOUT_S = 30 * 60 # hard per-run timeout, spec 09
```
A bare number inline in the logic (`if size > 209715200`) is forbidden.
## 3. The PostgreSQL DSN - one env var, never hardcoded, never logged
The only external resource is the local PostgreSQL 16 instance. Its DSN:
- Comes from the environment variable **`TRIBO_PG_DSN`**; default when
unset: `postgresql://postgres@localhost:5432/tribo`.
- Is read in ONE shared helper (e.g. `common/pg.py: get_dsn()`); scripts
call the helper, never `os.environ` directly.
- Is never hardcoded in a script, never committed in a config file, and
never logged (a DSN may embed a password - see
[obs-python-logging.md](obs-python-logging.md)).
There are no other secrets in this project. There is no `.env` file; do
not introduce one.
## 4. CLI flags - per-run choices only
Command-line flags are for choices that vary per invocation, not for
model parameters: `--log-level`, `--dry-run`, an output-root override, a
`--limit` for smoke runs. A physical coefficient or a volume as a CLI flag
is forbidden - it would silently fork the config out of `lab_config.yaml`.
## 5. Anti-patterns
- **Hardcoding a value that exists in `lab_config.yaml`** (seed, volumes,
coefficients) - the YAML is the single source; drift here breaks
reproducibility silently.
- **Adding a `.env` file or reading `os.environ` scattered through
scripts** - the DSN goes through the one shared helper; nothing else
lives in the environment.
- **Inline magic numbers** for spec-fixed gates and timeouts - name them
at the top with the spec reference.
- **Dollar prices inlined in `extrapolate.py`** - they live in
`hw_prices.yaml` so the operator can re-cost without touching code.
- **A model parameter as a CLI flag** - per-run flags are for run
mechanics, not for the lab model.
- **Committing `./out/config/` generated files to git as if they were
sources** - the YAML under `./out/` is a generated artifact; its source
of truth is spec 01 (see
[build-pipeline-tasks.md](build-pipeline-tasks.md)).
## 6. Related documents
- [data-determinism.md](data-determinism.md) - the seed contract.
- [build-pipeline-tasks.md](build-pipeline-tasks.md) - the `./out/` tree
and what is git-ignored.
- [obs-python-logging.md](obs-python-logging.md) - `--log-level` flag; DSN
never logged.

132
docs/rules/code-data-formatting.md Normal file
View File

@@ -0,0 +1,132 @@
# Code / Data / Formatting - ASCII-only typographic characters in source files
**Binding scope.** Strings that end up in any of the following MUST use ASCII
typographic characters; never their Unicode "smart" equivalents:
- Python source, SQL and SPARQL query files
- Generated data and artifacts written to disk (CSV, JSON-LD, N-Triples,
Turtle, MANIFEST and benchmark result CSVs, log messages)
- Configuration files (`.yaml`, `.json`, `.toml`)
- PowerShell / shell scripts (any file processed by Windows tooling)
- Markdown documentation (`docs/`, `README.md`, `CLAUDE.md`) - the specs
were normalized to ASCII typography on 2026-07-11 and stay that way
**Long dashes are an absolute exception, no carve-outs.** Em-dash (U+2014)
and en-dash (U+2013) MUST be replaced with ASCII hyphen-minus (`-`) in
**every** file in this repository, including markdown documentation, code
comments, and report prose. Reason: long dashes are visually
indistinguishable from a hyphen in many monospace fonts, break diffs / grep /
IDE search, and round-trip badly through Windows tooling chains (section 2).
---
## 1. The substitution table
| Unicode char | Codepoint | Name | ASCII replacement |
|---|---|---|---|
| em-dash | U+2014 | em-dash | `-` (single hyphen-minus) |
| en-dash | U+2013 | en-dash | `-` (single hyphen-minus) |
| horizontal bar | U+2015 | horizontal bar | `-` |
| minus sign | U+2212 | minus sign | `-` |
| ellipsis | U+2026 | horizontal ellipsis | `...` (three ASCII dots) |
| left double quote | U+201C | left double quotation mark | `"` |
| right double quote | U+201D | right double quotation mark | `"` |
| left single quote | U+2018 | left single quotation mark | `'` |
| right single quote | U+2019 | right single quotation mark | `'` |
| left guillemet | U+00AB | left guillemet | `"` |
| right guillemet | U+00BB | right guillemet | `"` |
| nbsp | U+00A0 | non-breaking space | regular space |
**Single hyphen, not double.** When replacing an em-dash that separates two
clauses, use a single `-` surrounded by spaces (`text - clause`), NOT `--`.
**Ranges use a single hyphen:** `Q1-Q7`, `120-200 MB`, `0.3-1.1 um` - never
an en-dash.
## 2. Why - documented failure modes
1. **PowerShell 5.1 reads no-BOM files as Windows-1252.** Non-ASCII UTF-8
bytes become mojibake and break string-literal parsing. This project is
developed on Windows; any script or config touched by PowerShell is
exposed.
2. **Windows -> git-bash -> tar / packaging round-trip.** At any step an
unset locale or wrong encoding assumption produces mojibake. A UTF-8 byte
sequence for U+2014 interpreted under a non-UTF-8 locale renders as a
`?` replacement character in the packaged output.
3. **Copy-paste from Word / Google Docs / Notion / web articles.** These
tools auto-substitute typography (`"` -> smart quotes, `'` -> smart
apostrophe, `--` -> em-dash, `...` -> ellipsis) silently. The
substitution survives copy into any editor and lands in source files
unnoticed. The original specs arrived with en/em dashes this way.
4. **Data files must diff and grep cleanly.** The CSV corpus, MANIFEST, and
benchmark results are compared by checksum and processed by streaming
parsers; a stray typographic character breaks byte-level reproducibility.
## 3. Where the rule does NOT apply
- **Scientific and unit notation is content, not typography.** Characters
such as `µ` (micro), `±`, `°C`, `×`, `Ø`, and Greek letters (`τ`, `σ`) in
formulas inside the specs are meaningful scientific notation and are kept.
Note the boundary: in **column names and code identifiers** the units are
always plain ASCII (`_um`, `_nm`, `_GPa` - see
[data-naming-units.md](data-naming-units.md)); the scientific characters
live only in explanatory prose.
- **International language strings** (Cyrillic, CJK, etc.) are meaningful
content, not typographic substitutions. UTF-8 throughout is the right
answer. (Repository artifacts are English; this matters only for quoted
material.)
- **Imported reference materials** in `docs/examples/` are kept verbatim as
received; they are not binding documents.
## 4. Detection / enforcement
There is no automated check for this rule yet - **not in CI yet**. The rule is
enforced by review and by the editing discipline in this file. The intent of a
future detector is documented here so it can be built consistently:
- **What it flags.** Any occurrence of the Unicode codepoints in the section 1
substitution table inside in-scope files. The section 3 exemptions
(scientific notation, international content, `docs/examples/`) are the only
carve-outs.
- **Exit behavior.** Non-zero exit when any flagged codepoint is found under
the scan roots, so it can later gate a commit hook or a build step.
- **Tooling.** When this detector is created it MUST be a PowerShell script
(this is a Windows project) and live under `tools/`.
## 5. Examples
### Good
```python
# safe through every Windows tooling step
print("Corpus size 152.3 MB - within the 120-200 MB gate.")
label = "steady-state COF, cycles 500-1000"
```
### Bad
```python
# em-dash literal (U+2014) - mojibake risk on PowerShell read/write
print("Corpus size 152.3 MB [U+2014] within the gate.")
# en-dash range (U+2013) - breaks grep for "Q1-Q7"
label = "queries Q1[U+2013]Q7"
```
## 6. Anti-patterns
- **Copy-pasting strings from a word processor or web page** into source /
specs / config without normalization.
- **Trusting tar / packaging to "preserve as-is".** They preserve bytes; the
*encoding interpretation* is what gets lost when locale or encoding
declaration is wrong on either end.
- **Assuming the source file encoding alone is enough.** The source file may
be fine - the failure is at downstream consumers (PowerShell read, build
write, OS locale).
- **Using `--` instead of `-` "for emphasis".** Single ASCII hyphen
surrounded by spaces is the project's chosen separator.
- **Putting `µ` or `°` into a column name or identifier** - units in names
are ASCII (`_um`, `_C`); see [data-naming-units.md](data-naming-units.md).

181
docs/rules/code-error-handling.md Normal file
View File

@@ -0,0 +1,181 @@
# Code / Error handling - never swallow, always surface, fail loud
**Purpose + scope.** Governs every `try:` / `except:` / `raise` site in
Python code in this repository (data generation, format converters, benchmark
runner, extrapolation, reporting, helper tools). Binding for all contributors.
The pipeline's core failure principle: **a wrong artifact is worse than no
artifact.** Every task's output feeds downstream tasks; silently degraded
data poisons every result built on it. When in doubt, abort with a clear
message and a non-zero exit code.
---
## 1. The rule
1. **Never `except: pass`.** A bare swallow is always a bug. Forbidden
absolutely.
2. **Never `except Exception: logger.warning("...") + continue`** for
anything that was not an EXPECTED, documented, transient condition. If
you swallow and continue, you must be able to state why the program can
correctly proceed.
3. **Always `logger.error(..., exc_info=True)` in `except` blocks.**
`exc_info=True` captures the traceback into the log record (see
[obs-python-logging.md](obs-python-logging.md)). Without it the original
exception is lost.
4. **Re-raise unless you have a defined recovery path.** If the caller
cannot continue meaningfully without the failed operation, re-raise
after logging.
5. **Surface to the right channel:**
- **CLI script (every pipeline task):** print the error to stderr and
exit non-zero. A task that failed MUST NOT write its
`./out/.done/<task>.ok` marker (see
[build-pipeline-tasks.md](build-pipeline-tasks.md)).
- **Validation gate (row counts, checksums, size bounds):** a failed
gate is an abort, not a warning. Example: task 03 aborts when the
generated tree is outside 120-200 MB; converters abort when row counts
do not match `MANIFEST.csv`.
- **Library / helper function:** re-raise; let the caller decide.
6. **Defensive "this cannot happen" branches must surface, not silently
`return`.** If you reach a defensive branch, log at ERROR with the
branch name and the inputs that got you there, then raise.
## 2. The exception ladder - the only three sanctioned swallow patterns
Code may swallow ONLY in these three documented patterns. Anything else
re-raises after logging.
### Pattern A: documented, recoverable transient (retry)
```python
for attempt in range(MAX_RETRIES):
try:
return do_thing()
except (psycopg.OperationalError, TimeoutError) as exc:
logger.warning("attempt %d/%d failed: %s", attempt + 1, MAX_RETRIES, exc)
time.sleep(backoff(attempt))
# All retries exhausted - surface.
logger.error("do_thing failed after %d retries", MAX_RETRIES)
raise
```
Justified: the per-attempt warning is acceptable; the final failure
re-raises.
### Pattern B: optional best-effort cleanup
```python
def _close_conn(conn) -> None:
try:
conn.close()
except Exception:
# Cleanup is best-effort; the operation already finished, log and continue.
logger.warning("closing connection failed", exc_info=True)
```
Justified: the comment names why continuing is correct; the traceback is
still captured.
### Pattern C: benchmark-cell isolation - record and continue the matrix
```python
try:
result = run_cell(fmt, query, timeout_s=30 * 60)
except CellTimeout:
logger.error("cell %s/%s timed out", fmt, query)
write_result_row(fmt, query, status="TIMEOUT") # recorded, not hidden
except Exception:
logger.error("cell %s/%s failed", fmt, query, exc_info=True)
write_result_row(fmt, query, status="ERROR")
```
Justified: spec 09 requires the benchmark matrix to survive a failing cell
(RDF Q5 is expected to time out). The failure is RECORDED as data
(`TIMEOUT` / `ERROR` in `results_raw.csv`), never silently dropped, and the
run continues with the next cell. This pattern applies ONLY to the benchmark
runner's per-cell boundary - a failure in the runner itself (setup, result
writing) still aborts.
## 3. Loading into SQLite / PostgreSQL - all-or-nothing per table
A converter that half-loads a table leaves a database that LOOKS complete
and silently skews every benchmark run against it.
1. Wrap each table's bulk load in a transaction; on any error roll back
and abort the converter with a non-zero exit.
2. After loading, validate row counts against `MANIFEST.csv` and fail loud
on any mismatch (see
[test-pipeline-validation.md](test-pipeline-validation.md)).
3. Never write the task's `.done` marker after a partial load.
## 4. Anti-patterns
- **`except: pass`** - forbidden absolutely.
- **`except Exception: logger.warning("could not X")` + continue** -
forbidden unless the function documents that "could not X" is a
recoverable expected outcome. If it is, phrase it "X skipped (expected
condition Y)" so a reader does not grep "could not" hunting for bugs.
- **Catching `BaseException`** - forbidden (swallows `KeyboardInterrupt`,
`SystemExit`). Use `Exception`.
- **`logger.error("X")` without `exc_info=True` in an `except` block** -
forbidden. The traceback IS the diagnostic.
- **`logger.error(str(exc))`** - forbidden; redundant with `exc_info=True`
and loses type and stack.
- **Returning `None` to signal failure** from a function whose contract is
"return T" - forbidden. Either raise, or return `T | None` with
documented semantics the caller checks.
- **`raise Exception("...")`** - forbidden. Use a specific class:
`ValueError` / `RuntimeError` / `KeyError` for trivial cases, a custom
domain exception subclass (e.g. `ManifestMismatchError`) for domain
conditions.
- **`raise` chained without context** - inside an `except`, prefer
`raise NewError(...) from exc` so the original cause stays visible in the
chained traceback. A bare `raise` that re-raises the same exception is
fine; `raise SomethingElse(...)` without `from exc` is forbidden.
- **Writing a `.done` marker in a `finally` block** - the marker means
"completed successfully", never "attempted".
- **Clamping / repairing out-of-range generated values silently** - a
physical-model bug must surface, not be masked by a `max(0, ...)` added
"to be safe". Deliberate clamps defined by the spec (e.g. COF floor 0.12)
are model logic, not error handling.
## 5. Defensive branches
```python
# Good:
def open_store(fmt: str) -> Store:
if fmt == "sqlite":
return SqliteStore()
if fmt == "postgresql":
return PgStore()
# Defensive: a new format added without updating this factory lands here.
logger.error("open_store: unknown format=%r", fmt)
raise ValueError(f"unknown storage format: {fmt!r}")
```
The silent-`return None` form lets the unknown value propagate and breaks
downstream code with a confusing `AttributeError` on the wrong line.
## 6. Logging vs surfacing - the decision
For a caught exception, ask in order:
1. **Will the pipeline produce a wrong artifact if I continue?** Re-raise /
abort.
2. **Is this expected and recoverable in the caller's contract?** Log at
WARNING with `exc_info=True`, return a sentinel the caller checks.
3. **Is this an isolated benchmark cell (pattern C)?** Log at ERROR, record
the status as data, continue the matrix.
4. **Is the process going down anyway?** Log at CRITICAL with
`exc_info=True`, then `sys.exit(1)` or let it propagate.
## 7. Related documents
- [obs-python-logging.md](obs-python-logging.md) - logging API and severity
levels.
- [test-pipeline-validation.md](test-pipeline-validation.md) - the
validation gates that turn silent corruption into loud failures.
- [build-pipeline-tasks.md](build-pipeline-tasks.md) - `.done` marker
semantics.
- [bench-methodology.md](bench-methodology.md) - the TIMEOUT contract for
benchmark cells.

93
docs/rules/code-python-style.md Normal file
View File

@@ -0,0 +1,93 @@
# Code / Python style - tabs, 3.11+, closed dependency list, streaming
**Purpose + scope.** Baseline style and dependency policy for every Python
file in this repository (pipeline tasks 01-11, query implementations,
helper tools). Binding for all contributors.
---
## 1. Language and formatting
- **Python 3.11+.** Use modern stdlib freely (`pathlib`, `dataclasses`,
`tomllib` if needed); no compatibility shims for older versions.
- **Indentation: TABS.** Set by the project owner and repeated in the
specs (00_PLAN, 03, 09). Every Python file in this repository indents
with tab characters, never spaces. Configure the editor per file type;
do not "fix" tab-indented files to spaces.
- `from __future__ import annotations` at the top of every module.
- Type hints on public function signatures; internal helpers at author's
discretion.
- English for all identifiers, docstrings, and comments (see
[meta-language-style.md](meta-language-style.md) - Russian is the
conversation language, not the code language).
- Comments follow [code-comments-policy.md](code-comments-policy.md):
default to none, keep only non-obvious WHY.
## 2. Dependencies - a closed list
The stdlib is the default. The ONLY permitted third-party packages:
| Package | Used by | For |
|---|---|---|
| `numpy` | 03 | vectorized generation, seeded RNG |
| `PyYAML` | 01, 03, 10 | reading/writing the YAML configs |
| `rdflib` | 04, 07 | JSON-LD validation, RDF serialization |
| `oxigraph` | 07, 09 | embedded queryable triplestore |
| `ijson` | 08, 09 | streaming JSON parsing |
| `psycopg` (v3) | 06, 08, 09 | PostgreSQL COPY and queries |
| `psutil` | 09 | RSS / CPU / IO measurement |
| `matplotlib` | 11 | report charts |
| `pandas` | 08 (optional) | ONLY as the separately-measured second CSV query variant; never in the pipeline itself |
Adding any other package requires updating this table first. In
particular forbidden: ORMs, logging frameworks
([obs-python-logging.md](obs-python-logging.md)), CLI frameworks
(argparse suffices), `requests`/HTTP clients (nothing to call).
Task 03 (data generation) is stricter by spec: **stdlib + numpy only.**
## 3. Script shape - every task is a self-contained CLI
- One entry script per pipeline task, runnable as
`python <script>.py [flags]`.
- `argparse` with `--log-level` (see
[obs-python-logging.md](obs-python-logging.md)) and task-appropriate
per-run flags ([code-config-yaml.md](code-config-yaml.md) section 4).
- A module docstring stating the task number, what it produces, and its
spec file.
- `main() -> int` returning the exit code; `sys.exit(main())` under
`if __name__ == "__main__":`.
- Exit code 0 only on full success (outputs written, validation passed,
marker written).
## 4. Streaming discipline - the corpus never fits in memory by design
The generated corpus is ~150 MB and its scaled-up target is 600 GB; code
must behave as if the data never fits in RAM:
- **Write generated rows as you produce them**; never accumulate the full
cycle/loop dataset in a list (spec 03: "stream rows; never hold the
full cycle dataset in memory").
- **Read CSV/JSON with iterators** (`csv.reader` row-by-row, `ijson`
events); never `read()` a bulk file whole or `json.load` the full
corpus.
- Batch database work (`executemany` / `COPY` in chunks), keeping only
the current batch in memory.
- Bounded aggregation state is fine (per-track summaries, counters);
unbounded accumulation of raw rows is the thing being forbidden.
## 5. Anti-patterns
- **Spaces for indentation** in Python files - the project standard is
tabs.
- **An unlisted third-party import** - extend the section 2 table first
or use the stdlib.
- **`pd.read_csv` in pipeline code** - pandas exists only as the
separately-measured Q-variant in spec 08.
- **Loading a bulk artifact whole** (`json.load` on the full corpus,
`readlines()` on a cycles CSV) - stream it.
- **A task importable only from a specific CWD** - resolve paths from a
repo-root anchor, not the current directory.
- **Two tasks sharing mutable module state** - tasks communicate through
`./out/` artifacts only (see
[build-pipeline-tasks.md](build-pipeline-tasks.md)).

85
docs/rules/data-determinism.md Normal file
View File

@@ -0,0 +1,85 @@
# Data / Determinism - one seed, byte-identical regeneration
**Purpose + scope.** The reproducibility contract for all generated data.
Binding for task 03 (CSV generation) and for every piece of code that
produces values which end up inside the corpus or its conversions.
The entire evaluation rests on this: benchmark results are only comparable
because every format holds EXACTLY the same data, and the study is only
reviewable because anyone can regenerate that data bit-for-bit.
---
## 1. The seed
- **The global seed is `20260711`.** It lives in `lab_config.yaml`
(written by task 01) and is READ from there - never hardcoded in a
generation script (see [code-config-yaml.md](code-config-yaml.md)).
- Changing the seed is a project-level decision, never a side effect. It
invalidates the whole `./out/` tree, every checksum, and every measured
result (see [meta-no-assumptions.md](meta-no-assumptions.md)).
## 2. Sanctioned randomness - seeded generators only
- The ONLY random sources are `random.Random(seed)` and
`numpy.random.default_rng(seed)`, constructed explicitly from the
config seed.
- Derive per-entity generators deterministically when parallel or
out-of-order generation needs them (e.g.
`default_rng([seed, batch_index, coupon_index])`) - never from time,
never from a global unseeded state.
- Forbidden anywhere in generation code: module-level `random.*` calls,
`numpy.random.*` legacy global functions, `os.urandom`, `uuid.uuid4`,
`hash()`-dependent ordering (Python string hashing is salted per
process).
## 3. No wall-clock in the data
- Every timestamp inside the corpus is SIMULATED - computed from the lab
schedule model - never `datetime.now()` / `time.time()`.
- Real wall-clock time may appear only in logs and in measurement records
(benchmark wall times in `./out/bench/` are measurements, not corpus
data).
## 4. Deterministic everything else
Randomness is not the only nondeterminism; all of these are binding:
- **Iteration order.** Iterate in a defined order (sorted paths, explicit
index loops). Never rely on `os.listdir` / `glob` / `set` / dict-of-set
ordering for anything that affects output. Sort directory listings
before use.
- **Float formatting.** Format floats with an explicit spec (e.g.
`f"{x:.6g}"`), chosen once per column family; repr drift between runs
or Python versions must not change file bytes.
- **Line endings and encoding.** Generated files are UTF-8 with `\n`
newlines, on every platform (`open(..., newline="")` for csv writers).
- **Parallelism.** If generation is parallelized, each worker's output is
deterministic (derived generators, fixed partitioning) and the merge
order is fixed.
## 5. The proof - MANIFEST sha256
- `./out/csv/MANIFEST.csv` records sha256 per generated file (spec 03).
**Regenerating with the same `lab_config.yaml` MUST reproduce every
sha256 exactly.** This is the acceptance test for this whole rule -
run generation twice when touching generator code and diff the
manifests.
- Downstream conversions must be equally deterministic in CONTENT (same
rows, same values); byte-identity of database files is not required
(SQLite/PostgreSQL internals may differ), which is why their gate is
row counts + query-result checksums instead
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
## 6. Anti-patterns
- **Hardcoding `20260711`** in a script instead of reading the config.
- **An unseeded or globally-seeded random call** anywhere in generation.
- **`datetime.now()` in corpus values** - simulated schedule only.
- **Ordering that depends on the filesystem or hash salting** - sort
explicitly.
- **"Harmless" float repr changes** - a formatting change is a corpus
change; it must be deliberate and re-baselines every checksum.
- **Regenerating the corpus to "refresh" it** - identical input config
must yield identical output; if bytes changed without a config change,
that is a determinism bug to fix, not a new baseline to accept.

120
docs/rules/data-naming-units.md Normal file
View File

@@ -0,0 +1,120 @@
# Data / Naming and units - IDs, timestamps, SI-explicit column names
**Purpose + scope.** The naming grammar for entities, timestamps, and
measurement columns across ALL representations of the corpus: CSV, JSON-LD,
SQLite, PostgreSQL, RDF. Binding for generation, conversion, queries, and
reporting. This file also fixes two points the specs leave ambiguous
(sections 3 and 4).
---
## 1. Entity identifiers
The hierarchy encodes containment left-to-right; each level appends one
segment:
| Entity | Code grammar | Example |
|---|---|---|
| batch | `B<nnn>` | `B721` |
| wafer | `<batch>-W<n>` | `B721-W1` |
| coupon | `<wafer>-C<nn>` (01..49, the 7x7 grid) | `B721-W1-C07` |
| track | `<coupon>-T<n>` (1..3) | `B721-W1-C07-T2` |
| run | `R<n>` (1..4) | `R3` |
- **CSV** stores the bare code (`B721-W1-C07`) in `<entity>_code`-style
columns.
- **JSON-LD / RDF** use the prefixed CURIE form as the node identity:
`batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`,
`track:B721-W1-C07-T2`, `run:R1` (spec 00_PLAN global conventions).
- **SQL** stores the bare code as the `<singular>_code` TEXT UNIQUE column
next to the integer surrogate key
(see [db-sql-schema.md](db-sql-schema.md)).
- A code is never parsed to recover its parent where a proper reference
exists; the parent relation is explicit in every format (directory
nesting, `partOf`, FK).
## 2. Timestamps
- ISO-8601, UTC, `Z` suffix, second precision:
`2026-03-14T09:30:00Z`.
- All corpus timestamps are simulated
(see [data-determinism.md](data-determinism.md) section 3).
- In SQL: `TIMESTAMPTZ` in PostgreSQL; TEXT in ISO-8601 form in SQLite
(sortable lexicographically by construction).
## 3. Measurement columns - unit suffix is part of the name
Every numeric measurement column ends in an explicit SI unit token. The
canonical tokens (as they appear in CSV headers):
| Token | Unit | Example columns |
|---|---|---|
| `_um` | micrometre | `thickness_um`, `position_um` |
| `_nm` | nanometre | `ra_nm` |
| `_GPa` | gigapascal | `hardness_GPa`, `reduced_modulus_GPa` |
| `_mN` | millinewton | `max_load_mN`, `friction_force_mN` |
| `_wtpct` | weight percent | `au_wtpct` |
| `_um3` | cubic micrometre | `wear_volume_um3` |
| `_m` | metre | `sliding_distance_m` |
| `_mm_s` | millimetre per second | `speed_mm_s` |
| `_V` | volt | `discharge_voltage_V` |
| `_W` | watt | `power_W` |
| `_C` | degree Celsius | `temperature_C` |
| `_pct` | percent (non-composition) | `rh_pct`, `cpu_util_pct` |
| `_s` / `_mb` | seconds / megabytes (benchmark outputs) | `wall_s`, `peak_rss_mb` |
- Unit tokens are plain ASCII - never `µ`, `°`, or Unicode superscripts
(see [code-data-formatting.md](code-data-formatting.md)).
- Dimensionless quantities carry no token (`cof`, `cycle`, `k_archard`).
- New measurement columns pick from this table or add a row to it in the
same commit.
## 4. Canonical case mapping CSV -> SQL (ambiguity resolved)
The specs write CSV headers with SI capitalization (`hardness_GPa`, spec
03) but SQL DDL in lowercase (`hardness_gpa`, spec 05). Binding
resolution:
- **CSV headers keep SI capitalization** (`_GPa`, `_mN`, `_V`, `_W`,
`_C`) - they mirror instrument output conventions.
- **SQL / JSON-LD / RDF property names are the plain lowercase of the CSV
name** (`hardness_gpa`, `max_load_mn`). One mechanical rule, no
per-column table to maintain.
- Converters map with `csv_name.lower()` - never hand-maintained rename
dicts that can drift.
## 5. `au_wtpct_mean` (spec gap closed)
Specs 05/06 index `coupons(au_wtpct_mean)` but spec 03 never names the
column. Binding definition:
- `au_wtpct_mean` = the arithmetic mean of `au_wtpct` over the coupon's
400 XRF map points, computed by task 03.
- It is written into `coupon_info.csv` by generation (not derived later
by converters), so every format inherits the identical value, and Q2
("coupons with mean Au = 10 +/- 0.5 wt%") means the same rows
everywhere.
## 6. Benchmark artifact columns
Result files under `./out/bench/` follow the same discipline:
`results_raw.csv` columns are
`format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb,
rows, checksum_ok` (spec 09); `storage_sizes.csv` columns are
`format, variant, bytes` (see
[build-pipeline-tasks.md](build-pipeline-tasks.md)). Extending either
file extends this list in the same commit.
## 7. Anti-patterns
- **A measurement column without a unit token** (`thickness`, `load`) -
unreadable and un-greppable across formats.
- **Unit in a comment instead of the name** - comments do not survive
format conversion; names do.
- **Parsing `B721-W1-C07` to find the wafer** where a real reference
exists in the format at hand.
- **A hand-maintained CSV->SQL rename mapping** - the mapping is
`lower()`, nothing else.
- **Recomputing `au_wtpct_mean` in a converter** - it is generated once
into `coupon_info.csv`; converters copy it.
- **Unicode units in identifiers** (`µm`, `°C`) - ASCII tokens only.

122
docs/rules/db-sql-schema.md Normal file
View File

@@ -0,0 +1,122 @@
# DB / SQL / Schema - conventions for the SQLite and PostgreSQL benchmark targets
**Purpose.** Naming, key, and loading conventions for the two relational
storage formats under evaluation: the SQLite file (`./out/sqlite/tribo.db`,
spec [05](../specs/05_convert_sqlite.md)) and the PostgreSQL 16 database
`tribo` (spec [06](../specs/06_convert_postgresql.md)).
**Binding scope.** Every table, index, and load routine in either engine.
The two engines carry the SAME logical schema; they differ only in types,
storage options, and load mechanics. The full table list and DDL signatures
live in the specs - this file codifies the conventions the DDL must follow,
and it wins where a spec detail is ambiguous.
---
## 1. Two kinds of tables
- **Dimension tables** (`batches`, `wafers`, `coupons`, `runs`,
`instruments`, `tracks`): small, joined constantly.
- PK: integer surrogate, named `<singular>_id` (`coupon_id`, `track_id`).
Never a bare `id`, never TEXT.
- The natural key from the CSV corpus (`B721`, `B721-W1-C07`,
`B721-W1-C07-T2` - see [data-naming-units.md](data-naming-units.md))
is stored as a TEXT column named `<singular>_code`, declared `UNIQUE`.
It is for lookups and human-readable output, never the join key.
- **Bulk tables** (`friction_cycles`, `friction_loop_points`, `xrf_points`,
`profilometry_points`, `nanoindentation`, `afm`, `wear`): millions of
rows, scanned and filtered.
- Reference dimensions by integer FK columns only. **TEXT foreign keys in
bulk tables are forbidden** - they bloat the file and slow every scan.
- PK is the natural composite where one exists: `(track_id, cycle)` for
cycles, `(track_id, cycle, pt)` for loop points; single-column
`track_id` / `coupon_id` where the grain is one row per entity. In
SQLite, composite-PK bulk tables are `WITHOUT ROWID`.
- No surrogate `bigserial` on bulk tables - it adds bytes and nothing
else.
## 2. Table and column naming
- Tables: lowercase, snake_case, plural (`friction_cycles`, not
`FrictionCycle`).
- Columns: lowercase snake_case; measurement columns carry the SI unit
suffix in **lowercase** (`hardness_gpa`, `max_load_mn`,
`wear_volume_um3`). The CSV corpus keeps SI capitalization
(`hardness_GPa`); the SQL mapping is a plain lowercase of the CSV name -
see the canonical mapping in
[data-naming-units.md](data-naming-units.md).
- Denormalization is deliberate and limited: `environment` and `load_mn`
are copied onto `tracks` (from `runs` / test conditions) because
Q4 and Q7 filter on them constantly. Do not denormalize further without
updating this rule.
## 3. Indexes - only what Q1-Q7 use
- Create indexes AFTER the bulk load, never before.
- The index set is driven by the benchmark queries: `coupons(au_wtpct_mean)`,
`tracks(run_id)`, `tracks(coupon_id)`, `runs(environment)`, composite
`tracks(environment, load_mn)`; PostgreSQL adds BRIN on
`friction_cycles(cycle)` per partition.
- Naming: `ux_<table>_<cols>` for unique, `ix_<table>_<cols>` for
non-unique.
- Do not over-index: an index no query uses is disk footprint that skews
the storage comparison AND load-time cost. Every index must name the
query it serves (a one-line comment in the DDL).
## 4. Loading
- **SQLite:** during load `PRAGMA journal_mode=OFF, synchronous=OFF,
cache_size=-262144`; insert via `executemany` in transactions of 50k
rows; after load `PRAGMA journal_mode=WAL`, `ANALYZE`, `VACUUM`.
- **PostgreSQL:** load via `COPY FROM STDIN` (psycopg3 `copy`), never
row-by-row INSERT; `friction_cycles` and `friction_loop_points` are
declaratively RANGE-partitioned by `track_id` (12 partitions of 120
tracks); `VACUUM ANALYZE` after load.
- Loads are all-or-nothing per table and validated against `MANIFEST.csv`
row counts - see [code-error-handling.md](code-error-handling.md)
section 3 and [test-pipeline-validation.md](test-pipeline-validation.md).
- On every SQLite connection that relies on FK behaviour:
`PRAGMA foreign_keys = ON` (SQLite silently ignores FK clauses without
it). Bulk tables MAY declare FK columns without enforced constraints for
load speed - row-count validation is the integrity gate; whichever choice
is made must be the same in both engines.
## 5. `track_summary` - precomputed, and off-limits to Q5
Both engines carry a `track_summary` (plain table computed during load in
SQLite; materialized view in PostgreSQL) with per-track derived values
(`cof_ss_mean`, `run_in_cycles`, ...). It exists for Q2/Q3/Q4/Q6. **Q5 must
never read it** - Q5 is the forced full-scan benchmark over raw
`friction_cycles` in every format (spec
[08](../specs/08_benchmark_queries.md)).
## 6. Engine parity
- Same logical schema, same table names, same column names, same row
counts in both engines.
- Q1-Q7 executed against SQLite and PostgreSQL MUST return identical
result sets (checksum parity with 1e-9 float tolerance - the canonical
expected checksums are computed from PostgreSQL and cross-validated
against SQLite; see
[test-pipeline-validation.md](test-pipeline-validation.md)).
- A schema change in one engine is a schema change in both, in the same
commit.
## 7. Anti-patterns
- **Bare `id` PK, TEXT PK, or a GUID column** - this is a load-once
analytical schema; surrogate integer `<singular>_id` on dimensions,
natural composites on bulk tables.
- **TEXT FK in a bulk table** (`friction_cycles.track_code`) - integer FKs
only.
- **A `bigserial` surrogate on `friction_cycles` / `friction_loop_points`**
- dead weight at 1.4M / 2.9M rows.
- **Creating indexes before the bulk load** - doubles load time for
nothing.
- **An index without a named query** - footprint that skews the format
comparison.
- **Rewriting Q5 to use `track_summary`** - breaks the benchmark's
full-scan contract.
- **`updated_at` / audit columns / triggers** - this is a read-only
benchmark target rebuilt from CSV; it has no mutation lifecycle.
- **Divergent schemas between the engines** - parity is what makes the
benchmark comparison valid.

59
docs/rules/meta-concise-answers.md Normal file
View File

@@ -0,0 +1,59 @@
# Meta / Concise answers
**Binding on every response.** Match length to the question. Stop after the answer.
## Rule
Do not append: background, rationale, anticipated follow-ups, recap, summary,
TL;DR, "why this matters", offers to do more. If the user said "concise",
"short", "one line", or "just" - obey literally.
## Length budget
| Question shape | Max response |
|---|---|
| "What is X?" / "Where is Y?" / yes-no | 1 line |
| "How do I do X?" | 1-3 lines, commands only |
| "How do I test X?" | 3-5 lines, commands + check |
| "Show me X" (file, diagram, output) | The thing itself, no preamble |
| "Compare A and B" | 1 short table or 2-3 bullets/side |
| Open-ended architecture / discussion | Full prose acceptable |
Diagrams / lists / tables: **8-line ceiling** unless user asks for more.
No ASCII box-drawing unless asked. When the user says "I don't care about X" -
omit X, do not list it "for completeness".
## Examples
**Bad** - user asks "what does this option do?", assistant returns 6-option
table with pros/cons. **Good** - one sentence about the one option.
**Bad** - user asks "show ER diagram, no fields", assistant returns 80-line
ASCII with cardinality table and "what changes" section. **Good** - 4-5
lines of `entity --< child` arrows for the asked entities only.
**Bad** - yes/no question answered with a paragraph of nuance before the
yes/no. **Good** - yes/no first, nuance only if asked.
## Exceptions (these ARE the answer, no length cap)
File contents requested, code requested, test/log output, final summary
after multi-step tool work, plans the user asked to be detailed.
## Self-check before sending
- Did the user ask a yes/no? -> answer yes/no in the first line, nuance
only after, only if useful.
- Did the user ask for one thing? -> deliver that one thing only.
- About to write more than ~10 lines for a question that needs ~5? -> cut
everything past line 10 unless it is code, output, or a file path.
- Did the user say "concise" / "short" / "one line" / "just"? -> obey
literally: one sentence or one command/path.
## Recovery
If the user complains the output is too long:
1. Acknowledge in one line.
2. Re-answer the original question in one line.
3. Capture it as an anti-pattern in the Examples section above.

33
docs/rules/meta-concise-questions.md Normal file
View File

@@ -0,0 +1,33 @@
# Ask extremely short, concise questions
Binding on EVERY question put to the user.
## Rules
- **One decision per question.** Do not bundle unrelated questions.
- **Minimal context.** State only what is needed to answer: the situation in one
short clause, the concrete options, and what the answer changes. Nothing else.
- **No preamble.** Do NOT precede a question with analysis, restated reasoning,
background, evidence, or a summary of what you just did. The user has that
context; if a fact is truly needed to decide, compress it to one clause.
- **Prefer the smallest form.** A yes/no, or a 2-3 option pick. Put the question
itself first or make it unmissable.
- **Self-contained but short.** The options must be understandable without opening
another document, but spell them out in a few words, not paragraphs.
- **No useless information.** Every word must help the user decide. Delete anything
that does not.
## Anti-patterns
- A multi-paragraph lead-in before the actual question.
- Re-explaining the problem the user already stated.
- Listing options with a paragraph of trade-offs each.
- Asking two or more separate decisions in one run of prose.
## Shape to aim for
```
<one-clause situation>. <Question>?
1. <option> - <few words>
2. <option> - <few words>
```

62
docs/rules/meta-cursor-rules.md Normal file
View File

@@ -0,0 +1,62 @@
# Meta / Cursor rule artifact - commit-message generation
**Binding** for keeping Cursor's commit-message generation aligned with this
project's commit convention.
The development environment is the Cursor IDE with the Claude Code extension.
Claude Code reads the root `CLAUDE.md` and `docs/rules/` directly and needs no
`.cursorrules`. Cursor's OWN built-in AI (Chat/Composer and the SCM "sparkle"
commit-message generator) does not read those files - it reads `.cursorrules`.
Since the user creates all commits themselves (see
[`meta-git-commits.md`](meta-git-commits.md)) and may use the one-click SCM
generator to draft the message, `.cursorrules` must exist and must encode the
commit convention.
This project ships a SINGLE Cursor rules file: the root
[`.cursorrules`](../../.cursorrules). There is no `.cursor/rules/*.mdc`
directory here, and none should be added. All Cursor rule content lives in
`.cursorrules`.
---
## 1. The requirement
`.cursorrules` MUST encode this project's commit-message convention so that
Cursor's generator emits conforming messages. The authoritative convention
lives in [`meta-git-commits.md`](meta-git-commits.md); `.cursorrules` mirrors
it as plain-text instructions to the generator.
The failure mode this rule prevents: a missing or stale `.cursorrules`, so the
one-click SCM generator falls back to Cursor defaults and emits unprefixed junk
(`Update files`, `Improve project`).
## 2. What `.cursorrules` must encode
1. **First-line format** - `type(scope): description`, lowercase imperative
summary in English, under 72 chars, no trailing period.
2. **Allowed types (closed list)** - `feat fix perf refactor security docs ci
chore test`.
3. **Allowed scopes (closed list)** - `specs rules docs config datagen convert
bench report tools all`, with the area mapping from
[`meta-git-commits.md`](meta-git-commits.md).
4. **Coverage rule** - the subject is only a title; multi-area diffs need a
blank line and a 2-6 bullet body covering every meaningful changed area; use
`all` when no narrower scope honestly fits; split unrelated work.
5. **Anti-examples** - forbid vague/unprefixed lines (`Update files`, `Improve
project`, `clarify rules`).
6. **Allowed special forms** - `Merge ...` commits are allowed as-is;
`release(scope): vX.Y.Z` is allowed for releases.
## 3. Keeping the mirror in sync
No commit-msg hook is installed in this repository; the convention is enforced
by review. `.cursorrules` and [`meta-git-commits.md`](meta-git-commits.md)
must agree: same type list, same scope list, same first-line format. If the
convention changes, update both files in the same change set.
## 4. Scope boundary
This rule governs the commit-message *convention materialization* - not who may
create commits. Commit authorship is governed by
[`meta-git-commits.md`](meta-git-commits.md): rules files describe the
convention; only the user creates actual commits.

99
docs/rules/meta-git-commits.md Normal file
View File

@@ -0,0 +1,99 @@
# Meta / Git commits - never on behalf of the user
**Binding** for every contributor (human or AI agent) touching this
repository. Applies to **all** git history-modifying operations.
---
## 1. The rule
**Never create a git commit on behalf of the user. Ever.** The user is the
only entity allowed to materialize a commit in this repo.
This covers, without exception:
- `git commit` (with or without `-a`, `-m`, `--amend`)
- `git merge` - never finalize one
- `git rebase` (interactive or otherwise) - never start one
- `git revert` - never run; the user decides whether a change is undone
- `git cherry-pick`
- `git stash`
- `git tag` (annotated or lightweight)
- `git push` of any kind - pushing is the user's call
- `git reset` / `git checkout` that changes the working tree or refs
- Any wrapper script or `gh` API call that would create a commit / tag /
merge on the user's behalf
For an AI agent, git is **read-only**: reading history (`git log`,
`git diff`, `git status`, `git show`) and inspecting branches is fine;
anything that writes to the index, refs, or `objects/` - including
`git add` - is left to the user. (This matches the machine-level
enforcement: git write commands by the agent are denied by a hook.)
## 2. Why
- Commit authorship and message wording are decisions the user owns - they are
durable, public, and signed against the user's name and email.
- "Tiny" commits that look harmless ("fix typo", "delete unused file") still
rewrite history and cannot be quietly reversed once pushed.
- Generated data and measured results in this project take hours to rebuild;
what enters history and when is a conscious decision, not a side effect.
## 3. What to do instead
- Make the working-tree changes.
- Report what changed and offer a draft commit message **as plain text** in the
conversation, together with the exact `git add` / `git commit` commands for
the user to run themselves.
## 4. Commit-message format
When you draft a message, follow this project's convention:
- **First line** - `type(scope): description`, lowercase imperative summary in
English, under 72 chars, no trailing period.
- **Types (closed list)** - `feat fix perf refactor security docs ci chore
test`.
- **Scopes (closed list, owner-approved 2026-07-11)** - `specs rules docs
config datagen convert bench report tools all`. Pick by the area of the
changed files:
- `specs` = `docs/specs/**`
- `rules` = `docs/rules/**`
- `docs` = other prose docs (`README.md`, root `CLAUDE.md`,
`docs/examples/**`)
- `config` = configuration files (`.gitignore`, `.cursorrules`, editable
YAML defaults such as hardware prices)
- `datagen` = lab-configuration and data-generation code (tasks 01-03:
`lab_config` generation, process diagrams, `generate_data.py`)
- `convert` = format converters (tasks 04-07: JSON-LD, SQLite,
PostgreSQL, RDF)
- `bench` = benchmark queries, runner, and extrapolation (tasks 08-10)
- `report` = reporting code and templates (task 11)
- `tools` = helper scripts under `tools/`
- `all` = multiple areas or in doubt
- **Coverage** - multi-area diffs get a blank line and a 2-6 bullet body
covering every meaningful changed area; split unrelated work.
- **Special forms** - `Merge ...` and `release(scope): vX.Y.Z` are allowed
as-is.
No `commit-msg` hook is installed in this repository yet; the convention is
enforced by review. If a hook is added later, it must implement exactly this
format and this file must be updated in the same change. The same convention
is mirrored into `.cursorrules` for Cursor's built-in commit-message
generator - see [meta-cursor-rules.md](meta-cursor-rules.md).
## 5. Anti-patterns
- **"I'll just commit it so we have a checkpoint."** No. Working-tree state is
the checkpoint; the user decides when it becomes history.
- **Amending the user's last commit to fix something you noticed.** No. Surface
the issue, let the user choose between amend and new commit.
- **Suggesting `git add -A` in the draft commands.** List explicitly named
files, so the user sees exactly what would land.
- **Vague subjects** (`Update files`, `Improve project`, `clarify rules`).
Say what changed and where.
## 6. Exception (closed list)
There are no exceptions. If a workflow seems to require a commit, pause,
explain the dependency, and let the user run the git command.

35
docs/rules/meta-language-style.md Normal file
View File

@@ -0,0 +1,35 @@
# Meta / Communication language and register
**Binding on every response and on all prose directed at the user**
(explanations, status updates, questions, summaries, commit messages, PR text).
## Rule
1. **Language: Russian.** All communication with the user is in standard
literary Russian.
2. **Register: neutral and professional. No jargon, no slang, no profanity.**
- No profanity or vulgar words - in any language, under any circumstances.
- No slang or jargon: street slang, gamer/internet slang, or untranslated
English buzzwords used as slang.
- Prefer the established Russian word over a transliterated English one
whenever a normal Russian equivalent exists.
- Plain, grammatical, complete sentences.
## Allowed terminology (not slang)
Field-standard technical terms are fine - they are terminology, not jargon:
e.g. «коммит», «билд», «дебаг», «релиз», «мерж». When unsure whether a word
is accepted terminology or slang, choose the neutral Russian equivalent.
## Scope
Does **not** apply to source code, identifiers, file paths, code comments,
shell/tool output, or verbatim quotes of errors and logs - those keep the
language of the codebase.
## Canonical source
This file is the project-level binding rule. It restates the project owner's
standing preference (communication in Russian; code, docs, and artifacts in
English). If a global user-level rule ever diverges from this file, the
stricter wording wins.

51
docs/rules/meta-no-assumptions.md Normal file
View File

@@ -0,0 +1,51 @@
# Meta / No assumptions
**Binding** on every interaction. Strictest when the cost of a wrong
guess is high: deleting or regenerating `./out/` artifacts, dropping or
reloading the `tribo` PostgreSQL database, changing the random seed or
`lab_config.yaml`, editing specs in ways that invalidate generated data,
git history, anything with data-loss consequences.
## Rule
Before producing a plan or change:
1. Re-read the user's most recent message. Watch for enumerations
("both", "and also", "in addition to"), explicit purpose statements
("X for A, Y for B"), and distinct entities that sound similar.
2. Don't collapse two distinct entities into one. "X for A and Y for B"
means **two** setups, not one.
3. If anything is ambiguous after careful re-reading - **ask**. Don't
pick the most likely interpretation and proceed.
4. Echo back the parsed interpretation before producing the plan, so a
misread is cheap to correct.
## When the rule is most strict
Strictness scales with the cost of a wrong guess:
| Domain | Why a wrong assumption is expensive |
|---|---|
| **Deleting / regenerating `./out/`** | The artifact tree holds hours of generation, conversion, and benchmark work. Regenerating CSV invalidates every downstream format, benchmark result, and report. Confirm which subtree is affected before deleting or overwriting. |
| **The `tribo` PostgreSQL database** | Dropping or reloading it discards a long `COPY` load and invalidates benchmark results derived from it. Confirm before any `DROP` / `TRUNCATE` / reload. |
| **The random seed / `lab_config.yaml`** | Seed 20260711 and the lab configuration are the root of the whole reproducibility chain. Changing either silently invalidates all generated data, all conversions, all checksums, and all measured results. Never change them as a side effect of another task. |
| **Editing specs (`docs/specs/`)** | Specs are the task contract. A "small clarification" that changes volumes, schemas, or file layouts desynchronizes already-generated artifacts. Confirm intent when an edit is more than wording. |
| **Benchmark results (`./out/bench/`)** | Measured results cannot be re-derived without re-running the full matrix (hours). Never overwrite `results_raw.csv` / `results_median.csv` without confirmation. |
| **git history** | A wrong rebase/reset/force-push rewrites shared history; recovery is manual and error-prone. (The AI agent never runs git write operations at all - see [meta-git-commits.md](meta-git-commits.md).) |
For all of the above: **the cost of one extra clarifying question is
zero; the cost of a wrong assumption is hours-to-days** of recovery.
Always ask.
## Forms a clarifying question takes
- "Confirm: you want X for A AND Y for B (two setups), not just one?"
- "Two readings: (a) ..., (b) ... - which?"
- "What does '<word>' refer to here - (a) ... or (b) ...?"
## When the assumption was already wrong
1. Acknowledge the misread explicitly. Quote the user's words back.
2. Discard the wrong artifact entirely; don't repair piecemeal unless
the user asks.
3. Restart with the corrected interpretation, echo it first.

119
docs/rules/meta-rules-naming.md Normal file
View File

@@ -0,0 +1,119 @@
# Meta / Rule file naming
**Binding** for files inside [`docs/rules/`](docs/rules/). Adding, renaming, or
splitting a rule file follows this document.
## Layout
- `docs/rules/` - binding rules (this folder): reproducible conventions on HOW
work is done. Reference content that supports a rule also lives here,
named `<rule>-reference.md`.
- `docs/specs/` - the pipeline task specifications (WHAT to build: tasks
01-11, their inputs, outputs, and dependencies). Binding for scope and
deliverables, but conventions that span tasks are codified in `docs/rules/`.
- `docs/examples/` - imported reference materials (master plan, ontology).
Descriptive, not binding.
Files live flat at the top level of `docs/rules/`. No subfolders.
## Naming
```
<domain>-<subdomain>-<topic>[-<aspect>].md
```
- Lowercase, dashes between segments. No underscores, no CamelCase.
- Max 4 segments. Most general first.
- Reading the filename should answer: category -> subcategory -> topic.
## Approved domain prefixes
| Prefix | Covers |
|---|---|
| `bench-` | Benchmark execution, resource measurement, extrapolation, reporting methodology |
| `build-` | Pipeline task orchestration: the `./out/` tree, `.done` markers, task order |
| `code-` | General code style across languages (Python, SQL, PowerShell): comments, formatting, error handling, configuration |
| `data-` | The simulated dataset: determinism, identifiers, units, column naming |
| `db-` | SQLite / PostgreSQL benchmark target schemas, loading, indexing |
| `meta-` | Rules about the rules system and working conventions |
| `obs-` | Observability and logging |
| `test-` | Validation gates, checksums, pytest conventions |
Don't invent a new prefix without adding it to this table in the same
commit.
### Subdomains (second / third segment)
Within a domain, the second segment is a recognised tool, subsystem, or
concern; the third is the concrete topic. Established families:
- `db-sql-*` - relational benchmark targets (SQLite, PostgreSQL): schema,
loading, indexing.
- `data-*` - dataset conventions (`data-determinism`, `data-naming-units`).
- `bench-*` - measurement methodology (`bench-methodology`).
- `build-pipeline-*` - task orchestration and artifact layout.
- `code-*` - general code style (comments, formatting, error handling,
config, Python style).
- `meta-*` - rules about the rules system and working conventions.
- `obs-*` - observability and logging (`obs-python-logging`).
- `test-*` - validation and testing (`test-pipeline-validation`).
## Examples
| Good | Reads as |
|---|---|
| `data-determinism.md` | Data -> deterministic generation |
| `db-sql-schema.md` | DB -> SQL -> schema conventions |
| `bench-methodology.md` | Benchmarks -> measurement methodology |
| `code-python-style.md` | Code -> Python -> style |
| `obs-python-logging.md` | Observability -> Python logging |
| `meta-rules-naming.md` | Meta -> rules system -> naming |
| Bad | Why |
|---|---|
| `BenchMethodology.md` | CamelCase forbidden |
| `data_determinism.md` | Underscores forbidden |
| `sqlite-loading-conventions.md` | Use `db-` prefix, drop redundant `conventions` |
| `bench-execution-measurement-rules-doc.md` | 5+ segments + redundant `rules` suffix |
## Adding a rule
1. Pick the domain prefix; add new ones to the table above first.
2. Check `ls docs/rules/` - extend a sibling rule before creating a new one.
3. Create the file with binding content, examples, and exceptions.
4. Add a row for the new file to the rule-trigger table in the root
[`CLAUDE.md`](../../CLAUDE.md) in the same commit.
5. If replacing an older doc, `git mv` to preserve history and `Grep` the
repo for the old path.
## Rules vs specs vs memory files
Three homes for a statement; pick by its kind:
| Source | Stores |
|---|---|
| `docs/rules/<name>.md` | A reproducible, mandatory behaviour - the same answer for every contributor on every task within the topic. |
| `docs/specs/NN_<task>.md` | The definition of one pipeline task: scope, inputs, outputs, dependencies. |
| `memory/<name>.md` | A fact about *this user*, *this project*, or *the current moment* - preferences, ongoing initiatives, locked decisions, infra specifics. |
- A convention that must hold across several tasks -> `docs/rules/`.
- A deliverable of one task -> its spec in `docs/specs/`.
- A fact about the user or the present state of things -> `memory/`.
A rule file may reference a spec or a memory note for context, but the
binding convention lives in the rule file. When a rule and a spec disagree
on a convention (naming, formats, thresholds), the rule wins and the spec
is updated to match.
## Anti-patterns
- **Duplicating rule content across rule files** - pick one canonical
location and cross-link from siblings instead of copying.
- **Restating a whole spec inside a rule** - the rule codifies the
reusable convention and links the spec for task detail.
- **Adding rules to the body of CLAUDE.md** instead of creating a rule
file - CLAUDE.md carries only the trigger table, the project preamble,
and the canonical-source-of-truth note.
- **Naming a file after the implementation** instead of the domain
(`oxigraph-loading.md` vs `db-rdf-loading.md`) - the implementation may
be replaced; the topic endures.

113
docs/rules/obs-python-logging.md Normal file
View File

@@ -0,0 +1,113 @@
# Observability / Python logging - standard library only, namespaced, plain text
**Purpose + scope.** Every Python module in this repository (data generation,
converters, benchmark runner, extrapolation, reporting, helper tools) MUST
use the standard-library `logging` module following these conventions. No
external logging dependency.
---
## 1. The rule
1. **Standard library `logging` only.** No `loguru`, no `structlog`, no
`python-json-logger`. Stdlib is sufficient and keeps the dependency list
closed (see [code-python-style.md](code-python-style.md)).
2. **Namespaced loggers.** Every module obtains its logger via
`logging.getLogger(__name__)`. Never the root logger directly.
3. **`print()` is for the user-facing CLI protocol only.** The specs
explicitly require certain stdout output: progress summaries, the real
generated tree size (task 03), per-cell benchmark progress, final
counts. That protocol output goes through `print()`. Everything
diagnostic - internal flow, warnings, errors, per-row detail - goes
through a logger.
4. **Log level from a CLI flag.** Every pipeline script accepts
`--log-level {DEBUG,INFO,WARNING,ERROR}`, default `INFO`. There is no
settings table and no `.env` in this project (see
[code-config-yaml.md](code-config-yaml.md)).
5. **Plain-text format on stderr.** These are one-shot CLI scripts, not
services; JSON logs are overkill. One line per record via
`logging.basicConfig`. Keeping the logger on stderr leaves stdout clean
for the protocol output of item 3.
6. **Never log secrets.** The only credential material in this project is
the PostgreSQL DSN - never log it (a DSN may embed a password). Log the
database name or host, never the full connection string.
## 2. API by severity
| Method | Use for |
|---|---|
| `logger.critical(...)` | The process cannot continue - before `sys.exit(1)` or an unrecoverable state. |
| `logger.error(...)` | Failed operation, exception caught and handled, broken invariant, a benchmark cell recorded as TIMEOUT/ERROR. Always include `exc_info=True` when an exception is in scope. |
| `logger.warning(...)` | Recoverable problem, retry triggered, suspicious state (e.g. a value near a validation bound), an optional step skipped (e.g. cold-cache pass unavailable). |
| `logger.info(...)` | Normal flow milestones - task start/finish, per-batch progress, row counts written, validation gates passed. |
| `logger.debug(...)` | Verbose tracing - per-coupon detail, generated value samples, SQL statements, per-run raw measurements. Hidden at default `INFO`. |
**Default level: `INFO`.** `DEBUG` is opt-in via `--log-level DEBUG` for a
diagnosis session.
## 3. Logger acquisition and setup
```python
# Top of every module that logs:
import logging
logger = logging.getLogger(__name__)
# Once, in each script's main():
logging.basicConfig(
level=args.log_level,
stream=sys.stderr,
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
```
Inside functions:
```python
def load_table(conn, name: str, rows) -> int:
logger.info("loading %s", name)
try:
...
except psycopg.Error:
logger.error("loading %s failed", name, exc_info=True)
raise
```
`__name__` gives the module path as the namespace tree used for filtering.
Never hardcode a logger name string. Clamp noisy third-party libraries
(`psycopg`, `matplotlib`) to `WARNING` in the same setup block if they
pollute the output.
## 4. Progress reporting in long loops
Generation and loading loops run over millions of rows. Report progress at
INFO on a coarse cadence (per batch / per wafer / per table, or every N
seconds), never per row. Per-row output belongs at DEBUG, and even there
prefer sampled output ("first row of each file") over full dumps.
## 5. Anti-patterns
- **`print()` for diagnostics** - forbidden; `print()` is only the
user-facing CLI protocol (progress summaries, final counts, sizes).
- **Diagnostics on stdout** - stdout is the protocol channel; the logger
writes to stderr.
- **Root logger calls** (`logging.info(...)`) - forbidden; always
`logger = logging.getLogger(__name__)`.
- **Third-party logging frameworks** (`loguru`, `structlog`) - forbidden.
- **Hardcoded logger names** (`logging.getLogger("datagen")`) - forbidden;
pass `__name__`.
- **Logging the PostgreSQL DSN / connection string** - forbidden; log
host/database name only.
- **Log level from an environment variable or a config file** - the level
is a per-run diagnostic choice; it comes from `--log-level`.
- **`logger.error(...)` without re-raise** when the function semantically
cannot proceed - see [code-error-handling.md](code-error-handling.md).
- **Per-row INFO logging in bulk loops** - drowns the signal and slows the
run; use coarse milestones.
## 6. Related documents
- [code-error-handling.md](code-error-handling.md) - `exc_info=True`
requirement and the swallow ladder.
- [code-config-yaml.md](code-config-yaml.md) - configuration scope (no
`.env`, no settings table).
- [code-python-style.md](code-python-style.md) - the closed dependency list.

125
docs/rules/test-pipeline-validation.md Normal file
View File

@@ -0,0 +1,125 @@
# Test / Pipeline validation - markers, manifests, checksums
**Purpose + scope.** The validation contract between pipeline tasks: how a
task proves it completed correctly and how downstream tasks verify their
inputs before trusting them. Binding for every task implementation (01-11)
and for any pytest suite added to this repository.
The pipeline has no human QA step between tasks - these gates are the only
thing standing between a silent generation/conversion bug and a report full
of wrong numbers.
---
## 1. Completion markers - written last, verified first
- Every task writes `./out/.done/<NN>.ok` **only after** all its outputs
are complete and its own validation passed. The marker contains the
task's headline counts (rows, bytes, triples) for downstream checks.
- A task MUST verify the `.done` markers of its declared dependencies
before starting, and abort with a clear message if one is missing.
- A task that re-runs deletes its own marker first and rewrites it at the
end. A marker is never written on failure and never from a `finally`
block (see [code-error-handling.md](code-error-handling.md)).
## 2. The CSV corpus manifest - the root of trust
- Task 03 writes `./out/csv/MANIFEST.csv`: one row per generated file with
path, row count, byte size, sha256.
- Every converter (04-07) validates what it read against the manifest: row
counts per source file, and sha256 where the file is consumed whole. A
mismatch is an abort, not a warning.
- The manifest is also the reproducibility proof: regenerating with the
same `lab_config.yaml` must reproduce identical sha256 values (see
[data-determinism.md](data-determinism.md)).
## 3. Canonical expected volumes
The lab configuration fixes the corpus size exactly; validation asserts
these numbers, not "roughly that many":
| Entity | Count |
|---|---|
| batches | 4 |
| wafers | 12 |
| coupons | 588 (480 friction + 108 characterization-only) |
| tracks | 1,440 |
| friction cycle rows | 1,440,000 |
| friction loop points | 2,880,000 |
| XRF points | 235,200 (588 x 400) |
| generated tree size | 120-200 MB (hard abort bounds) |
If a deliberate `lab_config.yaml` change moves these numbers, this table
and the validation code change in the same commit.
## 4. Query-result checksums - all formats agree
- Each benchmark query Q1-Q7 has ONE canonical expected result: rows
sorted, floats compared with tolerance 1e-9, checksummed to
`./out/bench/expected/q<N>.sha256`.
- The canonical checksums are computed from **PostgreSQL** and
cross-validated against **SQLite** before any benchmarking starts; a
disagreement between the two engines is a converter or query bug and
blocks task 09.
- Every measured benchmark run records a `checksum_ok` flag; a cell whose
result does not match the expected checksum is an invalid measurement
regardless of how fast it ran.
## 5. Format smoke tests
Each converter proves basic well-formedness of its output before writing
its marker:
- **JSON-LD (04):** one sample coupon file passes `json.load` AND an
rdflib JSON-LD parse.
- **SQLite (05) / PostgreSQL (06):** row counts per table match the
manifest-derived expectations; `track_summary` row count equals 1,440.
- **RDF (07):** the loaded store answers a SPARQL `COUNT` matching the
serialized triple count recorded in the `.done` marker.
## 6. If a pytest suite is added
The specs define no pytest suite; validation is built into the tasks. If
unit/integration tests are added later:
- **Skip, don't fail, when PostgreSQL is unreachable.** A checkout without
a running PostgreSQL still gets a green (skipped) run;
connection failure in a fixture is `pytest.skip(...)`, not an error.
- **Never mutate real `./out/` artifacts.** Tests that need data build a
small corpus into a temp directory (a reduced `lab_config` is fine) or
copy the artifact; hours of generated/measured data are not test
fixtures to be overwritten.
- **Ephemeral databases only.** A test touching PostgreSQL creates and
drops its own database (e.g. `tribo_test`) next to `tribo`; it never
writes to `tribo` itself.
- **Independent and order-free.** Every test passes run alone; no residue
between tests, no execution-order dependencies.
- **Deterministic assertions.** With the fixed seed, exact counts and
exact values are assertable - prefer exact assertions over "greater than
zero".
## 7. Anti-patterns
- **Writing a `.done` marker before validation, on failure, or in
`finally`** - the marker means "completed and verified".
- **Skipping manifest validation "because generation just ran"** - the
gate exists precisely for the runs where the assumption is wrong.
- **Warning-and-continuing on a count mismatch** - a converter with a row
deficit produces a database that LOOKS complete; abort.
- **Regenerating expected checksums to make a failing format pass** - the
canonical result comes from PostgreSQL cross-validated with SQLite;
a mismatch in another format is a bug in that converter or query.
- **Tests writing into `./out/`** or connecting to the real `tribo`
database.
- **Failing (not skipping) when PostgreSQL is unavailable** in a pytest
run.
## 8. Related documents
- [build-pipeline-tasks.md](build-pipeline-tasks.md) - marker layout and
task order.
- [data-determinism.md](data-determinism.md) - why byte-identical
regeneration is assertable.
- [code-error-handling.md](code-error-handling.md) - fail-loud mechanics.
- [bench-methodology.md](bench-methodology.md) - how `checksum_ok` enters
the measurement protocol.

50
docs/specs/00_PLAN.md Normal file
View File

@@ -0,0 +1,50 @@
# 00_PLAN - Tribology Lab Data Storage Evaluation Pipeline
## Goal
Build a complete, reproducible pipeline that:
1. Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow.
2. Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format).
3. Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
4. Benchmarks 7 laboratory-standard retrieval scenarios (Q1-Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
5. Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
6. Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange).
## Execution environment
- Claude Code agent inside Cursor, local filesystem access, on-premise only (no cloud services).
- Python 3.11+ (tabs for indentation), PostgreSQL 16 local instance, SQLite via stdlib `sqlite3`, `rdflib` for RDF, optional local triplestore (Jena Fuseki / GraphDB Free) - if unavailable, use `rdflib` + on-disk `oxigraph` (pip) as the queryable store.
- All artifacts written under `./out/` relative to repo root:
- `./out/csv/` `./out/json/` `./out/sqlite/` `./out/pg/` (dumps) `./out/rdf/` `./out/bench/` `./out/report/`
## Task files and dependencies
```mermaid
flowchart TD
T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
T01 --> T03[03_csv_data_generation]
T03 --> T04[04_convert_json]
T03 --> T05[05_convert_sqlite]
T03 --> T06[06_convert_postgresql]
T03 --> T07[07_convert_rdf]
T04 --> T08[08_benchmark_queries]
T05 --> T08
T06 --> T08
T07 --> T08
T03 --> T08
T08 --> T09[09_benchmark_execution]
T09 --> T10[10_extrapolation_model]
T09 --> T11[11_reporting]
T10 --> T11
```
## Parallelism
- T04-T07 are independent; run in parallel after T03.
- T02 is independent of data generation; can run any time after T01.
## Execution order
01 → 02 → 03 → (04 | 05 | 06 | 07) → 08 → 09 → 10 → 11
## Global conventions
- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`, `track:B721-W1-C07-T2`, run IDs `run:R1..R4`.
- All timestamps ISO-8601 UTC. All units SI-explicit in column names (`_mN`, `_um`, `_GPa`, `_wtpct`, `_nm`).
- Random seed fixed at 20260711 for full reproducibility.
- Python indentation: tabs.
- Every task writes a completion marker `./out/.done/<task>.ok` containing row/byte counts for downstream validation.

56
docs/specs/01_lab_configuration.md Normal file
View File

@@ -0,0 +1,56 @@
# 01 - Laboratory Configuration
## Context / Goal
Define the authoritative laboratory model that all downstream tasks (data generation, ontology, schemas) must follow. This file is a specification, not code; produce `./out/config/lab_config.yaml` capturing everything below.
## Dependencies
None (root task).
## Material system
- Substrate: Ti-6Al-4V coupons, 10 × 10 × 3 mm.
- Adhesion layer: Cr (sputtered).
- Functional coating: Pt-Au deposited in a **composition gradient** across each wafer; film thickness range 0.3-1.1 µm.
## Instruments
| Role | Instrument | Data produced |
|---|---|---|
| Friction | RAPID custom high-throughput parallelized 6-probe tribometer | COF vs cycle per track; test conditions |
| Nanoindentation | Bruker TI980 TriboIndenter | Hardness, reduced modulus (25 indents/coupon) |
| Composition | Bruker M4 Tornado micro-XRF (assumed model) | Pt/Au wt% map per coupon (20×20 grid) |
| Deposition | Kurt J. Lesker PVD 200 sputter-down | Batch deposition parameters |
| Surface roughness | AFM | Topography image metadata + Ra/Rq per coupon |
| Film thickness | Optical profilometry | Thickness map per coupon (10×10 grid) |
| Simulation | SIMTRA (sputter transport Monte-Carlo) | Deposition atom-energy / composition profiles per deposition run |
## Deposition matrix (4 batches)
| Batch | Pt:Au gun tilt | Pt power | Au power | Pt discharge | Au discharge |
|---|---|---|---|---|---|
| B721 | 20°:0° | 150 W | 50 W | 432 V | 311 V |
| B722 | 20°:20° | 100 W | 100 W | 399 V | 352 V |
| B723 | 20°:20° | 150 W | 50 W | 430 V | 311 V |
| B724 | 0°:20° | 50 W | 150 W | 376 V | 340 V |
## Hierarchy and volumes
```
batch (×4) → wafer (×3 per batch) → coupon (×49 per wafer, 7×7 grid) = 588 coupons
```
- Friction testing: 480 coupons (4 RAPID runs × 120 coupons: 5 plates × 6 probes × 4 coupons/square).
- Reserve/QA: 108 coupons (no friction data; characterization only).
- Environments: run R1,R2 = Lab Air (µ_normal); run R3,R4 = Dry N2 (µ_dry_nit).
- Tracks: 3 replicate tracks per friction coupon, identical conditions, fresh counterface ball per track (3 balls per holder) → 1,440 tracks.
- Cycles: 1,000 reciprocating cycles per track → 1,440,000 cycle records.
- Friction loops: every 100th cycle stores a position-resolved loop of 200 points → 1,440 × 10 × 200 = 2,880,000 loop points.
- Nominal test conditions: normal load 100 mN, stroke 1 mm, sliding speed 1 mm/s, ruby (Al2O3) ball Ø 3.175 mm; RH 45% (Lab Air) / 2% (Dry N2); T 23 ± 1 °C.
## Physical models for value simulation (used in task 03)
- Coupon Au wt%: linear gradient across wafer x-position, batch-dependent center: B721 8%, B722 25%, B723 10%, B724 60%; ±5 wt% span across wafer; XRF grid adds N(0, 0.3) noise.
- COF run-in: `cof(c) = cof_ss + (cof_0 - cof_ss)·exp(-c/τ) + N(0, σ)`; cof_0 ∈ [0.35, 0.5]; τ ∈ [100, 400] cycles; σ = 0.01.
- Steady-state COF depends on Au% and environment: Lab Air `cof_ss = 0.32 - 0.0015·Au%`; Dry N2 `cof_ss = 0.24 - 0.0012·Au%`; clamp ≥ 0.12.
- Hardness: `H = 8.5 - 0.05·Au% + N(0, 0.25)` GPa; reduced modulus `Er = 190 - 0.6·Au% + N(0, 4)` GPa.
- Roughness Ra: log-normal, median 5 nm, σ_log 0.3.
- Thickness: 0.3-1.1 µm, radial wafer profile + noise.
- Wear volume per track: Archard-like `V = k·F·s`, k depends on Au% and environment; store as track-level derived measurement.
## Output
- `./out/config/lab_config.yaml` - machine-readable version of everything above (volumes, matrices, model coefficients, seed).
- Marker `./out/.done/01.ok`.

48
docs/specs/02_process_flow_diagrams.md Normal file
View File

@@ -0,0 +1,48 @@
# 02 - Process Flow Diagrams
## Context / Goal
Generate the comprehensive Mermaid documentation of the laboratory workflow (reconstructed from the 10 Miro slides) as standalone `.md` / `.mermaid` artifacts for the final report and for human review.
## Dependencies
01 (uses volumes/terminology from lab_config.yaml).
## Task
Produce `./out/report/process_flow.md` containing the following diagrams, each also saved as a separate `.mermaid` file in `./out/report/diagrams/`.
### D1 - Coupon Assembly
```mermaid
flowchart LR
A([Ti-6Al-4V Base 10x10x3 mm]) --> B[Cleaning]
B --> B1([Rinsed in deionized water])
B1 --> B2([Sonicated in cleaning solution])
B --> C[Smearing Adhesive]
C --> C1([Cr adhesive coating])
C --> D[Vapor Deposition System<br/>Kurt J. Lesker PVD 200]
D --> D1([PtAu sputter coating, gradient])
D --> E([Test Coupon])
```
### D2 - Characterization & Batch Assembly
Test Coupon → SIMTRA (composition / atom energies) → Optical profilometry (film thickness 0.3-1.1 µm) → assemble Test Wafer (×49 coupons) → assemble Test Batch (×3 wafers) → batches 721-724 with the deposition parameter table.
### D3 - Testing tree
Friction (RAPID; Lab Air → µ_normal; Dry N2 → µ_dry_nit), Nanoindentation (TI980 → hardness, reduced modulus), AFM (topography → Ra), XRF (M4 Tornado → Pt/Au map).
### D4 - Tribometer session sequence (sequenceDiagram)
Request test → Get samples → Load samples → Load ball holders (3 counterfaces/holder) → Create Excel test plan (Sample Plate, Plate Location, Sample ID, Save Location, Folder Name, X/Y/Z Offset, Load, Iterations) → Transfer to tribometer → Set up platter → Activate software → Load plan → Activate tribometer → ... → Software stop → Save avg files → Remove plates/platters → Close software → Pull equipment out.
### D5 - Execution loop (flowchart)
```
Load all counter-faces
FOR EACH of 5 plates
FOR EACH of 6 probes (parallel)
FOR EACH of 4 coupons on coupon square
3x per coupon: draw track -> rotate counter-face
```
### D6 - Data hierarchy ERD-style
batch → wafer → coupon → {xrf_map, nanoindentation, afm, profilometry} and coupon → track → {cycles, loops, wear}; deposition → simtra_profile.
## Output
- `./out/report/process_flow.md`, `./out/report/diagrams/D1..D6.mermaid`
- Marker `./out/.done/02.ok`.

46
docs/specs/03_csv_data_generation.md Normal file
View File

@@ -0,0 +1,46 @@
# 03 - Simulated CSV Data Generation (~150 MB)
## Context / Goal
Generate the canonical raw dataset as a hierarchical CSV file tree, exactly as instruments would write it. This corpus is the single source for all conversions and benchmarks. Target size: 140-170 MB.
## Dependencies
01 (`lab_config.yaml`).
## Input
`./out/config/lab_config.yaml` - volumes, deposition matrix, physical model coefficients, seed 20260711.
## Processing instructions
Write `generate_data.py` (tabs for indentation, pure stdlib + numpy). Stream rows; never hold the full cycle dataset in memory. Use `random.Random(seed)` / `numpy.random.default_rng(seed)` only.
### File tree to produce
```
out/csv/
├── batches.csv # 4 rows: deposition params
├── simtra/
│ └── simtra_profile_<batch>.csv # 4 files, 1000 rows each: angle, energy_eV, pt_flux, au_flux
├── batch_<B>/wafer_<W>/
│ ├── wafer_info.csv
│ └── coupon_<C>/
│ ├── coupon_info.csv # position on wafer, thickness_um, ra_nm, assigned run/plate/probe/square or RESERVE
│ ├── xrf_map.csv # 400 rows: grid_x, grid_y, pt_wtpct, au_wtpct
│ ├── profilometry.csv # 100 rows: grid_x, grid_y, thickness_um
│ ├── nanoindentation.csv # 25 rows: indent_id, x_um, y_um, hardness_GPa, reduced_modulus_GPa, max_load_mN
│ ├── afm.csv # 1 row: ra_nm, rq_nm, image_file (synthetic path)
│ └── track_<T>/ # only for 480 friction coupons, T in 1..3
│ ├── track_info.csv # run_id, environment, load_mN, stroke_mm, speed_mm_s, counterface_id, started_at
│ ├── cof_vs_cycle.csv # 1000 rows: cycle, cof
│ ├── friction_loops.csv # 2000 rows: cycle (every 100th), position_um (200 pts), friction_force_mN
│ └── wear.csv # 1 row: wear_volume_um3, k_archard, sliding_distance_m
└── runs.csv # 4 rows: run_id, environment, date, plates, operator
```
### Value generation
Apply exactly the physical models of task 01 (gradient Au%, run-in COF, hardness/modulus vs Au%, Archard wear). COF must correlate with the coupon's mean Au% and the run environment so that Q2/Q3/Q7 benchmarks return physically meaningful results.
### Size control
Expected: cycles 1.44 M rows (~30 MB) + loops 2.88 M rows (~95 MB) + XRF 235 K rows (~8 MB) + remainder (~10 MB) ≈ 150 MB. After generation, print actual tree size; abort with error if outside 120-200 MB.
## Output
- Full tree under `./out/csv/` (~150 MB).
- `./out/csv/MANIFEST.csv`: file path, rows, bytes, sha256 - used by conversion tasks for validation.
- Marker `./out/.done/03.ok` with total rows/bytes.

22
docs/specs/04_convert_json.md Normal file
View File

@@ -0,0 +1,22 @@
# 04 - Convert to JSON-LD
## Context / Goal
Produce the ontology-annotated JSON-LD representation of the corpus: the format evaluated for inter-lab data exchange (FAIR). Two variants: (a) full - bulk points embedded; (b) hybrid - metadata JSON-LD with `sourceFile` links to CSV.
## Dependencies
03 (`./out/csv/`, MANIFEST.csv).
## Input
CSV tree.
## Processing instructions
- `@context`: project vocab `https://sandia.gov/ontology/tribology#` + QUDT units + PROV-O provenance; predicates `partOf`, `cutFrom`, `performedOn`, `performedBy` as `@id` links.
- Entity classes: `Batch, Wafer, Coupon, Instrument, CompositionMeasurement, NanoindentationMeasurement, AFMMeasurement, ProfilometryMeasurement, FrictionTest (track), FrictionCycle, FrictionLoopPoint, WearMeasurement, SimtraProfile, Run`.
- Variant FULL: one JSON-LD file per coupon (embedding its tracks/cycles/loops), compact serialization (no pretty-print). Directory `./out/json/full/`.
- Variant HYBRID: single `./out/json/hybrid/dataset.jsonld` with all metadata entities; cycle/loop arrays replaced by `{"sourceFile": "<relative csv path>", "rows": N, "sha256": "..."}`.
- Stream-write; validate one sample file with `json.load` + rdflib JSON-LD parse.
## Output
- `./out/json/full/` (expect ~0.7-1.2 GB), `./out/json/hybrid/dataset.jsonld` (expect < 20 MB).
- Record byte sizes of both variants into `./out/bench/storage_sizes.csv` (append: format, variant, bytes).
- Marker `./out/.done/04.ok`.

30
docs/specs/05_convert_sqlite.md Normal file
View File

@@ -0,0 +1,30 @@
# 05 - Convert to SQLite (optimized)
## Context / Goal
Build the single-file relational variant optimized for size and single-user query speed.
## Dependencies
03.
## Input
CSV tree.
## Processing instructions
- One database `./out/sqlite/tribo.db`. PRAGMAs during load: `journal_mode=OFF, synchronous=OFF, cache_size=-262144`; after load: `journal_mode=WAL`, `ANALYZE`, `VACUUM`.
- Schema (16-table style, integer surrogate keys everywhere; no TEXT foreign keys in bulk tables):
- `batches, wafers, coupons, runs, instruments, tracks` - dimension tables, TEXT natural keys kept as unique columns.
- `xrf_points(coupon_id INT, gx INT, gy INT, pt_wtpct REAL, au_wtpct REAL)`.
- `profilometry_points(coupon_id INT, gx INT, gy INT, thickness_um REAL)`.
- `nanoindentation(coupon_id INT, indent_id INT, x_um REAL, y_um REAL, hardness_gpa REAL, reduced_modulus_gpa REAL)`.
- `afm(coupon_id INT PRIMARY KEY, ra_nm REAL, rq_nm REAL)`.
- `friction_cycles(track_id INT, cycle INT, cof REAL, PRIMARY KEY(track_id, cycle)) WITHOUT ROWID`.
- `friction_loop_points(track_id INT, cycle INT, pt INT, position_um REAL, force_mn REAL, PRIMARY KEY(track_id, cycle, pt)) WITHOUT ROWID`.
- `wear(track_id INT PRIMARY KEY, wear_volume_um3 REAL, k_archard REAL, sliding_distance_m REAL)`.
- `track_summary(track_id INT PRIMARY KEY, cof_ss_mean REAL, cof_ss_std REAL, run_in_cycles INT)` - precomputed during load.
- Indexes for Q1-Q7: `coupons(au_wtpct_mean)`, `tracks(run_id)`, `tracks(coupon_id)`, `runs(environment)`, composite `tracks(environment, load_mn)` (denormalize environment/load onto tracks).
- Batch inserts with `executemany`, 50k rows per transaction.
- Validate row counts against MANIFEST.
## Output
- `./out/sqlite/tribo.db`; size appended to `storage_sizes.csv`.
- Marker `./out/.done/05.ok`.

23
docs/specs/06_convert_postgresql.md Normal file
View File

@@ -0,0 +1,23 @@
# 06 - Convert to PostgreSQL (indexed + partitioned)
## Context / Goal
Build the production-grade relational variant: the system-of-record candidate per the existing Storage ADR (PostgreSQL primary, RDF as deferred virtual layer).
## Dependencies
03.
## Input
CSV tree.
## Processing instructions
- Local PostgreSQL 16, database `tribo`. Same logical schema as task 05 with PostgreSQL types (`INT`, `REAL`, `TIMESTAMPTZ`, `TEXT`).
- `friction_cycles` and `friction_loop_points`: declarative RANGE partitioning by `track_id` (12 partitions of 120 tracks) - models the partitioning strategy needed at 600 GB scale.
- Load via `COPY FROM STDIN` (psycopg3 `copy`), not INSERT.
- Indexes after load: PK b-trees; `tracks(environment, load_mn)`, `coupons(au_wtpct_mean)`, `nanoindentation(coupon_id)`, BRIN on `friction_cycles(cycle)` per partition.
- `track_summary` as a materialized view (`cof_ss_mean` = avg of cycles > run_in threshold; `run_in_cycles` via first cycle where |cof - cof_ss| < 2σ).
- `VACUUM ANALYZE` after load. Record `pg_total_relation_size` per table and total into `storage_sizes.csv`.
- Also export a compressed dump `./out/pg/tribo.dump` (pg_dump -Fc) for archiving-format comparison.
## Output
- Live `tribo` database + `./out/pg/tribo.dump`.
- Marker `./out/.done/06.ok`.

24
docs/specs/07_convert_rdf.md Normal file
View File

@@ -0,0 +1,24 @@
# 07 - Convert to RDF TripleStore
## Context / Goal
Build the semantic-web variant to quantify triple-store overhead and SPARQL performance on real hardware.
## Dependencies
03 (optionally reuse 04 hybrid JSON-LD as an intermediate).
## Input
CSV tree (or `./out/json/full/` for direct rdflib parsing - choose the faster path, document the choice).
## Processing instructions
- Vocabulary identical to task 04 (`tribology#` + QUDT + PROV-O).
- Triple modeling: every cycle → 3 triples (`rdf:type` optional at bulk level to save space - model cycles as `track :hasCycle [ :cycleNumber n ; :cof x ]` without explicit type; document the decision). Loop points likewise. Expected total: 25-45 M triples.
- Serialize `./out/rdf/dataset.nt.gz` (N-Triples, gzip) and `./out/rdf/dataset.ttl` (Turtle) - both sizes recorded.
- Queryable store, first available option:
1. `oxigraph` (pip, embedded, on-disk store `./out/rdf/oxigraph_store/`) - preferred, no external service.
2. Jena Fuseki with TDB2 if Java present.
- Bulk-load, record store on-disk size into `storage_sizes.csv` and load wall-time.
- Smoke-test with a SPARQL COUNT query.
## Output
- Serializations + populated store.
- Marker `./out/.done/07.ok` with triple count.

31
docs/specs/08_benchmark_queries.md Normal file
View File

@@ -0,0 +1,31 @@
# 08 - Benchmark Query Definitions (Q1-Q7 × 5 formats)
## Context / Goal
Define the 7 laboratory-standard retrieval scenarios and their concrete implementation for each of the 5 formats. Every implementation must return identical result sets (validated by checksum of sorted result rows, tolerance 1e-9 on floats).
## Dependencies
04, 05, 06, 07 (all stores populated) + 03 (CSV baseline).
## Scenarios
| ID | Scenario | Result shape |
|---|---|---|
| Q1 | COF vs cycle curve for one track (`B722-W2-C13-T2`) | 1,000 rows (cycle, cof) |
| Q2 | Steady-state COF for all coupons with mean Au = 10 ± 0.5 wt% | ~dozens of rows (coupon, env, cof_ss) |
| Q3 | Hardness vs steady-state COF across all batches (join nano × track_summary) | 480 rows |
| Q4 | Tracks in Dry N2, load 100 mN, cof_ss > 0.20 (anomaly filter) | filtered rows |
| Q5 | Mean run-in cycles per batch (aggregation over ALL cycle rows, no summary tables allowed - force the full scan) | 4 rows |
| Q6 | Wear volume vs load per coupon (join wear × tracks × coupons) | 480 rows |
| Q7 | Stribeck-style aggregation: mean COF grouped by (environment, speed·load bucket) over all cycles past run-in | ~10 rows |
## Implementations
- **CSV**: pure Python + csv module streaming (pandas allowed only as a second measured variant, labeled separately). Q1 = direct path read (this is the honest CSV strength). Q2-Q7 = directory walks + programmatic joins.
- **JSON**: variant FULL, `ijson` streaming parser; Q1 = load the single coupon file.
- **SQLite**: SQL over `tribo.db`, cold and warm cache runs (`PRAGMA cache_size` default; drop OS cache not required, just report both run 1 and run 3).
- **PostgreSQL**: SQL, `EXPLAIN (ANALYZE, BUFFERS)` captured alongside; parallel workers default.
- **RDF**: SPARQL against the populated store; capture query text.
- Q5 rule applies to all formats: query raw cycle data, not `track_summary`.
## Output
- `./out/bench/queries/` - one subfolder per format with 7 runnable query files (`q1.sql`, `q1.sparql`, `q1.py`, ...).
- `./out/bench/expected/q<N>.sha256` - canonical result checksums (computed once from PostgreSQL, cross-validated against SQLite).
- Marker `./out/.done/08.ok`.

22
docs/specs/09_benchmark_execution.md Normal file
View File

@@ -0,0 +1,22 @@
# 09 - Benchmark Execution & Resource Measurement
## Context / Goal
Execute all 35 (Q1-Q7 × 5) benchmark cells with rigorous resource metering; produce the raw measurement table that feeds extrapolation and reporting.
## Dependencies
08.
## Processing instructions
Write `bench_runner.py` (tabs):
- Each cell runs as a **separate subprocess** so RSS/CPU are attributable. For PostgreSQL, also sample the backend PID via `pg_stat_activity` + psutil.
- Metrics per run: wall time (s), peak RSS (MB, psutil `memory_info().rss` sampled at 50 ms), CPU time user+sys (s), CPU utilization % (cpu_time / wall / cores), read bytes (psutil `io_counters`), result row count, checksum-match flag.
- Protocol: 1 warm-up run (discarded) + 3 measured runs per cell; report median and min/max. Randomize cell order to spread cache effects; record run order.
- Cold-cache variant: repeat the full matrix once after dropping caches if permitted (`sync; echo 3 > /proc/sys/vm/drop_caches` requires root - if not available, skip and note it).
- Failures/timeouts: hard timeout 30 min per run; record `TIMEOUT` and continue (expected possibility: RDF Q5).
- Storage footprint: consolidate `storage_sizes.csv` (all formats, incl. gzip-compressed archival sizes of each: `tar.gz` of CSV tree, gzip JSON, sqlite db gz, pg dump, nt.gz).
## Output
- `./out/bench/results_raw.csv`: format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb, rows, checksum_ok.
- `./out/bench/results_median.csv` - aggregated.
- `./out/bench/storage_sizes.csv` - final consolidated.
- Marker `./out/.done/09.ok`.

38
docs/specs/10_extrapolation_model.md Normal file
View File

@@ -0,0 +1,38 @@
# 10 - Extrapolation to 600 GB and Beyond
## Context / Goal
Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.
## Dependencies
09 (`results_median.csv`, `storage_sizes.csv`).
## Processing instructions
Write `extrapolate.py`:
### Storage scaling
Linear in data volume: `size_at(S) = measured_size × S / S_measured`, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).
### Query-time scaling laws (apply per format × query)
| Access pattern | Law | Applies to |
|---|---|---|
| Direct path / keyed point read | O(1) / O(log n) | CSV Q1, SQLite/PG Q1, RDF Q1 |
| Indexed selective filter | O(log n + k) | SQLite/PG Q2, Q4, Q6; RDF Q2, Q4 |
| Indexed join over dimensions | O(k log n) | Q3, Q6 relational; RDF joins ×(2-5) constant penalty |
| Full scan / aggregation | O(n) | all CSV/JSON Q2-Q7; Q5 and Q7 everywhere |
| Single-threaded full scan | O(n), 1 core | SQLite Q5/Q7 |
| Parallel partitioned scan | O(n / min(cores, partitions)) | PostgreSQL Q5/Q7 |
Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".
### RAM model
- CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
- SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
- RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).
### Hardware sizing table
For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in `./out/config/hw_prices.yaml` (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.
## Output
- `./out/bench/extrapolation.csv`: format, query, scale_gb, projected_wall_s, flag.
- `./out/bench/hardware_sizing.csv`: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
- Marker `./out/.done/10.ok`.

37
docs/specs/11_reporting.md Normal file
View File

@@ -0,0 +1,37 @@
# 11 - Reporting: Tables, Charts, Weighted Scoring
## Context / Goal
Aggregate all measurements and projections into the final decision document that clearly ranks the storage formats for laboratory use cases: analysis, report generation, search/filtering, archiving, inter-lab exchange.
## Dependencies
09, 10 (and 02 for embedding process diagrams).
## Processing instructions
Write `report.py` using matplotlib (PNG, 150 dpi, log scales where ranges span decades) into `./out/report/charts/`, and assemble `./out/report/REPORT.md`.
### Tables
1. Storage footprint: measured @150 MB + coefficients vs CSV + projected @600 GB/1.2 TB/6 TB (incl. compressed archival sizes).
2. Q1-Q7 median wall time / peak RAM / CPU% / read MB, per format (measured).
3. Q1-Q7 projected wall time @600 GB with IMPRACTICAL/FAIL flags.
4. Hardware sizing & cost per format @600 GB.
5. Weighted scoring matrix (see below).
### Charts (each measured + each projected @600 GB)
- C1: disk footprint bar (log).
- C2: RAM requirement bar (log).
- C3: per-query grouped bars Q1-Q7 wall time (log), one chart per query, 5 formats each.
- C4: degradation curves - wall time vs data size (150 MB → 6 TB), one line per format, per query class (point read / indexed / full aggregation), log-log.
- C5: stacked-bar weighted score.
- C6: cost vs performance scatter @600 GB (x = est cost, y = geometric-mean query time, log-log).
### Weighted scoring (agreed methodology)
Normalized 0-10 per metric, then weights: Search speed ×5 (0-50), RAM economy ×2 (0-20), Disk economy ×1 (0-10). Max 80. Search score = normalized inverse geometric mean of Q1-Q7 projected @600 GB (TIMEOUT/FAIL → 0). Present measured-scale and 600 GB-scale scoreboards side by side.
### Narrative sections (concise, English)
- Executive summary with the recommendation (expected, to be confirmed by data: PostgreSQL system of record; CSV retained as raw archive; hybrid JSON-LD for exchange; RDF as virtual layer only - reference Ontop OBDA).
- Use-case mapping table: analysis / reports / search / archiving / exchange → recommended format(s).
- Limitations: single-node, simulated data, extrapolation assumptions.
## Output
- `./out/report/REPORT.md` + `./out/report/charts/*.png` + all tables as CSV in `./out/report/tables/`.
- Marker `./out/.done/11.ok`.

27
docs/specs_old/00_PLAN.md Normal file
View File

@@ -0,0 +1,27 @@
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
## Tasks
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
## Final Goal
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.

26
docs/specs_old/01_setup_workspace.md Normal file
View File

@@ -0,0 +1,26 @@
# Task 01 — Setup Workspace
## Goal
Create the working directory layout and versioned pipeline configuration.
## Input
- Path to the raw CSV root (read-only). Ask the user if unknown.
## Actions
1. Create:
```
work/
config/ # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
manifest/ # inventory, fingerprints, reports
canonical/ # cleansed partitioned dataset
rejects/ # quarantined anomalies
schema/ # DDL, data dictionary, instruments repository
exports/{json,sqlite,postgresql,rdf}/
lineage/
scripts/ # all Python scripts produced by tasks
```
2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
## Intermediate Result (done when)
- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.

17
docs/specs_old/02_fs_inventory.md Normal file
View File

@@ -0,0 +1,17 @@
# Task 02 — Filesystem Inventory
## Goal
Non-destructive inventory of every file under `SOURCE_ROOT`.
## Input
- `work/config/pipeline.json` (Task 01).
## Actions
1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
## Intermediate Result (done when)
- `inventory.jsonl` row count == filesystem file count.
- `inventory_summary.md` produced; non-CSV extensions listed explicitly.

17
docs/specs_old/03_filename_tokenization.md Normal file
View File

@@ -0,0 +1,17 @@
# Task 03 — Filename & Path Tokenization
## Goal
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
## Input
- `manifest/inventory.jsonl` (Task 02).
- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
## Actions
1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
## Intermediate Result (done when)
- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).

18
docs/specs_old/04_schema_profiling.md Normal file
View File

@@ -0,0 +1,18 @@
# Task 04 — Schema Profiling & Fingerprints
## Goal
Capture the record structure of every CSV without full load; compute schema fingerprints.
## Input
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
## Actions
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
## Intermediate Result (done when)
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
- Every profile carries a fingerprint.

16
docs/specs_old/05_schema_grouping.md Normal file
View File

@@ -0,0 +1,16 @@
# Task 05 — Schema Grouping & Inconsistency Report
## Goal
Enumerate distinct record structures per measurement type; expose structural inconsistencies.
## Input
- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
## Actions
1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
## Intermediate Result (done when)
- The number of distinct schemas per measurement type is known and documented.
- `inconsistencies.csv` complete — this is the work queue for Task 07.

19
docs/specs_old/06_storage_map.md Normal file
View File

@@ -0,0 +1,19 @@
# Task 06 — Storage Map (As-Is / As-Designed)
## Goal
Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
## Input
- All Task 02–05 artifacts.
## Actions
1. Write `manifest/storage_map.md`:
- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
- purpose of each file kind mapped to measurement types and attachment levels;
- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
## Intermediate Result (done when)
- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.

18
docs/specs_old/07_column_mapping.md Normal file
View File

@@ -0,0 +1,18 @@
# Task 07 — Column Mapping (raw → canonical)
## Goal
For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
## Input
- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
## Actions
1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
## Intermediate Result (done when)
- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
- Escalation list for Task 08 written to `schema/ontology_escalations.md`.

18
docs/specs_old/08_ontology_schema.md Normal file
View File

@@ -0,0 +1,18 @@
# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
## Goal
Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
## Input
- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
## Actions
1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
## Intermediate Result (done when)
- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
- No open items in `ontology_escalations.md`.

18
docs/specs_old/09_cleanse_normalize.md Normal file
View File

@@ -0,0 +1,18 @@
# Task 09 — Cleanse & Normalize → Canonical Dataset
## Goal
Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
## Input
- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
## Actions
1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
## Intermediate Result (done when)
- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
- Canonical partitions written; lineage present for every canonical row.

17
docs/specs_old/10_anomaly_report.md Normal file
View File

@@ -0,0 +1,17 @@
# Task 10 — Anomaly Report & Iteration Gate
## Goal
Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
## Input
- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
## Actions
1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
## Intermediate Result (done when)
- `anomaly_report.md` final; every reject class documented as a true anomaly.
- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.

17
docs/specs_old/11_export_json.md Normal file
View File

@@ -0,0 +1,17 @@
# Task 11 — Export: JSON
## Goal
Two JSON products from the frozen canonical snapshot.
## Input
- `canonical/` snapshot, `schema/` (Tasks 09–10).
## Actions
1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
## Intermediate Result (done when)
- Both products written; spot-validation: point counts per measurement match canonical.
- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).

16
docs/specs_old/12_export_sqlite.md Normal file
View File

@@ -0,0 +1,16 @@
# Task 12 — Export: SQLite
## Goal
Portable single-file SQLite database conforming to the ontology.
## Input
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
## Actions
1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
## Intermediate Result (done when)
- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
- `exports/sqlite/EXPORT_MANIFEST.json` written.

22
docs/specs_old/13_export_postgresql.md Normal file
View File

@@ -0,0 +1,22 @@
# Task 13 — Export: PostgreSQL
## Goal
System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
## Input
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
## Actions
1. Generate `exports/postgresql/`:
- `01_schema.sql` (DDL, versioned);
- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
- per-table CSV staging files from canonical (streamed);
- `03_copy_load.sql` (`\copy` statements in dependency order);
- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
- `load.sh` / `load.ps1` orchestration.
2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
3. Include `lineage` table load (row-range granularity).
## Intermediate Result (done when)
- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.

17
docs/specs_old/14_export_rdf.md Normal file
View File

@@ -0,0 +1,17 @@
# Task 14 — Export: RDF (Metadata & Provenance Graph)
## Goal
RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
## Input
- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
## Actions
1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
## Intermediate Result (done when)
- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.

18
docs/specs_old/15_lineage_build_manifest.md Normal file
View File

@@ -0,0 +1,18 @@
# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
## Goal
Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
## Input
- All `exports/*`, `lineage/`, all config/schema versions.
## Actions
1. Write `scripts/finalize.py`:
- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
## Final Result (pipeline done when)
- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.

350
make_lab_config.py Normal file
View File

@@ -0,0 +1,350 @@
"""Task 01 - Laboratory configuration.
Writes ./out/config/lab_config.yaml, the authoritative machine-readable
laboratory model (volumes, deposition matrix, physical-model coefficients,
random seed) that all downstream tasks read, plus the ./out/.done/01.ok
completion marker. Spec: docs/specs/01_lab_configuration.md.
"""
from __future__ import annotations
import argparse
import logging
import sys
from pathlib import Path
import yaml
logger = logging.getLogger(__name__)
REPO_ROOT = Path(__file__).resolve().parent
TASK_ID = "01"
# Transcription of docs/specs/01_lab_configuration.md. This literal is the
# ONLY sanctioned home of these values in code; every other task reads the
# YAML this script writes (docs/rules/code-config-yaml.md).
LAB_CONFIG: dict = {
"project": "LabDataStorageEvaluation",
"description": "Simulated tribology laboratory (Sandia Pt-Au LDRD context)",
"spec": "docs/specs/01_lab_configuration.md",
"seed": 20260711,
"material_system": {
"substrate": {"material": "Ti-6Al-4V", "dimensions_mm": [10, 10, 3]},
"adhesion_layer": {"material": "Cr", "process": "sputtered"},
"coating": {
"material": "Pt-Au",
"deposition": "composition gradient across each wafer",
"thickness_um_range": [0.3, 1.1],
},
},
"instruments": [
{
"instrument_id": "rapid",
"role": "friction",
"name": "RAPID custom high-throughput parallelized 6-probe tribometer",
"data_produced": "COF vs cycle per track; test conditions",
},
{
"instrument_id": "ti980",
"role": "nanoindentation",
"name": "Bruker TI980 TriboIndenter",
"data_produced": "hardness, reduced modulus (25 indents per coupon)",
},
{
"instrument_id": "m4_tornado",
"role": "composition",
"name": "Bruker M4 Tornado micro-XRF",
"data_produced": "Pt/Au wt% map per coupon (20x20 grid)",
},
{
"instrument_id": "pvd200",
"role": "deposition",
"name": "Kurt J. Lesker PVD 200 sputter-down",
"data_produced": "batch deposition parameters",
},
{
"instrument_id": "afm",
"role": "surface_roughness",
"name": "AFM",
"data_produced": "topography image metadata + Ra/Rq per coupon",
},
{
"instrument_id": "profilometer",
"role": "film_thickness",
"name": "Optical profilometry",
"data_produced": "thickness map per coupon (10x10 grid)",
},
{
"instrument_id": "simtra",
"role": "simulation",
"name": "SIMTRA sputter transport Monte-Carlo",
"data_produced": "deposition atom-energy / composition profiles per deposition run",
},
],
"deposition_matrix": [
{
"batch_code": "B721",
"pt_gun_tilt_deg": 20,
"au_gun_tilt_deg": 0,
"pt_power_W": 150,
"au_power_W": 50,
"pt_discharge_V": 432,
"au_discharge_V": 311,
},
{
"batch_code": "B722",
"pt_gun_tilt_deg": 20,
"au_gun_tilt_deg": 20,
"pt_power_W": 100,
"au_power_W": 100,
"pt_discharge_V": 399,
"au_discharge_V": 352,
},
{
"batch_code": "B723",
"pt_gun_tilt_deg": 20,
"au_gun_tilt_deg": 20,
"pt_power_W": 150,
"au_power_W": 50,
"pt_discharge_V": 430,
"au_discharge_V": 311,
},
{
"batch_code": "B724",
"pt_gun_tilt_deg": 0,
"au_gun_tilt_deg": 20,
"pt_power_W": 50,
"au_power_W": 150,
"pt_discharge_V": 376,
"au_discharge_V": 340,
},
],
"hierarchy": {
"batches": 4,
"wafers_per_batch": 3,
"coupon_grid_per_wafer": [7, 7],
"coupons_per_wafer": 49,
},
"friction_assignment": {
"friction_coupons_total": 480,
"reserve_coupons_total": 108,
"runs": 4,
"coupons_per_run": 120,
"plates_per_run": 5,
"probes_per_plate": 6,
"coupons_per_probe_square": 4,
"tracks_per_friction_coupon": 3,
"counterfaces_per_holder": 3,
"fresh_counterface_per_track": True,
},
"volumes": {
"coupons_total": 588,
"tracks_total": 1440,
"cycles_per_track": 1000,
"cycle_rows_total": 1440000,
"loop_every_n_cycles": 100,
"loops_per_track": 10,
"loop_points_per_loop": 200,
"loop_points_total": 2880000,
"xrf_grid": [20, 20],
"xrf_points_per_coupon": 400,
"xrf_points_total": 235200,
"profilometry_grid": [10, 10],
"profilometry_points_per_coupon": 100,
"nanoindentation_indents_per_coupon": 25,
"simtra_rows_per_batch": 1000,
},
"runs": [
{"run_code": "R1", "environment": "lab_air", "date": "2026-04-06", "operator": "A. Reyes"},
{"run_code": "R2", "environment": "lab_air", "date": "2026-04-13", "operator": "K. Patel"},
{"run_code": "R3", "environment": "dry_n2", "date": "2026-04-20", "operator": "A. Reyes"},
{"run_code": "R4", "environment": "dry_n2", "date": "2026-04-27", "operator": "M. Novak"},
],
"schedule": {
"deposition_start_date": "2026-03-02",
"batch_interval_days": 7,
"characterization_lag_days": 3,
},
"test_conditions": {
"normal_load_mN": 100,
"stroke_mm": 1.0,
"speed_mm_s": 1.0,
"counterface": {"material": "ruby (Al2O3)", "diameter_mm": 3.175},
"rh_pct": {"lab_air": 45, "dry_n2": 2},
"temperature_C": 23,
"temperature_tolerance_C": 1,
},
"physical_models": {
"au_gradient": {
"description": "linear Au wt% gradient along wafer x, batch-dependent center",
"batch_center_wtpct": {"B721": 8, "B722": 25, "B723": 10, "B724": 60},
"wafer_span_wtpct": 5,
"xrf_noise_sd_wtpct": 0.3,
},
"cof_run_in": {
"formula": "cof(c) = cof_ss + (cof_0 - cof_ss) * exp(-c / tau) + N(0, sigma)",
"cof_0_range": [0.35, 0.5],
"tau_cycles_range": [100, 400],
"noise_sd": 0.01,
},
"cof_steady_state": {
"formula": "cof_ss = intercept + slope_per_au_wtpct * au_wtpct_mean, clamped to >= floor",
"lab_air": {"intercept": 0.32, "slope_per_au_wtpct": -0.0015},
"dry_n2": {"intercept": 0.24, "slope_per_au_wtpct": -0.0012},
"floor": 0.12,
},
"hardness_GPa": {
"formula": "H = intercept + slope_per_au_wtpct * au_wtpct_mean + N(0, noise_sd)",
"intercept": 8.5,
"slope_per_au_wtpct": -0.05,
"noise_sd": 0.25,
},
"reduced_modulus_GPa": {
"formula": "Er = intercept + slope_per_au_wtpct * au_wtpct_mean + N(0, noise_sd)",
"intercept": 190,
"slope_per_au_wtpct": -0.6,
"noise_sd": 4,
},
"roughness_ra_nm": {
"distribution": "lognormal",
"median_nm": 5,
"sigma_log": 0.3,
"rq_over_ra": 1.25,
},
"thickness_um": {
"profile": "radial parabolic across wafer plus noise",
"center_um": 1.05,
"edge_um": 0.35,
"noise_sd_um": 0.02,
"range_um": [0.3, 1.1],
},
"wear_archard": {
# Spec 01 fixes only the model shape (V = k*F*s, k depends on Au%
# and environment); these coefficients are project defaults chosen
# to yield Sandia-plausible low-wear volumes at 100 mN / 2 m.
"formula": "V = k * F * s; k = k0 * (1 + slope_per_au_wtpct * au_wtpct_mean), lognormal noise, clamped to >= k_floor",
"k0_mm3_per_N_m": {"lab_air": 1.5e-07, "dry_n2": 6.0e-08},
"slope_per_au_wtpct": {"lab_air": -0.008, "dry_n2": -0.010},
"k_floor_mm3_per_N_m": 1.0e-09,
"noise_sigma_log": 0.15,
},
"nanoindentation": {
"max_load_mN": 10,
"indent_grid": [5, 5],
"grid_pitch_um": 20,
},
"simtra": {
"angle_deg_range": [0, 90],
"energy_eV_max": 50,
"surface_binding_energy_eV": {"pt": 5.84, "au": 3.81},
"flux_per_W": 0.02,
},
},
}
YAML_HEADER = (
"# Generated by make_lab_config.py (task 01) from docs/specs/01_lab_configuration.md.\n"
"# Do not hand-edit: regenerate via `python make_lab_config.py`.\n"
)
def validate_config(cfg: dict) -> None:
"""Assert the arithmetic identities between declared volumes (spec 01)."""
h = cfg["hierarchy"]
f = cfg["friction_assignment"]
v = cfg["volumes"]
checks = [
("coupons_total", v["coupons_total"], h["batches"] * h["wafers_per_batch"] * h["coupons_per_wafer"]),
("coupons_per_wafer", h["coupons_per_wafer"], h["coupon_grid_per_wafer"][0] * h["coupon_grid_per_wafer"][1]),
("friction+reserve", v["coupons_total"], f["friction_coupons_total"] + f["reserve_coupons_total"]),
("friction_coupons_total", f["friction_coupons_total"], f["runs"] * f["coupons_per_run"]),
("coupons_per_run", f["coupons_per_run"], f["plates_per_run"] * f["probes_per_plate"] * f["coupons_per_probe_square"]),
("tracks_total", v["tracks_total"], f["friction_coupons_total"] * f["tracks_per_friction_coupon"]),
("cycle_rows_total", v["cycle_rows_total"], v["tracks_total"] * v["cycles_per_track"]),
("loops_per_track", v["loops_per_track"], v["cycles_per_track"] // v["loop_every_n_cycles"]),
("loop_points_total", v["loop_points_total"], v["tracks_total"] * v["loops_per_track"] * v["loop_points_per_loop"]),
("xrf_points_per_coupon", v["xrf_points_per_coupon"], v["xrf_grid"][0] * v["xrf_grid"][1]),
("xrf_points_total", v["xrf_points_total"], v["coupons_total"] * v["xrf_points_per_coupon"]),
("profilometry_points_per_coupon", v["profilometry_points_per_coupon"], v["profilometry_grid"][0] * v["profilometry_grid"][1]),
("runs_count", f["runs"], len(cfg["runs"])),
("batches_count", h["batches"], len(cfg["deposition_matrix"])),
]
for name, declared, derived in checks:
if declared != derived:
raise ValueError(f"volume identity failed: {name}: declared {declared} != derived {derived}")
logger.info("all %d volume identities hold", len(checks))
def write_yaml(cfg: dict, config_path: Path) -> int:
config_path.parent.mkdir(parents=True, exist_ok=True)
body = yaml.safe_dump(cfg, sort_keys=False, allow_unicode=False, width=100)
text = YAML_HEADER + body
with open(config_path, "w", encoding="utf-8", newline="\n") as fh:
fh.write(text)
reloaded = yaml.safe_load(text)
if reloaded != cfg:
raise RuntimeError(f"YAML round-trip mismatch for {config_path}")
return len(text.encode("utf-8"))
def write_marker(marker_path: Path, config_path: Path, config_bytes: int, cfg: dict) -> None:
v = cfg["volumes"]
f = cfg["friction_assignment"]
h = cfg["hierarchy"]
lines = [
f"task: '{TASK_ID}'",
"status: ok",
f"config_file: {config_path.as_posix()}",
f"config_bytes: {config_bytes}",
f"seed: {cfg['seed']}",
f"batches: {h['batches']}",
f"wafers: {h['batches'] * h['wafers_per_batch']}",
f"coupons: {v['coupons_total']}",
f"friction_coupons: {f['friction_coupons_total']}",
f"tracks: {v['tracks_total']}",
f"cycle_rows: {v['cycle_rows_total']}",
f"loop_points: {v['loop_points_total']}",
f"xrf_points: {v['xrf_points_total']}",
]
marker_path.parent.mkdir(parents=True, exist_ok=True)
with open(marker_path, "w", encoding="utf-8", newline="\n") as fh:
fh.write("\n".join(lines) + "\n")
def main() -> int:
parser = argparse.ArgumentParser(description="Task 01: write out/config/lab_config.yaml")
parser.add_argument("--log-level", default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR"])
parser.add_argument("--out-root", type=Path, default=REPO_ROOT / "out", help="artifact tree root (default: ./out)")
args = parser.parse_args()
logging.basicConfig(
level=args.log_level,
stream=sys.stderr,
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
out_root: Path = args.out_root
config_path = out_root / "config" / "lab_config.yaml"
marker_path = out_root / ".done" / f"{TASK_ID}.ok"
try:
if marker_path.exists():
logger.info("re-run: removing stale marker %s", marker_path)
marker_path.unlink()
validate_config(LAB_CONFIG)
config_bytes = write_yaml(LAB_CONFIG, config_path)
write_marker(marker_path, config_path, config_bytes, LAB_CONFIG)
except Exception:
logger.critical("task %s failed", TASK_ID, exc_info=True)
return 1
v = LAB_CONFIG["volumes"]
print(f"task 01 ok: {config_path} ({config_bytes} bytes)")
print(
f"volumes: batches=4 wafers=12 coupons={v['coupons_total']} tracks={v['tracks_total']} "
f"cycle_rows={v['cycle_rows_total']} loop_points={v['loop_points_total']} xrf_points={v['xrf_points_total']}"
)
print(f"marker: {marker_path}")
return 0
if __name__ == "__main__":
sys.exit(main())

12
requirements.txt Normal file
View File

@@ -0,0 +1,12 @@
# Closed dependency list - docs/rules/code-python-style.md section 2.
# Adding a package here requires updating that rule first.
numpy # task 03: vectorized generation, seeded RNG
PyYAML # tasks 01, 03, 10: YAML configs
rdflib # tasks 04, 07: JSON-LD validation, RDF serialization
pyoxigraph # tasks 07, 09: embedded queryable triplestore (pip name of oxigraph)
ijson # tasks 08, 09: streaming JSON parsing
psycopg[binary] # tasks 06, 08, 09: PostgreSQL COPY and queries
psutil # task 09: RSS / CPU / IO measurement
matplotlib # task 11: report charts
pandas # task 08 only: separately-measured CSV query variant