Compare commits
2 Commits
d8353953b0
...
9144d74d4c
| Author | SHA1 | Date | |
|---|---|---|---|
| 9144d74d4c | |||
|
b173ac82a9 |
48
.cursorrules
Normal file
48
.cursorrules
Normal file
@@ -0,0 +1,48 @@
|
||||
# Cursor rules for LabDataStorageEvaluation
|
||||
|
||||
## Commit messages (mandatory format)
|
||||
|
||||
Every commit message you generate MUST follow this convention exactly.
|
||||
|
||||
First line: type(scope): description
|
||||
- lowercase imperative English summary, under 72 characters, no trailing period
|
||||
- type is one of (closed list): feat fix perf refactor security docs ci chore test
|
||||
- scope is one of (closed list): specs rules docs config datagen convert bench report tools all
|
||||
|
||||
Scope selection by changed files:
|
||||
- specs = docs/specs/**
|
||||
- rules = docs/rules/**
|
||||
- docs = other prose docs (README.md, CLAUDE.md, docs/examples/**)
|
||||
- config = configuration files (.gitignore, .cursorrules, editable YAML defaults)
|
||||
- datagen = lab-configuration and data-generation code (tasks 01-03)
|
||||
- convert = format converters: JSON-LD, SQLite, PostgreSQL, RDF (tasks 04-07)
|
||||
- bench = benchmark queries, runner, extrapolation (tasks 08-10)
|
||||
- report = reporting code and templates (task 11)
|
||||
- tools = helper scripts under tools/
|
||||
- all = multiple areas, or when no narrower scope honestly fits
|
||||
|
||||
Body: the subject is only a title. If the diff touches more than one area,
|
||||
add a blank line and 2-6 concise bullets covering every meaningful changed
|
||||
area. Split unrelated work into separate commits.
|
||||
|
||||
Special forms allowed as-is: "Merge ..." commits; "release(scope): vX.Y.Z".
|
||||
|
||||
Never emit vague or unprefixed subjects such as: "Update files",
|
||||
"Improve project", "clarify rules", "changes".
|
||||
|
||||
Examples of good subjects:
|
||||
- feat(datagen): stream friction cycle rows during CSV generation
|
||||
- fix(convert): match sqlite row counts against MANIFEST before commit
|
||||
- docs(rules): add benchmark methodology rule
|
||||
- chore(config): ignore out/ artifact tree
|
||||
|
||||
## Typography
|
||||
|
||||
Use plain ASCII typography in everything you write into this repository:
|
||||
hyphen-minus instead of en/em dashes, straight quotes, three dots instead
|
||||
of the ellipsis character. See docs/rules/code-data-formatting.md.
|
||||
|
||||
## Python
|
||||
|
||||
Python code in this repository uses TABS for indentation. See
|
||||
docs/rules/code-python-style.md for the closed dependency list.
|
||||
15
.gitignore
vendored
Normal file
15
.gitignore
vendored
Normal file
@@ -0,0 +1,15 @@
|
||||
# Generated pipeline artifacts - reproducible from specs + rules + code
|
||||
# (see docs/rules/build-pipeline-tasks.md)
|
||||
out/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Editor / OS noise
|
||||
.vscode/
|
||||
.idea/
|
||||
Thumbs.db
|
||||
.DS_Store
|
||||
112
CLAUDE.md
Normal file
112
CLAUDE.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# LabDataStorageEvaluation - Project Rules
|
||||
|
||||
A reproducible evaluation pipeline for laboratory data storage formats,
|
||||
built around a simulated tribology laboratory (Sandia Pt-Au LDRD context).
|
||||
The pipeline (tasks 01-11, specs in [docs/specs/](docs/specs/)):
|
||||
|
||||
1. Defines the lab model (`lab_config.yaml`) and process-flow diagrams.
|
||||
2. Generates **~150 MB of physically plausible measurement data in CSV**
|
||||
(the canonical raw format), deterministically from **seed 20260711**.
|
||||
3. Converts the corpus into 4 target formats: **JSON-LD** (full + hybrid),
|
||||
**SQLite**, **PostgreSQL 16** (indexed + partitioned), **RDF
|
||||
triplestore** (oxigraph).
|
||||
4. Benchmarks **7 retrieval scenarios (Q1-Q7) x 5 formats**, measuring
|
||||
wall time, peak RAM, CPU, disk I/O, and footprint.
|
||||
5. Extrapolates to 600 GB / 1.2 TB / 6 TB and produces a ranked report
|
||||
(`REPORT.md` + charts).
|
||||
|
||||
Everything runs **locally** (Windows development host, local PostgreSQL 16,
|
||||
no cloud services, no server deploy). The stack is **Python 3.11+ (tabs for
|
||||
indentation)** with a closed dependency list; there is **no UI, no HTTP
|
||||
API, no authentication** in this project. All artifacts land under
|
||||
`./out/` (git-ignored); tasks signal completion via `./out/.done/<NN>.ok`
|
||||
markers.
|
||||
|
||||
These rules are **binding** for every contributor (human or AI agent)
|
||||
touching this repository. They override any pattern found in imported
|
||||
specs, external tutorials, or older project docs that predate this file.
|
||||
|
||||
---
|
||||
|
||||
## Mandatory rules - load before starting matching work
|
||||
|
||||
When the user's request matches a topic below, **before writing any code or
|
||||
producing a plan**, run `Read` on the listed rule file(s). The rules are
|
||||
binding; ignoring them produces broken work that the user will reject.
|
||||
|
||||
| If the task involves... | You MUST first Read |
|
||||
|---|---|
|
||||
| Writing or changing any Python (style, tabs, dependency list, streaming discipline) | [docs/rules/code-python-style.md](docs/rules/code-python-style.md) |
|
||||
| Data generation, physical models, random values, seeds, reproducibility | [docs/rules/data-determinism.md](docs/rules/data-determinism.md) |
|
||||
| Naming entities/IDs, timestamps, measurement columns, units, `au_wtpct_mean` | [docs/rules/data-naming-units.md](docs/rules/data-naming-units.md) |
|
||||
| SQLite / PostgreSQL schema, DDL, keys, indexes, bulk loading, `track_summary` | [docs/rules/db-sql-schema.md](docs/rules/db-sql-schema.md) |
|
||||
| Benchmark queries, the runner, resource measurement, timeouts, extrapolation, scoring | [docs/rules/bench-methodology.md](docs/rules/bench-methodology.md) |
|
||||
| The `./out/` tree, `.done` markers, task order/parallelism, `storage_sizes.csv`, re-runs | [docs/rules/build-pipeline-tasks.md](docs/rules/build-pipeline-tasks.md) |
|
||||
| Validation: MANIFEST, row counts, result checksums, expected volumes, pytest | [docs/rules/test-pipeline-validation.md](docs/rules/test-pipeline-validation.md) |
|
||||
| Python error handling (`try` / `except` / `raise`, abort semantics) | [docs/rules/code-error-handling.md](docs/rules/code-error-handling.md) |
|
||||
| Logging, `print()` vs logger, log levels | [docs/rules/obs-python-logging.md](docs/rules/obs-python-logging.md) |
|
||||
| Configuration values: the YAML configs, constants, the PostgreSQL DSN, CLI flags | [docs/rules/code-config-yaml.md](docs/rules/code-config-yaml.md) |
|
||||
| Writing, editing, or reviewing comments in source code (any language) | [docs/rules/code-comments-policy.md](docs/rules/code-comments-policy.md) |
|
||||
| Prose strings in source/config/docs (typography / encoding / dashes) | [docs/rules/code-data-formatting.md](docs/rules/code-data-formatting.md) |
|
||||
| Adding or renaming a rule file in `docs/rules/` | [docs/rules/meta-rules-naming.md](docs/rules/meta-rules-naming.md) |
|
||||
| Anything where misinterpretation could be destructive (deleting/regenerating `./out/`, dropping the `tribo` DB, changing the seed or `lab_config.yaml`, editing specs, git history) | [docs/rules/meta-no-assumptions.md](docs/rules/meta-no-assumptions.md) |
|
||||
| Every response (length / formatting discipline) | [docs/rules/meta-concise-answers.md](docs/rules/meta-concise-answers.md) |
|
||||
| Asking the user any question | [docs/rules/meta-concise-questions.md](docs/rules/meta-concise-questions.md) |
|
||||
| Every response (communication language / register - Russian, no jargon or slang) | [docs/rules/meta-language-style.md](docs/rules/meta-language-style.md) |
|
||||
| Any git operation that would create a commit, tag, merge, revert, rebase, cherry-pick, or push | [docs/rules/meta-git-commits.md](docs/rules/meta-git-commits.md) |
|
||||
| Generating commit messages via Cursor, or maintaining the `.cursorrules` artifact | [docs/rules/meta-cursor-rules.md](docs/rules/meta-cursor-rules.md) |
|
||||
|
||||
Multiple matches -> Read all matching rule files. Unsure -> Read the closest one;
|
||||
over-loading is cheap, missing a rule is expensive.
|
||||
|
||||
---
|
||||
|
||||
## Other binding constraints
|
||||
|
||||
These are short, cross-cutting constraints. They are still **binding**; the
|
||||
linked rule file holds the detailed spec.
|
||||
|
||||
- **Determinism is the foundation.** Seed `20260711` lives in
|
||||
`lab_config.yaml` and is never hardcoded; only seeded RNGs; no
|
||||
wall-clock values inside the corpus; regeneration is byte-identical
|
||||
(proof: `MANIFEST.csv` sha256). See
|
||||
[docs/rules/data-determinism.md](docs/rules/data-determinism.md).
|
||||
- **A wrong artifact is worse than no artifact.** Validation gates (row
|
||||
counts vs MANIFEST, result checksums with 1e-9 tolerance, the
|
||||
120-200 MB size gate) abort loudly; a `.done` marker is written only
|
||||
after full success. See
|
||||
[docs/rules/code-error-handling.md](docs/rules/code-error-handling.md)
|
||||
and [docs/rules/test-pipeline-validation.md](docs/rules/test-pipeline-validation.md).
|
||||
- **Tasks communicate only through `./out/` artifacts.** The tree is
|
||||
git-ignored; sources of truth are specs + rules + code. Benchmarks run
|
||||
on a quiet machine. See
|
||||
[docs/rules/build-pipeline-tasks.md](docs/rules/build-pipeline-tasks.md).
|
||||
- **Measurement honesty.** Subprocess isolation per cell, 1 warm-up + 3
|
||||
measured runs, randomized order, 30-min timeout recorded as `TIMEOUT`,
|
||||
failed checksums invalidate the measurement, projected numbers are
|
||||
always labeled. See
|
||||
[docs/rules/bench-methodology.md](docs/rules/bench-methodology.md).
|
||||
- **Python: tabs, closed dependency list.** Stdlib first; only the
|
||||
packages listed in
|
||||
[docs/rules/code-python-style.md](docs/rules/code-python-style.md);
|
||||
task 03 is stdlib + numpy only; stream everything bulk.
|
||||
- **Commit message format (mandatory).** First line
|
||||
`type(scope): description`, lowercase imperative English, under 72
|
||||
chars. Types (closed): `feat fix perf refactor security docs ci chore
|
||||
test`. Scopes (closed): `specs rules docs config datagen convert bench
|
||||
report tools all`. The AI never creates commits - git is read-only for
|
||||
agents; the user runs all git write commands. See
|
||||
[docs/rules/meta-git-commits.md](docs/rules/meta-git-commits.md).
|
||||
|
||||
---
|
||||
|
||||
## Canonical source of truth
|
||||
|
||||
If this file and any `docs/rules/<name>.md` disagree, **the rule file wins**
|
||||
within its topic - that is where the detailed binding spec lives. If a rule
|
||||
and a task spec in `docs/specs/` disagree on a convention, **the rule wins**
|
||||
and the spec is updated to match; the specs remain authoritative for task
|
||||
scope, inputs, outputs, and dependencies. If this file and any other
|
||||
document (README, older docs, code comments) disagree, **this file plus the
|
||||
rule files win**. Update the other document to match; do not work around
|
||||
the rules.
|
||||
264
README.md
264
README.md
@@ -1,11 +1,257 @@
|
||||
# LabDataStorageEvaluation
|
||||
|
||||
Build a complete, reproducible pipeline that:
|
||||
|
||||
|
||||
Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow.
|
||||
Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format).
|
||||
Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
|
||||
Benchmarks 7 laboratory-standard retrieval scenarios (Q1–Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
|
||||
Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
|
||||
Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange).
|
||||
A reproducible pipeline that answers one practical question: **which storage
|
||||
format should a materials-science laboratory use for its measurement data?**
|
||||
|
||||
The pipeline simulates a tribology laboratory (Sandia Pt-Au LDRD context),
|
||||
generates ~150 MB of physically plausible raw measurement data in CSV,
|
||||
converts the corpus into four additional storage formats, benchmarks seven
|
||||
laboratory-standard retrieval scenarios against all five formats, extrapolates
|
||||
the measurements to 600 GB - 6 TB, and produces a ranked decision report for
|
||||
five laboratory use cases: analysis, reporting, search/filtering, archiving,
|
||||
and inter-lab exchange.
|
||||
|
||||
- Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11).
|
||||
- Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md).
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define a simulated tribology laboratory with real instruments and a
|
||||
realistic test workflow.
|
||||
2. Generate ~150 MB of physically plausible measurement data in CSV (the
|
||||
canonical raw format), deterministically from seed `20260711`.
|
||||
3. Convert the CSV corpus into 4 target storage formats: JSON-LD, SQLite
|
||||
(optimized), PostgreSQL 16 (indexed + partitioned), RDF triplestore.
|
||||
4. Benchmark 7 retrieval scenarios (Q1-Q7) against all 5 formats, measuring
|
||||
wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
|
||||
5. Extrapolate measured results to 600 GB / 1.2 TB / 6 TB with
|
||||
complexity-aware scaling models.
|
||||
6. Produce summary tables, charts, and a weighted scoreboard that rank the
|
||||
formats per use case.
|
||||
|
||||
## Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
|
||||
T01 --> T03[03_csv_data_generation]
|
||||
T03 --> T04[04_convert_json]
|
||||
T03 --> T05[05_convert_sqlite]
|
||||
T03 --> T06[06_convert_postgresql]
|
||||
T03 --> T07[07_convert_rdf]
|
||||
T04 --> T08[08_benchmark_queries]
|
||||
T05 --> T08
|
||||
T06 --> T08
|
||||
T07 --> T08
|
||||
T03 --> T08
|
||||
T08 --> T09[09_benchmark_execution]
|
||||
T09 --> T10[10_extrapolation_model]
|
||||
T10 --> T11[11_reporting]
|
||||
T09 --> T11
|
||||
```
|
||||
|
||||
Execution order: `01 -> 02 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11`.
|
||||
Tasks 04-07 are independent and may run in parallel; task 09 requires a quiet
|
||||
machine (no parallel work). Every task writes a completion marker
|
||||
`./out/.done/<NN>.ok` and validates its dependencies' markers before starting.
|
||||
|
||||
| Task | Script | Main output | Status |
|
||||
|---|---|---|---|
|
||||
| 01 Lab configuration | `make_lab_config.py` | `out/config/lab_config.yaml` | implemented |
|
||||
| 02 Process flow diagrams | `make_process_flow.py` | `out/report/process_flow.md`, `out/report/diagrams/` | planned |
|
||||
| 03 CSV data generation | `generate_data.py` | `out/csv/` tree + `MANIFEST.csv` | planned |
|
||||
| 04 Convert to JSON-LD | `convert_json.py` | `out/json/full/`, `out/json/hybrid/dataset.jsonld` | planned |
|
||||
| 05 Convert to SQLite | `convert_sqlite.py` | `out/sqlite/tribo.db` | planned |
|
||||
| 06 Convert to PostgreSQL | `convert_postgresql.py` | live `tribo` DB + `out/pg/tribo.dump` | planned |
|
||||
| 07 Convert to RDF | `convert_rdf.py` | `out/rdf/dataset.nt.gz`, oxigraph store | planned |
|
||||
| 08 Benchmark queries | `make_queries.py` | `out/bench/queries/`, `out/bench/expected/` | planned |
|
||||
| 09 Benchmark execution | `bench_runner.py` | `out/bench/results_raw.csv`, `results_median.csv` | planned |
|
||||
| 10 Extrapolation | `extrapolate.py` | `out/bench/extrapolation.csv`, `hardware_sizing.csv` | planned |
|
||||
| 11 Reporting | `report.py` | `out/report/REPORT.md`, charts, tables | planned |
|
||||
|
||||
## The simulated laboratory
|
||||
|
||||
Ti-6Al-4V coupons (10 x 10 x 3 mm) with a sputtered Cr adhesion layer and a
|
||||
Pt-Au coating deposited as a composition gradient (film thickness
|
||||
0.3-1.1 um). Four deposition batches (B721-B724) vary gun tilt, power, and
|
||||
discharge voltage. Instruments: RAPID 6-probe tribometer (friction), Bruker
|
||||
TI980 (nanoindentation), Bruker M4 Tornado micro-XRF (composition), Kurt J.
|
||||
Lesker PVD 200 (deposition), AFM (roughness), optical profilometry
|
||||
(thickness), SIMTRA (sputter transport simulation).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
B[Batch x4] --> W[Wafer x3 per batch]
|
||||
W --> C[Coupon x49 per wafer, 7x7 grid]
|
||||
C --> X[xrf_map: 400 points]
|
||||
C --> N[nanoindentation: 25 indents]
|
||||
C --> A[afm: Ra / Rq]
|
||||
C --> P[profilometry: 100 points]
|
||||
C --> T[Track x3, friction coupons only]
|
||||
T --> CY[cof_vs_cycle: 1000 cycles]
|
||||
T --> L[friction_loops: 10 x 200 points]
|
||||
T --> WR[wear: 1 row]
|
||||
```
|
||||
|
||||
| Entity | Count |
|
||||
|---|---|
|
||||
| Batches | 4 |
|
||||
| Wafers | 12 |
|
||||
| Coupons | 588 (480 friction-tested + 108 reserve/QA) |
|
||||
| Tracks | 1,440 |
|
||||
| Friction cycle rows | 1,440,000 |
|
||||
| Friction loop points | 2,880,000 |
|
||||
| XRF map points | 235,200 |
|
||||
|
||||
Friction runs R1/R2 execute in Lab Air, R3/R4 in Dry N2. Generated values
|
||||
follow explicit physical models (COF run-in exponential, steady-state COF and
|
||||
hardness vs Au content, Archard wear), so benchmark queries return physically
|
||||
meaningful results.
|
||||
|
||||
## Storage formats under evaluation
|
||||
|
||||
| # | Format | Role in the comparison |
|
||||
|---|---|---|
|
||||
| 1 | CSV (raw tree) | Canonical instrument output; baseline |
|
||||
| 2 | JSON-LD (full + hybrid) | Ontology-annotated exchange format (FAIR) |
|
||||
| 3 | SQLite | Single-file relational, optimized for size and speed |
|
||||
| 4 | PostgreSQL 16 | Production relational: indexed, partitioned |
|
||||
| 5 | RDF triplestore (oxigraph) | Semantic-web store queried via SPARQL |
|
||||
|
||||
## Benchmark scenarios (Q1-Q7)
|
||||
|
||||
| ID | Scenario | Access pattern |
|
||||
|---|---|---|
|
||||
| Q1 | COF vs cycle curve for one track | Point read |
|
||||
| Q2 | Steady-state COF for coupons with mean Au = 10 +/- 0.5 wt% | Indexed filter |
|
||||
| Q3 | Hardness vs steady-state COF across all batches | Indexed join |
|
||||
| Q4 | Tracks in Dry N2, 100 mN, cof_ss > 0.20 (anomaly filter) | Indexed filter |
|
||||
| Q5 | Mean run-in cycles per batch over ALL raw cycle rows | Forced full scan |
|
||||
| Q6 | Wear volume vs load per coupon | Indexed join |
|
||||
| Q7 | Stribeck-style mean COF by (environment, speed-load bucket) | Full aggregation |
|
||||
|
||||
Protocol: every cell (format x query) runs in its own subprocess; 1 warm-up +
|
||||
3 measured runs; randomized cell order; 30-minute hard timeout recorded as
|
||||
`TIMEOUT`; every result validated against a canonical checksum before the
|
||||
measurement counts. See
|
||||
[docs/rules/bench-methodology.md](docs/rules/bench-methodology.md).
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
docs/
|
||||
specs/ task specifications 00-11 (WHAT to build)
|
||||
rules/ binding conventions (HOW work is done)
|
||||
examples/ imported reference materials (not binding)
|
||||
out/ ALL generated artifacts (git-ignored, reproducible):
|
||||
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
|
||||
make_lab_config.py task 01 entry script (one script per task, repo root)
|
||||
requirements.txt closed dependency list (docs/rules/code-python-style.md)
|
||||
```
|
||||
|
||||
The `out/` tree - including the ~150 MB CSV corpus, the ~1 GB JSON-LD variant,
|
||||
databases, and benchmark results - is **excluded from git**. Sources of truth
|
||||
are the specs, the rules, and the code; every artifact regenerates
|
||||
deterministically from them.
|
||||
|
||||
## Getting started
|
||||
|
||||
Prerequisites: Python 3.11+ (3.12 tested). PostgreSQL 16 is required only for
|
||||
tasks 06/08/09; its DSN comes from the `TRIBO_PG_DSN` environment variable
|
||||
(default `postgresql://postgres@localhost:5432/tribo`).
|
||||
|
||||
Windows (PowerShell):
|
||||
|
||||
```powershell
|
||||
py -3 -m venv .venv
|
||||
.venv\Scripts\python -m pip install -r requirements.txt
|
||||
.venv\Scripts\python make_lab_config.py
|
||||
```
|
||||
|
||||
Linux:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt
|
||||
.venv/bin/python make_lab_config.py
|
||||
```
|
||||
|
||||
Each task script supports `--log-level {DEBUG,INFO,WARNING,ERROR}` and prints
|
||||
its protocol output (paths, counts, sizes) to stdout. A task refuses to run if
|
||||
a dependency's `out/.done/<NN>.ok` marker is missing.
|
||||
|
||||
Note on cold-cache runs: the page-cache drop used by the cold-cache benchmark
|
||||
pass is Linux-only; on Windows hosts that pass is skipped and marked as such.
|
||||
Benchmark execution is planned for a dedicated Linux host.
|
||||
|
||||
## Results
|
||||
|
||||
This section is populated by tasks 02, 09, 10, and 11. All artifacts land
|
||||
under `out/` (git-ignored); the placeholders below name the exact files the
|
||||
pipeline produces, so the section can be filled in by copying the generated
|
||||
tables and linking the generated images.
|
||||
|
||||
### Process flow diagrams (task 02)
|
||||
|
||||
`out/report/process_flow.md` with `out/report/diagrams/D1..D6.mermaid`:
|
||||
D1 coupon assembly, D2 characterization and batch assembly, D3 testing tree,
|
||||
D4 tribometer session sequence, D5 execution loop, D6 data hierarchy.
|
||||
|
||||
### Measured results (task 09) - to be generated
|
||||
|
||||
| Placeholder | Source file |
|
||||
|---|---|
|
||||
| Storage footprint per format (measured + compressed) | `out/bench/storage_sizes.csv` |
|
||||
| Q1-Q7 median wall time / peak RAM / CPU / read MB | `out/bench/results_median.csv` |
|
||||
|
||||
### Projections (task 10) - to be generated
|
||||
|
||||
| Placeholder | Source file |
|
||||
|---|---|
|
||||
| Projected wall time per format at 0.6 / 1.2 / 6 TB with IMPRACTICAL/FAIL flags | `out/bench/extrapolation.csv` |
|
||||
| Hardware sizing and cost per format at scale | `out/bench/hardware_sizing.csv` |
|
||||
|
||||
### Charts (task 11) - to be generated
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
(Images appear after running the full pipeline; task 11 writes them to
|
||||
`out/report/charts/` under exactly these names.)
|
||||
|
||||
### Final scoreboard (task 11) - to be generated
|
||||
|
||||
Weighted scoring, max 80 points: search speed x5 (0-50), RAM economy x2
|
||||
(0-20), disk economy x1 (0-10); TIMEOUT/FAIL projects to 0.
|
||||
|
||||
| Format | Search (0-50) | RAM (0-20) | Disk (0-10) | Total (0-80) |
|
||||
|---|---|---|---|---|
|
||||
| CSV | TBD | TBD | TBD | TBD |
|
||||
| JSON-LD | TBD | TBD | TBD | TBD |
|
||||
| SQLite | TBD | TBD | TBD | TBD |
|
||||
| PostgreSQL | TBD | TBD | TBD | TBD |
|
||||
| RDF | TBD | TBD | TBD | TBD |
|
||||
|
||||
| Use case | Recommended format |
|
||||
|---|---|
|
||||
| Analysis | TBD |
|
||||
| Report generation | TBD |
|
||||
| Search / filtering | TBD |
|
||||
| Archiving | TBD |
|
||||
| Inter-lab exchange | TBD |
|
||||
|
||||
## Conventions
|
||||
|
||||
- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`,
|
||||
`track:B721-W1-C07-T2`, runs `R1..R4`.
|
||||
- Timestamps: ISO-8601 UTC; all corpus timestamps are simulated.
|
||||
- Units are explicit ASCII suffixes in column names (`_mN`, `_um`, `_GPa`,
|
||||
`_wtpct`, `_nm`).
|
||||
- Random seed `20260711`, read from `lab_config.yaml`; regeneration is
|
||||
byte-identical (proof: `MANIFEST.csv` sha256).
|
||||
- Python: tabs for indentation; closed dependency list; streaming discipline
|
||||
(the corpus never fits in memory by design).
|
||||
|
||||
248
docs/examples/TribologyData_MasterPlan_v1.md
Normal file
248
docs/examples/TribologyData_MasterPlan_v1.md
Normal file
@@ -0,0 +1,248 @@
|
||||
# Tribology Lab Data — Master Plan
|
||||
|
||||
**Discovery · Cleansing/Normalization · Ontology · Conversion**
|
||||
|
||||
Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
|
||||
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
|
||||
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
|
||||
Status: Plan v1 — Draft
|
||||
|
||||
---
|
||||
|
||||
## 0. Objective
|
||||
|
||||
Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
|
||||
|
||||
The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Guiding principles
|
||||
|
||||
1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
|
||||
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
|
||||
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
|
||||
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
|
||||
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
|
||||
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
|
||||
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
|
||||
|
||||
---
|
||||
|
||||
## 2. Phase overview
|
||||
|
||||
| Phase | Name | Nature | Primary output |
|
||||
|-------|------|--------|----------------|
|
||||
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
|
||||
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
|
||||
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
|
||||
|
||||
Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
|
||||
|
||||
**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
|
||||
|
||||
### 3.1 Inputs
|
||||
- Root of the ~700 GB CSV tree (read-only).
|
||||
|
||||
### 3.2 Activities
|
||||
|
||||
**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
|
||||
|
||||
**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
|
||||
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
|
||||
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
|
||||
- atmosphere tokens (`ambient`, `dryNitrogen`, …)
|
||||
|
||||
Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
|
||||
|
||||
**C. Structural profiling (header-only + sampled body).** Per file, without full load:
|
||||
- read header row → column names, count, order;
|
||||
- infer per-column datatype from first/last N rows + a random sample;
|
||||
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
|
||||
|
||||
**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
|
||||
|
||||
### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
|
||||
|
||||
```
|
||||
for each file in walk(root):
|
||||
meta = fs_metadata(file)
|
||||
tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere
|
||||
header = read_first_line(file)
|
||||
columns = parse_columns(header)
|
||||
dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M))
|
||||
fp = fingerprint(columns) # ordered-set hash
|
||||
emit_manifest_row(meta, tokens, columns, dtypes, fp)
|
||||
```
|
||||
|
||||
### 3.4 Artifacts
|
||||
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
|
||||
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
|
||||
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
|
||||
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
|
||||
|
||||
### 3.5 Exit criteria
|
||||
- 100% of files inventoried and fingerprinted.
|
||||
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
|
||||
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
|
||||
|
||||
**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
|
||||
|
||||
The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
|
||||
|
||||
### 4.1 The iteration loop (per measurement type)
|
||||
|
||||
1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
|
||||
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
|
||||
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
|
||||
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
|
||||
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
|
||||
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
|
||||
|
||||
### 4.2 Instruments Repository (ontology sub-model)
|
||||
|
||||
Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
|
||||
- identity (type, model/label, measurement type it produces);
|
||||
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
|
||||
- association to `measurement_types` / `column_definitions` in the data dictionary.
|
||||
|
||||
Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
|
||||
|
||||
### 4.3 Anomaly detection
|
||||
|
||||
Criteria are **derived from the data**, not fixed a priori. Starting catalog:
|
||||
|
||||
| Class | Description | Source of criterion |
|
||||
|-------|-------------|---------------------|
|
||||
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
|
||||
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
|
||||
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
|
||||
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
|
||||
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
|
||||
| Missing required | Null in a non-nullable canonical field | Schema constraints |
|
||||
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
|
||||
|
||||
Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
|
||||
|
||||
### 4.4 Data dictionary / metadata catalog
|
||||
|
||||
Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
|
||||
|
||||
### 4.5 Artifacts
|
||||
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
|
||||
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
|
||||
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
|
||||
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
|
||||
- **`instruments_repository`** — instrument entities + characteristics + ranges.
|
||||
- **`rejects_store`** — quarantined anomalies with reason codes.
|
||||
- **`ontology_notes`** — running record of ERD changes and rationale.
|
||||
|
||||
### 4.6 Exit criteria
|
||||
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
|
||||
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
|
||||
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 4 — Conversion & Export
|
||||
|
||||
**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
|
||||
|
||||
### 5.1 Targets
|
||||
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
|
||||
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
|
||||
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
|
||||
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
|
||||
5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
|
||||
|
||||
### 5.2 Design
|
||||
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
|
||||
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
|
||||
- Lineage is emitted alongside every export (see §6).
|
||||
|
||||
### 5.3 Artifacts
|
||||
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
|
||||
|
||||
### 5.4 Exit criteria
|
||||
- All five target formats generated from one canonical snapshot.
|
||||
- Round-trip validation: record counts and key aggregates match across formats.
|
||||
- Lineage complete and resolvable for sampled records.
|
||||
|
||||
---
|
||||
|
||||
## 6. Cross-cutting concerns
|
||||
|
||||
### 6.1 Lineage & processing-flow documentation
|
||||
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
|
||||
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
|
||||
- Lineage is a first-class export, present in every target format.
|
||||
|
||||
### 6.2 Configuration & rule registry
|
||||
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
|
||||
|
||||
### 6.3 Validation gates
|
||||
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
|
||||
|
||||
---
|
||||
|
||||
## 7. Target artifacts catalog (consolidated)
|
||||
|
||||
| Artifact | Phase | Form |
|
||||
|----------|-------|------|
|
||||
| storage_manifest | 1 | JSONL + SQLite |
|
||||
| schema_fingerprint_report | 1 | JSON/CSV |
|
||||
| unmatched_report | 1 | CSV |
|
||||
| storage_map | 1 | Markdown |
|
||||
| column_mapping | 2↔3 | CSV/JSON (versioned) |
|
||||
| canonical_dataset | 2↔3 | partitioned intermediate |
|
||||
| relational_schema (DDL) | 2↔3 | SQL |
|
||||
| data_dictionary | 2↔3 | table/CSV |
|
||||
| instruments_repository | 2↔3 | table/CSV |
|
||||
| rejects_store | 2↔3 | JSONL |
|
||||
| ontology_notes | 2↔3 | Markdown |
|
||||
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
|
||||
| lineage + build manifest | all | JSONL |
|
||||
|
||||
---
|
||||
|
||||
## 8. Tooling & environment
|
||||
|
||||
- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
|
||||
- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
|
||||
- **PostgreSQL** materialized in Phase 4 (system of record).
|
||||
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
|
||||
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
|
||||
- Python indentation: **tabs**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Performance strategy for 700 GB
|
||||
|
||||
- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
|
||||
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
|
||||
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
|
||||
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
|
||||
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
|
||||
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open items & risks
|
||||
|
||||
1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
|
||||
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
|
||||
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
|
||||
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
|
||||
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
|
||||
|
||||
---
|
||||
|
||||
*End of Plan v1.*
|
||||
188
docs/examples/Tribology_Ontology.html
Normal file
188
docs/examples/Tribology_Ontology.html
Normal file
File diff suppressed because one or more lines are too long
112
docs/rules/bench-methodology.md
Normal file
112
docs/rules/bench-methodology.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# Bench / Methodology - isolation, repetitions, timeouts, honest reporting
|
||||
|
||||
**Purpose + scope.** The measurement discipline for the benchmark matrix
|
||||
(Q1-Q7 x 5 formats, spec [08](../specs/08_benchmark_queries.md) /
|
||||
[09](../specs/09_benchmark_execution.md)) and for the extrapolation and
|
||||
reporting built on it (specs 10-11). Binding for `bench_runner.py`, every
|
||||
query implementation, `extrapolate.py`, and `report.py`.
|
||||
|
||||
The product of this project IS these measurements; a sloppy protocol
|
||||
invalidates everything downstream.
|
||||
|
||||
---
|
||||
|
||||
## 1. Cell isolation - one subprocess per measurement
|
||||
|
||||
- Every benchmark cell (format x query x run) executes in its OWN
|
||||
subprocess, so peak RSS, CPU time, and IO counters attribute to exactly
|
||||
that cell.
|
||||
- The runner samples the child with `psutil` every 50 ms for peak RSS;
|
||||
CPU time is user+sys at exit; read bytes from `io_counters`.
|
||||
- Nothing else runs in the child: no logging setup beyond minimum, no
|
||||
imports the query does not need - the measurement must not pay for the
|
||||
harness.
|
||||
|
||||
## 2. Repetition protocol
|
||||
|
||||
- Per cell: **1 warm-up run (discarded) + 3 measured runs.** Report the
|
||||
median in `results_median.csv`; keep min/max in the raw data.
|
||||
- **Randomize the cell execution order** and record the actual order in
|
||||
the raw results - fixed order lets cache effects masquerade as format
|
||||
differences.
|
||||
- SQLite additionally reports run 1 (cold) and run 3 (warm) separately
|
||||
(spec 08).
|
||||
- Never average TIMEOUT/ERROR runs into medians; a cell with a failed run
|
||||
reports the failure, not a prettier subset.
|
||||
|
||||
## 3. Timeouts and failures are data
|
||||
|
||||
- Hard per-run timeout: **30 minutes.** On expiry the runner records
|
||||
`TIMEOUT` for that cell and CONTINUES the matrix (an expected outcome:
|
||||
RDF Q5). Never retry a timeout hoping for a better number.
|
||||
- A crashed cell records `ERROR` the same way (see
|
||||
[code-error-handling.md](code-error-handling.md) pattern C).
|
||||
- In extrapolation and scoring, TIMEOUT/FAIL project to score 0 - they
|
||||
are never silently dropped from tables or charts (spec 10/11 flags:
|
||||
projection > 1 h = `IMPRACTICAL`, > 24 h = `FAIL`).
|
||||
|
||||
## 4. Result correctness gates the measurement
|
||||
|
||||
- Every measured run validates its result rows against the canonical
|
||||
checksum (sorted rows, float tolerance 1e-9) and records `checksum_ok`
|
||||
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
|
||||
- A fast run with `checksum_ok = false` is an INVALID measurement - it
|
||||
must never enter medians, extrapolation, or the report.
|
||||
- **Q5 reads raw `friction_cycles` in every format** - summary tables and
|
||||
materialized views are forbidden for it; verify PostgreSQL did not
|
||||
satisfy it from `track_summary`
|
||||
(`EXPLAIN (ANALYZE, BUFFERS)` is recorded for PostgreSQL runs anyway).
|
||||
|
||||
## 5. Fairness across formats
|
||||
|
||||
- Identical result-set contract for all 5 implementations of each query;
|
||||
no format answers a simplified question.
|
||||
- Each format uses its idiomatic strength honestly: CSV Q1 reads the one
|
||||
target file by path; JSON Q1 loads the one coupon file; SQL uses its
|
||||
indexes. What is forbidden is capping or sampling (`LIMIT`, partial
|
||||
scans) that changes the answer.
|
||||
- The pandas CSV variant is measured as a SEPARATE variant, never mixed
|
||||
into the plain-CSV numbers (spec 08).
|
||||
- Storage sizes compare like with like: for every format record both the
|
||||
working footprint and the gzip archival footprint in
|
||||
`storage_sizes.csv` (spec 09).
|
||||
|
||||
## 6. Environment honesty - this machine, stated limits
|
||||
|
||||
- Record the hardware and software environment (CPU model, cores, RAM,
|
||||
disk type, OS, engine versions) once per benchmark session into the
|
||||
raw results; the report cites it.
|
||||
- **Cold-cache protocol on this host:** the spec's page-cache drop
|
||||
(`/proc/sys/vm/drop_caches`) is Linux-only; this project runs on
|
||||
Windows Server 2025, where it is unavailable. The cold-cache matrix
|
||||
pass is therefore SKIPPED and marked as such in results and report -
|
||||
never approximated by a fake (e.g. file re-copy tricks) without a
|
||||
rules update describing the method.
|
||||
- Quiesce the machine during measured runs (no parallel pipeline tasks,
|
||||
no background loads); the runner runs cells sequentially by design.
|
||||
|
||||
## 7. Extrapolation discipline (task 10)
|
||||
|
||||
- Scaling laws per access pattern are those fixed in spec 10 (O(1),
|
||||
O(log n + k), O(k log n), O(n), O(n / min(cores, partitions)));
|
||||
constants calibrate on the measured 150 MB point.
|
||||
- Every projected number in the report is labeled as projected, with its
|
||||
scaling assumption; measured and projected values never mix in one
|
||||
column unlabeled.
|
||||
- Hardware cost figures come from `hw_prices.yaml`
|
||||
(see [code-config-yaml.md](code-config-yaml.md)), never inlined.
|
||||
|
||||
## 8. Anti-patterns
|
||||
|
||||
- **Running cells in-process** - RSS/CPU attribution becomes fiction.
|
||||
- **Fixed execution order** - cache warmth leaks between formats.
|
||||
- **Retrying or dropping slow runs** - the slow run IS the result.
|
||||
- **Averaging over a TIMEOUT** or reporting medians of fewer than 3 valid
|
||||
runs without flagging it.
|
||||
- **A "fast" result that fails the checksum** entering any downstream
|
||||
table.
|
||||
- **Tuning one format's query while leaving others naive** - fairness is
|
||||
per-scenario, not per-format-champion.
|
||||
- **Faking cold-cache on Windows** without a documented, rules-approved
|
||||
method.
|
||||
- **Presenting projected numbers as measured** in tables or charts.
|
||||
103
docs/rules/build-pipeline-tasks.md
Normal file
103
docs/rules/build-pipeline-tasks.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Build / Pipeline tasks - the ./out tree, .done markers, task order
|
||||
|
||||
**Purpose + scope.** How the 11 pipeline tasks compose: where artifacts
|
||||
live, how completion is signaled, what order tasks run in, and what git
|
||||
does and does not track. Binding for every task implementation and for
|
||||
anyone running the pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 1. The artifact tree - everything under `./out/`
|
||||
|
||||
All generated artifacts live under `./out/` relative to the repo root
|
||||
(spec 00_PLAN):
|
||||
|
||||
```
|
||||
out/
|
||||
config/ lab_config.yaml, hw_prices.yaml
|
||||
csv/ the canonical raw corpus + MANIFEST.csv
|
||||
json/ full/ (per-coupon JSON-LD), hybrid/dataset.jsonld
|
||||
sqlite/ tribo.db
|
||||
pg/ tribo.dump (pg_dump -Fc; the live DB is in PostgreSQL)
|
||||
rdf/ dataset.nt.gz, dataset.ttl, oxigraph_store/
|
||||
bench/ queries/, expected/, results_raw.csv, results_median.csv,
|
||||
storage_sizes.csv, extrapolation.csv, hardware_sizing.csv
|
||||
report/ REPORT.md, charts/, tables/, process_flow.md, diagrams/
|
||||
.done/ completion markers, one per task
|
||||
```
|
||||
|
||||
- A task writes ONLY under its own output paths; it treats its inputs as
|
||||
read-only.
|
||||
- Tasks communicate exclusively through these artifacts - never through
|
||||
shared in-process state
|
||||
(see [code-python-style.md](code-python-style.md)).
|
||||
- **`./out/` is git-ignored** (except nothing - the whole tree). Sources
|
||||
of truth are the specs, the rules, and the code; artifacts are
|
||||
reproducible from them ([data-determinism.md](data-determinism.md)).
|
||||
Never commit generated artifacts "for convenience".
|
||||
|
||||
## 2. Completion markers
|
||||
|
||||
- Each task NN writes `./out/.done/<NN>.ok` after - and only after - its
|
||||
outputs are complete and validated. The marker body carries the
|
||||
headline counts (rows, bytes, triples) downstream tasks check
|
||||
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
|
||||
- A task verifies its dependencies' markers before starting and aborts
|
||||
loudly if one is missing.
|
||||
- Re-running a task: delete its own marker first, regenerate outputs,
|
||||
rewrite the marker. If its outputs changed, downstream markers are now
|
||||
stale - delete them too (the dependency graph below says which).
|
||||
|
||||
## 3. Task graph and execution order
|
||||
|
||||
```
|
||||
01 -> 02
|
||||
01 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11
|
||||
09 ------> 11
|
||||
```
|
||||
|
||||
- Sequential order: 01, 02, 03, then 04-07 (independent, may run in
|
||||
parallel), 08, 09, 10, 11.
|
||||
- 02 (process diagrams) needs only 01 and can run any time after it.
|
||||
- Task 09 (benchmark execution) must run WITHOUT 04-07 style parallel
|
||||
work in the background - measurements need a quiet machine
|
||||
([bench-methodology.md](bench-methodology.md) section 6).
|
||||
|
||||
## 4. `storage_sizes.csv` - the shared append contract
|
||||
|
||||
- Location: `./out/bench/storage_sizes.csv`; columns:
|
||||
`format, variant, bytes`.
|
||||
- Tasks 04-07 APPEND their format's rows (e.g. `json,full,...`,
|
||||
`json,hybrid,...`, `sqlite,db,...`, `pg,live,...`, `pg,dump,...`,
|
||||
`rdf,nt_gz,...`, `rdf,store,...`); task 09 appends the gzip archival
|
||||
variants and consolidates.
|
||||
- Appends are idempotent per (format, variant): a re-run REPLACES its own
|
||||
prior rows, never duplicates them and never touches other formats'
|
||||
rows.
|
||||
- RDF load wall-time and similar per-format load metrics recorded per
|
||||
spec 07 go into the task's `.done` marker and the report, not into
|
||||
extra ad-hoc files.
|
||||
|
||||
## 5. Runs are resumable, not restartable-from-zero
|
||||
|
||||
- The marker system exists so a failed pipeline resumes at the failed
|
||||
task; do not delete the whole `./out/` tree to retry one task.
|
||||
- Conversely, after a deliberate corpus regeneration (03), ALL downstream
|
||||
markers and artifacts (04-11) are stale and must be removed - a mixed
|
||||
tree of old and new artifacts is worse than an empty one
|
||||
([meta-no-assumptions.md](meta-no-assumptions.md) applies before any
|
||||
such deletion).
|
||||
|
||||
## 6. Anti-patterns
|
||||
|
||||
- **Writing an artifact outside `./out/`** or a task writing into another
|
||||
task's output directory.
|
||||
- **Committing `./out/` content to git.**
|
||||
- **A marker written on partial success** - see
|
||||
[code-error-handling.md](code-error-handling.md).
|
||||
- **Running benchmarks while converters are still running.**
|
||||
- **Duplicate rows in `storage_sizes.csv`** after re-runs - replace your
|
||||
own rows.
|
||||
- **Hand-editing generated artifacts** (a "quick fix" inside
|
||||
`MANIFEST.csv` or a result CSV) - regenerate them through the owning
|
||||
task instead.
|
||||
61
docs/rules/code-comments-policy.md
Normal file
61
docs/rules/code-comments-policy.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Code comments - write WHY, not WHAT
|
||||
|
||||
**Binding** for all source files: Python, SQL, SPARQL, PowerShell, YAML, shell.
|
||||
|
||||
## Rule
|
||||
|
||||
Default to **no comment**. Add one only when the *why* is non-obvious from
|
||||
the code. If removing the comment wouldn't confuse a future reader, delete it.
|
||||
|
||||
A comment earns its place when it captures something the code can't:
|
||||
hidden constraint, ordering invariant, workaround for a named bug, vendor
|
||||
quirk, cross-file invariant the type system can't enforce.
|
||||
|
||||
## Delete on sight
|
||||
|
||||
```python
|
||||
# Read the manifest file
|
||||
manifest = read_manifest(csv_root) # function name says it
|
||||
```
|
||||
|
||||
```python
|
||||
# Increment counter
|
||||
counter += 1 # pure restatement
|
||||
```
|
||||
|
||||
```python
|
||||
# Added for the benchmark rework (see task 09)
|
||||
def run_cell(...): # belongs in commit message, rots in code
|
||||
```
|
||||
|
||||
```python
|
||||
# TODO: refactor later
|
||||
# no owner, no trigger - decoration
|
||||
```
|
||||
|
||||
## Keep
|
||||
|
||||
```python
|
||||
# Q5 must scan raw friction_cycles - track_summary is forbidden for this
|
||||
# query in every format (spec 08); do not "optimize" this into the summary.
|
||||
```
|
||||
Cross-file invariant a refactor must preserve.
|
||||
|
||||
```python
|
||||
# Column order matches MANIFEST.csv writer in generate_data.py - keep in sync.
|
||||
```
|
||||
Cross-file invariant the type system can't enforce.
|
||||
|
||||
```sql
|
||||
-- BRIN, not b-tree: cycle values are physically ordered within each
|
||||
-- partition, so BRIN gives the same pruning at ~1% of the index size.
|
||||
```
|
||||
Non-obvious engineering decision with a reason.
|
||||
|
||||
## Length
|
||||
|
||||
Inline: one line. Block: 1-4 lines. If you need more than a paragraph,
|
||||
the right home is a rule or spec doc - link it from a one-line comment.
|
||||
|
||||
Stale comments are worse than missing ones. When you touch a file, fix or
|
||||
delete comments whose *why* no longer holds.
|
||||
94
docs/rules/code-config-yaml.md
Normal file
94
docs/rules/code-config-yaml.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# Code / Config - all tunables live in YAML, no magic numbers, no .env
|
||||
|
||||
**Purpose + scope.** Governs where every configurable value lives. Binding
|
||||
for all Python code in this repository. Any new configurable parameter MUST
|
||||
follow this rule when deciding where it goes.
|
||||
|
||||
The goal: one authoritative source per parameter, so regenerating any
|
||||
artifact with the same config is byte-identical, and changing a parameter
|
||||
never requires hunting constants across scripts.
|
||||
|
||||
---
|
||||
|
||||
## 1. The two YAML configs
|
||||
|
||||
| File | Owns | Written by |
|
||||
|---|---|---|
|
||||
| `./out/config/lab_config.yaml` | The laboratory model: volumes (batches / wafers / coupons / tracks / cycles), the deposition matrix, physical-model coefficients, nominal test conditions, **the random seed 20260711** | Task 01, from the spec; regenerated only deliberately |
|
||||
| `./out/config/hw_prices.yaml` | Hardware cost parameters for extrapolation (RAM $/GB, NVMe $/TB, server base prices) | Task 10 seeds editable defaults; the operator may edit |
|
||||
|
||||
Rules:
|
||||
|
||||
- **Scripts READ these files; they never duplicate values from them.**
|
||||
Task 03 reads volumes, coefficients, and the seed from
|
||||
`lab_config.yaml` - it does not hardcode 588 coupons or seed 20260711.
|
||||
Task 10 reads prices from `hw_prices.yaml` - it does not inline dollar
|
||||
figures.
|
||||
- A parameter needed by more than one task lives in the YAML, never
|
||||
copy-pasted between scripts.
|
||||
- New tunables go into the appropriate YAML with a self-describing key;
|
||||
if neither file fits, propose a new config file in a docs/rules update
|
||||
first.
|
||||
|
||||
## 2. Spec-fixed constants - named, sourced, at the top
|
||||
|
||||
Values fixed by the specs that are NOT operator-tunable (the 120-200 MB
|
||||
size gate, the 30-minute cell timeout, the 50k-row transaction batch, the
|
||||
50 ms RSS sampling interval) are module-level named constants at the top of
|
||||
the owning script, each with a one-line comment naming the spec:
|
||||
|
||||
```python
|
||||
SIZE_GATE_MB = (120, 200) # abort bounds, spec 03
|
||||
CELL_TIMEOUT_S = 30 * 60 # hard per-run timeout, spec 09
|
||||
```
|
||||
|
||||
A bare number inline in the logic (`if size > 209715200`) is forbidden.
|
||||
|
||||
## 3. The PostgreSQL DSN - one env var, never hardcoded, never logged
|
||||
|
||||
The only external resource is the local PostgreSQL 16 instance. Its DSN:
|
||||
|
||||
- Comes from the environment variable **`TRIBO_PG_DSN`**; default when
|
||||
unset: `postgresql://postgres@localhost:5432/tribo`.
|
||||
- Is read in ONE shared helper (e.g. `common/pg.py: get_dsn()`); scripts
|
||||
call the helper, never `os.environ` directly.
|
||||
- Is never hardcoded in a script, never committed in a config file, and
|
||||
never logged (a DSN may embed a password - see
|
||||
[obs-python-logging.md](obs-python-logging.md)).
|
||||
|
||||
There are no other secrets in this project. There is no `.env` file; do
|
||||
not introduce one.
|
||||
|
||||
## 4. CLI flags - per-run choices only
|
||||
|
||||
Command-line flags are for choices that vary per invocation, not for
|
||||
model parameters: `--log-level`, `--dry-run`, an output-root override, a
|
||||
`--limit` for smoke runs. A physical coefficient or a volume as a CLI flag
|
||||
is forbidden - it would silently fork the config out of `lab_config.yaml`.
|
||||
|
||||
## 5. Anti-patterns
|
||||
|
||||
- **Hardcoding a value that exists in `lab_config.yaml`** (seed, volumes,
|
||||
coefficients) - the YAML is the single source; drift here breaks
|
||||
reproducibility silently.
|
||||
- **Adding a `.env` file or reading `os.environ` scattered through
|
||||
scripts** - the DSN goes through the one shared helper; nothing else
|
||||
lives in the environment.
|
||||
- **Inline magic numbers** for spec-fixed gates and timeouts - name them
|
||||
at the top with the spec reference.
|
||||
- **Dollar prices inlined in `extrapolate.py`** - they live in
|
||||
`hw_prices.yaml` so the operator can re-cost without touching code.
|
||||
- **A model parameter as a CLI flag** - per-run flags are for run
|
||||
mechanics, not for the lab model.
|
||||
- **Committing `./out/config/` generated files to git as if they were
|
||||
sources** - the YAML under `./out/` is a generated artifact; its source
|
||||
of truth is spec 01 (see
|
||||
[build-pipeline-tasks.md](build-pipeline-tasks.md)).
|
||||
|
||||
## 6. Related documents
|
||||
|
||||
- [data-determinism.md](data-determinism.md) - the seed contract.
|
||||
- [build-pipeline-tasks.md](build-pipeline-tasks.md) - the `./out/` tree
|
||||
and what is git-ignored.
|
||||
- [obs-python-logging.md](obs-python-logging.md) - `--log-level` flag; DSN
|
||||
never logged.
|
||||
132
docs/rules/code-data-formatting.md
Normal file
132
docs/rules/code-data-formatting.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# Code / Data / Formatting - ASCII-only typographic characters in source files
|
||||
|
||||
**Binding scope.** Strings that end up in any of the following MUST use ASCII
|
||||
typographic characters; never their Unicode "smart" equivalents:
|
||||
|
||||
- Python source, SQL and SPARQL query files
|
||||
- Generated data and artifacts written to disk (CSV, JSON-LD, N-Triples,
|
||||
Turtle, MANIFEST and benchmark result CSVs, log messages)
|
||||
- Configuration files (`.yaml`, `.json`, `.toml`)
|
||||
- PowerShell / shell scripts (any file processed by Windows tooling)
|
||||
- Markdown documentation (`docs/`, `README.md`, `CLAUDE.md`) - the specs
|
||||
were normalized to ASCII typography on 2026-07-11 and stay that way
|
||||
|
||||
**Long dashes are an absolute exception, no carve-outs.** Em-dash (U+2014)
|
||||
and en-dash (U+2013) MUST be replaced with ASCII hyphen-minus (`-`) in
|
||||
**every** file in this repository, including markdown documentation, code
|
||||
comments, and report prose. Reason: long dashes are visually
|
||||
indistinguishable from a hyphen in many monospace fonts, break diffs / grep /
|
||||
IDE search, and round-trip badly through Windows tooling chains (section 2).
|
||||
|
||||
---
|
||||
|
||||
## 1. The substitution table
|
||||
|
||||
| Unicode char | Codepoint | Name | ASCII replacement |
|
||||
|---|---|---|---|
|
||||
| em-dash | U+2014 | em-dash | `-` (single hyphen-minus) |
|
||||
| en-dash | U+2013 | en-dash | `-` (single hyphen-minus) |
|
||||
| horizontal bar | U+2015 | horizontal bar | `-` |
|
||||
| minus sign | U+2212 | minus sign | `-` |
|
||||
| ellipsis | U+2026 | horizontal ellipsis | `...` (three ASCII dots) |
|
||||
| left double quote | U+201C | left double quotation mark | `"` |
|
||||
| right double quote | U+201D | right double quotation mark | `"` |
|
||||
| left single quote | U+2018 | left single quotation mark | `'` |
|
||||
| right single quote | U+2019 | right single quotation mark | `'` |
|
||||
| left guillemet | U+00AB | left guillemet | `"` |
|
||||
| right guillemet | U+00BB | right guillemet | `"` |
|
||||
| nbsp | U+00A0 | non-breaking space | regular space |
|
||||
|
||||
**Single hyphen, not double.** When replacing an em-dash that separates two
|
||||
clauses, use a single `-` surrounded by spaces (`text - clause`), NOT `--`.
|
||||
|
||||
**Ranges use a single hyphen:** `Q1-Q7`, `120-200 MB`, `0.3-1.1 um` - never
|
||||
an en-dash.
|
||||
|
||||
## 2. Why - documented failure modes
|
||||
|
||||
1. **PowerShell 5.1 reads no-BOM files as Windows-1252.** Non-ASCII UTF-8
|
||||
bytes become mojibake and break string-literal parsing. This project is
|
||||
developed on Windows; any script or config touched by PowerShell is
|
||||
exposed.
|
||||
|
||||
2. **Windows -> git-bash -> tar / packaging round-trip.** At any step an
|
||||
unset locale or wrong encoding assumption produces mojibake. A UTF-8 byte
|
||||
sequence for U+2014 interpreted under a non-UTF-8 locale renders as a
|
||||
`?` replacement character in the packaged output.
|
||||
|
||||
3. **Copy-paste from Word / Google Docs / Notion / web articles.** These
|
||||
tools auto-substitute typography (`"` -> smart quotes, `'` -> smart
|
||||
apostrophe, `--` -> em-dash, `...` -> ellipsis) silently. The
|
||||
substitution survives copy into any editor and lands in source files
|
||||
unnoticed. The original specs arrived with en/em dashes this way.
|
||||
|
||||
4. **Data files must diff and grep cleanly.** The CSV corpus, MANIFEST, and
|
||||
benchmark results are compared by checksum and processed by streaming
|
||||
parsers; a stray typographic character breaks byte-level reproducibility.
|
||||
|
||||
## 3. Where the rule does NOT apply
|
||||
|
||||
- **Scientific and unit notation is content, not typography.** Characters
|
||||
such as `µ` (micro), `±`, `°C`, `×`, `Ø`, and Greek letters (`τ`, `σ`) in
|
||||
formulas inside the specs are meaningful scientific notation and are kept.
|
||||
Note the boundary: in **column names and code identifiers** the units are
|
||||
always plain ASCII (`_um`, `_nm`, `_GPa` - see
|
||||
[data-naming-units.md](data-naming-units.md)); the scientific characters
|
||||
live only in explanatory prose.
|
||||
- **International language strings** (Cyrillic, CJK, etc.) are meaningful
|
||||
content, not typographic substitutions. UTF-8 throughout is the right
|
||||
answer. (Repository artifacts are English; this matters only for quoted
|
||||
material.)
|
||||
- **Imported reference materials** in `docs/examples/` are kept verbatim as
|
||||
received; they are not binding documents.
|
||||
|
||||
## 4. Detection / enforcement
|
||||
|
||||
There is no automated check for this rule yet - **not in CI yet**. The rule is
|
||||
enforced by review and by the editing discipline in this file. The intent of a
|
||||
future detector is documented here so it can be built consistently:
|
||||
|
||||
- **What it flags.** Any occurrence of the Unicode codepoints in the section 1
|
||||
substitution table inside in-scope files. The section 3 exemptions
|
||||
(scientific notation, international content, `docs/examples/`) are the only
|
||||
carve-outs.
|
||||
- **Exit behavior.** Non-zero exit when any flagged codepoint is found under
|
||||
the scan roots, so it can later gate a commit hook or a build step.
|
||||
- **Tooling.** When this detector is created it MUST be a PowerShell script
|
||||
(this is a Windows project) and live under `tools/`.
|
||||
|
||||
## 5. Examples
|
||||
|
||||
### Good
|
||||
|
||||
```python
|
||||
# safe through every Windows tooling step
|
||||
print("Corpus size 152.3 MB - within the 120-200 MB gate.")
|
||||
label = "steady-state COF, cycles 500-1000"
|
||||
```
|
||||
|
||||
### Bad
|
||||
|
||||
```python
|
||||
# em-dash literal (U+2014) - mojibake risk on PowerShell read/write
|
||||
print("Corpus size 152.3 MB [U+2014] within the gate.")
|
||||
|
||||
# en-dash range (U+2013) - breaks grep for "Q1-Q7"
|
||||
label = "queries Q1[U+2013]Q7"
|
||||
```
|
||||
|
||||
## 6. Anti-patterns
|
||||
|
||||
- **Copy-pasting strings from a word processor or web page** into source /
|
||||
specs / config without normalization.
|
||||
- **Trusting tar / packaging to "preserve as-is".** They preserve bytes; the
|
||||
*encoding interpretation* is what gets lost when locale or encoding
|
||||
declaration is wrong on either end.
|
||||
- **Assuming the source file encoding alone is enough.** The source file may
|
||||
be fine - the failure is at downstream consumers (PowerShell read, build
|
||||
write, OS locale).
|
||||
- **Using `--` instead of `-` "for emphasis".** Single ASCII hyphen
|
||||
surrounded by spaces is the project's chosen separator.
|
||||
- **Putting `µ` or `°` into a column name or identifier** - units in names
|
||||
are ASCII (`_um`, `_C`); see [data-naming-units.md](data-naming-units.md).
|
||||
181
docs/rules/code-error-handling.md
Normal file
181
docs/rules/code-error-handling.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# Code / Error handling - never swallow, always surface, fail loud
|
||||
|
||||
**Purpose + scope.** Governs every `try:` / `except:` / `raise` site in
|
||||
Python code in this repository (data generation, format converters, benchmark
|
||||
runner, extrapolation, reporting, helper tools). Binding for all contributors.
|
||||
|
||||
The pipeline's core failure principle: **a wrong artifact is worse than no
|
||||
artifact.** Every task's output feeds downstream tasks; silently degraded
|
||||
data poisons every result built on it. When in doubt, abort with a clear
|
||||
message and a non-zero exit code.
|
||||
|
||||
---
|
||||
|
||||
## 1. The rule
|
||||
|
||||
1. **Never `except: pass`.** A bare swallow is always a bug. Forbidden
|
||||
absolutely.
|
||||
2. **Never `except Exception: logger.warning("...") + continue`** for
|
||||
anything that was not an EXPECTED, documented, transient condition. If
|
||||
you swallow and continue, you must be able to state why the program can
|
||||
correctly proceed.
|
||||
3. **Always `logger.error(..., exc_info=True)` in `except` blocks.**
|
||||
`exc_info=True` captures the traceback into the log record (see
|
||||
[obs-python-logging.md](obs-python-logging.md)). Without it the original
|
||||
exception is lost.
|
||||
4. **Re-raise unless you have a defined recovery path.** If the caller
|
||||
cannot continue meaningfully without the failed operation, re-raise
|
||||
after logging.
|
||||
5. **Surface to the right channel:**
|
||||
- **CLI script (every pipeline task):** print the error to stderr and
|
||||
exit non-zero. A task that failed MUST NOT write its
|
||||
`./out/.done/<task>.ok` marker (see
|
||||
[build-pipeline-tasks.md](build-pipeline-tasks.md)).
|
||||
- **Validation gate (row counts, checksums, size bounds):** a failed
|
||||
gate is an abort, not a warning. Example: task 03 aborts when the
|
||||
generated tree is outside 120-200 MB; converters abort when row counts
|
||||
do not match `MANIFEST.csv`.
|
||||
- **Library / helper function:** re-raise; let the caller decide.
|
||||
6. **Defensive "this cannot happen" branches must surface, not silently
|
||||
`return`.** If you reach a defensive branch, log at ERROR with the
|
||||
branch name and the inputs that got you there, then raise.
|
||||
|
||||
## 2. The exception ladder - the only three sanctioned swallow patterns
|
||||
|
||||
Code may swallow ONLY in these three documented patterns. Anything else
|
||||
re-raises after logging.
|
||||
|
||||
### Pattern A: documented, recoverable transient (retry)
|
||||
|
||||
```python
|
||||
for attempt in range(MAX_RETRIES):
|
||||
try:
|
||||
return do_thing()
|
||||
except (psycopg.OperationalError, TimeoutError) as exc:
|
||||
logger.warning("attempt %d/%d failed: %s", attempt + 1, MAX_RETRIES, exc)
|
||||
time.sleep(backoff(attempt))
|
||||
# All retries exhausted - surface.
|
||||
logger.error("do_thing failed after %d retries", MAX_RETRIES)
|
||||
raise
|
||||
```
|
||||
|
||||
Justified: the per-attempt warning is acceptable; the final failure
|
||||
re-raises.
|
||||
|
||||
### Pattern B: optional best-effort cleanup
|
||||
|
||||
```python
|
||||
def _close_conn(conn) -> None:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
# Cleanup is best-effort; the operation already finished, log and continue.
|
||||
logger.warning("closing connection failed", exc_info=True)
|
||||
```
|
||||
|
||||
Justified: the comment names why continuing is correct; the traceback is
|
||||
still captured.
|
||||
|
||||
### Pattern C: benchmark-cell isolation - record and continue the matrix
|
||||
|
||||
```python
|
||||
try:
|
||||
result = run_cell(fmt, query, timeout_s=30 * 60)
|
||||
except CellTimeout:
|
||||
logger.error("cell %s/%s timed out", fmt, query)
|
||||
write_result_row(fmt, query, status="TIMEOUT") # recorded, not hidden
|
||||
except Exception:
|
||||
logger.error("cell %s/%s failed", fmt, query, exc_info=True)
|
||||
write_result_row(fmt, query, status="ERROR")
|
||||
```
|
||||
|
||||
Justified: spec 09 requires the benchmark matrix to survive a failing cell
|
||||
(RDF Q5 is expected to time out). The failure is RECORDED as data
|
||||
(`TIMEOUT` / `ERROR` in `results_raw.csv`), never silently dropped, and the
|
||||
run continues with the next cell. This pattern applies ONLY to the benchmark
|
||||
runner's per-cell boundary - a failure in the runner itself (setup, result
|
||||
writing) still aborts.
|
||||
|
||||
## 3. Loading into SQLite / PostgreSQL - all-or-nothing per table
|
||||
|
||||
A converter that half-loads a table leaves a database that LOOKS complete
|
||||
and silently skews every benchmark run against it.
|
||||
|
||||
1. Wrap each table's bulk load in a transaction; on any error roll back
|
||||
and abort the converter with a non-zero exit.
|
||||
2. After loading, validate row counts against `MANIFEST.csv` and fail loud
|
||||
on any mismatch (see
|
||||
[test-pipeline-validation.md](test-pipeline-validation.md)).
|
||||
3. Never write the task's `.done` marker after a partial load.
|
||||
|
||||
## 4. Anti-patterns
|
||||
|
||||
- **`except: pass`** - forbidden absolutely.
|
||||
- **`except Exception: logger.warning("could not X")` + continue** -
|
||||
forbidden unless the function documents that "could not X" is a
|
||||
recoverable expected outcome. If it is, phrase it "X skipped (expected
|
||||
condition Y)" so a reader does not grep "could not" hunting for bugs.
|
||||
- **Catching `BaseException`** - forbidden (swallows `KeyboardInterrupt`,
|
||||
`SystemExit`). Use `Exception`.
|
||||
- **`logger.error("X")` without `exc_info=True` in an `except` block** -
|
||||
forbidden. The traceback IS the diagnostic.
|
||||
- **`logger.error(str(exc))`** - forbidden; redundant with `exc_info=True`
|
||||
and loses type and stack.
|
||||
- **Returning `None` to signal failure** from a function whose contract is
|
||||
"return T" - forbidden. Either raise, or return `T | None` with
|
||||
documented semantics the caller checks.
|
||||
- **`raise Exception("...")`** - forbidden. Use a specific class:
|
||||
`ValueError` / `RuntimeError` / `KeyError` for trivial cases, a custom
|
||||
domain exception subclass (e.g. `ManifestMismatchError`) for domain
|
||||
conditions.
|
||||
- **`raise` chained without context** - inside an `except`, prefer
|
||||
`raise NewError(...) from exc` so the original cause stays visible in the
|
||||
chained traceback. A bare `raise` that re-raises the same exception is
|
||||
fine; `raise SomethingElse(...)` without `from exc` is forbidden.
|
||||
- **Writing a `.done` marker in a `finally` block** - the marker means
|
||||
"completed successfully", never "attempted".
|
||||
- **Clamping / repairing out-of-range generated values silently** - a
|
||||
physical-model bug must surface, not be masked by a `max(0, ...)` added
|
||||
"to be safe". Deliberate clamps defined by the spec (e.g. COF floor 0.12)
|
||||
are model logic, not error handling.
|
||||
|
||||
## 5. Defensive branches
|
||||
|
||||
```python
|
||||
# Good:
|
||||
def open_store(fmt: str) -> Store:
|
||||
if fmt == "sqlite":
|
||||
return SqliteStore()
|
||||
if fmt == "postgresql":
|
||||
return PgStore()
|
||||
# Defensive: a new format added without updating this factory lands here.
|
||||
logger.error("open_store: unknown format=%r", fmt)
|
||||
raise ValueError(f"unknown storage format: {fmt!r}")
|
||||
```
|
||||
|
||||
The silent-`return None` form lets the unknown value propagate and breaks
|
||||
downstream code with a confusing `AttributeError` on the wrong line.
|
||||
|
||||
## 6. Logging vs surfacing - the decision
|
||||
|
||||
For a caught exception, ask in order:
|
||||
|
||||
1. **Will the pipeline produce a wrong artifact if I continue?** Re-raise /
|
||||
abort.
|
||||
2. **Is this expected and recoverable in the caller's contract?** Log at
|
||||
WARNING with `exc_info=True`, return a sentinel the caller checks.
|
||||
3. **Is this an isolated benchmark cell (pattern C)?** Log at ERROR, record
|
||||
the status as data, continue the matrix.
|
||||
4. **Is the process going down anyway?** Log at CRITICAL with
|
||||
`exc_info=True`, then `sys.exit(1)` or let it propagate.
|
||||
|
||||
## 7. Related documents
|
||||
|
||||
- [obs-python-logging.md](obs-python-logging.md) - logging API and severity
|
||||
levels.
|
||||
- [test-pipeline-validation.md](test-pipeline-validation.md) - the
|
||||
validation gates that turn silent corruption into loud failures.
|
||||
- [build-pipeline-tasks.md](build-pipeline-tasks.md) - `.done` marker
|
||||
semantics.
|
||||
- [bench-methodology.md](bench-methodology.md) - the TIMEOUT contract for
|
||||
benchmark cells.
|
||||
93
docs/rules/code-python-style.md
Normal file
93
docs/rules/code-python-style.md
Normal file
@@ -0,0 +1,93 @@
|
||||
# Code / Python style - tabs, 3.11+, closed dependency list, streaming
|
||||
|
||||
**Purpose + scope.** Baseline style and dependency policy for every Python
|
||||
file in this repository (pipeline tasks 01-11, query implementations,
|
||||
helper tools). Binding for all contributors.
|
||||
|
||||
---
|
||||
|
||||
## 1. Language and formatting
|
||||
|
||||
- **Python 3.11+.** Use modern stdlib freely (`pathlib`, `dataclasses`,
|
||||
`tomllib` if needed); no compatibility shims for older versions.
|
||||
- **Indentation: TABS.** Set by the project owner and repeated in the
|
||||
specs (00_PLAN, 03, 09). Every Python file in this repository indents
|
||||
with tab characters, never spaces. Configure the editor per file type;
|
||||
do not "fix" tab-indented files to spaces.
|
||||
- `from __future__ import annotations` at the top of every module.
|
||||
- Type hints on public function signatures; internal helpers at author's
|
||||
discretion.
|
||||
- English for all identifiers, docstrings, and comments (see
|
||||
[meta-language-style.md](meta-language-style.md) - Russian is the
|
||||
conversation language, not the code language).
|
||||
- Comments follow [code-comments-policy.md](code-comments-policy.md):
|
||||
default to none, keep only non-obvious WHY.
|
||||
|
||||
## 2. Dependencies - a closed list
|
||||
|
||||
The stdlib is the default. The ONLY permitted third-party packages:
|
||||
|
||||
| Package | Used by | For |
|
||||
|---|---|---|
|
||||
| `numpy` | 03 | vectorized generation, seeded RNG |
|
||||
| `PyYAML` | 01, 03, 10 | reading/writing the YAML configs |
|
||||
| `rdflib` | 04, 07 | JSON-LD validation, RDF serialization |
|
||||
| `oxigraph` | 07, 09 | embedded queryable triplestore |
|
||||
| `ijson` | 08, 09 | streaming JSON parsing |
|
||||
| `psycopg` (v3) | 06, 08, 09 | PostgreSQL COPY and queries |
|
||||
| `psutil` | 09 | RSS / CPU / IO measurement |
|
||||
| `matplotlib` | 11 | report charts |
|
||||
| `pandas` | 08 (optional) | ONLY as the separately-measured second CSV query variant; never in the pipeline itself |
|
||||
|
||||
Adding any other package requires updating this table first. In
|
||||
particular forbidden: ORMs, logging frameworks
|
||||
([obs-python-logging.md](obs-python-logging.md)), CLI frameworks
|
||||
(argparse suffices), `requests`/HTTP clients (nothing to call).
|
||||
|
||||
Task 03 (data generation) is stricter by spec: **stdlib + numpy only.**
|
||||
|
||||
## 3. Script shape - every task is a self-contained CLI
|
||||
|
||||
- One entry script per pipeline task, runnable as
|
||||
`python <script>.py [flags]`.
|
||||
- `argparse` with `--log-level` (see
|
||||
[obs-python-logging.md](obs-python-logging.md)) and task-appropriate
|
||||
per-run flags ([code-config-yaml.md](code-config-yaml.md) section 4).
|
||||
- A module docstring stating the task number, what it produces, and its
|
||||
spec file.
|
||||
- `main() -> int` returning the exit code; `sys.exit(main())` under
|
||||
`if __name__ == "__main__":`.
|
||||
- Exit code 0 only on full success (outputs written, validation passed,
|
||||
marker written).
|
||||
|
||||
## 4. Streaming discipline - the corpus never fits in memory by design
|
||||
|
||||
The generated corpus is ~150 MB and its scaled-up target is 600 GB; code
|
||||
must behave as if the data never fits in RAM:
|
||||
|
||||
- **Write generated rows as you produce them**; never accumulate the full
|
||||
cycle/loop dataset in a list (spec 03: "stream rows; never hold the
|
||||
full cycle dataset in memory").
|
||||
- **Read CSV/JSON with iterators** (`csv.reader` row-by-row, `ijson`
|
||||
events); never `read()` a bulk file whole or `json.load` the full
|
||||
corpus.
|
||||
- Batch database work (`executemany` / `COPY` in chunks), keeping only
|
||||
the current batch in memory.
|
||||
- Bounded aggregation state is fine (per-track summaries, counters);
|
||||
unbounded accumulation of raw rows is the thing being forbidden.
|
||||
|
||||
## 5. Anti-patterns
|
||||
|
||||
- **Spaces for indentation** in Python files - the project standard is
|
||||
tabs.
|
||||
- **An unlisted third-party import** - extend the section 2 table first
|
||||
or use the stdlib.
|
||||
- **`pd.read_csv` in pipeline code** - pandas exists only as the
|
||||
separately-measured Q-variant in spec 08.
|
||||
- **Loading a bulk artifact whole** (`json.load` on the full corpus,
|
||||
`readlines()` on a cycles CSV) - stream it.
|
||||
- **A task importable only from a specific CWD** - resolve paths from a
|
||||
repo-root anchor, not the current directory.
|
||||
- **Two tasks sharing mutable module state** - tasks communicate through
|
||||
`./out/` artifacts only (see
|
||||
[build-pipeline-tasks.md](build-pipeline-tasks.md)).
|
||||
85
docs/rules/data-determinism.md
Normal file
85
docs/rules/data-determinism.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# Data / Determinism - one seed, byte-identical regeneration
|
||||
|
||||
**Purpose + scope.** The reproducibility contract for all generated data.
|
||||
Binding for task 03 (CSV generation) and for every piece of code that
|
||||
produces values which end up inside the corpus or its conversions.
|
||||
|
||||
The entire evaluation rests on this: benchmark results are only comparable
|
||||
because every format holds EXACTLY the same data, and the study is only
|
||||
reviewable because anyone can regenerate that data bit-for-bit.
|
||||
|
||||
---
|
||||
|
||||
## 1. The seed
|
||||
|
||||
- **The global seed is `20260711`.** It lives in `lab_config.yaml`
|
||||
(written by task 01) and is READ from there - never hardcoded in a
|
||||
generation script (see [code-config-yaml.md](code-config-yaml.md)).
|
||||
- Changing the seed is a project-level decision, never a side effect. It
|
||||
invalidates the whole `./out/` tree, every checksum, and every measured
|
||||
result (see [meta-no-assumptions.md](meta-no-assumptions.md)).
|
||||
|
||||
## 2. Sanctioned randomness - seeded generators only
|
||||
|
||||
- The ONLY random sources are `random.Random(seed)` and
|
||||
`numpy.random.default_rng(seed)`, constructed explicitly from the
|
||||
config seed.
|
||||
- Derive per-entity generators deterministically when parallel or
|
||||
out-of-order generation needs them (e.g.
|
||||
`default_rng([seed, batch_index, coupon_index])`) - never from time,
|
||||
never from a global unseeded state.
|
||||
- Forbidden anywhere in generation code: module-level `random.*` calls,
|
||||
`numpy.random.*` legacy global functions, `os.urandom`, `uuid.uuid4`,
|
||||
`hash()`-dependent ordering (Python string hashing is salted per
|
||||
process).
|
||||
|
||||
## 3. No wall-clock in the data
|
||||
|
||||
- Every timestamp inside the corpus is SIMULATED - computed from the lab
|
||||
schedule model - never `datetime.now()` / `time.time()`.
|
||||
- Real wall-clock time may appear only in logs and in measurement records
|
||||
(benchmark wall times in `./out/bench/` are measurements, not corpus
|
||||
data).
|
||||
|
||||
## 4. Deterministic everything else
|
||||
|
||||
Randomness is not the only nondeterminism; all of these are binding:
|
||||
|
||||
- **Iteration order.** Iterate in a defined order (sorted paths, explicit
|
||||
index loops). Never rely on `os.listdir` / `glob` / `set` / dict-of-set
|
||||
ordering for anything that affects output. Sort directory listings
|
||||
before use.
|
||||
- **Float formatting.** Format floats with an explicit spec (e.g.
|
||||
`f"{x:.6g}"`), chosen once per column family; repr drift between runs
|
||||
or Python versions must not change file bytes.
|
||||
- **Line endings and encoding.** Generated files are UTF-8 with `\n`
|
||||
newlines, on every platform (`open(..., newline="")` for csv writers).
|
||||
- **Parallelism.** If generation is parallelized, each worker's output is
|
||||
deterministic (derived generators, fixed partitioning) and the merge
|
||||
order is fixed.
|
||||
|
||||
## 5. The proof - MANIFEST sha256
|
||||
|
||||
- `./out/csv/MANIFEST.csv` records sha256 per generated file (spec 03).
|
||||
**Regenerating with the same `lab_config.yaml` MUST reproduce every
|
||||
sha256 exactly.** This is the acceptance test for this whole rule -
|
||||
run generation twice when touching generator code and diff the
|
||||
manifests.
|
||||
- Downstream conversions must be equally deterministic in CONTENT (same
|
||||
rows, same values); byte-identity of database files is not required
|
||||
(SQLite/PostgreSQL internals may differ), which is why their gate is
|
||||
row counts + query-result checksums instead
|
||||
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
|
||||
|
||||
## 6. Anti-patterns
|
||||
|
||||
- **Hardcoding `20260711`** in a script instead of reading the config.
|
||||
- **An unseeded or globally-seeded random call** anywhere in generation.
|
||||
- **`datetime.now()` in corpus values** - simulated schedule only.
|
||||
- **Ordering that depends on the filesystem or hash salting** - sort
|
||||
explicitly.
|
||||
- **"Harmless" float repr changes** - a formatting change is a corpus
|
||||
change; it must be deliberate and re-baselines every checksum.
|
||||
- **Regenerating the corpus to "refresh" it** - identical input config
|
||||
must yield identical output; if bytes changed without a config change,
|
||||
that is a determinism bug to fix, not a new baseline to accept.
|
||||
120
docs/rules/data-naming-units.md
Normal file
120
docs/rules/data-naming-units.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Data / Naming and units - IDs, timestamps, SI-explicit column names
|
||||
|
||||
**Purpose + scope.** The naming grammar for entities, timestamps, and
|
||||
measurement columns across ALL representations of the corpus: CSV, JSON-LD,
|
||||
SQLite, PostgreSQL, RDF. Binding for generation, conversion, queries, and
|
||||
reporting. This file also fixes two points the specs leave ambiguous
|
||||
(sections 3 and 4).
|
||||
|
||||
---
|
||||
|
||||
## 1. Entity identifiers
|
||||
|
||||
The hierarchy encodes containment left-to-right; each level appends one
|
||||
segment:
|
||||
|
||||
| Entity | Code grammar | Example |
|
||||
|---|---|---|
|
||||
| batch | `B<nnn>` | `B721` |
|
||||
| wafer | `<batch>-W<n>` | `B721-W1` |
|
||||
| coupon | `<wafer>-C<nn>` (01..49, the 7x7 grid) | `B721-W1-C07` |
|
||||
| track | `<coupon>-T<n>` (1..3) | `B721-W1-C07-T2` |
|
||||
| run | `R<n>` (1..4) | `R3` |
|
||||
|
||||
- **CSV** stores the bare code (`B721-W1-C07`) in `<entity>_code`-style
|
||||
columns.
|
||||
- **JSON-LD / RDF** use the prefixed CURIE form as the node identity:
|
||||
`batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`,
|
||||
`track:B721-W1-C07-T2`, `run:R1` (spec 00_PLAN global conventions).
|
||||
- **SQL** stores the bare code as the `<singular>_code` TEXT UNIQUE column
|
||||
next to the integer surrogate key
|
||||
(see [db-sql-schema.md](db-sql-schema.md)).
|
||||
- A code is never parsed to recover its parent where a proper reference
|
||||
exists; the parent relation is explicit in every format (directory
|
||||
nesting, `partOf`, FK).
|
||||
|
||||
## 2. Timestamps
|
||||
|
||||
- ISO-8601, UTC, `Z` suffix, second precision:
|
||||
`2026-03-14T09:30:00Z`.
|
||||
- All corpus timestamps are simulated
|
||||
(see [data-determinism.md](data-determinism.md) section 3).
|
||||
- In SQL: `TIMESTAMPTZ` in PostgreSQL; TEXT in ISO-8601 form in SQLite
|
||||
(sortable lexicographically by construction).
|
||||
|
||||
## 3. Measurement columns - unit suffix is part of the name
|
||||
|
||||
Every numeric measurement column ends in an explicit SI unit token. The
|
||||
canonical tokens (as they appear in CSV headers):
|
||||
|
||||
| Token | Unit | Example columns |
|
||||
|---|---|---|
|
||||
| `_um` | micrometre | `thickness_um`, `position_um` |
|
||||
| `_nm` | nanometre | `ra_nm` |
|
||||
| `_GPa` | gigapascal | `hardness_GPa`, `reduced_modulus_GPa` |
|
||||
| `_mN` | millinewton | `max_load_mN`, `friction_force_mN` |
|
||||
| `_wtpct` | weight percent | `au_wtpct` |
|
||||
| `_um3` | cubic micrometre | `wear_volume_um3` |
|
||||
| `_m` | metre | `sliding_distance_m` |
|
||||
| `_mm_s` | millimetre per second | `speed_mm_s` |
|
||||
| `_V` | volt | `discharge_voltage_V` |
|
||||
| `_W` | watt | `power_W` |
|
||||
| `_C` | degree Celsius | `temperature_C` |
|
||||
| `_pct` | percent (non-composition) | `rh_pct`, `cpu_util_pct` |
|
||||
| `_s` / `_mb` | seconds / megabytes (benchmark outputs) | `wall_s`, `peak_rss_mb` |
|
||||
|
||||
- Unit tokens are plain ASCII - never `µ`, `°`, or Unicode superscripts
|
||||
(see [code-data-formatting.md](code-data-formatting.md)).
|
||||
- Dimensionless quantities carry no token (`cof`, `cycle`, `k_archard`).
|
||||
- New measurement columns pick from this table or add a row to it in the
|
||||
same commit.
|
||||
|
||||
## 4. Canonical case mapping CSV -> SQL (ambiguity resolved)
|
||||
|
||||
The specs write CSV headers with SI capitalization (`hardness_GPa`, spec
|
||||
03) but SQL DDL in lowercase (`hardness_gpa`, spec 05). Binding
|
||||
resolution:
|
||||
|
||||
- **CSV headers keep SI capitalization** (`_GPa`, `_mN`, `_V`, `_W`,
|
||||
`_C`) - they mirror instrument output conventions.
|
||||
- **SQL / JSON-LD / RDF property names are the plain lowercase of the CSV
|
||||
name** (`hardness_gpa`, `max_load_mn`). One mechanical rule, no
|
||||
per-column table to maintain.
|
||||
- Converters map with `csv_name.lower()` - never hand-maintained rename
|
||||
dicts that can drift.
|
||||
|
||||
## 5. `au_wtpct_mean` (spec gap closed)
|
||||
|
||||
Specs 05/06 index `coupons(au_wtpct_mean)` but spec 03 never names the
|
||||
column. Binding definition:
|
||||
|
||||
- `au_wtpct_mean` = the arithmetic mean of `au_wtpct` over the coupon's
|
||||
400 XRF map points, computed by task 03.
|
||||
- It is written into `coupon_info.csv` by generation (not derived later
|
||||
by converters), so every format inherits the identical value, and Q2
|
||||
("coupons with mean Au = 10 +/- 0.5 wt%") means the same rows
|
||||
everywhere.
|
||||
|
||||
## 6. Benchmark artifact columns
|
||||
|
||||
Result files under `./out/bench/` follow the same discipline:
|
||||
`results_raw.csv` columns are
|
||||
`format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb,
|
||||
rows, checksum_ok` (spec 09); `storage_sizes.csv` columns are
|
||||
`format, variant, bytes` (see
|
||||
[build-pipeline-tasks.md](build-pipeline-tasks.md)). Extending either
|
||||
file extends this list in the same commit.
|
||||
|
||||
## 7. Anti-patterns
|
||||
|
||||
- **A measurement column without a unit token** (`thickness`, `load`) -
|
||||
unreadable and un-greppable across formats.
|
||||
- **Unit in a comment instead of the name** - comments do not survive
|
||||
format conversion; names do.
|
||||
- **Parsing `B721-W1-C07` to find the wafer** where a real reference
|
||||
exists in the format at hand.
|
||||
- **A hand-maintained CSV->SQL rename mapping** - the mapping is
|
||||
`lower()`, nothing else.
|
||||
- **Recomputing `au_wtpct_mean` in a converter** - it is generated once
|
||||
into `coupon_info.csv`; converters copy it.
|
||||
- **Unicode units in identifiers** (`µm`, `°C`) - ASCII tokens only.
|
||||
122
docs/rules/db-sql-schema.md
Normal file
122
docs/rules/db-sql-schema.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# DB / SQL / Schema - conventions for the SQLite and PostgreSQL benchmark targets
|
||||
|
||||
**Purpose.** Naming, key, and loading conventions for the two relational
|
||||
storage formats under evaluation: the SQLite file (`./out/sqlite/tribo.db`,
|
||||
spec [05](../specs/05_convert_sqlite.md)) and the PostgreSQL 16 database
|
||||
`tribo` (spec [06](../specs/06_convert_postgresql.md)).
|
||||
**Binding scope.** Every table, index, and load routine in either engine.
|
||||
The two engines carry the SAME logical schema; they differ only in types,
|
||||
storage options, and load mechanics. The full table list and DDL signatures
|
||||
live in the specs - this file codifies the conventions the DDL must follow,
|
||||
and it wins where a spec detail is ambiguous.
|
||||
|
||||
---
|
||||
|
||||
## 1. Two kinds of tables
|
||||
|
||||
- **Dimension tables** (`batches`, `wafers`, `coupons`, `runs`,
|
||||
`instruments`, `tracks`): small, joined constantly.
|
||||
- PK: integer surrogate, named `<singular>_id` (`coupon_id`, `track_id`).
|
||||
Never a bare `id`, never TEXT.
|
||||
- The natural key from the CSV corpus (`B721`, `B721-W1-C07`,
|
||||
`B721-W1-C07-T2` - see [data-naming-units.md](data-naming-units.md))
|
||||
is stored as a TEXT column named `<singular>_code`, declared `UNIQUE`.
|
||||
It is for lookups and human-readable output, never the join key.
|
||||
- **Bulk tables** (`friction_cycles`, `friction_loop_points`, `xrf_points`,
|
||||
`profilometry_points`, `nanoindentation`, `afm`, `wear`): millions of
|
||||
rows, scanned and filtered.
|
||||
- Reference dimensions by integer FK columns only. **TEXT foreign keys in
|
||||
bulk tables are forbidden** - they bloat the file and slow every scan.
|
||||
- PK is the natural composite where one exists: `(track_id, cycle)` for
|
||||
cycles, `(track_id, cycle, pt)` for loop points; single-column
|
||||
`track_id` / `coupon_id` where the grain is one row per entity. In
|
||||
SQLite, composite-PK bulk tables are `WITHOUT ROWID`.
|
||||
- No surrogate `bigserial` on bulk tables - it adds bytes and nothing
|
||||
else.
|
||||
|
||||
## 2. Table and column naming
|
||||
|
||||
- Tables: lowercase, snake_case, plural (`friction_cycles`, not
|
||||
`FrictionCycle`).
|
||||
- Columns: lowercase snake_case; measurement columns carry the SI unit
|
||||
suffix in **lowercase** (`hardness_gpa`, `max_load_mn`,
|
||||
`wear_volume_um3`). The CSV corpus keeps SI capitalization
|
||||
(`hardness_GPa`); the SQL mapping is a plain lowercase of the CSV name -
|
||||
see the canonical mapping in
|
||||
[data-naming-units.md](data-naming-units.md).
|
||||
- Denormalization is deliberate and limited: `environment` and `load_mn`
|
||||
are copied onto `tracks` (from `runs` / test conditions) because
|
||||
Q4 and Q7 filter on them constantly. Do not denormalize further without
|
||||
updating this rule.
|
||||
|
||||
## 3. Indexes - only what Q1-Q7 use
|
||||
|
||||
- Create indexes AFTER the bulk load, never before.
|
||||
- The index set is driven by the benchmark queries: `coupons(au_wtpct_mean)`,
|
||||
`tracks(run_id)`, `tracks(coupon_id)`, `runs(environment)`, composite
|
||||
`tracks(environment, load_mn)`; PostgreSQL adds BRIN on
|
||||
`friction_cycles(cycle)` per partition.
|
||||
- Naming: `ux_<table>_<cols>` for unique, `ix_<table>_<cols>` for
|
||||
non-unique.
|
||||
- Do not over-index: an index no query uses is disk footprint that skews
|
||||
the storage comparison AND load-time cost. Every index must name the
|
||||
query it serves (a one-line comment in the DDL).
|
||||
|
||||
## 4. Loading
|
||||
|
||||
- **SQLite:** during load `PRAGMA journal_mode=OFF, synchronous=OFF,
|
||||
cache_size=-262144`; insert via `executemany` in transactions of 50k
|
||||
rows; after load `PRAGMA journal_mode=WAL`, `ANALYZE`, `VACUUM`.
|
||||
- **PostgreSQL:** load via `COPY FROM STDIN` (psycopg3 `copy`), never
|
||||
row-by-row INSERT; `friction_cycles` and `friction_loop_points` are
|
||||
declaratively RANGE-partitioned by `track_id` (12 partitions of 120
|
||||
tracks); `VACUUM ANALYZE` after load.
|
||||
- Loads are all-or-nothing per table and validated against `MANIFEST.csv`
|
||||
row counts - see [code-error-handling.md](code-error-handling.md)
|
||||
section 3 and [test-pipeline-validation.md](test-pipeline-validation.md).
|
||||
- On every SQLite connection that relies on FK behaviour:
|
||||
`PRAGMA foreign_keys = ON` (SQLite silently ignores FK clauses without
|
||||
it). Bulk tables MAY declare FK columns without enforced constraints for
|
||||
load speed - row-count validation is the integrity gate; whichever choice
|
||||
is made must be the same in both engines.
|
||||
|
||||
## 5. `track_summary` - precomputed, and off-limits to Q5
|
||||
|
||||
Both engines carry a `track_summary` (plain table computed during load in
|
||||
SQLite; materialized view in PostgreSQL) with per-track derived values
|
||||
(`cof_ss_mean`, `run_in_cycles`, ...). It exists for Q2/Q3/Q4/Q6. **Q5 must
|
||||
never read it** - Q5 is the forced full-scan benchmark over raw
|
||||
`friction_cycles` in every format (spec
|
||||
[08](../specs/08_benchmark_queries.md)).
|
||||
|
||||
## 6. Engine parity
|
||||
|
||||
- Same logical schema, same table names, same column names, same row
|
||||
counts in both engines.
|
||||
- Q1-Q7 executed against SQLite and PostgreSQL MUST return identical
|
||||
result sets (checksum parity with 1e-9 float tolerance - the canonical
|
||||
expected checksums are computed from PostgreSQL and cross-validated
|
||||
against SQLite; see
|
||||
[test-pipeline-validation.md](test-pipeline-validation.md)).
|
||||
- A schema change in one engine is a schema change in both, in the same
|
||||
commit.
|
||||
|
||||
## 7. Anti-patterns
|
||||
|
||||
- **Bare `id` PK, TEXT PK, or a GUID column** - this is a load-once
|
||||
analytical schema; surrogate integer `<singular>_id` on dimensions,
|
||||
natural composites on bulk tables.
|
||||
- **TEXT FK in a bulk table** (`friction_cycles.track_code`) - integer FKs
|
||||
only.
|
||||
- **A `bigserial` surrogate on `friction_cycles` / `friction_loop_points`**
|
||||
- dead weight at 1.4M / 2.9M rows.
|
||||
- **Creating indexes before the bulk load** - doubles load time for
|
||||
nothing.
|
||||
- **An index without a named query** - footprint that skews the format
|
||||
comparison.
|
||||
- **Rewriting Q5 to use `track_summary`** - breaks the benchmark's
|
||||
full-scan contract.
|
||||
- **`updated_at` / audit columns / triggers** - this is a read-only
|
||||
benchmark target rebuilt from CSV; it has no mutation lifecycle.
|
||||
- **Divergent schemas between the engines** - parity is what makes the
|
||||
benchmark comparison valid.
|
||||
59
docs/rules/meta-concise-answers.md
Normal file
59
docs/rules/meta-concise-answers.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# Meta / Concise answers
|
||||
|
||||
**Binding on every response.** Match length to the question. Stop after the answer.
|
||||
|
||||
## Rule
|
||||
|
||||
Do not append: background, rationale, anticipated follow-ups, recap, summary,
|
||||
TL;DR, "why this matters", offers to do more. If the user said "concise",
|
||||
"short", "one line", or "just" - obey literally.
|
||||
|
||||
## Length budget
|
||||
|
||||
| Question shape | Max response |
|
||||
|---|---|
|
||||
| "What is X?" / "Where is Y?" / yes-no | 1 line |
|
||||
| "How do I do X?" | 1-3 lines, commands only |
|
||||
| "How do I test X?" | 3-5 lines, commands + check |
|
||||
| "Show me X" (file, diagram, output) | The thing itself, no preamble |
|
||||
| "Compare A and B" | 1 short table or 2-3 bullets/side |
|
||||
| Open-ended architecture / discussion | Full prose acceptable |
|
||||
|
||||
Diagrams / lists / tables: **8-line ceiling** unless user asks for more.
|
||||
No ASCII box-drawing unless asked. When the user says "I don't care about X" -
|
||||
omit X, do not list it "for completeness".
|
||||
|
||||
## Examples
|
||||
|
||||
**Bad** - user asks "what does this option do?", assistant returns 6-option
|
||||
table with pros/cons. **Good** - one sentence about the one option.
|
||||
|
||||
**Bad** - user asks "show ER diagram, no fields", assistant returns 80-line
|
||||
ASCII with cardinality table and "what changes" section. **Good** - 4-5
|
||||
lines of `entity --< child` arrows for the asked entities only.
|
||||
|
||||
**Bad** - yes/no question answered with a paragraph of nuance before the
|
||||
yes/no. **Good** - yes/no first, nuance only if asked.
|
||||
|
||||
## Exceptions (these ARE the answer, no length cap)
|
||||
|
||||
File contents requested, code requested, test/log output, final summary
|
||||
after multi-step tool work, plans the user asked to be detailed.
|
||||
|
||||
## Self-check before sending
|
||||
|
||||
- Did the user ask a yes/no? -> answer yes/no in the first line, nuance
|
||||
only after, only if useful.
|
||||
- Did the user ask for one thing? -> deliver that one thing only.
|
||||
- About to write more than ~10 lines for a question that needs ~5? -> cut
|
||||
everything past line 10 unless it is code, output, or a file path.
|
||||
- Did the user say "concise" / "short" / "one line" / "just"? -> obey
|
||||
literally: one sentence or one command/path.
|
||||
|
||||
## Recovery
|
||||
|
||||
If the user complains the output is too long:
|
||||
|
||||
1. Acknowledge in one line.
|
||||
2. Re-answer the original question in one line.
|
||||
3. Capture it as an anti-pattern in the Examples section above.
|
||||
33
docs/rules/meta-concise-questions.md
Normal file
33
docs/rules/meta-concise-questions.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# Ask extremely short, concise questions
|
||||
|
||||
Binding on EVERY question put to the user.
|
||||
|
||||
## Rules
|
||||
|
||||
- **One decision per question.** Do not bundle unrelated questions.
|
||||
- **Minimal context.** State only what is needed to answer: the situation in one
|
||||
short clause, the concrete options, and what the answer changes. Nothing else.
|
||||
- **No preamble.** Do NOT precede a question with analysis, restated reasoning,
|
||||
background, evidence, or a summary of what you just did. The user has that
|
||||
context; if a fact is truly needed to decide, compress it to one clause.
|
||||
- **Prefer the smallest form.** A yes/no, or a 2-3 option pick. Put the question
|
||||
itself first or make it unmissable.
|
||||
- **Self-contained but short.** The options must be understandable without opening
|
||||
another document, but spell them out in a few words, not paragraphs.
|
||||
- **No useless information.** Every word must help the user decide. Delete anything
|
||||
that does not.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- A multi-paragraph lead-in before the actual question.
|
||||
- Re-explaining the problem the user already stated.
|
||||
- Listing options with a paragraph of trade-offs each.
|
||||
- Asking two or more separate decisions in one run of prose.
|
||||
|
||||
## Shape to aim for
|
||||
|
||||
```
|
||||
<one-clause situation>. <Question>?
|
||||
1. <option> - <few words>
|
||||
2. <option> - <few words>
|
||||
```
|
||||
62
docs/rules/meta-cursor-rules.md
Normal file
62
docs/rules/meta-cursor-rules.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# Meta / Cursor rule artifact - commit-message generation
|
||||
|
||||
**Binding** for keeping Cursor's commit-message generation aligned with this
|
||||
project's commit convention.
|
||||
|
||||
The development environment is the Cursor IDE with the Claude Code extension.
|
||||
Claude Code reads the root `CLAUDE.md` and `docs/rules/` directly and needs no
|
||||
`.cursorrules`. Cursor's OWN built-in AI (Chat/Composer and the SCM "sparkle"
|
||||
commit-message generator) does not read those files - it reads `.cursorrules`.
|
||||
Since the user creates all commits themselves (see
|
||||
[`meta-git-commits.md`](meta-git-commits.md)) and may use the one-click SCM
|
||||
generator to draft the message, `.cursorrules` must exist and must encode the
|
||||
commit convention.
|
||||
|
||||
This project ships a SINGLE Cursor rules file: the root
|
||||
[`.cursorrules`](../../.cursorrules). There is no `.cursor/rules/*.mdc`
|
||||
directory here, and none should be added. All Cursor rule content lives in
|
||||
`.cursorrules`.
|
||||
|
||||
---
|
||||
|
||||
## 1. The requirement
|
||||
|
||||
`.cursorrules` MUST encode this project's commit-message convention so that
|
||||
Cursor's generator emits conforming messages. The authoritative convention
|
||||
lives in [`meta-git-commits.md`](meta-git-commits.md); `.cursorrules` mirrors
|
||||
it as plain-text instructions to the generator.
|
||||
|
||||
The failure mode this rule prevents: a missing or stale `.cursorrules`, so the
|
||||
one-click SCM generator falls back to Cursor defaults and emits unprefixed junk
|
||||
(`Update files`, `Improve project`).
|
||||
|
||||
## 2. What `.cursorrules` must encode
|
||||
|
||||
1. **First-line format** - `type(scope): description`, lowercase imperative
|
||||
summary in English, under 72 chars, no trailing period.
|
||||
2. **Allowed types (closed list)** - `feat fix perf refactor security docs ci
|
||||
chore test`.
|
||||
3. **Allowed scopes (closed list)** - `specs rules docs config datagen convert
|
||||
bench report tools all`, with the area mapping from
|
||||
[`meta-git-commits.md`](meta-git-commits.md).
|
||||
4. **Coverage rule** - the subject is only a title; multi-area diffs need a
|
||||
blank line and a 2-6 bullet body covering every meaningful changed area; use
|
||||
`all` when no narrower scope honestly fits; split unrelated work.
|
||||
5. **Anti-examples** - forbid vague/unprefixed lines (`Update files`, `Improve
|
||||
project`, `clarify rules`).
|
||||
6. **Allowed special forms** - `Merge ...` commits are allowed as-is;
|
||||
`release(scope): vX.Y.Z` is allowed for releases.
|
||||
|
||||
## 3. Keeping the mirror in sync
|
||||
|
||||
No commit-msg hook is installed in this repository; the convention is enforced
|
||||
by review. `.cursorrules` and [`meta-git-commits.md`](meta-git-commits.md)
|
||||
must agree: same type list, same scope list, same first-line format. If the
|
||||
convention changes, update both files in the same change set.
|
||||
|
||||
## 4. Scope boundary
|
||||
|
||||
This rule governs the commit-message *convention materialization* - not who may
|
||||
create commits. Commit authorship is governed by
|
||||
[`meta-git-commits.md`](meta-git-commits.md): rules files describe the
|
||||
convention; only the user creates actual commits.
|
||||
99
docs/rules/meta-git-commits.md
Normal file
99
docs/rules/meta-git-commits.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Meta / Git commits - never on behalf of the user
|
||||
|
||||
**Binding** for every contributor (human or AI agent) touching this
|
||||
repository. Applies to **all** git history-modifying operations.
|
||||
|
||||
---
|
||||
|
||||
## 1. The rule
|
||||
|
||||
**Never create a git commit on behalf of the user. Ever.** The user is the
|
||||
only entity allowed to materialize a commit in this repo.
|
||||
|
||||
This covers, without exception:
|
||||
|
||||
- `git commit` (with or without `-a`, `-m`, `--amend`)
|
||||
- `git merge` - never finalize one
|
||||
- `git rebase` (interactive or otherwise) - never start one
|
||||
- `git revert` - never run; the user decides whether a change is undone
|
||||
- `git cherry-pick`
|
||||
- `git stash`
|
||||
- `git tag` (annotated or lightweight)
|
||||
- `git push` of any kind - pushing is the user's call
|
||||
- `git reset` / `git checkout` that changes the working tree or refs
|
||||
- Any wrapper script or `gh` API call that would create a commit / tag /
|
||||
merge on the user's behalf
|
||||
|
||||
For an AI agent, git is **read-only**: reading history (`git log`,
|
||||
`git diff`, `git status`, `git show`) and inspecting branches is fine;
|
||||
anything that writes to the index, refs, or `objects/` - including
|
||||
`git add` - is left to the user. (This matches the machine-level
|
||||
enforcement: git write commands by the agent are denied by a hook.)
|
||||
|
||||
## 2. Why
|
||||
|
||||
- Commit authorship and message wording are decisions the user owns - they are
|
||||
durable, public, and signed against the user's name and email.
|
||||
- "Tiny" commits that look harmless ("fix typo", "delete unused file") still
|
||||
rewrite history and cannot be quietly reversed once pushed.
|
||||
- Generated data and measured results in this project take hours to rebuild;
|
||||
what enters history and when is a conscious decision, not a side effect.
|
||||
|
||||
## 3. What to do instead
|
||||
|
||||
- Make the working-tree changes.
|
||||
- Report what changed and offer a draft commit message **as plain text** in the
|
||||
conversation, together with the exact `git add` / `git commit` commands for
|
||||
the user to run themselves.
|
||||
|
||||
## 4. Commit-message format
|
||||
|
||||
When you draft a message, follow this project's convention:
|
||||
|
||||
- **First line** - `type(scope): description`, lowercase imperative summary in
|
||||
English, under 72 chars, no trailing period.
|
||||
- **Types (closed list)** - `feat fix perf refactor security docs ci chore
|
||||
test`.
|
||||
- **Scopes (closed list, owner-approved 2026-07-11)** - `specs rules docs
|
||||
config datagen convert bench report tools all`. Pick by the area of the
|
||||
changed files:
|
||||
- `specs` = `docs/specs/**`
|
||||
- `rules` = `docs/rules/**`
|
||||
- `docs` = other prose docs (`README.md`, root `CLAUDE.md`,
|
||||
`docs/examples/**`)
|
||||
- `config` = configuration files (`.gitignore`, `.cursorrules`, editable
|
||||
YAML defaults such as hardware prices)
|
||||
- `datagen` = lab-configuration and data-generation code (tasks 01-03:
|
||||
`lab_config` generation, process diagrams, `generate_data.py`)
|
||||
- `convert` = format converters (tasks 04-07: JSON-LD, SQLite,
|
||||
PostgreSQL, RDF)
|
||||
- `bench` = benchmark queries, runner, and extrapolation (tasks 08-10)
|
||||
- `report` = reporting code and templates (task 11)
|
||||
- `tools` = helper scripts under `tools/`
|
||||
- `all` = multiple areas or in doubt
|
||||
- **Coverage** - multi-area diffs get a blank line and a 2-6 bullet body
|
||||
covering every meaningful changed area; split unrelated work.
|
||||
- **Special forms** - `Merge ...` and `release(scope): vX.Y.Z` are allowed
|
||||
as-is.
|
||||
|
||||
No `commit-msg` hook is installed in this repository yet; the convention is
|
||||
enforced by review. If a hook is added later, it must implement exactly this
|
||||
format and this file must be updated in the same change. The same convention
|
||||
is mirrored into `.cursorrules` for Cursor's built-in commit-message
|
||||
generator - see [meta-cursor-rules.md](meta-cursor-rules.md).
|
||||
|
||||
## 5. Anti-patterns
|
||||
|
||||
- **"I'll just commit it so we have a checkpoint."** No. Working-tree state is
|
||||
the checkpoint; the user decides when it becomes history.
|
||||
- **Amending the user's last commit to fix something you noticed.** No. Surface
|
||||
the issue, let the user choose between amend and new commit.
|
||||
- **Suggesting `git add -A` in the draft commands.** List explicitly named
|
||||
files, so the user sees exactly what would land.
|
||||
- **Vague subjects** (`Update files`, `Improve project`, `clarify rules`).
|
||||
Say what changed and where.
|
||||
|
||||
## 6. Exception (closed list)
|
||||
|
||||
There are no exceptions. If a workflow seems to require a commit, pause,
|
||||
explain the dependency, and let the user run the git command.
|
||||
35
docs/rules/meta-language-style.md
Normal file
35
docs/rules/meta-language-style.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# Meta / Communication language and register
|
||||
|
||||
**Binding on every response and on all prose directed at the user**
|
||||
(explanations, status updates, questions, summaries, commit messages, PR text).
|
||||
|
||||
## Rule
|
||||
|
||||
1. **Language: Russian.** All communication with the user is in standard
|
||||
literary Russian.
|
||||
2. **Register: neutral and professional. No jargon, no slang, no profanity.**
|
||||
- No profanity or vulgar words - in any language, under any circumstances.
|
||||
- No slang or jargon: street slang, gamer/internet slang, or untranslated
|
||||
English buzzwords used as slang.
|
||||
- Prefer the established Russian word over a transliterated English one
|
||||
whenever a normal Russian equivalent exists.
|
||||
- Plain, grammatical, complete sentences.
|
||||
|
||||
## Allowed terminology (not slang)
|
||||
|
||||
Field-standard technical terms are fine - they are terminology, not jargon:
|
||||
e.g. «коммит», «билд», «дебаг», «релиз», «мерж». When unsure whether a word
|
||||
is accepted terminology or slang, choose the neutral Russian equivalent.
|
||||
|
||||
## Scope
|
||||
|
||||
Does **not** apply to source code, identifiers, file paths, code comments,
|
||||
shell/tool output, or verbatim quotes of errors and logs - those keep the
|
||||
language of the codebase.
|
||||
|
||||
## Canonical source
|
||||
|
||||
This file is the project-level binding rule. It restates the project owner's
|
||||
standing preference (communication in Russian; code, docs, and artifacts in
|
||||
English). If a global user-level rule ever diverges from this file, the
|
||||
stricter wording wins.
|
||||
51
docs/rules/meta-no-assumptions.md
Normal file
51
docs/rules/meta-no-assumptions.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Meta / No assumptions
|
||||
|
||||
**Binding** on every interaction. Strictest when the cost of a wrong
|
||||
guess is high: deleting or regenerating `./out/` artifacts, dropping or
|
||||
reloading the `tribo` PostgreSQL database, changing the random seed or
|
||||
`lab_config.yaml`, editing specs in ways that invalidate generated data,
|
||||
git history, anything with data-loss consequences.
|
||||
|
||||
## Rule
|
||||
|
||||
Before producing a plan or change:
|
||||
|
||||
1. Re-read the user's most recent message. Watch for enumerations
|
||||
("both", "and also", "in addition to"), explicit purpose statements
|
||||
("X for A, Y for B"), and distinct entities that sound similar.
|
||||
2. Don't collapse two distinct entities into one. "X for A and Y for B"
|
||||
means **two** setups, not one.
|
||||
3. If anything is ambiguous after careful re-reading - **ask**. Don't
|
||||
pick the most likely interpretation and proceed.
|
||||
4. Echo back the parsed interpretation before producing the plan, so a
|
||||
misread is cheap to correct.
|
||||
|
||||
## When the rule is most strict
|
||||
|
||||
Strictness scales with the cost of a wrong guess:
|
||||
|
||||
| Domain | Why a wrong assumption is expensive |
|
||||
|---|---|
|
||||
| **Deleting / regenerating `./out/`** | The artifact tree holds hours of generation, conversion, and benchmark work. Regenerating CSV invalidates every downstream format, benchmark result, and report. Confirm which subtree is affected before deleting or overwriting. |
|
||||
| **The `tribo` PostgreSQL database** | Dropping or reloading it discards a long `COPY` load and invalidates benchmark results derived from it. Confirm before any `DROP` / `TRUNCATE` / reload. |
|
||||
| **The random seed / `lab_config.yaml`** | Seed 20260711 and the lab configuration are the root of the whole reproducibility chain. Changing either silently invalidates all generated data, all conversions, all checksums, and all measured results. Never change them as a side effect of another task. |
|
||||
| **Editing specs (`docs/specs/`)** | Specs are the task contract. A "small clarification" that changes volumes, schemas, or file layouts desynchronizes already-generated artifacts. Confirm intent when an edit is more than wording. |
|
||||
| **Benchmark results (`./out/bench/`)** | Measured results cannot be re-derived without re-running the full matrix (hours). Never overwrite `results_raw.csv` / `results_median.csv` without confirmation. |
|
||||
| **git history** | A wrong rebase/reset/force-push rewrites shared history; recovery is manual and error-prone. (The AI agent never runs git write operations at all - see [meta-git-commits.md](meta-git-commits.md).) |
|
||||
|
||||
For all of the above: **the cost of one extra clarifying question is
|
||||
zero; the cost of a wrong assumption is hours-to-days** of recovery.
|
||||
Always ask.
|
||||
|
||||
## Forms a clarifying question takes
|
||||
|
||||
- "Confirm: you want X for A AND Y for B (two setups), not just one?"
|
||||
- "Two readings: (a) ..., (b) ... - which?"
|
||||
- "What does '<word>' refer to here - (a) ... or (b) ...?"
|
||||
|
||||
## When the assumption was already wrong
|
||||
|
||||
1. Acknowledge the misread explicitly. Quote the user's words back.
|
||||
2. Discard the wrong artifact entirely; don't repair piecemeal unless
|
||||
the user asks.
|
||||
3. Restart with the corrected interpretation, echo it first.
|
||||
119
docs/rules/meta-rules-naming.md
Normal file
119
docs/rules/meta-rules-naming.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Meta / Rule file naming
|
||||
|
||||
**Binding** for files inside [`docs/rules/`](docs/rules/). Adding, renaming, or
|
||||
splitting a rule file follows this document.
|
||||
|
||||
## Layout
|
||||
|
||||
- `docs/rules/` - binding rules (this folder): reproducible conventions on HOW
|
||||
work is done. Reference content that supports a rule also lives here,
|
||||
named `<rule>-reference.md`.
|
||||
- `docs/specs/` - the pipeline task specifications (WHAT to build: tasks
|
||||
01-11, their inputs, outputs, and dependencies). Binding for scope and
|
||||
deliverables, but conventions that span tasks are codified in `docs/rules/`.
|
||||
- `docs/examples/` - imported reference materials (master plan, ontology).
|
||||
Descriptive, not binding.
|
||||
|
||||
Files live flat at the top level of `docs/rules/`. No subfolders.
|
||||
|
||||
## Naming
|
||||
|
||||
```
|
||||
<domain>-<subdomain>-<topic>[-<aspect>].md
|
||||
```
|
||||
|
||||
- Lowercase, dashes between segments. No underscores, no CamelCase.
|
||||
- Max 4 segments. Most general first.
|
||||
- Reading the filename should answer: category -> subcategory -> topic.
|
||||
|
||||
## Approved domain prefixes
|
||||
|
||||
| Prefix | Covers |
|
||||
|---|---|
|
||||
| `bench-` | Benchmark execution, resource measurement, extrapolation, reporting methodology |
|
||||
| `build-` | Pipeline task orchestration: the `./out/` tree, `.done` markers, task order |
|
||||
| `code-` | General code style across languages (Python, SQL, PowerShell): comments, formatting, error handling, configuration |
|
||||
| `data-` | The simulated dataset: determinism, identifiers, units, column naming |
|
||||
| `db-` | SQLite / PostgreSQL benchmark target schemas, loading, indexing |
|
||||
| `meta-` | Rules about the rules system and working conventions |
|
||||
| `obs-` | Observability and logging |
|
||||
| `test-` | Validation gates, checksums, pytest conventions |
|
||||
|
||||
Don't invent a new prefix without adding it to this table in the same
|
||||
commit.
|
||||
|
||||
### Subdomains (second / third segment)
|
||||
|
||||
Within a domain, the second segment is a recognised tool, subsystem, or
|
||||
concern; the third is the concrete topic. Established families:
|
||||
|
||||
- `db-sql-*` - relational benchmark targets (SQLite, PostgreSQL): schema,
|
||||
loading, indexing.
|
||||
- `data-*` - dataset conventions (`data-determinism`, `data-naming-units`).
|
||||
- `bench-*` - measurement methodology (`bench-methodology`).
|
||||
- `build-pipeline-*` - task orchestration and artifact layout.
|
||||
- `code-*` - general code style (comments, formatting, error handling,
|
||||
config, Python style).
|
||||
- `meta-*` - rules about the rules system and working conventions.
|
||||
- `obs-*` - observability and logging (`obs-python-logging`).
|
||||
- `test-*` - validation and testing (`test-pipeline-validation`).
|
||||
|
||||
## Examples
|
||||
|
||||
| Good | Reads as |
|
||||
|---|---|
|
||||
| `data-determinism.md` | Data -> deterministic generation |
|
||||
| `db-sql-schema.md` | DB -> SQL -> schema conventions |
|
||||
| `bench-methodology.md` | Benchmarks -> measurement methodology |
|
||||
| `code-python-style.md` | Code -> Python -> style |
|
||||
| `obs-python-logging.md` | Observability -> Python logging |
|
||||
| `meta-rules-naming.md` | Meta -> rules system -> naming |
|
||||
|
||||
| Bad | Why |
|
||||
|---|---|
|
||||
| `BenchMethodology.md` | CamelCase forbidden |
|
||||
| `data_determinism.md` | Underscores forbidden |
|
||||
| `sqlite-loading-conventions.md` | Use `db-` prefix, drop redundant `conventions` |
|
||||
| `bench-execution-measurement-rules-doc.md` | 5+ segments + redundant `rules` suffix |
|
||||
|
||||
## Adding a rule
|
||||
|
||||
1. Pick the domain prefix; add new ones to the table above first.
|
||||
2. Check `ls docs/rules/` - extend a sibling rule before creating a new one.
|
||||
3. Create the file with binding content, examples, and exceptions.
|
||||
4. Add a row for the new file to the rule-trigger table in the root
|
||||
[`CLAUDE.md`](../../CLAUDE.md) in the same commit.
|
||||
5. If replacing an older doc, `git mv` to preserve history and `Grep` the
|
||||
repo for the old path.
|
||||
|
||||
## Rules vs specs vs memory files
|
||||
|
||||
Three homes for a statement; pick by its kind:
|
||||
|
||||
| Source | Stores |
|
||||
|---|---|
|
||||
| `docs/rules/<name>.md` | A reproducible, mandatory behaviour - the same answer for every contributor on every task within the topic. |
|
||||
| `docs/specs/NN_<task>.md` | The definition of one pipeline task: scope, inputs, outputs, dependencies. |
|
||||
| `memory/<name>.md` | A fact about *this user*, *this project*, or *the current moment* - preferences, ongoing initiatives, locked decisions, infra specifics. |
|
||||
|
||||
- A convention that must hold across several tasks -> `docs/rules/`.
|
||||
- A deliverable of one task -> its spec in `docs/specs/`.
|
||||
- A fact about the user or the present state of things -> `memory/`.
|
||||
|
||||
A rule file may reference a spec or a memory note for context, but the
|
||||
binding convention lives in the rule file. When a rule and a spec disagree
|
||||
on a convention (naming, formats, thresholds), the rule wins and the spec
|
||||
is updated to match.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Duplicating rule content across rule files** - pick one canonical
|
||||
location and cross-link from siblings instead of copying.
|
||||
- **Restating a whole spec inside a rule** - the rule codifies the
|
||||
reusable convention and links the spec for task detail.
|
||||
- **Adding rules to the body of CLAUDE.md** instead of creating a rule
|
||||
file - CLAUDE.md carries only the trigger table, the project preamble,
|
||||
and the canonical-source-of-truth note.
|
||||
- **Naming a file after the implementation** instead of the domain
|
||||
(`oxigraph-loading.md` vs `db-rdf-loading.md`) - the implementation may
|
||||
be replaced; the topic endures.
|
||||
113
docs/rules/obs-python-logging.md
Normal file
113
docs/rules/obs-python-logging.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Observability / Python logging - standard library only, namespaced, plain text
|
||||
|
||||
**Purpose + scope.** Every Python module in this repository (data generation,
|
||||
converters, benchmark runner, extrapolation, reporting, helper tools) MUST
|
||||
use the standard-library `logging` module following these conventions. No
|
||||
external logging dependency.
|
||||
|
||||
---
|
||||
|
||||
## 1. The rule
|
||||
|
||||
1. **Standard library `logging` only.** No `loguru`, no `structlog`, no
|
||||
`python-json-logger`. Stdlib is sufficient and keeps the dependency list
|
||||
closed (see [code-python-style.md](code-python-style.md)).
|
||||
2. **Namespaced loggers.** Every module obtains its logger via
|
||||
`logging.getLogger(__name__)`. Never the root logger directly.
|
||||
3. **`print()` is for the user-facing CLI protocol only.** The specs
|
||||
explicitly require certain stdout output: progress summaries, the real
|
||||
generated tree size (task 03), per-cell benchmark progress, final
|
||||
counts. That protocol output goes through `print()`. Everything
|
||||
diagnostic - internal flow, warnings, errors, per-row detail - goes
|
||||
through a logger.
|
||||
4. **Log level from a CLI flag.** Every pipeline script accepts
|
||||
`--log-level {DEBUG,INFO,WARNING,ERROR}`, default `INFO`. There is no
|
||||
settings table and no `.env` in this project (see
|
||||
[code-config-yaml.md](code-config-yaml.md)).
|
||||
5. **Plain-text format on stderr.** These are one-shot CLI scripts, not
|
||||
services; JSON logs are overkill. One line per record via
|
||||
`logging.basicConfig`. Keeping the logger on stderr leaves stdout clean
|
||||
for the protocol output of item 3.
|
||||
6. **Never log secrets.** The only credential material in this project is
|
||||
the PostgreSQL DSN - never log it (a DSN may embed a password). Log the
|
||||
database name or host, never the full connection string.
|
||||
|
||||
## 2. API by severity
|
||||
|
||||
| Method | Use for |
|
||||
|---|---|
|
||||
| `logger.critical(...)` | The process cannot continue - before `sys.exit(1)` or an unrecoverable state. |
|
||||
| `logger.error(...)` | Failed operation, exception caught and handled, broken invariant, a benchmark cell recorded as TIMEOUT/ERROR. Always include `exc_info=True` when an exception is in scope. |
|
||||
| `logger.warning(...)` | Recoverable problem, retry triggered, suspicious state (e.g. a value near a validation bound), an optional step skipped (e.g. cold-cache pass unavailable). |
|
||||
| `logger.info(...)` | Normal flow milestones - task start/finish, per-batch progress, row counts written, validation gates passed. |
|
||||
| `logger.debug(...)` | Verbose tracing - per-coupon detail, generated value samples, SQL statements, per-run raw measurements. Hidden at default `INFO`. |
|
||||
|
||||
**Default level: `INFO`.** `DEBUG` is opt-in via `--log-level DEBUG` for a
|
||||
diagnosis session.
|
||||
|
||||
## 3. Logger acquisition and setup
|
||||
|
||||
```python
|
||||
# Top of every module that logs:
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Once, in each script's main():
|
||||
logging.basicConfig(
|
||||
level=args.log_level,
|
||||
stream=sys.stderr,
|
||||
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
|
||||
)
|
||||
```
|
||||
|
||||
Inside functions:
|
||||
|
||||
```python
|
||||
def load_table(conn, name: str, rows) -> int:
|
||||
logger.info("loading %s", name)
|
||||
try:
|
||||
...
|
||||
except psycopg.Error:
|
||||
logger.error("loading %s failed", name, exc_info=True)
|
||||
raise
|
||||
```
|
||||
|
||||
`__name__` gives the module path as the namespace tree used for filtering.
|
||||
Never hardcode a logger name string. Clamp noisy third-party libraries
|
||||
(`psycopg`, `matplotlib`) to `WARNING` in the same setup block if they
|
||||
pollute the output.
|
||||
|
||||
## 4. Progress reporting in long loops
|
||||
|
||||
Generation and loading loops run over millions of rows. Report progress at
|
||||
INFO on a coarse cadence (per batch / per wafer / per table, or every N
|
||||
seconds), never per row. Per-row output belongs at DEBUG, and even there
|
||||
prefer sampled output ("first row of each file") over full dumps.
|
||||
|
||||
## 5. Anti-patterns
|
||||
|
||||
- **`print()` for diagnostics** - forbidden; `print()` is only the
|
||||
user-facing CLI protocol (progress summaries, final counts, sizes).
|
||||
- **Diagnostics on stdout** - stdout is the protocol channel; the logger
|
||||
writes to stderr.
|
||||
- **Root logger calls** (`logging.info(...)`) - forbidden; always
|
||||
`logger = logging.getLogger(__name__)`.
|
||||
- **Third-party logging frameworks** (`loguru`, `structlog`) - forbidden.
|
||||
- **Hardcoded logger names** (`logging.getLogger("datagen")`) - forbidden;
|
||||
pass `__name__`.
|
||||
- **Logging the PostgreSQL DSN / connection string** - forbidden; log
|
||||
host/database name only.
|
||||
- **Log level from an environment variable or a config file** - the level
|
||||
is a per-run diagnostic choice; it comes from `--log-level`.
|
||||
- **`logger.error(...)` without re-raise** when the function semantically
|
||||
cannot proceed - see [code-error-handling.md](code-error-handling.md).
|
||||
- **Per-row INFO logging in bulk loops** - drowns the signal and slows the
|
||||
run; use coarse milestones.
|
||||
|
||||
## 6. Related documents
|
||||
|
||||
- [code-error-handling.md](code-error-handling.md) - `exc_info=True`
|
||||
requirement and the swallow ladder.
|
||||
- [code-config-yaml.md](code-config-yaml.md) - configuration scope (no
|
||||
`.env`, no settings table).
|
||||
- [code-python-style.md](code-python-style.md) - the closed dependency list.
|
||||
125
docs/rules/test-pipeline-validation.md
Normal file
125
docs/rules/test-pipeline-validation.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Test / Pipeline validation - markers, manifests, checksums
|
||||
|
||||
**Purpose + scope.** The validation contract between pipeline tasks: how a
|
||||
task proves it completed correctly and how downstream tasks verify their
|
||||
inputs before trusting them. Binding for every task implementation (01-11)
|
||||
and for any pytest suite added to this repository.
|
||||
|
||||
The pipeline has no human QA step between tasks - these gates are the only
|
||||
thing standing between a silent generation/conversion bug and a report full
|
||||
of wrong numbers.
|
||||
|
||||
---
|
||||
|
||||
## 1. Completion markers - written last, verified first
|
||||
|
||||
- Every task writes `./out/.done/<NN>.ok` **only after** all its outputs
|
||||
are complete and its own validation passed. The marker contains the
|
||||
task's headline counts (rows, bytes, triples) for downstream checks.
|
||||
- A task MUST verify the `.done` markers of its declared dependencies
|
||||
before starting, and abort with a clear message if one is missing.
|
||||
- A task that re-runs deletes its own marker first and rewrites it at the
|
||||
end. A marker is never written on failure and never from a `finally`
|
||||
block (see [code-error-handling.md](code-error-handling.md)).
|
||||
|
||||
## 2. The CSV corpus manifest - the root of trust
|
||||
|
||||
- Task 03 writes `./out/csv/MANIFEST.csv`: one row per generated file with
|
||||
path, row count, byte size, sha256.
|
||||
- Every converter (04-07) validates what it read against the manifest: row
|
||||
counts per source file, and sha256 where the file is consumed whole. A
|
||||
mismatch is an abort, not a warning.
|
||||
- The manifest is also the reproducibility proof: regenerating with the
|
||||
same `lab_config.yaml` must reproduce identical sha256 values (see
|
||||
[data-determinism.md](data-determinism.md)).
|
||||
|
||||
## 3. Canonical expected volumes
|
||||
|
||||
The lab configuration fixes the corpus size exactly; validation asserts
|
||||
these numbers, not "roughly that many":
|
||||
|
||||
| Entity | Count |
|
||||
|---|---|
|
||||
| batches | 4 |
|
||||
| wafers | 12 |
|
||||
| coupons | 588 (480 friction + 108 characterization-only) |
|
||||
| tracks | 1,440 |
|
||||
| friction cycle rows | 1,440,000 |
|
||||
| friction loop points | 2,880,000 |
|
||||
| XRF points | 235,200 (588 x 400) |
|
||||
| generated tree size | 120-200 MB (hard abort bounds) |
|
||||
|
||||
If a deliberate `lab_config.yaml` change moves these numbers, this table
|
||||
and the validation code change in the same commit.
|
||||
|
||||
## 4. Query-result checksums - all formats agree
|
||||
|
||||
- Each benchmark query Q1-Q7 has ONE canonical expected result: rows
|
||||
sorted, floats compared with tolerance 1e-9, checksummed to
|
||||
`./out/bench/expected/q<N>.sha256`.
|
||||
- The canonical checksums are computed from **PostgreSQL** and
|
||||
cross-validated against **SQLite** before any benchmarking starts; a
|
||||
disagreement between the two engines is a converter or query bug and
|
||||
blocks task 09.
|
||||
- Every measured benchmark run records a `checksum_ok` flag; a cell whose
|
||||
result does not match the expected checksum is an invalid measurement
|
||||
regardless of how fast it ran.
|
||||
|
||||
## 5. Format smoke tests
|
||||
|
||||
Each converter proves basic well-formedness of its output before writing
|
||||
its marker:
|
||||
|
||||
- **JSON-LD (04):** one sample coupon file passes `json.load` AND an
|
||||
rdflib JSON-LD parse.
|
||||
- **SQLite (05) / PostgreSQL (06):** row counts per table match the
|
||||
manifest-derived expectations; `track_summary` row count equals 1,440.
|
||||
- **RDF (07):** the loaded store answers a SPARQL `COUNT` matching the
|
||||
serialized triple count recorded in the `.done` marker.
|
||||
|
||||
## 6. If a pytest suite is added
|
||||
|
||||
The specs define no pytest suite; validation is built into the tasks. If
|
||||
unit/integration tests are added later:
|
||||
|
||||
- **Skip, don't fail, when PostgreSQL is unreachable.** A checkout without
|
||||
a running PostgreSQL still gets a green (skipped) run;
|
||||
connection failure in a fixture is `pytest.skip(...)`, not an error.
|
||||
- **Never mutate real `./out/` artifacts.** Tests that need data build a
|
||||
small corpus into a temp directory (a reduced `lab_config` is fine) or
|
||||
copy the artifact; hours of generated/measured data are not test
|
||||
fixtures to be overwritten.
|
||||
- **Ephemeral databases only.** A test touching PostgreSQL creates and
|
||||
drops its own database (e.g. `tribo_test`) next to `tribo`; it never
|
||||
writes to `tribo` itself.
|
||||
- **Independent and order-free.** Every test passes run alone; no residue
|
||||
between tests, no execution-order dependencies.
|
||||
- **Deterministic assertions.** With the fixed seed, exact counts and
|
||||
exact values are assertable - prefer exact assertions over "greater than
|
||||
zero".
|
||||
|
||||
## 7. Anti-patterns
|
||||
|
||||
- **Writing a `.done` marker before validation, on failure, or in
|
||||
`finally`** - the marker means "completed and verified".
|
||||
- **Skipping manifest validation "because generation just ran"** - the
|
||||
gate exists precisely for the runs where the assumption is wrong.
|
||||
- **Warning-and-continuing on a count mismatch** - a converter with a row
|
||||
deficit produces a database that LOOKS complete; abort.
|
||||
- **Regenerating expected checksums to make a failing format pass** - the
|
||||
canonical result comes from PostgreSQL cross-validated with SQLite;
|
||||
a mismatch in another format is a bug in that converter or query.
|
||||
- **Tests writing into `./out/`** or connecting to the real `tribo`
|
||||
database.
|
||||
- **Failing (not skipping) when PostgreSQL is unavailable** in a pytest
|
||||
run.
|
||||
|
||||
## 8. Related documents
|
||||
|
||||
- [build-pipeline-tasks.md](build-pipeline-tasks.md) - marker layout and
|
||||
task order.
|
||||
- [data-determinism.md](data-determinism.md) - why byte-identical
|
||||
regeneration is assertable.
|
||||
- [code-error-handling.md](code-error-handling.md) - fail-loud mechanics.
|
||||
- [bench-methodology.md](bench-methodology.md) - how `checksum_ok` enters
|
||||
the measurement protocol.
|
||||
50
docs/specs/00_PLAN.md
Normal file
50
docs/specs/00_PLAN.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# 00_PLAN - Tribology Lab Data Storage Evaluation Pipeline
|
||||
|
||||
## Goal
|
||||
Build a complete, reproducible pipeline that:
|
||||
1. Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow.
|
||||
2. Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format).
|
||||
3. Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore.
|
||||
4. Benchmarks 7 laboratory-standard retrieval scenarios (Q1-Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint.
|
||||
5. Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models.
|
||||
6. Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange).
|
||||
|
||||
## Execution environment
|
||||
- Claude Code agent inside Cursor, local filesystem access, on-premise only (no cloud services).
|
||||
- Python 3.11+ (tabs for indentation), PostgreSQL 16 local instance, SQLite via stdlib `sqlite3`, `rdflib` for RDF, optional local triplestore (Jena Fuseki / GraphDB Free) - if unavailable, use `rdflib` + on-disk `oxigraph` (pip) as the queryable store.
|
||||
- All artifacts written under `./out/` relative to repo root:
|
||||
- `./out/csv/` `./out/json/` `./out/sqlite/` `./out/pg/` (dumps) `./out/rdf/` `./out/bench/` `./out/report/`
|
||||
|
||||
## Task files and dependencies
|
||||
```mermaid
|
||||
flowchart TD
|
||||
T01[01_lab_configuration] --> T02[02_process_flow_diagrams]
|
||||
T01 --> T03[03_csv_data_generation]
|
||||
T03 --> T04[04_convert_json]
|
||||
T03 --> T05[05_convert_sqlite]
|
||||
T03 --> T06[06_convert_postgresql]
|
||||
T03 --> T07[07_convert_rdf]
|
||||
T04 --> T08[08_benchmark_queries]
|
||||
T05 --> T08
|
||||
T06 --> T08
|
||||
T07 --> T08
|
||||
T03 --> T08
|
||||
T08 --> T09[09_benchmark_execution]
|
||||
T09 --> T10[10_extrapolation_model]
|
||||
T09 --> T11[11_reporting]
|
||||
T10 --> T11
|
||||
```
|
||||
|
||||
## Parallelism
|
||||
- T04-T07 are independent; run in parallel after T03.
|
||||
- T02 is independent of data generation; can run any time after T01.
|
||||
|
||||
## Execution order
|
||||
01 → 02 → 03 → (04 | 05 | 06 | 07) → 08 → 09 → 10 → 11
|
||||
|
||||
## Global conventions
|
||||
- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`, `track:B721-W1-C07-T2`, run IDs `run:R1..R4`.
|
||||
- All timestamps ISO-8601 UTC. All units SI-explicit in column names (`_mN`, `_um`, `_GPa`, `_wtpct`, `_nm`).
|
||||
- Random seed fixed at 20260711 for full reproducibility.
|
||||
- Python indentation: tabs.
|
||||
- Every task writes a completion marker `./out/.done/<task>.ok` containing row/byte counts for downstream validation.
|
||||
56
docs/specs/01_lab_configuration.md
Normal file
56
docs/specs/01_lab_configuration.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# 01 - Laboratory Configuration
|
||||
|
||||
## Context / Goal
|
||||
Define the authoritative laboratory model that all downstream tasks (data generation, ontology, schemas) must follow. This file is a specification, not code; produce `./out/config/lab_config.yaml` capturing everything below.
|
||||
|
||||
## Dependencies
|
||||
None (root task).
|
||||
|
||||
## Material system
|
||||
- Substrate: Ti-6Al-4V coupons, 10 × 10 × 3 mm.
|
||||
- Adhesion layer: Cr (sputtered).
|
||||
- Functional coating: Pt-Au deposited in a **composition gradient** across each wafer; film thickness range 0.3-1.1 µm.
|
||||
|
||||
## Instruments
|
||||
| Role | Instrument | Data produced |
|
||||
|---|---|---|
|
||||
| Friction | RAPID custom high-throughput parallelized 6-probe tribometer | COF vs cycle per track; test conditions |
|
||||
| Nanoindentation | Bruker TI980 TriboIndenter | Hardness, reduced modulus (25 indents/coupon) |
|
||||
| Composition | Bruker M4 Tornado micro-XRF (assumed model) | Pt/Au wt% map per coupon (20×20 grid) |
|
||||
| Deposition | Kurt J. Lesker PVD 200 sputter-down | Batch deposition parameters |
|
||||
| Surface roughness | AFM | Topography image metadata + Ra/Rq per coupon |
|
||||
| Film thickness | Optical profilometry | Thickness map per coupon (10×10 grid) |
|
||||
| Simulation | SIMTRA (sputter transport Monte-Carlo) | Deposition atom-energy / composition profiles per deposition run |
|
||||
|
||||
## Deposition matrix (4 batches)
|
||||
| Batch | Pt:Au gun tilt | Pt power | Au power | Pt discharge | Au discharge |
|
||||
|---|---|---|---|---|---|
|
||||
| B721 | 20°:0° | 150 W | 50 W | 432 V | 311 V |
|
||||
| B722 | 20°:20° | 100 W | 100 W | 399 V | 352 V |
|
||||
| B723 | 20°:20° | 150 W | 50 W | 430 V | 311 V |
|
||||
| B724 | 0°:20° | 50 W | 150 W | 376 V | 340 V |
|
||||
|
||||
## Hierarchy and volumes
|
||||
```
|
||||
batch (×4) → wafer (×3 per batch) → coupon (×49 per wafer, 7×7 grid) = 588 coupons
|
||||
```
|
||||
- Friction testing: 480 coupons (4 RAPID runs × 120 coupons: 5 plates × 6 probes × 4 coupons/square).
|
||||
- Reserve/QA: 108 coupons (no friction data; characterization only).
|
||||
- Environments: run R1,R2 = Lab Air (µ_normal); run R3,R4 = Dry N2 (µ_dry_nit).
|
||||
- Tracks: 3 replicate tracks per friction coupon, identical conditions, fresh counterface ball per track (3 balls per holder) → 1,440 tracks.
|
||||
- Cycles: 1,000 reciprocating cycles per track → 1,440,000 cycle records.
|
||||
- Friction loops: every 100th cycle stores a position-resolved loop of 200 points → 1,440 × 10 × 200 = 2,880,000 loop points.
|
||||
- Nominal test conditions: normal load 100 mN, stroke 1 mm, sliding speed 1 mm/s, ruby (Al2O3) ball Ø 3.175 mm; RH 45% (Lab Air) / 2% (Dry N2); T 23 ± 1 °C.
|
||||
|
||||
## Physical models for value simulation (used in task 03)
|
||||
- Coupon Au wt%: linear gradient across wafer x-position, batch-dependent center: B721 8%, B722 25%, B723 10%, B724 60%; ±5 wt% span across wafer; XRF grid adds N(0, 0.3) noise.
|
||||
- COF run-in: `cof(c) = cof_ss + (cof_0 - cof_ss)·exp(-c/τ) + N(0, σ)`; cof_0 ∈ [0.35, 0.5]; τ ∈ [100, 400] cycles; σ = 0.01.
|
||||
- Steady-state COF depends on Au% and environment: Lab Air `cof_ss = 0.32 - 0.0015·Au%`; Dry N2 `cof_ss = 0.24 - 0.0012·Au%`; clamp ≥ 0.12.
|
||||
- Hardness: `H = 8.5 - 0.05·Au% + N(0, 0.25)` GPa; reduced modulus `Er = 190 - 0.6·Au% + N(0, 4)` GPa.
|
||||
- Roughness Ra: log-normal, median 5 nm, σ_log 0.3.
|
||||
- Thickness: 0.3-1.1 µm, radial wafer profile + noise.
|
||||
- Wear volume per track: Archard-like `V = k·F·s`, k depends on Au% and environment; store as track-level derived measurement.
|
||||
|
||||
## Output
|
||||
- `./out/config/lab_config.yaml` - machine-readable version of everything above (volumes, matrices, model coefficients, seed).
|
||||
- Marker `./out/.done/01.ok`.
|
||||
48
docs/specs/02_process_flow_diagrams.md
Normal file
48
docs/specs/02_process_flow_diagrams.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# 02 - Process Flow Diagrams
|
||||
|
||||
## Context / Goal
|
||||
Generate the comprehensive Mermaid documentation of the laboratory workflow (reconstructed from the 10 Miro slides) as standalone `.md` / `.mermaid` artifacts for the final report and for human review.
|
||||
|
||||
## Dependencies
|
||||
01 (uses volumes/terminology from lab_config.yaml).
|
||||
|
||||
## Task
|
||||
Produce `./out/report/process_flow.md` containing the following diagrams, each also saved as a separate `.mermaid` file in `./out/report/diagrams/`.
|
||||
|
||||
### D1 - Coupon Assembly
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A([Ti-6Al-4V Base 10x10x3 mm]) --> B[Cleaning]
|
||||
B --> B1([Rinsed in deionized water])
|
||||
B1 --> B2([Sonicated in cleaning solution])
|
||||
B --> C[Smearing Adhesive]
|
||||
C --> C1([Cr adhesive coating])
|
||||
C --> D[Vapor Deposition System<br/>Kurt J. Lesker PVD 200]
|
||||
D --> D1([PtAu sputter coating, gradient])
|
||||
D --> E([Test Coupon])
|
||||
```
|
||||
|
||||
### D2 - Characterization & Batch Assembly
|
||||
Test Coupon → SIMTRA (composition / atom energies) → Optical profilometry (film thickness 0.3-1.1 µm) → assemble Test Wafer (×49 coupons) → assemble Test Batch (×3 wafers) → batches 721-724 with the deposition parameter table.
|
||||
|
||||
### D3 - Testing tree
|
||||
Friction (RAPID; Lab Air → µ_normal; Dry N2 → µ_dry_nit), Nanoindentation (TI980 → hardness, reduced modulus), AFM (topography → Ra), XRF (M4 Tornado → Pt/Au map).
|
||||
|
||||
### D4 - Tribometer session sequence (sequenceDiagram)
|
||||
Request test → Get samples → Load samples → Load ball holders (3 counterfaces/holder) → Create Excel test plan (Sample Plate, Plate Location, Sample ID, Save Location, Folder Name, X/Y/Z Offset, Load, Iterations) → Transfer to tribometer → Set up platter → Activate software → Load plan → Activate tribometer → ... → Software stop → Save avg files → Remove plates/platters → Close software → Pull equipment out.
|
||||
|
||||
### D5 - Execution loop (flowchart)
|
||||
```
|
||||
Load all counter-faces
|
||||
FOR EACH of 5 plates
|
||||
FOR EACH of 6 probes (parallel)
|
||||
FOR EACH of 4 coupons on coupon square
|
||||
3x per coupon: draw track -> rotate counter-face
|
||||
```
|
||||
|
||||
### D6 - Data hierarchy ERD-style
|
||||
batch → wafer → coupon → {xrf_map, nanoindentation, afm, profilometry} and coupon → track → {cycles, loops, wear}; deposition → simtra_profile.
|
||||
|
||||
## Output
|
||||
- `./out/report/process_flow.md`, `./out/report/diagrams/D1..D6.mermaid`
|
||||
- Marker `./out/.done/02.ok`.
|
||||
46
docs/specs/03_csv_data_generation.md
Normal file
46
docs/specs/03_csv_data_generation.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# 03 - Simulated CSV Data Generation (~150 MB)
|
||||
|
||||
## Context / Goal
|
||||
Generate the canonical raw dataset as a hierarchical CSV file tree, exactly as instruments would write it. This corpus is the single source for all conversions and benchmarks. Target size: 140-170 MB.
|
||||
|
||||
## Dependencies
|
||||
01 (`lab_config.yaml`).
|
||||
|
||||
## Input
|
||||
`./out/config/lab_config.yaml` - volumes, deposition matrix, physical model coefficients, seed 20260711.
|
||||
|
||||
## Processing instructions
|
||||
Write `generate_data.py` (tabs for indentation, pure stdlib + numpy). Stream rows; never hold the full cycle dataset in memory. Use `random.Random(seed)` / `numpy.random.default_rng(seed)` only.
|
||||
|
||||
### File tree to produce
|
||||
```
|
||||
out/csv/
|
||||
├── batches.csv # 4 rows: deposition params
|
||||
├── simtra/
|
||||
│ └── simtra_profile_<batch>.csv # 4 files, 1000 rows each: angle, energy_eV, pt_flux, au_flux
|
||||
├── batch_<B>/wafer_<W>/
|
||||
│ ├── wafer_info.csv
|
||||
│ └── coupon_<C>/
|
||||
│ ├── coupon_info.csv # position on wafer, thickness_um, ra_nm, assigned run/plate/probe/square or RESERVE
|
||||
│ ├── xrf_map.csv # 400 rows: grid_x, grid_y, pt_wtpct, au_wtpct
|
||||
│ ├── profilometry.csv # 100 rows: grid_x, grid_y, thickness_um
|
||||
│ ├── nanoindentation.csv # 25 rows: indent_id, x_um, y_um, hardness_GPa, reduced_modulus_GPa, max_load_mN
|
||||
│ ├── afm.csv # 1 row: ra_nm, rq_nm, image_file (synthetic path)
|
||||
│ └── track_<T>/ # only for 480 friction coupons, T in 1..3
|
||||
│ ├── track_info.csv # run_id, environment, load_mN, stroke_mm, speed_mm_s, counterface_id, started_at
|
||||
│ ├── cof_vs_cycle.csv # 1000 rows: cycle, cof
|
||||
│ ├── friction_loops.csv # 2000 rows: cycle (every 100th), position_um (200 pts), friction_force_mN
|
||||
│ └── wear.csv # 1 row: wear_volume_um3, k_archard, sliding_distance_m
|
||||
└── runs.csv # 4 rows: run_id, environment, date, plates, operator
|
||||
```
|
||||
|
||||
### Value generation
|
||||
Apply exactly the physical models of task 01 (gradient Au%, run-in COF, hardness/modulus vs Au%, Archard wear). COF must correlate with the coupon's mean Au% and the run environment so that Q2/Q3/Q7 benchmarks return physically meaningful results.
|
||||
|
||||
### Size control
|
||||
Expected: cycles 1.44 M rows (~30 MB) + loops 2.88 M rows (~95 MB) + XRF 235 K rows (~8 MB) + remainder (~10 MB) ≈ 150 MB. After generation, print actual tree size; abort with error if outside 120-200 MB.
|
||||
|
||||
## Output
|
||||
- Full tree under `./out/csv/` (~150 MB).
|
||||
- `./out/csv/MANIFEST.csv`: file path, rows, bytes, sha256 - used by conversion tasks for validation.
|
||||
- Marker `./out/.done/03.ok` with total rows/bytes.
|
||||
22
docs/specs/04_convert_json.md
Normal file
22
docs/specs/04_convert_json.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# 04 - Convert to JSON-LD
|
||||
|
||||
## Context / Goal
|
||||
Produce the ontology-annotated JSON-LD representation of the corpus: the format evaluated for inter-lab data exchange (FAIR). Two variants: (a) full - bulk points embedded; (b) hybrid - metadata JSON-LD with `sourceFile` links to CSV.
|
||||
|
||||
## Dependencies
|
||||
03 (`./out/csv/`, MANIFEST.csv).
|
||||
|
||||
## Input
|
||||
CSV tree.
|
||||
|
||||
## Processing instructions
|
||||
- `@context`: project vocab `https://sandia.gov/ontology/tribology#` + QUDT units + PROV-O provenance; predicates `partOf`, `cutFrom`, `performedOn`, `performedBy` as `@id` links.
|
||||
- Entity classes: `Batch, Wafer, Coupon, Instrument, CompositionMeasurement, NanoindentationMeasurement, AFMMeasurement, ProfilometryMeasurement, FrictionTest (track), FrictionCycle, FrictionLoopPoint, WearMeasurement, SimtraProfile, Run`.
|
||||
- Variant FULL: one JSON-LD file per coupon (embedding its tracks/cycles/loops), compact serialization (no pretty-print). Directory `./out/json/full/`.
|
||||
- Variant HYBRID: single `./out/json/hybrid/dataset.jsonld` with all metadata entities; cycle/loop arrays replaced by `{"sourceFile": "<relative csv path>", "rows": N, "sha256": "..."}`.
|
||||
- Stream-write; validate one sample file with `json.load` + rdflib JSON-LD parse.
|
||||
|
||||
## Output
|
||||
- `./out/json/full/` (expect ~0.7-1.2 GB), `./out/json/hybrid/dataset.jsonld` (expect < 20 MB).
|
||||
- Record byte sizes of both variants into `./out/bench/storage_sizes.csv` (append: format, variant, bytes).
|
||||
- Marker `./out/.done/04.ok`.
|
||||
30
docs/specs/05_convert_sqlite.md
Normal file
30
docs/specs/05_convert_sqlite.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# 05 - Convert to SQLite (optimized)
|
||||
|
||||
## Context / Goal
|
||||
Build the single-file relational variant optimized for size and single-user query speed.
|
||||
|
||||
## Dependencies
|
||||
03.
|
||||
|
||||
## Input
|
||||
CSV tree.
|
||||
|
||||
## Processing instructions
|
||||
- One database `./out/sqlite/tribo.db`. PRAGMAs during load: `journal_mode=OFF, synchronous=OFF, cache_size=-262144`; after load: `journal_mode=WAL`, `ANALYZE`, `VACUUM`.
|
||||
- Schema (16-table style, integer surrogate keys everywhere; no TEXT foreign keys in bulk tables):
|
||||
- `batches, wafers, coupons, runs, instruments, tracks` - dimension tables, TEXT natural keys kept as unique columns.
|
||||
- `xrf_points(coupon_id INT, gx INT, gy INT, pt_wtpct REAL, au_wtpct REAL)`.
|
||||
- `profilometry_points(coupon_id INT, gx INT, gy INT, thickness_um REAL)`.
|
||||
- `nanoindentation(coupon_id INT, indent_id INT, x_um REAL, y_um REAL, hardness_gpa REAL, reduced_modulus_gpa REAL)`.
|
||||
- `afm(coupon_id INT PRIMARY KEY, ra_nm REAL, rq_nm REAL)`.
|
||||
- `friction_cycles(track_id INT, cycle INT, cof REAL, PRIMARY KEY(track_id, cycle)) WITHOUT ROWID`.
|
||||
- `friction_loop_points(track_id INT, cycle INT, pt INT, position_um REAL, force_mn REAL, PRIMARY KEY(track_id, cycle, pt)) WITHOUT ROWID`.
|
||||
- `wear(track_id INT PRIMARY KEY, wear_volume_um3 REAL, k_archard REAL, sliding_distance_m REAL)`.
|
||||
- `track_summary(track_id INT PRIMARY KEY, cof_ss_mean REAL, cof_ss_std REAL, run_in_cycles INT)` - precomputed during load.
|
||||
- Indexes for Q1-Q7: `coupons(au_wtpct_mean)`, `tracks(run_id)`, `tracks(coupon_id)`, `runs(environment)`, composite `tracks(environment, load_mn)` (denormalize environment/load onto tracks).
|
||||
- Batch inserts with `executemany`, 50k rows per transaction.
|
||||
- Validate row counts against MANIFEST.
|
||||
|
||||
## Output
|
||||
- `./out/sqlite/tribo.db`; size appended to `storage_sizes.csv`.
|
||||
- Marker `./out/.done/05.ok`.
|
||||
23
docs/specs/06_convert_postgresql.md
Normal file
23
docs/specs/06_convert_postgresql.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# 06 - Convert to PostgreSQL (indexed + partitioned)
|
||||
|
||||
## Context / Goal
|
||||
Build the production-grade relational variant: the system-of-record candidate per the existing Storage ADR (PostgreSQL primary, RDF as deferred virtual layer).
|
||||
|
||||
## Dependencies
|
||||
03.
|
||||
|
||||
## Input
|
||||
CSV tree.
|
||||
|
||||
## Processing instructions
|
||||
- Local PostgreSQL 16, database `tribo`. Same logical schema as task 05 with PostgreSQL types (`INT`, `REAL`, `TIMESTAMPTZ`, `TEXT`).
|
||||
- `friction_cycles` and `friction_loop_points`: declarative RANGE partitioning by `track_id` (12 partitions of 120 tracks) - models the partitioning strategy needed at 600 GB scale.
|
||||
- Load via `COPY FROM STDIN` (psycopg3 `copy`), not INSERT.
|
||||
- Indexes after load: PK b-trees; `tracks(environment, load_mn)`, `coupons(au_wtpct_mean)`, `nanoindentation(coupon_id)`, BRIN on `friction_cycles(cycle)` per partition.
|
||||
- `track_summary` as a materialized view (`cof_ss_mean` = avg of cycles > run_in threshold; `run_in_cycles` via first cycle where |cof - cof_ss| < 2σ).
|
||||
- `VACUUM ANALYZE` after load. Record `pg_total_relation_size` per table and total into `storage_sizes.csv`.
|
||||
- Also export a compressed dump `./out/pg/tribo.dump` (pg_dump -Fc) for archiving-format comparison.
|
||||
|
||||
## Output
|
||||
- Live `tribo` database + `./out/pg/tribo.dump`.
|
||||
- Marker `./out/.done/06.ok`.
|
||||
24
docs/specs/07_convert_rdf.md
Normal file
24
docs/specs/07_convert_rdf.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# 07 - Convert to RDF TripleStore
|
||||
|
||||
## Context / Goal
|
||||
Build the semantic-web variant to quantify triple-store overhead and SPARQL performance on real hardware.
|
||||
|
||||
## Dependencies
|
||||
03 (optionally reuse 04 hybrid JSON-LD as an intermediate).
|
||||
|
||||
## Input
|
||||
CSV tree (or `./out/json/full/` for direct rdflib parsing - choose the faster path, document the choice).
|
||||
|
||||
## Processing instructions
|
||||
- Vocabulary identical to task 04 (`tribology#` + QUDT + PROV-O).
|
||||
- Triple modeling: every cycle → 3 triples (`rdf:type` optional at bulk level to save space - model cycles as `track :hasCycle [ :cycleNumber n ; :cof x ]` without explicit type; document the decision). Loop points likewise. Expected total: 25-45 M triples.
|
||||
- Serialize `./out/rdf/dataset.nt.gz` (N-Triples, gzip) and `./out/rdf/dataset.ttl` (Turtle) - both sizes recorded.
|
||||
- Queryable store, first available option:
|
||||
1. `oxigraph` (pip, embedded, on-disk store `./out/rdf/oxigraph_store/`) - preferred, no external service.
|
||||
2. Jena Fuseki with TDB2 if Java present.
|
||||
- Bulk-load, record store on-disk size into `storage_sizes.csv` and load wall-time.
|
||||
- Smoke-test with a SPARQL COUNT query.
|
||||
|
||||
## Output
|
||||
- Serializations + populated store.
|
||||
- Marker `./out/.done/07.ok` with triple count.
|
||||
31
docs/specs/08_benchmark_queries.md
Normal file
31
docs/specs/08_benchmark_queries.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# 08 - Benchmark Query Definitions (Q1-Q7 × 5 formats)
|
||||
|
||||
## Context / Goal
|
||||
Define the 7 laboratory-standard retrieval scenarios and their concrete implementation for each of the 5 formats. Every implementation must return identical result sets (validated by checksum of sorted result rows, tolerance 1e-9 on floats).
|
||||
|
||||
## Dependencies
|
||||
04, 05, 06, 07 (all stores populated) + 03 (CSV baseline).
|
||||
|
||||
## Scenarios
|
||||
| ID | Scenario | Result shape |
|
||||
|---|---|---|
|
||||
| Q1 | COF vs cycle curve for one track (`B722-W2-C13-T2`) | 1,000 rows (cycle, cof) |
|
||||
| Q2 | Steady-state COF for all coupons with mean Au = 10 ± 0.5 wt% | ~dozens of rows (coupon, env, cof_ss) |
|
||||
| Q3 | Hardness vs steady-state COF across all batches (join nano × track_summary) | 480 rows |
|
||||
| Q4 | Tracks in Dry N2, load 100 mN, cof_ss > 0.20 (anomaly filter) | filtered rows |
|
||||
| Q5 | Mean run-in cycles per batch (aggregation over ALL cycle rows, no summary tables allowed - force the full scan) | 4 rows |
|
||||
| Q6 | Wear volume vs load per coupon (join wear × tracks × coupons) | 480 rows |
|
||||
| Q7 | Stribeck-style aggregation: mean COF grouped by (environment, speed·load bucket) over all cycles past run-in | ~10 rows |
|
||||
|
||||
## Implementations
|
||||
- **CSV**: pure Python + csv module streaming (pandas allowed only as a second measured variant, labeled separately). Q1 = direct path read (this is the honest CSV strength). Q2-Q7 = directory walks + programmatic joins.
|
||||
- **JSON**: variant FULL, `ijson` streaming parser; Q1 = load the single coupon file.
|
||||
- **SQLite**: SQL over `tribo.db`, cold and warm cache runs (`PRAGMA cache_size` default; drop OS cache not required, just report both run 1 and run 3).
|
||||
- **PostgreSQL**: SQL, `EXPLAIN (ANALYZE, BUFFERS)` captured alongside; parallel workers default.
|
||||
- **RDF**: SPARQL against the populated store; capture query text.
|
||||
- Q5 rule applies to all formats: query raw cycle data, not `track_summary`.
|
||||
|
||||
## Output
|
||||
- `./out/bench/queries/` - one subfolder per format with 7 runnable query files (`q1.sql`, `q1.sparql`, `q1.py`, ...).
|
||||
- `./out/bench/expected/q<N>.sha256` - canonical result checksums (computed once from PostgreSQL, cross-validated against SQLite).
|
||||
- Marker `./out/.done/08.ok`.
|
||||
22
docs/specs/09_benchmark_execution.md
Normal file
22
docs/specs/09_benchmark_execution.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# 09 - Benchmark Execution & Resource Measurement
|
||||
|
||||
## Context / Goal
|
||||
Execute all 35 (Q1-Q7 × 5) benchmark cells with rigorous resource metering; produce the raw measurement table that feeds extrapolation and reporting.
|
||||
|
||||
## Dependencies
|
||||
08.
|
||||
|
||||
## Processing instructions
|
||||
Write `bench_runner.py` (tabs):
|
||||
- Each cell runs as a **separate subprocess** so RSS/CPU are attributable. For PostgreSQL, also sample the backend PID via `pg_stat_activity` + psutil.
|
||||
- Metrics per run: wall time (s), peak RSS (MB, psutil `memory_info().rss` sampled at 50 ms), CPU time user+sys (s), CPU utilization % (cpu_time / wall / cores), read bytes (psutil `io_counters`), result row count, checksum-match flag.
|
||||
- Protocol: 1 warm-up run (discarded) + 3 measured runs per cell; report median and min/max. Randomize cell order to spread cache effects; record run order.
|
||||
- Cold-cache variant: repeat the full matrix once after dropping caches if permitted (`sync; echo 3 > /proc/sys/vm/drop_caches` requires root - if not available, skip and note it).
|
||||
- Failures/timeouts: hard timeout 30 min per run; record `TIMEOUT` and continue (expected possibility: RDF Q5).
|
||||
- Storage footprint: consolidate `storage_sizes.csv` (all formats, incl. gzip-compressed archival sizes of each: `tar.gz` of CSV tree, gzip JSON, sqlite db gz, pg dump, nt.gz).
|
||||
|
||||
## Output
|
||||
- `./out/bench/results_raw.csv`: format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb, rows, checksum_ok.
|
||||
- `./out/bench/results_median.csv` - aggregated.
|
||||
- `./out/bench/storage_sizes.csv` - final consolidated.
|
||||
- Marker `./out/.done/09.ok`.
|
||||
38
docs/specs/10_extrapolation_model.md
Normal file
38
docs/specs/10_extrapolation_model.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# 10 - Extrapolation to 600 GB and Beyond
|
||||
|
||||
## Context / Goal
|
||||
Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.
|
||||
|
||||
## Dependencies
|
||||
09 (`results_median.csv`, `storage_sizes.csv`).
|
||||
|
||||
## Processing instructions
|
||||
Write `extrapolate.py`:
|
||||
|
||||
### Storage scaling
|
||||
Linear in data volume: `size_at(S) = measured_size × S / S_measured`, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).
|
||||
|
||||
### Query-time scaling laws (apply per format × query)
|
||||
| Access pattern | Law | Applies to |
|
||||
|---|---|---|
|
||||
| Direct path / keyed point read | O(1) / O(log n) | CSV Q1, SQLite/PG Q1, RDF Q1 |
|
||||
| Indexed selective filter | O(log n + k) | SQLite/PG Q2, Q4, Q6; RDF Q2, Q4 |
|
||||
| Indexed join over dimensions | O(k log n) | Q3, Q6 relational; RDF joins ×(2-5) constant penalty |
|
||||
| Full scan / aggregation | O(n) | all CSV/JSON Q2-Q7; Q5 and Q7 everywhere |
|
||||
| Single-threaded full scan | O(n), 1 core | SQLite Q5/Q7 |
|
||||
| Parallel partitioned scan | O(n / min(cores, partitions)) | PostgreSQL Q5/Q7 |
|
||||
|
||||
Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".
|
||||
|
||||
### RAM model
|
||||
- CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
|
||||
- SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
|
||||
- RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).
|
||||
|
||||
### Hardware sizing table
|
||||
For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in `./out/config/hw_prices.yaml` (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.
|
||||
|
||||
## Output
|
||||
- `./out/bench/extrapolation.csv`: format, query, scale_gb, projected_wall_s, flag.
|
||||
- `./out/bench/hardware_sizing.csv`: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
|
||||
- Marker `./out/.done/10.ok`.
|
||||
37
docs/specs/11_reporting.md
Normal file
37
docs/specs/11_reporting.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# 11 - Reporting: Tables, Charts, Weighted Scoring
|
||||
|
||||
## Context / Goal
|
||||
Aggregate all measurements and projections into the final decision document that clearly ranks the storage formats for laboratory use cases: analysis, report generation, search/filtering, archiving, inter-lab exchange.
|
||||
|
||||
## Dependencies
|
||||
09, 10 (and 02 for embedding process diagrams).
|
||||
|
||||
## Processing instructions
|
||||
Write `report.py` using matplotlib (PNG, 150 dpi, log scales where ranges span decades) into `./out/report/charts/`, and assemble `./out/report/REPORT.md`.
|
||||
|
||||
### Tables
|
||||
1. Storage footprint: measured @150 MB + coefficients vs CSV + projected @600 GB/1.2 TB/6 TB (incl. compressed archival sizes).
|
||||
2. Q1-Q7 median wall time / peak RAM / CPU% / read MB, per format (measured).
|
||||
3. Q1-Q7 projected wall time @600 GB with IMPRACTICAL/FAIL flags.
|
||||
4. Hardware sizing & cost per format @600 GB.
|
||||
5. Weighted scoring matrix (see below).
|
||||
|
||||
### Charts (each measured + each projected @600 GB)
|
||||
- C1: disk footprint bar (log).
|
||||
- C2: RAM requirement bar (log).
|
||||
- C3: per-query grouped bars Q1-Q7 wall time (log), one chart per query, 5 formats each.
|
||||
- C4: degradation curves - wall time vs data size (150 MB → 6 TB), one line per format, per query class (point read / indexed / full aggregation), log-log.
|
||||
- C5: stacked-bar weighted score.
|
||||
- C6: cost vs performance scatter @600 GB (x = est cost, y = geometric-mean query time, log-log).
|
||||
|
||||
### Weighted scoring (agreed methodology)
|
||||
Normalized 0-10 per metric, then weights: Search speed ×5 (0-50), RAM economy ×2 (0-20), Disk economy ×1 (0-10). Max 80. Search score = normalized inverse geometric mean of Q1-Q7 projected @600 GB (TIMEOUT/FAIL → 0). Present measured-scale and 600 GB-scale scoreboards side by side.
|
||||
|
||||
### Narrative sections (concise, English)
|
||||
- Executive summary with the recommendation (expected, to be confirmed by data: PostgreSQL system of record; CSV retained as raw archive; hybrid JSON-LD for exchange; RDF as virtual layer only - reference Ontop OBDA).
|
||||
- Use-case mapping table: analysis / reports / search / archiving / exchange → recommended format(s).
|
||||
- Limitations: single-node, simulated data, extrapolation assumptions.
|
||||
|
||||
## Output
|
||||
- `./out/report/REPORT.md` + `./out/report/charts/*.png` + all tables as CSV in `./out/report/tables/`.
|
||||
- Marker `./out/.done/11.ok`.
|
||||
27
docs/specs_old/00_PLAN.md
Normal file
27
docs/specs_old/00_PLAN.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
|
||||
|
||||
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
|
||||
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
|
||||
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
|
||||
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
|
||||
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
|
||||
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
|
||||
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
|
||||
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
|
||||
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
|
||||
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
|
||||
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
|
||||
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
|
||||
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
|
||||
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
|
||||
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
|
||||
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
|
||||
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
|
||||
|
||||
## Final Goal
|
||||
|
||||
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.
|
||||
26
docs/specs_old/01_setup_workspace.md
Normal file
26
docs/specs_old/01_setup_workspace.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# Task 01 — Setup Workspace
|
||||
|
||||
## Goal
|
||||
Create the working directory layout and versioned pipeline configuration.
|
||||
|
||||
## Input
|
||||
- Path to the raw CSV root (read-only). Ask the user if unknown.
|
||||
|
||||
## Actions
|
||||
1. Create:
|
||||
```
|
||||
work/
|
||||
config/ # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
|
||||
manifest/ # inventory, fingerprints, reports
|
||||
canonical/ # cleansed partitioned dataset
|
||||
rejects/ # quarantined anomalies
|
||||
schema/ # DDL, data dictionary, instruments repository
|
||||
exports/{json,sqlite,postgresql,rdf}/
|
||||
lineage/
|
||||
scripts/ # all Python scripts produced by tasks
|
||||
```
|
||||
2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
|
||||
3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.
|
||||
17
docs/specs_old/02_fs_inventory.md
Normal file
17
docs/specs_old/02_fs_inventory.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Task 02 — Filesystem Inventory
|
||||
|
||||
## Goal
|
||||
Non-destructive inventory of every file under `SOURCE_ROOT`.
|
||||
|
||||
## Input
|
||||
- `work/config/pipeline.json` (Task 01).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
|
||||
2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
|
||||
3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
|
||||
4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `inventory.jsonl` row count == filesystem file count.
|
||||
- `inventory_summary.md` produced; non-CSV extensions listed explicitly.
|
||||
17
docs/specs_old/03_filename_tokenization.md
Normal file
17
docs/specs_old/03_filename_tokenization.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Task 03 — Filename & Path Tokenization
|
||||
|
||||
## Goal
|
||||
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl` (Task 02).
|
||||
- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
|
||||
|
||||
## Actions
|
||||
1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
|
||||
2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
|
||||
3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
|
||||
- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).
|
||||
18
docs/specs_old/04_schema_profiling.md
Normal file
18
docs/specs_old/04_schema_profiling.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Task 04 — Schema Profiling & Fingerprints
|
||||
|
||||
## Goal
|
||||
Capture the record structure of every CSV without full load; compute schema fingerprints.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
|
||||
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
|
||||
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
|
||||
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
|
||||
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
|
||||
- Every profile carries a fingerprint.
|
||||
16
docs/specs_old/05_schema_grouping.md
Normal file
16
docs/specs_old/05_schema_grouping.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Task 05 — Schema Grouping & Inconsistency Report
|
||||
|
||||
## Goal
|
||||
Enumerate distinct record structures per measurement type; expose structural inconsistencies.
|
||||
|
||||
## Input
|
||||
- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
|
||||
2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
|
||||
3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- The number of distinct schemas per measurement type is known and documented.
|
||||
- `inconsistencies.csv` complete — this is the work queue for Task 07.
|
||||
19
docs/specs_old/06_storage_map.md
Normal file
19
docs/specs_old/06_storage_map.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# Task 06 — Storage Map (As-Is / As-Designed)
|
||||
|
||||
## Goal
|
||||
Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
|
||||
|
||||
## Input
|
||||
- All Task 02–05 artifacts.
|
||||
|
||||
## Actions
|
||||
1. Write `manifest/storage_map.md`:
|
||||
- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
|
||||
- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
|
||||
- purpose of each file kind mapped to measurement types and attachment levels;
|
||||
- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
|
||||
2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
|
||||
- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.
|
||||
18
docs/specs_old/07_column_mapping.md
Normal file
18
docs/specs_old/07_column_mapping.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Task 07 — Column Mapping (raw → canonical)
|
||||
|
||||
## Goal
|
||||
For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
|
||||
|
||||
## Input
|
||||
- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
|
||||
- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
|
||||
|
||||
## Actions
|
||||
1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
|
||||
2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
|
||||
3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
|
||||
4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
|
||||
- Escalation list for Task 08 written to `schema/ontology_escalations.md`.
|
||||
18
docs/specs_old/08_ontology_schema.md
Normal file
18
docs/specs_old/08_ontology_schema.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
|
||||
|
||||
## Goal
|
||||
Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
|
||||
|
||||
## Input
|
||||
- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
|
||||
|
||||
## Actions
|
||||
1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
|
||||
2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
|
||||
3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
|
||||
4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
|
||||
5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
|
||||
- No open items in `ontology_escalations.md`.
|
||||
18
docs/specs_old/09_cleanse_normalize.md
Normal file
18
docs/specs_old/09_cleanse_normalize.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Task 09 — Cleanse & Normalize → Canonical Dataset
|
||||
|
||||
## Goal
|
||||
Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
|
||||
|
||||
## Input
|
||||
- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
|
||||
2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
|
||||
3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
|
||||
4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
|
||||
5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
|
||||
- Canonical partitions written; lineage present for every canonical row.
|
||||
17
docs/specs_old/10_anomaly_report.md
Normal file
17
docs/specs_old/10_anomaly_report.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Task 10 — Anomaly Report & Iteration Gate
|
||||
|
||||
## Goal
|
||||
Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
|
||||
|
||||
## Input
|
||||
- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
|
||||
2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
|
||||
3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
|
||||
4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `anomaly_report.md` final; every reject class documented as a true anomaly.
|
||||
- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.
|
||||
17
docs/specs_old/11_export_json.md
Normal file
17
docs/specs_old/11_export_json.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Task 11 — Export: JSON
|
||||
|
||||
## Goal
|
||||
Two JSON products from the frozen canonical snapshot.
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/` (Tasks 09–10).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
|
||||
2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
|
||||
3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
|
||||
4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Both products written; spot-validation: point counts per measurement match canonical.
|
||||
- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).
|
||||
16
docs/specs_old/12_export_sqlite.md
Normal file
16
docs/specs_old/12_export_sqlite.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Task 12 — Export: SQLite
|
||||
|
||||
## Goal
|
||||
Portable single-file SQLite database conforming to the ontology.
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
|
||||
2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
|
||||
3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
|
||||
- `exports/sqlite/EXPORT_MANIFEST.json` written.
|
||||
22
docs/specs_old/13_export_postgresql.md
Normal file
22
docs/specs_old/13_export_postgresql.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Task 13 — Export: PostgreSQL
|
||||
|
||||
## Goal
|
||||
System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
|
||||
|
||||
## Actions
|
||||
1. Generate `exports/postgresql/`:
|
||||
- `01_schema.sql` (DDL, versioned);
|
||||
- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
|
||||
- per-table CSV staging files from canonical (streamed);
|
||||
- `03_copy_load.sql` (`\copy` statements in dependency order);
|
||||
- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
|
||||
- `load.sh` / `load.ps1` orchestration.
|
||||
2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
|
||||
3. Include `lineage` table load (row-range granularity).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
|
||||
- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.
|
||||
17
docs/specs_old/14_export_rdf.md
Normal file
17
docs/specs_old/14_export_rdf.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Task 14 — Export: RDF (Metadata & Provenance Graph)
|
||||
|
||||
## Goal
|
||||
RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
|
||||
|
||||
## Actions
|
||||
1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
|
||||
2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
|
||||
3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
|
||||
4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
|
||||
- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.
|
||||
18
docs/specs_old/15_lineage_build_manifest.md
Normal file
18
docs/specs_old/15_lineage_build_manifest.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
|
||||
|
||||
## Goal
|
||||
Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
|
||||
|
||||
## Input
|
||||
- All `exports/*`, `lineage/`, all config/schema versions.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/finalize.py`:
|
||||
- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
|
||||
- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
|
||||
2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
|
||||
3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
|
||||
|
||||
## Final Result (pipeline done when)
|
||||
- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
|
||||
- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.
|
||||
350
make_lab_config.py
Normal file
350
make_lab_config.py
Normal file
@@ -0,0 +1,350 @@
|
||||
"""Task 01 - Laboratory configuration.
|
||||
|
||||
Writes ./out/config/lab_config.yaml, the authoritative machine-readable
|
||||
laboratory model (volumes, deposition matrix, physical-model coefficients,
|
||||
random seed) that all downstream tasks read, plus the ./out/.done/01.ok
|
||||
completion marker. Spec: docs/specs/01_lab_configuration.md.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent
|
||||
TASK_ID = "01"
|
||||
|
||||
# Transcription of docs/specs/01_lab_configuration.md. This literal is the
|
||||
# ONLY sanctioned home of these values in code; every other task reads the
|
||||
# YAML this script writes (docs/rules/code-config-yaml.md).
|
||||
LAB_CONFIG: dict = {
|
||||
"project": "LabDataStorageEvaluation",
|
||||
"description": "Simulated tribology laboratory (Sandia Pt-Au LDRD context)",
|
||||
"spec": "docs/specs/01_lab_configuration.md",
|
||||
"seed": 20260711,
|
||||
"material_system": {
|
||||
"substrate": {"material": "Ti-6Al-4V", "dimensions_mm": [10, 10, 3]},
|
||||
"adhesion_layer": {"material": "Cr", "process": "sputtered"},
|
||||
"coating": {
|
||||
"material": "Pt-Au",
|
||||
"deposition": "composition gradient across each wafer",
|
||||
"thickness_um_range": [0.3, 1.1],
|
||||
},
|
||||
},
|
||||
"instruments": [
|
||||
{
|
||||
"instrument_id": "rapid",
|
||||
"role": "friction",
|
||||
"name": "RAPID custom high-throughput parallelized 6-probe tribometer",
|
||||
"data_produced": "COF vs cycle per track; test conditions",
|
||||
},
|
||||
{
|
||||
"instrument_id": "ti980",
|
||||
"role": "nanoindentation",
|
||||
"name": "Bruker TI980 TriboIndenter",
|
||||
"data_produced": "hardness, reduced modulus (25 indents per coupon)",
|
||||
},
|
||||
{
|
||||
"instrument_id": "m4_tornado",
|
||||
"role": "composition",
|
||||
"name": "Bruker M4 Tornado micro-XRF",
|
||||
"data_produced": "Pt/Au wt% map per coupon (20x20 grid)",
|
||||
},
|
||||
{
|
||||
"instrument_id": "pvd200",
|
||||
"role": "deposition",
|
||||
"name": "Kurt J. Lesker PVD 200 sputter-down",
|
||||
"data_produced": "batch deposition parameters",
|
||||
},
|
||||
{
|
||||
"instrument_id": "afm",
|
||||
"role": "surface_roughness",
|
||||
"name": "AFM",
|
||||
"data_produced": "topography image metadata + Ra/Rq per coupon",
|
||||
},
|
||||
{
|
||||
"instrument_id": "profilometer",
|
||||
"role": "film_thickness",
|
||||
"name": "Optical profilometry",
|
||||
"data_produced": "thickness map per coupon (10x10 grid)",
|
||||
},
|
||||
{
|
||||
"instrument_id": "simtra",
|
||||
"role": "simulation",
|
||||
"name": "SIMTRA sputter transport Monte-Carlo",
|
||||
"data_produced": "deposition atom-energy / composition profiles per deposition run",
|
||||
},
|
||||
],
|
||||
"deposition_matrix": [
|
||||
{
|
||||
"batch_code": "B721",
|
||||
"pt_gun_tilt_deg": 20,
|
||||
"au_gun_tilt_deg": 0,
|
||||
"pt_power_W": 150,
|
||||
"au_power_W": 50,
|
||||
"pt_discharge_V": 432,
|
||||
"au_discharge_V": 311,
|
||||
},
|
||||
{
|
||||
"batch_code": "B722",
|
||||
"pt_gun_tilt_deg": 20,
|
||||
"au_gun_tilt_deg": 20,
|
||||
"pt_power_W": 100,
|
||||
"au_power_W": 100,
|
||||
"pt_discharge_V": 399,
|
||||
"au_discharge_V": 352,
|
||||
},
|
||||
{
|
||||
"batch_code": "B723",
|
||||
"pt_gun_tilt_deg": 20,
|
||||
"au_gun_tilt_deg": 20,
|
||||
"pt_power_W": 150,
|
||||
"au_power_W": 50,
|
||||
"pt_discharge_V": 430,
|
||||
"au_discharge_V": 311,
|
||||
},
|
||||
{
|
||||
"batch_code": "B724",
|
||||
"pt_gun_tilt_deg": 0,
|
||||
"au_gun_tilt_deg": 20,
|
||||
"pt_power_W": 50,
|
||||
"au_power_W": 150,
|
||||
"pt_discharge_V": 376,
|
||||
"au_discharge_V": 340,
|
||||
},
|
||||
],
|
||||
"hierarchy": {
|
||||
"batches": 4,
|
||||
"wafers_per_batch": 3,
|
||||
"coupon_grid_per_wafer": [7, 7],
|
||||
"coupons_per_wafer": 49,
|
||||
},
|
||||
"friction_assignment": {
|
||||
"friction_coupons_total": 480,
|
||||
"reserve_coupons_total": 108,
|
||||
"runs": 4,
|
||||
"coupons_per_run": 120,
|
||||
"plates_per_run": 5,
|
||||
"probes_per_plate": 6,
|
||||
"coupons_per_probe_square": 4,
|
||||
"tracks_per_friction_coupon": 3,
|
||||
"counterfaces_per_holder": 3,
|
||||
"fresh_counterface_per_track": True,
|
||||
},
|
||||
"volumes": {
|
||||
"coupons_total": 588,
|
||||
"tracks_total": 1440,
|
||||
"cycles_per_track": 1000,
|
||||
"cycle_rows_total": 1440000,
|
||||
"loop_every_n_cycles": 100,
|
||||
"loops_per_track": 10,
|
||||
"loop_points_per_loop": 200,
|
||||
"loop_points_total": 2880000,
|
||||
"xrf_grid": [20, 20],
|
||||
"xrf_points_per_coupon": 400,
|
||||
"xrf_points_total": 235200,
|
||||
"profilometry_grid": [10, 10],
|
||||
"profilometry_points_per_coupon": 100,
|
||||
"nanoindentation_indents_per_coupon": 25,
|
||||
"simtra_rows_per_batch": 1000,
|
||||
},
|
||||
"runs": [
|
||||
{"run_code": "R1", "environment": "lab_air", "date": "2026-04-06", "operator": "A. Reyes"},
|
||||
{"run_code": "R2", "environment": "lab_air", "date": "2026-04-13", "operator": "K. Patel"},
|
||||
{"run_code": "R3", "environment": "dry_n2", "date": "2026-04-20", "operator": "A. Reyes"},
|
||||
{"run_code": "R4", "environment": "dry_n2", "date": "2026-04-27", "operator": "M. Novak"},
|
||||
],
|
||||
"schedule": {
|
||||
"deposition_start_date": "2026-03-02",
|
||||
"batch_interval_days": 7,
|
||||
"characterization_lag_days": 3,
|
||||
},
|
||||
"test_conditions": {
|
||||
"normal_load_mN": 100,
|
||||
"stroke_mm": 1.0,
|
||||
"speed_mm_s": 1.0,
|
||||
"counterface": {"material": "ruby (Al2O3)", "diameter_mm": 3.175},
|
||||
"rh_pct": {"lab_air": 45, "dry_n2": 2},
|
||||
"temperature_C": 23,
|
||||
"temperature_tolerance_C": 1,
|
||||
},
|
||||
"physical_models": {
|
||||
"au_gradient": {
|
||||
"description": "linear Au wt% gradient along wafer x, batch-dependent center",
|
||||
"batch_center_wtpct": {"B721": 8, "B722": 25, "B723": 10, "B724": 60},
|
||||
"wafer_span_wtpct": 5,
|
||||
"xrf_noise_sd_wtpct": 0.3,
|
||||
},
|
||||
"cof_run_in": {
|
||||
"formula": "cof(c) = cof_ss + (cof_0 - cof_ss) * exp(-c / tau) + N(0, sigma)",
|
||||
"cof_0_range": [0.35, 0.5],
|
||||
"tau_cycles_range": [100, 400],
|
||||
"noise_sd": 0.01,
|
||||
},
|
||||
"cof_steady_state": {
|
||||
"formula": "cof_ss = intercept + slope_per_au_wtpct * au_wtpct_mean, clamped to >= floor",
|
||||
"lab_air": {"intercept": 0.32, "slope_per_au_wtpct": -0.0015},
|
||||
"dry_n2": {"intercept": 0.24, "slope_per_au_wtpct": -0.0012},
|
||||
"floor": 0.12,
|
||||
},
|
||||
"hardness_GPa": {
|
||||
"formula": "H = intercept + slope_per_au_wtpct * au_wtpct_mean + N(0, noise_sd)",
|
||||
"intercept": 8.5,
|
||||
"slope_per_au_wtpct": -0.05,
|
||||
"noise_sd": 0.25,
|
||||
},
|
||||
"reduced_modulus_GPa": {
|
||||
"formula": "Er = intercept + slope_per_au_wtpct * au_wtpct_mean + N(0, noise_sd)",
|
||||
"intercept": 190,
|
||||
"slope_per_au_wtpct": -0.6,
|
||||
"noise_sd": 4,
|
||||
},
|
||||
"roughness_ra_nm": {
|
||||
"distribution": "lognormal",
|
||||
"median_nm": 5,
|
||||
"sigma_log": 0.3,
|
||||
"rq_over_ra": 1.25,
|
||||
},
|
||||
"thickness_um": {
|
||||
"profile": "radial parabolic across wafer plus noise",
|
||||
"center_um": 1.05,
|
||||
"edge_um": 0.35,
|
||||
"noise_sd_um": 0.02,
|
||||
"range_um": [0.3, 1.1],
|
||||
},
|
||||
"wear_archard": {
|
||||
# Spec 01 fixes only the model shape (V = k*F*s, k depends on Au%
|
||||
# and environment); these coefficients are project defaults chosen
|
||||
# to yield Sandia-plausible low-wear volumes at 100 mN / 2 m.
|
||||
"formula": "V = k * F * s; k = k0 * (1 + slope_per_au_wtpct * au_wtpct_mean), lognormal noise, clamped to >= k_floor",
|
||||
"k0_mm3_per_N_m": {"lab_air": 1.5e-07, "dry_n2": 6.0e-08},
|
||||
"slope_per_au_wtpct": {"lab_air": -0.008, "dry_n2": -0.010},
|
||||
"k_floor_mm3_per_N_m": 1.0e-09,
|
||||
"noise_sigma_log": 0.15,
|
||||
},
|
||||
"nanoindentation": {
|
||||
"max_load_mN": 10,
|
||||
"indent_grid": [5, 5],
|
||||
"grid_pitch_um": 20,
|
||||
},
|
||||
"simtra": {
|
||||
"angle_deg_range": [0, 90],
|
||||
"energy_eV_max": 50,
|
||||
"surface_binding_energy_eV": {"pt": 5.84, "au": 3.81},
|
||||
"flux_per_W": 0.02,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
YAML_HEADER = (
|
||||
"# Generated by make_lab_config.py (task 01) from docs/specs/01_lab_configuration.md.\n"
|
||||
"# Do not hand-edit: regenerate via `python make_lab_config.py`.\n"
|
||||
)
|
||||
|
||||
|
||||
def validate_config(cfg: dict) -> None:
|
||||
"""Assert the arithmetic identities between declared volumes (spec 01)."""
|
||||
h = cfg["hierarchy"]
|
||||
f = cfg["friction_assignment"]
|
||||
v = cfg["volumes"]
|
||||
checks = [
|
||||
("coupons_total", v["coupons_total"], h["batches"] * h["wafers_per_batch"] * h["coupons_per_wafer"]),
|
||||
("coupons_per_wafer", h["coupons_per_wafer"], h["coupon_grid_per_wafer"][0] * h["coupon_grid_per_wafer"][1]),
|
||||
("friction+reserve", v["coupons_total"], f["friction_coupons_total"] + f["reserve_coupons_total"]),
|
||||
("friction_coupons_total", f["friction_coupons_total"], f["runs"] * f["coupons_per_run"]),
|
||||
("coupons_per_run", f["coupons_per_run"], f["plates_per_run"] * f["probes_per_plate"] * f["coupons_per_probe_square"]),
|
||||
("tracks_total", v["tracks_total"], f["friction_coupons_total"] * f["tracks_per_friction_coupon"]),
|
||||
("cycle_rows_total", v["cycle_rows_total"], v["tracks_total"] * v["cycles_per_track"]),
|
||||
("loops_per_track", v["loops_per_track"], v["cycles_per_track"] // v["loop_every_n_cycles"]),
|
||||
("loop_points_total", v["loop_points_total"], v["tracks_total"] * v["loops_per_track"] * v["loop_points_per_loop"]),
|
||||
("xrf_points_per_coupon", v["xrf_points_per_coupon"], v["xrf_grid"][0] * v["xrf_grid"][1]),
|
||||
("xrf_points_total", v["xrf_points_total"], v["coupons_total"] * v["xrf_points_per_coupon"]),
|
||||
("profilometry_points_per_coupon", v["profilometry_points_per_coupon"], v["profilometry_grid"][0] * v["profilometry_grid"][1]),
|
||||
("runs_count", f["runs"], len(cfg["runs"])),
|
||||
("batches_count", h["batches"], len(cfg["deposition_matrix"])),
|
||||
]
|
||||
for name, declared, derived in checks:
|
||||
if declared != derived:
|
||||
raise ValueError(f"volume identity failed: {name}: declared {declared} != derived {derived}")
|
||||
logger.info("all %d volume identities hold", len(checks))
|
||||
|
||||
|
||||
def write_yaml(cfg: dict, config_path: Path) -> int:
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
body = yaml.safe_dump(cfg, sort_keys=False, allow_unicode=False, width=100)
|
||||
text = YAML_HEADER + body
|
||||
with open(config_path, "w", encoding="utf-8", newline="\n") as fh:
|
||||
fh.write(text)
|
||||
reloaded = yaml.safe_load(text)
|
||||
if reloaded != cfg:
|
||||
raise RuntimeError(f"YAML round-trip mismatch for {config_path}")
|
||||
return len(text.encode("utf-8"))
|
||||
|
||||
|
||||
def write_marker(marker_path: Path, config_path: Path, config_bytes: int, cfg: dict) -> None:
|
||||
v = cfg["volumes"]
|
||||
f = cfg["friction_assignment"]
|
||||
h = cfg["hierarchy"]
|
||||
lines = [
|
||||
f"task: '{TASK_ID}'",
|
||||
"status: ok",
|
||||
f"config_file: {config_path.as_posix()}",
|
||||
f"config_bytes: {config_bytes}",
|
||||
f"seed: {cfg['seed']}",
|
||||
f"batches: {h['batches']}",
|
||||
f"wafers: {h['batches'] * h['wafers_per_batch']}",
|
||||
f"coupons: {v['coupons_total']}",
|
||||
f"friction_coupons: {f['friction_coupons_total']}",
|
||||
f"tracks: {v['tracks_total']}",
|
||||
f"cycle_rows: {v['cycle_rows_total']}",
|
||||
f"loop_points: {v['loop_points_total']}",
|
||||
f"xrf_points: {v['xrf_points_total']}",
|
||||
]
|
||||
marker_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(marker_path, "w", encoding="utf-8", newline="\n") as fh:
|
||||
fh.write("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Task 01: write out/config/lab_config.yaml")
|
||||
parser.add_argument("--log-level", default="INFO", choices=["DEBUG", "INFO", "WARNING", "ERROR"])
|
||||
parser.add_argument("--out-root", type=Path, default=REPO_ROOT / "out", help="artifact tree root (default: ./out)")
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(
|
||||
level=args.log_level,
|
||||
stream=sys.stderr,
|
||||
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
|
||||
)
|
||||
|
||||
out_root: Path = args.out_root
|
||||
config_path = out_root / "config" / "lab_config.yaml"
|
||||
marker_path = out_root / ".done" / f"{TASK_ID}.ok"
|
||||
try:
|
||||
if marker_path.exists():
|
||||
logger.info("re-run: removing stale marker %s", marker_path)
|
||||
marker_path.unlink()
|
||||
validate_config(LAB_CONFIG)
|
||||
config_bytes = write_yaml(LAB_CONFIG, config_path)
|
||||
write_marker(marker_path, config_path, config_bytes, LAB_CONFIG)
|
||||
except Exception:
|
||||
logger.critical("task %s failed", TASK_ID, exc_info=True)
|
||||
return 1
|
||||
|
||||
v = LAB_CONFIG["volumes"]
|
||||
print(f"task 01 ok: {config_path} ({config_bytes} bytes)")
|
||||
print(
|
||||
f"volumes: batches=4 wafers=12 coupons={v['coupons_total']} tracks={v['tracks_total']} "
|
||||
f"cycle_rows={v['cycle_rows_total']} loop_points={v['loop_points_total']} xrf_points={v['xrf_points_total']}"
|
||||
)
|
||||
print(f"marker: {marker_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
12
requirements.txt
Normal file
12
requirements.txt
Normal file
@@ -0,0 +1,12 @@
|
||||
# Closed dependency list - docs/rules/code-python-style.md section 2.
|
||||
# Adding a package here requires updating that rule first.
|
||||
|
||||
numpy # task 03: vectorized generation, seeded RNG
|
||||
PyYAML # tasks 01, 03, 10: YAML configs
|
||||
rdflib # tasks 04, 07: JSON-LD validation, RDF serialization
|
||||
pyoxigraph # tasks 07, 09: embedded queryable triplestore (pip name of oxigraph)
|
||||
ijson # tasks 08, 09: streaming JSON parsing
|
||||
psycopg[binary] # tasks 06, 08, 09: PostgreSQL COPY and queries
|
||||
psutil # task 09: RSS / CPU / IO measurement
|
||||
matplotlib # task 11: report charts
|
||||
pandas # task 08 only: separately-measured CSV query variant
|
||||
Reference in New Issue
Block a user