diff --git a/.cursorrules b/.cursorrules new file mode 100644 index 0000000..c93b983 --- /dev/null +++ b/.cursorrules @@ -0,0 +1,48 @@ +# Cursor rules for LabDataStorageEvaluation + +## Commit messages (mandatory format) + +Every commit message you generate MUST follow this convention exactly. + +First line: type(scope): description +- lowercase imperative English summary, under 72 characters, no trailing period +- type is one of (closed list): feat fix perf refactor security docs ci chore test +- scope is one of (closed list): specs rules docs config datagen convert bench report tools all + +Scope selection by changed files: +- specs = docs/specs/** +- rules = docs/rules/** +- docs = other prose docs (README.md, CLAUDE.md, docs/examples/**) +- config = configuration files (.gitignore, .cursorrules, editable YAML defaults) +- datagen = lab-configuration and data-generation code (tasks 01-03) +- convert = format converters: JSON-LD, SQLite, PostgreSQL, RDF (tasks 04-07) +- bench = benchmark queries, runner, extrapolation (tasks 08-10) +- report = reporting code and templates (task 11) +- tools = helper scripts under tools/ +- all = multiple areas, or when no narrower scope honestly fits + +Body: the subject is only a title. If the diff touches more than one area, +add a blank line and 2-6 concise bullets covering every meaningful changed +area. Split unrelated work into separate commits. + +Special forms allowed as-is: "Merge ..." commits; "release(scope): vX.Y.Z". + +Never emit vague or unprefixed subjects such as: "Update files", +"Improve project", "clarify rules", "changes". + +Examples of good subjects: +- feat(datagen): stream friction cycle rows during CSV generation +- fix(convert): match sqlite row counts against MANIFEST before commit +- docs(rules): add benchmark methodology rule +- chore(config): ignore out/ artifact tree + +## Typography + +Use plain ASCII typography in everything you write into this repository: +hyphen-minus instead of en/em dashes, straight quotes, three dots instead +of the ellipsis character. See docs/rules/code-data-formatting.md. + +## Python + +Python code in this repository uses TABS for indentation. See +docs/rules/code-python-style.md for the closed dependency list. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d07d4cb --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +# Generated pipeline artifacts - reproducible from specs + rules + code +# (see docs/rules/build-pipeline-tasks.md) +out/ + +# Python +__pycache__/ +*.pyc +.venv/ +venv/ + +# Editor / OS noise +.vscode/ +.idea/ +Thumbs.db +.DS_Store diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..ca8b7c9 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,112 @@ +# LabDataStorageEvaluation - Project Rules + +A reproducible evaluation pipeline for laboratory data storage formats, +built around a simulated tribology laboratory (Sandia Pt-Au LDRD context). +The pipeline (tasks 01-11, specs in [docs/specs/](docs/specs/)): + +1. Defines the lab model (`lab_config.yaml`) and process-flow diagrams. +2. Generates **~150 MB of physically plausible measurement data in CSV** + (the canonical raw format), deterministically from **seed 20260711**. +3. Converts the corpus into 4 target formats: **JSON-LD** (full + hybrid), + **SQLite**, **PostgreSQL 16** (indexed + partitioned), **RDF + triplestore** (oxigraph). +4. Benchmarks **7 retrieval scenarios (Q1-Q7) x 5 formats**, measuring + wall time, peak RAM, CPU, disk I/O, and footprint. +5. Extrapolates to 600 GB / 1.2 TB / 6 TB and produces a ranked report + (`REPORT.md` + charts). + +Everything runs **locally** (Windows development host, local PostgreSQL 16, +no cloud services, no server deploy). The stack is **Python 3.11+ (tabs for +indentation)** with a closed dependency list; there is **no UI, no HTTP +API, no authentication** in this project. All artifacts land under +`./out/` (git-ignored); tasks signal completion via `./out/.done/.ok` +markers. + +These rules are **binding** for every contributor (human or AI agent) +touching this repository. They override any pattern found in imported +specs, external tutorials, or older project docs that predate this file. + +--- + +## Mandatory rules - load before starting matching work + +When the user's request matches a topic below, **before writing any code or +producing a plan**, run `Read` on the listed rule file(s). The rules are +binding; ignoring them produces broken work that the user will reject. + +| If the task involves... | You MUST first Read | +|---|---| +| Writing or changing any Python (style, tabs, dependency list, streaming discipline) | [docs/rules/code-python-style.md](docs/rules/code-python-style.md) | +| Data generation, physical models, random values, seeds, reproducibility | [docs/rules/data-determinism.md](docs/rules/data-determinism.md) | +| Naming entities/IDs, timestamps, measurement columns, units, `au_wtpct_mean` | [docs/rules/data-naming-units.md](docs/rules/data-naming-units.md) | +| SQLite / PostgreSQL schema, DDL, keys, indexes, bulk loading, `track_summary` | [docs/rules/db-sql-schema.md](docs/rules/db-sql-schema.md) | +| Benchmark queries, the runner, resource measurement, timeouts, extrapolation, scoring | [docs/rules/bench-methodology.md](docs/rules/bench-methodology.md) | +| The `./out/` tree, `.done` markers, task order/parallelism, `storage_sizes.csv`, re-runs | [docs/rules/build-pipeline-tasks.md](docs/rules/build-pipeline-tasks.md) | +| Validation: MANIFEST, row counts, result checksums, expected volumes, pytest | [docs/rules/test-pipeline-validation.md](docs/rules/test-pipeline-validation.md) | +| Python error handling (`try` / `except` / `raise`, abort semantics) | [docs/rules/code-error-handling.md](docs/rules/code-error-handling.md) | +| Logging, `print()` vs logger, log levels | [docs/rules/obs-python-logging.md](docs/rules/obs-python-logging.md) | +| Configuration values: the YAML configs, constants, the PostgreSQL DSN, CLI flags | [docs/rules/code-config-yaml.md](docs/rules/code-config-yaml.md) | +| Writing, editing, or reviewing comments in source code (any language) | [docs/rules/code-comments-policy.md](docs/rules/code-comments-policy.md) | +| Prose strings in source/config/docs (typography / encoding / dashes) | [docs/rules/code-data-formatting.md](docs/rules/code-data-formatting.md) | +| Adding or renaming a rule file in `docs/rules/` | [docs/rules/meta-rules-naming.md](docs/rules/meta-rules-naming.md) | +| Anything where misinterpretation could be destructive (deleting/regenerating `./out/`, dropping the `tribo` DB, changing the seed or `lab_config.yaml`, editing specs, git history) | [docs/rules/meta-no-assumptions.md](docs/rules/meta-no-assumptions.md) | +| Every response (length / formatting discipline) | [docs/rules/meta-concise-answers.md](docs/rules/meta-concise-answers.md) | +| Asking the user any question | [docs/rules/meta-concise-questions.md](docs/rules/meta-concise-questions.md) | +| Every response (communication language / register - Russian, no jargon or slang) | [docs/rules/meta-language-style.md](docs/rules/meta-language-style.md) | +| Any git operation that would create a commit, tag, merge, revert, rebase, cherry-pick, or push | [docs/rules/meta-git-commits.md](docs/rules/meta-git-commits.md) | +| Generating commit messages via Cursor, or maintaining the `.cursorrules` artifact | [docs/rules/meta-cursor-rules.md](docs/rules/meta-cursor-rules.md) | + +Multiple matches -> Read all matching rule files. Unsure -> Read the closest one; +over-loading is cheap, missing a rule is expensive. + +--- + +## Other binding constraints + +These are short, cross-cutting constraints. They are still **binding**; the +linked rule file holds the detailed spec. + +- **Determinism is the foundation.** Seed `20260711` lives in + `lab_config.yaml` and is never hardcoded; only seeded RNGs; no + wall-clock values inside the corpus; regeneration is byte-identical + (proof: `MANIFEST.csv` sha256). See + [docs/rules/data-determinism.md](docs/rules/data-determinism.md). +- **A wrong artifact is worse than no artifact.** Validation gates (row + counts vs MANIFEST, result checksums with 1e-9 tolerance, the + 120-200 MB size gate) abort loudly; a `.done` marker is written only + after full success. See + [docs/rules/code-error-handling.md](docs/rules/code-error-handling.md) + and [docs/rules/test-pipeline-validation.md](docs/rules/test-pipeline-validation.md). +- **Tasks communicate only through `./out/` artifacts.** The tree is + git-ignored; sources of truth are specs + rules + code. Benchmarks run + on a quiet machine. See + [docs/rules/build-pipeline-tasks.md](docs/rules/build-pipeline-tasks.md). +- **Measurement honesty.** Subprocess isolation per cell, 1 warm-up + 3 + measured runs, randomized order, 30-min timeout recorded as `TIMEOUT`, + failed checksums invalidate the measurement, projected numbers are + always labeled. See + [docs/rules/bench-methodology.md](docs/rules/bench-methodology.md). +- **Python: tabs, closed dependency list.** Stdlib first; only the + packages listed in + [docs/rules/code-python-style.md](docs/rules/code-python-style.md); + task 03 is stdlib + numpy only; stream everything bulk. +- **Commit message format (mandatory).** First line + `type(scope): description`, lowercase imperative English, under 72 + chars. Types (closed): `feat fix perf refactor security docs ci chore + test`. Scopes (closed): `specs rules docs config datagen convert bench + report tools all`. The AI never creates commits - git is read-only for + agents; the user runs all git write commands. See + [docs/rules/meta-git-commits.md](docs/rules/meta-git-commits.md). + +--- + +## Canonical source of truth + +If this file and any `docs/rules/.md` disagree, **the rule file wins** +within its topic - that is where the detailed binding spec lives. If a rule +and a task spec in `docs/specs/` disagree on a convention, **the rule wins** +and the spec is updated to match; the specs remain authoritative for task +scope, inputs, outputs, and dependencies. If this file and any other +document (README, older docs, code comments) disagree, **this file plus the +rule files win**. Update the other document to match; do not work around +the rules. diff --git a/README.md b/README.md index 1a2ee4e..66eb8ed 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,257 @@ # LabDataStorageEvaluation -Build a complete, reproducible pipeline that: - - -Defines a simulated tribology laboratory (Sandia Pt-Au LDRD context) with real instruments and a realistic test workflow. -Generates ~150 MB of physically plausible measurement data in CSV (the canonical raw format). -Converts the CSV corpus into 4 target storage formats: JSON-LD, SQLite (optimized), PostgreSQL (indexed + partitioned), RDF TripleStore. -Benchmarks 7 laboratory-standard retrieval scenarios (Q1–Q7) against all 5 formats, measuring wall time, peak RAM, CPU utilization, disk I/O, and disk footprint. -Extrapolates measured results to 600 GB and beyond (0.6 / 1.2 / 6 TB) using complexity-aware scaling models. -Produces summary tables and charts that clearly rank storage formats for laboratory use (analysis, reporting, search/filter, archiving, inter-lab exchange). \ No newline at end of file +A reproducible pipeline that answers one practical question: **which storage +format should a materials-science laboratory use for its measurement data?** + +The pipeline simulates a tribology laboratory (Sandia Pt-Au LDRD context), +generates ~150 MB of physically plausible raw measurement data in CSV, +converts the corpus into four additional storage formats, benchmarks seven +laboratory-standard retrieval scenarios against all five formats, extrapolates +the measurements to 600 GB - 6 TB, and produces a ranked decision report for +five laboratory use cases: analysis, reporting, search/filtering, archiving, +and inter-lab exchange. + +- Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11). +- Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md). + +## Goals + +1. Define a simulated tribology laboratory with real instruments and a + realistic test workflow. +2. Generate ~150 MB of physically plausible measurement data in CSV (the + canonical raw format), deterministically from seed `20260711`. +3. Convert the CSV corpus into 4 target storage formats: JSON-LD, SQLite + (optimized), PostgreSQL 16 (indexed + partitioned), RDF triplestore. +4. Benchmark 7 retrieval scenarios (Q1-Q7) against all 5 formats, measuring + wall time, peak RAM, CPU utilization, disk I/O, and disk footprint. +5. Extrapolate measured results to 600 GB / 1.2 TB / 6 TB with + complexity-aware scaling models. +6. Produce summary tables, charts, and a weighted scoreboard that rank the + formats per use case. + +## Pipeline + +```mermaid +flowchart TD + T01[01_lab_configuration] --> T02[02_process_flow_diagrams] + T01 --> T03[03_csv_data_generation] + T03 --> T04[04_convert_json] + T03 --> T05[05_convert_sqlite] + T03 --> T06[06_convert_postgresql] + T03 --> T07[07_convert_rdf] + T04 --> T08[08_benchmark_queries] + T05 --> T08 + T06 --> T08 + T07 --> T08 + T03 --> T08 + T08 --> T09[09_benchmark_execution] + T09 --> T10[10_extrapolation_model] + T10 --> T11[11_reporting] + T09 --> T11 +``` + +Execution order: `01 -> 02 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11`. +Tasks 04-07 are independent and may run in parallel; task 09 requires a quiet +machine (no parallel work). Every task writes a completion marker +`./out/.done/.ok` and validates its dependencies' markers before starting. + +| Task | Script | Main output | Status | +|---|---|---|---| +| 01 Lab configuration | `make_lab_config.py` | `out/config/lab_config.yaml` | implemented | +| 02 Process flow diagrams | `make_process_flow.py` | `out/report/process_flow.md`, `out/report/diagrams/` | planned | +| 03 CSV data generation | `generate_data.py` | `out/csv/` tree + `MANIFEST.csv` | planned | +| 04 Convert to JSON-LD | `convert_json.py` | `out/json/full/`, `out/json/hybrid/dataset.jsonld` | planned | +| 05 Convert to SQLite | `convert_sqlite.py` | `out/sqlite/tribo.db` | planned | +| 06 Convert to PostgreSQL | `convert_postgresql.py` | live `tribo` DB + `out/pg/tribo.dump` | planned | +| 07 Convert to RDF | `convert_rdf.py` | `out/rdf/dataset.nt.gz`, oxigraph store | planned | +| 08 Benchmark queries | `make_queries.py` | `out/bench/queries/`, `out/bench/expected/` | planned | +| 09 Benchmark execution | `bench_runner.py` | `out/bench/results_raw.csv`, `results_median.csv` | planned | +| 10 Extrapolation | `extrapolate.py` | `out/bench/extrapolation.csv`, `hardware_sizing.csv` | planned | +| 11 Reporting | `report.py` | `out/report/REPORT.md`, charts, tables | planned | + +## The simulated laboratory + +Ti-6Al-4V coupons (10 x 10 x 3 mm) with a sputtered Cr adhesion layer and a +Pt-Au coating deposited as a composition gradient (film thickness +0.3-1.1 um). Four deposition batches (B721-B724) vary gun tilt, power, and +discharge voltage. Instruments: RAPID 6-probe tribometer (friction), Bruker +TI980 (nanoindentation), Bruker M4 Tornado micro-XRF (composition), Kurt J. +Lesker PVD 200 (deposition), AFM (roughness), optical profilometry +(thickness), SIMTRA (sputter transport simulation). + +```mermaid +flowchart LR + B[Batch x4] --> W[Wafer x3 per batch] + W --> C[Coupon x49 per wafer, 7x7 grid] + C --> X[xrf_map: 400 points] + C --> N[nanoindentation: 25 indents] + C --> A[afm: Ra / Rq] + C --> P[profilometry: 100 points] + C --> T[Track x3, friction coupons only] + T --> CY[cof_vs_cycle: 1000 cycles] + T --> L[friction_loops: 10 x 200 points] + T --> WR[wear: 1 row] +``` + +| Entity | Count | +|---|---| +| Batches | 4 | +| Wafers | 12 | +| Coupons | 588 (480 friction-tested + 108 reserve/QA) | +| Tracks | 1,440 | +| Friction cycle rows | 1,440,000 | +| Friction loop points | 2,880,000 | +| XRF map points | 235,200 | + +Friction runs R1/R2 execute in Lab Air, R3/R4 in Dry N2. Generated values +follow explicit physical models (COF run-in exponential, steady-state COF and +hardness vs Au content, Archard wear), so benchmark queries return physically +meaningful results. + +## Storage formats under evaluation + +| # | Format | Role in the comparison | +|---|---|---| +| 1 | CSV (raw tree) | Canonical instrument output; baseline | +| 2 | JSON-LD (full + hybrid) | Ontology-annotated exchange format (FAIR) | +| 3 | SQLite | Single-file relational, optimized for size and speed | +| 4 | PostgreSQL 16 | Production relational: indexed, partitioned | +| 5 | RDF triplestore (oxigraph) | Semantic-web store queried via SPARQL | + +## Benchmark scenarios (Q1-Q7) + +| ID | Scenario | Access pattern | +|---|---|---| +| Q1 | COF vs cycle curve for one track | Point read | +| Q2 | Steady-state COF for coupons with mean Au = 10 +/- 0.5 wt% | Indexed filter | +| Q3 | Hardness vs steady-state COF across all batches | Indexed join | +| Q4 | Tracks in Dry N2, 100 mN, cof_ss > 0.20 (anomaly filter) | Indexed filter | +| Q5 | Mean run-in cycles per batch over ALL raw cycle rows | Forced full scan | +| Q6 | Wear volume vs load per coupon | Indexed join | +| Q7 | Stribeck-style mean COF by (environment, speed-load bucket) | Full aggregation | + +Protocol: every cell (format x query) runs in its own subprocess; 1 warm-up + +3 measured runs; randomized cell order; 30-minute hard timeout recorded as +`TIMEOUT`; every result validated against a canonical checksum before the +measurement counts. See +[docs/rules/bench-methodology.md](docs/rules/bench-methodology.md). + +## Repository layout + +``` +docs/ + specs/ task specifications 00-11 (WHAT to build) + rules/ binding conventions (HOW work is done) + examples/ imported reference materials (not binding) +out/ ALL generated artifacts (git-ignored, reproducible): + config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/ +make_lab_config.py task 01 entry script (one script per task, repo root) +requirements.txt closed dependency list (docs/rules/code-python-style.md) +``` + +The `out/` tree - including the ~150 MB CSV corpus, the ~1 GB JSON-LD variant, +databases, and benchmark results - is **excluded from git**. Sources of truth +are the specs, the rules, and the code; every artifact regenerates +deterministically from them. + +## Getting started + +Prerequisites: Python 3.11+ (3.12 tested). PostgreSQL 16 is required only for +tasks 06/08/09; its DSN comes from the `TRIBO_PG_DSN` environment variable +(default `postgresql://postgres@localhost:5432/tribo`). + +Windows (PowerShell): + +```powershell +py -3 -m venv .venv +.venv\Scripts\python -m pip install -r requirements.txt +.venv\Scripts\python make_lab_config.py +``` + +Linux: + +```bash +python3 -m venv .venv +.venv/bin/python -m pip install -r requirements.txt +.venv/bin/python make_lab_config.py +``` + +Each task script supports `--log-level {DEBUG,INFO,WARNING,ERROR}` and prints +its protocol output (paths, counts, sizes) to stdout. A task refuses to run if +a dependency's `out/.done/.ok` marker is missing. + +Note on cold-cache runs: the page-cache drop used by the cold-cache benchmark +pass is Linux-only; on Windows hosts that pass is skipped and marked as such. +Benchmark execution is planned for a dedicated Linux host. + +## Results + +This section is populated by tasks 02, 09, 10, and 11. All artifacts land +under `out/` (git-ignored); the placeholders below name the exact files the +pipeline produces, so the section can be filled in by copying the generated +tables and linking the generated images. + +### Process flow diagrams (task 02) + +`out/report/process_flow.md` with `out/report/diagrams/D1..D6.mermaid`: +D1 coupon assembly, D2 characterization and batch assembly, D3 testing tree, +D4 tribometer session sequence, D5 execution loop, D6 data hierarchy. + +### Measured results (task 09) - to be generated + +| Placeholder | Source file | +|---|---| +| Storage footprint per format (measured + compressed) | `out/bench/storage_sizes.csv` | +| Q1-Q7 median wall time / peak RAM / CPU / read MB | `out/bench/results_median.csv` | + +### Projections (task 10) - to be generated + +| Placeholder | Source file | +|---|---| +| Projected wall time per format at 0.6 / 1.2 / 6 TB with IMPRACTICAL/FAIL flags | `out/bench/extrapolation.csv` | +| Hardware sizing and cost per format at scale | `out/bench/hardware_sizing.csv` | + +### Charts (task 11) - to be generated + +![C1 - disk footprint per format, log scale](out/report/charts/c1_disk_footprint.png) +![C2 - RAM requirement per format, log scale](out/report/charts/c2_ram_requirement.png) +![C3 - per-query wall time, one chart per query](out/report/charts/c3_q1_wall_time.png) +![C4 - degradation curves 150 MB to 6 TB, log-log](out/report/charts/c4_degradation_full_scan.png) +![C5 - weighted score, stacked bars](out/report/charts/c5_weighted_score.png) +![C6 - cost vs performance at 600 GB, log-log](out/report/charts/c6_cost_vs_performance.png) + +(Images appear after running the full pipeline; task 11 writes them to +`out/report/charts/` under exactly these names.) + +### Final scoreboard (task 11) - to be generated + +Weighted scoring, max 80 points: search speed x5 (0-50), RAM economy x2 +(0-20), disk economy x1 (0-10); TIMEOUT/FAIL projects to 0. + +| Format | Search (0-50) | RAM (0-20) | Disk (0-10) | Total (0-80) | +|---|---|---|---|---| +| CSV | TBD | TBD | TBD | TBD | +| JSON-LD | TBD | TBD | TBD | TBD | +| SQLite | TBD | TBD | TBD | TBD | +| PostgreSQL | TBD | TBD | TBD | TBD | +| RDF | TBD | TBD | TBD | TBD | + +| Use case | Recommended format | +|---|---| +| Analysis | TBD | +| Report generation | TBD | +| Search / filtering | TBD | +| Archiving | TBD | +| Inter-lab exchange | TBD | + +## Conventions + +- IDs: `batch:B721`, `wafer:B721-W1`, `coupon:B721-W1-C07`, + `track:B721-W1-C07-T2`, runs `R1..R4`. +- Timestamps: ISO-8601 UTC; all corpus timestamps are simulated. +- Units are explicit ASCII suffixes in column names (`_mN`, `_um`, `_GPa`, + `_wtpct`, `_nm`). +- Random seed `20260711`, read from `lab_config.yaml`; regeneration is + byte-identical (proof: `MANIFEST.csv` sha256). +- Python: tabs for indentation; closed dependency list; streaming discipline + (the corpus never fits in memory by design). diff --git a/docs/examples/TribologyData_MasterPlan_v1.md b/docs/examples/TribologyData_MasterPlan_v1.md new file mode 100644 index 0000000..4fb69a7 --- /dev/null +++ b/docs/examples/TribologyData_MasterPlan_v1.md @@ -0,0 +1,248 @@ +# Tribology Lab Data — Master Plan + +**Discovery · Cleansing/Normalization · Ontology · Conversion** + +Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility +Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store +Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation +Status: Plan v1 — Draft + +--- + +## 0. Objective + +Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate. + +The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats. + +--- + +## 1. Guiding principles + +1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated. +2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought. +3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize. +4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria. +5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred. +6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge. +7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates. + +--- + +## 2. Phase overview + +| Phase | Name | Nature | Primary output | +|-------|------|--------|----------------| +| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints | +| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository | +| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs | + +Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**. + +--- + +## 3. Phase 1 — Storage Discovery & Non-destructive Profiling + +**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies. + +### 3.1 Inputs +- Root of the ~700 GB CSV tree (read-only). + +### 3.2 Activities + +**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy). + +**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions: +- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`) +- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen) +- atmosphere tokens (`ambient`, `dryNitrogen`, …) + +Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies). + +**C. Structural profiling (header-only + sampled body).** Per file, without full load: +- read header row → column names, count, order; +- infer per-column datatype from first/last N rows + a random sample; +- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure. + +**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming. + +### 3.3 Algorithm sketch (streaming, O(files), bounded memory) + +``` +for each file in walk(root): + meta = fs_metadata(file) + tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere + header = read_first_line(file) + columns = parse_columns(header) + dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M)) + fp = fingerprint(columns) # ordered-set hash + emit_manifest_row(meta, tokens, columns, dtypes, fp) +``` + +### 3.4 Artifacts +- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint. +- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type. +- **`unmatched_report`** — files/filenames not matching known folder or naming rules. +- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed). + +### 3.5 Exit criteria +- 100% of files inventoried and fingerprinted. +- Every file assigned a (measurement type, attachment level) or flagged as unmatched. +- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated. + +--- + +## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative) + +**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies. + +The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity. + +### 4.1 The iteration loop (per measurement type) + +1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`). +2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering. +3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution). +4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes. +5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository. +6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns. + +### 4.2 Instruments Repository (ontology sub-model) + +Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries: +- identity (type, model/label, measurement type it produces); +- measured quantities, units, and **valid physical ranges** (min/max, resolution); +- association to `measurement_types` / `column_definitions` in the data dictionary. + +Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing. + +### 4.3 Anomaly detection + +Criteria are **derived from the data**, not fixed a priori. Starting catalog: + +| Class | Description | Source of criterion | +|-------|-------------|---------------------| +| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges | +| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check | +| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical | +| Type violation | Non-coercible value in a typed column | Mapping/coercion | +| Duplicate | Repeated rows / repeated identity keys | Key + row hashing | +| Missing required | Null in a non-nullable canonical field | Schema constraints | +| Encoding/format | Malformed rows, delimiter/decimal issues | Parser | + +Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement. + +### 4.4 Data dictionary / metadata catalog + +Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model. + +### 4.5 Artifacts +- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules. +- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7). +- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned. +- **`data_dictionary`** — `measurement_types` + `column_definitions`. +- **`instruments_repository`** — instrument entities + characteristics + ranges. +- **`rejects_store`** — quarantined anomalies with reason codes. +- **`ontology_notes`** — running record of ERD changes and rationale. + +### 4.6 Exit criteria +- Every fingerprint of every measurement type has a complete, unambiguous mapping. +- Canonical dataset produced for all in-scope files; rejects quarantined and categorized. +- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release. + +--- + +## 5. Phase 4 — Conversion & Export + +**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge. + +### 5.1 Targets +1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size. +2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy. +3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use). +4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path. +5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs. + +### 5.2 Design +- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers. +- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps). +- Lineage is emitted alongside every export (see §6). + +### 5.3 Artifacts +- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions). + +### 5.4 Exit criteria +- All five target formats generated from one canonical snapshot. +- Round-trip validation: record counts and key aggregates match across formats. +- Lineage complete and resolvable for sampled records. + +--- + +## 6. Cross-cutting concerns + +### 6.1 Lineage & processing-flow documentation +- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied. +- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**. +- Lineage is a first-class export, present in every target format. + +### 6.2 Configuration & rule registry +- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites. + +### 6.3 Validation gates +- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%. + +--- + +## 7. Target artifacts catalog (consolidated) + +| Artifact | Phase | Form | +|----------|-------|------| +| storage_manifest | 1 | JSONL + SQLite | +| schema_fingerprint_report | 1 | JSON/CSV | +| unmatched_report | 1 | CSV | +| storage_map | 1 | Markdown | +| column_mapping | 2↔3 | CSV/JSON (versioned) | +| canonical_dataset | 2↔3 | partitioned intermediate | +| relational_schema (DDL) | 2↔3 | SQL | +| data_dictionary | 2↔3 | table/CSV | +| instruments_repository | 2↔3 | table/CSV | +| rejects_store | 2↔3 | JSONL | +| ontology_notes | 2↔3 | Markdown | +| exports (JSON/SQLite/PG/RDF) | 4 | per format | +| lineage + build manifest | all | JSONL | + +--- + +## 8. Tooling & environment + +- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB. +- **No DB engine in Phases 1–2** beyond SQLite for the manifest index. +- **PostgreSQL** materialized in Phase 4 (system of record). +- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation). +- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge. +- Python indentation: **tabs**. + +--- + +## 9. Performance strategy for 700 GB + +- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row. +- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series. +- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1. +- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after. +- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset). +- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4. + +--- + +## 10. Open items & risks + +1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type. +2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it. +3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations. +4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned. +5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage. + +--- + +*End of Plan v1.* diff --git a/docs/examples/Tribology_Ontology.html b/docs/examples/Tribology_Ontology.html new file mode 100644 index 0000000..740a54d --- /dev/null +++ b/docs/examples/Tribology_Ontology.html @@ -0,0 +1,188 @@ + + + + + Bundled Page + + + + +
+ + + + τ + + + + + + + + + +
+
Unpacking...
+ + + + + + + + + + \ No newline at end of file diff --git a/docs/rules/bench-methodology.md b/docs/rules/bench-methodology.md new file mode 100644 index 0000000..4cf8338 --- /dev/null +++ b/docs/rules/bench-methodology.md @@ -0,0 +1,112 @@ +# Bench / Methodology - isolation, repetitions, timeouts, honest reporting + +**Purpose + scope.** The measurement discipline for the benchmark matrix +(Q1-Q7 x 5 formats, spec [08](../specs/08_benchmark_queries.md) / +[09](../specs/09_benchmark_execution.md)) and for the extrapolation and +reporting built on it (specs 10-11). Binding for `bench_runner.py`, every +query implementation, `extrapolate.py`, and `report.py`. + +The product of this project IS these measurements; a sloppy protocol +invalidates everything downstream. + +--- + +## 1. Cell isolation - one subprocess per measurement + +- Every benchmark cell (format x query x run) executes in its OWN + subprocess, so peak RSS, CPU time, and IO counters attribute to exactly + that cell. +- The runner samples the child with `psutil` every 50 ms for peak RSS; + CPU time is user+sys at exit; read bytes from `io_counters`. +- Nothing else runs in the child: no logging setup beyond minimum, no + imports the query does not need - the measurement must not pay for the + harness. + +## 2. Repetition protocol + +- Per cell: **1 warm-up run (discarded) + 3 measured runs.** Report the + median in `results_median.csv`; keep min/max in the raw data. +- **Randomize the cell execution order** and record the actual order in + the raw results - fixed order lets cache effects masquerade as format + differences. +- SQLite additionally reports run 1 (cold) and run 3 (warm) separately + (spec 08). +- Never average TIMEOUT/ERROR runs into medians; a cell with a failed run + reports the failure, not a prettier subset. + +## 3. Timeouts and failures are data + +- Hard per-run timeout: **30 minutes.** On expiry the runner records + `TIMEOUT` for that cell and CONTINUES the matrix (an expected outcome: + RDF Q5). Never retry a timeout hoping for a better number. +- A crashed cell records `ERROR` the same way (see + [code-error-handling.md](code-error-handling.md) pattern C). +- In extrapolation and scoring, TIMEOUT/FAIL project to score 0 - they + are never silently dropped from tables or charts (spec 10/11 flags: + projection > 1 h = `IMPRACTICAL`, > 24 h = `FAIL`). + +## 4. Result correctness gates the measurement + +- Every measured run validates its result rows against the canonical + checksum (sorted rows, float tolerance 1e-9) and records `checksum_ok` + (see [test-pipeline-validation.md](test-pipeline-validation.md)). +- A fast run with `checksum_ok = false` is an INVALID measurement - it + must never enter medians, extrapolation, or the report. +- **Q5 reads raw `friction_cycles` in every format** - summary tables and + materialized views are forbidden for it; verify PostgreSQL did not + satisfy it from `track_summary` + (`EXPLAIN (ANALYZE, BUFFERS)` is recorded for PostgreSQL runs anyway). + +## 5. Fairness across formats + +- Identical result-set contract for all 5 implementations of each query; + no format answers a simplified question. +- Each format uses its idiomatic strength honestly: CSV Q1 reads the one + target file by path; JSON Q1 loads the one coupon file; SQL uses its + indexes. What is forbidden is capping or sampling (`LIMIT`, partial + scans) that changes the answer. +- The pandas CSV variant is measured as a SEPARATE variant, never mixed + into the plain-CSV numbers (spec 08). +- Storage sizes compare like with like: for every format record both the + working footprint and the gzip archival footprint in + `storage_sizes.csv` (spec 09). + +## 6. Environment honesty - this machine, stated limits + +- Record the hardware and software environment (CPU model, cores, RAM, + disk type, OS, engine versions) once per benchmark session into the + raw results; the report cites it. +- **Cold-cache protocol on this host:** the spec's page-cache drop + (`/proc/sys/vm/drop_caches`) is Linux-only; this project runs on + Windows Server 2025, where it is unavailable. The cold-cache matrix + pass is therefore SKIPPED and marked as such in results and report - + never approximated by a fake (e.g. file re-copy tricks) without a + rules update describing the method. +- Quiesce the machine during measured runs (no parallel pipeline tasks, + no background loads); the runner runs cells sequentially by design. + +## 7. Extrapolation discipline (task 10) + +- Scaling laws per access pattern are those fixed in spec 10 (O(1), + O(log n + k), O(k log n), O(n), O(n / min(cores, partitions))); + constants calibrate on the measured 150 MB point. +- Every projected number in the report is labeled as projected, with its + scaling assumption; measured and projected values never mix in one + column unlabeled. +- Hardware cost figures come from `hw_prices.yaml` + (see [code-config-yaml.md](code-config-yaml.md)), never inlined. + +## 8. Anti-patterns + +- **Running cells in-process** - RSS/CPU attribution becomes fiction. +- **Fixed execution order** - cache warmth leaks between formats. +- **Retrying or dropping slow runs** - the slow run IS the result. +- **Averaging over a TIMEOUT** or reporting medians of fewer than 3 valid + runs without flagging it. +- **A "fast" result that fails the checksum** entering any downstream + table. +- **Tuning one format's query while leaving others naive** - fairness is + per-scenario, not per-format-champion. +- **Faking cold-cache on Windows** without a documented, rules-approved + method. +- **Presenting projected numbers as measured** in tables or charts. diff --git a/docs/rules/build-pipeline-tasks.md b/docs/rules/build-pipeline-tasks.md new file mode 100644 index 0000000..2ac57c2 --- /dev/null +++ b/docs/rules/build-pipeline-tasks.md @@ -0,0 +1,103 @@ +# Build / Pipeline tasks - the ./out tree, .done markers, task order + +**Purpose + scope.** How the 11 pipeline tasks compose: where artifacts +live, how completion is signaled, what order tasks run in, and what git +does and does not track. Binding for every task implementation and for +anyone running the pipeline. + +--- + +## 1. The artifact tree - everything under `./out/` + +All generated artifacts live under `./out/` relative to the repo root +(spec 00_PLAN): + +``` +out/ + config/ lab_config.yaml, hw_prices.yaml + csv/ the canonical raw corpus + MANIFEST.csv + json/ full/ (per-coupon JSON-LD), hybrid/dataset.jsonld + sqlite/ tribo.db + pg/ tribo.dump (pg_dump -Fc; the live DB is in PostgreSQL) + rdf/ dataset.nt.gz, dataset.ttl, oxigraph_store/ + bench/ queries/, expected/, results_raw.csv, results_median.csv, + storage_sizes.csv, extrapolation.csv, hardware_sizing.csv + report/ REPORT.md, charts/, tables/, process_flow.md, diagrams/ + .done/ completion markers, one per task +``` + +- A task writes ONLY under its own output paths; it treats its inputs as + read-only. +- Tasks communicate exclusively through these artifacts - never through + shared in-process state + (see [code-python-style.md](code-python-style.md)). +- **`./out/` is git-ignored** (except nothing - the whole tree). Sources + of truth are the specs, the rules, and the code; artifacts are + reproducible from them ([data-determinism.md](data-determinism.md)). + Never commit generated artifacts "for convenience". + +## 2. Completion markers + +- Each task NN writes `./out/.done/.ok` after - and only after - its + outputs are complete and validated. The marker body carries the + headline counts (rows, bytes, triples) downstream tasks check + (see [test-pipeline-validation.md](test-pipeline-validation.md)). +- A task verifies its dependencies' markers before starting and aborts + loudly if one is missing. +- Re-running a task: delete its own marker first, regenerate outputs, + rewrite the marker. If its outputs changed, downstream markers are now + stale - delete them too (the dependency graph below says which). + +## 3. Task graph and execution order + +``` +01 -> 02 +01 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11 + 09 ------> 11 +``` + +- Sequential order: 01, 02, 03, then 04-07 (independent, may run in + parallel), 08, 09, 10, 11. +- 02 (process diagrams) needs only 01 and can run any time after it. +- Task 09 (benchmark execution) must run WITHOUT 04-07 style parallel + work in the background - measurements need a quiet machine + ([bench-methodology.md](bench-methodology.md) section 6). + +## 4. `storage_sizes.csv` - the shared append contract + +- Location: `./out/bench/storage_sizes.csv`; columns: + `format, variant, bytes`. +- Tasks 04-07 APPEND their format's rows (e.g. `json,full,...`, + `json,hybrid,...`, `sqlite,db,...`, `pg,live,...`, `pg,dump,...`, + `rdf,nt_gz,...`, `rdf,store,...`); task 09 appends the gzip archival + variants and consolidates. +- Appends are idempotent per (format, variant): a re-run REPLACES its own + prior rows, never duplicates them and never touches other formats' + rows. +- RDF load wall-time and similar per-format load metrics recorded per + spec 07 go into the task's `.done` marker and the report, not into + extra ad-hoc files. + +## 5. Runs are resumable, not restartable-from-zero + +- The marker system exists so a failed pipeline resumes at the failed + task; do not delete the whole `./out/` tree to retry one task. +- Conversely, after a deliberate corpus regeneration (03), ALL downstream + markers and artifacts (04-11) are stale and must be removed - a mixed + tree of old and new artifacts is worse than an empty one + ([meta-no-assumptions.md](meta-no-assumptions.md) applies before any + such deletion). + +## 6. Anti-patterns + +- **Writing an artifact outside `./out/`** or a task writing into another + task's output directory. +- **Committing `./out/` content to git.** +- **A marker written on partial success** - see + [code-error-handling.md](code-error-handling.md). +- **Running benchmarks while converters are still running.** +- **Duplicate rows in `storage_sizes.csv`** after re-runs - replace your + own rows. +- **Hand-editing generated artifacts** (a "quick fix" inside + `MANIFEST.csv` or a result CSV) - regenerate them through the owning + task instead. diff --git a/docs/rules/code-comments-policy.md b/docs/rules/code-comments-policy.md new file mode 100644 index 0000000..90d0dc5 --- /dev/null +++ b/docs/rules/code-comments-policy.md @@ -0,0 +1,61 @@ +# Code comments - write WHY, not WHAT + +**Binding** for all source files: Python, SQL, SPARQL, PowerShell, YAML, shell. + +## Rule + +Default to **no comment**. Add one only when the *why* is non-obvious from +the code. If removing the comment wouldn't confuse a future reader, delete it. + +A comment earns its place when it captures something the code can't: +hidden constraint, ordering invariant, workaround for a named bug, vendor +quirk, cross-file invariant the type system can't enforce. + +## Delete on sight + +```python +# Read the manifest file +manifest = read_manifest(csv_root) # function name says it +``` + +```python +# Increment counter +counter += 1 # pure restatement +``` + +```python +# Added for the benchmark rework (see task 09) +def run_cell(...): # belongs in commit message, rots in code +``` + +```python +# TODO: refactor later + # no owner, no trigger - decoration +``` + +## Keep + +```python +# Q5 must scan raw friction_cycles - track_summary is forbidden for this +# query in every format (spec 08); do not "optimize" this into the summary. +``` +Cross-file invariant a refactor must preserve. + +```python +# Column order matches MANIFEST.csv writer in generate_data.py - keep in sync. +``` +Cross-file invariant the type system can't enforce. + +```sql +-- BRIN, not b-tree: cycle values are physically ordered within each +-- partition, so BRIN gives the same pruning at ~1% of the index size. +``` +Non-obvious engineering decision with a reason. + +## Length + +Inline: one line. Block: 1-4 lines. If you need more than a paragraph, +the right home is a rule or spec doc - link it from a one-line comment. + +Stale comments are worse than missing ones. When you touch a file, fix or +delete comments whose *why* no longer holds. diff --git a/docs/rules/code-config-yaml.md b/docs/rules/code-config-yaml.md new file mode 100644 index 0000000..0a5d98b --- /dev/null +++ b/docs/rules/code-config-yaml.md @@ -0,0 +1,94 @@ +# Code / Config - all tunables live in YAML, no magic numbers, no .env + +**Purpose + scope.** Governs where every configurable value lives. Binding +for all Python code in this repository. Any new configurable parameter MUST +follow this rule when deciding where it goes. + +The goal: one authoritative source per parameter, so regenerating any +artifact with the same config is byte-identical, and changing a parameter +never requires hunting constants across scripts. + +--- + +## 1. The two YAML configs + +| File | Owns | Written by | +|---|---|---| +| `./out/config/lab_config.yaml` | The laboratory model: volumes (batches / wafers / coupons / tracks / cycles), the deposition matrix, physical-model coefficients, nominal test conditions, **the random seed 20260711** | Task 01, from the spec; regenerated only deliberately | +| `./out/config/hw_prices.yaml` | Hardware cost parameters for extrapolation (RAM $/GB, NVMe $/TB, server base prices) | Task 10 seeds editable defaults; the operator may edit | + +Rules: + +- **Scripts READ these files; they never duplicate values from them.** + Task 03 reads volumes, coefficients, and the seed from + `lab_config.yaml` - it does not hardcode 588 coupons or seed 20260711. + Task 10 reads prices from `hw_prices.yaml` - it does not inline dollar + figures. +- A parameter needed by more than one task lives in the YAML, never + copy-pasted between scripts. +- New tunables go into the appropriate YAML with a self-describing key; + if neither file fits, propose a new config file in a docs/rules update + first. + +## 2. Spec-fixed constants - named, sourced, at the top + +Values fixed by the specs that are NOT operator-tunable (the 120-200 MB +size gate, the 30-minute cell timeout, the 50k-row transaction batch, the +50 ms RSS sampling interval) are module-level named constants at the top of +the owning script, each with a one-line comment naming the spec: + +```python +SIZE_GATE_MB = (120, 200) # abort bounds, spec 03 +CELL_TIMEOUT_S = 30 * 60 # hard per-run timeout, spec 09 +``` + +A bare number inline in the logic (`if size > 209715200`) is forbidden. + +## 3. The PostgreSQL DSN - one env var, never hardcoded, never logged + +The only external resource is the local PostgreSQL 16 instance. Its DSN: + +- Comes from the environment variable **`TRIBO_PG_DSN`**; default when + unset: `postgresql://postgres@localhost:5432/tribo`. +- Is read in ONE shared helper (e.g. `common/pg.py: get_dsn()`); scripts + call the helper, never `os.environ` directly. +- Is never hardcoded in a script, never committed in a config file, and + never logged (a DSN may embed a password - see + [obs-python-logging.md](obs-python-logging.md)). + +There are no other secrets in this project. There is no `.env` file; do +not introduce one. + +## 4. CLI flags - per-run choices only + +Command-line flags are for choices that vary per invocation, not for +model parameters: `--log-level`, `--dry-run`, an output-root override, a +`--limit` for smoke runs. A physical coefficient or a volume as a CLI flag +is forbidden - it would silently fork the config out of `lab_config.yaml`. + +## 5. Anti-patterns + +- **Hardcoding a value that exists in `lab_config.yaml`** (seed, volumes, + coefficients) - the YAML is the single source; drift here breaks + reproducibility silently. +- **Adding a `.env` file or reading `os.environ` scattered through + scripts** - the DSN goes through the one shared helper; nothing else + lives in the environment. +- **Inline magic numbers** for spec-fixed gates and timeouts - name them + at the top with the spec reference. +- **Dollar prices inlined in `extrapolate.py`** - they live in + `hw_prices.yaml` so the operator can re-cost without touching code. +- **A model parameter as a CLI flag** - per-run flags are for run + mechanics, not for the lab model. +- **Committing `./out/config/` generated files to git as if they were + sources** - the YAML under `./out/` is a generated artifact; its source + of truth is spec 01 (see + [build-pipeline-tasks.md](build-pipeline-tasks.md)). + +## 6. Related documents + +- [data-determinism.md](data-determinism.md) - the seed contract. +- [build-pipeline-tasks.md](build-pipeline-tasks.md) - the `./out/` tree + and what is git-ignored. +- [obs-python-logging.md](obs-python-logging.md) - `--log-level` flag; DSN + never logged. diff --git a/docs/rules/code-data-formatting.md b/docs/rules/code-data-formatting.md new file mode 100644 index 0000000..73813b1 --- /dev/null +++ b/docs/rules/code-data-formatting.md @@ -0,0 +1,132 @@ +# Code / Data / Formatting - ASCII-only typographic characters in source files + +**Binding scope.** Strings that end up in any of the following MUST use ASCII +typographic characters; never their Unicode "smart" equivalents: + +- Python source, SQL and SPARQL query files +- Generated data and artifacts written to disk (CSV, JSON-LD, N-Triples, + Turtle, MANIFEST and benchmark result CSVs, log messages) +- Configuration files (`.yaml`, `.json`, `.toml`) +- PowerShell / shell scripts (any file processed by Windows tooling) +- Markdown documentation (`docs/`, `README.md`, `CLAUDE.md`) - the specs + were normalized to ASCII typography on 2026-07-11 and stay that way + +**Long dashes are an absolute exception, no carve-outs.** Em-dash (U+2014) +and en-dash (U+2013) MUST be replaced with ASCII hyphen-minus (`-`) in +**every** file in this repository, including markdown documentation, code +comments, and report prose. Reason: long dashes are visually +indistinguishable from a hyphen in many monospace fonts, break diffs / grep / +IDE search, and round-trip badly through Windows tooling chains (section 2). + +--- + +## 1. The substitution table + +| Unicode char | Codepoint | Name | ASCII replacement | +|---|---|---|---| +| em-dash | U+2014 | em-dash | `-` (single hyphen-minus) | +| en-dash | U+2013 | en-dash | `-` (single hyphen-minus) | +| horizontal bar | U+2015 | horizontal bar | `-` | +| minus sign | U+2212 | minus sign | `-` | +| ellipsis | U+2026 | horizontal ellipsis | `...` (three ASCII dots) | +| left double quote | U+201C | left double quotation mark | `"` | +| right double quote | U+201D | right double quotation mark | `"` | +| left single quote | U+2018 | left single quotation mark | `'` | +| right single quote | U+2019 | right single quotation mark | `'` | +| left guillemet | U+00AB | left guillemet | `"` | +| right guillemet | U+00BB | right guillemet | `"` | +| nbsp | U+00A0 | non-breaking space | regular space | + +**Single hyphen, not double.** When replacing an em-dash that separates two +clauses, use a single `-` surrounded by spaces (`text - clause`), NOT `--`. + +**Ranges use a single hyphen:** `Q1-Q7`, `120-200 MB`, `0.3-1.1 um` - never +an en-dash. + +## 2. Why - documented failure modes + +1. **PowerShell 5.1 reads no-BOM files as Windows-1252.** Non-ASCII UTF-8 + bytes become mojibake and break string-literal parsing. This project is + developed on Windows; any script or config touched by PowerShell is + exposed. + +2. **Windows -> git-bash -> tar / packaging round-trip.** At any step an + unset locale or wrong encoding assumption produces mojibake. A UTF-8 byte + sequence for U+2014 interpreted under a non-UTF-8 locale renders as a + `?` replacement character in the packaged output. + +3. **Copy-paste from Word / Google Docs / Notion / web articles.** These + tools auto-substitute typography (`"` -> smart quotes, `'` -> smart + apostrophe, `--` -> em-dash, `...` -> ellipsis) silently. The + substitution survives copy into any editor and lands in source files + unnoticed. The original specs arrived with en/em dashes this way. + +4. **Data files must diff and grep cleanly.** The CSV corpus, MANIFEST, and + benchmark results are compared by checksum and processed by streaming + parsers; a stray typographic character breaks byte-level reproducibility. + +## 3. Where the rule does NOT apply + +- **Scientific and unit notation is content, not typography.** Characters + such as `µ` (micro), `±`, `°C`, `×`, `Ø`, and Greek letters (`τ`, `σ`) in + formulas inside the specs are meaningful scientific notation and are kept. + Note the boundary: in **column names and code identifiers** the units are + always plain ASCII (`_um`, `_nm`, `_GPa` - see + [data-naming-units.md](data-naming-units.md)); the scientific characters + live only in explanatory prose. +- **International language strings** (Cyrillic, CJK, etc.) are meaningful + content, not typographic substitutions. UTF-8 throughout is the right + answer. (Repository artifacts are English; this matters only for quoted + material.) +- **Imported reference materials** in `docs/examples/` are kept verbatim as + received; they are not binding documents. + +## 4. Detection / enforcement + +There is no automated check for this rule yet - **not in CI yet**. The rule is +enforced by review and by the editing discipline in this file. The intent of a +future detector is documented here so it can be built consistently: + +- **What it flags.** Any occurrence of the Unicode codepoints in the section 1 + substitution table inside in-scope files. The section 3 exemptions + (scientific notation, international content, `docs/examples/`) are the only + carve-outs. +- **Exit behavior.** Non-zero exit when any flagged codepoint is found under + the scan roots, so it can later gate a commit hook or a build step. +- **Tooling.** When this detector is created it MUST be a PowerShell script + (this is a Windows project) and live under `tools/`. + +## 5. Examples + +### Good + +```python +# safe through every Windows tooling step +print("Corpus size 152.3 MB - within the 120-200 MB gate.") +label = "steady-state COF, cycles 500-1000" +``` + +### Bad + +```python +# em-dash literal (U+2014) - mojibake risk on PowerShell read/write +print("Corpus size 152.3 MB [U+2014] within the gate.") + +# en-dash range (U+2013) - breaks grep for "Q1-Q7" +label = "queries Q1[U+2013]Q7" +``` + +## 6. Anti-patterns + +- **Copy-pasting strings from a word processor or web page** into source / + specs / config without normalization. +- **Trusting tar / packaging to "preserve as-is".** They preserve bytes; the + *encoding interpretation* is what gets lost when locale or encoding + declaration is wrong on either end. +- **Assuming the source file encoding alone is enough.** The source file may + be fine - the failure is at downstream consumers (PowerShell read, build + write, OS locale). +- **Using `--` instead of `-` "for emphasis".** Single ASCII hyphen + surrounded by spaces is the project's chosen separator. +- **Putting `µ` or `°` into a column name or identifier** - units in names + are ASCII (`_um`, `_C`); see [data-naming-units.md](data-naming-units.md). diff --git a/docs/rules/code-error-handling.md b/docs/rules/code-error-handling.md new file mode 100644 index 0000000..93c031f --- /dev/null +++ b/docs/rules/code-error-handling.md @@ -0,0 +1,181 @@ +# Code / Error handling - never swallow, always surface, fail loud + +**Purpose + scope.** Governs every `try:` / `except:` / `raise` site in +Python code in this repository (data generation, format converters, benchmark +runner, extrapolation, reporting, helper tools). Binding for all contributors. + +The pipeline's core failure principle: **a wrong artifact is worse than no +artifact.** Every task's output feeds downstream tasks; silently degraded +data poisons every result built on it. When in doubt, abort with a clear +message and a non-zero exit code. + +--- + +## 1. The rule + +1. **Never `except: pass`.** A bare swallow is always a bug. Forbidden + absolutely. +2. **Never `except Exception: logger.warning("...") + continue`** for + anything that was not an EXPECTED, documented, transient condition. If + you swallow and continue, you must be able to state why the program can + correctly proceed. +3. **Always `logger.error(..., exc_info=True)` in `except` blocks.** + `exc_info=True` captures the traceback into the log record (see + [obs-python-logging.md](obs-python-logging.md)). Without it the original + exception is lost. +4. **Re-raise unless you have a defined recovery path.** If the caller + cannot continue meaningfully without the failed operation, re-raise + after logging. +5. **Surface to the right channel:** + - **CLI script (every pipeline task):** print the error to stderr and + exit non-zero. A task that failed MUST NOT write its + `./out/.done/.ok` marker (see + [build-pipeline-tasks.md](build-pipeline-tasks.md)). + - **Validation gate (row counts, checksums, size bounds):** a failed + gate is an abort, not a warning. Example: task 03 aborts when the + generated tree is outside 120-200 MB; converters abort when row counts + do not match `MANIFEST.csv`. + - **Library / helper function:** re-raise; let the caller decide. +6. **Defensive "this cannot happen" branches must surface, not silently + `return`.** If you reach a defensive branch, log at ERROR with the + branch name and the inputs that got you there, then raise. + +## 2. The exception ladder - the only three sanctioned swallow patterns + +Code may swallow ONLY in these three documented patterns. Anything else +re-raises after logging. + +### Pattern A: documented, recoverable transient (retry) + +```python +for attempt in range(MAX_RETRIES): + try: + return do_thing() + except (psycopg.OperationalError, TimeoutError) as exc: + logger.warning("attempt %d/%d failed: %s", attempt + 1, MAX_RETRIES, exc) + time.sleep(backoff(attempt)) +# All retries exhausted - surface. +logger.error("do_thing failed after %d retries", MAX_RETRIES) +raise +``` + +Justified: the per-attempt warning is acceptable; the final failure +re-raises. + +### Pattern B: optional best-effort cleanup + +```python +def _close_conn(conn) -> None: + try: + conn.close() + except Exception: + # Cleanup is best-effort; the operation already finished, log and continue. + logger.warning("closing connection failed", exc_info=True) +``` + +Justified: the comment names why continuing is correct; the traceback is +still captured. + +### Pattern C: benchmark-cell isolation - record and continue the matrix + +```python +try: + result = run_cell(fmt, query, timeout_s=30 * 60) +except CellTimeout: + logger.error("cell %s/%s timed out", fmt, query) + write_result_row(fmt, query, status="TIMEOUT") # recorded, not hidden +except Exception: + logger.error("cell %s/%s failed", fmt, query, exc_info=True) + write_result_row(fmt, query, status="ERROR") +``` + +Justified: spec 09 requires the benchmark matrix to survive a failing cell +(RDF Q5 is expected to time out). The failure is RECORDED as data +(`TIMEOUT` / `ERROR` in `results_raw.csv`), never silently dropped, and the +run continues with the next cell. This pattern applies ONLY to the benchmark +runner's per-cell boundary - a failure in the runner itself (setup, result +writing) still aborts. + +## 3. Loading into SQLite / PostgreSQL - all-or-nothing per table + +A converter that half-loads a table leaves a database that LOOKS complete +and silently skews every benchmark run against it. + +1. Wrap each table's bulk load in a transaction; on any error roll back + and abort the converter with a non-zero exit. +2. After loading, validate row counts against `MANIFEST.csv` and fail loud + on any mismatch (see + [test-pipeline-validation.md](test-pipeline-validation.md)). +3. Never write the task's `.done` marker after a partial load. + +## 4. Anti-patterns + +- **`except: pass`** - forbidden absolutely. +- **`except Exception: logger.warning("could not X")` + continue** - + forbidden unless the function documents that "could not X" is a + recoverable expected outcome. If it is, phrase it "X skipped (expected + condition Y)" so a reader does not grep "could not" hunting for bugs. +- **Catching `BaseException`** - forbidden (swallows `KeyboardInterrupt`, + `SystemExit`). Use `Exception`. +- **`logger.error("X")` without `exc_info=True` in an `except` block** - + forbidden. The traceback IS the diagnostic. +- **`logger.error(str(exc))`** - forbidden; redundant with `exc_info=True` + and loses type and stack. +- **Returning `None` to signal failure** from a function whose contract is + "return T" - forbidden. Either raise, or return `T | None` with + documented semantics the caller checks. +- **`raise Exception("...")`** - forbidden. Use a specific class: + `ValueError` / `RuntimeError` / `KeyError` for trivial cases, a custom + domain exception subclass (e.g. `ManifestMismatchError`) for domain + conditions. +- **`raise` chained without context** - inside an `except`, prefer + `raise NewError(...) from exc` so the original cause stays visible in the + chained traceback. A bare `raise` that re-raises the same exception is + fine; `raise SomethingElse(...)` without `from exc` is forbidden. +- **Writing a `.done` marker in a `finally` block** - the marker means + "completed successfully", never "attempted". +- **Clamping / repairing out-of-range generated values silently** - a + physical-model bug must surface, not be masked by a `max(0, ...)` added + "to be safe". Deliberate clamps defined by the spec (e.g. COF floor 0.12) + are model logic, not error handling. + +## 5. Defensive branches + +```python +# Good: +def open_store(fmt: str) -> Store: + if fmt == "sqlite": + return SqliteStore() + if fmt == "postgresql": + return PgStore() + # Defensive: a new format added without updating this factory lands here. + logger.error("open_store: unknown format=%r", fmt) + raise ValueError(f"unknown storage format: {fmt!r}") +``` + +The silent-`return None` form lets the unknown value propagate and breaks +downstream code with a confusing `AttributeError` on the wrong line. + +## 6. Logging vs surfacing - the decision + +For a caught exception, ask in order: + +1. **Will the pipeline produce a wrong artifact if I continue?** Re-raise / + abort. +2. **Is this expected and recoverable in the caller's contract?** Log at + WARNING with `exc_info=True`, return a sentinel the caller checks. +3. **Is this an isolated benchmark cell (pattern C)?** Log at ERROR, record + the status as data, continue the matrix. +4. **Is the process going down anyway?** Log at CRITICAL with + `exc_info=True`, then `sys.exit(1)` or let it propagate. + +## 7. Related documents + +- [obs-python-logging.md](obs-python-logging.md) - logging API and severity + levels. +- [test-pipeline-validation.md](test-pipeline-validation.md) - the + validation gates that turn silent corruption into loud failures. +- [build-pipeline-tasks.md](build-pipeline-tasks.md) - `.done` marker + semantics. +- [bench-methodology.md](bench-methodology.md) - the TIMEOUT contract for + benchmark cells. diff --git a/docs/rules/code-python-style.md b/docs/rules/code-python-style.md new file mode 100644 index 0000000..6387eae --- /dev/null +++ b/docs/rules/code-python-style.md @@ -0,0 +1,93 @@ +# Code / Python style - tabs, 3.11+, closed dependency list, streaming + +**Purpose + scope.** Baseline style and dependency policy for every Python +file in this repository (pipeline tasks 01-11, query implementations, +helper tools). Binding for all contributors. + +--- + +## 1. Language and formatting + +- **Python 3.11+.** Use modern stdlib freely (`pathlib`, `dataclasses`, + `tomllib` if needed); no compatibility shims for older versions. +- **Indentation: TABS.** Set by the project owner and repeated in the + specs (00_PLAN, 03, 09). Every Python file in this repository indents + with tab characters, never spaces. Configure the editor per file type; + do not "fix" tab-indented files to spaces. +- `from __future__ import annotations` at the top of every module. +- Type hints on public function signatures; internal helpers at author's + discretion. +- English for all identifiers, docstrings, and comments (see + [meta-language-style.md](meta-language-style.md) - Russian is the + conversation language, not the code language). +- Comments follow [code-comments-policy.md](code-comments-policy.md): + default to none, keep only non-obvious WHY. + +## 2. Dependencies - a closed list + +The stdlib is the default. The ONLY permitted third-party packages: + +| Package | Used by | For | +|---|---|---| +| `numpy` | 03 | vectorized generation, seeded RNG | +| `PyYAML` | 01, 03, 10 | reading/writing the YAML configs | +| `rdflib` | 04, 07 | JSON-LD validation, RDF serialization | +| `oxigraph` | 07, 09 | embedded queryable triplestore | +| `ijson` | 08, 09 | streaming JSON parsing | +| `psycopg` (v3) | 06, 08, 09 | PostgreSQL COPY and queries | +| `psutil` | 09 | RSS / CPU / IO measurement | +| `matplotlib` | 11 | report charts | +| `pandas` | 08 (optional) | ONLY as the separately-measured second CSV query variant; never in the pipeline itself | + +Adding any other package requires updating this table first. In +particular forbidden: ORMs, logging frameworks +([obs-python-logging.md](obs-python-logging.md)), CLI frameworks +(argparse suffices), `requests`/HTTP clients (nothing to call). + +Task 03 (data generation) is stricter by spec: **stdlib + numpy only.** + +## 3. Script shape - every task is a self-contained CLI + +- One entry script per pipeline task, runnable as + `python