fix(docs): fix formatting issues in README.md
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
112
docs/rules/bench-methodology.md
Normal file
112
docs/rules/bench-methodology.md
Normal file
@@ -0,0 +1,112 @@
|
||||
# Bench / Methodology - isolation, repetitions, timeouts, honest reporting
|
||||
|
||||
**Purpose + scope.** The measurement discipline for the benchmark matrix
|
||||
(Q1-Q7 x 5 formats, spec [08](../specs/08_benchmark_queries.md) /
|
||||
[09](../specs/09_benchmark_execution.md)) and for the extrapolation and
|
||||
reporting built on it (specs 10-11). Binding for `bench_runner.py`, every
|
||||
query implementation, `extrapolate.py`, and `report.py`.
|
||||
|
||||
The product of this project IS these measurements; a sloppy protocol
|
||||
invalidates everything downstream.
|
||||
|
||||
---
|
||||
|
||||
## 1. Cell isolation - one subprocess per measurement
|
||||
|
||||
- Every benchmark cell (format x query x run) executes in its OWN
|
||||
subprocess, so peak RSS, CPU time, and IO counters attribute to exactly
|
||||
that cell.
|
||||
- The runner samples the child with `psutil` every 50 ms for peak RSS;
|
||||
CPU time is user+sys at exit; read bytes from `io_counters`.
|
||||
- Nothing else runs in the child: no logging setup beyond minimum, no
|
||||
imports the query does not need - the measurement must not pay for the
|
||||
harness.
|
||||
|
||||
## 2. Repetition protocol
|
||||
|
||||
- Per cell: **1 warm-up run (discarded) + 3 measured runs.** Report the
|
||||
median in `results_median.csv`; keep min/max in the raw data.
|
||||
- **Randomize the cell execution order** and record the actual order in
|
||||
the raw results - fixed order lets cache effects masquerade as format
|
||||
differences.
|
||||
- SQLite additionally reports run 1 (cold) and run 3 (warm) separately
|
||||
(spec 08).
|
||||
- Never average TIMEOUT/ERROR runs into medians; a cell with a failed run
|
||||
reports the failure, not a prettier subset.
|
||||
|
||||
## 3. Timeouts and failures are data
|
||||
|
||||
- Hard per-run timeout: **30 minutes.** On expiry the runner records
|
||||
`TIMEOUT` for that cell and CONTINUES the matrix (an expected outcome:
|
||||
RDF Q5). Never retry a timeout hoping for a better number.
|
||||
- A crashed cell records `ERROR` the same way (see
|
||||
[code-error-handling.md](code-error-handling.md) pattern C).
|
||||
- In extrapolation and scoring, TIMEOUT/FAIL project to score 0 - they
|
||||
are never silently dropped from tables or charts (spec 10/11 flags:
|
||||
projection > 1 h = `IMPRACTICAL`, > 24 h = `FAIL`).
|
||||
|
||||
## 4. Result correctness gates the measurement
|
||||
|
||||
- Every measured run validates its result rows against the canonical
|
||||
checksum (sorted rows, float tolerance 1e-9) and records `checksum_ok`
|
||||
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
|
||||
- A fast run with `checksum_ok = false` is an INVALID measurement - it
|
||||
must never enter medians, extrapolation, or the report.
|
||||
- **Q5 reads raw `friction_cycles` in every format** - summary tables and
|
||||
materialized views are forbidden for it; verify PostgreSQL did not
|
||||
satisfy it from `track_summary`
|
||||
(`EXPLAIN (ANALYZE, BUFFERS)` is recorded for PostgreSQL runs anyway).
|
||||
|
||||
## 5. Fairness across formats
|
||||
|
||||
- Identical result-set contract for all 5 implementations of each query;
|
||||
no format answers a simplified question.
|
||||
- Each format uses its idiomatic strength honestly: CSV Q1 reads the one
|
||||
target file by path; JSON Q1 loads the one coupon file; SQL uses its
|
||||
indexes. What is forbidden is capping or sampling (`LIMIT`, partial
|
||||
scans) that changes the answer.
|
||||
- The pandas CSV variant is measured as a SEPARATE variant, never mixed
|
||||
into the plain-CSV numbers (spec 08).
|
||||
- Storage sizes compare like with like: for every format record both the
|
||||
working footprint and the gzip archival footprint in
|
||||
`storage_sizes.csv` (spec 09).
|
||||
|
||||
## 6. Environment honesty - this machine, stated limits
|
||||
|
||||
- Record the hardware and software environment (CPU model, cores, RAM,
|
||||
disk type, OS, engine versions) once per benchmark session into the
|
||||
raw results; the report cites it.
|
||||
- **Cold-cache protocol on this host:** the spec's page-cache drop
|
||||
(`/proc/sys/vm/drop_caches`) is Linux-only; this project runs on
|
||||
Windows Server 2025, where it is unavailable. The cold-cache matrix
|
||||
pass is therefore SKIPPED and marked as such in results and report -
|
||||
never approximated by a fake (e.g. file re-copy tricks) without a
|
||||
rules update describing the method.
|
||||
- Quiesce the machine during measured runs (no parallel pipeline tasks,
|
||||
no background loads); the runner runs cells sequentially by design.
|
||||
|
||||
## 7. Extrapolation discipline (task 10)
|
||||
|
||||
- Scaling laws per access pattern are those fixed in spec 10 (O(1),
|
||||
O(log n + k), O(k log n), O(n), O(n / min(cores, partitions)));
|
||||
constants calibrate on the measured 150 MB point.
|
||||
- Every projected number in the report is labeled as projected, with its
|
||||
scaling assumption; measured and projected values never mix in one
|
||||
column unlabeled.
|
||||
- Hardware cost figures come from `hw_prices.yaml`
|
||||
(see [code-config-yaml.md](code-config-yaml.md)), never inlined.
|
||||
|
||||
## 8. Anti-patterns
|
||||
|
||||
- **Running cells in-process** - RSS/CPU attribution becomes fiction.
|
||||
- **Fixed execution order** - cache warmth leaks between formats.
|
||||
- **Retrying or dropping slow runs** - the slow run IS the result.
|
||||
- **Averaging over a TIMEOUT** or reporting medians of fewer than 3 valid
|
||||
runs without flagging it.
|
||||
- **A "fast" result that fails the checksum** entering any downstream
|
||||
table.
|
||||
- **Tuning one format's query while leaving others naive** - fairness is
|
||||
per-scenario, not per-format-champion.
|
||||
- **Faking cold-cache on Windows** without a documented, rules-approved
|
||||
method.
|
||||
- **Presenting projected numbers as measured** in tables or charts.
|
||||
Reference in New Issue
Block a user