LabDataStorageEvaluation/docs/rules/bench-methodology.md

Bench / Methodology - isolation, repetitions, timeouts, honest reporting

Purpose + scope. The measurement discipline for the benchmark matrix (Q1-Q7 x 5 formats, spec 08 / 09) and for the extrapolation and reporting built on it (specs 10-11). Binding for bench_runner.py, every query implementation, extrapolate.py, and report.py.

The product of this project IS these measurements; a sloppy protocol invalidates everything downstream.

---

1. Cell isolation - one subprocess per measurement

• Every benchmark cell (format x query x run) executes in its OWN subprocess, so peak RSS, CPU time, and IO counters attribute to exactly that cell.
• The runner samples the child with psutil every 50 ms for peak RSS; CPU time is user+sys at exit; read bytes from io_counters.
• Nothing else runs in the child: no logging setup beyond minimum, no imports the query does not need - the measurement must not pay for the harness.

2. Repetition protocol

• Per cell: 1 warm-up run (discarded) + 3 measured runs. Report the median in results_median.csv; keep min/max in the raw data.
• Randomize the cell execution order and record the actual order in the raw results - fixed order lets cache effects masquerade as format differences.
• SQLite additionally reports run 1 (cold) and run 3 (warm) separately (spec 08).
• Never average TIMEOUT/ERROR runs into medians; a cell with a failed run reports the failure, not a prettier subset.

3. Timeouts and failures are data

• Hard per-run timeout: 30 minutes. On expiry the runner records TIMEOUT for that cell and CONTINUES the matrix (an expected outcome: RDF Q5). Never retry a timeout hoping for a better number.
• A crashed cell records ERROR the same way (see code-error-handling.md pattern C).
• In extrapolation and scoring, TIMEOUT/FAIL project to score 0 - they are never silently dropped from tables or charts (spec 10/11 flags: projection > 1 h = IMPRACTICAL, > 24 h = FAIL).

4. Result correctness gates the measurement

• Every measured run validates its result rows against the canonical checksum (sorted rows, float tolerance 1e-9) and records checksum_ok (see test-pipeline-validation.md).
• A fast run with checksum_ok = false is an INVALID measurement - it must never enter medians, extrapolation, or the report.
• Q5 reads raw friction_cycles in every format - summary tables and materialized views are forbidden for it; verify PostgreSQL did not satisfy it from track_summary (EXPLAIN (ANALYZE, BUFFERS) is recorded for PostgreSQL runs anyway).

5. Fairness across formats

• Identical result-set contract for all 5 implementations of each query; no format answers a simplified question.
• Each format uses its idiomatic strength honestly: CSV Q1 reads the one target file by path; JSON Q1 loads the one coupon file; SQL uses its indexes. What is forbidden is capping or sampling (LIMIT, partial scans) that changes the answer.
• The pandas CSV variant is measured as a SEPARATE variant, never mixed into the plain-CSV numbers (spec 08).
• Storage sizes compare like with like: for every format record both the working footprint and the gzip archival footprint in storage_sizes.csv (spec 09).

6. Environment honesty - this machine, stated limits

• Record the hardware and software environment (CPU model, cores, RAM, disk type, OS, engine versions) once per benchmark session into the raw results; the report cites it.
• Cold-cache protocol on this host: the spec's page-cache drop (/proc/sys/vm/drop_caches) is Linux-only; this project runs on Windows Server 2025, where it is unavailable. The cold-cache matrix pass is therefore SKIPPED and marked as such in results and report - never approximated by a fake (e.g. file re-copy tricks) without a rules update describing the method.
• Quiesce the machine during measured runs (no parallel pipeline tasks, no background loads); the runner runs cells sequentially by design.

7. Extrapolation discipline (task 10)

• Scaling laws per access pattern are those fixed in spec 10 (O(1), O(log n + k), O(k log n), O(n), O(n / min(cores, partitions))); constants calibrate on the measured 150 MB point.
• Every projected number in the report is labeled as projected, with its scaling assumption; measured and projected values never mix in one column unlabeled.
• Hardware cost figures come from hw_prices.yaml (see code-config-yaml.md), never inlined.

8. Anti-patterns

• Running cells in-process - RSS/CPU attribution becomes fiction.
• Fixed execution order - cache warmth leaks between formats.
• Retrying or dropping slow runs - the slow run IS the result.
• Averaging over a TIMEOUT or reporting medians of fewer than 3 valid runs without flagging it.
• A "fast" result that fails the checksum entering any downstream table.
• Tuning one format's query while leaving others naive - fairness is per-scenario, not per-format-champion.
• Faking cold-cache on Windows without a documented, rules-approved method.
• Presenting projected numbers as measured in tables or charts.