LabDataStorageEvaluation/docs/rules/bench-methodology.md

# Bench / Methodology - isolation, repetitions, timeouts, honest reporting

**Purpose + scope.** The measurement discipline for the benchmark matrix
(Q1-Q7 x 5 formats, spec [08](../specs/08_benchmark_queries.md) /
[09](../specs/09_benchmark_execution.md)) and for the extrapolation and
reporting built on it (specs 10-11). Binding for `bench_runner.py`, every
query implementation, `extrapolate.py`, and `report.py`.

The product of this project IS these measurements; a sloppy protocol
invalidates everything downstream.

---

## 1. Cell isolation - one subprocess per measurement

- Every benchmark cell (format x query x run) executes in its OWN
  subprocess, so peak RSS, CPU time, and IO counters attribute to exactly
  that cell.
- The runner samples the child with `psutil` every 50 ms for peak RSS;
  CPU time is user+sys at exit; read bytes from `io_counters`.
- Nothing else runs in the child: no logging setup beyond minimum, no
  imports the query does not need - the measurement must not pay for the
  harness.

## 2. Repetition protocol

- Per cell: **1 warm-up run (discarded) + 3 measured runs.** Report the
  median in `results_median.csv`; keep min/max in the raw data.
- **Randomize the cell execution order** and record the actual order in
  the raw results - fixed order lets cache effects masquerade as format
  differences.
- SQLite additionally reports run 1 (cold) and run 3 (warm) separately
  (spec 08).
- Never average TIMEOUT/ERROR runs into medians; a cell with a failed run
  reports the failure, not a prettier subset.

## 3. Timeouts and failures are data

- Hard per-run timeout: **30 minutes.** On expiry the runner records
  `TIMEOUT` for that cell and CONTINUES the matrix (an expected outcome:
  RDF Q5). Never retry a timeout hoping for a better number.
- A crashed cell records `ERROR` the same way (see
  [code-error-handling.md](code-error-handling.md) pattern C).
- In extrapolation and scoring, TIMEOUT/FAIL project to score 0 - they
  are never silently dropped from tables or charts (spec 10/11 flags:
  projection > 1 h = `IMPRACTICAL`, > 24 h = `FAIL`).

## 4. Result correctness gates the measurement

- Every measured run validates its result rows against the canonical
  checksum (sorted rows, float tolerance 1e-9) and records `checksum_ok`
  (see [test-pipeline-validation.md](test-pipeline-validation.md)).
- A fast run with `checksum_ok = false` is an INVALID measurement - it
  must never enter medians, extrapolation, or the report.
- **Q5 reads raw `friction_cycles` in every format** - summary tables and
  materialized views are forbidden for it; verify PostgreSQL did not
  satisfy it from `track_summary`
  (`EXPLAIN (ANALYZE, BUFFERS)` is recorded for PostgreSQL runs anyway).

## 5. Fairness across formats

- Identical result-set contract for all 5 implementations of each query;
  no format answers a simplified question.
- Each format uses its idiomatic strength honestly: CSV Q1 reads the one
  target file by path; JSON Q1 loads the one coupon file; SQL uses its
  indexes. What is forbidden is capping or sampling (`LIMIT`, partial
  scans) that changes the answer.
- The pandas CSV variant is measured as a SEPARATE variant, never mixed
  into the plain-CSV numbers (spec 08).
- Storage sizes compare like with like: for every format record both the
  working footprint and the gzip archival footprint in
  `storage_sizes.csv` (spec 09).

## 6. Environment honesty - this machine, stated limits

- Record the hardware and software environment (CPU model, cores, RAM,
  disk type, OS, engine versions) once per benchmark session into the
  raw results; the report cites it.
- **Cold-cache protocol on this host:** the spec's page-cache drop
  (`/proc/sys/vm/drop_caches`) is Linux-only; this project runs on
  Windows Server 2025, where it is unavailable. The cold-cache matrix
  pass is therefore SKIPPED and marked as such in results and report -
  never approximated by a fake (e.g. file re-copy tricks) without a
  rules update describing the method.
- Quiesce the machine during measured runs (no parallel pipeline tasks,
  no background loads); the runner runs cells sequentially by design.

## 7. Extrapolation discipline (task 10)

- Scaling laws per access pattern are those fixed in spec 10 (O(1),
  O(log n + k), O(k log n), O(n), O(n / min(cores, partitions)));
  constants calibrate on the measured 150 MB point.
- Every projected number in the report is labeled as projected, with its
  scaling assumption; measured and projected values never mix in one
  column unlabeled.
- Hardware cost figures come from `hw_prices.yaml`
  (see [code-config-yaml.md](code-config-yaml.md)), never inlined.

## 8. Anti-patterns

- **Running cells in-process** - RSS/CPU attribution becomes fiction.
- **Fixed execution order** - cache warmth leaks between formats.
- **Retrying or dropping slow runs** - the slow run IS the result.
- **Averaging over a TIMEOUT** or reporting medians of fewer than 3 valid
  runs without flagging it.
- **A "fast" result that fails the checksum** entering any downstream
  table.
- **Tuning one format's query while leaving others naive** - fairness is
  per-scenario, not per-format-champion.
- **Faking cold-cache on Windows** without a documented, rules-approved
  method.
- **Presenting projected numbers as measured** in tables or charts.