b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
23 lines
1.5 KiB
Markdown
23 lines
1.5 KiB
Markdown
# 09 - Benchmark Execution & Resource Measurement
|
||
|
||
## Context / Goal
|
||
Execute all 35 (Q1-Q7 × 5) benchmark cells with rigorous resource metering; produce the raw measurement table that feeds extrapolation and reporting.
|
||
|
||
## Dependencies
|
||
08.
|
||
|
||
## Processing instructions
|
||
Write `bench_runner.py` (tabs):
|
||
- Each cell runs as a **separate subprocess** so RSS/CPU are attributable. For PostgreSQL, also sample the backend PID via `pg_stat_activity` + psutil.
|
||
- Metrics per run: wall time (s), peak RSS (MB, psutil `memory_info().rss` sampled at 50 ms), CPU time user+sys (s), CPU utilization % (cpu_time / wall / cores), read bytes (psutil `io_counters`), result row count, checksum-match flag.
|
||
- Protocol: 1 warm-up run (discarded) + 3 measured runs per cell; report median and min/max. Randomize cell order to spread cache effects; record run order.
|
||
- Cold-cache variant: repeat the full matrix once after dropping caches if permitted (`sync; echo 3 > /proc/sys/vm/drop_caches` requires root - if not available, skip and note it).
|
||
- Failures/timeouts: hard timeout 30 min per run; record `TIMEOUT` and continue (expected possibility: RDF Q5).
|
||
- Storage footprint: consolidate `storage_sizes.csv` (all formats, incl. gzip-compressed archival sizes of each: `tar.gz` of CSV tree, gzip JSON, sqlite db gz, pg dump, nt.gz).
|
||
|
||
## Output
|
||
- `./out/bench/results_raw.csv`: format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb, rows, checksum_ok.
|
||
- `./out/bench/results_median.csv` - aggregated.
|
||
- `./out/bench/storage_sizes.csv` - final consolidated.
|
||
- Marker `./out/.done/09.ok`.
|