fix(docs): fix formatting issues in README.md
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
22
docs/specs/09_benchmark_execution.md
Normal file
22
docs/specs/09_benchmark_execution.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# 09 - Benchmark Execution & Resource Measurement
|
||||
|
||||
## Context / Goal
|
||||
Execute all 35 (Q1-Q7 × 5) benchmark cells with rigorous resource metering; produce the raw measurement table that feeds extrapolation and reporting.
|
||||
|
||||
## Dependencies
|
||||
08.
|
||||
|
||||
## Processing instructions
|
||||
Write `bench_runner.py` (tabs):
|
||||
- Each cell runs as a **separate subprocess** so RSS/CPU are attributable. For PostgreSQL, also sample the backend PID via `pg_stat_activity` + psutil.
|
||||
- Metrics per run: wall time (s), peak RSS (MB, psutil `memory_info().rss` sampled at 50 ms), CPU time user+sys (s), CPU utilization % (cpu_time / wall / cores), read bytes (psutil `io_counters`), result row count, checksum-match flag.
|
||||
- Protocol: 1 warm-up run (discarded) + 3 measured runs per cell; report median and min/max. Randomize cell order to spread cache effects; record run order.
|
||||
- Cold-cache variant: repeat the full matrix once after dropping caches if permitted (`sync; echo 3 > /proc/sys/vm/drop_caches` requires root - if not available, skip and note it).
|
||||
- Failures/timeouts: hard timeout 30 min per run; record `TIMEOUT` and continue (expected possibility: RDF Q5).
|
||||
- Storage footprint: consolidate `storage_sizes.csv` (all formats, incl. gzip-compressed archival sizes of each: `tar.gz` of CSV tree, gzip JSON, sqlite db gz, pg dump, nt.gz).
|
||||
|
||||
## Output
|
||||
- `./out/bench/results_raw.csv`: format, query, run, wall_s, peak_rss_mb, cpu_s, cpu_util_pct, read_mb, rows, checksum_ok.
|
||||
- `./out/bench/results_median.csv` - aggregated.
|
||||
- `./out/bench/storage_sizes.csv` - final consolidated.
|
||||
- Marker `./out/.done/09.ok`.
|
||||
Reference in New Issue
Block a user