b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
39 lines
2.2 KiB
Markdown
39 lines
2.2 KiB
Markdown
# 10 - Extrapolation to 600 GB and Beyond
|
||
|
||
## Context / Goal
|
||
Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.
|
||
|
||
## Dependencies
|
||
09 (`results_median.csv`, `storage_sizes.csv`).
|
||
|
||
## Processing instructions
|
||
Write `extrapolate.py`:
|
||
|
||
### Storage scaling
|
||
Linear in data volume: `size_at(S) = measured_size × S / S_measured`, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).
|
||
|
||
### Query-time scaling laws (apply per format × query)
|
||
| Access pattern | Law | Applies to |
|
||
|---|---|---|
|
||
| Direct path / keyed point read | O(1) / O(log n) | CSV Q1, SQLite/PG Q1, RDF Q1 |
|
||
| Indexed selective filter | O(log n + k) | SQLite/PG Q2, Q4, Q6; RDF Q2, Q4 |
|
||
| Indexed join over dimensions | O(k log n) | Q3, Q6 relational; RDF joins ×(2-5) constant penalty |
|
||
| Full scan / aggregation | O(n) | all CSV/JSON Q2-Q7; Q5 and Q7 everywhere |
|
||
| Single-threaded full scan | O(n), 1 core | SQLite Q5/Q7 |
|
||
| Parallel partitioned scan | O(n / min(cores, partitions)) | PostgreSQL Q5/Q7 |
|
||
|
||
Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".
|
||
|
||
### RAM model
|
||
- CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
|
||
- SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
|
||
- RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).
|
||
|
||
### Hardware sizing table
|
||
For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in `./out/config/hw_prices.yaml` (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.
|
||
|
||
## Output
|
||
- `./out/bench/extrapolation.csv`: format, query, scale_gb, projected_wall_s, flag.
|
||
- `./out/bench/hardware_sizing.csv`: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
|
||
- Marker `./out/.done/10.ok`.
|