fix(docs): fix formatting issues in README.md

- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
2026-07-11 13:39:13 -04:00
parent d8353953b0
commit b173ac82a9
55 changed files with 3755 additions and 9 deletions

View File

@@ -0,0 +1,38 @@
# 10 - Extrapolation to 600 GB and Beyond
## Context / Goal
Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.
## Dependencies
09 (`results_median.csv`, `storage_sizes.csv`).
## Processing instructions
Write `extrapolate.py`:
### Storage scaling
Linear in data volume: `size_at(S) = measured_size × S / S_measured`, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).
### Query-time scaling laws (apply per format × query)
| Access pattern | Law | Applies to |
|---|---|---|
| Direct path / keyed point read | O(1) / O(log n) | CSV Q1, SQLite/PG Q1, RDF Q1 |
| Indexed selective filter | O(log n + k) | SQLite/PG Q2, Q4, Q6; RDF Q2, Q4 |
| Indexed join over dimensions | O(k log n) | Q3, Q6 relational; RDF joins ×(2-5) constant penalty |
| Full scan / aggregation | O(n) | all CSV/JSON Q2-Q7; Q5 and Q7 everywhere |
| Single-threaded full scan | O(n), 1 core | SQLite Q5/Q7 |
| Parallel partitioned scan | O(n / min(cores, partitions)) | PostgreSQL Q5/Q7 |
Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".
### RAM model
- CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
- SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
- RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).
### Hardware sizing table
For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in `./out/config/hw_prices.yaml` (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.
## Output
- `./out/bench/extrapolation.csv`: format, query, scale_gb, projected_wall_s, flag.
- `./out/bench/hardware_sizing.csv`: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
- Marker `./out/.done/10.ok`.