# 10 - Extrapolation to 600 GB and Beyond

## Context / Goal
Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.

## Dependencies
09 (`results_median.csv`, `storage_sizes.csv`).

## Processing instructions
Write `extrapolate.py`:

### Storage scaling
Linear in data volume: `size_at(S) = measured_size × S / S_measured`, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).

### Query-time scaling laws (apply per format × query)
| Access pattern | Law | Applies to |
|---|---|---|
| Direct path / keyed point read | O(1) / O(log n) | CSV Q1, SQLite/PG Q1, RDF Q1 |
| Indexed selective filter | O(log n + k) | SQLite/PG Q2, Q4, Q6; RDF Q2, Q4 |
| Indexed join over dimensions | O(k log n) | Q3, Q6 relational; RDF joins ×(2-5) constant penalty |
| Full scan / aggregation | O(n) | all CSV/JSON Q2-Q7; Q5 and Q7 everywhere |
| Single-threaded full scan | O(n), 1 core | SQLite Q5/Q7 |
| Parallel partitioned scan | O(n / min(cores, partitions)) | PostgreSQL Q5/Q7 |

Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".

### RAM model
- CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
- SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
- RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).

### Hardware sizing table
For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in `./out/config/hw_prices.yaml` (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.

## Output
- `./out/bench/extrapolation.csv`: format, query, scale_gb, projected_wall_s, flag.
- `./out/bench/hardware_sizing.csv`: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
- Marker `./out/.done/10.ok`.