Files

administrator b173ac82a9 fix(docs): fix formatting issues in README.md

- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker

2026-07-11 13:39:13 -04:00

2.2 KiB

Raw Blame History

10 - Extrapolation to 600 GB and Beyond

Context / Goal

Project the 150 MB measurements to production scale: 600 GB (×4000), 1.2 TB (×8000), 6 TB (×40000) of raw CSV. Model both storage growth and query-performance degradation to understand system requirements as laboratory data grows.

Dependencies

09 (results_median.csv, storage_sizes.csv).

Processing instructions

Write extrapolate.py:

Storage scaling

Linear in data volume: size_at(S) = measured_size × S / S_measured, per format (CSV 1×; report the measured coefficients for JSON, SQLite, PG incl. index share, RDF store).

Query-time scaling laws (apply per format × query)

Access pattern	Law	Applies to
Direct path / keyed point read	O(1) / O(log n)	CSV Q1, SQLite/PG Q1, RDF Q1
Indexed selective filter	O(log n + k)	SQLite/PG Q2, Q4, Q6; RDF Q2, Q4
Indexed join over dimensions	O(k log n)	Q3, Q6 relational; RDF joins ×(2-5) constant penalty
Full scan / aggregation	O(n)	all CSV/JSON Q2-Q7; Q5 and Q7 everywhere
Single-threaded full scan	O(n), 1 core	SQLite Q5/Q7
Parallel partitioned scan	O(n / min(cores, partitions))	PostgreSQL Q5/Q7

Calibrate the constant from the measured 150 MB point; project wall time at each scale. Flag any projection > 1 h as "IMPRACTICAL", > 24 h as "FAIL".

RAM model

CSV/JSON: streaming - flat RAM; note that pandas variant would be O(n) and mark it infeasible > ~10 GB.
SQLite/PG: recommended cache = hot-set model: 10% of index size + working set; report per scale.
RDF: dictionary + index working set ≈ 15-25% of store size for acceptable latency; report where it exceeds 1 TB RAM (single-node infeasibility threshold).

Hardware sizing table

For each format × scale: required disk (with 30% headroom), recommended RAM, recommended cores; approximate cost using parameters in ./out/config/hw_prices.yaml (defaults, editable): DDR5 RDIMM $14/GB, enterprise NVMe $250/TB, base 16-core 1U $6k, 32-core $10k, per extra node $6k.

Output

./out/bench/extrapolation.csv: format, query, scale_gb, projected_wall_s, flag.
./out/bench/hardware_sizing.csv: format, scale_gb, disk_tb, ram_gb, cores, nodes, est_cost_usd.
Marker ./out/.done/10.ok.

2.2 KiB Raw Blame History Unescape Escape