b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
2.7 KiB
03 - Simulated CSV Data Generation (~150 MB)
Context / Goal
Generate the canonical raw dataset as a hierarchical CSV file tree, exactly as instruments would write it. This corpus is the single source for all conversions and benchmarks. Target size: 140-170 MB.
Dependencies
01 (lab_config.yaml).
Input
./out/config/lab_config.yaml - volumes, deposition matrix, physical model coefficients, seed 20260711.
Processing instructions
Write generate_data.py (tabs for indentation, pure stdlib + numpy). Stream rows; never hold the full cycle dataset in memory. Use random.Random(seed) / numpy.random.default_rng(seed) only.
File tree to produce
out/csv/
├── batches.csv # 4 rows: deposition params
├── simtra/
│ └── simtra_profile_<batch>.csv # 4 files, 1000 rows each: angle, energy_eV, pt_flux, au_flux
├── batch_<B>/wafer_<W>/
│ ├── wafer_info.csv
│ └── coupon_<C>/
│ ├── coupon_info.csv # position on wafer, thickness_um, ra_nm, assigned run/plate/probe/square or RESERVE
│ ├── xrf_map.csv # 400 rows: grid_x, grid_y, pt_wtpct, au_wtpct
│ ├── profilometry.csv # 100 rows: grid_x, grid_y, thickness_um
│ ├── nanoindentation.csv # 25 rows: indent_id, x_um, y_um, hardness_GPa, reduced_modulus_GPa, max_load_mN
│ ├── afm.csv # 1 row: ra_nm, rq_nm, image_file (synthetic path)
│ └── track_<T>/ # only for 480 friction coupons, T in 1..3
│ ├── track_info.csv # run_id, environment, load_mN, stroke_mm, speed_mm_s, counterface_id, started_at
│ ├── cof_vs_cycle.csv # 1000 rows: cycle, cof
│ ├── friction_loops.csv # 2000 rows: cycle (every 100th), position_um (200 pts), friction_force_mN
│ └── wear.csv # 1 row: wear_volume_um3, k_archard, sliding_distance_m
└── runs.csv # 4 rows: run_id, environment, date, plates, operator
Value generation
Apply exactly the physical models of task 01 (gradient Au%, run-in COF, hardness/modulus vs Au%, Archard wear). COF must correlate with the coupon's mean Au% and the run environment so that Q2/Q3/Q7 benchmarks return physically meaningful results.
Size control
Expected: cycles 1.44 M rows (~30 MB) + loops 2.88 M rows (~95 MB) + XRF 235 K rows (~8 MB) + remainder (~10 MB) ≈ 150 MB. After generation, print actual tree size; abort with error if outside 120-200 MB.
Output
- Full tree under
./out/csv/(~150 MB). ./out/csv/MANIFEST.csv: file path, rows, bytes, sha256 - used by conversion tasks for validation.- Marker
./out/.done/03.okwith total rows/bytes.