e3359383c0
- introduce reserve grid positions for QA witness pattern - add batch codes to run entries in lab configuration - update schedule with run start time and track interval - enhance validation checks for run-batch mapping
2.7 KiB
03 - Simulated CSV Data Generation (~150 MB)
Context / Goal
Generate the canonical raw dataset as a hierarchical CSV file tree, exactly as instruments would write it. This corpus is the single source for all conversions and benchmarks. Target size: 140-170 MB.
Dependencies
01 (lab_config.yaml).
Input
./out/config/lab_config.yaml - volumes, deposition matrix, physical model coefficients, seed 20260711.
Processing instructions
Write generate_data.py (tabs for indentation, pure stdlib + numpy). Stream rows; never hold the full cycle dataset in memory. Use random.Random(seed) / numpy.random.default_rng(seed) only.
File tree to produce
out/csv/
├── batches.csv # 4 rows: deposition params
├── simtra/
│ └── simtra_profile_<batch>.csv # 4 files, 1000 rows each: angle_deg, energy_eV, pt_flux, au_flux
├── batch_<B>/wafer_<W>/
│ ├── wafer_info.csv
│ └── coupon_<C>/
│ ├── coupon_info.csv # position on wafer, thickness_um, ra_nm, assigned run/plate/probe/square or RESERVE
│ ├── xrf_map.csv # 400 rows: grid_x, grid_y, pt_wtpct, au_wtpct
│ ├── profilometry.csv # 100 rows: grid_x, grid_y, thickness_um
│ ├── nanoindentation.csv # 25 rows: indent_id, x_um, y_um, hardness_GPa, reduced_modulus_GPa, max_load_mN
│ ├── afm.csv # 1 row: ra_nm, rq_nm, image_file (synthetic path)
│ └── track_<T>/ # only for 480 friction coupons, T in 1..3
│ ├── track_info.csv # run_id, environment, load_mN, stroke_mm, speed_mm_s, counterface_id, started_at
│ ├── cof_vs_cycle.csv # 1000 rows: cycle, cof
│ ├── friction_loops.csv # 2000 rows: cycle (every 100th), position_um (200 pts), friction_force_mN
│ └── wear.csv # 1 row: wear_volume_um3, k_archard, sliding_distance_m
└── runs.csv # 4 rows: run_id, environment, date, plates, operator
Value generation
Apply exactly the physical models of task 01 (gradient Au%, run-in COF, hardness/modulus vs Au%, Archard wear). COF must correlate with the coupon's mean Au% and the run environment so that Q2/Q3/Q7 benchmarks return physically meaningful results.
Size control
Expected: cycles 1.44 M rows (~30 MB) + loops 2.88 M rows (~95 MB) + XRF 235 K rows (~8 MB) + remainder (~10 MB) ≈ 150 MB. After generation, print actual tree size; abort with error if outside 120-200 MB.
Output
- Full tree under
./out/csv/(~150 MB). ./out/csv/MANIFEST.csv: file path, rows, bytes, sha256 - used by conversion tasks for validation.- Marker
./out/.done/03.okwith total rows/bytes.