e3359383c0
- introduce reserve grid positions for QA witness pattern - add batch codes to run entries in lab configuration - update schedule with run start time and track interval - enhance validation checks for run-batch mapping
47 lines
2.7 KiB
Markdown
47 lines
2.7 KiB
Markdown
# 03 - Simulated CSV Data Generation (~150 MB)
|
|
|
|
## Context / Goal
|
|
Generate the canonical raw dataset as a hierarchical CSV file tree, exactly as instruments would write it. This corpus is the single source for all conversions and benchmarks. Target size: 140-170 MB.
|
|
|
|
## Dependencies
|
|
01 (`lab_config.yaml`).
|
|
|
|
## Input
|
|
`./out/config/lab_config.yaml` - volumes, deposition matrix, physical model coefficients, seed 20260711.
|
|
|
|
## Processing instructions
|
|
Write `generate_data.py` (tabs for indentation, pure stdlib + numpy). Stream rows; never hold the full cycle dataset in memory. Use `random.Random(seed)` / `numpy.random.default_rng(seed)` only.
|
|
|
|
### File tree to produce
|
|
```
|
|
out/csv/
|
|
├── batches.csv # 4 rows: deposition params
|
|
├── simtra/
|
|
│ └── simtra_profile_<batch>.csv # 4 files, 1000 rows each: angle_deg, energy_eV, pt_flux, au_flux
|
|
├── batch_<B>/wafer_<W>/
|
|
│ ├── wafer_info.csv
|
|
│ └── coupon_<C>/
|
|
│ ├── coupon_info.csv # position on wafer, thickness_um, ra_nm, assigned run/plate/probe/square or RESERVE
|
|
│ ├── xrf_map.csv # 400 rows: grid_x, grid_y, pt_wtpct, au_wtpct
|
|
│ ├── profilometry.csv # 100 rows: grid_x, grid_y, thickness_um
|
|
│ ├── nanoindentation.csv # 25 rows: indent_id, x_um, y_um, hardness_GPa, reduced_modulus_GPa, max_load_mN
|
|
│ ├── afm.csv # 1 row: ra_nm, rq_nm, image_file (synthetic path)
|
|
│ └── track_<T>/ # only for 480 friction coupons, T in 1..3
|
|
│ ├── track_info.csv # run_id, environment, load_mN, stroke_mm, speed_mm_s, counterface_id, started_at
|
|
│ ├── cof_vs_cycle.csv # 1000 rows: cycle, cof
|
|
│ ├── friction_loops.csv # 2000 rows: cycle (every 100th), position_um (200 pts), friction_force_mN
|
|
│ └── wear.csv # 1 row: wear_volume_um3, k_archard, sliding_distance_m
|
|
└── runs.csv # 4 rows: run_id, environment, date, plates, operator
|
|
```
|
|
|
|
### Value generation
|
|
Apply exactly the physical models of task 01 (gradient Au%, run-in COF, hardness/modulus vs Au%, Archard wear). COF must correlate with the coupon's mean Au% and the run environment so that Q2/Q3/Q7 benchmarks return physically meaningful results.
|
|
|
|
### Size control
|
|
Expected: cycles 1.44 M rows (~30 MB) + loops 2.88 M rows (~95 MB) + XRF 235 K rows (~8 MB) + remainder (~10 MB) ≈ 150 MB. After generation, print actual tree size; abort with error if outside 120-200 MB.
|
|
|
|
## Output
|
|
- Full tree under `./out/csv/` (~150 MB).
|
|
- `./out/csv/MANIFEST.csv`: file path, rows, bytes, sha256 - used by conversion tasks for validation.
|
|
- Marker `./out/.done/03.ok` with total rows/bytes.
|