Public/LabDataStorageEvaluation
1
0
Fork 0
Code Issues Pull Requests Actions Releases Wiki Activity

feat(convert): add task 05 sqlite converter with variant B keys" -m "- convert: convert_sqlite.py builds tribo.db (composite PKs, enforced FKs, Q1-Q7 indexes, track_summary)

Browse Source 
- tools: common/track_summary.py fixes the cross-engine steady-state algorithm
- rules: db-sql-schema records the FK enforcement decision and track_summary definition
- specs: 05 aligned with data-naming-units, simtra_profiles added
- docs: FK key-schema study added under docs/research/, README updated
- replace manual CSV reading with CorpusReader for better data handling
- streamline argument parsing and dependency checks using common pipeline functions
- enhance marker writing for task completion tracking
- remove unused regex and validation error classes for cleaner code
This commit is contained in:
administrator
2026-07-11 16:08:29 -04:00
parent 9364808b3d
commit 96ed7bc918
12 changed files with 1294 additions and 240 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/specs/05_convert_sqlite.md
View File

@@ -10,17 +10,19 @@ Build the single-file relational variant optimized for size and single-user quer
CSV tree.
## Processing instructions
- One database `./out/sqlite/tribo.db`. PRAGMAs during load: `journal_mode=OFF, synchronous=OFF, cache_size=-262144`; after load: `journal_mode=WAL`, `ANALYZE`, `VACUUM`.
- Schema (16-table style, integer surrogate keys everywhere; no TEXT foreign keys in bulk tables):
- One database `./out/sqlite/tribo.db`. PRAGMAs during load: `journal_mode=OFF, synchronous=OFF, cache_size=-262144, foreign_keys=ON`; after load: `journal_mode=WAL`, `ANALYZE`, `VACUUM`.
- Foreign keys are enforced on every table, including bulk tables (variant B of the measured key-schema study; owner decision 2026-07-11, see db-sql-schema rule). Full study with ER diagrams and measurements: [../research/Tribology_FK_Architecture.html](../research/Tribology_FK_Architecture.html).
- Schema (16-table style, integer surrogate keys everywhere; no TEXT foreign keys in bulk tables; SQL column names are the lowercase of the CSV headers per data-naming-units):
- `batches, wafers, coupons, runs, instruments, tracks` - dimension tables, TEXT natural keys kept as unique columns.
- `xrf_points(coupon_id INT, gx INT, gy INT, pt_wtpct REAL, au_wtpct REAL)`.
- `profilometry_points(coupon_id INT, gx INT, gy INT, thickness_um REAL)`.
- `nanoindentation(coupon_id INT, indent_id INT, x_um REAL, y_um REAL, hardness_gpa REAL, reduced_modulus_gpa REAL)`.
- `afm(coupon_id INT PRIMARY KEY, ra_nm REAL, rq_nm REAL)`.
- `simtra_profiles(batch_id INT, row_no INT, angle_deg REAL, energy_ev REAL, pt_flux REAL, au_flux REAL, PRIMARY KEY(batch_id, row_no)) WITHOUT ROWID`.
- `xrf_points(coupon_id INT, grid_x INT, grid_y INT, pt_wtpct REAL, au_wtpct REAL, PRIMARY KEY(coupon_id, grid_x, grid_y)) WITHOUT ROWID`.
- `profilometry_points(coupon_id INT, grid_x INT, grid_y INT, thickness_um REAL, PRIMARY KEY(coupon_id, grid_x, grid_y)) WITHOUT ROWID`.
- `nanoindentation(coupon_id INT, indent_id INT, x_um REAL, y_um REAL, hardness_gpa REAL, reduced_modulus_gpa REAL, max_load_mn REAL, PRIMARY KEY(coupon_id, indent_id)) WITHOUT ROWID`.
- `afm(coupon_id INT PRIMARY KEY, ra_nm REAL, rq_nm REAL, image_file TEXT)`.
- `friction_cycles(track_id INT, cycle INT, cof REAL, PRIMARY KEY(track_id, cycle)) WITHOUT ROWID`.
- `friction_loop_points(track_id INT, cycle INT, pt INT, position_um REAL, force_mn REAL, PRIMARY KEY(track_id, cycle, pt)) WITHOUT ROWID`.
- `friction_loop_points(track_id INT, cycle INT, pt INT, position_um REAL, friction_force_mn REAL, PRIMARY KEY(track_id, cycle, pt)) WITHOUT ROWID`.
- `wear(track_id INT PRIMARY KEY, wear_volume_um3 REAL, k_archard REAL, sliding_distance_m REAL)`.
- `track_summary(track_id INT PRIMARY KEY, cof_ss_mean REAL, cof_ss_std REAL, run_in_cycles INT)` - precomputed during load.
- `track_summary(track_id INT PRIMARY KEY, cof_ss_mean REAL, cof_ss_std REAL, run_in_cycles INT)` - precomputed during load (algorithm fixed in db-sql-schema rule, section 5).
- Indexes for Q1-Q7: `coupons(au_wtpct_mean)`, `tracks(run_id)`, `tracks(coupon_id)`, `runs(environment)`, composite `tracks(environment, load_mn)` (denormalize environment/load onto tracks).
- Batch inserts with `executemany`, 50k rows per transaction.
- Validate row counts against MANIFEST.