feat(convert): add task 05 sqlite converter with variant B keys" -m "- convert: convert_sqlite.py builds tribo.db (composite PKs, enforced FKs, Q1-Q7 indexes, track_summary)

- tools: common/track_summary.py fixes the cross-engine steady-state algorithm
- rules: db-sql-schema records the FK enforcement decision and track_summary definition
- specs: 05 aligned with data-naming-units, simtra_profiles added
- docs: FK key-schema study added under docs/research/, README updated
- replace manual CSV reading with CorpusReader for better data handling
- streamline argument parsing and dependency checks using common pipeline functions
- enhance marker writing for task completion tracking
- remove unused regex and validation error classes for cleaner code
This commit is contained in:
administrator
2026-07-11 16:08:29 -04:00
parent 9364808b3d
commit 96ed7bc918
12 changed files with 1294 additions and 240 deletions

View File

@@ -74,11 +74,14 @@ and it wins where a spec detail is ambiguous.
- Loads are all-or-nothing per table and validated against `MANIFEST.csv`
row counts - see [code-error-handling.md](code-error-handling.md)
section 3 and [test-pipeline-validation.md](test-pipeline-validation.md).
- On every SQLite connection that relies on FK behaviour:
`PRAGMA foreign_keys = ON` (SQLite silently ignores FK clauses without
it). Bulk tables MAY declare FK columns without enforced constraints for
load speed - row-count validation is the integrity gate; whichever choice
is made must be the same in both engines.
- **Foreign keys are ENFORCED in both engines on every table, including
bulk tables** (owner decision 2026-07-11, after measurement on the real
corpus: +0.5 s load on 4.3M rows in SQLite, zero disk and read cost;
full study with ER diagrams and measurements:
[docs/research/Tribology_FK_Architecture.html](../research/Tribology_FK_Architecture.html)).
Every SQLite connection sets `PRAGMA foreign_keys = ON` (SQLite silently
ignores FK clauses without it). Row-count validation against
`MANIFEST.csv` remains a second, independent integrity gate.
## 5. `track_summary` - precomputed, and off-limits to Q5
@@ -89,6 +92,20 @@ never read it** - Q5 is the forced full-scan benchmark over raw
`friction_cycles` in every format (spec
[08](../specs/08_benchmark_queries.md)).
Binding computation (identical semantics in both engines; float results
agree within the 1e-9 checksum tolerance):
1. Steady-state proxy window = cycles 501-1000 (the second half);
`m` = avg(cof), `s` = stddev_pop(cof) over that window.
2. `run_in_cycles` = the first cycle `c` with `|cof(c) - m| < 2*s`;
fallback 500 if no cycle qualifies.
3. `cof_ss_mean` / `cof_ss_std` = avg / stddev_pop of cof over cycles
`> run_in_cycles`.
SQLite computes this in Python during load
([common/track_summary.py](../../common/track_summary.py)); PostgreSQL as
a materialized view implementing the same three steps.
## 6. Engine parity
- Same logical schema, same table names, same column names, same row