fix(docs): fix formatting issues in README.md

- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
2026-07-11 13:39:13 -04:00
parent d8353953b0
commit b173ac82a9
55 changed files with 3755 additions and 9 deletions

View File

@@ -0,0 +1,85 @@
# Data / Determinism - one seed, byte-identical regeneration
**Purpose + scope.** The reproducibility contract for all generated data.
Binding for task 03 (CSV generation) and for every piece of code that
produces values which end up inside the corpus or its conversions.
The entire evaluation rests on this: benchmark results are only comparable
because every format holds EXACTLY the same data, and the study is only
reviewable because anyone can regenerate that data bit-for-bit.
---
## 1. The seed
- **The global seed is `20260711`.** It lives in `lab_config.yaml`
(written by task 01) and is READ from there - never hardcoded in a
generation script (see [code-config-yaml.md](code-config-yaml.md)).
- Changing the seed is a project-level decision, never a side effect. It
invalidates the whole `./out/` tree, every checksum, and every measured
result (see [meta-no-assumptions.md](meta-no-assumptions.md)).
## 2. Sanctioned randomness - seeded generators only
- The ONLY random sources are `random.Random(seed)` and
`numpy.random.default_rng(seed)`, constructed explicitly from the
config seed.
- Derive per-entity generators deterministically when parallel or
out-of-order generation needs them (e.g.
`default_rng([seed, batch_index, coupon_index])`) - never from time,
never from a global unseeded state.
- Forbidden anywhere in generation code: module-level `random.*` calls,
`numpy.random.*` legacy global functions, `os.urandom`, `uuid.uuid4`,
`hash()`-dependent ordering (Python string hashing is salted per
process).
## 3. No wall-clock in the data
- Every timestamp inside the corpus is SIMULATED - computed from the lab
schedule model - never `datetime.now()` / `time.time()`.
- Real wall-clock time may appear only in logs and in measurement records
(benchmark wall times in `./out/bench/` are measurements, not corpus
data).
## 4. Deterministic everything else
Randomness is not the only nondeterminism; all of these are binding:
- **Iteration order.** Iterate in a defined order (sorted paths, explicit
index loops). Never rely on `os.listdir` / `glob` / `set` / dict-of-set
ordering for anything that affects output. Sort directory listings
before use.
- **Float formatting.** Format floats with an explicit spec (e.g.
`f"{x:.6g}"`), chosen once per column family; repr drift between runs
or Python versions must not change file bytes.
- **Line endings and encoding.** Generated files are UTF-8 with `\n`
newlines, on every platform (`open(..., newline="")` for csv writers).
- **Parallelism.** If generation is parallelized, each worker's output is
deterministic (derived generators, fixed partitioning) and the merge
order is fixed.
## 5. The proof - MANIFEST sha256
- `./out/csv/MANIFEST.csv` records sha256 per generated file (spec 03).
**Regenerating with the same `lab_config.yaml` MUST reproduce every
sha256 exactly.** This is the acceptance test for this whole rule -
run generation twice when touching generator code and diff the
manifests.
- Downstream conversions must be equally deterministic in CONTENT (same
rows, same values); byte-identity of database files is not required
(SQLite/PostgreSQL internals may differ), which is why their gate is
row counts + query-result checksums instead
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
## 6. Anti-patterns
- **Hardcoding `20260711`** in a script instead of reading the config.
- **An unseeded or globally-seeded random call** anywhere in generation.
- **`datetime.now()` in corpus values** - simulated schedule only.
- **Ordering that depends on the filesystem or hash salting** - sort
explicitly.
- **"Harmless" float repr changes** - a formatting change is a corpus
change; it must be deliberate and re-baselines every checksum.
- **Regenerating the corpus to "refresh" it** - identical input config
must yield identical output; if bytes changed without a config change,
that is a determinism bug to fix, not a new baseline to accept.