LabDataStorageEvaluation/docs/rules/data-determinism.md

Data / Determinism - one seed, byte-identical regeneration

Purpose + scope. The reproducibility contract for all generated data. Binding for task 03 (CSV generation) and for every piece of code that produces values which end up inside the corpus or its conversions.

The entire evaluation rests on this: benchmark results are only comparable because every format holds EXACTLY the same data, and the study is only reviewable because anyone can regenerate that data bit-for-bit.

---

1. The seed

• The global seed is 20260711. It lives in lab_config.yaml (written by task 01) and is READ from there - never hardcoded in a generation script (see code-config-yaml.md).
• Changing the seed is a project-level decision, never a side effect. It invalidates the whole ./out/ tree, every checksum, and every measured result (see meta-no-assumptions.md).

2. Sanctioned randomness - seeded generators only

• The ONLY random sources are random.Random(seed) and numpy.random.default_rng(seed), constructed explicitly from the config seed.
• Derive per-entity generators deterministically when parallel or out-of-order generation needs them (e.g. default_rng([seed, batch_index, coupon_index])) - never from time, never from a global unseeded state.
• Forbidden anywhere in generation code: module-level random.* calls, numpy.random.* legacy global functions, os.urandom, uuid.uuid4, hash()-dependent ordering (Python string hashing is salted per process).

3. No wall-clock in the data

• Every timestamp inside the corpus is SIMULATED - computed from the lab schedule model - never datetime.now() / time.time().
• Real wall-clock time may appear only in logs and in measurement records (benchmark wall times in ./out/bench/ are measurements, not corpus data).

4. Deterministic everything else

Randomness is not the only nondeterminism; all of these are binding:

• Iteration order. Iterate in a defined order (sorted paths, explicit index loops). Never rely on os.listdir / glob / set / dict-of-set ordering for anything that affects output. Sort directory listings before use.
• Float formatting. Format floats with an explicit spec (e.g. f"{x:.6g}"), chosen once per column family; repr drift between runs or Python versions must not change file bytes.
• Line endings and encoding. Generated files are UTF-8 with \n newlines, on every platform (open(..., newline="") for csv writers).
• Parallelism. If generation is parallelized, each worker's output is deterministic (derived generators, fixed partitioning) and the merge order is fixed.

5. The proof - MANIFEST sha256

• ./out/csv/MANIFEST.csv records sha256 per generated file (spec 03). Regenerating with the same lab_config.yaml MUST reproduce every sha256 exactly. This is the acceptance test for this whole rule - run generation twice when touching generator code and diff the manifests.
• Downstream conversions must be equally deterministic in CONTENT (same rows, same values); byte-identity of database files is not required (SQLite/PostgreSQL internals may differ), which is why their gate is row counts + query-result checksums instead (see test-pipeline-validation.md).

6. Anti-patterns

• Hardcoding 20260711 in a script instead of reading the config.
• An unseeded or globally-seeded random call anywhere in generation.
• datetime.now() in corpus values - simulated schedule only.
• Ordering that depends on the filesystem or hash salting - sort explicitly.
• "Harmless" float repr changes - a formatting change is a corpus change; it must be deliberate and re-baselines every checksum.
• Regenerating the corpus to "refresh" it - identical input config must yield identical output; if bytes changed without a config change, that is a determinism bug to fix, not a new baseline to accept.