b4270b0247
- update README to clarify the exclusion of bulk artifacts from git - specify committed curated results subset in README and rules - improve descriptions of output directories and their contents - refine details on the pipeline's data generation and reporting processes
110 lines
4.7 KiB
Markdown
110 lines
4.7 KiB
Markdown
# Build / Pipeline tasks - the ./out tree, .done markers, task order
|
|
|
|
**Purpose + scope.** How the 11 pipeline tasks compose: where artifacts
|
|
live, how completion is signaled, what order tasks run in, and what git
|
|
does and does not track. Binding for every task implementation and for
|
|
anyone running the pipeline.
|
|
|
|
---
|
|
|
|
## 1. The artifact tree - everything under `./out/`
|
|
|
|
All generated artifacts live under `./out/` relative to the repo root
|
|
(spec 00_PLAN):
|
|
|
|
```
|
|
out/
|
|
config/ lab_config.yaml, hw_prices.yaml
|
|
csv/ the canonical raw corpus + MANIFEST.csv
|
|
json/ full/ (per-coupon JSON-LD), hybrid/dataset.jsonld
|
|
sqlite/ tribo.db
|
|
pg/ tribo.dump (pg_dump -Fc; the live DB is in PostgreSQL)
|
|
rdf/ dataset.nt.gz, dataset.ttl, oxigraph_store/
|
|
bench/ queries/, expected/, results_raw.csv, results_median.csv,
|
|
storage_sizes.csv, extrapolation.csv, hardware_sizing.csv
|
|
report/ REPORT.md, charts/, tables/, process_flow.md, diagrams/
|
|
.done/ completion markers, one per task
|
|
```
|
|
|
|
- A task writes ONLY under its own output paths; it treats its inputs as
|
|
read-only.
|
|
- Tasks communicate exclusively through these artifacts - never through
|
|
shared in-process state
|
|
(see [code-python-style.md](code-python-style.md)).
|
|
- **The bulk of `./out/` is git-ignored.** Sources of truth are the
|
|
specs, the rules, and the code; artifacts are reproducible from them
|
|
([data-determinism.md](data-determinism.md)). A curated results subset
|
|
IS committed (owner decision, 2026-07-11): `out/report/` (REPORT.md,
|
|
charts, tables, diagrams), `out/config/` (lab_config.yaml,
|
|
hw_prices.yaml), and the `out/.done/` markers - exactly the exceptions
|
|
listed in `.gitignore`. Never commit the bulk artifacts (corpus,
|
|
databases, stores, archives, raw benchmark data) "for convenience".
|
|
|
|
## 2. Completion markers
|
|
|
|
- Each task NN writes `./out/.done/<NN>.ok` after - and only after - its
|
|
outputs are complete and validated. The marker body carries the
|
|
headline counts (rows, bytes, triples) downstream tasks check
|
|
(see [test-pipeline-validation.md](test-pipeline-validation.md)).
|
|
- A task verifies its dependencies' markers before starting and aborts
|
|
loudly if one is missing.
|
|
- Re-running a task: delete its own marker first, regenerate outputs,
|
|
rewrite the marker. If its outputs changed, downstream markers are now
|
|
stale - delete them too (the dependency graph below says which).
|
|
|
|
## 3. Task graph and execution order
|
|
|
|
```
|
|
01 -> 02
|
|
01 -> 03 -> (04 | 05 | 06 | 07) -> 08 -> 09 -> 10 -> 11
|
|
09 ------> 11
|
|
```
|
|
|
|
- Sequential order: 01, 02, 03, then 04-07 (independent, may run in
|
|
parallel), 08, 09, 10, 11.
|
|
- 02 (process diagrams) needs only 01 and can run any time after it.
|
|
- Task 09 (benchmark execution) must run WITHOUT 04-07 style parallel
|
|
work in the background - measurements need a quiet machine
|
|
([bench-methodology.md](bench-methodology.md) section 6).
|
|
|
|
## 4. `storage_sizes.csv` - the shared append contract
|
|
|
|
- Location: `./out/bench/storage_sizes.csv`; columns:
|
|
`format, variant, bytes`.
|
|
- Tasks 04-07 APPEND their format's rows (e.g. `json,full,...`,
|
|
`json,hybrid,...`, `sqlite,db,...`, `pg,live,...`, `pg,dump,...`,
|
|
`rdf,nt_gz,...`, `rdf,store,...`); task 09 appends the gzip archival
|
|
variants and consolidates.
|
|
- Appends are idempotent per (format, variant): a re-run REPLACES its own
|
|
prior rows, never duplicates them and never touches other formats'
|
|
rows.
|
|
- RDF load wall-time and similar per-format load metrics recorded per
|
|
spec 07 go into the task's `.done` marker and the report, not into
|
|
extra ad-hoc files.
|
|
|
|
## 5. Runs are resumable, not restartable-from-zero
|
|
|
|
- The marker system exists so a failed pipeline resumes at the failed
|
|
task; do not delete the whole `./out/` tree to retry one task.
|
|
- Conversely, after a deliberate corpus regeneration (03), ALL downstream
|
|
markers and artifacts (04-11) are stale and must be removed - a mixed
|
|
tree of old and new artifacts is worse than an empty one
|
|
([meta-no-assumptions.md](meta-no-assumptions.md) applies before any
|
|
such deletion).
|
|
|
|
## 6. Anti-patterns
|
|
|
|
- **Writing an artifact outside `./out/`** or a task writing into another
|
|
task's output directory.
|
|
- **Committing bulk `./out/` content to git** (corpus, databases, stores,
|
|
archives, raw benchmark data) - only the curated subset in `.gitignore`
|
|
(`report/`, `config/`, `.done/`) is committed.
|
|
- **A marker written on partial success** - see
|
|
[code-error-handling.md](code-error-handling.md).
|
|
- **Running benchmarks while converters are still running.**
|
|
- **Duplicate rows in `storage_sizes.csv`** after re-runs - replace your
|
|
own rows.
|
|
- **Hand-editing generated artifacts** (a "quick fix" inside
|
|
`MANIFEST.csv` or a result CSV) - regenerate them through the owning
|
|
task instead.
|