fix(docs): fix formatting issues in README.md

- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
2026-07-11 13:39:13 -04:00
parent d8353953b0
commit b173ac82a9
55 changed files with 3755 additions and 9 deletions

View File

@@ -0,0 +1,113 @@
# Observability / Python logging - standard library only, namespaced, plain text
**Purpose + scope.** Every Python module in this repository (data generation,
converters, benchmark runner, extrapolation, reporting, helper tools) MUST
use the standard-library `logging` module following these conventions. No
external logging dependency.
---
## 1. The rule
1. **Standard library `logging` only.** No `loguru`, no `structlog`, no
`python-json-logger`. Stdlib is sufficient and keeps the dependency list
closed (see [code-python-style.md](code-python-style.md)).
2. **Namespaced loggers.** Every module obtains its logger via
`logging.getLogger(__name__)`. Never the root logger directly.
3. **`print()` is for the user-facing CLI protocol only.** The specs
explicitly require certain stdout output: progress summaries, the real
generated tree size (task 03), per-cell benchmark progress, final
counts. That protocol output goes through `print()`. Everything
diagnostic - internal flow, warnings, errors, per-row detail - goes
through a logger.
4. **Log level from a CLI flag.** Every pipeline script accepts
`--log-level {DEBUG,INFO,WARNING,ERROR}`, default `INFO`. There is no
settings table and no `.env` in this project (see
[code-config-yaml.md](code-config-yaml.md)).
5. **Plain-text format on stderr.** These are one-shot CLI scripts, not
services; JSON logs are overkill. One line per record via
`logging.basicConfig`. Keeping the logger on stderr leaves stdout clean
for the protocol output of item 3.
6. **Never log secrets.** The only credential material in this project is
the PostgreSQL DSN - never log it (a DSN may embed a password). Log the
database name or host, never the full connection string.
## 2. API by severity
| Method | Use for |
|---|---|
| `logger.critical(...)` | The process cannot continue - before `sys.exit(1)` or an unrecoverable state. |
| `logger.error(...)` | Failed operation, exception caught and handled, broken invariant, a benchmark cell recorded as TIMEOUT/ERROR. Always include `exc_info=True` when an exception is in scope. |
| `logger.warning(...)` | Recoverable problem, retry triggered, suspicious state (e.g. a value near a validation bound), an optional step skipped (e.g. cold-cache pass unavailable). |
| `logger.info(...)` | Normal flow milestones - task start/finish, per-batch progress, row counts written, validation gates passed. |
| `logger.debug(...)` | Verbose tracing - per-coupon detail, generated value samples, SQL statements, per-run raw measurements. Hidden at default `INFO`. |
**Default level: `INFO`.** `DEBUG` is opt-in via `--log-level DEBUG` for a
diagnosis session.
## 3. Logger acquisition and setup
```python
# Top of every module that logs:
import logging
logger = logging.getLogger(__name__)
# Once, in each script's main():
logging.basicConfig(
level=args.log_level,
stream=sys.stderr,
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
```
Inside functions:
```python
def load_table(conn, name: str, rows) -> int:
logger.info("loading %s", name)
try:
...
except psycopg.Error:
logger.error("loading %s failed", name, exc_info=True)
raise
```
`__name__` gives the module path as the namespace tree used for filtering.
Never hardcode a logger name string. Clamp noisy third-party libraries
(`psycopg`, `matplotlib`) to `WARNING` in the same setup block if they
pollute the output.
## 4. Progress reporting in long loops
Generation and loading loops run over millions of rows. Report progress at
INFO on a coarse cadence (per batch / per wafer / per table, or every N
seconds), never per row. Per-row output belongs at DEBUG, and even there
prefer sampled output ("first row of each file") over full dumps.
## 5. Anti-patterns
- **`print()` for diagnostics** - forbidden; `print()` is only the
user-facing CLI protocol (progress summaries, final counts, sizes).
- **Diagnostics on stdout** - stdout is the protocol channel; the logger
writes to stderr.
- **Root logger calls** (`logging.info(...)`) - forbidden; always
`logger = logging.getLogger(__name__)`.
- **Third-party logging frameworks** (`loguru`, `structlog`) - forbidden.
- **Hardcoded logger names** (`logging.getLogger("datagen")`) - forbidden;
pass `__name__`.
- **Logging the PostgreSQL DSN / connection string** - forbidden; log
host/database name only.
- **Log level from an environment variable or a config file** - the level
is a per-run diagnostic choice; it comes from `--log-level`.
- **`logger.error(...)` without re-raise** when the function semantically
cannot proceed - see [code-error-handling.md](code-error-handling.md).
- **Per-row INFO logging in bulk loops** - drowns the signal and slows the
run; use coarse milestones.
## 6. Related documents
- [code-error-handling.md](code-error-handling.md) - `exc_info=True`
requirement and the swallow ladder.
- [code-config-yaml.md](code-config-yaml.md) - configuration scope (no
`.env`, no settings table).
- [code-python-style.md](code-python-style.md) - the closed dependency list.