LabDataStorageEvaluation/docs/rules/obs-python-logging.md

# Observability / Python logging - standard library only, namespaced, plain text

**Purpose + scope.** Every Python module in this repository (data generation,
converters, benchmark runner, extrapolation, reporting, helper tools) MUST
use the standard-library `logging` module following these conventions. No
external logging dependency.

---

## 1. The rule

1. **Standard library `logging` only.** No `loguru`, no `structlog`, no
   `python-json-logger`. Stdlib is sufficient and keeps the dependency list
   closed (see [code-python-style.md](code-python-style.md)).
2. **Namespaced loggers.** Every module obtains its logger via
   `logging.getLogger(__name__)`. Never the root logger directly.
3. **`print()` is for the user-facing CLI protocol only.** The specs
   explicitly require certain stdout output: progress summaries, the real
   generated tree size (task 03), per-cell benchmark progress, final
   counts. That protocol output goes through `print()`. Everything
   diagnostic - internal flow, warnings, errors, per-row detail - goes
   through a logger.
4. **Log level from a CLI flag.** Every pipeline script accepts
   `--log-level {DEBUG,INFO,WARNING,ERROR}`, default `INFO`. There is no
   settings table and no `.env` in this project (see
   [code-config-yaml.md](code-config-yaml.md)).
5. **Plain-text format on stderr.** These are one-shot CLI scripts, not
   services; JSON logs are overkill. One line per record via
   `logging.basicConfig`. Keeping the logger on stderr leaves stdout clean
   for the protocol output of item 3.
6. **Never log secrets.** The only credential material in this project is
   the PostgreSQL DSN - never log it (a DSN may embed a password). Log the
   database name or host, never the full connection string.

## 2. API by severity

| Method | Use for |
|---|---|
| `logger.critical(...)` | The process cannot continue - before `sys.exit(1)` or an unrecoverable state. |
| `logger.error(...)` | Failed operation, exception caught and handled, broken invariant, a benchmark cell recorded as TIMEOUT/ERROR. Always include `exc_info=True` when an exception is in scope. |
| `logger.warning(...)` | Recoverable problem, retry triggered, suspicious state (e.g. a value near a validation bound), an optional step skipped (e.g. cold-cache pass unavailable). |
| `logger.info(...)` | Normal flow milestones - task start/finish, per-batch progress, row counts written, validation gates passed. |
| `logger.debug(...)` | Verbose tracing - per-coupon detail, generated value samples, SQL statements, per-run raw measurements. Hidden at default `INFO`. |

**Default level: `INFO`.** `DEBUG` is opt-in via `--log-level DEBUG` for a
diagnosis session.

## 3. Logger acquisition and setup

```python
# Top of every module that logs:
import logging
logger = logging.getLogger(__name__)

# Once, in each script's main():
logging.basicConfig(
	level=args.log_level,
	stream=sys.stderr,
	format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)
```

Inside functions:

```python
def load_table(conn, name: str, rows) -> int:
	logger.info("loading %s", name)
	try:
		...
	except psycopg.Error:
		logger.error("loading %s failed", name, exc_info=True)
		raise
```

`__name__` gives the module path as the namespace tree used for filtering.
Never hardcode a logger name string. Clamp noisy third-party libraries
(`psycopg`, `matplotlib`) to `WARNING` in the same setup block if they
pollute the output.

## 4. Progress reporting in long loops

Generation and loading loops run over millions of rows. Report progress at
INFO on a coarse cadence (per batch / per wafer / per table, or every N
seconds), never per row. Per-row output belongs at DEBUG, and even there
prefer sampled output ("first row of each file") over full dumps.

## 5. Anti-patterns

- **`print()` for diagnostics** - forbidden; `print()` is only the
  user-facing CLI protocol (progress summaries, final counts, sizes).
- **Diagnostics on stdout** - stdout is the protocol channel; the logger
  writes to stderr.
- **Root logger calls** (`logging.info(...)`) - forbidden; always
  `logger = logging.getLogger(__name__)`.
- **Third-party logging frameworks** (`loguru`, `structlog`) - forbidden.
- **Hardcoded logger names** (`logging.getLogger("datagen")`) - forbidden;
  pass `__name__`.
- **Logging the PostgreSQL DSN / connection string** - forbidden; log
  host/database name only.
- **Log level from an environment variable or a config file** - the level
  is a per-run diagnostic choice; it comes from `--log-level`.
- **`logger.error(...)` without re-raise** when the function semantically
  cannot proceed - see [code-error-handling.md](code-error-handling.md).
- **Per-row INFO logging in bulk loops** - drowns the signal and slows the
  run; use coarse milestones.

## 6. Related documents

- [code-error-handling.md](code-error-handling.md) - `exc_info=True`
  requirement and the swallow ladder.
- [code-config-yaml.md](code-config-yaml.md) - configuration scope (no
  `.env`, no settings table).
- [code-python-style.md](code-python-style.md) - the closed dependency list.