LabDataStorageEvaluation/docs/rules/obs-python-logging.md

Observability / Python logging - standard library only, namespaced, plain text

Purpose + scope. Every Python module in this repository (data generation, converters, benchmark runner, extrapolation, reporting, helper tools) MUST use the standard-library logging module following these conventions. No external logging dependency.

---

1. The rule

1. Standard library logging only. No loguru, no structlog, no python-json-logger. Stdlib is sufficient and keeps the dependency list closed (see code-python-style.md).
2. Namespaced loggers. Every module obtains its logger via logging.getLogger(__name__). Never the root logger directly.
3. print() is for the user-facing CLI protocol only. The specs explicitly require certain stdout output: progress summaries, the real generated tree size (task 03), per-cell benchmark progress, final counts. That protocol output goes through print(). Everything diagnostic - internal flow, warnings, errors, per-row detail - goes through a logger.
4. Log level from a CLI flag. Every pipeline script accepts --log-level {DEBUG,INFO,WARNING,ERROR}, default INFO. There is no settings table and no .env in this project (see code-config-yaml.md).
5. Plain-text format on stderr. These are one-shot CLI scripts, not services; JSON logs are overkill. One line per record via logging.basicConfig. Keeping the logger on stderr leaves stdout clean for the protocol output of item 3.
6. Never log secrets. The only credential material in this project is the PostgreSQL DSN - never log it (a DSN may embed a password). Log the database name or host, never the full connection string.

2. API by severity

Method	Use for
logger.critical(...)	The process cannot continue - before sys.exit(1) or an unrecoverable state.
logger.error(...)	Failed operation, exception caught and handled, broken invariant, a benchmark cell recorded as TIMEOUT/ERROR. Always include exc_info=True when an exception is in scope.
logger.warning(...)	Recoverable problem, retry triggered, suspicious state (e.g. a value near a validation bound), an optional step skipped (e.g. cold-cache pass unavailable).
logger.info(...)	Normal flow milestones - task start/finish, per-batch progress, row counts written, validation gates passed.
logger.debug(...)	Verbose tracing - per-coupon detail, generated value samples, SQL statements, per-run raw measurements. Hidden at default INFO.

Default level: INFO. DEBUG is opt-in via --log-level DEBUG for a diagnosis session.

3. Logger acquisition and setup

# Top of every module that logs:
import logging
logger = logging.getLogger(__name__)

# Once, in each script's main():
logging.basicConfig(
	level=args.log_level,
	stream=sys.stderr,
	format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
)

Inside functions:

def load_table(conn, name: str, rows) -> int:
	logger.info("loading %s", name)
	try:
		...
	except psycopg.Error:
		logger.error("loading %s failed", name, exc_info=True)
		raise

__name__ gives the module path as the namespace tree used for filtering. Never hardcode a logger name string. Clamp noisy third-party libraries (psycopg, matplotlib) to WARNING in the same setup block if they pollute the output.

4. Progress reporting in long loops

Generation and loading loops run over millions of rows. Report progress at INFO on a coarse cadence (per batch / per wafer / per table, or every N seconds), never per row. Per-row output belongs at DEBUG, and even there prefer sampled output ("first row of each file") over full dumps.

5. Anti-patterns

• print() for diagnostics - forbidden; print() is only the user-facing CLI protocol (progress summaries, final counts, sizes).
• Diagnostics on stdout - stdout is the protocol channel; the logger writes to stderr.
• Root logger calls (logging.info(...)) - forbidden; always logger = logging.getLogger(__name__).
• Third-party logging frameworks (loguru, structlog) - forbidden.
• Hardcoded logger names (logging.getLogger("datagen")) - forbidden; pass __name__.
• Logging the PostgreSQL DSN / connection string - forbidden; log host/database name only.
• Log level from an environment variable or a config file - the level is a per-run diagnostic choice; it comes from --log-level.
• logger.error(...) without re-raise when the function semantically cannot proceed - see code-error-handling.md.
• Per-row INFO logging in bulk loops - drowns the signal and slows the run; use coarse milestones.

6. Related documents

• code-error-handling.md - exc_info=True requirement and the swallow ladder.
• code-config-yaml.md - configuration scope (no .env, no settings table).
• code-python-style.md - the closed dependency list.