fix(docs): fix formatting issues in README.md

- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
2026-07-11 13:39:13 -04:00
parent d8353953b0
commit b173ac82a9
55 changed files with 3755 additions and 9 deletions

View File

@@ -0,0 +1,18 @@
# Task 04 — Schema Profiling & Fingerprints
## Goal
Capture the record structure of every CSV without full load; compute schema fingerprints.
## Input
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
## Actions
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
## Intermediate Result (done when)
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
- Every profile carries a fingerprint.