fix(docs): fix formatting issues in README.md
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
18
docs/specs_old/04_schema_profiling.md
Normal file
18
docs/specs_old/04_schema_profiling.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Task 04 — Schema Profiling & Fingerprints
|
||||
|
||||
## Goal
|
||||
Capture the record structure of every CSV without full load; compute schema fingerprints.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
|
||||
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
|
||||
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
|
||||
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
|
||||
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
|
||||
- Every profile carries a fingerprint.
|
||||
Reference in New Issue
Block a user