b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
18 lines
1.2 KiB
Markdown
18 lines
1.2 KiB
Markdown
# Task 03 — Filename & Path Tokenization
|
|
|
|
## Goal
|
|
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
|
|
|
|
## Input
|
|
- `manifest/inventory.jsonl` (Task 02).
|
|
- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
|
|
|
|
## Actions
|
|
1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
|
|
2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
|
|
3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
|
|
|
|
## Intermediate Result (done when)
|
|
- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
|
|
- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).
|