b173ac82a9
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline - replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents - add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks - normalize specs and README typography to ASCII per code-data-formatting - rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore - specs: pipeline plan and task specs 00-11 - rules: 19 binding rule files adapted for this project - docs: CLAUDE.md rule-trigger table - config: .cursorrules commit convention, .gitignore excluding out/ and .venv/ - docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders - config: requirements.txt for the closed dependency list - datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
1.2 KiB
1.2 KiB
Task 03 — Filename & Path Tokenization
Goal
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
Input
manifest/inventory.jsonl(Task 02).- Known conventions: hierarchy
data_note → batch → coupon(die) → sample; XRFL_<MMDDYY>_<seq>; macro frictiondata1..data3(+ atmosphere); nanoindentationrun1..run9; frictiontest_N/cycleN(ambient) or_N(dryNitrogen); sample tokens like456/457.
Actions
- Encode patterns as regex rules in
config/token_patterns.json(versioned; extend as new patterns are found). - Write
scripts/tokenize.py: apply rules to each inventory row; extract{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}. - Emit
manifest/tokens.jsonlandmanifest/unmatched.csv(files matching no rule, with best-guess notes).
Intermediate Result (done when)
- Every inventory row is in
tokens.jsonlwith either a full classification or anunmatchedflag. unmatched.csvreviewed; genuinely new conventions added totoken_patterns.jsonand the script re-run (repeat until unmatched are only true anomalies).