dev_masha #11
16
README.md
Protected
16
README.md
@@ -12,8 +12,21 @@ the measurements to 600 GB - 6 TB, and produces a ranked decision report for
|
||||
five laboratory use cases: analysis, reporting, search/filtering, archiving,
|
||||
and inter-lab exchange.
|
||||
|
||||
**Results - interactive, single-file HTML (open in a browser or download and share):**
|
||||
|
||||
- **[Storage Format Evaluation](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html)** -
|
||||
the full study results: dashboard, measured matrix, projections to 6 TB,
|
||||
hardware bill of materials, weighted scoring and verdict
|
||||
(source: [docs/research/Tribology_Storage_Evaluation.html](docs/research/Tribology_Storage_Evaluation.html)).
|
||||
- **[Foreign-Key Architecture Study](https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_FK_Architecture.html)** -
|
||||
the measured key-schema study behind the relational design decision
|
||||
(source: [docs/research/Tribology_FK_Architecture.html](docs/research/Tribology_FK_Architecture.html)).
|
||||
|
||||
Reference documents:
|
||||
|
||||
- Task specifications: [docs/specs/](docs/specs/) (plan + tasks 01-11).
|
||||
- Binding conventions: [docs/rules/](docs/rules/) and [CLAUDE.md](CLAUDE.md).
|
||||
- Full decision report (markdown): [out/report/REPORT.md](out/report/REPORT.md).
|
||||
|
||||
## Goals
|
||||
|
||||
@@ -144,8 +157,7 @@ docs/
|
||||
Initial_Prompt.md the generative prompt behind specs 00-11 (provenance, non-binding)
|
||||
specs/ task specifications 00-11 (WHAT to build)
|
||||
rules/ binding conventions (HOW work is done)
|
||||
research/ measured design studies (e.g. FK key-schema study)
|
||||
examples/ imported reference materials (not binding)
|
||||
research/ measured design studies and interactive result pages
|
||||
out/ ALL generated artifacts (reproducible; bulk git-ignored,
|
||||
report/ config/ .done/ committed - see .gitignore):
|
||||
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
|
||||
|
||||
@@ -1,248 +0,0 @@
|
||||
# Tribology Lab Data — Master Plan
|
||||
|
||||
**Discovery · Cleansing/Normalization · Ontology · Conversion**
|
||||
|
||||
Project: LDRD Mechanochemical Alloys FY25-27 — Sandia National Laboratories, Tribology Facility
|
||||
Scope: ~700 GB raw CSV → normalized, self-documenting, multi-format store
|
||||
Target of record: PostgreSQL (per ADR v1.0); deferred virtual RDF (Ontop OBDA) for FAIR federation
|
||||
Status: Plan v1 — Draft
|
||||
|
||||
---
|
||||
|
||||
## 0. Objective
|
||||
|
||||
Transform ~700 GB of heterogeneous instrument-produced CSV into a formally modeled, cleansed, and normalized dataset with full source→target lineage, then generate reproducible exports (JSON, SQLite, PostgreSQL, RDF/triplestore) from a single canonical intermediate.
|
||||
|
||||
The existing artifacts (Business Description, ERD, ADR) already define the **target domain model**. This plan governs how the raw filesystem is reconciled *against* that model and materialized into the export formats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Guiding principles
|
||||
|
||||
1. **Non-destructive.** Source CSV is read-only. All work products are derived; the raw tree is never mutated.
|
||||
2. **Lineage-first.** Every target row carries provenance back to `(source_file, byte/row offset, transform rules applied)`. Lineage is a build-time output, not an afterthought.
|
||||
3. **Phases 2 and 3 are iterative and co-dependent.** A draft ontology guides cleansing; findings in the data refine the ontology. Neither is frozen until both stabilize.
|
||||
4. **Instruments Repository is built, not referenced.** No external etalon exists. Instruments and their characteristics accrete as an ontology sub-model as the data reveals them; instrument ranges later become anomaly criteria.
|
||||
5. **Pure Python first.** Lightweight streaming pipeline, no DB engine during Phases 1–2. Migration to JSX Viewer / better-sqlite3 is deferred.
|
||||
6. **Canonical intermediate.** All exports derive from one normalized representation, so formats never diverge.
|
||||
7. **Reproducibility.** The pipeline is deterministic and re-runnable end-to-end from source + config; no manual editing of intermediates.
|
||||
|
||||
---
|
||||
|
||||
## 2. Phase overview
|
||||
|
||||
| Phase | Name | Nature | Primary output |
|
||||
|-------|------|--------|----------------|
|
||||
| 1 | Storage Discovery & Profiling | Non-destructive scan | Storage manifest + schema fingerprints |
|
||||
| 2 ↔ 3 | Cleansing/Normalization ↔ Ontology | Iterative loop | Cleansed canonical data + relational schema + data dictionary + Instruments Repository |
|
||||
| 4 | Conversion & Export | Deterministic build | JSON / SQLite / PostgreSQL / RDF outputs |
|
||||
|
||||
Cross-cutting throughout: **lineage**, **anomaly detection**, **configuration/rule registry**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase 1 — Storage Discovery & Non-destructive Profiling
|
||||
|
||||
**Goal.** Reconstruct the actual (as-is) and intended (as-designed) structure of the filesystem and record structures, without loading full file bodies.
|
||||
|
||||
### 3.1 Inputs
|
||||
- Root of the ~700 GB CSV tree (read-only).
|
||||
|
||||
### 3.2 Activities
|
||||
|
||||
**A. Filesystem inventory.** Walk the tree; for each file capture: full path, relative path components (folder semantics), filename, size, mtime, extension. Reconstruct the folder-level meaning (project → batch → coupon → sample, per the known hierarchy).
|
||||
|
||||
**B. Filename tokenization.** Parse filenames into structured tokens against the known conventions:
|
||||
- coupon-level (no sample token) → XRF (`L_081423_1`), macro friction (`data1..data3`)
|
||||
- sample-level (sample token `456/457`) → nanoindentation (`run1..run9`), friction test (`test_N`) / cycle (`cycleN` ambient, `_N` dry nitrogen)
|
||||
- atmosphere tokens (`ambient`, `dryNitrogen`, …)
|
||||
|
||||
Produce a token-extraction report and a list of filenames that **fail** to match any known pattern (candidates for new rules or anomalies).
|
||||
|
||||
**C. Structural profiling (header-only + sampled body).** Per file, without full load:
|
||||
- read header row → column names, count, order;
|
||||
- infer per-column datatype from first/last N rows + a random sample;
|
||||
- compute a **schema fingerprint** (normalized hash of the ordered column set) to group files of identical structure.
|
||||
|
||||
**D. Schema grouping.** Cluster files by (measurement type inferred from path/name) × (schema fingerprint). This surfaces the core Phase-2 problem: files of the *same purpose* with *different* column sets/naming.
|
||||
|
||||
### 3.3 Algorithm sketch (streaming, O(files), bounded memory)
|
||||
|
||||
```
|
||||
for each file in walk(root):
|
||||
meta = fs_metadata(file)
|
||||
tokens = tokenize_filename(file) # -> measurement type, ids, atmosphere
|
||||
header = read_first_line(file)
|
||||
columns = parse_columns(header)
|
||||
dtypes = infer_types(sample_rows(file, head=N, tail=N, random=M))
|
||||
fp = fingerprint(columns) # ordered-set hash
|
||||
emit_manifest_row(meta, tokens, columns, dtypes, fp)
|
||||
```
|
||||
|
||||
### 3.4 Artifacts
|
||||
- **`storage_manifest`** — machine-readable catalog (JSON lines + SQLite index) of every file with metadata, tokens, columns, dtypes, fingerprint.
|
||||
- **`schema_fingerprint_report`** — fingerprint → {file count, member files, canonical column set} per measurement type.
|
||||
- **`unmatched_report`** — files/filenames not matching known folder or naming rules.
|
||||
- **`storage_map`** — human-readable reconstruction of folder/file purpose (as-is vs as-designed).
|
||||
|
||||
### 3.5 Exit criteria
|
||||
- 100% of files inventoried and fingerprinted.
|
||||
- Every file assigned a (measurement type, attachment level) or flagged as unmatched.
|
||||
- Fingerprint clusters reviewed; the set of distinct schemas per measurement type is enumerated.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase 2 ↔ 3 — Cleansing/Normalization & Ontology (iterative)
|
||||
|
||||
**Goal.** Reconcile the enumerated raw schemas into the canonical relational model while building the ontology and Instruments Repository. Detect and quarantine anomalies.
|
||||
|
||||
The two phases run as a loop over each measurement type (XRF → nanoindentation → friction → macro friction), since each has its own schema and granularity.
|
||||
|
||||
### 4.1 The iteration loop (per measurement type)
|
||||
|
||||
1. **Draft canonical schema** from the target ERD (e.g., `friction_data_points`, `xrf_spectrum_points`).
|
||||
2. **Map raw columns → canonical columns.** Build a column-mapping table per fingerprint: raw name/position → canonical name, unit, dtype. Resolve synonyms, unit differences, column reordering.
|
||||
3. **Cleanse & normalize** on a streamed pass: apply mapping, coerce types, normalize units, standardize atmosphere/instrument codes, derive keys from filename tokens (FK resolution).
|
||||
4. **Detect anomalies** (see 4.3); route offending rows/files to quarantine with reason codes.
|
||||
5. **Feed findings back to the ontology** — new instrument, new column, new atmosphere value, unit variant, or an entity/attribute the ERD does not yet cover → update schema + data dictionary + Instruments Repository.
|
||||
6. **Repeat** until the mapping covers all fingerprints of that type with no unresolved columns.
|
||||
|
||||
### 4.2 Instruments Repository (ontology sub-model)
|
||||
|
||||
Built incrementally as instruments are identified from path/name + column signatures. Each instrument entity carries:
|
||||
- identity (type, model/label, measurement type it produces);
|
||||
- measured quantities, units, and **valid physical ranges** (min/max, resolution);
|
||||
- association to `measurement_types` / `column_definitions` in the data dictionary.
|
||||
|
||||
Instrument ranges become the source of "physically impossible value" criteria in anomaly detection — closing the loop between the repository and cleansing.
|
||||
|
||||
### 4.3 Anomaly detection
|
||||
|
||||
Criteria are **derived from the data**, not fixed a priori. Starting catalog:
|
||||
|
||||
| Class | Description | Source of criterion |
|
||||
|-------|-------------|---------------------|
|
||||
| Out-of-range | Value outside instrument's physical capability | Instruments Repository ranges |
|
||||
| Sequence gap | Missing steps in ordered series (time, cycle, run) | Series continuity check |
|
||||
| Structural mismatch | File of a purpose with unexpected column set | Fingerprint vs canonical |
|
||||
| Type violation | Non-coercible value in a typed column | Mapping/coercion |
|
||||
| Duplicate | Repeated rows / repeated identity keys | Key + row hashing |
|
||||
| Missing required | Null in a non-nullable canonical field | Schema constraints |
|
||||
| Encoding/format | Malformed rows, delimiter/decimal issues | Parser |
|
||||
|
||||
Anomalies are **quarantined, not dropped**: written to a rejects store with `(source_file, row_offset, reason_code, raw_payload)` for review and rule refinement.
|
||||
|
||||
### 4.4 Data dictionary / metadata catalog
|
||||
|
||||
Maintained in lockstep with the schema: `measurement_types` and `column_definitions` (column name, unit, data type, ordinal, description) — keeping heterogeneous raw series self-documenting, exactly as specified in the domain model.
|
||||
|
||||
### 4.5 Artifacts
|
||||
- **`column_mapping`** — per (measurement type, fingerprint): raw→canonical mapping with unit/type rules.
|
||||
- **`canonical_dataset`** — cleansed, normalized rows in the canonical intermediate (partitioned by measurement type; see §7).
|
||||
- **`relational_schema`** — evolving DDL (PostgreSQL dialect) matching the ERD, versioned.
|
||||
- **`data_dictionary`** — `measurement_types` + `column_definitions`.
|
||||
- **`instruments_repository`** — instrument entities + characteristics + ranges.
|
||||
- **`rejects_store`** — quarantined anomalies with reason codes.
|
||||
- **`ontology_notes`** — running record of ERD changes and rationale.
|
||||
|
||||
### 4.6 Exit criteria
|
||||
- Every fingerprint of every measurement type has a complete, unambiguous mapping.
|
||||
- Canonical dataset produced for all in-scope files; rejects quarantined and categorized.
|
||||
- Relational schema, data dictionary, and Instruments Repository mutually consistent and frozen for the release.
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 4 — Conversion & Export
|
||||
|
||||
**Goal.** Deterministically materialize the frozen canonical dataset into the required output formats. All formats derive from the same canonical intermediate, so they cannot diverge.
|
||||
|
||||
### 5.1 Targets
|
||||
1. **JSON mega-file** — single hierarchical document (data note → … → measurements → points). Suitable for small/whole-dataset transport; streamed writer for size.
|
||||
2. **Multiple JSON files** — split by natural boundary (per coupon / per sample / per measurement type). Directory layout mirrors the ontology hierarchy.
|
||||
3. **SQLite** — schema generated from the relational model; bulk-loaded. Portable single-file store (aligns with future better-sqlite3 / JSX Viewer use).
|
||||
4. **PostgreSQL** — system of record. DDL + `COPY`-based bulk load (parse tokens → FKs, insert headers, bulk-load numeric rows), per the ADR ingestion path.
|
||||
5. **RDF / triplestore** — **metadata & provenance graph only** (never the bulk numeric series), per ADR. Preferred: virtual RDF via Ontop (SPARQL→SQL over PostgreSQL). Optional materialized export via R2RML/RML to Jena/GraphDB. Vocabularies: PROV-O, QUDT, Dublin Core, DCAT, materials/tribology terms; mint stable IRIs.
|
||||
|
||||
### 5.2 Design
|
||||
- One **export driver** reads the canonical intermediate + schema and dispatches to per-format writers.
|
||||
- Each writer is streaming and idempotent; re-running reproduces byte-comparable output (modulo timestamps).
|
||||
- Lineage is emitted alongside every export (see §6).
|
||||
|
||||
### 5.3 Artifacts
|
||||
- Per-format output sets + a **build manifest** (what was produced, from which canonical snapshot, with which config/rule versions).
|
||||
|
||||
### 5.4 Exit criteria
|
||||
- All five target formats generated from one canonical snapshot.
|
||||
- Round-trip validation: record counts and key aggregates match across formats.
|
||||
- Lineage complete and resolvable for sampled records.
|
||||
|
||||
---
|
||||
|
||||
## 6. Cross-cutting concerns
|
||||
|
||||
### 6.1 Lineage & processing-flow documentation
|
||||
- Per target row: `source_file`, source row offset, `source_modified_at`, ordered list of transform rules applied.
|
||||
- Per pipeline run: config snapshot, rule/version set, input inventory hash, output manifest — a reproducible **processing-flow record**.
|
||||
- Lineage is a first-class export, present in every target format.
|
||||
|
||||
### 6.2 Configuration & rule registry
|
||||
- Filename-token patterns, column mappings, unit conversions, atmosphere/instrument code normalization, and anomaly thresholds live in **versioned config**, not code. This is what makes Phases 2↔3 iterative without rewrites.
|
||||
|
||||
### 6.3 Validation gates
|
||||
- Between phases: schema conformance, referential completeness (all FKs resolvable), reject-rate thresholds, fingerprint coverage = 100%.
|
||||
|
||||
---
|
||||
|
||||
## 7. Target artifacts catalog (consolidated)
|
||||
|
||||
| Artifact | Phase | Form |
|
||||
|----------|-------|------|
|
||||
| storage_manifest | 1 | JSONL + SQLite |
|
||||
| schema_fingerprint_report | 1 | JSON/CSV |
|
||||
| unmatched_report | 1 | CSV |
|
||||
| storage_map | 1 | Markdown |
|
||||
| column_mapping | 2↔3 | CSV/JSON (versioned) |
|
||||
| canonical_dataset | 2↔3 | partitioned intermediate |
|
||||
| relational_schema (DDL) | 2↔3 | SQL |
|
||||
| data_dictionary | 2↔3 | table/CSV |
|
||||
| instruments_repository | 2↔3 | table/CSV |
|
||||
| rejects_store | 2↔3 | JSONL |
|
||||
| ontology_notes | 2↔3 | Markdown |
|
||||
| exports (JSON/SQLite/PG/RDF) | 4 | per format |
|
||||
| lineage + build manifest | all | JSONL |
|
||||
|
||||
---
|
||||
|
||||
## 8. Tooling & environment
|
||||
|
||||
- **Language:** Python (streaming, stdlib-first; `csv`, `pathlib`, `hashlib`, `json`, `sqlite3`). Add `pyarrow`/Parquet only if the canonical intermediate needs columnar efficiency at 700 GB.
|
||||
- **No DB engine in Phases 1–2** beyond SQLite for the manifest index.
|
||||
- **PostgreSQL** materialized in Phase 4 (system of record).
|
||||
- **Ontop / R2RML** for the RDF surface (Phase 4, deferred activation).
|
||||
- **Future:** migration of browse/query surface to JSX Viewer + better-sqlite3; canonical SQLite export is the natural bridge.
|
||||
- Python indentation: **tabs**.
|
||||
|
||||
---
|
||||
|
||||
## 9. Performance strategy for 700 GB
|
||||
|
||||
- **Never full-load.** Phase 1 is header + sampled-row only; Phases 2/4 stream row-by-row.
|
||||
- **Bounded memory.** Constant-memory transforms; no whole-file DataFrames on large series.
|
||||
- **Partition the canonical intermediate** by measurement type and, for the high-volume series (`friction_data_points`), by a coarse key (e.g., test/cycle) — informed by volume estimates gathered in Phase 1.
|
||||
- **Parallelize by file** (embarrassingly parallel scan/transform); aggregate manifests after.
|
||||
- **Deterministic ordering** for reproducible, resumable runs (checkpoint by manifest offset).
|
||||
- Volume estimates from Phase 1 drive the `friction_data_points` partitioning decision before Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open items & risks
|
||||
|
||||
1. **`macro_friction_points` schema is provisional** (columns `normal_load_mn`, `average_coefficient_of_friction` pending an actual macro-friction file). Resolve during Phase 2 mapping of that type.
|
||||
2. **Unknown fingerprints.** True count of distinct raw schemas per type is unknown until Phase 1 completes; mapping effort in Phase 2 scales with it.
|
||||
3. **Anomaly criteria maturity.** Instrument ranges (and thus out-of-range detection) are only as complete as the Instruments Repository, which is itself being built — expect multiple loop iterations.
|
||||
4. **Atmosphere/instrument code drift.** Filename conventions vary (`cycleN` vs `_N`, atmosphere tokens); normalization rules must be discovered and versioned.
|
||||
5. **Lineage granularity vs. size.** Row-level lineage on billion-row series must be compact (offsets/rule-ids, not copied payloads) to avoid inflating storage.
|
||||
|
||||
---
|
||||
|
||||
*End of Plan v1.*
|
||||
File diff suppressed because one or more lines are too long
@@ -533,7 +533,7 @@ CREATE TABLE friction_cycles (
|
||||
<div class="foot">
|
||||
Measured 2026-07-11 on the deterministic simulated corpus (seed 20260711, 141.8 MiB CSV) · SQLite, warm OS cache, best of 3 runs ·
|
||||
scripts: <span class="mono">measure_key_variants.py</span> (build + size + load + Q1) and <span class="mono">read_benchmarks.py</span> (six read patterns) ·
|
||||
companion page: <span class="mono">Tribology_Ontology.html</span> (measurement-result registry).
|
||||
companion page: <a href="https://git.boskadoff.com/Public/LabDataStorageEvaluation/raw/branch/master/docs/research/Tribology_Storage_Evaluation.html" style="color:#1f5fdb;text-decoration:none">Storage Format Evaluation</a> (full study results).
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -78,8 +78,6 @@ an en-dash.
|
||||
content, not typographic substitutions. UTF-8 throughout is the right
|
||||
answer. (Repository artifacts are English; this matters only for quoted
|
||||
material.)
|
||||
- **Imported reference materials** in `docs/examples/` are kept verbatim as
|
||||
received; they are not binding documents.
|
||||
|
||||
## 4. Detection / enforcement
|
||||
|
||||
@@ -89,8 +87,7 @@ future detector is documented here so it can be built consistently:
|
||||
|
||||
- **What it flags.** Any occurrence of the Unicode codepoints in the section 1
|
||||
substitution table inside in-scope files. The section 3 exemptions
|
||||
(scientific notation, international content, `docs/examples/`) are the only
|
||||
carve-outs.
|
||||
(scientific notation, international content) are the only carve-outs.
|
||||
- **Exit behavior.** Non-zero exit when any flagged codepoint is found under
|
||||
the scan roots, so it can later gate a commit hook or a build step.
|
||||
- **Tooling.** When this detector is created it MUST be a PowerShell script
|
||||
|
||||
@@ -1,27 +0,0 @@
|
||||
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
|
||||
|
||||
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
|
||||
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
|
||||
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
|
||||
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
|
||||
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
|
||||
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
|
||||
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
|
||||
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
|
||||
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
|
||||
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
|
||||
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
|
||||
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
|
||||
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
|
||||
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
|
||||
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
|
||||
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
|
||||
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
|
||||
|
||||
## Final Goal
|
||||
|
||||
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.
|
||||
@@ -1,26 +0,0 @@
|
||||
# Task 01 — Setup Workspace
|
||||
|
||||
## Goal
|
||||
Create the working directory layout and versioned pipeline configuration.
|
||||
|
||||
## Input
|
||||
- Path to the raw CSV root (read-only). Ask the user if unknown.
|
||||
|
||||
## Actions
|
||||
1. Create:
|
||||
```
|
||||
work/
|
||||
config/ # pipeline.json, token_patterns.json, column_mappings/, anomaly_rules.json
|
||||
manifest/ # inventory, fingerprints, reports
|
||||
canonical/ # cleansed partitioned dataset
|
||||
rejects/ # quarantined anomalies
|
||||
schema/ # DDL, data dictionary, instruments repository
|
||||
exports/{json,sqlite,postgresql,rdf}/
|
||||
lineage/
|
||||
scripts/ # all Python scripts produced by tasks
|
||||
```
|
||||
2. Write `work/config/pipeline.json`: `SOURCE_ROOT`, sample sizes (head/tail/random rows), version stamp.
|
||||
3. Write empty `work/config/token_patterns.json` and `work/config/anomaly_rules.json` skeletons.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Directory tree exists; `pipeline.json` valid and points to an existing `SOURCE_ROOT`.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 02 — Filesystem Inventory
|
||||
|
||||
## Goal
|
||||
Non-destructive inventory of every file under `SOURCE_ROOT`.
|
||||
|
||||
## Input
|
||||
- `work/config/pipeline.json` (Task 01).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/fs_inventory.py` (streaming walk, constant memory).
|
||||
2. For each file capture: relative path, path components (folder tokens), filename, extension, size bytes, mtime (ISO), sha-free quick id (path hash).
|
||||
3. Emit `manifest/inventory.jsonl` (one JSON object per file) and load an index copy into `manifest/manifest.sqlite` table `inventory`.
|
||||
4. Emit `manifest/inventory_summary.md`: file count, total size, count by extension, count and size by top-level folder, depth histogram.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `inventory.jsonl` row count == filesystem file count.
|
||||
- `inventory_summary.md` produced; non-CSV extensions listed explicitly.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 03 — Filename & Path Tokenization
|
||||
|
||||
## Goal
|
||||
Parse every path/filename into structured tokens; classify each file by measurement type and attachment level.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl` (Task 02).
|
||||
- Known conventions: hierarchy `data_note → batch → coupon(die) → sample`; XRF `L_<MMDDYY>_<seq>`; macro friction `data1..data3` (+ atmosphere); nanoindentation `run1..run9`; friction `test_N` / `cycleN` (ambient) or `_N` (dryNitrogen); sample tokens like `456/457`.
|
||||
|
||||
## Actions
|
||||
1. Encode patterns as regex rules in `config/token_patterns.json` (versioned; extend as new patterns are found).
|
||||
2. Write `scripts/tokenize.py`: apply rules to each inventory row; extract `{batch, coupon, die, sample, measurement_type, attachment_level(coupon|sample), atmosphere, run/test/cycle/seq, lot, date}`.
|
||||
3. Emit `manifest/tokens.jsonl` and `manifest/unmatched.csv` (files matching no rule, with best-guess notes).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every inventory row is in `tokens.jsonl` with either a full classification or an `unmatched` flag.
|
||||
- `unmatched.csv` reviewed; genuinely new conventions added to `token_patterns.json` and the script re-run (repeat until unmatched are only true anomalies).
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 04 — Schema Profiling & Fingerprints
|
||||
|
||||
## Goal
|
||||
Capture the record structure of every CSV without full load; compute schema fingerprints.
|
||||
|
||||
## Input
|
||||
- `manifest/inventory.jsonl`, `manifest/tokens.jsonl`; sample sizes from `pipeline.json`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/profile_schema.py` (streaming; head N + tail N + random M rows only).
|
||||
2. Per file capture: delimiter, header presence, column names (raw, ordered), column count, inferred dtype per column, null-rate in sample, min/max of numeric columns in sample, total row estimate (size/avg-row-len).
|
||||
3. Fingerprint = hash of normalized ordered column-name list (lowercased, trimmed). Also compute a name-insensitive positional fingerprint (dtype sequence) for headerless files.
|
||||
4. Emit `manifest/schema_profiles.jsonl`; add table `schema_profiles` to `manifest.sqlite`.
|
||||
5. Log unreadable/malformed files to `manifest/profile_errors.csv`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- 100% of CSV files have a profile row or an entry in `profile_errors.csv`.
|
||||
- Every profile carries a fingerprint.
|
||||
@@ -1,16 +0,0 @@
|
||||
# Task 05 — Schema Grouping & Inconsistency Report
|
||||
|
||||
## Goal
|
||||
Enumerate distinct record structures per measurement type; expose structural inconsistencies.
|
||||
|
||||
## Input
|
||||
- `manifest/tokens.jsonl`, `manifest/schema_profiles.jsonl`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/group_schemas.py`: group files by `measurement_type × fingerprint`.
|
||||
2. Emit `manifest/fingerprint_report.md` per measurement type: list of distinct fingerprints, file count each, canonical column set each, column-name diffs between fingerprints (added/removed/renamed candidates), dtype conflicts.
|
||||
3. Emit `manifest/inconsistencies.csv`: one row per detected inconsistency `{measurement_type, kind(column_set|naming|dtype|placement), fingerprints involved, example files}`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- The number of distinct schemas per measurement type is known and documented.
|
||||
- `inconsistencies.csv` complete — this is the work queue for Task 07.
|
||||
@@ -1,19 +0,0 @@
|
||||
# Task 06 — Storage Map (As-Is / As-Designed)
|
||||
|
||||
## Goal
|
||||
Human-readable reconstruction of the storage logic: purpose of each folder level and file kind.
|
||||
|
||||
## Input
|
||||
- All Task 02–05 artifacts.
|
||||
|
||||
## Actions
|
||||
1. Write `manifest/storage_map.md`:
|
||||
- annotated folder-tree template (one exemplar branch per pattern, not all 700 GB);
|
||||
- purpose of each nesting level mapped to entities (`data_note → batch → coupon/die → sample`);
|
||||
- purpose of each file kind mapped to measurement types and attachment levels;
|
||||
- deviations: folders/files violating the intended structure (from `unmatched.csv`, `inconsistencies.csv`).
|
||||
2. Mark reconstructed intent vs observed fact explicitly (`AS-DESIGNED` / `AS-IS` labels).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `storage_map.md` explains every folder level and file kind; deviations enumerated with examples.
|
||||
- Phase 1 complete. Gate: fingerprint coverage 100%, unmatched files are catalogued anomalies only.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 07 — Column Mapping (raw → canonical)
|
||||
|
||||
## Goal
|
||||
For every `measurement_type × fingerprint`, a complete mapping of raw columns to canonical columns.
|
||||
|
||||
## Input
|
||||
- `manifest/fingerprint_report.md`, `manifest/inconsistencies.csv`.
|
||||
- Target canonical schemas (ERD): `xrf_spectrum_points`, `nanoindentation_points`, `friction_data_points`, `macro_friction_points`.
|
||||
|
||||
## Actions
|
||||
1. For each fingerprint create `config/column_mappings/<type>__<fingerprint>.json`: `{raw_name|position → canonical_name, unit_in, unit_out, dtype, transform(optional)}`.
|
||||
2. Resolve synonyms, reordering, unit variants; record every decision inline as `"note"`.
|
||||
3. Columns with no canonical target → escalate to Task 08 (ontology change) instead of dropping silently.
|
||||
4. `macro_friction_points` schema is PROVISIONAL — finalize from an actual macro-friction file profile here; propagate to Task 08.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every fingerprint has a mapping file; no raw column is unmapped or undocumented.
|
||||
- Escalation list for Task 08 written to `schema/ontology_escalations.md`.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 08 — Ontology: Relational Schema, Data Dictionary, Instruments Repository
|
||||
|
||||
## Goal
|
||||
Formal, versioned data model driving cleansing and all exports. Iterates with Tasks 07/09/10.
|
||||
|
||||
## Input
|
||||
- ERD (project artifact `Mary_G__LabResults__ERD.txt`), `schema/ontology_escalations.md`.
|
||||
|
||||
## Actions
|
||||
1. Generate `schema/ddl_postgresql.sql` from the ERD: full hierarchy `ptau_data_notes → batches → coupons → samples`, four measurement branches, `atmospheres` lookup, `measurement_types` + `column_definitions`.
|
||||
2. Generate `schema/data_dictionary.csv`: per measurement type — column name, unit, dtype, ordinal, description (source: the mappings of Task 07).
|
||||
3. Create `schema/instruments_repository.csv`: instrument id, type, measurement type produced, measured quantities, units, valid min/max ranges, resolution. Populate incrementally from tokens + column signatures; ranges feed Task 09 anomaly rules.
|
||||
4. Maintain `schema/ontology_notes.md`: every schema change with rationale (append-only).
|
||||
5. Version the DDL (`-- schema_version: N` header); bump on each iteration.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- DDL, data dictionary, and Instruments Repository are mutually consistent and cover all mapped fingerprints.
|
||||
- No open items in `ontology_escalations.md`.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 09 — Cleanse & Normalize → Canonical Dataset
|
||||
|
||||
## Goal
|
||||
Streamed transformation of all raw CSV into the canonical dataset; anomalies quarantined, lineage recorded.
|
||||
|
||||
## Input
|
||||
- Mappings (Task 07), schema + instrument ranges (Task 08), `config/anomaly_rules.json`, tokens (Task 03).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/cleanse.py` — streaming, constant memory, parallel by file, resumable (checkpoint by inventory offset).
|
||||
2. Per file: apply column mapping → coerce dtypes → normalize units → standardize codes (atmosphere, instrument) → resolve hierarchy keys from tokens (natural keys: batch/coupon/sample/run/test/cycle numbers).
|
||||
3. Anomaly checks per row: out-of-instrument-range, non-coercible type, missing required, duplicate key, sequence gap (per ordered series), malformed row. Reasons coded per `anomaly_rules.json`.
|
||||
4. Clean rows → `canonical/<measurement_type>/…` partitioned files (JSONL or Parquet; partition friction points by test/cycle per Phase-1 volume estimates). Rejected rows → `rejects/<type>.jsonl` with `{source_file, row_offset, reason_code, raw_payload}`.
|
||||
5. Lineage per output row: `{source_file_id, row_offset, mapping_version, schema_version}` → compact companion files in `lineage/`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Every in-scope file processed; row counts reconcile: `source ≈ canonical + rejects` per file.
|
||||
- Canonical partitions written; lineage present for every canonical row.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 10 — Anomaly Report & Iteration Gate
|
||||
|
||||
## Goal
|
||||
Consolidate anomalies, refine rules/mappings/ontology, and gate the Phase 2↔3 loop.
|
||||
|
||||
## Input
|
||||
- `rejects/*.jsonl`, `manifest/unmatched.csv`, current rule/mapping/schema versions.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/anomaly_report.py` → `manifest/anomaly_report.md`: counts by reason code × measurement type, top offending files, examples per class, reject-rate per fingerprint.
|
||||
2. Triage each anomaly class: (a) true data anomaly — keep quarantined, document; (b) rule defect — fix `anomaly_rules.json` / mapping / token pattern; (c) ontology gap — escalate to Task 08.
|
||||
3. If (b) or (c) occurred: bump versions and re-run Tasks 07→09 for affected types.
|
||||
4. Repeat until: fingerprint mapping coverage 100%, reject classes are all category (a), reject rate stable between runs.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `anomaly_report.md` final; every reject class documented as a true anomaly.
|
||||
- Canonical dataset frozen: `canonical/SNAPSHOT.md` written with versions + row counts. Phases 2↔3 complete.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 11 — Export: JSON
|
||||
|
||||
## Goal
|
||||
Two JSON products from the frozen canonical snapshot.
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/` (Tasks 09–10).
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/export_json.py` (streaming writer; never build the whole document in memory).
|
||||
2. **Mega-file** `exports/json/dataset.json`: full hierarchy `data_note → batches → coupons(die) → samples → measurements → points`, plus embedded `data_dictionary` and `instruments`.
|
||||
3. **Split set** `exports/json/split/`: directory mirrors the hierarchy; one file per measurement (points inline), plus `index.json` per level.
|
||||
4. Each measurement object carries its lineage block `{source_file, mapping_version, schema_version}`.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Both products written; spot-validation: point counts per measurement match canonical.
|
||||
- `exports/json/EXPORT_MANIFEST.json` written (snapshot id, versions, counts).
|
||||
@@ -1,16 +0,0 @@
|
||||
# Task 12 — Export: SQLite
|
||||
|
||||
## Goal
|
||||
Portable single-file SQLite database conforming to the ontology.
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/export_sqlite.py`: translate DDL to SQLite dialect (bigint→INTEGER, timestamptz→TEXT ISO, keep FKs), create `exports/sqlite/labdata.sqlite`.
|
||||
2. Bulk-load in dependency order (hierarchy → lookups → headers → points); `executemany` in transactions; create indexes after load; `PRAGMA journal_mode=OFF, synchronous=OFF` during load, `ANALYZE` after.
|
||||
3. Load `measurement_types`, `column_definitions`, `instruments` tables; add `lineage` table (row-range level, not per-row, to bound size).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `labdata.sqlite` opens; per-table row counts equal canonical counts; FK integrity check passes (`PRAGMA foreign_key_check` empty).
|
||||
- `exports/sqlite/EXPORT_MANIFEST.json` written.
|
||||
@@ -1,22 +0,0 @@
|
||||
# Task 13 — Export: PostgreSQL
|
||||
|
||||
## Goal
|
||||
System-of-record load package: DDL + COPY-based bulk load (per ADR ingestion path).
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot, `schema/ddl_postgresql.sql`.
|
||||
|
||||
## Actions
|
||||
1. Generate `exports/postgresql/`:
|
||||
- `01_schema.sql` (DDL, versioned);
|
||||
- `02_lookups.sql` (atmospheres, measurement_types, column_definitions, instruments);
|
||||
- per-table CSV staging files from canonical (streamed);
|
||||
- `03_copy_load.sql` (`\copy` statements in dependency order);
|
||||
- `04_post_load.sql` (indexes, constraints validation, ANALYZE);
|
||||
- `load.sh` / `load.ps1` orchestration.
|
||||
2. Point tables loaded via COPY; headers via INSERT; partitioning DDL for `friction_data_points` per Phase-1 volume estimates.
|
||||
3. Include `lineage` table load (row-range granularity).
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- Package executes cleanly on an empty database (test locally if a PG instance is available; otherwise dry-validate SQL syntax).
|
||||
- Post-load counts script confirms parity with canonical. `EXPORT_MANIFEST.json` written.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Task 14 — Export: RDF (Metadata & Provenance Graph)
|
||||
|
||||
## Goal
|
||||
RDF surface for the metadata/provenance layer only — never the numeric series (per ADR).
|
||||
|
||||
## Input
|
||||
- `canonical/` snapshot headers + hierarchy, `schema/`, lineage.
|
||||
|
||||
## Actions
|
||||
1. Mint stable IRIs for every data note, batch, coupon, sample, measurement (pattern `https://tribology.ufl.example/id/<entity>/<key>`; base IRI in config).
|
||||
2. Write `scripts/export_rdf.py` → `exports/rdf/metadata.ttl` (Turtle): hierarchy, measurement headers, atmosphere, instrument, source-file provenance. Vocabularies: PROV-O (provenance), QUDT (units), Dublin Core (citation), DCAT (dataset catalog), local `lab:` terms.
|
||||
3. Emit `exports/rdf/mappings.r2rml.ttl` — R2RML mappings from the PostgreSQL schema, ready for Ontop virtual deployment.
|
||||
4. Numeric series: reference-only — each measurement IRI carries a `lab:dataService` pointer, no point triples.
|
||||
|
||||
## Intermediate Result (done when)
|
||||
- `metadata.ttl` parses (validate with `rdflib`); triple counts reported per class.
|
||||
- R2RML mappings cover all metadata tables. `EXPORT_MANIFEST.json` written.
|
||||
@@ -1,18 +0,0 @@
|
||||
# Task 15 — Lineage Consolidation, Build Manifest, Final Validation
|
||||
|
||||
## Goal
|
||||
Close the pipeline: consolidated lineage, reproducible build record, cross-format validation.
|
||||
|
||||
## Input
|
||||
- All `exports/*`, `lineage/`, all config/schema versions.
|
||||
|
||||
## Actions
|
||||
1. Write `scripts/finalize.py`:
|
||||
- consolidate `lineage/` into `lineage/lineage.sqlite` (queryable: target row-range → source file + offsets + rule versions);
|
||||
- produce `BUILD_MANIFEST.md`: snapshot id, all config/mapping/schema versions, input inventory hash, per-artifact counts, run timestamps — the processing-flow documentation.
|
||||
2. Cross-format validation: row counts and key aggregates (e.g., sum of cycles per test, point counts per measurement) identical across canonical, JSON, SQLite, PostgreSQL staging.
|
||||
3. Lineage spot-check: resolve 20 random target rows back to source file + row offset; verify raw values match.
|
||||
|
||||
## Final Result (pipeline done when)
|
||||
- `BUILD_MANIFEST.md` complete; all validations pass; lineage resolvable.
|
||||
- **Final goal achieved:** cleansed canonical dataset conforming to the ontology, all anomalies quarantined and explained, four export formats generated from one snapshot, full source→target lineage, entire pipeline reproducible from source + versioned config.
|
||||
Reference in New Issue
Block a user