Public/LabDataStorageEvaluation
1
0
Fork 0
Code Issues Pull Requests Actions Releases Wiki Activity

fix(docs): fix formatting issues in README.md

Browse Source 
- adapt meta-/code-/obs- rules to the local Python benchmark pipeline
- replace db-sql-ddl, code-config-env-scope, test-e2e-pytest with pipeline equivalents
- add code-python-style, data-determinism, data-naming-units, bench-methodology, build-pipeline-tasks
- normalize specs and README typography to ASCII per code-data-formatting
- rewrite root CLAUDE.md trigger table; add .cursorrules and .gitignore
- specs: pipeline plan and task specs 00-11
- rules: 19 binding rule files adapted for this project
- docs: CLAUDE.md rule-trigger table
- config: .cursorrules commit convention, .gitignore excluding out/ and .venv/
- docs: rewrite README.md with pipeline diagrams, setup guide, result placeholders
- config: requirements.txt for the closed dependency list
- datagen: make_lab_config.py writes out/config/lab_config.yaml and .done marker
This commit is contained in:
administrator
2026-07-11 13:39:13 -04:00
parent d8353953b0
commit b173ac82a9
55 changed files with 3755 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
docs/specs_old/00_PLAN.md Normal file
View File

@@ -0,0 +1,27 @@
# Data Processing Plan — Tribology Lab Data (LDRD Mechanochemical Alloys FY25-27)
Executor: Cursor AI agent with filesystem access to the raw CSV tree (read-only) and a writable `work/` directory.
Source tree: `SOURCE_ROOT` (set in `config/pipeline.json`, Task 01). All outputs under `work/`.
Rules: source is never mutated; every task is re-runnable; Python, tabs for indentation.
## Tasks
1. **01_setup_workspace.md** — create `work/` layout + pipeline config.
2. **02_fs_inventory.md** — walk source tree; produce file inventory.
3. **03_filename_tokenization.md** — parse filenames into tokens; flag unmatched.
4. **04_schema_profiling.md** — read headers + sampled rows; infer dtypes; compute schema fingerprints.
5. **05_schema_grouping.md** — cluster files by (measurement type × fingerprint); report structural inconsistencies.
6. **06_storage_map.md** — human-readable as-is / as-designed storage map.
7. **07_column_mapping.md** — per fingerprint: raw→canonical column mapping (ERD-aligned).
8. **08_ontology_schema.md** — relational DDL + data dictionary + Instruments Repository (iterates with 07/09).
9. **09_cleanse_normalize.md** — streamed cleanse → canonical dataset; quarantine anomalies with reason codes.
10. **10_anomaly_report.md** — consolidate rejects; refine rules; loop 07–09 until coverage = 100%.
11. **11_export_json.md** — JSON mega-file + per-entity JSON set from canonical.
12. **12_export_sqlite.md** — SQLite database per DDL, bulk-loaded.
13. **13_export_postgresql.md** — PostgreSQL DDL + COPY load scripts.
14. **14_export_rdf.md** — RDF metadata/provenance graph (R2RML mappings; no numeric series).
15. **15_lineage_build_manifest.md** — lineage consolidation + final build manifest; end-to-end validation.
## Final Goal
From ~700 GB of raw heterogeneous CSV: a single cleansed **canonical dataset** conforming to the ontology (ERD + data dictionary + Instruments Repository), with all anomalies quarantined and explained, materialized into **JSON, SQLite, PostgreSQL, and RDF** exports, each carrying complete **source→target lineage**, reproducible from source + versioned config in one pipeline run.