Public/LabDataStorageEvaluation
1
0
Fork 0
Code Issues Pull Requests Actions Releases Wiki Activity

feat(convert): integrate process metrics into JSON and SQLite converters

Browse Source 
- add process_metrics function to record peak RSS and CPU time
- update JSON converter to include metrics in completion marker
- modify SQLite converter to utilize new metrics for performance tracking
- enhance storage size updates with locking mechanism for concurrent access
This commit is contained in:
administrator
2026-07-11 21:39:08 -04:00
parent 96ed7bc918
commit 48c102d2a4
17 changed files with 1928 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@@ -62,9 +62,9 @@ machine (no parallel work). Every task writes a completion marker
| 03 CSV data generation | `generate_data.py` | `out/csv/` tree + `MANIFEST.csv` | implemented |
| 04 Convert to JSON-LD | `convert_json.py` | `out/json/full/`, `out/json/hybrid/dataset.jsonld` | implemented |
| 05 Convert to SQLite | `convert_sqlite.py` | `out/sqlite/tribo.db` | implemented |
| 06 Convert to PostgreSQL | `convert_postgresql.py` | live `tribo` DB + `out/pg/tribo.dump` | planned |
| 07 Convert to RDF | `convert_rdf.py` | `out/rdf/dataset.nt.gz`, oxigraph store | planned |
| 08 Benchmark queries | `make_queries.py` | `out/bench/queries/`, `out/bench/expected/` | planned |
| 06 Convert to PostgreSQL | `convert_postgresql.py` | live PostgreSQL DB (+ dump when pg_dump is available) | implemented |
| 07 Convert to RDF | `convert_rdf.py` | `out/rdf/dataset.nt.gz`, `dataset.ttl`, oxigraph store | implemented |
| 08 Benchmark queries | `make_queries.py` | `out/bench/queries/`, `out/bench/expected/` | implemented |
| 09 Benchmark execution | `bench_runner.py` | `out/bench/results_raw.csv`, `results_median.csv` | planned |
| 10 Extrapolation | `extrapolate.py` | `out/bench/extrapolation.csv`, `hardware_sizing.csv` | planned |
| 11 Reporting | `report.py` | `out/report/REPORT.md`, charts, tables | planned |
@@ -147,11 +147,15 @@ docs/
out/ ALL generated artifacts (git-ignored, reproducible):
config/ csv/ json/ sqlite/ pg/ rdf/ bench/ report/ .done/
common/ shared helpers (storage_sizes.csv contract, ...)
queries/ Q1-Q7 implementations for csv / json / rdf formats
make_lab_config.py task 01 entry script (one script per task, repo root)
make_process_flow.py task 02 entry script
generate_data.py task 03 entry script
convert_json.py task 04 entry script
convert_sqlite.py task 05 entry script
convert_postgresql.py task 06 entry script (TRIBO_PG_DSN, TRIBO_PROM_URL)
convert_rdf.py task 07 entry script
make_queries.py task 08 entry script (canonical results need TRIBO_PG_DSN)
requirements.txt closed dependency list (docs/rules/code-python-style.md)
```