LabDataStorageEvaluation/docs/rules/code-data-formatting.md

Code / Data / Formatting - ASCII-only typographic characters in source files

Binding scope. Strings that end up in any of the following MUST use ASCII typographic characters; never their Unicode "smart" equivalents:

• Python source, SQL and SPARQL query files
• Generated data and artifacts written to disk (CSV, JSON-LD, N-Triples, Turtle, MANIFEST and benchmark result CSVs, log messages)
• Configuration files (.yaml, .json, .toml)
• PowerShell / shell scripts (any file processed by Windows tooling)
• Markdown documentation (docs/, README.md, CLAUDE.md) - the specs were normalized to ASCII typography on 2026-07-11 and stay that way

Long dashes are an absolute exception, no carve-outs. Em-dash (U+2014) and en-dash (U+2013) MUST be replaced with ASCII hyphen-minus (-) in every file in this repository, including markdown documentation, code comments, and report prose. Reason: long dashes are visually indistinguishable from a hyphen in many monospace fonts, break diffs / grep / IDE search, and round-trip badly through Windows tooling chains (section 2).

---

1. The substitution table

Unicode char	Codepoint	Name	ASCII replacement
em-dash	U+2014	em-dash	- (single hyphen-minus)
en-dash	U+2013	en-dash	- (single hyphen-minus)
horizontal bar	U+2015	horizontal bar	-
minus sign	U+2212	minus sign	-
ellipsis	U+2026	horizontal ellipsis	... (three ASCII dots)
left double quote	U+201C	left double quotation mark	"
right double quote	U+201D	right double quotation mark	"
left single quote	U+2018	left single quotation mark	'
right single quote	U+2019	right single quotation mark	'
left guillemet	U+00AB	left guillemet	"
right guillemet	U+00BB	right guillemet	"
nbsp	U+00A0	non-breaking space	regular space

Single hyphen, not double. When replacing an em-dash that separates two clauses, use a single - surrounded by spaces (text - clause), NOT --.

Ranges use a single hyphen: Q1-Q7, 120-200 MB, 0.3-1.1 um - never an en-dash.

2. Why - documented failure modes

1. PowerShell 5.1 reads no-BOM files as Windows-1252. Non-ASCII UTF-8 bytes become mojibake and break string-literal parsing. This project is developed on Windows; any script or config touched by PowerShell is exposed.

2. Windows -> git-bash -> tar / packaging round-trip. At any step an unset locale or wrong encoding assumption produces mojibake. A UTF-8 byte sequence for U+2014 interpreted under a non-UTF-8 locale renders as a ? replacement character in the packaged output.

3. Copy-paste from Word / Google Docs / Notion / web articles. These tools auto-substitute typography (" -> smart quotes, ' -> smart apostrophe, -- -> em-dash, ... -> ellipsis) silently. The substitution survives copy into any editor and lands in source files unnoticed. The original specs arrived with en/em dashes this way.

4. Data files must diff and grep cleanly. The CSV corpus, MANIFEST, and benchmark results are compared by checksum and processed by streaming parsers; a stray typographic character breaks byte-level reproducibility.

3. Where the rule does NOT apply

• Scientific and unit notation is content, not typography. Characters such as µ (micro), ±, °C, ×, Ø, and Greek letters (τ, σ) in formulas inside the specs are meaningful scientific notation and are kept. Note the boundary: in column names and code identifiers the units are always plain ASCII (_um, _nm, _GPa - see data-naming-units.md); the scientific characters live only in explanatory prose.
• International language strings (Cyrillic, CJK, etc.) are meaningful content, not typographic substitutions. UTF-8 throughout is the right answer. (Repository artifacts are English; this matters only for quoted material.)

4. Detection / enforcement

There is no automated check for this rule yet - not in CI yet. The rule is enforced by review and by the editing discipline in this file. The intent of a future detector is documented here so it can be built consistently:

• What it flags. Any occurrence of the Unicode codepoints in the section 1 substitution table inside in-scope files. The section 3 exemptions (scientific notation, international content) are the only carve-outs.
• Exit behavior. Non-zero exit when any flagged codepoint is found under the scan roots, so it can later gate a commit hook or a build step.
• Tooling. When this detector is created it MUST be a PowerShell script (this is a Windows project) and live under tools/.

5. Examples

Good

# safe through every Windows tooling step
print("Corpus size 152.3 MB - within the 120-200 MB gate.")
label = "steady-state COF, cycles 500-1000"

Bad

# em-dash literal (U+2014) - mojibake risk on PowerShell read/write
print("Corpus size 152.3 MB [U+2014] within the gate.")

# en-dash range (U+2013) - breaks grep for "Q1-Q7"
label = "queries Q1[U+2013]Q7"

6. Anti-patterns

• Copy-pasting strings from a word processor or web page into source / specs / config without normalization.
• Trusting tar / packaging to "preserve as-is". They preserve bytes; the encoding interpretation is what gets lost when locale or encoding declaration is wrong on either end.
• Assuming the source file encoding alone is enough. The source file may be fine - the failure is at downstream consumers (PowerShell read, build write, OS locale).
• Using -- instead of - "for emphasis". Single ASCII hyphen surrounded by spaces is the project's chosen separator.
• Putting µ or ° into a column name or identifier - units in names are ASCII (_um, _C); see data-naming-units.md.