Testing & Quality

Orbitron uses unit tests, integration tests, corpus-backed smoke tests, and a small set of benchmarks. Keep tests fast and deterministic by default, and make large-corpus coverage opt-in.

11.1 What Testing Does

  • Catches regressions in parsing, rendering preparation, selection, editing, and analysis.
  • Ensures file loaders handle malformed inputs without panics.
  • Keeps refactors safe by locking down expected behavior at API boundaries.

11.2 Test Types & Locations

  • Unit tests: alongside code (#[cfg(test)]) or crate tests.
  • Integration tests:
    • IO workflows: io/pipelines/tests/
    • CLI workflows: automation/cli/tests/
  • Smoke tests: load many fixtures and validate “does not crash” behavior (typically under io/pipelines/tests/).
  • Malformed input tests: validate error surfaces (typically under io/pipelines/tests/).
  • Canonical fingerprint goldens: io/pipelines/tests/canonical_golden_formats.rs and nwchem_canonical_golden.rs pin a path-free fingerprint of every committed fixture’s canonical document against io/pipelines/tests/golden/*_canonical_fingerprint.txt, so any change to a parse trips them. Regenerate after an intentional change with UPDATE_GOLDEN=1 on that test target, and read the diff: a golden pins whatever the parser produced, bugs included.
  • CLI tests launch the binary through common::orbitron() (automation/cli/tests/support/common.rs), which points ORBITRON_CONFIG at a path nothing creates so every run gets the built-in defaults rather than your own config.toml. Never spawn it with Command::new(cargo_bin!("orbitron")): automation/cli/tests/config_isolation.rs scans the test sources and fails on a direct call site.
  • Render-parity tests: tests/render-parity/ — a headless golden-image harness that renders a set of molecular scenes on the GPU offscreen path and pixel-delta diffs them against committed baselines, catching visual regressions in the renderer. A pixel counts as differing when any RGBA channel exceeds the channel tolerance, and the run passes when the differing-pixel fraction stays within tolerance. Baselines regenerate with ORBITRON_REGEN_BASELINES=1. Skips automatically on machines without a usable GPU adapter — except in the dedicated CI job, which runs on Mesa lavapipe with ORBITRON_RENDER_PARITY_REQUIRE_GPU=1 so a missing adapter fails the job instead of silently skipping it.

11.3 Running Tests

cargo nextest run is the canonical runner (it is what [workspace.metadata.toolbox.commands] in the root Cargo.toml declares, and what CI uses for the workspace suite). CI still reaches for plain cargo test in four places nextest cannot cover: doctests, the GUI and TUI smoke tests, and the render-parity golden. All commands run inside the conda environment.

# Workspace unit + integration tests
CARGO_BUILD_JOBS=16 conda run -n orbitron-dev cargo nextest run --workspace

# Single crate
conda run -n orbitron-dev cargo nextest run -p orbitron-io-pipelines

# One test by name
conda run -n orbitron-dev cargo nextest run test_parse_xyz_atom_count

# Show println!/logs
conda run -n orbitron-dev cargo nextest run --no-capture

# Doctests — nextest runs none of them, so this is a separate invocation
conda run -n orbitron-dev cargo test --workspace --doc

CARGO_BUILD_JOBS caps rustc during the build; concurrency during the run comes from .config/nextest.toml:

  • [profile.default] test-threads = 16. Nextest otherwise starts one test process per logical CPU. On macOS, also keep target/debug/deps small: Cargo never prunes it, and once it is large every newly linked binary makes Gatekeeper’s syspolicyd walk that directory, which can stall the whole desktop session. Check it with ls -f target/debug/deps | wc -l (about 300 after a clean; five figures is a problem) and cargo clean when it is large. The thread cap is insurance, not the mechanism.
  • [test-groups] gpu = { max-threads = 1 } — every test that creates a wgpu device shares one slot, because on macOS that device is the same GPU WindowServer composites with. The membership is the list of test-group = 'gpu' overrides in .config/nextest.toml; read it there rather than from a copy here. It includes all of orbitron-render-parity (which additionally takes threads-required = 16, so the heaviest test gets the whole runner) and orbitron-ui-shell’s a_blank_session_opens_and_presents_a_frame (ui/shell/tests/viewer_starts.rs), the desktop’s own start-up smoke test. orbitron never links the viewer any more (the gui feature and the two CLI GUI smokes this replaced, view_blank_session_autocloses and view_keeps_cli_prepared_scene_instead_of_reloading_source, were removed with it), so this is the one surviving start-up smoke, now owned by the crate that builds the binary it launches. A new test that builds a renderer has to be added to the gpu group.
  • [profile.ci] test-threads = "num-cpus" — CI runners are headless with far fewer cores, where a fixed 16 would oversubscribe.

With logging:

RUST_LOG=debug conda run -n orbitron-dev cargo nextest run --no-capture
RUST_LOG=orbitron=debug conda run -n orbitron-dev cargo nextest run

Quality gates, including doctests that nextest does not run:

conda run -n orbitron-dev cargo fmt --all
conda run -n orbitron-dev ./scripts/check-rust-lints.sh
CARGO_BUILD_JOBS=16 conda run -n orbitron-dev cargo nextest run --workspace
conda run -n orbitron-dev cargo test --workspace --doc

On a macOS workstation where high build-artifact churn makes Spotlight or syspolicyd destabilize the desktop session, run a Cargo phase through the cleanup wrapper:

./scripts/cargo-campaign.sh nextest run --workspace

The wrapper creates its target and canonical cache under a Spotlight-excluded directory in /private/tmp, caps compilation at 12 jobs by default, and removes the whole campaign directory on success, failure, or an interrupt. Each invocation starts from a cold target, so keep related tests in one Cargo command when practical. Ordinary developer and CI runs can continue using the standard commands above.

Optional smoke tests:

  • Packaged embedding gates
    • Run ./scripts/dev-smoke.sh from the repository root after changing the Python bridge, staged WASM assets, widget frontend, web export, or local viewer server. It builds the production chain and checks package and server contracts.
    • Run conda run -n orbitron-dev python scripts/dev-smoke-browser.py when a change can affect browser startup, scene transfer, rendering, multiple viewers, trajectories, density, or retry behavior. The browser gate opens the production bundle in Playwright and requires painted pixels rather than accepting a page that merely loaded.
    • Install the matching browser once with conda run -n orbitron-dev python -m playwright install chromium.
    • On macOS, DYLD_LIBRARY_PATH must remain unset for the Playwright launch. The repository setup script configures the conda environment that way.
  • Wasm viewer smoke test
    • Generate ~/scratch/orbitron-wasm/scene.bin with the release-CLI steps in Extensions.
    • Build + serve the wasm dist/ output (scene and wasm on same origin):
      • cd viewer/wasm && conda run -n orbitron-dev trunk build --release
      • cp ~/scratch/orbitron-wasm/scene.bin dist/scene.bin
      • cd dist && conda run -n orbitron-dev python -m http.server 8081 --bind 127.0.0.1
    • Then run: WASM_VIEWER_URL="http://127.0.0.1:8081/?scene=/scene.bin" npm run test:smoke
    • One-command alternative (from viewer/wasm): npm run test:smoke:dist (scripts/run_smoke_dist.sh builds, serves dist/ on 8081 and tears the server down).
    • First-time Playwright setup may require npx playwright install.
  • Wasm viewer release smoke test — npm run test:smoke:release, from viewer/wasm. This is the only suite that builds and exercises the release artifacts. test:smoke:dist runs trunk build, a debug wasm of about 21 MB; a release is trunk build --release, about 6 MB, assembled into the ZIP bundle and the single-file standalone by scripts/build-wasm-viewer.sh. The difference is not cosmetic — a double-init() race in index.html threw closure invoked recursively or after being dropped on the release build and not on the debug one, so the dist suite passed while the shipped page threw on every load, and the standalone shipped dead for months behind a build that exited 0 and wrote a plausible 8 MB file. test:smoke:release builds the real artifacts, serves them, and opens both. Needs viewer/wasm/ts/node_modules (for esbuild) as well as trunk.
  • Published-site smoke test — npm run test:smoke:site, from viewer/wasm. Serves docs/ as published and opens a committed bundle through the committed viewer, because docs/viewer/ carries its own tracked runtime and bundles that can drift apart. It builds nothing on purpose. Needs a rendered docs/.
  • CLI page smoke test — npm run test:smoke:cli-page, from viewer/wasm. Writes pages with target/release/orbitron web and opens them over file://, asserting atom counts, mode counts and a settled scene. Stage the viewer assets, then build the release CLI. It is the only check that a page the CLI writes actually loads.
Import test from tests/fixtures/harness.cjs, not from @playwright/test

The harness is Playwright’s test plus an auto fixture that fails a test when the page raised an uncaught error. Fifteen tests drove a page that threw on every load and all fifteen passed, because a thrown module script or a rejected init is not a console error and breaks no assertion about the DOM. A test that means to provoke an error requests the pageErrors fixture and calls .allow(pattern); the pattern is required, since allowing everything would put the hole back.

Note the limit: this catches deterministic throws, not races. The double-init() bug above reproduces on the Linux CI runner and not under Playwright on macOS.

Two front ends share one wasm module, and a bug fixed in one is not fixed in the other: the standalone demo page (index.html + app.js, hand-written) and the <orbitron-viewer> custom element (ts/src/, bundled to orbitron-viewer.js). Both app.js and orbitron-viewer.js are committed build artifacts — regenerate the element bundle with npm run build in viewer/wasm/ts after touching ts/src, or the change ships only in source. Camera and input arithmetic belongs in Rust (src/camera_math.rs), host-testable, precisely so the two pages cannot drift. - Widget frontend scaffold - Build the scaffold bundle: cd extensions/python-bridge/widget_frontend && npm install && npm run build - Verify output via ./scripts/verify_widget_frontend.sh (from extensions/python-bridge). - Widget packaging - Run extensions/python-bridge/scripts/verify_widget_frontend.sh after the frontend build. - Run extensions/python-bridge/scripts/verify_wheel_assets.sh against a finished wheel. See extensions/python-bridge/PACKAGING.md for the staged classic, JupyterLab, and WASM asset paths.

The Python bridge suite

cargo nextest does not run it and neither does the Rust gate. It is pytest, from extensions/python-bridge, and CI runs it as its own workflow (.github/workflows/python-tests.yml) on Python 3.10 and 3.13:

cd extensions/python-bridge
conda run -n orbitron-dev python -m pytest tests/ -q

Two environment variables keep the run from being quietly thinner than it looks:

  • ORBITRON_TRAJECTORY_NOTEBOOK_PATH=<a real optimisation log> un-skips the trajectory notebook test, which otherwise skips in silence. Anything with an optimisation in it works.
  • ORBITRON_CANONICAL_CACHE=$(mktemp -d) whenever parser behaviour is in play, for the reason in IO Pipelines §6.2 — a cache hit skips the parser, and the bridge has already returned band_gap: None for a file the CLI reported 0.572 eV for on that account.

Notebooks are executed by the suite through jupyter nbconvert --to notebook --execute, and all of them are expected to pass, including examples/api_viewer_widget.ipynb, which starts its own delta and scene servers in its first cell rather than assuming they are up. test_execute_quickstart_notebook fails rather than skips on a cell error. The one test that skips is the trajectory notebook.

Notebooks write their output under Path(tempfile.gettempdir()) / f"orbitron-{getpass.getuser()}", defined in each one’s first code cell — not a fixed /tmp/<name>. On Linux that is a shared namespace, so whichever account ran first owned the files and every later one got PermissionError. Moving CI to its own user hit that five separate times, once disguised as a failing determinism test.

CI names its test files individually rather than running pytest tests/, because the notebook file needs its own -m "not slow". A file left off that line is a suite nothing runs — three sat uncollected until 2026-09-04 — so add new files to the workflow invocation in the same commit that adds them.

11.4 Performance Benchmarks

Orbitron uses Criterion for targeted performance regression checks.

The conda environment includes the Python Playwright bindings used by the packaged-viewer smoke and browser benchmark. Install their matching Chromium build once after creating the environment:

conda run -n orbitron-dev python -m playwright install chromium
cargo bench -p orbitron-viewer-core --bench cartoon_build

# File-load RSS, capacity-model, and repeated-retention probe
ORBITRON_MEM_REPEAT_CYCLES=25 \
ORBITRON_MEM_REPEAT_FILE=io/pipelines/tests/fixtures/pdb/1ubq.pdb \
cargo bench -p orbitron-services --bench memory_footprint

Bench sources live under each crate’s benches/ directory — viewer/core/, ui/shell/, core/backbone/, core/inchi/, core/render/, core/services/, io/pipelines/, and automation/tui/ all ship one. The TUI bench is the one exception to a plain cargo bench: its target declares required-features = ["bench-stats"], so it needs cargo bench -p orbitron-tui --bench render --features bench-stats and otherwise fails with requires the features: bench-stats rather than running. Benches should prefer small fixtures from io/pipelines/tests/fixtures/ to keep runs fast and reproducible. Set ORBITRON_MEM_JSON=/path/to/result.json when the memory-footprint run needs a machine-readable release baseline. Each fixture record keeps the structured FileOpenEstimate beside measured peak and retained RSS growth. It also reports the projection-to-peak ratio, signed byte margin, and whether the projection bounded the observed peak. Only the first fixture in a process has a clean cold allocator baseline; run one fixture per invocation when calibrating a format multiplier.

On macOS, scripts/macos-process-monitor.py can bound a native GUI or other long-running workflow while recording the summed RSS for its command tree:

scripts/macos-process-monitor.py \
  --timeout 40 \
  --summary \
  --write ~/scratch/orbitron-viewer-rss.json \
  -- ./target/release/orbitron view path/to/structure.pdb

The monitor follows descendants even when a child starts another process group, as Chromium does for its browser and helper processes. The full JSON file contains the sample timeline. --summary only shortens the terminal output. On timeout or interruption, the monitor terminates every observed process group and escalates to SIGKILL after three seconds so child processes do not outlive the campaign. final_rss_bytes and final_process_count are sampled afterward across those observed groups. They report cleanup state rather than the last workload sample.

11.5 Test Fixtures

Orbitron uses two fixture roots under io/pipelines/tests/:

  • io/pipelines/tests/fixtures/ (in git): small canonical fixtures and malformed samples.
  • io/pipelines/tests/corpus/ (git-ignored): large real-world corpora. Tests should skip gracefully when corpus files are missing.

The committed fixture tree has a 150 MiB budget. Its README.md documents the rules for scratch files, volumetric downsampling, and optional stress corpora.

Helpers:

  • io/pipelines/tests/common/fixture.rs::fixture_path(...)
  • io/pipelines/tests/common/corpus.rs::corpus_path(...)

Pin every new text fixture in .gitattributes in the same commit that adds it: <path> text eol=lf. Git for Windows defaults to core.autocrlf=true, so a fixture that is byte-compared against a golden, or split on line structure, otherwise fails on a Windows checkout. Scope the pattern to text extensions, because the fixture trees also hold binary companions (.movecs, .hess, .chk, .db) that must never be eol-normalised.

A test that can skip work asserts how much it did. A CRLF checkout once made the InChI corpus ratchet compare nothing and pass in 0.01 s, because records whose id fails to parse are skipped rather than counted. Write assert_eq!(compared, reference.len()), not only “nothing went wrong”, so a broken loader fails instead of passing vacuously. The audit sweeps below follow the same rule.

11.6 Regression Checklist

Use this quick pass before requesting review, especially when touching canonical pipelines, helpers, or loader behaviour.

  • Environment prep: run everything through conda run -n orbitron-dev <command> (or activate the env once with conda activate orbitron-dev). Run the formatting and lint commands from §11.3 until clean.
  • Core parsers & canonical helpers: conda run -n orbitron-dev cargo nextest run -p orbitron-io-pipelines plus any targeted release-mode example probes. When adding or modifying helpers, ensure at least one affected format-specific test exercises them.
  • CLI & bundle workflows: conda run -n orbitron-dev cargo nextest run -p orbitron-cli (covers canonical export/import, cache hydration, QE/VASP/Molcas/Molpro regressions). If fixtures changed, re-run the manual smoke: conda run -n orbitron-dev cargo run --release -p orbitron-cli -- canonical export <fixture> and inspect the manifest and attachments.
  • SMILES differential: run conda run -n myrdkit --no-capture-output python scripts/differential_smiles.py and conda run -n myopenbabel --no-capture-output python scripts/differential_smiles.py --reference openbabel after changing SMILES parsing, explicit hydrogens, stereo constraints, or MOL/SDF export. Each environment is an independent reference. The JSON reports preserve tool refusals separately from structural or stereochemical disagreements.
  • Docs: update the Quarto sources under website/ whenever expectations shift (e.g., new helper contracts, format extras, canonical bundle layout). docs/ is generated output — website/_quarto.yml sets output-dir: ../docs, so an edit there is overwritten on the next render. Run quarto render website and inspect changed pages before committing.
  • Embedding: after a Python, WASM, widget, web-export, or local-server change, run both packaged embedding gates from §11.3. The first checks the built artifacts and HTTP contracts; the second proves the resulting scenes paint in a browser.
  • Fixtures & hashing: for new attachments/fixtures, record SHA256 in the relevant test assertions and verify register_attachment_refs + outcome_with_attachments are wired so caches stay deterministic.
  • UI/TUI spots (if touched): run conda run -n orbitron-dev cargo nextest run -p orbitron-ui-shell and conda run -n orbitron-dev cargo nextest run -p orbitron-tui, then perform a quick manual load of a representative canonical bundle via the CLI or viewer.

11.7 SMILES differential check

scripts/differential_smiles.py compares Orbitron independently against RDKit and Open Babel over tests/differential/smiles-corpus.json. Build the release CLI first, then run:

conda run -n myrdkit --no-capture-output python scripts/differential_smiles.py
conda run -n myopenbabel --no-capture-output python scripts/differential_smiles.py \
  --reference openbabel

The report distinguishes agree, disagree, tool-refused, reference-refused, both-refused, and no-reference. A missing reference is a no-reference outcome with exit status 2, not a passing comparison. Keep the references in their existing conda environments rather than installing either package into the system Python or Orbitron’s build environment.

11.8 Audit sweeps and mutation testing

Every real defect found in this repository came from looking at output or asserting an invariant. The suite was green through all of them. Three kinds of tooling exist to close that gap; reach for them before reading code hunting for bugs, because they generate candidates mechanically.

Invariant sweeps (gating). Each asserts a physical or arithmetic relation that reads several parsed quantities together, so a wrong one cannot agree with itself the way a golden can.

  • io/pipelines/tests/frequency_frame.rs — a run’s normal modes must belong to the geometry parsed with them (sum_i m_i (r_i x d_i) = 0). Judge the worst mode rather than the typical one: symmetry cancels the moment for most modes even in a wrong frame, and a median-based version passed a deliberately reverted fix. Compare each run against itself in a generic wrong frame rather than a fixed tolerance. It found the Gaussian parser pairing standard-orientation modes with an input-orientation geometry, and NWChem dividing every displacement by sqrt(mass) since the first commit. See notes/archive/frequency-frame-sweep.md.
  • io/pipelines/tests/thermochemistry_sweep.rs — H − E = RT, G = H − TS, and the entropy and Cv components summing to their totals, over 38 fixtures across four packages. It came back clean and is kept because it is arithmetic on already-parsed numbers and costs nothing. Verified to fire: leaving one Gaussian correction in Hartree fails 32 relations. See notes/archive/thermochemistry-sweep.md.
  • core/services/tests/mo_normalization.rs — ∫|ψ|² dV = 1, integrated on a grid through generate_orbital_grid rather than as analytic overlaps, because CᵀSC = I built from the evaluator’s own normalisation constants would agree with itself while both were wrong. The basis-function form ∫|φ|² = 1 is the sharper one, since it needs no coefficients. It found MOLDEN files carrying no electron count (so density slices returned identically zero) and NBO [GTO] coefficients being normalised twice. See notes/archive/mo-normalization-sweep.md.
  • io/pipelines/tests/frontier_gap.rs — Orbitron’s frontier against the one each package computes for itself: Molpro’s LUMO-HOMO line, DIRAC’s E(HOMO)/E(LUMO)/gap block, CP2K’s two spellings of its own. A printed answer is a free oracle, needing no reference implementation. Check the endpoints as well as the difference — two energies shifted together leave the gap intact. Six defects were living in that difference: Molpro dropped orbital rows whose first energy is positive (reading them as labels), did not recognise the restricted Orbital energies: spelling so six fixtures carried no orbitals at all, and placed an ROHF run’s SOMO by a positional rule that ignores symmetry blocking; DIRAC had only its poorer of two orbital listings read, so four runs carried no orbitals while the file printed its whole manifold and three more had a HOMO and no LUMO. The frontier is the sharp quantity because it reads occupancies, energies and symmetry blocking together, so any one of the three being wrong moves it. See notes/archive/frontier-gap-sweep.md.
  • io/pipelines/tests/orbital_occupation_sum.rs — occupancies must account for the electrons, which couples the orbital table, the effective core potential and the molecular charge, three things parsed by different readers. It is the complement to the sweep above, not a duplicate: a defect that halves every occupancy leaves the frontier exactly where it was. Three defects on the first run — DIRAC prints one eigenvalue per Kramers pair, so every DIRAC run reported half its electrons (U(2+) summed to 45 against 90); Molpro states an unrestricted run’s occupancy per symmetry and only the first count was read, so a CH₃ doublet had three alpha electrons instead of five; and a multi-step Gaussian log had every printing’s orbitals concatenated, giving HCN 55 orbitals holding 70 electrons. Five exemptions are named for physical reasons (ECP, pseudopotential, valence-only semi-empirical, NWChem printing only a window around the frontier, a QM/MM partition).
  • io/pipelines/tests/total_energy_sign.rs — a run’s total energy is negative. It asserts its own coverage (at least 340 fields read, or the sweep has lost its corpus) and pins the handful of runs whose total is legitimately positive in a POSITIVE_IS_CORRECT list, checked to still be that many. The cheapest thing here and clean today, kept for the same reason as the thermochemistry sweep. It would have caught a QE “benzene” reporting +14.55 Ry for four months: its nearest carbons were 0.43 Å apart against 1.39, its ewald term was +532.64 Ry where a real benzene gives +86.49, and the decomposition sat four lines below the number. What hid it was a test pinning the value under a comment explaining that a positive total is fine for a PAW run — a plausible explanation beside a nonsense value is how an unexamined number comes to look examined. Note the mechanism: an allowlist of field names, not a substring match. energy_hartree is also an orbital eigenvalue and DIRAC’s uranium run has a virtual at 1.7 million Hartree, while matching total_energy would miss final_energy_ry in the other direction.

The sweeps above are the ones with a story worth telling. The rest gate in the same way, and each one’s module doc is the canonical account of what it asserts (the table in the repository’s CLAUDE.md lists them with their notes):

  • io/pipelines/tests/bond_perception_invariants.rs — a rigid motion and a renumbering leave the bond set alone.
  • io/pipelines/tests/bond_length_floor.rs — no bond is shorter than its pair’s two triple-bond radii.
  • io/pipelines/tests/unit_cell_agreement.rs — the parsed cell matches the file’s own stated volume, and CIF angles round-trip.
  • io/pipelines/tests/basis_set_consistency.rs — shells, primitives and MO coefficient indices are mutually consistent.
  • io/pipelines/tests/point_group_order.rs — PointGroup::order() equals operations.len().
  • io/pipelines/tests/isosurface_topology.rs — an isosurface is a surface: no degenerate triangle, non-manifold edge or fold, and a sign-following enclosed volume.
  • io/pipelines/tests/trajectory_composition.rs — the element at index i is the same in every frame, compared as a sequence.
  • io/pipelines/tests/population_atom_id_invariant.rs — a parsed Mulliken population resolves to the scene atom it belongs to, on the right element.
  • io/pipelines/tests/parser_invariants.rs — components sum to their total, at most 3N−6 vibrations, charges add to the molecular charge, excitation energies ascend from a nonzero floor (one test is #[ignore]d).
  • io/pipelines/tests/truncated_input_sweep.rs — every byte prefix of a corpus file parses without panicking and gives a structurally sound scene.
  • io/pipelines/tests/run_provenance_sweep.rs — every run’s start time is read, with coverage asserted per format.
  • core/services/tests/hull_invariants.rs — a coordination polyhedron is a polyhedron (V−E+F=2, every edge twice, normals outward).
  • core/services/tests/render_data_invariants.rs — what build_scene_data hands the GPU matches the scene it came from.
  • core/services/tests/export_roundtrip_sweep.rs — every corpus scene written through every writer reads back as the same molecule, to the format’s precision.
  • core/edit/tests/undo_restores_the_scene.rs — apply then undo leaves the scene digest unchanged.
  • core/surface/tests/exact_invariants.rs — a solvent-accessible surface matches its closed form, converging as the grid refines.
  • automation/cli/tests/missing_atoms_sweep.rs — every heavy atom a PDB’s REMARK 470 or an mmCIF’s _pdbx_unobs_or_zero_occ_atoms states missing from a polymer residue is reported by analyze validation.
  • automation/cli/tests/metal_sites_link_sweep.rs — every metal contact a PDB’s LINK records or an mmCIF’s metalc rows state is a donor analyze metal-sites finds. Set ORBITRON_METAL_PDB_DIR to sweep full entries too.

Two diagnostic sweeps. Both are #[ignore]d: they print a report rather than gating, and a hit is a lead to check against the source file, not a defect.

# Fields null or empty in EVERY canonical document. An asymmetry — one field
# empty in all records while its sibling is populated in all of them — has no
# physical cause.
conda run -n orbitron-dev cargo nextest run -p orbitron-io-pipelines \
  --run-ignored all report_fields_empty_in_every_record --no-capture

# Section-marker literals that no file in the corpus contains. A search that
# finds nothing is indistinguishable from a file that contains nothing.
conda run -n orbitron-dev cargo nextest run -p orbitron-io-pipelines \
  --run-ignored all report_markers_no_file_contains --no-capture

Both assert their own coverage (records compared, markers extracted, files scanned) so a broken extractor fails rather than reporting a triumphant zero.

Mutation testing. cargo mutants answers “would a test notice if this were wrong”. Run it through scripts/nightly-mutants.sh, which caps build parallelism, cleans target/debug/deps first, shards a crate across nights, and writes a report under notes/audits/ (gitignored).

The config must live at .cargo/mutants.toml. That is the only path cargo-mutants reads by default; a copy at the workspace root is ignored in silence, and the symptom is indirect. The run falls back to cargo test, which fails the baseline on io/pipelines/tests/canonical_cache.rs (its tests set ORBITRON_CANONICAL_CACHE process-wide and need nextest’s process-per-test isolation), so cargo-mutants mutates nothing and reports MISSED 0 — which reads exactly like a clean sweep. The script treats “tested nothing” as a non-zero exit.

Two environment variables change what a run covers.

MUTANTS_TEST_WORKSPACE=1 runs the whole workspace’s tests against each mutant instead of only the mutated crate’s. It costs about 1.5x per mutant and it is the difference between a survivor list worth reading and one that is mostly noise: the guard for many files lives in another crate’s tests, so a mutant whose only guard is elsewhere is reported MISSED otherwise. On orbitron-edit all 24 survivors in command/ were false that way, because changes_scene is only called from viewer/core. It is off by default because a night then covers a third less ground.

With that flag the script also runs the unmutated workspace baseline itself and refuses the campaign if it fails. (Without the flag it does not.) cargo-mutants runs its own baseline against --package only, then runs every mutant against the whole workspace, so a test elsewhere that already fails “catches” every mutant. A Molpro run on 2026-09-21 reported MISSED 0 that way, with most of its catches coming from one CLI test failing on every run.

MUTANTS_FILE='<glob>' runs one area of a crate instead of a numbered shard, for example MUTANTS_FILE='io/pipelines/src/formats/cif/**' with orbitron-io-pipelines. The filter replaces sharding entirely, so the rotation is neither consulted nor advanced, and each filter gets its own work directory. It exists because io/pipelines partitions cleanly by format and a format is a unit whose survivors one person can read in a sitting.

Triaging survivors. “No test failed” cannot tell an equivalent mutant from an untested one. Re-run the survivors under cargo-mutants with a single throwaway test as the whole suite: one that serialises the canonical parse of every file in the format’s corpus and fails on any difference from an unmutated baseline. Then caught means the mutant changes what a real file parses to and no committed test notices, and missed means the whole corpus parses identically (an equivalent mutant, or code no file reaches). Normalise absolute paths first. The method and its results are in notes/archive/molpro-mutation-survivors.md and notes/archive/cp2k-mutation-survivors.md.

Sizing: core/diagram runs about 2 s per mutant; io/pipelines needs a full crate rebuild per mutant, about a minute each, and generates roughly 9,700 of them. Expect whole nights per shard there, and expect most mutants to be unviable (they do not compile) in crates full of typed builders.

11.9 Geometry harnesses against outside references

Four #[ignore]d reports measure the geometry tools against data Orbitron had no hand in. They print, they do not gate, and they need corpora that are not in the repository. They exist because every other check on this code scores it against its own conventions — the coordination-sphere builder places donors on a polyhedron’s vertices and is then scored by how close they are to that polyhedron, which cannot fail.

Harness Asks Needs
automation/cli/tests/templates_against_crystals.rs How far snapping a real sphere onto our nearest reference polyhedron moves its angles tmQM
automation/cli/tests/uff4mof_against_crystals.rs Whether UFF4MOF’s eight-coordinate types relax a crystal structure less far than the standard ones, and (report_typing_of_four_to_six_coordinate_metals) how far each four- to six-coordinate d-block sphere drifts under the type it is given, as TSV for a before/after tmQM
automation/cli/tests/builder_against_corpus.rs Whether a rebuilt complex beats the one in a corpus on distance, ligand integrity and atoms collapsed onto the metal reackt staging
automation/cli/tests/relax_drift.rs How far a force field moves a structure that is already settled reackt staging

tmQM is 108,541 CSD-derived complexes at TPSSh geometries. Clone it and decompress a shard; each is about 165 MB.

gunzip -c ~/scratch/tmqm/tmQM/tmQM_X1.xyz.gz > ~/scratch/tmqm/tmQM_X1.xyz
ORBITRON_TMQM=~/scratch/tmqm/tmQM_X1.xyz \
cargo nextest run --release -p orbitron-cli --run-ignored all \
  report_templates_against_crystals --no-capture

Each harness carries its findings and its caveats in its own header. Three are worth knowing before running any of them:

The donor-kind split is why the report carries one. An η⁵ cyclopentadienyl puts five carbons within bonding distance and an η⁶ arene six, so counting neighbours made a metallocene CN 10 and asked which ten-vertex polyhedron it resembled. Those spheres were the majority above CN 8 — Ti, Zr, Ru, Ir, Mo, Cr, W — and the median over them read as a template failure at high coordination that does not exist. coordination_sites now collapses a face-bound ring to the one site it occupies, which took no_clear_match over that shard from 1,204 to 37 and the median angle error from 0.201 to 0.155 rad while leaving the heteroatom spheres where they were. The split stays in the report because it is the check that the collapse has not quietly swallowed a real sphere.

Run them in release. A debug builder_against_corpus was still going after half an hour and finishes in three and a half minutes optimised. relax_drift settles about ninety structures a minute either way and writes a per-structure CSV so the analysis does not need a second run.

Drift compares fairly only between structures built the same way. These structures are built to an experiment-anchored distance table and then scored by how far a force field moves them; UFF’s metal–donor equilibrium is about 0.14 Å shorter than that table, so a template build drifts further than a force-field minimum without being worse.