Testing & Quality

Orbitron uses unit tests, integration tests, corpus-backed smoke tests, and a small set of benchmarks. Keep tests fast and deterministic by default, and make large-corpus coverage opt-in.

11.1 What Testing Does

  • Catches regressions in parsing, rendering preparation, selection, editing, and analysis.
  • Ensures file loaders handle malformed inputs without panics.
  • Keeps refactors safe by locking down expected behavior at API boundaries.

11.2 Test Types & Locations

  • Unit tests: alongside code (#[cfg(test)]) or crate tests.
  • Integration tests:
    • IO workflows: io/pipelines/tests/
    • CLI workflows: automation/cli/tests/
  • Smoke tests: load many fixtures and validate “does not crash” behavior (typically under io/pipelines/tests/).
  • Malformed input tests: validate error surfaces (typically under io/pipelines/tests/).
  • Regression tests: tests/regression/.
  • Render-parity tests: tests/render-parity/ — a headless golden-image harness that renders a set of molecular scenes on the GPU offscreen path and perceptually diffs them against committed baselines, catching visual regressions in the renderer. Baselines regenerate with ORBITRON_REGEN_BASELINES=1. Skips automatically on machines without a usable GPU adapter — except in the dedicated CI job, which runs on Mesa lavapipe with ORBITRON_RENDER_PARITY_REQUIRE_GPU=1 so a missing adapter fails the job instead of silently skipping it.

11.3 Running Tests

cargo nextest run is the canonical runner (it is what [workspace.metadata.toolbox.commands] in the root Cargo.toml declares, and what CI uses exclusively). All commands run inside the conda environment.

# Workspace unit + integration tests
CARGO_BUILD_JOBS=16 conda run -n orbitron-dev cargo nextest run --workspace

# Single crate
conda run -n orbitron-dev cargo nextest run -p orbitron-io-pipelines

# One test by name
conda run -n orbitron-dev cargo nextest run test_parse_xyz_atom_count

# Show println!/logs
conda run -n orbitron-dev cargo nextest run --no-capture

# Doctests — nextest runs none of them, so this is a separate invocation
conda run -n orbitron-dev cargo test --workspace --doc

CARGO_BUILD_JOBS caps rustc during the build; concurrency during the run comes from .config/nextest.toml:

  • [profile.default] test-threads = 16. Nextest otherwise starts one test process per logical CPU, and on the 24-core dev machine a sustained full-tilt run has been enough for macOS to watchdog-kill WindowServer and restart the GUI session four times, taking every terminal with it.
  • [test-groups] gpu = { max-threads = 1 } — every test that creates a wgpu device shares one slot, because on macOS that device is the same GPU WindowServer composites with. Membership is all of orbitron-render-parity (which additionally takes threads-required = 16, so the heaviest test gets the whole runner) and orbitron-render’s offscreen_gpu_render_draws_atoms. A new test that builds a renderer has to be added to the gpu group.
  • [profile.ci] test-threads = "num-cpus" — CI runners are headless with far fewer cores, where a fixed 16 would oversubscribe.

With logging:

RUST_LOG=debug conda run -n orbitron-dev cargo nextest run --no-capture
RUST_LOG=orbitron=debug conda run -n orbitron-dev cargo nextest run

Quality gates, including doctests that nextest does not run:

conda run -n orbitron-dev cargo fmt --all
conda run -n orbitron-dev ./scripts/check-rust-lints.sh
CARGO_BUILD_JOBS=16 conda run -n orbitron-dev cargo nextest run --workspace
conda run -n orbitron-dev cargo test --workspace --doc

On a macOS workstation where high build-artifact churn makes Spotlight or syspolicyd destabilize the desktop session, run a Cargo phase through the cleanup wrapper:

./scripts/cargo-campaign.sh nextest run --workspace

The wrapper creates its target and canonical cache under a Spotlight-excluded directory in /private/tmp, caps compilation at 12 jobs by default, and removes the whole campaign directory on success, failure, or an interrupt. Each invocation starts from a cold target, so keep related tests in one Cargo command when practical. Ordinary developer and CI runs can continue using the standard commands above.

Optional smoke tests:

  • Packaged embedding gates
    • Run ./scripts/dev-smoke.sh from the repository root after changing the Python bridge, staged WASM assets, widget frontend, web export, or local viewer server. It builds the production chain and checks package and server contracts.
    • Run conda run -n orbitron-dev python scripts/dev-smoke-browser.py when a change can affect browser startup, scene transfer, rendering, multiple viewers, trajectories, density, or retry behavior. The browser gate opens the production bundle in Playwright and requires painted pixels rather than accepting a page that merely loaded.
    • Install the matching browser once with conda run -n orbitron-dev python -m playwright install chromium.
    • On macOS, DYLD_LIBRARY_PATH must remain unset for the Playwright launch. The repository setup script configures the conda environment that way.
  • Wasm viewer smoke test
    • Generate ~/scratch/orbitron-wasm/scene.bin with the release-CLI steps in Extensions.
    • Build + serve the wasm dist/ output (scene and wasm on same origin):
      • cd viewer/wasm && conda run -n orbitron-dev trunk build --release
      • cp ~/scratch/orbitron-wasm/scene.bin dist/scene.bin
      • cd dist && conda run -n orbitron-dev python -m http.server 8081 --bind 127.0.0.1
    • Then run: WASM_VIEWER_URL="http://127.0.0.1:8081/?scene=/scene.bin" npm run test:smoke
    • One-command alternative (from viewer/wasm): npm run test:smoke:dist
    • First-time Playwright setup may require npx playwright install.
  • Widget frontend scaffold
    • Build the scaffold bundle: cd extensions/python-bridge/widget_frontend && npm install && npm run build
    • Verify output via ./scripts/verify_widget_frontend.sh (from extensions/python-bridge).
  • Widget packaging
    • Run extensions/python-bridge/scripts/verify_widget_frontend.sh after the frontend build.
    • Run extensions/python-bridge/scripts/verify_wheel_assets.sh against a finished wheel. See extensions/python-bridge/PACKAGING.md for the staged classic, JupyterLab, and WASM asset paths.

11.4 Performance Benchmarks

Orbitron uses Criterion for targeted performance regression checks.

The conda environment includes the Python Playwright bindings used by the packaged-viewer smoke and browser benchmark. Install their matching Chromium build once after creating the environment:

conda run -n orbitron-dev python -m playwright install chromium
cargo bench -p orbitron-viewer-core --bench cartoon_build

# File-load RSS, capacity-model, and repeated-retention probe
ORBITRON_MEM_REPEAT_CYCLES=25 \
ORBITRON_MEM_REPEAT_FILE=io/pipelines/tests/fixtures/pdb/1ubq.pdb \
cargo bench -p orbitron-services --bench memory_footprint

Bench sources live in viewer/core/benches/ and should prefer small fixtures from io/pipelines/tests/fixtures/ to keep runs fast and reproducible. Set ORBITRON_MEM_JSON=/path/to/result.json when the memory-footprint run needs a machine-readable release baseline. Each fixture record keeps the structured FileOpenEstimate beside measured peak and retained RSS growth. It also reports the projection-to-peak ratio, signed byte margin, and whether the projection bounded the observed peak. Only the first fixture in a process has a clean cold allocator baseline; run one fixture per invocation when calibrating a format multiplier.

On macOS, scripts/macos-process-monitor.py can bound a native GUI or other long-running workflow while recording the summed RSS for its command tree:

scripts/macos-process-monitor.py \
  --timeout 40 \
  --summary \
  --write ~/scratch/orbitron-viewer-rss.json \
  -- ./target/release/orbitron view path/to/structure.pdb

The monitor follows descendants even when a child starts another process group, as Chromium does for its browser and helper processes. The full JSON file contains the sample timeline. --summary only shortens the terminal output. On timeout or interruption, the monitor terminates every observed process group and escalates to SIGKILL after three seconds so child processes do not outlive the campaign. final_rss_bytes and final_process_count are sampled afterward across those observed groups. They report cleanup state rather than the last workload sample.

11.5 Test Fixtures

Orbitron uses two fixture roots under io/pipelines/tests/:

  • io/pipelines/tests/fixtures/ (in git): small canonical fixtures and malformed samples.
  • io/pipelines/tests/corpus/ (git-ignored): large real-world corpora. Tests should skip gracefully when corpus files are missing.

The committed fixture tree has a 150 MiB budget. Its README.md documents the rules for scratch files, volumetric downsampling, and optional stress corpora.

Helpers:

  • io/pipelines/tests/common/fixture.rs::fixture_path(...)
  • io/pipelines/tests/common/corpus.rs::corpus_path(...)

11.6 Regression Checklist

Use this quick pass before requesting review, especially when touching canonical pipelines, helpers, or loader behaviour.

  • Environment prep: run everything through conda run -n orbitron-dev <command> (or activate the env once with conda activate orbitron-dev). Run the formatting and lint commands from §11.3 until clean.
  • Core parsers & canonical helpers: conda run -n orbitron-dev cargo nextest run -p orbitron-io-pipelines plus any targeted release-mode example probes. When adding or modifying helpers, ensure at least one affected format-specific test exercises them.
  • CLI & bundle workflows: conda run -n orbitron-dev cargo nextest run -p orbitron-cli (covers canonical export/import, cache hydration, QE/VASP/Molcas/Molpro regressions). If fixtures changed, re-run the manual smoke: conda run -n orbitron-dev cargo run --release -p orbitron-cli -- canonical export <fixture> and inspect the manifest and attachments.
  • Regression crate: conda run -n orbitron-dev cargo nextest run -p orbitron-regression for loader digests whenever you change parsing logic, canonical metadata, or attachments.
  • SMILES differential: run conda run -n myrdkit --no-capture-output python scripts/differential_smiles.py and conda run -n myopenbabel --no-capture-output python scripts/differential_smiles.py --reference openbabel after changing SMILES parsing, explicit hydrogens, stereo constraints, or MOL/SDF export. Each environment is an independent reference. The JSON reports preserve tool refusals separately from structural or stereochemical disagreements.
  • Docs: update the Quarto sources under website/ whenever expectations shift (e.g., new helper contracts, format extras, canonical bundle layout). docs/ is generated output — website/_quarto.yml sets output-dir: ../docs, so an edit there is overwritten on the next render. Run quarto render website and inspect changed pages before committing.
  • Embedding: after a Python, WASM, widget, web-export, or local-server change, run both packaged embedding gates from §11.3. The first checks the built artifacts and HTTP contracts; the second proves the resulting scenes paint in a browser.
  • Fixtures & hashing: for new attachments/fixtures, record SHA256 in the relevant test assertions and verify register_attachment_refs + outcome_with_attachments are wired so caches stay deterministic.
  • UI/TUI spots (if touched): run conda run -n orbitron-dev cargo nextest run -p orbitron-ui-shell and conda run -n orbitron-dev cargo nextest run -p orbitron-tui, then perform a quick manual load of a representative canonical bundle via the CLI or viewer.

11.7 SMILES differential check

scripts/differential_smiles.py compares Orbitron independently against RDKit and Open Babel over tests/differential/smiles-corpus.json. Build the release CLI first, then run:

conda run -n myrdkit --no-capture-output python scripts/differential_smiles.py
conda run -n myopenbabel --no-capture-output python scripts/differential_smiles.py \
  --reference openbabel

The report distinguishes agree, disagree, tool-refused, reference-refused, both-refused, and no-reference. A missing reference is a no-reference outcome with exit status 2, not a passing comparison. Keep the references in their existing conda environments rather than installing either package into the system Python or Orbitron’s build environment.