The repository now has a measurable PR quality path instead of a loose set of manual checks. Coverage, quarantine governance, provider smoke, desktop smoke, and workflow wiring all produce durable reports that contributors and maintainers can inspect without reconstructing terminal output. This also fixes the desktop smoke current-runtime path so browser-driven smoke runs use the desktop default active provider instead of forcing the official current model, and records that runtime decision as an artifact. Constraint: Default PR gates must remain non-live and contributor-safe while live model checks stay explicit. Constraint: Release packaging is still GitHub Actions based, so release preflight must run before the build matrix. Rejected: Make live provider or desktop smoke mandatory on every PR | secrets, quotas, and model availability are maintainer-controlled. Rejected: Let PRs lower coverage baselines in the same change | base-branch ratchet comparison must remain authoritative. Confidence: high Scope-risk: moderate Directive: Do not relax coverage or quarantine policy without a maintainer approval label and a fresh quality report. Tested: ALLOW_CLI_CORE_CHANGE=1 ALLOW_COVERAGE_BASELINE_CHANGE=1 bun run quality:gate --mode pr Tested: bun run quality:gate --mode baseline --allow-live --only provider-smoke:* --provider-model nvidia-custom:main:nvidia-custom-main --artifacts-dir /tmp/quality-gate-live-smoke Tested: bun run quality:gate --mode baseline --allow-live --only desktop-smoke:* --provider-model current:current:current-runtime --artifacts-dir /tmp/quality-gate-desktop-smoke-fixed Tested: git diff --check Not-tested: Full live release mode with multiple providers in hosted CI; provider credentials and quota remain maintainer-controlled.
7.1 KiB
Contributing and Local Quality Gates
This guide explains how to install, develop, test, and run the local quality gates before opening a PR. The goal is to help maintainers and contributors answer one question before review: did this change break the core Coding Agent workflow?
Setup
Install root dependencies with Bun:
bun install
If your change touches desktop/, also install desktop dependencies:
cd desktop
bun install
If your change touches adapters/, or if you run check:adapters / check:native, install adapter dependencies:
cd adapters
bun install
Do not commit local artifacts such as artifacts/quality-runs/, node_modules/, or desktop/node_modules/.
Required PR Gate
Before opening a normal PR, run:
bun run quality:pr
This gate does not call real models, so every contributor can run it locally. It writes reports to:
artifacts/quality-runs/<timestamp>/report.md
artifacts/quality-runs/<timestamp>/report.json
artifacts/quality-runs/<timestamp>/junit.xml
artifacts/quality-runs/<timestamp>/logs/*.log
artifacts/coverage/<timestamp>/coverage-report.md
artifacts/coverage/<timestamp>/coverage-report.json
Include the commands you ran and the report summary in your PR description.
quality:pr also runs quarantine and coverage gates. Coverage uses a ratchet policy: the current baseline lives in scripts/quality-gate/coverage-baseline.json, and CI compares against the base branch baseline when available. New PRs must not lower overall coverage beyond the allowed window. Changes to coverage-baseline.json or coverage-thresholds.json require the maintainer-only allow-coverage-baseline-change label. Quarantine entries must keep an owner, reviewAfter date, and exit criteria; once reviewAfter expires, the default server and coverage gates fail until maintainers review the entry.
Area-Specific Checks
Run the checks that match the files you changed:
bun run check:server # Server API, WebSocket, providers, sessions, and related tests
bun run check:desktop # Desktop lint, Vitest, and production build
bun run check:adapters # IM adapter tests
bun run check:native # Desktop sidecars and Tauri native checks
bun run check:docs # Docs build, using npm ci + docs:build
bun run check:quarantine # Quarantine owners, exit criteria, and review windows
bun run check:coverage # Root, desktop, and adapter coverage reports plus ratchet enforcement
Focused tests are fine while developing, but run bun run quality:pr before sending the PR.
Production code changes must include matching tests. Changes under desktop/src/**, src/server/**, src/tools/**, src/utils/**, or adapters/** without a same-area test file are blocked unless a maintainer applies allow-missing-tests. Coverage baseline/threshold changes are also blocked unless a maintainer applies allow-coverage-baseline-change.
Live Model Baseline
quality:baseline runs real Coding Agent tasks: it starts the local server, creates isolated fixtures, asks a model through chat to fix code, runs tests, and saves transcripts, diffs, verification logs, and a report. It also runs provider live smoke: saved or active OpenAI-compatible providers validate connectivity, proxy conversion, and streaming proxy behavior; env-only provider smoke validates upstream connectivity and the transform pipeline.
The default baseline command does not call real models:
bun run quality:baseline
To actually call models, pass --allow-live and choose a local provider.
First list your local providers and copyable selectors:
bun run quality:providers
Example output:
Saved providers:
MiniMax
selector: minimax
main: MiniMax-M2.7-highspeed
--provider-model minimax:main:minimax-main
Copy one of the listed values:
bun run quality:gate --mode baseline --allow-live --provider-model minimax:main:minimax-main
To run only provider smoke plus desktop agent-browser smoke, use:
bun run quality:smoke --provider-model minimax:main:minimax-main
You can run multiple models in one pass:
bun run quality:gate --mode baseline --allow-live \
--provider-model codingplan:main:codingplan-main \
--provider-model minimax:main:minimax-main
Provider selectors come from the providers saved in your local Desktop Settings > Providers page. Contributors do not need the maintainer's provider UUIDs or vendor accounts. They can add their own provider locally, run bun run quality:providers, and choose their own model.
If you do not have a saved provider, you can run one unsaved provider smoke with environment variables:
QUALITY_GATE_PROVIDER_BASE_URL=https://example.com \
QUALITY_GATE_PROVIDER_API_KEY=... \
QUALITY_GATE_PROVIDER_MODEL=model-id \
QUALITY_GATE_PROVIDER_API_FORMAT=openai_chat \
bun run quality:gate --mode baseline --allow-live
When To Run The Baseline
Run the live baseline for changes touching:
- Desktop chat, session resume, WebSocket, or the CLI bridge
- Provider, model, or runtime selection
- Permissions, tool calls, file edits, and task execution
- agent-browser smoke, Computer Use, Skills, or MCP
- Release preparation or broad cross-module refactors
If you do not have model access, still run bun run quality:pr and state in the PR why the live baseline was not run.
Release Gate
Before a release, run release mode:
bun run quality:gate --mode release --allow-live --provider-model <selector>:main
Release mode composes PR checks, baseline catalog validation, live baseline cases, provider smoke, desktop smoke, and native checks. Reports are written to artifacts/quality-runs/<timestamp>/. The hosted release workflow now runs quality:gate --mode pr as a non-live preflight before the packaging matrix and uploads a release-quality-gate artifact; maintainers still need to run the live release gate explicitly with an available provider.
In release mode, live lanes are not allowed to be silently skipped. Missing providers, model quota, or external account access will fail the gate and must be recorded as a release blocker.
PR Workflow
- Create a product branch such as
fix/session-reconnectorfeat/provider-quality-gate. - Install dependencies and make the change.
- Add tests for behavior changes.
- Run focused checks for the affected area.
- Run
bun run quality:pr. - Run the live baseline for high-risk changes.
- In the PR description, include user impact, verification commands, coverage/quality report summary, and known risks.
FAQ
Can I run checks without a provider?
Yes. Run the normal PR gate:
bun run quality:pr
Only the live baseline needs a real model. Add your provider in Desktop Settings > Providers, then run:
bun run quality:providers
What if provider selectors conflict?
If two provider names produce the same selector, quality:providers falls back to the provider ID. Copy the --provider-model ... value it prints.
What if a model ID contains a colon?
Prefer role selectors:
--provider-model custom:haiku:custom-haiku
The runner resolves haiku to the real model ID from your local provider configuration.