Release quality gates are used as maintainer evidence, so a live baseline failure must not hide later provider or desktop-smoke lanes. Run every lane and summarize all pass/fail statuses so reviewers can tell whether the whole matrix was exercised.
Constraint: Live model lanes can fail independently and may be slow, but release evidence needs complete coverage.
Rejected: Keep fail-fast behavior | it makes reports ambiguous after the first live failure.
Confidence: high
Scope-risk: narrow
Directive: Do not reintroduce fail-fast behavior for release gates without adding explicit unexecuted-lane reporting.
Tested: bun test scripts/quality-gate/runner.test.ts scripts/quality-gate/baseline/cases.test.ts scripts/quality-gate/providerTargets.test.ts
Tested: bun run quality:gate --mode release --allow-live --provider-model codingplan:main:codingplan-main --provider-model minimax:main:minimax-main
Not-tested: clean-worktree release rerun before this commit; run immediately after committing.
The live baseline previously accepted provider UUIDs, which made the gate hard to run on another contributor's machine. Add a local provider listing command and resolve quality-gate targets from stable provider-name selectors while keeping UUIDs and current runtime support.
Constraint: Provider configuration is local machine state under CLAUDE_CONFIG_DIR and must not expose API keys.
Rejected: Require contributors to inspect providers.json manually | too error-prone and leaks implementation detail into the workflow
Confidence: high
Scope-risk: narrow
Directive: Keep live-provider baseline selection copyable from quality:providers before adding more live test lanes.
Tested: bun test scripts/quality-gate/providerTargets.test.ts
Tested: bun test scripts/quality-gate/*.test.ts scripts/quality-gate/baseline/*.test.ts
Tested: bun run quality:providers
Tested: bun run quality:gate --mode baseline --dry-run --provider-model codingplan:main --provider-model minimax:main
Tested: bun run quality:gate --mode baseline --dry-run --provider-model custom:haiku
Tested: bun run quality:gate --mode pr --dry-run
Tested: bun run check:server
The live baseline covered the server WebSocket path, but it still did not prove the desktop app can open a session, apply a selected provider/model, send a chat turn, and surface model/tool progress through the UI. This adds an agent-browser driven smoke lane that starts the local server and Vite desktop app, restores an isolated session tab with the requested runtime selection, submits a small coding task through the composer, and accepts the run only when the fixture diff and tests pass.
Constraint: Desktop smoke must stay behind --allow-live because it launches browsers and real models.
Constraint: The smoke temporarily enables bypassPermissions for the isolated run and restores the previous mode afterward.
Rejected: Wait for a final marker phrase | model reasoning and echoed prompts can contain the same marker before the work is actually done.
Rejected: Use only DOM text as success proof | the browser can show progress while the project files are still unchanged.
Confidence: high
Scope-risk: moderate
Directive: Future desktop smoke cases should verify project state and artifacts, not only UI copy.
Tested: bun test scripts/quality-gate/*.test.ts scripts/quality-gate/baseline/*.test.ts
Tested: bun run quality:gate --mode baseline --dry-run --provider-model minimax-m2.7
Tested: bun run quality:gate --mode baseline --allow-live --provider-model minimax-m2.7 (9 passed, 0 failed)
Tested: bun run check:server
Not-tested: Kimi desktop smoke completion because the provider returned AccountQuotaExceeded / API Error 429 until its reset window.
The first quality gate proved the execution path, but two tiny fixtures were not enough to reveal regressions in real agent behavior. This expands the maintained baseline with deeper failure recovery, workspace search, artifact creation, and cross-module refactor cases, and records per-case diffs so reviewers can see exactly what each model changed.
Constraint: Baseline runs must remain explicit maintainer-controlled live checks, not default PR checks.
Rejected: Require models to edit tests in every contract-change case | correct fixes can satisfy prewritten acceptance tests without modifying tests.
Rejected: Trust only command exit codes | passing tests alone does not show whether the agent edited forbidden or unrelated files.
Confidence: high
Scope-risk: moderate
Directive: Keep live baseline cases product-shaped and inspect both verification output and diff.patch before accepting future fixture changes.
Tested: bun test scripts/quality-gate/*.test.ts scripts/quality-gate/baseline/*.test.ts
Tested: bun run quality:gate --mode baseline --dry-run --provider-model volc-kimi-k2.6 --provider-model minimax-m2.7
Tested: bun run quality:gate --mode baseline --allow-live --provider-model volc-kimi-k2.6 --provider-model minimax-m2.7 (14 passed, 0 failed)
Not-tested: Desktop UI browser smoke and native Tauri release packaging remain outside this baseline expansion.
The desktop product needs a repeatable local gate that can prove the core Coding Agent loop still works after changes, not only that unit tests pass. This adds a quality-gate runner with PR, baseline, and release modes, structured reports, explicit quarantine metadata, and fixture-based live baseline cases that can run across provider/model targets.
Constraint: Existing check:pr and CI policy behavior must remain usable while the stronger baseline grows around it
Constraint: Default PR gates must not require real model credentials or provider quota
Rejected: Build a standalone QA platform first | too heavy before the baseline task shape is proven
Rejected: Keep unstable server exclusions hardcoded in run-server-tests | hides quarantine policy from reports and future review
Confidence: medium
Scope-risk: moderate
Directive: Expand baseline cases by adding focused fixtures and verifiers; do not make normal PR checks depend on live providers
Tested: bun test scripts/quality-gate/*.test.ts scripts/quality-gate/baseline/*.test.ts
Tested: bun run check:server
Tested: bun run quality:gate --mode baseline --allow-live --provider-model 2944f963-ce75-45b7-bac1-6e4f57df0970:kimi-k2.6:volc-kimi-k2.6 --provider-model 9c78d3df-7fb5-44c7-8436-3a41c3a59231:MiniMax-M2.7-highspeed:minimax-m2.7
Not-tested: desktop UI browser smoke and native release mode in this commit
