mirror of
https://github.com/NanmiCoder/cc-haha
synced 2026-07-16 13:03:31 +08:00
5ed9903d2e
The live baseline covered the server WebSocket path, but it still did not prove the desktop app can open a session, apply a selected provider/model, send a chat turn, and surface model/tool progress through the UI. This adds an agent-browser driven smoke lane that starts the local server and Vite desktop app, restores an isolated session tab with the requested runtime selection, submits a small coding task through the composer, and accepts the run only when the fixture diff and tests pass. Constraint: Desktop smoke must stay behind --allow-live because it launches browsers and real models. Constraint: The smoke temporarily enables bypassPermissions for the isolated run and restores the previous mode afterward. Rejected: Wait for a final marker phrase | model reasoning and echoed prompts can contain the same marker before the work is actually done. Rejected: Use only DOM text as success proof | the browser can show progress while the project files are still unchanged. Confidence: high Scope-risk: moderate Directive: Future desktop smoke cases should verify project state and artifacts, not only UI copy. Tested: bun test scripts/quality-gate/*.test.ts scripts/quality-gate/baseline/*.test.ts Tested: bun run quality:gate --mode baseline --dry-run --provider-model minimax-m2.7 Tested: bun run quality:gate --mode baseline --allow-live --provider-model minimax-m2.7 (9 passed, 0 failed) Tested: bun run check:server Not-tested: Kimi desktop smoke completion because the provider returned AccountQuotaExceeded / API Error 429 until its reset window.
86 lines
1.7 KiB
TypeScript
86 lines
1.7 KiB
TypeScript
export type QualityGateMode = 'pr' | 'baseline' | 'release'
|
|
|
|
export type LaneKind = 'command' | 'baseline-case' | 'desktop-smoke'
|
|
|
|
export type LaneDefinition = {
|
|
id: string
|
|
title: string
|
|
description: string
|
|
kind: LaneKind
|
|
command?: string[]
|
|
baselineCaseId?: string
|
|
baselineTarget?: BaselineTarget
|
|
requiredForModes: QualityGateMode[]
|
|
live?: boolean
|
|
}
|
|
|
|
export type BaselineCase = {
|
|
id: string
|
|
title: string
|
|
description: string
|
|
fixture: string
|
|
prompt: string
|
|
mode: 'ui' | 'websocket'
|
|
requiredCapabilities: Array<'model' | 'file-edit' | 'shell' | 'permission' | 'browser'>
|
|
timeoutMs: number
|
|
verify: {
|
|
commands: string[][]
|
|
requiredFiles?: string[]
|
|
expectedFiles?: string[]
|
|
forbiddenFiles?: string[]
|
|
transcriptAssertions?: string[]
|
|
}
|
|
}
|
|
|
|
export type BaselineTarget = {
|
|
providerId: string | null
|
|
modelId: string
|
|
label: string
|
|
}
|
|
|
|
export type LaneStatus = 'passed' | 'failed' | 'skipped'
|
|
|
|
export type LaneResult = {
|
|
id: string
|
|
title: string
|
|
status: LaneStatus
|
|
command?: string[]
|
|
durationMs: number
|
|
exitCode?: number
|
|
skipReason?: string
|
|
error?: string
|
|
artifactDir?: string
|
|
}
|
|
|
|
export type QualityGateOptions = {
|
|
mode: QualityGateMode
|
|
dryRun: boolean
|
|
allowLive: boolean
|
|
baselineTargets: BaselineTarget[]
|
|
rootDir: string
|
|
artifactsDir?: string
|
|
runOutputDir?: string
|
|
runId?: string
|
|
}
|
|
|
|
export type QualityGateReport = {
|
|
schemaVersion: 1
|
|
runId: string
|
|
mode: QualityGateMode
|
|
dryRun: boolean
|
|
allowLive: boolean
|
|
startedAt: string
|
|
finishedAt: string
|
|
rootDir: string
|
|
git: {
|
|
sha: string | null
|
|
dirty: boolean
|
|
}
|
|
results: LaneResult[]
|
|
summary: {
|
|
passed: number
|
|
failed: number
|
|
skipped: number
|
|
}
|
|
}
|