mirror of
https://github.com/NanmiCoder/cc-haha
synced 2026-07-17 13:13:35 +08:00
feat: add Computer Use architecture docs and fix all anchor links
- Add Computer Use architecture deep dive documentation (CN/EN) with 4 generated diagrams (architecture, security gates, Python Bridge, patch environment comparison) - Move Computer Use section below Skills in sidebar and README - Fix 153 broken anchor links across 18 documentation files: - CN docs: add missing `、` after Chinese numeral prefixes - EN docs: add missing `_` prefix for number-starting heading IDs - Fix em-dash and special character encoding in anchor hrefs
This commit is contained in:
程序员阿江(Relakkes)
parent
89e6ff3781
commit
cf55c55d00
@ -33,10 +33,10 @@ A **locally runnable version** repaired from the leaked Claude Code source, with
|
||||
- `--print` headless mode for scripts and CI
|
||||
- MCP server, plugin, and Skills support
|
||||
- Custom API endpoint and model support ([Third-Party Models Guide](docs/en/guide/third-party-models.md))
|
||||
- **Computer Use desktop control** — [Guide](docs/en/features/computer-use.md)
|
||||
- **Memory System** (cross-session persistent memory) — [Usage Guide](docs/memory/01-usage-guide.md)
|
||||
- **Multi-Agent System** (agent orchestration, parallel tasks, Teams collaboration) — [Usage Guide](docs/agent/01-usage-guide.md) | [Implementation](docs/agent/02-implementation.md)
|
||||
- **Skills System** (extensible capability plugins, custom workflows) — [Usage Guide](docs/skills/01-usage-guide.md) | [Implementation](docs/skills/02-implementation.md)
|
||||
- **Computer Use desktop control** — [Guide](docs/en/features/computer-use.md) | [Architecture](docs/en/features/computer-use-architecture.md)
|
||||
- Fallback Recovery CLI mode (`CLAUDE_CODE_FORCE_RECOVERY_CLI=1 ./bin/claude-haha`)
|
||||
|
||||
---
|
||||
@ -136,10 +136,10 @@ export PATH="$HOME/path/to/claude-code-haha/bin:$PATH"
|
||||
|------|------|
|
||||
| [Environment Variables](docs/en/guide/env-vars.md) | Full env var reference and configuration methods |
|
||||
| [Third-Party Models](docs/en/guide/third-party-models.md) | Using OpenAI / DeepSeek / Ollama and other non-Anthropic models |
|
||||
| [Computer Use](docs/en/features/computer-use.md) | Desktop control (screenshots, mouse, keyboard) |
|
||||
| [Memory System](docs/memory/01-usage-guide.md) | Cross-session persistent memory usage and implementation |
|
||||
| [Multi-Agent System](docs/agent/01-usage-guide.md) | Agent orchestration, parallel tasks and Teams collaboration |
|
||||
| [Skills System](docs/skills/01-usage-guide.md) | Extensible capability plugins, custom workflows and conditional activation |
|
||||
| [Computer Use](docs/en/features/computer-use.md) | Desktop control (screenshots, mouse, keyboard) — [Architecture](docs/en/features/computer-use-architecture.md) |
|
||||
| [Global Usage](docs/en/guide/global-usage.md) | Run claude-haha from any directory |
|
||||
| [FAQ](docs/en/guide/faq.md) | Common error troubleshooting |
|
||||
| [Source Fixes](docs/en/reference/fixes.md) | Fixes compared with the original leaked source |
|
||||
|
||||
@ -33,10 +33,10 @@
|
||||
- `--print` 无头模式（脚本/CI 场景）
|
||||
- 支持 MCP 服务器、插件、Skills
|
||||
- 支持自定义 API 端点和模型（[第三方模型使用指南](docs/guide/third-party-models.md)）
|
||||
- **Computer Use 桌面控制** — [使用指南](docs/features/computer-use.md)
|
||||
- **记忆系统**（跨会话持久化记忆）— [使用指南](docs/memory/01-usage-guide.md)
|
||||
- **多 Agent 系统**（多代理编排、并行任务、Teams 协作）— [使用指南](docs/agent/01-usage-guide.md) | [实现原理](docs/agent/02-implementation.md)
|
||||
- **Skills 系统**（可扩展能力插件、自定义工作流）— [使用指南](docs/skills/01-usage-guide.md) | [实现原理](docs/skills/02-implementation.md)
|
||||
- **Computer Use 桌面控制** — [功能指南](docs/features/computer-use.md) | [架构解析](docs/features/computer-use-architecture.md)
|
||||
- 降级 Recovery CLI 模式（`CLAUDE_CODE_FORCE_RECOVERY_CLI=1 ./bin/claude-haha`）
|
||||
|
||||
---
|
||||
@ -136,10 +136,10 @@ export PATH="$HOME/path/to/claude-code-haha/bin:$PATH"
|
||||
|------|------|
|
||||
| [环境变量](docs/guide/env-vars.md) | 完整环境变量参考和配置方式 |
|
||||
| [第三方模型](docs/guide/third-party-models.md) | 接入 OpenAI / DeepSeek / Ollama 等非 Anthropic 模型 |
|
||||
| [Computer Use](docs/features/computer-use.md) | 桌面控制功能（截屏、鼠标、键盘） |
|
||||
| [记忆系统](docs/memory/01-usage-guide.md) | 跨会话持久化记忆的使用与实现 |
|
||||
| [多 Agent 系统](docs/agent/01-usage-guide.md) | 多代理编排、并行任务执行与 Teams 协作 |
|
||||
| [Skills 系统](docs/skills/01-usage-guide.md) | 可扩展能力插件、自定义工作流与条件激活 |
|
||||
| [Computer Use](docs/features/computer-use.md) | 桌面控制功能（截屏、鼠标、键盘）— [架构解析](docs/features/computer-use-architecture.md) |
|
||||
| [全局使用](docs/guide/global-usage.md) | 在任意目录启动 claude-haha |
|
||||
| [常见问题](docs/guide/faq.md) | 常见错误排查 |
|
||||
| [源码修复记录](docs/reference/fixes.md) | 相对于原始泄露源码的修复内容 |
|
||||
|
||||
@ -11,12 +11,6 @@ const zhSidebar = [
|
||||
{ text: '常见问题', link: '/guide/faq' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: '功能',
|
||||
items: [
|
||||
{ text: 'Computer Use', link: '/features/computer-use' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: '记忆系统',
|
||||
collapsed: false,
|
||||
@ -45,6 +39,14 @@ const zhSidebar = [
|
||||
{ text: '实现原理', link: '/skills/02-implementation' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'Computer Use',
|
||||
collapsed: false,
|
||||
items: [
|
||||
{ text: '功能指南', link: '/features/computer-use' },
|
||||
{ text: '架构解析', link: '/features/computer-use-architecture' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: '参考',
|
||||
collapsed: true,
|
||||
@ -66,12 +68,6 @@ const enSidebar = [
|
||||
{ text: 'FAQ', link: '/en/guide/faq' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'Features',
|
||||
items: [
|
||||
{ text: 'Computer Use', link: '/en/features/computer-use' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'Memory System',
|
||||
collapsed: false,
|
||||
@ -100,6 +96,14 @@ const enSidebar = [
|
||||
{ text: 'Implementation', link: '/en/skills/02-implementation' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'Computer Use',
|
||||
collapsed: false,
|
||||
items: [
|
||||
{ text: 'Guide', link: '/en/features/computer-use' },
|
||||
{ text: 'Architecture', link: '/en/features/computer-use-architecture' },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: 'Reference',
|
||||
collapsed: true,
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> 让 Claude Code 同时调度多个专业代理，并行处理复杂任务。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一什么是多-agent-系统">多 Agent 系统</a> · <a href="#二六种内置-agent">六种内置 Agent</a> · <a href="#三如何生成-agent">如何生成 Agent</a> · <a href="#四后台任务管理">后台任务管理</a> · <a href="#五agent-teams--多代理协作">Agent Teams</a> · <a href="#六自定义-agent">自定义 Agent</a> · <a href="#七权限模式">权限模式</a> · <a href="#八快速参考">快速参考</a>
|
||||
<a href="#一、什么是多-agent-系统">多 Agent 系统</a> · <a href="#二、六种内置-agent">六种内置 Agent</a> · <a href="#三、如何生成-agent">如何生成 Agent</a> · <a href="#四、后台任务管理">后台任务管理</a> · <a href="#五、agent-teams-—-多代理协作">Agent Teams</a> · <a href="#六、自定义-agent">自定义 Agent</a> · <a href="#七、权限模式">权限模式</a> · <a href="#八、快速参考">快速参考</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> 深入剖析多 Agent 编排的架构设计、生成流程、上下文传递和协作机制。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一架构总览">架构总览</a> · <a href="#二agent-生成流程--四条路径">生成流程</a> · <a href="#三工具池系统--三层过滤">工具池系统</a> · <a href="#四上下文传递机制">上下文传递</a> · <a href="#五agent-teams-内部机制">Teams 内部机制</a> · <a href="#六后台任务引擎">后台任务引擎</a> · <a href="#七dreamtask--自动记忆整合">DreamTask</a> · <a href="#八worktree-隔离实现">Worktree 隔离</a> · <a href="#九权限同步机制">权限同步</a> · <a href="#十agent-生命周期完整数据流">生命周期数据流</a> · <a href="#十一关键源文件索引">源文件索引</a> · <a href="#十二feature-flags">Feature Flags</a>
|
||||
<a href="#一、架构总览">架构总览</a> · <a href="#二、agent-生成流程-—-四条路径">生成流程</a> · <a href="#三、工具池系统-—-三层过滤">工具池系统</a> · <a href="#四、上下文传递机制">上下文传递</a> · <a href="#五、agent-teams-内部机制">Teams 内部机制</a> · <a href="#六、后台任务引擎">后台任务引擎</a> · <a href="#七、dreamtask-—-自动记忆整合">DreamTask</a> · <a href="#八、worktree-隔离实现">Worktree 隔离</a> · <a href="#九、权限同步机制">权限同步</a> · <a href="#十、agent-生命周期完整数据流">生命周期数据流</a> · <a href="#十一、关键源文件索引">源文件索引</a> · <a href="#十二、feature-flags">Feature Flags</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> 从源码视角剖析全球最流行 AI Code Editor 背后的 Agent 架构设计哲学。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一核心-agent-循环">核心循环</a> · <a href="#二系统提示词工程">提示词工程</a> · <a href="#三工具系统设计">工具系统</a> · <a href="#四上下文管理与压缩">上下文管理</a> · <a href="#五技能与插件生态">技能与插件</a> · <a href="#六权限与安全体系">权限与安全</a> · <a href="#七故障恢复机制">故障恢复</a> · <a href="#八与-langchainreact-的本质区别">对比分析</a> · <a href="#九为什么-claude-code-能做到这么好">成功之道</a>
|
||||
<a href="#一、核心-agent-循环">核心循环</a> · <a href="#二、系统提示词工程">提示词工程</a> · <a href="#三、工具系统设计">工具系统</a> · <a href="#四、上下文管理与压缩">上下文管理</a> · <a href="#五、技能与插件生态">技能与插件</a> · <a href="#六、权限与安全体系">权限与安全</a> · <a href="#七、故障恢复机制">故障恢复</a> · <a href="#八、与-langchain-react-的本质区别">对比分析</a> · <a href="#九、为什么-claude-code-能做到这么好">成功之道</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> Let Claude Code orchestrate multiple specialized agents to handle complex tasks in parallel.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-what-is-the-multi-agent-system">Multi-Agent System</a> · <a href="#2-six-built-in-agents">Six Built-in Agents</a> · <a href="#3-how-to-spawn-agents">Spawning Agents</a> · <a href="#4-background-task-management">Background Tasks</a> · <a href="#5-agent-teams--multi-agent-collaboration">Agent Teams</a> · <a href="#6-custom-agents">Custom Agents</a> · <a href="#7-permission-modes">Permission Modes</a> · <a href="#8-quick-reference">Quick Reference</a>
|
||||
<a href="#_1-what-is-the-multi-agent-system">Multi-Agent System</a> · <a href="#_2-six-built-in-agents">Six Built-in Agents</a> · <a href="#_3-how-to-spawn-agents">Spawning Agents</a> · <a href="#_4-background-task-management">Background Tasks</a> · <a href="#_5-agent-teams-—-multi-agent-collaboration">Agent Teams</a> · <a href="#_6-custom-agents">Custom Agents</a> · <a href="#_7-permission-modes">Permission Modes</a> · <a href="#_8-quick-reference">Quick Reference</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> A deep dive into the architecture, spawn flow, context passing, and collaboration mechanisms of multi-agent orchestration.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-architecture-overview">Architecture</a> · <a href="#2-agent-spawn-flow--four-paths">Spawn Flow</a> · <a href="#3-tool-pool-system--three-layer-filtering">Tool Pool</a> · <a href="#4-context-passing-mechanism">Context Passing</a> · <a href="#5-agent-teams-internals">Teams Internals</a> · <a href="#6-background-task-engine">Task Engine</a> · <a href="#7-dreamtask--automatic-memory-consolidation">DreamTask</a> · <a href="#8-worktree-isolation-implementation">Worktree Isolation</a> · <a href="#9-permission-synchronization">Permission Sync</a> · <a href="#10-agent-lifecycle-end-to-end-data-flow">Lifecycle Data Flow</a> · <a href="#11-key-source-file-index">Source Index</a> · <a href="#12-feature-flags">Feature Flags</a>
|
||||
<a href="#_1-architecture-overview">Architecture</a> · <a href="#_2-agent-spawn-flow-—-four-paths">Spawn Flow</a> · <a href="#_3-tool-pool-system-—-three-layer-filtering">Tool Pool</a> · <a href="#_4-context-passing-mechanism">Context Passing</a> · <a href="#_5-agent-teams-internals">Teams Internals</a> · <a href="#_6-background-task-engine">Task Engine</a> · <a href="#_7-dreamtask-—-automatic-memory-consolidation">DreamTask</a> · <a href="#_8-worktree-isolation-implementation">Worktree Isolation</a> · <a href="#_9-permission-synchronization">Permission Sync</a> · <a href="#_10-agent-lifecycle-end-to-end-data-flow">Lifecycle Data Flow</a> · <a href="#_11-key-source-file-index">Source Index</a> · <a href="#_12-feature-flags">Feature Flags</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> Deconstructing the architecture behind the world's most popular AI code editor — from source code to design philosophy.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-the-core-agent-loop">Core Loop</a> · <a href="#2-system-prompt-engineering">Prompt Engineering</a> · <a href="#3-tool-system-design">Tool System</a> · <a href="#4-context-management--compression">Context Management</a> · <a href="#5-skills--plugin-ecosystem">Skills & Plugins</a> · <a href="#6-permission--security-model">Permissions</a> · <a href="#7-fault-recovery-mechanisms">Recovery</a> · <a href="#8-how-it-differs-from-langchainreact">vs LangChain</a> · <a href="#9-why-claude-code-is-so-good">Why It Works</a>
|
||||
<a href="#_1-the-core-agent-loop">Core Loop</a> · <a href="#_2-system-prompt-engineering">Prompt Engineering</a> · <a href="#_3-tool-system-design">Tool System</a> · <a href="#_4-context-management-compression">Context Management</a> · <a href="#_5-skills-plugin-ecosystem">Skills & Plugins</a> · <a href="#_6-permission-security-model">Permissions</a> · <a href="#_7-fault-recovery-mechanisms">Recovery</a> · <a href="#_8-how-it-differs-from-langchain-react">vs LangChain</a> · <a href="#_9-why-claude-code-is-so-good">Why It Works</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
602
docs/en/features/computer-use-architecture.md
Normal file
602
docs/en/features/computer-use-architecture.md
Normal file
@ -0,0 +1,602 @@
|
||||
# Computer Use Architecture Deep Dive
|
||||
|
||||
> A deep dive into the Computer Use implementation: from MCP tool definitions to Python Bridge, from 9-layer security gates to feature flag bypasses.
|
||||
|
||||
<p align="center">
|
||||
<a href="#_1-patch-environment-overview">Patch Environment</a> ·
|
||||
<a href="#_2-layered-architecture">Layered Architecture</a> ·
|
||||
<a href="#_3-mcp-tool-layer">MCP Tool Layer</a> ·
|
||||
<a href="#_4-security-gate-system">Security Gates</a> ·
|
||||
<a href="#_5-python-bridge-mechanism">Python Bridge</a> ·
|
||||
<a href="#_6-screenshot-analyze-act-loop">Interaction Loop</a> ·
|
||||
<a href="#_7-key-source-file-index">Source File Index</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 1. Patch Environment Overview
|
||||
|
||||
The original Claude Code's Computer Use feature (internal codename **Chicago**) depends on three components that are not publicly available:
|
||||
|
||||
| Component | Purpose | Availability |
|
||||
|-----------|---------|-------------|
|
||||
| `@ant/computer-use-swift` | Screenshots, display enumeration | Anthropic private npm package |
|
||||
| `@ant/computer-use-input` | Mouse/keyboard simulation | Anthropic private npm package |
|
||||
| GrowthBook remote config | Feature flags, kill switch | Anthropic internal service |
|
||||
|
||||
Our approach: **preserve the original MCP tool definitions and security mechanisms, only replace the execution layer and feature flag controls**.
|
||||
|
||||
|
||||
|
||||
### What We Changed
|
||||
|
||||
```
|
||||
Original Claude Code Claude Code Haha (Patched)
|
||||
──────────────────── ─────────────────────────
|
||||
@ant/computer-use-swift ──replaced──→ Python Bridge (mac_helper.py)
|
||||
@ant/computer-use-input ──replaced──→ pyautogui + pyobjc
|
||||
GrowthBook feature flags ──bypassed──→ gates.ts hardcoded return true
|
||||
Subscription check (Max/Pro) ──bypassed──→ getChicagoEnabled() = true
|
||||
Build macro CHICAGO_MCP ──replaced──→ true
|
||||
isDefaultDisabledBuiltin ──modified──→ returns false
|
||||
```
|
||||
|
||||
### What We Kept Intact
|
||||
|
||||
- **MCP tool definitions** (24 tools with their schema and parameter validation)
|
||||
- **9-layer security gates** (TCC permissions, app allowlist, permission tiers, pixel validation, etc.)
|
||||
- **App classification system** (191 bundle IDs categorized with permission mappings)
|
||||
- **Session context management** (global lock, screenshot cache, state synchronization)
|
||||
- **Keyboard shortcut blocklist** (system-level dangerous operation interception)
|
||||
|
||||
### Feature Flag Bypass Details
|
||||
|
||||
The original code uses three layers of gating to restrict Computer Use access:
|
||||
|
||||
```typescript
|
||||
// Original code (simplified)
|
||||
function getChicagoEnabled(): boolean {
|
||||
// Layer 1: GrowthBook remote config
|
||||
const config = getDynamicConfig('tengu_malort_pedway')
|
||||
// Layer 2: Subscription check
|
||||
const hasSubscription = hasRequiredSubscription() // Max/Pro
|
||||
// Layer 3: Build-time macro
|
||||
return feature('CHICAGO_MCP') && config.enabled && hasSubscription
|
||||
}
|
||||
```
|
||||
|
||||
Our modification:
|
||||
|
||||
```typescript
|
||||
// gates.ts — our change
|
||||
export function getChicagoEnabled(): boolean {
|
||||
return true // ← all three gate layers bypassed
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: Sub-gates (pixelValidation, mouseAnimation, etc.) still retain the original logic and can be controlled via configuration.
|
||||
|
||||
---
|
||||
|
||||
## 2. Layered Architecture
|
||||
|
||||
Computer Use uses a **6-layer architecture** with clear responsibilities and boundaries:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Layer 1 — MCP Tool Interface │
|
||||
│ tools.ts: 24 tool schemas + parameter validation │
|
||||
│ buildComputerUseTools() → MCP Tool Definition │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 2 — Tool Dispatch & Security Control │
|
||||
│ toolCalls.ts: handleToolCall() + 9 security gates │
|
||||
│ deniedApps.ts: 191 app classifications + permission tiers │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 3 — MCP Server Binding │
|
||||
│ mcpServer.ts: session context + global lock + screenshot │
|
||||
│ bindSessionContext() → per-call overrides │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 4 — CLI Integration │
|
||||
│ wrapper.tsx: permission dialogs + state read/write │
|
||||
│ setup.ts: MCP config initialization │
|
||||
│ gates.ts: feature flags (bypassed) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 5 — Python Bridge IPC [PATCH] │
|
||||
│ pythonBridge.ts: venv mgmt + JSON RPC + error handling │
|
||||
│ callPythonHelper<T>(command, payload) → T │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 6 — Python Runtime Execution [PATCH] │
|
||||
│ mac_helper.py: pyautogui + mss + pyobjc │
|
||||
│ 660 lines of Python implementing all system interactions │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Layers marked **[PATCH]** are our replaced/new code. **All other layers** are preserved from the original Claude Code.
|
||||
|
||||
### Why This Layering?
|
||||
|
||||
| Layer | Source | Reusability |
|
||||
|-------|--------|------------|
|
||||
| Layer 1-2 | `vendor/computer-use-mcp/` | Platform-agnostic, reusable for Electron, Web hosts |
|
||||
| Layer 3 | `vendor/computer-use-mcp/` | Platform-agnostic, standard MCP protocol |
|
||||
| Layer 4 | `utils/computerUse/` | CLI-specific, bound to app state |
|
||||
| Layer 5-6 | `utils/computerUse/` + `runtime/` | macOS-specific, Python implementation |
|
||||
|
||||
---
|
||||
|
||||
## 3. MCP Tool Layer
|
||||
|
||||
### 24 Tools Overview
|
||||
|
||||
Computer Use exposes 24 tools to the model via MCP (Model Context Protocol):
|
||||
|
||||
| Category | Tools | Permission Tier |
|
||||
|----------|-------|----------------|
|
||||
| **Permission** | `request_access`, `list_granted_applications` | None required |
|
||||
| **Screenshot** | `screenshot`, `zoom` | read |
|
||||
| **Mouse Click** | `left_click`, `double_click`, `triple_click` | click |
|
||||
| **Mouse Advanced** | `right_click`, `middle_click`, `left_click_drag` | full |
|
||||
| **Mouse Move** | `mouse_move`, `cursor_position`, `scroll` | click |
|
||||
| **Mouse Low-level** | `left_mouse_down`, `left_mouse_up` | full |
|
||||
| **Keyboard** | `type`, `key`, `hold_key` | full |
|
||||
| **Application** | `open_application`, `switch_display` | full |
|
||||
| **Clipboard** | `read_clipboard`, `write_clipboard` | full |
|
||||
| **Batch** | `computer_batch` | Inherits from sub-operations |
|
||||
| **Wait** | `wait` | None required |
|
||||
|
||||
### Coordinate System
|
||||
|
||||
The model interacts with the screen through two coordinate modes:
|
||||
|
||||
```
|
||||
pixels mode (default):
|
||||
Model sees screenshot size (1176 x 784)
|
||||
Model outputs coordinate [588, 392]
|
||||
scaleCoord() conversion:
|
||||
x_logical = (588 * displayWidth / 1176) + originX
|
||||
y_logical = (392 * displayHeight / 784) + originY
|
||||
|
||||
normalized_0_100 mode:
|
||||
Model outputs coordinate [50, 50] (percentage)
|
||||
scaleCoord() conversion:
|
||||
x_logical = (50 / 100) * displayWidth + originX
|
||||
y_logical = (50 / 100) * displayHeight + originY
|
||||
```
|
||||
|
||||
Screenshot dimensions are calculated by `imageResize.ts` to ensure:
|
||||
- Long edge <= 1568 pixels
|
||||
- Token budget <= 1568 (vision encoder at 28px/token)
|
||||
- Aspect ratio preserved
|
||||
|
||||
### App Classification System
|
||||
|
||||
`deniedApps.ts` precisely classifies 191 applications:
|
||||
|
||||
**Browsers** (55 bundle IDs) -> tier `read`
|
||||
```
|
||||
Safari, Chrome, Firefox, Arc, Edge, Opera, Brave, Vivaldi...
|
||||
Reason: browser operations should use Chrome MCP, not blind clicking
|
||||
```
|
||||
|
||||
**Terminals** (102 bundle IDs) -> tier `click`
|
||||
```
|
||||
Terminal, iTerm2, VS Code, Cursor, JetBrains IDEs, Xcode...
|
||||
Reason: terminal operations should use Bash Tool, limited to click only
|
||||
```
|
||||
|
||||
**Trading** (34 bundle IDs) -> tier `read`
|
||||
```
|
||||
Webull, Fidelity, Interactive Brokers, Binance, Kraken...
|
||||
Reason: financial operations are extremely high-risk, screenshot only
|
||||
```
|
||||
|
||||
**Completely Blocked** (policy deny list):
|
||||
```
|
||||
Netflix, Spotify, Apple Music, Kindle...
|
||||
Reason: copyright compliance, rejected without permission dialog
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Security Gate System
|
||||
|
||||
Every input action (click, keyboard, drag) must pass through **9 security gates** before execution:
|
||||
|
||||
|
||||
|
||||
### Gate Details
|
||||
|
||||
#### Gate 1: Kill Switch
|
||||
```typescript
|
||||
if (adapter.isDisabled()) return errorResult("Computer Use is disabled")
|
||||
```
|
||||
Reads `getChicagoEnabled()` — always returns `true` in patched version.
|
||||
|
||||
#### Gate 2: TCC Permission Check
|
||||
```typescript
|
||||
await adapter.ensureOsPermissions()
|
||||
// → Python: check_permissions
|
||||
// → Accessibility: osascript "tell System Events..."
|
||||
// → Screen Recording: CGDisplayCaptureDisplay()
|
||||
```
|
||||
Reports error if macOS Accessibility or Screen Recording permissions are missing.
|
||||
|
||||
#### Gate 3: Global Mutex Lock
|
||||
```typescript
|
||||
await tryAcquireComputerUseLock(sessionId)
|
||||
// File lock: ~/.claude/computer-use.lock
|
||||
// JSON: { sessionId, pid, acquiredAt }
|
||||
```
|
||||
Ensures only one Claude session can control the computer at a time. Supports stale PID recovery.
|
||||
|
||||
#### Gate 4: Hide Non-Allowlisted Apps
|
||||
```typescript
|
||||
await executor.prepareForAction(allowlistBundleIds)
|
||||
// Hide all app windows not in the allowlist
|
||||
// Ensures screenshots only contain authorized apps
|
||||
```
|
||||
|
||||
#### Gate 5: Frontmost App Check
|
||||
```typescript
|
||||
const frontmost = await executor.getFrontmostApp()
|
||||
if (!allowlist.includes(frontmost.bundleId)) {
|
||||
return errorResult("Application not authorized")
|
||||
}
|
||||
```
|
||||
Even after passing the allowlist, verifies the current foreground app is authorized.
|
||||
|
||||
#### Gate 6: Permission Tier Check
|
||||
|
||||
Three-tier permission model:
|
||||
|
||||
| Tier | Allowed Operations | Prohibited Operations |
|
||||
|------|-------------------|----------------------|
|
||||
| `read` | Screenshot viewing | Any input action |
|
||||
| `click` | Left click, scroll | Right-click, drag, keyboard input |
|
||||
| `full` | All operations | No restrictions |
|
||||
|
||||
```typescript
|
||||
function tierSatisfies(tier: CuAppPermTier, required: ActionKind): boolean {
|
||||
const order = { read: 0, click: 1, full: 2 }
|
||||
return order[tier] >= order[required]
|
||||
}
|
||||
```
|
||||
|
||||
**Anti-subversion**: If permissions are insufficient, the response includes a `TIER_ANTI_SUBVERSION` hint to prevent the model from bypassing restrictions via AppleScript or System Events.
|
||||
|
||||
#### Gate 7: Clipboard Guard
|
||||
|
||||
**Threat model**:
|
||||
```
|
||||
1. Agent calls write_clipboard("rm -rf /")
|
||||
2. Switches to Terminal (click-tier allows clicking)
|
||||
3. Model clicks Terminal's paste button
|
||||
4. Malicious command executed
|
||||
```
|
||||
|
||||
**Protection**:
|
||||
```
|
||||
When click-tier app becomes frontmost:
|
||||
→ Save current clipboard content (stash)
|
||||
→ Clear clipboard
|
||||
→ Re-clear after each operation
|
||||
|
||||
When non-click-tier app becomes frontmost:
|
||||
→ Restore original clipboard content
|
||||
```
|
||||
|
||||
#### Gate 8: Pixel Validation (Staleness Guard)
|
||||
|
||||
```
|
||||
Last screenshot Current actual screen
|
||||
┌────────────┐ ┌────────────┐
|
||||
│ Button A │ │ Dialog │ ← UI has changed
|
||||
│ [756,342] │ │ Confirm? │
|
||||
└────────────┘ └────────────┘
|
||||
|
||||
Validation: sample 9x9 pixel grid at [756,342]
|
||||
→ Compare last screenshot vs live screenshot pixels
|
||||
→ Different → reject click + prompt to re-screenshot
|
||||
→ Same → allow click
|
||||
```
|
||||
|
||||
**Note**: In patched version, `pixelValidation` is off by default (`hostAdapter.cropRawPatch()` returns null).
|
||||
|
||||
#### Gate 9: System Shortcut Interception
|
||||
|
||||
`keyBlocklist.ts` blocks dangerous shortcuts:
|
||||
|
||||
| Shortcut | Dangerous Action |
|
||||
|----------|-----------------|
|
||||
| `Cmd+Q` | Quit application |
|
||||
| `Shift+Cmd+Q` | Log out |
|
||||
| `Option+Cmd+Esc` | Force quit dialog |
|
||||
| `Cmd+Tab` | App switcher |
|
||||
| `Cmd+Space` | Spotlight |
|
||||
| `Ctrl+Cmd+Q` | Lock screen |
|
||||
|
||||
---
|
||||
|
||||
## 5. Python Bridge Mechanism
|
||||
|
||||
|
||||
|
||||
### Architecture Design
|
||||
|
||||
Original Claude Code uses compiled Swift native modules (.node NAPI plugins) to directly call macOS APIs. We replaced this layer with **Python subprocess + JSON RPC**:
|
||||
|
||||
```
|
||||
TypeScript (Bun Runtime) Python (venv)
|
||||
┌────────────────────┐ ┌────────────────────┐
|
||||
│ executor.ts │ │ mac_helper.py │
|
||||
│ │ execFile() │ │
|
||||
│ callPythonHelper │ ──────────────→ │ main() │
|
||||
│ ('click', │ command + │ ├─ parse argv │
|
||||
│ {x:756,y:342}) │ --payload JSON │ ├─ dispatch() │
|
||||
│ │ │ └─ click() │
|
||||
│ ← JSON.parse ──── │ ←────────────── │ pyautogui │
|
||||
│ {ok:true, │ stdout JSON │ │
|
||||
│ result:true} │ │ json_output(...) │
|
||||
└────────────────────┘ └────────────────────┘
|
||||
```
|
||||
|
||||
### Bootstrap Flow
|
||||
|
||||
First call to `callPythonHelper()` automatically completes environment setup:
|
||||
|
||||
```
|
||||
ensureBootstrapped()
|
||||
│
|
||||
├─ Check if .runtime/venv/bin/python3 exists
|
||||
│ └─ Missing → python3 -m venv .runtime/venv/
|
||||
│
|
||||
├─ Check if pip is available
|
||||
│ └─ Missing → python3 -m ensurepip --upgrade
|
||||
│
|
||||
├─ Compute SHA256 of runtime/requirements.txt
|
||||
│ └─ Compare with .runtime/requirements.sha256
|
||||
│ └─ Different → pip install -r requirements.txt
|
||||
│ Write new SHA256 hash
|
||||
│ └─ Same → skip installation
|
||||
│
|
||||
└─ Ready, return venv Python path
|
||||
```
|
||||
|
||||
**Dependencies** (`runtime/requirements.txt`):
|
||||
|
||||
| Library | Purpose |
|
||||
|---------|---------|
|
||||
| `mss` | High-performance screen capture |
|
||||
| `Pillow` | JPEG encoding and image processing |
|
||||
| `pyautogui` | Mouse click, keyboard input |
|
||||
| `pyobjc-core` | macOS Objective-C bridge |
|
||||
| `pyobjc-framework-Cocoa` | NSWorkspace (app management), NSPasteboard (clipboard) |
|
||||
| `pyobjc-framework-Quartz` | CGDisplay (monitors), CGWindow (window list) |
|
||||
|
||||
### Command Mapping
|
||||
|
||||
`mac_helper.py` (660 lines) implements the following commands:
|
||||
|
||||
| Command | Python Implementation | Return Value |
|
||||
|---------|----------------------|-------------|
|
||||
| `screenshot` | `mss.grab()` + `PIL.Image` JPEG encoding | `{base64, width, height, displayWidth, displayHeight}` |
|
||||
| `zoom` | `mss.grab(region)` region capture | `{base64, width, height}` |
|
||||
| `click` | `pyautogui.moveTo()` + `pyautogui.click()` | `true` |
|
||||
| `key` | `pyautogui.hotkey()` / `pyautogui.press()` | `true` |
|
||||
| `type` | `pyautogui.write(interval=0.008)` | `true` |
|
||||
| `drag` | `pyautogui.dragTo(duration=0.2)` | `true` |
|
||||
| `scroll` | `pyautogui.scroll()` / `pyautogui.hscroll()` | `true` |
|
||||
| `hold_key` | `pyautogui.keyDown()` + `sleep` + `pyautogui.keyUp()` | `true` |
|
||||
| `frontmost_app` | `NSWorkspace.frontmostApplication()` | `{bundleId, displayName}` |
|
||||
| `list_displays` | `CGGetActiveDisplayList()` + `CGDisplayBounds()` | `[DisplayGeometry...]` |
|
||||
| `open_app` | `NSWorkspace.launchApplicationAtURL_options_` | `void` |
|
||||
| `read_clipboard` | `NSPasteboard.stringForType_()` | `string` |
|
||||
| `write_clipboard` | `NSPasteboard.setString_forType_()` | `void` |
|
||||
| `check_permissions` | `osascript` + `CGDisplayCaptureDisplay` | `{accessibility, screenRecording}` |
|
||||
|
||||
### Error Handling
|
||||
|
||||
```python
|
||||
# mac_helper.py unified error handling
|
||||
def main():
|
||||
try:
|
||||
result = dispatch(command, payload)
|
||||
json_output({"ok": True, "result": result})
|
||||
except Exception as e:
|
||||
error_output({"ok": False, "error": {"message": str(e)}})
|
||||
```
|
||||
|
||||
TypeScript side:
|
||||
```typescript
|
||||
// pythonBridge.ts
|
||||
const parsed = JSON.parse(stdout)
|
||||
if (!parsed.ok) {
|
||||
throw new Error(parsed.error.message) // → MCP tool error
|
||||
}
|
||||
return parsed.result
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Screenshot-Analyze-Act Loop
|
||||
|
||||
A complete Computer Use interaction consists of multiple **screenshot-analyze-act** cycles:
|
||||
|
||||
### Typical Interaction Flow
|
||||
|
||||
```
|
||||
User: "Open NetEase Music and search for a song"
|
||||
|
||||
┌─ Cycle 1: Discover and open application ─────────────────┐
|
||||
│ │
|
||||
│ Step 1: request_access │
|
||||
│ → Permission dialog, user authorizes allowed apps │
|
||||
│ → Set allowedApps, grantFlags │
|
||||
│ │
|
||||
│ Step 2: screenshot │
|
||||
│ → Full screen capture → JPEG encode → base64 │
|
||||
│ → Cache screenshot dimensions (lastScreenshotDims) │
|
||||
│ → Return to model │
|
||||
│ │
|
||||
│ Step 3: Model analyzes screenshot │
|
||||
│ → "NetEase Music not on desktop, need to open it" │
|
||||
│ → Decides to call open_application │
|
||||
│ │
|
||||
│ Step 4: open_application("com.netease.163music") │
|
||||
│ → Gates 1-9 all pass │
|
||||
│ → Python: NSWorkspace.launchApplicationAtURL_() │
|
||||
│ │
|
||||
└───────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─ Cycle 2: Locate search box ─────────────────────────────┐
|
||||
│ │
|
||||
│ Step 5: screenshot │
|
||||
│ → Full screen (app now open) │
|
||||
│ → Update lastScreenshotDims │
|
||||
│ │
|
||||
│ Step 6: Model analyzes screenshot │
|
||||
│ → Vision identifies search box at (756, 342) │
|
||||
│ → Decides to click search box │
|
||||
│ │
|
||||
│ Step 7: left_click({coordinate: [756, 342]}) │
|
||||
│ → Gate 4: Hide non-allowlisted apps │
|
||||
│ → Gate 5: Frontmost is NetEase Music ✓ │
|
||||
│ → Gate 6: tier=full >= click ✓ │
|
||||
│ → scaleCoord(756, 342) → screen coordinates │
|
||||
│ → Python: pyautogui.click(x_logical, y_logical) │
|
||||
│ │
|
||||
└───────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─ Cycle 3: Type search query ─────────────────────────────┐
|
||||
│ │
|
||||
│ Step 8: type({text: "my favorite song"}) │
|
||||
│ → Gate 6: tier=full >= keyboard ✓ │
|
||||
│ → Python: pyautogui.write("...", interval=0.008) │
|
||||
│ │
|
||||
│ Step 9: screenshot │
|
||||
│ → Confirm search results appeared │
|
||||
│ │
|
||||
│ Step 10: left_click({coordinate: [...]}) │
|
||||
│ → Click target song │
|
||||
│ │
|
||||
└───────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Coordinate Conversion
|
||||
|
||||
**What the model sees** vs **actual screen** are different coordinate spaces:
|
||||
|
||||
```
|
||||
Physical screen (2560 x 1600, Retina 2x)
|
||||
├─ Logical size: 1280 x 800
|
||||
└─ Physical pixels: 2560 x 1600
|
||||
|
||||
After imageResize:
|
||||
├─ Scaled size: 1176 x 735 (≤1568px budget)
|
||||
└─ This is the screenshot size the model "sees"
|
||||
|
||||
Model outputs coordinate: [588, 368] (in image space)
|
||||
|
||||
scaleCoord conversion:
|
||||
x_logical = (588 / 1176) * 1280 + originX = 640
|
||||
y_logical = (368 / 735) * 800 + originY = 400
|
||||
|
||||
Python executes:
|
||||
pyautogui.moveTo(640, 400) ← logical coords (macOS handles Retina)
|
||||
```
|
||||
|
||||
### Screenshot Cache & State Sync
|
||||
|
||||
```
|
||||
bindSessionContext closure
|
||||
│
|
||||
├─ lastScreenshot (in-memory)
|
||||
│ ├─ base64: JPEG data (for pixel validation)
|
||||
│ ├─ width/height: model-visible dimensions
|
||||
│ └─ displayWidth/displayHeight/originX/originY: display geometry
|
||||
│
|
||||
└─ AppState.computerUseMcpState (persisted)
|
||||
├─ allowedApps: AppGrant[] — authorized app list
|
||||
├─ grantFlags: {...} — clipboard/system shortcut permissions
|
||||
├─ selectedDisplayId?: number — selected display
|
||||
├─ lastScreenshotDims?: {...} — screenshot geometry (survives restart)
|
||||
└─ hiddenDuringTurn?: Set<string> — apps hidden this turn
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Key Source File Index
|
||||
|
||||
### vendor/computer-use-mcp/ (Original Code Layer)
|
||||
|
||||
| File | Lines | Responsibility |
|
||||
|------|-------|---------------|
|
||||
| `types.ts` | 622 | Permission model, session context, all type definitions |
|
||||
| `tools.ts` | 707 | 24 MCP tool schemas and parameter validation |
|
||||
| `toolCalls.ts` | 1600+ | **Core**: tool dispatch, 9 security gates, permission flow |
|
||||
| `deniedApps.ts` | 554 | 191 app classifications (browser/terminal/trading) and permission mappings |
|
||||
| `sentinelApps.ts` | 44 | Sensitive app warning labels (shell/filesystem/system_settings) |
|
||||
| `mcpServer.ts` | 314 | MCP server factory, session context binding, global lock |
|
||||
| `pixelCompare.ts` | 172 | Click target pixel validation (staleness guard) |
|
||||
| `imageResize.ts` | 109 | Screenshot dimension calculation (API image transcoder) |
|
||||
| `keyBlocklist.ts` | 154 | System shortcut interception (Cmd+Q, Cmd+Tab, etc.) |
|
||||
| `executor.ts` | 101 | ComputerExecutor interface definition |
|
||||
| `subGates.ts` | 20 | Feature flag sub-gate presets |
|
||||
|
||||
### utils/computerUse/ (CLI Adaptation Layer)
|
||||
|
||||
| File | Lines | Responsibility | Patched? |
|
||||
|------|-------|---------------|----------|
|
||||
| `executor.ts` | 231 | ComputerExecutor Python bridge implementation | Yes, rewritten |
|
||||
| `pythonBridge.ts` | 111 | Python subprocess management, venv bootstrap, JSON RPC | Yes, new |
|
||||
| `hostAdapter.ts` | 54 | HostAdapter implementation (permission checks, flag reading) | Partially |
|
||||
| `gates.ts` | 51 | GrowthBook feature flags (`getChicagoEnabled` bypass) | Yes, modified |
|
||||
| `wrapper.tsx` | 300+ | Session context construction, permission dialogs, lock management | Unchanged |
|
||||
| `setup.ts` | 54 | MCP config initialization | Unchanged |
|
||||
| `computerUseLock.ts` | 216 | Global file lock (`~/.claude/computer-use.lock`) | Unchanged |
|
||||
| `common.ts` | 62 | Constants (server name, bundle ID) | Unchanged |
|
||||
| `cleanup.ts` | — | Turn-end cleanup (app restore, clipboard restore) | Unchanged |
|
||||
| `toolRendering.tsx` | — | Tool result UI rendering | Unchanged |
|
||||
|
||||
### runtime/ (Python Runtime)
|
||||
|
||||
| File | Lines | Responsibility | Patched? |
|
||||
|------|-------|---------------|----------|
|
||||
| `mac_helper.py` | 660 | All system interactions in Python | Yes, new |
|
||||
| `requirements.txt` | 6 | Python dependency declarations | Yes, new |
|
||||
|
||||
---
|
||||
|
||||
## 8. Design Trade-offs
|
||||
|
||||
### Why Python Bridge?
|
||||
|
||||
| Dimension | Native Swift (.node) | Python Bridge |
|
||||
|-----------|---------------------|---------------|
|
||||
| **Performance** | ~0ms (in-process call) | ~50-100ms (subprocess startup) |
|
||||
| **Readability** | Not readable after compilation | 660 lines of clear Python |
|
||||
| **Modifiability** | Requires Swift build environment | Edit .py files directly |
|
||||
| **Dependencies** | Specific Bun version NAPI | Any Python 3.8+ |
|
||||
| **Cross-platform** | macOS only | pyautogui/mss are natively cross-platform |
|
||||
| **User experience** | Imperceptible | Imperceptible (model thinking takes 2-5s) |
|
||||
|
||||
**Conclusion**: The 50-100ms extra latency is completely negligible in Computer Use scenarios — the model typically takes 2-5 seconds for screenshot analysis and decision-making, so users won't notice the additional 100ms for underlying operations.
|
||||
|
||||
### Approaches We Tried But Abandoned
|
||||
|
||||
**Approach 1: Extract native .node modules**
|
||||
- Successfully extracted `computer-use-swift.node` (ARM64 424KB) from the Claude Code binary
|
||||
- Synchronous methods worked, but **Swift async method continuations never resumed**
|
||||
- Root cause: .node files compiled for Claude Code's built-in Bun, incompatible with user's Bun version
|
||||
|
||||
**Approach 2: Empty stub packages**
|
||||
- Code compiled but all operations threw errors — no actual execution capability
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Computer Use Guide](./computer-use.md) — Usage, quick start, environment variables
|
||||
- [Source Fixes](/en/reference/fixes) — Detailed records of other fixes and patches
|
||||
BIN
docs/en/features/images/01-computer-use-architecture.jpg
Normal file
BIN
docs/en/features/images/01-computer-use-architecture.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 888 KiB |
BIN
docs/en/features/images/02-computer-use-security-gates.jpg
Normal file
BIN
docs/en/features/images/02-computer-use-security-gates.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 523 KiB |
BIN
docs/en/features/images/03-computer-use-python-bridge.jpg
Normal file
BIN
docs/en/features/images/03-computer-use-python-bridge.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 912 KiB |
BIN
docs/en/features/images/04-computer-use-patch.jpg
Normal file
BIN
docs/en/features/images/04-computer-use-patch.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 800 KiB |
@ -3,7 +3,7 @@
|
||||
> Let Claude Code remember who you are, what you prefer, and what's happening in your project across sessions.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-what-is-the-memory-system">Memory System</a> · <a href="#2-four-memory-types">Four Memory Types</a> · <a href="#3-how-to-trigger-memory-saving">Trigger Saving</a> · <a href="#4-where-are-memories-stored">Storage Location</a> · <a href="#5-how-to-manage-memories">Manage Memories</a> · <a href="#6-memory-lifecycle">Lifecycle</a> · <a href="#7-quick-reference">Quick Reference</a>
|
||||
<a href="#_1-what-is-the-memory-system">Memory System</a> · <a href="#_2-four-memory-types">Four Memory Types</a> · <a href="#_3-how-to-trigger-memory-saving">Trigger Saving</a> · <a href="#_4-where-are-memories-stored">Storage Location</a> · <a href="#_5-how-to-manage-memories">Manage Memories</a> · <a href="#_6-memory-lifecycle">Lifecycle</a> · <a href="#_7-quick-reference">Quick Reference</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> From system prompt injection to background auto-extraction, dissecting every technical detail of the memory system.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-overall-architecture">Architecture</a> · <a href="#2-path-resolution-system">Path Resolution</a> · <a href="#3-system-prompt-injection">Prompt Injection</a> · <a href="#4-automatic-memory-extraction">Auto-Extraction</a> · <a href="#5-intelligent-memory-retrieval">Intelligent Retrieval</a> · <a href="#6-memory-scanning-in-detail">Memory Scanning</a> · <a href="#7-agent-memory">Agent Memory</a> · <a href="#8-team-memory-sync">Team Sync</a> · <a href="#9-key-constants-quick-reference">Constants</a> · <a href="#10-data-flow-overview">Data Flow</a>
|
||||
<a href="#_1-overall-architecture">Architecture</a> · <a href="#_2-path-resolution-system">Path Resolution</a> · <a href="#_3-system-prompt-injection">Prompt Injection</a> · <a href="#_4-automatic-memory-extraction">Auto-Extraction</a> · <a href="#_5-intelligent-memory-retrieval">Intelligent Retrieval</a> · <a href="#_6-memory-scanning-in-detail">Memory Scanning</a> · <a href="#_7-agent-memory">Agent Memory</a> · <a href="#_8-team-memory-sync">Team Sync</a> · <a href="#_9-key-constants-quick-reference">Constants</a> · <a href="#_10-data-flow-overview">Data Flow</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> Claude "dreams" -- silently reviewing recent sessions in the background to consolidate, update, and prune memories, much like the human brain organizes memories during sleep.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-what-is-autodream">AutoDream</a> · <a href="#2-trigger-conditions">Trigger Conditions</a> · <a href="#3-four-phase-consolidation-process">Consolidation Process</a> · <a href="#4-security-restrictions">Security</a> · <a href="#5-ui-presentation">UI</a> · <a href="#6-configuration-and-toggles">Configuration</a> · <a href="#7-relationship-with-extractmemories">Comparison</a> · <a href="#8-source-code-navigation">Source Code</a>
|
||||
<a href="#_1-what-is-autodream">AutoDream</a> · <a href="#_2-trigger-conditions">Trigger Conditions</a> · <a href="#_3-four-phase-consolidation-process">Consolidation Process</a> · <a href="#_4-security-restrictions">Security</a> · <a href="#_5-ui-presentation">UI</a> · <a href="#_6-configuration-and-toggles">Configuration</a> · <a href="#_7-relationship-with-extractmemories">Comparison</a> · <a href="#_8-source-code-navigation">Source Code</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> Skills are the extensible capability engine of Claude Code, allowing you to define custom automated workflows using Markdown files.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-what-are-skills">What Are Skills</a> · <a href="#2-six-skill-sources">Six Sources</a> · <a href="#3-skill-definition-format">Definition Format</a> · <a href="#4-invocation-methods">Invocation</a> · <a href="#5-execution-context">Execution Context</a> · <a href="#6-conditional-activation">Conditional Activation</a> · <a href="#7-permission-control">Permissions</a> · <a href="#8-quick-reference">Quick Reference</a>
|
||||
<a href="#_1-what-are-skills">What Are Skills</a> · <a href="#_2-six-skill-sources">Six Sources</a> · <a href="#_3-skill-definition-format">Definition Format</a> · <a href="#_4-invocation-methods">Invocation</a> · <a href="#_5-execution-context">Execution Context</a> · <a href="#_6-conditional-activation">Conditional Activation</a> · <a href="#_7-permission-control">Permissions</a> · <a href="#_8-quick-reference">Quick Reference</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> A deep dive into how Skills are discovered, loaded, injected, executed, and managed.
|
||||
|
||||
<p align="center">
|
||||
<a href="#1-overall-architecture">Architecture</a> · <a href="#2-skill-discovery-and-loading">Discovery & Loading</a> · <a href="#3-frontmatter-parsing">Frontmatter Parsing</a> · <a href="#4-skill-injection-into-conversations">Injection</a> · <a href="#5-skilltool-execution-engine">Execution Engine</a> · <a href="#6-fork-sub-agent-execution">Fork Execution</a> · <a href="#7-conditional-activation-and-dynamic-discovery">Conditional Activation</a> · <a href="#8-hook-integration">Hook Integration</a> · <a href="#9-permission-system">Permission System</a> · <a href="#10-complete-lifecycle">Complete Lifecycle</a> · <a href="#11-source-code-index">Source Index</a>
|
||||
<a href="#_1-overall-architecture">Architecture</a> · <a href="#_2-skill-discovery-and-loading">Discovery & Loading</a> · <a href="#_3-frontmatter-parsing">Frontmatter Parsing</a> · <a href="#_4-skill-injection-into-conversations">Injection</a> · <a href="#_5-skilltool-execution-engine">Execution Engine</a> · <a href="#_6-fork-sub-agent-execution">Fork Execution</a> · <a href="#_7-conditional-activation-and-dynamic-discovery">Conditional Activation</a> · <a href="#_8-hook-integration">Hook Integration</a> · <a href="#_9-permission-system">Permission System</a> · <a href="#_10-complete-lifecycle">Complete Lifecycle</a> · <a href="#_11-source-code-index">Source Index</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
607
docs/features/computer-use-architecture.md
Normal file
607
docs/features/computer-use-architecture.md
Normal file
@ -0,0 +1,607 @@
|
||||
# Computer Use 架构深度解析
|
||||
|
||||
> 深入解析 Computer Use 功能的底层实现：从 MCP 工具定义到 Python Bridge，从 9 层安全关卡到灰度控制绕过。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一、补丁环境总览">补丁环境</a> ·
|
||||
<a href="#二、分层架构">分层架构</a> ·
|
||||
<a href="#三、mcp-工具层">MCP 工具层</a> ·
|
||||
<a href="#四、安全关卡体系">安全关卡</a> ·
|
||||
<a href="#五、python-bridge-机制">Python Bridge</a> ·
|
||||
<a href="#六、截图-分析-操作闭环">交互闭环</a> ·
|
||||
<a href="#七、关键源文件索引">源文件索引</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 一、补丁环境总览
|
||||
|
||||
Claude Code 原版的 Computer Use 功能（内部代号 **Chicago**）依赖三个不可公开获取的组件：
|
||||
|
||||
| 组件 | 作用 | 获取方式 |
|
||||
|------|------|----------|
|
||||
| `@ant/computer-use-swift` | 屏幕截图、显示器枚举 | Anthropic 私有 npm 包 |
|
||||
| `@ant/computer-use-input` | 鼠标/键盘模拟 | Anthropic 私有 npm 包 |
|
||||
| GrowthBook 远程配置 | 灰度控制、功能开关 | Anthropic 内部服务 |
|
||||
|
||||
我们的方案是：**保留原始 MCP 工具定义和安全机制不变，仅替换底层执行层和灰度控制**。
|
||||
|
||||
|
||||
|
||||
### 我们做了什么
|
||||
|
||||
```
|
||||
原始 Claude Code Claude Code Haha (补丁版)
|
||||
───────────────── ─────────────────────────
|
||||
@ant/computer-use-swift ──替换为──→ Python Bridge (mac_helper.py)
|
||||
@ant/computer-use-input ──替换为──→ pyautogui + pyobjc
|
||||
GrowthBook 灰度控制 ──绕过──→ gates.ts 硬编码 return true
|
||||
订阅检查 (Max/Pro) ──绕过──→ getChicagoEnabled() = true
|
||||
编译宏 CHICAGO_MCP ──替换为──→ true
|
||||
isDefaultDisabledBuiltin ──修改──→ 返回 false
|
||||
```
|
||||
|
||||
### 我们没有改什么
|
||||
|
||||
- **MCP 工具定义**（24 个工具的 schema 和参数校验）
|
||||
- **9 层安全关卡**（TCC 权限、应用白名单、权限等级、像素验证等）
|
||||
- **应用分类系统**（191 个 bundle ID 的分类和权限映射）
|
||||
- **会话上下文管理**（全局锁、截图缓存、状态同步）
|
||||
- **键盘快捷键阻止列表**（系统级危险操作拦截）
|
||||
|
||||
### 灰度绕过细节
|
||||
|
||||
原始代码通过三层门控限制 Computer Use 的访问：
|
||||
|
||||
```typescript
|
||||
// 原始代码（简化）
|
||||
function getChicagoEnabled(): boolean {
|
||||
// 层1：GrowthBook 远程配置
|
||||
const config = getDynamicConfig('tengu_malort_pedway')
|
||||
// 层2：订阅检查
|
||||
const hasSubscription = hasRequiredSubscription() // Max/Pro
|
||||
// 层3：编译时宏
|
||||
return feature('CHICAGO_MCP') && config.enabled && hasSubscription
|
||||
}
|
||||
```
|
||||
|
||||
我们的处理：
|
||||
|
||||
```typescript
|
||||
// gates.ts — 我们的修改
|
||||
export function getChicagoEnabled(): boolean {
|
||||
return true // ← 三层门控全部绕过
|
||||
}
|
||||
```
|
||||
|
||||
**注意**：子开关（pixelValidation、mouseAnimation 等）仍然保留原始逻辑，可通过配置控制。
|
||||
|
||||
---
|
||||
|
||||
## 二、分层架构
|
||||
|
||||
Computer Use 采用 **6 层架构**，每层职责清晰、边界明确：
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Layer 1 — MCP 工具接口层 │
|
||||
│ tools.ts: 24 个工具 schema + 参数校验 │
|
||||
│ buildComputerUseTools() → MCP Tool Definition │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 2 — 工具调度与安全控制层 │
|
||||
│ toolCalls.ts: handleToolCall() + 9 层安全关卡 │
|
||||
│ deniedApps.ts: 191 个应用分类 + 权限等级 │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 3 — MCP 服务器绑定层 │
|
||||
│ mcpServer.ts: 会话上下文 + 全局锁 + 截图缓存 │
|
||||
│ bindSessionContext() → per-call overrides │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 4 — CLI 集成层 │
|
||||
│ wrapper.tsx: 权限对话框 + 状态读写 │
|
||||
│ setup.ts: MCP 配置初始化 │
|
||||
│ gates.ts: 灰度控制（已绕过） │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 5 — Python Bridge 进程通信层 [补丁] │
|
||||
│ pythonBridge.ts: venv 管理 + JSON RPC + 错误处理 │
|
||||
│ callPythonHelper<T>(command, payload) → T │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Layer 6 — Python 运行时执行层 [补丁] │
|
||||
│ mac_helper.py: pyautogui + mss + pyobjc │
|
||||
│ 660 行 Python 实现所有系统交互 │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**标 `[补丁]` 的层**是我们替换/新增的代码，**其余层完全保留**原始 Claude Code 的逻辑。
|
||||
|
||||
### 为什么这样分层？
|
||||
|
||||
| 层 | 来源 | 可复用性 |
|
||||
|----|------|----------|
|
||||
| Layer 1-2 | `vendor/computer-use-mcp/` | 平台无关，可用于 Electron、Web 等宿主 |
|
||||
| Layer 3 | `vendor/computer-use-mcp/` | 平台无关，MCP 标准协议 |
|
||||
| Layer 4 | `utils/computerUse/` | CLI 专用，绑定应用状态 |
|
||||
| Layer 5-6 | `utils/computerUse/` + `runtime/` | macOS 专用，Python 实现 |
|
||||
|
||||
---
|
||||
|
||||
## 三、MCP 工具层
|
||||
|
||||
### 24 个工具一览
|
||||
|
||||
Computer Use 通过 MCP（Model Context Protocol）暴露 24 个工具给模型：
|
||||
|
||||
| 类别 | 工具 | 权限等级 |
|
||||
|------|------|----------|
|
||||
| **权限** | `request_access`, `list_granted_applications` | 无需权限 |
|
||||
| **截屏** | `screenshot`, `zoom` | read |
|
||||
| **鼠标点击** | `left_click`, `double_click`, `triple_click` | click |
|
||||
| **鼠标高级** | `right_click`, `middle_click`, `left_click_drag` | full |
|
||||
| **鼠标移动** | `mouse_move`, `cursor_position`, `scroll` | click |
|
||||
| **鼠标底层** | `left_mouse_down`, `left_mouse_up` | full |
|
||||
| **键盘** | `type`, `key`, `hold_key` | full |
|
||||
| **应用** | `open_application`, `switch_display` | full |
|
||||
| **剪贴板** | `read_clipboard`, `write_clipboard` | full |
|
||||
| **批量** | `computer_batch` | 继承子操作 |
|
||||
| **等待** | `wait` | 无需权限 |
|
||||
|
||||
### 坐标系统
|
||||
|
||||
模型通过两种坐标模式与屏幕交互：
|
||||
|
||||
```
|
||||
pixels 模式（默认）：
|
||||
模型看到截图尺寸 (1176 x 784)
|
||||
模型输出坐标 [588, 392]
|
||||
scaleCoord() 转换：
|
||||
x_logical = (588 * displayWidth / 1176) + originX
|
||||
y_logical = (392 * displayHeight / 784) + originY
|
||||
|
||||
normalized_0_100 模式：
|
||||
模型输出坐标 [50, 50]（百分比）
|
||||
scaleCoord() 转换：
|
||||
x_logical = (50 / 100) * displayWidth + originX
|
||||
y_logical = (50 / 100) * displayHeight + originY
|
||||
```
|
||||
|
||||
截图尺寸经过 `imageResize.ts` 计算，确保：
|
||||
- 长边 ≤ 1568 像素
|
||||
- Token 预算 ≤ 1568（视觉编码器 28px/token）
|
||||
- 保持宽高比
|
||||
|
||||
### 应用分类系统
|
||||
|
||||
`deniedApps.ts` 对 191 个应用进行精细分类：
|
||||
|
||||
**浏览器类**（55 个 bundle ID）→ 权限等级 `read`
|
||||
```
|
||||
Safari, Chrome, Firefox, Arc, Edge, Opera, Brave, Vivaldi...
|
||||
理由：浏览器操作应使用 Chrome MCP，而非盲点击
|
||||
```
|
||||
|
||||
**终端类**（102 个 bundle ID）→ 权限等级 `click`
|
||||
```
|
||||
Terminal, iTerm2, VS Code, Cursor, JetBrains 全家桶, Xcode...
|
||||
理由：终端操作应使用 Bash Tool，限制为只能点击不能输入
|
||||
```
|
||||
|
||||
**交易类**（34 个 bundle ID）→ 权限等级 `read`
|
||||
```
|
||||
Webull, Fidelity, Interactive Brokers, Binance, Kraken...
|
||||
理由：金融操作风险极高，仅允许截图查看
|
||||
```
|
||||
|
||||
**完全禁止**（策略拒绝名单）：
|
||||
```
|
||||
Netflix, Spotify, Apple Music, Kindle...
|
||||
理由：版权合规，不进入权限对话框直接拒绝
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 四、安全关卡体系
|
||||
|
||||
每个输入操作（点击、键盘、拖拽）在执行前需通过 **9 层安全关卡**：
|
||||
|
||||
|
||||
|
||||
### 关卡详解
|
||||
|
||||
#### Gate 1：Kill Switch
|
||||
```typescript
|
||||
if (adapter.isDisabled()) return errorResult("Computer Use is disabled")
|
||||
```
|
||||
读取 `getChicagoEnabled()` — 在补丁版中永远返回 `true`。
|
||||
|
||||
#### Gate 2：TCC 权限检查
|
||||
```typescript
|
||||
await adapter.ensureOsPermissions()
|
||||
// → Python: check_permissions
|
||||
// → Accessibility: osascript "tell System Events..."
|
||||
// → Screen Recording: CGDisplayCaptureDisplay()
|
||||
```
|
||||
如果缺少 macOS Accessibility 或 Screen Recording 权限，直接报错。
|
||||
|
||||
#### Gate 3：全局互斥锁
|
||||
```typescript
|
||||
await tryAcquireComputerUseLock(sessionId)
|
||||
// 文件锁: ~/.claude/computer-use.lock
|
||||
// JSON: { sessionId, pid, acquiredAt }
|
||||
```
|
||||
确保同一时间只有一个 Claude 会话可以控制电脑。支持 stale PID 恢复。
|
||||
|
||||
#### Gate 4：隐藏非白名单应用
|
||||
```typescript
|
||||
await executor.prepareForAction(allowlistBundleIds)
|
||||
// 隐藏所有不在白名单中的应用窗口
|
||||
// 确保截图中只出现授权应用
|
||||
```
|
||||
|
||||
#### Gate 5：前台应用检查
|
||||
```typescript
|
||||
const frontmost = await executor.getFrontmostApp()
|
||||
if (!allowlist.includes(frontmost.bundleId)) {
|
||||
return errorResult("Application not authorized")
|
||||
}
|
||||
```
|
||||
即使通过了白名单，还需确认当前前台应用是已授权的。
|
||||
|
||||
#### Gate 6：权限等级检查
|
||||
|
||||
三级权限模型：
|
||||
|
||||
| Tier | 允许的操作 | 禁止的操作 |
|
||||
|------|-----------|-----------|
|
||||
| `read` | 截图查看 | 任何输入操作 |
|
||||
| `click` | 左键点击、滚动 | 右键、拖拽、键盘输入 |
|
||||
| `full` | 全部操作 | 无限制 |
|
||||
|
||||
```typescript
|
||||
function tierSatisfies(tier: CuAppPermTier, required: ActionKind): boolean {
|
||||
const order = { read: 0, click: 1, full: 2 }
|
||||
return order[tier] >= order[required]
|
||||
}
|
||||
```
|
||||
|
||||
**反绕过机制**：如果权限不足，响应中会附加 `TIER_ANTI_SUBVERSION` 提示，防止模型通过 AppleScript 或 System Events 绕过限制。
|
||||
|
||||
#### Gate 7：剪贴板防护
|
||||
|
||||
**威胁模型**：
|
||||
```
|
||||
1. Agent 调用 write_clipboard("rm -rf /")
|
||||
2. 切换到 Terminal（click-tier 允许点击）
|
||||
3. 模型点击 Terminal 的粘贴按钮
|
||||
4. 恶意命令被执行
|
||||
```
|
||||
|
||||
**防护机制**：
|
||||
```
|
||||
当 click-tier 应用成为前台时：
|
||||
→ 保存当前剪贴板内容（stash）
|
||||
→ 清空剪贴板
|
||||
→ 每次操作后再次清空
|
||||
|
||||
当非 click-tier 应用成为前台时：
|
||||
→ 恢复原始剪贴板内容
|
||||
```
|
||||
|
||||
#### Gate 8：像素验证（Staleness Guard）
|
||||
|
||||
```
|
||||
上次截图 当前实际屏幕
|
||||
┌────────────┐ ┌────────────┐
|
||||
│ 按钮A │ │ 对话框 │ ← UI 已变化
|
||||
│ [756,342] │ │ 确认? │
|
||||
└────────────┘ └────────────┘
|
||||
|
||||
像素验证：在 [756,342] 取 9×9 网格
|
||||
→ 对比上次截图 vs 实时截图的像素
|
||||
→ 不同 → 拒绝点击 + 提示重新截图
|
||||
→ 相同 → 允许点击
|
||||
```
|
||||
|
||||
**注意**：补丁版中 `pixelValidation` 默认关闭（`hostAdapter.cropRawPatch()` 返回 null）。
|
||||
|
||||
#### Gate 9：系统快捷键拦截
|
||||
|
||||
`keyBlocklist.ts` 阻止危险快捷键：
|
||||
|
||||
| 快捷键 | 危险操作 |
|
||||
|--------|---------|
|
||||
| `⌘Q` | 退出应用 |
|
||||
| `⇧⌘Q` | 登出系统 |
|
||||
| `⌥⌘⎋` | 强制退出对话框 |
|
||||
| `⌘Tab` | 应用切换器 |
|
||||
| `⌘Space` | Spotlight |
|
||||
| `⌃⌘Q` | 锁屏 |
|
||||
|
||||
---
|
||||
|
||||
## 五、Python Bridge 机制
|
||||
|
||||
|
||||
|
||||
### 架构设计
|
||||
|
||||
原始 Claude Code 使用编译好的 Swift 原生模块（.node NAPI 插件）直接调用 macOS API。我们用 **Python 子进程 + JSON RPC** 替代了这一层：
|
||||
|
||||
```
|
||||
TypeScript (Bun 运行时) Python (venv)
|
||||
┌────────────────────┐ ┌────────────────────┐
|
||||
│ executor.ts │ │ mac_helper.py │
|
||||
│ │ execFile() │ │
|
||||
│ callPythonHelper │ ──────────────→ │ main() │
|
||||
│ ('click', │ command + │ ├─ 解析 argv │
|
||||
│ {x:756,y:342}) │ --payload JSON │ ├─ dispatch() │
|
||||
│ │ │ └─ click() │
|
||||
│ ← JSON.parse ──── │ ←────────────── │ pyautogui │
|
||||
│ {ok:true, │ stdout JSON │ │
|
||||
│ result:true} │ │ json_output(...) │
|
||||
└────────────────────┘ └────────────────────┘
|
||||
```
|
||||
|
||||
### 启动引导流程
|
||||
|
||||
首次调用 `callPythonHelper()` 时自动完成环境初始化：
|
||||
|
||||
```
|
||||
ensureBootstrapped()
|
||||
│
|
||||
├─ 检查 .runtime/venv/bin/python3 是否存在
|
||||
│ └─ 不存在 → python3 -m venv .runtime/venv/
|
||||
│
|
||||
├─ 检查 pip 是否可用
|
||||
│ └─ 不可用 → python3 -m ensurepip --upgrade
|
||||
│
|
||||
├─ 计算 runtime/requirements.txt 的 SHA256
|
||||
│ └─ 与 .runtime/requirements.sha256 对比
|
||||
│ └─ 不同 → pip install -r requirements.txt
|
||||
│ 写入新的 SHA256 哈希
|
||||
│ └─ 相同 → 跳过安装
|
||||
│
|
||||
└─ 就绪，返回 venv Python 路径
|
||||
```
|
||||
|
||||
**依赖清单**（`runtime/requirements.txt`）：
|
||||
|
||||
| 库 | 用途 |
|
||||
|----|------|
|
||||
| `mss` | 高性能屏幕截图 |
|
||||
| `Pillow` | JPEG 编码和图像处理 |
|
||||
| `pyautogui` | 鼠标点击、键盘输入 |
|
||||
| `pyobjc-core` | macOS Objective-C 桥接 |
|
||||
| `pyobjc-framework-Cocoa` | NSWorkspace（应用管理）、NSPasteboard（剪贴板） |
|
||||
| `pyobjc-framework-Quartz` | CGDisplay（显示器）、CGWindow（窗口列表） |
|
||||
|
||||
### 命令映射表
|
||||
|
||||
`mac_helper.py`（660 行）实现了以下命令：
|
||||
|
||||
| 命令 | Python 实现 | 返回值 |
|
||||
|------|------------|--------|
|
||||
| `screenshot` | `mss.grab()` + `PIL.Image` JPEG 编码 | `{base64, width, height, displayWidth, displayHeight}` |
|
||||
| `zoom` | `mss.grab(region)` 区域截图 | `{base64, width, height}` |
|
||||
| `click` | `pyautogui.moveTo()` + `pyautogui.click()` | `true` |
|
||||
| `key` | `pyautogui.hotkey()` / `pyautogui.press()` | `true` |
|
||||
| `type` | `pyautogui.write(interval=0.008)` | `true` |
|
||||
| `drag` | `pyautogui.dragTo(duration=0.2)` | `true` |
|
||||
| `scroll` | `pyautogui.scroll()` / `pyautogui.hscroll()` | `true` |
|
||||
| `hold_key` | `pyautogui.keyDown()` + `sleep` + `pyautogui.keyUp()` | `true` |
|
||||
| `frontmost_app` | `NSWorkspace.frontmostApplication()` | `{bundleId, displayName}` |
|
||||
| `list_displays` | `CGGetActiveDisplayList()` + `CGDisplayBounds()` | `[DisplayGeometry...]` |
|
||||
| `find_window_displays` | `CGWindowListCopyWindowInfo()` + 交集计算 | `[{bundleId, displayIds}...]` |
|
||||
| `list_installed_apps` | 递归扫描 `/Applications` + `plistlib` | `[InstalledApp...]` |
|
||||
| `list_running_apps` | `NSWorkspace.runningApplications()` | `[RunningApp...]` |
|
||||
| `open_app` | `NSWorkspace.launchApplicationAtURL_options_` | `void` |
|
||||
| `read_clipboard` | `NSPasteboard.stringForType_()` | `string` |
|
||||
| `write_clipboard` | `NSPasteboard.setString_forType_()` | `void` |
|
||||
| `check_permissions` | `osascript` + `CGDisplayCaptureDisplay` | `{accessibility, screenRecording}` |
|
||||
|
||||
### 错误处理
|
||||
|
||||
```python
|
||||
# mac_helper.py 的统一错误处理
|
||||
def main():
|
||||
try:
|
||||
result = dispatch(command, payload)
|
||||
json_output({"ok": True, "result": result})
|
||||
except Exception as e:
|
||||
error_output({"ok": False, "error": {"message": str(e)}})
|
||||
```
|
||||
|
||||
TypeScript 端：
|
||||
```typescript
|
||||
// pythonBridge.ts
|
||||
const parsed = JSON.parse(stdout)
|
||||
if (!parsed.ok) {
|
||||
throw new Error(parsed.error.message) // → MCP tool error
|
||||
}
|
||||
return parsed.result
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 六、截图-分析-操作闭环
|
||||
|
||||
一个完整的 Computer Use 交互由多个 **截图-分析-操作** 循环组成：
|
||||
|
||||
### 典型交互流程
|
||||
|
||||
```
|
||||
用户: "帮我打开网易云音乐，搜索一首歌"
|
||||
|
||||
┌─ 循环 1：发现并打开应用 ─────────────────────────────────┐
|
||||
│ │
|
||||
│ Step 1: request_access │
|
||||
│ → 弹出权限对话框，用户授权允许操作的应用 │
|
||||
│ → 设置 allowedApps, grantFlags │
|
||||
│ │
|
||||
│ Step 2: screenshot │
|
||||
│ → 截取全屏 → JPEG 编码 → base64 │
|
||||
│ → 缓存截图尺寸 (lastScreenshotDims) │
|
||||
│ → 返回给模型 │
|
||||
│ │
|
||||
│ Step 3: 模型分析截图 │
|
||||
│ → "桌面上没有网易云音乐，需要打开它" │
|
||||
│ → 决定调用 open_application │
|
||||
│ │
|
||||
│ Step 4: open_application("com.netease.163music") │
|
||||
│ → Gate 1-9 全部通过 │
|
||||
│ → Python: NSWorkspace.launchApplicationAtURL_() │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─ 循环 2：定位搜索框 ────────────────────────────────────┐
|
||||
│ │
|
||||
│ Step 5: screenshot │
|
||||
│ → 截取全屏（网易云已打开） │
|
||||
│ → 更新 lastScreenshotDims │
|
||||
│ │
|
||||
│ Step 6: 模型分析截图 │
|
||||
│ → 视觉识别搜索框位置 → (756, 342) │
|
||||
│ → 决定点击搜索框 │
|
||||
│ │
|
||||
│ Step 7: left_click({coordinate: [756, 342]}) │
|
||||
│ → Gate 4: 隐藏非白名单应用 │
|
||||
│ → Gate 5: 前台是网易云 ✓ │
|
||||
│ → Gate 6: tier=full ≥ click ✓ │
|
||||
│ → Gate 7: 非 click-tier 应用，跳过 │
|
||||
│ → scaleCoord(756, 342) 转换为屏幕坐标 │
|
||||
│ → Python: pyautogui.click(x_logical, y_logical) │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─ 循环 3：输入搜索词 ────────────────────────────────────┐
|
||||
│ │
|
||||
│ Step 8: type({text: "喜欢你"}) │
|
||||
│ → Gate 6: tier=full ≥ keyboard ✓ │
|
||||
│ → Python: pyautogui.write("喜欢你", interval=0.008) │
|
||||
│ │
|
||||
│ Step 9: screenshot │
|
||||
│ → 确认搜索结果已出现 │
|
||||
│ → 模型分析："搜索结果列表中第一个就是目标歌曲" │
|
||||
│ │
|
||||
│ Step 10: left_click({coordinate: [...]}) │
|
||||
│ → 点击目标歌曲 │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 坐标转换的关键
|
||||
|
||||
**模型看到的**和**屏幕实际的**是不同的坐标空间：
|
||||
|
||||
```
|
||||
原始屏幕 (2560 x 1600, Retina 2x)
|
||||
├─ 逻辑尺寸: 1280 x 800
|
||||
└─ 物理像素: 2560 x 1600
|
||||
|
||||
imageResize 计算后：
|
||||
├─ 缩放后尺寸: 1176 x 735 (≤1568px 预算)
|
||||
└─ 这就是模型"看到"的截图尺寸
|
||||
|
||||
模型输出坐标: [588, 368]（图像空间的像素位置）
|
||||
|
||||
scaleCoord 转换:
|
||||
x_logical = (588 / 1176) * 1280 + originX = 640 + 0 = 640
|
||||
y_logical = (368 / 735) * 800 + originY = 400 + 0 = 400
|
||||
|
||||
Python 执行:
|
||||
pyautogui.moveTo(640, 400) ← 逻辑坐标（macOS 自动处理 Retina）
|
||||
```
|
||||
|
||||
### 截图缓存与状态同步
|
||||
|
||||
```
|
||||
bindSessionContext 闭包
|
||||
│
|
||||
├─ lastScreenshot (内存)
|
||||
│ ├─ base64: JPEG 数据（用于 pixel validation）
|
||||
│ ├─ width/height: 模型看到的尺寸
|
||||
│ └─ displayWidth/displayHeight/originX/originY: 显示器几何
|
||||
│
|
||||
└─ AppState.computerUseMcpState (持久化)
|
||||
├─ allowedApps: AppGrant[] — 已授权应用列表
|
||||
├─ grantFlags: {...} — 剪贴板/系统快捷键权限
|
||||
├─ selectedDisplayId?: number — 选定的显示器
|
||||
├─ lastScreenshotDims?: {...} — 截图几何（跨重启恢复）
|
||||
└─ hiddenDuringTurn?: Set<string> — 本轮隐藏的应用
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 七、关键源文件索引
|
||||
|
||||
### vendor/computer-use-mcp/（原始代码层）
|
||||
|
||||
| 文件 | 行数 | 职责 |
|
||||
|------|------|------|
|
||||
| `types.ts` | 622 | 权限模型、会话上下文、全部类型定义 |
|
||||
| `tools.ts` | 707 | 24 个 MCP 工具的 schema 和参数校验 |
|
||||
| `toolCalls.ts` | 1600+ | **核心**：工具派发、9 层安全关卡、权限流程 |
|
||||
| `deniedApps.ts` | 554 | 191 个应用的分类（browser/terminal/trading）和权限映射 |
|
||||
| `sentinelApps.ts` | 44 | 敏感应用预警标签（shell/filesystem/system_settings） |
|
||||
| `mcpServer.ts` | 314 | MCP 服务器工厂、会话上下文绑定、全局锁 |
|
||||
| `pixelCompare.ts` | 172 | 点击目标像素验证（staleness guard） |
|
||||
| `imageResize.ts` | 109 | 截图尺寸计算（API 图像转码算法） |
|
||||
| `keyBlocklist.ts` | 154 | 系统快捷键拦截（⌘Q、⌘Tab 等） |
|
||||
| `executor.ts` | 101 | ComputerExecutor 接口定义 |
|
||||
| `subGates.ts` | 20 | 灰度子开关预设组合 |
|
||||
|
||||
### utils/computerUse/（CLI 适配层）
|
||||
|
||||
| 文件 | 行数 | 职责 | 补丁? |
|
||||
|------|------|------|-------|
|
||||
| `executor.ts` | 231 | ComputerExecutor 的 Python bridge 实现 | ✅ 重写 |
|
||||
| `pythonBridge.ts` | 111 | Python 子进程管理、venv 引导、JSON RPC | ✅ 新增 |
|
||||
| `hostAdapter.ts` | 54 | HostAdapter 实现（权限检查、灰度读取） | 部分修改 |
|
||||
| `gates.ts` | 51 | GrowthBook 灰度控制（`getChicagoEnabled` 绕过） | ✅ 修改 |
|
||||
| `wrapper.tsx` | 300+ | 会话上下文构建、权限对话框、锁管理 | 未修改 |
|
||||
| `setup.ts` | 54 | MCP 配置初始化 | 未修改 |
|
||||
| `computerUseLock.ts` | 216 | 全局文件锁（`~/.claude/computer-use.lock`） | 未修改 |
|
||||
| `common.ts` | 62 | 常量定义（server name、bundle ID） | 未修改 |
|
||||
| `cleanup.ts` | — | turn-end 清理（应用恢复、剪贴板恢复） | 未修改 |
|
||||
| `toolRendering.tsx` | — | 工具结果 UI 渲染 | 未修改 |
|
||||
|
||||
### runtime/（Python 运行时）
|
||||
|
||||
| 文件 | 行数 | 职责 | 补丁? |
|
||||
|------|------|------|-------|
|
||||
| `mac_helper.py` | 660 | 所有系统交互的 Python 实现 | ✅ 新增 |
|
||||
| `requirements.txt` | 6 | Python 依赖声明 | ✅ 新增 |
|
||||
|
||||
---
|
||||
|
||||
## 八、设计权衡
|
||||
|
||||
### 为什么选择 Python Bridge？
|
||||
|
||||
| 维度 | 原生 Swift (.node) | Python Bridge |
|
||||
|------|-------------------|---------------|
|
||||
| **性能** | ~0ms（进程内调用） | ~50-100ms（子进程启动） |
|
||||
| **可读性** | 编译后不可读 | 660 行清晰的 Python |
|
||||
| **可修改性** | 需要 Swift 编译环境 | 直接编辑 .py 文件 |
|
||||
| **依赖** | 特定 Bun 版本的 NAPI | 任意 Python 3.8+ |
|
||||
| **跨平台** | 仅 macOS | pyautogui/mss 天然跨平台 |
|
||||
| **用户体验** | 无感知 | 无感知（模型思考时间秒级） |
|
||||
|
||||
**结论**：50-100ms 的额外延迟在 Computer Use 的场景下完全可以忽略——模型分析截图和决策的时间通常是 2-5 秒，用户不会注意到底层操作多了 100ms。
|
||||
|
||||
### 我们尝试过但放弃的方案
|
||||
|
||||
**方案一：提取原生 .node 模块**
|
||||
- 从 Claude Code 二进制中成功提取了 `computer-use-swift.node`（ARM64 424KB）
|
||||
- 同步方法正常工作，但 **Swift 异步方法的 continuation 永远不会 resume**
|
||||
- 根因：.node 文件针对 Claude Code 内置 Bun 编译，与用户 Bun 版本不兼容
|
||||
|
||||
**方案二：空 Stub 包**
|
||||
- 代码能编译但所有操作报错——没有实际执行能力
|
||||
|
||||
---
|
||||
|
||||
## 相关文档
|
||||
|
||||
- [Computer Use 功能指南](./computer-use.md) — 使用方式、快速开始、环境变量
|
||||
- [源码修复记录](/reference/fixes) — 其他修复和补丁的详细记录
|
||||
BIN
docs/features/images/01-computer-use-architecture.jpg
Normal file
BIN
docs/features/images/01-computer-use-architecture.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 888 KiB |
BIN
docs/features/images/02-computer-use-security-gates.jpg
Normal file
BIN
docs/features/images/02-computer-use-security-gates.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 523 KiB |
BIN
docs/features/images/03-computer-use-python-bridge.jpg
Normal file
BIN
docs/features/images/03-computer-use-python-bridge.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 912 KiB |
BIN
docs/features/images/04-computer-use-patch.jpg
Normal file
BIN
docs/features/images/04-computer-use-patch.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 800 KiB |
@ -32,12 +32,12 @@ features:
|
||||
title: Skills 系统
|
||||
details: 可扩展能力插件、自定义工作流、条件激活
|
||||
link: /skills/01-usage-guide
|
||||
- icon: "\U0001F4BB"
|
||||
title: Computer Use
|
||||
details: 桌面控制功能 — 截屏、鼠标、键盘操作
|
||||
link: /features/computer-use
|
||||
- icon: "\U0001F310"
|
||||
title: 第三方模型支持
|
||||
details: 接入 OpenAI、DeepSeek、Ollama 等任意兼容模型
|
||||
link: /guide/third-party-models
|
||||
- icon: "\U0001F4BB"
|
||||
title: Computer Use
|
||||
details: 桌面控制功能 — 截屏、鼠标、键盘操作（Python Bridge 实现）
|
||||
link: /features/computer-use
|
||||
---
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> 让 Claude Code 跨会话记住你是谁、你偏好什么、项目正在发生什么。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一什么是记忆系统">记忆系统</a> · <a href="#二四种记忆类型">四种记忆类型</a> · <a href="#三如何触发记忆保存">触发保存</a> · <a href="#四记忆存储在哪里">存储位置</a> · <a href="#五如何管理记忆">管理记忆</a> · <a href="#六记忆的生命周期">生命周期</a> · <a href="#七快速参考">快速参考</a>
|
||||
<a href="#一、什么是记忆系统">记忆系统</a> · <a href="#二、四种记忆类型">四种记忆类型</a> · <a href="#三、如何触发记忆保存">触发保存</a> · <a href="#四、记忆存储在哪里">存储位置</a> · <a href="#五、如何管理记忆">管理记忆</a> · <a href="#六、记忆的生命周期">生命周期</a> · <a href="#七、快速参考">快速参考</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> 从系统提示词注入到后台自动提取，拆解记忆系统的每一个技术细节。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一整体架构">整体架构</a> · <a href="#二路径解析系统">路径解析</a> · <a href="#三系统提示词注入">提示词注入</a> · <a href="#四自动记忆提取">自动提取</a> · <a href="#五智能记忆检索">智能检索</a> · <a href="#六记忆扫描详解">记忆扫描</a> · <a href="#七代理记忆agent-memory">代理记忆</a> · <a href="#八团队记忆同步">团队同步</a> · <a href="#九关键常量速查">常量速查</a> · <a href="#十数据流总览">数据流总览</a>
|
||||
<a href="#一、整体架构">整体架构</a> · <a href="#二、路径解析系统">路径解析</a> · <a href="#三、系统提示词注入">提示词注入</a> · <a href="#四、自动记忆提取">自动提取</a> · <a href="#五、智能记忆检索">智能检索</a> · <a href="#六、记忆扫描详解">记忆扫描</a> · <a href="#七、代理记忆-agent-memory">代理记忆</a> · <a href="#八、团队记忆同步">团队同步</a> · <a href="#九、关键常量速查">常量速查</a> · <a href="#十、数据流总览">数据流总览</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> Claude 会"做梦"——在后台静默回顾近期会话，整合、更新、修剪记忆，就像人类睡眠中整理白天的记忆一样。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一什么是-autodream">AutoDream</a> · <a href="#二触发条件">触发条件</a> · <a href="#三四阶段整合流程">整合流程</a> · <a href="#四安全限制">安全限制</a> · <a href="#五ui-展示">UI 展示</a> · <a href="#六配置与开关">配置开关</a> · <a href="#七与-extractmemories-的关系">对比</a> · <a href="#八源码导航">源码导航</a>
|
||||
<a href="#一、什么是-autodream">AutoDream</a> · <a href="#二、触发条件">触发条件</a> · <a href="#三、四阶段整合流程">整合流程</a> · <a href="#四、安全限制">安全限制</a> · <a href="#五、ui-展示">UI 展示</a> · <a href="#六、配置与开关">配置开关</a> · <a href="#七、与-extractmemories-的关系">对比</a> · <a href="#八、源码导航">源码导航</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> Skills 是 Claude Code 的扩展能力引擎，让你用 Markdown 文件定义专属的自动化工作流。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一什么是-skills">Skills 是什么</a> · <a href="#二六种-skill-来源">六种来源</a> · <a href="#三skill-定义格式">定义格式</a> · <a href="#四调用方式">调用方式</a> · <a href="#五执行上下文">执行上下文</a> · <a href="#六条件激活">条件激活</a> · <a href="#七权限控制">权限控制</a> · <a href="#八快速参考">快速参考</a>
|
||||
<a href="#一、什么是-skills">Skills 是什么</a> · <a href="#二、六种-skill-来源">六种来源</a> · <a href="#三、skill-定义格式">定义格式</a> · <a href="#四、调用方式">调用方式</a> · <a href="#五、执行上下文">执行上下文</a> · <a href="#六、条件激活">条件激活</a> · <a href="#七、权限控制">权限控制</a> · <a href="#八、快速参考">快速参考</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
> 深度剖析 Skills 如何被发现、加载、注入、执行和管理。
|
||||
|
||||
<p align="center">
|
||||
<a href="#一整体架构">整体架构</a> · <a href="#二skill-发现与加载">发现与加载</a> · <a href="#三frontmatter-解析">Frontmatter 解析</a> · <a href="#四skill-注入到对话">注入对话</a> · <a href="#五skilltool-执行引擎">执行引擎</a> · <a href="#六fork-子代理执行">Fork 执行</a> · <a href="#七条件激活与动态发现">条件激活</a> · <a href="#八hook-集成">Hook 集成</a> · <a href="#九权限系统">权限系统</a> · <a href="#十完整生命周期">完整生命周期</a> · <a href="#十一源码索引">源码索引</a>
|
||||
<a href="#一、整体架构">整体架构</a> · <a href="#二、skill-发现与加载">发现与加载</a> · <a href="#三、frontmatter-解析">Frontmatter 解析</a> · <a href="#四、skill-注入到对话">注入对话</a> · <a href="#五、skilltool-执行引擎">执行引擎</a> · <a href="#六、fork-子代理执行">Fork 执行</a> · <a href="#七、条件激活与动态发现">条件激活</a> · <a href="#八、hook-集成">Hook 集成</a> · <a href="#九、权限系统">权限系统</a> · <a href="#十、完整生命周期">完整生命周期</a> · <a href="#十一、源码索引">源码索引</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user