From 5fa8c3603d24497344e9e9e26d3231ac7a746297 Mon Sep 17 00:00:00 2001 From: Colin McDonnell Date: Fri, 9 Jan 2026 16:03:25 -0800 Subject: [PATCH] Add writeups --- docs/webfetch.md | 520 ++++++++++++++++++++++++++++++++++++++++++++++ docs/websearch.md | 435 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 955 insertions(+) create mode 100644 docs/webfetch.md create mode 100644 docs/websearch.md diff --git a/docs/webfetch.md b/docs/webfetch.md new file mode 100644 index 0000000..16bcc0d --- /dev/null +++ b/docs/webfetch.md @@ -0,0 +1,520 @@ +# WebFetch Tool Analysis + +Analysis of webfetch/URL fetching implementations across three AI coding agents to inform the design of pullfrog's custom webfetch MCP tool. + +--- + +## 1. OpenCode Implementation + +**Source**: `packages/opencode/src/tool/webfetch.ts` + +### Architecture + +OpenCode's webfetch is straightforward - a simple fetch wrapper with HTML-to-markdown conversion: + +```typescript +const response = await fetch(params.url, { + signal: AbortSignal.any([controller.signal, ctx.abort]), + headers: { + "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...", + Accept: acceptHeader, + "Accept-Language": "en-US,en;q=0.9", + }, +}) +``` + +### Key Features + +| Feature | Implementation | +|---------|---------------| +| **Output formats** | `text`, `markdown`, `html` (default: markdown) | +| **HTML→Markdown** | Uses `turndown` library | +| **Max response size** | 5MB hard limit | +| **Timeout** | 30s default, 120s max | +| **Permission system** | Application-level `ctx.ask()` prompt | +| **Domain blocking** | None - relies on user approval | +| **Caching** | None | +| **Redirect handling** | Native fetch behavior | + +### HTML Processing + +Two methods depending on output format: + +1. **`extractTextFromHTML()`** - Uses Bun's `HTMLRewriter` to strip scripts/styles and extract text +2. **`convertHTMLToMarkdown()`** - Uses `turndown` with sensible defaults (ATX headings, fenced code blocks) + +### Permission Model + +```typescript +await ctx.ask({ + permission: "webfetch", + patterns: [params.url], + always: ["*"], // User can allow all future requests + metadata: { url, format, timeout }, +}) +``` + +**Verdict**: No enforcement - purely advisory. If user approves, the fetch proceeds with no restrictions. + +### What I Like +- Clean, minimal implementation +- Good HTML processing with `turndown` +- Sensible size limits (5MB) +- Format flexibility + +### What I Don't Like +- No domain whitelisting/blocklisting +- No caching (repeated requests to same URL are wasteful) +- Permission system is advisory-only +- No redirect safety checks + +--- + +## 2. Claude Code Implementation + +**Source**: Extracted from bundled `claude` CLI binary + +### Architecture + +Claude Code uses a more sophisticated approach with server-side domain validation: + +```javascript +// Domain validation before fetch +async function Ci5(domain) { + const response = await fetch( + `https://claude.ai/api/web/domain_info?domain=${encodeURIComponent(domain)}` + ); + if (response.status === 200) { + return response.data.can_fetch === true + ? { status: "allowed" } + : { status: "blocked" }; + } + return { status: "check_failed" }; +} +``` + +### Key Features + +| Feature | Implementation | +|---------|---------------| +| **Domain blocklist** | Server-side API at `claude.ai/api/web/domain_info` | +| **Permission format** | `WebFetch(domain:example.com)` - domain-only, not URLs | +| **Wildcard support** | `domain:*.google.com` patterns | +| **HTTP→HTTPS upgrade** | Automatic protocol upgrade | +| **Caching** | 15-minute self-cleaning cache | +| **HTML→Markdown** | Uses `turndown` | +| **Redirect handling** | Special handling - informs user of cross-host redirects | +| **Enterprise override** | `skipWebFetchPreflight` setting | + +### Domain Permission Model + +Claude Code enforces domain-level permissions, not URL-level: + +```javascript +WebFetch: (A) => { + if (A.includes("://") || A.startsWith("http")) + return { + valid: false, + error: "WebFetch permissions use domain format, not URLs", + suggestion: 'Use "domain:hostname" format', + examples: ["WebFetch(domain:example.com)", "WebFetch(domain:github.com)"] + }; + if (!A.startsWith("domain:")) + return { + valid: false, + error: 'WebFetch permissions must use "domain:" prefix', + examples: ["WebFetch(domain:example.com)", "WebFetch(domain:*.google.com)"] + }; + return { valid: true }; +} +``` + +### Blocklist Enforcement Flow + +``` +User requests URL + ↓ +Extract hostname + ↓ +Check claude.ai/api/web/domain_info?domain=hostname + ↓ +┌──────────────────────────────────┐ +│ allowed → proceed with fetch │ +│ blocked → throw AC0 error │ +│ check_failed → throw QC0 error │ +└──────────────────────────────────┘ +``` + +### Redirect Handling + +When a URL redirects to a different host: +```javascript +// Returns special response asking user to manually re-request +return `To complete your request, I need to fetch content from the redirected URL. +Please use WebFetch again with these parameters: +- url: "${redirectUrl}" +- prompt: "${originalPrompt}"`; +``` + +This prevents open redirect attacks where a "safe" domain redirects to a malicious one. + +### What I Like +- **Server-side blocklist** - centralized, updateable without client changes +- **Domain-level permissions** - prevents path-based bypasses +- **Redirect safety** - cross-host redirects require explicit user action +- **15-minute caching** - reduces redundant requests +- **Enterprise override** - `skipWebFetchPreflight` for corporate environments + +### What I Don't Like +- **External dependency** - requires `claude.ai` API to be available +- **No local blocklist** - can't work offline or with custom blocklists +- **Opaque blocklist** - users can't see what's blocked or why + +--- + +## 3. Gemini CLI Implementation + +**Source**: `@google/gemini-cli` npm package + +### Architecture + +Gemini CLI takes a fundamentally different approach - it doesn't have a dedicated webfetch tool. Instead it relies on: + +1. **Google Search grounding** - built into the Gemini API +2. **MCP servers** - external tools can provide fetch capabilities +3. **No native URL fetching** - by design + +### Key Observations + +From searching the codebase: +- No `webfetch`, `url_fetch`, or similar tool definitions +- Has `github_fetch.ts` for fetching GitHub releases (internal use) +- Relies on model's built-in capabilities or MCP extensions + +### Why No WebFetch? + +Gemini's design philosophy appears to be: +1. Use the model's grounding capabilities for web information +2. Delegate specialized fetching to MCP servers +3. Avoid building network access into the CLI itself + +### What I Like +- **Clean separation** - network access is opt-in via MCP +- **Security by default** - no built-in way to exfiltrate data + +### What I Don't Like +- **Missing functionality** - can't fetch arbitrary URLs +- **Requires MCP setup** - more complex for users who need fetching + +--- + +## 4. Pullfrog Design Decisions + +### Core Requirements + +1. **Domain-level whitelisting** - enforced in-tool, not advisory +2. **Simple implementation** - no external API dependencies +3. **GitHub-focused** - optimized for common development URLs + +### Proposed Architecture + +``` +┌─────────────────────────────────────────────────────┐ +│ WebFetch Tool │ +├─────────────────────────────────────────────────────┤ +│ 1. Parse URL → extract hostname │ +│ 2. Check against DOMAIN_ALLOWLIST │ +│ 3. If not allowed → return error (not throw) │ +│ 4. Fetch with timeout + size limits │ +│ 5. Convert HTML → Markdown if needed │ +│ 6. Return content │ +└─────────────────────────────────────────────────────┘ +``` + +### Domain Allowlist Strategy + +**Included in initial allowlist**: +```typescript +const DOMAIN_ALLOWLIST = new Set([ + // Documentation sites + "docs.github.com", + "developer.mozilla.org", + "nodejs.org", + "docs.python.org", + "go.dev", + "doc.rust-lang.org", + "docs.microsoft.com", + "learn.microsoft.com", + + // Package registries (documentation) + "npmjs.com", + "www.npmjs.com", + "pypi.org", + "crates.io", + "pkg.go.dev", + + // GitHub (raw content, gists) + "raw.githubusercontent.com", + "gist.githubusercontent.com", + + // Common API documentation + "api.github.com", // Already have GitHub tools, but for reference docs +]); +``` + +**Explicitly NOT included**: +- `github.com` itself - we have dedicated GitHub MCP tools +- Social media sites +- General web pages +- Arbitrary user-provided domains + +### Features Borrowed from Each Agent + +| Feature | Source | Included? | Rationale | +|---------|--------|-----------|-----------| +| HTML→Markdown via turndown | OpenCode | ✅ | Clean, proven library | +| 5MB size limit | OpenCode | ✅ | Sensible default | +| Domain-level permissions | Claude Code | ✅ | Core requirement | +| Redirect safety checks | Claude Code | ✅ | Prevents open redirect attacks | +| 15-minute caching | Claude Code | ❌ | Adds complexity, MCP is stateless | +| Server-side blocklist | Claude Code | ❌ | External dependency | +| Enterprise override | Claude Code | ❌ | Not needed for GitHub Actions | +| No built-in fetching | Gemini | ❌ | We need this functionality | + +### Features NOT Included (and why) + +1. **Caching** - MCP tools are stateless by design. Caching would require shared state across requests. The agent can cache results itself. + +2. **Server-side blocklist** - Would require standing up an API endpoint. The allowlist approach is simpler and more transparent. + +3. **User permission prompts** - In GitHub Actions context, there's no interactive user. Allowlist is enforced automatically. + +4. **Wildcard domain patterns** - Adds complexity. Start with explicit domains, add patterns if needed. + +5. **Multiple output formats** - Start with markdown only. Can add `text` and `html` later if needed. + +### Error Handling Strategy + +Unlike OpenCode/Claude which throw errors, we return errors as content: + +```typescript +// Domain not allowed - return message, don't throw +if (!isDomainAllowed(hostname)) { + return { + output: `Domain "${hostname}" is not in the allowlist. Allowed domains: ${Array.from(DOMAIN_ALLOWLIST).join(", ")}`, + error: true, + }; +} +``` + +This lets the agent understand the limitation and potentially find alternative approaches. + +### Redirect Handling + +Adopt Claude Code's approach with modification: + +```typescript +// If redirect crosses domains, check the new domain +if (response.redirected) { + const redirectUrl = new URL(response.url); + if (!isDomainAllowed(redirectUrl.hostname)) { + return { + output: `URL redirected to "${redirectUrl.hostname}" which is not in the allowlist.`, + error: true, + }; + } +} +``` + +--- + +## 5. Implementation Plan + +### Summary + +Add a new `webfetch` MCP tool that fetches web content with domain-level whitelisting enforced server-side. The whitelist is configured via the payload (from GitHub App), and non-whitelisted domains return a helpful message guiding the LLM to alternative approaches. + +### Key Design Decisions + +| Aspect | OpenCode | Claude Code | Our Implementation | +|--------|----------|-------------|-------------------| +| **Whitelisting** | Permission prompt (advisory) | External API `domain_info` | Payload-configured whitelist | +| **Enforcement** | None (user approval) | Server-side check | Server-side check | +| **HTML Processing** | Turndown for markdown | Turndown for markdown | Turndown for markdown | +| **Redirects** | Follows automatically | Detects cross-host redirects | Follow with host check | +| **Timeout** | 30s default, 120s max | Configurable | 30s default, 120s max | + +### Step 1: Add whitelist to payload type + +Update `index.ts` to include `allowedWebFetchDomains`: + +```typescript +interface Payload { + // ... existing fields + allowedWebFetchDomains?: string[]; // e.g. ["github.com", "*.npmjs.com", "docs.python.org"] +} +``` + +### Step 2: Create `mcp/webfetch.ts` + +```typescript +// Core structure +export const WebFetchParams = type({ + url: "string", + "format?": "'markdown' | 'text' | 'html'", + "timeout?": "number", +}); + +export function WebFetchTool(ctx: ToolContext) { + return tool({ + name: "webfetch", + description: `Fetch content from whitelisted web URLs...`, + parameters: WebFetchParams, + execute: execute(async (params) => { + // 1. Validate URL format + // 2. Check domain against whitelist (from ctx.payload) + // 3. Fetch with timeout and size limits + // 4. Convert HTML to markdown if needed + // 5. Return content or guidance message + }), + }); +} +``` + +### Step 3: Domain Matching Logic + +Support wildcards for subdomains: + +- `github.com` - exact match +- `*.github.com` - any subdomain (e.g., `docs.github.com`, `api.github.com`) +- `*.npmjs.com` - matches `www.npmjs.com`, `registry.npmjs.com`, etc. + +```typescript +function isDomainAllowed(hostname: string, whitelist: string[]): boolean { + for (const pattern of whitelist) { + if (pattern.startsWith("*.")) { + const suffix = pattern.slice(1); // ".github.com" + if (hostname.endsWith(suffix) || hostname === pattern.slice(2)) { + return true; + } + } else if (hostname === pattern) { + return true; + } + } + return false; +} +``` + +### Step 4: Response for Non-Whitelisted Domains + +When domain is not whitelisted, return guidance (not an error): + +```typescript +return { + allowed: false, + message: `The domain "${hostname}" is not in the allowed list for direct fetching. ` + + `Consider using web_search to find relevant information, or ask the user to ` + + `provide the content directly. Allowed domains: ${whitelist.join(", ")}`, +}; +``` + +### Step 5: HTML to Markdown Conversion + +Use Turndown (same as OpenCode) for HTML-to-markdown conversion: + +```typescript +import TurndownService from "turndown"; + +function htmlToMarkdown(html: string): string { + const turndown = new TurndownService({ + headingStyle: "atx", + codeBlockStyle: "fenced", + }); + turndown.remove(["script", "style", "meta", "link"]); + return turndown.turndown(html); +} +``` + +### Step 6: Register the Tool + +Add to `mcp/index.ts`: + +```typescript +import { WebFetchTool } from "./webfetch.ts"; + +// In the tools array +WebFetchTool(ctx), +``` + +### Data Flow + +```mermaid +sequenceDiagram + participant LLM + participant MCP as MCP Server + participant WF as WebFetch Tool + participant Web as External URL + + LLM->>MCP: webfetch(url, format) + MCP->>WF: execute(params) + WF->>WF: Parse URL, extract hostname + WF->>WF: Check whitelist from payload + alt Domain allowed + WF->>Web: fetch(url) + Web-->>WF: Response + WF->>WF: Convert to markdown + WF-->>MCP: {content, contentType} + MCP-->>LLM: Success result + else Domain not allowed + WF-->>MCP: {allowed: false, guidance} + MCP-->>LLM: Guidance message + end +``` + +### Files to Create/Modify + +| File | Action | +|------|--------| +| `mcp/webfetch.ts` | Create - main tool implementation | +| `mcp/index.ts` | Modify - register the tool | +| `index.ts` | Modify - add `allowedWebFetchDomains` to payload type | +| `package.json` | Modify - add `turndown` dependency | + +### Dependencies + +Add to `package.json`: + +- `turndown` - HTML to markdown conversion (same as OpenCode) +- `@types/turndown` - TypeScript types + +--- + +## 6. Implementation Checklist + +- [ ] Add `allowedWebFetchDomains` field to payload type in `index.ts` +- [ ] Create `mcp/webfetch.ts` with domain whitelisting and HTML conversion +- [ ] Register `WebFetchTool` in `mcp/index.ts` +- [ ] Add `turndown` and `@types/turndown` dependencies to `package.json` +- [ ] Test with allowed domains +- [ ] Test with blocked domains +- [ ] Test redirect behavior + +--- + +## 7. Open Questions + +1. **Should we support query parameters in allowlist?** + - e.g., allow `api.example.com/v1/*` but not `api.example.com/admin/*` + - Initial decision: No, domain-level only + +2. **Should we allow configurable allowlists?** + - Via environment variable or config file? + - Initial decision: No, hardcoded for simplicity + +3. **Should we support authentication headers?** + - For private documentation sites + - Initial decision: No, security risk + +4. **Rate limiting?** + - Prevent agent from hammering a site + - Initial decision: Rely on timeout, add if needed diff --git a/docs/websearch.md b/docs/websearch.md new file mode 100644 index 0000000..6601b18 --- /dev/null +++ b/docs/websearch.md @@ -0,0 +1,435 @@ +# Web Search Functionality by Agent + +This document describes how each supported agent implements web search functionality. + +## Summary + +| Agent | Tool Name | Search Provider | API/Method | +|-------|-----------|-----------------|------------| +| Claude Code | `WebSearch` | Anthropic internal | Claude Code SDK | +| Gemini CLI | `google_web_search` | Google Search via Gemini API | `generateContent` with `model: 'web-search'` | +| OpenCode | `websearch` | Exa AI | MCP protocol to `https://mcp.exa.ai/mcp` | + +All three agents also support a separate **web fetch** tool for directly retrieving and parsing web page content. + +--- + +## Claude Code + +### Tools +- `WebSearch` - Search the web for information +- `WebFetch` - Fetch and process web content + +### Implementation +Native functionality through `@anthropic-ai/claude-agent-sdk` (closed source). The actual search provider is internal to Anthropic's infrastructure. + +### Configuration in Pullfrog + +Web search can be disabled via the `disallowedTools` option: + +```typescript +// In sandbox mode, web tools are disabled +disallowedTools: ["Bash", "WebSearch", "WebFetch", "Write"] +``` + +--- + +## Gemini CLI + +### Tools +- `google_web_search` - Perform web searches using Google Search +- `web_fetch` - Fetch and process content from URLs + +### Implementation + +Source: [`packages/core/src/tools/web-search.ts`](https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/tools/web-search.ts) + +**How it works:** +1. Sends query to Gemini API using `generateContent` with `model: 'web-search'` +2. Google performs the search and returns results with grounding metadata +3. Response includes inline citations, source URLs, and titles + +```typescript +const response = await geminiClient.generateContent( + { model: 'web-search' }, + [{ role: 'user', parts: [{ text: this.params.query }] }], + signal, +); +``` + +### Features +- Returns processed summary (not raw search results) +- Inline citations with grounding metadata +- Sources list with titles and URIs +- UTF-8 byte position handling for accurate citation insertion + +### Parameters +- `query` (string, required): The search query + +### Web Fetch + +The `web_fetch` tool processes content from URLs: +- Uses Gemini API's `urlContext` feature +- Fallback to direct HTTP fetch with `html-to-text` conversion +- Supports up to 20 URLs per request +- Converts GitHub blob URLs to raw URLs automatically + +--- + +## OpenCode + +### Tools +- `websearch` - Search the web using Exa AI +- `webfetch` - Fetch and read web pages + +### Implementation + +Source: [`packages/opencode/src/tool/websearch.ts`](https://github.com/sst/opencode/blob/main/packages/opencode/src/tool/websearch.ts) + +**How it works:** +1. Calls Exa AI's MCP endpoint at `https://mcp.exa.ai/mcp` +2. Uses JSON-RPC protocol to invoke the `web_search_exa` tool +3. Parses SSE response for search results + +```typescript +const searchRequest: McpSearchRequest = { + jsonrpc: "2.0", + id: 1, + method: "tools/call", + params: { + name: "web_search_exa", + arguments: { + query: params.query, + type: params.type || "auto", + numResults: params.numResults || 8, + livecrawl: params.livecrawl || "fallback", + contextMaxCharacters: params.contextMaxCharacters, + }, + }, +} +``` + +### Features +- Real-time web searches with content scraping +- Configurable result count (default: 8) +- Live crawl modes: `fallback` (backup if cached unavailable) or `preferred` (prioritize live crawling) +- Search types: `auto` (balanced), `fast` (quick results), `deep` (comprehensive) +- Context max characters for LLM optimization + +### Parameters +- `query` (string, required): The search query +- `numResults` (number, optional): Number of results to return (default: 8) +- `livecrawl` (enum, optional): `"fallback"` | `"preferred"` +- `type` (enum, optional): `"auto"` | `"fast"` | `"deep"` +- `contextMaxCharacters` (number, optional): Maximum characters for context + +### Configuration in Pullfrog + +Web tools are configured via the permission config in `opencode.json`: + +```typescript +// In sandbox mode +permission: { + webfetch: "deny", + // ... +} + +// In normal mode +permission: { + webfetch: "allow", + // ... +} +``` + +### Environment Variables +- `OPENCODE_ENABLE_EXA` - Enable Exa web search tools (required for "zen" users) + +### Web Fetch + +The `webfetch` tool directly fetches URLs: +- Direct HTTP fetch with browser-like User-Agent +- HTML to Markdown conversion using Turndown +- Configurable timeout (max 120 seconds) +- 5MB response size limit + +--- + +## Comparison + +| Feature | Claude Code | Gemini CLI | OpenCode | +|---------|-------------|------------|----------| +| Search Provider | Anthropic | Google | Exa AI | +| Result Format | Summary | Summary + Citations | Raw content | +| URL Fetching | Yes (`WebFetch`) | Yes (`web_fetch`) | Yes (`webfetch`) | +| Grounding/Citations | Unknown | Yes | No | +| Configurable Results | No | No | Yes (numResults) | +| Search Depth Options | No | No | Yes (auto/fast/deep) | +| Live Crawling | Unknown | Fallback only | Configurable | + +--- + +## Security Considerations + +In Pullfrog's sandbox mode: +- **Claude Code**: `WebSearch` and `WebFetch` are explicitly disabled via `disallowedTools` +- **Gemini CLI**: No explicit disable mechanism in the wrapper (relies on default behavior) +- **OpenCode**: `webfetch` permission set to `"deny"` in sandbox mode + +For public repositories, consider the implications of web search/fetch: +- Fetched content could potentially be used to inject prompts +- Search queries might leak information about the codebase context + +--- + +## Proposed Implementation Plan + +### Option 1: Use Native Agent Web Search (Current State) + +Each agent uses its own built-in web search: +- **Pros**: No additional implementation, leverages each provider's strengths +- **Cons**: Inconsistent behavior across agents, no unified control + +**Current gaps:** +- Gemini CLI has no explicit disable mechanism for web search in sandbox mode +- No unified way to configure web search across all agents + +### Option 2: Unified MCP Web Search Tool + +Add a `web_search` tool to the Pullfrog MCP server (`mcp/`) that all agents can use: + +``` +mcp/ +├── bash.ts +├── webSearch.ts # New unified web search tool +└── ... +``` + +**Implementation approach:** + +1. **Create `mcp/webSearch.ts`** with a provider-agnostic interface: + ```typescript + export const webSearchTool = { + name: "web_search", + description: "Search the web for information", + inputSchema: { + type: "object", + properties: { + query: { type: "string", description: "Search query" }, + numResults: { type: "number", description: "Number of results (default: 5)" }, + }, + required: ["query"], + }, + }; + ``` + +2. **Choose a search provider** (options): + - **Exa AI** - Already used by OpenCode, good LLM-optimized results + - **Tavily** - Popular for AI agents, provides search + content extraction + - **SerpAPI** - Google results via API + - **Brave Search API** - Privacy-focused alternative + +3. **Add to MCP server** in `mcp/server.ts`: + ```typescript + import { webSearchTool, handleWebSearch } from "./webSearch.ts"; + // Register tool... + ``` + +4. **Disable native web search** for each agent: + - Claude: Add `"WebSearch"` to `disallowedTools` + - Gemini: Add `"google_web_search"` to `excludeTools` in settings.json + - OpenCode: Set `websearch: "deny"` in permission config + +**Pros:** +- Consistent behavior across all agents +- Centralized control for security/sandbox modes +- Can filter/sanitize results before returning to agent +- Single API key management + +**Cons:** +- Additional API costs (search provider) +- Loses provider-specific features (e.g., Gemini's grounding metadata) + +### Option 3: Hybrid Approach + +Allow native web search for private repos, use MCP tool for public repos: + +```typescript +// In agent configuration +const useNativeWebSearch = !repo.isPublic; + +// Claude +disallowedTools: repo.isPublic ? ["WebSearch", "WebFetch"] : []; + +// Gemini +excludeTools: repo.isPublic ? ["google_web_search", "web_fetch"] : []; + +// OpenCode +permission: { + websearch: repo.isPublic ? "deny" : "allow", +} +``` + +Then for public repos, agents would use the MCP `web_search` tool which: +- Filters sensitive queries +- Sanitizes returned content +- Logs all searches for audit + +### Recommended Approach + +**Short-term**: Implement Option 3 (Hybrid) with these steps: + +1. [ ] Add `excludeTools: ["google_web_search"]` for Gemini in public repo mode +2. [ ] Ensure OpenCode `websearch` permission is properly set for sandbox mode +3. [ ] Document the current native web search behavior for each agent + +**Medium-term**: Implement Option 2 (Unified MCP) for public repos: + +1. [ ] Create `mcp/webSearch.ts` using Exa AI (consistent with OpenCode) +2. [ ] Add `EXA_API_KEY` to secrets handling +3. [ ] Register web search in MCP server +4. [ ] Disable native web search for all agents when MCP tool is available +5. [ ] Add result sanitization to prevent prompt injection + +### API Key Requirements + +| Provider | Environment Variable | Notes | +|----------|---------------------|-------| +| Exa AI | `EXA_API_KEY` | Already used by OpenCode | +| Tavily | `TAVILY_API_KEY` | Popular alternative | +| Brave | `BRAVE_API_KEY` | Privacy-focused | + +For the unified MCP approach, only one search provider API key would be needed. + +--- + +## Proposed Implementation Plan + +### Option A: Unified MCP Web Search Tool + +Create a custom MCP tool that provides consistent web search across all agents. + +**Pros:** +- Consistent behavior and results across agents +- Full control over search provider and rate limiting +- Can implement caching and deduplication +- Single point for security filtering + +**Cons:** +- Additional infrastructure (need a search API key) +- Latency from proxying through MCP server + +**Implementation:** +1. Add `websearch` tool to `mcp/` directory +2. Integrate with a search provider (options: Exa AI, SerpAPI, Brave Search, Tavily) +3. Configure each agent to use MCP tool instead of native: + - Claude: Add to `disallowedTools` and provide via MCP + - Gemini: Use `excludeTools` in settings.json for `google_web_search` + - OpenCode: Disable native via permission config + +```typescript +// mcp/websearch.ts +export const websearchTool = { + name: "websearch", + description: "Search the web for current information", + inputSchema: { + type: "object", + properties: { + query: { type: "string", description: "Search query" }, + numResults: { type: "number", description: "Number of results (1-10)" }, + }, + required: ["query"], + }, + handler: async ({ query, numResults = 5 }) => { + // Use Exa, Brave, or other search API + const results = await searchProvider.search(query, numResults); + return formatResults(results); + }, +}; +``` + +### Option B: Native Tools with Configuration + +Keep using each agent's native web search but add consistent configuration. + +**Pros:** +- No additional infrastructure +- Agents can use optimized native implementations +- Less latency + +**Cons:** +- Inconsistent results across agents +- Different capabilities per agent +- Harder to control/audit searches + +**Implementation:** +1. Add `websearch_enabled` option to payload/config +2. Update each agent wrapper: + - Claude: Toggle `WebSearch` in `disallowedTools` + - Gemini: Add `google_web_search` to `excludeTools` in settings.json + - OpenCode: Set `websearch` permission in config + +```typescript +// agents/claude.ts +const disallowedTools = payload.websearchEnabled + ? ["Bash"] + : ["Bash", "WebSearch", "WebFetch"]; + +// agents/gemini.ts +if (!payload.websearchEnabled) { + newSettings.excludeTools = [...(newSettings.excludeTools || []), "google_web_search"]; +} + +// agents/opencode.ts +permission: { + websearch: payload.websearchEnabled ? "allow" : "deny", + // ... +} +``` + +### Option C: Hybrid Approach (Recommended) + +Use native tools when available, with MCP fallback for consistency. + +**Implementation:** +1. Define a `websearch` MCP tool as fallback +2. For agents with good native search (Claude, Gemini): use native +3. For agents without (or with unreliable) search: use MCP tool +4. Add configuration to force MCP-only mode if needed + +```typescript +// Per-agent configuration +const agentWebSearchConfig = { + claude: { useNative: true, mcpFallback: false }, + gemini: { useNative: true, mcpFallback: false }, + opencode: { useNative: false, mcpFallback: true }, // Exa requires API key +}; +``` + +### Required Changes by Option + +| Change | Option A | Option B | Option C | +|--------|----------|----------|----------| +| New MCP tool | Yes | No | Yes | +| Search API key | Yes | No | Optional | +| Agent wrapper changes | Yes | Yes | Yes | +| Action input changes | No | Yes | Yes | +| External dependencies | Yes | No | Optional | + +### Recommended Next Steps + +1. **Decide on search provider** - If going with MCP approach: + - Exa AI: Already used by OpenCode, good for code-related searches + - Brave Search: Privacy-focused, good general search + - Tavily: Designed for AI agents, includes content extraction + +2. **Add configuration** - New action inputs: + ```yaml + websearch: + description: 'Enable web search functionality' + required: false + default: 'false' + ``` + +3. **Implement per-agent** - Start with Option B (simplest), upgrade to C if needed + +4. **Add security controls** - Query filtering, domain allowlists, rate limiting