shockbot/docs/websearch.md

Web Search Functionality by Agent

This document describes how each supported agent implements web search functionality.

Summary

Agent	Tool Name	Search Provider	API/Method
Claude Code	WebSearch	Anthropic internal	Claude Code SDK
Gemini CLI	google_web_search	Google Search via Gemini API	generateContent with model: 'web-search'
OpenCode	websearch	Exa AI	MCP protocol to https://mcp.exa.ai/mcp

All three agents also support a separate web fetch tool for directly retrieving and parsing web page content.

---

Claude Code

Tools

• WebSearch - Search the web for information
• WebFetch - Fetch and process web content

Implementation

Native functionality through @anthropic-ai/claude-agent-sdk (closed source). The actual search provider is internal to Anthropic's infrastructure.

Configuration in Pullfrog

Web search can be disabled via the disallowedTools option:

// In sandbox mode, web tools are disabled
disallowedTools: ["Bash", "WebSearch", "WebFetch", "Write"]

---

Gemini CLI

Tools

• google_web_search - Perform web searches using Google Search
• web_fetch - Fetch and process content from URLs

Implementation

Source: packages/core/src/tools/web-search.ts

How it works:

1. Sends query to Gemini API using generateContent with model: 'web-search'
2. Google performs the search and returns results with grounding metadata
3. Response includes inline citations, source URLs, and titles

const response = await geminiClient.generateContent(
  { model: 'web-search' },
  [{ role: 'user', parts: [{ text: this.params.query }] }],
  signal,
);

Features

• Returns processed summary (not raw search results)
• Inline citations with grounding metadata
• Sources list with titles and URIs
• UTF-8 byte position handling for accurate citation insertion

Parameters

• query (string, required): The search query

Web Fetch

The web_fetch tool processes content from URLs:

• Uses Gemini API's urlContext feature
• Fallback to direct HTTP fetch with html-to-text conversion
• Supports up to 20 URLs per request
• Converts GitHub blob URLs to raw URLs automatically

---

OpenCode

Tools

• websearch - Search the web using Exa AI
• webfetch - Fetch and read web pages

Implementation

Source: packages/opencode/src/tool/websearch.ts

How it works:

1. Calls Exa AI's MCP endpoint at https://mcp.exa.ai/mcp
2. Uses JSON-RPC protocol to invoke the web_search_exa tool
3. Parses SSE response for search results

const searchRequest: McpSearchRequest = {
  jsonrpc: "2.0",
  id: 1,
  method: "tools/call",
  params: {
    name: "web_search_exa",
    arguments: {
      query: params.query,
      type: params.type || "auto",
      numResults: params.numResults || 8,
      livecrawl: params.livecrawl || "fallback",
      contextMaxCharacters: params.contextMaxCharacters,
    },
  },
}

Features

• Real-time web searches with content scraping
• Configurable result count (default: 8)
• Live crawl modes: fallback (backup if cached unavailable) or preferred (prioritize live crawling)
• Search types: auto (balanced), fast (quick results), deep (comprehensive)
• Context max characters for LLM optimization

Parameters

• query (string, required): The search query
• numResults (number, optional): Number of results to return (default: 8)
• livecrawl (enum, optional): "fallback" | "preferred"
• type (enum, optional): "auto" | "fast" | "deep"
• contextMaxCharacters (number, optional): Maximum characters for context

Configuration in Pullfrog

Web tools are configured via the permission config in opencode.json:

// In sandbox mode
permission: {
  webfetch: "deny",
  // ...
}

// In normal mode
permission: {
  webfetch: "allow",
  // ...
}

Environment Variables

• OPENCODE_ENABLE_EXA - Enable Exa web search tools (required for "zen" users)

Web Fetch

The webfetch tool directly fetches URLs:

• Direct HTTP fetch with browser-like User-Agent
• HTML to Markdown conversion using Turndown
• Configurable timeout (max 120 seconds)
• 5MB response size limit

---

Comparison

Feature	Claude Code	Gemini CLI	OpenCode
Search Provider	Anthropic	Google	Exa AI
Result Format	Summary	Summary + Citations	Raw content
URL Fetching	Yes (WebFetch)	Yes (web_fetch)	Yes (webfetch)
Grounding/Citations	Unknown	Yes	No
Configurable Results	No	No	Yes (numResults)
Search Depth Options	No	No	Yes (auto/fast/deep)
Live Crawling	Unknown	Fallback only	Configurable

---

Security Considerations

In Pullfrog's sandbox mode:

• Claude Code: WebSearch and WebFetch are explicitly disabled via disallowedTools
• Gemini CLI: No explicit disable mechanism in the wrapper (relies on default behavior)
• OpenCode: webfetch permission set to "deny" in sandbox mode

For public repositories, consider the implications of web search/fetch:

• Fetched content could potentially be used to inject prompts
• Search queries might leak information about the codebase context

---

Proposed Implementation Plan

Option 1: Use Native Agent Web Search (Current State)

Each agent uses its own built-in web search:

• Pros: No additional implementation, leverages each provider's strengths
• Cons: Inconsistent behavior across agents, no unified control

Current gaps:

• Gemini CLI has no explicit disable mechanism for web search in sandbox mode
• No unified way to configure web search across all agents

Option 2: Unified MCP Web Search Tool

Add a web_search tool to the Pullfrog MCP server (mcp/) that all agents can use:

mcp/
├── bash.ts
├── webSearch.ts    # New unified web search tool
└── ...

Implementation approach:

1. Create mcp/webSearch.ts with a provider-agnostic interface:

   export const webSearchTool = {
     name: "web_search",
     description: "Search the web for information",
     inputSchema: {
       type: "object",
       properties: {
         query: { type: "string", description: "Search query" },
         numResults: { type: "number", description: "Number of results (default: 5)" },
       },
       required: ["query"],
     },
   };
   

2. Choose a search provider (options):

   • Exa AI - Already used by OpenCode, good LLM-optimized results
   • Tavily - Popular for AI agents, provides search + content extraction
   • SerpAPI - Google results via API
   • Brave Search API - Privacy-focused alternative

3. Add to MCP server in mcp/server.ts:

   import { webSearchTool, handleWebSearch } from "./webSearch.ts";
   // Register tool...
   

4. Disable native web search for each agent:

   • Claude: Add "WebSearch" to disallowedTools
   • Gemini: Add "google_web_search" to excludeTools in settings.json
   • OpenCode: Set websearch: "deny" in permission config

Pros:

• Consistent behavior across all agents
• Centralized control for security/sandbox modes
• Can filter/sanitize results before returning to agent
• Single API key management

Cons:

• Additional API costs (search provider)
• Loses provider-specific features (e.g., Gemini's grounding metadata)

Option 3: Hybrid Approach

Allow native web search for private repos, use MCP tool for public repos:

// In agent configuration
const useNativeWebSearch = !repo.isPublic;

// Claude
disallowedTools: repo.isPublic ? ["WebSearch", "WebFetch"] : [];

// Gemini  
excludeTools: repo.isPublic ? ["google_web_search", "web_fetch"] : [];

// OpenCode
permission: {
  websearch: repo.isPublic ? "deny" : "allow",
}

Then for public repos, agents would use the MCP web_search tool which:

• Filters sensitive queries
• Sanitizes returned content
• Logs all searches for audit

Recommended Approach

Short-term: Implement Option 3 (Hybrid) with these steps:

1. Add excludeTools: ["google_web_search"] for Gemini in public repo mode
2. Ensure OpenCode websearch permission is properly set for sandbox mode
3. Document the current native web search behavior for each agent

Medium-term: Implement Option 2 (Unified MCP) for public repos:

1. Create mcp/webSearch.ts using Exa AI (consistent with OpenCode)
2. Add EXA_API_KEY to secrets handling
3. Register web search in MCP server
4. Disable native web search for all agents when MCP tool is available
5. Add result sanitization to prevent prompt injection

API Key Requirements

Provider	Environment Variable	Notes
Exa AI	EXA_API_KEY	Already used by OpenCode
Tavily	TAVILY_API_KEY	Popular alternative
Brave	BRAVE_API_KEY	Privacy-focused

For the unified MCP approach, only one search provider API key would be needed.

---

Proposed Implementation Plan

Option A: Unified MCP Web Search Tool

Create a custom MCP tool that provides consistent web search across all agents.

Pros:

• Consistent behavior and results across agents
• Full control over search provider and rate limiting
• Can implement caching and deduplication
• Single point for security filtering

Cons:

• Additional infrastructure (need a search API key)
• Latency from proxying through MCP server

Implementation:

1. Add websearch tool to mcp/ directory
2. Integrate with a search provider (options: Exa AI, SerpAPI, Brave Search, Tavily)
3. Configure each agent to use MCP tool instead of native:
   • Claude: Add to disallowedTools and provide via MCP
   • Gemini: Use excludeTools in settings.json for google_web_search
   • OpenCode: Disable native via permission config

// mcp/websearch.ts
export const websearchTool = {
  name: "websearch",
  description: "Search the web for current information",
  inputSchema: {
    type: "object",
    properties: {
      query: { type: "string", description: "Search query" },
      numResults: { type: "number", description: "Number of results (1-10)" },
    },
    required: ["query"],
  },
  handler: async ({ query, numResults = 5 }) => {
    // Use Exa, Brave, or other search API
    const results = await searchProvider.search(query, numResults);
    return formatResults(results);
  },
};

Option B: Native Tools with Configuration

Keep using each agent's native web search but add consistent configuration.

Pros:

• No additional infrastructure
• Agents can use optimized native implementations
• Less latency

Cons:

• Inconsistent results across agents
• Different capabilities per agent
• Harder to control/audit searches

Implementation:

1. Add websearch_enabled option to payload/config
2. Update each agent wrapper:
   • Claude: Toggle WebSearch in disallowedTools
   • Gemini: Add google_web_search to excludeTools in settings.json
   • OpenCode: Set websearch permission in config

// agents/claude.ts
const disallowedTools = payload.websearchEnabled 
  ? ["Bash"] 
  : ["Bash", "WebSearch", "WebFetch"];

// agents/gemini.ts  
if (!payload.websearchEnabled) {
  newSettings.excludeTools = [...(newSettings.excludeTools || []), "google_web_search"];
}

// agents/opencode.ts
permission: {
  websearch: payload.websearchEnabled ? "allow" : "deny",
  // ...
}

Option C: Hybrid Approach (Recommended)

Use native tools when available, with MCP fallback for consistency.

Implementation:

1. Define a websearch MCP tool as fallback
2. For agents with good native search (Claude, Gemini): use native
3. For agents without (or with unreliable) search: use MCP tool
4. Add configuration to force MCP-only mode if needed

// Per-agent configuration
const agentWebSearchConfig = {
  claude: { useNative: true, mcpFallback: false },
  gemini: { useNative: true, mcpFallback: false },
  opencode: { useNative: false, mcpFallback: true }, // Exa requires API key
};

Required Changes by Option

Change	Option A	Option B	Option C
New MCP tool	Yes	No	Yes
Search API key	Yes	No	Optional
Agent wrapper changes	Yes	Yes	Yes
Action input changes	No	Yes	Yes
External dependencies	Yes	No	Optional

Recommended Next Steps

1. Decide on search provider - If going with MCP approach:

   • Exa AI: Already used by OpenCode, good for code-related searches
   • Brave Search: Privacy-focused, good general search
   • Tavily: Designed for AI agents, includes content extraction

2. Add configuration - New action inputs:

   websearch:
     description: 'Enable web search functionality'
     required: false
     default: 'false'
   

3. Implement per-agent - Start with Option B (simplest), upgrade to C if needed

4. Add security controls - Query filtering, domain allowlists, rate limiting