refactor delegation system, add PR summary comments, and improve code quality (#334)

* refactor delegation system and add PR summary comments

Delegation system:
- replace mode-based delegation with select_mode → delegate two-step flow
- orchestrator crafts self-contained subagent prompts (clean context — no system/repo/event instructions leak)
- add role-based tool filtering via FastMCP authenticate hook (?role=subagent hides orchestrator-only tools)
- add select_mode tool for orchestrator guidance per mode
- add ask_question tool for lightweight research subagents
- extract shared subagent lifecycle into subagent.ts (create, complete, stdout, instructions)
- route set_output to per-subagent state when activeSubagentId is set
- track per-subagent state (SubagentState Map) replacing boolean delegationActive flag
- capture and aggregate AgentUsage across all agents (claude, codex, gemini, opencode)
- write usage summary table to GitHub job summary
- block built-in subagent spawning (Task for Claude, Task(*) for Cursor)
- increase activity timeout from 60s to 300s (subagent thinking phases)
- fix gh CLI misguidance in system prompt — explicitly forbid usage

PR summary comments:
- add prSummaryComment trigger (DB schema + migrations + Zod + UI toggle)
- dispatch mini-effort summary job alongside PR review on pr.created
- add update_pull_request_body MCP tool
- add defaultEffort option to webhook dispatch

Hardening:
- rewrite delegate/selectMode tests with simulated state management
- add toolFiltering.test.ts for role extraction, canAccess, set_output routing
- remove non-null assertions for PULLFROG_TEMP_DIR (proper error throws)
- use fetchWithRetry for direct tarball downloads
- DRY fix for rate limit check in test runner

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: add type keyword to Effort import in handleWebhook.ts

Co-authored-by: Cursor <cursoragent@cursor.com>

* clean up delegation system, improve code quality across the codebase

- simplify delegate tool to instructions + effort params with subagent lifecycle in subagent.ts
- add select_mode and ask_question orchestrator-only tools with canAccess filtering
- replace delegate.test.ts/selectMode.test.ts with toolFiltering.test.ts (live MCP integration)
- add set_output routing for subagent context and AgentUsage tracking across all agents
- add PR summary comment trigger (schema, UI, webhook dispatch with silent flag)
- add update_pull_request_body MCP tool
- fix changed-agents.sh to always include claude canary for non-agent action changes
- fix cursor pagination bug in getSelectedInstallationReposPage
- remove destructuring patterns, inline type definitions, and unsafe type casts
- replace non-null assertions with explicit checks in install.ts
- convert multi-param functions to single param objects (postCleanup, runActionLocal, etc.)
- use isHttpError helper in API routes instead of catch-any patterns
- add adhoc test fixtures for delegation scenarios (context isolation, error handling, synthesis, etc.)

Co-authored-by: Cursor <cursoragent@cursor.com>

* no subagent mutation, one mcp per subagent

* address review feedback: parallel-safe usage tracking, subagent isolation, minor improvements

* fix subagent state isolation: replace Object.freeze with shallow copy

Object.freeze throws TypeErrors when subagent tools (checkout_pr,
report_progress) write scalar properties to toolState. A shallow copy
achieves the same isolation for scalar fields while allowing tools to
work normally. Shared references (subagents Map, usageEntries array)
remain shared for coordination.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>

test/adhoc/delegateAskQuestion.ts

@@ -0,0 +1,62 @@
+import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
+import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts";
+
+/**
+ * delegate-ask-question — orchestrator uses ask_question to gather codebase
+ * info, then uses that answer to craft a targeted delegation.
+ *
+ * tests the ask_question → delegate pipeline: information gathering first,
+ * then action based on gathered context. this validates that the orchestrator
+ * can chain ask_question and delegate as a two-step workflow.
+ */
+
+const fixture = defineFixture(
+  {
+    prompt: `You are an orchestrator. Your task has TWO steps:
+
+STEP 1 — GATHER INFO:
+Use gh_pullfrog/ask_question to ask: "What files are in the root directory of this repository? List them."
+
+STEP 2 — DELEGATE WITH CONTEXT:
+After receiving the answer, select Plan mode via select_mode, then delegate to a subagent with mini effort.
+Your subagent instructions MUST include:
+- The list of files you learned about from step 1
+- Tell the subagent to call gh_pullfrog/set_output with EXACTLY this format: "FILES_FOUND=true,COUNT=<N>" where <N> is the number of files from the list you gave it
+- Do NOT create any branches, commits, or PRs
+
+After delegation completes, call set_output yourself with the subagent's result.
+
+IMPORTANT: You MUST use ask_question BEFORE delegating. The subagent prompt must reference specific files from the ask_question answer.`,
+    effort: "auto",
+    timeout: "10m",
+  },
+  { localOnly: true }
+);
+
+function validator(result: AgentResult): ValidationCheck[] {
+  const output = getStructuredOutput(result);
+  const agentOutput = getAgentOutput(result);
+
+  const setOutputCalled = output !== null;
+  const hasFilesFound = setOutputCalled && /FILES_FOUND=true/i.test(output ?? "");
+  const countMatch = output ? /COUNT=(\d+)/i.exec(output) : null;
+  const hasFileCount = countMatch !== null && parseInt(countMatch[1], 10) > 0;
+  const askQuestionUsed = /» ask_question subagent=/i.test(agentOutput);
+  const delegationOccurred = /» delegating subagent=/i.test(agentOutput);
+
+  return [
+    { name: "set_output", passed: setOutputCalled },
+    { name: "files_found", passed: hasFilesFound },
+    { name: "file_count", passed: hasFileCount },
+    { name: "ask_question_used", passed: askQuestionUsed },
+    { name: "delegation_occurred", passed: delegationOccurred },
+  ];
+}
+
+export const test: TestRunnerOptions = {
+  name: "delegate-ask-question",
+  fixture,
+  validator,
+  env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
+  tags: ["adhoc"],
+};

test/adhoc/delegateContextIsolation.ts

@@ -0,0 +1,71 @@
+import { randomUUID } from "node:crypto";
+import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
+import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts";
+
+/**
+ * delegate-context-isolation — verifies that the subagent's "clean room"
+ * actually works: the orchestrator receives a secret marker in its prompt but
+ * MUST NOT include it in the subagent's instructions.
+ *
+ * the subagent is told to report everything it knows about the task context.
+ * the validator checks that the secret marker does NOT appear in the subagent's
+ * reported context, proving it only sees what the orchestrator explicitly passes.
+ *
+ * this is a critical architectural property of the delegation system.
+ */
+
+const SECRET = randomUUID();
+
+const fixture = defineFixture(
+  {
+    prompt: `You are an orchestrator. READ CAREFULLY.
+
+Your SECRET value is: ${SECRET}
+You MUST remember this secret and include it in your OWN final set_output call.
+You MUST NOT include this secret in the subagent's instructions.
+
+Your task:
+1. Select Plan mode via select_mode.
+2. Delegate to a subagent with mini effort. Your subagent instructions should be EXACTLY:
+   "You are a research subagent. Your only task is to report what you know. Call gh_pullfrog/set_output with a summary of: (a) the instructions you were given, (b) any context you have about the parent task. Start your output with CONTEXT_REPORT:"
+   DO NOT mention the secret value anywhere in the subagent instructions.
+3. After delegation, call set_output with: "SECRET=${SECRET},SUBAGENT_SAID=<paste the subagent's output here>"
+
+CRITICAL: Your final output MUST contain "SECRET=${SECRET}" exactly.`,
+    effort: "auto",
+    timeout: "8m",
+  },
+  { localOnly: true }
+);
+
+function validator(result: AgentResult): ValidationCheck[] {
+  const output = getStructuredOutput(result);
+  const agentOutput = getAgentOutput(result);
+
+  const setOutputCalled = output !== null;
+  // orchestrator should include at least the first segment of the UUID (proving it read it).
+  // some agents paraphrase or truncate, so matching the first 8 hex chars is sufficient.
+  const secretPrefix = SECRET.slice(0, 8);
+  const secretInOutput = setOutputCalled && output !== null && output.includes(secretPrefix);
+  const delegationOccurred = /» delegating subagent=/i.test(agentOutput);
+
+  // the subagent's context report should NOT contain any part of the secret
+  const subagentMatch = output ? /SUBAGENT_SAID=([\s\S]*)/i.exec(output) : null;
+  const subagentOutput = subagentMatch ? subagentMatch[1] : "";
+  const secretLeaked = subagentOutput.includes(secretPrefix);
+
+  return [
+    { name: "set_output", passed: setOutputCalled },
+    { name: "secret_in_output", passed: secretInOutput },
+    { name: "delegation_occurred", passed: delegationOccurred },
+    { name: "no_secret_leak", passed: !secretLeaked },
+  ];
+}
+
+export const test: TestRunnerOptions = {
+  name: "delegate-context-isolation",
+  fixture,
+  validator,
+  env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
+  tags: ["adhoc"],
+};

test/adhoc/delegateErrorHandling.ts

@@ -0,0 +1,58 @@
+import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
+import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts";
+
+/**
+ * delegate-error-handling — orchestrator delegates a task that will fail,
+ * then must handle the failure gracefully and report it.
+ *
+ * the subagent is told to read a file that doesn't exist, which will cause
+ * file_read to return an error. the orchestrator should detect the subagent
+ * failure (via the delegate tool's return value) and report it clearly.
+ *
+ * tests error propagation through the delegation system and the orchestrator's
+ * ability to reason about failure modes rather than blindly forwarding results.
+ */
+
+const fixture = defineFixture(
+  {
+    prompt: `You are an orchestrator. This test validates error handling.
+
+1. Select Plan mode via select_mode.
+2. Delegate to a subagent with mini effort. Subagent instructions:
+   "Use gh_pullfrog/file_read to read the file 'this-file-does-not-exist-anywhere.xyz'. Report what you find by calling gh_pullfrog/set_output with the file content. If the file cannot be read, call gh_pullfrog/set_output with 'FILE_NOT_FOUND'."
+3. After the delegation completes, examine the result. The subagent should have reported FILE_NOT_FOUND or an error.
+4. Call set_output with EXACTLY: "ERROR_HANDLED=true,REASON=<brief description of what went wrong>"
+
+If the delegation failed entirely (subagent crashed), still call set_output with "ERROR_HANDLED=true,REASON=delegation_failed".
+
+The point of this test is that you handle the error gracefully and report it — not that you succeed at reading the file.`,
+    effort: "auto",
+    timeout: "8m",
+  },
+  { localOnly: true }
+);
+
+function validator(result: AgentResult): ValidationCheck[] {
+  const output = getStructuredOutput(result);
+  const agentOutput = getAgentOutput(result);
+
+  const setOutputCalled = output !== null;
+  const errorHandled = setOutputCalled && /ERROR_HANDLED=true/i.test(output ?? "");
+  const hasReason = setOutputCalled && /REASON=\S+/i.test(output ?? "");
+  const delegationOccurred = /» delegating subagent=/i.test(agentOutput);
+
+  return [
+    { name: "set_output", passed: setOutputCalled },
+    { name: "error_handled", passed: errorHandled },
+    { name: "reason_provided", passed: hasReason },
+    { name: "delegation_occurred", passed: delegationOccurred },
+  ];
+}
+
+export const test: TestRunnerOptions = {
+  name: "delegate-error-handling",
+  fixture,
+  validator,
+  env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
+  tags: ["adhoc"],
+};

test/adhoc/delegateFileRead.ts

@@ -0,0 +1,57 @@
+import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
+import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts";
+
+/**
+ * delegate-file-read — orchestrator delegates a subagent to read a real file
+ * from the repository and return its content.
+ *
+ * tests the full delegation pipeline: mode selection → prompt crafting with MCP
+ * tool references → subagent file read → result propagation back to orchestrator.
+ *
+ * unlike the basic delegate test (which just echoes a hardcoded string), this
+ * requires the subagent to actually use MCP tools (file_read) to interact with
+ * the repo and return derived data.
+ */
+
+const fixture = defineFixture(
+  {
+    prompt: `You are an orchestrator. Your task:
+
+1. Select the Plan mode via select_mode.
+2. Delegate to a subagent with mini effort. Craft instructions telling it to:
+   - Use gh_pullfrog/file_read to read the file "README.md" from the repository root
+   - Count the total number of lines in the file
+   - Call gh_pullfrog/set_output with EXACTLY this format: "LINES=<number>" where <number> is the line count (e.g., "LINES=42")
+   - Do NOT create any branches, commits, or PRs
+3. After the delegation completes, call set_output with the subagent's result (the LINES=<number> string).
+
+IMPORTANT: Your subagent prompt must include the exact MCP tool names (gh_pullfrog/file_read, gh_pullfrog/set_output).`,
+    effort: "auto",
+    timeout: "8m",
+  },
+  { localOnly: true }
+);
+
+function validator(result: AgentResult): ValidationCheck[] {
+  const output = getStructuredOutput(result);
+  const agentOutput = getAgentOutput(result);
+
+  const setOutputCalled = output !== null;
+  const linesMatch = output ? /LINES=(\d+)/i.exec(output) : null;
+  const hasLineCount = linesMatch !== null && parseInt(linesMatch[1], 10) > 0;
+  const delegationOccurred = /» delegating subagent=/i.test(agentOutput);
+
+  return [
+    { name: "set_output", passed: setOutputCalled },
+    { name: "line_count_reported", passed: hasLineCount },
+    { name: "delegation_occurred", passed: delegationOccurred },
+  ];
+}
+
+export const test: TestRunnerOptions = {
+  name: "delegate-file-read",
+  fixture,
+  validator,
+  env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
+  tags: ["adhoc"],
+};

test/adhoc/delegateSynthesis.ts

@@ -0,0 +1,74 @@
+import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
+import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts";
+
+/**
+ * delegate-synthesis — orchestrator delegates two research tasks to separate
+ * subagents, then synthesizes their results into a combined answer.
+ *
+ * phase 1: subagent reads README.md and extracts the first line.
+ * phase 2: subagent counts how many .md files exist via list_directory.
+ * synthesis: orchestrator combines both pieces of info into the final output.
+ *
+ * this tests the orchestrator's ability to:
+ * - run multiple sequential delegations
+ * - pass specific, different instructions to each subagent
+ * - extract and combine results from separate delegation phases
+ * - produce a structured final output from heterogeneous subagent responses
+ */
+
+const fixture = defineFixture(
+  {
+    prompt: `You are an orchestrator. You must delegate TWO research tasks and SYNTHESIZE the results.
+
+PHASE 1 — GET FIRST LINE:
+Select Plan mode via select_mode, then delegate with mini effort.
+Subagent instructions: "Use gh_pullfrog/file_read to read 'README.md'. Extract the FIRST LINE of the file. Call gh_pullfrog/set_output with just the first line of text (nothing else)."
+
+PHASE 2 — COUNT FILES:
+Select Plan mode again, then delegate with mini effort.
+Subagent instructions: "Use gh_pullfrog/list_directory to list the root directory '.'. Count how many items are listed. Call gh_pullfrog/set_output with just the number (nothing else)."
+
+SYNTHESIS:
+After both phases complete, YOU (the orchestrator) must call set_output with EXACTLY:
+"FIRST_LINE=<first line from phase 1>,FILE_COUNT=<number from phase 2>"
+
+Both pieces must come from the respective subagent results. Do NOT read the files yourself.`,
+    effort: "auto",
+    timeout: "10m",
+  },
+  { localOnly: true }
+);
+
+function validator(result: AgentResult): ValidationCheck[] {
+  const output = getStructuredOutput(result);
+  const agentOutput = getAgentOutput(result);
+
+  const setOutputCalled = output !== null;
+
+  // should have two delegation calls
+  const delegationMatches = agentOutput.match(/» delegating subagent=/g);
+  const twoDelegations = delegationMatches !== null && delegationMatches.length >= 2;
+
+  // FIRST_LINE should be a non-empty string (the first line of README.md)
+  const firstLineMatch = output ? /FIRST_LINE=([^,]+)/i.exec(output) : null;
+  const hasFirstLine = firstLineMatch !== null && firstLineMatch[1].trim().length > 0;
+
+  // FILE_COUNT should be a positive number
+  const countMatch = output ? /FILE_COUNT=(\d+)/i.exec(output) : null;
+  const hasFileCount = countMatch !== null && parseInt(countMatch[1], 10) > 0;
+
+  return [
+    { name: "set_output", passed: setOutputCalled },
+    { name: "two_delegations", passed: twoDelegations },
+    { name: "first_line_extracted", passed: hasFirstLine },
+    { name: "file_count_extracted", passed: hasFileCount },
+  ];
+}
+
+export const test: TestRunnerOptions = {
+  name: "delegate-synthesis",
+  fixture,
+  validator,
+  env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
+  tags: ["adhoc"],
+};

test/adhoc/delegateTimeout.ts

@@ -12,8 +12,8 @@ import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts"
 
 const fixture = defineFixture(
   {
-    prompt: `Delegate to the Plan mode with auto effort. Pass these instructions to the subagent:
-"Carefully analyze the following engineering question and provide a thorough response, then call set_output with the value 'DELEGATE_TIMEOUT_PASSED'.
+    prompt: `Select the Plan mode via select_mode, then delegate with auto effort. Your subagent instructions should be:
+"Carefully analyze the following engineering question. Think through each point thoroughly before finishing.
 
 Question: Design a comprehensive error handling strategy for a distributed microservices architecture. Consider:
 1. Circuit breaker patterns — when to open, half-open, close. What thresholds to use.
@@ -22,7 +22,9 @@ Question: Design a comprehensive error handling strategy for a distributed micro
 4. Health check endpoints — liveness vs readiness probes, dependency health checks.
 5. Graceful degradation — fallback responses, feature flags, bulkhead pattern.
 
-Provide a detailed analysis covering ALL 5 points with concrete examples before calling set_output."`,
+After you have finished your analysis, call gh_pullfrog/set_output with EXACTLY the string 'DELEGATE_TIMEOUT_PASSED' — not your analysis, just that exact string."
+
+After the delegation completes, call set_output yourself with the subagent's result (forward it verbatim).`,
     effort: "auto",
     timeout: "8m",
   },
@@ -35,8 +37,7 @@ function validator(result: AgentResult): ValidationCheck[] {
 
   const setOutputCalled = output !== null;
   const correctValue = setOutputCalled && /DELEGATE_TIMEOUT_PASSED/i.test(output);
-  const delegationOccurred = /» delegating to \w+ mode/i.test(agentOutput);
-  // the critical check: no activity timeout occurred
+  const delegationOccurred = /» delegating subagent=/i.test(agentOutput);
   const noActivityTimeout = !/activity timeout/i.test(agentOutput);
 
   return [

test/adhoc/delegateTwoPhase.ts

@@ -0,0 +1,79 @@
+import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
+import {
+  defineFixture,
+  generateTestMarker,
+  getAgentOutput,
+  getStructuredOutput,
+} from "../utils.ts";
+
+/**
+ * delegate-two-phase — orchestrator runs two sequential delegations where
+ * the second phase depends on state created by the first.
+ *
+ * phase 1: subagent writes a file with a unique marker.
+ * phase 2: subagent reads the file and reports its content.
+ *
+ * tests that file state persists across delegation phases (both subagents
+ * run in the same working directory) and that the orchestrator correctly
+ * chains phases by passing context from phase 1 into phase 2's instructions.
+ */
+
+const marker = generateTestMarker("PULLFROG_PHASE_MARKER");
+
+const fixture = defineFixture(
+  {
+    prompt: `You are an orchestrator. You must run TWO sequential delegation phases.
+
+First, read the marker value: run echo $PULLFROG_PHASE_MARKER
+
+PHASE 1 — WRITE:
+Select Plan mode via select_mode, then delegate with mini effort.
+Subagent instructions: "Use gh_pullfrog/file_write to write a file called 'delegation-test.txt' with the content '<MARKER_VALUE>'. Then call gh_pullfrog/set_output with 'PHASE1_DONE'. Do not create branches or PRs."
+(Replace <MARKER_VALUE> with the actual marker value you read.)
+
+PHASE 2 — READ AND VERIFY:
+After Phase 1 completes, select Plan mode again and delegate with mini effort.
+Subagent instructions: "Use gh_pullfrog/file_read to read the file 'delegation-test.txt'. Call gh_pullfrog/set_output with the EXACT content of the file. Do not create branches or PRs."
+
+After both phases complete, call set_output with: "WRITTEN=<marker>,READ=<what_phase2_returned>"`,
+    effort: "auto",
+    timeout: "10m",
+    bash: "enabled",
+  },
+  { localOnly: true }
+);
+
+function validator(result: AgentResult): ValidationCheck[] {
+  const output = getStructuredOutput(result);
+  const agentOutput = getAgentOutput(result);
+  const secret = marker.value;
+
+  const setOutputCalled = output !== null;
+
+  // two delegation calls should appear in logs
+  const delegationMatches = agentOutput.match(/» delegating subagent=/g);
+  const twoDelegations = delegationMatches !== null && delegationMatches.length >= 2;
+
+  // the marker should appear in both WRITTEN= and READ= sections.
+  // use greedy match for READ= since subagents may prefix with "content:" etc.
+  const writtenMatch = output ? /WRITTEN=([^\s,]+)/i.exec(output) : null;
+  const markerWritten = writtenMatch?.[1].includes(secret) ?? false;
+  const readSection = output ? /READ=(.+)/i.exec(output) : null;
+  const markerRead = readSection?.[1].includes(secret) ?? false;
+
+  return [
+    { name: "set_output", passed: setOutputCalled },
+    { name: "two_delegations", passed: twoDelegations },
+    { name: "marker_written", passed: markerWritten },
+    { name: "marker_read_back", passed: markerRead },
+  ];
+}
+
+export const test: TestRunnerOptions = {
+  name: "delegate-two-phase",
+  fixture,
+  validator,
+  agentEnv: marker.agentEnv,
+  env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
+  tags: ["adhoc"],
+};