refactor delegation system, add PR summary comments, and improve code quality (#334)
* refactor delegation system and add PR summary comments Delegation system: - replace mode-based delegation with select_mode → delegate two-step flow - orchestrator crafts self-contained subagent prompts (clean context — no system/repo/event instructions leak) - add role-based tool filtering via FastMCP authenticate hook (?role=subagent hides orchestrator-only tools) - add select_mode tool for orchestrator guidance per mode - add ask_question tool for lightweight research subagents - extract shared subagent lifecycle into subagent.ts (create, complete, stdout, instructions) - route set_output to per-subagent state when activeSubagentId is set - track per-subagent state (SubagentState Map) replacing boolean delegationActive flag - capture and aggregate AgentUsage across all agents (claude, codex, gemini, opencode) - write usage summary table to GitHub job summary - block built-in subagent spawning (Task for Claude, Task(*) for Cursor) - increase activity timeout from 60s to 300s (subagent thinking phases) - fix gh CLI misguidance in system prompt — explicitly forbid usage PR summary comments: - add prSummaryComment trigger (DB schema + migrations + Zod + UI toggle) - dispatch mini-effort summary job alongside PR review on pr.created - add update_pull_request_body MCP tool - add defaultEffort option to webhook dispatch Hardening: - rewrite delegate/selectMode tests with simulated state management - add toolFiltering.test.ts for role extraction, canAccess, set_output routing - remove non-null assertions for PULLFROG_TEMP_DIR (proper error throws) - use fetchWithRetry for direct tarball downloads - DRY fix for rate limit check in test runner Co-authored-by: Cursor <cursoragent@cursor.com> * fix: add type keyword to Effort import in handleWebhook.ts Co-authored-by: Cursor <cursoragent@cursor.com> * clean up delegation system, improve code quality across the codebase - simplify delegate tool to instructions + effort params with subagent lifecycle in subagent.ts - add select_mode and ask_question orchestrator-only tools with canAccess filtering - replace delegate.test.ts/selectMode.test.ts with toolFiltering.test.ts (live MCP integration) - add set_output routing for subagent context and AgentUsage tracking across all agents - add PR summary comment trigger (schema, UI, webhook dispatch with silent flag) - add update_pull_request_body MCP tool - fix changed-agents.sh to always include claude canary for non-agent action changes - fix cursor pagination bug in getSelectedInstallationReposPage - remove destructuring patterns, inline type definitions, and unsafe type casts - replace non-null assertions with explicit checks in install.ts - convert multi-param functions to single param objects (postCleanup, runActionLocal, etc.) - use isHttpError helper in API routes instead of catch-any patterns - add adhoc test fixtures for delegation scenarios (context isolation, error handling, synthesis, etc.) Co-authored-by: Cursor <cursoragent@cursor.com> * no subagent mutation, one mcp per subagent * address review feedback: parallel-safe usage tracking, subagent isolation, minor improvements * fix subagent state isolation: replace Object.freeze with shallow copy Object.freeze throws TypeErrors when subagent tools (checkout_pr, report_progress) write scalar properties to toolState. A shallow copy achieves the same isolation for scalar fields while allowing tools to work normally. Shared references (subagents Map, usageEntries array) remain shared for coordination. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
This commit is contained in:
committed by
pullfrog[bot]
parent
a90743e9fe
commit
cfd38d82fc
@@ -0,0 +1,74 @@
|
||||
import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
|
||||
import { defineFixture, getAgentOutput, getStructuredOutput } from "../utils.ts";
|
||||
|
||||
/**
|
||||
* delegate-synthesis — orchestrator delegates two research tasks to separate
|
||||
* subagents, then synthesizes their results into a combined answer.
|
||||
*
|
||||
* phase 1: subagent reads README.md and extracts the first line.
|
||||
* phase 2: subagent counts how many .md files exist via list_directory.
|
||||
* synthesis: orchestrator combines both pieces of info into the final output.
|
||||
*
|
||||
* this tests the orchestrator's ability to:
|
||||
* - run multiple sequential delegations
|
||||
* - pass specific, different instructions to each subagent
|
||||
* - extract and combine results from separate delegation phases
|
||||
* - produce a structured final output from heterogeneous subagent responses
|
||||
*/
|
||||
|
||||
const fixture = defineFixture(
|
||||
{
|
||||
prompt: `You are an orchestrator. You must delegate TWO research tasks and SYNTHESIZE the results.
|
||||
|
||||
PHASE 1 — GET FIRST LINE:
|
||||
Select Plan mode via select_mode, then delegate with mini effort.
|
||||
Subagent instructions: "Use gh_pullfrog/file_read to read 'README.md'. Extract the FIRST LINE of the file. Call gh_pullfrog/set_output with just the first line of text (nothing else)."
|
||||
|
||||
PHASE 2 — COUNT FILES:
|
||||
Select Plan mode again, then delegate with mini effort.
|
||||
Subagent instructions: "Use gh_pullfrog/list_directory to list the root directory '.'. Count how many items are listed. Call gh_pullfrog/set_output with just the number (nothing else)."
|
||||
|
||||
SYNTHESIS:
|
||||
After both phases complete, YOU (the orchestrator) must call set_output with EXACTLY:
|
||||
"FIRST_LINE=<first line from phase 1>,FILE_COUNT=<number from phase 2>"
|
||||
|
||||
Both pieces must come from the respective subagent results. Do NOT read the files yourself.`,
|
||||
effort: "auto",
|
||||
timeout: "10m",
|
||||
},
|
||||
{ localOnly: true }
|
||||
);
|
||||
|
||||
function validator(result: AgentResult): ValidationCheck[] {
|
||||
const output = getStructuredOutput(result);
|
||||
const agentOutput = getAgentOutput(result);
|
||||
|
||||
const setOutputCalled = output !== null;
|
||||
|
||||
// should have two delegation calls
|
||||
const delegationMatches = agentOutput.match(/» delegating subagent=/g);
|
||||
const twoDelegations = delegationMatches !== null && delegationMatches.length >= 2;
|
||||
|
||||
// FIRST_LINE should be a non-empty string (the first line of README.md)
|
||||
const firstLineMatch = output ? /FIRST_LINE=([^,]+)/i.exec(output) : null;
|
||||
const hasFirstLine = firstLineMatch !== null && firstLineMatch[1].trim().length > 0;
|
||||
|
||||
// FILE_COUNT should be a positive number
|
||||
const countMatch = output ? /FILE_COUNT=(\d+)/i.exec(output) : null;
|
||||
const hasFileCount = countMatch !== null && parseInt(countMatch[1], 10) > 0;
|
||||
|
||||
return [
|
||||
{ name: "set_output", passed: setOutputCalled },
|
||||
{ name: "two_delegations", passed: twoDelegations },
|
||||
{ name: "first_line_extracted", passed: hasFirstLine },
|
||||
{ name: "file_count_extracted", passed: hasFileCount },
|
||||
];
|
||||
}
|
||||
|
||||
export const test: TestRunnerOptions = {
|
||||
name: "delegate-synthesis",
|
||||
fixture,
|
||||
validator,
|
||||
env: { GITHUB_REPOSITORY: "pullfrog/test-repo" },
|
||||
tags: ["adhoc"],
|
||||
};
|
||||
Reference in New Issue
Block a user