refactor delegation system, add PR summary comments, and improve code quality (#334)

* refactor delegation system and add PR summary comments

Delegation system:
- replace mode-based delegation with select_mode → delegate two-step flow
- orchestrator crafts self-contained subagent prompts (clean context — no system/repo/event instructions leak)
- add role-based tool filtering via FastMCP authenticate hook (?role=subagent hides orchestrator-only tools)
- add select_mode tool for orchestrator guidance per mode
- add ask_question tool for lightweight research subagents
- extract shared subagent lifecycle into subagent.ts (create, complete, stdout, instructions)
- route set_output to per-subagent state when activeSubagentId is set
- track per-subagent state (SubagentState Map) replacing boolean delegationActive flag
- capture and aggregate AgentUsage across all agents (claude, codex, gemini, opencode)
- write usage summary table to GitHub job summary
- block built-in subagent spawning (Task for Claude, Task(*) for Cursor)
- increase activity timeout from 60s to 300s (subagent thinking phases)
- fix gh CLI misguidance in system prompt — explicitly forbid usage

PR summary comments:
- add prSummaryComment trigger (DB schema + migrations + Zod + UI toggle)
- dispatch mini-effort summary job alongside PR review on pr.created
- add update_pull_request_body MCP tool
- add defaultEffort option to webhook dispatch

Hardening:
- rewrite delegate/selectMode tests with simulated state management
- add toolFiltering.test.ts for role extraction, canAccess, set_output routing
- remove non-null assertions for PULLFROG_TEMP_DIR (proper error throws)
- use fetchWithRetry for direct tarball downloads
- DRY fix for rate limit check in test runner

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: add type keyword to Effort import in handleWebhook.ts

Co-authored-by: Cursor <cursoragent@cursor.com>

* clean up delegation system, improve code quality across the codebase

- simplify delegate tool to instructions + effort params with subagent lifecycle in subagent.ts
- add select_mode and ask_question orchestrator-only tools with canAccess filtering
- replace delegate.test.ts/selectMode.test.ts with toolFiltering.test.ts (live MCP integration)
- add set_output routing for subagent context and AgentUsage tracking across all agents
- add PR summary comment trigger (schema, UI, webhook dispatch with silent flag)
- add update_pull_request_body MCP tool
- fix changed-agents.sh to always include claude canary for non-agent action changes
- fix cursor pagination bug in getSelectedInstallationReposPage
- remove destructuring patterns, inline type definitions, and unsafe type casts
- replace non-null assertions with explicit checks in install.ts
- convert multi-param functions to single param objects (postCleanup, runActionLocal, etc.)
- use isHttpError helper in API routes instead of catch-any patterns
- add adhoc test fixtures for delegation scenarios (context isolation, error handling, synthesis, etc.)

Co-authored-by: Cursor <cursoragent@cursor.com>

* no subagent mutation, one mcp per subagent

* address review feedback: parallel-safe usage tracking, subagent isolation, minor improvements

* fix subagent state isolation: replace Object.freeze with shallow copy

Object.freeze throws TypeErrors when subagent tools (checkout_pr,
report_progress) write scalar properties to toolState. A shallow copy
achieves the same isolation for scalar fields while allowing tools to
work normally. Shared references (subagents Map, usageEntries array)
remain shared for coordination.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
This commit is contained in:
David Blass
2026-02-22 14:12:43 +00:00
committed by pullfrog[bot]
parent a90743e9fe
commit cfd38d82fc
43 changed files with 3293 additions and 2158 deletions
+1 -1
View File
@@ -1,7 +1,7 @@
import { performance } from "node:perf_hooks";
import { log } from "./log.ts";
export const DEFAULT_ACTIVITY_TIMEOUT_MS = 60_000;
export const DEFAULT_ACTIVITY_TIMEOUT_MS = 300_000;
export const DEFAULT_ACTIVITY_CHECK_INTERVAL_MS = 5_000;
type ActivityTimeoutContext = {
+7 -1
View File
@@ -6,7 +6,13 @@ import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
// re-export logging utilities for backward compatibility
export { formatIndentedField, formatJsonValue, log, writeSummary } from "./log.ts";
export {
formatIndentedField,
formatJsonValue,
formatUsageSummary,
log,
writeSummary,
} from "./log.ts";
/**
* Finds a CLI executable path by checking if it's installed globally
+15 -10
View File
@@ -80,7 +80,9 @@ export async function installFromNpmTarball(params: InstallFromNpmTarballParams)
log.debug(`» installing ${params.packageName}@${resolvedVersion}...`);
const tempDir = process.env.PULLFROG_TEMP_DIR!;
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) throw new Error("PULLFROG_TEMP_DIR is not set");
const tarballPath = join(tempDir, "package.tgz");
// Download tarball from npm
@@ -302,7 +304,9 @@ export async function installFromGithubTarball(
log.debug(`» downloading asset from ${assetUrl}...`);
const tempDir = process.env.PULLFROG_TEMP_DIR!;
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) throw new Error("PULLFROG_TEMP_DIR is not set");
const tarballPath = join(tempDir, assetName);
// download the asset
@@ -349,13 +353,12 @@ export async function installFromDirectTarball(
): Promise<string> {
log.info(`» downloading tarball from ${params.url}...`);
const tempDir = process.env.PULLFROG_TEMP_DIR!;
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) throw new Error("PULLFROG_TEMP_DIR is not set");
const tarballPath = join(tempDir, "direct-package.tgz");
const response = await fetch(params.url);
if (!response.ok) {
throw new Error(`failed to download tarball: ${response.status} ${response.statusText}`);
}
const response = await fetchWithRetry(params.url, {}, "failed to download tarball");
if (!response.body) throw new Error("response body is null");
const fileStream = createWriteStream(tarballPath);
@@ -367,8 +370,8 @@ export async function installFromDirectTarball(
mkdirSync(extractDir, { recursive: true });
const tarArgs = ["-xzf", tarballPath, "-C", extractDir];
if (params.stripComponents) {
tarArgs.push(`--strip-components=${params.stripComponents}`);
if (params.stripComponents !== undefined && params.stripComponents > 0) {
tarArgs.push(`--strip-components=${Math.floor(params.stripComponents)}`);
}
log.debug(`» extracting tarball...`);
@@ -401,7 +404,9 @@ export async function installFromDirectTarball(
export async function installFromCurl(params: InstallFromCurlParams): Promise<string> {
log.info(`» installing ${params.executableName}...`);
const tempDir = process.env.PULLFROG_TEMP_DIR!;
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) throw new Error("PULLFROG_TEMP_DIR is not set");
const installScriptPath = join(tempDir, "install.sh");
// Download the install script
+33 -101
View File
@@ -167,7 +167,7 @@ Protected branches (default branch) are blocked from direct pushes in restricted
**Do not attempt to configure git credentials manually** - the ${ghPullfrogMcpName} server handles all authentication internally.
**GitHub** — Prefer using MCP tools from ${ghPullfrogMcpName} for GitHub operations. The \`gh\` CLI is available as a fallback if needed, but MCP tools handle authentication and provide better integration.
**GitHub** — Use MCP tools from ${ghPullfrogMcpName} for all GitHub operations. Never use the \`gh\` CLI — it is not authenticated and will fail. The MCP tools handle authentication, enforce permissions, and integrate with the delegation system.
**Efficiency**: Trust the tools - do not repeatedly verify file contents or git status after operations. If a tool reports success, proceed to the next step. Only verify if you encounter an actual error.
@@ -208,15 +208,6 @@ In case of conflict between instructions, follow this precedence (highest to low
3. Event-level instructions
4. Repo-level instructions`;
const subagentPriorityOrder = `## Priority Order
In case of conflict between instructions, follow this precedence (highest to lowest):
1. Security rules and system instructions (non-overridable)
2. User prompt
3. Orchestrator context
4. Event-level instructions
5. Repo-level instructions`;
export interface ResolvedInstructions {
full: string;
system: string;
@@ -235,7 +226,6 @@ interface ContextSectionsInput {
eventTitleBody: string;
eventMetadata: string;
userQuoted: string;
orchestratorSection?: string | undefined;
}
function buildContextSections(ctx: ContextSectionsInput): string {
@@ -254,12 +244,6 @@ ${ctx.repo}`
${ctx.eventInstructions}`
: "";
const orchestratorSection = ctx.orchestratorSection
? `************* ORCHESTRATOR CONTEXT *************
${ctx.orchestratorSection}`
: "";
const titleBodySection = ctx.eventTitleBody ? `${relatedLabel}\n\n${ctx.eventTitleBody}` : "";
const metadataSection = ctx.eventMetadata ? `--- event context ---\n\n${ctx.eventMetadata}` : "";
@@ -277,9 +261,7 @@ ${titleBodySection}
${metadataSection}`;
return [repoSection, orchestratorSection, eventInstructionsSection, userSection]
.filter(Boolean)
.join("\n\n");
return [repoSection, eventInstructionsSection, userSection].filter(Boolean).join("\n\n");
}
// shared computation for all instruction builders
@@ -340,42 +322,48 @@ ${ctx.contextSections}`;
export function resolveInstructions(ctx: InstructionsContext): ResolvedInstructions {
const inputs = buildCommonInputs(ctx);
const orchestratorTaskSection = `**Required!** You are an orchestrator. Evaluate the task below, then delegate to specialized subagents using \`${ghPullfrogMcpName}/delegate\`.
const orchestratorTaskSection = `**Required!** You are an orchestrator. Evaluate the task below, then delegate to specialized subagents.
### How to delegate
### Step 1: Select a mode
Call \`delegate\` with a mode, effort level, and optional instructions:
- \`mode\`: The workflow to run (see available modes below)
- \`effort\`:
- \`"mini"\`: low-effort and fast, for simple tasks
- \`"auto"\`: medium-effort, good for typical tasks that don't require significant reasoning
- \`"max"\`: high-effort, good for PR reviews and complex coding tasks.
- \`instructions\`: Optional additional context for the subagent. Use this to pass results from earlier delegations or narrow the subagent's focus.
Call \`${ghPullfrogMcpName}/select_mode\` with the appropriate mode name. This returns orchestrator-level guidance on how to handle the task — including suggested delegation phases and prompt-crafting tips.
### Single vs. multi-phase delegation
Available modes:
${ctx.modes.map((m) => `- "${m.name}": ${m.description}`).join("\n")}
**Single delegation** (most common): Evaluate the task, pick the right mode and effort, delegate once. This is the default for most tasks.
### Step 2: Craft subagent prompts and delegate
**Multi-phase delegation** (for complex tasks that benefit from distinct phases):
- Plan then Build: delegate to Plan, read the result, then delegate to Build with the plan as instructions
- Review then Build: delegate to Review for analysis, then delegate to Build to address the findings
- Any combination that makes sense for the task
Based on the guidance from select_mode, craft a focused, self-contained prompt for each subagent, then call \`${ghPullfrogMcpName}/delegate\` with:
- \`instructions\`: Your crafted prompt. **The subagent receives ONLY this text — no other context is added.** Include everything it needs: file paths, constraints, conventions, tool usage instructions, and any relevant context from the codebase or previous phases.
- \`effort\`: \`"mini"\` (simple tasks), \`"auto"\` (typical tasks), or \`"max"\` (complex tasks requiring deep reasoning).
After each delegation, you receive the subagent's result. Use it to decide whether to delegate again and what context to pass.
### Step 3: Post-delegation (your responsibility)
### Effort guidelines
After each delegation, you receive the subagent's summary (via set_output) and a path to its full stdout log (which you can inspect via \`${ghPullfrogMcpName}/file_read\` if needed). Use this to decide whether to delegate again or finalize.
- \`"auto"\` (default): Use for most tasks. Maps to the most capable model.
- \`"mini"\`: Simple, mechanical tasks — issue labeling, adding a comment, trivial changes.
- \`"max"\`: Deep architectural analysis, complex debugging, tasks requiring maximum reasoning.
**Remote operations are YOUR job.** Subagents do NOT have push, PR creation, or other remote-mutating tools. After a subagent that makes code changes completes, you must:
- Push the branch via \`${ghPullfrogMcpName}/push_branch\`
- Create a PR via \`${ghPullfrogMcpName}/create_pull_request\` (if needed)
- Call \`${ghPullfrogMcpName}/report_progress\` with the final summary including PR links
When all delegations are complete, call \`${ghPullfrogMcpName}/set_output\` with the final result — the last subagent's summary or a synthesis of all phases. This is required: it makes the result available as the GitHub Action output for downstream steps.
### Information gathering
Use \`${ghPullfrogMcpName}/ask_question\` to spawn a lightweight subagent that answers a specific question about the codebase. The intermediate exploration context stays in the subagent — only the concise answer returns to you.
### Prompt-crafting rules
- Your subagent has NO context beyond what you write. No repo instructions, no event instructions, no user prompt — only your crafted instructions.
- Include MCP tool names when the subagent needs them (e.g., "commit via \`${ghPullfrogMcpName}/git\`").
- Subagents do NOT have \`push_branch\`, \`create_pull_request\`, \`update_pull_request_body\`, \`delete_branch\`, or \`push_tags\`. Never instruct a subagent to push or create PRs — that is your job as the orchestrator.
- Include branch naming conventions, testing expectations, and commit instructions when relevant.
- For multi-phase flows, pass results from earlier phases directly into the next subagent's prompt.
- The subagent should call \`${ghPullfrogMcpName}/set_output\` with a concise summary when done (include the branch name if code changes were made).
### No-action cases
If the task clearly requires no work (e.g., irrelevant event, duplicate request), you may skip delegation entirely. Call \`${ghPullfrogMcpName}/report_progress\` directly to explain why no action is needed.
### Available modes
${ctx.modes.map((m) => `- "${m.name}": ${m.description}`).join("\n")}`;
If the task clearly requires no work (e.g., irrelevant event, duplicate request), skip delegation entirely. Call \`${ghPullfrogMcpName}/report_progress\` directly to explain why no action is needed.`;
const system = buildSystemPrompt({
bash: ctx.payload.bash,
@@ -409,59 +397,3 @@ ${ctx.modes.map((m) => `- "${m.name}": ${m.description}`).join("\n")}`;
runtime: inputs.runtime,
};
}
// --- subagent instructions (used by delegate tool) ---
interface SubagentInstructionsContext extends InstructionsContext {
mode: Mode;
orchestratorInstructions: string | undefined;
}
export function resolveSubagentInstructions(
ctx: SubagentInstructionsContext
): ResolvedInstructions {
const inputs = buildCommonInputs(ctx);
const subagentTaskSection = `You are operating in **${ctx.mode.name}** mode as a delegated subagent. An orchestrator spawned you and will read your final output to decide what to do next.
### Delegation rules
- The \`delegate\` tool is NOT available to you — complete your task directly using the available tools.
- When you finish, end with a clear, concise summary: what you did, what succeeded, what failed, and any blockers or next steps. The orchestrator uses this to decide whether to delegate again or report final results.
- If you encounter an error you cannot resolve, report it clearly — do not attempt to delegate or re-run yourself.
${ctx.mode.prompt}`;
const system = buildSystemPrompt({
bash: ctx.payload.bash,
trigger: ctx.payload.event.trigger,
priorityOrder: subagentPriorityOrder,
taskSection: subagentTaskSection,
});
const contextSections = buildContextSections({
payload: ctx.payload,
repo: inputs.repo,
eventInstructions: inputs.eventInstructions,
eventTitleBody: inputs.eventTitleBody,
eventMetadata: inputs.eventMetadata,
userQuoted: inputs.userQuoted,
orchestratorSection: ctx.orchestratorInstructions,
});
const full = assembleFullPrompt({
runtime: inputs.runtime,
system,
contextSections,
});
return {
full,
system,
user: inputs.user,
eventInstructions: inputs.eventInstructions,
repo: inputs.repo,
event: inputs.event,
runtime: inputs.runtime,
};
}
+56
View File
@@ -4,6 +4,7 @@
import * as core from "@actions/core";
import { table } from "table";
import type { AgentUsage } from "../agents/shared.ts";
import { isGitHubActions, isInsideDocker } from "./globals.ts";
const isRunnerDebugEnabled = () => core.isDebug();
@@ -298,3 +299,58 @@ export function formatIndentedField(label: string, content: string): string {
}
return formatted;
}
/**
* format aggregated usage data as a markdown table for the GitHub step summary
*/
export function formatUsageSummary(entries: AgentUsage[]): string {
if (entries.length === 0) return "";
const hasCost = entries.some((e) => e.costUsd !== undefined);
const header = hasCost
? "| Agent | Input | Output | Cache Read | Cache Write | Cost |"
: "| Agent | Input | Output | Cache Read | Cache Write |";
const fmt = (n: number) => n.toLocaleString("en-US");
const separatorRow = hasCost
? "| --- | ---: | ---: | ---: | ---: | ---: |"
: "| --- | ---: | ---: | ---: | ---: |";
const rows = entries.map((e) => {
const base = `| ${e.agent} | ${fmt(e.inputTokens)} | ${fmt(e.outputTokens)} | ${fmt(e.cacheReadTokens ?? 0)} | ${fmt(e.cacheWriteTokens ?? 0)} |`;
return hasCost
? `${base} ${e.costUsd !== undefined ? `$${e.costUsd.toFixed(4)}` : "-"} |`
: base;
});
// totals row (only useful when there are multiple entries)
const totalsRows: string[] = [];
if (entries.length > 1) {
const totalInput = entries.reduce((sum, e) => sum + e.inputTokens, 0);
const totalOutput = entries.reduce((sum, e) => sum + e.outputTokens, 0);
const totalCacheRead = entries.reduce((sum, e) => sum + (e.cacheReadTokens ?? 0), 0);
const totalCacheWrite = entries.reduce((sum, e) => sum + (e.cacheWriteTokens ?? 0), 0);
const totalBase = `| **Total** | **${fmt(totalInput)}** | **${fmt(totalOutput)}** | **${fmt(totalCacheRead)}** | **${fmt(totalCacheWrite)}** |`;
if (hasCost) {
const totalCost = entries.reduce((sum, e) => sum + (e.costUsd ?? 0), 0);
totalsRows.push(`${totalBase} **$${totalCost.toFixed(4)}** |`);
} else {
totalsRows.push(totalBase);
}
}
return [
"<details>",
"<summary>Usage</summary>",
"",
header,
separatorRow,
...rows,
...totalsRows,
"",
"</details>",
].join("\n");
}
+37 -44
View File
@@ -7,22 +7,19 @@ import { getJobToken } from "./token.ts";
type JsonPromptInput = Extract<ResolvedPromptInput, object>; // not string
/**
* Controls whether the script should check the reason for the workflow termination.
* It can be either canceled or failed.
* YAML file cannot supply it (not in ENV), so an extra request is required to check it.
* */
// controls whether the script should check the reason for the workflow termination.
// it can be either canceled or failed.
// YAML file cannot supply it (not in ENV), so an extra request is required to check it.
const SHOULD_CHECK_REASON = true;
/**
* Build error comment body with error message and footer
*/
function buildErrorCommentBody(params: {
type BuildErrorCommentBodyParams = {
owner: string;
repo: string;
runId: string | undefined;
isCancellation: boolean;
}): string {
};
function buildErrorCommentBody(params: BuildErrorCommentBodyParams): string {
const workflowRunLink = params.runId
? `[workflow run logs](https://github.com/${params.owner}/${params.repo}/actions/runs/${params.runId})`
: "workflow run logs";
@@ -38,34 +35,31 @@ function buildErrorCommentBody(params: {
return `${errorMessage}${footer}`;
}
/**
* Validate that the progress comment is stuck on "Leaping into action"
* Fetches the comment by ID and checks if it starts with LEAPING_INTO_ACTION_PREFIX
* Returns the comment ID if stuck, null otherwise
*/
type ValidateStuckCommentParams = {
promptInput: JsonPromptInput | null;
octokit: ReturnType<typeof createOctokit>;
owner: string;
repo: string;
};
async function validateStuckProgressComment(
promptInput: JsonPromptInput | null,
octokit: ReturnType<typeof createOctokit>,
owner: string,
repo: string
params: ValidateStuckCommentParams
): Promise<number | null> {
if (!promptInput?.progressCommentId) {
if (!params.promptInput?.progressCommentId) {
log.info("[post] no progressCommentId in prompt input, skipping cleanup");
return null;
}
const commentId = parseInt(promptInput.progressCommentId, 10);
const commentId = parseInt(params.promptInput.progressCommentId, 10);
log.info(`[post] validating progressCommentId from prompt input: ${commentId}`);
try {
const { data: comment } = await octokit.rest.issues.getComment({
owner,
repo,
const commentResult = await params.octokit.rest.issues.getComment({
owner: params.owner,
repo: params.repo,
comment_id: commentId,
});
// check if comment is stuck on "Leaping into action"
if (comment.body?.startsWith(LEAPING_INTO_ACTION_PREFIX)) {
if (commentResult.data.body?.startsWith(LEAPING_INTO_ACTION_PREFIX)) {
log.info(`[post] comment ${commentId} is stuck on "Leaping into action"`);
return commentId;
}
@@ -79,31 +73,31 @@ async function validateStuckProgressComment(
}
}
/**
* Detect if the workflow or its steps is cancelled.
* While the job is still in_progress, the individual steps may have their conclusions set.
*/
async function getIsCancelled(params: {
type GetIsCancelledParams = {
repoContext: ReturnType<typeof parseRepoContext>;
octokit: ReturnType<typeof createOctokit>;
runIdStr: string | undefined;
}): Promise<boolean> {
};
async function getIsCancelled(params: GetIsCancelledParams): Promise<boolean> {
if (!params.runIdStr) return false; // can't check without a run ID — assume failure
try {
const { data: jobs } = await params.octokit.rest.actions.listJobsForWorkflowRun({
const jobsResult = await params.octokit.rest.actions.listJobsForWorkflowRun({
owner: params.repoContext.owner,
repo: params.repoContext.name,
run_id: Number.parseInt(params.runIdStr, 10),
});
// find current job by matching GITHUB_JOB env var
// Note: GITHUB_JOB is the job ID (yaml key), but job.name is the display name
// For matrix jobs, the name includes matrix values like "build (ubuntu-latest, node-18)"
// So we match jobs that START with the job ID
// find current job by matching GITHUB_JOB env var.
// GITHUB_JOB is the job ID (yaml key), but job.name is the display name.
// for matrix jobs, the name includes matrix values like "build (ubuntu-latest, node-18)"
// so we match jobs that START with the job ID
const currentJobName = process.env.GITHUB_JOB;
const currentJob = currentJobName
? jobs.jobs.find((j) => j.name === currentJobName || j.name.startsWith(`${currentJobName} (`))
: jobs.jobs[0]; // fallback to first job
? jobsResult.data.jobs.find(
(j) => j.name === currentJobName || j.name.startsWith(`${currentJobName} (`)
)
: jobsResult.data.jobs[0]; // fallback to first job
if (!currentJob) {
log.warning("[post] could not find current job");
@@ -150,13 +144,12 @@ export async function runPostCleanup(): Promise<void> {
const repoContext = parseRepoContext();
const octokit = createOctokit(token);
// validate that progressCommentId from prompt input is stuck on "Leaping into action"
const commentId = await validateStuckProgressComment(
const commentId = await validateStuckProgressComment({
promptInput,
octokit,
repoContext.owner,
repoContext.name
);
owner: repoContext.owner,
repo: repoContext.name,
});
if (!commentId) return log.info("» [post] no stuck progress comment to update, skipping cleanup");