From e4e93ea6d3c787a7bc6cc285d168deb0eab7ac83 Mon Sep 17 00:00:00 2001 From: Colin McDonnell Date: Fri, 8 May 2026 19:28:24 +0000 Subject: [PATCH] PR summary as agent-edited tmpfile snapshot (#568) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * PR summary as agent-edited tmpfile snapshot Replaces the comment-based PR summary path (and the in-progress update_pr_summary tool from #534) with a snapshot file the agent edits in place during Review / IncrementalReview / pr-summary Task runs. The server seeds the tmpfile with the previous snapshot (incremental) or a stable scaffold (first run), exposes the path via select_mode, and reads it back at end-of-run to persist to WorkflowRun.summarySnapshot and (when the prSummaryComment toggle is on) splice into the PR description body. Why a tmpfile rather than a tool call: incremental snapshot edits are output-token-cheap when the agent uses native file-editing tools, and range-diff cleanly across runs because section headings are stable. The agent never has to regurgitate the full snapshot to update it. Gating: snapshot generation is opt-in via either prSummaryComment="enabled" (splice into PR body) or prReReview="enabled" (snapshot feeds future incremental review runs as context). Users who disable both pay nothing end-to-end — no seeding, DB write, or body splice. Behavior changes: - Drop the Summarize mode and the Summary comment type entirely; the rolling summary is no longer a separate run shape. - pull_request_synchronize with re-review off and summary on still dispatches a silent pr-summary Task, but it edits the snapshot file instead of posting a fresh comment. - /api/repo/.../pr/.../summary-comment now returns { snapshot: string | null } from the DB instead of fetching a comment via GraphQL. URL kept stable so deployed older actions degrade gracefully. - summaryCommentNodeId is retained on WorkflowRun for legacy data and a future backfill of pre-snapshot comment-based summaries. Supersedes #534. The commit-tool/sub-agent direction in that PR is abandoned in favor of this file-based shape. * address review pass #1: synchronize fallback, splice idempotency, docs * address review pass #2: in-flight skip should not race summary fallback * address review pass #3: signal-handler flush, doc clarifications * address review pass #4: in-flight persist promise + bounded body-splice timeout * address review pass #5: defensive catch on persist worker, doc nit * add summary-stale post-run gate When generateSummary is set, we capture the bytes of the seeded snapshot file and pass them to the agent's post-run loop alongside the file path. After each agent attempt, the loop diffs the current file against the seed; if they're byte-identical the agent never touched it, and we nudge once via a resume turn (similar to the dirty-tree gate, but soft and fire-once so smaller models that legitimately decide no edit is warranted don't burn the retry budget). Mostly defends against forgetful smaller models on the Review path — their mode prompt asks them to edit the snapshot file, but the multi-step instruction can fall through when the diff is large. * trigger: retry vercel preview build * fix(action): drop unused re-export that pulled node:fs/promises into next bundle action/internal/index.ts was re-exporting DEFAULT_PR_SUMMARY_INSTRUCTIONS from action/utils/prSummary.ts, but nothing in the next.js app imports it. prSummary.ts uses node:fs/promises, and pullfrog/internal is aliased into the next bundle by next.config.ts, which made turbopack try to resolve node:fs/promises in client chunks and fail with: the chunking context (unknown) does not support external modules (request: node:fs/promises) drop the re-export — selectMode.ts (the only real consumer) already imports it directly from action/utils/prSummary.ts. * firewall PR summary snapshot from user instructions; resurrect rich format for Review The agent-internal snapshot (the markdown file the agent edits in place across runs) is exclusively durable context for future agent runs — user-supplied summarization instructions warp it and degrade that context. Drop the prSummaryCommentInstructions read path end-to-end: - handleWebhook: stop reading prSummaryCommentInstructions, stop passing prSummaryInstructions through dispatch options - action payload + ToolState + selectMode addendum: drop the instructions appendix; the snapshot prompt is fixed, not user-shaped - TriggersSettings: drop the InstructionsEditor for prSummaryCommentInstructions - prSummary.ts: reframe DEFAULT_PR_SUMMARY_INSTRUCTIONS as agent-targeted (durable context, not human-facing prose) Prisma columns (prSummaryComment, prSummaryCommentInstructions) and the matching zod schema entry stay for graceful retreat. Separately, resurrect PR_SUMMARY_FORMAT (deleted along with the Summarize mode in the original PR) and wire it into Review mode only. Initial PR reviews now include a structured summary section in the review body using the rich format (TL;DR, key changes, ## sections with before/after, file-link trails). IncrementalReview keeps its existing terser bullet-list shape since re-review bodies are deltas, not introductions. The user-facing review summary and the agent-internal snapshot are deliberately separate artifacts with separate prompts and zero shared content. * address review comments: prompt self-consistency + stale-doc cleanup PR 568 self-review (4232488109) flagged a self-contradiction the firewall commit introduced and three stale doc references that survived. - action/modes.ts: Review-mode step 2's trivial-PR shortcut said `submit "Reviewed — no issues found." per step 5`, but step 5's rewrite removed exactly that preamble. Aligned both: trivial PRs and no-actionable-issues PRs now produce a body that opens with "No new issues found." followed by the PR summary, so the user gets the headline up front and still sees what was reviewed. - docs/pr-reviews.mdx: dropped the "customize the summary style with Summary instructions in the console" sentence (the editor was removed in the firewall commit). Replaced with a note that the snapshot uses Pullfrog's built-in format and is not user-customizable. - wiki/prompt.md, wiki/modes.md: rewrote the snapshot-prompt entries to reflect the firewall — DEFAULT_PR_SUMMARY_INSTRUCTIONS is the entire prompt, prSummaryCommentInstructions is no longer wired in. * drop orphaned prSummaryCommentInstructions column Prod audit (455 repos): 5 non-null rows on a single account, all containing the literal placeholder text from the InstructionsEditor we removed in the firewall commit. No account has an intentional preference set, so silent-ignore (the keep-for-retreat option) costs us nothing meaningful while leaving an orphan column in the schema. Drop it. - prisma/schema.prisma: remove the column - prisma/migrations/20260506000000_drop_pr_summary_comment_instructions: ALTER TABLE ... DROP COLUMN - utils/schemas/triggers.ts: drop the matching zod entry * drop body splicing; snapshot is internal-only User-visible PR summarization continues to ship in Review and IncrementalReview review bodies (which already render PR_SUMMARY_FORMAT and "Reviewed changes" respectively). The snapshot tmpfile is now purely durable cross-run agent context — seed, edit-in-place, save to DB, feed the next run. Massive simplification: the body splice mechanics, the two-toggle gating matrix, the summaryHandlingCovered race tracking, and the synchronize summary-only Task fallback all go away. Code: - prSummary.ts: drop splice/strip/marker code (`splicePrSummary`, `stripExistingSummaryBlock`, `buildSummaryBlock`, `extractPrSummary`, PULLFROG_SUMMARY_START/END). keep scaffold, instructions, seed/read. - main.ts: rename persistAndPostSummary -> persistSummary; collapse to a single DB PATCH. drop pulls.get/pulls.update, drop AbortSignal timeout, drop in-flight promise machinery, drop prSummaryToBody plumbing. - ToolState: add summarySeed (replaces local var in main.ts so persist can compare). drop prSummaryToBody and summaryPersistInFlight. - persistSummary now compares against the seed and skips the DB write with a warning when unchanged — saving the seed verbatim is either a no-op or persists the placeholder scaffold, neither useful. - postRun.ts: when summary-stale is the only failing gate and the resume turn itself fails, restore the pre-resume successful result and break. symmetric with the existing reflection-failure preservation. summary-stale can no longer flip a successful run to failed. Webhook: - pull_request_opened: generateSummary follows prReReview only (the snapshot has no consumer when re-review is off). - pull_request_synchronize: collapses to "if prReReview enabled, dispatch IncrementalReview". the summaryHandlingCovered flag, the same-SHA/in-flight coordination it was protecting, and the summary-only Task fallback all delete cleanly. UI / config: - drop SummarizePRsTrigger (the toggle gated body splice; with that gone it has no behavior). drop sidebar entry, console import, Text icon import. - drop prSummaryComment from triggers zod schema, prisma schema, preview settings script. Migration: squash the two existing migrations into one timestamped 20260507000000_pr_summary_snapshot covering all three column changes (add summarySnapshot on workflow_runs, drop prSummaryCommentInstructions and prSummaryComment on repos). repo convention is one migration per PR. Action: bump 0.0.203 -> 0.0.205 (payload contract changed: prSummaryToBody removed; main is at 0.0.204). Out-of-diff cleanup: - review.ts:190 + review.test.ts:651 — "Reviewed — no issues found." -> "No new issues found." to match the canonical body in modes.ts. Verified: pnpm typecheck clean, pnpm lint clean, postRun + review tests pass, dev DB reset against production and the squashed migration applied cleanly (summarySnapshot present, prSummaryComment / prSummaryCommentInstructions both gone). * re-orient snapshot toward functional summary; drop prior-review-feedback section Empirical audit on preview-568 PR #5 showed the snapshot IS load-bearing for the orchestrator: lens-dispatch prompts on incremental runs carried forward context from the snapshot's risk register (e.g. "the JSDoc explicitly scopes to code points — do not flag grapheme-cluster issues" on the surrogate-pair fix run, "consistency with native padStart" on the padStart-added run). The orchestrator was reading the snapshot, reasoning about it, and using it to anti-prime / focus subagents — exactly the high-leverage path. My earlier "snapshot is write-only" claim was wrong. The shape, however, was steering it toward review-history-log instead of functional summary. This commit re-orients: - prSummary.ts: replace the four-section scaffold (~580 chars of placeholder italics under "What this PR does / Key changes / Risk / Reviewed in prior runs") with a minimal seed (~150 chars: just a header + a one-line comment about what the file is for). different PRs warrant different organization; forcing a refactor and a feature into the same template is procrustean. minimal seed also makes the unchanged-from-seed gate in persistSummary more sensitive. - selectMode.ts addendum: rewrite around three principles. (1) the snapshot is a FUNCTIONAL summary of what the PR does and the risks it carries, not a chronological review log — commit history can already be reconstructed from list_pull_request_reviews. (2) the orchestrator should USE the snapshot during triage and dispatch — concrete example given of carrying snapshot context into subagent lens prompts. (3) structure is the agent's call; stable headings make snapshots range-diff cleanly when they fit, but riff when they don't. - modes.ts IncrementalReview: drop the "Prior review feedback" checklist from the user-facing review body (step 6b gone, step 7 ELSE IFs cleaned up). It duplicated content that's already covered by the Reviewed-changes bullets and tracked durably in the snapshot for the next agent run; in the user-facing body it was noise. step 3 still fetches prior reviews but its role is now just filtering aggregation in step 5, not rendering. - AGENTS.md: codify "no follow-ups" rule. when an issue is identified during code review, fix it in this PR — PR scope does not constrain quality. follow-up TODOs are forbidden as a substitute for doing the work now. Empirical evidence supporting the re-orientation: - Run 25568912293 (PR#5 incr1, surrogate-pair fix): orchestrator's correctness lens dispatch said "Do NOT flag grapheme-cluster issues — the JSDoc scopes to code points." The grapheme-cluster framing was not in the diff; it was downstream of the snapshot's prior risk-section framing of truncate's contract. Snapshot influencing dispatch. - Run 25569054779 (PR#5 incr2, padStart added): orchestrator's correctness lens dispatch enumerated edge cases including "consistency with native String.prototype.padStart contract" and "fill = multi-code-point string (e.g. emoji)". Both threads carried over from the snapshot's prior truncate code-point-vs-code-unit discussion. Snapshot informing the shape of what was looked for. The cost of maintaining the snapshot (~800 tokens, ~$0.005/run) is trivially affordable when it materially improves orchestrator triage on the 1-5 lenses dispatched per review. --- agents/claude.ts | 2 + agents/opencode.ts | 2 + agents/postRun.ts | 85 +++++++++++++++++++++++- agents/shared.ts | 29 ++++++++- external.ts | 2 + main.ts | 112 ++++++++++++++++++++++++++++++++ mcp/comment.ts | 40 ++---------- mcp/review.test.ts | 4 +- mcp/review.ts | 4 +- mcp/selectMode.ts | 100 +++++++++++----------------- mcp/server.ts | 17 ++++- modes.ts | 71 +++++++++----------- package.json | 2 +- utils/instructions.ts | 2 +- utils/patchWorkflowRunFields.ts | 4 +- utils/payload.ts | 2 + utils/prSummary.ts | 78 ++++++++++++++++++++++ 17 files changed, 405 insertions(+), 151 deletions(-) create mode 100644 utils/prSummary.ts diff --git a/agents/claude.ts b/agents/claude.ts index 53ddc45..225bca5 100644 --- a/agents/claude.ts +++ b/agents/claude.ts @@ -719,6 +719,8 @@ export const claude = agent({ initialResult: result, initialUsage: result.usage, stopScript: ctx.stopScript, + summaryFilePath: ctx.summaryFilePath, + summarySeed: ctx.summarySeed, reflectionPrompt: buildLearningsReflectionPrompt("claude"), canResume: (r) => Boolean(r.sessionId), resume: async (c) => { diff --git a/agents/opencode.ts b/agents/opencode.ts index 3ffed68..89739d3 100644 --- a/agents/opencode.ts +++ b/agents/opencode.ts @@ -962,6 +962,8 @@ export const opencode = agent({ initialResult: result, initialUsage: result.usage, stopScript: ctx.stopScript, + summaryFilePath: ctx.summaryFilePath, + summarySeed: ctx.summarySeed, reflectionPrompt: buildLearningsReflectionPrompt("opencode"), resume: async (c) => runOpenCode({ diff --git a/agents/postRun.ts b/agents/postRun.ts index 02f6fba..9b9e504 100644 --- a/agents/postRun.ts +++ b/agents/postRun.ts @@ -1,3 +1,4 @@ +import { readFile } from "node:fs/promises"; import { type AgentId, formatMcpToolRef } from "../external.ts"; import { LIFECYCLE_HOOK_TIMEOUT_MS } from "../lifecycle.ts"; import { log } from "../utils/cli.ts"; @@ -92,13 +93,46 @@ export function buildStopHookPrompt(failure: StopHookFailure): string { ].join("\n"); } +/** check whether the seeded summary file is byte-identical to its seed. + * a missing or unreadable file returns false (don't nudge — the agent + * may have legitimately deleted it, or the seed step failed; the read- + * back path in main.ts handles both cases by skipping persist). */ +async function isSummaryUnchanged(filePath: string, seed: string): Promise { + try { + const current = await readFile(filePath, "utf8"); + return current === seed; + } catch { + return false; + } +} + +export function buildSummaryStalePrompt(filePath: string): string { + return [ + `PR SUMMARY UNTOUCHED — the rolling PR summary file at \`${filePath}\` is byte-identical to its seed; this run did not edit it.`, + "", + "review the diff and update the file in place to reflect what changed in the PR. update intent, key changes, and any risks worth flagging — keep the existing section headings stable so incremental runs produce clean diffs.", + "", + "if the diff is genuinely too small or noisy to warrant rewriting (e.g. a one-line typo fix, a comment tweak, a formatting-only change), it's fine to leave the structure as-is — but at minimum confirm you considered it by appending one line to the appropriate section noting the run. silence is not an option; the snapshot is what the next review run reads as context.", + ].join("\n"); +} + /** - * check the two post-run gates: did the stop hook pass and is the working - * tree clean? returns everything that still needs fixing so the caller can + * check the post-run gates: did the stop hook pass, is the working tree + * clean, and (when applicable) did the agent touch the rolling PR summary + * snapshot? returns everything that still needs nudging so the caller can * render a single combined resume prompt. + * + * the summary-stale check is skipped when `summaryFilePath` / `summarySeed` + * are not provided; this is the common case (non-PR runs, runs where the + * dispatcher didn't request snapshot generation, runs where the seed step + * failed). loop callers also pass these as undefined after the agent has + * already been nudged once, to avoid burning the retry budget on a soft + * non-blocking gate. */ export async function collectPostRunIssues(params: { stopScript: string | null | undefined; + summaryFilePath?: string | undefined; + summarySeed?: string | undefined; }): Promise { const issues: PostRunIssues = {}; if (params.stopScript) { @@ -107,6 +141,10 @@ export async function collectPostRunIssues(params: { } const status = getGitStatus(); if (status) issues.dirtyTree = status; + if (params.summaryFilePath && params.summarySeed !== undefined) { + const stale = await isSummaryUnchanged(params.summaryFilePath, params.summarySeed); + if (stale) issues.summaryStale = { filePath: params.summaryFilePath }; + } return issues; } @@ -114,6 +152,7 @@ export function buildPostRunPrompt(issues: PostRunIssues): string { const parts: string[] = []; if (issues.stopHook) parts.push(buildStopHookPrompt(issues.stopHook)); if (issues.dirtyTree) parts.push(buildCommitPrompt(issues.dirtyTree)); + if (issues.summaryStale) parts.push(buildSummaryStalePrompt(issues.summaryStale.filePath)); return parts.join("\n\n---\n\n"); } @@ -164,6 +203,14 @@ export async function runPostRunRetryLoop(params: { initialResult: R; initialUsage: AgentUsage | undefined; stopScript: string | null | undefined; + /** absolute path to the seeded PR summary file. when set together with + * `summarySeed`, the loop checks after each agent attempt whether the + * file has been edited; if not, it nudges the agent ONCE via a resume + * turn (subsequent iterations skip the check so we don't keep burning + * retries on a soft gate when the agent has decided no edit is warranted). */ + summaryFilePath?: string | undefined; + /** exact bytes of the seeded summary file used for the unchanged-check. */ + summarySeed?: string | undefined; resume: (context: { prompt: string; previousResult: R }) => Promise; canResume?: ((result: R) => boolean) | undefined; reflectionPrompt?: string | undefined; @@ -173,10 +220,21 @@ export async function runPostRunRetryLoop(params: { let finalIssues: PostRunIssues = {}; let gateResumeCount = 0; let pendingReflection = params.reflectionPrompt; + // nudge for an untouched summary file fires AT MOST ONCE per run. after + // we've delivered the prompt, subsequent gate checks pass undefined so + // the loop doesn't keep flagging the same condition — the agent may have + // legitimately decided no edit is warranted, and re-prompting would + // burn the retry budget without adding signal. + let summaryStaleNudged = false; while (gateResumeCount < MAX_POST_RUN_RETRIES) { if (!result.success) break; - const issues = await collectPostRunIssues({ stopScript: params.stopScript }); + const issues = await collectPostRunIssues({ + stopScript: params.stopScript, + summaryFilePath: summaryStaleNudged ? undefined : params.summaryFilePath, + summarySeed: summaryStaleNudged ? undefined : params.summarySeed, + }); + if (issues.summaryStale) summaryStaleNudged = true; finalIssues = issues; if (!hasPostRunIssues(issues)) { @@ -230,8 +288,25 @@ export async function runPostRunRetryLoop(params: { log.info(`» post-run retry (attempt ${gateResumeCount + 1}/${MAX_POST_RUN_RETRIES})`); const prompt = buildPostRunPrompt(issues); + // summary-stale is a soft gate that must never flip a successful run to + // failed. when it's the only issue and the resume itself errors out, + // restore the pre-resume successful result and break — persistSummary + // detects the unchanged file via its seed comparison and skips the DB + // write on its own, so no further coordination is needed here. + const onlySummaryStale = + issues.summaryStale !== undefined && + issues.stopHook === undefined && + issues.dirtyTree === undefined; + const preResume = result; result = await params.resume({ prompt, previousResult: result }); aggregatedUsage = mergeAgentUsage(aggregatedUsage, result.usage); + if (!result.success && onlySummaryStale) { + log.warning( + `» summary-stale resume turn failed (${result.error ?? "unknown error"}), preserving prior successful result` + ); + result = preResume; + break; + } gateResumeCount++; } @@ -242,6 +317,10 @@ export async function runPostRunRetryLoop(params: { // already observed a clean state we skip: re-running the hook risks flaky // false-positive failures right after it just passed. if (gateResumeCount > 0 && result.success && hasPostRunIssues(finalIssues)) { + // re-check the gates that can actually fail the run (stop hook / + // dirty tree). summary-stale is intentionally NOT re-checked here: + // we already delivered the one-shot nudge, and a still-unchanged + // file at this point is the agent's deliberate choice. finalIssues = await collectPostRunIssues({ stopScript: params.stopScript }); } diff --git a/agents/shared.ts b/agents/shared.ts index 3fec569..a8169a0 100644 --- a/agents/shared.ts +++ b/agents/shared.ts @@ -42,13 +42,26 @@ export interface StopHookFailure { output: string; } +export interface SummaryStale { + /** absolute path to the seeded snapshot file the agent was meant to edit. */ + filePath: string; +} + export interface PostRunIssues { stopHook?: StopHookFailure; dirtyTree?: string; + /** populated when the rolling PR summary file is byte-identical to its + * seed, i.e. the agent never touched it. soft gate — nudges once via a + * resume turn but never fails the run, parallel to dirtyTree semantics. */ + summaryStale?: SummaryStale; } export function hasPostRunIssues(issues: PostRunIssues): boolean { - return issues.stopHook !== undefined || issues.dirtyTree !== undefined; + return ( + issues.stopHook !== undefined || + issues.dirtyTree !== undefined || + issues.summaryStale !== undefined + ); } /** @@ -106,6 +119,20 @@ export interface AgentRunContext { * guidance. null when the repo has no stop hook configured. */ stopScript?: string | null | undefined; + /** + * absolute path to the rolling PR summary tmpfile, when one was seeded + * for this run (Review / IncrementalReview / pr-summary Task). enables + * a post-run sanity nudge that prompts the agent if the file is still + * byte-identical to its seed. + */ + summaryFilePath?: string | undefined; + /** + * exact bytes of the seeded summary file. compared against the current + * file content after each agent attempt to detect "agent forgot to edit + * the summary" — particularly common with smaller models that lose + * track of multi-step instructions. + */ + summarySeed?: string | undefined; /** * called synchronously when the agent subprocess is killed for inner * activity timeout. lets main.ts tear down shared resources (MCP HTTP diff --git a/external.ts b/external.ts index 01b1d92..9a4d88a 100644 --- a/external.ts +++ b/external.ts @@ -281,6 +281,8 @@ export interface WriteablePayload { cwd?: string | undefined; /** pre-created progress comment (ID + type) for updating status */ progressComment?: { id: string; type: "issue" | "review" } | undefined; + /** when true, seed the PR summary tmpfile + persist edits at run end */ + generateSummary?: boolean | undefined; } // immutable payload type for agent execution diff --git a/main.ts b/main.ts index cbcb681..00760b6 100644 --- a/main.ts +++ b/main.ts @@ -1,6 +1,7 @@ // changes to tool permissions should be reflected in wiki/granular-tools.md import { existsSync, readdirSync } from "node:fs"; +import { readFile } from "node:fs/promises"; import { join } from "node:path"; import * as core from "@actions/core"; import { deleteProgressComment, reportProgress } from "./mcp/comment.ts"; @@ -34,6 +35,7 @@ import { executeLifecycleHook } from "./utils/lifecycle.ts"; import { normalizeEnv } from "./utils/normalizeEnv.ts"; import { aggregateUsage, patchWorkflowRunFields } from "./utils/patchWorkflowRunFields.ts"; import { resolvePayload, resolvePromptInput } from "./utils/payload.ts"; +import { readSummaryFile, seedSummaryFile } from "./utils/prSummary.ts"; import { postReviewCleanup } from "./utils/reviewCleanup.ts"; import { handleAgentResult } from "./utils/run.ts"; import { type AccountPlan, isInfraCovered } from "./utils/runContext.ts"; @@ -333,6 +335,67 @@ async function resolveProxyModel(ctx: { log.info(`» proxy: ${label} → ${ctx.proxyModel}`); } +/** + * Fetch the most recent persisted PR summary snapshot for this PR. + * Returns null on first-time PRs, when summary is disabled, or on any error. + * Best-effort: a transient API failure should not block the run. + */ +async function fetchPreviousSnapshot(ctx: ToolContext, prNumber: number): Promise { + if (!ctx.githubInstallationToken) return null; + try { + const response = await apiFetch({ + path: `/api/repo/${ctx.repo.owner}/${ctx.repo.name}/pr/${prNumber}/summary-comment`, + method: "GET", + headers: { authorization: `Bearer ${ctx.githubInstallationToken}` }, + signal: AbortSignal.timeout(10_000), + }); + if (!response.ok) return null; + const data = (await response.json()) as { snapshot?: string | null }; + return typeof data.snapshot === "string" && data.snapshot.length > 0 ? data.snapshot : null; + } catch { + return null; + } +} + +/** + * Read the agent-edited PR summary tmpfile and persist to `WorkflowRun.summarySnapshot`. + * + * Best-effort: any failure is logged and does not affect the run's success + * status. Skips the PATCH when the file is byte-identical to its seed — + * persisting the seed verbatim would either re-write what the DB already has + * (on incremental runs) or serialize the placeholder scaffold (on first + * runs), neither of which is useful. + */ +async function persistSummary(ctx: ToolContext): Promise { + const filePath = ctx.toolState.summaryFilePath; + if (!filePath) return; + // already-completed guard: the error-path call (success path persisted, + // then a late step threw) and the SIGINT/SIGTERM handler all funnel + // through here; the first one to arrive wins. + if (ctx.toolState.summaryPersistAttempted) return; + ctx.toolState.summaryPersistAttempted = true; + const snapshot = await readSummaryFile(filePath); + if (!snapshot) { + log.debug(`pr summary tmpfile missing or invalid at ${filePath} — skipping persist`); + return; + } + // soft gate: agent never touched the seeded file. saving the seed back + // is a no-op at best (incremental run — DB already has it) and a bug at + // worst (first run — serializes the placeholder italics). log a warning + // so the failure mode is visible in CI without flipping the run to + // failed. + const seed = ctx.toolState.summarySeed?.trim(); + if (seed !== undefined && snapshot === seed) { + log.warning( + "» pr summary tmpfile unchanged from seed — skipping persist (agent did not edit it)" + ); + return; + } + await patchWorkflowRunFields(ctx, { summarySnapshot: snapshot }).catch((err) => { + log.debug(`pr summary persist failed: ${err instanceof Error ? err.message : String(err)}`); + }); +} + async function writeJobSummary(toolState: ToolState): Promise { const usageSummary = formatUsageSummary(toolState.usageEntries); const summaryParts = [toolState.lastProgressBody, usageSummary].filter(Boolean); @@ -549,6 +612,40 @@ export async function main(): Promise { log.info(`» MCP server started at ${mcpHttpServer.url}`); timer.checkpoint("mcpServer"); + // seed the rolling PR summary tmpfile when the dispatcher requested it. + // gated on event being a PR — issue/workflow_dispatch runs have no + // summarySnapshot to maintain. file path is exposed to the agent via + // the select_mode response addendum (action/mcp/selectMode.ts). + if (payload.generateSummary && payload.event.is_pr && payload.event.issue_number) { + const previousSnapshot = await fetchPreviousSnapshot(toolContext, payload.event.issue_number); + const filePath = await seedSummaryFile({ tmpdir, previousSnapshot }); + toolState.summaryFilePath = filePath; + // capture the exact bytes the agent will see at startup. used by + // the post-run retry loop to detect the agent forgetting to edit + // the file (byte-identical to seed → nudge once via resume turn) + // and by persistSummary to skip the DB write when nothing changed. + // we just wrote the file, so the read shouldn't fail; the catch + // leaves summarySeed unset (its default), in which case the unchanged + // checks downstream are simply skipped. + try { + toolState.summarySeed = await readFile(filePath, "utf8"); + } catch { + // intentionally empty — summarySeed stays undefined + } + log.info( + `» summary snapshot seeded at ${filePath} (previous=${previousSnapshot ? "yes" : "no"})` + ); + // on SIGINT/SIGTERM we still want to persist whatever the agent has + // written so far. handler is best-effort: any failure inside is + // swallowed by Promise.allSettled in exitHandler.ts, and the + // summaryPersistAttempted guard prevents double-execution if the + // signal arrives after the normal path already persisted. capture a + // narrowed reference so the closure doesn't depend on the outer + // `toolContext` variable being defined later. + const ctxForExit = toolContext; + onExitSignal(() => persistSummary(ctxForExit)); + } + startInstallation(toolContext); const modelForLog = resolveModelForLog({ payload, resolvedModel }); @@ -661,6 +758,8 @@ export async function main(): Promise { instructions, todoTracker, stopScript: runContext.repoSettings.stopScript, + summaryFilePath: toolState.summaryFilePath, + summarySeed: toolState.summarySeed, onActivityTimeout: onInnerActivityTimeout, onToolUse: (event) => { const wasTracked = recordDiffReadFromToolUse({ @@ -742,6 +841,12 @@ export async function main(): Promise { }); } + // read the agent-edited summary tmpfile and persist to the DB. happens + // after the agent exits so the file is in its final state. + if (toolContext) { + await persistSummary(toolContext); + } + // clean up stranded progress comments. the comment is stale unless // report_progress wrote a final summary to it — three sub-cases all reduce // to !finalSummaryWritten: @@ -803,6 +908,13 @@ export async function main(): Promise { }); } + // best-effort summary persist on the error path: if the agent successfully + // edited the summary file before timing out / crashing, those edits are + // worth keeping for the next incremental run. + if (toolContext) { + await persistSummary(toolContext); + } + return { success: false, error: errorMessage, diff --git a/mcp/comment.ts b/mcp/comment.ts index 8731160..e867330 100644 --- a/mcp/comment.ts +++ b/mcp/comment.ts @@ -56,10 +56,8 @@ export const Comment = type({ issueNumber: type.number.describe("the issue number to comment on"), body: type.string.describe("the comment body content"), type: type - .enumerated("Plan", "Summary", "Comment") - .describe( - "Plan: record as the plan for this run. Summary: record as the PR summary comment (one per PR, updated in place). Comment: regular comment (default)." - ) + .enumerated("Plan", "Comment") + .describe("Plan: record as the plan for this run. Comment: regular comment (default).") .optional(), }); @@ -67,37 +65,11 @@ export function CreateCommentTool(ctx: ToolContext) { return tool({ name: "create_issue_comment", description: - "Create a comment on a GitHub issue or PR. For progress/plan updates on the current run use report_progress instead. Use type: 'Plan' for plan comments, type: 'Summary' for PR summary comments.", + "Create a comment on a GitHub issue or PR. For progress/plan updates on the current run use report_progress instead. Use type: 'Plan' for plan comments.", parameters: Comment, execute: execute(async ({ issueNumber, body, type: commentType }) => { const bodyWithFooter = addFooter(ctx, body); - // if a summary comment already exists (found by select_mode), update instead of creating - if (commentType === "Summary" && ctx.toolState.existingSummaryCommentId) { - log.info( - `» redirecting create_issue_comment(Summary) to update existing comment ${ctx.toolState.existingSummaryCommentId}` - ); - const result = await ctx.octokit.rest.issues.updateComment({ - owner: ctx.repo.owner, - repo: ctx.repo.name, - comment_id: ctx.toolState.existingSummaryCommentId, - body: bodyWithFooter, - }); - - ctx.toolState.wasUpdated = true; - - if (result.data.node_id) { - await patchWorkflowRunFields(ctx, { summaryCommentNodeId: result.data.node_id }); - } - - return { - success: true, - commentId: result.data.id, - url: result.data.html_url, - body: result.data.body, - }; - } - const result = await ctx.octokit.rest.issues.createComment({ owner: ctx.repo.owner, repo: ctx.repo.name, @@ -131,10 +103,6 @@ export function CreateCommentTool(ctx: ToolContext) { }; } - if (commentType === "Summary" && result.data.node_id) { - await patchWorkflowRunFields(ctx, { summaryCommentNodeId: result.data.node_id }); - } - return { success: true, commentId: result.data.id, @@ -209,7 +177,7 @@ export async function reportProgress( // always track the body for job summary ctx.toolState.lastProgressBody = body; - // silent events (e.g., auto-label, PR summary) should never create or update progress comments. + // silent events (e.g., auto-label, pr-summary Task) should never create or update progress comments. // the body is still tracked above for the GitHub Actions job summary. if (ctx.payload.event.silent) { return { body, action: "skipped" }; diff --git a/mcp/review.test.ts b/mcp/review.test.ts index a83010b..0077ebd 100644 --- a/mcp/review.test.ts +++ b/mcp/review.test.ts @@ -648,8 +648,8 @@ describe("reviewSkipDecision", () => { describe("duplicateReviewDecision", () => { // regression: colinhacks/zod#5897 had two reviews submitted from the same // workflow run 8 seconds apart — a substantive review followed by an empty - // "Reviewed — no issues found." follow-up. the agent re-classified the - // first review's non-blocking observations as "no actionable issues" and + // "No new issues found." follow-up. the agent re-classified the first + // review's non-blocking observations as "no actionable issues" and // submitted the canonical body per modes.ts. this guard makes the second // call a no-op without burning a GitHub API call or polluting the PR. diff --git a/mcp/review.ts b/mcp/review.ts index 3b10cb8..b0c48a3 100644 --- a/mcp/review.ts +++ b/mcp/review.ts @@ -187,8 +187,8 @@ export type DuplicateReviewDecision = { * the agent is instructed to call create_pull_request_review exactly once per * Review-mode session (see action/modes.ts), but in practice it sometimes * submits twice — once with substantive feedback, then again with the - * canonical "Reviewed — no issues found." body when the prompt's branch - * logic re-classifies non-blocking observations. the second submission is + * canonical "No new issues found." body when the prompt's branch logic + * re-classifies non-blocking observations. the second submission is * always redundant: the first review is the record, and the duplicate just * adds noise to the PR. * diff --git a/mcp/selectMode.ts b/mcp/selectMode.ts index d94e6ed..06daf4b 100644 --- a/mcp/selectMode.ts +++ b/mcp/selectMode.ts @@ -1,14 +1,13 @@ import { type } from "arktype"; import { formatMcpToolRef } from "../external.ts"; -import { type Mode, PR_SUMMARY_FORMAT } from "../modes.ts"; +import type { Mode } from "../modes.ts"; import { apiFetch } from "../utils/apiFetch.ts"; -import { log } from "../utils/log.ts"; import type { ToolContext } from "./server.ts"; import { execute, tool } from "./shared.ts"; export const SelectModeParams = type({ mode: type.string.describe( - "the name of the mode to select (e.g., 'Build', 'Plan', 'Review', 'IncrementalReview', 'Fix', 'AddressReviews', 'Task', 'ResolveConflicts', 'Summarize')" + "the name of the mode to select (e.g., 'Build', 'Plan', 'Review', 'IncrementalReview', 'Fix', 'AddressReviews', 'Task', 'ResolveConflicts')" ), "issue_number?": type("number").describe( "optional issue number; when provided with Plan mode, used to look up an existing plan comment for this issue (edit vs create)" @@ -32,18 +31,6 @@ An existing plan comment was found for this issue. Update that comment with the - produce a structured plan with clear milestones 3. Call \`${t("report_progress")}\` with the full revised plan text and \`{ target_plan_comment: true }\` so it updates the existing plan comment (not the progress comment). 4. Then post a short note to the progress comment (e.g. "Plan has been updated in the comment above.") via \`${t("report_progress")}\` so it is not left as "Leaping...".`, - - SummaryUpdate: `### Checklist (updating existing summary) - -An existing summary comment was found for this PR. Update it rather than creating a new one. - -1. Use \`previousSummaryBody\` from this response as the current summary to revise. -2. Checkout the PR via \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. -3. Read the diff using the TOC to selectively read relevant sections. Produce an updated summary reflecting the current state of the PR, using the existing summary (\`previousSummaryBody\`) as a starting point. If EVENT INSTRUCTIONS specify a custom format, follow that instead of the default format below. -4. Call \`${t("edit_issue_comment")}\` with \`commentId: existingSummaryCommentId\` (from this response) and the updated summary body. -5. Call \`${t("report_progress")}\` with a brief note (e.g., "Updated PR summary."). - -${PR_SUMMARY_FORMAT}`, }; } @@ -78,10 +65,7 @@ function buildOrchestratorGuidance( // matches the API response for /repo/[owner]/[repo]/issue/[issueNumber]/plan-comment export type PlanCommentResponsePayload = { error: string } | { commentId: number; body: string }; -// matches the API response for /repo/[owner]/[repo]/pr/[prNumber]/summary-comment -export type SummaryCommentResponsePayload = { error: string } | { commentId: number; body: string }; - -// IMPORTANT: these routes authenticate via GitHub installation token (getEnrichedRepo), +// IMPORTANT: this route authenticates via GitHub installation token (getEnrichedRepo), // NOT the Pullfrog API JWT (ctx.apiToken). use ctx.githubInstallationToken here. // see wiki/api-auth.md for the two auth patterns. async function fetchExistingPlanComment( @@ -103,33 +87,30 @@ async function fetchExistingPlanComment( } } -async function fetchExistingSummaryComment( - ctx: ToolContext, - prNumber: number -): Promise | null> { - if (!ctx.githubInstallationToken) { - log.warning("fetchExistingSummaryComment: no token, skipping"); - return null; - } - const path = `/api/repo/${ctx.repo.owner}/${ctx.repo.name}/pr/${prNumber}/summary-comment`; - try { - const response = await apiFetch({ - path, - method: "GET", - headers: { authorization: `Bearer ${ctx.githubInstallationToken}` }, - signal: AbortSignal.timeout(10_000), - }); - const data = (await response.json()) as SummaryCommentResponsePayload; - if (response.ok && "commentId" in data) { - return data; - } - const errMsg = "error" in data ? data.error : "(no error body)"; - log.warning(`fetchExistingSummaryComment: ${response.status} ${path} — ${errMsg}`); - return null; - } catch (error) { - log.warning("fetchExistingSummaryComment failed:", error); - return null; - } +const SUMMARY_MODES = new Set(["Review", "IncrementalReview", "Task"]); + +/** modes that gain the PR summary edit step when toolState.summaryFilePath is set. + * + * NOTE: this snapshot is an internal artifact consumed by future agent runs. it is + * deliberately NOT shaped by user-supplied summary instructions — those would warp + * the durable agent context. user-facing summarization (e.g. the review body's + * "Reviewed changes" section) is governed by review-mode prompts and review + * instructions, separately from this snapshot. */ +function buildSummaryAddendum(t: (name: string) => string, ctx: ToolContext): string { + const filePath = ctx.toolState.summaryFilePath; + if (!filePath) return ""; + return `### PR summary snapshot — required step + +A rolling PR summary lives at \`${filePath}\`. It is your durable cross-run agent context — a functional summary of what this PR does, the subsystems and files it touches, the material behavior of its changes, and any risks or open questions worth carrying forward. It is NOT a chronological log of past review runs; commit-level history can already be reconstructed from \`${t("list_pull_request_reviews")}\`. + +How to use it: + +- read \`${filePath}\` at the START of the run, alongside the diff. it represents what previous agent runs already understood about this PR — absorb it before picking lenses or crafting subagent dispatch prompts. if it's a fresh seed (file is one or two lines), this is a first review and you'll be filling it in from the diff. +- let the snapshot inform triage and dispatch. when it already tracks a risk, your lens prompts to subagents are stronger when they reference that context (e.g. "the JSDoc explicitly scopes to code points — do not flag grapheme-cluster issues" if the snapshot already documents that contract). when something the snapshot tracks is now resolved by new commits, note that. when new commits introduce something the snapshot doesn't yet describe, that's exactly where your fan-out should focus. +- update the file in place to reflect the PR's CURRENT state. revise stale claims, drop resolved risks, add new behavior or risks. accuracy over breadth — every claim must be grounded in the diff. write for the next agent run, not for a human. +- structure however serves THIS PR. there is no required section template. a refactor might organize by renamed export and call-site impact; a feature by capability; a billing change by money path. a compact note of which commit ranges have been reviewed should always be present so future runs scope correctly, but the rest is your call. when the structure works across runs, keep it stable so range-diffs are clean; when the PR's character changes (e.g. scope expands), reshape. + +Do NOT call \`${t("create_issue_comment")}\` for the summary — the server reads this file at end-of-run and persists it. The file edit is mandatory regardless of whether a review is submitted; the snapshot feeds the next run.`; } export function SelectModeTool(ctx: ToolContext) { @@ -180,22 +161,19 @@ export function SelectModeTool(ctx: ToolContext) { } } - if (selectedMode.name === "Summarize") { - const prNumber = ctx.payload.event.issue_number; - if (prNumber !== undefined) { - const existing = await fetchExistingSummaryComment(ctx, prNumber); - if (existing !== null) { - ctx.toolState.existingSummaryCommentId = existing.commentId; - return { - ...buildOrchestratorGuidance(ctx, selectedMode, overrides.SummaryUpdate), - existingSummaryCommentId: existing.commentId, - previousSummaryBody: existing.body, - }; - } - } - } + const summaryAddendum = SUMMARY_MODES.has(selectedMode.name) + ? buildSummaryAddendum(t, ctx) + : ""; - return buildOrchestratorGuidance(ctx, selectedMode); + const base = buildOrchestratorGuidance(ctx, selectedMode); + if (summaryAddendum.length > 0) { + return { + ...base, + orchestratorGuidance: `${base.orchestratorGuidance}\n\n${summaryAddendum}`, + summaryFilePath: ctx.toolState.summaryFilePath, + }; + } + return base; }), }); } diff --git a/mcp/server.ts b/mcp/server.ts index df3fd38..0d45d9f 100644 --- a/mcp/server.ts +++ b/mcp/server.ts @@ -128,8 +128,21 @@ export interface ToolState { // set by select_mode when Plan + issue_number and plan-comment API returns existing plan (for report_progress target_plan_comment) existingPlanCommentId?: number; previousPlanBody?: string; - // set by select_mode when Summarize mode and summary-comment API returns existing summary - existingSummaryCommentId?: number; + // absolute path to the PR summary markdown file the agent edits in place. + // seeded by main.ts before the agent starts when payload.generateSummary is set; + // read back at end-of-run to persist to DB. + summaryFilePath?: string; + // exact bytes of the seeded snapshot file at run start. compared against + // the file content at end-of-run to detect "agent never touched it" — in + // that case persistSummary skips the DB write (saving the seed verbatim + // would either re-write what the DB already has, on incremental runs, or + // serialize the placeholder scaffold, on first runs). + summarySeed?: string; + // set to true after persistSummary completes once. prevents the error-path + // call (which exists so a successful agent edit before a crash still gets + // persisted) from redundantly re-running the DB PATCH on the + // success-then-late-throw path. + summaryPersistAttempted?: boolean; output?: string; usageEntries: AgentUsage[]; model?: string | undefined; diff --git a/modes.ts b/modes.ts index 18b7f78..810cbf0 100644 --- a/modes.ts +++ b/modes.ts @@ -10,6 +10,12 @@ export interface Mode { prompt?: string | undefined; } +// Default user-facing summary format embedded in Review mode review bodies. +// Deliberately scoped to Review (initial PR review). IncrementalReview keeps +// its own terser bullet-list "Reviewed changes" shape since re-review bodies +// are deltas, not introductions. Distinct from the agent-internal snapshot +// (action/utils/prSummary.ts) which has its own stable scaffold and is never +// shaped by user instructions — see selectMode.ts for the firewall. export const PR_SUMMARY_FORMAT = `### Default format Follow this structure exactly: @@ -175,7 +181,7 @@ ${learningsStep(t, 6)}`, 2. **triage**: orient yourself on the PR — identify *what kind of thing this is* (domain it touches, seams it crosses, external contracts it depends on, user-facing surfaces it changes). orientation only — defer specific defect-hunting to the subagents; pre-reviewing biases the lenses you pick. use \`${t("get_pull_request")}\` and other read-only GitHub tools for additional context if needed. - if the PR is **genuinely trivial**, skip steps 3–4 entirely and submit \`Reviewed — no issues found.\` per step 5. there's no value in dispatching even one lens for a typo. + if the PR is **genuinely trivial**, skip steps 3–4 entirely and submit a \`No new issues found.\` review per step 5. there's no value in dispatching even one lens for a typo. "Genuinely trivial" (skip): - single-word doc typo, whitespace/format-only, comment-only across any number of files @@ -243,26 +249,28 @@ ${learningsStep(t, 6)}`, note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session. + The review body is structured as: \`[optional alert blockquote]\` → \`[PR summary using the default format below]\`. Inline comments are passed via the \`comments\` parameter, not in the body. + - **critical issues** (blocks merge — bugs, security, data loss): - \`approved: false\`. Body begins with a GitHub alert blockquote, e.g.: - \`> [!CAUTION]\\n> This PR introduces a race condition in ...\` - Follow with a brief summary if needed. Include all inline comments. + \`approved: false\`. Body opens with \`> [!CAUTION]\\n> This PR introduces ...\`, followed by the PR summary. Include all inline comments via \`comments\`. - **recommended changes** (non-critical): - \`approved: false\`. Body begins with a GitHub alert blockquote, e.g.: - \`> [!IMPORTANT]\\n> Consider adding input validation for ...\` - Follow with a brief summary if needed. Include all inline comments. + \`approved: false\`. Body opens with \`> [!IMPORTANT]\\n> Consider ...\`, followed by the PR summary. Include all inline comments via \`comments\`. - **no actionable issues**: - \`approved: true\`, body: "Reviewed — no issues found."`, + \`approved: true\`. Body opens with \`No new issues found.\` followed by the PR summary. + +${PR_SUMMARY_FORMAT}`, }, // IncrementalReview shares Review's multi-lens orchestrator pattern but - // scopes the target to the incremental diff and adds prior-review-feedback - // tracking. The "issues must be NEW since the last Pullfrog review" filter - // lives at aggregation time (step 5), NOT in the subagent prompt — pushing - // the filter into subagents matches the canonical anneal anti-pattern of - // "list known pre-existing failures — don't flag these" and suppresses - // signal on regressions the new commits amplified. The body-format rules - // (Reviewed changes / Prior review feedback) are unchanged from the prior - // version. Same severity-table omission as Review. + // scopes the target to the incremental diff. The "issues must be NEW + // since the last Pullfrog review" filter lives at aggregation time + // (step 5), NOT in the subagent prompt — pushing the filter into + // subagents matches the canonical anneal anti-pattern of "list known + // pre-existing failures — don't flag these" and suppresses signal on + // regressions the new commits amplified. The review body is just + // "Reviewed changes" — a separate "Prior review feedback" checklist + // would duplicate the rolling PR summary snapshot's record of what + // earlier runs already addressed and add noise to the user-facing + // body. Same severity-table omission as Review. { name: "IncrementalReview", description: @@ -273,7 +281,7 @@ ${learningsStep(t, 6)}`, 2. **incremental scope**: if \`incrementalDiffPath\` is present, read it to see what changed since the last review. this is a range-diff that isolates the net changes, filtering out base branch noise. if not present, fall back to reviewing the full PR diff and determine what changed since Pullfrog's most recent review. -3. **prior feedback**: fetch previous reviews via \`${t("list_pull_request_reviews")}\`. for the most recent Pullfrog review, call \`${t("get_review_comments")}\` with the review ID to retrieve specific prior line-level feedback. you'll need this in step 6 to track which prior comments were addressed. +3. **prior feedback**: fetch previous reviews via \`${t("list_pull_request_reviews")}\`. for the most recent Pullfrog review, call \`${t("get_review_comments")}\` with the review ID to retrieve specific prior line-level feedback. you'll use this to filter your aggregation in step 5 — anything already flagged in a prior review and not changed by the new commits should not be re-raised. you do NOT need to render this in the review body; the rolling PR summary snapshot is the durable record of what's been addressed. 4. **triage & fan out**: orient on the *incremental* changes — domain, seams, external contracts, user-facing surfaces. @@ -302,20 +310,14 @@ ${learningsStep(t, 6)}`, 5. **aggregate, draft, self-critique**: merge findings; de-dup overlaps; trace each finding yourself. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the new commits, anything not actionable, and anything that re-states prior review feedback (heuristic: if the finding's root cause lives in lines the *new commits* added or modified, it's in scope; otherwise drop). also drop **bloat-shaped findings** — proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or degrades elegance to nominally improve correctness) makes the codebase worse, not better. To compute "lines the new commits added or modified": if \`incrementalDiffPath\` from step 1 is present, use it directly. Otherwise, take the prior Pullfrog review's \`commit_id\` (returned alongside each entry from \`${t("list_pull_request_reviews")}\` in step 3) and run \`git diff ..HEAD\` to isolate the lines added since that review. draft inline comments with NEW line numbers from the full PR diff — every comment must be actionable, 2-3 sentences max. - then check: which prior review comments were addressed by the new commits? track the addressed ones for step 6b. +6. **build the review body** — a single "Reviewed changes" section: summarize at the logical-change level, not per-file. each bullet starts with a past-tense verb (e.g. \`- Extracted shared CLI runtime into a single module\`, \`- Renamed package to pullfrog\`). avoid file paths unless they add clarity. if the changes can be described in one sentence, use one sentence — no bullets needed. do NOT include a separate "Prior review feedback" checklist; that's tracked in the rolling PR summary snapshot for the next agent run, and surfacing it in the user-facing body is noise (changes that addressed prior feedback are already covered by the Reviewed-changes bullets). in some cases you may receive a complete diff for the whole pull request instead of an incremental one — when this happens, you will need to determine what changes have happened since Pullfrog's most recent review. -6. **build the review body** — two distinct sections: - a. **Reviewed changes**: summarize at the logical-change level, not per-file. each bullet starts with a past-tense verb (e.g. \`- Extracted shared CLI runtime into a single module\`, \`- Renamed package to pullfrog\`). avoid file paths unless they add clarity. if the changes can be described in one sentence, use one sentence — no bullets needed. - b. **Prior review feedback** (only if any were addressed): list only the prior review comments that WERE addressed by the new commits (\`- [x] safeParse instead of parse — addressed\`). omit unaddressed comments. omit this entire section if nothing was addressed. a change can appear in both sections. - - no headings, no tables, no prose paragraphs in either section — just bullets - - in some cases you may receive a complete diff for the whole pull request instead of an incremental one. when this happens, you will need to determine what changes have happened since Pullfrog's most recent review. - -7. Submit — Do NOT call \`report_progress\` or \`create_issue_comment\` — the review is the final record and the progress comment will be cleaned up automatically. the review body always includes the reviewed changes from step 6a. append \`Prior review feedback:\\n\` with the checklist from step 6b only if any prior comments were addressed. Follow these rules: +7. Submit — Do NOT call \`report_progress\` or \`create_issue_comment\` — the review is the final record and the progress comment will be cleaned up automatically. Follow these rules: - note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session. - IF NO NEW ISSUES, NON-SUBSTANTIVE CHANGES ONLY (trivial formatting, import reordering, comment tweaks): do NOT submit a review. Do NOT call \`report_progress\`. Exit — the progress comment will be cleaned up automatically. - - ELSE IF NEW CRITICAL ISSUES (blocks merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with a GitHub alert blockquote (e.g. \`> [!CAUTION]\\n> This PR introduces ...\`), then the reviewed changes summary and prior feedback (if any). - - ELSE IF NEW RECOMMENDED CHANGES (non-critical): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\` alert, then the reviewed changes summary and prior feedback (if any). - - ELSE IF NO NEW ISSUES, SUBSTANTIVE CHANGES (new functionality, behavior changes, or fixes to prior review feedback): call \`${t("create_pull_request_review")}\` to create a PR review. If all previous reviews have been properly addressed and no new issues were discovered, you can set \`approved: true\`. body opens with \`No new issues. Reviewed the following changes:\\n\`, then the reviewed changes summary and prior feedback (if any).`, + - ELSE IF NEW CRITICAL ISSUES (blocks merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with a GitHub alert blockquote (e.g. \`> [!CAUTION]\\n> This PR introduces ...\`), then the Reviewed-changes summary. + - ELSE IF NEW RECOMMENDED CHANGES (non-critical): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\` alert, then the Reviewed-changes summary. + - ELSE IF NO NEW ISSUES, SUBSTANTIVE CHANGES (new functionality, behavior changes, or fixes to prior review feedback): call \`${t("create_pull_request_review")}\` to create a PR review. If all previous reviews have been properly addressed and no new issues were discovered, you can set \`approved: true\`. body opens with \`No new issues. Reviewed the following changes:\\n\`, then the Reviewed-changes summary.`, }, { name: "Plan", @@ -405,19 +407,6 @@ ${learningsStep(t, 6)}`, ${learningsStep(t, 4)}`, }, - { - name: "Summarize", - description: - "Summarize a PR with a structured comment that is updated in place on subsequent pushes", - prompt: `### Checklist - -1. Checkout the PR via \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. -2. Read the diff using the TOC to selectively read relevant sections (not the entire file). Produce a structured summary. If EVENT INSTRUCTIONS specify a custom format, follow that instead of the default format below. -3. Call \`${t("create_issue_comment")}\` with \`type: "Summary"\` and the summary body. -4. Call \`${t("report_progress")}\` with a brief note (e.g., "Posted PR summary."). - -${PR_SUMMARY_FORMAT}`, - }, ]; } diff --git a/package.json b/package.json index 5112f8d..4c88784 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "pullfrog", - "version": "0.0.204", + "version": "0.0.205", "type": "module", "bin": { "pullfrog": "dist/cli.mjs", diff --git a/utils/instructions.ts b/utils/instructions.ts index e5bd134..9aa89ec 100644 --- a/utils/instructions.ts +++ b/utils/instructions.ts @@ -289,7 +289,7 @@ When embedding images (e.g. uploaded screenshots) in comments or PR bodies, alwa **\`report_progress\`**: call this exactly once at the end of every run with a brief final summary (1-3 sentences) unless the mode guidance instructs otherwise. Never call it for intermediate status updates (e.g., "Checking for changes...", "Starting review...") — the task list handles live progress automatically. Calling \`report_progress\` replaces the task list with your summary and preserves the current task list in a collapsible section. Keep the summary concise — do not repeat what the task list already shows. Focus on the outcome (what was accomplished, links to artifacts) rather than listing individual steps. If something failed, include the tool's error text even when that makes the summary longer. -Never use \`create_issue_comment\` for task progress — that creates duplicate comments and leaves the progress comment stuck in its initial state. \`create_issue_comment\` is only for standalone comments unrelated to your current task (e.g., Plan comments, PR Summary comments). +Never use \`create_issue_comment\` for task progress — that creates duplicate comments and leaves the progress comment stuck in its initial state. \`create_issue_comment\` is only for standalone comments unrelated to your current task (e.g., Plan comments). ### If you get stuck diff --git a/utils/patchWorkflowRunFields.ts b/utils/patchWorkflowRunFields.ts index df0c3e1..57c29ae 100644 --- a/utils/patchWorkflowRunFields.ts +++ b/utils/patchWorkflowRunFields.ts @@ -14,7 +14,8 @@ export type WorkflowRunArtifactPatchKey = | "issueNodeId" | "reviewNodeId" | "planCommentNodeId" - | "summaryCommentNodeId"; + | "summaryCommentNodeId" + | "summarySnapshot"; /** * Usage fields — aggregated across all agent calls and PATCHed once at @@ -38,6 +39,7 @@ const STRING_KEYS: WorkflowRunArtifactPatchKey[] = [ "reviewNodeId", "planCommentNodeId", "summaryCommentNodeId", + "summarySnapshot", ]; const NUMBER_KEYS: WorkflowRunUsagePatchKey[] = [ diff --git a/utils/payload.ts b/utils/payload.ts index 37c7f2b..fd526ae 100644 --- a/utils/payload.ts +++ b/utils/payload.ts @@ -27,6 +27,7 @@ export const JsonPayload = type({ id: "string", type: "'issue' | 'review'", }).or("undefined"), + "generateSummary?": "boolean | undefined", }); // permission levels that indicate collaborator status (have push access) @@ -160,6 +161,7 @@ export function resolvePayload( timeout: inputs.timeout ?? jsonPayload?.timeout, cwd: resolveCwd(inputs.cwd), progressComment: jsonPayload?.progressComment, + generateSummary: jsonPayload?.generateSummary, // permissions: inputs > repoSettings > fallbacks push: inputs.push ?? repoSettings.push ?? "restricted", diff --git a/utils/prSummary.ts b/utils/prSummary.ts new file mode 100644 index 0000000..dbbfbec --- /dev/null +++ b/utils/prSummary.ts @@ -0,0 +1,78 @@ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; + +/** + * The PR-level summary snapshot is a markdown file the agent edits in place + * during a Review / IncrementalReview run. The server seeds the file with + * either the previous run's snapshot (incremental) or a stub scaffold (first + * run), lets the agent edit it with its native file-editing tools, then + * reads it back at end-of-run and persists it to `WorkflowRun.summarySnapshot`. + * + * The snapshot is an internal artifact — it is consumed by future agent runs + * as durable cross-run context, not surfaced to humans. User-visible summary + * content lives in the Review / IncrementalReview review bodies, governed by + * `action/modes.ts`. + * + * Edit-in-place avoids the output-token tax of a tool call that regurgitates + * the full snapshot, and gives incremental runs a clean surface that + * range-diffs cleanly across runs because the section headings are stable. + */ + +export const SUMMARY_FILE_NAME = "pullfrog-summary.md"; + +/** + * minimal seed for first-run PRs. just a header + a one-line note about + * what this file is for. structure is intentionally NOT prescribed — + * different PRs warrant different organization, and the agent should pick + * a shape that fits this PR. the agent's prompt (see selectMode.ts + * `buildSummaryAddendum`) carries the actual instructions for what to + * capture and how. + * + * keeping the seed short also makes the unchanged-from-seed gate more + * sensitive — any meaningful edit moves the file off the seed, so + * `persistSummary` can reliably skip the DB write when the agent didn't + * touch the file. + */ +export const SUMMARY_SCAFFOLD = `# PR summary + + +`; + +const MIN_SNAPSHOT_LENGTH = 60; +/** PG TEXT can hold ~1GB but a sane cap protects the DB / API payloads. */ +const MAX_SNAPSHOT_LENGTH = 32_768; + +export function summaryFilePath(tmpdir: string): string { + return join(tmpdir, SUMMARY_FILE_NAME); +} + +/** seed the summary file with previous snapshot (incremental) or scaffold (first run). */ +export async function seedSummaryFile(params: { + tmpdir: string; + previousSnapshot: string | null; +}): Promise { + const path = summaryFilePath(params.tmpdir); + await mkdir(dirname(path), { recursive: true }); + const seed = + params.previousSnapshot && params.previousSnapshot.trim().length >= MIN_SNAPSHOT_LENGTH + ? params.previousSnapshot + : SUMMARY_SCAFFOLD; + await writeFile(path, seed, "utf8"); + return path; +} + +/** read + validate the summary file written by the agent. + * returns null when the file is missing or fails sanity checks. */ +export async function readSummaryFile(path: string): Promise { + let raw: string; + try { + raw = await readFile(path, "utf8"); + } catch { + return null; + } + const trimmed = raw.trim(); + if (trimmed.length < MIN_SNAPSHOT_LENGTH) return null; + if (trimmed.length > MAX_SNAPSHOT_LENGTH) return trimmed.slice(0, MAX_SNAPSHOT_LENGTH); + return trimmed; +}