Files
shockbot/mcp/review.test.ts
T
Colin McDonnell e4e93ea6d3 PR summary as agent-edited tmpfile snapshot (#568)
* PR summary as agent-edited tmpfile snapshot

Replaces the comment-based PR summary path (and the in-progress
update_pr_summary tool from #534) with a snapshot file the agent edits in
place during Review / IncrementalReview / pr-summary Task runs.

The server seeds the tmpfile with the previous snapshot (incremental) or a
stable scaffold (first run), exposes the path via select_mode, and reads it
back at end-of-run to persist to WorkflowRun.summarySnapshot and (when the
prSummaryComment toggle is on) splice into the PR description body.

Why a tmpfile rather than a tool call: incremental snapshot edits are
output-token-cheap when the agent uses native file-editing tools, and
range-diff cleanly across runs because section headings are stable. The
agent never has to regurgitate the full snapshot to update it.

Gating: snapshot generation is opt-in via either prSummaryComment="enabled"
(splice into PR body) or prReReview="enabled" (snapshot feeds future
incremental review runs as context). Users who disable both pay nothing
end-to-end — no seeding, DB write, or body splice.

Behavior changes:
- Drop the Summarize mode and the Summary comment type entirely; the
  rolling summary is no longer a separate run shape.
- pull_request_synchronize with re-review off and summary on still
  dispatches a silent pr-summary Task, but it edits the snapshot file
  instead of posting a fresh comment.
- /api/repo/.../pr/.../summary-comment now returns
  { snapshot: string | null } from the DB instead of fetching a comment via
  GraphQL. URL kept stable so deployed older actions degrade gracefully.
- summaryCommentNodeId is retained on WorkflowRun for legacy data and a
  future backfill of pre-snapshot comment-based summaries.

Supersedes #534. The commit-tool/sub-agent direction in that PR is
abandoned in favor of this file-based shape.

* address review pass #1: synchronize fallback, splice idempotency, docs

* address review pass #2: in-flight skip should not race summary fallback

* address review pass #3: signal-handler flush, doc clarifications

* address review pass #4: in-flight persist promise + bounded body-splice timeout

* address review pass #5: defensive catch on persist worker, doc nit

* add summary-stale post-run gate

When generateSummary is set, we capture the bytes of the seeded snapshot
file and pass them to the agent's post-run loop alongside the file path.
After each agent attempt, the loop diffs the current file against the
seed; if they're byte-identical the agent never touched it, and we nudge
once via a resume turn (similar to the dirty-tree gate, but soft and
fire-once so smaller models that legitimately decide no edit is warranted
don't burn the retry budget).

Mostly defends against forgetful smaller models on the Review path —
their mode prompt asks them to edit the snapshot file, but the
multi-step instruction can fall through when the diff is large.

* trigger: retry vercel preview build

* fix(action): drop unused re-export that pulled node:fs/promises into next bundle

action/internal/index.ts was re-exporting DEFAULT_PR_SUMMARY_INSTRUCTIONS
from action/utils/prSummary.ts, but nothing in the next.js app imports
it. prSummary.ts uses node:fs/promises, and pullfrog/internal is aliased
into the next bundle by next.config.ts, which made turbopack try to
resolve node:fs/promises in client chunks and fail with:

  the chunking context (unknown) does not support external modules
  (request: node:fs/promises)

drop the re-export — selectMode.ts (the only real consumer) already
imports it directly from action/utils/prSummary.ts.

* firewall PR summary snapshot from user instructions; resurrect rich format for Review

The agent-internal snapshot (the markdown file the agent edits in place across
runs) is exclusively durable context for future agent runs — user-supplied
summarization instructions warp it and degrade that context. Drop the
prSummaryCommentInstructions read path end-to-end:

- handleWebhook: stop reading prSummaryCommentInstructions, stop passing
  prSummaryInstructions through dispatch options
- action payload + ToolState + selectMode addendum: drop the instructions
  appendix; the snapshot prompt is fixed, not user-shaped
- TriggersSettings: drop the InstructionsEditor for prSummaryCommentInstructions
- prSummary.ts: reframe DEFAULT_PR_SUMMARY_INSTRUCTIONS as agent-targeted
  (durable context, not human-facing prose)

Prisma columns (prSummaryComment, prSummaryCommentInstructions) and the
matching zod schema entry stay for graceful retreat.

Separately, resurrect PR_SUMMARY_FORMAT (deleted along with the Summarize mode
in the original PR) and wire it into Review mode only. Initial PR reviews now
include a structured summary section in the review body using the rich format
(TL;DR, key changes, ## sections with before/after, file-link trails).
IncrementalReview keeps its existing terser bullet-list shape since re-review
bodies are deltas, not introductions. The user-facing review summary and the
agent-internal snapshot are deliberately separate artifacts with separate
prompts and zero shared content.

* address review comments: prompt self-consistency + stale-doc cleanup

PR 568 self-review (4232488109) flagged a self-contradiction the firewall
commit introduced and three stale doc references that survived.

- action/modes.ts: Review-mode step 2's trivial-PR shortcut said `submit
  "Reviewed — no issues found." per step 5`, but step 5's rewrite removed
  exactly that preamble. Aligned both: trivial PRs and no-actionable-issues
  PRs now produce a body that opens with "No new issues found." followed by
  the PR summary, so the user gets the headline up front and still sees what
  was reviewed.
- docs/pr-reviews.mdx: dropped the "customize the summary style with Summary
  instructions in the console" sentence (the editor was removed in the
  firewall commit). Replaced with a note that the snapshot uses Pullfrog's
  built-in format and is not user-customizable.
- wiki/prompt.md, wiki/modes.md: rewrote the snapshot-prompt entries to
  reflect the firewall — DEFAULT_PR_SUMMARY_INSTRUCTIONS is the entire
  prompt, prSummaryCommentInstructions is no longer wired in.

* drop orphaned prSummaryCommentInstructions column

Prod audit (455 repos): 5 non-null rows on a single account, all containing the
literal placeholder text from the InstructionsEditor we removed in the firewall
commit. No account has an intentional preference set, so silent-ignore (the
keep-for-retreat option) costs us nothing meaningful while leaving an orphan
column in the schema. Drop it.

- prisma/schema.prisma: remove the column
- prisma/migrations/20260506000000_drop_pr_summary_comment_instructions:
  ALTER TABLE ... DROP COLUMN
- utils/schemas/triggers.ts: drop the matching zod entry

* drop body splicing; snapshot is internal-only

User-visible PR summarization continues to ship in Review and IncrementalReview
review bodies (which already render PR_SUMMARY_FORMAT and "Reviewed changes"
respectively). The snapshot tmpfile is now purely durable cross-run agent
context — seed, edit-in-place, save to DB, feed the next run. Massive
simplification: the body splice mechanics, the two-toggle gating matrix, the
summaryHandlingCovered race tracking, and the synchronize summary-only Task
fallback all go away.

Code:
- prSummary.ts: drop splice/strip/marker code (`splicePrSummary`,
  `stripExistingSummaryBlock`, `buildSummaryBlock`, `extractPrSummary`,
  PULLFROG_SUMMARY_START/END). keep scaffold, instructions, seed/read.
- main.ts: rename persistAndPostSummary -> persistSummary; collapse to a
  single DB PATCH. drop pulls.get/pulls.update, drop AbortSignal timeout,
  drop in-flight promise machinery, drop prSummaryToBody plumbing.
- ToolState: add summarySeed (replaces local var in main.ts so persist can
  compare). drop prSummaryToBody and summaryPersistInFlight.
- persistSummary now compares against the seed and skips the DB write
  with a warning when unchanged — saving the seed verbatim is either a
  no-op or persists the placeholder scaffold, neither useful.
- postRun.ts: when summary-stale is the only failing gate and the resume
  turn itself fails, restore the pre-resume successful result and break.
  symmetric with the existing reflection-failure preservation. summary-stale
  can no longer flip a successful run to failed.

Webhook:
- pull_request_opened: generateSummary follows prReReview only (the snapshot
  has no consumer when re-review is off).
- pull_request_synchronize: collapses to "if prReReview enabled, dispatch
  IncrementalReview". the summaryHandlingCovered flag, the same-SHA/in-flight
  coordination it was protecting, and the summary-only Task fallback all
  delete cleanly.

UI / config:
- drop SummarizePRsTrigger (the toggle gated body splice; with that gone
  it has no behavior). drop sidebar entry, console import, Text icon import.
- drop prSummaryComment from triggers zod schema, prisma schema, preview
  settings script.

Migration: squash the two existing migrations into one timestamped
20260507000000_pr_summary_snapshot covering all three column changes
(add summarySnapshot on workflow_runs, drop prSummaryCommentInstructions
and prSummaryComment on repos). repo convention is one migration per PR.

Action: bump 0.0.203 -> 0.0.205 (payload contract changed: prSummaryToBody
removed; main is at 0.0.204).

Out-of-diff cleanup:
- review.ts:190 + review.test.ts:651 — "Reviewed — no issues found." ->
  "No new issues found." to match the canonical body in modes.ts.

Verified: pnpm typecheck clean, pnpm lint clean, postRun + review tests
pass, dev DB reset against production and the squashed migration applied
cleanly (summarySnapshot present, prSummaryComment / prSummaryCommentInstructions
both gone).

* re-orient snapshot toward functional summary; drop prior-review-feedback section

Empirical audit on preview-568 PR #5 showed the snapshot IS load-bearing
for the orchestrator: lens-dispatch prompts on incremental runs carried
forward context from the snapshot's risk register (e.g. "the JSDoc
explicitly scopes to code points — do not flag grapheme-cluster issues"
on the surrogate-pair fix run, "consistency with native padStart" on the
padStart-added run). The orchestrator was reading the snapshot, reasoning
about it, and using it to anti-prime / focus subagents — exactly the
high-leverage path. My earlier "snapshot is write-only" claim was wrong.

The shape, however, was steering it toward review-history-log instead of
functional summary. This commit re-orients:

- prSummary.ts: replace the four-section scaffold (~580 chars of placeholder
  italics under "What this PR does / Key changes / Risk / Reviewed in prior
  runs") with a minimal seed (~150 chars: just a header + a one-line
  comment about what the file is for). different PRs warrant different
  organization; forcing a refactor and a feature into the same template
  is procrustean. minimal seed also makes the unchanged-from-seed gate
  in persistSummary more sensitive.

- selectMode.ts addendum: rewrite around three principles. (1) the snapshot
  is a FUNCTIONAL summary of what the PR does and the risks it carries,
  not a chronological review log — commit history can already be
  reconstructed from list_pull_request_reviews. (2) the orchestrator should
  USE the snapshot during triage and dispatch — concrete example given of
  carrying snapshot context into subagent lens prompts. (3) structure is
  the agent's call; stable headings make snapshots range-diff cleanly when
  they fit, but riff when they don't.

- modes.ts IncrementalReview: drop the "Prior review feedback" checklist
  from the user-facing review body (step 6b gone, step 7 ELSE IFs cleaned
  up). It duplicated content that's already covered by the Reviewed-changes
  bullets and tracked durably in the snapshot for the next agent run; in
  the user-facing body it was noise. step 3 still fetches prior reviews
  but its role is now just filtering aggregation in step 5, not rendering.

- AGENTS.md: codify "no follow-ups" rule. when an issue is identified
  during code review, fix it in this PR — PR scope does not constrain
  quality. follow-up TODOs are forbidden as a substitute for doing the
  work now.

Empirical evidence supporting the re-orientation:

- Run 25568912293 (PR#5 incr1, surrogate-pair fix): orchestrator's
  correctness lens dispatch said "Do NOT flag grapheme-cluster issues
  — the JSDoc scopes to code points." The grapheme-cluster framing was
  not in the diff; it was downstream of the snapshot's prior risk-section
  framing of truncate's contract. Snapshot influencing dispatch.

- Run 25569054779 (PR#5 incr2, padStart added): orchestrator's correctness
  lens dispatch enumerated edge cases including "consistency with native
  String.prototype.padStart contract" and "fill = multi-code-point string
  (e.g. emoji)". Both threads carried over from the snapshot's prior
  truncate code-point-vs-code-unit discussion. Snapshot informing the
  shape of what was looked for.

The cost of maintaining the snapshot (~800 tokens, ~$0.005/run) is
trivially affordable when it materially improves orchestrator triage
on the 1-5 lenses dispatched per review.
2026-05-08 19:28:24 +00:00

710 lines
28 KiB
TypeScript

import { describe, expect, it, vi } from "vitest";
import {
buildCommentableMap,
type CommentableLines,
clearStrandedPendingReview,
commentableLinesForFile,
createReviewWithStrandedRecovery,
type DroppedComment,
duplicateReviewDecision,
formatDroppedCommentsNote,
MAX_DROPPED_COMMENT_LINES,
type ReviewCommentInput,
reviewSkipDecision,
validateInlineComments,
} from "./review.ts";
import type { ToolContext } from "./server.ts";
describe("commentableLinesForFile", () => {
it("returns empty sets for missing patches (binary or no changes)", () => {
const result = commentableLinesForFile(undefined);
expect(result.LEFT.size).toBe(0);
expect(result.RIGHT.size).toBe(0);
});
it("collects added lines on RIGHT, removed lines on LEFT, context on both", () => {
const patch = ["@@ -10,3 +10,4 @@", " ctx1", "-old", "+new", "+new2", " ctx2"].join("\n");
const { LEFT, RIGHT } = commentableLinesForFile(patch);
expect([...LEFT].sort((a, b) => a - b)).toEqual([10, 11, 12]);
expect([...RIGHT].sort((a, b) => a - b)).toEqual([10, 11, 12, 13]);
});
it("handles multiple hunks", () => {
const patch = ["@@ -1,2 +1,2 @@", " a", "-b", "+B", "@@ -20,1 +20,2 @@", " x", "+y"].join("\n");
const { LEFT, RIGHT } = commentableLinesForFile(patch);
expect(RIGHT.has(2)).toBe(true); // +B
expect(RIGHT.has(21)).toBe(true); // +y
expect(LEFT.has(2)).toBe(true); // -b
expect(LEFT.has(20)).toBe(true); // context x
expect(RIGHT.has(20)).toBe(true); // context x
});
it("ignores the 'no newline at end of file' marker", () => {
const patch = ["@@ -1,1 +1,1 @@", "-old", "\\ No newline at end of file", "+new"].join("\n");
const { LEFT, RIGHT } = commentableLinesForFile(patch);
expect(LEFT.has(1)).toBe(true);
expect(RIGHT.has(1)).toBe(true);
expect(LEFT.size).toBe(1);
expect(RIGHT.size).toBe(1);
});
it("parses hunk headers without explicit counts", () => {
// single-line hunks can omit ",<count>"
const patch = ["@@ -5 +5 @@", "-old", "+new"].join("\n");
const { LEFT, RIGHT } = commentableLinesForFile(patch);
expect(LEFT.has(5)).toBe(true);
expect(RIGHT.has(5)).toBe(true);
});
});
function buildMap(entries: Array<[string, string]>): Map<string, CommentableLines> {
const map = new Map<string, CommentableLines>();
for (const [file, patch] of entries) {
map.set(file, commentableLinesForFile(patch));
}
return map;
}
describe("validateInlineComments", () => {
const patch = ["@@ -10,2 +10,3 @@", " ctx", "-old", "+new", "+new2"].join("\n");
const diffMap = buildMap([["src/foo.ts", patch]]);
const base = (overrides: Partial<ReviewCommentInput>): ReviewCommentInput => ({
path: "src/foo.ts",
line: 11,
side: "RIGHT",
body: "LGTM",
...overrides,
});
it("keeps comments anchored to added lines on RIGHT", () => {
const result = validateInlineComments([base({ line: 12 })], diffMap);
expect(result.valid).toHaveLength(1);
expect(result.dropped).toHaveLength(0);
});
it("keeps comments anchored to removed lines on LEFT", () => {
const result = validateInlineComments([base({ line: 11, side: "LEFT" })], diffMap);
expect(result.valid).toHaveLength(1);
expect(result.dropped).toHaveLength(0);
});
it("drops comments on files not in the diff", () => {
const result = validateInlineComments([base({ path: "other/bar.ts" })], diffMap);
expect(result.valid).toHaveLength(0);
expect(result.dropped).toHaveLength(1);
expect(result.dropped[0].reason).toContain("file not in PR diff");
});
it("distinguishes binary/no-patch files from files with hunks", () => {
// file present in the PR but with no patch data (binary file).
const binaryMap = buildMap([
["src/foo.ts", patch],
["assets/logo.png", undefined as unknown as string],
]);
const result = validateInlineComments([base({ path: "assets/logo.png", line: 1 })], binaryMap);
expect(result.valid).toHaveLength(0);
expect(result.dropped).toHaveLength(1);
expect(result.dropped[0].reason).toContain("no textual diff");
expect(result.dropped[0].reason).not.toContain("not inside a diff hunk");
});
it("drops comments on lines outside diff hunks", () => {
const result = validateInlineComments([base({ line: 500 })], diffMap);
expect(result.valid).toHaveLength(0);
expect(result.dropped).toHaveLength(1);
expect(result.dropped[0].reason).toContain("line 500");
expect(result.dropped[0].reason).toContain("RIGHT");
});
it("drops comments whose side mismatches the hunk (added line on LEFT)", () => {
// line 12 is "+new" — only in RIGHT. Asking for it on LEFT should drop.
const result = validateInlineComments([base({ line: 12, side: "LEFT" })], diffMap);
expect(result.valid).toHaveLength(0);
expect(result.dropped).toHaveLength(1);
});
it("drops multi-line comments where start_line is out of range", () => {
const result = validateInlineComments([base({ line: 12, start_line: 3 })], diffMap);
expect(result.valid).toHaveLength(0);
expect(result.dropped).toHaveLength(1);
expect(result.dropped[0].reason).toContain("start_line 3");
});
it("keeps multi-line comments fully inside a hunk", () => {
const result = validateInlineComments([base({ line: 12, start_line: 11 })], diffMap);
expect(result.valid).toHaveLength(1);
expect(result.dropped).toHaveLength(0);
});
it("drops inverted ranges (start_line > line) with a precise reason", () => {
// both 11 and 12 anchor in the hunk, but GitHub 422s with "invalid line
// numbers" when start_line > line. dropping locally avoids the opaque
// remote failure and tells the agent exactly what to fix.
const result = validateInlineComments([base({ line: 11, start_line: 12 })], diffMap);
expect(result.valid).toHaveLength(0);
expect(result.dropped).toHaveLength(1);
expect(result.dropped[0].reason).toMatch(/start_line 12 is after line 11/);
expect(result.dropped[0].reason).toMatch(/start_line <= line/);
});
it("partitions a batch — valid and invalid comments survive independently", () => {
const result = validateInlineComments(
[base({ line: 12 }), base({ line: 9999 }), base({ path: "missing.ts" })],
diffMap
);
expect(result.valid).toHaveLength(1);
expect(result.dropped).toHaveLength(2);
});
it("defaults side to RIGHT when omitted", () => {
const result = validateInlineComments([{ path: "src/foo.ts", line: 12, body: "" }], diffMap);
expect(result.valid).toHaveLength(1);
});
});
describe("buildCommentableMap", () => {
it("returns the cached snapshot when toolState matches PR and checkoutSha", async () => {
// simulates checkout_pr having pre-populated the cache. the cache pins the
// commentable lines to checkoutSha so review-time validation matches what
// GitHub anchors to, even if the PR is updated mid-run.
const cached = buildMap([["src/foo.ts", "@@ -1,1 +1,2 @@\n ctx\n+new"]]);
const paginate = vi.fn();
const ctx = {
octokit: { paginate, rest: { pulls: { listFiles: {} } } },
repo: { owner: "o", name: "r" },
toolState: {
commentableLinesByFile: cached,
commentableLinesPullNumber: 42,
commentableLinesCheckoutSha: "sha1",
checkoutSha: "sha1",
},
} as unknown as ToolContext;
const result = await buildCommentableMap(ctx, 42);
expect(result).toBe(cached);
expect(paginate).not.toHaveBeenCalled();
});
it("ignores the cached snapshot when it was built for a different PR", async () => {
// without this guard, checkout_pr(B) followed by review(A) would validate
// A's inline comments against B's diff — silently dropping valid anchors.
const cached = buildMap([["src/foo.ts", "@@ -1,1 +1,2 @@\n ctx\n+new"]]);
const freshFile = { filename: "src/bar.ts", patch: "@@ -1,1 +1,2 @@\n ctx\n+added" };
const paginate = vi.fn().mockResolvedValue([freshFile]);
const ctx = {
octokit: { paginate, rest: { pulls: { listFiles: {} } } },
repo: { owner: "o", name: "r" },
toolState: {
commentableLinesByFile: cached,
commentableLinesPullNumber: 99,
commentableLinesCheckoutSha: "sha1",
checkoutSha: "sha1",
},
} as unknown as ToolContext;
const result = await buildCommentableMap(ctx, 42);
expect(paginate).toHaveBeenCalledTimes(1);
expect(result).not.toBe(cached);
expect(result.get("src/bar.ts")?.RIGHT.has(2)).toBe(true);
});
it("ignores the cached snapshot when checkoutSha has moved since it was built", async () => {
// simulates a second checkout_pr(42) that bumped checkoutSha but failed
// before repopulating the cache (e.g., listFiles rate-limited). without
// the sha guard, review would reuse the stale snapshot against the new
// anchor and either drop valid comments or let invalid ones through.
const cached = buildMap([["src/foo.ts", "@@ -1,1 +1,2 @@\n ctx\n+new"]]);
const freshFile = { filename: "src/bar.ts", patch: "@@ -1,1 +1,2 @@\n ctx\n+added" };
const paginate = vi.fn().mockResolvedValue([freshFile]);
const ctx = {
octokit: { paginate, rest: { pulls: { listFiles: {} } } },
repo: { owner: "o", name: "r" },
toolState: {
commentableLinesByFile: cached,
commentableLinesPullNumber: 42,
commentableLinesCheckoutSha: "sha-old",
checkoutSha: "sha-new",
},
} as unknown as ToolContext;
const result = await buildCommentableMap(ctx, 42);
expect(paginate).toHaveBeenCalledTimes(1);
expect(result).not.toBe(cached);
});
it("falls back to listFiles when no cache exists", async () => {
const file = { filename: "src/bar.ts", patch: "@@ -1,1 +1,2 @@\n ctx\n+added" };
const paginate = vi.fn().mockResolvedValue([file]);
const ctx = {
octokit: { paginate, rest: { pulls: { listFiles: {} } } },
repo: { owner: "o", name: "r" },
toolState: {},
} as unknown as ToolContext;
const result = await buildCommentableMap(ctx, 42);
expect(paginate).toHaveBeenCalledTimes(1);
expect(result.get("src/bar.ts")?.RIGHT.has(2)).toBe(true);
});
});
describe("formatDroppedCommentsNote", () => {
it("renders single-line dropped entries with `path:line`", () => {
const dropped: DroppedComment[] = [
{
path: "src/foo.ts",
line: 42,
side: "RIGHT",
reason: "line 42 (RIGHT) is not inside a diff hunk",
},
];
const note = formatDroppedCommentsNote(dropped);
expect(note).toContain("**Note:** 1 inline comment(s) dropped");
expect(note).toContain("`src/foo.ts:42` (RIGHT)");
expect(note).toContain("line 42 (RIGHT) is not inside a diff hunk");
});
it("renders multi-line dropped entries with `path:start-end`", () => {
const dropped: DroppedComment[] = [
{
path: "src/bar.ts",
line: 20,
startLine: 15,
side: "LEFT",
reason: "start_line 15 (LEFT) is not inside a diff hunk",
},
];
const note = formatDroppedCommentsNote(dropped);
expect(note).toContain("`src/bar.ts:15-20` (LEFT)");
});
it("falls back to single-line format when startLine equals line", () => {
const dropped: DroppedComment[] = [
{ path: "src/baz.ts", line: 7, startLine: 7, side: "RIGHT", reason: "file not in PR diff" },
];
const note = formatDroppedCommentsNote(dropped);
expect(note).toContain("`src/baz.ts:7` (RIGHT)");
expect(note).not.toContain("7-7");
});
it("caps detail lines and reports the remainder so body stays under GitHub's size limit", () => {
const overflow = MAX_DROPPED_COMMENT_LINES + 7;
const dropped: DroppedComment[] = Array.from({ length: overflow }, (_, i) => ({
path: `src/file${i}.ts`,
line: i + 1,
side: "RIGHT" as const,
reason: "file not in PR diff",
}));
const note = formatDroppedCommentsNote(dropped);
expect(note).toContain(`**Note:** ${overflow} inline comment(s) dropped`);
// still reports the full count in the header
expect(note).toContain(`${overflow} inline comment(s)`);
// first entry shown, last entry elided
expect(note).toContain("`src/file0.ts:1` (RIGHT)");
expect(note).not.toContain(`src/file${overflow - 1}.ts`);
expect(note).toContain("…and 7 more dropped comment(s) not shown");
});
it("does not add a truncation line when drops fit under the cap", () => {
const dropped: DroppedComment[] = Array.from({ length: MAX_DROPPED_COMMENT_LINES }, (_, i) => ({
path: `src/f${i}.ts`,
line: i + 1,
side: "RIGHT" as const,
reason: "file not in PR diff",
}));
const note = formatDroppedCommentsNote(dropped);
expect(note).not.toContain("more dropped comment(s) not shown");
});
});
describe("clearStrandedPendingReview", () => {
function pendingReviewError(status: number, message: string): Error {
const err = new Error(message) as Error & { status: number };
err.status = status;
return err;
}
const baseParams = { owner: "o", repo: "r", pull_number: 42 };
it("rethrows the original error when status is not 422", async () => {
const err = pendingReviewError(500, "server exploded");
const ctx = {
octokit: {
paginate: vi.fn(),
rest: { pulls: { listReviews: {}, deletePendingReview: vi.fn() } },
},
} as unknown as ToolContext;
await expect(clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })).rejects.toBe(
err
);
expect(ctx.octokit.paginate).not.toHaveBeenCalled();
});
it("rethrows the original error when 422 does not mention pending review", async () => {
// a 422 from an unrelated validation (e.g., invalid anchor) must not
// trigger a destructive delete of the user's own draft.
const err = pendingReviewError(422, "pull_request_review_thread is not part of the diff");
const deletePendingReview = vi.fn();
const ctx = {
octokit: {
paginate: vi.fn(),
rest: { pulls: { listReviews: {}, deletePendingReview } },
},
} as unknown as ToolContext;
await expect(clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })).rejects.toBe(
err
);
expect(ctx.octokit.paginate).not.toHaveBeenCalled();
expect(deletePendingReview).not.toHaveBeenCalled();
});
it("rethrows the original error when no PENDING review is found", async () => {
// 422 claimed a pending exists but listReviews returns only SUBMITTED —
// likely a transient GitHub inconsistency. retry won't help; surface the
// original error so the caller sees why createReview failed.
const err = pendingReviewError(422, "User already has a pending review for this pull request");
const paginate = vi.fn().mockResolvedValue([{ id: 1, state: "COMMENTED" } as unknown as never]);
const deletePendingReview = vi.fn();
const ctx = {
octokit: { paginate, rest: { pulls: { listReviews: {}, deletePendingReview } } },
} as unknown as ToolContext;
await expect(clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })).rejects.toBe(
err
);
expect(paginate).toHaveBeenCalledTimes(1);
expect(deletePendingReview).not.toHaveBeenCalled();
});
it("deletes the leftover PENDING review and resolves on success", async () => {
const err = pendingReviewError(422, "User already has a pending review for this pull request");
const paginate = vi.fn().mockResolvedValue([
{ id: 100, state: "COMMENTED" },
{ id: 101, state: "PENDING" },
] as unknown as never);
const deletePendingReview = vi.fn().mockResolvedValue({ status: 204 });
const ctx = {
octokit: { paginate, rest: { pulls: { listReviews: {}, deletePendingReview } } },
} as unknown as ToolContext;
await expect(
clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })
).resolves.toBeUndefined();
expect(deletePendingReview).toHaveBeenCalledWith({
owner: "o",
repo: "r",
pull_number: 42,
review_id: 101,
});
});
it("swallows a 404 from deletePendingReview (raced with another cleanup)", async () => {
const err = pendingReviewError(422, "User already has a pending review for this pull request");
const paginate = vi.fn().mockResolvedValue([{ id: 101, state: "PENDING" }] as unknown as never);
const deletePendingReview = vi.fn().mockRejectedValue(pendingReviewError(404, "not found"));
const ctx = {
octokit: { paginate, rest: { pulls: { listReviews: {}, deletePendingReview } } },
} as unknown as ToolContext;
await expect(
clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })
).resolves.toBeUndefined();
});
it("swallows a 422 from deletePendingReview (draft submitted by a concurrent caller)", async () => {
const err = pendingReviewError(422, "User already has a pending review for this pull request");
const paginate = vi.fn().mockResolvedValue([{ id: 101, state: "PENDING" }] as unknown as never);
const deletePendingReview = vi
.fn()
.mockRejectedValue(pendingReviewError(422, "review has already been submitted"));
const ctx = {
octokit: { paginate, rest: { pulls: { listReviews: {}, deletePendingReview } } },
} as unknown as ToolContext;
await expect(
clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })
).resolves.toBeUndefined();
});
it("rethrows the ORIGINAL 422 when listReviews fails so the real blocker isn't masked", async () => {
// if listReviews throws a transient 502 during cleanup, we must surface
// the pending-review 422 — not the 502 — so the caller sees the actual
// reason createReview failed and can retry the cleanup. masking the 422
// with a 502 previously sent agents chasing phantom server errors.
const err = pendingReviewError(422, "User already has a pending review for this pull request");
const paginate = vi.fn().mockRejectedValue(pendingReviewError(502, "bad gateway"));
const deletePendingReview = vi.fn();
const ctx = {
octokit: { paginate, rest: { pulls: { listReviews: {}, deletePendingReview } } },
} as unknown as ToolContext;
await expect(clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })).rejects.toBe(
err
);
expect(deletePendingReview).not.toHaveBeenCalled();
});
it("rethrows non-404/422 errors from deletePendingReview so the real cause surfaces", async () => {
const err = pendingReviewError(422, "User already has a pending review for this pull request");
const paginate = vi.fn().mockResolvedValue([{ id: 101, state: "PENDING" }] as unknown as never);
const cleanupErr = pendingReviewError(500, "internal server error");
const deletePendingReview = vi.fn().mockRejectedValue(cleanupErr);
const ctx = {
octokit: { paginate, rest: { pulls: { listReviews: {}, deletePendingReview } } },
} as unknown as ToolContext;
await expect(clearStrandedPendingReview(ctx, { ...baseParams, originalErr: err })).rejects.toBe(
cleanupErr
);
});
});
describe("createReviewWithStrandedRecovery", () => {
function pendingReviewError(status: number, message: string): Error {
const err = new Error(message) as Error & { status: number };
err.status = status;
return err;
}
const params = {
owner: "o",
repo: "r",
pull_number: 42,
event: "COMMENT" as const,
};
it("returns createReview result directly when no stranded draft exists", async () => {
const response = { data: { id: 1, node_id: "n1" } };
const createReview = vi.fn().mockResolvedValue(response);
const ctx = {
octokit: {
paginate: vi.fn(),
rest: { pulls: { createReview, listReviews: {}, deletePendingReview: vi.fn() } },
},
} as unknown as ToolContext;
await expect(createReviewWithStrandedRecovery(ctx, params)).resolves.toBe(response);
expect(createReview).toHaveBeenCalledTimes(1);
});
it("clears a stranded PENDING draft and retries on pending-review 422 — covers the no-body path", async () => {
// regression: the no-body review path (approve-with-no-feedback,
// comments-only) used to call createReview directly. a prior body-path run
// that crashed between createReview(PENDING) and submitReview would leave
// a stranded PENDING draft; every subsequent no-body review would 422
// with "already has a pending review" until a body-path run happened to
// clear it. this test exercises the recovery: first createReview 422s,
// clearStranded deletes the leftover, and the retry succeeds.
const stranded = pendingReviewError(
422,
"User already has a pending review for this pull request"
);
const response = { data: { id: 2, node_id: "n2" } };
const createReview = vi.fn().mockRejectedValueOnce(stranded).mockResolvedValueOnce(response);
const paginate = vi.fn().mockResolvedValue([{ id: 77, state: "PENDING" }] as unknown as never);
const deletePendingReview = vi.fn().mockResolvedValue({ status: 204 });
const ctx = {
octokit: {
paginate,
rest: { pulls: { createReview, listReviews: {}, deletePendingReview } },
},
} as unknown as ToolContext;
await expect(createReviewWithStrandedRecovery(ctx, params)).resolves.toBe(response);
expect(createReview).toHaveBeenCalledTimes(2);
expect(deletePendingReview).toHaveBeenCalledWith({
owner: "o",
repo: "r",
pull_number: 42,
review_id: 77,
});
});
it("rethrows non-pending 422s without retrying — avoids masking a real validation error", async () => {
// if the 422 is unrelated to a stranded draft (e.g. body too long, bad
// anchor), clearStrandedPendingReview rethrows and we must not retry
// blindly — a retry would just hit the same validation and double the
// GitHub API traffic for nothing.
const err = pendingReviewError(422, "body is too long");
const createReview = vi.fn().mockRejectedValue(err);
const paginate = vi.fn();
const deletePendingReview = vi.fn();
const ctx = {
octokit: {
paginate,
rest: { pulls: { createReview, listReviews: {}, deletePendingReview } },
},
} as unknown as ToolContext;
await expect(createReviewWithStrandedRecovery(ctx, params)).rejects.toBe(err);
expect(createReview).toHaveBeenCalledTimes(1);
expect(deletePendingReview).not.toHaveBeenCalled();
});
});
describe("reviewSkipDecision", () => {
// GitHub 422s `event: "COMMENT"` reviews with no body + no comments
// ("{\"message\":\"Unprocessable Entity\",\"errors\":[\"\"]}"). verified
// empirically against repos/pullfrog/preview-546-run-issues-fixes/pulls/1
// with and without commit_id set. the skip function must return a decision
// for every shape that lands on that API call.
it("skips with 'no-issues' when !approved + empty body + no comments", () => {
const decision = reviewSkipDecision({
approved: false,
body: "",
hasComments: false,
prApproveEnabled: true,
});
expect(decision?.kind).toBe("no-issues");
expect(decision?.reason).toContain("nothing to post");
});
it("treats null body the same as empty string", () => {
const decision = reviewSkipDecision({
approved: false,
body: null,
hasComments: false,
prApproveEnabled: true,
});
expect(decision?.kind).toBe("no-issues");
});
it("treats undefined body the same as empty string", () => {
const decision = reviewSkipDecision({
approved: false,
body: undefined,
hasComments: false,
prApproveEnabled: true,
});
expect(decision?.kind).toBe("no-issues");
});
it("skips with 'empty-downgraded-approve' when approved + !prApproveEnabled + empty", () => {
// this is the F3 regression case — agent requests APPROVE, runtime
// downgrades to COMMENT (prApproveEnabled off), and the empty COMMENT
// 422s at GitHub. before this fix, the tool returned a stranded-success
// shape that didn't map to any persisted review.
const decision = reviewSkipDecision({
approved: true,
body: "",
hasComments: false,
prApproveEnabled: false,
});
expect(decision?.kind).toBe("empty-downgraded-approve");
expect(decision?.reason).toContain("prApproveEnabled is disabled");
});
it("does NOT skip legitimate bare APPROVE (approved + prApproveEnabled + empty)", () => {
// GitHub accepts empty APPROVE reviews — the stamp itself is the content.
// skipping here would silently drop agents' real approvals.
const decision = reviewSkipDecision({
approved: true,
body: "",
hasComments: false,
prApproveEnabled: true,
});
expect(decision).toBeNull();
});
it("does NOT skip when body is present (no-issues path)", () => {
const decision = reviewSkipDecision({
approved: false,
body: "found some issues",
hasComments: false,
prApproveEnabled: true,
});
expect(decision).toBeNull();
});
it("does NOT skip when body is present (downgrade path)", () => {
// approved+!prApproveEnabled with a body becomes a real COMMENT review
// (downgrade + body). GitHub accepts those; don't skip.
const decision = reviewSkipDecision({
approved: true,
body: "nits follow",
hasComments: false,
prApproveEnabled: false,
});
expect(decision).toBeNull();
});
it("does NOT skip when comments are present (no-issues path)", () => {
const decision = reviewSkipDecision({
approved: false,
body: "",
hasComments: true,
prApproveEnabled: true,
});
expect(decision).toBeNull();
});
it("does NOT skip when comments are present (downgrade path)", () => {
const decision = reviewSkipDecision({
approved: true,
body: "",
hasComments: true,
prApproveEnabled: false,
});
expect(decision).toBeNull();
});
});
describe("duplicateReviewDecision", () => {
// regression: colinhacks/zod#5897 had two reviews submitted from the same
// workflow run 8 seconds apart — a substantive review followed by an empty
// "No new issues found." follow-up. the agent re-classified the first
// review's non-blocking observations as "no actionable issues" and
// submitted the canonical body per modes.ts. this guard makes the second
// call a no-op without burning a GitHub API call or polluting the PR.
it("allows the first submission when no prior review exists", () => {
const decision = duplicateReviewDecision({
existing: undefined,
currentCheckoutSha: "sha1",
});
expect(decision).toBeNull();
});
it("blocks a second submission when checkoutSha matches the prior reviewedSha", () => {
// exact reproduction of the zod#5897 shape: same session, same checked-out
// SHA, second create_pull_request_review call.
const decision = duplicateReviewDecision({
existing: { id: 100, reviewedSha: "sha1" },
currentCheckoutSha: "sha1",
});
expect(decision?.kind).toBe("already-submitted");
expect(decision?.reviewId).toBe(100);
expect(decision?.reason).toContain("already submitted");
expect(decision?.reason).toContain("checkout_pr");
});
it("allows a follow-up when checkoutSha advanced past the prior reviewedSha", () => {
// the new-commits-mid-review path advances toolState.checkoutSha to the
// new HEAD before returning, and the agent is told to call checkout_pr
// again — both paths leave checkoutSha != reviewedSha. those are real
// follow-up reviews and must go through.
const decision = duplicateReviewDecision({
existing: { id: 100, reviewedSha: "sha-old" },
currentCheckoutSha: "sha-new",
});
expect(decision).toBeNull();
});
it("blocks when checkoutSha is missing — cannot prove the SHA moved", () => {
// if the agent never called checkout_pr, we have no anchor to compare
// against. assume duplicate rather than letting a second review through
// — the prior review still satisfies the agent's intent.
const decision = duplicateReviewDecision({
existing: { id: 100, reviewedSha: "sha1" },
currentCheckoutSha: undefined,
});
expect(decision?.kind).toBe("already-submitted");
});
it("blocks when prior reviewedSha is missing — cannot prove the SHA moved", () => {
// belt-and-suspenders: if for any reason the prior review didn't capture
// a reviewedSha, treat the second call as a duplicate to be safe.
const decision = duplicateReviewDecision({
existing: { id: 100, reviewedSha: undefined },
currentCheckoutSha: "sha1",
});
expect(decision?.kind).toBe("already-submitted");
});
});