post-run gate: fail review-mode runs that don't submit a review or progress (#638)

* post-run gate: fail the run when review mode finishes without a review or progress review-mode runs that ended in a text-only assistant turn ("now I have enough to draft the review...") were silently swallowed: the progress comment was deleted by stranded-comment cleanup and no review appeared on the PR. user- visible result was identical to "the agent never ran." caught in https://github.com/pullfrog/app/actions/runs/25583698781. new post-run gate alongside stopHook / dirtyTree / summaryStale: derived inline from toolState (selectedMode in {Review, IncrementalReview} && !review && !finalSummaryWritten && hadProgressComment) — no parallel toolState flag. when it fires, the resume prompt nudges the agent to call either create_pull_request_review or report_progress; persistent failure after MAX_POST_RUN_RETRIES surfaces as AgentResult.error. also: when the post-run loop returns success=false, write the error to the progress comment before the stranded-comment cleanup runs, and skip the delete in that case. previously a !success run from the loop would lose the error message into the void. IncrementalReview's trivial-skip branch now calls report_progress with a brief "no review warranted" note instead of exiting silently — keeps the contract symmetric with the gate and gives the user a visible signal even on no-op review runs. documents the literal-record design rule on the ToolState interface so future fields don't drift back into derived/absence-encoding state. * review feedback: mode-aware nudge, gate-error preservation, prompt order addresses three findings from the auto-review on this PR: 1. Review mode nudge no longer offers `report_progress` as an exit. Review mode's contract (modes.ts step 5) forbids it; the gate previously sent contradictory copy. IncrementalReview's nudge still offers both since its trivial-skip path legitimately allows `report_progress`. 2. `writeJobSummary` is now wrapped in try/catch on the success-path cleanup. without this, a throw there jumped to the outer catch and overwrote the gate's failure message in the progress comment with the (less actionable) writeJobSummary error — restoring exactly the invisible-failure UX this PR fixes. step-summary writes are informational; let them fail silently. 3. `buildPostRunPrompt` reorders gates to match the terminal hard-fail order: `stopHook` → `unsubmittedReview` → `dirtyTree` → `summaryStale`. when both hard-fail gates co-fire (rare in review modes), the prompt's emphasis now matches the user-visible failure message. new test asserts the IncrementalReview nudge offers both exits while the Review nudge offers only `create_pull_request_review`. e2e validation already passed against pullfrog/preview-638-review-stop-hook PR #1 (gate fired once; agent recovered on second turn). * mode-aware terminal error copy second auto-review caught a residual contradiction: the terminal hard-fail error string reported "create_pull_request_review or report_progress" for both modes, even though the new mode-aware nudge tells Review-mode agents "Review mode does not have a no-submit exit". the error message now mirrors the nudge — Review names only `create_pull_request_review`, IncrementalReview lists both. additional Review-mode hard-fail test asserts the absence of `report_progress` in the error.
2026-05-09 00:14:31 +00:00
parent 653fae47a5
commit 85d25a6fe6
8 changed files with 301 additions and 10 deletions
@@ -328,6 +328,138 @@ describe("runPostRunRetryLoop — reflection turn", () => {
    expect(mockedGetGitStatus).not.toHaveBeenCalled();
  });

+  it("nudges the agent when Review mode finished without submitting (no report_progress exit offered)", async () => {
+    // simulate the canonical bug: the agent picked Review mode, talked itself
+    // out of acting, and stopped without calling create_pull_request_review.
+    // the gate should fire a single resume that ONLY offers
+    // create_pull_request_review (Review mode's contract forbids
+    // report_progress as an exit). once the resume flips the toolState
+    // (modeled here by the closure returning null on the second call), the
+    // loop should exit cleanly with success=true.
+    let submitted = false;
+    const resume = vi
+      .fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
+      .mockImplementation(async () => {
+        submitted = true;
+        return successResult({ output: "review submitted" });
+      });
+
+    const result = await runPostRunRetryLoop({
+      initialResult: successResult({ output: "analysis turn" }),
+      initialUsage: undefined,
+      stopScript: null,
+      getUnsubmittedReview: () => (submitted ? null : "Review"),
+      resume,
+    });
+
+    expect(result.success).toBe(true);
+    expect(resume).toHaveBeenCalledTimes(1);
+    const prompt = resume.mock.calls[0]?.[0].prompt ?? "";
+    expect(prompt).toContain("MISSING REVIEW OUTPUT");
+    expect(prompt).toContain("Review mode");
+    expect(prompt).toContain("create_pull_request_review");
+    // Review mode contract: nudge must NOT offer report_progress as an exit
+    // (would contradict modes.ts step 5 "ALWAYS submit ... do NOT call
+    // report_progress").
+    expect(prompt).not.toContain("report_progress");
+  });
+
+  it("offers `report_progress` in the IncrementalReview nudge for the trivial-skip path", async () => {
+    // IncrementalReview's mode prompt step 7 has a legitimate "no new
+    // issues, non-substantive changes only → report_progress" exit. the
+    // nudge must offer it so the agent can satisfy the gate without
+    // submitting a vacuous re-review.
+    let satisfied = false;
+    const resume = vi
+      .fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
+      .mockImplementation(async () => {
+        satisfied = true;
+        return successResult({ output: "no review warranted, reported" });
+      });
+
+    const result = await runPostRunRetryLoop({
+      initialResult: successResult({ output: "analysis turn" }),
+      initialUsage: undefined,
+      stopScript: null,
+      getUnsubmittedReview: () => (satisfied ? null : "IncrementalReview"),
+      resume,
+    });
+
+    expect(result.success).toBe(true);
+    const prompt = resume.mock.calls[0]?.[0].prompt ?? "";
+    expect(prompt).toContain("IncrementalReview mode");
+    expect(prompt).toContain("create_pull_request_review");
+    expect(prompt).toContain("report_progress");
+  });
+
+  it("hard-fails IncrementalReview after MAX_POST_RUN_RETRIES with mode-appropriate error copy", async () => {
+    // parallel to the persistent-stop-hook test: if the agent keeps stopping
+    // without producing a review, the run must fail loudly with a clear error
+    // so the user knows something went wrong instead of seeing a deleted
+    // progress comment and silence. IncrementalReview's error string lists
+    // both exits the gate accepts.
+    const resume = vi
+      .fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
+      .mockResolvedValue(successResult({ output: "still didn't submit" }));
+
+    const result = await runPostRunRetryLoop({
+      initialResult: successResult({ output: "analysis turn" }),
+      initialUsage: undefined,
+      stopScript: null,
+      getUnsubmittedReview: () => "IncrementalReview",
+      resume,
+    });
+
+    expect(result.success).toBe(false);
+    expect(result.error).toContain("IncrementalReview");
+    expect(result.error).toContain("create_pull_request_review");
+    expect(result.error).toContain("report_progress");
+    expect(result.error).toContain("3 retry attempts");
+    expect(resume).toHaveBeenCalledTimes(3);
+  });
+
+  it("hard-fails Review with a mode-appropriate error string (no `report_progress`)", async () => {
+    // Review mode's contract is "always submit a review" (modes.ts step 5).
+    // the terminal error must mirror the nudge copy: only
+    // create_pull_request_review is named — listing report_progress would
+    // contradict the mode prompt and confuse triage.
+    const resume = vi
+      .fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
+      .mockResolvedValue(successResult({ output: "still didn't submit" }));
+
+    const result = await runPostRunRetryLoop({
+      initialResult: successResult({ output: "analysis turn" }),
+      initialUsage: undefined,
+      stopScript: null,
+      getUnsubmittedReview: () => "Review",
+      resume,
+    });
+
+    expect(result.success).toBe(false);
+    expect(result.error).toContain("Review mode");
+    expect(result.error).toContain("create_pull_request_review");
+    expect(result.error).not.toContain("report_progress");
+  });
+
+  it("does not fire the unsubmitted-review gate when review was submitted", async () => {
+    // negative case: the closure returns null (review or progress was
+    // recorded), the gate must stay silent regardless of mode.
+    const resume = vi
+      .fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
+      .mockResolvedValue(successResult());
+
+    const result = await runPostRunRetryLoop({
+      initialResult: successResult({ output: "review submitted inline" }),
+      initialUsage: undefined,
+      stopScript: null,
+      getUnsubmittedReview: () => null,
+      resume,
+    });
+
+    expect(result.success).toBe(true);
+    expect(resume).not.toHaveBeenCalled();
+  });
+
  it("aggregates usage across every gate retry", async () => {
    // billing/reporting rely on the usage summary reflecting the full run,
    // not just the final retry's slice. regression gate.