post-run gate: fail review-mode runs that don't submit a review or progress (#638)

* post-run gate: fail the run when review mode finishes without a review or progress

review-mode runs that ended in a text-only assistant turn ("now I have enough
to draft the review...") were silently swallowed: the progress comment was
deleted by stranded-comment cleanup and no review appeared on the PR. user-
visible result was identical to "the agent never ran." caught in
https://github.com/pullfrog/app/actions/runs/25583698781.

new post-run gate alongside stopHook / dirtyTree / summaryStale: derived
inline from toolState (selectedMode in {Review, IncrementalReview} && !review
&& !finalSummaryWritten && hadProgressComment) — no parallel toolState flag.
when it fires, the resume prompt nudges the agent to call either
create_pull_request_review or report_progress; persistent failure after
MAX_POST_RUN_RETRIES surfaces as AgentResult.error.

also: when the post-run loop returns success=false, write the error to the
progress comment before the stranded-comment cleanup runs, and skip the
delete in that case. previously a !success run from the loop would lose the
error message into the void.

IncrementalReview's trivial-skip branch now calls report_progress with a
brief "no review warranted" note instead of exiting silently — keeps the
contract symmetric with the gate and gives the user a visible signal even
on no-op review runs.

documents the literal-record design rule on the ToolState interface so
future fields don't drift back into derived/absence-encoding state.

* review feedback: mode-aware nudge, gate-error preservation, prompt order

addresses three findings from the auto-review on this PR:

1. Review mode nudge no longer offers `report_progress` as an exit. Review
   mode's contract (modes.ts step 5) forbids it; the gate previously sent
   contradictory copy. IncrementalReview's nudge still offers both since
   its trivial-skip path legitimately allows `report_progress`.

2. `writeJobSummary` is now wrapped in try/catch on the success-path
   cleanup. without this, a throw there jumped to the outer catch and
   overwrote the gate's failure message in the progress comment with the
   (less actionable) writeJobSummary error — restoring exactly the
   invisible-failure UX this PR fixes. step-summary writes are
   informational; let them fail silently.

3. `buildPostRunPrompt` reorders gates to match the terminal hard-fail
   order: `stopHook` → `unsubmittedReview` → `dirtyTree` → `summaryStale`.
   when both hard-fail gates co-fire (rare in review modes), the prompt's
   emphasis now matches the user-visible failure message.

new test asserts the IncrementalReview nudge offers both exits while the
Review nudge offers only `create_pull_request_review`. e2e validation
already passed against pullfrog/preview-638-review-stop-hook PR #1
(gate fired once; agent recovered on second turn).

* mode-aware terminal error copy

second auto-review caught a residual contradiction: the terminal hard-fail
error string reported "create_pull_request_review or report_progress" for
both modes, even though the new mode-aware nudge tells Review-mode agents
"Review mode does not have a no-submit exit". the error message now mirrors
the nudge — Review names only `create_pull_request_review`,
IncrementalReview lists both. additional Review-mode hard-fail test asserts
the absence of `report_progress` in the error.
This commit is contained in:
Colin McDonnell
2026-05-09 00:14:31 +00:00
committed by pullfrog[bot]
parent 653fae47a5
commit 85d25a6fe6
8 changed files with 301 additions and 10 deletions
+52 -3
View File
@@ -927,6 +927,19 @@ export async function main(): Promise<MainResult> {
summaryFilePath: toolState.summaryFilePath,
summarySeed: toolState.summarySeed,
learningsFilePath: toolState.learningsFilePath,
// post-run gate: derive "review mode finished without producing
// anything visible" inline from toolState. no parallel toolState flag —
// the absence of `review` and `finalSummaryWritten` is the signal.
// skipped when there's no progress comment to anchor the failure to
// (e.g. silent runs / non-issue events) so the gate doesn't fire
// on runs where there's nothing to display anyway.
getUnsubmittedReview: () => {
const mode = toolState.selectedMode;
if (mode !== "Review" && mode !== "IncrementalReview") return null;
if (toolState.review || toolState.finalSummaryWritten) return null;
if (!toolState.hadProgressComment) return null;
return mode;
},
onActivityTimeout: onInnerActivityTimeout,
onToolUse: (event) => {
const wasTracked = recordDiffReadFromToolUse({
@@ -1021,6 +1034,22 @@ export async function main(): Promise<MainResult> {
await persistLearnings(toolContext);
}
// when the agent harness returns success=false (e.g. unsubmitted-review
// gate exhausted retries, stop-hook persistently failing), surface the
// error in the progress comment so the user sees it instead of a
// deleted-comment void. mirrors the catch-block error reporting for
// thrown errors. runs before the stranded-comment cleanup below so
// the comment is still around to update; reportErrorToComment sets
// wasUpdated=true and the !result.success guard skips deletion.
if (!result.success && toolContext && toolState.progressComment) {
await reportErrorToComment({
toolState,
error: result.error || "agent run failed",
}).catch((error) => {
log.debug(`failure error report failed: ${error}`);
});
}
// clean up stranded progress comments. the comment is stale unless
// report_progress wrote a final summary to it — three sub-cases all reduce
// to !finalSummaryWritten:
@@ -1034,14 +1063,34 @@ export async function main(): Promise<MainResult> {
// so progressComment is already null by the time we get here for that path.
// uses finalSummaryWritten (not todoTracker.enabled or wasUpdated) so cleanup
// survives API failures in report_progress where cancel() ran but the write
// didn't succeed, and isn't fooled by writes to *other* artifacts.
if (toolContext && toolState.progressComment && !toolState.finalSummaryWritten) {
// didn't succeed, and isn't fooled by writes to *other* artifacts. skipped
// entirely on result.success===false: the error message just written above
// is the user's only signal that the run happened — deleting it would
// restore the same empty-void UX this commit fixes.
if (
toolContext &&
result.success &&
toolState.progressComment &&
!toolState.finalSummaryWritten
) {
await deleteProgressComment(toolContext).catch((error) => {
log.debug(`stranded progress comment cleanup failed: ${error}`);
});
}
await writeJobSummary(toolState);
// best-effort: failures writing the actions step summary must not throw
// past this point. on the result.success===false branch above we already
// wrote `result.error` to the progress comment, and a throw here would
// jump to the outer catch which calls reportErrorToComment again with
// the (less actionable) writeJobSummary error — silently overwriting the
// gate's failure message in the progress comment. the step-summary write
// is informational; let it fail silently rather than corrupt user-facing
// output.
try {
await writeJobSummary(toolState);
} catch (error) {
log.debug(`job summary write failed: ${error}`);
}
// emit structured output marker for test validation
if (toolState.output) {