8c6cd2bda2d6be66367d7d4f93b00c3efbefc214
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colin McDonnell
|
a4a5010441 |
gemini-3: default thinkingLevel to medium + restrict eager prep to frozen install (#663)
* gemini-3: default thinkingLevel to medium + don't `npm ci` without a lockfile upstream opencode hardcodes `thinkingLevel: "high"` for every gemini-3 model on the direct google SDK (see `packages/opencode/src/provider/transform.ts` `options()`). that added 30-60s of pre-tool-call TTFT and 5-46s of post-tool jabber per turn, which is overkill for the tool-routing decisions that dominate agentic loops — and the variance caused the `providers-live (google/gemini-pro)` smoke job to time out at 4 minutes (see job 75405504847 on run 25684766415). three changes: - inject `provider.google.models.<api-id>.options.thinkingConfig.thinkingLevel = "medium"` for the two curated gemini-3 slugs in `buildSecurityConfig`. deep-merges over the upstream default; explicit `--variant high` / user opencode config still wins. flash stays at medium too — low-effort flash is visibly worse and the latency win isn't meaningful (flash is already fast). - bump the `providers-live` harness step from 4 → 6 minutes. the job-level 8-minute cap stays as the upper bound, but gemini's intrinsic TTFT variance was eating most of the 4-minute slack on its own. - in `installNodeDependencies`, pick `frozen` only when a lockfile was actually detected. previously a package.json-only repo (like the smoke fixture's `pullfrog/test-repo`) always triggered `npm ci` and emitted a noisy `EUSAGE` error before falling through. * prep: skip eager install when neither lockfile nor `packageManager` field present the previous commit changed the no-lockfile path from `npm ci` (always errored `EUSAGE`, never wrote any artifact) to a successful `npm install`, which had an unintended side effect: it generated `package-lock.json` in the working tree, tripping the post-run dirty-tree gate. the agent then committed the lockfile and opened a real PR — and in the openai/gpt smoke run on PR #663, the agent overwrote the `SMOKE TEST PASSED` output with the PR URL, failing the smoke validator. a repo with `package.json` but no lockfile and no `packageManager` field has not committed dependency state. eagerly installing produces state the repo doesn't track, which is the dirty-tree problem above. skip the eager install entirely in that case; the agent can opt in via `await_dependency_installation` when it actually needs deps. repos with a lockfile or a `packageManager` field keep the existing frozen-install behavior unchanged. * post-run: suppress dirty-tree gate in non-committing modes (Review / IncrementalReview / Plan) the dirty-tree post-run gate currently fires for every mode and tells the agent to commit and push whatever is in the working tree. that's wrong for modes that complete by submitting a review (`Review` / `IncrementalReview`) or posting a Plan comment (`Plan`) — those modes never touch files as part of their contract, so any tree dirt at end-of-run is incidental tool noise on an ephemeral worktree. nudging the agent to commit it can produce a spurious PR, as seen in the openai/gpt smoke run on PR #663 where a stray `package-lock.json` from `npm install` led the agent to open pullfrog/test-repo#32 and overwrite the smoke output. introduce `NON_COMMITTING_MODES` in `action/modes.ts` and consult it in `collectPostRunIssues`. when the selected mode is read-only, log the suppression for visibility but skip populating `issues.dirtyTree`. modes that legitimately commit (`Build`, `AddressReviews`, `Fix`, `ResolveConflicts`, `Task`) keep the existing nudge. * prep: restore eager frozen-install, drop non-frozen fallback eager dependency prep is non-mutating by contract — it runs before the agent starts and any artifact it leaves in the tree (e.g. a generated `package-lock.json`) trips the dirty-tree post-run gate and can lead the agent to open a spurious PR (seen on the openai/gpt smoke run earlier in this PR). revert the previous skip-when-no-lockfile branch: that was the wrong layer to enforce the invariant. instead, run `frozen` (`npm ci` / `pnpm install --frozen-lockfile` / etc.) unconditionally and drop the `|| install` fallback that could silently mutate the tree when `frozen` is missing. frozen commands fail cleanly without writing artifacts when there's no lockfile, which is exactly the safety contract we want. repos that need a real install must opt in explicitly via a `setup` lifecycle hook. * review nits: single getGitStatus call, tighten gemini-3 override scope comment addresses two inline nits from the PR review: - `collectPostRunIssues` was calling `getGitStatus()` (spawns `git status --porcelain`) in both branches of the mode check. lift the call above the conditional and branch on the result; same behavior, one git invocation. - the JSDoc on `GEMINI_3_DIRECT_API_IDS` said the override applies "across the board," but the constant only covers the two curated slugs in `action/models.ts`. tighten the wording to call out that other gemini-3 ids in models.dev keep the upstream "high" default. skipped the bot's yarn-1 concern after reading yarn 1's `install.js`: `bailout()` (lines 461-465) throws `frozenLockfileError` when `frozenLockfile && (!lockfileClean || missingPatterns.length > 0)`, which fires before `linker.init()` writes node_modules or runs lifecycle scripts. the existing comment's claim that frozen commands fail without artifacts holds for yarn 1 too. |
||
|
Colin McDonnell
|
cf94773bf0 |
modes: make task-list authoring the explicit first step in every mode checklist (#665)
* modes: make task-list authoring the explicit first step in every mode checklist The system prompt already instructs the agent to author an internal task list at the start of every run (action/utils/instructions.ts:291), but the rule lives several hundred tokens above the agent's first decision point and references the mode's checklist before the agent has it. Compliance is roughly coin-flip across opus runs — PR #610 dead-air for 9m20s was the extreme case; my own #664 e2e runs split 1-for-1 on `todowrite` compliance. Putting the directive *inside* the checklist that `select_mode` returns co-locates instruction with referent at the moment the agent decides what to do next. Same vocabulary as the existing rule (`task list`, agent-agnostic; the harness already maps to `todowrite`/`TodoWrite` per-agent in agents/opencode.ts and agents/claude.ts). The directive is deliberately non-prescriptive about list contents — the agent authors items based on the work it's about to do, not from a hand-shaped template. Touches all 8 built-in modes and the PlanEdit override: - Build / AddressReviews / Review / IncrementalReview / Plan / Fix / ResolveConflicts / Task: inserts `1. **task list**: create your task list for this run as your first action.` and renumbers existing steps. - action/mcp/selectMode.ts: same insertion in the PlanEdit override checklist. - All internal step cross-references shifted +1 (`step 5` → `step 6`, `skip steps 3–4` → `skip steps 4–5`, etc.) across Review, IncrementalReview, and ResolveConflicts modes. One code-comment reference in IncrementalReview's preamble updated to match. Complements #664 (live progress streaming): streaming guarantees the user sees *something* regardless of compliance; this PR raises the ceiling on what they see when the agent does comply (clean numbered checklist tracking through the run instead of just the latest assistant message). 488 action tests pass; typecheck, lint, format all clean. * postRun: fix stale 'step 7' reference missed during +1 renumbering |
||
|
Colin McDonnell
|
8e36f76cfa |
postrun: thread AgentRunContext through the retry loop instead of repackaging (#652)
* postrun: thread AgentRunContext through the retry loop instead of repackaging
drop the per-gate plumbing in `runPostRunRetryLoop`: the loop now receives
`ctx: AgentRunContext` whole and reads `ctx.stopScript` + `ctx.toolState.*`
directly. `getUnsubmittedReview` becomes a pure utility in postRun.ts
instead of a closure shipped over `AgentRunContext`. `AgentRunContext`
loses 4 fields that duplicated `toolState` (`summaryFilePath`,
`summarySeed`, `learningsFilePath`, `getUnsubmittedReview`) and gains
`toolState: ToolState`. both harness call sites collapse from 11 lines to
7; main.ts deletes the inline closure.
`ToolState` and friends move from `action/mcp/server.ts` to
`action/toolState.ts` so non-MCP code (agents, post-run loop) stops
importing run-state types from the MCP server module.
no behavior change. 503/503 tests green.
* toolState: relocate `CommentableLines` to break dep cycle with mcp/review
`action/toolState.ts` was importing `CommentableLines` from
`mcp/review.ts`, which pulled the entire MCP server compile graph (24
files) into any consumer of `ToolState` — including `cf-worker-indexing`
via the `pullfrog/internal` re-export chain through `utils/log.ts` →
`agents/shared.ts` → `toolState.ts`. that exposed a pre-existing TS
error in `mcp/issueEvents.ts` (octokit types resolve differently under
cf-worker's `moduleResolution: bundler`).
move `CommentableLines` (a small `{ RIGHT: Set<number>; LEFT: Set<number> }`
state-shape type) to `toolState.ts` where it's used; re-export from
`mcp/review.ts` for back-compat with test and call-site imports. cuts
cf-worker's mcp/ compile inclusion from 24 files back to 0.
* postRun: drop mock-heavy retry-loop tests; keep pure gate predicate
`runPostRunRetryLoop` and `executeStopHook` were covered by ~560 lines
of mock-heavy regression-gate tests that stubbed `spawn` / `getGitStatus`
and fabricated `AgentRunContext` to drive orchestration paths. per
AGENTS.md ("prefer no test over a mock-heavy test that only catches the
most obvious form of regression") and the empirical track record — the
one real production failure of this code path (#646) was a missing npm
release, not a logic bug a unit test could catch — the value-to-ceremony
ratio is poor. delete them.
keep only the pure predicate: `getUnsubmittedReview(toolState)` is a
decision function whose four input conditions have user-visible
consequences when wrong. 5 assertions, no mocks, no ctx fabrication.
488 tests still pass.
* toolState: import PrepResult from prep/types.ts, not the barrel
same dep-cycle class as the previous CommentableLines fix. importing
PrepResult from prep/index.ts pulled prep/installNodeDependencies.ts
into the Next.js production build's typecheck graph (via
pullfrog/internal → utils/log.ts → agents/shared.ts → toolState.ts →
prep/index.ts → installNodeDependencies.ts), and Next.js's stricter
NODE_ENV-required ProcessEnv shape rejected an existing
`env: { PATH: ... }` literal.
prep/types.ts is a leaf module with zero imports — re-routing the type
import severs the chain. Vercel preview deploy goes from Error → Ready;
preview-sync stops racing the deploy.
|
||
|
Colin McDonnell
|
ef394277c1 |
review: synthesize [!NOTE] informational tier with #644 alert judiciousness — 4-callout visual ladder + approved Fix-gate (#653)
* review: NOTE-tier callout + `actionable` flag to suppress Fix buttons
Adds an `actionable` parameter to the `create_pull_request_review` tool
(defaults true) so the agent can opt out of the Fix-it/Fix-all/Fix-👍s
footer affordance on informational reviews. Threaded through
`createAndSubmitWithFooter` so the buttons are omitted when
`actionable: false`.
Updates `Review` and `IncrementalReview` mode prompts with a 4th tier:
`> [!NOTE]` + `actionable: false` for mergeable, FYI-style observations
(prior feedback addressed cleanly, minor stale doc reference, etc.).
Calibration note: `[!IMPORTANT]`/`[!CAUTION]` are reserved for findings
that warrant code changes, because that's what trains users to click
Fix. `[!NOTE]` reviews must not carry inline comments — if a point is
concrete enough to anchor to a line, upgrade the whole review tier.
* review: drop redundant `actionable` flag, key Fix buttons off `approved`
`approved` already encodes "this PR is mergeable, nothing for the Fix
button to act on" — `actionable` was a second flag carrying the same
signal. Drop it from the tool schema and `FooterOpts`; the footer gate
stays `if (!opts.approved)` (unchanged from pre-PR behavior, with a new
comment documenting the UX rationale).
NOTE-tier reviews now use `approved: true` + `> [!NOTE]` body instead of
`approved: false` + `actionable: false`. For repos with
`prApproveEnabled: false`, the runtime already downgrades APPROVE to
COMMENT, so the GitHub-side shape is identical to the prior design.
* review: address Pullfrog feedback — drop ambiguous parenthetical + update postRun nudge
- Review-mode calibration: drop the "(or no callout at all)" parenthetical
that didn't map cleanly to a bullet; replace with explicit "both the
`[!NOTE]` tier and the 'no actionable issues' tier below use approved:
true" so the bullet-list anchor is obvious.
- `buildUnsubmittedReviewPrompt` (Review mode): the fallback nudge for
unsubmitted reviews now defers to the mode prompt's tier matrix and
acknowledges that `> [!NOTE]` informational reviews submit with
`approved: true` alongside the canonical "No new issues found." path.
Previously the nudge only described the pre-NOTE binary world.
|
||
|
Colin McDonnell
|
85d25a6fe6 |
post-run gate: fail review-mode runs that don't submit a review or progress (#638)
* post-run gate: fail the run when review mode finishes without a review or progress
review-mode runs that ended in a text-only assistant turn ("now I have enough
to draft the review...") were silently swallowed: the progress comment was
deleted by stranded-comment cleanup and no review appeared on the PR. user-
visible result was identical to "the agent never ran." caught in
https://github.com/pullfrog/app/actions/runs/25583698781.
new post-run gate alongside stopHook / dirtyTree / summaryStale: derived
inline from toolState (selectedMode in {Review, IncrementalReview} && !review
&& !finalSummaryWritten && hadProgressComment) — no parallel toolState flag.
when it fires, the resume prompt nudges the agent to call either
create_pull_request_review or report_progress; persistent failure after
MAX_POST_RUN_RETRIES surfaces as AgentResult.error.
also: when the post-run loop returns success=false, write the error to the
progress comment before the stranded-comment cleanup runs, and skip the
delete in that case. previously a !success run from the loop would lose the
error message into the void.
IncrementalReview's trivial-skip branch now calls report_progress with a
brief "no review warranted" note instead of exiting silently — keeps the
contract symmetric with the gate and gives the user a visible signal even
on no-op review runs.
documents the literal-record design rule on the ToolState interface so
future fields don't drift back into derived/absence-encoding state.
* review feedback: mode-aware nudge, gate-error preservation, prompt order
addresses three findings from the auto-review on this PR:
1. Review mode nudge no longer offers `report_progress` as an exit. Review
mode's contract (modes.ts step 5) forbids it; the gate previously sent
contradictory copy. IncrementalReview's nudge still offers both since
its trivial-skip path legitimately allows `report_progress`.
2. `writeJobSummary` is now wrapped in try/catch on the success-path
cleanup. without this, a throw there jumped to the outer catch and
overwrote the gate's failure message in the progress comment with the
(less actionable) writeJobSummary error — restoring exactly the
invisible-failure UX this PR fixes. step-summary writes are
informational; let them fail silently.
3. `buildPostRunPrompt` reorders gates to match the terminal hard-fail
order: `stopHook` → `unsubmittedReview` → `dirtyTree` → `summaryStale`.
when both hard-fail gates co-fire (rare in review modes), the prompt's
emphasis now matches the user-visible failure message.
new test asserts the IncrementalReview nudge offers both exits while the
Review nudge offers only `create_pull_request_review`. e2e validation
already passed against pullfrog/preview-638-review-stop-hook PR #1
(gate fired once; agent recovered on second turn).
* mode-aware terminal error copy
second auto-review caught a residual contradiction: the terminal hard-fail
error string reported "create_pull_request_review or report_progress" for
both modes, even though the new mode-aware nudge tells Review-mode agents
"Review mode does not have a no-submit exit". the error message now mirrors
the nudge — Review names only `create_pull_request_review`,
IncrementalReview lists both. additional Review-mode hard-fail test asserts
the absence of `report_progress` in the error.
|
||
|
Colin McDonnell
|
d6de1c369a |
learnings: edit-in-place tmpfile (drop update_learnings tool) (#635)
* learnings: edit-in-place tmpfile (drop update_learnings tool)
learnings now follow the PR-summary file pattern: server seeds
`pullfrog-learnings.md` from `Repo.learnings` at startup, agent reads
it as part of context, may edit in place during the post-run reflection
turn, server reads back at end-of-run and PATCHes if changed.
motivation: `update_learnings` required the agent to pass the FULL
merged list as a string parameter — an output-token tax that grew
linearly with the learnings size, and a constant prompt-context
expansion since the contents were also inlined into the LEARNINGS
section. for repos with mature learnings the prompt was getting
visibly noisy in CI logs.
key changes:
- new `action/utils/learnings.ts` (seed/read helpers + 10k cap)
- `main.ts`: always seed; `persistLearnings` mirrors `persistSummary`
(success path, error path, exit-signal handler, idempotent guard,
byte-trim equality skip); forwards `model` for `LearningsRevision.model`
- `LEARNINGS` prompt section now contains only the file path + a
one-line "read it" instruction (no contents inlined)
- `update_learnings` MCP tool deleted; `action/mcp/learnings.ts` removed
- reflection turn (`buildLearningsReflectionPrompt`) reframed around
file editing with explicit prune-stale + leave-alone-if-nothing-new
framing
- `learningsStep` removed from every mode checklist — surface lives only
in the LEARNINGS prompt section + the reflection turn now
* learnings: harden seed step + refresh stale docs (review feedback)
Three findings from PR review, all implemented:
1. wrap learnings seed in best-effort try/catch (action/main.ts) —
the always-on seed block ran unconditionally and an unwrapped
`seedLearningsFile` (mkdir + writeFile) failure (ENOSPC, EACCES,
hostile sandbox) would unwind into the outer main() catch and flip
an otherwise-successful run to "❌ Pullfrog failed" before the
agent even started. asymmetric with `persistLearnings`'s own
best-effort contract. wrap and log on failure; downstream
consumers (`persistLearnings`, agent harnesses, `resolveInstructions`)
already handle `learningsFilePath: undefined` cleanly.
2. refresh wiki/main.md — `resolveInstructions` parameter renamed
from `learnings` to `learningsFilePath` in this PR; the data-flow
diagram and the resolver dependency table both still showed the
pre-refactor signature.
3. drop deleted `learnings.ts` from ROADMAP.md + RESEARCH.md
"missing MCP tool tests" bullets — `action/mcp/learnings.ts` was
removed in this PR; the bullets are otherwise still accurate.
|
||
|
Colin McDonnell
|
e4e93ea6d3 |
PR summary as agent-edited tmpfile snapshot (#568)
* PR summary as agent-edited tmpfile snapshot Replaces the comment-based PR summary path (and the in-progress update_pr_summary tool from #534) with a snapshot file the agent edits in place during Review / IncrementalReview / pr-summary Task runs. The server seeds the tmpfile with the previous snapshot (incremental) or a stable scaffold (first run), exposes the path via select_mode, and reads it back at end-of-run to persist to WorkflowRun.summarySnapshot and (when the prSummaryComment toggle is on) splice into the PR description body. Why a tmpfile rather than a tool call: incremental snapshot edits are output-token-cheap when the agent uses native file-editing tools, and range-diff cleanly across runs because section headings are stable. The agent never has to regurgitate the full snapshot to update it. Gating: snapshot generation is opt-in via either prSummaryComment="enabled" (splice into PR body) or prReReview="enabled" (snapshot feeds future incremental review runs as context). Users who disable both pay nothing end-to-end — no seeding, DB write, or body splice. Behavior changes: - Drop the Summarize mode and the Summary comment type entirely; the rolling summary is no longer a separate run shape. - pull_request_synchronize with re-review off and summary on still dispatches a silent pr-summary Task, but it edits the snapshot file instead of posting a fresh comment. - /api/repo/.../pr/.../summary-comment now returns { snapshot: string | null } from the DB instead of fetching a comment via GraphQL. URL kept stable so deployed older actions degrade gracefully. - summaryCommentNodeId is retained on WorkflowRun for legacy data and a future backfill of pre-snapshot comment-based summaries. Supersedes #534. The commit-tool/sub-agent direction in that PR is abandoned in favor of this file-based shape. * address review pass #1: synchronize fallback, splice idempotency, docs * address review pass #2: in-flight skip should not race summary fallback * address review pass #3: signal-handler flush, doc clarifications * address review pass #4: in-flight persist promise + bounded body-splice timeout * address review pass #5: defensive catch on persist worker, doc nit * add summary-stale post-run gate When generateSummary is set, we capture the bytes of the seeded snapshot file and pass them to the agent's post-run loop alongside the file path. After each agent attempt, the loop diffs the current file against the seed; if they're byte-identical the agent never touched it, and we nudge once via a resume turn (similar to the dirty-tree gate, but soft and fire-once so smaller models that legitimately decide no edit is warranted don't burn the retry budget). Mostly defends against forgetful smaller models on the Review path — their mode prompt asks them to edit the snapshot file, but the multi-step instruction can fall through when the diff is large. * trigger: retry vercel preview build * fix(action): drop unused re-export that pulled node:fs/promises into next bundle action/internal/index.ts was re-exporting DEFAULT_PR_SUMMARY_INSTRUCTIONS from action/utils/prSummary.ts, but nothing in the next.js app imports it. prSummary.ts uses node:fs/promises, and pullfrog/internal is aliased into the next bundle by next.config.ts, which made turbopack try to resolve node:fs/promises in client chunks and fail with: the chunking context (unknown) does not support external modules (request: node:fs/promises) drop the re-export — selectMode.ts (the only real consumer) already imports it directly from action/utils/prSummary.ts. * firewall PR summary snapshot from user instructions; resurrect rich format for Review The agent-internal snapshot (the markdown file the agent edits in place across runs) is exclusively durable context for future agent runs — user-supplied summarization instructions warp it and degrade that context. Drop the prSummaryCommentInstructions read path end-to-end: - handleWebhook: stop reading prSummaryCommentInstructions, stop passing prSummaryInstructions through dispatch options - action payload + ToolState + selectMode addendum: drop the instructions appendix; the snapshot prompt is fixed, not user-shaped - TriggersSettings: drop the InstructionsEditor for prSummaryCommentInstructions - prSummary.ts: reframe DEFAULT_PR_SUMMARY_INSTRUCTIONS as agent-targeted (durable context, not human-facing prose) Prisma columns (prSummaryComment, prSummaryCommentInstructions) and the matching zod schema entry stay for graceful retreat. Separately, resurrect PR_SUMMARY_FORMAT (deleted along with the Summarize mode in the original PR) and wire it into Review mode only. Initial PR reviews now include a structured summary section in the review body using the rich format (TL;DR, key changes, ## sections with before/after, file-link trails). IncrementalReview keeps its existing terser bullet-list shape since re-review bodies are deltas, not introductions. The user-facing review summary and the agent-internal snapshot are deliberately separate artifacts with separate prompts and zero shared content. * address review comments: prompt self-consistency + stale-doc cleanup PR 568 self-review (4232488109) flagged a self-contradiction the firewall commit introduced and three stale doc references that survived. - action/modes.ts: Review-mode step 2's trivial-PR shortcut said `submit "Reviewed — no issues found." per step 5`, but step 5's rewrite removed exactly that preamble. Aligned both: trivial PRs and no-actionable-issues PRs now produce a body that opens with "No new issues found." followed by the PR summary, so the user gets the headline up front and still sees what was reviewed. - docs/pr-reviews.mdx: dropped the "customize the summary style with Summary instructions in the console" sentence (the editor was removed in the firewall commit). Replaced with a note that the snapshot uses Pullfrog's built-in format and is not user-customizable. - wiki/prompt.md, wiki/modes.md: rewrote the snapshot-prompt entries to reflect the firewall — DEFAULT_PR_SUMMARY_INSTRUCTIONS is the entire prompt, prSummaryCommentInstructions is no longer wired in. * drop orphaned prSummaryCommentInstructions column Prod audit (455 repos): 5 non-null rows on a single account, all containing the literal placeholder text from the InstructionsEditor we removed in the firewall commit. No account has an intentional preference set, so silent-ignore (the keep-for-retreat option) costs us nothing meaningful while leaving an orphan column in the schema. Drop it. - prisma/schema.prisma: remove the column - prisma/migrations/20260506000000_drop_pr_summary_comment_instructions: ALTER TABLE ... DROP COLUMN - utils/schemas/triggers.ts: drop the matching zod entry * drop body splicing; snapshot is internal-only User-visible PR summarization continues to ship in Review and IncrementalReview review bodies (which already render PR_SUMMARY_FORMAT and "Reviewed changes" respectively). The snapshot tmpfile is now purely durable cross-run agent context — seed, edit-in-place, save to DB, feed the next run. Massive simplification: the body splice mechanics, the two-toggle gating matrix, the summaryHandlingCovered race tracking, and the synchronize summary-only Task fallback all go away. Code: - prSummary.ts: drop splice/strip/marker code (`splicePrSummary`, `stripExistingSummaryBlock`, `buildSummaryBlock`, `extractPrSummary`, PULLFROG_SUMMARY_START/END). keep scaffold, instructions, seed/read. - main.ts: rename persistAndPostSummary -> persistSummary; collapse to a single DB PATCH. drop pulls.get/pulls.update, drop AbortSignal timeout, drop in-flight promise machinery, drop prSummaryToBody plumbing. - ToolState: add summarySeed (replaces local var in main.ts so persist can compare). drop prSummaryToBody and summaryPersistInFlight. - persistSummary now compares against the seed and skips the DB write with a warning when unchanged — saving the seed verbatim is either a no-op or persists the placeholder scaffold, neither useful. - postRun.ts: when summary-stale is the only failing gate and the resume turn itself fails, restore the pre-resume successful result and break. symmetric with the existing reflection-failure preservation. summary-stale can no longer flip a successful run to failed. Webhook: - pull_request_opened: generateSummary follows prReReview only (the snapshot has no consumer when re-review is off). - pull_request_synchronize: collapses to "if prReReview enabled, dispatch IncrementalReview". the summaryHandlingCovered flag, the same-SHA/in-flight coordination it was protecting, and the summary-only Task fallback all delete cleanly. UI / config: - drop SummarizePRsTrigger (the toggle gated body splice; with that gone it has no behavior). drop sidebar entry, console import, Text icon import. - drop prSummaryComment from triggers zod schema, prisma schema, preview settings script. Migration: squash the two existing migrations into one timestamped 20260507000000_pr_summary_snapshot covering all three column changes (add summarySnapshot on workflow_runs, drop prSummaryCommentInstructions and prSummaryComment on repos). repo convention is one migration per PR. Action: bump 0.0.203 -> 0.0.205 (payload contract changed: prSummaryToBody removed; main is at 0.0.204). Out-of-diff cleanup: - review.ts:190 + review.test.ts:651 — "Reviewed — no issues found." -> "No new issues found." to match the canonical body in modes.ts. Verified: pnpm typecheck clean, pnpm lint clean, postRun + review tests pass, dev DB reset against production and the squashed migration applied cleanly (summarySnapshot present, prSummaryComment / prSummaryCommentInstructions both gone). * re-orient snapshot toward functional summary; drop prior-review-feedback section Empirical audit on preview-568 PR #5 showed the snapshot IS load-bearing for the orchestrator: lens-dispatch prompts on incremental runs carried forward context from the snapshot's risk register (e.g. "the JSDoc explicitly scopes to code points — do not flag grapheme-cluster issues" on the surrogate-pair fix run, "consistency with native padStart" on the padStart-added run). The orchestrator was reading the snapshot, reasoning about it, and using it to anti-prime / focus subagents — exactly the high-leverage path. My earlier "snapshot is write-only" claim was wrong. The shape, however, was steering it toward review-history-log instead of functional summary. This commit re-orients: - prSummary.ts: replace the four-section scaffold (~580 chars of placeholder italics under "What this PR does / Key changes / Risk / Reviewed in prior runs") with a minimal seed (~150 chars: just a header + a one-line comment about what the file is for). different PRs warrant different organization; forcing a refactor and a feature into the same template is procrustean. minimal seed also makes the unchanged-from-seed gate in persistSummary more sensitive. - selectMode.ts addendum: rewrite around three principles. (1) the snapshot is a FUNCTIONAL summary of what the PR does and the risks it carries, not a chronological review log — commit history can already be reconstructed from list_pull_request_reviews. (2) the orchestrator should USE the snapshot during triage and dispatch — concrete example given of carrying snapshot context into subagent lens prompts. (3) structure is the agent's call; stable headings make snapshots range-diff cleanly when they fit, but riff when they don't. - modes.ts IncrementalReview: drop the "Prior review feedback" checklist from the user-facing review body (step 6b gone, step 7 ELSE IFs cleaned up). It duplicated content that's already covered by the Reviewed-changes bullets and tracked durably in the snapshot for the next agent run; in the user-facing body it was noise. step 3 still fetches prior reviews but its role is now just filtering aggregation in step 5, not rendering. - AGENTS.md: codify "no follow-ups" rule. when an issue is identified during code review, fix it in this PR — PR scope does not constrain quality. follow-up TODOs are forbidden as a substitute for doing the work now. Empirical evidence supporting the re-orientation: - Run 25568912293 (PR#5 incr1, surrogate-pair fix): orchestrator's correctness lens dispatch said "Do NOT flag grapheme-cluster issues — the JSDoc scopes to code points." The grapheme-cluster framing was not in the diff; it was downstream of the snapshot's prior risk-section framing of truncate's contract. Snapshot influencing dispatch. - Run 25569054779 (PR#5 incr2, padStart added): orchestrator's correctness lens dispatch enumerated edge cases including "consistency with native String.prototype.padStart contract" and "fill = multi-code-point string (e.g. emoji)". Both threads carried over from the snapshot's prior truncate code-point-vs-code-unit discussion. Snapshot informing the shape of what was looked for. The cost of maintaining the snapshot (~800 tokens, ~$0.005/run) is trivially affordable when it materially improves orchestrator triage on the 1-5 lenses dispatched per review. |
||
|
David Blass
|
c6a757424c |
Stop hook + learnings reflection via post-run loop (#515) (#548)
* add stop hook + learnings reflection to post-run loop (#515) stop hook (#515): repo-configured script that runs after the agent finishes. non-zero exit resumes the agent with the hook output as guidance; persistent failure (3 attempts) marks the run failed. the dirty-tree and stop-hook gates share a single retry loop so a fix + push happen in one turn. learnings reflection: per Colin, the learnings step baked into mode checklists rarely fires — the agent stays focused on the task and the meta-ask falls through. the post-run loop now delivers a dedicated one-shot --continue turn asking the agent to call update_learnings if relevant, nothing else competing for attention. reflection doesn't consume the gate-retry budget; if it dirties the tree, the next loop iteration catches it via the dirty-tree gate. plumbing: Repo.stopScript column + migration, zod schema, run-context api, AgentSettings UI. RepoSettings.stopScript threads through to AgentRunContext and into each agent harness. subprocess-dependent logic lives in action/agents/postRun.ts to keep action/agents/shared.ts lean — shared.ts is reachable from pullfrog/internal, and pulling node:child_process through it leaks into root tsc (which uses bundler resolution, not NodeNext). * fix: preserve successful run when reflection turn fails The post-run reflection turn (update_learnings nudge) is a best-effort one-shot; its failure must not flip a successful run to failed. Prior code overwrote `result` with the reflection's return value, so a model API error during reflection caused the whole run to be reported as failed even though the gated work had already completed cleanly. Now: save the pre-reflection result, and if reflection returns `success: false`, log a warning, restore the prior success, and exit without re-invoking the gates (re-running a freshly-green stop hook risks a flaky false-positive failure). Adds action/agents/postRun.test.ts covering the reflection path — previously uncovered. * fix: surface both stop-hook stdout and stderr to the agent The `(stderr || stdout)` heuristic in executeStopHook dropped stdout entirely whenever stderr had any content. Scripts that emit a benign warning to stderr and the actionable error to stdout (common for wrapper scripts) starved the agent of the information it needed to fix the issue. Now concatenate both streams (stderr first, stdout second, skipping empty ones) before truncation. This keeps stdout's tail — usually where summaries and totals live — intact under the 4096-char cap. * test: lock in the core post-run retry + reflection invariants PR #548's test plan ships four manual verification scenarios. Convert three to vitest coverage, catching regressions on the hottest code paths: - persistent stop hook failure exhausts MAX_POST_RUN_RETRIES and surfaces as AgentResult.error with both the retry count and the verbatim hook output (so the GitHub-comment rendering stays actionable). - every gate retry is fed the hook output as the resume prompt. - usage aggregates across the initial run plus every retry (billing relies on this). - reflection turn still fires when no stop hook is configured and the tree is clean. Manual item remaining is the full UI round-trip of the settings form, which is out of scope for unit tests. * test: cover executeStopHook soft-fail and truncation invariants Three paths the PR documents but previously had no regression gates: - timeout (SPAWN_TIMEOUT_CODE) and activity-timeout (SPAWN_ACTIVITY_TIMEOUT_CODE) must return null, not a failure. a hook that times out is an infra problem; retrying with an agent turn risks an infinite loop. - spawn errors (ENOENT from a typoed binary, etc.) take the same soft-fail path for the same reason. - oversize hook output is truncated to the last 4096 chars with a "truncated" marker, keeping the tail (where summaries live) and protecting the 65535-char GitHub-comment budget downstream. Regression targets — a refactor that accidentally surfaces an infra failure as a gate failure, or blows the comment budget, will now fail loudly in CI. * test: cover soft-fail, no-resume, and short-circuit invariants Three more documented behaviors that previously had no regression gates: - dirty-tree-only is a soft-fail: persistent uncommitted changes log and warn but DO NOT flip the run to failed. a regression that started surfacing this as AgentResult.error would break every run that leaves a test fixture untracked. - canResume=false + stop hook failure still surfaces the hook failure as AgentResult.error. the retry budget is zero so "N retry attempts" is correctly omitted from the message, but the run still reports WHY it failed rather than silently reporting success. - initial result with success=false short-circuits the loop: no gate checks, no reflection, no resume calls. the original agent error flows through verbatim for clean triage. Also reset mockedSpawn in beforeEach so test state doesn't leak between cases. * test: lock in the reflection-dirties-tree → dirty-tree-gate path The PR description claims: "if the reflection turn dirties the tree, the loop picks that up on the next iteration via the normal dirty-tree gate." There was no regression gate on this invariant. Without it, a refactor that moved the reflection out of the retry loop (e.g., into a one-shot post-loop call) would silently bypass the commit-before-you-finish contract whenever the agent misbehaves during reflection — uncommitted changes would ship as part of the run's "success" state. The test sequences three getGitStatus returns (clean → dirty → clean) and asserts two resume calls: REFLECTION first, then UNCOMMITTED CHANGES with the dirtying file in the prompt. * fix: preserve pre-reflection task output when reflection succeeds the reflection turn's reply ("done" or "updated learnings with N bullets") is a meta-ask, not a task summary. before this fix, result = reflectionResult clobbered the original task's output on the returned AgentResult, so downstream consumers (handleAgentResult's fallback path when toolState is empty, programmatic callers of main()) saw the reflection's trivial reply instead of the real summary. spread reflectionResult to inherit fields subsequent gate retries need (e.g. the new sessionId claude emits per --resume invocation), but keep the pre-reflection output verbatim. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: fall back to reflection's output when pre-reflection output is empty the prior fix used `??` which only fell through on null/undefined. runs that communicate exclusively through MCP tools (e.g. report_progress) and emit no plain text leave result.output = "", which `??` preserved as-is — dropping the reflection's reply and leaving handleAgentResult's fallback path with nothing to show. switch to `||` so empty-string pre-reflection output yields the reflection's output instead of ""; non-empty task output still wins as intended. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: drop reflection-failure-skips-hook test (over-specified control flow) the test pinned the literal `break` in the post-reflection failure branch with stopScript=null, asserting only that getGitStatus was called once. that's not a behavior contract — a reasonable refactor (e.g. `continue` to re-check gates with explicit flake guards) would fail this test even though the new behavior would be fine. the "does not flip a successful run to failed" test already covers the only thing callers depend on. * test: drop low-value mock-driven tests from postRun - "fires the reflection turn when no stop hook is configured" — fully subsumed by the output-preservation test (asserts task output survives, which is only possible if reflection fired). - "uses stdout alone" / "uses stderr alone" — pin format trivia (`filter(Boolean).join`) that LLMs ignore. - "returns empty output (not undefined) when both streams are empty" — guards a TS-impossible case; every consumer uses `output || "(no output)"`. - "returns null on activity-timeout" — duplicate of the timeout test; same `return null` branch with a different constant. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |