Compare commits

..

63 Commits

Author SHA1 Message Date
Colin McDonnell 5abb3072c7 release: action v0.1.8 2026-05-14 05:37:33 +00:00
Colin McDonnell 74b7329f64 fix(action): dedupe concurrent checkout_pr + guard cross-PR clobber (#735)
* fix(action): dedupe concurrent checkout_pr calls + guard cross-PR clobber (#642)

agents occasionally emit duplicate parallel `checkout_pr` tool_use blocks
in one turn, causing two `checkoutPrBranch` invocations to race the same
`.git/shallow.lock` and one to fail with `File exists`. the prior fix
(#564) added a 30s staleness sweep, but that very threshold protects the
within-run concurrent case from itself.

dedupe at the tool layer: a module-level `Map<pull_number, Promise>`
shares a single in-flight promise across concurrent same-PR calls. the
fetch race becomes architecturally impossible — first call does the work,
duplicate gets the same `CheckoutPrResult`. cleared in `finally` so
subsequent same-PR calls re-do the work normally.

also reject cross-PR checkouts when the working tree is dirty, surfacing
a clear error instead of silently overwriting uncommitted work from a
prior PR. uses existing `toolState.issueNumber` (no new state).

* review: use dedicated `pullNumber` toolState field for cross-PR guard

per copilot review: the prior guard used `toolState.issueNumber`, which
is also set by issue/comment lookup tools (issueInfo, issueComments,
issueEvents, review). that conflation is intentional and correct for
its only consumer (`report_progress` falls back to `issueNumber` to
choose which issue/PR to comment on, and GitHub treats both via the
same comment API). but it makes the field wrong for the cross-PR
guard: a same-PR re-checkout after `get_issue(other)` would falsely
fire and surface a misleading "from PR #other" message.

introduce a separate `pullNumber` field, set only by `checkoutPrBranch`
alongside `issueNumber` and `checkoutSha`. narrower invariant, no
disturbance to the existing `issueNumber` semantics.

* review: drop dual-write — single `issueNumber` is sufficient for the guard

reverting the `pullNumber` addition. setting both `issueNumber` and
`pullNumber` to the same value at the same site was a code smell — there
is no scenario where they diverge. issues and PRs share GitHub's number
space, and the cross-PR guard's actual job is "refuse to clobber a dirty
tree when switching to a different number"; that's expressible with
`issueNumber` alone.

addresses copilot's original concern (misleading "from PR #X" message
when X was an issue) by removing the prior-number reference from the
error message entirely. the dirty paths are the actionable detail.
2026-05-14 05:08:11 +00:00
Colin McDonnell ba7f5a0b89 action: surface agent hang context in progress comment (#733)
* action: surface agent hang context in progress comment

When the activity-timeout watchdog kills a stalled opencode subprocess,
the user used to see a bare "activity timeout: no output for 30Xs" — no
provider context, no stderr trace, no clue why the run died. Investigation
of the six runs in #728 showed the same shape every time: opencode hangs
after a non-retryable provider event (auth 401, 502 stream lost, free-tier
flake), and the only useful signal was buried in stderr where the user
couldn't see it without diving into Actions logs.

Stop trying to prevent the hang. Surface it.

Add a small `AgentDiagnostic` handle on `toolState` that the harness
mutates as a run progresses (recent stderr ring buffer reference, last
provider-error label, event count). `formatAgentHangBody` renders that
into a markdown body — bold headline, one-line explanation, collapsible
`<details>` with the last ~10 stderr lines (capped to 3KB) — used by
both the agent harness's own catch path and main.ts's outer catch when
the watchdog wins the race against the harness.

Both paths converge on one formatter; the existing
"View workflow run ➔" footer affordance in `reportErrorToComment` is
unchanged, so the user still has one click from the comment to the raw
logs to develop their own thesis.

* address review: gate hang body on isHang; fix contradictory copy

- Only render `hangBody` when `isHang`. The harness sets
  `agentDiagnostic` on entry, so any non-hang throw past `runOpenCode`'s
  own catch (post-success `output_schema` validator, late cleanup throws)
  was rendering "Pullfrog failed — N events processed…" with the real
  exception message dropped — including for runs that actually succeeded
  before a late throw.

- When `lastProviderError` already names the cause in the headline, the
  zero-events sentence "check whether the model provider is reachable"
  contradicts it (a 401 produces zero events but isn't a reachability
  issue). Drop the nudge in that case; keep it for the silent-stall path
  where it's still actionable.

* address copilot review: fence escape, idle parsing, secret redaction, tests

- pick a backtick fence longer than any backtick run in the rendered
  stderr tail. opencode error JSON occasionally embeds triple backticks
  in tool input dumps; the fixed three-tick fence let those terminate
  the fence early and corrupt the rest of the comment markdown.

- parse idle seconds out of the timer reject string ("activity timeout:
  no output for 301s") and use that for the hang explanation. previously
  rendered total runtime, which overstated the stall by 20+ minutes for
  runs that streamed for a long time before going quiet (e.g.
  Rohithgilla12/data-peek#25784038918, 1230s elapsed but 304s idle).

- redact sensitive env-var values from the rendered stderr tail before
  it lands in the PR comment / job summary. workflow log writes already
  go through `core.setSecret` masking; PR comments and summaries bypass
  that pipeline entirely. matches against `isSensitiveEnvName` (the same
  *_KEY/*_TOKEN/*_SECRET/*_PASSWORD/*_CREDENTIAL surface that
  `normalizeEnv` registers with the runner) and only redacts values
  >= 8 chars to avoid false-positive substring hits.

- add `agentHangReport.test.ts` covering the branchy bits: idle-seconds
  parsing, eventCount-zero copy with and without provider error,
  fence-escape against embedded triple backticks, 3 KB tail truncation,
  null-on-no-diagnostic, and secret redaction.

`startedAtMs` is dropped from `AgentDiagnostic` — total runtime was the
only consumer and idle seconds replaces it.

* strip slop: drop tests, drop redactSecrets, simplify ternary

- delete `agentHangReport.test.ts`. half the cases just pinned literal
  copy ("**Pullfrog stalled**", "check whether the model provider is
  reachable") which is exactly the "performative tests to every string
  utility" pattern AGENTS.md flags. the other half tested 2-5 line pure
  helpers (parseIdleSec / pickFence / truncation) that code review
  catches. the formatter is a best-effort string output; pinning it in
  tests creates churn without catching real regressions.

- remove `redactSecrets` and revert the formatter's import. theatrical
  defense: opencode doesn't dump env on startup, bearer tokens aren't
  in request bodies, bash is denied. the action has many other
  PR-comment write paths that don't redact (comment.ts, errorReport.ts,
  the progress writer) — if PR-comment secret hygiene matters, it's a
  cross-cutting concern at the comment-write layer, not bolted onto
  one formatter.

- factor the explanation triple-ternary into `formatExplanation` with
  early returns. same logic, easier to read.

`isHang` gate, fence-length escaping, and idle-seconds parsing stay —
those are real correctness fixes.
2026-05-14 04:13:26 +00:00
Colin McDonnell b9383bbcfd action: center provider-error log excerpt on the matched line (closes #703)
the `» provider error detected (...)` excerpt was `chunk.substring(0, 500)`
— the head of whatever stderr buffer node delivered. on big writes that's
the front of an mcp tool-schema dump, not the matched error text. label
was correct (regex.test on the whole chunk), excerpt was misleading.

introduce findProviderErrorMatch(text) that returns { label, excerpt }
where excerpt is a windowed slice centered on the regex match index:
the matched line plus 1 line before and 2 lines after, hard-capped at
600 bytes. detectProviderError stays as a thin wrapper for label-only
callers. both opencode and claude harnesses log match.excerpt instead
of chunk.substring(0, 500).

regression tests cover the multi-line buffer case, surrounding-line
context, byte-cap fallback to matched-line-only, and head truncation
of a single oversize line.
2026-05-14 03:59:45 +00:00
Colin McDonnell 8d6460da1c fix: surface real tool error string in opencode log handler (#736)
opencode's `ToolStateError` carries the failure reason on `state.error`,
not `state.output`. our log handler was reading `state.output` and
falling back to `(no error message)`, so every tool failure logged a
useless line. type the state as a discriminated union (mirrors
@opencode-ai/sdk) so the field misread becomes a compile error.

operator-facing only: the model already received the real error via
opencode's tool-result envelope (verified by running webfetch against
a known-404 URL — model reported "Error: Request failed with status
code: 404" verbatim).

closes #662
2026-05-14 03:56:24 +00:00
Colin McDonnell 1f4c3031be ci: filter test matrices by per-test coverage globs (#730)
* ci: filter test matrices by per-test coverage globs to cut LLM spend

every test in `crossagent/`, `agnostic/`, and every provider entry now
declares a `coverage: string[]` of repo-relative globs. the new `changes`
job runs `paths-filter` for a docs-only short-circuit, then pipes the
changed-file list into `action/test/matrix.ts`, which intersects each
entry's coverage against the diff and emits filtered `agents`,
`agnostic`, `flagships`, and `aliases` matrices. main pushes and
`workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety
net.

retires `changed-agents.sh` and the `MODE=flagships` branch in
`list-aliases.ts` in favor of one consistent model.

* ci(matrix): switch test discovery to dep-free static parsing

the GHA `changes` job has no `node_modules` installed. the previous
dynamic-import path pulled the test files transitively through
`utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with
ERR_MODULE_NOT_FOUND. parse the test files via regex instead so
matrix.ts stays zero-dep — the chain (matrix -> coverage / providers /
list-aliases / models) imports only node builtins and relative TS files.

* ci(matrix): address PR #730 review feedback

- drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`,
  `restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but
  the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`,
  both already covered).
- drop unused `coverageForProvider` export and its `byName` map from
  `providers.ts` (matrix.ts builds its own lookup inline).
- derive the active agent list from `agents/index.ts` via the same
  dep-free regex tactic as `parseTestFile` instead of hardcoding
  `["claude", "opencode"]` — adding a new harness file now wires it
  into the dynamic matrix automatically.
- treat `coverage: []` as `coverage: undefined` in `shouldRun` so an
  accidentally-empty array doesn't silently skip CI on every PR.
- add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the
  `timeout` test's coverage — the activity-timeout enforcement path
  was the original reason the test exists.
- ungate the `root` job (lint/format/typecheck/vitest). it's a required
  status check on `main`, so gating it on `code == 'true'` would make
  docs-only PRs unmergeable (skipped jobs don't satisfy required-check
  rules). the real LLM savings come from skipping the four matrices,
  not from skipping `root`.
- harden the four matrix-job `if:` guards from `outputs.matrix && ...`
  to `outputs.matrix != '' && ...` — explicit > implicit short-circuit.
- document `expandBraces`'s flat-only support so a future author isn't
  surprised by `{a,{b,c}}` not expanding.
- fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".
2026-05-14 03:55:33 +00:00
Colin McDonnell 4ad649ebb9 action: extend shallow-unreachable deepen-retry to checkout_pr fetches (#734)
extracts the deepen-retry helper from `GitFetchTool` into shared
`$gitFetchWithDeepen` and applies it to every fetch in `checkoutPrBranch`
(baseRef, pull/N/head, before_sha temp branch). on shallow clones with
deep PR ancestry — the failure mode behind ~10 of 51 `heuristic:very-slow`
runs in 24h on `remotion-dev/remotion` — the baseRef fetch was throwing
`Could not read <sha>` to the agent before the compare-api deepen block
could run. agents then burned 10+ minutes retrying `checkout_pr` and
falling back to ad-hoc shell `git fetch --deepen` workarounds.

also splits the analyzer's `heuristic:git-error-recovered` into
`heuristic:git-shallow-unreachable` and `heuristic:git-shallow-lock`
buckets so future audits surface this without manual log-grep.

closes #656.
2026-05-14 03:44:08 +00:00
Colin McDonnell 2960d51493 shell tool: cap output at 5K chars and spill overflow to tempfile (#732)
unbounded shell tool output blows the agent's context window on commands
that dump big logs (test runners, build tools, grep on large trees). cap
the inline body at 5000 chars; on overflow, persist the full output to
${PULLFROG_TEMP_DIR}/shell-<id>.log and return the tail prefixed with a
sentinel pointing at the saved path. agents re-read the tempfile with
cat/tail/grep when they need more.
2026-05-14 03:18:54 +00:00
Colin McDonnell b6df2860c3 action: bump to 0.1.7 2026-05-14 02:48:31 +00:00
Colin McDonnell d495f0b984 surface BYOK failures + chronic-failures card + WorkflowRunStatus mirrors GitHub conclusions (#722)
- Migrates `WorkflowRunStatus` from `running | completed | cancelled` to a 9-state mirror of `workflow_run.conclusion`. Backfill: old `completed → success`, `cancelled → failure`. New rows write `hook.workflow_run.conclusion` verbatim via `statusFromConclusion`.
- Adds Discord links to `formatApiKeyErrorSummary` (both missing-key and 401 invalid-key shapes).
- Repo console: `<ChronicFailuresCard>` fires when the last 3 terminal-state runs are all `failure`. Pure DB read; latest-run button hidden for pre-dispatch failures (`runId: null`).
- `StatusIcon` distinguishes `cancelled` (gray X, intentional stop) from `failure` (red X) so the visual matches the chronic-card threshold.
- Pre-dispatch failures (workflow lookup miss, dispatch API error) write `failure` instead of `cancelled` so they feed the card.
- Cascade: every `status: "completed"` filter in billing routes / cron / cohort queries / analyzer becomes `status: "success"`.

Verified end-to-end on `pullfrog/preview-722-failure-surfaces` — Better Stack logs confirm webhooks reached the preview deploy and all three e2e runs got `marked as failure (conclusion=failure)` via the new mapper.

Closes #679, #702.
2026-05-14 02:39:41 +00:00
Colin McDonnell 206c11fe7c review: drop misleading 'with the same arguments' from diff-coverage nudge
agent is free to refine review body/comments on retry — there's no
enforcement that the second call matches the first, and if reading the
nudged region surfaces a new finding the agent should add it.
2026-05-14 02:36:06 +00:00
Colin McDonnell 7414c1e9ca review: clarify diff-coverage nudge gives explicit license to skip generated artifacts
the one-time pre-flight nudge said "optionally read" but never told the agent
it's free to retry without reading when every unread region is generated
(lockfiles, codegen, snapshots, migration metadata). audit #677 surfaced ~21
runs/24h burning an extra model turn re-reading drizzle snapshots, pnpm-lock,
and *.gen.ts files purely to satisfy the gate. mode prompts only mention
generated content in the "skip self-review entirely" path, not the
"in-progress substantive review" path, so the in-the-moment error message
was the gap. behavior unchanged for legitimately-unread source regions.
2026-05-14 02:30:37 +00:00
Colin McDonnell 8f9208bd3f feat: Amazon Bedrock support via routing slug (#720)
* add Amazon Bedrock as a routing slug

introduces a single `bedrock/byok` catalog entry that the harness translates
to the appropriate Bedrock model ID at run time via `BEDROCK_MODEL_ID`. routes
Anthropic IDs through claude-code (with `CLAUDE_CODE_USE_BEDROCK=1`) and
everything else through opencode's `amazon-bedrock` provider — keeps the
catalog flat for an audience that needs version pinning rather than aliasing.

accepts either `AWS_BEARER_TOKEN_BEDROCK` or `AWS_ACCESS_KEY_ID` +
`AWS_SECRET_ACCESS_KEY` for auth; both validated alongside `AWS_REGION` and
`BEDROCK_MODEL_ID` in `validateAgentApiKey`. catalog drift tests, the bumps
cron, and per-alias smoke scripts all skip routing slugs since there's no
fixed `resolve` to validate.

docs/bedrock.mdx walks through setup; wiki/model-resolution.md has a section
explaining why bedrock breaks the usual alias pattern.

closes pullfrog/pullfrog#40

* ci: add bedrock env vars to test workflows

mirrors the new bedrock provider's required env vars (AWS_BEARER_TOKEN_BEDROCK
inherited from org secret + AWS_REGION + BEDROCK_MODEL_ID hardcoded) into both
.github/workflows/test.yml files so the ci.test "env vars cover all provider
API keys" assertion passes.

* docs(bedrock): clearer setup flow + screenshot of model selector

restructures the setup section into three concrete steps in execution order:
select Bedrock from the dropdown, store the bearer token as a secret (Pullfrog
or GitHub — links to keys.mdx for the trade-off), then add region + model id
directly in pullfrog.yml since neither is sensitive. enable-model-access in
the Bedrock console moved to step 4 (only required once per model and only
when AWS rejects the call, not blocking on first run).

adds a screenshot of the console model selector with Amazon Bedrock selected
so readers can recognize the UI state they're aiming for.

* fix(bedrock): tolerate raw Bedrock model IDs in validateAgentApiKey

main.ts passes the resolved model into validateAgentApiKey
(`payload.proxyModel ?? resolvedModel ?? payload.model`). For Bedrock,
`resolveModel` translates `bedrock/byok` into the raw AWS model ID
(e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/` and so
trips parseModel inside getModelEnvVars.

Detect the no-slash case and re-run the bedrock setup check (auth +
region; BEDROCK_MODEL_ID is already enforced upstream by resolveModel).

Caught by PR #720 e2e dispatch on pullfrog/preview-720-bedrock —
"invalid model slug 'us.anthropic.claude-opus-4-6-v1' — expected
'provider/model'". Two regression tests cover the raw-ID path.

* fix(bedrock): always prepend amazon-bedrock/ prefix when bedrock-routed

opencode.ts was gating the prefix-injection on `!isBedrockAnthropicId(rawModel)`,
on the theory that Anthropic Bedrock IDs always go through claude-code. But
`PULLFROG_AGENT=opencode` is a documented escape hatch — when it forces
opencode for an Anthropic Bedrock model, the prefix still has to be added or
opencode fails with 'Model not found: <modelId>/.'.

The Anthropic-vs-other discriminant only belongs in resolveAgent. Once an
agent is selected, it should consistently honor the bedrock route.

Caught by the PULLFROG_AGENT=opencode + Opus 4.6 e2e on
pullfrog/preview-720-bedrock — run 25823437606.

* ui+docs(bedrock): bespoke setup callout + clearer docs

UI:
- BedrockSetupCallout in components/AgentSettings.tsx covers both the
  Model costs section and the onboarding card. Detects bedrock via
  resolveDisplayAlias().routing === "bedrock", shows a dedicated message
  ("store AWS_BEARER_TOKEN_BEDROCK as a secret, then put AWS_REGION +
  BEDROCK_MODEL_ID directly in pullfrog.yml") + link to the setup guide.
  Replaces the generic "X, Y, or Z is required" prompt that misrepresented
  the three values as three separate secrets to add (and used the wrong
  "or" connector for what's actually an AND).
- OnboardingCard re-uses the same callout with the gradient-card variant.

Docs:
- Drop the obsolete "Enable model access" step. AWS retired the manual
  enrollment page; foundation models auto-enable on first invocation.
  Anthropic models still need a one-time use-case form for first-time
  users — surfaced under the AccessDenied troubleshooting entry.
- Drop the "Testing a different model in one run" PULLFROG_MODEL note.
  It introduced the secrets-vs-vars distinction we want to keep out of
  the bedrock setup story.
- Step 3 already recommends hardcoding region + model id in pullfrog.yml.

Workflow template:
- The default pullfrog.yml customers receive (utils/github/pullfrog.yml.ts)
  now references AWS_BEARER_TOKEN_BEDROCK from secrets but inlines
  AWS_REGION and BEDROCK_MODEL_ID as plain values. Matches the docs.

* fix(bedrock): three review-caught edges in routing + UI copy

Addresses three real issues from PR #720 review:

1. agent.ts: PULLFROG_MODEL=bedrock/byok no longer leaks the literal
   sentinel "bedrock" downstream. resolveCliModel returns the alias's
   resolve field verbatim, which for routing entries IS the sentinel.
   Refactored both the env-override and slug-lookup paths through a
   shared resolveSlug() that recognizes routing aliases and defers to
   their backing env var (BEDROCK_MODEL_ID).

2. models.ts: isBedrockAnthropicId() now anchors on a discrete
   dot/slash/colon-segment match (case-insensitive) instead of a
   substring contains. The substring check was fragile in both
   directions for inference-profile ARNs (BEDROCK_MODEL_ID accepts
   ARNs per AWS docs) — a non-Anthropic profile whose user-chosen name
   contained "anthropic" would mis-route to claude-code, and an
   Anthropic profile whose name omitted it would miss
   CLAUDE_CODE_USE_BEDROCK=1.

3. AgentSettings.tsx: BedrockSetupCallout's configured-state copy
   showed "AWS_BEARER_TOKEN_BEDROCK configured" even when the user
   satisfied the gate via AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY,
   gaslighting access-key users about a secret they never set.
   Detect which auth method is actually present and name the right
   secret(s) in the success message.

Regression tests in models.test.ts (5 new isBedrockAnthropicId cases
including positive and negative ARN forms) and agent.test.ts (2 new
PULLFROG_MODEL=bedrock/byok cases). 171/171 action tests pass.

* yml template: add commented AWS access-key alternative for Bedrock auth

Mirrors the IAM access-key path verified end-to-end on PR #720 e2e
run 25830764987. Bearer token stays as the primary nudge; the access-key
pair is the fallback for users who can't mint Bedrock API keys.

* yml template: drop redundant 'or, alternatively' annotation

* ui+docs(bedrock): rewrite callout copy + refresh screenshot

Reframes the BedrockSetupCallout away from generic BYOK language to a
Bedrock-specific message: leads with "Amazon Bedrock is configured
entirely via environment variables", lists all four (auth, region,
model id), and ends with the requested CTA sentence ("click below to
learn more about Bedrock support in Pullfrog").

Promotes the "Bedrock setup guide" docs link from an inline anchor to
a prominent button (always visible, regardless of auth state). The
"Add AWS_BEARER_TOKEN_BEDROCK" affordance is now a secondary chip
shown only when no auth secret is configured.

Refreshes docs/images/model-selector-bedrock.png to capture the new
callout — the prior screenshot still showed the old generic
"BYOK / X, Y, or Z required" wording.
2026-05-14 02:12:38 +00:00
Colin McDonnell 1a9d3c1f82 fix bootstrap ETARGET when customer has npm min-release-age policy (#725)
* fix bootstrap ETARGET when customer has npm min-release-age policy

set npm_config_min_release_age=0 in the action runtime env so
`npx --yes pullfrog@<spec>` doesn't get rejected by a customer-side
release-age gate (npm 11.5+'s min-release-age / pnpm's
minimumReleaseAge). env vars beat .npmrc in npm config precedence,
so this neutralises the policy regardless of where it's defined.

pullfrog's npm version is server-stamped from a SHA-pinned action
ref customers already vet at the action layer — it isn't a
customer-vetted dep, so the release-age policy is the wrong
affordance for our bootstrap and would otherwise hard-fail every
run while the latest publish ages into the customer's window.

closes #713

* also cover pnpm's minimumReleaseAge key for corepack fallback path

* correct pnpm env var (pnpm v11+ uses pnpm_config_*, not npm_config_*)

the prior commit set `npm_config_minimum_release_age=0` to cover the
pnpm corepack-dlx fallback path, but pnpm v11+ only reads env vars
prefixed `pnpm_config_*` / `PNPM_CONFIG_*` (the v10→v11 migration
explicitly renamed the prefix). swap to the correct env var so the
fallback path actually neutralises pnpm's `minimumReleaseAge`.

also tighten the comment block, and add an AGENTS.md rule reminding
us to fetch top-level reviews AND inline review comments together —
they live on different endpoints and the inline set is easy to miss
with `gh pr view --json reviews,comments` alone.

* add scripts/pr-reviews.ts for one-shot review evaluation

dumps top-level reviews + inline review threads (with resolved/outdated
state) + PR-level conversation in a single GraphQL round trip, so agents
don't miss inline-comment feedback. fixes the trap where
`gh pr view --json reviews,comments` silently omits the inline
`pulls/{n}/comments` set.

borrows `gh auth token` so no env vars are required. registered in
`wiki/scripts.md`; AGENTS.md rule updated to point at the script
instead of the two-step gh-CLI workaround.

* pr-reviews: dump raw JSON for jq piping
2026-05-14 01:48:14 +00:00
Colin McDonnell 951745ec89 disable stop hook (runtime + dashboard) (#727) 2026-05-14 01:44:32 +00:00
Colin McDonnell 56793d4a81 claude: prefer non-JSON stdout over NDJSON tail in exit-1 fallback (#643) (#726)
Claude CLI under CLAUDE_CODE_OAUTH_TOKEN exits 1 without setting `is_error`
when the OAuth subscription's quota is exhausted. The existing fallback
chain (`lastResultError || stderr || tailLines(stdout)`) had nothing
structured to grab and dumped ~2KB of `system/init` NDJSON into the
progress comment, hiding the actionable quota notice the CLI had already
printed as plain text.

Capture non-JSON stdout lines into a 20-line ring buffer (mirroring the
existing `recentStderr` pattern) and prefer it over the raw NDJSON tail.
Generic — no regex on bubble text — so any human-readable line the CLI
emits surfaces instead of the event stream.

Also adds a `failure:claude-oauth-quota` bucket to `analyze-logs.ts`,
ordered before the SIGTERM check so the NDJSON tail's `cancelled` /
`cancel_url` substrings (from learnings content) stop shadowing it.
2026-05-14 01:26:10 +00:00
Colin McDonnell d857e06731 postrun: tighten unsubmitted-review gate to require create_pull_request_review for Review mode (#724)
The gate at `getUnsubmittedReview` accepted `toolState.finalSummaryWritten`
as a valid Review exit, contradicting the post-failure error message which
already says Review's only valid exit is `create_pull_request_review`.
This let any caller that flipped `finalSummaryWritten` — including a
`task`-dispatched `reviewfrog` subagent calling `pullfrog_report_progress`
in violation of its prose-only read-only contract — silence the gate even
when the orchestrator never submitted a review.

Split per-mode: Review requires `toolState.review`, IncrementalReview keeps
the existing `||` (its post-failure message explicitly accepts
`report_progress` as a "no review warranted" exit). Test split mirrors the
new semantics.

closes #648
2026-05-14 00:01:15 +00:00
Colin McDonnell b9f0938405 mcp: restore operational guidance dropped in #723
#723's revision pass cut four substantive strings along with the
negative anchors. those strings address real, audit-observed failure
modes and the positive examples don't carry them.

restored:
- push_branch: "if the response reports a timeout, the underlying
  push may have actually succeeded — verify with git log
  origin/<branch> before retrying" (was on the tool description)
- create_pull_request_review commit_id .describe(): "must be the FULL
  40-character SHA — abbreviated SHAs are rejected by GitHub with 422"
- create_pull_request_review comments[].line .describe(): "must sit
  inside a `@@` hunk... dropped entries are reported under
  droppedComments in the response"
- create_pull_request_review comments[].start_line .describe(): "both
  start_line and line must sit inside the same @@ hunk"

also: get_commit_info example used a 31-character SHA (non-standard
truncation). swapped to a 7-char short form, which is what git
log --oneline emits and what agents see in practice. note that this
tool accepts either full or abbreviated, unlike create_pull_request_review
which requires full.
2026-05-13 22:49:06 +00:00
Colin McDonnell b8ac42e875 mcp: embed example calls in top-level tool descriptions (#723)
* mcp: embed example calls in top-level tool descriptions

agents (esp. claude sonnet) hallucinate param names from training-data
priors — `pr_number` instead of `pull_number`, `summary` instead of
`body`, full subcommand strings jammed into `git({command})` like it
were `shell({command})`. each error burns a tool round-trip plus a
follow-up ToolSearch, ~40+ events / 24h, no observable recovery cost
to us but visible to users in agent logs.

cheapest fix: add a sample formatted function call to every affected
tool's top-level description. example anchors are more reliable than
schema descriptions alone because the model treats descriptions as
narrative but call examples as canonical structure. for `git` and
`shell` (whose `command` fields collide), include explicit
counter-examples disambiguating which tool owns which shape.

no schema aliases / coercion yet — try the cheap thing first; if the
next audit window still shows the same hallucination rate, layer
aliases on top per #585's recommendation.

closes #585, closes #701

* mcp: drop negative anchors from tool descriptions

negation is a footgun in tool descriptions — telling the model "NOT
pr_number" makes pr_number more salient, not less. let the positive
example carry the schema and trust the model to read it.

removes:
- "the parameter is pull_number (a number), NOT pr_number" and
  similar across checkout_pr, get_pull_request, list_pull_request_reviews,
  get_review_comments, create_pull_request_review
- "NOT summary, message, or content" on report_progress
- "WRONG: git({ command: 'log --oneline' })" counter-example on git
- redundant param-type restatements after the example (e.g. "depth is a
  number, not a string" on git_fetch, "description is required" on shell)

keeps a single positive example per tool. for tools with multiple call
shapes (git, git_fetch, push_branch), two positive examples instead of
one + a counter-example.
2026-05-13 22:45:08 +00:00
Colin McDonnell 868576a474 audit: format byok auth errors actionably + tighten audit prompt
- `action/utils/apiKeys.ts`: rewrite the missing-key body as Markdown with
  linked CTAs (repo secrets / model settings / docs). add
  `isApiKeyAuthError` + `formatApiKeyErrorSummary` covering both shapes:
  missing key (#679) and revoked/invalid 401 key (#702).
- `action/main.ts`: reclassify in the result-failure branch and the catch
  block so the PR progress comment surfaces the actionable CTA instead of
  the raw `Invalid API key · Fix external API key` / numbered-list dump.
- `scripts/analyze-logs.ts`: split `failure:user-misconfig` into
  `:no-key` and `:invalid-key` so both buckets are visible separately
  and the audit can ignore them as user-correctable.
- `.github/workflows/run-audit.yml`: add three explicit prompt rules —
  cross-customer signal required (≥3 distinct accounts; single-customer
  concentration is not enough), recovered failures are not actionable,
  user misconfig is out of scope. closes the loop on #679 / #702 being
  filed in the first place.
2026-05-13 21:59:47 +00:00
Colin McDonnell b2b1e588e7 biome: exclude .scripts/ — gitignored operator scratchpad
Mirrors the gitignore. Same shape as the existing !**/logs / !**/.logs
/ !.worktrees exclusions in files.includes. Matches the upstream
.gitignore policy for the .scripts/ directory.

Without this, .scripts/ scripts (`.scripts/kyle-*.ts`,
`.scripts/check-comment.ts`, etc.) get scanned by `pnpm lint` and
`pnpm format` from the repo root and routinely fail husky pre-push
even though they're explicitly intended to be local-only / personal.
The companion to .gitignore — both are operator-owned scratchpads;
neither participates in repo-wide hygiene.
2026-05-13 21:25:43 +00:00
Colin McDonnell 5caeb75344 review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models (#710)
* review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models

PR review wall-time was dominated by two failure modes: orchestrator
serial-dispatching subagents (despite prompt asking for parallel) and
running every lens on the same Opus tier as the orchestrator. Sample of
recent runs showed 25-60min reviews on small PRs, with 8-10min idle
gaps between subagent dispatches.

Three changes:

1. `action/modes.ts` — replace the soft "1 trivial / 2-3 typical /
   4-5 high-stakes" lens calibration with a binary 0-or-2+ rule. Default
   is 0 lenses (orchestrator handles review solo with optional cheap
   tracerfrog dispatches). 2+ parallel lenses only fire for substantive
   PRs (>5 files AND >200 lines) or high-stakes-subsystem touches. Never
   exactly one. Both Review and IncrementalReview prompts get loud
   ALL-CAPS framing on parallel dispatch — emit ALL Task tool_use blocks
   in a single assistant turn before reading any result. Drop the
   "do NOT lens-review the diff yourself" advice; orchestrator pulls
   context aggressively, in parallel with the lens fan-out.

2. New `tracerfrog` subagent for mechanical code tracing ("where is X
   used / who calls Y / what depends on Z"). Pure read+grep+report with
   no judgment — orchestrator can dispatch many tracers cheaply in
   parallel. Defined in `action/agents/reviewer.ts`. Wired into both
   claude.ts (`--agents` JSON) and opencode.ts (`agent` config block).

3. Per-subagent model downshifts via `deriveSubagentModels`:
   - Anthropic: reviewfrog → Sonnet, tracerfrog → Haiku
   - OpenAI: both → gpt-5.4-mini
   - other providers (xai, deepseek, gemini, etc.): inherit (no
     standard tier triplet to downshift to)

Claude Code path always runs Anthropic so the downshift is hardcoded
inline in claude.ts. OpenCode uses the helper since orchestrator
provider varies.

Both runtimes' subagent-definition formats verified directly against
their source: `--agents` JSON `model` field (claude-code's
`AgentJsonSchema` accepts model+effort+maxTurns+more) and OpenCode's
`agent.{name}.model` config field (parsed via Provider.parseModel,
applied per-task in tool/task.ts line 92). Parallel dispatch is
infra-supported in both — only the orchestrator model's tool_use
emission pattern was the bottleneck.

Tests: subagentModels.test.ts (14 tests covering provider matrix),
subagentRegistration.test.ts (6 source-asserts catching shape
regressions in buildAgentsJson / buildReviewerAgentConfig).

* subagentModels: add openrouter routes (proxy/router mode)

Initial helper missed the openrouter prefix used by Pullfrog's router
proxy. preview-710 e2e showed the OpenCode + openrouter path receiving
no downshift — orchestrator and lenses both ran on opus-4.7 because
'openrouter/anthropic/claude-opus-4.7' didn't match any of the
anthropic/openai prefixes the helper checked.

Add explicit branches for 'openrouter/anthropic/...' (uses dot notation:
claude-sonnet-4.6 / claude-haiku-4.5) and 'openrouter/openai/...'
(gpt-5.4-mini for both reviewer and tracer). Same opus->sonnet,
sonnet->keep-but-haiku-tracer, haiku->no-op semantics as the direct
anthropic path.

* opencode: log resolved subagent models at startup

So we can verify per-subagent model overrides actually take effect at
runtime. Prints once per run alongside the existing model/effort log
lines.

* drop tracerfrog: keep reviewfrog only, LSP-powered tracer planned later

Removes the cheap-haiku-tracer subagent (TRACER_AGENT_NAME +
TRACER_SYSTEM_PROMPT, registrations in claude.ts/opencode.ts, dispatch
guidance in modes.ts). The mechanical-tracing use case will be served
better by an LSP-powered tool than by a separately-prompted subagent.

deriveSubagentModels collapses to a single { reviewer } shape; the
reviewfrog-on-Sonnet downshift stays. Same source-assert + provider-
matrix tests, minus the tracer-specific cases.

modes.ts wording: drop the 'subagent type cheat sheet' bullet, drop
the parenthetical 'often better served by tracerfrog than reviewfrog'
on the impact lens, drop tracerfrog from the same-turn-context-pulling
hint. The 0-or-2+ rule and ALL-CAPS parallel emphasis are unchanged.

* subagentModels: broader downshift coverage (gpt-pro, gemini-pro, grok); drop gpt-mini target

Scanned every resolved orchestrator slug in action/models.ts against
models.dev pricing data. Identified five clear cases where the
orchestrator is meaningfully expensive AND has a cheaper sibling that
remains capable enough for review-style judgment work.

Changes:
- Anthropic: opus → sonnet  (kept; -40%)
- OpenAI: gpt → gpt-5.4 (was: gpt-mini; -54% instead of -85% but
  preserves review-quality judgment — gpt-mini was too dumb)
- OpenAI: gpt-pro → gpt   (NEW; -93%, biggest single unlock —
  gpt-5.5-pro is $30/Mtok in vs gpt-5.5 at $5)
- Google: gemini-pro → gemini-flash  (NEW; -75%)
- xAI: grok-4.3 → grok-4-1-fast  (NEW; -80%)

Every branch handles the three routes in use: direct provider slug,
opencode-vendored, and openrouter-proxied. Variants below the downshift
target (mini/nano/flash/fast/sonnet/haiku) inherit (no further drop).

Skipped:
- DeepSeek: v4-flash ($0.14/Mtok) is too far below review judgment
  threshold; v4-pro orchestrator already cheap ($0.55 blended).
- Moonshot: kimi-k2-thinking would only save 32% and slug stability on
  OpenRouter is uncertain; revisit if cost matters.
- o3: already mid-tier in OpenAI's reasoning family; no clean target.

* models: hoist subagent downshift into the registry, add hidden flag

The downshift relationship now lives next to each alias's resolve /
openRouterResolve as a sibling field. Two new ModelDef fields:

- subagentModel?: string — alias key (within same provider) of the
  cheaper sibling reviewfrog should use as a lens-fanout subagent.
  e.g. claude-opus → 'claude-sonnet'.
- hidden?: boolean — exclude from selectable lists (UI dropdown,
  CLI init picker). Does NOT affect resolution; for that use
  fallback. Used so internal-only subagent targets like openai/gpt-5.4
  exist in the registry but never appear as a user-facing pick.

Wiring:
- anthropic.claude-opus → claude-sonnet (-40%)
- openai.gpt-pro → gpt (-93%, biggest unlock)
- openai.gpt → gpt-5.4 (-54%); gpt-5.4 added with hidden:true
- google.gemini-pro → gemini-flash (-75%)
- mirrored across opencode + openrouter providers (each provider
  declares its own three-route data so the downshift declaration
  is colocated with the rest of the alias definition).

deriveSubagentModels collapses from ~85 lines of prefix-matching to
a ~15-line registry reverse-lookup: find the alias whose resolve OR
openRouterResolve matches the orchestrator's spec, follow its
subagentModel pointer, return the matching field of the target alias.

Filter sites updated:
- components/ModelSelector.tsx: !a.fallback && !a.hidden
- action/commands/init.ts:       same

Tests rewritten to exercise the registry through the public surface;
the matrix collapses to one assertion per (provider × route) pair.

* TEMP: log per-step cost+tokens for subagent model verification (PR #710)

* TEMP: also log SUBAGENT step_finish from bus envelope handler

* remove temporary per-step diagnostic logs (verification done)

Verified subagent model downshift takes effect end-to-end on the OpenCode
+ openrouter path. PR #8 in pullfrog/preview-710-review-perf dispatched
3 lenses (billing-subsystem / security / correctness) on the orchestrator's
opus-4.7 session, and per-subagent step_finish events showed actual cost
exactly matching Sonnet pricing rates (60% of what Opus would have cost):

  session       n  actual    if-Opus   if-Sonnet  match
  T3VrUuF...    5  $0.2425   $0.4042   $0.2425    Sonnet ✓
  93ZZR7E...    4  $0.2253   $0.3754   $0.2253    Sonnet ✓
  Fb1Kr7b...    4  $0.2495   $0.4158   $0.2495    Sonnet ✓

The startup '» subagent models: reviewfrog=...' line stays — useful
permanent diagnostic showing the resolved subagent model per-run.

* TEMP: log per-event model from claude.ts assistant handler

* remove temporary per-event model log (claude.ts verification done)

Verified subagent model downshift takes effect end-to-end on the Claude
Code path. PR #9 in pullfrog/preview-710-review-perf dispatched 2 lenses
on an opus-4-7 orchestrator. Per-assistant-event model field from the
SDK's stream-json output, partitioned by parent_tool_use_id:

  ORCH (parent_tool_use_id=null):  17 events all model=claude-opus-4-7
  SUBAGENT lens:billing-subsystem: 17 events all model=claude-sonnet-4-6
  SUBAGENT lens:security:          21 events all model=claude-sonnet-4-6

Zero leakage to opus from either subagent session. The per-subagent
'model' field in --agents JSON is honored by claude-code at the SDK
level, identical to the OpenCode path verified earlier.

* opencode: bump per-call output cap 5K → 16K to unblock large reviews

The 5K cap (added in #616 to lower OpenRouter upfront budget reservation
for low-wallet runs) was capping the entire response of a single LLM call,
not just the budget reservation. A single tool_use response — like a
`create_pull_request_review` with many inline comments — would truncate
mid-stream past 5K output tokens, leave the JSON unparseable, and the tool
would never actually invoke. We hit this on PR #710's verify-downshift PR:
review aggregated from 3 lenses had 11 inline comments + a long body,
truncated at out=5000 on every retry attempt, action exited with 'Review
mode finished without calling create_pull_request_review after 3 retry
attempts'.

Investigated whether OpenCode (or OpenAI/Anthropic/OpenRouter directly)
exposes a separate budget-reservation parameter that could stay small
while letting the response exceed it. They don't — `max_tokens` /
`max_completion_tokens` is the single value all four use for both the
upfront reservation and the hard output ceiling. No way to decouple them
at the API surface.

Bumped to 16K as a middle ground: 8× the prior cap (handles every review
shape we've observed plus headroom), still half of OpenCode's 32K default
so the wallet-burn benefit for low-balance accounts is preserved, just
smaller. For Opus 4.7 a typical ~50K-input call now reserves roughly
$0.65 instead of the prior $0.38.

Updated the constant comment to spell out the trade-off clearly so this
doesn't happen again.

* opencode: drop OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX override entirely

Verified the original rationale for the override is obsolete. From #616
the cap shrunk OpenRouter's per-call upfront budget reservation so a
single call's reservation wouldn't exceed the per-run key cap
(`ROUTER_PER_RUN_LIMIT_USD = 25`) and lock low-balance accounts out of
starting a run.

That per-run gate is gone. `app/api/proxy-token/route.ts` ~line 422
explicitly says: 'No upper cap (the old ROUTER_PER_RUN_LIMIT_USD = 25 is
gone). The natural ceiling is whatever the user has + their buffer.'
Router now mints keys with `keyLimitCents = balance + buffer` ($50 for
autoreload+card, $5 for card-only, $0 for no-card). A single call's
upfront reservation fits comfortably within that — no separate per-call
gate to fail past.

The cap had a real downside as a hard per-call output truncation. A
single `create_pull_request_review` tool_use with many inline comments
would truncate mid-stream past 5K output tokens, the JSON would be
unparseable, and the tool never invoked. Hit on PR #710's
verify-downshift PR.

Removing the override entirely; OpenCode falls back to its 32K default.
Left an explanatory note above the env-var assignment site so the next
person doesn't unknowingly re-add it.
2026-05-13 21:05:52 +00:00
David Blass 5518890b18 learnings: TOC + section taxonomy + 100k cap, hygiene rules, tool-quirk descriptions (#717)
* audit learnings: reshape reflection prompt + bake tool quirks into descriptions (#619)

Cross-repo audit of the 48 repos with non-null learnings turned up two
recurring failure modes:

1. ~25-30% of bullets across the most-active repos are pullfrog-tool
   quirks ("shell timeout is in milliseconds", "git args must be a JSON
   array", "create_pull_request_review drops out-of-hunk comments",
   "push_branch may report timeout when push succeeded", "checkout_pr
   shallow.lock retries", "commit_id needs full 40-char SHA"). These are
   universal across repos and should live in tool descriptions, not be
   rediscovered and stored 48 times. Tool descriptions now surface them.

2. Bullets are routinely 200-1000 chars (paragraph-length), and 12 of 48
   repos are at the 10k cap. The reflection prompt now: caps bullets at
   ~240 chars (one specific fact), bans PR/review/commit/date-anchored
   facts that decay within weeks, bans tool-quirk learnings, and tells
   the agent that cap pressure means compress+prune existing bullets,
   not skip new findings.

Co-authored-by: Cursor <cursoragent@cursor.com>

* learnings: add server-generated TOC, fixed section taxonomy, raise cap to 100k (#707)

Cap goes 10k → 100k. Reads stay bounded because the seeded file now
opens with a server-generated table of contents listing every `## `
section's line range — agents read the TOC, then `read_file offset/limit`
just the sections relevant to the current task instead of slurping the
whole file.

## Section taxonomy (fixed)

`## Build & test`, `## CI`, `## Conventions`, `## Architecture`,
`## Gotchas`. Free-form `### ` sub-headings inside a section are fine.
Pre-taxonomy free-text rows get wrapped in a `## Legacy` carve-out on
first seed so they remain visible while the agent gradually re-curates
them during reflection turns.

## Storage shape unchanged

`Repo.learnings` still holds raw markdown (no schema migration). The TOC
is a pure read-side affordance: prepended at seed time, stripped from
the agent-edited file before persist. Markers
`<!-- pullfrog-learnings-toc:* -->` delimit the strip region. Agent
edits inside the markers are discarded.

## Round-trip semantics

`seedLearningsFile` now returns `{ path, canonicalSeed }` where
`canonicalSeed` is the post-TOC body — same shape `readLearningsFile`
returns at end-of-run, so `persistLearnings` byte-compares them
directly to skip the no-op PATCH. Empty-repo first runs end up with the
section scaffold both as seed and as read-back, so untouched runs still
short-circuit cleanly.

## Reflection prompt

Adds explicit section-placement guidance (place each new bullet under
the most relevant `## `; do NOT add new top-level headings; do NOT
edit anything between the TOC markers). Carries forward the bullet
hygiene from the previous commit: ≤240 chars per bullet, no
pullfrog-tool quirks (those belong in tool descriptions), no
PR/review/commit/date references. The "near cap" framing is replaced
with "compress and prune within a section when it grows noisy" since
the cap pressure that drove cramming is gone.

Co-authored-by: Cursor <cursoragent@cursor.com>

* anneal round 1: line-anchored taxonomy detect, partial-merge, line-boundary truncation, scaffold-empty UI

Multi-lens review of the TOC + taxonomy diff surfaced a cluster of
correctness and operational bugs. Fixes:

- `hasAnyTaxonomyHeading` used `String.includes("## X")` which
  false-positives on `### X` (the `## ` substring sits inside `### `),
  prose containing `## CI`, fenced code documenting markdown, etc.
  Replaced with a line-anchored predicate that reuses `parseHeadings`
  so detection and TOC construction stay consistent.

- The "any heading present → pass through verbatim" rule meant a body
  with one taxonomy heading would seed without the other four. Worse,
  requiring all five would flip a body back into Legacy when the agent
  legitimately pruned a section to empty. New `partial` kind: keep
  existing content in place, append missing sections in canonical order
  so the agent always has the full scaffold without losing pruning
  intent.

- `stripLearningsToc` collapsed `\n{3,}` globally; `canonicalSeed`
  doesn't, so an untouched body with intentional triple-newline spacing
  would compare unequal and burn a spurious LearningsRevision row each
  run. Drop the global collapse — only the leading newlines that the
  strip itself introduces are normalized.

- 100k truncation via `slice(0, 100_000)` could cut mid-line, breaking
  `parseHeadings` (whole-line `^## `) on the next seed and flipping a
  cut body back into Legacy. New `truncateAtLineBoundary` cuts at the
  last newline before the cap.

- `LearningsSection.tsx` rendered a scaffold-only body as "has
  learnings" instead of the empty placeholder. Added a
  `hasOnlyEmptyScaffold` guard so the console behaves the same as
  pre-PR for the empty case.

- Seed log line distinguishes `kind=structured/partial/legacy-wrapped/
  empty` instead of `existing=yes/no`, so operators can spot legacy
  migration activity in logs.

- New tests cover: substring false-positive (`### Build & test`,
  in-prose mentions), partial-taxonomy merge (no Legacy wrap),
  full-taxonomy structured pass-through, last-newline truncation,
  triple-newline preservation.

Deferred (documented in PR body): deploy-ordering footgun (action
before API), rollback for rows >10k, Gemini sanitizer dropping
`description` on `anyOf` branches, reflection-on-failed-runs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* anneal r2: hard-truncate fallback when line boundary discards >4k

Round-2 review caught a regression in `truncateAtLineBoundary`: when the
only newline within the first 100k chars sits near the start (e.g. one
heading + 100k+ char single line — pathological pasted log dumps), the
line-boundary cut discards almost all of the body. losing one partial
line is preferable to losing kilobytes; threshold the fallback at 4k.

Co-authored-by: Cursor <cursoragent@cursor.com>

* move TOC out of file: prompt-side rendering, server-parsed headings

drops the in-file TOC + fixed taxonomy in favor of:
- file on disk = verbatim Repo.learnings (no markers, no scaffold)
- server parses headings (mdast-util-from-markdown) at run-context time
  and returns them as RepoSettings.learningsHeadings
- action renders heading TOC into the LEARNINGS prompt section as
  parenthesized line ranges like `Build & test (L1-L42)` with hierarchy
  via 2-space indent off the shallowest depth
- reflection prompt teaches agent-curated structure with a soft 300-line
  per-section cap and explicit guidance to restructure flat legacy lists

cuts 8 helpers (ensureSections, stripLearningsToc, assembleFile,
buildTocBlock, parseHeadings, buildSectionScaffold, hasAnyTaxonomyHeading,
LEARNINGS_SECTIONS) and the canonicalSeed round-trip dance.

action seedLearningsFile is now { path } only; main.ts byte-compares the
trimmed read-back against (current ?? "").trim() to gate the persist
PATCH. truncateAtLineBoundary kept for safety.

new tests:
- test/learningsToc.test.ts (11 parser cases incl. fenced-code, blockquote,
  arbitrary h1-h6 nesting, startLine-points-at-heading invariant)
- action/utils/learningsTocRender.test.ts (7 renderer cases)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-13 20:14:26 +00:00
Colin McDonnell d04c1ca3da action: bump to 0.1.6 2026-05-13 18:23:45 +00:00
Colin McDonnell ae976e7159 parallel tool execution: enable opencode batch + nudge agents to parallelize (#719)
opencode: opt into `experimental.batch_tool` (anomalyco/opencode#2983) so the
`batch` tool registers and the model can bundle 1-25 independent calls into one
round trip. edit calls are excluded upstream.

instructions.ts: add a "Parallel tool execution" section to the SYSTEM Workflow
block, agent-specialized via ctx.agentId. uses Anthropic's canonical wording
("invoke all relevant tools simultaneously...") so Claude reliably emits multiple
tool_use blocks per message; tells OpenCode about the new `batch` affordance.

verified end-to-end against haiku-class models (sonnet for claude, default for
opencode) with a "read 3 files and report first lines" fixture. results:
- opencode used `batch` with 3 nested reads AND emitted 3 native parallel
  read calls in the same assistant turn
- claude went from 3 serial turns (1 read each) to 1 message with 3 parallel
  Read tool_use blocks
2026-05-13 18:05:39 +00:00
Colin McDonnell 5aabd1e4a9 fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680) (#715)
* fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680)

unbounded `stdoutBuffer += chunk` / `stderrBuffer += chunk` in
`action/utils/subprocess.ts` previously crashed the wrapper with
`RangeError: Invalid string length` once V8's ~1 GiB kMaxLength was
breached on long-lived agent runs. multi-lens opencode Reviews on large
monorepos (e.g. tambo-ai/buildy) hit this consistently — 23 runs in the
last 24h, 100% of Review-mode hard failures on that repo.

- add `retain: "tail" | "none"` to SpawnOptions, defaulting to "tail"
  with an 8 MiB cap. tail-mode prepends a `... [N MiB truncated] ...`
  sentinel so downstream consumers can detect truncation.
- export `TailBuffer` helper for callers that need the same bounded
  accumulator semantics at their own layer.
- wrap stream `data` listeners in try/catch as defense in depth — any
  synchronous throw inside a stream handler is otherwise fatal.
- opencode + claude pass `retain: "none"` (they drain via onStdout /
  onStderr) and switch their own `output` accumulators to TailBuffer.
  their error paths read the agent-layer bounded mirrors instead of
  the now-empty `result.stdout` / `result.stderr`.
- add `failure:string-length-overflow` heuristic to scripts/analyze-logs.ts
  so post-fix recurrences are visible at a glance instead of bucketing
  into `failure:unknown`.
- regression tests cover >1 MiB stderr without crash, retain:"none"
  contract, and TailBuffer truncation semantics.

* fix: avoid TS parameter property syntax in TailBuffer for strip-only node loader

* address review: clarify try/catch scope + lock retain default to "tail"

- the original comment claimed the try/catch caught "any synchronous throw"
  in the data listener, but `options.onStdout?.(chunk)` returns a Promise
  in the agent callers (claude.ts:569, opencode.ts:933) — a throw inside
  an async user callback surfaces as an unhandled Promise rejection, not
  a synchronous exception. reword to describe the actual protection:
  defense-in-depth for synchronous throws in the listener body, which is
  exactly the shape of the original RangeError on `+= chunk`.
- add a test that locks `retain` default to "tail" by spawning without
  the option and asserting `result.stderr` is non-empty. a future refactor
  that flipped the default to "none" would silently break gitAuth,
  package installs, and lifecycle hooks that read result.stderr for
  failure messages, and the rest of the suite wouldn't catch it.
2026-05-13 17:54:28 +00:00
Colin McDonnell 60cc8772a6 fix(log-audit): kill 404 noise from /api/github/installation-token at source (#693) (#708)
* fix(log-audit): kill 404 noise from `/api/github/installation-token` at source (#693)

Closes #693. Issue diagnosed a surface symptom (`log.error` on expected
404s) but missed the actual root causes. Investigation revealed two
distinct populations producing identical 3-call 404 bursts:

1. **Fork-CI on `pullfrog/pullfrog`**: `test-token.yml` and
   `trigger-sync.yml` ship with `on: push: main`, so every fork inherits
   them and 404s our token endpoint on first push. Self-inflicted noise
   that scales with fork count.
2. **Real users hitting the full action without installing the App**:
   `/api/repo/.../run-context` uses the caller's `GITHUB_TOKEN` to read
   the repo from GitHub and then unconditionally lazy-provisions
   Account+Repo rows via `fetchOrCreateRepo`, even when the App isn't
   installed. Generates phantom DB rows and false `new account created`
   team@ alerts. (Confirmed via Prisma: `ezcorp-org` has an Account row
   with `installerLogin: null`, never installed our App.)

Both populations then trip the client retry loop in
`acquireTokenViaOIDC`, which matched `"Token exchange failed"` and
retried 3× on terminal 4xx — tripling log volume and wasting CI time.

## Changes

- `action/.github/workflows/{test-token,trigger-sync}.yml`: gate jobs
  with `if: github.repository == 'pullfrog/pullfrog'`. Forks inherit
  the files but the jobs no-op.
- `app/api/repo/[owner]/[repo]/run-context/route.ts`: call
  `getRepoInstallation` first; return 404 with install URL if the App
  isn't installed, before any DB writes or GitHub repo fetch.
- `action/utils/github.ts`: introduce `TokenExchangeError` for non-2xx
  server responses; `acquireNewToken` no longer retries it. Retry now
  fires only on genuine network/timeout failures. 404 surfaces a
  user-actionable error pointing at the install URL.
- `app/api/github/installation-token/route.ts`: move `log.error` inside
  the 500 branch only. 404 branch is silent (expected user-state) and
  returns the same install URL message for consistency.

## Effect

- Better Stack `level=error` lines from this path: 6/day → 0.
- Failed user-trial CI time: 3 wasted token requests → 1.
- User-facing error: opaque `Token exchange failed: 404` → actionable
  install URL.
- No more phantom Account rows from never-installed callers.

Skipped per design discussion: phantom-account cleanup (conservative —
stop the bleed, leave history), `AGENTS.md` rule (overgeneralized).

* review: address oracle leak + per-env install URL + retryable 5xx

Addresses pullfrog[bot] (IMPORTANT) and Copilot review findings on #708:

- **Install-status oracle in `run-context`** [pullfrog, Copilot]:
  `getRepoInstallation` runs with our App's JWT, *before* the caller's
  bearer token is validated against the repo. Pre-PR the route was
  uniformly bad-token-shaped; the new install-specific 404 turned it
  into an unauthenticated oracle distinguishing "Pullfrog installed
  here" from "not installed". Collapsed the 404 message to match the
  outer catch's ambiguous "repository not found or token lacks access".
  Legit runners still get the actionable install URL from
  `/api/github/installation-token`, which IS gated by OIDC.

- **Hardcoded `github.com/apps/pullfrog`** [Copilot]: server-side
  `installation-token` now uses `GITHUB_APP_INSTALL_URL` from
  `app/globals.ts`, so dev/staging deployments with a different
  `GITHUB_APP_SLUG` direct users to the correct app. Action-side
  echoes the server's `error` body when present (single source of
  truth) and falls back to a generic message only if the body isn't
  JSON.

- **Transient 5xx/429 made terminal** [Copilot]: `shouldRetry` now
  returns `true` for `TokenExchangeError` with `status >= 500` or
  `status === 429`. 4xx remains terminal (the actual #693 fix). Real
  outages no longer fail the workflow immediately.

- **Stale comment** [pullfrog, Copilot]: reworded the comment at
  `installation-token/route.ts:141` to reflect the new retry policy
  ("the action surfaces this once (no retry)" instead of "the action
  retries on this").

* review: restore caller-token-first auth in run-context

Pre-PR, `getEnrichedRepo({owner, repo, token})` used the caller's
token as the auth boundary — `getRepo({token})` succeeding was the
proof-of-access check. My initial install-gate inverted the order
and ran the App-credentialed `getRepoInstallation` first, which is
how it became:

- an install-status oracle (pullfrog bot, addressed previously by
  matching the outer-catch wording), and
- an outbound amplifier against our App JWT for arbitrary `owner/repo`
  (pullfrog bot, this commit).

Reordered so `getRepo({token})` runs first. Garbage / unauthorized
bearers get rejected by github (mapped to 403 by the outer catch)
before any App-credentialed call fires. `getRepo` is cached 5min,
so `getEnrichedRepo` below remains a free re-hit.
2026-05-13 17:47:13 +00:00
Colin McDonnell 4260984257 attribute claude subagent log lines + per-session thinking timer; tighten lens calibration (#700)
* attribute claude subagent log lines + per-session thinking timer; tighten lens calibration

three orthogonal fixes diagnosed from the 10m PR-699 review run:

1. wire SessionLabeler into the Claude Code harness. claude-agent-sdk
   stamps every Assistant/User/System message with session_id and a
   non-null parent_tool_use_id when emitted from a subagent context, so
   the same FIFO labeler the OpenCode harness uses works here too.
   parallel reviewfrog dispatches now log with [lens:correctness] /
   [lens:operational-readiness] / etc. prefixes instead of being
   indistinguishable from the orchestrator. matches both "Task" and
   "Agent" tool names per the v2.1.63 rename.

2. one ThinkingTimer per session. the global timer treated cross-session
   interleaving (parent thinks → child tool_call, child returns →
   parent dispatches next) as parent thinking time, so individual
   "thought for Xs" numbers were untrustworthy. each session now owns
   its own timer and prefixes its own log line.

3. tighten the Review/IncrementalReview lens-add discipline. PR-699
   triggered 4 lenses on a typical refactor (no auth/billing/schema)
   when the prompt's own calibration says 2-3 is typical; the
   research-validated lens went deep on Resend idempotency window +
   prisma updateMany lost-updates without either being load-bearing.
   adds an explicit "name the failure mode this lens would catch
   that the diff plausibly introduces" bar, and tightens
   research-validated specifically: only when correctness depends on
   the third-party contract, not when the API is merely used.

side benefits from #1: subagents' TodoWrite events no longer clobber
the orchestrator's progress comment; subagent text no longer overwrites
finalOutput; system-event handler safely routes through eventLabel even
though SDK only emits system:init for the top-level query today.

* fix node strip-only mode: declare formatLine as field, not parameter property

* key claude subagent labels by parent_tool_use_id, not session_id

claude-agent-sdk runs subagents inside the orchestrator's session — they
share session_id — and stamps subagent messages with parent_tool_use_id
pointing at the Agent tool_use that spawned them. e2e on PR-700 with
preview-700-claude-labeling#1 confirmed the original session_id-keyed
wiring never differentiated subagent activity (only the dispatch line
got [lens:correctness] in the log; the subagent's reads, writes, and
todos all rendered as orchestrator).

extend SessionLabeler so labelFor accepts an optional parent_tool_use_id
and short-circuits to a direct map keyed by Agent tool_use id when set.
recordTaskDispatch optionally takes the Agent tool_use id (block.id at
dispatch time) and binds it. orchestrator events keep flowing through
the sessionID/FIFO path unchanged so opencode wiring is untouched.

* drop weak timer test that asserted only field isolation

per pullfrog review on PR-700: the 'two timers do not bleed timestamps'
test only verified that two ThinkingTimer instances have separate
private fields, which has always been true. doesn't earn its keep —
the per-session behavior is exercised by integration through claude.ts
+ opencode.ts.
2026-05-13 15:28:08 +00:00
Colin McDonnell d5f881e9fc action: trim sensitive env values before GitHub Actions log masking (#698)
* action: trim sensitive env values before GitHub Actions log masking

GitHub Actions' log masking is line-based: a secret value containing a
newline only registers the first line as a mask, leaving the remainder
exposed verbatim in logs. A trailing newline copied from a terminal into
a GitHub Actions secret (e.g. ANTHROPIC_API_KEY) was enough to leak
"a large part of the key" in run logs (pullfrog/pullfrog#41).

normalizeEnv now trims leading/trailing whitespace from any value whose
key matches the sensitive name pattern, masks the cleaned value, and
warns when whitespace was stripped so the user notices the source.
sanitizeSecret is reused for dbSecrets injection in main.ts. The three
secret-store PUT/POST routes also trim values defensively, matching the
existing name.trim() pattern.

Real multi-line secrets are not used in practice — even GITHUB_PRIVATE_KEY
PEMs are stored single-line with escaped \n and unescaped at the point of
use — so a straight trim() is safe.

* action: address review — use core.setSecret for masking, don't zero whitespace-only

Pullfrog's review of #698 caught two real issues in the original fix:

1. `console.log(\`::add-mask::\${trimmed}\`)` doesn't escape \r/\n. If a
   value survives trim with an embedded newline (PEMs, kubeconfigs, JSON),
   the runner only registers the first line as a mask and the rest leaks.
   `core.setSecret(trimmed)` routes through @actions/core which
   percent-encodes \r/\n so the runner V2 parser decodes back to the full
   value and registers every non-empty line as a separate mask. Removes
   the load-bearing "no embedded newlines" invariant from the fix.

2. Whitespace-only sensitive values silently became "". Downstream
   truthy checks would flip from "set" to "missing" with no log. Now
   sanitizeSecret returns null in that case and callers skip the
   process.env write, surfacing a clear missing-key error instead.

Tests rewritten to assert process.env state directly — no stdout spies.
Masking correctness is delegated to @actions/core (trusted dependency).
2026-05-13 15:27:13 +00:00
Colin McDonnell 1dc53043a6 chore: bump action to 0.1.5 2026-05-13 04:56:01 +00:00
Colin McDonnell 076e5a17b5 default Claude Code effort to high
max effort burns roughly 2x the wall time per turn for marginal quality
gain. high is the model's tuned default ('equivalent to not setting the
parameter' per Anthropic docs). full-send can be reintroduced as an
opt-in per-run override later if needed.
2026-05-13 04:49:07 +00:00
Colin McDonnell d5d8a0d7ac fix(#691): drop opencode/gpt-5-nano + opencode/mimo-v2-pro-free (not actually keyless on Zen) (#695)
* remove opencode/gpt-5-nano and opencode/mimo-v2-pro-free from catalog

#7 delete aliases. both were listed as `isFree: true, envVars: []` but
neither is keyless on opencode zen, producing a hard-fail
`UnknownError: Model not found: opencode/<id>` on every run without an
opencode_api_key. fixes pullfrog/app#691 (5 runs across 3 repos, 100%
failure rate in the last 24h).

root cause: opencode's provider gate
(`packages/opencode/src/provider/provider.ts` `opencode:` loader) keeps
a zen model only when models.dev reports `cost.input === 0` for it,
then signs requests with `apiKey: "public"`. paid zen models get
deleted from the autoloaded set and opencode surfaces the deletion as
"model not found".

- `opencode/gpt-5-nano`: models.dev reports `cost: {input: 0.05, output:
  0.4, cache_read: 0.005}`. paid → requires `OPENCODE_API_KEY`.
- `opencode/mimo-v2-pro-free`: free on models.dev but not in
  `https://opencode.ai/zen/v1/models` — zen never served it, so even
  the public-key path fails.

remaining free aliases (`opencode/big-pickle`,
`opencode/minimax-m2.5-free`) both pass both checks (cost.input === 0
in models.dev AND present in zen's served list) and continue to work
without a key — verified against the opencode source.

callers swept: `action/utils/apiKeys.test.ts`, `action/models.test.ts`,
`action/test/list-aliases.ts`, `action/test/model-smoke.ts`,
`components/ModelSelector.tsx` (`modelIdToUpstream`),
`wiki/model-resolution.md`, `wiki/models-catalog.md`. wrote up the
free-zen verification rule in models-catalog so the next maintainer
can sanity-check both conditions before adding any `isFree` alias.

users with a stored `opencode/gpt-5-nano` or `opencode/mimo-v2-pro-free`
will now fall through `resolveCliModel → undefined` into the auto-select
path — a strict improvement over today's hard fail. no DB migration
needed; the slugs are simply unknown and treated like any other
unrecognized stored value.

* rework: keep mimo deprecated, demote gpt-5-nano to paid, add free-zen invariants

revised approach after the first commit over-corrected. mimo was never
broken at runtime — `fallback: "opencode/big-pickle"` already routes
stored values through to a real free model before any zen call. the
literal `opencode/mimo-v2-pro-free` being absent from zen's served list
is irrelevant because `resolveCliModel` walks the chain first. restoring
it as-is.

the actual bug was `opencode/gpt-5-nano`: marked `isFree: true,
envVars: []` but `models.dev` reports `cost: {input: 0.05, output: 0.4}`
on the opencode provider, so opencode's keyless gate
(`packages/opencode/src/provider/provider.ts` `opencode:`) deletes it
when `OPENCODE_API_KEY` is missing and the run hard-fails with
`UnknownError: Model not found: opencode/gpt-5-nano`. demoting it to a
regular paid zen alias (drop `isFree`/`envVars: []`, add
`openRouterResolve: "openrouter/openai/gpt-5-nano"` — verified to exist
on openrouter at the same price). users without `OPENCODE_API_KEY` now
get our explicit "no API key found" error pointing at the secrets page
instead of opencode's cryptic upstream error. confirmed via
`https://opencode.ai/zen/v1/models` that zen serves no free GPT
variants, so there's no cheaper-than-`gpt-mini` free option to suggest
in its place.

CI gap analysis (why this slipped through):

- `models-catalog.main.test.ts` only checked existence + `status !==
  "deprecated"` on models.dev. paid-model-marked-free regressions and
  zen-served-list drift both passed.
- `models-live` (`model-smoke.ts`) runs with `OPENCODE_API_KEY` in env,
  so the keyless deletion gate never fires. `gpt-5-nano` returned "OK"
  in CI even though end users hit a hard fail.
- `model-smoke.ts` walks the fallback chain, so mimo would have been
  smoked as big-pickle anyway — the dead resolve target was never
  exercised directly. (this is the right design; the gap is at the
  catalog layer, not the smoke layer.)

new tests:

- PR-blocking, static (`action/test/models.test.ts`, `isFree
  invariants`): every `isFree` alias must live under `opencode`, have
  `envVars: []`, omit `openRouterResolve`, AND have a fallback chain
  whose terminal alias is also `isFree` (catches "deprecate a free
  alias to a paid target" — the worst silent-charge regression).
- main-only, network (`action/test/models-catalog.main.test.ts`,
  `opencode Zen served list`): every alias whose terminal-fallback
  resolve is `opencode/*` must appear in
  `https://opencode.ai/zen/v1/models`. catches zen dropping a model
  from its served list.
- main-only, network (same file, `isFree models.dev cost`): every
  `isFree` alias's terminal-fallback resolve must have `cost.input ===
  0` in the `opencode` provider block on `models.dev`. would have
  caught `gpt-5-nano` at the next models-bump run.

both network tests dedupe on terminal resolve, so deprecated aliases
sharing a target aren't double-counted. `pnpm vitest run`: 113 static
tests pass. `pnpm test:catalog`: 142 network tests pass against the
live `models.dev`, `openrouter.ai`, and `opencode.ai/zen/v1/models`
endpoints.

wiki/models-catalog.md: rewrote the new "Free-Zen aliases need Zen-side
verification" section to (a) describe the two conditions, (b) note
that a fallback to an isFree alias is the legitimate escape hatch
(mimo's pattern), and (c) point at the three tests by name so the next
maintainer can find the enforcement surface. wiki/model-resolution.md
points at the new section.

* make gpt-5-nano a deprecated free alias falling back to big-pickle

revising the previous "demote to paid" approach. the user-facing
ergonomics are cleaner: anyone who picked gpt-5-nano under the "Free"
badge gets transparent-upgraded to a real free model (big-pickle)
instead of suddenly being asked to set OPENCODE_API_KEY. matches the
existing mimo pattern exactly. the dropdown already filters
`!a.fallback`, so the slug disappears from the picker on its own and
the trigger renders it as "Big Pickle" via `resolveDisplayAlias`.

no other catalog or test surface changes — the isFree invariants and
the main-only zen/cost checks still pass (gpt-5-nano's terminal is
now big-pickle, which is both isFree and zero-cost on models.dev,
deduping with big-pickle's own row in both network tests).

* revise: keep gpt-5-nano as paid alias, backfill affected DB rows instead

dropping the deprecated-alias approach. `opencode/gpt-5-nano` is a
legitimate cheap paid model people may want with BYOK
(`OPENCODE_API_KEY`) — giving it `fallback: "opencode/big-pickle"`
would foreclose that for everyone going forward. correct fix is two
parts:

(a) reclassify in the catalog as a regular paid OpenCode alias:
  - drop `isFree: true` and `envVars: []` so the local validator
    demands `OPENCODE_API_KEY`
  - add `openRouterResolve: "openrouter/openai/gpt-5-nano"` to satisfy
    the completeness test and route BYOK-via-OpenRouter users
  - no `fallback` — slug stays visible in the picker as a paid option

(b) one-shot DB backfill of provably-affected repos
(`scripts/backfill-gpt5-nano-affected.ts`). scope:
  - `Repo.model = "opencode/gpt-5-nano"`
  - AND at least one `WorkflowRun` with `inputTokens IS NULL` (evidence
    of an attempted run that didn't get past the model-init gate)

skipped intentionally:
  - repos whose runs have `inputTokens > 0` — they have a key, gpt-5-
    nano works for them
  - repos with zero WorkflowRun rows — never dispatched; touching them
    would be presumptuous
  - `LearningsRevision.model` — audit trail of which model authored a
    revision, rewriting it would falsify history

ran against .env.prod: 2 repos stored the slug; 1 was provably
affected (sodown4thecause/seobot, 5/5 zero-token runs — matches #691's
3 failed runs from this repo plus 2 outside the 24h audit window).
1 was an internal test account that never dispatched (left as-is).
applied: 1 row updated. confirmed idempotent on re-run.

the other two repos in #691 (Nantiee/ALTA-breast-pump-tool,
keksiqc/ansible-setup-linux) don't store the slug in `Repo.model`;
their failed dispatches passed the model inline in the
`workflow_dispatch` `prompt` payload, so the catalog fix alone (no
longer offering it as free) is what helps them.

tests:
  - models.test.ts: `getModelEnvVars("opencode/gpt-5-nano")` now
    returns `["OPENCODE_API_KEY"]`, moved into the keyed-model group
  - apiKeys.test.ts: added "throws without OPENCODE_API_KEY" case
  - isFree invariants from the previous commit still pass — gpt-5-nano
    no longer triggers them since it's no longer isFree
  - main-only catalog tests still pass (gpt-5-nano served by Zen, just
    paid; no isFree cost check applies)

* docs: drop stale GPT Nano + MiMo V2 Pro from free-tier lists

addressing pullfrog auto-review feedback on #695. three mintlify pages
still advertised both as keyless after the catalog pivot, which now
makes the docs affirmatively wrong rather than merely stale:

- gpt nano is paid in the catalog (no `isFree`, inherits
  `OPENCODE_API_KEY`); a user following the docs would hit the same
  "missing API key" failure that's described 4 lines below in
  `docs/keys.mdx`.
- mimo v2 pro is hidden from the picker (`fallback` triggers
  `ModelSelector`'s `!a.fallback` filter); the alias only exists for
  legacy stored-value resolution. a user reading the docs cannot
  actually pick it.

surviving picker-visible free set: Big Pickle and MiniMax M2.5.

- `docs/keys.mdx`: drop both bullets from the "Free models" list
- `docs/billing.mdx`: drop both bullets from the "Free models" list
- `docs/getting-started.mdx`: collapse the inline mention from a
  4-model list to "Big Pickle and MiniMax M2.5"

* address third review: picker grouping + backfill classifier honesty

i had not pulled the third pullfrog review (`02:17:28Z`) when i declared
reviews triaged after the docs sweep — the fourth review flagged that
three findings remained pending. addressing them now.

1. picker grouping for now-selectable paid gpt-5-nano. when i removed
   `"gpt-5-nano": "OpenAI"` from `modelIdToUpstream` in the previous
   pivot-to-paid commit, i mistook it for dead code. it's not — the map
   IS consulted for paid opencode aliases via `groupByUpstream →
   getUpstreamLabel` inside the OpenCode submenu's
   `renderSubContent`. without the entry, `gpt-5-nano` falls back to
   `getProviderDisplayName("opencode")` = "OpenCode" and gets dropped
   into its own sub-header instead of joining opencode/gpt,
   opencode/gpt-pro, opencode/gpt-mini under the "OpenAI" upstream
   group. re-added with an explanatory comment so the next refactor
   doesn't make the same mistake.

2. JSDoc / code mismatch in `scripts/backfill-gpt5-nano-affected.ts`.
   the JSDoc said "at least one `WorkflowRun` with `inputTokens IS
   NULL`" but the code is `no WorkflowRun has inputTokens > 0` — a
   strictly broader filter (catches `null` AND `0`). rewrote the scope
   block to describe what the code actually does, with the operative
   classifier spelled out: "a billable run with `inputTokens > 0` is
   proof the agent successfully reached and called the model".

3. classifier breadth (raised in the same review). honest answer: the
   "no positive-token run" filter IS a heuristic — a repo whose only
   dispatches happened to fail or cancel for unrelated reasons would
   get false-positive-classified A. for THIS one-shot population (2
   repos, 1 with 5/5 zero-token runs — strong systematic-failure
   signal) the heuristic was good enough and the dry-run inspection
   confirmed before APPLY. for any larger reuse of this pattern, you
   need to cross-reference the runtime error string (`UnknownError:
   Model not found: opencode/gpt-5-nano`) from GitHub Actions logs or
   Better Stack — that error doesn't live on `WorkflowRun` rows. added
   a "Classifier limitations" section to the JSDoc making this
   explicit.

nothing about the actual applied backfill changes — the prod write
(1 repo: sodown4thecause/seobot → opencode/big-pickle) is unchanged
and re-running the script remains idempotent.
2026-05-13 02:43:08 +00:00
Colin McDonnell 159389fad2 fix(mcp): sanitize for gemini when model is unresolved (#697)
* fix(mcp): sanitize for gemini when model is unresolved

isGeminiRouted() previously required the effective model string to
contain "gemini" — but when payload.model="auto" (or any unresolved
slug) reaches addTools(), `effective` is the literal "auto", which
doesn't match. opencode then auto-selects gemini *after* the MCP
server has registered raw arktype schemas, and every tool turn dies
on `function_declarations[*].properties[*].any_of[*].enum: only
allowed for STRING type`.

widen the gate: any unresolved specifier (undefined / "auto" / a
slug without a `provider/` prefix) is treated as gemini-routed and
sanitized. the transforms are universally compatible normalizations
so the false-positive cost is negligible. tighten case 3 to preserve
`description` so the only lossy path no longer drops operator-facing
context.

fixes #676.

* revert case-3 description preservation

per pullfrog review on #697: keeping `description` as a peer of
`anyOf`/`oneOf` directly contradicts the file's own header (lines
19-21) and the upstream opencode #14659 rationale that gates this
sanitizer — gemini requires anyOf to be the ONLY field on a schema
node, sibling keywords trigger
`anyOf must be the only field in a schema node`. the change was
speculative scope creep with no evidence, and would silently
re-introduce a different gemini failure for any future schema using
`.describe().or(...)`. the bug fix for #676 doesn't need it (arktype
doesn't emit non-collapsible anyOf for current tool schemas).
2026-05-13 02:31:59 +00:00
Colin McDonnell 43bb14bf87 action: strip Content-Type on body-less apiFetch requests (#692) (#694)
* action: strip Content-Type on body-less apiFetch requests (#692)

Vercel's Next.js lambda adapter (Next 16.1.x) attempts to decode a
request body when Content-Type is set and throws
`SyntaxError: Unexpected end of data` before delegating to the route
handler, returning a 500. Hit /run-context exclusively because it was
the only body-less GET that sent `Content-Type: application/json`.

- Drop `Content-Type: application/json` from the GET in
  `action/utils/runContext.ts` (meaningless on a body-less request).
- Defensively strip any `content-type` header in `action/utils/apiFetch.ts`
  when no body is present so future callers can't reintroduce this.

* apiFetch: soften comment — empirical observation, RFC 9110 §8.3 framing
2026-05-13 02:03:24 +00:00
Colin McDonnell d8f825034f billing: $10 signup credit + lazy claim modal; disable welcome credit promo (#674)
* billing: $10 signup credit + lazy claim modal; disable welcome credit promo

Adds a per-Account $10 Router signup credit granted on first Router-tab
mount via a new admin-gated POST /api/account/[owner]/signup-credit/claim.
The endpoint is idempotent — the inserted CreditGrant row IS the dedup
state, so subsequent calls return granted:false. Client SignupCreditModal
fires the POST on mount (only when modelAccessMode === "router") and
opens a celebratory dialog when granted:true.

Disables the legacy welcome credit ($10 on first card add) via a new
WELCOME_CREDIT_PROMO_ACTIVE = false flag in utils/stripe.ts. Code path
stays intact — flip the flag to revive. Strips the now-untruthful
"$10 on enabling billing" copy from BillingCard, EnableRouterPrompt,
triggerWorkflow paywall comment, action router_requires_card summary,
email snippet, billing/pricing docs and wiki.

Cuts WELCOME_CREDIT_CENTS from 2000 to 1000 to reflect the lower amount
that would land if the flag is ever re-enabled. Adds "signup" reason
mapping to BillingCard wallet history.

Verified end-to-end against dev: admin+Router fires modal, admin+BYOK
gate-blocks mount, BYOK→Router transition fires modal on click, member
and collaborator paths skip the mount entirely, reload after grant is
idempotent. Wallet history shows "Router signup credit +$10.00".

* billing: address PR review (race fix, copy sweep, modal retry)

Correctness:
- Add @@unique([accountId, reason]) on CreditGrant + migration. The prior
  check-then-insert pattern in /signup-credit/claim and finalizeCheckoutSession
  raced at READ COMMITTED — two concurrent admin tabs could land two grants of
  the same reason on a fresh account ($10 each). Both write sites now rely on
  the unique index for dedup (P2002 = "already granted") and route updated to
  catch P2002 cleanly. Verified zero existing duplicates in prod before
  migration.
- Add log.info on signup grant insert so a successful grant has any chance of
  being caught by ops monitoring.
- Add retry: 2 with backoff to the claim mutation. Endpoint is idempotent so
  a server-side success that lost its response cleanly returns granted:false
  on retry.

Public copy that still advertised the (now-deleted) $20 welcome credit:
- app/page.tsx landing pricing card
- emails/announceBilling.ts broadcast template
- docs/keys.mdx BYOK note
- components/AgentSettings.tsx Router-without-billing warning
- utils/stripe.ts finalizeCheckoutSession JSDoc
- utils/email/snippets.ts ROUTER_CREDIT_PS_HTML JSDoc

Wiki staleness sweep:
- wiki/billing.md TOC, mermaid diagram (signup edge added; welcome marked
  dormant), test coverage list, key modules section, no-card wallet narrative
- wiki/pricing.md welcome-credit drawdown reference
- Rewrote my own internally-inconsistent dormancy paragraph to be honest
  about the $20-historical / $10-on-revival framing.

Trivia:
- ModelAccessCard JSX comment had a literal \\u2192 instead of →.

* billing: address PR review round 2

- Replace try/catch P2002 inside finalizeCheckoutSession's prisma.$transaction
  with createMany skipDuplicates. The previous form is broken on Postgres: a
  unique-violation poisons the surrounding TX, so the catch block returns
  cleanly but the outer commit fails and the account.update (stripeCustomerId)
  silently rolls back too. Currently armed only behind the dormant welcome-
  credit flag, but would have broken billing enablement the moment the flag
  flipped. createMany skipDuplicates yields a single ON CONFLICT DO NOTHING
  statement that returns count: 0 cleanly without aborting the TX.
- Apply the same createMany skipDuplicates pattern to the signup-credit route
  too — drops the exception-as-control-flow Prisma namespace import and is
  more uniform with the welcome path.
- Drop the now-orphaned credit_grants_accountId_idx in the same migration.
  The schema removed @@index([accountId]) when @@unique([accountId, reason])
  was added (covered by the leftmost prefix), but the migration only added
  the unique index, leaving prod drifted.

* billing: fix stale finalizeCheckoutSession JSDoc

The function-level JSDoc still described the abandoned try/catch P2002
mechanism after switching to createMany skipDuplicates. The inline
comment + code now agree on the new ON CONFLICT DO NOTHING shape.

* billing: decouple first-card alert, drop vestigial billing field, fix modal cents; sync copy

* docs+homepage: align Router credit copy with signup claim (no card-on-add carrot)

* homepage: add pricing screenshot and pay-as-you-go promo line

* billing: fix once-per-lifetime misframe on first-card alert

* billing: suppress signup credit for prior welcome-credit recipients

* billing: drop bogus '1000 users' cap; invalidate billing on signup-credit settle
2026-05-12 23:47:52 +00:00
Colin McDonnell f0805b78f5 learnings: surface persist failures as warnings, not debug
`persistLearnings` only emitted `log.info("» learnings updated")` on
success; every failure path (non-2xx, fetch throw, 10s timeout) was
`log.debug`, which is hidden unless `ACTIONS_RUNNER_DEBUG=true`. Survey
of recent runs caught at least one case where the agent definitively
edited the tmpfile but no DB row was written and no warning surfaced.

Promote both failure paths to `log.warning` so dropped agent work is
visible in CI logs. The unchanged-from-seed short-circuit stays at
debug — that's a genuine no-op.
2026-05-11 23:51:46 +00:00
Colin McDonnell e20b4d5515 action: bump to 0.1.4 2026-05-11 23:22:47 +00:00
David Blass 8c6cd2bda2 cancel + restart workflow run when @pullfrog mention is edited (#612)
* cancel + restart workflow run when @pullfrog mention is edited

- add `WorkflowRun.triggeringCommentId` (BigInt?, indexed) so the webhook
  handler can find the run that was fired by a given comment
- thread `triggeringCommentId` through `reserveRun` / `triggerWorkflow`
- factor `dispatchMentionRun` out of `issue_comment_created` so the same
  shape is reused on edit
- replace the `issue_comment_edited` stub: re-evaluates the trigger gate,
  cancels prior runs (`octokit.rest.actions.cancelWorkflowRun` + DB
  status='cancelled'), then re-dispatches with a `previousRunsNote`
  appended to `eventInstructions` so the agent acknowledges the prior
  run/PR/artifacts in its summary
- if the edit removes `@pullfrog`, cancel only (no restart)

Co-authored-by: Cursor <cursoragent@cursor.com>

* thread previousRunsNote via dedicated payload field

user prompt has precedence over eventInstructions, so stuffing the
prior-runs note into eventInstructions made it vanish whenever the
trigger comment contained an @pullfrog mention (which is always for the
edit path). pass it as its own payload field and render it alongside the
user's task so the agent actually sees it.

* delete cancelled run's progress comment on edit-restart

so the issue thread doesn't accumulate "This run was cancelled" stubs
on every edit. only deletes for runs we actively cancel; runs that were
already terminal (e.g. completed before the edit) keep their summary
comment in the thread, and `previousRunsNote` links to it so the new
agent can reference prior work.

post-cleanup is race-safe: the action's `validateStuckProgressComment`
swallows the 404 from the deleted comment and exits cleanly, so the
old run's post step cannot clobber the new run's leaping comment.

Co-authored-by: Cursor <cursoragent@cursor.com>

* also cancel + delete progress comment when triggering comment is deleted

mirrors the edit-removes-@pullfrog path: when an @pullfrog comment that
fired a run is hard-deleted, look up any prior runs by triggeringCommentId,
GH-cancel running ones, and delete their leaping progress comments.

skips trigger-gate re-eval (we're tearing down a run, not firing one) and
performs no restart. reuses the existing cancelRunsForTriggeringComment
helper; the returned previousRunsNote is discarded since no dispatch
follows.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: move cancellation before trigger gate in issue_comment_edited

cancelRunsForTriggeringComment now runs before the triggerEnabled check,
so edits that remove @pullfrog still cancel in-flight runs even when the
repo mention trigger is currently disabled (e.g. for non-collaborators).

* anneal: scope cancel updates per-row + simplify edit gate

- replace blanket updateMany on (triggeringCommentId, repoId) with per-row, status-guarded updates so a parallel handler's freshly-reserved run cannot be clobbered into cancelled by a racing edit delivery.
- drop wasMention/isMention early-break in issue_comment_edited; always run cancelRunsForTriggeringComment (DB is the canonical "did this comment ever trigger a run" source). closes the missing-changes.body.from edge and lets us tear down a still-running prior run even if the admin disabled the mention trigger mid-flight.
- buildPreviousRunsNote returns undefined (not "") when no link lines materialize.
- doc cleanups + wiki/modes.md addendum noting issue_comment_edited / _deleted now drive cancel + restart.

Co-authored-by: Cursor <cursoragent@cursor.com>

* address review feedback on cancel/restart semantics

- guard workflow_run.completed update against status='cancelled' so a
  successful-but-uncancellable GH Actions job can't resurrect a cancelled
  row (and re-bill it) via the completed webhook.
- bucket only status='completed' runs into `preserved` in
  cancelRunsForTriggeringComment; cancelled/failed prior runs have stubs
  as their progress comment, not summaries worth referencing.
- emit previousRunsNote for the runId-null cancel case so the restarted
  agent always knows when it's superseding a prior dispatch.
- drop the agent-forbidden `gh pr list` hint and soften 'was cancelled'
  to 'was signalled to cancel' in the note body.
- post a fallback comment when the edit-path dispatch fails (prior run
  already torn down and progress comment already deleted).
- symmetrize the delete-handler's pullfrog guard with the edit handler
  (key off hook.comment.user, not hook.sender).
- trim misleading comments on the per-row DB update guard.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-11 23:20:44 +00:00
Colin McDonnell e4d0fc7e3d biome: ignore .logs/ (was matching only logs/) 2026-05-11 23:06:33 +00:00
Colin McDonnell a4a5010441 gemini-3: default thinkingLevel to medium + restrict eager prep to frozen install (#663)
* gemini-3: default thinkingLevel to medium + don't `npm ci` without a lockfile

upstream opencode hardcodes `thinkingLevel: "high"` for every gemini-3 model on
the direct google SDK (see `packages/opencode/src/provider/transform.ts`
`options()`). that added 30-60s of pre-tool-call TTFT and 5-46s of post-tool
jabber per turn, which is overkill for the tool-routing decisions that dominate
agentic loops — and the variance caused the `providers-live (google/gemini-pro)`
smoke job to time out at 4 minutes (see job 75405504847 on run 25684766415).

three changes:

- inject `provider.google.models.<api-id>.options.thinkingConfig.thinkingLevel = "medium"`
  for the two curated gemini-3 slugs in `buildSecurityConfig`. deep-merges over
  the upstream default; explicit `--variant high` / user opencode config still
  wins. flash stays at medium too — low-effort flash is visibly worse and the
  latency win isn't meaningful (flash is already fast).
- bump the `providers-live` harness step from 4 → 6 minutes. the job-level
  8-minute cap stays as the upper bound, but gemini's intrinsic TTFT variance
  was eating most of the 4-minute slack on its own.
- in `installNodeDependencies`, pick `frozen` only when a lockfile was actually
  detected. previously a package.json-only repo (like the smoke fixture's
  `pullfrog/test-repo`) always triggered `npm ci` and emitted a noisy
  `EUSAGE` error before falling through.

* prep: skip eager install when neither lockfile nor `packageManager` field present

the previous commit changed the no-lockfile path from `npm ci` (always errored
`EUSAGE`, never wrote any artifact) to a successful `npm install`, which had
an unintended side effect: it generated `package-lock.json` in the working
tree, tripping the post-run dirty-tree gate. the agent then committed the
lockfile and opened a real PR — and in the openai/gpt smoke run on PR #663,
the agent overwrote the `SMOKE TEST PASSED` output with the PR URL, failing
the smoke validator.

a repo with `package.json` but no lockfile and no `packageManager` field has
not committed dependency state. eagerly installing produces state the repo
doesn't track, which is the dirty-tree problem above. skip the eager install
entirely in that case; the agent can opt in via `await_dependency_installation`
when it actually needs deps. repos with a lockfile or a `packageManager` field
keep the existing frozen-install behavior unchanged.

* post-run: suppress dirty-tree gate in non-committing modes (Review / IncrementalReview / Plan)

the dirty-tree post-run gate currently fires for every mode and tells the agent
to commit and push whatever is in the working tree. that's wrong for modes
that complete by submitting a review (`Review` / `IncrementalReview`) or
posting a Plan comment (`Plan`) — those modes never touch files as part of
their contract, so any tree dirt at end-of-run is incidental tool noise on an
ephemeral worktree. nudging the agent to commit it can produce a spurious PR,
as seen in the openai/gpt smoke run on PR #663 where a stray
`package-lock.json` from `npm install` led the agent to open
pullfrog/test-repo#32 and overwrite the smoke output.

introduce `NON_COMMITTING_MODES` in `action/modes.ts` and consult it in
`collectPostRunIssues`. when the selected mode is read-only, log the
suppression for visibility but skip populating `issues.dirtyTree`. modes that
legitimately commit (`Build`, `AddressReviews`, `Fix`, `ResolveConflicts`,
`Task`) keep the existing nudge.

* prep: restore eager frozen-install, drop non-frozen fallback

eager dependency prep is non-mutating by contract — it runs before the agent
starts and any artifact it leaves in the tree (e.g. a generated
`package-lock.json`) trips the dirty-tree post-run gate and can lead the agent
to open a spurious PR (seen on the openai/gpt smoke run earlier in this PR).

revert the previous skip-when-no-lockfile branch: that was the wrong layer to
enforce the invariant. instead, run `frozen` (`npm ci` / `pnpm install
--frozen-lockfile` / etc.) unconditionally and drop the `|| install` fallback
that could silently mutate the tree when `frozen` is missing. frozen commands
fail cleanly without writing artifacts when there's no lockfile, which is
exactly the safety contract we want. repos that need a real install must opt
in explicitly via a `setup` lifecycle hook.

* review nits: single getGitStatus call, tighten gemini-3 override scope comment

addresses two inline nits from the PR review:

- `collectPostRunIssues` was calling `getGitStatus()` (spawns `git status
  --porcelain`) in both branches of the mode check. lift the call above the
  conditional and branch on the result; same behavior, one git invocation.
- the JSDoc on `GEMINI_3_DIRECT_API_IDS` said the override applies "across
  the board," but the constant only covers the two curated slugs in
  `action/models.ts`. tighten the wording to call out that other gemini-3
  ids in models.dev keep the upstream "high" default.

skipped the bot's yarn-1 concern after reading yarn 1's `install.js`:
`bailout()` (lines 461-465) throws `frozenLockfileError` when
`frozenLockfile && (!lockfileClean || missingPatterns.length > 0)`, which
fires before `linker.init()` writes node_modules or runs lifecycle scripts.
the existing comment's claim that frozen commands fail without artifacts
holds for yarn 1 too.
2026-05-11 22:04:19 +00:00
Colin McDonnell cf94773bf0 modes: make task-list authoring the explicit first step in every mode checklist (#665)
* modes: make task-list authoring the explicit first step in every mode checklist

The system prompt already instructs the agent to author an internal task list
at the start of every run (action/utils/instructions.ts:291), but the rule
lives several hundred tokens above the agent's first decision point and
references the mode's checklist before the agent has it. Compliance is
roughly coin-flip across opus runs — PR #610 dead-air for 9m20s was the
extreme case; my own #664 e2e runs split 1-for-1 on `todowrite` compliance.

Putting the directive *inside* the checklist that `select_mode` returns
co-locates instruction with referent at the moment the agent decides what to
do next. Same vocabulary as the existing rule (`task list`, agent-agnostic;
the harness already maps to `todowrite`/`TodoWrite` per-agent in
agents/opencode.ts and agents/claude.ts). The directive is deliberately
non-prescriptive about list contents — the agent authors items based on the
work it's about to do, not from a hand-shaped template.

Touches all 8 built-in modes and the PlanEdit override:

- Build / AddressReviews / Review / IncrementalReview / Plan / Fix /
  ResolveConflicts / Task: inserts `1. **task list**: create your task list
  for this run as your first action.` and renumbers existing steps.
- action/mcp/selectMode.ts: same insertion in the PlanEdit override checklist.
- All internal step cross-references shifted +1 (`step 5` → `step 6`,
  `skip steps 3–4` → `skip steps 4–5`, etc.) across Review,
  IncrementalReview, and ResolveConflicts modes. One code-comment reference
  in IncrementalReview's preamble updated to match.

Complements #664 (live progress streaming): streaming guarantees the user
sees *something* regardless of compliance; this PR raises the ceiling on
what they see when the agent does comply (clean numbered checklist tracking
through the run instead of just the latest assistant message).

488 action tests pass; typecheck, lint, format all clean.

* postRun: fix stale 'step 7' reference missed during +1 renumbering
2026-05-11 21:57:11 +00:00
Colin McDonnell 8e36f76cfa postrun: thread AgentRunContext through the retry loop instead of repackaging (#652)
* postrun: thread AgentRunContext through the retry loop instead of repackaging

drop the per-gate plumbing in `runPostRunRetryLoop`: the loop now receives
`ctx: AgentRunContext` whole and reads `ctx.stopScript` + `ctx.toolState.*`
directly. `getUnsubmittedReview` becomes a pure utility in postRun.ts
instead of a closure shipped over `AgentRunContext`. `AgentRunContext`
loses 4 fields that duplicated `toolState` (`summaryFilePath`,
`summarySeed`, `learningsFilePath`, `getUnsubmittedReview`) and gains
`toolState: ToolState`. both harness call sites collapse from 11 lines to
7; main.ts deletes the inline closure.

`ToolState` and friends move from `action/mcp/server.ts` to
`action/toolState.ts` so non-MCP code (agents, post-run loop) stops
importing run-state types from the MCP server module.

no behavior change. 503/503 tests green.

* toolState: relocate `CommentableLines` to break dep cycle with mcp/review

`action/toolState.ts` was importing `CommentableLines` from
`mcp/review.ts`, which pulled the entire MCP server compile graph (24
files) into any consumer of `ToolState` — including `cf-worker-indexing`
via the `pullfrog/internal` re-export chain through `utils/log.ts` →
`agents/shared.ts` → `toolState.ts`. that exposed a pre-existing TS
error in `mcp/issueEvents.ts` (octokit types resolve differently under
cf-worker's `moduleResolution: bundler`).

move `CommentableLines` (a small `{ RIGHT: Set<number>; LEFT: Set<number> }`
state-shape type) to `toolState.ts` where it's used; re-export from
`mcp/review.ts` for back-compat with test and call-site imports. cuts
cf-worker's mcp/ compile inclusion from 24 files back to 0.

* postRun: drop mock-heavy retry-loop tests; keep pure gate predicate

`runPostRunRetryLoop` and `executeStopHook` were covered by ~560 lines
of mock-heavy regression-gate tests that stubbed `spawn` / `getGitStatus`
and fabricated `AgentRunContext` to drive orchestration paths. per
AGENTS.md ("prefer no test over a mock-heavy test that only catches the
most obvious form of regression") and the empirical track record — the
one real production failure of this code path (#646) was a missing npm
release, not a logic bug a unit test could catch — the value-to-ceremony
ratio is poor. delete them.

keep only the pure predicate: `getUnsubmittedReview(toolState)` is a
decision function whose four input conditions have user-visible
consequences when wrong. 5 assertions, no mocks, no ctx fabrication.

488 tests still pass.

* toolState: import PrepResult from prep/types.ts, not the barrel

same dep-cycle class as the previous CommentableLines fix. importing
PrepResult from prep/index.ts pulled prep/installNodeDependencies.ts
into the Next.js production build's typecheck graph (via
pullfrog/internal → utils/log.ts → agents/shared.ts → toolState.ts →
prep/index.ts → installNodeDependencies.ts), and Next.js's stricter
NODE_ENV-required ProcessEnv shape rejected an existing
`env: { PATH: ... }` literal.

prep/types.ts is a leaf module with zero imports — re-routing the type
import severs the chain. Vercel preview deploy goes from Error → Ready;
preview-sync stops racing the deploy.
2026-05-11 18:47:08 +00:00
Colin McDonnell dee13b160f console: case-insensitive owner/repo slug resolution (#649)
* console: case-insensitive owner/repo slug resolution

URL slugs may be any case but GitHub treats logins and repo names as
case-insensitive (and 301-redirects to canonical case). Internal
find/filter sites compared with `===`, so mixed-case slugs (e.g.
`/console/Pullfrog`) hard-403'd in resolveOwnerAccess and silently
redirected from the per-repo console when currentRepo lookup missed.

Lowercase both sides at every slug comparison: resolveOwnerAccess
installation lookup, currentRepo lookup in repo + history pages,
ConsoleHeader installation/repo lookups, getInstallations personal
split, getOrgMembership user/org checks, getInstallationRepos node
filter, getUserRole owner-as-collaborator check, and the action
runtime's installation-repo access check.

Caches keyed by raw input remain case-split across casings; that's
fine since both entries resolve to the same canonical GitHub data and
TTLs are short.

* api: resolve targetAccountId by gh node id

getAuthenticatedAccountContext was looking up Account by `name` using
the raw URL slug, but `Account.name` is plain String populated from
canonical GitHub login. Mixed-case URLs would render the page (since
resolveOwnerAccess is now case-insensitive) but every billing/secrets
API call would 403 on the find-by-name miss.

Resolve by gh_${access.installation.account.node_id} instead — invariant
to case-folding and login renames. Same pattern as the sibling owner
page route already uses.
2026-05-11 18:45:20 +00:00
Colin McDonnell ef394277c1 review: synthesize [!NOTE] informational tier with #644 alert judiciousness — 4-callout visual ladder + approved Fix-gate (#653)
* review: NOTE-tier callout + `actionable` flag to suppress Fix buttons

Adds an `actionable` parameter to the `create_pull_request_review` tool
(defaults true) so the agent can opt out of the Fix-it/Fix-all/Fix-👍s
footer affordance on informational reviews. Threaded through
`createAndSubmitWithFooter` so the buttons are omitted when
`actionable: false`.

Updates `Review` and `IncrementalReview` mode prompts with a 4th tier:
`> [!NOTE]` + `actionable: false` for mergeable, FYI-style observations
(prior feedback addressed cleanly, minor stale doc reference, etc.).
Calibration note: `[!IMPORTANT]`/`[!CAUTION]` are reserved for findings
that warrant code changes, because that's what trains users to click
Fix. `[!NOTE]` reviews must not carry inline comments — if a point is
concrete enough to anchor to a line, upgrade the whole review tier.

* review: drop redundant `actionable` flag, key Fix buttons off `approved`

`approved` already encodes "this PR is mergeable, nothing for the Fix
button to act on" — `actionable` was a second flag carrying the same
signal. Drop it from the tool schema and `FooterOpts`; the footer gate
stays `if (!opts.approved)` (unchanged from pre-PR behavior, with a new
comment documenting the UX rationale).

NOTE-tier reviews now use `approved: true` + `> [!NOTE]` body instead of
`approved: false` + `actionable: false`. For repos with
`prApproveEnabled: false`, the runtime already downgrades APPROVE to
COMMENT, so the GitHub-side shape is identical to the prior design.

* review: address Pullfrog feedback — drop ambiguous parenthetical + update postRun nudge

- Review-mode calibration: drop the "(or no callout at all)" parenthetical
  that didn't map cleanly to a bullet; replace with explicit "both the
  `[!NOTE]` tier and the 'no actionable issues' tier below use approved:
  true" so the bullet-list anchor is obvious.
- `buildUnsubmittedReviewPrompt` (Review mode): the fallback nudge for
  unsubmitted reviews now defers to the mode prompt's tier matrix and
  acknowledges that `> [!NOTE]` informational reviews submit with
  `approved: true` alongside the canonical "No new issues found." path.
  Previously the nudge only described the pre-NOTE binary world.
2026-05-11 17:14:03 +00:00
David Blass ee479474ce action: tighten review alert judiciousness in prompts (#644)
The Review and IncrementalReview prompts unconditionally wrapped any
non-critical review body in `> [!IMPORTANT]`, even for trivial nits or
"rough edge" observations. The result is alert fatigue — full-width
colored callouts dominate the page when the actual finding is a single
JSDoc tweak.

Adds an explicit judiciousness preamble to both Review step 5 and
IncrementalReview step 7, and splits the prior single non-critical tier
into two:

- must-address non-critical (`[!IMPORTANT]`) — gated on real
  consequences if shipped (incorrect behavior, missing validation,
  regressions the author should fix before merge)
- minor suggestions only (no alert) — single-line nits, doc/comment
  polish, defer-able observations, "rough edges"

Critical tier wording also tightened to spell out the bar (`bugs,
security, data loss, broken core flows`).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-11 17:07:00 +00:00
Colin McDonnell 96910f0f50 fix(run-audit): drop summary comment, fall back to agent final message in job summary
the audit agent's final 'post a short summary' instruction was ambiguous
and, with no PR/issue context on schedule runs, caused the agent to invent
a target — landing the summary as a comment on the most recent open PR
(see #650). drop the comment instruction outright.

writeJobSummary now falls back to the agent's final assistant message
(result.output) when lastProgressBody is empty, so non-PR runs surface a
real summary in the GitHub Actions job summary tab instead of just the
usage table. lastProgressBody still wins when present to avoid duplicating
the progress comment body.
2026-05-11 16:56:26 +00:00
Colin McDonnell 4cc6d95a91 ci: split per-alias resolution smoke from per-provider harness smoke (#650)
* ci: split per-alias resolution smoke from per-provider harness smoke

`models-live` previously ran the full Pullfrog harness (Docker + MCP +
agent + structured-output validation) once per alias on every PR that
touched `models.ts` or `agents/**`. That cost minutes and dollars per
alias and re-validated tool-calling for every routing wrapper.

The per-alias signal we actually need from `models.ts` changes is just
"does this alias resolve and authenticate." Tool-calling correctness is
a property of the underlying model, not the alias, and it doesn't change
when someone adds a row to the catalog. Splitting the two concerns:

- `models-live` now runs `action/test/model-smoke.ts` per alias — a
  top-level CLI invocation (`opencode run -m <resolve> "reply OK"` or
  `claude -p "reply OK" --model <bare>`) with no Docker, MCP, or
  Pullfrog harness. Validates resolution + auth in seconds at fractions
  of a cent. Lets us drop the `EXPENSIVE_RESOLVE_SUBSTRINGS` carve-out
  for `gpt-pro` since the cheap smoke covers it for free.

- `providers-live` (new) runs the full harness smoke once per provider
  against a hand-curated standard-tier model (`anthropic/claude-sonnet`,
  `openai/gpt`, `google/gemini-pro`, `xai/grok`,
  `deepseek/deepseek-pro`, `moonshotai/kimi-k2`,
  `opencode/big-pickle`, `openrouter/claude-sonnet`). Catches
  provider-class regressions like the Gemini schema sanitizer or
  OpenAI tool-call format drift. ~8 jobs, ~$0.40/push, ~4min critical
  path in parallel.

Net change per push that touches `models.ts`: ~$20 → ~$0.40.

`list-aliases.ts` now branches on `MODE` to emit either matrix; the
flagship list asserts each slug exists in `modelAliases` so renames
break CI loudly. Wiki updated to reflect the new two-tier coverage and
the operational rule for new Gemini aliases (cheap smoke covers
auth, manual harness run still needed for sanitizer compatibility on
non-flagship Gemini additions).

* fix(model-smoke): walk fallback chain; address pr review comments

- model-smoke now uses `resolveCliModel(slug)` instead of `alias.resolve`
  so deprecated aliases (those with `fallback` set, e.g.
  `opencode/mimo-v2-pro-free` → `opencode/big-pickle`) hit the
  replacement model the way production does. mimo-v2-pro-free was
  failing CI because the underlying opencode model is dead — the
  fallback chain is the whole point of marking it deprecated.

- tighten stale `agentForSlug()` reference in model-smoke.ts comment
  (function was deleted in this same PR; classification is now inline
  in `list-aliases.ts toMatrixEntry`).

- tighten `FLAGSHIPS` drift comment to call out that the assertion is
  one-way (catches slug-rename, but silently omits new providers).
  Update wiki step 4 of "To add a provider" to require adding the
  standard-tier slug to `FLAGSHIPS` for harness coverage.

* docs: scrub stale env-knob refs in models-catalog parity section

`wiki/models-catalog.md` cross-provider parity paragraph still pointed
at `INCLUDE_ALL_PASSTHROUGHS` / `INCLUDE_EXPENSIVE` and the implicit
filter→expensive-gate coupling — all removed in this PR. Aligned the
copy with Step 9 (which was already updated): `INCLUDE_PASSTHROUGHS`,
no expensive gate, `MATRIX_FILTER` applies to both aliases and
flagships modes.
2026-05-11 16:36:14 +00:00
David Blass 10590993f4 checkout_pr: retry missing pull/N/head ref with PR-state guard (#627)
* checkout_pr: retry missing pull/N/head ref with PR-state guard

Co-authored-by: Cursor <cursoragent@cursor.com>

* checkout_pr tests: satisfy ToolState required fields

Co-authored-by: Cursor <cursoragent@cursor.com>

* checkout_pr: tighten retry-helper semantics (anneal round 1)

Co-authored-by: Cursor <cursoragent@cursor.com>

* checkout_pr: use retry util, drop retry tests

* Update action/mcp/checkout.ts

Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
2026-05-09 00:58:55 +00:00
David Blass 10aeaf8c11 action: dedupe identical reply_to_review_comment calls within a session (#623)
* action: dedupe identical reply_to_review_comment calls within a session

PR #610 reproduced a Kimi K2 stutter where the agent's tool_use surface
showed one `pullfrog_reply_to_review_comment` call but GitHub recorded
two byte-identical POSTs 3s apart, leaving a duplicate response on
`action/mcp/review.ts:14`.

Add `duplicateReplyDecision` (mirrors `duplicateReviewDecision`) and
track per-session replies on `ToolState.reviewReplies`, keyed by
parent `comment_id` + `bodyWithFooter`. Identical re-emissions short
circuit with `{ skipped: true, reason }` instead of POSTing again.
Body-keyed (not just id-keyed) so legitimate follow-up replies with
different content still go through.

Tighten `AddressReviews` step 5 to say *exactly once per comment* and
note that the runtime dedupes identical bodies, so the agent has both
prompt-level guidance and a server-side guarantee.

Co-authored-by: Cursor <cursoragent@cursor.com>

* address review: drop stale file ref in dedupe comment; soften tool description

* remove comment.test.ts

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
2026-05-09 00:30:51 +00:00
Colin McDonnell 85d25a6fe6 post-run gate: fail review-mode runs that don't submit a review or progress (#638)
* post-run gate: fail the run when review mode finishes without a review or progress

review-mode runs that ended in a text-only assistant turn ("now I have enough
to draft the review...") were silently swallowed: the progress comment was
deleted by stranded-comment cleanup and no review appeared on the PR. user-
visible result was identical to "the agent never ran." caught in
https://github.com/pullfrog/app/actions/runs/25583698781.

new post-run gate alongside stopHook / dirtyTree / summaryStale: derived
inline from toolState (selectedMode in {Review, IncrementalReview} && !review
&& !finalSummaryWritten && hadProgressComment) — no parallel toolState flag.
when it fires, the resume prompt nudges the agent to call either
create_pull_request_review or report_progress; persistent failure after
MAX_POST_RUN_RETRIES surfaces as AgentResult.error.

also: when the post-run loop returns success=false, write the error to the
progress comment before the stranded-comment cleanup runs, and skip the
delete in that case. previously a !success run from the loop would lose the
error message into the void.

IncrementalReview's trivial-skip branch now calls report_progress with a
brief "no review warranted" note instead of exiting silently — keeps the
contract symmetric with the gate and gives the user a visible signal even
on no-op review runs.

documents the literal-record design rule on the ToolState interface so
future fields don't drift back into derived/absence-encoding state.

* review feedback: mode-aware nudge, gate-error preservation, prompt order

addresses three findings from the auto-review on this PR:

1. Review mode nudge no longer offers `report_progress` as an exit. Review
   mode's contract (modes.ts step 5) forbids it; the gate previously sent
   contradictory copy. IncrementalReview's nudge still offers both since
   its trivial-skip path legitimately allows `report_progress`.

2. `writeJobSummary` is now wrapped in try/catch on the success-path
   cleanup. without this, a throw there jumped to the outer catch and
   overwrote the gate's failure message in the progress comment with the
   (less actionable) writeJobSummary error — restoring exactly the
   invisible-failure UX this PR fixes. step-summary writes are
   informational; let them fail silently.

3. `buildPostRunPrompt` reorders gates to match the terminal hard-fail
   order: `stopHook` → `unsubmittedReview` → `dirtyTree` → `summaryStale`.
   when both hard-fail gates co-fire (rare in review modes), the prompt's
   emphasis now matches the user-visible failure message.

new test asserts the IncrementalReview nudge offers both exits while the
Review nudge offers only `create_pull_request_review`. e2e validation
already passed against pullfrog/preview-638-review-stop-hook PR #1
(gate fired once; agent recovered on second turn).

* mode-aware terminal error copy

second auto-review caught a residual contradiction: the terminal hard-fail
error string reported "create_pull_request_review or report_progress" for
both modes, even though the new mode-aware nudge tells Review-mode agents
"Review mode does not have a no-submit exit". the error message now mirrors
the nudge — Review names only `create_pull_request_review`,
IncrementalReview lists both. additional Review-mode hard-fail test asserts
the absence of `report_progress` in the error.
2026-05-09 00:14:31 +00:00
David Blass 653fae47a5 claude: surface structured error from is_error result events instead of dumping NDJSON (#626)
* claude: surface structured error from is_error result events instead of dumping NDJSON

Co-authored-by: Cursor <cursoragent@cursor.com>

* claude: tighten error-surface fixes (anneal round 1)

Co-authored-by: Cursor <cursoragent@cursor.com>

* claude: remove tests per request

* claude: gate is_error short-circuit on subtype=success, restore error_* branches

* claude: preserve fallback token table for error_* subtypes

the `lastResultError === null` guard was too broad — `error_max_turns` /
`error_during_execution` / `error_*` subtypes set `lastResultError` from
`event.errors[]` and represent runs that genuinely consumed tokens, so
suppressing the fallback table silently dropped billing visibility for
those cases. gate on a dedicated `syntheticStopFailure` flag that's set
only for the `subtype: "success"` + `is_error: true` case where
`accumulatedTokens` is stale.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
2026-05-09 00:07:50 +00:00
Colin McDonnell 363e4cbed8 ci: gate gpt-5.5-pro by resolve, refresh stale matrix docs (#639)
* address review: gate by resolve, refresh stale doc claims

- list-aliases.ts: gate EXPENSIVE on alias.resolve substring (catches
  opencode/gpt-pro and openrouter/gpt-pro, which both resolve to a
  gpt-5.5-pro variant — would have re-entered the matrix under
  INCLUDE_ALL_PASSTHROUGHS=1 and tripled the cost).
- test.yml + models-catalog.md: stop describing the matrix as
  exhaustive. Mention pruning + INCLUDE_EXPENSIVE/MATRIX_FILTER opt-ins.

* address review: clarify env vars are local-only, filter input is the CI knob

Pullfrog review caught that wiki/models-catalog.md was advertising
INCLUDE_EXPENSIVE / INCLUDE_ALL_PASSTHROUGHS as workflow_dispatch knobs
— they're not, only `filter` (→ MATRIX_FILTER) is wired through. The
filter coupling already implicitly opens the expensive gate, so dispatch
+ filter is the canonical CI path.
2026-05-09 00:01:30 +00:00
Colin McDonnell c8888cecde bump action version to 0.1.2 2026-05-08 23:37:52 +00:00
Colin McDonnell c0de70431e ci: prune openai/gpt-pro from default models-live matrix (#637)
* ci: prune openai/gpt-pro from default models-live matrix

gpt-5.5-pro burns ~$2.40/run ($30/M input, $180/M output) — flagship
reasoning tier with hidden reasoning tokens dominating cost. Multiplied
by every push that touches a resolution-affecting file, the bill is
untenable for a smoke that just verifies set_output works.

Pruned by default; re-enable with INCLUDE_EXPENSIVE=1 or MATRIX_FILTER
when validating the alias on demand.

Also adds a comment-frugality rule to AGENTS.md.

* ci: include list-aliases.ts in models paths-filter

The matrix builder is resolution-affecting from a validation standpoint
— a regression to it (e.g. accidentally pruning all aliases) wouldn't
trigger models-live on its own commit.
2026-05-08 23:36:26 +00:00
Colin McDonnell b0274e3265 local proxy-key testing via x-dev-repo bypass (#629)
* local proxy-key testing via x-dev-repo bypass

`pnpm play` previously couldn't exercise the proxy/router/oss code path
— `resolveProxyModel` early-exits without OIDC credentials, and
`mintProxyKey` always sends an OIDC bearer to `/api/proxy-token`. since
GitHub Actions OIDC only exists in real workflow runs, billing flows
(auto-reload, balance gates, key rotation, OSS subsidy) had no local
feedback loop.

a server-side dev bypass already exists at `app/api/proxy-token/route.ts`
that accepts an `x-dev-repo: owner/repo` header instead of an OIDC bearer
when `NODE_ENV === "development"`. wire the action side so it sends that
header when there are no OIDC credentials AND `API_URL` resolves to
localhost (i.e. the developer is talking to their own `pnpm dev`
server). production is unreachable through this path because vercel
never sets `NODE_ENV=development`.

document the affordance in `wiki/action-tests.md` so the next person
doesn't have to re-discover it (the server bypass had been sitting
there undocumented since the WIP billing rewrite).

verified end-to-end: `PLAY_LOCAL=1 GITHUB_REPOSITORY=pullfrog/app
API_URL=http://localhost:3100 pnpm play …` now logs `» proxy: dev
bypass (x-dev-repo) for pullfrog/app` → `» proxy: router → openrouter/
anthropic/claude-opus-4.7` → `» model: …(proxy)`, mints a real
OpenRouter key against the dev DB, and the agent runs through the
proxy.

* wiki: cross-reference dev proxy-key affordance from main/e2e/stripe

action-tests.md already documents the localhost+x-dev-repo path; mention
it from the natural discovery points so the next person finds it without
spelunking through git history again:

- main.md: resolveProxyModel row in the dependencies table notes the
  two auth paths (OIDC bearer in prod, x-dev-repo in dev).
- e2e-testing.md: "When to use this" calls out the lighter-weight
  alternative for proxy-only changes.
- stripe.md: new "Loop including the action" subsection in the Dev
  workflow section, alongside the existing dev-script and cron-endpoint
  loops.
2026-05-08 23:35:58 +00:00
Colin McDonnell 8f36eca62a action: use log.success for skill install confirmations 2026-05-08 23:32:20 +00:00
Colin McDonnell 3c9799adda add models-bump cron + drop snapshot test
every 12h, scripts/find-newer-models.ts scans models.dev for newer GA
versions of every alias in action/models.ts and writes a focused
per-alias diff. .github/workflows/models-bump.yml short-circuits when
no candidates exist; otherwise hands the diff to pullfrog/pullfrog@main
to evaluate against the policy in wiki/model-resolution.md and open a
single living PR on the pullfrog/models-bump branch.

drops the brittle "latest model per provider" snapshot block in
action/test/models-catalog.main.test.ts (and its .snap file) — the cron
keeps the registry in sync with upstreams, and the remaining validity
tests act as the integrity gate on the bump PR.
2026-05-08 23:27:42 +00:00
Colin McDonnell 5f3e46c42d fix: don't reuse disabled proxy key on workflow re-runs; non-fatal title-gen errors (#636)
* fix: don't reuse disabled proxy key on workflow re-runs; non-fatal title-gen errors

Three small surgical fixes addressing run https://github.com/pullfrog/app/actions/runs/25580969379:

1. **`/api/proxy-token` idempotency now checks `finalizedAt`.** GitHub re-runs
   share the same `run_id` (only `run_attempt` increments), so attempt N+1's
   action calls /api/proxy-token and inherits attempt N's `proxyKeyId`. The
   `workflow_run.completed` webhook between attempts retires that key on
   OpenRouter (`disableKey`), so attempt N+1 was getting back a disabled key
   and OpenRouter responded with `401 User not found` on every call. Falling
   through when finalized routes through the same billing gate
   (`handleRouterBilling` balance check), so no new attack surface.

2. **OpenCode title-gen / small-model errors no longer fatal.** OpenCode
   auto-spawns a small `agent=title small=true` background call at session
   start to name the thread, defaulting to `anthropic/claude-haiku-4.5`
   (anomalyco/opencode#1243). Pre-fix, the wrapper's `error` event handler
   treated any `type=error` as fatal, so a cosmetic title failure killed the
   run before primary inference even started. Now: stderr matching `small=true`
   sets a one-shot suppression flag for the next stdout `error` event, which
   is logged as a warning instead.

3. **Provider-error classifier puts auth patterns above rate-limit.** OpenRouter
   401 payloads bundle `x-ratelimit-*` response headers, and the loose
   `\brate[_ ]limit/i` pattern was winning. Added 401/403 status, `User not
   found`, `Invalid authentication`, `No auth credentials found` patterns
   ahead of rate-limit. Updated the existing 401-headers regression test to
   assert correct auth classification rather than `null`.

* opencode: correlate small-model error suppression by message, not by next-event

Pullfrog self-review on #636 flagged a real concurrency hole. OpenCode forks
the title-gen call (`session/prompt.ts:1452-1457` via `Effect.forkIn(scope)`)
so it races primary inference. The previous one-shot `suppressNextErrorEvent`
boolean had no per-call correlation: it was consumed by whichever stdout
`type=error` event landed next, regardless of which subagent produced it.
Under concurrent failures, a primary-agent error landing first could be
silently downgraded to a warning while the small-model error then propagated
fatally — the inverse of the bug the suppression was meant to prevent.

Replaced the boolean with a `Set<string>` of pending small-model error
messages. stderr extracts the inner `"message":"..."` from any classified
provider error tagged `small=true`; the stdout `error` handler suppresses
only when `event.error.data.message` matches a pending entry. Set is capped
at 32 entries so a long stream of small-model failures can't wedge memory.

Also corrected the comment that referenced "session summarizer" — verified
in opencode source that summarize() does NOT use `small: true`; only the
title generator does today (only `small: true` match in the codebase).

* revert: drop opencode title-gen suppression

We have no evidence — and can't construct a realistic scenario — where
title-gen fails on an otherwise-successful run. Title-gen and primary share
the same OPENROUTER_API_KEY and hit the same proxy/upstream; whatever breaks
one breaks the other. The original repro on run 25580969379 is fully
explained by the stale proxy key (fix #1) — title-gen happened to be the
first call that surfaced the auth error, but every subsequent primary call
would have died the same way.

Suppression code adds complexity (cross-stream correlation logic, message
matching, set capping) and a real failure mode of its own (a small-model
error with a unique message could mask an unrelated primary error landing
shortly after). Net negative. Removing.
2026-05-08 23:00:41 +00:00
Colin McDonnell 3d393c36a3 opencode: surface subagent events via injected plugin (#634)
* opencode: surface subagent events via injected plugin

opencode's cli/cmd/run.ts event loop filters all message.part.updated
events to the orchestrator's session id (`part.sessionID !== sessionID`
continue), so subagent-internal tool_use / text / step events were
silently discarded by the CLI in --format json mode. opencode plugins,
by contrast, receive every bus event via bus.subscribeAll() regardless
of session.

ship a per-run plugin (action/agents/opencodePlugin.ts) that re-emits
non-orchestrator message.part.updated events as `pullfrog_bus_event`
envelopes on opencode's stdout. the plugin is staged into
<XDG_CONFIG_HOME>/opencode/plugin/pullfrog-events.ts which is already
redirected to ctx.tmpdir — never the user's repo working tree.

the plugin also forwards the orchestrator's task tool dispatch at
state.status="running" — that's the first moment state.input is
populated with description / subagent_type / prompt and it lands
BEFORE the subagent's first message.part.updated. forwarding this
lets SessionLabeler register the lens label early, so subagent
events bind to the correct lens name (e.g. lens:correctness) instead
of the subagent#N fallback. the existing tool_use handler dedupes
on callID so the late status=completed event from the CLI doesn't
double-record.

the parent's pullfrog_bus_event handler synthesizes the equivalent
CLI-style event for each part type (tool/step-start/step-finish/text)
and dispatches through the same handlers used by orchestrator events,
so labeling, tool-call rendering, and the formatWithLabel magenta
prefix all share one code path.

verified end-to-end via `pnpm play --local --raw` with a prompt that
dispatches a reviewfrog subagent: orchestrator's task call now logs
"» dispatching subagent: lens:read-readme-and-report-purpose" before
the subagent runs, the subagent's read tool call surfaces with
[lens:...] magenta prefix, and the run-end "subagent finished"
attribution shows the lens name.

also adds an AGENTS.md rule formalizing the no-write-to-repo
invariant: action runtime must never write into the user's working
tree; auxiliary files go in ctx.tmpdir via HOME / XDG_CONFIG_HOME.

* drop opencodePlugin.test.ts — bullshit-test cleanup

these tests spied on process.stdout.write, loaded the plugin source
into a temp file via dynamic import, and asserted the output strings
matched the plugin source i'd just hand-written. zero unique signal
over the e2e run in preview repo, plus they violate AGENTS.md's
"mocks tend to add ceremony and brittleness" rule. real signal lives
in the e2e: lens label rendering, dispatch attribution, no double
events. if a syntactic regression in the plugin source ever ships,
opencode logs it on plugin load and the e2e fails fast — the unit
tests would catch the same regression no faster.

* remove isPausedExternally — plugin makes it unnecessary

empirical proof from PR #634's e2e debug trace: ~3.3 pullfrog_bus_event
lines per second arrive on the parent's child.stdout pipe during a
typical subagent run. each one fires updateActivity() and resets
lastActivityTime, so the inner spawn activity timer naturally stays
armed-but-not-fired throughout the subagent's lifetime — no suspend
predicate needed.

drop:
- SpawnOptions.isPausedExternally + the check in spawn()'s activity loop
- isSubagentInFlight() in opencode.ts + its callsite
- two isPausedExternally unit tests in subprocess.test.ts

keep:
- killGroup (the actual zombie-prevention fix; still tested)
- the plugin (action/agents/opencodePlugin.ts; the architectural fix)
- everything in opencode.ts that derives lens labels from task dispatches

the only edge case isPausedExternally covered that the plugin doesn't
is a non-streaming provider going silent for >5min during a single
LLM call inside a subagent. that's a provider-behavior question, not
a harness-architecture one — best fixed at the provider level if it
shows up. defense-in-depth that adds indirection is harmful when the
upstream architectural fix is already in place.

* opencode: address review feedback on bus envelope routing

three findings from PR #634 review (2026-05-08T22:13:44Z):

1. token/cost double-count: routing subagent step_finish through the
   orchestrator's handler folded subagent tokens/cost into the run-wide
   accumulators that flow to logTokenTable + AgentUsage. neighbouring
   init/text handlers all gate on ORCHESTRATOR_LABEL for exactly this
   reason. fix: drop step_start AND step_finish from the bus envelope
   handler — those carry orchestrator-scoped state (currentStepId,
   stepHistory, token accumulators) that subagent events shouldn't
   touch. tool calls and text from subagents still surface — that's
   the user-visible activity.

2. subagent tool errors invisible: routed status="error" tool parts
   into handlers.tool_use which only emits "» <tool>(...)" with no
   error indication. fix: extend handlers.tool_use itself to log
   "» tool call failed: <msg>" when state.status==="error". benefits
   the orchestrator path too — opencode CLI also emits failed tool
   calls as tool_use at status=error and we were swallowing the
   failure signal there as well.

3. stale comments + leaked local paths: plugin source had
   /tmp/opencode-investigate/... paths from my local clone, specific
   line numbers from opencode's dev branch that don't match v1.1.56,
   forkDetach claim that's wrong for the pinned version, and JSDoc
   that still listed message.updated/session.error in the forwarded
   set after the runtime filter narrowed to message.part.updated only.
   fix: drop machine-local paths, drop version-fragile line numbers,
   correct the forwarded-set list, generalize the
   "why no @opencode-ai/plugin import" rationale to be version-agnostic.

second review (2026-05-08T22:27:58Z) confirms these are the only
findings still open — no new issues from the isPausedExternally
removal.
2026-05-08 22:46:43 +00:00
Colin McDonnell d6de1c369a learnings: edit-in-place tmpfile (drop update_learnings tool) (#635)
* learnings: edit-in-place tmpfile (drop update_learnings tool)

learnings now follow the PR-summary file pattern: server seeds
`pullfrog-learnings.md` from `Repo.learnings` at startup, agent reads
it as part of context, may edit in place during the post-run reflection
turn, server reads back at end-of-run and PATCHes if changed.

motivation: `update_learnings` required the agent to pass the FULL
merged list as a string parameter — an output-token tax that grew
linearly with the learnings size, and a constant prompt-context
expansion since the contents were also inlined into the LEARNINGS
section. for repos with mature learnings the prompt was getting
visibly noisy in CI logs.

key changes:
- new `action/utils/learnings.ts` (seed/read helpers + 10k cap)
- `main.ts`: always seed; `persistLearnings` mirrors `persistSummary`
  (success path, error path, exit-signal handler, idempotent guard,
  byte-trim equality skip); forwards `model` for `LearningsRevision.model`
- `LEARNINGS` prompt section now contains only the file path + a
  one-line "read it" instruction (no contents inlined)
- `update_learnings` MCP tool deleted; `action/mcp/learnings.ts` removed
- reflection turn (`buildLearningsReflectionPrompt`) reframed around
  file editing with explicit prune-stale + leave-alone-if-nothing-new
  framing
- `learningsStep` removed from every mode checklist — surface lives only
  in the LEARNINGS prompt section + the reflection turn now

* learnings: harden seed step + refresh stale docs (review feedback)

Three findings from PR review, all implemented:

1. wrap learnings seed in best-effort try/catch (action/main.ts) —
   the always-on seed block ran unconditionally and an unwrapped
   `seedLearningsFile` (mkdir + writeFile) failure (ENOSPC, EACCES,
   hostile sandbox) would unwind into the outer main() catch and flip
   an otherwise-successful run to " Pullfrog failed" before the
   agent even started. asymmetric with `persistLearnings`'s own
   best-effort contract. wrap and log on failure; downstream
   consumers (`persistLearnings`, agent harnesses, `resolveInstructions`)
   already handle `learningsFilePath: undefined` cleanly.

2. refresh wiki/main.md — `resolveInstructions` parameter renamed
   from `learnings` to `learningsFilePath` in this PR; the data-flow
   diagram and the resolver dependency table both still showed the
   pre-refactor signature.

3. drop deleted `learnings.ts` from ROADMAP.md + RESEARCH.md
   "missing MCP tool tests" bullets — `action/mcp/learnings.ts` was
   removed in this PR; the bullets are otherwise still accurate.
2026-05-08 22:45:26 +00:00
Colin McDonnell 2e6c01670e mcp: log artifact id after every github write (#633)
makes debugging easier by emitting a single `» <verb> <kind> <id>` line
after every successful GitHub write (and upload) the agent performs via
the Pullfrog MCP, mirroring the chevron convention used elsewhere.
2026-05-08 21:48:28 +00:00
Colin McDonnell 17b610e1a1 bump action version to 0.1.1 2026-05-08 21:32:06 +00:00
Colin McDonnell ca913c76ea spawn: kill process group + heartbeat subagent activity (#631)
* spawn: kill process group + heartbeat subagent activity

two compounding bugs produced zombie agent runs that stalled until the
GitHub-Actions job-level timeout (observed on PR #622, run 25577068620).

1. SIGKILL hit the wrong process. node_modules/opencode-ai/bin/opencode
   is a Node shim that spawnSyncs the native opencode-<plat>-<arch>
   binary with stdio:"inherit". our spawn() ran without detached, so
   child.kill("SIGKILL") killed only the shim. the native binary was
   reparented to PID 1, kept holding our stdout pipe via inherited fds,
   and child.on("close") never fired — leaving the agent promise
   pending past the 5min outer safety-net timer ("agent still pending
   5min after inner activity kill — forcing exit") and the grandchild
   running until the runner timed out.

   fix: SpawnOptions gains killGroup; when set, we spawn detached and
   route all kill paths (timeout, activity timeout, ctrl-c) through
   process.kill(-pid, signal). opencode + claude opt in.

2. inner activity timer false-fired during long task subagents.
   opencode's `task` tool encapsulates subagent execution in-process —
   subagent-internal events don't reach the parent NDJSON stream — so
   the parent looked idle for the full subagent duration even when
   real work was happening, and the 5min DEFAULT_ACTIVITY_TIMEOUT_MS
   would fire mid-subagent.

   fix: SpawnOptions gains externalActivitySource; the timer fires on
   min(local stdout idle, external idle). opencode passes getIdleMs()
   from the global activity tracker and runs a 30s heartbeat
   (markActivity()) while at least one task dispatch is in flight.

action/utils/subprocess.test.ts covers both: a bash+sleep grandchild
that proves close fires <10s with killGroup, and externalActivitySource
keeping the timer armed during 8s of stdout silence.

* opencode: suspend activity timer instead of heartbeat during subagent runs

addresses review on prior commit: replace the 30s markActivity()
heartbeat with a boolean isPausedExternally predicate keyed off
opencode's existing taskDispatchByCallID + pendingTaskDispatches.
no fake activity, no race window between a 30s tick and a subagent
that finishes between ticks.

while the predicate returns true, spawn's activity check skips the
kill decision *and* advances lastActivityTime so a clean unpause
can't fire on a stale baseline. tests cover both the suspended case
(8s of stdout silence + activityTimeout=1s but paused → process
exits cleanly) and the resume case (paused for 500ms then unpaused
→ 30s sleep gets killed by activity timeout as normal).
2026-05-08 21:29:22 +00:00
96 changed files with 5337 additions and 1538 deletions
+4
View File
@@ -11,6 +11,10 @@ permissions:
jobs:
test-token:
# only run in the upstream publish target. forks inherit this file but
# haven't installed the pullfrog github app — running it there 404s our
# token endpoint and pollutes our error logs (see #693).
if: github.repository == 'pullfrog/pullfrog'
runs-on: ubuntu-latest
steps:
- name: Get installation token
+3
View File
@@ -55,6 +55,9 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
OPENCODE_API_KEY: ${{ secrets.OPENCODE_API_KEY }}
AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
AWS_REGION: us-east-1
BEDROCK_MODEL_ID: us.anthropic.claude-opus-4-6-v1
PULLFROG_MODEL: ${{ vars.PULLFROG_MODEL }}
steps:
- uses: actions/checkout@v6
+4 -2
View File
@@ -10,8 +10,10 @@ permissions:
jobs:
trigger:
# skip if pushed by our bot (breaks the loop)
if: github.actor != 'pullfrog[bot]'
# only run in the upstream publish target (forks inherit this file but
# can't dispatch into pullfrog/app), and skip if pushed by our bot (breaks
# the loop).
if: github.repository == 'pullfrog/pullfrog' && github.actor != 'pullfrog[bot]'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
+290 -53
View File
@@ -16,19 +16,26 @@ import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { performance } from "node:perf_hooks";
import { pullfrogMcpName } from "../external.ts";
import { BEDROCK_MODEL_ID_ENV, isBedrockAnthropicId } from "../models.ts";
import { getIdleMs, markActivity } from "../utils/activity.ts";
import { log } from "../utils/cli.ts";
import { formatJsonValue, log } from "../utils/cli.ts";
import { installFromNpmTarball } from "../utils/install.ts";
import { detectProviderError } from "../utils/providerErrors.ts";
import { findProviderErrorMatch } from "../utils/providerErrors.ts";
import { addSkill, installBundledSkills } from "../utils/skills.ts";
import { SPAWN_ACTIVITY_TIMEOUT_CODE, SpawnTimeoutError, spawn } from "../utils/subprocess.ts";
import {
DEFAULT_MAX_RETAINED_BYTES,
SPAWN_ACTIVITY_TIMEOUT_CODE,
SpawnTimeoutError,
spawn,
TailBuffer,
} from "../utils/subprocess.ts";
import { ThinkingTimer } from "../utils/timer.ts";
import type { TodoTracker } from "../utils/todoTracking.ts";
import { getDevDependencyVersion } from "../utils/version.ts";
import { buildLearningsReflectionPrompt, runPostRunRetryLoop } from "./postRun.ts";
import { REVIEWER_AGENT_NAME, REVIEWER_SYSTEM_PROMPT } from "./reviewer.ts";
import { deriveLabelFromTaskInput } from "./sessionLabeler.ts";
import { formatWithLabel, ORCHESTRATOR_LABEL, SessionLabeler } from "./sessionLabeler.ts";
import {
type AgentResult,
type AgentRunContext,
@@ -66,17 +73,27 @@ function writeMcpConfig(ctx: AgentRunContext): string {
/**
* Build the `--agents` JSON definition for the `reviewfrog` subagent.
*
* The Claude Code path always runs against an Anthropic model (see
* resolveAgent), so we hardcode the cheaper-sibling downshift: lenses run
* on Sonnet, the orchestrator stays on whatever model `--model` was passed.
*
* Per-call model override is also possible (Task tool's `model` arg accepts
* 'sonnet' | 'opus' | 'haiku') and takes precedence over what's set here —
* we don't pass it; the per-subagent `model` field is the right default.
*
* The non-mutative + non-recursive contract is enforced by the prose system
* prompt baked into the agent — see action/agents/reviewer.ts for why we no
* longer wire per-agent `disallowedTools` here.
* prompt baked into the agent — see action/agents/reviewer.ts for why we
* no longer wire per-agent `disallowedTools` here.
*/
function buildAgentsJson(): string {
const agents = {
[REVIEWER_AGENT_NAME]: {
description:
"Read-only review subagent for self-review and lens-based code review. " +
"Read-only review subagent for lens-based code review (correctness, security, billing-subsystem, etc.). " +
"Reads only — no writes, no state-changing shell or MCP calls, no nested subagent dispatch.",
prompt: REVIEWER_SYSTEM_PROMPT,
model: "claude-sonnet-4-6",
},
};
return JSON.stringify(agents);
@@ -90,10 +107,13 @@ function stripProviderPrefix(specifier: string): string {
return slashIndex > 0 ? specifier.slice(slashIndex + 1) : specifier;
}
// `max` effort is supported on Opus 4.6 / 4.7; other models fall back to `high`.
// claude-code deny-lists older opus/sonnet generations from `max` at invocation time.
function resolveEffort(model: string | undefined): "max" | "high" {
if (model?.includes("opus")) return "max";
// `high` is the model's tuned default ("equivalent to not setting the parameter"
// per Anthropic docs). `max` is "absolute maximum capability with no constraints
// on token spending" — meaningfully slower and burns more thinking budget per
// turn. We default everyone to `high`; PRs that genuinely need full-send can
// opt in via a future per-run override rather than paying the wall-time cost on
// every Opus run.
function resolveEffort(_model: string | undefined): "high" {
return "high";
}
@@ -111,13 +131,21 @@ interface ContentBlock {
[key: string]: unknown;
}
// SDK schema (per claude-agent-sdk docs) puts `session_id` and
// `parent_tool_use_id` at the top level of every Assistant/User/System/Result
// message, not inside `message`. Subagent events carry a non-null
// `parent_tool_use_id` pointing at the orchestrator's Task/Agent tool_use id.
interface ClaudeSystemEvent {
type: "system";
session_id?: string;
parent_tool_use_id?: string | null;
[key: string]: unknown;
}
interface ClaudeAssistantEvent {
type: "assistant";
session_id?: string;
parent_tool_use_id?: string | null;
message?: {
role?: string;
content?: ContentBlock[];
@@ -135,6 +163,8 @@ interface ClaudeAssistantEvent {
interface ClaudeUserEvent {
type: "user";
session_id?: string;
parent_tool_use_id?: string | null;
message?: {
role?: string;
content?: ContentBlock[];
@@ -146,6 +176,15 @@ interface ClaudeUserEvent {
interface ClaudeResultEvent {
type: "result";
subtype?: string;
// claude CLI sets `is_error: true` (alongside `subtype: "success"`) when
// an upstream provider fails mid-stream. `api_error_status` carries the
// provider HTTP status (e.g. 401 for invalid API key). per the official
// SDK types, `api_error_status` is `number | null`, and the `error_*`
// subtypes carry their actionable payload in `errors: string[]` instead
// of `result`.
is_error?: boolean;
api_error_status?: number | null;
errors?: string[];
result?: string;
session_id?: string;
num_turns?: number;
@@ -203,14 +242,78 @@ type RunParams = {
type ClaudeRunResult = AgentResult & { sessionId?: string | undefined };
async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
/**
* Return the tail of `text` capped at `maxCodeUnits` UTF-16 code units,
* dropping any partial first line. used in the exit-non-zero stdout fallback
* so we never surface a truncated NDJSON event to operators —
* `result.stdout.slice(-2048)` would otherwise cut mid-line and produce a
* syntactically broken JSON fragment. code units rather than bytes because
* `String.prototype.slice` operates on UTF-16 units; for multi-byte UTF-8
* content the effective byte budget can be up to 4× the nominal limit.
*/
function tailLines(text: string, maxCodeUnits: number): string {
if (text.length <= maxCodeUnits) return text;
const tail = text.slice(-maxCodeUnits);
const firstNewline = tail.indexOf("\n");
// if no newline in window or it's at the very start, return as-is;
// otherwise drop the partial first line.
return firstNewline > 0 && firstNewline < tail.length - 1 ? tail.slice(firstNewline + 1) : tail;
}
export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
const startTime = performance.now();
let eventCount = 0;
const thinkingTimer = new ThinkingTimer();
// per-session labeler so parallel subagent log lines can be differentiated.
// claude-agent-sdk runs subagents inside the orchestrator's session — they
// share `session_id` — and stamps every subagent message with a non-null
// `parent_tool_use_id` pointing at the Agent tool_use that spawned them.
// we bind each Agent tool_use id to its dispatched label up front, then
// labelFor short-circuits to the direct mapping when parent_tool_use_id is
// set. orchestrator events (parent_tool_use_id === null) flow through the
// sessionID path and bind to ORCHESTRATOR_LABEL on first sighting.
const labeler = new SessionLabeler();
function eventLabel(event: { session_id?: string; parent_tool_use_id?: string | null }): string {
return labeler.labelFor(event.session_id ?? null, event.parent_tool_use_id ?? null);
}
function withLabel(label: string, message: string): string {
return label === ORCHESTRATOR_LABEL ? message : formatWithLabel(label, message);
}
// one ThinkingTimer per session — sharing a single timer across sessions
// conflated cross-session interleaving as parent thinking time. each timer
// formats its log lines through the session label so attribution is visible.
const thinkingTimers = new Map<string, ThinkingTimer>();
function timerFor(label: string): ThinkingTimer {
let t = thinkingTimers.get(label);
if (!t) {
const formatLine = (line: string) =>
label === ORCHESTRATOR_LABEL ? line : formatWithLabel(label, line);
t = new ThinkingTimer(formatLine);
thinkingTimers.set(label, t);
}
return t;
}
let finalOutput = "";
let sessionId: string | undefined;
let resultErrorSubtype: string | null = null;
// captures the structured error string from a result event with
// `is_error: true` (e.g. mid-stream provider auth failures the CLI
// surfaces as `subtype: "success"` synthetic-stop events, or the
// `errors[]` array from `error_*` subtypes). preferred over raw
// stdout/stderr in the exit-non-zero path so the GitHub Actions
// `##[error]` line shows the actionable message instead of an 8KB+
// NDJSON dump.
let lastResultError: string | null = null;
// set only for synthetic-stop `subtype: "success"` + `is_error: true`
// events, where `accumulatedTokens` from prior `assistant` events is
// stale and logging it would mislead operators into thinking billable
// tokens were spent on a successful turn. deliberately NOT set for
// `error_max_turns` / `error_during_execution` / `error_*` subtypes
// because those runs genuinely consumed tokens and operators need
// billing visibility for them.
let syntheticStopFailure = false;
let accumulatedTokens = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
// Claude CLI reports a single end-of-run `total_cost_usd` on the result
// event. per-message events don't carry cost, so there's nothing to sum —
@@ -234,18 +337,30 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
}
const handlers = {
system: (_event: ClaudeSystemEvent) => {
log.debug(`» ${params.label} system event`);
system: (event: ClaudeSystemEvent) => {
// claude-agent-sdk only emits system:init for the top-level query, so
// this binds the orchestrator label and never appears in subagent flow.
// we still route through eventLabel so a subagent system event (if the
// SDK ever adds one) wouldn't go silently misattributed.
const label = eventLabel(event);
log.debug(withLabel(label, `» ${params.label} system event`));
},
assistant: (event: ClaudeAssistantEvent) => {
const content = event.message?.content;
if (!content) return;
const label = eventLabel(event);
const boxTitle = label === ORCHESTRATOR_LABEL ? params.label : `${params.label} [${label}]`;
for (const block of content) {
if (block.type === "text" && block.text?.trim()) {
const message = block.text.trim();
log.box(message, { title: params.label });
finalOutput = message;
log.box(message, { title: boxTitle });
// only the orchestrator's text becomes the run's "output" — subagent
// report-back text would otherwise clobber the parent's final answer.
if (label === ORCHESTRATOR_LABEL) {
finalOutput = message;
}
} else if (block.type === "tool_use") {
const toolName = block.name || "unknown";
if (params.onToolUse) {
@@ -254,23 +369,34 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
input: block.input,
});
}
thinkingTimer.markToolCall();
log.toolCall({ toolName, input: block.input || {} });
timerFor(label).markToolCall();
const inputFormatted = formatJsonValue(block.input || {});
const toolCallLine =
inputFormatted !== "{}" ? `» ${toolName}(${inputFormatted})` : `» ${toolName}()`;
log.info(withLabel(label, toolCallLine));
// surface the subagent identity when the orchestrator dispatches a
// Task — claude rolls subagent activity up into a single tool_result
// (no per-event session_id in its stream), so this log line is the
// only attribution available before the subagent's report-back.
if (toolName === "Task" && block.input && typeof block.input === "object") {
// when the orchestrator dispatches a subagent, bind the Agent
// tool_use id to the dispatched label so future events carrying
// `parent_tool_use_id === block.id` resolve directly to the right
// lens. v2.1.63+ renamed the tool to "Agent"; older versions
// emitted "Task". match both for forward-compat.
if (
(toolName === "Task" || toolName === "Agent") &&
block.input &&
typeof block.input === "object"
) {
const taskInput = block.input as {
description?: string;
subagent_type?: string;
prompt?: string;
};
const label = deriveLabelFromTaskInput(taskInput);
const dispatchedLabel = labeler.recordTaskDispatch(taskInput, block.id ?? null);
log.info(
`» dispatching subagent: ${label}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
withLabel(
label,
`» dispatching subagent: ${dispatchedLabel}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
)
);
}
@@ -280,8 +406,14 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
params.todoTracker.cancel();
}
// parse TodoWrite events for live progress tracking
if (toolName === "TodoWrite" && params.todoTracker?.enabled) {
// parse TodoWrite events for live progress tracking. only honor the
// orchestrator's todos — subagents emit their own todo lists which
// would otherwise clobber the visible progress comment.
if (
toolName === "TodoWrite" &&
params.todoTracker?.enabled &&
label === ORCHESTRATOR_LABEL
) {
params.todoTracker.update(block.input);
}
}
@@ -302,10 +434,12 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
const content = event.message?.content;
if (!content) return;
const label = eventLabel(event);
for (const block of content) {
if (typeof block === "string") continue;
if (block.type === "tool_result") {
thinkingTimer.markToolResult();
timerFor(label).markToolResult();
const outputContent =
typeof block.content === "string"
@@ -323,9 +457,9 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
: String(block.content);
if (block.is_error) {
log.info(`» tool error: ${outputContent}`);
log.info(withLabel(label, `» tool error: ${outputContent}`));
} else {
log.debug(`» tool output: ${outputContent}`);
log.debug(withLabel(label, `» tool output: ${outputContent}`));
}
}
}
@@ -335,6 +469,27 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
const subtype = event.subtype || "unknown";
const numTurns = event.num_turns || 0;
// claude CLI emits synthetic-stop result events with `subtype: "success"`
// but `is_error: true` when an upstream provider fails mid-stream (e.g.
// 401 from anthropic). short-circuit before the usage/token-table path
// so we don't log a usage table for a failed attempt and so downstream
// (`resultErrorSubtype` branch) surfaces the structured error. gated on
// `subtype === "success"` because the `error_*` subtypes also set
// `is_error: true` but carry their payload in `errors: string[]` and
// are handled by the dedicated branches below.
if (event.is_error === true && subtype === "success") {
const apiStatus = event.api_error_status;
lastResultError =
event.result?.trim() ||
`claude reported is_error=true with no result text (api_error_status=${apiStatus ?? "unknown"})`;
resultErrorSubtype = subtype;
syntheticStopFailure = true;
log.info(
`» ${params.label} result error: subtype=${subtype}, api_error_status=${apiStatus ?? "unknown"}, message=${lastResultError}`
);
return;
}
if (subtype === "success") {
// extract detailed usage from result event (most accurate source).
// note: `input` here is non-cached input tokens only, matching the
@@ -369,12 +524,15 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
}
} else if (subtype === "error_max_turns") {
resultErrorSubtype = subtype;
lastResultError = event.errors?.join("\n").trim() || null;
log.info(`» ${params.label} max turns reached: ${JSON.stringify(event)}`);
} else if (subtype === "error_during_execution") {
resultErrorSubtype = subtype;
lastResultError = event.errors?.join("\n").trim() || null;
log.info(`» ${params.label} execution error: ${JSON.stringify(event)}`);
} else if (subtype.startsWith("error")) {
resultErrorSubtype = subtype;
lastResultError = event.errors?.join("\n").trim() || null;
log.info(`» ${params.label} result: subtype=${subtype}, data=${JSON.stringify(event)}`);
} else {
log.info(`» ${params.label} result: subtype=${subtype}, data=${JSON.stringify(event)}`);
@@ -392,10 +550,18 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
};
const recentStderr: string[] = [];
// ring buffer of recent non-JSON stdout lines. Claude CLI prints
// human-readable TTY chrome (status bubbles, quota notices, etc.)
// alongside the NDJSON event stream. when the CLI exits non-zero without
// emitting a structured error event, these lines are the only actionable
// signal — preferring them over the NDJSON tail keeps progress comments
// readable. issue #643.
const recentNonJsonStdout: string[] = [];
let lastProviderError: string | null = null;
let output = "";
// capped accumulator — see opencode.ts for rationale (issue #680).
const output = new TailBuffer(DEFAULT_MAX_RETAINED_BYTES);
let stdoutBuffer = "";
try {
@@ -407,9 +573,20 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
activityTimeout: 300_000,
onActivityTimeout: params.onActivityTimeout,
stdio: ["ignore", "pipe", "pipe"],
// run claude in its own process group so SIGKILL on activity timeout /
// outer cancellation reaches any subprocesses it spawns (rg, file
// watchers, mcp transports, etc). claude itself is a node bundle so
// there's no shim-orphan issue like opencode-ai/bin/opencode, but
// detached + killGroup is the right default for any agent runtime.
killGroup: true,
// claude already drains every chunk via onStdout (NDJSON parsing) and
// onStderr (recentStderr ring buffer). retaining a second copy in the
// spawn wrapper would grow unbounded for long sessions and previously
// crashed the wrapper with RangeError. see issue #680.
retain: "none",
onStdout: async (chunk) => {
const text = chunk.toString();
output += text;
output.append(text);
markActivity();
stdoutBuffer += text;
@@ -425,6 +602,8 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
event = JSON.parse(trimmed) as ClaudeEvent;
} catch {
log.debug(`» non-JSON stdout line: ${trimmed.substring(0, 200)}`);
recentNonJsonStdout.push(trimmed);
if (recentNonJsonStdout.length > MAX_STDERR_LINES) recentNonJsonStdout.shift();
continue;
}
@@ -460,10 +639,10 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
recentStderr.push(trimmed);
if (recentStderr.length > MAX_STDERR_LINES) recentStderr.shift();
const providerError = detectProviderError(trimmed);
if (providerError) {
lastProviderError = providerError;
log.info(`» provider error detected (${providerError}): ${trimmed.substring(0, 500)}`);
const match = findProviderErrorMatch(trimmed);
if (match) {
lastProviderError = match.label;
log.info(`» provider error detected (${match.label}): ${match.excerpt}`);
} else {
log.debug(trimmed);
}
@@ -490,8 +669,16 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
if (stderrContext) log.info(`» last stderr output:\n${stderrContext}`);
}
// skip the fallback token table only for the synthetic-stop
// `subtype: "success"` + `is_error: true` case: `accumulatedTokens` from
// prior `assistant` events is stale there and logging it would mislead
// operators into thinking billable tokens were spent on a successful turn.
// `error_max_turns` / `error_during_execution` / `error_*` subtypes
// represent runs that genuinely consumed tokens, so they still get the
// table for billing visibility.
if (
!tokensLogged &&
!syntheticStopFailure &&
(accumulatedTokens.input > 0 ||
accumulatedTokens.output > 0 ||
accumulatedTokens.cacheRead > 0 ||
@@ -505,18 +692,39 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
if (result.exitCode !== 0) {
const errorContext = lastProviderError ? ` (${lastProviderError})` : "";
// prefer the structured `lastResultError` (parsed from a result event
// with `is_error: true`) over raw stdout. raw stdout is the full NDJSON
// event stream — dumping it into a GitHub Actions `##[error]` line both
// hides the actionable provider message and pollutes the run log. cap
// the stdout fallback to the last 2KB so it stays readable when neither
// a structured error nor stderr is available.
//
// result.stdout / result.stderr are empty because we pass retain:"none"
// to spawn (see issue #680); the agent layer keeps its own bounded
// mirrors via `output` (TailBuffer) and `recentStderr` (ring buffer).
const stdoutSnapshot = output.toString();
const stderrSnapshot = recentStderr.join("\n");
const truncatedStdout = stdoutSnapshot ? tailLines(stdoutSnapshot, 2048) : "";
// prefer non-JSON stdout (human-readable TTY chrome the CLI prints,
// including status bubbles and quota notices) over the raw NDJSON
// tail. when the CLI exits 1 without emitting `is_error` (issue #643),
// the NDJSON fallback would otherwise dump 2KB of `system/init` events
// into the progress comment with no mention of the actual cause.
const nonJsonStdoutSnapshot = recentNonJsonStdout.join("\n");
const errorMessage =
result.stderr ||
result.stdout ||
lastResultError ||
stderrSnapshot ||
nonJsonStdoutSnapshot ||
truncatedStdout ||
`unknown error - no output from Claude CLI${errorContext}`;
log.error(
`${params.label} exited with code ${result.exitCode}${errorContext}: ${errorMessage}`
);
log.debug(`stdout: ${result.stdout?.substring(0, 500)}`);
log.debug(`stderr: ${result.stderr?.substring(0, 500)}`);
log.debug(`stdout: ${stdoutSnapshot.substring(0, 500)}`);
log.debug(`stderr: ${stderrSnapshot.substring(0, 500)}`);
return {
success: false,
output: finalOutput || output,
output: finalOutput || stdoutSnapshot,
error: errorMessage,
usage,
sessionId,
@@ -526,7 +734,7 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
if (eventCount === 0 && lastProviderError) {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `provider error: ${lastProviderError}`,
usage,
sessionId,
@@ -536,14 +744,14 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
if (resultErrorSubtype) {
return {
success: false,
output: finalOutput || output,
error: `result subtype: ${resultErrorSubtype}`,
output: finalOutput || output.toString(),
error: lastResultError || `result subtype: ${resultErrorSubtype}`,
usage,
sessionId,
};
}
return { success: true, output: finalOutput || output, usage, sessionId };
return { success: true, output: finalOutput || output.toString(), usage, sessionId };
} catch (error) {
params.todoTracker?.cancel();
const duration = performance.now() - startTime;
@@ -569,7 +777,7 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `${errorMessage} [${diagnosis}]`,
usage: buildUsage(),
sessionId,
@@ -637,7 +845,22 @@ export const claude = agent({
const cliPath = await installClaudeCli();
const specifier = ctx.payload.proxyModel ?? ctx.resolvedModel;
const model = specifier ? stripProviderPrefix(specifier) : undefined;
// claude-code on Bedrock takes the bare AWS model ID — no provider prefix
// to strip, since the ID is already in `provider.model` form (e.g.
// `us.anthropic.claude-opus-4-7`). detect via the env-var sentinel: if
// BEDROCK_MODEL_ID is set and matches the resolved specifier, this is a
// bedrock route. see `wiki/model-resolution.md` for the routing pattern.
const bedrockModelId = process.env[BEDROCK_MODEL_ID_ENV]?.trim();
const isBedrockRoute =
specifier !== undefined &&
bedrockModelId !== undefined &&
bedrockModelId === specifier &&
isBedrockAnthropicId(specifier);
const model = !specifier
? undefined
: isBedrockRoute
? specifier
: stripProviderPrefix(specifier);
const homeEnv = {
HOME: ctx.tmpdir,
@@ -684,10 +907,24 @@ export const claude = agent({
// agent process gets full env — needs LLM API keys, PATH, locale, etc.
// security is enforced via managed-settings.json, --disallowedTools (Bash), and MCP tool filtering.
//
// bedrock route: claude-code reads `CLAUDE_CODE_USE_BEDROCK=1` to switch
// its provider implementation from the direct Anthropic API to Bedrock.
// AWS_BEARER_TOKEN_BEDROCK / AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY +
// AWS_REGION are already in process.env from the workflow's `env:` block.
// see https://docs.claude.com/en/docs/claude-code/amazon-bedrock.
//
// we only force CLAUDE_CODE_USE_BEDROCK=1 when this is a Pullfrog-routed
// bedrock run; if the user has set the env var manually for some other
// reason (e.g. always-Bedrock org policy), `...process.env` already
// carries it through and we don't disturb it.
const env: Record<string, string | undefined> = {
...process.env,
...homeEnv,
};
if (isBedrockRoute) {
env.CLAUDE_CODE_USE_BEDROCK = "1";
}
const repoDir = process.cwd();
@@ -716,12 +953,12 @@ export const claude = agent({
// the run). the reflection prompt fires once after gates go clean, as a
// dedicated turn that nudges the agent to persist learnings.
return runPostRunRetryLoop({
ctx,
initialResult: result,
initialUsage: result.usage,
stopScript: ctx.stopScript,
summaryFilePath: ctx.summaryFilePath,
summarySeed: ctx.summarySeed,
reflectionPrompt: buildLearningsReflectionPrompt("claude"),
reflectionPrompt: ctx.toolState.learningsFilePath
? buildLearningsReflectionPrompt(ctx.toolState.learningsFilePath)
: undefined,
canResume: (r) => Boolean(r.sessionId),
resume: async (c) => {
const sessionId = c.previousResult.sessionId;
+422 -97
View File
@@ -12,20 +12,33 @@
* security is enforced at the tool layer, not the process layer.
*/
import { execFileSync } from "node:child_process";
import { mkdirSync } from "node:fs";
import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { performance } from "node:perf_hooks";
import { pullfrogMcpName } from "../external.ts";
import { modelAliases } from "../models.ts";
import { BEDROCK_MODEL_ID_ENV, modelAliases } from "../models.ts";
import type { ToolState } from "../toolState.ts";
import { getIdleMs, markActivity } from "../utils/activity.ts";
import { type AgentDiagnostic, formatAgentHangBody } from "../utils/agentHangReport.ts";
import { formatJsonValue, log } from "../utils/cli.ts";
import { installFromNpmTarball } from "../utils/install.ts";
import { detectProviderError } from "../utils/providerErrors.ts";
import { findProviderErrorMatch } from "../utils/providerErrors.ts";
import { addSkill, installBundledSkills } from "../utils/skills.ts";
import { SPAWN_ACTIVITY_TIMEOUT_CODE, SpawnTimeoutError, spawn } from "../utils/subprocess.ts";
import {
DEFAULT_MAX_RETAINED_BYTES,
SPAWN_ACTIVITY_TIMEOUT_CODE,
SpawnTimeoutError,
spawn,
TailBuffer,
} from "../utils/subprocess.ts";
import { ThinkingTimer } from "../utils/timer.ts";
import type { TodoTracker } from "../utils/todoTracking.ts";
import { getDevDependencyVersion } from "../utils/version.ts";
import {
PULLFROG_BUS_EVENT_TYPE,
PULLFROG_OPENCODE_PLUGIN_FILENAME,
PULLFROG_OPENCODE_PLUGIN_SOURCE,
} from "./opencodePlugin.ts";
import { buildLearningsReflectionPrompt, runPostRunRetryLoop } from "./postRun.ts";
import { REVIEWER_AGENT_NAME, REVIEWER_SYSTEM_PROMPT } from "./reviewer.ts";
import { formatWithLabel, ORCHESTRATOR_LABEL, SessionLabeler } from "./sessionLabeler.ts";
@@ -37,6 +50,7 @@ import {
logTokenTable,
MAX_STDERR_LINES,
} from "./shared.ts";
import { deriveSubagentModels } from "./subagentModels.ts";
async function installOpencodeCli(): Promise<string> {
return await installFromNpmTarball({
@@ -54,26 +68,51 @@ type OpenCodeConfig = {
permission?: Record<string, unknown>;
provider?: Record<string, unknown>;
agent?: Record<string, unknown>;
experimental?: Record<string, unknown>;
model?: string;
enabled_providers?: string[];
[key: string]: unknown;
};
// NOTE: OpenCode's per-call `max_tokens` defaults to 32_000. We previously
// overrode this via `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX = 5000` in #616
// to lower OpenRouter's per-call upfront budget reservation — back when the
// `ROUTER_PER_RUN_LIMIT_USD = 25` per-run key cap meant that reservation was
// a hard gate that could lock low-balance accounts out of starting a run.
//
// That gate is gone (see `app/api/proxy-token/route.ts` ~line 422 — "Per-run
// key budget … is decoupled from wallet balance"); the router now mints
// keys with `keyLimitCents = balance + buffer` ($50 / $5 / $0). The override
// no longer materially helps, and as a hard per-call output truncation it
// actively hurt: a single `create_pull_request_review` tool_use with many
// inline comments would truncate mid-stream past 5K output tokens, the JSON
// was unparseable, and the tool never invoked. We hit this on PR #710's
// verify-downshift PR. Removed in #710 — using OpenCode's 32K default.
//
// If you need to re-cap output for some reason, set
// `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX` in the action env. OpenCode's
// top-level `limit.output` config field has no read site (silently dropped
// on merge in session/llm.ts), so the env var is the only working knob.
/**
* Per-inference `max_tokens` reservation the agent sends to the upstream
* model. OpenCode's default is 32_000 (sized for long-running TUI sessions
* where a human user might want big outputs). Pullfrog runs are headless and
* short — typical outputs are 1-3K tokens — so we cap at 5_000. This
* drastically reduces the upfront budget reservation OpenRouter requires per
* call (~$0.38 vs ~$2.40 for Opus), which is what lets low-wallet runs
* actually start.
* upstream opencode hardcodes `thinkingLevel: "high"` as the default for every
* gemini-3 model on the direct google SDK (`provider/transform.ts` `options()`).
* that adds 30-60s of pre-tool-call TTFT and 5-46s of post-tool jabber per turn,
* which is overkill for agentic loops where most steps are tool-routing
* decisions. we override to "medium" for the curated slugs we ship in
* `action/models.ts`; users who want max quality can still pick the `-high`
* variant explicitly. flash stays at "medium" too — low-effort flash is
* visibly worse on harder tasks and the latency savings aren't meaningful
* (flash is already fast). other gemini-3 ids that exist in models.dev but
* aren't in our curated alias map keep the upstream `"high"` default.
*
* Plumbed via `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX` env var rather than the
* config JSON. OpenCode's `OUTPUT_TOKEN_MAX` (session/llm.ts) is sourced
* exclusively from this env var; top-level `limit.output` in the config
* has no read site and is silently dropped on merge.
* keyed by upstream api id (matches the slugs in `action/models.ts`). the
* merge order in opencode `session/llm.ts` is `base ← model.options ← agent.options ← variant`,
* deep-merged — so an explicit `--variant high` still wins, and explicit
* model.options in a user-provided opencode config would also win.
*/
const PULLFROG_OPENCODE_OUTPUT_LIMIT = 5000;
const GEMINI_3_DIRECT_THINKING_LEVEL = "medium";
const GEMINI_3_DIRECT_API_IDS = ["gemini-3.1-pro-preview", "gemini-3-flash-preview"];
function buildSecurityConfig(ctx: AgentRunContext, model: string | undefined): string {
const config: OpenCodeConfig = {
@@ -88,7 +127,35 @@ function buildSecurityConfig(ctx: AgentRunContext, model: string | undefined): s
mcp: {
[pullfrogMcpName]: { type: "remote", url: ctx.mcpServerUrl },
},
agent: buildReviewerAgentConfig(),
agent: (() => {
const cfg = buildReviewerAgentConfig(model);
const reviewerModel = (cfg[REVIEWER_AGENT_NAME] as { model?: string })?.model ?? "(inherit)";
log.info(`» subagent models: reviewfrog=${reviewerModel}`);
return cfg;
})(),
// opt into opencode's experimental `batch` tool (added in
// anomalyco/opencode PR #2983, opt-in via `experimental.batch_tool`). it
// exposes a single `batch` tool that runs 1-25 independent tool calls
// (read/grep/glob/bash/etc.) concurrently in one assistant turn, which
// collapses the dominant grep→20×read pattern into a single round trip.
// edits are explicitly disallowed inside the batch upstream. paired with
// the "Parallel tool execution" guidance in utils/instructions.ts so the
// model actually reaches for it. see wiki/prompt.md.
experimental: { batch_tool: true },
provider: {
google: {
models: Object.fromEntries(
GEMINI_3_DIRECT_API_IDS.map((id) => [
id,
{
options: {
thinkingConfig: { thinkingLevel: GEMINI_3_DIRECT_THINKING_LEVEL },
},
},
])
),
},
},
};
if (model) {
@@ -104,19 +171,30 @@ function buildSecurityConfig(ctx: AgentRunContext, model: string | undefined): s
}
/**
* Read-only subagent for self-review and /anneal lens dispatch. The
* non-mutative + non-recursive contract is enforced by the prose system
* prompt — see action/agents/reviewer.ts for why we no longer wire per-agent
* tool/permission denies here.
* Read-only `reviewfrog` subagent for lens-based review.
*
* Non-mutative + non-recursive — enforced by the prose system prompt in
* reviewer.ts.
*
* Per-subagent `model:` override is driven by the registry in
* `action/models.ts` via each alias's `subagentModel` field — see
* `deriveSubagentModels` for the reverse-lookup. Currently wired:
* Anthropic opus → sonnet, OpenAI gpt-pro → gpt and gpt → gpt-5.4,
* Google gemini-pro → gemini-flash. Other providers (xai, deepseek,
* moonshot) and already-cheap tiers inherit (no override) — either the
* absolute savings are too small to justify or there's no clean
* cheaper-but-capable sibling.
*/
function buildReviewerAgentConfig(): Record<string, unknown> {
function buildReviewerAgentConfig(orchestratorModel: string | undefined): Record<string, unknown> {
const overrides = deriveSubagentModels(orchestratorModel);
return {
[REVIEWER_AGENT_NAME]: {
description:
"Read-only review subagent for self-review and lens-based code review. " +
"Read-only review subagent for lens-based code review (correctness, security, billing-subsystem, etc.). " +
"Reads only — no writes, no state-changing shell or MCP calls, no nested subagent dispatch.",
mode: "subagent",
prompt: REVIEWER_SYSTEM_PROMPT,
...(overrides.reviewer !== undefined ? { model: overrides.reviewer } : {}),
},
};
}
@@ -154,9 +232,12 @@ function autoSelectModel(cliPath: string): string | undefined {
const availableSet = new Set(availableModels);
if (availableSet.size > 0) {
log.debug(`» opencode models (${availableSet.size}): ${availableModels.join(", ")}`);
// skip hidden aliases (internal subagent-tier targets like opencode/gpt-5.4) —
// they should never surface as a user-facing orchestrator pick. mirrors the
// selectable-list filter in components/ModelSelector.tsx and action/commands/init.ts.
const match =
modelAliases.find((a) => a.preferred && availableSet.has(a.resolve)) ??
modelAliases.find((a) => availableSet.has(a.resolve));
modelAliases.find((a) => !a.hidden && a.preferred && availableSet.has(a.resolve)) ??
modelAliases.find((a) => !a.hidden && availableSet.has(a.resolve));
if (match) {
log.info(
`» model: ${match.resolve} (auto-selected${match.preferred ? " — preferred" : ""} curated match)`
@@ -228,6 +309,20 @@ interface OpenCodeStepFinishEvent {
[key: string]: unknown;
}
/**
* tool-part state, mirroring opencode's `ToolState` (anomalyco/opencode
* `session/message-v2.ts`). error parts carry the reason on `error`,
* completed parts on `output` — reading the wrong field is what caused
* the silent `(no error message)` log in #662.
*
* Named `ToolPartState` locally (not `ToolState`) so it doesn't shadow the
* action-wide `ToolState` imported above.
*/
type ToolPartState =
| { status: "pending" | "running"; input?: unknown }
| { status: "completed"; input?: unknown; output: string }
| { status: "error"; input?: unknown; error: string };
interface OpenCodeToolUseEvent {
type: "tool_use";
timestamp?: number;
@@ -236,7 +331,7 @@ interface OpenCodeToolUseEvent {
id?: string;
callID?: string;
tool?: string;
state?: { status?: string; input?: unknown; output?: string };
state?: ToolPartState;
};
[key: string]: unknown;
}
@@ -245,7 +340,7 @@ interface OpenCodeToolResultEvent {
type: "tool_result";
timestamp?: number;
sessionID?: string;
part?: { callID?: string; state?: { status?: string; output?: string } };
part?: { callID?: string; state?: ToolPartState };
tool_id?: string;
status?: "success" | "error";
output?: string;
@@ -280,6 +375,36 @@ interface OpenCodeErrorEvent {
[key: string]: unknown;
}
/**
* Envelope event emitted by our `.opencode/plugin/pullfrog-events.ts` (the
* source lives in `opencodePlugin.ts`). The plugin subscribes to opencode's
* bus via `bus.subscribeAll()` and re-emits non-orchestrator
* `message.part.updated` events on stdout so subagent activity surfaces here.
*
* `bus_event.properties.part` matches the same `Part` shape that opencode's
* `cli/cmd/run.ts` uses to drive its own emit() calls, so we can route the
* inner part through the existing `tool_use` / `step_start` / `step_finish`
* / `text` handlers by synthesizing the equivalent OpenCode-style event.
*/
interface OpenCodeBusEnvelopeEvent {
type: "pullfrog_bus_event";
bus_event?: {
type?: string;
properties?: {
part?: {
sessionID?: string;
type?: string;
time?: { end?: number | string };
state?: { status?: string };
[key: string]: unknown;
};
[key: string]: unknown;
};
[key: string]: unknown;
};
[key: string]: unknown;
}
type OpenCodeEvent =
| OpenCodeInitEvent
| OpenCodeMessageEvent
@@ -289,7 +414,8 @@ type OpenCodeEvent =
| OpenCodeToolUseEvent
| OpenCodeToolResultEvent
| OpenCodeResultEvent
| OpenCodeErrorEvent;
| OpenCodeErrorEvent
| OpenCodeBusEnvelopeEvent;
// ── runner ──────────────────────────────────────────────────────────────────────
@@ -299,6 +425,7 @@ type RunParams = {
args: string[];
cwd: string;
env: Record<string, string | undefined>;
toolState: ToolState;
todoTracker?: TodoTracker | undefined;
onActivityTimeout?: (() => void) | undefined;
onToolUse?: ((event: { toolName: string; input: unknown }) => void) | undefined;
@@ -307,7 +434,6 @@ type RunParams = {
async function runOpenCode(params: RunParams): Promise<AgentResult> {
const startTime = performance.now();
let eventCount = 0;
const thinkingTimer = new ThinkingTimer();
let finalOutput = "";
let accumulatedTokens = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
@@ -324,15 +450,12 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
// per-session labeler so parallel subagent log lines can be differentiated.
// the orchestrator's task tool_use events seed the labeler; the next
// previously-unseen sessionID consumes the head of the pending-label queue.
// NB: opencode's runtime currently encapsulates subagent execution inside
// the `task` tool — subagent-internal tool_use/tool_result events do not
// surface on the parent's NDJSON stream. The labeler is therefore mostly
// dormant in practice for opencode (no per-event session differentiation
// is needed because there are no per-subagent events). The orchestrator's
// `task` dispatch log (with `description: <lens>`) and the per-task
// duration log below are the actual attribution surface available today.
// The labeler is kept in place defensively so that if/when opencode begins
// streaming subagent sessions, attribution flips on with no further work.
// upstream opencode's `cli/cmd/run.ts` filters subagent events out of its
// NDJSON stream (`part.sessionID !== sessionID`), so we ship a per-run
// plugin (`action/agents/opencodePlugin.ts`, written into the tmpdir at
// setup) that re-emits non-orchestrator `message.part.updated` events. those
// arrive here as `pullfrog_bus_event` envelopes and feed the labeler with
// real data per subagent session.
const labeler = new SessionLabeler();
function eventLabel(event: Record<string, unknown>): string {
const sid = event.sessionID ?? event.session_id;
@@ -342,6 +465,23 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
return label === ORCHESTRATOR_LABEL ? message : formatWithLabel(label, message);
}
// one ThinkingTimer per session — sharing a single timer across sessions
// conflated cross-session interleaving (parent thinks → child tool_call,
// or child returns → parent dispatches next) as parent thinking time. each
// timer formats its log lines through the session label so the "thought
// for X" attribution is visible in the merged stream.
const thinkingTimers = new Map<string, ThinkingTimer>();
function timerFor(label: string): ThinkingTimer {
let t = thinkingTimers.get(label);
if (!t) {
const formatLine = (line: string) =>
label === ORCHESTRATOR_LABEL ? line : formatWithLabel(label, line);
t = new ThinkingTimer(formatLine);
thinkingTimers.set(label, t);
}
return t;
}
// tracks per-task dispatch metadata so the matching tool_result can log a
// labeled "» subagent finished: lens=X duration=Ys" line. this is the most
// useful per-lens observability available given that subagent-internal
@@ -522,25 +662,31 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
// label is already bound); the dispatch label is for the next new
// sessionID that appears.
if (toolName === "task") {
const taskInput = (event.part?.state?.input ?? {}) as {
description?: string;
subagent_type?: string;
prompt?: string;
};
const dispatchedLabel = labeler.recordTaskDispatch(taskInput);
// dual-index by callID (fast path) AND in a FIFO queue (fallback path
// for when opencode's task tool_result carries a different callID).
const dispatch: TaskDispatch = {
label: dispatchedLabel,
startedAt: performance.now(),
toolUseCallID: toolId,
};
taskDispatchByCallID.set(toolId, dispatch);
pendingTaskDispatches.push(dispatch);
log.info(
`» dispatching subagent: ${dispatchedLabel}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
);
// may have been pre-registered via the plugin's early task-dispatch
// announcement (`pullfrog_bus_event` handler). dedupe on callID so
// we don't record the same dispatch twice (which would corrupt the
// FIFO label queue).
if (!taskDispatchByCallID.has(toolId)) {
const taskInput = (event.part?.state?.input ?? {}) as {
description?: string;
subagent_type?: string;
prompt?: string;
};
const dispatchedLabel = labeler.recordTaskDispatch(taskInput);
// dual-index by callID (fast path) AND in a FIFO queue (fallback path
// for when opencode's task tool_result carries a different callID).
const dispatch: TaskDispatch = {
label: dispatchedLabel,
startedAt: performance.now(),
toolUseCallID: toolId,
};
taskDispatchByCallID.set(toolId, dispatch);
pendingTaskDispatches.push(dispatch);
log.info(
`» dispatching subagent: ${dispatchedLabel}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
);
}
} else {
// remember non-task callIDs so a later tool_result with that callID
// is correctly identified as not-a-task (and we don't FIFO-pop a
@@ -561,7 +707,7 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
});
}
thinkingTimer.markToolCall();
timerFor(label).markToolCall();
const inputFormatted = formatJsonValue(event.part?.state?.input || {});
const toolCallLine =
inputFormatted !== "{}" ? `» ${toolName}(${inputFormatted})` : `» ${toolName}()`;
@@ -570,6 +716,14 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
if (event.part?.state?.status === "completed" && event.part.state.output) {
log.debug(withLabel(label, ` output: ${event.part.state.output}`));
}
// surface tool errors at info level. opencode emits tool parts at
// status="error" through the same `tool_use` event the CLI's run-loop
// (and our injected plugin for subagent parts) emits — without this
// branch the only signal in the user's logs is `» <tool>(...)` with
// no indication the call failed.
if (event.part?.state?.status === "error") {
log.info(withLabel(label, `» tool call failed: ${event.part.state.error}`));
}
// agent's explicit MCP report_progress takes priority over todo tracking
if (toolName.includes("report_progress") && params.todoTracker) {
@@ -584,11 +738,17 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
},
tool_result: (event: OpenCodeToolResultEvent) => {
const toolId = event.part?.callID || event.tool_id;
const status = event.part?.state?.status || event.status || "unknown";
const output = event.part?.state?.output || event.output;
const state = event.part?.state;
const status = state?.status ?? event.status ?? "unknown";
const payload =
state?.status === "completed"
? state.output
: state?.status === "error"
? state.error
: event.output;
const label = eventLabel(event);
thinkingTimer.markToolResult();
timerFor(label).markToolResult();
// surface subagent completion at info level — opencode otherwise hides
// per-task timing in debug-only logs, so a parallel multi-lens fan-out
@@ -604,12 +764,12 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
if (taskDispatchByCallID.size > 0 || pendingTaskDispatches.length > 0) {
if (toolId && taskDispatchByCallID.has(toolId)) {
const dispatch = taskDispatchByCallID.get(toolId);
if (dispatch) emitSubagentFinished(dispatch, status, output, "exact");
if (dispatch) emitSubagentFinished(dispatch, status, payload, "exact");
} else {
const callIDIsKnownNonTask = toolId ? knownNonTaskCallIDs.has(toolId) : false;
if (!callIDIsKnownNonTask && pendingTaskDispatches.length > 0) {
const dispatch = pendingTaskDispatches[0]!;
emitSubagentFinished(dispatch, status, output, "fifo");
emitSubagentFinished(dispatch, status, payload, "fifo");
}
}
}
@@ -626,13 +786,8 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
`» ${params.label} tool_result${stepContext}: id=${toolId}, status=${status}, duration=${Math.round(toolDuration)}ms`
)
);
if (output) {
log.debug(
withLabel(
label,
` output: ${typeof output === "string" ? output : JSON.stringify(output)}`
)
);
if (payload) {
log.debug(withLabel(label, ` output: ${payload}`));
}
if (toolDuration > 5000) {
log.info(
@@ -645,11 +800,9 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
}
}
if (status === "error") {
const errorMsg = typeof output === "string" ? output : JSON.stringify(output);
log.info(withLabel(label, `» tool call failed: ${errorMsg}`));
} else if (output) {
const outputStr = typeof output === "string" ? output : JSON.stringify(output);
log.debug(withLabel(label, `tool output: ${outputStr}`));
log.info(withLabel(label, `» tool call failed: ${payload ?? "(no error message)"}`));
} else if (payload) {
log.debug(withLabel(label, `tool output: ${payload}`));
}
},
error: (event: OpenCodeErrorEvent) => {
@@ -690,6 +843,98 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
}
}
},
[PULLFROG_BUS_EVENT_TYPE]: async (event: OpenCodeBusEnvelopeEvent) => {
// surface subagent activity that opencode's CLI run-loop discards (it
// filters `part.sessionID !== sessionID`). our injected plugin
// (action/agents/opencodePlugin.ts) re-emits non-orchestrator
// `message.part.updated` bus events; here we synthesize the equivalent
// CLI-style event for each known part type and dispatch through the
// existing handlers so labeling, attribution, and logging all reuse the
// same code path as the orchestrator's events. mirrors the dispatch
// logic in opencode-ai's `cli/cmd/run.ts` `loop()` function.
const busEvent = event.bus_event;
if (!busEvent || busEvent.type !== "message.part.updated") return;
const part = busEvent.properties?.part;
if (!part || typeof part.sessionID !== "string") return;
const sessionID = part.sessionID;
const partType = part.type;
// early task dispatch: the orchestrator's task tool fires bus events at
// status=running BEFORE the subagent's first message.part.updated, but
// the CLI's run-loop only emits the matching tool_use NDJSON event at
// status=completed (after the subagent finishes). without
// pre-registering the dispatch label here, the labeler binds the
// subagent's sessionID to a generic `subagent#N` fallback before the
// CLI's tool_use ever fires recordTaskDispatch. dedupe against
// taskDispatchByCallID so the late tool_use handler doesn't double-add.
if (partType === "tool") {
const status = part.state?.status;
const partWithToolFields = part as {
tool?: string;
callID?: string;
state?: { status?: string; input?: unknown };
};
// only running (not pending) — at pending state.input is still {}.
// by running, the LLM has filled in description/subagent_type/prompt.
// mirrors the same check in the plugin source.
const isOrchestratorTaskDispatch =
partWithToolFields.tool === "task" && status === "running";
if (isOrchestratorTaskDispatch) {
const callID = partWithToolFields.callID;
if (typeof callID === "string" && !taskDispatchByCallID.has(callID)) {
const taskInput = (partWithToolFields.state?.input ?? {}) as {
description?: string;
subagent_type?: string;
prompt?: string;
};
const dispatchedLabel = labeler.recordTaskDispatch(taskInput);
const dispatch: TaskDispatch = {
label: dispatchedLabel,
startedAt: performance.now(),
toolUseCallID: callID,
};
taskDispatchByCallID.set(callID, dispatch);
pendingTaskDispatches.push(dispatch);
log.info(
`» dispatching subagent: ${dispatchedLabel}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
);
}
return;
}
if (status !== "completed" && status !== "error") return;
await handlers.tool_use({
type: "tool_use",
sessionID,
part,
} as OpenCodeToolUseEvent);
return;
}
// intentionally NOT routing subagent step_start / step_finish through
// the orchestrator's handlers:
// - step_finish carries `tokens` and `cost` and the handler folds
// them into the run-wide accumulators. surfacing subagent steps
// here would inflate the orchestrator's usage telemetry — and
// either double-count (if opencode also bills child tokens back
// up to the parent session) or just over-report. the existing
// init/message/text handlers all gate on ORCHESTRATOR_LABEL for
// the same reason.
// - step_start mutates `currentStepId` / `currentStepType` /
// `stepHistory`, which are orchestrator-scoped — using them to
// attribute subagent activity in the orchestrator's tool-use
// timing log would be wrong.
// the subagent's tool calls and text still surface (handled below)
// — that's the user-visible activity.
if (partType === "step-start" || partType === "step-finish") return;
if (partType === "text" && part.time?.end !== undefined) {
await handlers.text({
type: "text",
sessionID,
part,
} as OpenCodeTextEvent);
return;
}
},
};
const recentStderr: string[] = [];
@@ -697,7 +942,25 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
let lastProviderError: string | null = null;
let agentErrorEvent: OpenCodeErrorEvent | null = null;
let output = "";
// shared with main.ts via toolState. updated in place as events stream and
// stderr accumulates so the outer activity-timeout catch sees the same
// context the harness's own catch path uses to format `result.error`.
// recentStderr is shared by reference; the scalar fields are mirrored on
// each update below.
const diagnostic: AgentDiagnostic = {
label: params.label,
recentStderr,
lastProviderError: undefined,
eventCount: 0,
};
params.toolState.agentDiagnostic = diagnostic;
// capped accumulator for the agent's narration. used as a post-run fallback
// when `finalOutput` (the orchestrator's final assistant message) is empty.
// unbounded `output += text` previously grew to ~1 GiB on multi-lens Reviews
// and contributed to the wrapper-level RangeError. retain:"none" on spawn
// skips the duplicate buffer there; this TailBuffer caps the agent layer.
const output = new TailBuffer(DEFAULT_MAX_RETAINED_BYTES);
let stdoutBuffer = "";
try {
@@ -709,9 +972,28 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
activityTimeout: 300_000,
onActivityTimeout: params.onActivityTimeout,
stdio: ["ignore", "pipe", "pipe"],
// node_modules/opencode-ai/bin/opencode is a Node shim that spawnSyncs
// the native opencode-<plat>-<arch> binary with stdio:"inherit". without
// a process-group kill, SIGKILL hits only the shim, the native binary
// is reparented to PID 1, holds our stdout pipe open, and `child.close`
// never fires — producing zombie runs. detached + killGroup nukes the
// whole tree.
killGroup: true,
// we already drain every chunk via onStdout/onStderr (NDJSON parsing
// + recentStderr ring buffer). retaining a second copy in the spawn
// wrapper would grow unbounded for multi-lens Reviews and previously
// crashed the wrapper with RangeError at ~1 GiB. see issue #680.
retain: "none",
// NB: we used to pass `isPausedExternally: isSubagentInFlight` to suspend
// the activity timer during subagent dispatches. unnecessary now that
// our injected plugin (action/agents/opencodePlugin.ts) re-emits
// subagent `message.part.updated` events on opencode's stdout — those
// arrive at child.stdout here, fire updateActivity(), and reset
// lastActivityTime naturally. verified empirically in PR #634
// (~3.3 plugin events/sec during a typical subagent run).
onStdout: async (chunk) => {
const text = chunk.toString();
output += text;
output.append(text);
markActivity();
stdoutBuffer += text;
@@ -731,6 +1013,7 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
}
eventCount++;
diagnostic.eventCount = eventCount;
log.debug(JSON.stringify(event, null, 2));
const timeSinceLastActivity = getIdleMs();
@@ -769,10 +1052,11 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
recentStderr.push(trimmed);
if (recentStderr.length > MAX_STDERR_LINES) recentStderr.shift();
const providerError = detectProviderError(trimmed);
if (providerError) {
lastProviderError = providerError;
log.info(`» provider error detected (${providerError}): ${trimmed.substring(0, 500)}`);
const match = findProviderErrorMatch(trimmed);
if (match) {
lastProviderError = match.label;
diagnostic.lastProviderError = match.label;
log.info(`» provider error detected (${match.label}): ${match.excerpt}`);
} else {
log.debug(trimmed);
}
@@ -836,22 +1120,31 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
if (result.exitCode !== 0) {
const errorContext = lastProviderError ? ` (${lastProviderError})` : "";
// result.stdout / result.stderr are empty because we pass retain:"none"
// to spawn (see issue #680); use the agent's bounded mirrors instead.
const stdoutSnapshot = output.toString();
const stderrSnapshot = recentStderr.join("\n");
const errorMessage =
result.stderr ||
result.stdout ||
stderrSnapshot ||
stdoutSnapshot ||
`unknown error - no output from OpenCode CLI${errorContext}`;
log.error(
`${params.label} exited with code ${result.exitCode}${errorContext}: ${errorMessage}`
);
log.debug(`stdout: ${result.stdout?.substring(0, 500)}`);
log.debug(`stderr: ${result.stderr?.substring(0, 500)}`);
return { success: false, output: finalOutput || output, error: errorMessage, usage };
log.debug(`stdout: ${stdoutSnapshot.substring(0, 500)}`);
log.debug(`stderr: ${stderrSnapshot.substring(0, 500)}`);
return {
success: false,
output: finalOutput || stdoutSnapshot,
error: errorMessage,
usage,
};
}
if (eventCount === 0 && lastProviderError) {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `provider error: ${lastProviderError}`,
usage,
};
@@ -864,13 +1157,13 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
errorEvent.error?.data?.message || errorEvent.error?.name || JSON.stringify(errorEvent);
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `${errorName}: ${errorMessage}`,
usage,
};
}
return { success: true, output: finalOutput || output, usage };
return { success: true, output: finalOutput || output.toString(), usage };
} catch (error) {
params.todoTracker?.cancel();
const duration = performance.now() - startTime;
@@ -894,10 +1187,11 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
`» recent stderr (last ${Math.min(recentStderr.length, 10)} lines):\n${stderrContext}`
);
const body = formatAgentHangBody({ diagnostic, isHang: isActivityTimeout, errorMessage });
return {
success: false,
output: finalOutput || output,
error: `${errorMessage} [${diagnosis}]`,
output: finalOutput || output.toString(),
error: body ?? `${errorMessage} [${diagnosis}]`,
usage: buildUsage(),
};
}
@@ -911,7 +1205,24 @@ export const opencode = agent({
run: async (ctx) => {
const cliPath = await installOpencodeCli();
const model = ctx.payload.proxyModel ?? ctx.resolvedModel ?? autoSelectModel(cliPath);
const rawModel = ctx.payload.proxyModel ?? ctx.resolvedModel ?? autoSelectModel(cliPath);
// bedrock route: opencode's `amazon-bedrock` provider expects the model
// string in `amazon-bedrock/<bedrock-id>` form. the bare AWS model ID
// (what the user puts in `BEDROCK_MODEL_ID`) needs the prefix added.
// detect via env-var sentinel — same pattern as claude.ts.
//
// we deliberately do NOT gate on `!isBedrockAnthropicId(rawModel)` here:
// Anthropic-on-Bedrock normally routes to claude-code (per `resolveAgent`),
// but `PULLFROG_AGENT=opencode` is the documented escape hatch for forcing
// opencode regardless. when that override fires, opencode still needs the
// `amazon-bedrock/` prefix or the provider lookup fails with
// "Model not found: <modelId>/.". the Anthropic-vs-other discriminant
// only belongs in `resolveAgent`.
const bedrockModelId = process.env[BEDROCK_MODEL_ID_ENV]?.trim();
const isBedrockRoute =
rawModel !== undefined && bedrockModelId !== undefined && bedrockModelId === rawModel;
const model = isBedrockRoute ? `amazon-bedrock/${rawModel}` : rawModel;
const homeEnv = {
HOME: ctx.tmpdir,
@@ -920,6 +1231,20 @@ export const opencode = agent({
mkdirSync(join(homeEnv.XDG_CONFIG_HOME, "opencode"), { recursive: true });
// drop our bus-event surfacing plugin into opencode's global config dir
// (which we've redirected to the per-run tmpdir via XDG_CONFIG_HOME).
// opencode auto-discovers plugins from `<Global.Path.config>/{plugin,plugins}/*.{ts,js}`
// (see `packages/opencode/src/config/config.ts:633` calling
// `ConfigPlugin.load(dir)`), so this lands in the loader without any
// config wiring. critically: this MUST be inside the tmpdir, never the
// user's repo working tree — see AGENTS.md.
const opencodePluginDir = join(homeEnv.XDG_CONFIG_HOME, "opencode", "plugin");
mkdirSync(opencodePluginDir, { recursive: true });
writeFileSync(
join(opencodePluginDir, PULLFROG_OPENCODE_PLUGIN_FILENAME),
PULLFROG_OPENCODE_PLUGIN_SOURCE
);
const agentBrowserVersion = getDevDependencyVersion("agent-browser");
addSkill({
ref: `vercel-labs/agent-browser@v${agentBrowserVersion}`,
@@ -945,7 +1270,6 @@ export const opencode = agent({
...homeEnv,
OPENCODE_CONFIG_CONTENT: buildSecurityConfig(ctx, model),
OPENCODE_PERMISSION: permissionOverride,
OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX: PULLFROG_OPENCODE_OUTPUT_LIMIT.toString(),
GOOGLE_GENERATIVE_AI_API_KEY:
process.env.GOOGLE_GENERATIVE_AI_API_KEY || process.env.GEMINI_API_KEY,
};
@@ -960,6 +1284,7 @@ export const opencode = agent({
cliPath,
cwd: repoDir,
env,
toolState: ctx.toolState,
todoTracker: ctx.todoTracker,
onActivityTimeout: ctx.onActivityTimeout,
onToolUse: ctx.onToolUse,
@@ -976,12 +1301,12 @@ export const opencode = agent({
// the reflection prompt fires once after gates go clean, as a dedicated
// turn that nudges the agent to persist learnings.
return runPostRunRetryLoop({
ctx,
initialResult: result,
initialUsage: result.usage,
stopScript: ctx.stopScript,
summaryFilePath: ctx.summaryFilePath,
summarySeed: ctx.summarySeed,
reflectionPrompt: buildLearningsReflectionPrompt("opencode"),
reflectionPrompt: ctx.toolState.learningsFilePath
? buildLearningsReflectionPrompt(ctx.toolState.learningsFilePath)
: undefined,
resume: async (c) =>
runOpenCode({
...runParams,
+139
View File
@@ -0,0 +1,139 @@
/**
* Source for the opencode plugin we drop into the per-run tmpdir at
* `<XDG_CONFIG_HOME>/opencode/plugin/pullfrog-events.ts`. The harness already
* redirects `XDG_CONFIG_HOME` to `ctx.tmpdir/.config` (see `opencode.ts`
* `homeEnv`), so opencode's auto-discovery scans the tmpdir, never the user's
* working tree. opencode's `Global.Path.config` resolves to
* `path.join(xdgConfig, "opencode")` and the config layer auto-discovers
* plugins from every directory in its scan list — including
* `Global.Path.config` — by globbing `{plugin,plugins}/*.{ts,js}` via
* `ConfigPlugin.load(dir)`.
*
* We MUST NOT write into the user's repo working tree. The repo is a checkout
* the agent operates on; only the agent's own tools (gated by
* `OPENCODE_PERMISSION`) may modify it. The whole reason we redirect HOME and
* XDG_CONFIG_HOME is so harness-side files (config, plugins, scratch state)
* land in the tmpdir.
*
* Why this plugin exists: opencode's `task` tool runs subagents in-process and
* the CLI's `cli/cmd/run.ts` event loop filters `part.sessionID !== sessionID`,
* so subagent-internal `message.part.updated` events are silently discarded
* before reaching our parent NDJSON stream. plugins, by contrast, receive
* EVERY bus event via `bus.subscribeAll()` regardless of session.
*
* The plugin re-emits every relevant bus event onto opencode's stdout as a
* single JSON line wrapped in a sentinel envelope. our `runOpenCode` parser
* recognises the envelope, unpacks it, and routes the inner part through the
* existing handlers with a per-session label from `SessionLabeler` so each
* subagent's tool calls / text appear inline alongside the orchestrator's.
*
* Dumb plugin / smart parent split: the plugin emits every part for every
* session. the parent dedupes against the orchestrator's own session id (which
* it already knows from the `init` event). this keeps the plugin trivial and
* keeps the per-session attribution logic on the parent side where the
* SessionLabeler already lives.
*
* Event-name prefixing: the wrapped event-type sentinel is
* `pullfrog_bus_event` — picked to be unmistakably ours so a future opencode
* release that introduces a coincidentally-named event type won't collide.
*/
export const PULLFROG_BUS_EVENT_TYPE = "pullfrog_bus_event" as const;
export const PULLFROG_OPENCODE_PLUGIN_FILENAME = "pullfrog-events.ts" as const;
/**
* Source written verbatim to `<XDG_CONFIG_HOME>/opencode/plugin/pullfrog-events.ts`.
*
* - Structural typing only (no runtime import of `@opencode-ai/plugin`):
* opencode installs that dep into the directory containing the plugin
* alongside discovery, but a) the dep isn't required for the structural
* shape we use, and b) keeping zero imports avoids any module-resolution
* coupling to opencode's plugin-loader internals across versions.
* - default export is the plugin factory (opencode's plugin loader accepts
* default exports as the server entrypoint).
* - we only forward `message.part.updated`. that's where the user-visible
* subagent activity (tool calls, text, step transitions) lives. add more
* event types here if the parent needs them.
* - JSON.stringify+single write keeps the line atomic up to PIPE_BUF (4KB on
* Linux). longer parts may interleave with concurrent stdout writers; the
* parser tolerates non-JSON lines (logs them at debug) so a torn line is a
* missed event, not a crash.
*/
export const PULLFROG_OPENCODE_PLUGIN_SOURCE = `// AUTOGENERATED by Pullfrog. do not edit; it'll be overwritten on the next run.
// surfaces opencode subagent activity that the CLI's run-loop discards. see
// action/agents/opencodePlugin.ts in pullfrog/app for why this exists. lives
// inside the per-run tmpdir (XDG_CONFIG_HOME/opencode/plugin/), never inside
// the user's working tree.
const PULLFROG_BUS_EVENT_TYPE = ${JSON.stringify(PULLFROG_BUS_EVENT_TYPE)};
// the first sessionID we see on a message.part.updated event is the
// orchestrator — opencode's run command creates exactly one top-level session
// before any subagent is dispatched, and the user-prompt text part fires
// before the first task tool_use. we lock that sessionID in here and use it
// to filter: the orchestrator's events are already streamed by the CLI's
// run-loop, so we only forward (a) all subagent events, and (b) the
// orchestrator's task tool dispatches at status="running". the CLI only
// emits task tool_use at status=completed (after the subagent finishes), so
// without the early announce the parent's labeler binds subagent sessions
// before recordTaskDispatch fires and the lens label is lost.
let orchestratorSessionID: string | undefined;
function isOrchestratorTaskDispatch(part: {
type?: string;
tool?: string;
state?: { status?: string };
}): boolean {
if (part.type !== "tool") return false;
if (part.tool !== "task") return false;
// only forward at status="running" (not "pending"). at pending the
// state.input is still {} — the orchestrator has emitted the part shell
// but the LLM hasn't filled in description/subagent_type/prompt yet. by
// running, input is populated and recordTaskDispatch can derive the lens
// label correctly.
return part.state?.status === "running";
}
export default async function pullfrogEventsPlugin() {
return {
event: async (input: {
event: {
type: string;
properties?: {
part?: {
sessionID?: string;
type?: string;
tool?: string;
state?: { status?: string };
};
};
};
}) => {
const event = input.event;
if (!event || typeof event !== "object") return;
if (event.type !== "message.part.updated") return;
const part = event.properties?.part;
const sessionID = part?.sessionID;
if (typeof sessionID !== "string" || sessionID.length === 0) return;
if (orchestratorSessionID === undefined) orchestratorSessionID = sessionID;
if (sessionID === orchestratorSessionID) {
// skip orchestrator events EXCEPT early task dispatches.
if (!part || !isOrchestratorTaskDispatch(part)) return;
}
try {
const line = JSON.stringify({
type: PULLFROG_BUS_EVENT_TYPE,
bus_event: event,
});
process.stdout.write(line + "\\n");
} catch {
// a circular reference or BigInt etc. would throw; swallow rather
// than letting a single bad event take down the plugin.
}
},
};
}
`;
+46 -412
View File
@@ -1,429 +1,63 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { SPAWN_TIMEOUT_CODE, SpawnTimeoutError } from "../utils/subprocess.ts";
import type { AgentResult } from "./shared.ts";
import { describe, expect, it } from "vitest";
import type { ToolState } from "../toolState.ts";
import { getUnsubmittedReview } from "./postRun.ts";
vi.mock("./shared.ts", async (importOriginal) => {
const actual = await importOriginal<typeof import("./shared.ts")>();
function makeToolState(overrides: Partial<ToolState> = {}): ToolState {
return {
...actual,
getGitStatus: vi.fn(() => ""),
progressComment: undefined,
hadProgressComment: true,
backgroundProcesses: new Map(),
usageEntries: [],
...overrides,
};
});
}
vi.mock("../utils/subprocess.ts", async (importOriginal) => {
const actual = await importOriginal<typeof import("../utils/subprocess.ts")>();
return {
...actual,
spawn: vi.fn(),
};
});
const { runPostRunRetryLoop, executeStopHook } = await import("./postRun.ts");
const { getGitStatus } = await import("./shared.ts");
const { spawn } = await import("../utils/subprocess.ts");
const mockedGetGitStatus = vi.mocked(getGitStatus);
const mockedSpawn = vi.mocked(spawn);
const successResult = (overrides: Partial<AgentResult> = {}): AgentResult => ({
success: true,
output: "ok",
...overrides,
});
describe("runPostRunRetryLoop — reflection turn", () => {
beforeEach(() => {
mockedGetGitStatus.mockReset();
mockedGetGitStatus.mockReturnValue("");
mockedSpawn.mockReset();
describe("getUnsubmittedReview", () => {
it("returns null when mode is not a review mode", () => {
expect(getUnsubmittedReview(makeToolState({ selectedMode: "Build" }))).toBeNull();
expect(getUnsubmittedReview(makeToolState())).toBeNull();
});
afterEach(() => {
vi.restoreAllMocks();
it("returns null when a review was already submitted", () => {
expect(
getUnsubmittedReview(
makeToolState({
selectedMode: "Review",
review: { id: 1, nodeId: "n", reviewedSha: undefined },
})
)
).toBeNull();
});
it("does not flip a successful run to failed when reflection returns success:false", async () => {
// the reflection turn is a best-effort nudge (update_learnings). if it
// fails — e.g. the model API errors mid-turn — the underlying task has
// already completed and been gated cleanly, so the run as a whole must
// still be reported as successful.
const initial = successResult({ output: "task done" });
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue({ success: false, error: "model API transient failure" });
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: call update_learnings if anything is worth saving",
});
expect(result.success).toBe(true);
expect(result.output).toBe("task done");
expect(result.error).toBeUndefined();
expect(resume).toHaveBeenCalledTimes(1);
expect(resume.mock.calls[0]?.[0].prompt).toMatch(/REFLECTION/);
it("fires for Review even when report_progress wrote a final summary", () => {
// Review's only valid exit is `create_pull_request_review`. a summary
// comment is not a substitute, and accepting it here previously let
// subagent-flipped `finalSummaryWritten` silence the gate.
expect(
getUnsubmittedReview(makeToolState({ selectedMode: "Review", finalSummaryWritten: true }))
).toBe("Review");
});
it("still aggregates usage from a failed reflection turn", async () => {
// the reflection consumed tokens even if it didn't produce useful output;
// the run total must reflect that so billing/reporting stays accurate.
const initial = successResult({
usage: { agent: "claude", inputTokens: 100, outputTokens: 50 },
});
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue({
success: false,
error: "model API transient failure",
usage: { agent: "claude", inputTokens: 10, outputTokens: 5 },
});
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: initial.usage,
stopScript: null,
resume,
reflectionPrompt: "reflect",
});
expect(result.success).toBe(true);
expect(result.usage?.inputTokens).toBe(110);
expect(result.usage?.outputTokens).toBe(55);
it("returns null for IncrementalReview when report_progress wrote a final summary", () => {
// IncrementalReview treats `report_progress` as a legitimate
// "no review warranted" exit, matching the post-failure error message.
expect(
getUnsubmittedReview(
makeToolState({ selectedMode: "IncrementalReview", finalSummaryWritten: true })
)
).toBeNull();
});
it("falls back to the reflection's output when the pre-reflection output is empty", async () => {
// the preservation fix must only kick in when the task actually produced
// meaningful output. runs that communicate exclusively through MCP tools
// (e.g. report_progress) leave result.output = "" — using `??` here kept
// the empty string and dropped the reflection's reply, leaving the
// fallback `handleAgentResult` path with nothing to show. prefer the
// reflection's output (even a trivial "done") over no output at all.
const initial = successResult({ output: "" });
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult({ output: "done" }));
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: consider update_learnings",
});
expect(result.success).toBe(true);
expect(result.output).toBe("done");
it("returns null when there is no progress comment to anchor the failure to", () => {
expect(
getUnsubmittedReview(makeToolState({ selectedMode: "Review", hadProgressComment: false }))
).toBeNull();
});
it("preserves the pre-reflection task output when a trivial reflection ('done') succeeds", async () => {
// the reflection turn is a meta-ask — its literal reply ("done" or a
// short "updated learnings with N bullets") is not the task summary the
// caller wants to see. before this fix, `result = reflectionResult`
// clobbered the task's output on the returned AgentResult, so downstream
// consumers (handleAgentResult's fallback path when toolState is empty,
// programmatic callers of main()) saw "done" instead of the real
// summary. assert the task's output survives a successful reflection.
const initial = successResult({ output: "Implemented feature X; tests pass; pushed PR #42" });
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult({ output: "done" }));
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: consider update_learnings",
});
expect(result.success).toBe(true);
expect(result.output).toBe("Implemented feature X; tests pass; pushed PR #42");
});
it("skips reflection entirely when canResume returns false", async () => {
const initial = successResult();
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult());
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: null,
resume,
canResume: () => false,
reflectionPrompt: "reflect",
});
expect(result.success).toBe(true);
expect(resume).not.toHaveBeenCalled();
});
it("catches a reflection turn that dirties the tree via the dirty-tree gate on the next iteration", async () => {
// PR claims: "if the reflection turn dirties the tree, the loop picks
// that up on the next iteration via the normal dirty-tree gate." lock
// it in — without this invariant the reflection prompt could bypass the
// commit-before-you-finish contract whenever the agent misbehaves.
//
// three getGitStatus calls in sequence:
// 1. clean (triggers reflection)
// 2. reflection left the tree dirty
// 3. retry committed the changes — now clean, loop exits
mockedGetGitStatus
.mockReturnValueOnce("")
.mockReturnValueOnce(" M scratch/notes.md")
.mockReturnValueOnce("");
const initial = successResult();
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult({ output: "resumed" }));
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: consider update_learnings",
});
expect(result.success).toBe(true);
// call 0: reflection; call 1: dirty-tree retry
expect(resume).toHaveBeenCalledTimes(2);
expect(resume.mock.calls[0]?.[0].prompt).toContain("REFLECTION");
expect(resume.mock.calls[1]?.[0].prompt).toContain("UNCOMMITTED CHANGES");
expect(resume.mock.calls[1]?.[0].prompt).toContain("scratch/notes.md");
});
it("surfaces a persistent stop hook failure as AgentResult.error after MAX_POST_RUN_RETRIES", async () => {
// PR test plan item #1: "confirm the agent is resumed with the hook
// output and the run fails after 3 attempts if never resolved."
//
// stop the hook from passing on every invocation, have `resume` always
// return success (the agent tried but couldn't fix the issue), and
// verify: (a) the loop exhausts all retries, (b) the final result is
// success=false, (c) the error mentions the retry count and the hook
// output verbatim so the GitHub comment surfaces what actually failed.
const hookFailure = {
stdout: "lint: 3 issues in src/foo.ts",
stderr: "",
exitCode: 7,
durationMs: 5,
};
mockedSpawn.mockResolvedValue(hookFailure);
const initial = successResult({ output: "agent done" });
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult({ output: "retry done" }));
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: "pnpm lint",
resume,
reflectionPrompt: undefined,
});
expect(result.success).toBe(false);
expect(result.error).toContain("stop hook failed");
expect(result.error).toContain("exit code 7");
expect(result.error).toContain("3 retry attempts");
expect(result.error).toContain("lint: 3 issues in src/foo.ts");
// each retry feeds the hook output back into the agent as the resume prompt
expect(resume).toHaveBeenCalledTimes(3);
for (const call of resume.mock.calls) {
expect(call[0].prompt).toContain("STOP HOOK FAILED");
expect(call[0].prompt).toContain("lint: 3 issues in src/foo.ts");
}
});
it("treats a persistently dirty tree (no stop hook failure) as a soft-fail", async () => {
// the PR documents: "dirty-tree-only failures preserve prior behavior:
// they're logged but don't fail the run." a regression that started
// surfacing dirty-tree as AgentResult.error would make every run that
// leaves untracked test fixtures around fail spuriously. guard it.
mockedGetGitStatus.mockReturnValue(" M src/foo.ts");
const initial = successResult();
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult({ output: "tried but tree still dirty" }));
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: null,
resume,
});
expect(result.success).toBe(true);
expect(result.error).toBeUndefined();
// retries were attempted (the loop fed the dirty-tree prompt back to the agent)
expect(resume).toHaveBeenCalledTimes(3);
for (const call of resume.mock.calls) {
expect(call[0].prompt).toContain("UNCOMMITTED CHANGES");
}
});
it("surfaces a stop hook failure even when canResume is false (no retry budget, still fails the run)", async () => {
// the retry loop is best-effort. when canResume says no (e.g. claude
// without a sessionId), we still need the failure gate to fire so the
// user sees WHY the run failed instead of an opaque success. covers the
// "checks still ran even if we can't resume" comment in postRun.ts.
mockedSpawn.mockResolvedValue({
stdout: "typecheck: 2 errors",
stderr: "",
exitCode: 1,
durationMs: 1,
});
const initial = successResult();
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult());
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: "pnpm typecheck",
resume,
canResume: () => false,
});
expect(result.success).toBe(false);
expect(result.error).toContain("stop hook failed");
expect(result.error).toContain("typecheck: 2 errors");
// no retries were attempted because canResume said no — error lists no
// retry count (that would be a lie).
expect(result.error).not.toContain("retry attempt");
expect(resume).not.toHaveBeenCalled();
});
it("short-circuits the loop when the initial result is already failed", async () => {
// if the agent already failed (timeout, model error) there's no point
// running gates or a reflection — the run is toast. preserve the original
// error verbatim so triage is straightforward.
const initial: AgentResult = {
success: false,
error: "agent died mid-turn",
output: "partial",
};
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue(successResult());
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: undefined,
stopScript: "pnpm lint",
resume,
reflectionPrompt: "reflect",
});
expect(result.success).toBe(false);
expect(result.error).toBe("agent died mid-turn");
expect(resume).not.toHaveBeenCalled();
expect(mockedSpawn).not.toHaveBeenCalled();
expect(mockedGetGitStatus).not.toHaveBeenCalled();
});
it("aggregates usage across every gate retry", async () => {
// billing/reporting rely on the usage summary reflecting the full run,
// not just the final retry's slice. regression gate.
mockedSpawn.mockResolvedValue({
stdout: "fail",
stderr: "",
exitCode: 1,
durationMs: 1,
});
const initial = successResult({
usage: { agent: "claude", inputTokens: 100, outputTokens: 50 },
});
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
.mockResolvedValue({
success: true,
output: "retry",
usage: { agent: "claude", inputTokens: 10, outputTokens: 5 },
});
const result = await runPostRunRetryLoop({
initialResult: initial,
initialUsage: initial.usage,
stopScript: "flaky",
resume,
});
// 100 initial + 10 * 3 retries = 130
expect(result.usage?.inputTokens).toBe(130);
expect(result.usage?.outputTokens).toBe(65);
});
});
describe("executeStopHook — output capture", () => {
beforeEach(() => {
mockedSpawn.mockReset();
});
afterEach(() => {
vi.restoreAllMocks();
});
it("includes both stdout and stderr in the failure output when both are populated", async () => {
// hooks that wrap other tools commonly emit a benign warning to stderr
// (e.g. "config file not found, using defaults") and the actionable error
// to stdout. a `(stderr || stdout)` heuristic drops stdout entirely
// whenever stderr is non-empty, starving the agent of the information it
// needs to fix the issue.
mockedSpawn.mockResolvedValue({
stdout: "ERROR: lint check failed at path/to/file.ts:42",
stderr: "Warning: config file not found, using defaults",
exitCode: 1,
durationMs: 5,
});
const failure = await executeStopHook("run-lint");
expect(failure).not.toBeNull();
expect(failure?.output).toContain("ERROR: lint check failed at path/to/file.ts:42");
expect(failure?.output).toContain("Warning: config file not found, using defaults");
});
it("returns null (treated as passed) when spawn throws a timeout", async () => {
// infra-level failures can't be fixed by the agent. surfacing them as a
// hook failure would put the loop into a retry cycle that never
// terminates. soft-fail and let the run succeed.
mockedSpawn.mockRejectedValue(
new SpawnTimeoutError("hook exceeded 10 minutes", SPAWN_TIMEOUT_CODE)
it("returns the selected mode when the gate should fire", () => {
expect(getUnsubmittedReview(makeToolState({ selectedMode: "Review" }))).toBe("Review");
expect(getUnsubmittedReview(makeToolState({ selectedMode: "IncrementalReview" }))).toBe(
"IncrementalReview"
);
const failure = await executeStopHook("slow-hook");
expect(failure).toBeNull();
});
it("returns null (treated as passed) on spawn ENOENT (command not found)", async () => {
// if the user misconfigures the hook (wrong binary, typo), the spawn
// itself throws. same rationale as timeouts: soft-fail, don't retry.
mockedSpawn.mockRejectedValue(
Object.assign(new Error("spawn nosuchbin ENOENT"), { code: "ENOENT" })
);
const failure = await executeStopHook("nosuchbin");
expect(failure).toBeNull();
});
it("truncates oversize output, keeping the tail", async () => {
// the error is embedded in AgentResult.error and flows into GitHub
// comments (65535-char cap). the 4096-char truncation is our guardrail;
// lock it in so a well-meaning refactor can't blow the comment budget.
const longTail = "LAST_LINE_IS_ACTIONABLE";
const longOutput = "x".repeat(10_000) + longTail;
mockedSpawn.mockResolvedValue({
stdout: longOutput,
stderr: "",
exitCode: 1,
durationMs: 1,
});
const failure = await executeStopHook("noisy");
expect(failure?.output).toContain(longTail);
expect(failure?.output).toContain("truncated");
expect(failure?.output.length).toBeLessThan(longOutput.length);
});
});
+184 -59
View File
@@ -1,6 +1,7 @@
import { readFile } from "node:fs/promises";
import { type AgentId, formatMcpToolRef } from "../external.ts";
import { LIFECYCLE_HOOK_TIMEOUT_MS } from "../lifecycle.ts";
import { NON_COMMITTING_MODES } from "../modes.ts";
import type { ToolState } from "../toolState.ts";
import { log } from "../utils/cli.ts";
import {
SPAWN_ACTIVITY_TIMEOUT_CODE,
@@ -10,6 +11,7 @@ import {
} from "../utils/subprocess.ts";
import {
type AgentResult,
type AgentRunContext,
type AgentUsage,
buildCommitPrompt,
getGitStatus,
@@ -20,6 +22,36 @@ import {
type StopHookFailure,
} from "./shared.ts";
/**
* derive "agent picked a review mode but never produced visible output" from
* the literal facts on `toolState`. returns the selected mode when the gate
* should fire, `null` otherwise — pure read, no side effects, safe to invoke
* after every agent attempt.
*
* the gate is anchored to `hadProgressComment` so silent runs (non-issue
* events, dispatcher skipped seeding) don't fire a nudge there's no UI for.
*
* `Review` and `IncrementalReview` have different valid exits:
* - Review: only `create_pull_request_review` counts. `report_progress` is
* not a substitute — a Review run that exits with just a summary comment
* has produced nothing reviewable on the PR. matches the hard-fail
* message at `expected = "create_pull_request_review"` below.
* - IncrementalReview: `report_progress` is a legitimate "no review
* warranted" exit, so either toolState flag short-circuits.
* splitting per mode also closes the bypass where a subagent (e.g. a
* `task`-dispatched `reviewfrog` lens) calls `report_progress` and silences
* the gate even though the orchestrator never submitted a review.
*/
export function getUnsubmittedReview(toolState: ToolState): "Review" | "IncrementalReview" | null {
const mode = toolState.selectedMode;
if (!toolState.hadProgressComment) return null;
if (mode === "Review") return toolState.review ? null : "Review";
if (mode === "IncrementalReview") {
return toolState.review || toolState.finalSummaryWritten ? null : "IncrementalReview";
}
return null;
}
/**
* hook output can flow into two size-sensitive places: the LLM resume prompt
* (context window) and AgentResult.error (surfaced in GitHub comments capped
@@ -116,67 +148,150 @@ export function buildSummaryStalePrompt(filePath: string): string {
].join("\n");
}
export function buildUnsubmittedReviewPrompt(mode: "Review" | "IncrementalReview"): string {
// mode-aware: Review mode's contract is "always submit one review" — its
// mode prompt forbids `report_progress`, so the nudge here must not offer
// it as an exit. IncrementalReview legitimately allows a report_progress
// exit when there are no new issues since the last review (mode prompt
// step 8), so the nudge mirrors that contract.
if (mode === "Review") {
return [
`MISSING REVIEW OUTPUT — you selected Review mode but stopped without calling \`create_pull_request_review\`. the user has no visible signal that this run produced anything; the progress comment will be deleted on exit and no review will appear on the PR.`,
"",
"call `create_pull_request_review` now with your aggregated review (body + inline comments). pick the tier per the mode prompt — Review mode has no no-submit exit, so even informational `> [!NOTE]` reviews and `No new issues found.` reviews must be submitted (both use `approved: true`). the first call may error once with a diff-coverage nudge — retry the same call to proceed.",
"",
"do NOT stop again until `create_pull_request_review` has been called successfully.",
].join("\n");
}
return [
`MISSING REVIEW OUTPUT — you selected IncrementalReview mode but stopped without calling \`create_pull_request_review\` or \`report_progress\`. the user has no visible signal that this run produced anything; the progress comment will be deleted on exit and no review will appear on the PR.`,
"",
"do exactly one of:",
"- if you have findings: call `create_pull_request_review` now with your aggregated review (body + inline comments). the first call may error once with a diff-coverage nudge — retry the same call to proceed.",
"- if there are genuinely no actionable findings since the last review (e.g. only formatting / comment / lockfile changes): call `report_progress` with a 1-2 sentence summary explaining that no review was warranted.",
"",
"do NOT stop again until one of those tools has been called successfully.",
].join("\n");
}
/**
* check the post-run gates: did the stop hook pass, is the working tree
* clean, and (when applicable) did the agent touch the rolling PR summary
* snapshot? returns everything that still needs nudging so the caller can
* render a single combined resume prompt.
* snapshot or produce review output? returns everything that still needs
* nudging so the caller can render a single combined resume prompt.
*
* the summary-stale check is skipped when `summaryFilePath` / `summarySeed`
* are not provided; this is the common case (non-PR runs, runs where the
* dispatcher didn't request snapshot generation, runs where the seed step
* failed). loop callers also pass these as undefined after the agent has
* already been nudged once, to avoid burning the retry budget on a soft
* non-blocking gate.
* reads run state directly off `ctx.toolState` so each invocation sees the
* latest mutations from MCP tool calls. `skipSummaryStale` lets the loop
* suppress the summary-stale check after the one-shot nudge has been
* delivered (re-firing it would burn the retry budget on a soft gate the
* agent has already decided not to act on).
*/
export async function collectPostRunIssues(params: {
stopScript: string | null | undefined;
summaryFilePath?: string | undefined;
summarySeed?: string | undefined;
}): Promise<PostRunIssues> {
export async function collectPostRunIssues(
ctx: AgentRunContext,
options: { skipSummaryStale?: boolean } = {}
): Promise<PostRunIssues> {
const issues: PostRunIssues = {};
if (params.stopScript) {
const failure = await executeStopHook(params.stopScript);
if (failure) issues.stopHook = failure;
}
// stop hook is disabled — production audit (May 2026) showed 8/9 configured
// scripts are foot-guns (duplicates of prepushScript, run on non-committing
// modes against unchanged trees) burning the retry budget on un-fixable
// gates. re-enable here + the dashboard block in `AgentSettings.tsx` once
// we've decided on the right semantics (mode-gating vs. HEAD-changed gating
// vs. deletion). see issue #714.
// if (ctx.stopScript) {
// const failure = await executeStopHook(ctx.stopScript);
// if (failure) issues.stopHook = failure;
// }
// dirty-tree gate fires only in modes that legitimately commit. Review /
// IncrementalReview / Plan complete via review submission or a Plan
// comment, not by touching files — any tree dirt is incidental (e.g. a
// tool-installed `node_modules/`) and the worktree is ephemeral, so
// nudging the agent to commit it would produce a spurious PR. see
// `NON_COMMITTING_MODES` in `action/modes.ts`.
const status = getGitStatus();
if (status) issues.dirtyTree = status;
if (params.summaryFilePath && params.summarySeed !== undefined) {
const stale = await isSummaryUnchanged(params.summaryFilePath, params.summarySeed);
if (stale) issues.summaryStale = { filePath: params.summaryFilePath };
const mode = ctx.toolState.selectedMode;
if (status) {
if (mode && NON_COMMITTING_MODES.has(mode)) {
log.info(`» dirty-tree gate suppressed: mode \`${mode}\` does not commit`);
} else {
issues.dirtyTree = status;
}
}
const summaryFilePath = ctx.toolState.summaryFilePath;
const summarySeed = ctx.toolState.summarySeed;
if (!options.skipSummaryStale && summaryFilePath && summarySeed !== undefined) {
const stale = await isSummaryUnchanged(summaryFilePath, summarySeed);
if (stale) issues.summaryStale = { filePath: summaryFilePath };
}
const unsubmittedMode = getUnsubmittedReview(ctx.toolState);
if (unsubmittedMode) issues.unsubmittedReview = unsubmittedMode;
return issues;
}
export function buildPostRunPrompt(issues: PostRunIssues): string {
// order matches the terminal hard-fail order in `runPostRunRetryLoop` so
// the prompt's emphasis (which gate the agent should fix first) lines up
// with the user-visible failure message reported when retries exhaust.
// both hard-fail gates first (`stopHook` → `unsubmittedReview`), then the
// soft gates (`dirtyTree` → `summaryStale`).
const parts: string[] = [];
if (issues.stopHook) parts.push(buildStopHookPrompt(issues.stopHook));
if (issues.unsubmittedReview) {
parts.push(buildUnsubmittedReviewPrompt(issues.unsubmittedReview));
}
if (issues.dirtyTree) parts.push(buildCommitPrompt(issues.dirtyTree));
if (issues.summaryStale) parts.push(buildSummaryStalePrompt(issues.summaryStale.filePath));
return parts.join("\n\n---\n\n");
}
/**
* prompt for a dedicated post-run reflection turn nudging the agent to call
* `update_learnings` if it discovered anything worth persisting.
* prompt for a dedicated post-run reflection turn nudging the agent to edit
* the rolling learnings file if it discovered anything worth persisting.
*
* this exists because the learnings step baked into mode checklists is
* frequently ignored — the agent stays focused on the task and the meta-ask
* falls through. delivering it as its own resume turn, with nothing competing
* for attention, raises the fire rate substantially.
* this exists because passive "if you learned something, write it down"
* instructions baked into mode checklists are frequently ignored — the agent
* stays focused on the task and the meta-ask falls through. delivering it
* as its own resume turn, with nothing competing for attention, raises the
* fire rate substantially.
*
* the file is the single source of truth — there is no separate MCP tool
* call. the server reads the file at end-of-run and persists any edits to
* `Repo.learnings`.
*
* the prompt copy is shaped by repo-wide audits of the actual content the
* agent has been writing (issue #619 in pullfrog/app). recurring failure
* modes the framing pushes back on:
* - massive multi-paragraph "bullets" that are really mini-articles
* - PR-/review-/commit-/date-anchored facts that decay within weeks
* - rediscovery of pullfrog-tool quirks that belong in tool descriptions,
* not per-repo learnings
* - sections growing into giant flat lists with no internal structure,
* forcing future runs to read kilobytes to find one fact
*/
export function buildLearningsReflectionPrompt(agentId: AgentId): string {
const t = (name: string) => formatMcpToolRef(agentId, name);
export function buildLearningsReflectionPrompt(filePath: string): string {
return [
`REFLECTION — before you finish, think back over this task: did you discover anything about this repo's setup, test commands, conventions, or patterns that you are confident is correct and would reliably help future runs?`,
`REFLECTION — before you finish, think back over this task: did you discover anything about this repo's setup, test commands, conventions, or patterns that is high-confidence and would reliably help future runs?`,
"",
`if so, call \`${t("update_learnings")}\` to persist it.`,
`the rolling learnings file is at \`${filePath}\`. read it first if you haven't already, then edit it in place using your native file tools. the server reads this file at end-of-run and persists any changes — there is no tool to call.`,
"",
`rules:`,
`- only call \`${t("update_learnings")}\` when the finding is high-confidence and broadly useful. skip if unsure, speculative, or one-off.`,
`- pass the FULL merged list: existing learnings from the original prompt + your new discoveries. one fact per bullet, lines starting with \`- \`.`,
`- deduplicate, and drop bullets that are clearly wrong or no longer relevant to the current codebase.`,
`- if you already called \`${t("update_learnings")}\` earlier in this run, or nothing new is worth capturing, just reply "done" and stop — do not edit the repo for this reflection.`,
`structure:`,
`- markdown hierarchy: \`## \` for top-level themes, \`### \` and deeper for sub-themes when a section grows. there is no fixed taxonomy — choose headings that fit THIS repo (e.g. for one repo \`## Migrations\` / \`## Local dev\` may make sense; for another, \`## API quirks\` / \`## Failure modes\`).`,
`- **no section over ~300 lines.** when a section is approaching that, split it: introduce \`### \` subsections grouping related bullets, or hoist a coherent group into a new top-level \`## \` section. granular sections mean future runs read targeted line ranges instead of slurping the whole file. this is the most important hygiene rule on long-lived repos.`,
`- if you find a flat unstructured list (legacy content from before this format), restructure it: read it, group related bullets, rewrite the file with \`## \` / \`### \` headings around them. don't preserve bad structure — fix it.`,
"",
`bullet hygiene:`,
`- one fact per line starting with \`- \`. each bullet is ONE specific durable fact, not a paragraph or essay.`,
`- aim for ≤ 240 chars per bullet. longer bullets are almost always mixing multiple facts that should be split, or burying the durable claim under PR-specific context that should be cut.`,
`- only add bullets when the finding is high-confidence AND broadly useful AND will still be true in 3+ months. skip speculative, one-off, or "maybe" findings.`,
`- prune bullets that are clearly wrong, no longer relevant, or low-signal. a focused, accurate file beats a long stale one. compressing two overlapping bullets into one tighter bullet counts as progress.`,
`- deduplicate against existing entries (in any section) — if a bullet covers the same fact, update it in place instead of adding a duplicate.`,
"",
`do NOT add bullets for:`,
`- pullfrog tool quirks (e.g. "\`shell\` timeout is in milliseconds", "\`git\` args must be a JSON array", "\`create_pull_request_review\` drops out-of-hunk comments", "\`push_branch\` may report timeout when push succeeded"). these are universal across repos and belong in the tool descriptions — flag the gap rather than hoarding the workaround per-repo.`,
`- references to specific PR numbers, review IDs, commit SHAs, branch names, or person handles ("PR #595 introduced X", "flagged in review 12345", "as of commit abc123"). repo state changes; these decay into noise within weeks.`,
`- dated assertions ("as of May 2026", "currently...", "for now..."). if a fact needs a date to be true, it isn't durable enough to belong here.`,
`- play-by-play of what THIS run did. learnings are for the NEXT run, not a retrospective.`,
"",
`if you have nothing substantively new to add AND the existing entries still look healthy and well-structured, leave the file alone — just reply "done" and stop. silence is a valid outcome.`,
].join("\n");
}
@@ -200,17 +315,9 @@ export function buildLearningsReflectionPrompt(agentId: AgentId): string {
* behavior: they're logged but don't fail the run.
*/
export async function runPostRunRetryLoop<R extends AgentResult>(params: {
ctx: AgentRunContext;
initialResult: R;
initialUsage: AgentUsage | undefined;
stopScript: string | null | undefined;
/** absolute path to the seeded PR summary file. when set together with
* `summarySeed`, the loop checks after each agent attempt whether the
* file has been edited; if not, it nudges the agent ONCE via a resume
* turn (subsequent iterations skip the check so we don't keep burning
* retries on a soft gate when the agent has decided no edit is warranted). */
summaryFilePath?: string | undefined;
/** exact bytes of the seeded summary file used for the unchanged-check. */
summarySeed?: string | undefined;
resume: (context: { prompt: string; previousResult: R }) => Promise<R>;
canResume?: ((result: R) => boolean) | undefined;
reflectionPrompt?: string | undefined;
@@ -220,19 +327,16 @@ export async function runPostRunRetryLoop<R extends AgentResult>(params: {
let finalIssues: PostRunIssues = {};
let gateResumeCount = 0;
let pendingReflection = params.reflectionPrompt;
// nudge for an untouched summary file fires AT MOST ONCE per run. after
// we've delivered the prompt, subsequent gate checks pass undefined so
// the loop doesn't keep flagging the same condition — the agent may have
// legitimately decided no edit is warranted, and re-prompting would
// burn the retry budget without adding signal.
// nudge for an untouched summary file fires AT MOST ONCE per run. once
// delivered, subsequent collectPostRunIssues calls skip the check — the
// agent may have legitimately decided no edit is warranted, and
// re-prompting would burn the retry budget without adding signal.
let summaryStaleNudged = false;
while (gateResumeCount < MAX_POST_RUN_RETRIES) {
if (!result.success) break;
const issues = await collectPostRunIssues({
stopScript: params.stopScript,
summaryFilePath: summaryStaleNudged ? undefined : params.summaryFilePath,
summarySeed: summaryStaleNudged ? undefined : params.summarySeed,
const issues = await collectPostRunIssues(params.ctx, {
skipSummaryStale: summaryStaleNudged,
});
if (issues.summaryStale) summaryStaleNudged = true;
finalIssues = issues;
@@ -318,10 +422,11 @@ export async function runPostRunRetryLoop<R extends AgentResult>(params: {
// false-positive failures right after it just passed.
if (gateResumeCount > 0 && result.success && hasPostRunIssues(finalIssues)) {
// re-check the gates that can actually fail the run (stop hook /
// dirty tree). summary-stale is intentionally NOT re-checked here:
// we already delivered the one-shot nudge, and a still-unchanged
// file at this point is the agent's deliberate choice.
finalIssues = await collectPostRunIssues({ stopScript: params.stopScript });
// dirty tree / unsubmitted review). summary-stale is intentionally
// NOT re-checked here: we already delivered the one-shot nudge, and
// a still-unchanged file at this point is the agent's deliberate
// choice.
finalIssues = await collectPostRunIssues(params.ctx, { skipSummaryStale: true });
}
if (result.success && finalIssues.stopHook) {
@@ -337,5 +442,25 @@ export async function runPostRunRetryLoop<R extends AgentResult>(params: {
};
}
if (result.success && finalIssues.unsubmittedReview) {
const retryNote =
gateResumeCount > 0
? ` after ${gateResumeCount} retry ${gateResumeCount === 1 ? "attempt" : "attempts"}`
: "";
// mode-aware: Review's contract requires a review submission; only
// IncrementalReview accepts `report_progress` as an exit. mirroring
// the nudge prompt avoids contradicting the agent-facing copy.
const expected =
finalIssues.unsubmittedReview === "Review"
? "create_pull_request_review"
: "create_pull_request_review or report_progress";
return {
...result,
success: false,
error: `${finalIssues.unsubmittedReview} mode finished without calling ${expected}${retryNote}`,
usage: aggregatedUsage,
};
}
return { ...result, usage: aggregatedUsage };
}
+34
View File
@@ -146,6 +146,40 @@ describe("SessionLabeler", () => {
]);
});
test("Claude path: parent_tool_use_id resolves directly without consuming FIFO", () => {
// Claude runs subagents inside the orchestrator's session — they share
// session_id — and stamps subagent messages with parent_tool_use_id.
// recording dispatch with the Agent tool_use id binds it directly so
// future events resolve regardless of session_id.
const labeler = new SessionLabeler();
expect(labeler.labelFor("shared-session", null)).toBe(ORCHESTRATOR_LABEL);
labeler.recordTaskDispatch({ description: "correctness" }, "toolu_01");
labeler.recordTaskDispatch({ description: "security" }, "toolu_02");
// subagent events come through with shared session_id but distinct
// parent_tool_use_id — direct mapping wins
expect(labeler.labelFor("shared-session", "toolu_01")).toBe("lens:correctness");
expect(labeler.labelFor("shared-session", "toolu_02")).toBe("lens:security");
// orchestrator events on the same session still resolve correctly
expect(labeler.labelFor("shared-session", null)).toBe(ORCHESTRATOR_LABEL);
// pendingLabels is unused on the Claude path — FIFO never consumed
expect(labeler.pendingDispatchCount()).toBe(2);
expect(labeler.size()).toBe(1);
});
test("Claude path: unknown parent_tool_use_id falls through to sessionID/FIFO logic", () => {
// defensive: if a subagent event arrives with a parent_tool_use_id we
// never recorded (e.g. orchestrator dispatched off-stream, or a tool we
// didn't track), the labeler shouldn't crash — it should fall through
// to the sessionID-keyed path.
const labeler = new SessionLabeler();
labeler.labelFor("shared", null);
expect(labeler.labelFor("shared", "unknown-tool-id")).toBe(ORCHESTRATOR_LABEL);
});
test("realistic four-lens parallel fan-out — interleaved tool_use stream", () => {
// simulates the event order we'd see when the orchestrator dispatches
// 4 lens subagents in a single assistant turn and they all start emitting
+48 -18
View File
@@ -67,38 +67,68 @@ export function deriveLabelFromTaskInput(input: TaskDispatchInput): string {
}
/**
* Stateful tracker mapping sessionIDs to human labels.
* Stateful tracker mapping subagent activity back to human-readable labels.
*
* Lifecycle:
* - First call to `labelFor()` returns ORCHESTRATOR_LABEL and binds that
* sessionID to it. Every subsequent event from that session gets the
* same label.
* - When the orchestrator emits a Task tool_use, the harness calls
* `recordTaskDispatch()` to push the dispatch's derived label onto a
* pending FIFO queue.
* - The next previously-unseen sessionID consumes the head of the queue.
* - If `labelFor()` is called for a new session with an empty queue
* (e.g. a subagent emitted events before the parent's tool_use was
* parsed, or the runtime spawned a session we didn't expect), the
* labeler falls back to `subagent#N` so log lines remain attributable.
* Two attribution channels are supported because the runtimes differ:
*
* - **OpenCode** spawns each subagent as its own opencode `Session` with
* a distinct `sessionID`. The harness records each Task dispatch into a
* pending FIFO queue; the next previously-unseen sessionID consumes the
* head of the queue and binds it to that label.
*
* - **Claude Code** runs subagents inside the orchestrator's session — they
* all share `session_id` — and instead stamps every subagent message with
* `parent_tool_use_id` pointing at the Agent tool_use id that spawned them.
* The harness binds each Agent tool_use id to its dispatched label up
* front, then `labelFor` looks the label up directly when an event arrives
* carrying that `parent_tool_use_id`.
*
* `labelFor(sessionID, parentToolUseId?)` accepts both: when
* `parentToolUseId` is set and known it short-circuits to the direct mapping;
* otherwise it falls through to the FIFO/sessionID path.
*/
export class SessionLabeler {
private readonly labels = new Map<string, string>();
private readonly labelsByToolUseId = new Map<string, string>();
private readonly pendingLabels: string[] = [];
private fallbackCounter = 0;
recordTaskDispatch(input: TaskDispatchInput): string {
/**
* Record a Task/Agent tool dispatch.
*
* @param input Task tool input — used to derive the lens label.
* @param toolUseId Optional Agent tool_use id. When provided, future events
* carrying `parent_tool_use_id === toolUseId` resolve
* directly to this label without consuming the FIFO queue
* (Claude path). Always also pushed to the FIFO queue so
* the OpenCode path still works when toolUseId is absent.
*/
recordTaskDispatch(input: TaskDispatchInput, toolUseId?: string | null): string {
const label = deriveLabelFromTaskInput(input);
this.pendingLabels.push(label);
if (toolUseId) this.labelsByToolUseId.set(toolUseId, label);
return label;
}
/**
* Return a label for the given sessionID. Binds on first call.
* Pass undefined/empty for events that lack a session id — the caller
* gets ORCHESTRATOR_LABEL so the line is still attributable.
* Return a label for the given event.
*
* @param sessionID Session id from the event (OpenCode: per-session;
* Claude: shared across orchestrator + subagents).
* @param parentToolUseId Claude's `parent_tool_use_id` — non-null on
* subagent messages. When set and known, takes
* priority over the FIFO/sessionID path.
*/
labelFor(sessionID: string | undefined | null): string {
labelFor(sessionID: string | undefined | null, parentToolUseId?: string | null): string {
// Claude path: subagent messages carry parent_tool_use_id pointing at
// the Agent tool_use that spawned them. resolve directly without
// touching the sessionID-keyed map (which is bound to the orchestrator
// for the shared session_id and would otherwise misattribute).
if (parentToolUseId) {
const direct = this.labelsByToolUseId.get(parentToolUseId);
if (direct) return direct;
}
if (!sessionID) return ORCHESTRATOR_LABEL;
const existing = this.labels.get(sessionID);
if (existing) return existing;
+28 -14
View File
@@ -1,5 +1,6 @@
import { execFileSync } from "node:child_process";
import type { AgentId } from "../external.ts";
import type { ToolState } from "../toolState.ts";
import { log } from "../utils/cli.ts";
import type { ResolvedInstructions } from "../utils/instructions.ts";
import type { ResolvedPayload } from "../utils/payload.ts";
@@ -54,13 +55,25 @@ export interface PostRunIssues {
* seed, i.e. the agent never touched it. soft gate — nudges once via a
* resume turn but never fails the run, parallel to dirtyTree semantics. */
summaryStale?: SummaryStale;
/**
* populated when the agent selected a review mode but the post-run check
* over toolState shows neither a `create_pull_request_review` submission
* nor a final `report_progress` write happened. derived inline from
* `toolState.selectedMode` + `toolState.review` + `toolState.finalSummaryWritten`
* via {@link getUnsubmittedReview} — no parallel toolState flag is stored.
* carries the mode name so the resume prompt can reference it. handled like
* `stopHook`: nudge via resume, hard-fail if still unsatisfied after
* `MAX_POST_RUN_RETRIES`.
*/
unsubmittedReview?: "Review" | "IncrementalReview";
}
export function hasPostRunIssues(issues: PostRunIssues): boolean {
return (
issues.stopHook !== undefined ||
issues.dirtyTree !== undefined ||
issues.summaryStale !== undefined
issues.summaryStale !== undefined ||
issues.unsubmittedReview !== undefined
);
}
@@ -104,7 +117,14 @@ export interface AgentResult {
}
/**
* Minimal context passed to agent.run()
* Context passed to agent.run() and threaded through the post-run loop.
*
* design rule: this is the single object that flows through the harness and
* downstream utilities by reference. derived predicates (e.g.
* `getUnsubmittedReview`), tmpfile paths, and seed bytes live on
* `toolState` — read them at the call site, do not duplicate them onto this
* interface. utilities that need run state should accept `ctx` whole, not
* destructure a narrow subset.
*/
export interface AgentRunContext {
payload: ResolvedPayload;
@@ -120,19 +140,13 @@ export interface AgentRunContext {
*/
stopScript?: string | null | undefined;
/**
* absolute path to the rolling PR summary tmpfile, when one was seeded
* for this run (Review / IncrementalReview / pr-summary Task). enables
* a post-run sanity nudge that prompts the agent if the file is still
* byte-identical to its seed.
* mutable per-run state shared with the MCP server (by reference). post-run
* gates read fresh values from it after each agent attempt — `summaryFilePath`,
* `summarySeed`, `selectedMode`, `review`, `finalSummaryWritten`,
* `hadProgressComment` are all consulted by `collectPostRunIssues`. see
* `action/toolState.ts` for the literal-state design rule.
*/
summaryFilePath?: string | undefined;
/**
* exact bytes of the seeded summary file. compared against the current
* file content after each agent attempt to detect "agent forgot to edit
* the summary" — particularly common with smaller models that lose
* track of multi-step instructions.
*/
summarySeed?: string | undefined;
toolState: ToolState;
/**
* called synchronously when the agent subprocess is killed for inner
* activity timeout. lets main.ts tear down shared resources (MCP HTTP
+110
View File
@@ -0,0 +1,110 @@
import { describe, expect, it } from "vitest";
import { deriveSubagentModels } from "./subagentModels.ts";
describe("deriveSubagentModels", () => {
it("returns no override when orchestrator is undefined", () => {
expect(deriveSubagentModels(undefined)).toEqual({ reviewer: undefined });
});
it("returns no override when orchestrator slug isn't registered", () => {
expect(deriveSubagentModels("nonexistent/model")).toEqual({ reviewer: undefined });
});
describe("anthropic family — opus → sonnet", () => {
it("direct anthropic opus", () => {
expect(deriveSubagentModels("anthropic/claude-opus-4-7")).toEqual({
reviewer: "anthropic/claude-sonnet-4-6",
});
});
it("opencode-vendored opus stays on opencode prefix", () => {
expect(deriveSubagentModels("opencode/claude-opus-4-7")).toEqual({
reviewer: "opencode/claude-sonnet-4-6",
});
});
it("openrouter-anthropic-opus-via-anthropic-direct hits anthropic alias's openRouterResolve", () => {
// both the anthropic alias and the opencode alias have the same
// openRouterResolve. first-match-wins by alias declaration order
// (anthropic declared first in providers).
expect(deriveSubagentModels("openrouter/anthropic/claude-opus-4.7")).toEqual({
reviewer: "openrouter/anthropic/claude-sonnet-4.6",
});
});
it("sonnet has no further downshift", () => {
expect(deriveSubagentModels("anthropic/claude-sonnet-4-6")).toEqual({ reviewer: undefined });
expect(deriveSubagentModels("opencode/claude-sonnet-4-6")).toEqual({ reviewer: undefined });
});
it("haiku has no downshift", () => {
expect(deriveSubagentModels("anthropic/claude-haiku-4-5")).toEqual({ reviewer: undefined });
});
});
describe("openai family", () => {
it("gpt-pro → gpt (direct)", () => {
expect(deriveSubagentModels("openai/gpt-5.5-pro")).toEqual({ reviewer: "openai/gpt-5.5" });
});
it("gpt → gpt-5.4 (direct)", () => {
expect(deriveSubagentModels("openai/gpt-5.5")).toEqual({ reviewer: "openai/gpt-5.4" });
});
it("gpt → gpt-5.4 (opencode-vendored)", () => {
expect(deriveSubagentModels("opencode/gpt-5.5")).toEqual({ reviewer: "opencode/gpt-5.4" });
});
it("gpt-pro → gpt (openrouter)", () => {
expect(deriveSubagentModels("openrouter/openai/gpt-5.5-pro")).toEqual({
reviewer: "openrouter/openai/gpt-5.5",
});
});
it("gpt → gpt-5.4 (openrouter)", () => {
expect(deriveSubagentModels("openrouter/openai/gpt-5.5")).toEqual({
reviewer: "openrouter/openai/gpt-5.4",
});
});
it("gpt-5.4 itself (the hidden subagent target) has no further downshift", () => {
expect(deriveSubagentModels("openai/gpt-5.4")).toEqual({ reviewer: undefined });
});
it("gpt-mini has no downshift", () => {
expect(deriveSubagentModels("openai/gpt-5.4-mini")).toEqual({ reviewer: undefined });
});
});
describe("google (gemini) — pro → flash", () => {
it("direct google", () => {
expect(deriveSubagentModels("google/gemini-3.1-pro-preview")).toEqual({
reviewer: "google/gemini-3-flash-preview",
});
});
it("opencode-vendored gemini-pro", () => {
expect(deriveSubagentModels("opencode/gemini-3.1-pro")).toEqual({
reviewer: "opencode/gemini-3-flash",
});
});
it("openrouter-google-gemini-pro", () => {
expect(deriveSubagentModels("openrouter/google/gemini-3.1-pro-preview")).toEqual({
reviewer: "openrouter/google/gemini-3-flash-preview",
});
});
it("flash has no downshift", () => {
expect(deriveSubagentModels("google/gemini-3-flash-preview")).toEqual({
reviewer: undefined,
});
});
});
describe("providers / models without a subagentModel — inherit", () => {
it("xai grok (already cheap flagship)", () => {
expect(deriveSubagentModels("xai/grok-4.3")).toEqual({ reviewer: undefined });
});
it("deepseek", () => {
expect(deriveSubagentModels("deepseek/deepseek-v4-pro")).toEqual({ reviewer: undefined });
});
it("moonshot kimi", () => {
expect(deriveSubagentModels("moonshotai/kimi-k2.6")).toEqual({ reviewer: undefined });
});
it("opencode big-pickle", () => {
expect(deriveSubagentModels("opencode/big-pickle")).toEqual({ reviewer: undefined });
});
it("legacy fallback aliases (gpt-codex, deepseek-reasoner)", () => {
expect(deriveSubagentModels("openai/gpt-5.3-codex")).toEqual({ reviewer: undefined });
expect(deriveSubagentModels("deepseek/deepseek-reasoner")).toEqual({ reviewer: undefined });
});
});
});
+40
View File
@@ -0,0 +1,40 @@
import { modelAliases } from "../models.ts";
/**
* Derive a cheaper subagent model override from the orchestrator's resolved
* model spec.
*
* This is a pure registry lookup: every alias in `action/models.ts` declares
* its own `subagentModel` (alias key in the same provider). At runtime we
* reverse-lookup the orchestrator's resolved slug to find the alias that
* produced it, follow the `subagentModel` pointer, and return the target
* alias's resolve / openRouterResolve depending on which route the
* orchestrator was using.
*
* Returns `{ reviewer: undefined }` when the orchestrator's alias has no
* `subagentModel` (e.g. it's already at a sufficiently cheap tier, or its
* provider doesn't have a clean cheaper-but-capable sibling). See models.ts
* for the wiring + per-provider rationale.
*/
export function deriveSubagentModels(orchestratorSpec: string | undefined): {
reviewer: string | undefined;
} {
if (!orchestratorSpec) return { reviewer: undefined };
// Reverse-lookup. The same resolve string appears in only one alias
// (within its provider), so first match wins. We track which field
// matched (resolve vs openRouterResolve) so we can pick the same field
// off the subagent target — keeping the orchestrator's route consistent.
for (const source of modelAliases) {
const matchedDirect = source.resolve === orchestratorSpec;
const matchedOR = source.openRouterResolve === orchestratorSpec;
if (!matchedDirect && !matchedOR) continue;
if (!source.subagentModel) return { reviewer: undefined };
const target = modelAliases.find((a) => a.slug === source.subagentModel);
if (!target) return { reviewer: undefined };
const reviewer = matchedOR ? target.openRouterResolve : target.resolve;
return { reviewer };
}
return { reviewer: undefined };
}
+40
View File
@@ -0,0 +1,40 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
const claudeSource = readFileSync(join(__dirname, "claude.ts"), "utf-8");
const opencodeSource = readFileSync(join(__dirname, "opencode.ts"), "utf-8");
/**
* The Claude Code `--agents` JSON and OpenCode `agent` config block are the
* only places where per-subagent model overrides take effect. They're built
* by string-only helpers we don't export, so this test reads the source and
* asserts the literal model strings + agent names are wired in. A regression
* here means the next review run silently runs lenses on Opus instead of
* Sonnet.
*/
describe("subagent registration source asserts", () => {
describe("claude.ts buildAgentsJson", () => {
it("registers reviewfrog with sonnet model", () => {
expect(claudeSource).toMatch(
/\[REVIEWER_AGENT_NAME\]:\s*\{[^}]*model:\s*"claude-sonnet-4-6"/s
);
});
it("imports the reviewer name constant", () => {
expect(claudeSource).toMatch(/REVIEWER_AGENT_NAME/);
});
});
describe("opencode.ts buildReviewerAgentConfig", () => {
it("registers reviewfrog with mode: subagent", () => {
expect(opencodeSource).toMatch(/\[REVIEWER_AGENT_NAME\]:[^}]*mode:\s*"subagent"/s);
});
it("uses deriveSubagentModels for the reviewer model override", () => {
expect(opencodeSource).toMatch(/deriveSubagentModels\(/);
expect(opencodeSource).toMatch(/overrides\.reviewer/);
});
it("passes orchestrator model to buildReviewerAgentConfig", () => {
expect(opencodeSource).toMatch(/buildReviewerAgentConfig\(model\)/);
});
});
});
+9 -2
View File
@@ -22,9 +22,16 @@ type CliProvider = {
function buildProviders(): CliProvider[] {
return Object.entries(providers)
.filter(([key]) => key !== "opencode" && key !== "openrouter")
.filter(([key]) => key !== "opencode" && key !== "openrouter" && key !== "bedrock")
.map(([key, config]: [string, ProviderConfig]) => {
const aliases = modelAliases.filter((a) => a.provider === key && !a.fallback);
// bedrock requires multi-secret setup (auth + region + model id) that
// doesn't fit the single-paste flow below — direct users to
// https://docs.pullfrog.com/bedrock instead. revisit once the init flow
// supports multi-value setup. `hidden` excludes internal-only subagent
// targets (e.g. openai/gpt-5.4) per #710.
const aliases = modelAliases.filter(
(a) => a.provider === key && !a.fallback && !a.routing && !a.hidden
);
const recommended = aliases.find((a) => a.preferred);
const sorted = [...aliases].sort((a, b) => {
if (a.preferred && !b.preferred) return -1;
+7
View File
@@ -273,6 +273,13 @@ export interface WriteablePayload {
triggerer?: string | undefined;
/** event-level instructions for this trigger type (flag-expanded server-side) */
eventInstructions?: string | undefined;
/**
* system-injected note about prior superseded runs (e.g. when the
* triggering @pullfrog comment is edited). rendered alongside the user's
* prompt rather than via eventInstructions so it survives user-prompt
* precedence.
*/
previousRunsNote?: string | undefined;
/** event data from webhook payload - discriminated union based on trigger field */
event: PayloadEvent;
/** timeout for agent run (e.g., "10m", "1h30m") - defaults to "1h" */
+262 -33
View File
@@ -6,13 +6,9 @@ import { join } from "node:path";
import * as core from "@actions/core";
import { deleteProgressComment, reportProgress } from "./mcp/comment.ts";
import { startInstallation } from "./mcp/dependencies.ts";
import {
initToolState,
startMcpHttpServer,
type ToolContext,
type ToolState,
} from "./mcp/server.ts";
import { startMcpHttpServer, type ToolContext } from "./mcp/server.ts";
import { computeModes } from "./modes.ts";
import { initToolState, type ToolState } from "./toolState.ts";
import {
type ActivityTimeout,
createProcessOutputActivityTimeout,
@@ -20,8 +16,14 @@ import {
DEFAULT_ACTIVITY_TIMEOUT_MS,
} from "./utils/activity.ts";
import { resolveAgent, resolveModel } from "./utils/agent.ts";
import { formatAgentHangBody } from "./utils/agentHangReport.ts";
import { apiFetch } from "./utils/apiFetch.ts";
import { validateAgentApiKey } from "./utils/apiKeys.ts";
import {
formatApiKeyErrorSummary,
isApiKeyAuthError,
validateAgentApiKey,
} from "./utils/apiKeys.ts";
import { isLocalApiUrl } from "./utils/apiUrl.ts";
import { resolveBody } from "./utils/body.ts";
import { formatUsageSummary, log, writeSummary } from "./utils/cli.ts";
import { recordDiffReadFromToolUse } from "./utils/diffCoverage.ts";
@@ -31,8 +33,9 @@ import { resolveGit, setGitAuthServer } from "./utils/gitAuth.ts";
import { startGitAuthServer } from "./utils/gitAuthServer.ts";
import { createOctokit, writeGitHubUsageSummaryToFile } from "./utils/github.ts";
import { resolveInstructions } from "./utils/instructions.ts";
import { readLearningsFile, seedLearningsFile } from "./utils/learnings.ts";
import { executeLifecycleHook } from "./utils/lifecycle.ts";
import { normalizeEnv } from "./utils/normalizeEnv.ts";
import { normalizeEnv, sanitizeSecret } from "./utils/normalizeEnv.ts";
import { aggregateUsage, patchWorkflowRunFields } from "./utils/patchWorkflowRunFields.ts";
import { resolvePayload, resolvePromptInput } from "./utils/payload.ts";
import { isRouterKeylimitExhaustedError } from "./utils/providerErrors.ts";
@@ -185,8 +188,9 @@ function billingConsoleUrl(owner: string, anchor: "billing" | "model-access"): s
*
* Branches:
* - `router_requires_card`: user is on Router mode with no card AND no
* wallet balance. Lead with the carrot ($20 free credit), link to
* `#model-access` where the Add Card flow lives.
* wallet balance (signup credit exhausted or not granted). Frame as
* "add a card to continue", link to `#model-access` where the Add
* Card flow lives.
* - `router_balance_exhausted`: user has a card on file but auto-reload is
* disabled and they've spent past their $5 overdraft buffer. Frame as
* "balance ran out" and surface both remediation paths (top up, or flip
@@ -208,7 +212,7 @@ function formatBillingErrorSummary(error: BillingError, owner: string): string {
return [
"**Add a card to start using Pullfrog Router.**",
"",
"Router proxies OpenRouter at raw cost — no platform markup, and your first $20 of usage is on us.",
"Router proxies OpenRouter at raw cost — no platform markup. Add a card and we'll auto-reload your wallet so runs keep flowing.",
"",
`[Add a card →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
@@ -279,18 +283,18 @@ function formatTransientErrorSummary(error: TransientError, owner: string): stri
].join("\n");
}
async function mintProxyKey(ctx: { oidcCredentials: OidcCredentials }): Promise<string | null> {
async function mintProxyKey(ctx: {
oidcCredentials: OidcCredentials | null;
repo: { owner: string; name: string };
}): Promise<string | null> {
try {
process.env.ACTIONS_ID_TOKEN_REQUEST_URL = ctx.oidcCredentials.requestUrl;
process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN = ctx.oidcCredentials.requestToken;
const oidcToken = await core.getIDToken("pullfrog-api");
delete process.env.ACTIONS_ID_TOKEN_REQUEST_URL;
delete process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN;
const headers = await buildProxyTokenHeaders(ctx);
if (!headers) return null;
const response = await apiFetch({
path: "/api/proxy-token",
method: "POST",
headers: { Authorization: `Bearer ${oidcToken}` },
headers,
});
if (response.status === 402) {
@@ -336,12 +340,44 @@ async function mintProxyKey(ctx: { oidcCredentials: OidcCredentials }): Promise<
}
}
/**
* choose how to authenticate the `/api/proxy-token` request:
*
* - production: mint a fresh OIDC token via `core.getIDToken` and send as
* `Authorization: Bearer …` (the server verifies it cryptographically).
* - local dev (no OIDC + `API_URL` is localhost): send `x-dev-repo:
* owner/repo` instead. the server-side route only honors this header
* when `NODE_ENV === "development"`, so prod is never reachable through
* this branch even if the action is misconfigured.
*
* returns null when neither path is available — caller treats as soft skip.
*/
async function buildProxyTokenHeaders(ctx: {
oidcCredentials: OidcCredentials | null;
repo: { owner: string; name: string };
}): Promise<Record<string, string> | null> {
if (ctx.oidcCredentials) {
process.env.ACTIONS_ID_TOKEN_REQUEST_URL = ctx.oidcCredentials.requestUrl;
process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN = ctx.oidcCredentials.requestToken;
const oidcToken = await core.getIDToken("pullfrog-api");
delete process.env.ACTIONS_ID_TOKEN_REQUEST_URL;
delete process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN;
return { Authorization: `Bearer ${oidcToken}` };
}
if (isLocalApiUrl()) {
log.info(`» proxy: dev bypass (x-dev-repo) for ${ctx.repo.owner}/${ctx.repo.name}`);
return { "x-dev-repo": `${ctx.repo.owner}/${ctx.repo.name}` };
}
return null;
}
async function resolveProxyModel(ctx: {
payload: ResolvedPayload;
oss: boolean;
plan: AccountPlan;
proxyModel?: string | undefined;
oidcCredentials: OidcCredentials | null;
repo: { owner: string; name: string };
}): Promise<void> {
// env override = BYOK escape hatch, don't proxy
if (process.env.PULLFROG_MODEL?.trim()) return;
@@ -349,12 +385,15 @@ async function resolveProxyModel(ctx: {
const needsProxy = isInfraCovered({ isOss: ctx.oss, plan: ctx.plan }) && ctx.proxyModel;
if (!needsProxy) return;
if (!ctx.oidcCredentials) {
// dev affordance: when talking to a localhost API, the server-side
// x-dev-repo bypass replaces OIDC verification, so a play run can
// exercise the proxy/router/oss path without GitHub Actions OIDC.
if (!ctx.oidcCredentials && !isLocalApiUrl()) {
log.warning("» proxy requested but no OIDC credentials available — skipping");
return;
}
const key = await mintProxyKey({ oidcCredentials: ctx.oidcCredentials });
const key = await mintProxyKey({ oidcCredentials: ctx.oidcCredentials, repo: ctx.repo });
if (!key) return;
process.env.OPENROUTER_API_KEY = key;
@@ -395,6 +434,62 @@ async function fetchPreviousSnapshot(ctx: ToolContext, prNumber: number): Promis
* (on incremental runs) or serialize the placeholder scaffold (on first
* runs), neither of which is useful.
*/
/**
* Read the agent-edited repo-level learnings tmpfile and PATCH it to
* `Repo.learnings`.
*
* Best-effort: any failure is logged and does not affect the run's success
* status. Skips the PATCH when the file is byte-trim-identical to its seed —
* the agent didn't touch it, so writing the same content back would just
* burn a `LearningsRevision` row and an API round-trip.
*
* `model` is forwarded so `LearningsRevision.model` keeps populating; it
* powers the per-revision attribution badge in the UI history view.
*/
async function persistLearnings(ctx: ToolContext): Promise<void> {
const filePath = ctx.toolState.learningsFilePath;
if (!filePath) return;
if (ctx.toolState.learningsPersistAttempted) return;
ctx.toolState.learningsPersistAttempted = true;
const current = await readLearningsFile(filePath);
if (current === null) {
log.debug(`learnings tmpfile missing or unreadable at ${filePath} — skipping persist`);
return;
}
const seed = ctx.toolState.learningsSeed?.trim() ?? "";
if (current === seed) {
log.debug("learnings tmpfile unchanged from seed — skipping persist");
return;
}
try {
const response = await apiFetch({
path: `/api/repo/${ctx.repo.owner}/${ctx.repo.name}/learnings`,
method: "PATCH",
headers: {
authorization: `Bearer ${ctx.apiToken}`,
"content-type": "application/json",
},
body: JSON.stringify({
learnings: current,
model: ctx.toolState.model,
}),
signal: AbortSignal.timeout(10_000),
});
if (!response.ok) {
const error = await response.text().catch(() => "(no body)");
// promoted from debug → warning: this path means the agent edited the
// file (we already short-circuited the unchanged-from-seed case above)
// but the PATCH dropped it on the floor. silently losing real work is
// worse than the noise of a CI warning.
log.warning(`learnings persist failed (${response.status}): ${error}`);
return;
}
log.info("» learnings updated");
} catch (err) {
log.warning(`learnings persist failed: ${err instanceof Error ? err.message : String(err)}`);
}
}
async function persistSummary(ctx: ToolContext): Promise<void> {
const filePath = ctx.toolState.summaryFilePath;
if (!filePath) return;
@@ -425,9 +520,14 @@ async function persistSummary(ctx: ToolContext): Promise<void> {
});
}
async function writeJobSummary(toolState: ToolState): Promise<void> {
// fall back to the agent's final assistant message when the agent never
// called report_progress (e.g. schedule/workflow_dispatch runs that have no
// PR/issue context to comment on). lastProgressBody wins when present so we
// don't double up the progress comment body in the job summary.
async function writeJobSummary(toolState: ToolState, finalOutput?: string): Promise<void> {
const usageSummary = formatUsageSummary(toolState.usageEntries);
const summaryParts = [toolState.lastProgressBody, usageSummary].filter(Boolean);
const body = toolState.lastProgressBody || finalOutput;
const summaryParts = [body, usageSummary].filter(Boolean);
if (summaryParts.length > 0) {
await writeSummary(summaryParts.join("\n\n"));
}
@@ -464,12 +564,15 @@ export async function main(): Promise<MainResult> {
const runContext = await resolveRunContextData({ octokit: initialOctokit, token: jobToken });
timer.checkpoint("runContextData");
// inject account-level secrets into process.env (YAML secrets take precedence)
// inject account-level secrets into process.env (YAML secrets take precedence).
// sanitizeSecret trims + masks so accidental trailing whitespace doesn't leak
// through GitHub Actions' line-based log masking. whitespace-only values
// return null and skip injection so the user sees a clear missing-key error.
if (runContext.dbSecrets) {
for (const [key, value] of Object.entries(runContext.dbSecrets)) {
if (!process.env[key]) {
process.env[key] = value;
core.setSecret(value);
const sanitized = sanitizeSecret(key, value);
if (sanitized !== null) process.env[key] = sanitized;
}
}
const count = Object.keys(runContext.dbSecrets).length;
@@ -520,6 +623,7 @@ export async function main(): Promise<MainResult> {
plan: runContext.plan,
proxyModel: runContext.proxyModel,
oidcCredentials,
repo: runContext.repo,
});
} catch (error) {
if (error instanceof BillingError) {
@@ -647,6 +751,45 @@ export async function main(): Promise<MainResult> {
log.info(`» MCP server started at ${mcpHttpServer.url}`);
timer.checkpoint("mcpServer");
// seed the rolling repo-level learnings tmpfile for every run. the
// agent reads the file at startup (path is surfaced in the LEARNINGS
// section of the prompt) and may edit it during the post-run
// reflection turn. persistLearnings reads it back at end-of-run and
// PATCHes any changes to Repo.learnings, byte-trim equality against
// the seed gates the API call. always-seed (vs gated): learnings are
// universal — any run can produce them, and gating just hides the
// affordance.
//
// wrapped in best-effort try/catch: this block runs unconditionally,
// and an unwrapped filesystem failure (ENOSPC, EACCES, hostile sandbox)
// would unwind into the outer main() catch and flip an otherwise-
// successful run to "❌ Pullfrog failed" before the agent even starts.
// matches `persistLearnings`'s own best-effort contract — learnings
// are a peripheral artifact, not a load-bearing capability. on failure
// toolState.learningsFilePath stays unset, and downstream consumers
// (`persistLearnings`, agent harnesses, `resolveInstructions`) all
// treat undefined as "no learnings affordance this run".
try {
const learningsPath = await seedLearningsFile({
tmpdir,
current: runContext.repoSettings.learnings,
});
toolState.learningsFilePath = learningsPath;
// file on disk is the verbatim DB body, so the seed used for
// change-detection is just `current ?? ""` (trimmed). persistLearnings
// byte-compares against the trimmed read-back to skip no-op PATCHes.
toolState.learningsSeed = (runContext.repoSettings.learnings ?? "").trim();
log.info(
`» learnings seeded at ${learningsPath} (existing=${runContext.repoSettings.learnings ? "yes" : "no"})`
);
const ctxForExit = toolContext;
onExitSignal(() => persistLearnings(ctxForExit));
} catch (err) {
log.warning(
`» learnings seed failed: ${err instanceof Error ? err.message : String(err)} — continuing without learnings file`
);
}
// seed the rolling PR summary tmpfile when the dispatcher requested it.
// gated on event being a PR — issue/workflow_dispatch runs have no
// summarySnapshot to maintain. file path is exposed to the agent via
@@ -698,7 +841,8 @@ export async function main(): Promise<MainResult> {
modes,
agentId,
outputSchema,
learnings: runContext.repoSettings.learnings,
learningsFilePath: toolState.learningsFilePath ?? null,
learningsHeadings: runContext.repoSettings.learningsHeadings,
});
const logParts = [
instructions.eventInstructions
@@ -793,8 +937,7 @@ export async function main(): Promise<MainResult> {
instructions,
todoTracker,
stopScript: runContext.repoSettings.stopScript,
summaryFilePath: toolState.summaryFilePath,
summarySeed: toolState.summarySeed,
toolState,
onActivityTimeout: onInnerActivityTimeout,
onToolUse: (event) => {
const wasTracked = recordDiffReadFromToolUse({
@@ -882,6 +1025,34 @@ export async function main(): Promise<MainResult> {
await persistSummary(toolContext);
}
// same for the rolling repo-level learnings tmpfile. always seeded, so
// always read back; persistLearnings short-circuits when the file is
// unchanged from its seed.
if (toolContext) {
await persistLearnings(toolContext);
}
// when the agent harness returns success=false (e.g. unsubmitted-review
// gate exhausted retries, stop-hook persistently failing), surface the
// error in the progress comment so the user sees it instead of a
// deleted-comment void. mirrors the catch-block error reporting for
// thrown errors. runs before the stranded-comment cleanup below so
// the comment is still around to update; reportErrorToComment sets
// wasUpdated=true and the !result.success guard skips deletion.
if (!result.success && toolContext && toolState.progressComment) {
const rawError = result.error || "agent run failed";
const errorBody = isApiKeyAuthError(rawError)
? formatApiKeyErrorSummary({
owner: runContext.repo.owner,
name: runContext.repo.name,
raw: rawError,
})
: rawError;
await reportErrorToComment({ toolState, error: errorBody }).catch((error) => {
log.debug(`failure error report failed: ${error}`);
});
}
// clean up stranded progress comments. the comment is stale unless
// report_progress wrote a final summary to it — three sub-cases all reduce
// to !finalSummaryWritten:
@@ -895,14 +1066,34 @@ export async function main(): Promise<MainResult> {
// so progressComment is already null by the time we get here for that path.
// uses finalSummaryWritten (not todoTracker.enabled or wasUpdated) so cleanup
// survives API failures in report_progress where cancel() ran but the write
// didn't succeed, and isn't fooled by writes to *other* artifacts.
if (toolContext && toolState.progressComment && !toolState.finalSummaryWritten) {
// didn't succeed, and isn't fooled by writes to *other* artifacts. skipped
// entirely on result.success===false: the error message just written above
// is the user's only signal that the run happened — deleting it would
// restore the same empty-void UX this commit fixes.
if (
toolContext &&
result.success &&
toolState.progressComment &&
!toolState.finalSummaryWritten
) {
await deleteProgressComment(toolContext).catch((error) => {
log.debug(`stranded progress comment cleanup failed: ${error}`);
});
}
await writeJobSummary(toolState);
// best-effort: failures writing the actions step summary must not throw
// past this point. on the result.success===false branch above we already
// wrote `result.error` to the progress comment, and a throw here would
// jump to the outer catch which calls reportErrorToComment again with
// the (less actionable) writeJobSummary error — silently overwriting the
// gate's failure message in the progress comment. the step-summary write
// is informational; let it fail silently rather than corrupt user-facing
// output.
try {
await writeJobSummary(toolState, result.output);
} catch (error) {
log.debug(`job summary write failed: ${error}`);
}
// emit structured output marker for test validation
if (toolState.output) {
@@ -933,11 +1124,43 @@ export async function main(): Promise<MainResult> {
? new BillingError(errorMessage, { code: "router_keylimit_exhausted" })
: null;
// when the activity-timeout watchdog wins the race against the agent
// harness's own catch, the bare timer reject reason ("activity timeout:
// no output for 302s") tells the user nothing actionable. the harness
// keeps a structured diagnostic on toolState as it runs — recent stderr,
// last provider-error label, event count — and `formatAgentHangBody`
// renders that as a markdown body suitable for both the job summary tab
// and the PR progress comment.
//
// gated on isHang because the harness sets `agentDiagnostic` on entry,
// so any non-hang throw that hits the outer catch (e.g. the post-success
// output_schema validator, or a late cleanup throw after the run already
// succeeded) would otherwise render "Pullfrog failed" with stale event
// counts and silently drop the real `errorMessage`.
const isHang =
errorMessage.startsWith("activity timeout") || errorMessage.startsWith("agent still pending");
const hangBody = isHang
? formatAgentHangBody({ diagnostic: toolState.agentDiagnostic, isHang: true, errorMessage })
: null;
const apiKeySource = hangBody ?? errorMessage;
const apiKeyErrorSummary =
!billingError && isApiKeyAuthError(apiKeySource)
? formatApiKeyErrorSummary({
owner: runContext.repo.owner,
name: runContext.repo.name,
raw: apiKeySource,
})
: null;
// best-effort summary — write the error so it's visible in the Actions summary tab
try {
const errorSummary = billingError
? formatBillingErrorSummary(billingError, runContext.repo.owner)
: `### ❌ Pullfrog failed\n\n\`\`\`\n${errorMessage}\n\`\`\``;
: (apiKeyErrorSummary ??
(hangBody
? `### ❌ Pullfrog failed\n\n${hangBody}`
: `### ❌ Pullfrog failed\n\n\`\`\`\n${errorMessage}\n\`\`\``));
const usageSummary = formatUsageSummary(toolState.usageEntries);
const parts = [errorSummary, toolState.lastProgressBody, usageSummary].filter(Boolean);
await writeSummary(parts.join("\n\n"));
@@ -946,7 +1169,7 @@ export async function main(): Promise<MainResult> {
try {
const commentBody = billingError
? formatBillingErrorSummary(billingError, runContext.repo.owner)
: errorMessage;
: (apiKeyErrorSummary ?? hangBody ?? errorMessage);
await reportErrorToComment({ toolState, error: commentBody });
} catch {
// error reporting failed, but don't let it mask the original error
@@ -966,6 +1189,12 @@ export async function main(): Promise<MainResult> {
await persistSummary(toolContext);
}
// same rationale for learnings: a partial edit before a crash is still
// worth keeping. persistLearnings is idempotent via learningsPersistAttempted.
if (toolContext) {
await persistLearnings(toolContext);
}
return {
success: false,
error: errorMessage,
+323 -175
View File
@@ -5,9 +5,10 @@ import type { Octokit, RestEndpointMethodTypes } from "@octokit/rest";
import { type } from "arktype";
import { log } from "../utils/cli.ts";
import { countLines, createDiffCoverageState } from "../utils/diffCoverage.ts";
import { $git } from "../utils/gitAuth.ts";
import { $git, $gitFetchWithDeepen } from "../utils/gitAuth.ts";
import { executeLifecycleHook } from "../utils/lifecycle.ts";
import { computeIncrementalDiff } from "../utils/rangeDiff.ts";
import { retry } from "../utils/retry.ts";
import { $ } from "../utils/shell.ts";
import { rejectIfLeadingDash } from "./git.ts";
import { commentableLinesForFile } from "./review.ts";
@@ -258,10 +259,10 @@ async function ensureBeforeShaReachable(params: EnsureBeforeShaParams): Promise<
sha: params.sha,
ref: tempBranch,
});
await $git(
"fetch",
await $gitFetchWithDeepen(
["--no-tags", ...(params.isShallow ? ["--depth=1"] : []), "origin", tempBranch],
{ token: params.gitToken }
{ token: params.gitToken },
`before_sha temp branch ${tempBranch}`
);
log.debug(`» fetched before_sha via temp branch ${tempBranch}`);
return true;
@@ -281,6 +282,12 @@ type CheckoutPrBranchParams = GitContext & {
// legitimate git op that's holding the lock.
const STALE_LOCK_AGE_MS = 30_000;
// PR head refs (refs/pull/N/head) sometimes lag the pull_request.opened
// webhook by a few seconds. retry the missing-ref case with backoff
// before giving up — see issue #591.
const PULL_REF_RETRY_DELAYS_MS = [2_000, 5_000, 10_000];
const PULL_REF_MISSING_PATTERN = /couldn't find remote ref pull\/\d+\/head/i;
const GIT_LOCK_PATHS = [
".git/shallow.lock",
".git/index.lock",
@@ -308,6 +315,62 @@ function cleanupStaleGitLocks(): void {
}
}
/**
* Returns false when a PR's current state diverges from what we dispatched
* on (closed/merged, or head SHA differs from pr.headSha). Used to short-
* circuit the pull/N/head retry loop when the ref is missing because the
* PR has moved on, not because of a webhook race.
*
* Network failures here are treated as "still valid" — we'd rather burn the
* retry budget than wrongly abort on a transient API blip.
*
* Note: this answers "should we keep trying?", NOT "will the next fetch
* succeed?". `pulls.get` (REST API) and `pull/N/head` (git ref) are served
* by independent GitHub replicas with their own propagation lag, so
* `pulls.get` reporting an open PR with a matching head SHA does not
* guarantee the git ref is yet visible — and vice versa (see issue #591
* for the original webhook-vs-ref replication-lag context).
*/
async function isPullRequestStillDispatchable(args: {
octokit: Octokit;
owner: string;
repo: string;
pr: PrData;
}): Promise<boolean> {
try {
const { data } = await args.octokit.rest.pulls.get({
owner: args.owner,
repo: args.repo,
pull_number: args.pr.number,
});
if (data.state !== "open") return false;
if (data.head.sha !== args.pr.headSha) return false;
return true;
} catch {
// lenient — don't abort on API hiccups
return true;
}
}
/**
* Throws the friendly clean-abort error when the PR has moved on since
* dispatch. Wraps `isPullRequestStillDispatchable` so the abort message
* lives in one place and is invoked from the inner `catch` around the
* `pull/N/head` fetch on every missing-ref failure.
*/
async function abortIfPullRequestMoved(args: {
octokit: Octokit;
owner: string;
repo: string;
pr: PrData;
}): Promise<void> {
const stillValid = await isPullRequestStillDispatchable(args);
if (stillValid) return;
throw new Error(
`PR #${args.pr.number} is no longer in the state it was at dispatch (likely closed, merged, or force-pushed between webhook fire and run start). aborting checkout — re-trigger the run if this PR is still active.`
);
}
/**
* Shared helper to checkout a PR branch and configure fork remotes.
* Assumes origin remote is already configured with authentication.
@@ -347,9 +410,17 @@ export async function checkoutPrBranch(
toolState.checkoutSha = $("git", ["rev-parse", "HEAD"], { log: false }).trim();
const alreadyOnBranch = toolState.checkoutSha === pr.headSha;
// fetch base branch so origin/<base> exists for diff operations
// fetch base branch so origin/<base> exists for diff operations.
// wrap with deepen-retry: on shallow clones (the actions/checkout default
// is depth=1), repos with deep PR ancestry can't reach the baseRef tip in
// a single round trip, surfacing as `Could not read <sha>` / `remote did
// not send all necessary objects` (issue #656).
log.debug(`» fetching base branch (${pr.baseRef})...`);
await $git("fetch", ["--no-tags", "origin", pr.baseRef], { token: gitToken });
await $gitFetchWithDeepen(
["--no-tags", "origin", pr.baseRef],
{ token: gitToken },
`base branch ${pr.baseRef}`
);
// alreadyOnBranch only matches for repeated checkout_pr calls for the same PR in one session
// (without the tip moving), or if an external setup already checked out the PR head.
@@ -363,11 +434,40 @@ export async function checkoutPrBranch(
// -B creates or resets the branch to match origin/baseBranch
$("git", ["checkout", "-B", pr.baseRef, `origin/${pr.baseRef}`], { log: false });
// fetch PR branch using pull/{n}/head refspec (works for both fork and same-repo PRs)
// fetch PR branch using pull/{n}/head refspec (works for both fork and same-repo PRs).
// two transient classes wrap this fetch:
// - shallow-unreachable (`Could not read <sha>` etc.) — handled by the
// inner `$gitFetchWithDeepen` deepen-retry (one shot, see issue #656)
// - pull/N/head webhook race (`couldn't find remote ref pull/N/head`) —
// handled by the outer retry below (see issue #591)
log.debug(`» fetching PR #${pr.number} (${localBranch})...`);
await $git("fetch", ["--no-tags", "origin", `+pull/${pr.number}/head:${localBranch}`], {
token: gitToken,
});
await retry(
async () => {
try {
await $gitFetchWithDeepen(
["--no-tags", "origin", `+pull/${pr.number}/head:${localBranch}`],
{ token: gitToken },
`PR #${pr.number}`
);
} catch (e) {
// on the webhook race, check whether the PR still matches what we
// dispatched on. if it's been closed/merged or the head SHA moved,
// no amount of retrying will populate the expected ref — surface a
// clean abort error instead of burning the full retry budget.
const msg = e instanceof Error ? e.message : String(e);
if (PULL_REF_MISSING_PATTERN.test(msg)) {
await abortIfPullRequestMoved({ octokit, owner, repo: name, pr });
}
throw e;
}
},
{
delaysMs: PULL_REF_RETRY_DELAYS_MS,
label: `pull/${pr.number}/head fetch`,
shouldRetry: (e) =>
PULL_REF_MISSING_PATTERN.test(e instanceof Error ? e.message : String(e)),
}
);
// checkout the branch
$("git", ["checkout", localBranch], { log: false });
@@ -503,186 +603,234 @@ export async function checkoutPrBranch(
return { hookWarning: postCheckoutHook.warning };
}
/**
* dedupes concurrent `checkout_pr` calls for the same PR. agents (notably
* Sonnet/Claude) occasionally emit duplicate parallel tool_use blocks for the
* same args in one turn; without this, both invocations race
* `checkoutPrBranch` against the same `.git/shallow.lock` and one fails with
* `File exists` (issue #642). cleared in `finally` so subsequent same-PR
* calls re-do the work normally.
*/
const inFlightCheckouts = new Map<number, Promise<CheckoutPrResult>>();
export function CheckoutPrTool(ctx: ToolContext) {
const runCheckout = async (pull_number: number): Promise<CheckoutPrResult> => {
const prResponse = await ctx.octokit.rest.pulls.get({
owner: ctx.repo.owner,
repo: ctx.repo.name,
pull_number,
});
const headRepo = prResponse.data.head.repo;
if (!headRepo) {
throw new Error(`PR #${pull_number} source repository was deleted`);
}
const pr: PrData = {
number: pull_number,
headSha: prResponse.data.head.sha,
headRef: prResponse.data.head.ref,
headRepoFullName: headRepo.full_name,
baseRef: prResponse.data.base.ref,
baseRepoFullName: prResponse.data.base.repo.full_name,
maintainerCanModify: prResponse.data.maintainer_can_modify,
};
const checkoutResult = await checkoutPrBranch(pr, {
octokit: ctx.octokit,
owner: ctx.repo.owner,
name: ctx.repo.name,
gitToken: ctx.gitToken,
toolState: ctx.toolState,
shell: ctx.payload.shell,
postCheckoutScript: ctx.postCheckoutScript,
beforeSha: ctx.toolState.beforeSha,
});
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) {
throw new Error(
"PULLFROG_TEMP_DIR not set - checkout_pr must run in pullfrog action context"
);
}
const headShort = ctx.toolState.checkoutSha!.slice(0, 7);
// compute incremental diff if we have a beforeSha to compare against
let incrementalDiffPath: string | undefined;
if (ctx.toolState.beforeSha && ctx.toolState.checkoutSha) {
const beforeShort = ctx.toolState.beforeSha.slice(0, 7);
const incremental = computeIncrementalDiff({
baseBranch: pr.baseRef,
beforeSha: ctx.toolState.beforeSha,
headSha: ctx.toolState.checkoutSha,
});
if (incremental) {
incrementalDiffPath = join(
tempDir,
`pr-${pull_number}-${beforeShort}-${headShort}-incremental.diff`
);
writeFileSync(incrementalDiffPath, incremental);
log.info(
`» incremental diff computed (${incremental.length} bytes) → ${incrementalDiffPath}`
);
}
}
// fetch PR files and format with line numbers
const formatResult = await fetchAndFormatPrDiff(ctx, pull_number);
const diffPreview = formatResult.content.split("\n").slice(0, 100).join("\n");
log.debug(`formatted diff preview (first 100 lines):\n${diffPreview}`);
const diffPath = join(tempDir, `pr-${pull_number}-${headShort}.diff`);
writeFileSync(diffPath, formatResult.content);
log.debug(`wrote diff to ${diffPath} (${formatResult.content.length} bytes)`);
ctx.toolState.diffCoverage = createDiffCoverageState({
diffPath,
totalLines: countLines({ content: formatResult.content }),
toc: formatResult.toc,
previous: ctx.toolState.diffCoverage,
});
log.debug(
`» diff coverage initialized: diffPath=${diffPath}, totalLines=${ctx.toolState.diffCoverage.totalLines}, tocEntries=${ctx.toolState.diffCoverage.tocEntries.length}`
);
// cache commentable-lines snapshot so review-time validation matches what
// GitHub will anchor to (commit_id=checkoutSha), even if the PR is updated
// between checkout and review.
const cached = new Map<string, ReturnType<typeof commentableLinesForFile>>();
for (const file of formatResult.files) {
cached.set(file.filename, commentableLinesForFile(file.patch));
}
ctx.toolState.commentableLinesByFile = cached;
ctx.toolState.commentableLinesPullNumber = pull_number;
ctx.toolState.commentableLinesCheckoutSha = ctx.toolState.checkoutSha;
const incrementalInstructions = incrementalDiffPath
? ` IMPORTANT: incrementalDiffPath contains ONLY the changes since the last reviewed version ` +
`(computed via range-diff). you MUST read incrementalDiffPath FIRST to understand what changed, ` +
`then use diffPath for full PR context. do NOT skip the incremental diff.`
: "";
// commit metadata relative to the PR base (e.g. main). use origin/<base>
// because the local base ref may not exist after a shallow fetch. cap
// the log so a PR with thousands of commits doesn't blow up the tool
// response. if the base ref can't be resolved (e.g. shallow fetch that
// didn't pull down origin/<base>), degrade gracefully rather than
// failing the whole checkout_pr call over metadata.
const COMMIT_LOG_MAX = 200;
const baseRange = `origin/${pr.baseRef}..HEAD`;
let commitCount = 0;
let commitLog = "";
let commitLogUnavailable = false;
try {
commitCount = parseInt(
$("git", ["rev-list", "--count", baseRange], { log: false }).trim() || "0",
10
);
commitLog = $("git", ["log", "--oneline", `--max-count=${COMMIT_LOG_MAX}`, baseRange], {
log: false,
});
} catch (err) {
commitLogUnavailable = true;
log.debug(
`» unable to compute commit metadata for ${baseRange}: ${err instanceof Error ? err.message : String(err)}`
);
}
const commitLogTruncated = commitCount > COMMIT_LOG_MAX;
const hookWarningInstructions = checkoutResult.hookWarning
? ` HOOK WARNING: the post-checkout lifecycle hook reported a non-fatal failure (see hookWarning). ` +
`decide whether to retry based on the guidance in that field before proceeding.`
: "";
const commitLogInstructions = commitLogUnavailable
? ` NOTE: commit metadata is partial (base ref unreachable, likely a shallow fetch). ` +
`commitCount/commitLog may be 0/empty or incomplete; treat them as "unknown" rather than "no commits", ` +
`and use \`git log\` directly if you need the full history.`
: commitLogTruncated
? ` NOTE: commitLog was capped at ${COMMIT_LOG_MAX} entries out of ${commitCount} commits; ` +
`use \`git log\` directly if you need the full history.`
: "";
return {
success: true,
number: prResponse.data.number,
title: prResponse.data.title,
body: prResponse.data.body,
base: pr.baseRef,
localBranch: `pr-${pull_number}`,
remoteBranch: `refs/heads/${pr.headRef}`,
isFork: pr.headRepoFullName !== pr.baseRepoFullName,
maintainerCanModify: pr.maintainerCanModify,
url: prResponse.data.html_url,
headRepo: pr.headRepoFullName,
diffPath,
incrementalDiffPath,
toc: formatResult.toc,
commitCount,
commitLog,
commitLogTruncated,
commitLogUnavailable,
hookWarning: checkoutResult.hookWarning,
instructions:
`the diff file at diffPath contains a table of contents (TOC) at the top listing every changed file with its line range. ` +
`use the TOC line ranges as your checklist and read specific files from the diff instead of reading the entire file. ` +
`for example, if the TOC says "src/foo.ts → lines 5-42", read lines 5-42 from diffPath to see that file's changes. ` +
`review files selectively based on relevance rather than reading everything sequentially. ` +
`to inspect the PR's changed files, use diffPath — do NOT run \`git diff <base>..<head>\` to re-derive what's already in diffPath. the formatted diff with line numbers is authoritative. ` +
`\`git log\` and \`git diff --stat\` are fine for commit-range overview, and \`git diff\` / \`git diff --cached\` are fine for inspecting *your own* uncommitted changes — but PR review content MUST come from diffPath. ` +
`before your review is submitted, a one-time coverage pre-flight may error listing unread TOC regions. ` +
`retry the same create_pull_request_review call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session. ` +
`the local branch is 'localBranch' (pr-{number}), not the remote branch name. ` +
`when pushing, omit branchName to use the current branch. do not use remoteBranch as a local branch name.` +
incrementalInstructions +
hookWarningInstructions +
commitLogInstructions,
} satisfies CheckoutPrResult;
};
return tool({
name: "checkout_pr",
description:
"Checkout a pull request branch locally. This fetches the PR branch and sets up push configuration for fork PRs. " +
"Returns diffPath pointing to the formatted diff file.",
"Returns diffPath pointing to the formatted diff file. " +
"Example: `checkout_pr({ pull_number: 1234 })`. " +
"Transient fetch timeouts are common — retry the same call up to a few times before treating the failure as terminal. " +
"If the error mentions `.git/shallow.lock: File exists` or `.git/index.lock: File exists`, that's a stale lock from a prior timed-out fetch — remove it via the shell tool (`rm -f .git/shallow.lock .git/index.lock`) and retry.",
parameters: CheckoutPr,
execute: execute(async ({ pull_number }) => {
const prResponse = await ctx.octokit.rest.pulls.get({
owner: ctx.repo.owner,
repo: ctx.repo.name,
pull_number,
});
const headRepo = prResponse.data.head.repo;
if (!headRepo) {
throw new Error(`PR #${pull_number} source repository was deleted`);
const inFlight = inFlightCheckouts.get(pull_number);
if (inFlight) {
log.info(`» checkout_pr({pull_number:${pull_number}}) already in flight — sharing result`);
return inFlight;
}
const pr: PrData = {
number: pull_number,
headSha: prResponse.data.head.sha,
headRef: prResponse.data.head.ref,
headRepoFullName: headRepo.full_name,
baseRef: prResponse.data.base.ref,
baseRepoFullName: prResponse.data.base.repo.full_name,
maintainerCanModify: prResponse.data.maintainer_can_modify,
};
const checkoutResult = await checkoutPrBranch(pr, {
octokit: ctx.octokit,
owner: ctx.repo.owner,
name: ctx.repo.name,
gitToken: ctx.gitToken,
toolState: ctx.toolState,
shell: ctx.payload.shell,
postCheckoutScript: ctx.postCheckoutScript,
beforeSha: ctx.toolState.beforeSha,
});
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) {
throw new Error(
"PULLFROG_TEMP_DIR not set - checkout_pr must run in pullfrog action context"
);
}
const headShort = ctx.toolState.checkoutSha!.slice(0, 7);
// compute incremental diff if we have a beforeSha to compare against
let incrementalDiffPath: string | undefined;
if (ctx.toolState.beforeSha && ctx.toolState.checkoutSha) {
const beforeShort = ctx.toolState.beforeSha.slice(0, 7);
const incremental = computeIncrementalDiff({
baseBranch: pr.baseRef,
beforeSha: ctx.toolState.beforeSha,
headSha: ctx.toolState.checkoutSha,
});
if (incremental) {
incrementalDiffPath = join(
tempDir,
`pr-${pull_number}-${beforeShort}-${headShort}-incremental.diff`
);
writeFileSync(incrementalDiffPath, incremental);
log.info(
`» incremental diff computed (${incremental.length} bytes) → ${incrementalDiffPath}`
// refuse to clobber an active checkout if the working tree is dirty —
// forces the agent to commit/push or discard before switching contexts
// instead of silently overwriting uncommitted work. `issueNumber`
// tracks any issue/PR the agent has touched (issues and PRs share
// GitHub's number space); the guard fires only when the agent is
// switching to a *different* number with a dirty tree, which captures
// the legitimate "stop, you have unsaved work" case regardless of
// whether the prior number was an issue or a PR.
const current = ctx.toolState.issueNumber;
if (current !== undefined && current !== pull_number) {
const dirty = $("git", ["status", "--porcelain"], { log: false }).trim();
if (dirty) {
throw new Error(
`cannot checkout PR #${pull_number} while the working tree has uncommitted changes. ` +
`commit, push, or discard them before switching. dirty paths:\n${dirty}`
);
}
}
// fetch PR files and format with line numbers
const formatResult = await fetchAndFormatPrDiff(ctx, pull_number);
const diffPreview = formatResult.content.split("\n").slice(0, 100).join("\n");
log.debug(`formatted diff preview (first 100 lines):\n${diffPreview}`);
const diffPath = join(tempDir, `pr-${pull_number}-${headShort}.diff`);
writeFileSync(diffPath, formatResult.content);
log.debug(`wrote diff to ${diffPath} (${formatResult.content.length} bytes)`);
ctx.toolState.diffCoverage = createDiffCoverageState({
diffPath,
totalLines: countLines({ content: formatResult.content }),
toc: formatResult.toc,
previous: ctx.toolState.diffCoverage,
});
log.debug(
`» diff coverage initialized: diffPath=${diffPath}, totalLines=${ctx.toolState.diffCoverage.totalLines}, tocEntries=${ctx.toolState.diffCoverage.tocEntries.length}`
);
// cache commentable-lines snapshot so review-time validation matches what
// GitHub will anchor to (commit_id=checkoutSha), even if the PR is updated
// between checkout and review.
const cached = new Map<string, ReturnType<typeof commentableLinesForFile>>();
for (const file of formatResult.files) {
cached.set(file.filename, commentableLinesForFile(file.patch));
}
ctx.toolState.commentableLinesByFile = cached;
ctx.toolState.commentableLinesPullNumber = pull_number;
ctx.toolState.commentableLinesCheckoutSha = ctx.toolState.checkoutSha;
const incrementalInstructions = incrementalDiffPath
? ` IMPORTANT: incrementalDiffPath contains ONLY the changes since the last reviewed version ` +
`(computed via range-diff). you MUST read incrementalDiffPath FIRST to understand what changed, ` +
`then use diffPath for full PR context. do NOT skip the incremental diff.`
: "";
// commit metadata relative to the PR base (e.g. main). use origin/<base>
// because the local base ref may not exist after a shallow fetch. cap
// the log so a PR with thousands of commits doesn't blow up the tool
// response. if the base ref can't be resolved (e.g. shallow fetch that
// didn't pull down origin/<base>), degrade gracefully rather than
// failing the whole checkout_pr call over metadata.
const COMMIT_LOG_MAX = 200;
const baseRange = `origin/${pr.baseRef}..HEAD`;
let commitCount = 0;
let commitLog = "";
let commitLogUnavailable = false;
const promise = runCheckout(pull_number);
inFlightCheckouts.set(pull_number, promise);
try {
commitCount = parseInt(
$("git", ["rev-list", "--count", baseRange], { log: false }).trim() || "0",
10
);
commitLog = $("git", ["log", "--oneline", `--max-count=${COMMIT_LOG_MAX}`, baseRange], {
log: false,
});
} catch (err) {
commitLogUnavailable = true;
log.debug(
`» unable to compute commit metadata for ${baseRange}: ${err instanceof Error ? err.message : String(err)}`
);
return await promise;
} finally {
inFlightCheckouts.delete(pull_number);
}
const commitLogTruncated = commitCount > COMMIT_LOG_MAX;
const hookWarningInstructions = checkoutResult.hookWarning
? ` HOOK WARNING: the post-checkout lifecycle hook reported a non-fatal failure (see hookWarning). ` +
`decide whether to retry based on the guidance in that field before proceeding.`
: "";
const commitLogInstructions = commitLogUnavailable
? ` NOTE: commit metadata is partial (base ref unreachable, likely a shallow fetch). ` +
`commitCount/commitLog may be 0/empty or incomplete; treat them as "unknown" rather than "no commits", ` +
`and use \`git log\` directly if you need the full history.`
: commitLogTruncated
? ` NOTE: commitLog was capped at ${COMMIT_LOG_MAX} entries out of ${commitCount} commits; ` +
`use \`git log\` directly if you need the full history.`
: "";
return {
success: true,
number: prResponse.data.number,
title: prResponse.data.title,
body: prResponse.data.body,
base: pr.baseRef,
localBranch: `pr-${pull_number}`,
remoteBranch: `refs/heads/${pr.headRef}`,
isFork: pr.headRepoFullName !== pr.baseRepoFullName,
maintainerCanModify: pr.maintainerCanModify,
url: prResponse.data.html_url,
headRepo: pr.headRepoFullName,
diffPath,
incrementalDiffPath,
toc: formatResult.toc,
commitCount,
commitLog,
commitLogTruncated,
commitLogUnavailable,
hookWarning: checkoutResult.hookWarning,
instructions:
`the diff file at diffPath contains a table of contents (TOC) at the top listing every changed file with its line range. ` +
`use the TOC line ranges as your checklist and read specific files from the diff instead of reading the entire file. ` +
`for example, if the TOC says "src/foo.ts → lines 5-42", read lines 5-42 from diffPath to see that file's changes. ` +
`review files selectively based on relevance rather than reading everything sequentially. ` +
`to inspect the PR's changed files, use diffPath — do NOT run \`git diff <base>..<head>\` to re-derive what's already in diffPath. the formatted diff with line numbers is authoritative. ` +
`\`git log\` and \`git diff --stat\` are fine for commit-range overview, and \`git diff\` / \`git diff --cached\` are fine for inspecting *your own* uncommitted changes — but PR review content MUST come from diffPath. ` +
`before your review is submitted, a one-time coverage pre-flight may error listing unread TOC regions. ` +
`retry the same create_pull_request_review call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session. ` +
`the local branch is 'localBranch' (pr-{number}), not the remote branch name. ` +
`when pushing, omit branchName to use the current branch. do not use remoteBranch as a local branch name.` +
incrementalInstructions +
hookWarningInstructions +
commitLogInstructions,
} satisfies CheckoutPrResult;
}),
});
}
+85 -3
View File
@@ -65,7 +65,9 @@ export function CreateCommentTool(ctx: ToolContext) {
return tool({
name: "create_issue_comment",
description:
"Create a comment on a GitHub issue or PR. For progress/plan updates on the current run use report_progress instead. Use type: 'Plan' for plan comments.",
"Create a comment on a GitHub issue or PR. " +
'Example: `create_issue_comment({ issueNumber: 1234, body: "Thanks for the report." })`. ' +
"For progress/plan updates on the current run use report_progress instead. Use type: 'Plan' for plan comments.",
parameters: Comment,
execute: execute(async ({ issueNumber, body, type: commentType }) => {
const bodyWithFooter = addFooter(ctx, body);
@@ -78,6 +80,7 @@ export function CreateCommentTool(ctx: ToolContext) {
});
ctx.toolState.wasUpdated = true;
log.info(`» created comment ${result.data.id}`);
if (commentType === "Plan") {
if (result.data.node_id) {
@@ -94,6 +97,7 @@ export function CreateCommentTool(ctx: ToolContext) {
comment_id: result.data.id,
body: bodyWithPlanLink,
});
log.info(`» updated comment ${updateResult.data.id}`);
return {
success: true,
@@ -132,6 +136,7 @@ export function EditCommentTool(ctx: ToolContext) {
comment_id: commentId,
body: bodyWithFooter,
});
log.info(`» updated comment ${result.data.id}`);
return {
success: true,
@@ -307,7 +312,9 @@ export function ReportProgressTool(ctx: ToolContext) {
return tool({
name: "report_progress",
description:
"Share progress on the associated GitHub issue/PR. The first call creates a comment; subsequent calls update it in place. Call this at the end of every run with a brief final summary (1-3 sentences) unless the mode guidance instructs otherwise. The current task list is automatically appended in a collapsible section — do not restate individual steps.",
"Share progress on the associated GitHub issue/PR. The first call creates a comment; subsequent calls update it in place. " +
'Example: `report_progress({ body: "Implemented the auth check and added tests." })`. ' +
"Call this at the end of every run with a brief final summary (1-3 sentences) unless the mode guidance instructs otherwise. The current task list is automatically appended in a collapsible section — do not restate individual steps.",
parameters: ReportProgress,
execute: execute(async (params) => {
let body = params.body;
@@ -339,6 +346,10 @@ export function ReportProgressTool(ctx: ToolContext) {
};
}
if (result.commentId !== undefined) {
log.info(`» ${result.action} comment ${result.commentId}`);
}
if (!params.target_plan_comment) {
ctx.toolState.finalSummaryWritten = true;
}
@@ -391,15 +402,77 @@ export const ReplyToReviewComment = type({
),
});
/**
* decision returned by `duplicateReplyDecision` when a session has already
* posted an identical reply to the same parent review comment.
*/
export interface DuplicateReplyDecision {
kind: "already-replied";
commentId: number;
url: string | undefined;
reason: string;
}
/**
* decide whether a second reply_to_review_comment call in the same session
* is a duplicate of an earlier reply to the same parent comment.
*
* the agent is instructed to call reply_to_review_comment exactly once per
* parent comment per AddressReviews session, but in practice it sometimes
* emits the same call twice. PR #610 reproduced this with Kimi K2:
* identical body posted 3 seconds apart, only one tool_use event in the
* agent log. the second post is always redundant and clutters the PR thread.
*
* we key on (comment_id, bodyWithFooter) so a legitimate follow-up reply
* with different content still goes through. within a single run the
* footer is constant (workflow run + model + jobId), so byte-equal bodies
* catch the stutter without blocking real follow-ups.
*
* mirrors the shape of `duplicateReviewDecision` in mcp/review.ts.
*/
export function duplicateReplyDecision(params: {
existing: { commentId: number; url: string | undefined; bodyWithFooter: string } | undefined;
bodyWithFooter: string;
}): DuplicateReplyDecision | null {
const existing = params.existing;
if (!existing) return null;
if (existing.bodyWithFooter !== params.bodyWithFooter) return null;
return {
kind: "already-replied",
commentId: existing.commentId,
url: existing.url,
reason: `reply ${existing.commentId} with identical body was already posted in this session; ignoring duplicate call`,
};
}
export function ReplyToReviewCommentTool(ctx: ToolContext) {
return tool({
name: "reply_to_review_comment",
description:
"Reply to a PR review comment thread (NOT issue comments — this only works for inline review comments on PR diffs). Call this for EACH comment you address in AddressReviews mode. Keep replies extremely brief (1 sentence max).",
"Reply to a PR review comment thread (NOT issue comments — this only works for inline review comments on PR diffs). " +
'Example: `reply_to_review_comment({ pull_number: 1234, comment_id: 567890, body: "Fixed by adding a null check." })`. ' +
"Call exactly ONCE per parent comment you address in AddressReviews mode — duplicate calls with the same body are a no-op. Keep replies extremely brief (1 sentence max).",
parameters: ReplyToReviewComment,
execute: execute(async ({ pull_number, comment_id, body }) => {
const bodyWithFooter = addFooter(ctx, body);
// guard against duplicate reply submissions in the same session.
// see duplicateReplyDecision for the rationale.
const dup = duplicateReplyDecision({
existing: ctx.toolState.reviewReplies?.get(comment_id),
bodyWithFooter,
});
if (dup) {
log.info(`skipping duplicate review reply: ${dup.reason}`);
return {
success: true,
skipped: true,
reason: dup.reason,
commentId: dup.commentId,
url: dup.url,
};
}
const result = await ctx.octokit.rest.pulls.createReplyForReviewComment({
owner: ctx.repo.owner,
repo: ctx.repo.name,
@@ -407,11 +480,20 @@ export function ReplyToReviewCommentTool(ctx: ToolContext) {
comment_id,
body: bodyWithFooter,
});
log.info(`» created review comment ${result.data.id} (in reply to ${comment_id})`);
// mark progress as updated so error reporting + run-result handling know
// a substantive write happened (used by reportErrorToComment / handleAgentResult)
ctx.toolState.wasUpdated = true;
// record this reply for in-session dedupe of subsequent identical calls.
ctx.toolState.reviewReplies ??= new Map();
ctx.toolState.reviewReplies.set(comment_id, {
commentId: result.data.id,
url: result.data.html_url,
bodyWithFooter,
});
return {
success: true,
commentId: result.data.id,
+2 -1
View File
@@ -15,7 +15,8 @@ export function CommitInfoTool(ctx: ToolContext) {
name: "get_commit_info",
description:
"Retrieve commit metadata and diff via GitHub API. Use this instead of git show for reviewing commits - " +
"it works with shallow clones and shows the actual changes in the commit. Returns diffPath pointing to formatted diff file.",
"it works with shallow clones and shows the actual changes in the commit. Returns diffPath pointing to formatted diff file. " +
'Example: `get_commit_info({ sha: "2a6ab5d" })`.',
parameters: CommitInfo,
execute: execute(async ({ sha }) => {
const response = await ctx.octokit.rest.repos.getCommit({
+25 -6
View File
@@ -176,13 +176,32 @@ export function sanitizeToolForGemini<T extends Tool<any, any>>(tool: T): T {
}
/**
* true when the effective upstream model is served by google's generative
* language API directly (`google/*`), via opencode (`opencode/gemini-*`),
* or via openrouter (`openrouter/google/gemini-*`). slug-substring match
* works because every gemini route's model id contains "gemini".
* true when the effective upstream model is or might become google
* generative language API traffic. matches:
* - direct `google/*`, opencode `opencode/gemini-*`, openrouter
* `openrouter/google/gemini-*` (slug substring "gemini" wins).
* - any unresolved specifier: `undefined`, `"auto"`, or a slug that
* didn't map through the alias registry (no `provider/` prefix).
* these flow through the agent's own auto-select, which may land
* on gemini *after* the MCP server has already registered tools
* at which point sanitization is too late to apply. erring on the
* side of sanitizing is safe: cases 1 + 2 are universally
* compatible JSON-Schema normalizations (enum-only typed string,
* collapsible const-unions string enum); case 3 is gemini-
* specific but only fires on non-collapsible unions, which arktype
* does not emit for our current tool schemas. see issue #676 for
* the prod failure that motivated this widening.
*/
export function isGeminiRouted(ctx: ToolContext): boolean {
const effective = ctx.payload.proxyModel ?? ctx.resolvedModel ?? ctx.payload.model;
if (!effective) return false;
return effective.toLowerCase().includes("gemini");
if (!effective) return true;
const normalized = effective.toLowerCase();
if (normalized.includes("gemini")) return true;
// every concrete model resolved through the registry carries a
// `provider/` prefix (e.g. "anthropic/claude-opus-4-7"). anything
// without a slash is either the literal `"auto"` alias or an
// unrecognized slug that resolveModel logged a warning for — both
// route through the agent's late auto-select, which may pick gemini.
if (!normalized.includes("/")) return true;
return false;
}
+21 -36
View File
@@ -1,10 +1,11 @@
import { regex } from "arkregex";
import { type } from "arktype";
import type { StoredPushDest } from "../toolState.ts";
import { log } from "../utils/cli.ts";
import { $git } from "../utils/gitAuth.ts";
import { $git, $gitFetchWithDeepen } from "../utils/gitAuth.ts";
import { executeLifecycleHook } from "../utils/lifecycle.ts";
import { $ } from "../utils/shell.ts";
import type { StoredPushDest, ToolContext } from "./server.ts";
import type { ToolContext } from "./server.ts";
import { execute, tool } from "./shared.ts";
type PushDestination = {
@@ -217,10 +218,12 @@ export function PushBranchTool(ctx: ToolContext) {
name: "push_branch",
description:
"Push the current branch to the remote repository. Omit branchName to push the current branch (recommended). " +
'Example: `push_branch({})` to push the current branch. Example: `push_branch({ branchName: "pr-1" })` to push a specific local branch. ' +
"If specifying branchName, use the LOCAL branch name (e.g., 'pr-1'), not the remote branch name. " +
"The correct remote and remote branch are determined automatically from branch config set by checkout_pr. " +
"Requires a clean working tree. Runs the repository prepush hook (if configured) before the network push — hook failure means tests/lint or similar in that script failed, not necessarily a Pullfrog timeout. " +
"Never force push unless explicitly requested. Pushes to the default branch are blocked in restricted mode.",
"Never force push unless explicitly requested. Pushes to the default branch are blocked in restricted mode. " +
"If the response reports a timeout, the underlying push may have actually succeeded — verify with `git log origin/<branch>` (or this tool with command 'log') before retrying, otherwise you'll push a duplicate.",
parameters: PushBranch,
execute: execute(async ({ branchName, force }) => {
// permission check
@@ -351,6 +354,11 @@ export function PushBranchTool(ctx: ToolContext) {
throw lastErr instanceof Error ? lastErr : new Error(String(lastErr));
}
const pushedSha = $("git", ["rev-parse", "HEAD"], { log: false }).trim();
log.info(
`» pushed branch ${branch} to ${pushDest.remoteName}/${pushDest.remoteBranch} (sha ${pushedSha})`
);
return {
success: true,
branch,
@@ -445,7 +453,10 @@ export function GitTool(ctx: ToolContext) {
return tool({
name: "git",
description:
"Run git commands. For push/fetch, use the dedicated MCP tools (push_branch, git_fetch). " +
"Run a git subcommand. `command` is a single subcommand; flags and positional args go in `args`. " +
'Example: `git({ command: "log", args: ["--oneline", "-n", "20"] })`. ' +
'Example: `git({ command: "diff", args: ["origin/main..HEAD"] })`. ' +
"For push/fetch, use the dedicated MCP tools (push_branch, git_fetch). " +
"git pull is not available — use git_fetch then this tool with command 'merge'.",
parameters: Git,
execute: execute(async (params) => {
@@ -499,25 +510,12 @@ const GitFetch = type({
depth: type.number.describe("Fetch depth (for shallow clones)").optional(),
});
// when an agent-supplied depth is too shallow to reach the merge base, git
// surfaces "Could not read <sha>" and "remote did not send all necessary
// objects". detect both wordings so a single deepen retry can recover before
// the error reaches the agent (issue #564). git emits the full OID via
// oid_to_hex, so the bound is 40 (SHA-1) or 64 (SHA-256).
const SHALLOW_UNREACHABLE_PATTERNS: RegExp[] = [
/Could not read [a-f0-9]{40,64}/,
/remote did not send all necessary objects/,
];
// large enough to clear the merge base on most real-world PRs without
// downloading the full history; matches the fallback used by checkoutPrBranch
// when the compare API is unavailable.
const DEEPEN_RETRY_DEPTH = 1000;
export function GitFetchTool(ctx: ToolContext) {
return tool({
name: "git_fetch",
description: "Fetch refs from remote repository. Use this instead of git fetch directly.",
description:
"Fetch refs from remote repository. Use this instead of git fetch directly. " +
'Example: `git_fetch({ ref: "main" })`. With depth: `git_fetch({ ref: "pull/1234/head", depth: 1 })`.',
parameters: GitFetch,
execute: execute(async (params) => {
rejectIfLeadingDash(params.ref, "ref");
@@ -525,22 +523,7 @@ export function GitFetchTool(ctx: ToolContext) {
if (params.depth !== undefined) {
fetchArgs.push(`--depth=${params.depth}`);
}
try {
await $git("fetch", fetchArgs, { token: ctx.gitToken });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const isShallowUnreachable = SHALLOW_UNREACHABLE_PATTERNS.some((p) => p.test(msg));
const isShallow =
isShallowUnreachable &&
$("git", ["rev-parse", "--is-shallow-repository"], { log: false }).trim() === "true";
if (!isShallow) throw err;
log.info(
`» git_fetch hit shallow-unreachable error, retrying with --deepen=${DEEPEN_RETRY_DEPTH}`
);
await $git("fetch", [`--deepen=${DEEPEN_RETRY_DEPTH}`, "--no-tags", "origin", params.ref], {
token: ctx.gitToken,
});
}
await $gitFetchWithDeepen(fetchArgs, { token: ctx.gitToken }, "git_fetch");
return { success: true, ref: params.ref };
}),
});
@@ -595,6 +578,7 @@ export function DeleteBranchTool(ctx: ToolContext) {
await $git("push", ["origin", "--delete", `refs/heads/${params.branchName}`], {
token: ctx.gitToken,
});
log.info(`» deleted branch ${params.branchName}`);
return { success: true, deleted: params.branchName };
}),
});
@@ -625,6 +609,7 @@ export function PushTagsTool(ctx: ToolContext) {
await $git("push", pushArgs, {
token: ctx.gitToken,
});
log.info(`» pushed tag ${params.tag}`);
return { success: true, tag: params.tag };
}),
});
+3
View File
@@ -1,4 +1,5 @@
import { type } from "arktype";
import { log } from "../utils/cli.ts";
import { fixDoubleEscapedString } from "../utils/fixDoubleEscapedString.ts";
import { patchWorkflowRunFields } from "../utils/patchWorkflowRunFields.ts";
import type { ToolContext } from "./server.ts";
@@ -32,6 +33,8 @@ export function IssueTool(ctx: ToolContext) {
assignees: params.assignees ?? [],
});
log.info(`» created issue #${result.data.number} (id ${result.data.id})`);
const nodeId = result.data.node_id;
if (typeof nodeId === "string" && nodeId.length > 0) {
await patchWorkflowRunFields(ctx, {
+2 -1
View File
@@ -10,7 +10,8 @@ export function GetIssueCommentsTool(ctx: ToolContext) {
return tool({
name: "get_issue_comments",
description:
"Get all comments for a GitHub issue. Returns all comments including the issue body and all subsequent discussion comments.",
"Get all comments for a GitHub issue. Returns all comments including the issue body and all subsequent discussion comments. " +
"Example: `get_issue_comments({ issue_number: 1234 })`.",
parameters: GetIssueComments,
execute: execute(async ({ issue_number }) => {
// set issue context
+3 -1
View File
@@ -9,7 +9,9 @@ export const IssueInfo = type({
export function IssueInfoTool(ctx: ToolContext) {
return tool({
name: "get_issue",
description: "Retrieve GitHub issue information by issue number",
description:
"Retrieve GitHub issue information by issue number. " +
"Example: `get_issue({ issue_number: 1234 })`.",
parameters: IssueInfo,
execute: execute(async ({ issue_number }) => {
const issue = await ctx.octokit.rest.issues.get({
+2
View File
@@ -1,4 +1,5 @@
import { type } from "arktype";
import { log } from "../utils/cli.ts";
import type { ToolContext } from "./server.ts";
import { execute, tool } from "./shared.ts";
@@ -20,6 +21,7 @@ export function AddLabelsTool(ctx: ToolContext) {
issue_number,
labels,
});
log.info(`» added labels [${labels.join(", ")}] to issue #${issue_number}`);
return {
success: true,
-41
View File
@@ -1,41 +0,0 @@
import { type } from "arktype";
import { apiFetch } from "../utils/apiFetch.ts";
import type { ToolContext } from "./server.ts";
import { execute, tool } from "./shared.ts";
const UpdateLearningsParams = type({
learnings: type.string.describe(
"the FULL merged learnings as a flat bullet list. each line starts with `- `. one discrete, actionable fact per bullet. combine existing bullets from the prompt with your new discoveries. deduplicate — if an existing bullet covers the same fact, update it in place rather than adding a new one. drop bullets that are clearly wrong or no longer relevant to the current codebase. keep the list focused and concise."
),
});
export function UpdateLearningsTool(ctx: ToolContext) {
return tool({
name: "update_learnings",
description:
"persist operational learnings about this repository (setup steps, test commands, key conventions, patterns). ONLY call this when you have high confidence the information is correct and broadly useful for future runs — not for one-off findings or uncertain observations. format: flat bullet list (`- ` per line, one fact per bullet). pass the FULL merged list — combine existing learnings from the prompt with new discoveries. deduplicate, and drop bullets that are clearly wrong or no longer relevant to the current codebase.",
parameters: UpdateLearningsParams,
execute: execute(async (params) => {
const response = await apiFetch({
path: `/api/repo/${ctx.repo.owner}/${ctx.repo.name}/learnings`,
method: "PATCH",
headers: {
authorization: `Bearer ${ctx.apiToken}`,
"content-type": "application/json",
},
body: JSON.stringify({
learnings: params.learnings,
model: ctx.toolState.model,
}),
signal: AbortSignal.timeout(10_000),
});
if (!response.ok) {
const error = await response.text();
throw new Error(`failed to update learnings: ${error}`);
}
return { success: true };
}),
});
}
+2
View File
@@ -48,6 +48,7 @@ export function UpdatePullRequestBodyTool(ctx: ToolContext) {
pull_number: params.pull_number,
body: bodyWithFooter,
});
log.info(`» updated pull request #${result.data.number}`);
ctx.toolState.wasUpdated = true;
@@ -80,6 +81,7 @@ export function CreatePullRequestTool(ctx: ToolContext) {
base: params.base,
draft: params.draft ?? false,
});
log.info(`» created pull request #${result.data.number} (id ${result.data.id})`);
// best-effort: request review from the user who triggered the workflow
const reviewer = ctx.payload.triggerer;
+3 -1
View File
@@ -30,7 +30,9 @@ export function PullRequestInfoTool(ctx: ToolContext) {
return tool({
name: "get_pull_request",
description:
"Retrieve PR metadata (title, body, state, branches, author, labels, linked issues). To checkout a PR branch locally, use checkout_pr instead.",
"Retrieve PR metadata (title, body, state, branches, author, labels, linked issues). " +
"Example: `get_pull_request({ pull_number: 1234 })`. " +
"To checkout a PR branch locally, use checkout_pr instead.",
parameters: PullRequestInfo,
execute: execute(async ({ pull_number }) => {
// fetch REST and GraphQL in parallel
+17 -6
View File
@@ -1,6 +1,7 @@
import type { RestEndpointMethodTypes } from "@octokit/rest";
import { type } from "arktype";
import { formatMcpToolRef } from "../external.ts";
import type { CommentableLines } from "../toolState.ts";
import { getApiUrl } from "../utils/apiUrl.ts";
import { buildPullfrogFooter } from "../utils/buildPullfrogFooter.ts";
import { log } from "../utils/cli.ts";
@@ -16,6 +17,8 @@ import { deleteProgressComment } from "./comment.ts";
import type { ToolContext } from "./server.ts";
import { execute, tool } from "./shared.ts";
export type { CommentableLines };
function getHttpStatus(err: unknown): number | undefined {
if (typeof err !== "object" || err === null) return undefined;
const status = (err as Record<string, unknown>).status;
@@ -46,7 +49,6 @@ export function isTransientReviewError(err: unknown): boolean {
export const TRANSIENT_REVIEW_RETRY_DELAYS_MS = [1_000, 3_000];
type PullFile = RestEndpointMethodTypes["pulls"]["listFiles"]["response"]["data"][number];
export type CommentableLines = { RIGHT: Set<number>; LEFT: Set<number> };
/**
* parse a PR file's patch to determine which line numbers on each side are
@@ -314,18 +316,20 @@ export const CreatePullRequestReview = type({
.optional(),
approved: type.boolean
.describe(
"Set to true to submit as an approval. ONLY when the review contains no actionable feedback — neither inline comments nor actionable content in the body. Defaults to false (comment-only review). Rejections are not supported."
"Set to true to submit as an approval. Use for both 'no issues found' and informational `> [!NOTE]` reviews where the PR is mergeable as-is and nothing in the body warrants code changes — approving also suppresses the Fix-button footer affordance so users don't dispatch a fix run on non-actionable feedback. Reserve approved: false for `> [!IMPORTANT]` (recommended changes) and `> [!CAUTION]` (critical) reviews. Defaults to false (comment-only review). Rejections are not supported."
)
.optional(),
commit_id: type.string
.describe("Optional SHA of the commit being reviewed. Defaults to latest.")
.describe(
"Optional SHA of the commit being reviewed. Defaults to latest. Must be the FULL 40-character SHA — abbreviated SHAs are rejected by GitHub with `422 Unprocessable Entity`. The PR-synchronize event payload's `head_sha` is already full-length."
)
.optional(),
comments: type({
path: type.string.describe(
"The file path to comment on (relative to repo root). Must be a file that appears in the PR diff."
),
line: type.number.describe(
"Line number to comment on. For multi-line ranges, this is the end line. Use NEW column from diff format."
"Line number to comment on. For multi-line ranges, this is the end line. Use NEW column from diff format. Must sit inside a `@@` hunk in the PR diff — anchors on context-only or untouched lines are dropped silently (the rest of the review still posts; dropped entries are reported under `droppedComments` in the response)."
),
side: type
.enumerated("LEFT", "RIGHT")
@@ -343,7 +347,7 @@ export const CreatePullRequestReview = type({
.optional(),
start_line: type.number
.describe(
"Start line for multi-line comment ranges. Omit for single-line comments. The range [start_line, line] defines which lines a suggestion replaces."
"Start line for multi-line comment ranges. Omit for single-line comments. The range [start_line, line] defines which lines a suggestion replaces. Both `start_line` and `line` must sit inside the same `@@` hunk — a `start_line` outside the hunk causes the whole comment to be dropped even when `line` is valid. If you need to comment on context just above/below a hunk, shrink the range to a single line that is provably modified."
)
.optional(),
})
@@ -359,6 +363,7 @@ export function CreatePullRequestReviewTool(ctx: ToolContext) {
name: "create_pull_request_review",
description:
"Submit a review for an existing pull request. " +
'Example: `create_pull_request_review({ pull_number: 1234, body: "LGTM", approved: true, comments: [{ path: "src/api.ts", line: 42, body: "nit: rename" }] })`. ' +
"Each call creates a permanent, visible review on the PR — NEVER submit test or diagnostic reviews. " +
"Reviews with no body AND no comments are silently skipped (nothing to post). " +
"IMPORTANT: 95%+ of feedback should be in 'comments' array with file paths and line numbers. " +
@@ -584,6 +589,7 @@ export function CreatePullRequestReviewTool(ctx: ToolContext) {
}
const reviewId = result.data.id;
const reviewNodeId = result.data.node_id;
log.info(`» created review ${reviewId} on pull request #${pull_number}`);
// reviewedSha = what the agent actually reviewed (checkout SHA), not the
// submission anchor (current HEAD). this ensures postReviewCleanup dispatches
@@ -706,7 +712,9 @@ function runDiffCoveragePreflight(params: { ctx: ToolContext }): void {
.join("\n");
throw new Error(
`diff coverage pre-flight: some TOC regions were not read before review submission. ` +
`this is a one-time nudge — optionally read the ranges below from ${coverageState.diffPath}, then call create_pull_request_review again with the same arguments. ` +
`this is a one-time nudge — read the ranges below from ${coverageState.diffPath} on a best-effort basis, then call create_pull_request_review again. ` +
`you are NOT obligated to read generated artifacts (lockfiles like pnpm-lock.yaml / package-lock.json / yarn.lock / Cargo.lock; codegen output like *.gen.*, *.pb.go, *.generated.*; snapshot/fixture dirs like __snapshots__/; migration metadata like drizzle/meta/, prisma migration SQL). ` +
`if every unread region is generated, retry immediately without reading. ` +
`this pre-flight will not block again in this review session.\n\n` +
`unread TOC regions:\n${unreadText}\n\n` +
`${coverageState.lastBreakdown}`
@@ -841,6 +849,9 @@ async function createAndSubmitWithFooter(
// API_URL is misconfigured, and future footer-building changes could
// introduce new throw paths. keep the whole body wrapped.
try {
// Fix buttons are suppressed on approving reviews — those are mergeable
// by definition (either "no issues found" or `> [!NOTE]` informational
// observations), so dispatching a fix run would be a UX trap.
const customParts: string[] = [];
if (!opts.approved) {
const apiUrl = getApiUrl();
+4 -2
View File
@@ -602,6 +602,7 @@ export function GetReviewCommentsTool(ctx: ToolContext) {
name: "get_review_comments",
description:
"Get review comments for a pull request review with full thread context. " +
"Example: `get_review_comments({ pull_number: 1234, review_id: 567890 })`. " +
"Automatically filters to approved comments when applicable. " +
"Returns a TOC and commentsPath pointing to a markdown file with full comment details.",
parameters: GetReviewComments,
@@ -673,7 +674,8 @@ export function ListPullRequestReviewsTool(ctx: ToolContext) {
return tool({
name: "list_pull_request_reviews",
description:
"List all reviews for a pull request. Returns all reviews including approvals, request changes, and comments.",
"List all reviews for a pull request. Returns all reviews including approvals, request changes, and comments. " +
"Example: `list_pull_request_reviews({ pull_number: 1234 })`.",
parameters: ListPullRequestReviews,
execute: execute(async (params) => {
const reviews = await ctx.octokit.paginate(ctx.octokit.rest.pulls.listReviews, {
@@ -735,7 +737,7 @@ export function ResolveReviewThreadTool(ctx: ToolContext) {
});
const thread = response.resolveReviewThread.thread;
log.debug(`resolved thread ${thread.id}, isResolved=${thread.isResolved}`);
log.info(`» resolved review thread ${thread.id}`);
return {
thread_id: thread.id,
+7 -5
View File
@@ -24,13 +24,14 @@ function buildModeOverrides(t: (name: string) => string): Record<string, string>
An existing plan comment was found for this issue. Update that comment with the revised plan do not create a new plan comment.
1. Use \`previousPlanBody\` from this response as the plan to revise; do not call \`get_issue\` or \`get_issue_comments\`.
2. Revise the plan based on the user's request:
1. **task list**: create your task list for this run as your first action.
2. Use \`previousPlanBody\` from this response as the plan to revise; do not call \`get_issue\` or \`get_issue_comments\`.
3. Revise the plan based on the user's request:
- incorporate the current plan (\`previousPlanBody\`) and the user's revision request
- gather relevant codebase context (file paths, architecture notes from AGENTS.md)
- produce a structured plan with clear milestones
3. Call \`${t("report_progress")}\` with the full revised plan text and \`{ target_plan_comment: true }\` so it updates the existing plan comment (not the progress comment).
4. Then post a short note to the progress comment (e.g. "Plan has been updated in the comment above.") via \`${t("report_progress")}\` so it is not left as "Leaping...".`,
4. Call \`${t("report_progress")}\` with the full revised plan text and \`{ target_plan_comment: true }\` so it updates the existing plan comment (not the progress comment).
5. Then post a short note to the progress comment (e.g. "Plan has been updated in the comment above.") via \`${t("report_progress")}\` so it is not left as "Leaping...".`,
};
}
@@ -120,7 +121,8 @@ export function SelectModeTool(ctx: ToolContext) {
return tool({
name: "select_mode",
description:
"Select a mode and receive step-by-step guidance on how to handle the task. Call this to understand the best workflow for the current mode.",
"Select a mode and receive step-by-step guidance on how to handle the task. Call this to understand the best workflow for the current mode. " +
'Example: `select_mode({ mode: "Review" })` or `select_mode({ mode: "Plan", issue_number: 1234 })`.',
parameters: SelectModeParams,
execute: execute(async (params) => {
if (ctx.toolState.selectedMode) {
+3 -129
View File
@@ -3,23 +3,14 @@ import "./arkConfig.ts";
import { createServer } from "node:net";
import { setTimeout as sleep } from "node:timers/promises";
import { FastMCP, type Tool } from "fastmcp";
import type { AgentUsage } from "../agents/index.ts";
import { type AgentId, pullfrogMcpName } from "../external.ts";
import type { Mode } from "../modes.ts";
import type { PrepResult } from "../prep/index.ts";
import type { ToolState } from "../toolState.ts";
import { closeBrowserDaemon } from "../utils/browser.ts";
import { log } from "../utils/cli.ts";
import type { DiffCoverageState } from "../utils/diffCoverage.ts";
import type { OctokitWithPlugins } from "../utils/github.ts";
import type { ResolvedPayload } from "../utils/payload.ts";
import {
type ProgressComment,
type ProgressCommentType,
parseProgressComment,
} from "../utils/progressComment.ts";
import type { AccountPlan } from "../utils/runContext.ts";
import type { RunContextData } from "../utils/runContextData.ts";
import type { TodoTracker } from "../utils/todoTracking.ts";
import { CheckoutPrTool } from "./checkout.ts";
import { GetCheckSuiteLogsTool } from "./checkSuite.ts";
import {
@@ -39,11 +30,9 @@ import { GetIssueCommentsTool } from "./issueComments.ts";
import { GetIssueEventsTool } from "./issueEvents.ts";
import { IssueInfoTool } from "./issueInfo.ts";
import { AddLabelsTool } from "./labels.ts";
import { UpdateLearningsTool } from "./learnings.ts";
import { SetOutputTool } from "./output.ts";
import { CreatePullRequestTool, UpdatePullRequestBodyTool } from "./pr.ts";
import { PullRequestInfoTool } from "./prInfo.ts";
import type { CommentableLines } from "./review.ts";
import { CreatePullRequestReviewTool } from "./review.ts";
import {
GetReviewCommentsTool,
@@ -55,120 +44,6 @@ import { addTools } from "./shared.ts";
import { KillBackgroundTool, ShellTool } from "./shell.ts";
import { UploadFileTool } from "./upload.ts";
export type BackgroundProcess = {
pid: number;
outputPath: string;
pidPath: string;
};
export type BrowserDaemon = { binDir: string; error?: never } | { binDir?: never; error: string };
export type StoredPushDest = {
remoteName: string;
remoteBranch: string;
localBranch: string;
};
export interface ToolState {
// where we're allowed to push - base repo initially, fork URL for fork PRs
// set by setupGit, updated by checkout_pr. always set before push validation.
pushUrl?: string;
// push destination set by checkout_pr - used as primary source in push_branch
// because git config reads can fail in certain environments
pushDest?: StoredPushDest;
// issue or PR number (same number space in GitHub)
issueNumber?: number;
// PR HEAD sha at checkout time — used to detect new commits pushed during a review
checkoutSha?: string;
// commentable lines per file at checkoutSha — captured during checkout_pr so
// review-time inline-comment validation matches the diff GitHub will anchor
// to (commit_id=checkoutSha). without this, a PR update between checkout and
// review would make listFiles (latest HEAD) disagree with the anchor,
// silently dropping valid comments or letting invalid ones through.
//
// commentableLinesPullNumber records WHICH PR this snapshot belongs to. if
// the agent checks out PR B and then reviews PR A in the same session, the
// cached snapshot for B would silently mis-validate A's comments — keying
// by PR number forces a re-fetch when the target changes.
//
// commentableLinesCheckoutSha pins the snapshot to the SHA it was built
// against. if a second checkout_pr for the SAME PR bumps checkoutSha but
// fails before repopulating the cache (e.g., listFiles rate-limits), the
// stale snapshot would silently mis-validate comments against the new SHA.
// comparing both fields forces a re-fetch when either moves.
commentableLinesByFile?: Map<string, CommentableLines>;
commentableLinesPullNumber?: number;
commentableLinesCheckoutSha?: string | undefined;
// SHA to diff incrementally against — set from event payload on first checkout,
// then from checkoutSha when review.ts detects new commits mid-review
beforeSha?: string;
selectedMode?: string;
backgroundProcesses: Map<string, BackgroundProcess>;
browserDaemon?: BrowserDaemon | undefined;
review?: {
id: number;
nodeId: string;
reviewedSha: string | undefined;
};
dependencyInstallation?: {
status: "not_started" | "in_progress" | "completed" | "failed";
promise: Promise<PrepResult[]> | undefined;
results: PrepResult[] | undefined;
};
// undefined = no comment yet, object = active comment, null = deliberately deleted
progressComment: ProgressComment | null | undefined;
// immutable snapshot: true if a progress comment was pre-created at init time.
// survives deleteProgressComment so handleAgentResult can still detect "expected but never reported".
hadProgressComment: boolean;
lastProgressBody?: string;
wasUpdated?: boolean;
// set after a non-plan report_progress successfully writes the final summary.
// decoupled from todoTracker.enabled so cleanup detection survives API failures.
finalSummaryWritten?: boolean;
// set by select_mode when Plan + issue_number and plan-comment API returns existing plan (for report_progress target_plan_comment)
existingPlanCommentId?: number;
previousPlanBody?: string;
// absolute path to the PR summary markdown file the agent edits in place.
// seeded by main.ts before the agent starts when payload.generateSummary is set;
// read back at end-of-run to persist to DB.
summaryFilePath?: string;
// exact bytes of the seeded snapshot file at run start. compared against
// the file content at end-of-run to detect "agent never touched it" — in
// that case persistSummary skips the DB write (saving the seed verbatim
// would either re-write what the DB already has, on incremental runs, or
// serialize the placeholder scaffold, on first runs).
summarySeed?: string;
// set to true after persistSummary completes once. prevents the error-path
// call (which exists so a successful agent edit before a crash still gets
// persisted) from redundantly re-running the DB PATCH on the
// success-then-late-throw path.
summaryPersistAttempted?: boolean;
output?: string;
usageEntries: AgentUsage[];
model?: string | undefined;
todoTracker?: TodoTracker | undefined;
diffCoverage?: DiffCoverageState | undefined;
}
interface InitToolStateParams {
progressComment: { id: string; type: ProgressCommentType } | undefined;
}
export function initToolState(params: InitToolStateParams): ToolState {
const resolved = parseProgressComment(params.progressComment);
if (resolved) {
log.info(`» using pre-created progress comment: ${resolved.id} (${resolved.type})`);
}
return {
progressComment: resolved,
hadProgressComment: !!resolved,
backgroundProcesses: new Map(),
usageEntries: [],
};
}
export interface ToolContext {
agentId: AgentId;
repo: RunContextData["repo"];
@@ -189,8 +64,8 @@ export interface ToolContext {
tmpdir: string;
// repo-level OSS flag + account-level billing plan. together they decide
// whether pullfrog is paying for marginal infra — see isInfraCovered in
// utils/runContext.ts. plan gating for things like update_learnings is
// enforced server-side via 402, so we pass plan along mostly for future
// utils/runContext.ts. plan gating for endpoints like the learnings PATCH
// is enforced server-side via 402, so we pass plan along mostly for future
// use / observability. see wiki/pricing.md.
oss: boolean;
plan: AccountPlan;
@@ -288,7 +163,6 @@ function buildOrchestratorTools(ctx: ToolContext, outputSchema?: JsonSchema): To
DeleteBranchTool(ctx),
CreatePullRequestTool(ctx),
UpdatePullRequestBodyTool(ctx),
UpdateLearningsTool(ctx),
];
}
+27 -3
View File
@@ -15,7 +15,9 @@ import { execute, tool } from "./shared.ts";
export const ShellParams = type({
command: "string",
description: "string",
"timeout?": "number",
"timeout?": type.number.describe(
"Timeout in MILLISECONDS (not seconds). Default 30000 (30s), max 120000 (2m). e.g. timeout: 180000 for 3 minutes; timeout: 180 means 180ms and will kill the process almost immediately."
),
"working_directory?": "string",
"background?": "boolean",
});
@@ -174,6 +176,23 @@ function getTempDir(): string {
return tempDir;
}
/** chars of shell output kept inline in the agent reply. anything past this
* blows the agent's context budget on commands that dump big logs (test
* runners, build tools, grep on large trees), so the overflow is spilled
* to a tempfile the agent can re-read selectively (cat/tail/grep). */
export const MAX_OUTPUT_CHARS = 5000;
/** if `output` exceeds `MAX_OUTPUT_CHARS`, persist the full body to a
* tempfile and return the last `MAX_OUTPUT_CHARS` prefixed with a sentinel
* pointing at the saved path. otherwise return as-is. */
function capOutput(output: string): string {
if (output.length <= MAX_OUTPUT_CHARS) return output;
const fullPath = join(getTempDir(), `shell-${randomUUID().slice(0, 8)}.log`);
writeFileSync(fullPath, output);
const elided = output.length - MAX_OUTPUT_CHARS;
return `... [${elided} chars truncated; full output saved to ${fullPath}] ...\n${output.slice(-MAX_OUTPUT_CHARS)}`;
}
/** detect git as a command invocation (not as part of another word like .gitignore) */
function isGitCommand(command: string): boolean {
const trimmed = command.trim();
@@ -187,11 +206,15 @@ export function ShellTool(ctx: ToolContext) {
name: "shell",
description: `Execute shell commands securely. Environment is filtered to remove API keys and secrets.
Example: \`shell({ command: "pnpm test", description: "run the test suite" })\`.
Use this tool to:
- Run shell commands (ls, cat, grep, find, etc.)
- Execute build tools (npm, pnpm, cargo, make, etc.)
- Run tests and linters
Output is capped at ${MAX_OUTPUT_CHARS} chars: if exceeded, only the tail is returned and the full body is saved to a tempfile (path included in the response). Re-read the tempfile with cat/tail/grep when you need more.
Do NOT use this tool for git commands use the dedicated git tools instead.`,
parameters: ShellParams,
execute: execute(async (params) => {
@@ -297,13 +320,14 @@ Do NOT use this tool for git commands — use the dedicated git tools instead.`,
: `[timed out after ${timeout}ms]`;
const finalExitCode = exitCode ?? (timedOut ? 124 : -1);
const trimmed = output.trim();
if (finalExitCode !== 0) {
log.info(`shell command failed with exit code ${finalExitCode}: ${params.command}`);
if (output) log.info(`output: ${output.trim()}`);
if (trimmed) log.info(`output: ${trimmed}`);
}
return {
output: output.trim(),
output: capOutput(trimmed),
exit_code: finalExitCode,
timed_out: timedOut,
};
+3
View File
@@ -3,6 +3,7 @@ import * as path from "node:path";
import { type } from "arktype";
import { fileTypeFromBuffer } from "file-type";
import { apiFetch } from "../utils/apiFetch.ts";
import { log } from "../utils/cli.ts";
import type { ToolContext } from "./server.ts";
import { execute, tool } from "./shared.ts";
@@ -65,6 +66,8 @@ export function UploadFileTool(ctx: ToolContext) {
throw new Error(`failed to upload file: ${uploadResponse.statusText}`);
}
log.info(`» uploaded file ${publicUrl}`);
return { success: true, publicUrl, filename, contentLength, contentType };
}),
});
+60 -2
View File
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import {
getModelEnvVars,
getModelProvider,
isBedrockAnthropicId,
modelAliases,
parseModel,
providers,
@@ -55,13 +56,13 @@ describe("getModelEnvVars", () => {
it("returns empty env vars for free opencode models", () => {
expect(getModelEnvVars("opencode/big-pickle")).toEqual([]);
expect(getModelEnvVars("opencode/gpt-5-nano")).toEqual([]);
expect(getModelEnvVars("opencode/mimo-v2-pro-free")).toEqual([]);
expect(getModelEnvVars("opencode/minimax-m2.5-free")).toEqual([]);
});
it("still requires OPENCODE_API_KEY for non-free opencode models", () => {
expect(getModelEnvVars("opencode/claude-opus")).toEqual(["OPENCODE_API_KEY"]);
expect(getModelEnvVars("opencode/gpt-5-nano")).toEqual(["OPENCODE_API_KEY"]);
});
});
@@ -175,7 +176,12 @@ describe("modelAliases registry", () => {
it("has exactly one preferred model per provider", () => {
for (const providerKey of Object.keys(providers)) {
const preferred = modelAliases.filter((a) => a.provider === providerKey && a.preferred);
// routing-only providers (bedrock) deliberately have no preferred
// model — the user picks the actual model via a per-run env var, so
// there's no "preferred default" to surface to auto-select.
const aliases = modelAliases.filter((a) => a.provider === providerKey);
if (aliases.every((a) => a.routing)) continue;
const preferred = aliases.filter((a) => a.preferred);
expect(preferred.length, `${providerKey} should have exactly 1 preferred model`).toBe(1);
}
});
@@ -190,6 +196,10 @@ describe("modelAliases registry", () => {
it("all resolve values follow provider/model format", () => {
for (const alias of modelAliases) {
// routing slugs use a sentinel `resolve` (e.g. "bedrock") that's never
// passed to a CLI directly — the harness reads a separate env var to
// get the real model ID. format check doesn't apply.
if (alias.routing) continue;
expect(alias.resolve).toContain("/");
}
});
@@ -200,6 +210,54 @@ describe("modelAliases registry", () => {
});
});
describe("isBedrockAnthropicId", () => {
it("matches geo-prefixed Anthropic foundation IDs", () => {
expect(isBedrockAnthropicId("us.anthropic.claude-opus-4-7")).toBe(true);
expect(isBedrockAnthropicId("eu.anthropic.claude-sonnet-4-6")).toBe(true);
expect(isBedrockAnthropicId("global.anthropic.claude-haiku-4-5-20251001-v1:0")).toBe(true);
});
it("matches in-region Anthropic foundation IDs", () => {
expect(isBedrockAnthropicId("anthropic.claude-opus-4-7")).toBe(true);
});
it("rejects non-Anthropic foundation IDs", () => {
expect(isBedrockAnthropicId("amazon.nova-pro-v1:0")).toBe(false);
expect(isBedrockAnthropicId("us.meta.llama4-scout-17b-instruct-v1:0")).toBe(false);
expect(isBedrockAnthropicId("deepseek.v3.2")).toBe(false);
});
// regression: PR #720 review caught that a substring-only match was
// fragile for inference-profile ARNs (which BEDROCK_MODEL_ID accepts per
// the AWS docs). ARN names are user-chosen — both directions of the
// heuristic could break depending on what name the operator picked.
// We anchor on a discrete dot-segment match (case-insensitive) instead.
it("ignores 'anthropic' substrings inside non-segment text", () => {
// ARN whose user-chosen profile name happens to contain "anthropic" as
// part of a longer word — would route to claude-code under naive
// includes("anthropic") even though the backing model is unknown.
expect(
isBedrockAnthropicId(
"arn:aws:bedrock:us-east-2:123456789012:application-inference-profile/my-anthropicish-profile"
)
).toBe(false);
});
it("matches when 'anthropic' appears as its own dot-segment in ARN", () => {
// ARN whose profile name embeds the foundation segment correctly —
// operator chose to surface the backing model in the name.
expect(
isBedrockAnthropicId(
"arn:aws:bedrock:us-east-2:123456789012:application-inference-profile/anthropic.claude-opus-4-7"
)
).toBe(true);
});
it("is case-insensitive", () => {
expect(isBedrockAnthropicId("US.ANTHROPIC.CLAUDE-OPUS-4-7")).toBe(true);
});
});
describe("providers registry", () => {
it("every provider has envVars", () => {
for (const [key, config] of Object.entries(providers)) {
+128 -5
View File
@@ -7,6 +7,22 @@
// ── types ──────────────────────────────────────────────────────────────────────
/**
* routing discriminant for entries whose `resolve` is dynamic looked up
* from a separate env var at run time rather than fixed in the catalog.
*
* `"bedrock"` means the actual model ID comes from `BEDROCK_MODEL_ID`
* (an AWS-canonical Bedrock model ID like `us.anthropic.claude-opus-4-7`
* or `amazon.nova-pro-v1:0`). enterprise Bedrock customers self-select for
* version control silent alias bumps would break compliance review,
* model-access enrollment, and provisioned-throughput contracts. so the
* single `bedrock/byok` entry is a routing slug, not a model alias: the
* harness reads `BEDROCK_MODEL_ID` and routes to claude-code (when the ID
* contains "anthropic") or opencode (everything else, with an
* `amazon-bedrock/` prefix).
*/
export type ModelRouting = "bedrock";
export interface ModelAlias {
/** stable alias stored in DB, e.g. "anthropic/claude-opus" */
slug: string;
@@ -14,9 +30,9 @@ export interface ModelAlias {
provider: string;
/** human-readable name shown in dropdowns */
displayName: string;
/** concrete models.dev specifier, e.g. "anthropic/claude-opus-4-6" */
/** concrete models.dev specifier, e.g. "anthropic/claude-opus-4-6". sentinel for routing entries — never passed to a CLI directly. */
resolve: string;
/** full models.dev specifier for the OpenRouter equivalent (undefined for free models) */
/** full models.dev specifier for the OpenRouter equivalent (undefined for free models and routing entries) */
openRouterResolve: string | undefined;
/** top-tier pick for this provider — preferred during auto-select */
preferred: boolean;
@@ -24,6 +40,15 @@ export interface ModelAlias {
isFree: boolean;
/** slug of a replacement model — presence implies this model is deprecated */
fallback: string | undefined;
/** dynamic-resolution discriminant — see ModelRouting docs */
routing: ModelRouting | undefined;
/** alias key (within same provider) of the cheaper sibling reviewfrog should
* use as its lens-fanout subagent. e.g. claude-opus "claude-sonnet". */
subagentModel: string | undefined;
/** hide from selectable lists (UI dropdowns, CLI pickers). does NOT affect
* resolution for that use `fallback`. used for internal-only tier targets
* (e.g. gpt-5.4 as a subagent target without exposing it to users). */
hidden: boolean;
}
interface ModelDef {
@@ -37,6 +62,13 @@ interface ModelDef {
isFree?: boolean;
/** slug of a replacement model — presence implies this model is deprecated */
fallback?: string;
/** dynamic-resolution discriminant — see ModelRouting docs */
routing?: ModelRouting;
/** alias key (within same provider) of the cheaper sibling reviewfrog should
* use as its lens-fanout subagent (e.g. claude-opus "claude-sonnet"). */
subagentModel?: string;
/** hide from selectable lists. does NOT affect resolution; for that use `fallback`. */
hidden?: boolean;
}
export interface ProviderConfig {
@@ -61,6 +93,7 @@ export const providers = {
resolve: "anthropic/claude-opus-4-7",
openRouterResolve: "openrouter/anthropic/claude-opus-4.7",
preferred: true,
subagentModel: "claude-sonnet",
},
"claude-sonnet": {
displayName: "Claude Sonnet",
@@ -83,11 +116,22 @@ export const providers = {
resolve: "openai/gpt-5.5",
openRouterResolve: "openrouter/openai/gpt-5.5",
preferred: true,
subagentModel: "gpt-5.4",
},
"gpt-pro": {
displayName: "GPT Pro",
resolve: "openai/gpt-5.5-pro",
openRouterResolve: "openrouter/openai/gpt-5.5-pro",
subagentModel: "gpt",
},
// hidden subagent target — `gpt` lenses run against this. surfacing
// it in the picker would just confuse users (it's the prior-flagship,
// and they already have `gpt` and `gpt-mini` to choose from).
"gpt-5.4": {
displayName: "GPT 5.4",
resolve: "openai/gpt-5.4",
openRouterResolve: "openrouter/openai/gpt-5.4",
hidden: true,
},
"gpt-mini": {
displayName: "GPT Mini",
@@ -126,6 +170,7 @@ export const providers = {
resolve: "google/gemini-3.1-pro-preview",
openRouterResolve: "openrouter/google/gemini-3.1-pro-preview",
preferred: true,
subagentModel: "gemini-flash",
},
"gemini-flash": {
displayName: "Gemini Flash",
@@ -214,6 +259,7 @@ export const providers = {
displayName: "Claude Opus",
resolve: "opencode/claude-opus-4-7",
openRouterResolve: "openrouter/anthropic/claude-opus-4.7",
subagentModel: "claude-sonnet",
},
"claude-sonnet": {
displayName: "Claude Sonnet",
@@ -229,11 +275,20 @@ export const providers = {
displayName: "GPT",
resolve: "opencode/gpt-5.5",
openRouterResolve: "openrouter/openai/gpt-5.5",
subagentModel: "gpt-5.4",
},
"gpt-pro": {
displayName: "GPT Pro",
resolve: "opencode/gpt-5.5-pro",
openRouterResolve: "openrouter/openai/gpt-5.5-pro",
subagentModel: "gpt",
},
// hidden subagent target — see openai provider above for context.
"gpt-5.4": {
displayName: "GPT 5.4",
resolve: "opencode/gpt-5.4",
openRouterResolve: "openrouter/openai/gpt-5.4",
hidden: true,
},
"gpt-mini": {
displayName: "GPT Mini",
@@ -257,6 +312,7 @@ export const providers = {
displayName: "Gemini Pro",
resolve: "opencode/gemini-3.1-pro",
openRouterResolve: "openrouter/google/gemini-3.1-pro-preview",
subagentModel: "gemini-flash",
},
"gemini-flash": {
displayName: "Gemini Flash",
@@ -271,8 +327,7 @@ export const providers = {
"gpt-5-nano": {
displayName: "GPT Nano",
resolve: "opencode/gpt-5-nano",
envVars: [],
isFree: true,
openRouterResolve: "openrouter/openai/gpt-5-nano",
},
"mimo-v2-pro-free": {
displayName: "MiMo V2 Pro",
@@ -289,6 +344,20 @@ export const providers = {
},
},
}),
bedrock: provider({
displayName: "Amazon Bedrock",
envVars: ["AWS_BEARER_TOKEN_BEDROCK", "AWS_REGION", "BEDROCK_MODEL_ID"],
models: {
// single routing entry — the actual Bedrock model ID is read from
// BEDROCK_MODEL_ID at run time. see ModelRouting docs for why we
// don't catalog individual Bedrock models.
byok: {
displayName: "Amazon Bedrock",
resolve: "bedrock",
routing: "bedrock",
},
},
}),
openrouter: provider({
displayName: "OpenRouter",
envVars: ["OPENROUTER_API_KEY"],
@@ -298,6 +367,7 @@ export const providers = {
resolve: "openrouter/anthropic/claude-opus-4.7",
openRouterResolve: "openrouter/anthropic/claude-opus-4.7",
preferred: true,
subagentModel: "claude-sonnet",
},
"claude-sonnet": {
displayName: "Claude Sonnet",
@@ -313,11 +383,20 @@ export const providers = {
displayName: "GPT",
resolve: "openrouter/openai/gpt-5.5",
openRouterResolve: "openrouter/openai/gpt-5.5",
subagentModel: "gpt-5.4",
},
"gpt-pro": {
displayName: "GPT Pro",
resolve: "openrouter/openai/gpt-5.5-pro",
openRouterResolve: "openrouter/openai/gpt-5.5-pro",
subagentModel: "gpt",
},
// hidden subagent target — see openai provider above for context.
"gpt-5.4": {
displayName: "GPT 5.4",
resolve: "openrouter/openai/gpt-5.4",
openRouterResolve: "openrouter/openai/gpt-5.4",
hidden: true,
},
"gpt-mini": {
displayName: "GPT Mini",
@@ -346,6 +425,7 @@ export const providers = {
displayName: "Gemini Pro",
resolve: "openrouter/google/gemini-3.1-pro-preview",
openRouterResolve: "openrouter/google/gemini-3.1-pro-preview",
subagentModel: "gemini-flash",
},
"gemini-flash": {
displayName: "Gemini Flash",
@@ -433,6 +513,12 @@ export const modelAliases: ModelAlias[] = Object.entries(providers).flatMap(
preferred: def.preferred ?? false,
isFree: def.isFree ?? false,
fallback: def.fallback,
routing: def.routing,
// subagentModel is stored as an alias key local to the provider; expand
// here to a fully-qualified slug so callers can look up the target alias
// directly without re-deriving the provider.
subagentModel: def.subagentModel ? `${providerKey}/${def.subagentModel}` : undefined,
hidden: def.hidden ?? false,
}))
);
@@ -452,7 +538,7 @@ const MAX_FALLBACK_DEPTH = 10;
* use this in UI display sites (dropdown trigger labels, PR-comment footers,
* etc.) so a deprecated stored slug renders as the model the user actually
* runs against not the historical name. selectable lists should still hide
* deprecated aliases by filtering on `!a.fallback`.
* deprecated and internal-only aliases by filtering on `!a.fallback && !a.hidden`.
*/
export function resolveDisplayAlias(slug: string): ModelAlias | undefined {
let current = slug;
@@ -486,3 +572,40 @@ export function resolveCliModel(slug: string): string | undefined {
export function resolveOpenRouterModel(slug: string): string | undefined {
return resolveDisplayAlias(slug)?.openRouterResolve;
}
// ── bedrock routing ────────────────────────────────────────────────────────────
/** env var that supplies the Bedrock model ID for the `bedrock/byok` slug. */
export const BEDROCK_MODEL_ID_ENV = "BEDROCK_MODEL_ID";
/**
* the Bedrock model ID passed to claude-code or opencode is whatever the
* user set in `BEDROCK_MODEL_ID` Pullfrog never resolves or upgrades it.
* we route by checking whether the ID names an Anthropic model: claude-code
* handles Anthropic-on-Bedrock natively (with `CLAUDE_CODE_USE_BEDROCK=1`),
* everything else goes through opencode's `amazon-bedrock` provider.
*
* AWS Bedrock IDs come in two shapes:
* - dotted foundation IDs: `us.anthropic.claude-opus-4-7`,
* `anthropic.claude-haiku-4-5-20251001-v1:0`, `amazon.nova-pro-v1:0`,
* `meta.llama4-scout-17b-instruct-v1:0`. AWS-published, lowercase, the
* foundation provider always appears as a discrete dot-segment.
* - inference-profile ARNs: `arn:aws:bedrock:us-east-2:<acct>:application-inference-profile/<user-name>`.
* `<user-name>` is operator-chosen, so a naive substring check is fragile
* in both directions (Anthropic profile named without "anthropic" routes
* to opencode and misses CLAUDE_CODE_USE_BEDROCK; non-Anthropic profile
* whose name happens to contain "anthropic" routes to claude-code).
*
* we anchor on a discrete dot-segment match (case-insensitive). this catches
* every published foundation ID and is conservative for ARN-form IDs: ARN
* names that don't include "anthropic" as their own dot-segment route to
* opencode by default. operators using ARN-form IDs whose backing model is
* Anthropic should set `PULLFROG_AGENT=claude` to force the right route, or
* include the foundation segment in the profile name.
*/
export function isBedrockAnthropicId(bedrockModelId: string): boolean {
// split on `.`, `/`, and `:` so the check works for both dotted foundation
// IDs (anthropic.* / us.anthropic.*) and ARN-form IDs (where the relevant
// foundation segment sits between `/` and `.` inside the resource name).
return bedrockModelId.toLowerCase().split(/[./:]/).includes("anthropic");
}
+173 -100
View File
@@ -65,10 +65,6 @@ Rules:
- Focus on *intent*, not *what* the diff already shows what changed
- Get the file count and commit count from the checkout_pr metadata, not by counting manually`;
function learningsStep(t: (toolName: string) => string, n: number): string {
return `${n}. **learnings** (only if high confidence): if you discovered something about repo setup, test commands, conventions, or patterns that you are confident is correct and would reliably help future runs, call \`${t("update_learnings")}\` to persist it. skip this step if you are unsure or the finding is speculative/one-off. format as a flat bullet list (\`- \` per line, one fact per bullet). merge with existing learnings from the prompt — pass the FULL merged list. deduplicate, and drop bullets that are clearly wrong or no longer relevant to the current codebase.`;
}
export function computeModes(agentId: AgentId): Mode[] {
const t = (toolName: string) => formatMcpToolRef(agentId, toolName);
return [
@@ -78,18 +74,20 @@ export function computeModes(agentId: AgentId): Mode[] {
"Implement, build, create, or develop code changes; make specific changes to files or features; execute a plan; or handle tasks with specific implementation details",
prompt: `### Checklist
1. **plan** (optional, for complex tasks): analyze requirements, read AGENTS.md and relevant code, produce a step-by-step implementation plan.
1. **task list**: create your task list for this run as your first action.
2. **setup**: checkout or create the branch:
2. **plan** (optional, for complex tasks): analyze requirements, read AGENTS.md and relevant code, produce a step-by-step implementation plan.
3. **setup**: checkout or create the branch:
- **PR event, modifying the existing PR**: call \`${t("checkout_pr")}\`
- **new branch**: use \`${t("git")}\` to create a branch (\`git checkout -b pullfrog/branch-name\`)
3. **build**: implement changes using your native file and shell tools:
4. **build**: implement changes using your native file and shell tools:
- follow the plan (if you ran a plan phase)
- plan your approach before writing code: identify which files need to change, key design decisions, and edge cases. for non-trivial changes, consider whether there's a more elegant approach.
- run relevant tests/lints before committing
4. **self-review**: judgment call does YOUR diff warrant a fresh-eyes pass?
5. **self-review**: judgment call does YOUR diff warrant a fresh-eyes pass?
Skip self-review (commit directly) when the diff is **genuinely trivial**:
- doc typos, comment-only edits, whitespace/format-only, import reordering
@@ -120,13 +118,11 @@ export function computeModes(agentId: AgentId): Mode[] {
Review the findings, address valid points, and discard nitpicks or false positives. The reviewer is fallible it biases toward *recommending additions* (defensive checks for impossible cases, extra logging, new abstractions used once, comments restating code, tests asserting tautologies, "just-in-case" guards). For each finding, ask: would applying it leave the code more sound, correct, AND elegant? Two-out-of-three is usually a signal to look harder for a fix that gets all three before settling for one that trades elegance for correctness. Reject bloat-shaped findings without applying them, and after applying the rest re-read your diff and be discerning about what *you just changed*: if any fix turned out to be bloat in context, revert it. The goal is code that is sound and correct *while remaining elegant*; the smallest diff that fixes the real defect almost always wins. Then verify only intended changes are present, no debug artifacts or commented-out code remain, no unrelated files were modified. Commit locally via shell (\`git add . && git commit -m "..."\`).
5. **finalize**:
6. **finalize**:
- confirm a clean working tree, then push via \`${t("push_branch")}\` (see *SYSTEM* Git rules if this fails — prepush errors are usually the repo's tests/lint, not infra timeouts)
- create a PR via \`${t("create_pull_request")}\`
- call \`${t("report_progress")}\` with the PR link or the exact error if push/PR failed
${learningsStep(t, 6)}
### Notes
For simple, well-defined tasks, skip the plan phase and go straight to build.`,
@@ -137,51 +133,59 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
"Address PR review feedback; respond to reviewer comments; make requested changes to an existing PR",
prompt: `### Checklist
1. Checkout the PR branch via \`${t("checkout_pr")}\`.
1. **task list**: create your task list for this run as your first action.
2. Fetch review comments via \`${t("get_review_comments")}\`.
2. Checkout the PR branch via \`${t("checkout_pr")}\`.
3. For each comment:
3. Fetch review comments via \`${t("get_review_comments")}\`.
4. For each comment:
- understand the feedback
- evaluate whether applying it would leave the code more **sound, correct, AND elegant**. reviewers are fallible and bias toward *recommending additions* (defensive checks for impossible cases, extra abstractions, comments restating obvious code, tests asserting tautologies, "just-in-case" guards). if a request would add bloat ceremony without commensurate correctness benefit push back in your reply rather than mechanically applying it. two-out-of-three is usually a signal to look harder for a fix that gets all three before settling.
- if the request stands, make the code change using your native tools; otherwise reply explaining why
- record what was done (or why nothing was done)
4. Quality check:
5. Quality check:
- test changes, then review the diff before committing verify only intended changes are present, no debug artifacts remain, no fix turned out to be bloat in context (revert any that did), and the changes are clean enough that a senior engineer would approve without hesitation
- commit locally via shell (\`git add . && git commit -m "..."\`)
5. Finalize:
6. Finalize:
- confirm a clean working tree, then push via \`${t("push_branch")}\` (same push/prepush guidance as Build mode in *SYSTEM*)
- reply to each comment using \`${t("reply_to_review_comment")}\`
- reply to each comment **exactly once** using \`${t("reply_to_review_comment")}\` — do not re-emit the same call (the runtime dedupes identical bodies and the second call is wasted)
- resolve addressed threads via \`${t("resolve_review_thread")}\`
- call \`${t("report_progress")}\` with a brief summary (or the exact push error if push failed)
${learningsStep(t, 6)}`,
- call \`${t("report_progress")}\` with a brief summary (or the exact push error if push failed)`,
},
// Review and IncrementalReview use the multi-lens orchestrator pattern
// (canonical source: .claude/commands/anneal.md). The orchestrator does
// triage → parallel read-only subagent fan-out → aggregate → draft comments
// → submit. For someone else's PR, parallel lenses (correctness, security,
// research-validated claims, user-journey, etc.) provide breadth across
// angles that a single subagent can't carry coherently. Build mode keeps
// a single fresh-eyes subagent (different problem shape — orchestrator
// wrote the code and bias-mitigation comes from delegating to one
// subagent that doesn't share the implementation context).
// Deliberate omission vs canonical /anneal: severity categorization in the
// final message (the review body has its own CAUTION/IMPORTANT framing
// instead of a severity table).
// Review and IncrementalReview use a 0-or-2+ lens pattern. The default is
// 0 lenses (orchestrator handles the review solo). Multi-lens (2+
// reviewfrog subagents in parallel) only fires for substantive PRs or
// high-stakes-subsystem touches — and when it fires, ALL lenses must
// dispatch in a single assistant turn or the parallelism win disappears.
// We never dispatch exactly one lens: a single lens is just a worse,
// slower version of doing the work yourself.
//
// Build mode self-review is a different problem shape: the orchestrator
// wrote the code, so bias-mitigation comes from delegating to one
// fresh-eyes subagent that doesn't share the implementation context. A
// single subagent there is appropriate; the 0-or-2+ rule applies only to
// the Review/IncrementalReview lens fan-out where independence between
// perspectives is what's being purchased.
//
// Deliberate omission vs canonical /anneal: severity categorization in
// the final message (the review body has its own CAUTION/IMPORTANT
// framing instead of a severity table).
{
name: "Review",
description:
"Review code, PRs, or implementations; provide feedback or suggestions; identify issues; or check code quality, style, and correctness",
prompt: `### Checklist
1. **checkout**: call \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. read the diff TOC end-to-end and treat its file line ranges as your coverage checklist.
1. **task list**: create your task list for this run as your first action.
2. **triage**: orient yourself on the PR identify *what kind of thing this is* (domain it touches, seams it crosses, external contracts it depends on, user-facing surfaces it changes). orientation only defer specific defect-hunting to the subagents; pre-reviewing biases the lenses you pick. use \`${t("get_pull_request")}\` and other read-only GitHub tools for additional context if needed.
2. **checkout**: call \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. read the diff TOC end-to-end and treat its file line ranges as your coverage checklist.
if the PR is **genuinely trivial**, skip steps 34 entirely and submit a \`No new issues found.\` review per step 5. there's no value in dispatching even one lens for a typo.
3. **triage**: orient yourself on the PR identify *what kind of thing this is* (domain it touches, seams it crosses, external contracts it depends on, user-facing surfaces it changes). pull as much context as you need to render a confident, well-grounded review: read related files, grep for callers of changed symbols, check tests that exercise the touched paths, fetch related GitHub state. **you are the synthesizer** never delegate understanding to subagents.
if the PR is **genuinely trivial**, skip the fan-out entirely and submit a \`No new issues found.\` review per step 7.
"Genuinely trivial" (skip):
- single-word doc typo, whitespace/format-only, comment-only across any number of files
@@ -200,23 +204,25 @@ ${learningsStep(t, 6)}`,
- any "typo fix" in user-facing copy that changes meaning ("approved" "denied")
- mixed diffs where a semantic 1-liner is buried in whitespace/formatting changes
When unsure, treat as non-trivial. The cost of one extra subagent is cents; the cost of a missed billing/auth/data bug is much more.
4. **lens decision 0 or 2+, NEVER 1**.
otherwise pick lenses by where the PR concentrates risk **there's no fixed count**. lens count is judgment, not a formula. concrete shapes to anchor against:
The default is **0 lenses**: handle the review yourself end-to-end. Most PRs land here.
- **1 lens** pure refactor / mechanical rename across many files (impact); new test file with no source change (test-integrity); small isolated bug fix (correctness); doc-only PR with non-trivial technical content (research-validated or holistic)
- **23 lenses (most PRs land here)** new CRUD endpoint (correctness + security + test-integrity); new UI flow (user-journey + correctness); a single bug fix in a non-critical subsystem (correctness + test-integrity); design doc covering one domain (research-validated + correctness or holistic)
- **45 lenses (high-stakes subsystem touches)** any billing/payments change (billing-subsystem + correctness + security + operational-readiness); new auth flow (auth-subsystem + correctness + security + test-integrity); schema migration (schema-migration-subsystem + correctness + operational-readiness + impact); cross-subsystem PR that touches billing AND auth AND schema (one subsystem lens per domain + correctness)
- **6+ lenses** almost always a smell; you're either covering overlapping ground or this PR should have been split. push back via the review body rather than expanding lens count.
Dispatch **2+ \`${REVIEWER_AGENT_NAME}\` lenses in parallel** ONLY when ALL of the following are true:
- the PR is substantive (>5 files changed AND >200 net lines), OR touches a high-stakes subsystem (auth, billing, payments, schema migration, webhooks, secrets, RBAC, multi-tenant isolation, cron/scheduling)
- you can name 2+ distinct concrete failure modes that warrant independent lenses (one lens per failure mode; orthogonal, not overlapping)
- parallel-orchestrated independent perspectives meaningfully outperform what you'd find solo
lenses come in two flavors, and you can mix them:
**NEVER dispatch exactly one lens.** A single lens is just a more expensive version of doing the work yourself with a worse model it adds wall time and a context-handoff for no orthogonality benefit. Either you have at least two genuinely independent failure-mode hypotheses (dispatch all in one turn), or you don't (do the review yourself).
When you do go multi-lens, lens framings come in two flavors:
- **themed lenses** a perspective applied across the whole diff (correctness, security, user-journey, performance, etc.).
- **subsystem lenses** a domain-scoped frame for high-stakes subsystems the PR touches (e.g. "the auth lens", "the billing lens", "the schema-migration lens"). a subsystem lens is "review the PR specifically for what could go wrong in this subsystem" and naturally combines theme + scope. **for high-stakes domains, lead with the subsystem lens rather than the generic themed equivalent** "billing-subsystem" outperforms "correctness on billing code" because the framing primes the subagent to remember domain-specific failure modes (double-charges, refund races, currency rounding, dispute flows) the generic lens misses.
- **subsystem lenses** a domain-scoped frame for high-stakes subsystems the PR touches (e.g. "the auth lens", "the billing lens", "the schema-migration lens"). **for high-stakes domains, lead with the subsystem lens rather than the generic themed equivalent** "billing-subsystem" outperforms "correctness on billing code" because the framing primes the subagent to remember domain-specific failure modes (double-charges, refund races, currency rounding, dispute flows) the generic lens misses.
starter menu (combine, omit, or invent your own):
- **correctness & invariants** bugs, races, error handling, edge cases, state-machine boundaries
- **impact** when the PR removes features, deletes exports, renames identifiers, or changes architectural patterns: stale references in code, tests, docs (\`docs/\`, \`wiki/\`), comments, configs, UI
- **research-validated assumptions** third-party API contracts, SDK semantics, framework directives, version-gated behavior. the subagent must verify load-bearing claims via web search and quote source URLs.
- **impact** stale references in code/tests/docs/configs/UI after rename/remove
- **research-validated assumptions** third-party API contracts, SDK semantics, framework directives, version-gated behavior. **only pick when the PR's correctness depends on the contract behaving a specific way** not when the API is merely used. The bar is "if the third-party contract differs from what the diff assumes, the PR is incorrect." When dispatched, the subagent must verify load-bearing claims via web search and quote source URLs.
- **security** new endpoints, authZ, input validation, secrets handling, replay/CSRF/injection, cross-tenant isolation
- **user-journey** UX-touching flows: walk through happy path and failure modes as a user
- **operational readiness** observability, alerting, migrations (forward + rollback), feature flags, on-call burden
@@ -226,44 +232,67 @@ ${learningsStep(t, 6)}`,
- **holistic** does the PR make sense as a whole? symmetric flows (delete for every create, rollback for every migration)?
- **subsystem lenses** (invent as the PR demands) auth, billing, payments, schema migration, webhooks, secrets, RBAC, multi-tenant isolation, cron/scheduling, etc.
3. **fan out**: dispatch one \`${REVIEWER_AGENT_NAME}\` subagent per lens — its baked-in system prompt enforces the non-mutative + non-recursive contract (read-only file/search/web tools and read-only MCP queries; no writes, shell side effects, state-changing MCP calls, or nested subagent dispatch). when picking 2+ lenses, dispatch them in a **single assistant turn with multiple parallel subagent calls**; issuing one and awaiting reply before the next collapses the fan-out into a serial review. if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body — do not skip step 3 entirely on a single subagent failure. each subagent gets:
The only subagent type is \`${REVIEWER_AGENT_NAME}\` — used for lens judgment work ("is this safe / correct / well-tested?"), runs on a mid-tier model.
5. **fan out (only if step 4 said 2+ lenses)**: dispatch every \`${REVIEWER_AGENT_NAME}\` subagent for this run **IN A SINGLE ASSISTANT TURN, AS MULTIPLE PARALLEL TASK TOOL_USE BLOCKS IN ONE MESSAGE.**
CRITICAL PARALLELISM IS THE ONLY REASON LENSES EXIST.
The default tool-call behavior of Claude Code (and most agent runtimes) is **serial dispatch**: emit one Task call, await result, emit next, await, etc. This collapses your fan-out into a sequential review where each lens adds N × (orchestrator-think-time + lens-execution-time) to wall time. **YOU MUST OVERRIDE THIS DEFAULT.** Emit ALL of your Task tool_use blocks in the SAME assistant message, BEFORE you read ANY result from ANY of them. If you find yourself emitting one Task call, then thinking about the result, then emitting another STOP and re-issue them all together. The whole point of going multi-lens is the wall-clock speedup from parallel execution; serial dispatch defeats it entirely.
Right pattern: one assistant turn with N Task tool_use blocks wait N results arrive together aggregate.
Wrong pattern: turn 1 = Task(lens A) turn 2 (after A's result) = Task(lens B) → turn 3 (after B's result) = Task(lens C). This is the failure mode. Do not do this.
You can also include your own \`read\` / \`grep\` / \`webfetch\` calls in the SAME turn as the parallel \`${REVIEWER_AGENT_NAME}\` dispatches — concurrent context-pulling on the orchestrator side runs in parallel with the lens fan-out and costs zero extra wall time.
if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body do not skip the fan-out entirely on a single subagent failure. each subagent gets:
- the diff path / target reading the diff and the codebase is its job
- **only one lens** never a multi-section "review for X, Y, and Z" prompt
- **a Task \`description\` set to the lens name** (e.g. \`"security"\`, \`"correctness"\`, \`"billing-subsystem"\`) — the harness reads this field to label the subagent's log lines so parallel runs can be told apart in CI output. without it, every subagent shows up as \`subagent#N\`.
- the read-only contract restated in your dispatch instructions so the rule is present twice (the subagent's system prompt also enforces it). The test: would this call still be a no-op if reverted? If not (PR comments, branch pushes, issue updates, set_output, label changes, dependency installs, etc.), don't make it.
- if the lens touches external contracts, instruct the subagent to verify load-bearing claims via web search rather than trust training data, and to quote source URLs in its reasoning. action runs are non-interactive there's no human in the loop to catch "I'm pretty sure Stripe does X."
- ask the subagent to report findings with file paths and NEW line numbers from the diff so you can anchor inline comments without re-reading the entire diff.
delegation discipline:
- do NOT lens-review the diff yourself in parallel with the subagents (your job is dispatch + comment-drafting; doing the lens work yourself reintroduces the bias the fan-out avoids)
- do NOT summarize the PR for them (biases toward a validation frame)
- do NOT hand them a curated reading list (let them discover scope)
- do NOT pre-shape their output with a finding schema
- do NOT mention the other lenses (independence is the point overlapping findings are a strong signal)
4. **aggregate & draft**: merge findings; de-dup overlaps (two lenses catching the same issue = higher-confidence signal); trace each finding yourself before accepting it. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the PR (heuristic: if the finding's root cause lives in lines this PR added or modified, it's in scope; otherwise drop unless the PR plausibly introduced or amplified the regression), and anything not actionable. also drop **bloat-shaped findings** proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or worse, degrades elegance to nominally improve correctness) makes the codebase worse, not better.
6. **aggregate & draft**: when the fan-out lands, merge findings; de-dup overlaps (two lenses catching the same issue = higher-confidence signal); trace each finding yourself before accepting it. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the PR (heuristic: if the finding's root cause lives in lines this PR added or modified, it's in scope; otherwise drop unless the PR plausibly introduced or amplified the regression), and anything not actionable. also drop **bloat-shaped findings** proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or worse, degrades elegance to nominally improve correctness) makes the codebase worse, not better.
for surviving findings, draft inline comments with NEW line numbers from the diff. every comment must be actionable, 2-3 sentences max. use GitHub permalink format for code references. for impact-analysis findings (stale references after rename/remove), report them in the review body ordered by severity (runtime breakage > incorrect docs > stale comments) rather than as inline comments unless they're anchored to a specific line.
5. **submit**: ALWAYS submit exactly one review via \`${t("create_pull_request_review")}\`. Do NOT call \`report_progress\` — the review is the final record and the progress comment will be cleaned up automatically.
7. **submit**: ALWAYS submit exactly one review via \`${t("create_pull_request_review")}\`. Do NOT call \`report_progress\` — the review is the final record and the progress comment will be cleaned up automatically.
note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed optionally after reading the listed ranges. the pre-flight will not block again this session.
The review body is structured as: \`[optional alert blockquote]\`\`[PR summary using the default format below]\`. Inline comments are passed via the \`comments\` parameter, not in the body.
- **critical issues** (blocks merge bugs, security, data loss):
GitHub alert blockquotes render at four visual intensities the callout is what the author sees first, so pick the one that matches what you want them to do:
- \`[!CAUTION]\` — large red banner. Reads as "this will break something."
- \`[!IMPORTANT]\` — large purple banner. Reads as "you need to look at this before merging."
- \`[!NOTE]\` — small blue inline callout. Reads as "FYI, here's something worth noting."
- no callout plain text. Reads as routine review output.
Two reinforcing levers: callout intensity (above) and \`approved\` (which gates the footer Fix-button affordance — Fix renders on every non-approving review, so \`approved: true\` suppresses it). Wrapping mergeable feedback in \`[!IMPORTANT]\` trains users to click Fix on reviews that don't need fixing. Pick the tier the author's actual next action justifies.
- **critical issues** (blocks merge bugs, security, data loss, broken core flows):
\`approved: false\`. Body opens with \`> [!CAUTION]\\n> This PR introduces ...\`, followed by the PR summary. Include all inline comments via \`comments\`.
- **recommended changes** (non-critical):
\`approved: false\`. Body opens with \`> [!IMPORTANT]\\n> Consider ...\`, followed by the PR summary. Include all inline comments via \`comments\`.
- **must-address non-critical findings** (real consequences if shipped incorrect behavior in non-critical paths, missing validation on user input, regressions the author should fix before merge):
\`approved: false\`. Body opens with \`> [!IMPORTANT]\\n> ...\`, followed by the PR summary. Reserve this tier for findings with concrete fallout — do NOT use \`[!IMPORTANT]\` for nits, style preferences, or "consider also" suggestions. Include all inline comments via \`comments\`.
- **minor suggestions only** (single-line nits, doc/comment polish, defer-able observations, "rough edges"):
\`approved: false\`. NO alert blockquote. Body opens directly with the PR summary. Include all inline comments via \`comments\`.
- **informational observations** (mergeable as-is, nothing actionable e.g. prior feedback addressed cleanly, surfacing a minor stale doc reference, calling out something noteworthy without recommending a change):
\`approved: true\`. Body opens with \`> [!NOTE]\\n> ...\`, followed by the PR summary. Do NOT include inline \`comments\`\`[!NOTE]\` signals "no action needed", which contradicts an actionable anchor; if a point is concrete enough to anchor to a line, downgrade the whole review to "minor suggestions only" (\`approved: false\`) instead.
- **no actionable issues**:
\`approved: true\`. Body opens with \`No new issues found.\` followed by the PR summary.
${PR_SUMMARY_FORMAT}`,
},
// IncrementalReview shares Review's multi-lens orchestrator pattern but
// scopes the target to the incremental diff. The "issues must be NEW
// since the last Pullfrog review" filter lives at aggregation time
// (step 5), NOT in the subagent prompt — pushing the filter into
// IncrementalReview shares Review's 0-or-2+ lens pattern but scopes the
// target to the incremental diff. The "issues must be NEW since the last
// Pullfrog review" filter lives at aggregation time (step 8), NOT in the
// subagent prompt — pushing the filter into
// subagents matches the canonical anneal anti-pattern of "list known
// pre-existing failures — don't flag these" and suppresses signal on
// regressions the new commits amplified. The review body is just
@@ -277,46 +306,73 @@ ${PR_SUMMARY_FORMAT}`,
"Re-review a PR after new commits are pushed; focus on new changes since the last review",
prompt: `### Checklist
1. **checkout**: call \`${t("checkout_pr")}\` — this returns PR metadata, \`diffPath\` (full diff), and \`incrementalDiffPath\` (changes since last reviewed version, if available). read the diff TOC first and use its line ranges as your coverage checklist.
1. **task list**: create your task list for this run as your first action.
2. **incremental scope**: if \`incrementalDiffPath\` is present, read it to see what changed since the last review. this is a range-diff that isolates the net changes, filtering out base branch noise. if not present, fall back to reviewing the full PR diff and determine what changed since Pullfrog's most recent review.
2. **checkout**: call \`${t("checkout_pr")}\` — this returns PR metadata, \`diffPath\` (full diff), and \`incrementalDiffPath\` (changes since last reviewed version, if available). read the diff TOC first and use its line ranges as your coverage checklist.
3. **prior feedback**: fetch previous reviews via \`${t("list_pull_request_reviews")}\`. for the most recent Pullfrog review, call \`${t("get_review_comments")}\` with the review ID to retrieve specific prior line-level feedback. you'll use this to filter your aggregation in step 5 — anything already flagged in a prior review and not changed by the new commits should not be re-raised. you do NOT need to render this in the review body; the rolling PR summary snapshot is the durable record of what's been addressed.
3. **incremental scope**: if \`incrementalDiffPath\` is present, read it to see what changed since the last review. this is a range-diff that isolates the net changes, filtering out base branch noise. if not present, fall back to reviewing the full PR diff and determine what changed since Pullfrog's most recent review.
4. **triage & fan out**: orient on the *incremental* changes domain, seams, external contracts, user-facing surfaces.
4. **prior feedback**: fetch previous reviews via \`${t("list_pull_request_reviews")}\`. for the most recent Pullfrog review, call \`${t("get_review_comments")}\` with the review ID to retrieve specific prior line-level feedback. you'll use this to filter your aggregation in step 8 — anything already flagged in a prior review and not changed by the new commits should not be re-raised. you do NOT need to render this in the review body; the rolling PR summary snapshot is the durable record of what's been addressed.
if the incremental changes are **genuinely trivial**, skip the fan-out entirely and jump to step 7's non-substantive path (do NOT submit a review).
5. **triage**: orient on the *incremental* changes domain, seams, external contracts, user-facing surfaces. pull as much context as you need to render a confident review: read related files, grep for callers of changed symbols, check tests that exercise the touched paths. **you are the synthesizer.**
if the incremental changes are **genuinely trivial**, skip the fan-out entirely and jump to step 10's non-substantive path (do NOT submit a review).
"Genuinely trivial" (skip): formatting/comment tweaks, import reordering, lockfile regen, mechanical rename of import paths, whitespace-only.
"Looks trivial but isn't" (do NOT skip same anti-patterns as Review mode): 1-line changes to SQL/regex/auth/billing/permissions/signature-verification code; flipping feature-flag defaults or retry/timeout constants; money/tax/HTTP-method/redirect changes; tightening or loosening a comparison operator; mixed diffs with a semantic line buried in formatting.
When unsure, treat as non-trivial.
otherwise pick lenses by where the new commits concentrate risk **there's no fixed count**, same calibration as Review mode (1 lens for pure refactor / isolated fix; 23 for typical features; 45 for high-stakes subsystem touches; 6+ is a smell). lens framing follows Review mode: themed lenses (correctness & invariants, impact when new commits remove/rename/deprecate things, research-validated assumptions, security, user-journey, operational readiness, integration & cross-cutting, test integrity, performance, holistic) and subsystem lenses (auth, billing, schema migration, etc.) for high-stakes domains lead with the subsystem lens rather than the generic themed equivalent.
6. **lens decision 0 or 2+, NEVER 1**.
dispatch one \`${REVIEWER_AGENT_NAME}\` subagent per lens — its baked-in system prompt enforces the non-mutative + non-recursive contract (read-only file/search/web tools and read-only MCP queries; no writes, shell side effects, state-changing MCP calls, or nested subagent dispatch). dispatch them in a **single assistant turn with multiple parallel subagent calls** (serial dispatch collapses the fan-out). if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body — do not skip step 4 entirely on a single subagent failure. each subagent gets:
- the diff scope (incremental diff path if available, full diff otherwise). do NOT tell them to skip pre-existing issues that suppresses regressions the new commits amplified; the "issues must be NEW" filter lives at aggregation time (step 5), not in the subagent prompt
The default is **0 lenses**: handle the re-review yourself end-to-end. Most incremental reviews land here especially thread-reply re-reviews where the user is asking "did you address X?" rather than "review the diff again."
Dispatch **2+ \`${REVIEWER_AGENT_NAME}\` lenses in parallel** ONLY when ALL of the following are true:
- the incremental changes are substantive (>5 files changed AND >200 net new lines), OR touch a high-stakes subsystem (auth, billing, payments, schema migration, webhooks, secrets, RBAC, multi-tenant isolation, cron/scheduling)
- you can name 2+ distinct concrete failure modes the new commits plausibly introduce that warrant independent lenses
- parallel-orchestrated independent perspectives meaningfully outperform what you'd find solo
**NEVER dispatch exactly one lens.** Single-lens dispatch adds wall time and cost for no orthogonality benefit. Either go multi-lens (2 in parallel) or do the re-review yourself.
Lens framing follows Review mode: themed lenses (correctness, security, etc.) and subsystem lenses (auth, billing, schema-migration, etc.) for high-stakes domains lead with the subsystem lens.
7. **fan out (only if step 6 said 2+ lenses)**: dispatch every \`${REVIEWER_AGENT_NAME}\` subagent for this run **IN A SINGLE ASSISTANT TURN, AS MULTIPLE PARALLEL TASK TOOL_USE BLOCKS IN ONE MESSAGE.**
CRITICAL PARALLELISM IS THE ONLY REASON LENSES EXIST.
Default tool-call behavior is **serial dispatch**: emit one Task call, await result, emit next, await, etc. This collapses your fan-out into a sequential review where each lens adds N × (orchestrator-think-time + lens-execution-time) to wall time. **YOU MUST OVERRIDE THIS DEFAULT.** Emit ALL of your Task tool_use blocks in the SAME assistant message, BEFORE you read ANY result from ANY of them.
Right pattern: one assistant turn with N Task tool_use blocks wait N results arrive together aggregate.
Wrong pattern: turn 1 = Task(lens A) turn 2 (after A's result) = Task(lens B). This is the failure mode.
You can also include your own \`read\` / \`grep\` / \`webfetch\` calls in the SAME turn as the parallel \`${REVIEWER_AGENT_NAME}\` dispatches.
if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body. each subagent gets:
- the diff scope (incremental diff path if available, full diff otherwise). do NOT tell them to skip pre-existing issues that suppresses regressions the new commits amplified; the "issues must be NEW" filter lives at aggregation time (step 8), not in the subagent prompt
- **only one lens** never a multi-section "review for X, Y, and Z" prompt
- **a Task \`description\` set to the lens name** (e.g. \`"security"\`, \`"correctness"\`, \`"billing-subsystem"\`) — the harness reads this field to label the subagent's log lines so parallel runs can be told apart in CI output. without it, every subagent shows up as \`subagent#N\`.
- the read-only contract restated in your dispatch instructions so the rule is present twice (the subagent's system prompt also enforces it). The test: would this call still be a no-op if reverted? If not (PR comments, branch pushes, issue updates, set_output, label changes, dependency installs, etc.), don't make it.
- if the lens touches external contracts, instruct the subagent to verify load-bearing claims via web search and quote source URLs. action runs are non-interactive there's no human to catch "I'm pretty sure Stripe does X."
- **a Task \`description\` set to the lens name** — the harness reads this field to label log lines so parallel runs can be told apart.
- if the lens touches external contracts, instruct the subagent to verify load-bearing claims via web search and quote source URLs.
- ask the subagent to report findings with file paths and NEW line numbers from the full PR diff so you can anchor inline comments.
delegation discipline:
- do NOT lens-review the diff yourself in parallel with the subagents
- do NOT summarize the changes for them (biases toward validation frame)
- do NOT hand them a curated reading list (let them discover scope)
- do NOT pre-shape their output with a finding schema
- do NOT mention the other lenses (independence is the point)
5. **aggregate, draft, self-critique**: merge findings; de-dup overlaps; trace each finding yourself. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the new commits, anything not actionable, and anything that re-states prior review feedback (heuristic: if the finding's root cause lives in lines the *new commits* added or modified, it's in scope; otherwise drop). also drop **bloat-shaped findings** proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or degrades elegance to nominally improve correctness) makes the codebase worse, not better. To compute "lines the new commits added or modified": if \`incrementalDiffPath\` from step 1 is present, use it directly. Otherwise, take the prior Pullfrog review's \`commit_id\` (returned alongside each entry from \`${t("list_pull_request_reviews")}\` in step 3) and run \`git diff <prior-review-sha>..HEAD\` to isolate the lines added since that review. draft inline comments with NEW line numbers from the full PR diff — every comment must be actionable, 2-3 sentences max.
8. **aggregate, draft, self-critique**: merge findings (yours + any subagent output if you went multi-lens); de-dup overlaps; trace each finding yourself. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the new commits, anything not actionable, and anything that re-states prior review feedback (heuristic: if the finding's root cause lives in lines the *new commits* added or modified, it's in scope; otherwise drop). also drop **bloat-shaped findings** proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or degrades elegance to nominally improve correctness) makes the codebase worse, not better. To compute "lines the new commits added or modified": if \`incrementalDiffPath\` from step 2 is present, use it directly. Otherwise, take the prior Pullfrog review's \`commit_id\` (returned alongside each entry from \`${t("list_pull_request_reviews")}\` in step 4) and run \`git diff <prior-review-sha>..HEAD\` to isolate the lines added since that review. draft inline comments with NEW line numbers from the full PR diff — every comment must be actionable, 2-3 sentences max.
6. **build the review body** a single "Reviewed changes" section: summarize at the logical-change level, not per-file. each bullet starts with a past-tense verb (e.g. \`- Extracted shared CLI runtime into a single module\`, \`- Renamed package to pullfrog\`). avoid file paths unless they add clarity. if the changes can be described in one sentence, use one sentence — no bullets needed. do NOT include a separate "Prior review feedback" checklist; that's tracked in the rolling PR summary snapshot for the next agent run, and surfacing it in the user-facing body is noise (changes that addressed prior feedback are already covered by the Reviewed-changes bullets). in some cases you may receive a complete diff for the whole pull request instead of an incremental one — when this happens, you will need to determine what changes have happened since Pullfrog's most recent review.
9. **build the review body** a single "Reviewed changes" section: summarize at the logical-change level, not per-file. each bullet starts with a past-tense verb (e.g. \`- Extracted shared CLI runtime into a single module\`, \`- Renamed package to pullfrog\`). avoid file paths unless they add clarity. if the changes can be described in one sentence, use one sentence — no bullets needed. do NOT include a separate "Prior review feedback" checklist; that's tracked in the rolling PR summary snapshot for the next agent run, and surfacing it in the user-facing body is noise (changes that addressed prior feedback are already covered by the Reviewed-changes bullets). in some cases you may receive a complete diff for the whole pull request instead of an incremental one — when this happens, you will need to determine what changes have happened since Pullfrog's most recent review.
7. Submit Do NOT call \`report_progress\` or \`create_issue_comment\` — the review is the final record and the progress comment will be cleaned up automatically. Follow these rules:
10. Submit every run must end with EXACTLY ONE of \`${t("create_pull_request_review")}\` (substantive review) or \`${t("report_progress")}\` (no-review acknowledgement). do NOT call \`create_issue_comment\` for review output.
Same callout-intensity ladder as Review mode \`[!CAUTION]\` (large red, "will break") → \`[!IMPORTANT]\` (large purple, "must address before merging") → \`[!NOTE]\` (small blue, "FYI") → no callout (plain text). And the same Fix-button lever: the footer renders a Fix button on every non-approving review, so \`approved: true\` suppresses it. Wrapping mergeable feedback in \`[!IMPORTANT]\` trains users to click Fix on reviews that don't need fixing — pick the tier the author's actual next action justifies.
Follow these rules:
- note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed optionally after reading the listed ranges. the pre-flight will not block again this session.
- IF NO NEW ISSUES, NON-SUBSTANTIVE CHANGES ONLY (trivial formatting, import reordering, comment tweaks): do NOT submit a review. Do NOT call \`report_progress\`. Exit — the progress comment will be cleaned up automatically.
- ELSE IF NEW CRITICAL ISSUES (blocks merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with a GitHub alert blockquote (e.g. \`> [!CAUTION]\\n> This PR introduces ...\`), then the Reviewed-changes summary.
- ELSE IF NEW RECOMMENDED CHANGES (non-critical): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\` alert, then the Reviewed-changes summary.
- IF NO NEW ISSUES, NON-SUBSTANTIVE CHANGES ONLY (trivial formatting, import reordering, comment tweaks): do NOT submit a review. Instead call \`${t("report_progress")}\` with a 1-2 sentence note explaining no review was warranted (e.g. "No new issues. Changes since last review are formatting-only."). this leaves a visible signal that the run completed.
- ELSE IF NEW CRITICAL ISSUES (blocks merge bugs, security, data loss, broken core flows): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!CAUTION]\\n> This PR introduces ...\`, then the Reviewed-changes summary.
- ELSE IF NEW MUST-ADDRESS NON-CRITICAL FINDINGS (real consequences if shipped incorrect behavior, missing validation, regressions the author should fix before merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\`, then the Reviewed-changes summary. Do NOT use this tier for nits, style preferences, or "consider also" suggestions.
- ELSE IF NEW MINOR SUGGESTIONS ONLY (single-line nits, doc/comment polish, defer-able observations, "rough edges"): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens directly with \`Reviewed the following changes:\\n\` (NO alert blockquote), then the Reviewed-changes summary.
- ELSE IF INFORMATIONAL OBSERVATIONS (mergeable as-is, but worth surfacing e.g. prior feedback addressed cleanly with one minor stale doc reference, or a noteworthy positive observation): call \`${t("create_pull_request_review")}\` with \`approved: true\`, NO inline comments, and the review body. body opens with \`> [!NOTE]\\n> ...\` alert, then the Reviewed-changes summary. If a point is concrete enough to anchor to a line, downgrade the whole review to "minor suggestions only" (\`approved: false\`) instead — \`[!NOTE]\` and inline comments don't mix.
- ELSE IF NO NEW ISSUES, SUBSTANTIVE CHANGES (new functionality, behavior changes, or fixes to prior review feedback): call \`${t("create_pull_request_review")}\` to create a PR review. If all previous reviews have been properly addressed and no new issues were discovered, you can set \`approved: true\`. body opens with \`No new issues. Reviewed the following changes:\\n\`, then the Reviewed-changes summary.`,
},
{
@@ -325,15 +381,15 @@ ${PR_SUMMARY_FORMAT}`,
"Create plans, break down tasks, outline steps, analyze requirements, understand scope of work, or provide task breakdowns",
prompt: `### Checklist
1. Analyze the task and gather context:
1. **task list**: create your task list for this run as your first action.
2. Analyze the task and gather context:
- read AGENTS.md and relevant codebase files
- understand the architecture and constraints
2. Produce a structured, actionable plan with clear milestones.
3. Produce a structured, actionable plan with clear milestones.
3. Call \`${t("report_progress")}\` with the plan.
${learningsStep(t, 4)}`,
4. Call \`${t("report_progress")}\` with the plan.`,
},
{
name: "Fix",
@@ -341,46 +397,48 @@ ${learningsStep(t, 4)}`,
"Fix CI failures; debug failing tests or builds; investigate and resolve check suite failures",
prompt: `### Checklist
1. Checkout the PR branch via \`${t("checkout_pr")}\`.
1. **task list**: create your task list for this run as your first action.
2. Fetch check suite logs via \`${t("get_check_suite_logs")}\`.
2. Checkout the PR branch via \`${t("checkout_pr")}\`.
3. **CRITICAL**: verify the failure was INTRODUCED BY THIS PR before fixing. If unrelated, abort and report.
3. Fetch check suite logs via \`${t("get_check_suite_logs")}\`.
4. Diagnose and fix:
4. **CRITICAL**: verify the failure was INTRODUCED BY THIS PR before fixing. If unrelated, abort and report.
5. Diagnose and fix:
- read the workflow file, reproduce locally with the EXACT same commands CI runs
- fix the issue using your native file and shell tools
- verify the fix by re-running the exact CI command
- review the diff before committing verify only the fix is present, no debug artifacts, no unrelated changes. the fix should be clean enough that a senior engineer would approve without hesitation.
- commit locally via shell (\`git add . && git commit -m "..."\`)
5. Finalize:
6. Finalize:
- confirm a clean working tree, then push via \`${t("push_branch")}\` (same push/prepush guidance as Build mode in *SYSTEM*)
- call \`${t("report_progress")}\` with the diagnosis and fix summary (or the exact push error if push failed)
${learningsStep(t, 6)}`,
- call \`${t("report_progress")}\` with the diagnosis and fix summary (or the exact push error if push failed)`,
},
{
name: "ResolveConflicts",
description: "Resolve merge conflicts in a PR branch against the base branch",
prompt: `### Checklist
1. **Setup**:
1. **task list**: create your task list for this run as your first action.
2. **Setup**:
- Call \`${t("checkout_pr")}\` to get the PR branch.
- Call \`${t("get_pull_request")}\` to identify the base branch (e.g., 'main').
- Call \`${t("git_fetch")}\` to fetch the base branch.
2. **Merge Attempt**:
3. **Merge Attempt**:
- Run \`git merge origin/<base_branch>\` via shell.
- If it succeeds automatically, confirm a clean working tree, push via \`${t("push_branch")}\` (same push/prepush guidance as Build mode in *SYSTEM*), and call \`${t("report_progress")}\` with a brief success note or the exact push error if push failed — **then stop; do not run steps 34.**
- If it fails (conflicts), resolve them manually (continue to steps 34).
- If it succeeds automatically, confirm a clean working tree, push via \`${t("push_branch")}\` (same push/prepush guidance as Build mode in *SYSTEM*), and call \`${t("report_progress")}\` with a brief success note or the exact push error if push failed — **then stop; do not run steps 45.**
- If it fails (conflicts), resolve them manually (continue to steps 45).
3. **Resolve Conflicts**:
4. **Resolve Conflicts**:
- Run \`git status\` or parse the merge output to find the list of conflicting files.
- For each conflicting file: read it, find the conflict markers (\`<<<<<<<\`, \`=======\`, \`>>>>>>>\`), understand the code context, and rewrite the file with the correct resolution. Remove all markers.
- Verify the file syntax is correct after resolution.
4. **Finalize**:
5. **Finalize**:
- Run a final verification (build/test) to ensure the resolution works.
- \`git add . && git commit -m "resolve merge conflicts"\`
- confirm a clean working tree, then push via \`${t("push_branch")}\` (same push/prepush guidance as Build mode in *SYSTEM*)
@@ -392,23 +450,38 @@ ${learningsStep(t, 6)}`,
"General-purpose tasks that don't fit other modes: answering questions, adding comments, labeling, running ad-hoc commands, or any direct request",
prompt: `### Checklist
1. Analyze the task. For simple operations (labeling, commenting, answering questions, running a single command), handle directly.
1. **task list**: create your task list for this run as your first action.
2. For substantial work code changes across multiple files, multi-step investigations:
2. Analyze the task. For simple operations (labeling, commenting, answering questions, running a single command), handle directly.
3. For substantial work code changes across multiple files, multi-step investigations:
- plan your approach before starting
- use native file and shell tools for local operations
- use ${pullfrogMcpName} MCP tools for GitHub/git operations
- if code changes are needed: review your own diff before committing verify only intended changes are present, no debug artifacts remain, and the changes are clean enough that a senior engineer would approve without hesitation
3. Finalize:
4. Finalize:
- if code changes were made, push to a pull request (new or existing) using \`${t("push_branch")}\` and \`${t("create_pull_request")}\` as needed. \`git status\` must be clean before you finish (see *SYSTEM* Git rules if push fails).
- call \`${t("report_progress")}\` once with results — include exact tool errors if push or PR creation failed
- if the task involved labeling, commenting, or other GitHub operations, perform those directly
${learningsStep(t, 4)}`,
- if the task involved labeling, commenting, or other GitHub operations, perform those directly`,
},
];
}
// static export for UI display — uses opencode format as the readable default
export const modes: Mode[] = computeModes("opencode");
/**
* modes that legitimately never modify the working tree. used by the post-run
* dirty-tree gate to suppress the "commit and push" nudge those modes
* complete by submitting a review (`Review` / `IncrementalReview`) or by
* posting a Plan comment (`Plan`), not by touching files. any leftover in the
* tree at end-of-run is incidental tool noise (e.g. a `node_modules/` from a
* stray install attempt) on an ephemeral worktree; nudging the agent to
* commit it would produce a spurious PR.
*/
export const NON_COMMITTING_MODES: ReadonlySet<string> = new Set([
"Review",
"IncrementalReview",
"Plan",
]);
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "pullfrog",
"version": "0.1.0",
"version": "0.1.8",
"type": "module",
"bin": {
"pullfrog": "dist/cli.mjs",
+10 -3
View File
@@ -135,14 +135,21 @@ export const installNodeDependencies: PrepDefinition = {
}
}
// get the frozen install command (or fallback to regular install)
const resolved = resolveCommand(agent, "frozen", []) || resolveCommand(agent, "install", []);
// frozen-lockfile install only. eager prep is non-mutating by contract:
// we run it before the agent starts and any artifact it leaves in the
// tree (e.g. a generated `package-lock.json`) trips the dirty-tree
// post-run gate and produces a spurious PR. `frozen` commands
// (`npm ci`, `pnpm install --frozen-lockfile`, etc.) fail cleanly
// without modifying state when there's no lockfile, which is exactly
// what we want — repos that need a non-frozen install must opt in via
// a `setup` lifecycle hook (`action/utils/lifecycle.ts`).
const resolved = resolveCommand(agent, "frozen", []);
if (!resolved) {
return {
language: "node",
packageManager,
dependenciesInstalled: false,
issues: [`no install command found for ${agent}`],
issues: [`no frozen-install command available for ${agent}`],
};
}
+9
View File
@@ -104,6 +104,15 @@ function createRuntimeContext(): RuntimeContext {
const env: NodeJS.ProcessEnv = { ...process.env };
env.npm_config_registry = NPM_REGISTRY;
env.COREPACK_NPM_REGISTRY = NPM_REGISTRY;
// bypass customer-side release-age gates (npm's `min-release-age`, pnpm's
// `minimumReleaseAge`) so our bootstrap can resolve the latest publish.
// pullfrog's npm version is server-stamped from a SHA-pinned action ref the
// customer already vets at the action layer — not a customer-vetted dep, so
// the gate is the wrong affordance here. env beats .npmrc in both tools.
// npm uses `npm_config_*`; pnpm v11+ requires `pnpm_config_*` (the v10→v11
// migration renamed the prefix). tracked: #713
env.npm_config_min_release_age = "0";
env.pnpm_config_minimum_release_age = "0";
const currentPath = process.env.PATH ?? "";
env.PATH = currentPath ? `${nodeBinDir}${delimiter}${currentPath}` : nodeBinDir;
@@ -1,38 +0,0 @@
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
exports[`latest model per provider snapshot > matches snapshot 1`] = `
{
"anthropic": {
"modelId": "claude-opus-4-7",
"releaseDate": "2026-04-16",
},
"deepseek": {
"modelId": "deepseek-v4-pro",
"releaseDate": "2026-04-24",
},
"google": {
"modelId": "gemini-3.1-flash-lite",
"releaseDate": "2026-05-07",
},
"moonshotai": {
"modelId": "kimi-k2.6",
"releaseDate": "2026-04-21",
},
"openai": {
"modelId": "gpt-5.5-pro",
"releaseDate": "2026-04-23",
},
"opencode": {
"modelId": "gpt-5.5-pro",
"releaseDate": "2026-04-24",
},
"openrouter": {
"modelId": "x-ai/grok-4.3",
"releaseDate": "2026-05-01",
},
"xai": {
"modelId": "grok-4.3",
"releaseDate": "2026-05-01",
},
}
`;
+6
View File
@@ -96,4 +96,10 @@ export const test: TestRunnerOptions = {
repoSetup,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic", "security"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+6
View File
@@ -104,4 +104,10 @@ export const test: TestRunnerOptions = {
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+1
View File
@@ -92,4 +92,5 @@ export const test: TestRunnerOptions = {
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic", "security"],
coverage: ["action/mcp/dependencies.ts", "action/utils/install.ts"],
};
+6
View File
@@ -62,4 +62,10 @@ export const test: TestRunnerOptions = {
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+6
View File
@@ -74,4 +74,10 @@ export const test: TestRunnerOptions = {
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+6
View File
@@ -67,4 +67,10 @@ export const test: TestRunnerOptions = {
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+7
View File
@@ -29,4 +29,11 @@ export const test: TestRunnerOptions = {
expectFailure: true,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/timer.ts",
"action/utils/subprocess.ts",
"action/utils/exitHandler.ts",
"action/utils/activity.ts",
"action/mcp/selectMode.ts",
],
};
-70
View File
@@ -1,70 +0,0 @@
#!/usr/bin/env bash
# determines which agents need testing based on changed files.
# reads changed file paths from stdin (JSON array or newline-delimited).
# outputs a JSON array of agent names to stdout.
#
# only agents whose harness file changed AND are exported from index.ts are included.
# shared.ts/index.ts/postRun.ts and other non-harness action changes fall back to opencode as a canary.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
AGENTS_INDEX="$SCRIPT_DIR/../agents/index.ts"
# build the set of active agents from index.ts imports (portable, no -P)
active_agents=()
while IFS= read -r line; do
[[ -n "$line" ]] && active_agents+=("$line")
done < <(sed -n 's/.*from "\.\/\([^"]*\)\.ts".*/\1/p' "$AGENTS_INDEX" | grep -v shared)
# read stdin - auto-detect JSON array vs newline-delimited
input=$(cat)
if echo "$input" | jq -e 'type == "array"' > /dev/null 2>&1; then
files=$(echo "$input" | jq -r '.[]')
else
files="$input"
fi
is_active_agent() {
local name="$1"
for a in "${active_agents[@]}"; do
[[ "$a" == "$name" ]] && return 0
done
return 1
}
# find which agent harness files changed
changed_agents=()
has_non_agent_change=false
while IFS= read -r file; do
[[ -z "$file" ]] && continue
case "$file" in
action/agents/shared.ts|action/agents/index.ts|action/agents/postRun.ts)
has_non_agent_change=true
;;
action/agents/*.ts)
agent_name="$(basename "$file" .ts)"
if is_active_agent "$agent_name"; then
changed_agents+=("$agent_name")
else
# legacy/inactive agent file changed — treat as non-agent change
has_non_agent_change=true
fi
;;
action/*)
has_non_agent_change=true
;;
esac
done <<< "$files"
# output agents based on change type.
# non-agent action changes always include opencode as a canary.
if $has_non_agent_change; then
changed_agents+=("opencode")
fi
if [[ ${#changed_agents[@]} -gt 0 ]]; then
printf '%s\n' "${changed_agents[@]}" | sort -u | jq -R . | jq -sc .
else
echo '[]'
fi
+20 -51
View File
@@ -1,4 +1,3 @@
import { execFileSync } from "node:child_process";
import { readdirSync, readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -16,7 +15,7 @@ type WorkflowJob = {
"runs-on": string;
"timeout-minutes"?: number;
permissions?: WorkflowPermissions;
strategy?: { "fail-fast": boolean; matrix: Record<string, string[]> };
strategy?: { "fail-fast": boolean; matrix: Record<string, unknown> };
env?: Record<string, string>;
steps?: unknown[];
};
@@ -57,7 +56,6 @@ const expectedAgents = Object.keys(agents).sort();
const crossagentTests = getTestNamesFromDir("crossagent");
const agnosticTests = getTestNamesFromDir("agnostic");
const adhocTests = getTestNamesFromDir("adhoc");
const dynamicAgentsExpression = "$" + "{{ fromJSON(needs.changes.outputs.agents) }}";
// all provider API key names + GITHUB_TOKEN + model overrides
const expectedAgentEnvVars = [
@@ -83,53 +81,22 @@ describe("ci workflow consistency", () => {
const rootJob = rootWorkflow.jobs["action-agents"];
const actionJob = actionWorkflow.jobs.agents;
it("root agent matrix uses dynamic output from changes job", () => {
expect(rootJob.strategy!.matrix.agent).toBe(dynamicAgentsExpression);
});
it("changed-agents.sh falls back to opencode when shared agent code changed", () => {
const input = JSON.stringify(["action/agents/shared.ts"]);
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input,
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
});
it("changed-agents.sh falls back to opencode for non-agent action changes", () => {
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input: JSON.stringify(["action/mcp/server.ts"]),
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
});
it("changed-agents.sh includes opencode canary alongside changed agents", () => {
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input: JSON.stringify(["action/agents/opencode.ts", "action/mcp/server.ts"]),
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
});
it("changed-agents.sh treats legacy agent files as non-agent changes", () => {
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input: JSON.stringify(["action/agents/codex.ts", "action/agents/gemini.ts"]),
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
it("root agents matrix is wired to the dynamic matrix output", () => {
const include = rootJob.strategy?.matrix.include;
expect(typeof include).toBe("string");
expect(include as string).toContain("fromJSON(needs.changes.outputs.matrix).agents");
});
it("action agent matrix matches agents map", () => {
expect([...actionJob.strategy!.matrix.agent].sort()).toEqual(expectedAgents);
});
it("root test matrix matches crossagent/ directory", () => {
expect([...rootJob.strategy!.matrix.test].sort()).toEqual(crossagentTests);
expect((actionJob.strategy?.matrix.agent as string[])?.slice().sort()).toEqual(
expectedAgents
);
});
it("action test matrix matches crossagent/ directory", () => {
expect([...actionJob.strategy!.matrix.test].sort()).toEqual(crossagentTests);
expect((actionJob.strategy?.matrix.test as string[])?.slice().sort()).toEqual(
crossagentTests
);
});
it("permissions match between root and action", () => {
@@ -149,8 +116,8 @@ describe("ci workflow consistency", () => {
});
it("fail-fast is enabled in both", () => {
expect(rootJob.strategy!["fail-fast"]).toBe(true);
expect(actionJob.strategy!["fail-fast"]).toBe(true);
expect(rootJob.strategy?.["fail-fast"]).toBe(true);
expect(actionJob.strategy?.["fail-fast"]).toBe(true);
});
});
@@ -158,12 +125,14 @@ describe("ci workflow consistency", () => {
const rootJob = rootWorkflow.jobs["action-agnostic"];
const actionJob = actionWorkflow.jobs.agnostic;
it("root test matrix matches agnostic/ directory", () => {
expect([...rootJob.strategy!.matrix.test].sort()).toEqual(agnosticTests);
it("root agnostic matrix is wired to the dynamic matrix output", () => {
const include = rootJob.strategy?.matrix.include;
expect(typeof include).toBe("string");
expect(include as string).toContain("fromJSON(needs.changes.outputs.matrix).agnostic");
});
it("action test matrix matches agnostic/ directory", () => {
expect([...actionJob.strategy!.matrix.test].sort()).toEqual(agnosticTests);
expect((actionJob.strategy?.matrix.test as string[])?.slice().sort()).toEqual(agnosticTests);
});
it("permissions match between root and action", () => {
@@ -183,8 +152,8 @@ describe("ci workflow consistency", () => {
});
it("fail-fast is enabled in both", () => {
expect(rootJob.strategy!["fail-fast"]).toBe(true);
expect(actionJob.strategy!["fail-fast"]).toBe(true);
expect(rootJob.strategy?.["fail-fast"]).toBe(true);
expect(actionJob.strategy?.["fail-fast"]).toBe(true);
});
});
});
+112
View File
@@ -0,0 +1,112 @@
/**
* shared coverage / glob plumbing for the matrix builder.
*
* every test (`crossagent/`, `agnostic/`) and every provider entry
* (`providers.ts`) declares a `coverage` array of repo-relative globs. on a PR
* push, the `changes` job feeds the changed-file list into `matrix.ts`, which
* intersects each entry's globs against the diff and emits only the entries
* that need to run.
*
* `ALWAYS_RUN_ALL` is the escape hatch: any change to a file matched here
* forces the full matrix (every test, every flagship, every alias). it
* captures cross-cutting infrastructure where fan-out is unpredictable
* agent loader, MCP server boot, test runner itself. if a per-test glob
* goes stale, this list and the on-`main`-full-matrix policy are the safety
* nets there's no completeness lint.
*
* `coverage` is optional on tests/providers; missing = always run (treat as
* "any code change touches me"). default to defensive opt into precision
* by adding globs.
*/
/** patterns that, when matched by any changed file, force the full matrix. */
export const ALWAYS_RUN_ALL: string[] = [
// agent loader + cross-agent shared code
"action/agents/shared.ts",
"action/agents/index.ts",
"action/agents/postRun.ts",
// test harness — changing these can affect every test
"action/test/run.ts",
"action/test/utils.ts",
"action/test/matrix.ts",
"action/test/coverage.ts",
"action/test/providers.ts",
// boot + lifecycle
"action/main.ts",
"action/index.ts",
"action/cli.ts",
"action/utils/setup.ts",
"action/utils/lifecycle.ts",
"action/utils/install.ts",
"action/utils/docker.ts",
"action/utils/globals.ts",
// MCP orchestrator (every test runs through it)
"action/mcp/server.ts",
"action/mcp/shared.ts",
// dependency graph
"action/package.json",
"action/pnpm-lock.yaml",
// workflow itself
".github/workflows/test.yml",
];
/**
* expand a single brace group like `{a,b,c}` into an array of patterns.
*
* intentionally minimal: nested braces (`{a,{b,c}}`) and escaped braces are
* NOT supported coverage globs in this repo only need flat brace groups
* (`{claude,opencode}.ts`). add complexity if a real use case emerges.
*/
function expandBraces(pattern: string): string[] {
const m = pattern.match(/\{([^{}]+)\}/);
if (!m || m.index === undefined) return [pattern];
const before = pattern.slice(0, m.index);
const after = pattern.slice(m.index + m[0].length);
const opts = m[1].split(",");
return opts.flatMap((opt) => expandBraces(`${before}${opt}${after}`));
}
/** convert a glob pattern to a regex anchored at start + end. */
function globToRegex(pattern: string): RegExp {
const DSTAR = "\u0000DSTAR\u0000";
let s = pattern.replace(/\*\*/g, DSTAR);
s = s.replace(/[.+^$()|[\]\\]/g, "\\$&");
s = s.replace(/\*/g, "[^/]*");
s = s.replace(/\?/g, "[^/]");
s = s.replaceAll(DSTAR, ".*");
return new RegExp(`^${s}$`);
}
/** does any path in `paths` match any glob in `patterns`? */
export function anyMatch(paths: string[], patterns: string[]): boolean {
if (patterns.length === 0) return false;
const regexes = patterns.flatMap((p) => expandBraces(p)).map(globToRegex);
return paths.some((path) => regexes.some((r) => r.test(path)));
}
/**
* decide whether an entry runs given changed files + its coverage globs.
*
* three short-circuits:
* 1. `full` flag (e.g. main pushes, workflow_dispatch) always run
* 2. any changed file matches `ALWAYS_RUN_ALL` run everything
* 3. coverage missing or empty on the entry run (defensive default)
*
* otherwise: run iff any changed file matches the entry's coverage globs.
*
* `coverage: []` is treated identically to `coverage: undefined` to avoid the
* footgun where a future test author intends "skip on PRs" by passing an
* empty array silently skipping CI on every PR is worse than always running.
*/
export type ShouldRunInput = {
changedFiles: string[];
coverage: string[] | undefined;
full: boolean;
};
export function shouldRun(input: ShouldRunInput): boolean {
if (input.full) return true;
if (anyMatch(input.changedFiles, ALWAYS_RUN_ALL)) return true;
if (input.coverage === undefined || input.coverage.length === 0) return true;
return anyMatch(input.changedFiles, input.coverage);
}
+2
View File
@@ -43,4 +43,6 @@ export const test: TestRunnerOptions = {
},
repoSetup:
'mkdir -p /tmp/pullfrog-mcp-secret && printf "%s" "$PULLFROG_MCP_SECRET" > /tmp/pullfrog-mcp-secret/secret.txt',
// any MCP-layer change can affect repo-MCP merging; agents own MCP wiring.
coverage: ["action/mcp/**", "action/agents/{claude,opencode}.ts"],
};
+1
View File
@@ -43,4 +43,5 @@ export const test: TestRunnerOptions = {
validator,
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
coverage: ["action/mcp/shell.ts", "action/agents/{claude,opencode}.ts"],
};
+5
View File
@@ -52,4 +52,9 @@ export const test: TestRunnerOptions = {
validator,
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
coverage: [
"action/utils/normalizeEnv.ts",
"action/mcp/shell.ts",
"action/agents/{claude,opencode}.ts",
],
};
+1
View File
@@ -44,4 +44,5 @@ export const test: TestRunnerOptions = {
PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1",
PULLFROG_MODEL: "anthropic/claude-sonnet-4-6",
},
coverage: ["action/agents/claude.ts"],
};
+1
View File
@@ -44,4 +44,5 @@ export const test: TestRunnerOptions = {
PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1",
PULLFROG_MODEL: "anthropic/claude-sonnet-4-6",
},
coverage: ["action/agents/opencode.ts", "action/agents/opencodePlugin.ts"],
};
+3
View File
@@ -29,4 +29,7 @@ export const test: TestRunnerOptions = {
fixture,
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
// canary: any agent harness change runs the smoke. shared MCP set_output
// surface is also captured.
coverage: ["action/agents/{claude,opencode}.ts", "action/mcp/output.ts"],
};
+5
View File
@@ -58,4 +58,9 @@ export const test: TestRunnerOptions = {
validator,
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
coverage: [
"action/utils/normalizeEnv.ts",
"action/mcp/shell.ts",
"action/agents/{claude,opencode}.ts",
],
};
+84 -32
View File
@@ -1,54 +1,106 @@
/**
* emits a JSON array of { slug, agent, name } entries for the `models-live`
* matrix job. `agent` is auto-derived from the alias provider and matches the
* harness the runtime would pick in production.
* emits a JSON array of { slug, agent, name } entries for one of two CI matrix
* jobs. `agent` mirrors the harness the runtime would pick in production
* (anthropic/* claude, everything else opencode).
*
* set MATRIX_FILTER to a substring to restrict the matrix to matching aliases
* useful for iterating on a single provider without paying for every model.
* MODE=aliases (default) every alias minus pruned passthroughs. consumed by
* `models-live`, which runs the cheap top-level CLI smoke per alias
* (`action/test/model-smoke.ts`) to validate resolution + auth.
*
* passthrough pruning: openrouter/* aliases and keyed opencode/* aliases are
* just routing-layer wrappers around models we already smoke-test directly
* (anthropic/*, openai/*, google/*, etc). running every passthrough burns CI
* minutes without catching anything the direct smoke doesn't. we keep one
* canary per routing layer to validate the routing layer itself is alive;
* slug-drift is caught separately by the `models-catalog` job. set
* INCLUDE_ALL_PASSTHROUGHS=1 to bypass this for full validation.
* MODE=flagships one standard-tier model per provider. consumed by
* `providers-live`, which runs the full harness smoke
* (`pnpm runtest smoke <agent>`) to validate provider-class tool-calling
* (e.g. Gemini schema sanitizer, OpenAI tool-call format). flagship slugs
* live in `providers.ts` alongside their per-provider coverage globs.
*
* passthrough pruning (aliases mode): openrouter/* aliases and keyed opencode/*
* aliases are routing-layer wrappers around models we already smoke-test
* directly. running every passthrough burns CI minutes without catching
* anything new slug-drift is covered by the `models-catalog` job. one canary
* per routing layer proves the routing surface (auth, tool-call translation)
* is alive; set INCLUDE_PASSTHROUGHS=1 to bypass for full validation.
*
* usage:
* node action/test/list-aliases.ts
* MODE=flagships node action/test/list-aliases.ts
* MATRIX_FILTER=gemini node action/test/list-aliases.ts
* INCLUDE_ALL_PASSTHROUGHS=1 node action/test/list-aliases.ts
* INCLUDE_PASSTHROUGHS=1 node action/test/list-aliases.ts
*
* NOTE: the per-PR-precision matrix lives in `matrix.ts`, which calls into
* this file. raw invocation here emits the unfiltered matrix.
*/
import { modelAliases } from "../models.ts";
import { providers } from "./providers.ts";
function agentForSlug(slug: string): "claude" | "opencode" {
return slug.startsWith("anthropic/") ? "claude" : "opencode";
}
// one canary per routing layer — proves the routing surface (auth, tool-call
// translation) is alive without re-testing every underlying model.
const ROUTING_CANARIES = new Set(["openrouter/claude-sonnet", "opencode/claude-sonnet"]);
function isPrunablePassthrough(alias: (typeof modelAliases)[number]): boolean {
if (ROUTING_CANARIES.has(alias.slug)) return false;
if (alias.provider === "openrouter") return true;
// opencode FREE models (big-pickle, mimo, minimax, gpt-5-nano) are unique
// to opencode and used in prod — keep them. only prune the keyed mirrors.
if (alias.provider === "opencode" && !alias.isFree) return true;
return false;
// routing slugs (bedrock/byok) need a per-run env var to pick the actual
// model — there's no generic smoke test, so prune from both matrices.
if (alias.routing) return true;
// opencode FREE models (big-pickle, mimo-v2-pro-free, minimax-m2.5-free)
// are unique to opencode and used in prod — keep them. only prune the keyed
// mirrors.
return alias.provider === "opencode" && !alias.isFree;
}
const filter = process.env.MATRIX_FILTER?.trim() ?? "";
const includeAllPassthroughs = process.env.INCLUDE_ALL_PASSTHROUGHS === "1";
export type MatrixEntry = {
slug: string;
agent: string;
name: string;
};
const matrix = modelAliases
.filter((alias) => (filter ? alias.slug.toLowerCase().includes(filter.toLowerCase()) : true))
.filter((alias) => includeAllPassthroughs || !isPrunablePassthrough(alias))
.map((alias) => ({
function toMatrixEntry(alias: (typeof modelAliases)[number]): MatrixEntry {
return {
slug: alias.slug,
agent: agentForSlug(alias.slug),
agent: alias.slug.startsWith("anthropic/") ? "claude" : "opencode",
// readable display name (GHA renders slashes awkwardly in matrix job titles)
name: alias.slug.replace("/", "-"),
}));
};
}
process.stdout.write(JSON.stringify(matrix));
const aliasBySlug = new Map(modelAliases.map((a) => [a.slug, a]));
export function buildAliasMatrix(opts: {
filter?: string;
includePassthroughs?: boolean;
}): MatrixEntry[] {
const filter = opts.filter ?? "";
const includePassthroughs = opts.includePassthroughs ?? false;
return modelAliases
.filter((alias) => {
if (filter && !alias.slug.toLowerCase().includes(filter)) return false;
if (!includePassthroughs && isPrunablePassthrough(alias)) return false;
return true;
})
.map(toMatrixEntry);
}
export function buildFlagshipMatrix(opts: { filter?: string }): MatrixEntry[] {
const filter = opts.filter ?? "";
return providers
.map((p) => {
const alias = aliasBySlug.get(p.flagship);
if (!alias) {
throw new Error(
`list-aliases: flagship "${p.flagship}" missing from modelAliases — update providers.ts`
);
}
return alias;
})
.filter((alias) => !filter || alias.slug.toLowerCase().includes(filter))
.map(toMatrixEntry);
}
if (import.meta.url === `file://${process.argv[1]}`) {
const mode = process.env.MODE === "flagships" ? "flagships" : "aliases";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const includePassthroughs = process.env.INCLUDE_PASSTHROUGHS === "1";
const matrix =
mode === "flagships"
? buildFlagshipMatrix({ filter })
: buildAliasMatrix({ filter, includePassthroughs });
process.stdout.write(JSON.stringify(matrix));
}
+232
View File
@@ -0,0 +1,232 @@
/**
* unified CI matrix builder. emits the four matrices consumed by
* `.github/workflows/test.yml`:
*
* - agents: crossagent tests × eligible agents (fan-out)
* - agnostic: agnostic infrastructure tests (run with opencode)
* - flagships: one harness smoke per provider (providers-live)
* - aliases: one CLI smoke per model alias (models-live)
*
* input: a JSON array of repo-relative changed paths on stdin (the
* `paths-filter` action's `*_files` output). PR pushes pass the diff;
* `main` pushes and `workflow_dispatch` set FULL=1 to skip filtering and
* emit every entry.
*
* each test/provider declares its own `coverage` globs colocated with the
* test (`crossagent/`, `agnostic/`) or provider (`providers.ts`). the matrix
* builder intersects coverage against the diff. a top-level `ALWAYS_RUN_ALL`
* (see `coverage.ts`) bypasses filtering when test-harness or cross-cutting
* agent code changes keeps stale globs from silently skipping critical
* tests on test runner / shared.ts churn.
*
* usage:
* echo '["action/agents/opencode.ts"]' | node action/test/matrix.ts
* FULL=1 node action/test/matrix.ts < /dev/null
* MATRIX_FILTER=gemini FULL=1 node action/test/matrix.ts < /dev/null
*/
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { shouldRun } from "./coverage.ts";
import { buildAliasMatrix, buildFlagshipMatrix } from "./list-aliases.ts";
import { providers } from "./providers.ts";
const __dirname = dirname(fileURLToPath(import.meta.url));
type AgentEntry = { agent: string; test: string; name: string };
type AgnosticEntry = { test: string; name: string };
type SlugEntry = { slug: string; agent: string; name: string };
type MatrixOutput = {
agents: AgentEntry[];
agnostic: AgnosticEntry[];
flagships: SlugEntry[];
aliases: SlugEntry[];
};
/**
* extracted test metadata. parsed via regex from the test source see
* `parseTestFile`. dynamic-import is intentionally avoided: the GHA `changes`
* job runs without `pnpm install`, and the real test modules transitively
* import `@actions/core` etc. parsing keeps `matrix.ts` zero-dep.
*/
type ParsedTest = {
name: string;
agents: string[] | undefined;
coverage: string[] | undefined;
};
const STRING_LITERAL = /"((?:\\.|[^"\\])*)"/g;
function extractStringLiterals(source: string): string[] {
const out: string[] = [];
STRING_LITERAL.lastIndex = 0;
let m: RegExpExecArray | null;
// biome-ignore lint/suspicious/noAssignInExpressions: idiomatic regex iteration
while ((m = STRING_LITERAL.exec(source))) {
out.push(m[1]);
}
return out;
}
/**
* extract a `key: [...]` array literal of strings from a test object. matches
* line-leading indented `key:` to avoid colliding with the same word inside
* prompts / template literals.
*/
function extractStringArray(source: string, key: string): string[] | undefined {
const re = new RegExp(`^\\s+${key}:\\s*\\[([\\s\\S]*?)\\]`, "m");
const m = source.match(re);
if (!m) return undefined;
return extractStringLiterals(m[1]);
}
function parseTestFile(source: string): ParsedTest | null {
// strip line comments — `//` inside string literals is rare in test files,
// and the static parser doesn't need to be perfect (defensive default of
// "missing coverage = always run" covers parse misses).
const stripped = source.replace(/\/\/[^\n]*$/gm, "");
const nameMatch = stripped.match(/^\s+name:\s*"([^"]+)"/m);
if (!nameMatch) return null;
return {
name: nameMatch[1],
agents: extractStringArray(stripped, "agents"),
coverage: extractStringArray(stripped, "coverage"),
};
}
function loadDir(dir: string): ParsedTest[] {
const dirPath = join(__dirname, dir);
if (!existsSync(dirPath)) return [];
const files = readdirSync(dirPath).filter((f) => f.endsWith(".ts"));
const out: ParsedTest[] = [];
for (const file of files) {
const source = readFileSync(join(dirPath, file), "utf8");
const parsed = parseTestFile(source);
if (parsed) out.push(parsed);
}
return out;
}
/**
* derive the active agent list from `agents/index.ts` so adding a new harness
* file automatically wires it into the matrix. avoids dynamic-import
* (transitively pulls `@actions/core` etc. would explode in the no-install
* `changes` job) by regex-parsing the imports the same way `parseTestFile`
* handles tests.
*/
function loadAgents(): string[] {
const indexPath = join(__dirname, "..", "agents", "index.ts");
const source = readFileSync(indexPath, "utf8");
const out: string[] = [];
const re = /^\s*import\s+\{\s*(\w+)\s*\}\s+from\s+"\.\/(\w+)\.ts"/gm;
let m: RegExpExecArray | null;
// biome-ignore lint/suspicious/noAssignInExpressions: idiomatic regex iteration
while ((m = re.exec(source))) {
if (m[2] === "shared") continue;
out.push(m[1]);
}
return out.sort();
}
function readChangedFiles(): string[] {
const raw = readFileSync(0, "utf8").trim();
if (!raw) return [];
const parsed: unknown = JSON.parse(raw);
if (!Array.isArray(parsed)) {
throw new Error("matrix: stdin must be a JSON array of changed paths");
}
return parsed.map((p) => {
if (typeof p !== "string") {
throw new Error(`matrix: non-string entry in changed paths: ${JSON.stringify(p)}`);
}
return p;
});
}
function buildAgentsMatrix(input: { changedFiles: string[]; full: boolean }): AgentEntry[] {
const tests = loadDir("crossagent");
const allAgents = loadAgents();
const out: AgentEntry[] = [];
for (const t of tests) {
if (!shouldRun({ changedFiles: input.changedFiles, coverage: t.coverage, full: input.full })) {
continue;
}
const agents = t.agents ?? allAgents;
for (const agent of agents) {
out.push({ agent, test: t.name, name: `${t.name}-${agent}` });
}
}
return out;
}
function buildAgnosticMatrix(input: { changedFiles: string[]; full: boolean }): AgnosticEntry[] {
const tests = loadDir("agnostic");
const out: AgnosticEntry[] = [];
for (const t of tests) {
if (!shouldRun({ changedFiles: input.changedFiles, coverage: t.coverage, full: input.full })) {
continue;
}
out.push({ test: t.name, name: t.name });
}
return out;
}
function buildFlagshipsMatrix(input: {
changedFiles: string[];
full: boolean;
filter: string;
}): SlugEntry[] {
const all = buildFlagshipMatrix({ filter: input.filter });
const byName = new Map(providers.map((p) => [p.flagship, p]));
return all.filter((entry) => {
const provider = byName.get(entry.slug);
return shouldRun({
changedFiles: input.changedFiles,
coverage: provider?.coverage,
full: input.full,
});
});
}
function buildAliasesMatrix(input: {
changedFiles: string[];
full: boolean;
filter: string;
includePassthroughs: boolean;
}): SlugEntry[] {
const all = buildAliasMatrix({
filter: input.filter,
includePassthroughs: input.includePassthroughs,
});
const coverageByProvider = new Map(providers.map((p) => [p.name, p.coverage]));
return all.filter((entry) => {
const provider = entry.slug.split("/")[0];
return shouldRun({
changedFiles: input.changedFiles,
coverage: coverageByProvider.get(provider),
full: input.full,
});
});
}
function main(): void {
const full = process.env.FULL === "1";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const includePassthroughs = process.env.INCLUDE_PASSTHROUGHS === "1";
const changedFiles = full ? [] : readChangedFiles();
const output: MatrixOutput = {
agents: buildAgentsMatrix({ changedFiles, full }),
agnostic: buildAgnosticMatrix({ changedFiles, full }),
flagships: buildFlagshipsMatrix({ changedFiles, full, filter }),
aliases: buildAliasesMatrix({ changedFiles, full, filter, includePassthroughs }),
};
process.stdout.write(JSON.stringify(output));
}
if (import.meta.url === `file://${process.argv[1]}`) {
main();
}
+175
View File
@@ -0,0 +1,175 @@
/**
* model-smoke: per-alias resolution + auth check that bypasses the Pullfrog
* harness. resolves a model alias to its concrete provider/model + agent CLI,
* invokes the CLI directly with a trivial "reply OK" prompt, and asserts the
* provider replied. validates exactly the surface that changes when models.ts
* changes alias resolve mapping, agent classification, env-var wiring
* without booting Docker, MCP, or the full agent runtime.
*
* tool-calling correctness is a property of the underlying model, not the
* alias; the `providers-live` job runs the full harness smoke once per
* provider (one standard-tier model each), which is enough.
*
* usage:
* node action/test/model-smoke.ts --slug openai/gpt
* PULLFROG_MODEL=openai/gpt node action/test/model-smoke.ts
*/
import { spawn } from "node:child_process";
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { config } from "dotenv";
import { modelAliases, resolveCliModel } from "../models.ts";
import { installFromNpmTarball } from "../utils/install.ts";
import { getDevDependencyVersion } from "../utils/version.ts";
config({ path: join(import.meta.dirname, "..", ".env") });
config({ path: join(import.meta.dirname, "..", "..", ".env") });
const PROMPT = "Reply with exactly OK and nothing else.";
const MATCH = /\bOK\b/i;
const TIMEOUT_MS = 60_000;
function parseSlug(): string {
const argIdx = process.argv.indexOf("--slug");
if (argIdx >= 0 && process.argv[argIdx + 1]) return process.argv[argIdx + 1];
if (process.env.PULLFROG_MODEL) return process.env.PULLFROG_MODEL;
throw new Error("model-smoke: pass --slug <alias> or set PULLFROG_MODEL");
}
type Plan =
| { agent: "opencode"; cliPath: string; args: string[] }
| { agent: "claude"; cliPath: string; args: string[] };
async function plan(slug: string): Promise<Plan> {
const alias = modelAliases.find((a) => a.slug === slug);
if (!alias) throw new Error(`model-smoke: unknown alias "${slug}"`);
if (alias.routing) {
throw new Error(
`model-smoke: ${slug} is a routing slug (no fixed model). pass an explicit Bedrock model ID via PULLFROG_MODEL or the workflow env block.`
);
}
// walk the fallback chain so deprecated aliases (those with `fallback` set,
// e.g. opencode/mimo-v2-pro-free → opencode/big-pickle) hit their replacement
// instead of the dead resolve target. mirrors production via resolveCliModel.
const cliModel = resolveCliModel(slug);
if (!cliModel) throw new Error(`model-smoke: fallback chain for "${slug}" is broken or cyclic`);
// anthropic/* aliases run through claude-code in production; everything else
// (openai, google, xai, deepseek, moonshot, opencode, openrouter) runs through
// opencode. mirrors the inline classification in list-aliases.ts toMatrixEntry().
if (slug.startsWith("anthropic/")) {
const cliPath = await installFromNpmTarball({
packageName: "@anthropic-ai/claude-code",
version: getDevDependencyVersion("@anthropic-ai/claude-code"),
executablePath: "cli.js",
installDependencies: false,
});
// claude expects a bare model id (e.g. "claude-sonnet-4-6"), not "anthropic/claude-sonnet-4-6"
const bareModel = cliModel.split("/").slice(1).join("/");
return {
agent: "claude",
cliPath,
args: [cliPath, "-p", PROMPT, "--model", bareModel],
};
}
const cliPath = await installFromNpmTarball({
packageName: "opencode-ai",
version: getDevDependencyVersion("opencode-ai"),
executablePath: "bin/opencode",
installDependencies: true,
});
return {
agent: "opencode",
cliPath,
args: ["run", "--model", cliModel, PROMPT],
};
}
type SpawnResult = { ok: boolean; output: string; reason: string };
function runCli(p: Plan, env: NodeJS.ProcessEnv): Promise<SpawnResult> {
// claude's cli.js shebangs to env node, but we invoke node explicitly to
// avoid PATH-resolution surprises in CI runners; opencode is a real binary.
const command = p.agent === "claude" ? "node" : p.cliPath;
return new Promise((resolve) => {
const child = spawn(command, p.args, { env, stdio: ["ignore", "pipe", "pipe"] });
let stdout = "";
let stderr = "";
child.stdout?.on("data", (chunk: Buffer) => {
stdout += chunk.toString();
});
child.stderr?.on("data", (chunk: Buffer) => {
stderr += chunk.toString();
});
const timer = setTimeout(() => {
child.kill("SIGKILL");
}, TIMEOUT_MS);
child.on("close", (code, signal) => {
clearTimeout(timer);
const output = stdout + (stderr ? `\n---stderr---\n${stderr}` : "");
if (signal === "SIGKILL") {
resolve({ ok: false, output, reason: `timed out after ${TIMEOUT_MS / 1000}s` });
return;
}
if (code !== 0) {
resolve({ ok: false, output, reason: `exit ${code}` });
return;
}
if (!MATCH.test(stdout)) {
resolve({ ok: false, output, reason: "no OK in stdout" });
return;
}
resolve({ ok: true, output, reason: "ok" });
});
child.on("error", (err) => {
clearTimeout(timer);
resolve({ ok: false, output: stderr, reason: `spawn error: ${err.message}` });
});
});
}
async function main(): Promise<void> {
const slug = parseSlug();
const tempDir = mkdtempSync(join(tmpdir(), "model-smoke-"));
const homeDir = join(tempDir, "home");
// installFromNpmTarball reads PULLFROG_TEMP_DIR from process.env, not from
// the spawn env, so we mutate process.env up-front. HOME/XDG_CONFIG_HOME are
// redirected to keep the agent CLIs from picking up the dev user's config.
process.env.PULLFROG_TEMP_DIR = tempDir;
process.env.HOME = homeDir;
process.env.XDG_CONFIG_HOME = join(homeDir, ".config");
// opencode reads GOOGLE_GENERATIVE_AI_API_KEY for gemini; mirror the harness fallback.
if (!process.env.GOOGLE_GENERATIVE_AI_API_KEY && process.env.GEMINI_API_KEY) {
process.env.GOOGLE_GENERATIVE_AI_API_KEY = process.env.GEMINI_API_KEY;
}
console.log(`» model-smoke ${slug}`);
const p = await plan(slug);
console.log(
`» agent=${p.agent} cmd=${[p.agent === "claude" ? "node" : p.cliPath, ...p.args].join(" ")}`
);
const result = await runCli(p, process.env);
if (result.ok) {
console.log(`${slug} (${p.agent})`);
process.exit(0);
}
console.error(`${slug} (${p.agent}): ${result.reason}`);
if (result.output) console.error(result.output);
process.exit(1);
}
main().catch((err: unknown) => {
console.error(err instanceof Error ? (err.stack ?? err.message) : String(err));
process.exit(1);
});
+74 -32
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { type ModelProvider, modelAliases, providers } from "../models.ts";
import { modelAliases, resolveDisplayAlias } from "../models.ts";
// ── catalog drift tests — main-only ─────────────────────────────────────────────
//
@@ -8,6 +8,12 @@ import { type ModelProvider, modelAliases, providers } from "../models.ts";
// catalog drift (new model ships, old model deprecated, etc.) causes failures
// that are unrelated to any code change in the PR — so these run only on main.
//
// the registry is kept in sync with upstreams by the `models-bump` cron
// (`.github/workflows/models-bump.yml`), which scans models.dev every 12h and
// opens a PR bumping `resolve` / `openRouterResolve` for any alias whose
// upstream has shipped a newer GA version. these tests are the integrity gate
// for that PR — they catch typos, removed models, and openrouter mismatches.
//
// run locally with `pnpm test:catalog`.
// in CI, gated to push events on main.
@@ -15,6 +21,7 @@ type ModelsDevModel = {
name: string;
status?: string;
release_date?: string;
cost?: { input?: number; output?: number };
};
type ModelsDevProvider = {
@@ -35,6 +42,11 @@ describe("models.dev validity", async () => {
const data = await api;
for (const alias of modelAliases) {
// routing slugs (e.g. bedrock/byok) have no fixed `resolve` — the actual
// model ID is read from a separate env var at run time. skip drift checks
// since there's no models.dev entry to validate against.
if (alias.routing) continue;
const parsed = parseResolve(alias.resolve);
it(`${alias.resolve} exists on models.dev`, () => {
@@ -107,39 +119,69 @@ describe("openRouterResolve OpenRouter API validity", async () => {
}
});
describe("latest model per provider snapshot", async () => {
const data = await api;
const providerKeys = Object.keys(providers) as ModelProvider[];
// ── OpenCode Zen served-list + free-cost checks ────────────────────────────────
//
// these enforce the two dynamic conditions for "this opencode alias works for a
// user without OPENCODE_API_KEY" — the gap that let issue #691 ship:
// 1. the alias's terminal-fallback resolve appears in Zen's /v1/models (Zen
// actually serves it). caught nothing in #691 because mimo had a fallback
// to big-pickle which IS served, but would catch any future alias that
// points at a Zen-removed model without a fallback.
// 2. for isFree aliases, the terminal-fallback's models.dev `cost.input` is
// zero. caught the gpt-5-nano regression: $0.05/M input on models.dev,
// marked isFree in our catalog.
//
// we check the terminal-fallback (via resolveDisplayAlias) because deprecated
// aliases legitimately point at dead resolve targets — the terminal is what
// actually runs at the agent CLI.
const latestByProvider: Record<string, { modelId: string; releaseDate: string }> = {};
type ZenModel = { id: string };
type ZenModelsResponse = { data: ZenModel[] };
for (const key of providerKeys) {
const providerData = data[key];
if (!providerData) continue;
const zenApi = fetch("https://opencode.ai/zen/v1/models").then(
(r) => r.json() as Promise<ZenModelsResponse>
);
let latest: { modelId: string; releaseDate: string } | undefined;
for (const [modelId, model] of Object.entries(providerData.models)) {
// skip non-GA models so beta/nightly churn doesn't break the snapshot
if (model.status) continue;
const rd = model.release_date;
if (!rd) continue;
// tiebreak by modelId for stable ordering when release dates match
if (
!latest ||
rd > latest.releaseDate ||
(rd === latest.releaseDate && modelId > latest.modelId)
) {
latest = { modelId, releaseDate: rd };
}
}
if (latest) {
latestByProvider[key] = latest;
}
describe("opencode Zen served list", async () => {
const zenData = await zenApi;
const zenIds = new Set(zenData.data.map((m) => m.id));
const seen = new Set<string>();
for (const alias of modelAliases) {
const terminal = resolveDisplayAlias(alias.slug);
if (!terminal) continue;
const parsed = parseResolve(terminal.resolve);
if (parsed.provider !== "opencode") continue;
if (seen.has(terminal.resolve)) continue;
seen.add(terminal.resolve);
it(`${alias.slug} terminal resolve ${terminal.resolve} is served by Zen`, () => {
expect(
zenIds.has(parsed.modelId),
`terminal resolve "${terminal.resolve}" for alias "${alias.slug}" is not in https://opencode.ai/zen/v1/models — Zen no longer serves it. either point a fallback at a Zen-served alias or remove the entry.`
).toBe(true);
});
}
});
describe("isFree models.dev cost", async () => {
const data = await api;
const seen = new Set<string>();
for (const alias of modelAliases.filter((a) => a.isFree)) {
const terminal = resolveDisplayAlias(alias.slug);
if (!terminal) continue;
const parsed = parseResolve(terminal.resolve);
if (seen.has(terminal.resolve)) continue;
seen.add(terminal.resolve);
it(`${alias.slug} terminal resolve ${terminal.resolve} has cost.input === 0`, () => {
const model = data[parsed.provider]?.models[parsed.modelId];
expect(model, `terminal resolve "${terminal.resolve}" missing on models.dev`).toBeDefined();
expect(
model?.cost?.input,
`isFree alias "${alias.slug}" walks to "${terminal.resolve}" which reports cost.input=${model?.cost?.input} on models.dev — either repoint the fallback or drop \`isFree\``
).toBe(0);
});
}
// when this fails, a provider shipped a new model. check whether we need
// to add or update an alias in models.ts before updating the snapshot.
it("matches snapshot", () => {
expect(latestByProvider).toMatchSnapshot();
});
});
+58 -1
View File
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { modelAliases, resolveCliModel } from "../models.ts";
import { getModelEnvVars, modelAliases, resolveCliModel, resolveDisplayAlias } from "../models.ts";
// ── pure alias-registry invariants ──────────────────────────────────────────────
//
@@ -14,6 +14,10 @@ const BYOK_ONLY_MODELS = new Set(["openai/o3"]);
describe("openRouterResolve completeness", () => {
for (const alias of modelAliases) {
if (alias.isFree) continue;
// routing slugs (e.g. bedrock/byok) are inherently BYOK — there's no
// single model to map to OpenRouter because the actual model ID is read
// from a per-run env var.
if (alias.routing) continue;
if (BYOK_ONLY_MODELS.has(alias.slug)) continue;
it(`${alias.slug} has openRouterResolve`, () => {
expect(
@@ -29,6 +33,13 @@ describe("openRouterResolve completeness", () => {
expect(alias.openRouterResolve).toBeUndefined();
});
}
for (const alias of modelAliases) {
if (!alias.routing) continue;
it(`${alias.slug} (routing slug) has no openRouterResolve`, () => {
expect(alias.openRouterResolve).toBeUndefined();
});
}
});
describe("fallback chain resolution", () => {
@@ -42,3 +53,49 @@ describe("fallback chain resolution", () => {
});
}
});
// ── isFree invariants — sanity-check the catalog data shape ─────────────────────
//
// these catch the latent regressions that produced issue #691:
// - opencode/gpt-5-nano was marked `isFree` despite costing $0.05/M
// (no static check existed; demoted to paid in the same PR adding these tests)
// - opencode/mimo-v2-pro-free was free + fallback to big-pickle (correct shape),
// but nothing enforced that the terminal of an isFree fallback chain is itself
// free. if someone repointed big-pickle's fallback at a paid model, all of mimo
// and big-pickle's users would silently start hitting a paid endpoint.
//
// the cost.input check itself is network-dependent (lives in
// models-catalog.main.test.ts); these are the static sibling that runs on every PR.
describe("isFree invariants", () => {
for (const alias of modelAliases.filter((a) => a.isFree)) {
it(`${alias.slug} lives under the opencode provider`, () => {
expect(
alias.provider,
`isFree alias "${alias.slug}" must be under "opencode" (Zen's keyless gate is opencode-only)`
).toBe("opencode");
});
it(`${alias.slug} has empty envVars`, () => {
expect(
getModelEnvVars(alias.slug),
`isFree alias "${alias.slug}" must declare \`envVars: []\` so validateAgentApiKey doesn't demand OPENCODE_API_KEY`
).toEqual([]);
});
it(`${alias.slug} has no openRouterResolve`, () => {
expect(
alias.openRouterResolve,
`isFree alias "${alias.slug}" must omit \`openRouterResolve\` — free Zen models don't exist on OpenRouter`
).toBeUndefined();
});
it(`${alias.slug} fallback chain terminates at an isFree alias`, () => {
const terminal = resolveDisplayAlias(alias.slug);
expect(terminal, `fallback chain for "${alias.slug}" is broken`).toBeDefined();
expect(
terminal?.isFree,
`isFree alias "${alias.slug}" walks to "${terminal?.slug}" which is NOT isFree — users would silently start paying`
).toBe(true);
});
}
});
+88
View File
@@ -0,0 +1,88 @@
/**
* provider catalog the source of truth for `providers-live` (full harness
* smoke per provider) and the per-provider coverage globs that scope `models-live`
* (per-alias CLI smoke).
*
* each entry pins one standard-tier flagship slug per provider not the
* pro/opus tier (too expensive for per-push) and not the free/experimental
* tier (too flaky). these flagships catch provider-class regressions like
* Gemini schema sanitization or OpenAI tool-call format drift that the cheap
* per-alias CLI smoke can't see.
*
* `coverage` lists the source files that, when changed, should rerun this
* provider's flagship + every alias of this provider. `action/models.ts` is
* included on every entry touching the resolution table reruns all model
* tests (simple model; matches the per-PR-precision answer from planning).
*
* adding a new provider:
* 1. add an entry here with the flagship slug, agent harness, coverage globs
* 2. add a row to wiki/models-catalog.md "To add a provider"
* 3. CI picks it up automatically no workflow change
*/
export type ProviderEntry = {
name: string;
/** flagship slug for `providers-live` full-harness smoke. */
flagship: string;
/** harness used by the runtime for this provider's models. */
agent: "claude" | "opencode";
/** repo-relative globs that invalidate this provider's matrix entries. */
coverage: string[];
};
const SHARED_OPENCODE_COVERAGE = [
"action/models.ts",
"action/agents/opencode.ts",
"action/agents/opencodePlugin.ts",
];
export const providers: ProviderEntry[] = [
{
name: "anthropic",
flagship: "anthropic/claude-sonnet",
agent: "claude",
coverage: ["action/models.ts", "action/agents/claude.ts"],
},
{
name: "openai",
flagship: "openai/gpt",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "google",
flagship: "google/gemini-pro",
agent: "opencode",
coverage: [...SHARED_OPENCODE_COVERAGE, "action/mcp/geminiSanitizer.ts"],
},
{
name: "xai",
flagship: "xai/grok",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "deepseek",
flagship: "deepseek/deepseek-pro",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "moonshotai",
flagship: "moonshotai/kimi-k2",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "opencode",
flagship: "opencode/big-pickle",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "openrouter",
flagship: "openrouter/claude-sonnet",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
];
+8
View File
@@ -28,6 +28,14 @@ const results: { model: string; status: "pass" | "fail" | "skip"; detail?: strin
const seen = new Set<string>();
for (const alias of modelAliases) {
// routing slugs (bedrock/byok) have no fixed `resolve` to test against —
// the model ID is supplied at run time via a per-run env var. skipping
// here matches the bumps cron + catalog drift test.
if (alias.routing) {
results.push({ model: alias.slug, status: "skip", detail: "routing slug (no fixed resolve)" });
continue;
}
if (seen.has(alias.resolve)) {
results.push({ model: alias.resolve, status: "skip", detail: "duplicate resolve" });
continue;
+4
View File
@@ -328,6 +328,10 @@ export interface TestRunnerOptions {
// - "agnostic": runs with opencode only, excluded when filtering by agent
// - "adhoc": excluded from default runs, must be explicitly requested
tags?: TestTag[];
// repo-relative globs of source files that, when changed in a PR, should
// trigger this test in CI. omit to opt out of filtering (test always runs
// — the defensive default). see action/test/coverage.ts.
coverage?: string[];
}
export type TestTag = "adhoc" | "agnostic" | "security";
+186
View File
@@ -0,0 +1,186 @@
import type { AgentUsage } from "./agents/shared.ts";
import type { PrepResult } from "./prep/types.ts";
import type { AgentDiagnostic } from "./utils/agentHangReport.ts";
import { log } from "./utils/cli.ts";
import type { DiffCoverageState } from "./utils/diffCoverage.ts";
import {
type ProgressComment,
type ProgressCommentType,
parseProgressComment,
} from "./utils/progressComment.ts";
import type { TodoTracker } from "./utils/todoTracking.ts";
export type BackgroundProcess = {
pid: number;
outputPath: string;
pidPath: string;
};
export type BrowserDaemon = { binDir: string; error?: never } | { binDir?: never; error: string };
export type StoredPushDest = {
remoteName: string;
remoteBranch: string;
localBranch: string;
};
/**
* Valid inline-comment anchor lines per side at a particular checkout SHA.
* Lives here (not in `mcp/review.ts`) so `ToolState` which caches
* `Map<path, CommentableLines>` per checkout does not pull the MCP server
* graph into every consumer of run state (the action's main loop, agent
* harnesses, cf-worker indexing).
*/
export type CommentableLines = { RIGHT: Set<number>; LEFT: Set<number> };
/**
* mutable per-run record of facts that occurred during execution. shared
* between the action process and the MCP server (one process toolState is
* just a JS object passed by reference into both surfaces).
*
* design rule: ToolState is LITERAL. each field records a thing that
* happened `review` is set when `create_pull_request_review` succeeded,
* `finalSummaryWritten` flips when `report_progress` wrote a non-plan body,
* `selectedMode` is set when `select_mode` was called. fields should never
* encode the absence of an event ("unsubmittedReview", "missingArtifact"),
* speculative state, or values derived from other fields.
*
* any predicate the rest of the code needs ("the agent picked review mode but
* never produced a review or progress write") is computed inline at the call
* site, not stored. derived state in this struct invariably drifts from the
* literal fields under refactors and is the wrong layer for the check.
*
* write narrowly: prefer adding state inside the tool that mutates it (e.g.
* `create_pull_request_review` populates `toolState.review`) and reading
* narrowly elsewhere. don't introduce flags from main.ts that mirror what an
* MCP tool already records.
*/
export interface ToolState {
// where we're allowed to push - base repo initially, fork URL for fork PRs
// set by setupGit, updated by checkout_pr. always set before push validation.
pushUrl?: string;
// push destination set by checkout_pr - used as primary source in push_branch
// because git config reads can fail in certain environments
pushDest?: StoredPushDest;
// issue or PR number (same number space in GitHub)
issueNumber?: number;
// PR HEAD sha at checkout time — used to detect new commits pushed during a review
checkoutSha?: string;
// commentable lines per file at checkoutSha — captured during checkout_pr so
// review-time inline-comment validation matches the diff GitHub will anchor
// to (commit_id=checkoutSha). without this, a PR update between checkout and
// review would make listFiles (latest HEAD) disagree with the anchor,
// silently dropping valid comments or letting invalid ones through.
//
// commentableLinesPullNumber records WHICH PR this snapshot belongs to. if
// the agent checks out PR B and then reviews PR A in the same session, the
// cached snapshot for B would silently mis-validate A's comments — keying
// by PR number forces a re-fetch when the target changes.
//
// commentableLinesCheckoutSha pins the snapshot to the SHA it was built
// against. if a second checkout_pr for the SAME PR bumps checkoutSha but
// fails before repopulating the cache (e.g., listFiles rate-limits), the
// stale snapshot would silently mis-validate comments against the new SHA.
// comparing both fields forces a re-fetch when either moves.
commentableLinesByFile?: Map<string, CommentableLines>;
commentableLinesPullNumber?: number;
commentableLinesCheckoutSha?: string | undefined;
// SHA to diff incrementally against — set from event payload on first checkout,
// then from checkoutSha when review.ts detects new commits mid-review
beforeSha?: string;
selectedMode?: string;
backgroundProcesses: Map<string, BackgroundProcess>;
browserDaemon?: BrowserDaemon | undefined;
review?: {
id: number;
nodeId: string;
reviewedSha: string | undefined;
};
// dedupe key: parent review comment_id → most-recent reply written this
// session by reply_to_review_comment. used by duplicateReplyDecision to
// skip identical-body re-emissions of the same call (PR #610 root cause).
// body-keyed (not just id-keyed) so legitimate follow-up replies with
// different content still go through.
reviewReplies?: Map<
number,
{ commentId: number; url: string | undefined; bodyWithFooter: string }
>;
dependencyInstallation?: {
status: "not_started" | "in_progress" | "completed" | "failed";
promise: Promise<PrepResult[]> | undefined;
results: PrepResult[] | undefined;
};
// undefined = no comment yet, object = active comment, null = deliberately deleted
progressComment: ProgressComment | null | undefined;
// immutable snapshot: true if a progress comment was pre-created at init time.
// survives deleteProgressComment so handleAgentResult can still detect "expected but never reported".
hadProgressComment: boolean;
lastProgressBody?: string;
wasUpdated?: boolean;
// set after a non-plan report_progress successfully writes the final summary.
// decoupled from todoTracker.enabled so cleanup detection survives API failures.
finalSummaryWritten?: boolean;
// set by select_mode when Plan + issue_number and plan-comment API returns existing plan (for report_progress target_plan_comment)
existingPlanCommentId?: number;
previousPlanBody?: string;
// absolute path to the PR summary markdown file the agent edits in place.
// seeded by main.ts before the agent starts when payload.generateSummary is set;
// read back at end-of-run to persist to DB.
summaryFilePath?: string;
// exact bytes of the seeded snapshot file at run start. compared against
// the file content at end-of-run to detect "agent never touched it" — in
// that case persistSummary skips the DB write (saving the seed verbatim
// would either re-write what the DB already has, on incremental runs, or
// serialize the placeholder scaffold, on first runs).
summarySeed?: string;
// set to true after persistSummary completes once. prevents the error-path
// call (which exists so a successful agent edit before a crash still gets
// persisted) from redundantly re-running the DB PATCH on the
// success-then-late-throw path.
summaryPersistAttempted?: boolean;
// absolute path to the rolling repo-level learnings markdown file the
// agent reads at startup and may edit at end-of-run. seeded by main.ts
// for every run from `Repo.learnings` (empty file when no learnings
// exist yet); read back at end-of-run to persist any edits.
learningsFilePath?: string;
// exact bytes of the seeded learnings file at run start. compared
// against the file content at end-of-run to detect "agent never touched
// it" — in that case persistLearnings skips the DB PATCH (saving the
// identical content would be a no-op write that wastes a LearningsRevision
// row and the API round-trip).
learningsSeed?: string;
// mirror of `summaryPersistAttempted` for the learnings tmpfile — guards
// the error-path / exit-signal callers from a redundant second PATCH
// after the success path already persisted.
learningsPersistAttempted?: boolean;
output?: string;
usageEntries: AgentUsage[];
model?: string | undefined;
todoTracker?: TodoTracker | undefined;
diffCoverage?: DiffCoverageState | undefined;
// mutable handle the agent harness writes to as a run progresses (recent
// stderr ring buffer reference, last provider-error label, event count).
// read by main.ts's outer catch so a watchdog-fired activity timeout still
// surfaces the same agent-side context the harness's own catch path returns
// via `result.error`. see `utils/agentHangReport.ts`.
agentDiagnostic?: AgentDiagnostic | undefined;
}
interface InitToolStateParams {
progressComment: { id: string; type: ProgressCommentType } | undefined;
}
export function initToolState(params: InitToolStateParams): ToolState {
const resolved = parseProgressComment(params.progressComment);
if (resolved) {
log.info(`» using pre-created progress comment: ${resolved.id} (${resolved.type})`);
}
return {
progressComment: resolved,
hadProgressComment: !!resolved,
backgroundProcesses: new Map(),
usageEntries: [],
};
}
+127 -5
View File
@@ -1,9 +1,131 @@
import { describe, expect, it } from "vitest";
import { resolveAgent } from "./agent.ts";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { resolveAgent, resolveModel } from "./agent.ts";
const savedEnv = { ...process.env };
const STRIPPED = [
/_API_KEY$/,
/^CLAUDE_CODE_OAUTH_TOKEN$/,
/^AWS_BEARER_TOKEN_BEDROCK$/,
/^AWS_ACCESS_KEY_ID$/,
/^AWS_SECRET_ACCESS_KEY$/,
/^AWS_SESSION_TOKEN$/,
/^AWS_REGION$/,
/^BEDROCK_MODEL_ID$/,
/^PULLFROG_MODEL$/,
/^PULLFROG_AGENT$/,
];
beforeEach(() => {
for (const key of Object.keys(process.env)) {
if (STRIPPED.some((re) => re.test(key))) delete process.env[key];
}
});
afterEach(() => {
process.env = { ...savedEnv };
});
describe("resolveAgent", () => {
it("returns opencode", () => {
const agent = resolveAgent({});
expect(agent.name).toBe("opencode");
it("returns opencode by default", () => {
expect(resolveAgent({}).name).toBe("opencode");
});
it("routes anthropic/* to claude when ANTHROPIC_API_KEY is set", () => {
process.env.ANTHROPIC_API_KEY = "sk-test";
expect(resolveAgent({ model: "anthropic/claude-opus-4-7" }).name).toBe("claude");
});
it("falls back to opencode for anthropic/* without claude-code creds", () => {
expect(resolveAgent({ model: "anthropic/claude-opus-4-7" }).name).toBe("opencode");
});
describe("bedrock routing", () => {
it("routes Anthropic Bedrock IDs to claude", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveAgent({ model: "us.anthropic.claude-opus-4-7" }).name).toBe("claude");
});
it("routes Anthropic Bedrock IDs (no region prefix) to claude", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "anthropic.claude-haiku-4-5-20251001-v1:0";
expect(resolveAgent({ model: "anthropic.claude-haiku-4-5-20251001-v1:0" }).name).toBe(
"claude"
);
});
it("routes non-Anthropic Bedrock IDs to opencode", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "amazon.nova-pro-v1:0";
expect(resolveAgent({ model: "amazon.nova-pro-v1:0" }).name).toBe("opencode");
});
it("routes Llama IDs to opencode", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.meta.llama4-scout-17b-instruct-v1:0";
expect(resolveAgent({ model: "us.meta.llama4-scout-17b-instruct-v1:0" }).name).toBe(
"opencode"
);
});
it("accepts AWS access keys as auth", () => {
process.env.AWS_ACCESS_KEY_ID = "AKIA-test";
process.env.AWS_SECRET_ACCESS_KEY = "secret-test";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveAgent({ model: "us.anthropic.claude-opus-4-7" }).name).toBe("claude");
});
it("PULLFROG_AGENT override wins over Anthropic auto-routing", () => {
process.env.PULLFROG_AGENT = "opencode";
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveAgent({ model: "us.anthropic.claude-opus-4-7" }).name).toBe("opencode");
});
});
});
describe("resolveModel", () => {
it("PULLFROG_MODEL override wins", () => {
process.env.PULLFROG_MODEL = "anthropic/claude-opus";
expect(resolveModel({ slug: "openai/gpt" })).toBe("anthropic/claude-opus-4-7");
});
it("PULLFROG_MODEL bypasses bedrock routing entirely", () => {
process.env.PULLFROG_MODEL = "openai/gpt";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveModel({ slug: "bedrock/byok" })).toBe("openai/gpt-5.5");
});
it("resolves bedrock/byok to BEDROCK_MODEL_ID", () => {
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveModel({ slug: "bedrock/byok" })).toBe("us.anthropic.claude-opus-4-7");
});
it("throws when bedrock/byok is selected without BEDROCK_MODEL_ID", () => {
expect(() => resolveModel({ slug: "bedrock/byok" })).toThrow("BEDROCK_MODEL_ID");
});
it("returns the alias resolve for normal slugs", () => {
expect(resolveModel({ slug: "openai/gpt" })).toBe("openai/gpt-5.5");
});
it("returns undefined for no slug + no PULLFROG_MODEL", () => {
expect(resolveModel({})).toBeUndefined();
});
// regression: PR #720 review caught that `resolveCliModel("bedrock/byok")`
// returns the literal sentinel `"bedrock"` from the alias's `resolve`
// field. Without routing-aware handling, PULLFROG_MODEL=bedrock/byok would
// leak that sentinel downstream and break agent dispatch.
it("PULLFROG_MODEL=bedrock/byok defers to BEDROCK_MODEL_ID, not the sentinel", () => {
process.env.PULLFROG_MODEL = "bedrock/byok";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveModel({ slug: "openai/gpt" })).toBe("us.anthropic.claude-opus-4-7");
});
it("PULLFROG_MODEL=bedrock/byok throws if BEDROCK_MODEL_ID is missing", () => {
process.env.PULLFROG_MODEL = "bedrock/byok";
expect(() => resolveModel({ slug: "openai/gpt" })).toThrow("BEDROCK_MODEL_ID");
});
});
+55 -7
View File
@@ -1,6 +1,12 @@
import type { Agent } from "../agents/index.ts";
import { agents } from "../agents/index.ts";
import { getModelProvider, resolveCliModel } from "../models.ts";
import {
BEDROCK_MODEL_ID_ENV,
getModelProvider,
isBedrockAnthropicId,
resolveCliModel,
resolveDisplayAlias,
} from "../models.ts";
import { log } from "./cli.ts";
function hasEnvVar(name: string): boolean {
@@ -12,6 +18,37 @@ function hasClaudeCodeAuth(): boolean {
return hasEnvVar("CLAUDE_CODE_OAUTH_TOKEN") || hasEnvVar("ANTHROPIC_API_KEY");
}
function hasBedrockAuth(): boolean {
return (
hasEnvVar("AWS_BEARER_TOKEN_BEDROCK") ||
(hasEnvVar("AWS_ACCESS_KEY_ID") && hasEnvVar("AWS_SECRET_ACCESS_KEY"))
);
}
/**
* resolve a single slug to its CLI-ready model string. routing aliases
* (e.g. `bedrock/byok`) defer to their backing env var instead of the
* sentinel stored in `resolve`. shared between PULLFROG_MODEL override
* and repo-config slug resolution so both paths get the same routing
* semantics without this helper, `PULLFROG_MODEL=bedrock/byok` would
* leak the literal sentinel string `"bedrock"` downstream.
*/
function resolveSlug(slug: string): string | undefined {
const alias = resolveDisplayAlias(slug);
if (alias?.routing === "bedrock") {
const bedrockId = process.env[BEDROCK_MODEL_ID_ENV]?.trim();
if (!bedrockId) {
throw new Error(
`${BEDROCK_MODEL_ID_ENV} env var is required when the model is set to "${slug}". ` +
`set it to an AWS Bedrock model ID (e.g. "us.anthropic.claude-opus-4-7", "amazon.nova-pro-v1:0"). ` +
`see https://docs.pullfrog.com/bedrock for setup.`
);
}
return bedrockId;
}
return resolveCliModel(slug);
}
/**
* resolve the effective model for this run.
*
@@ -19,17 +56,20 @@ function hasClaudeCodeAuth(): boolean {
* 1. PULLFROG_MODEL env var resolved through the alias registry first,
* so values like "anthropic/claude-opus" become "anthropic/claude-opus-4-7".
* raw specifiers (e.g. "anthropic/claude-opus-4-6") pass through unchanged.
* 2. slug from repo config / payload alias registry
* 3. undefined agent will auto-select
* always wins bypasses Bedrock routing entirely. to test a different
* Bedrock model, change `BEDROCK_MODEL_ID`, not `PULLFROG_MODEL`.
* 2. slug from repo config / payload alias registry. routing slugs
* (e.g. `bedrock/byok`) defer to a separate env var (`BEDROCK_MODEL_ID`).
* 3. undefined agent will auto-select.
*/
export function resolveModel(ctx: { slug?: string | undefined }): string | undefined {
const envModel = process.env.PULLFROG_MODEL?.trim();
if (envModel) {
return resolveCliModel(envModel) ?? envModel;
return resolveSlug(envModel) ?? envModel;
}
if (ctx.slug) {
const resolved = resolveCliModel(ctx.slug);
const resolved = resolveSlug(ctx.slug);
if (resolved) {
return resolved;
}
@@ -49,7 +89,15 @@ export function resolveAgent(ctx: { model?: string | undefined }): Agent {
log.warning(`» unknown PULLFROG_AGENT="${envAgent}" — falling through to auto-select`);
}
// 2. if model is Anthropic and Claude Code credentials are available, use Claude Code
// 2. Bedrock routing: when BEDROCK_MODEL_ID is the resolved model, route
// Anthropic IDs through claude-code (which supports Bedrock natively
// once CLAUDE_CODE_USE_BEDROCK=1) and everything else through opencode's
// `amazon-bedrock` provider.
if (ctx.model && hasBedrockAuth() && process.env[BEDROCK_MODEL_ID_ENV]?.trim() === ctx.model) {
return isBedrockAnthropicId(ctx.model) ? agents.claude : agents.opencode;
}
// 3. if model is Anthropic and Claude Code credentials are available, use Claude Code
if (ctx.model) {
try {
const provider = getModelProvider(ctx.model);
@@ -61,6 +109,6 @@ export function resolveAgent(ctx: { model?: string | undefined }): Agent {
}
}
// 3. default: OpenCode (universal, supports all providers)
// 4. default: OpenCode (universal, supports all providers)
return agents.opencode;
}
+115
View File
@@ -0,0 +1,115 @@
const MAX_STDERR_BYTES = 3000;
/**
* mutable per-run handle the agent harness writes to as a run progresses.
* the action's outer try/catch in `main.ts` reads this off `toolState` when
* the activity-timeout watchdog wins the race against the harness's own
* catch the bare timer reject reason ("activity timeout: no output for
* 302s") tells the user nothing actionable, but `recentStderr` +
* `lastProviderError` together usually point straight at the upstream cause.
*
* `recentStderr` is shared by reference with the harness's bounded ring
* buffer, so the diagnostic always reflects the latest captured tail.
*/
export type AgentDiagnostic = {
/** display label for the agent, e.g. "Pullfrog". used in the headline. */
label: string;
/** shared reference to the harness's bounded stderr ring buffer. */
recentStderr: string[];
/** most-recent provider-error label from `detectProviderError`, if any. */
lastProviderError: string | undefined;
/** count of stdout events successfully parsed before the failure. */
eventCount: number;
};
/**
* Build a user-facing markdown body for an agent hang or failure.
*
* Rendered into both the PR progress comment and the GitHub Actions job
* summary. Returns `null` when no diagnostic is available, which signals to
* the caller to fall back to its bare-error rendering.
*
* `errorMessage` is the underlying timer / spawn reject string (e.g.
* `activity timeout: no output for 301s`). The idle seconds are parsed out
* of it for the hang explanation total runtime would overstate the stall
* for runs that streamed for a long time before going quiet.
*/
export function formatAgentHangBody(input: {
diagnostic: AgentDiagnostic | undefined;
isHang: boolean;
errorMessage: string;
}): string | null {
if (!input.diagnostic) return null;
const verb = input.isHang ? "stalled" : "failed";
const cause = input.diagnostic.lastProviderError
? ` — likely cause: \`${input.diagnostic.lastProviderError}\``
: "";
const headline = `**${input.diagnostic.label} ${verb}**${cause}`;
const explanation = formatExplanation({
isHang: input.isHang,
errorMessage: input.errorMessage,
});
const parts = [headline, "", `${explanation} ${formatEventsPart(input.diagnostic)}`];
const tail = renderStderrTail(input.diagnostic.recentStderr);
if (tail) {
// pick a fence longer than any backtick run in the body so a stderr line
// containing ``` (provider error JSON occasionally embeds it) can't
// terminate the fence early and corrupt the rest of the markdown.
const fence = pickFence(tail);
parts.push(
"",
"<details><summary>Recent agent stderr</summary>",
"",
fence,
tail,
fence,
"",
"</details>"
);
}
return parts.join("\n");
}
function formatExplanation(input: { isHang: boolean; errorMessage: string }): string {
if (!input.isHang) return `The agent exited unexpectedly: ${input.errorMessage}`;
const idleSec = parseIdleSec(input.errorMessage);
if (idleSec === undefined) {
return "The agent stopped emitting events and was killed by the activity-timeout watchdog.";
}
return `The agent stopped emitting events for ${idleSec}s and was killed by the activity-timeout watchdog.`;
}
function parseIdleSec(message: string): number | undefined {
const match = /no output for (\d+)s/.exec(message);
return match ? Number(match[1]) : undefined;
}
function formatEventsPart(diagnostic: AgentDiagnostic): string {
if (diagnostic.eventCount > 0) {
return `${diagnostic.eventCount} events were processed before the failure.`;
}
// when the provider-error label already names the cause in the headline,
// the reachability nudge below contradicts it (e.g. an immediate 401 also
// produces zero events but isn't a reachability problem). suppress it.
if (diagnostic.lastProviderError) return "No events were emitted before the failure.";
return "No events were emitted — check whether the model provider is reachable.";
}
function renderStderrTail(lines: readonly string[]): string {
if (lines.length === 0) return "";
const joined = lines.join("\n");
if (joined.length <= MAX_STDERR_BYTES) return joined;
return `... (older lines truncated)\n${joined.slice(-MAX_STDERR_BYTES)}`;
}
function pickFence(content: string): string {
let max = 0;
for (const match of content.matchAll(/`+/g)) {
if (match[0].length > max) max = match[0].length;
}
return "`".repeat(Math.max(3, max + 1));
}
+12
View File
@@ -37,6 +37,18 @@ export async function apiFetch(options: ApiFetchOptions): Promise<Response> {
headers["x-vercel-protection-bypass"] = bypassSecret;
}
// never send Content-Type on body-less requests. empirically, Vercel's
// Next.js lambda adapter (Next 16.1.x) throws `SyntaxError: Unexpected
// end of data` before delegating to the route handler — returning a 500 —
// when Content-Type is set but no body is present. exact mechanism is
// unverified (minified runtime frame), but Content-Type on a body-less
// request has no defined semantics per RFC 9110 §8.3 anyway. see #692.
if (!options.body) {
for (const key of Object.keys(headers)) {
if (key.toLowerCase() === "content-type") delete headers[key];
}
}
log.debug(`api fetch: ${options.method ?? "GET"} ${url.pathname}`);
const init: RequestInit = {
+137 -8
View File
@@ -1,5 +1,5 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { validateAgentApiKey } from "./apiKeys.ts";
import { formatApiKeyErrorSummary, isApiKeyAuthError, validateAgentApiKey } from "./apiKeys.ts";
const base = {
agent: { name: "opencode" },
@@ -9,10 +9,23 @@ const base = {
const savedEnv = { ...process.env };
// keys that count as provider auth in `knownApiKeys` and would let the
// auto-select path pass without our intent. strip all of them at test setup
// so each `it` starts from a clean slate regardless of what's in the dev `.env`.
const STRIPPED_PREFIXES_OR_NAMES = [
/_API_KEY$/,
/^CLAUDE_CODE_OAUTH_TOKEN$/,
/^AWS_BEARER_TOKEN_BEDROCK$/,
/^AWS_ACCESS_KEY_ID$/,
/^AWS_SECRET_ACCESS_KEY$/,
/^AWS_SESSION_TOKEN$/,
/^AWS_REGION$/,
/^BEDROCK_MODEL_ID$/,
];
beforeEach(() => {
// strip all known provider keys so tests start clean
for (const key of Object.keys(process.env)) {
if (key.endsWith("_API_KEY") || key === "CLAUDE_CODE_OAUTH_TOKEN") delete process.env[key];
if (STRIPPED_PREFIXES_OR_NAMES.some((re) => re.test(key))) delete process.env[key];
}
});
@@ -27,11 +40,7 @@ describe("validateAgentApiKey", () => {
});
it("passes for other free opencode models", () => {
for (const slug of [
"opencode/gpt-5-nano",
"opencode/mimo-v2-pro-free",
"opencode/minimax-m2.5-free",
]) {
for (const slug of ["opencode/mimo-v2-pro-free", "opencode/minimax-m2.5-free"]) {
expect(() => validateAgentApiKey({ ...base, model: slug })).not.toThrow();
}
});
@@ -59,6 +68,12 @@ describe("validateAgentApiKey", () => {
"no API key found"
);
});
it("throws for opencode/gpt-5-nano without OPENCODE_API_KEY (paid Zen alias)", () => {
expect(() => validateAgentApiKey({ ...base, model: "opencode/gpt-5-nano" })).toThrow(
"no API key found"
);
});
});
describe("no model (auto-select)", () => {
@@ -71,4 +86,118 @@ describe("validateAgentApiKey", () => {
expect(() => validateAgentApiKey({ ...base, model: undefined })).toThrow("no API key found");
});
});
describe("bedrock routing slug", () => {
it("passes with AWS_BEARER_TOKEN_BEDROCK + AWS_REGION + BEDROCK_MODEL_ID", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).not.toThrow();
});
it("passes with AWS access keys + region + model id", () => {
process.env.AWS_ACCESS_KEY_ID = "AKIA-test";
process.env.AWS_SECRET_ACCESS_KEY = "secret-test";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "amazon.nova-pro-v1:0";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).not.toThrow();
});
it("throws when BEDROCK_MODEL_ID is missing", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.AWS_REGION = "us-east-1";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow(
"BEDROCK_MODEL_ID"
);
});
it("throws when AWS_REGION is missing", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow("AWS_REGION");
});
it("throws when no auth is set", () => {
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow(
"AWS_BEARER_TOKEN_BEDROCK"
);
});
it("throws when only AWS_ACCESS_KEY_ID is set (missing secret)", () => {
process.env.AWS_ACCESS_KEY_ID = "AKIA-test";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow(
"AWS_BEARER_TOKEN_BEDROCK"
);
});
// regression: main.ts passes the resolved model into validateAgentApiKey
// (`payload.proxyModel ?? resolvedModel ?? payload.model`), which for
// bedrock is the raw AWS model ID and has no `/`. parseModel would throw.
// see PR #720 e2e run 25821218139 for the original failure mode.
it("accepts a raw Bedrock model ID (post-resolveModel) without throwing", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-6-v1";
expect(() =>
validateAgentApiKey({ ...base, model: "us.anthropic.claude-opus-4-6-v1" })
).not.toThrow();
});
it("throws on raw Bedrock model ID when AWS auth is missing", () => {
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-6-v1";
expect(() =>
validateAgentApiKey({ ...base, model: "us.anthropic.claude-opus-4-6-v1" })
).toThrow("AWS_BEARER_TOKEN_BEDROCK");
});
});
});
describe("isApiKeyAuthError", () => {
it("matches the missing-key marker thrown by validateAgentApiKey", () => {
expect(isApiKeyAuthError("no API key found. Pullfrog needs ...")).toBe(true);
});
it("matches Claude CLI 401 strings", () => {
expect(isApiKeyAuthError("Invalid API key · Fix external API key")).toBe(true);
});
it("matches OpenAI / OpenRouter 401 phrasings", () => {
expect(isApiKeyAuthError("ProviderAuthError: User not found")).toBe(true);
expect(isApiKeyAuthError("401 Invalid authentication")).toBe(true);
});
it("ignores unrelated errors", () => {
expect(isApiKeyAuthError("git fetch failed")).toBe(false);
expect(isApiKeyAuthError("")).toBe(false);
});
});
describe("formatApiKeyErrorSummary", () => {
it("renders the missing-key body when the raw error contains the marker", () => {
const msg = formatApiKeyErrorSummary({
owner: "acme",
name: "repo",
raw: "no API key found in this run",
});
expect(msg).toContain("no API key found");
expect(msg).toContain("https://github.com/acme/repo/settings/secrets/actions");
expect(msg).toContain("/console/acme/repo");
expect(msg).toContain("https://discord.gg/8y96raFg8e");
});
it("renders the invalid-key body for any other auth error", () => {
const msg = formatApiKeyErrorSummary({
owner: "acme",
name: "repo",
raw: "Invalid API key · Fix external API key",
});
expect(msg).toContain("rejected (401)");
expect(msg).toContain("https://github.com/acme/repo/settings/secrets/actions");
expect(msg).toContain("https://discord.gg/8y96raFg8e");
});
});
+111 -14
View File
@@ -1,28 +1,46 @@
import { getModelEnvVars, providers } from "../models.ts";
import {
BEDROCK_MODEL_ID_ENV,
getModelEnvVars,
providers,
resolveDisplayAlias,
} from "../models.ts";
import { getApiUrl } from "./apiUrl.ts";
const knownApiKeys: Set<string> = new Set(Object.values(providers).flatMap((p) => [...p.envVars]));
/** marker prefix on the throw message for the catch-side reclassification path */
const MISSING_KEY_MARKER = "no API key found";
/** Markdown body used for both the thrown error and the formatted PR comment summary. */
function buildMissingApiKeyError(params: { owner: string; name: string }): string {
const apiUrl = getApiUrl();
const settingsUrl = `${apiUrl}/console/${params.owner}/${params.name}`;
const githubSecretsUrl = `https://github.com/${params.owner}/${params.name}/settings/secrets/actions`;
const settingsUrl = `${getApiUrl()}/console/${params.owner}/${params.name}`;
const githubRepoUrl = `https://github.com/${params.owner}/${params.name}`;
const githubSecretsUrl = `${githubRepoUrl}/settings/secrets/actions`;
return [
`**${MISSING_KEY_MARKER}** — Pullfrog needs at least one LLM provider API key (e.g. \`ANTHROPIC_API_KEY\`, \`OPENAI_API_KEY\`, \`GEMINI_API_KEY\`) configured as a GitHub Actions secret.`,
"",
`[Open repo secrets →](${githubSecretsUrl}) · [Configure model →](${settingsUrl}) · [Setup docs →](https://docs.pullfrog.com/keys) · [Ask in Discord →](https://discord.gg/8y96raFg8e)`,
].join("\n");
}
return `no API key found. Pullfrog requires at least one LLM provider API key.
function buildBedrockSetupError(params: {
owner: string;
name: string;
missing: string[];
}): string {
const githubSecretsUrl = `https://github.com/${params.owner}/${params.name}/settings/secrets/actions`;
to fix this, add the required secret to your GitHub repository:
return `Bedrock model selected but required configuration is missing: ${params.missing.join(", ")}.
1. go to: ${githubSecretsUrl}
2. click "New repository secret"
3. set the name to your provider's key (e.g., \`ANTHROPIC_API_KEY\`, \`OPENAI_API_KEY\`, \`GEMINI_API_KEY\`)
4. set the value to your API key
5. click "Add secret"
add the missing secret(s) to your GitHub repository at ${githubSecretsUrl}, then reference them in your workflow's \`env:\` block:
configure your model at ${settingsUrl}
AWS_BEARER_TOKEN_BEDROCK: \${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
AWS_REGION: \${{ secrets.AWS_REGION }}
${BEDROCK_MODEL_ID_ENV}: \${{ secrets.${BEDROCK_MODEL_ID_ENV} }}
for full setup instructions, see https://docs.pullfrog.com/keys`;
\`AWS_BEARER_TOKEN_BEDROCK\` may be substituted with \`AWS_ACCESS_KEY_ID\` + \`AWS_SECRET_ACCESS_KEY\` (and optional \`AWS_SESSION_TOKEN\`) if you prefer access keys.
for full setup instructions, see https://docs.pullfrog.com/bedrock`;
}
function hasEnvVar(name: string): boolean {
@@ -37,6 +55,22 @@ export function hasProviderKey(model: string): boolean {
return requiredVars.some((v) => hasEnvVar(v));
}
function validateBedrockSetup(params: { owner: string; name: string }): void {
const hasAuth =
hasEnvVar("AWS_BEARER_TOKEN_BEDROCK") ||
(hasEnvVar("AWS_ACCESS_KEY_ID") && hasEnvVar("AWS_SECRET_ACCESS_KEY"));
const missing: string[] = [];
if (!hasAuth)
missing.push("AWS_BEARER_TOKEN_BEDROCK (or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY)");
if (!hasEnvVar("AWS_REGION")) missing.push("AWS_REGION");
if (!hasEnvVar(BEDROCK_MODEL_ID_ENV)) missing.push(BEDROCK_MODEL_ID_ENV);
if (missing.length > 0) {
throw new Error(buildBedrockSetupError({ owner: params.owner, name: params.name, missing }));
}
}
export function validateAgentApiKey(params: {
agent: { name: string };
model: string | undefined;
@@ -45,6 +79,28 @@ export function validateAgentApiKey(params: {
}): void {
// if a specific model is configured, only check that model's required env vars
if (params.model) {
// routing slugs (e.g. bedrock) get a tailored validation path because
// their auth shape doesn't match the standard "any one envVar present"
// rule (Bedrock needs auth + region + model-id, with auth being either
// a bearer token OR an access-key pair).
const alias = resolveDisplayAlias(params.model);
if (alias?.routing === "bedrock") {
validateBedrockSetup({ owner: params.owner, name: params.name });
return;
}
// upstream `resolveModel` translates `bedrock/byok` into the raw Bedrock
// model ID (e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/`
// and so isn't parseable as `provider/model`. these IDs only reach this
// function via routing aliases, so re-run the bedrock setup check rather
// than falling through to `getModelEnvVars` (which would throw inside
// parseModel). resolveModel itself already enforced BEDROCK_MODEL_ID,
// but auth + region are still validated here.
if (!params.model.includes("/")) {
validateBedrockSetup({ owner: params.owner, name: params.name });
return;
}
const requiredVars = getModelEnvVars(params.model);
// free models have no required env vars — skip validation entirely
if (requiredVars.length === 0) return;
@@ -59,3 +115,44 @@ export function validateAgentApiKey(params: {
throw new Error(buildMissingApiKeyError({ owner: params.owner, name: params.name }));
}
}
/**
* Detect agent-runtime auth failures that should be reformatted as an actionable
* key-fix CTA before being shown to the user. Covers the two shapes we see:
* - missing key (validateAgentApiKey throw): contains MISSING_KEY_MARKER
* - revoked / invalid key (Claude CLI 401 surfaced via api_error_status):
* "Invalid API key · Fix external API key" + similar provider variants
*/
export function isApiKeyAuthError(text: string): boolean {
if (!text) return false;
return (
text.includes(MISSING_KEY_MARKER) ||
/Invalid API key/i.test(text) ||
/\bUser not found\b/i.test(text) ||
/\bInvalid authentication\b/i.test(text)
);
}
/**
* Friendly Markdown summary for both the missing-key and invalid-key cases.
* Used in the catch / result-failure paths in `main.ts` to overwrite the raw
* agent error before it's posted to the PR progress comment.
*/
export function formatApiKeyErrorSummary(params: {
owner: string;
name: string;
raw: string;
}): string {
if (params.raw.includes(MISSING_KEY_MARKER)) {
return buildMissingApiKeyError({ owner: params.owner, name: params.name });
}
const githubSecretsUrl = `https://github.com/${params.owner}/${params.name}/settings/secrets/actions`;
const settingsUrl = `${getApiUrl()}/console/${params.owner}/${params.name}`;
return [
`**Your LLM provider API key was rejected (401).** Rotate the key in your provider dashboard, then update the matching GitHub Actions secret.`,
"",
`[Update repo secret →](${githubSecretsUrl}) · [Model settings →](${settingsUrl}) · [Setup docs →](https://docs.pullfrog.com/keys) · [Ask in Discord →](https://discord.gg/8y96raFg8e)`,
].join("\n");
}
+15
View File
@@ -25,3 +25,18 @@ export function getApiUrl(): string {
log.debug(`resolved API_URL: ${raw}`);
return raw;
}
/**
* true when the action is configured to talk to a localhost API server (i.e.
* `pnpm dev` running on the developer's box). signals we can use dev-only
* affordances like the `x-dev-repo` proxy-token bypass the corresponding
* server-side dev gates (`NODE_ENV === "development"`) ensure these paths
* never activate against prod regardless of what the action does.
*/
export function isLocalApiUrl(): boolean {
try {
return isLocalUrl(new URL(getApiUrl()));
} catch {
return false;
}
}
+1 -1
View File
@@ -1,7 +1,7 @@
import { execFileSync, spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import { dirname } from "node:path";
import type { ToolState } from "../mcp/server.ts";
import type { ToolState } from "../toolState.ts";
import { log } from "./cli.ts";
import { filterEnv } from "./secrets.ts";
import { getDevDependencyVersion } from "./version.ts";
+1 -1
View File
@@ -1,4 +1,4 @@
import type { ToolState } from "../mcp/server.ts";
import type { ToolState } from "../toolState.ts";
import { getApiUrl } from "./apiUrl.ts";
import { buildPullfrogFooter } from "./buildPullfrogFooter.ts";
import { createOctokit, parseRepoContext } from "./github.ts";
+54
View File
@@ -14,6 +14,7 @@ import { readFileSync, realpathSync, unlinkSync } from "node:fs";
import { log } from "./cli.ts";
import type { GitAuthServer } from "./gitAuthServer.ts";
import { filterEnv } from "./secrets.ts";
import { $ } from "./shell.ts";
import { spawn } from "./subprocess.ts";
type SafeGitSubcommand = "fetch" | "push";
@@ -181,3 +182,56 @@ export async function $git(
}
}
}
/**
* shallow-clone unreachable: when an existing local depth is too shallow for
* git to traverse to the requested ref's ancestry, the remote walk fails with
* one of these wordings (git emits the full OID via oid_to_hex, so the bound
* is 40 for SHA-1 or 64 for SHA-256). detecting both lets a single deepen
* retry recover before the error reaches the agent see issue #564 for the
* original `git_fetch` precedent and #656 for the `checkout_pr` follow-up.
*/
export const SHALLOW_UNREACHABLE_PATTERNS: RegExp[] = [
/Could not read [a-f0-9]{40,64}/,
/remote did not send all necessary objects/,
];
/**
* large enough to clear the merge base on most real-world PRs without
* downloading the full history; matches the fallback used by
* `checkoutPrBranch` when the GitHub compare API is unavailable.
*/
export const DEEPEN_RETRY_DEPTH = 1000;
/**
* authenticated `git fetch` that recovers from shallow-unreachable errors
* by retrying once with `--deepen=1000`. callers pass the same args they
* would to `$git("fetch", ...)`; on shallow-unreachable failures in a
* shallow repo, the second attempt prepends `--deepen=N` and strips any
* caller-supplied `--depth=` (the two flags are mutually exclusive, and
* the caller's depth is what got us into this mess).
*
* non-shallow-unreachable errors and non-shallow repos rethrow unchanged,
* so this is safe to wrap any fetch without changing fast-path behavior.
*/
export async function $gitFetchWithDeepen(
args: string[],
options: GitAuthOptions,
label?: string
): Promise<GitResult> {
try {
return await $git("fetch", args, options);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const isShallowUnreachable = SHALLOW_UNREACHABLE_PATTERNS.some((p) => p.test(msg));
if (!isShallowUnreachable) throw err;
const isShallow =
$("git", ["rev-parse", "--is-shallow-repository"], { log: false }).trim() === "true";
if (!isShallow) throw err;
log.info(
`» ${label ?? "git fetch"} hit shallow-unreachable error, retrying with --deepen=${DEEPEN_RETRY_DEPTH}`
);
const retryArgs = args.filter((a) => !a.startsWith("--depth="));
return await $git("fetch", [`--deepen=${DEEPEN_RETRY_DEPTH}`, ...retryArgs], options);
}
}
+50 -9
View File
@@ -99,6 +99,22 @@ type AcquireTokenOptions = {
permissions?: GitHubAppPermissions;
};
/**
* Thrown when our token-exchange endpoint returns a non-2xx response.
* The retry policy in `acquireNewToken` looks for this concrete type to
* skip retries 4xx is terminal user state (not-installed, not-authorized)
* and 5xx is rare enough that re-running the workflow is the right escape
* hatch. Genuine network failures throw plain `Error` and stay retryable.
*/
class TokenExchangeError extends Error {
readonly status: number;
constructor(status: number, message: string) {
super(message);
this.name = "TokenExchangeError";
this.status = status;
}
}
async function acquireTokenViaOIDC(opts?: AcquireTokenOptions): Promise<string> {
const oidcToken = await core.getIDToken("pullfrog-api");
@@ -128,7 +144,22 @@ async function acquireTokenViaOIDC(opts?: AcquireTokenOptions): Promise<string>
clearTimeout(timeoutId);
if (!tokenResponse.ok) {
throw new Error(`Token exchange failed: ${tokenResponse.status} ${tokenResponse.statusText}`);
// prefer the server-side `error` field — it's the single source of
// truth for the install URL (uses GITHUB_APP_INSTALL_URL, which
// varies per env / GITHUB_APP_SLUG). fall back to a generic message
// if the body isn't JSON or doesn't carry an `error` field.
let serverMessage: string | undefined;
try {
const body = (await tokenResponse.json()) as { error?: unknown };
if (typeof body.error === "string") serverMessage = body.error;
} catch {
// body wasn't JSON — fall through to the generic message
}
throw new TokenExchangeError(
tokenResponse.status,
serverMessage ??
`Token exchange failed: ${tokenResponse.status} ${tokenResponse.statusText}`
);
}
const tokenData = (await tokenResponse.json()) as InstallationToken;
@@ -221,8 +252,11 @@ const checkRepositoryAccess = async (
headers: { Authorization: `token ${token}` },
});
const ownerLower = repoOwner.toLowerCase();
const nameLower = repoName.toLowerCase();
return response.repositories.some(
(repo) => repo.owner.login === repoOwner && repo.name === repoName
(repo) =>
repo.owner.login.toLowerCase() === ownerLower && repo.name.toLowerCase() === nameLower
);
} catch {
return false;
@@ -330,13 +364,20 @@ export async function acquireNewToken(opts?: AcquireTokenOptions): Promise<strin
if (isOIDCAvailable()) {
return await retry(() => acquireTokenViaOIDC(opts), {
label: "token exchange",
shouldRetry: (error) =>
error instanceof Error &&
(error.name === "AbortError" ||
error.message.includes("fetch failed") ||
error.message.includes("ECONNRESET") ||
error.message.includes("ETIMEDOUT") ||
error.message.includes("Token exchange failed")),
shouldRetry: (error) => {
// 4xx is terminal user state (app not installed, permissions wrong) —
// retrying just triples our log noise and the user's CI bill (see
// #693). 5xx/429 are transient (vercel cold start, github outage,
// rate limit) and should ride the existing backoff.
if (error instanceof TokenExchangeError) return error.status >= 500 || error.status === 429;
return (
error instanceof Error &&
(error.message.includes("timed out") ||
error.message.includes("fetch failed") ||
error.message.includes("ECONNRESET") ||
error.message.includes("ETIMEDOUT"))
);
},
});
} else {
// local development via GitHub App
+86 -12
View File
@@ -4,6 +4,7 @@ import { encode as toonEncode } from "@toon-format/toon";
import { type AgentId, formatMcpToolRef, type PayloadEvent, pullfrogMcpName } from "../external.ts";
import type { Mode } from "../modes.ts";
import type { ResolvedPayload } from "./payload.ts";
import type { LearningsHeading } from "./runContext.ts";
import type { RunContextData } from "./runContextData.ts";
interface InstructionsContext {
@@ -12,7 +13,14 @@ interface InstructionsContext {
modes: Mode[];
agentId: AgentId;
outputSchema?: Record<string, unknown> | undefined;
learnings: string | null;
/** absolute path to the seeded learnings tmpfile, or null when the file
* couldn't be seeded for some reason. main.ts always seeds, so in
* practice this is always set; the null case keeps the type honest. */
learningsFilePath: string | null;
/** server-parsed TOC for the body of the learnings tmpfile. rendered
* inline into the LEARNINGS prompt section so the agent can `read_file`
* targeted line ranges instead of pulling the whole file into context. */
learningsHeadings: LearningsHeading[];
}
interface PromptContext extends InstructionsContext {
@@ -29,6 +37,7 @@ function buildRuntimeContext(ctx: InstructionsContext): string {
"~pullfrog": _,
prompt: _p,
eventInstructions: _ei,
previousRunsNote: _prn,
event: _e,
...payloadRest
} = ctx.payload;
@@ -143,17 +152,23 @@ In case of conflict between instructions, follow this precedence (highest to low
// section builders
// ---------------------------------------------------------------------------
// the user's task: blockquoted user prompt, or event-level instructions for auto-triggers
// the user's task: blockquoted user prompt, or event-level instructions for auto-triggers.
// `previousRunsNote` is system-injected context (e.g. prior runs superseded by a
// comment edit); it's appended regardless of which branch wins so it survives
// user-prompt precedence over eventInstructions.
function buildTaskSection(ctx: PromptContext): string {
const previousRunsNote = ctx.payload.previousRunsNote?.trim() ?? "";
if (ctx.userQuoted) {
const parts = [ctx.userQuoted, previousRunsNote].filter(Boolean);
return `************* YOUR TASK *************
${ctx.userQuoted}`;
${parts.join("\n\n")}`;
}
const eventInstructions = ctx.payload.eventInstructions ?? "";
if (eventInstructions) {
const parts = [ctx.eventTitle, eventInstructions].filter(Boolean);
if (eventInstructions || previousRunsNote) {
const parts = [ctx.eventTitle, eventInstructions, previousRunsNote].filter(Boolean);
return `************* YOUR TASK *************
${parts.join("\n\n")}`;
@@ -273,6 +288,21 @@ ${getStandaloneModeInstructions(ctx.payload.event.trigger, t, ctx.outputSchema)}
Trust the tools do not repeatedly verify file contents or git status after operations. If a tool reports success, proceed to the next step. Only verify if you encounter an actual error. Exception: right before \`${t("push_branch")}\`, ensure the working tree is clean — that tool rejects dirty trees, and tests you ran earlier often leave untracked output.
### Parallel tool execution
For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously in a single assistant turn rather than sequentially. The dominant failure mode is grep read read read read across separate turns when one round trip would do. Always parallelize when calls are independent:
- reading multiple files (especially after a grep returns candidates)
- multiple greps with different patterns
- glob + grep + read combos
- listing multiple directories
- inspecting multiple MCP tools or resources
Do NOT parallelize operations that depend on prior output (e.g. create a file then read it), or ordered stateful mutations. Edits are not parallelizable sequence those normally.${
ctx.agentId === "opencode"
? `\n\nOn OpenCode you also have a \`batch\` tool that bundles 1-25 independent calls into one wrapper call. Reach for it whenever you have >=2 independent calls. Native parallel tool_use and \`batch\` both achieve one round trip instead of N — use whichever your provider supports best.`
: `\n\nEmit multiple \`tool_use\` blocks in the same assistant message for independent calls — the runtime executes them concurrently. Do not wait for one tool result before issuing the next independent call.`
}
### Command execution
Never use \`sleep\` to wait for commands to complete. Commands run synchronously — when the shell tool returns, the command has finished.
@@ -344,18 +374,58 @@ export interface ResolvedInstructions {
runtime: string;
}
/** render the heading list as an indented bullet TOC. ranges shown in
* parentheses (`(L3-L18)`); the start line is always the heading line
* itself, so reading the listed range gives the agent the heading +
* body together. shallowest heading depth in the body sits at the root
* column; deeper levels indent by `(depth - rootDepth) * 2` spaces. */
export function renderLearningsToc(headings: LearningsHeading[]): string {
if (headings.length === 0) return "";
const rootDepth = Math.min(...headings.map((h) => h.depth));
return headings
.map((h) => {
const indent = " ".repeat((h.depth - rootDepth) * 2);
return `${indent}- ${h.title} (L${h.startLine}-L${h.endLine})`;
})
.join("\n");
}
/** assemble the LEARNINGS prompt section: file path + intro + either
* the rendered heading TOC (when the body has structure) or a no-headings
* affordance pointing the agent at the reflection turn for restructuring.
* empty string when the seed step failed and there's no path to surface. */
export function buildLearningsSection(ctx: {
filePath: string | null;
headings: LearningsHeading[];
}): string {
if (!ctx.filePath) return "";
const intro = `Repo-level learnings accumulated by previous agent runs live at \`${ctx.filePath}\`. Use this file as durable context (test commands, conventions, gotchas, architecture notes).`;
const tocBody =
ctx.headings.length === 0
? "(no headings yet — file is empty or a flat list. read the whole file. during the post-run reflection turn, structure it with `## ` / `### ` headings so future runs can read targeted ranges.)"
: `Read targeted line ranges via your native file tool — do NOT slurp the whole file. Each range starts at the section heading line, so reading the range gives you heading + body together.\n\n${renderLearningsToc(ctx.headings)}`;
return `************* LEARNINGS *************\n\n${intro}\n\n${tocBody}`;
}
function assembleFullPrompt(ctx: {
toc: string;
task: string;
procedure: string;
eventContext: string;
system: string;
learnings: string | null;
learningsFilePath: string | null;
learningsHeadings: LearningsHeading[];
runtime: string;
}): string {
const learningsSection = ctx.learnings
? `************* LEARNINGS *************\n\n${ctx.learnings}`
: "";
// server-parsed TOC is rendered inline so the agent can target line
// ranges via its native file tool. the file body itself is never
// inlined — that would re-inflate context every run and clutter CI
// logs. post-run reflection (action/agents/postRun.ts) is where
// editing is encouraged.
const learningsSection = buildLearningsSection({
filePath: ctx.learningsFilePath,
headings: ctx.learningsHeadings,
});
const runtimeSection = `************* RUNTIME *************\n\n${ctx.runtime}`;
@@ -389,8 +459,11 @@ export function resolveInstructions(ctx: InstructionsContext): ResolvedInstructi
if (eventContext)
tocEntries.push({ label: "EVENT CONTEXT", description: "related PR/issue data" });
tocEntries.push({ label: "SYSTEM", description: "persona, security, tools, workflow rules" });
if (pctx.learnings)
tocEntries.push({ label: "LEARNINGS", description: "repo-specific knowledge" });
if (pctx.learningsFilePath)
tocEntries.push({
label: "LEARNINGS",
description: "repo-specific knowledge file path + heading TOC",
});
tocEntries.push({ label: "RUNTIME", description: "environment metadata" });
const toc = buildToc(tocEntries);
@@ -401,7 +474,8 @@ export function resolveInstructions(ctx: InstructionsContext): ResolvedInstructi
procedure,
eventContext,
system,
learnings: pctx.learnings,
learningsFilePath: pctx.learningsFilePath,
learningsHeadings: pctx.learningsHeadings,
runtime: pctx.runtime,
});
+87
View File
@@ -0,0 +1,87 @@
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import {
LEARNINGS_FILE_NAME,
learningsFilePath,
readLearningsFile,
seedLearningsFile,
} from "./learnings.ts";
describe("learnings tmpfile round-trip", () => {
let dir: string;
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), "pullfrog-learnings-test-"));
});
afterEach(async () => {
await rm(dir, { recursive: true, force: true });
});
it("writes the verbatim DB body to disk and reads it back unchanged", async () => {
const current = [
"## Build & test",
"",
"- run tests with `pnpm -r test`",
"",
"## Architecture",
"",
"- workers in `worker/`",
].join("\n");
const path = await seedLearningsFile({ tmpdir: dir, current });
expect(path).toBe(learningsFilePath(dir));
expect(path.endsWith(LEARNINGS_FILE_NAME)).toBe(true);
expect(await readFile(path, "utf8")).toBe(current);
expect(await readLearningsFile(path)).toBe(current);
});
it("seeds an empty file when the repo has no learnings yet, round-trip is empty string", async () => {
const path = await seedLearningsFile({ tmpdir: dir, current: null });
expect(await readFile(path, "utf8")).toBe("");
expect(await readLearningsFile(path)).toBe("");
});
it("returns null when the file is missing (treated as no-change by persist)", async () => {
expect(await readLearningsFile(learningsFilePath(dir))).toBeNull();
});
it("trims trailing whitespace so editor newlines never trigger a spurious PATCH", async () => {
const current = "## Build & test\n\n- one fact";
const path = await seedLearningsFile({ tmpdir: dir, current });
await writeFile(path, `${current}\n\n `, "utf8");
expect(await readLearningsFile(path)).toBe(current);
});
it("truncates over-cap bodies at the last newline boundary so the next-seed TOC parse stays clean", async () => {
const padding = `${"x".repeat(80)}\n`.repeat(1300);
const oversized = `## Build & test\n\n${padding}`;
const path = await seedLearningsFile({ tmpdir: dir, current: null });
await writeFile(path, oversized, "utf8");
const read = await readLearningsFile(path);
expect(read).toBeTruthy();
expect(read?.length).toBeLessThanOrEqual(100_000);
const tailLine = read?.split("\n").pop() ?? "";
expect(/^x+$/.test(tailLine)).toBe(true);
expect(tailLine.length).toBe(80);
});
it("falls back to a hard truncate when the only newline is far above the cap (giant single line)", async () => {
const oversized = `## Build & test\n${"x".repeat(110_000)}`;
const path = await seedLearningsFile({ tmpdir: dir, current: null });
await writeFile(path, oversized, "utf8");
const read = await readLearningsFile(path);
expect(read).toBeTruthy();
expect(read?.length).toBe(100_000);
expect(read?.startsWith("## Build & test\n")).toBe(true);
});
it("preserves legacy free-text without scaffolding or wrapping", async () => {
const legacy = "- this is some old free-text bullet\n- another one";
const path = await seedLearningsFile({ tmpdir: dir, current: legacy });
expect(await readFile(path, "utf8")).toBe(legacy);
expect(await readLearningsFile(path)).toBe(legacy);
});
});
+90
View File
@@ -0,0 +1,90 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
/**
* Repo-level learnings operational facts about a repo (setup steps, test
* commands, conventions, gotchas) that accumulate across agent runs and feed
* back into future runs as durable context. Modeled on the PR-summary tmpfile
* pattern (see action/utils/prSummary.ts):
*
* 1. server seeds `pullfrog-learnings.md` with the verbatim body of
* `Repo.learnings` (or empty for fresh repos), and parses headings
* server-side (`utils/learningsToc.ts`) the parsed TOC is rendered
* into the LEARNINGS prompt section, not into the file
* 2. the agent reads the TOC in the prompt and uses listed line ranges
* to read just the sections relevant to the current task file can
* grow large, but only targeted ranges hit the agent's context
* 3. agent edits the file in place at end-of-run during the reflection
* turn (see action/agents/postRun.ts buildLearningsReflectionPrompt)
* 4. main.ts reads the file back at end-of-run and PATCHes
* `/api/repo/[owner]/[repo]/learnings` if the body changed
*
* Edit-in-place avoids stuffing the entire learnings list into both the
* prompt context and an `update_learnings` MCP tool call (which previously
* required passing the FULL merged list as a string parameter an
* output-token tax that grew linearly with the learnings size).
*
* Section structure is agent-curated. The reflection prompt teaches
* hierarchy + a soft 300-line-per-section cap to keep TOC ranges
* agent-targetable on long-lived repos; there is no fixed taxonomy.
*/
export const LEARNINGS_FILE_NAME = "pullfrog-learnings.md";
/** server-side cap mirrors `MAX_LEARNINGS_LENGTH` in
* `app/api/repo/[owner]/[repo]/learnings/route.ts`. truncating client-side
* keeps the PATCH from being rejected with a 400. raised from 10k 100k
* once the TOC affordance landed: with line-range reads via the
* server-parsed TOC the agent doesn't ingest the whole file, so the cap
* can grow to whatever curation discipline allows. 100k holds ~400-500
* short bullets. */
const MAX_LEARNINGS_LENGTH = 100_000;
export function learningsFilePath(tmpdir: string): string {
return join(tmpdir, LEARNINGS_FILE_NAME);
}
/** seed the rolling learnings tmpfile with the verbatim DB body (or empty
* string for fresh repos). returns the absolute path. the parsed TOC is
* carried separately via `RepoSettings.learningsHeadings` and rendered
* into the prompt by `resolveInstructions`, so the file on disk is just
* the body no markers, no scaffold, no in-file TOC. */
export async function seedLearningsFile(params: {
tmpdir: string;
current: string | null;
}): Promise<string> {
const path = learningsFilePath(params.tmpdir);
await mkdir(dirname(path), { recursive: true });
await writeFile(path, params.current ?? "", "utf8");
return path;
}
/** truncate at the last newline boundary before `cap` so we don't leave
* a partial line at the tail (a half-truncated `## Headi` confuses the
* server's next-seed TOC parse and shrinks visible structure). falls
* back to a hard `slice` when the line boundary would discard a large
* run of content i.e. when the tail of `head` is one giant line (rare:
* minified pastes, fenced log dumps). losing a partial last line is
* preferable to losing kilobytes of body. */
const TRUNCATION_LINE_BOUNDARY_TOLERANCE = 4096;
function truncateAtLineBoundary(body: string, cap: number): string {
if (body.length <= cap) return body;
const head = body.slice(0, cap);
const lastNewline = head.lastIndexOf("\n");
if (lastNewline <= 0) return head;
if (cap - lastNewline > TRUNCATION_LINE_BOUNDARY_TOLERANCE) return head;
return head.slice(0, lastNewline);
}
/** read the agent-edited learnings file. returns null when the file is
* missing or unreadable (treated as "no change"). caps content at the
* server's max length to avoid a 400 round-trip. */
export async function readLearningsFile(path: string): Promise<string | null> {
let raw: string;
try {
raw = await readFile(path, "utf8");
} catch {
return null;
}
return truncateAtLineBoundary(raw.trim(), MAX_LEARNINGS_LENGTH);
}
+99
View File
@@ -0,0 +1,99 @@
import { describe, expect, it } from "vitest";
import { buildLearningsSection, renderLearningsToc } from "./instructions.ts";
import type { LearningsHeading } from "./runContext.ts";
const h = (depth: 1 | 2 | 3 | 4 | 5 | 6, title: string, startLine: number, endLine: number) => ({
depth,
title,
startLine,
endLine,
});
describe("renderLearningsToc", () => {
it("renders flat h2 list with parenthesized ranges, no hashes or backticks", () => {
const headings: LearningsHeading[] = [
h(2, "Build & test", 1, 18),
h(2, "Architecture", 19, 60),
];
expect(renderLearningsToc(headings)).toBe(
`- Build & test (L1-L18)
- Architecture (L19-L60)`
);
});
it("indents deeper headings 2 spaces per depth level past the shallowest", () => {
const headings: LearningsHeading[] = [
h(2, "Build & test", 1, 42),
h(3, "Local", 3, 18),
h(3, "CI", 19, 42),
h(2, "Architecture", 43, 210),
h(3, "Background workers", 80, 210),
];
expect(renderLearningsToc(headings)).toBe(
`- Build & test (L1-L42)
- Local (L3-L18)
- CI (L19-L42)
- Architecture (L43-L210)
- Background workers (L80-L210)`
);
});
it("treats the shallowest depth as the root column when no h2 is present", () => {
const headings: LearningsHeading[] = [h(3, "Only h3", 1, 5), h(4, "Sub h4", 2, 5)];
expect(renderLearningsToc(headings)).toBe(
`- Only h3 (L1-L5)
- Sub h4 (L2-L5)`
);
});
it("supports depths up through h6 with stable 2-space indent steps", () => {
const headings: LearningsHeading[] = [
h(2, "Two", 1, 10),
h(3, "Three", 2, 10),
h(4, "Four", 3, 10),
h(5, "Five", 4, 10),
h(6, "Six", 5, 10),
];
expect(renderLearningsToc(headings)).toBe(
`- Two (L1-L10)
- Three (L2-L10)
- Four (L3-L10)
- Five (L4-L10)
- Six (L5-L10)`
);
});
});
describe("buildLearningsSection", () => {
it("returns empty string when no file path (seed step failed)", () => {
expect(buildLearningsSection({ filePath: null, headings: [] })).toBe("");
});
it("renders the no-headings affordance when the body has no structure", () => {
const out = buildLearningsSection({
filePath: "/tmp/run-1/pullfrog-learnings.md",
headings: [],
});
expect(out).toContain("************* LEARNINGS *************");
expect(out).toContain("/tmp/run-1/pullfrog-learnings.md");
expect(out).toContain("no headings yet");
expect(out).toContain("structure it with");
// does not include a TOC list when there are no headings
expect(out).not.toMatch(/\(L\d+-L\d+\)/);
});
it("renders the TOC inline with the file path and heading guidance", () => {
const out = buildLearningsSection({
filePath: "/tmp/run-1/pullfrog-learnings.md",
headings: [h(2, "Build & test", 1, 18), h(2, "Architecture", 19, 60)],
});
expect(out).toContain("************* LEARNINGS *************");
expect(out).toContain("/tmp/run-1/pullfrog-learnings.md");
expect(out).toContain("- Build & test (L1-L18)");
expect(out).toContain("- Architecture (L19-L60)");
expect(out).toContain("Each range starts at the section heading line");
// explicit "no hashes, no backticks" in the rendered list
expect(out).not.toContain("- `## Build");
expect(out).not.toContain("`## Build");
});
});
+91
View File
@@ -0,0 +1,91 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { normalizeEnv, sanitizeSecret } from "./normalizeEnv.ts";
/**
* These tests pin the load-bearing invariants of secret sanitisation:
* - sensitive values are trimmed before downstream code reads them
* - whitespace-only values are NOT silently zeroed (leave env unchanged)
* - case normalisation still happens
*
* Masking (`core.setSecret`) is delegated to `@actions/core` and trusted to
* work as documented we don't spy on stdout to re-test the toolkit.
*/
describe("normalizeEnv: process.env state contract", () => {
let originalEnv: NodeJS.ProcessEnv;
beforeEach(() => {
// normalizeEnv() iterates the entire process.env, so the test must
// control it. snapshot + full wipe + restore is the cleanest isolation.
originalEnv = { ...process.env };
for (const k of Object.keys(process.env)) delete process.env[k];
});
afterEach(() => {
for (const k of Object.keys(process.env)) delete process.env[k];
Object.assign(process.env, originalEnv);
});
it("trims trailing newline from sensitive env vars", () => {
process.env.ANTHROPIC_API_KEY = "sk-ant-secret-value\n";
normalizeEnv();
expect(process.env.ANTHROPIC_API_KEY).toBe("sk-ant-secret-value");
});
it("trims surrounding whitespace including \\r\\n and spaces", () => {
process.env.OPENAI_API_KEY = " sk-openai-value\r\n ";
normalizeEnv();
expect(process.env.OPENAI_API_KEY).toBe("sk-openai-value");
});
it("leaves clean sensitive values untouched", () => {
process.env.ANTHROPIC_API_KEY = "sk-ant-clean";
normalizeEnv();
expect(process.env.ANTHROPIC_API_KEY).toBe("sk-ant-clean");
});
it("ignores non-sensitive env vars", () => {
process.env.NODE_ENV = "production\n";
normalizeEnv();
expect(process.env.NODE_ENV).toBe("production\n");
});
it("canonicalises case and trims the value", () => {
process.env.anthropic_api_key = "sk-ant-lowercase\n";
normalizeEnv();
expect(process.env.ANTHROPIC_API_KEY).toBe("sk-ant-lowercase");
expect(process.env.anthropic_api_key).toBeUndefined();
});
it("preserves whitespace-only values rather than silently zeroing them", () => {
// contract: don't mutate when value is whitespace-only. caller sees the
// misconfigured value verbatim and either fails clearly downstream or
// logs a missing-key error.
process.env.ANTHROPIC_API_KEY = " \n ";
normalizeEnv();
expect(process.env.ANTHROPIC_API_KEY).toBe(" \n ");
});
it("preserves embedded newlines (toolkit masks each line)", () => {
// multi-line PEMs aren't used in practice, but if one slipped in via a
// DB secret we don't want to silently mutate it. trim() only touches
// the ends; @actions/core handles per-line masking via the runner.
process.env.ANTHROPIC_API_KEY = "line1\nline2";
normalizeEnv();
expect(process.env.ANTHROPIC_API_KEY).toBe("line1\nline2");
});
});
describe("sanitizeSecret return value", () => {
it("returns the trimmed value for a sensitive secret with trailing newline", () => {
expect(sanitizeSecret("ANTHROPIC_API_KEY", "sk-ant-secret\n")).toBe("sk-ant-secret");
});
it("returns the value unchanged when no trimming is needed", () => {
expect(sanitizeSecret("ANTHROPIC_API_KEY", "sk-ant-clean")).toBe("sk-ant-clean");
});
it("returns null for whitespace-only input so caller can skip injection", () => {
expect(sanitizeSecret("ANTHROPIC_API_KEY", " \n")).toBeNull();
});
});
+45 -13
View File
@@ -1,11 +1,40 @@
import * as core from "@actions/core";
import { log } from "./cli.ts";
import { isSensitiveEnvName } from "./secrets.ts";
function maskValue(value: string | undefined) {
if (value && typeof value === "string" && value.trim().length > 0) {
// ::add-mask::value tells GitHub Actions to mask this value in logs
console.log(`::add-mask::${value}`);
/**
* Trim surrounding whitespace from a sensitive value and register it as a
* GitHub Actions log mask. Trailing newlines from terminal-copy paste are a
* common footgun: the value travels through GH Actions logs and any tool
* that re-emits parts of it leaks the unmasked tail. Trimming canonicalises
* the value so the mask matches exactly what downstream tools will print.
*
* Masking is delegated to `core.setSecret` (not raw `console.log`) so the
* toolkit percent-encodes `\r`/`\n`; the runner V2 parser decodes them and
* registers the full value plus every non-empty line as separate masks. That
* keeps us safe for embedded-newline values (PEMs, kubeconfigs, JSON blobs)
* even though they aren't currently used.
*
* Returns the trimmed value, or `null` when the input was whitespace-only
* callers must leave `process.env` untouched in that case so a misconfigured
* value surfaces as a clear "missing key" downstream rather than silently
* mutating to the empty string.
*/
export function sanitizeSecret(key: string, value: string): string | null {
const trimmed = value.trim();
if (trimmed.length === 0) {
log.warning(
`» ${key} is whitespace-only — leaving env var unchanged. check your secret value.`
);
return null;
}
if (trimmed !== value) {
log.warning(
`» stripped whitespace from ${key} (whitespace in secret values breaks GitHub Actions log masking)`
);
}
core.setSecret(trimmed);
return trimmed;
}
/**
@@ -15,7 +44,8 @@ function maskValue(value: string | undefined) {
* If there are conflicts (same key with different capitalizations but different values),
* logs a warning and keeps the uppercase version.
*
* Also registers sensitive values as masks in GitHub Actions.
* Also trims and masks sensitive values so accidental trailing whitespace
* doesn't defeat GitHub Actions log masking.
*/
export function normalizeEnv(): void {
const upperKeys = new Map<string, string[]>();
@@ -30,14 +60,6 @@ export function normalizeEnv(): void {
// process each group
for (const [upperKey, keys] of upperKeys) {
// if sensitive, ensure we mask the value (regardless of whether we rename it or not)
if (isSensitiveEnvName(upperKey)) {
// mask all values associated with this key group
for (const key of keys) {
maskValue(process.env[key]);
}
}
if (keys.length === 1) {
const key = keys[0];
if (key !== upperKey) {
@@ -71,4 +93,14 @@ export function normalizeEnv(): void {
// set the uppercase version
process.env[upperKey] = preferredValue;
}
// trim + mask sensitive values after case normalisation so each key is
// visited exactly once with its final, canonical value
for (const key of Object.keys(process.env)) {
if (!isSensitiveEnvName(key)) continue;
const value = process.env[key];
if (typeof value !== "string" || value.length === 0) continue;
const sanitized = sanitizeSecret(key, value);
if (sanitized !== null) process.env[key] = sanitized;
}
}
+2
View File
@@ -21,6 +21,7 @@ export const JsonPayload = type({
"triggerer?": "string | undefined",
"eventInstructions?": "string",
"previousRunsNote?": "string",
"event?": "object",
"timeout?": "string | undefined",
"progressComment?": type({
@@ -157,6 +158,7 @@ export function resolvePayload(
// it's not a common use case but GITHUB_ACTOR can be a user when the workflow is manually triggered by a user through GitHub Actions UI
(!isPullfrog(process.env.GITHUB_ACTOR) ? process.env.GITHUB_ACTOR : undefined),
eventInstructions: jsonPayload?.eventInstructions,
previousRunsNote: jsonPayload?.previousRunsNote,
event,
timeout: inputs.timeout ?? jsonPayload?.timeout,
cwd: resolveCwd(inputs.cwd),
+102 -3
View File
@@ -1,4 +1,8 @@
import { detectProviderError, isRouterKeylimitExhaustedError } from "./providerErrors.ts";
import {
detectProviderError,
findProviderErrorMatch,
isRouterKeylimitExhaustedError,
} from "./providerErrors.ts";
describe("detectProviderError", () => {
describe("false positives previously seen in production", () => {
@@ -7,7 +11,13 @@ describe("detectProviderError", () => {
expect(detectProviderError("commit f609cc89e84596ab125d60dac568bfb2ef398396 429")).toBeNull();
});
it("returns null for x-ratelimit-* response headers in 401 error JSON", () => {
it("classifies 401 + x-ratelimit-* headers as auth, not rate-limited", () => {
// OpenRouter 401 responses bundle `x-ratelimit-*` rate-limit headers
// alongside the auth error. the auth patterns must win — pre-fix this
// got tagged as `rate limited` because of the loose `\brate[_ ]limit`
// match against header names like `ratelimit-limit-requests`. note: in
// OpenRouter's actual format the header name is `ratelimit` (one word),
// but the dumped JSON sometimes contains `rate-limit` separators too.
const stderr = JSON.stringify({
error: { name: "APIError", statusCode: 401, message: "Invalid authentication credentials" },
headers: {
@@ -16,7 +26,7 @@ describe("detectProviderError", () => {
"x-ratelimit-reset-tokens": "2025-01-01T00:00:00Z",
},
});
expect(detectProviderError(stderr)).toBeNull();
expect(detectProviderError(stderr)).toBe("auth error (401)");
});
it("returns null for INTERNAL_SERVER_ERROR substring", () => {
@@ -29,6 +39,37 @@ describe("detectProviderError", () => {
});
});
describe("auth errors", () => {
it("detects 401 / 403 status codes as auth errors", () => {
expect(detectProviderError('{"statusCode": 401}')).toBe("auth error (401)");
expect(detectProviderError('{"statusCode": 403}')).toBe("auth error (403)");
expect(detectProviderError("status_code: 401")).toBe("auth error (401)");
});
it("detects OpenRouter 'User not found' (disabled/invalid key)", () => {
// bare `"code":401` lacks a status-key prefix so the 401 status pattern
// intentionally doesn't fire; the User-not-found pattern catches it.
expect(detectProviderError('{"error":{"message":"User not found","code":401}}')).toBe(
"auth error (invalid/disabled key)"
);
expect(detectProviderError("APIError: User not found.")).toBe(
"auth error (invalid/disabled key)"
);
});
it("detects 'Invalid authentication' phrasing", () => {
expect(detectProviderError("Invalid authentication credentials")).toBe(
"auth error (invalid credentials)"
);
});
it("detects 'No auth credentials found' phrasing", () => {
expect(detectProviderError("AI_APICallError: No auth credentials found")).toBe(
"auth error (missing credentials)"
);
});
});
describe("real provider errors", () => {
it("detects 429 only when adjacent to a status key", () => {
expect(detectProviderError('{"statusCode": 429}')).toBe("rate limited (429)");
@@ -79,6 +120,64 @@ describe("detectProviderError", () => {
});
});
describe("findProviderErrorMatch", () => {
// regression for issue #703: when stderr arrives as a multi-KB buffer
// (mcp tool-schema dump + the actual error message), the old
// `chunk.substring(0, 500)` excerpt showed the head of the buffer
// (schema) instead of the matched error text. the windowed excerpt
// must center on the matched line.
it("excerpt centers on the matched line, not the head of the buffer", () => {
const schemaDump =
"{".repeat(2000) +
'"name":"pullfrog_create_pull_request_review","description":"Submit a review..."';
const errorLine = "ERROR 2026-05-13 service=session error=rate_limit_exceeded retry-after=30";
const chunk = `${schemaDump}\n${errorLine}\ncaller stack at handler.ts:42`;
const match = findProviderErrorMatch(chunk);
expect(match).not.toBeNull();
expect(match?.label).toBe("rate limited");
expect(match?.excerpt).toContain("rate_limit_exceeded");
expect(match?.excerpt).toContain("retry-after=30");
expect(match?.excerpt).not.toContain("pullfrog_create_pull_request_review");
});
it("includes a small surrounding-line window for stack-trace context", () => {
const chunk =
"» about to call session.processor\n" +
"ERROR rate_limit_exceeded for key=abc\n" +
"at handler.ts:42\n" +
"at runtime.ts:88";
const match = findProviderErrorMatch(chunk);
expect(match?.excerpt).toContain("about to call session.processor");
expect(match?.excerpt).toContain("rate_limit_exceeded");
expect(match?.excerpt).toContain("handler.ts:42");
expect(match?.excerpt).toContain("runtime.ts:88");
});
it("falls back to the matched line alone when adjacent lines are huge", () => {
const giantPrefix = "x".repeat(5000);
const errorLine = '"statusCode": 429, "message": "slow down"';
const giantSuffix = "y".repeat(5000);
const chunk = `${giantPrefix}\n${errorLine}\n${giantSuffix}`;
const match = findProviderErrorMatch(chunk);
expect(match?.label).toBe("rate limited (429)");
expect(match?.excerpt).toBe(errorLine);
});
it("head-truncates the matched line if it alone exceeds the byte cap", () => {
const padding = "z".repeat(700);
const chunk = `${padding} "statusCode": 429 ${padding}`;
const match = findProviderErrorMatch(chunk);
expect(match?.label).toBe("rate limited (429)");
expect(match?.excerpt.length).toBeLessThanOrEqual(600);
});
it("returns null when no pattern matches", () => {
expect(findProviderErrorMatch("just some normal log line\nnothing wrong here")).toBeNull();
});
});
describe("isRouterKeylimitExhaustedError", () => {
it("matches the canonical OpenRouter mid-run error", () => {
expect(
+71 -2
View File
@@ -6,6 +6,17 @@ type ProviderErrorPattern = { regex: RegExp; label: string };
const statusKey = `\\b(?:status[_ ]?code|http[_ ]?status|status)["']?\\s*[:=]\\s*["']?`;
const PROVIDER_ERROR_PATTERNS: ProviderErrorPattern[] = [
// auth patterns must come BEFORE rate-limit patterns. OpenRouter 401 error
// payloads carry `x-ratelimit-*` response headers in the dump, and the
// free-form rate-limit regex below would otherwise win on word-boundary
// matches inside header names. canonical 401 messages: OpenRouter returns
// `{"error":{"message":"User not found","code":401}}` for disabled or
// invalid keys (https://openai.luzhipeng.com/docs/api/reference/errors-and-debugging).
{ regex: new RegExp(`${statusKey}401\\b`, "i"), label: "auth error (401)" },
{ regex: new RegExp(`${statusKey}403\\b`, "i"), label: "auth error (403)" },
{ regex: /\bUser not found\b/i, label: "auth error (invalid/disabled key)" },
{ regex: /\bInvalid authentication\b/i, label: "auth error (invalid credentials)" },
{ regex: /\bNo auth credentials found\b/i, label: "auth error (missing credentials)" },
{ regex: new RegExp(`${statusKey}429\\b`, "i"), label: "rate limited (429)" },
{ regex: new RegExp(`${statusKey}500\\b`, "i"), label: "provider 500 error" },
{ regex: new RegExp(`${statusKey}503\\b`, "i"), label: "provider unavailable (503)" },
@@ -30,13 +41,71 @@ const PROVIDER_ERROR_PATTERNS: ProviderErrorPattern[] = [
{ regex: /["']?\blimit\b["']?\s*:\s*0\b/, label: "zero quota" },
];
export function detectProviderError(text: string): string | null {
/**
* Result of a provider-error scan: the classification label plus a
* human-readable excerpt centered on the matched line. The excerpt is what
* gets surfaced in `» provider error detected (...)` log lines see
* `extractExcerpt` for the windowing/byte-cap policy.
*/
export type ProviderErrorMatch = {
label: string;
excerpt: string;
};
// roughly half a wide terminal line by 45 lines of context; large enough
// to capture a structured error payload (request id, retry-after, model)
// plus its immediate stack/headers, small enough to not flood the log.
const EXCERPT_MAX_BYTES = 600;
const LINES_BEFORE = 1;
const LINES_AFTER = 2;
export function findProviderErrorMatch(text: string): ProviderErrorMatch | null {
for (const entry of PROVIDER_ERROR_PATTERNS) {
if (entry.regex.test(text)) return entry.label;
const m = entry.regex.exec(text);
if (!m) continue;
return { label: entry.label, excerpt: extractExcerpt(text, m.index) };
}
return null;
}
export function detectProviderError(text: string): string | null {
return findProviderErrorMatch(text)?.label ?? null;
}
/**
* Slice a context window around `matchIndex`: the matched line plus
* `LINES_BEFORE`/`LINES_AFTER` neighbours. If the windowed slice exceeds
* `EXCERPT_MAX_BYTES` (giant adjacent lines, e.g. JSON tool-schema dumps),
* fall back to the matched line alone, head-truncated if still too long.
* Replaces the old `chunk.substring(0, 500)` head-anchored excerpt which
* surfaced whatever happened to be at the front of the stderr buffer
* instead of the error itself. See issue #703.
*/
function extractExcerpt(text: string, matchIndex: number): string {
const lineStart = text.lastIndexOf("\n", matchIndex - 1) + 1;
const lineEndRaw = text.indexOf("\n", matchIndex);
const lineEnd = lineEndRaw === -1 ? text.length : lineEndRaw;
let start = lineStart;
for (let i = 0; i < LINES_BEFORE && start > 0; i++) {
const prev = text.lastIndexOf("\n", start - 2);
start = prev < 0 ? 0 : prev + 1;
}
let end = lineEnd;
for (let i = 0; i < LINES_AFTER && end < text.length; i++) {
const next = text.indexOf("\n", end + 1);
end = next < 0 ? text.length : next;
}
let excerpt = text.slice(start, end);
if (excerpt.length > EXCERPT_MAX_BYTES) {
excerpt = text.slice(lineStart, lineEnd);
if (excerpt.length > EXCERPT_MAX_BYTES) excerpt = excerpt.slice(0, EXCERPT_MAX_BYTES);
}
return excerpt.trim();
}
/**
* OpenRouter's response when the per-run key's remaining budget can't cover
* the agent's `max_tokens` reservation. Distinct from a generic provider error
+1 -1
View File
@@ -1,6 +1,6 @@
import type { AgentResult } from "../agents/shared.ts";
import type { MainResult } from "../main.ts";
import type { ToolState } from "../mcp/server.ts";
import type { ToolState } from "../toolState.ts";
import { log } from "./cli.ts";
import { reportErrorToComment } from "./errorReport.ts";
+19 -1
View File
@@ -9,6 +9,22 @@ export interface Mode {
prompt: string;
}
/**
* server-parsed TOC entry for `Repo.learnings`. depth is 1-6 (h1-h6),
* line numbers are 1-indexed against the raw body. computed by
* `parseLearningsHeadings` in `utils/learningsToc.ts` (server side) and
* shipped over the run-context JSON boundary; the canonical declaration
* lives there. duplicated here because the action runtime can't reach
* across into the proprietary root-level codebase, and the JSON wire
* means typecheck can't enforce shape equality across both sides.
*/
export interface LearningsHeading {
depth: 1 | 2 | 3 | 4 | 5 | 6;
title: string;
startLine: number;
endLine: number;
}
export interface RepoSettings {
model: string | null;
modes: Mode[];
@@ -21,6 +37,7 @@ export interface RepoSettings {
prApproveEnabled: boolean;
modeInstructions: Record<string, string>;
learnings: string | null;
learningsHeadings: LearningsHeading[];
envAllowlist: string | null;
}
@@ -61,6 +78,7 @@ const defaultSettings: RepoSettings = {
prApproveEnabled: false,
modeInstructions: {},
learnings: null,
learningsHeadings: [],
envAllowlist: null,
};
@@ -88,7 +106,6 @@ export async function fetchRunContext(params: {
try {
const headers: Record<string, string> = {
Authorization: `Bearer ${params.token}`,
"Content-Type": "application/json",
};
if (params.oidcToken) {
headers["X-GitHub-OIDC-Token"] = params.oidcToken;
@@ -128,6 +145,7 @@ export async function fetchRunContext(params: {
postCheckoutScript: data.settings?.postCheckoutScript ?? null,
prepushScript: data.settings?.prepushScript ?? null,
stopScript: data.settings?.stopScript ?? null,
learningsHeadings: data.settings?.learningsHeadings ?? [],
},
apiToken: data.apiToken,
oss: data.oss ?? false,
+1 -1
View File
@@ -3,7 +3,7 @@ import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { ShellPermission } from "../external.ts";
import type { ToolState } from "../mcp/server.ts";
import type { ToolState } from "../toolState.ts";
import { log } from "./cli.ts";
import type { OctokitWithPlugins } from "./github.ts";
import { isInsideDocker } from "./globals.ts";
+2 -2
View File
@@ -68,7 +68,7 @@ export function installBundledSkills(params: { home: string }): void {
writeFileSync(join(skillDir, "SKILL.md"), content);
}
}
log.info(`installed bundled skills: ${BUNDLED_SKILL_NAMES.join(", ")}`);
log.success(`installed bundled skills: ${BUNDLED_SKILL_NAMES.join(", ")}`);
}
/**
@@ -108,7 +108,7 @@ export function addSkill(params: {
}
);
if (result.status === 0) {
log.info(`installed ${params.skill} skill (${params.agent})`);
log.success(`installed ${params.skill} skill (${params.agent})`);
} else {
const stderr = (result.stderr?.toString() || "").trim();
const errorMsg = result.error ? result.error.message : stderr;
+102 -1
View File
@@ -1,5 +1,6 @@
import { performance } from "node:perf_hooks";
import { describe, expect, it } from "vitest";
import { spawn } from "./subprocess.ts";
import { spawn, TailBuffer } from "./subprocess.ts";
describe("spawn error path", () => {
it("surfaces ENOENT-style spawn failures in stderr so callers can diagnose", async () => {
@@ -48,6 +49,106 @@ describe("spawn error path", () => {
expect(afterHandles).toBeLessThanOrEqual(beforeHandles);
});
it("killGroup: true propagates SIGKILL to grandchildren so close fires promptly", async () => {
// regression: node_modules/opencode-ai/bin/opencode is a Node shim that
// spawnSyncs the native binary with stdio:"inherit". without killGroup,
// child.kill("SIGKILL") hit only the shim — the native binary was
// reparented to PID 1, kept holding our stdout pipe via the inherited
// fds, and `child.on("close")` never fired (because pipes stayed open).
// a 5-min outer safety-net timer eventually rejected the agent promise,
// but the grandchild kept running until the GitHub Actions job-level
// timeout. this test replicates the shape with bash + a backgrounded
// sleep grandchild: with killGroup, close fires promptly after SIGKILL;
// without it, the parent would wait for sleep to exit (30s).
//
// the activity-check interval is fixed at 5s so the earliest the kill
// can fire is ~5s after start. budget 15s end-to-end.
const before = performance.now();
const result = await spawn({
cmd: "bash",
args: ["-c", "sleep 30 & wait"],
env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "" },
activityTimeout: 1000,
killGroup: true,
}).catch((err) => err);
const elapsed = performance.now() - before;
expect(result).toBeInstanceOf(Error);
// 10s ceiling: 5s activity-check tick + signal delivery. a regression
// here (no killGroup) would hang for the full 30s sleep.
expect(elapsed).toBeLessThan(10_000);
}, 20_000);
it('retain:"tail" caps stderr at maxRetainedBytes and prepends a truncation sentinel', async () => {
// regression for issue #680: unbounded `stderrBuffer += chunk` previously
// crashed the wrapper with `RangeError: Invalid string length` once V8's
// ~1 GiB kMaxLength was breached on long-lived agent runs. the fix caps
// retention with a TailBuffer; this test exercises the cap end-to-end by
// emitting ~2 MiB of stderr against a 256 KiB ceiling and asserts the
// wrapper does not crash, the result is bounded, and the sentinel is
// present so downstream consumers can detect the truncation.
const result = await spawn({
cmd: "bash",
// print ~2 MiB to stderr in 64 KiB chunks. `yes` + head gives us a
// reliable byte budget that's well above the 256 KiB cap below.
args: ["-c", "yes ABCDEFGH | head -c 2097152 1>&2"],
env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "" },
activityTimeout: 0,
maxRetainedBytes: 256 * 1024,
});
expect(result.exitCode).toBe(0);
expect(result.stderr).toMatch(/truncated by retain:tail cap/);
expect(result.stderr.length).toBeLessThan(256 * 1024 + 200);
}, 15_000);
it('retain:"none" returns empty stdout/stderr regardless of child output', async () => {
// long-lived agent callers (opencode, claude) drain via onStdout/onStderr
// and never read result.stdout/result.stderr — they pass retain:"none"
// to skip the per-chunk concatenation entirely. assert that contract:
// empty strings out, but onStdout still fires.
const chunks: string[] = [];
const result = await spawn({
cmd: "bash",
args: ["-c", "echo hello; echo world 1>&2"],
env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "" },
activityTimeout: 0,
retain: "none",
onStdout: (chunk) => chunks.push(chunk),
});
expect(result.exitCode).toBe(0);
expect(result.stdout).toBe("");
expect(result.stderr).toBe("");
expect(chunks.join("")).toContain("hello");
});
it('retain defaults to "tail" so short-lived callers keep failure-surfacing snapshots', async () => {
// lock the default explicitly. gitAuth, package installs, and lifecycle
// hooks all rely on `result.stderr` being non-empty on failure — flipping
// the default to "none" would silently break their error messages while
// all other tests in this file kept passing.
const result = await spawn({
cmd: "bash",
args: ["-c", "echo -n diagnostic-output 1>&2; exit 7"],
env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "" },
activityTimeout: 0,
});
expect(result.exitCode).toBe(7);
expect(result.stderr).toBe("diagnostic-output");
});
it("TailBuffer drops oldest bytes once the cap is exceeded", () => {
const buf = new TailBuffer(10);
buf.append("0123456789");
expect(buf.toString()).toBe("0123456789");
buf.append("abcde");
// 0-9 plus abcde = 15 chars; cap is 10, so we keep the last 10 = "56789abcde"
expect(buf.toString()).toMatch(/truncated by retain:tail cap/);
expect(buf.toString()).toContain("56789abcde");
});
it("reports signal-killed subprocesses as failures, not success", async () => {
// regression: before the fix, `child.on("close", (exitCode) => ...)`
// discarded the signal parameter and `exitCode || 0` coerced the
+145 -19
View File
@@ -88,6 +88,29 @@ function installSignalHandler(): void {
});
}
/**
* Controls what the wrapper retains in memory across the child's lifetime
* for the post-hoc `SpawnResult.stdout` / `SpawnResult.stderr` snapshots.
*
* Streaming callbacks (`onStdout` / `onStderr`) fire regardless `retain`
* only governs the buffered snapshot returned in `SpawnResult`.
*
* - `"tail"` (default): keep the last `maxRetainedBytes` UTF-16 code units
* of each stream. Once the cap is exceeded, oldest bytes are sliced off
* and the result is prefixed with a `... [N MiB truncated] ...` sentinel.
* Right default for short-lived commands whose failure mode is in their
* final output (git errors, install failures, hook scripts).
* - `"none"`: skip the buffer entirely. `SpawnResult.stdout` / `.stderr`
* are empty strings. Use this for long-lived streaming agents that already
* drain via `onStdout` / `onStderr` and never read the buffered snapshot.
*
* Default cap is 8 MiB well below V8's ~1 GiB `kMaxLength` so `+= chunk`
* can never throw `RangeError: Invalid string length`.
*/
export type RetainMode = "tail" | "none";
export const DEFAULT_MAX_RETAINED_BYTES = 8 * 1024 * 1024;
export interface SpawnOptions {
cmd: string;
args: string[];
@@ -106,6 +129,55 @@ export interface SpawnOptions {
stdio?: ("pipe" | "ignore" | "inherit")[];
onStdout?: (chunk: string) => void;
onStderr?: (chunk: string) => void;
// when true, spawn the child detached (its own process group) and route all
// kill paths (timeout, activity timeout, ctrl-c) through `process.kill(-pid, ...)`
// so signals reach grandchildren too. critical for binaries that fork through
// a shim (e.g. node_modules/opencode-ai/bin/opencode is a Node shim that
// spawnSync's the native binary; without killGroup, SIGKILL only hits the
// shim and the native binary is reparented to PID 1, holds our stdout pipe
// open, keeps emitting NDJSON, and `child.on("close")` never fires —
// producing zombie runs that hang until the GitHub Actions job timeout).
killGroup?: boolean;
retain?: RetainMode;
maxRetainedBytes?: number;
}
/**
* Bounded string accumulator that keeps the tail of appended chunks.
* Once the cap is exceeded, oldest bytes are sliced off and `toString()`
* prefixes the survivors with a sentinel describing the elided byte count.
*
* Exported because long-lived agent runtimes (opencode, claude) also
* accumulate per-run narration strings independently of the spawn wrapper
* and need the same protection against V8's `kMaxLength`.
*/
export class TailBuffer {
// explicit field declarations rather than constructor parameter properties:
// node's strip-only TS loader (used by action/test/run.ts in CI) rejects
// `constructor(private readonly cap: number)` with ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX.
private readonly cap: number;
private buffer = "";
private truncatedBytes = 0;
constructor(cap: number) {
this.cap = cap;
}
append(chunk: string): void {
if (this.cap <= 0) return;
this.buffer += chunk;
if (this.buffer.length > this.cap) {
const drop = this.buffer.length - this.cap;
this.truncatedBytes += drop;
this.buffer = this.buffer.slice(drop);
}
}
toString(): string {
if (this.truncatedBytes === 0) return this.buffer;
const mib = (this.truncatedBytes / 1024 / 1024).toFixed(1);
return `... [${mib} MiB truncated by retain:tail cap] ...\n${this.buffer}`;
}
}
export interface SpawnResult {
@@ -124,8 +196,17 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
installSignalHandler();
const startTime = performance.now();
let stdoutBuffer = "";
let stderrBuffer = "";
// capped accumulators — unbounded `+= chunk` previously crashed the wrapper
// with `RangeError: Invalid string length` once V8's ~1 GiB kMaxLength was
// breached on long-lived agent subprocesses (e.g. multi-lens opencode
// Reviews on large monorepos). retain:"none" skips the buffer entirely
// for callers that already drain via onStdout/onStderr.
const retain: RetainMode = options.retain ?? "tail";
const cap = options.maxRetainedBytes ?? DEFAULT_MAX_RETAINED_BYTES;
const stdoutBuffer = retain === "none" ? null : new TailBuffer(cap);
const stderrBuffer = retain === "none" ? null : new TailBuffer(cap);
const killGroup = options.killGroup ?? false;
return new Promise((resolve, reject) => {
// security: caller must provide complete env object, not merged with process.env
@@ -136,10 +217,28 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
},
stdio: options.stdio || ["pipe", "pipe", "pipe"],
cwd: options.cwd || process.cwd(),
detached: killGroup,
});
// sends `signal` to the entire process group when killGroup is set, so
// grandchildren (e.g. the native opencode binary spawned by the
// opencode-ai Node shim) die with the parent. falls back to a direct
// child kill if the process-group send fails (common when the child
// already exited or was never made a process group leader).
const killSelf = (signal: NodeJS.Signals): void => {
if (killGroup && child.pid) {
try {
process.kill(-child.pid, signal);
return;
} catch {
// fall through to direct kill
}
}
child.kill(signal);
};
// track child for cleanup on Ctrl+C
trackChild({ child });
trackChild({ child, killGroup });
let timeoutId: NodeJS.Timeout | undefined;
let sigkillEscalatorId: NodeJS.Timeout | undefined;
@@ -157,7 +256,7 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
if (options.timeout) {
timeoutId = setTimeout(() => {
isTimedOut = true;
child.kill("SIGTERM");
killSelf("SIGTERM");
// track the escalator so a graceful SIGTERM response (close fires
// before the 5s elapses) can clear it. without capture, this timer
@@ -165,7 +264,7 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
// past a timed-out subprocess's clean exit.
sigkillEscalatorId = setTimeout(() => {
if (!child.killed) {
child.kill("SIGKILL");
killSelf("SIGKILL");
}
}, 5000);
}, options.timeout);
@@ -186,9 +285,9 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
killedAtIdleMs = idleMs;
const idleSec = Math.round(idleMs / 1000);
log.info(
`no output for ${idleSec}s from pid=${child.pid} (${options.cmd}), killing process`
`no output for ${idleSec}s from pid=${child.pid} (${options.cmd}), killing process${killGroup ? " group" : ""}`
);
child.kill("SIGKILL");
killSelf("SIGKILL");
clearInterval(activityCheckIntervalId);
try {
options.onActivityTimeout?.();
@@ -205,20 +304,46 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
lastActivityTime = performance.now();
}
// wrap handlers in try/catch as defense in depth for synchronous throws
// inside the listener body. the historical `+= chunk` RangeError was such
// a throw — synchronous and fatal under node's default uncaught-exception
// policy. with the TailBuffer cap in place the wrapper-side `append` can
// no longer throw, but the catch keeps protecting against any future
// synchronous regression in this path.
//
// note: this does NOT catch rejections from async user callbacks —
// `options.onStdout?.(chunk)` returns a Promise in the agent callers
// (claude.ts, opencode.ts) and a throw inside an async callback surfaces
// as an unhandled Promise rejection, not a synchronous exception. agent
// callers handle their own NDJSON-parse failures internally; the
// synchronous protection here is what matters for the RangeError class
// of bugs (issue #680).
if (child.stdout) {
child.stdout.on("data", (data: Buffer) => {
updateActivity();
const chunk = data.toString();
stdoutBuffer += chunk;
options.onStdout?.(chunk);
try {
updateActivity();
const chunk = data.toString();
stdoutBuffer?.append(chunk);
options.onStdout?.(chunk);
} catch (err) {
log.debug(
`spawn stdout handler threw: ${err instanceof Error ? err.message : String(err)}`
);
}
});
}
if (child.stderr) {
child.stderr.on("data", (data: Buffer) => {
const chunk = data.toString();
stderrBuffer += chunk;
options.onStderr?.(chunk);
try {
const chunk = data.toString();
stderrBuffer?.append(chunk);
options.onStderr?.(chunk);
} catch (err) {
log.debug(
`spawn stderr handler threw: ${err instanceof Error ? err.message : String(err)}`
);
}
});
}
@@ -260,7 +385,7 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
// appeared to succeed when they'd actually been killed — caller
// checked `result.exitCode !== 0` and moved on.
let resolvedExitCode = exitCode ?? 0;
let resolvedStderr = stderrBuffer;
let resolvedStderr = stderrBuffer?.toString() ?? "";
if (exitCode === null && signal) {
const killMsg = `[spawn] ${options.cmd}: killed by signal ${signal}`;
resolvedStderr = resolvedStderr ? `${resolvedStderr}\n${killMsg}` : killMsg;
@@ -268,7 +393,7 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
}
resolve({
stdout: stdoutBuffer,
stdout: stdoutBuffer?.toString() ?? "",
stderr: resolvedStderr,
exitCode: resolvedExitCode,
durationMs,
@@ -290,11 +415,12 @@ export async function spawn(options: SpawnOptions): Promise<SpawnResult> {
// per the guidance, and hit the same wall every run.
const errMsg = `[spawn] ${options.cmd}: ${error.message}`;
console.error(errMsg);
stderrBuffer = stderrBuffer ? `${stderrBuffer}\n${errMsg}` : errMsg;
const existingStderr = stderrBuffer?.toString() ?? "";
const finalStderr = existingStderr ? `${existingStderr}\n${errMsg}` : errMsg;
resolve({
stdout: stdoutBuffer,
stderr: stderrBuffer,
stdout: stdoutBuffer?.toString() ?? "",
stderr: finalStderr,
exitCode: 1,
durationMs,
});
+13
View File
@@ -203,5 +203,18 @@ describe("ThinkingTimer", () => {
expect(cli.log.info).toHaveBeenNthCalledWith(1, "» thought for 4 seconds");
expect(cli.log.info).toHaveBeenNthCalledWith(2, "» thought for 5 seconds");
});
it("routes log lines through the optional formatLine for per-session prefixing", () => {
const startTime = 1000000;
vi.mocked(performance.now)
.mockReturnValueOnce(startTime) // markToolResult
.mockReturnValueOnce(startTime + 4000); // markToolCall
const timer = new ThinkingTimer((line) => `[lens:security] ${line}`);
timer.markToolResult();
timer.markToolCall();
expect(cli.log.info).toHaveBeenCalledWith("[lens:security] » thought for 4 seconds");
});
});
});
+23 -3
View File
@@ -22,6 +22,15 @@ export class Timer {
const THINKING_THRESHOLD = 3000; // ms
/**
* Measures wall-clock gap between the last tool_result and the next tool_call,
* surfacing it as a "thought for Xs" log when over `THINKING_THRESHOLD`.
*
* Use one instance per logical session (orchestrator, each subagent) sharing
* a single timer across sessions conflates cross-session interleaving as
* thinking time. The optional `formatLine` lets the caller prefix output with
* a session label so attribution is visible in the merged log stream.
*/
export class ThinkingTimer {
private readonly durationFormatter = new Intl.NumberFormat("en-US", {
style: "unit",
@@ -32,21 +41,32 @@ export class ThinkingTimer {
});
private lastToolResultTimestamp: number | null = null;
private readonly formatLine: (line: string) => string;
// node's native TS strip-only mode does not support parameter properties,
// so the formatter is declared as a field and assigned in the body.
constructor(formatLine: (line: string) => string = (l) => l) {
this.formatLine = formatLine;
}
markToolResult(): void {
this.lastToolResultTimestamp = performance.now();
log.debug(`» thinking timer: markToolResult at ${this.lastToolResultTimestamp}`);
log.debug(
this.formatLine(`» thinking timer: markToolResult at ${this.lastToolResultTimestamp}`)
);
}
markToolCall(): void {
const now = performance.now();
log.debug(
`» thinking timer: markToolCall at ${now}, lastToolResult=${this.lastToolResultTimestamp}`
this.formatLine(
`» thinking timer: markToolCall at ${now}, lastToolResult=${this.lastToolResultTimestamp}`
)
);
if (this.lastToolResultTimestamp === null) return;
const elapsed = now - this.lastToolResultTimestamp;
if (elapsed < THINKING_THRESHOLD) return;
const seconds = elapsed / 1000;
log.info(`» thought for ${this.durationFormatter.format(seconds)}`);
log.info(this.formatLine(`» thought for ${this.durationFormatter.format(seconds)}`));
}
}