Compare commits

...

67 Commits

Author SHA1 Message Date
Colin McDonnell 09344a9ec9 release: bump action to 0.1.10 2026-05-20 15:01:35 +00:00
Colin McDonnell 1b201352b5 release: bump action to 0.1.9 2026-05-20 14:09:52 +00:00
Colin McDonnell 6c166ac1cc fix: prevent cross-PR push from subagent-induced branch switch (#796)
* fix: prevent cross-PR push from subagent-induced branch switch

A workflow_dispatch run for zed-industries/cloud (workflow run 26036155393)
force-pushed the orchestrator's work onto an unrelated engineer's PR branch
(origin/reactivate-pro-plan, PR #2582). The orchestrator's reviewfrog subagent
called checkout_pr({pull_number: 2582}), which (1) moved the shared working
tree to pr-2582 and (2) persisted pushDest pointing at reactivate-pro-plan.
The orchestrator's subsequent commit + push_branch then clobbered the victim
PR. Recovery + disclosure in PR #2584.

Three compounding bugs closed here:

1. checkout_pr dirty-tree guard had a first-call hole: the previous condition
   required ctx.toolState.issueNumber to already be set, so on workflow_dispatch
   runs the first checkout_pr (commonly from a subagent) bypassed the guard
   entirely. Now any PR switch with a dirty tree is refused, including the
   first switch of a run. Idempotent same-PR re-checkouts are still absorbed
   by alreadyOnBranch inside checkoutPrBranch.

2. push_branch trusted sticky pushDest blindly. Added a backstop: refuse
   pushes where the local branch matches /^pr-(\d+)$/ AND pushDest.remoteBranch
   differs from it AND the current run is not scoped to PR N (event.is_pr === true
   && event.issue_number === N). This catches subagent-induced silent branch
   switches even if a future bug reintroduces a first-call hole in fix #1.

3. Build-mode self-review prompt told the orchestrator to ship "the output of
   git diff" to the reviewer. The model in this run synthesized
   git diff main...HEAD, which excludes uncommitted work — and Build self-review
   runs BEFORE the commit, so the reviewer saw an empty diff and thrashed,
   eventually calling checkout_pr on a random PR to find something to look at.
   Prompt now specifies git diff origin/<base-branch> (two-dot, no HEAD),
   which compares the working tree against the remote base.

Refs:
  zed-industries/cloud workflow run 26036155393
  zed-industries/cloud#2582 (victim)
  zed-industries/cloud#2584 (disclosure)

* review: key dirty-tree guard on current branch + drop 'two-dot' misnomer

Address review feedback on PR #796.

1. checkout_pr dirty-tree guard now keys off the live current branch
   (git rev-parse --abbrev-ref HEAD), not ctx.toolState.issueNumber.
   issueNumber is ALSO set by get_issue / get_issue_comments /
   get_issue_events, so a subagent doing get_issue(N) followed by
   checkout_pr(N) on a dirty tree would have bypassed the original guard
   (issueNumber === pull_number). The current branch is the actual
   primitive for "would this call move HEAD" — querying it directly avoids
   correlating on toolState that other tools write to.

2. modes.ts: drop the wrong "two-dot" label on git diff origin/<base>.
   That's the single-rev form, not two-dot. Copilot was right that the
   label was confusing/contradictory with the actually-shown command.
2026-05-20 05:24:24 +00:00
Colin McDonnell a0576a702a opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767)
* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite

Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.

Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.

Surface area:

  - drop `init` / `message` / `result` / `tool_result` event types and
    handlers (no longer emitted at v1.14+ per upstream
    `cli/cmd/run.ts:588-601`).
  - `tool_use` is now a single event covering both `state.status:
    "completed"` and `"error"`. duration / subagent-finish bookkeeping
    moves from the v1 `tool_result` handler into the consolidated
    `tool_use` handler.
  - new `reasoning` event handler — gated on `--thinking`, surfaces
    Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
    `baseArgs`.
  - drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
    v1.15 the `task` tool callID is stable across the whole
    `tool-input-* → tool-call → tool-result/tool-error` chain
    (`session/processor.ts:282-330`). exact-match map is sufficient.
  - drop `experimental.batch_tool: true` from injected config — declared
    but inert at v1.15. re-add once upstream wires it back.
  - bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
    the platform-specific binary into `opencode.exe` for every OS now).

Validated locally:

  - `pnpm test` 610/610 ✓
  - `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
  - `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
    multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
    (this is the headline `thought_signature` fix)
  - runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
  - runtest opencode: skill-invoke and mcpmerge fail (model-behavior
    drift on the new system prompt; wiring confirmed intact via direct
    repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
    Tracked for follow-up; does not gate the migration.

Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.

* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)

The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.

* opencode v2: set PWD env explicitly to fix skill / project-config discovery

Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:

  const root = Filesystem.resolve(process.env.PWD ?? process.cwd())

We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.

Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.

Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).

* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly

opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).

Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.

Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now:    `» thinking (4.0s): <preview>` (one line, sourced)

For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.

Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.

* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment

claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.

Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.

Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.

* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup

Code-quality pass on the v2 work:

1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
   between v1 and v2:
   - `OpenCodeConfig` type
   - `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
   - `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
     and re-imported by v2 via a back-reference)
   - `installOpencodeCli({ binPath })` (parameterized — v1 passes
     `bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
     `installCli` lambda; matches each pinned version's npm shape)
   - `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback

   v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
   keeps a one-line `export { geminiHighThinkingOverrides }` re-export
   so `opencode.test.ts` keeps working unchanged. Once v1 is retired
   (post burn-in) opencodeShared collapses back into v2.

2. `opencode_v2.ts` cleanup:
   - drop dead state (`currentStepId`, `stepHistory` were write-only —
     their reader was the v1 `tool_result` handler we deleted)
   - hoist `state` in `tool_use` handler; replace nested-ternary payload
     extraction with a `terminalPayload(state)` helper
   - extract `formatPartDuration(time)` for the reasoning-block
     "(X.Ys)" suffix
   - tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
     `callID` fields directly, drop the `partWithToolFields` cast
   - trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
     handler, tool_use handler, bus envelope handler all shortened
   - `step_start` becomes an explicit `() => {}` no-op so the dispatcher
     doesn't log "unhandled event" for every step

3. `subagentRegistration.test.ts` retargeted at the new file split —
   reads opencodeShared.ts for the buildReviewerAgentConfig assertions
   and opencode_v2.ts for the orchestrator-model wire-through.

Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.

* opencode v2: address PR review feedback

Three fixes from the inline review threads on #767:

1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
   level `markActivity()` resets the module-level idle counter, so the
   per-event `getIdleMs()` sample inside the dispatch loop was always
   ~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
   a runner-local `lastEventAt` so we measure real event-to-event silence
   instead of chunk-arrival latency. Drop the unused `getIdleMs` import.

2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
   `lastProviderError`, and `recentStderr` are closed over by the
   `handlers` const but were declared after it. No current bug because
   handlers only fire inside the awaited `spawn()`, but a future
   refactor that triggers a handler synchronously during setup would
   surface a TDZ. Hoisted above `handlers`.

3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
   L566): leave a `TODO` comment marking the gap until `AgentUsage`
   grows a `reasoningTokens` field — separate PR with schema work.
   Cost totals stay correct because `part.cost` is summed independently.

Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
  fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
  fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
  `reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
  the bus-envelope flow; the plugin filters orchestrator events except
  for status=running task dispatches, and bus-envelope returns before
  calling handlers.tool_use on those. No double-fire under current code.

Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.

* DX: flip pnpm play / pnpm runtest to docker-by-default

Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.

scripts (root → action):
- pnpm play       → pnpm -C action gha play.ts          (docker, default)
- pnpm play:local → pnpm -C action play:local           (host)
- pnpm runtest    → pnpm -C action gha test/run.ts      (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local     (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)

action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.

CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.

Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.

File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
  `pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers

AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.

wiki/docker.md unchanged — already described the now-real shape.

* test/crossagent: add codex-auth smoke

Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:

  - installCodexAuth() materializes auth.json under the test HOME
  - opencode routes openai requests through ChatGPT subscription auth
    (no OPENAI_API_KEY in env, AT path forced via expires: 0)
  - the refresh chain advances during the run (refresh_token rotates)
  - detectCodexRefresh() would surface the rotation to entryPost.ts

The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.

CI wiring (already committed in a1c1fd4f as part of the DX flip):
  - .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
    action-agents env block
  - action/.github/workflows/test.yml: same; codex-auth in the
    hardcoded test matrix with a claude exclude

The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.

ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.

* docs(codex-auth): make storage requirement unmissable

A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).

- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
  with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
  enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
  loudly contrast Pullfrog secret store vs GH Actions and explain the
  writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
  future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
  comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
  CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
  unchanged per scope.

* auth codex: auto-open device URL, drop --scope flag

- detect `https://auth.openai.com/codex/device...` from codex CLI output
  and best-effort launch it in the user's default browser (open / xdg-open
  / cmd start, wslview fallback on linux). gated so we only open once per
  flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
  interactive (browser approval), so a "skip-the-prompt" flag for just one
  of the prompts was dead weight. collapses scope selection to "always
  prompt on org-owned, always account on user-owned".

* rename gha→docker, flip play/runtest defaults to host

the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.

- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
  tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
  tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
  `pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
  bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
  `README.md`, `AGENTS.md`), CI workflows
  (`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
  and code headers (`action/{play,test/run,utils/runFixture}.ts`,
  `webhook.ts`, `action/test/coverage.ts`)

`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).

* fix(codex): route post-hook writeback through apiFetch + conditional skip

Three threads addressing PR #767 followups.

action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).

action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.

doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
  mention (removed in 10be96db, scope is now always interactively
  prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
  is agent-gated (opencode/opencode_v2 harness), not model-gated;
  opencode runs with non-OpenAI models still materialize the file,
  it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
  → pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
  instead of the dropped --scope flag.

* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS

upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.

also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.

local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.

---------

Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>
2026-05-20 04:05:16 +00:00
David Blass 4d1fd5ea1a fix: 4 unaddressed log-audit / run-audit findings + close 10 already-resolved issues (#785)
* fix: 4 unaddressed log-audit / run-audit findings

closes 4 issues with code changes; 7 issues are already addressed by #769
and 3 are deferred — see PR description.

#782 Anthropic 401 → `isApiKeyAuthError` now matches the direct-Anthropic
401 shape (`Failed to authenticate. API Error: 401 ...`,
`authentication_error`, `Invalid bearer token`, `api_error_status=401`)
so revoked / mistyped / rotated `ANTHROPIC_API_KEY` users see the
formatted rotate-key CTA instead of a raw 401 JSON dump.

#778 billing-class provider errors → `providerErrors.ts` now classifies
`CreditsError` / `FreeUsageLimitError` / `Insufficient balance` /
`spending cap` as `provider billing exhausted` *before* status-code
patterns can win and tag them as transient `auth error (401)` /
`rate limited (429)`. `agentHangReport.ts` swaps the bare
"Pullfrog stalled — auth error" headline for a billing-specific CTA
(extracts the provider's billing URL when present).

#775 silent IncrementalReview swallows `BillingError` →
`reportErrorToComment` now optionally falls through to creating a fresh
issue comment on `toolState.issueNumber` when no progress comment
exists. Wired with `createIfMissing: true` from the `BillingError` /
`TransientError` paths in `proxy.ts` so silent triggers
(`pull_request_synchronize`) finally surface the router-balance signal
on the PR instead of only in the GH job summary.

#773 `currentUser()` inside `after()` →
`fillInstallerIdentityIfMissing` is split into
`resolveInstallerIdentity` (must run inside the request body) and
`fillInstallerIdentity` (DB-only, safe in `after()`). The
`/console/[owner]` caller now resolves Clerk identity up-front and
defers only the prisma write, fixing the broken installer-identity
backfill on org-console first-admin visits.

Co-authored-by: Cursor <cursoragent@cursor.com>

* add /audits cursor command for triaging run-audit + log-audit issues

Co-authored-by: Cursor <cursoragent@cursor.com>

* review prompt: tighten body-section bar + inline technical-details (#770)

* review prompt: tighten body-section bar + add inline technical-details

Two layers of tightening to the Review/IncrementalReview prompts in
PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step):

1. Reframe inline-vs-body split. Body `### ` sections are now reserved
   for concerns that genuinely have no line to anchor to — absence,
   sequencing, design decisions, scope questions, architectural risk.
   Drop the "cross-cutting concerns" framing (misled the agent into
   either filing nothing in the body or filing multi-file anchored
   findings there).

2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step
   6) and IncrementalReview (step 8) aggregate phases. Diagnosis from
   PR #767's auto-review: on substantial PRs the agent surfaced
   findings but routed all of them inline, producing reviews with zero
   `### ` body sections even on diffs where non-anchored concerns
   clearly existed.

3. Replace the abstract `### ` example with a concrete non-anchored
   one ("Legacy `opencode.ts` has no documented deletion plan") so the
   agent pattern-matches the absence-shaped finding, not a line-bug.

4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT
   so inline comments can carry a `<details>Technical details</details>`
   block when the fix has cross-file implications. Rename the existing
   "Agent details" inline collapsible to "Technical details" for
   consistency with body sections.

5. (Carried over from prior uncommitted work) Restructure the review
   metadata block from `<details>Review metadata</details>` into an
   HTML comment + an italic TL;DR commit-range line. The HTML comment
   keeps the metadata addressable for downstream agents without
   eating user-visible review real estate.

No tests touched.

* wiki: document multi-model end-to-end eval pattern

* feat(promo): cookie-stashed promo codes for onboarding rewards (#771)

* feat(promo): cookie-stashed promo codes for onboarding rewards

Operator hands out a link like https://pullfrog.com/start?promo=FROGGY;
middleware validates the code against an in-code registry, stashes it in
an HttpOnly cookie, and the install callback applies the reward once
the GH-side account exists. v1 reward: unlimited_runs (lifts the monthly
free-runs cap to 1M, same convention prod-grandfathered accounts use).

No schema changes. Idempotent across reinstalls via the lte: 100 gate.

* fix(promo): integrate handler into existing proxy.ts (Next 16 rename)

* docs(promo): clarify sentinel + sync plan doc with renamed paths

* feat(promo): add FOUNDATIONS code

* feat(promo): show applied promo code in console

* refactor(promo): move cookie set to client-side

* docs(promo): point JSDocs at PromoCookieSetter, not proxy.ts

* billing: cap counts only successful runs (#787)

* billing: cap counts only successful runs

`reserveRun` was counting `WorkflowRun` rows regardless of status against
`Account.includedMonthlyRuns`. Failed / cancelled / skipped / timed-out
runs consumed cap slots even though their `billableCents` got zeroed on
the completion webhook — pushing paying users into billable territory
earlier than the contract implies. `inthhq` paid for 2 extra runs this
month because 2 failed runs ate 2 of their 100 free slots.

Cap query now filters on `CAP_CONSUMING_STATUS = "success"`. Only
runs that actually deliver value consume slots; in-flight (`running`)
runs hold no slot until they terminate as success (burst-bypass risk
is theoretical given GH Actions concurrency limits).

Shared constant lives in `utils/billing.ts` and is used in lockstep by
three call sites: `reserveRun` (live cap gate), the billing API's
`runsThisMonth` (dashboard progress bar), and the billing-report
script's `cap` column. Script's `cap` cell was also broken
independently — it compared `monthBillableRuns` (overage count) against
`includedMonthlyRuns` (free cap), so `inthhq` rendered as `125/100
(over)` when the meaningful ratio is `223/100 (over)`. Fixed to use
`mRuns/cap`, which is the same predicate the live billing path uses.

* move CAP_CONSUMING_STATUS to workflowRunStatus.ts + wire script through it

Per copilot review: the JSDoc claimed the billing-report script used the
constant in lockstep, but the script kept `status: "success"` inline. The
script imports from raw-node ESM and can't pull in `next/server`, so it
couldn't import from `utils/billing.ts`. Moved the constant to
`utils/workflowRunStatus.ts` (already Next-free, already the home of
`CONCLUSION_VALUES`) and updated all three call sites to import from
there. Script's `mRuns` query now uses `CAP_CONSUMING_STATUS` directly,
making drift impossible.

* learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743)

* learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy

three audit fixes on top of the recent learnings overhaul (#717):

- `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry
  when a body has non-whitespace content before the first heading. the
  prompt instructs the agent NOT to slurp the whole file when a TOC is
  present, so without this any preamble lines were silently invisible
  (realistic transitional case: an agent partially restructures a
  legacy free-text body and leaves bullets above the first `## `).

- server-side PATCH route now applies the same line-boundary-aware
  truncation as the action (defense in depth via a shared
  `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from
  `action/internal`). the raw `.slice` it used before could leave a
  mid-heading tail on any caller that bypassed the client-side
  truncate, breaking the next-seed TOC parse. removes the duplicated
  cap constant.

- `buildLearningsSection` intro no longer asserts "accumulated by
  previous agent runs" — false for fresh repos with zero history. new
  copy is tense-neutral and works for empty + populated bodies. also
  nudges the agent to re-read after mid-run edits (the inlined TOC
  ranges are a run-start snapshot).

Co-authored-by: Cursor <cursoragent@cursor.com>

* learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste

The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned
calls discovering a quirk this run, recording the workaround prevents the
next run from repeating the waste. Reframe around one litmus ("would a
future run do its work better because this bullet exists?") and trust it
to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary)
and the four-example pullfrog/PR/date/play-by-play list (the rule
underneath is "don't anchor facts to repo state that will move"). Cuts
~10 lines from a prompt the model was already mostly ignoring; the
remaining anchor list is narrower and more enforceable.

* audit-learnings-r2: align wiki + tighten re-read nudge

- wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls.
- buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly.

* postRun: refresh JSDoc to match the reflection prompt rewrite

`buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets.

* fix(mcp/issueEvents): narrow event.event before Set.has lookup

octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup.

* learnings: split truncation helpers into MCP-free module

re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph.

move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>

* trim first-run celebration email to short personal note

drops the feature-dump bullet list (custom review instructions, github
iteration walkthrough, security model) — wrong moment to teach. keeps
the congrats, the reply CTA, adds discord/x links, keeps the router
credit P.S. handler no longer needs the workflowRun→repo lookup.

* signup-report: per-bucket histogram

Adds a UTC-aligned signups-per-bucket histogram between the overview
block and the company-email list. Empty buckets are pre-filled with 0
so dry spells render as gaps. New `BUCKET=hour|day` env flag with a
smart default (hour if window ≤ 48h, else day). Histogram is also
included in the JSON payload under `histogram: [{key, count}, ...]`.

* signup-report: drop hourly bucket, day-only histogram

* feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748)

* feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660)

Per-account ceiling on the sum of `router_topup` invoices (pending +
succeeded) for the current UTC calendar month. Closes a gap where a
runaway agent loop, leaked PR trigger, or stuck workflow could
auto-reload indefinitely with no aggregate per-month ceiling.

Two enforcement modes via `RouterLimitMode` enum:
  - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun;
    402 `router_monthly_limit` from /api/proxy-token; email + banner
  - `alert_only`: auto-reload keeps flowing; email + banner only,
    first breach per UTC month

Enforcement is split across reserveRun (pre-dispatch paywall comment)
and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through
the same `getRouterSpentThisMonthCents` helper so the dashboard, the
dispatch gate, and the auto-reload gate can't disagree.

Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string),
claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent
reloads breaching together send exactly one email. Read-time
comparison with the current month re-arms on rollover — no cron.

Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell)
above the Router/BYOK tabs in `ModelAccessCard`, with a popover
"Adjust limit" form that PATCHes the existing
/api/account/[owner]/billing/settings route. Same `assertBillingAdmin`
gate that owns the other billing settings — no new auth surface.

See wiki/billing.md § Router monthly spend limit for the full
contract + edge cases.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): anneal pass on monthly Router spend limit (#660)

Round-1 review across 5 lenses (billing-subsystem, correctness,
security, operational-readiness, research-validated-assumptions)
surfaced one critical + three actionable major findings on top of
[#748](https://github.com/pullfrog/app/pull/748).

**Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot`
used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles
to `field != value` — UNKNOWN (not TRUE) against the post-migration
`NULL` default. First breach for any account would never claim the
slot, never stamp the row, and never fire the email (hard_cap or
alert_only). Replaced with `OR: [{ field: null }, { field: { not:
monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern.

**Major — email gap on manual-top-up over cap.** Breach email was only
wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>`
that crosses the cap blocks dispatch via `reserveRun` but never hits
proxy-token, so the user got the PR comment but no email. Wired the
CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s
PaywallError catch (the SERIALIZABLE txn rolled back when we threw,
so we re-claim with the global client; single-statement CAS is its
own race boundary against concurrent proxy-token claims).

**Major — PR paywall comment leaked $ figures.** `router_limit` body
embedded `($X of $Y)` in a comment visible to anyone with PR read
access (public repos, forks, outside collaborators). Other paywall
types deliberately avoid amounts. Removed; deep link still points to
the authenticated console for the figures.

**Medium — observability.** Added `[router-limit]` structured logs at
the three enforcement sites (proxy-token hard_cap 402, proxy-token
alert_only breach, reserveRun paywall) so on-call can grep "did the
cap fire for customer X this month."

**Medium — customer docs.** Added a `### Monthly spend limit` section
to `docs/billing.mdx` (Mintlify) describing the two modes and the
manual-top-up caveat.

**Doc — refund/dispute interaction.** Documented in `wiki/billing.md`
that the cap inherits the existing webhook semantics: disputed
`router_topup` drops from the sum (cap briefly un-trips); refunds
don't flip status today so refunded top-ups keep counting. Matches
wallet behavior — not redefined here.

Accepted as-is (documented or pre-existing): `after()` reliability vs
stamp-before-send tradeoff, alert_only email fires before Stripe
phase-2, proxy-token reads limit fields outside SERIALIZABLE scope
(brief TOCTOU on admin lowering cap), stale paywall comment on cap
clear, no global kill switch (per-account `alert_only` flip is the
practical kill switch), no audit log on cap changes (no existing
audit infra), action version not bumped (separate release commit).

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): anneal round 2 on monthly Router spend limit

Round-2 anneal (billing-subsystem, correctness, research-validated,
user-journey, operational-readiness) surfaced a critical merge conflict
and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748).

**Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted
`formatBillingErrorSummary` from `action/main.ts` to
`action/utils/billingErrors.ts`. The PR's new `router_monthly_limit`
arm still lived in `action/main.ts`. Took main's slim orchestrator
wholesale; moved the arm into the extracted file.

**Major — cap = payments only, not dispatch.** `reserveRun` was
pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit`
regardless of wallet balance, contradicting the cap's positioning as
"ceiling on what you pay." An account with $500 of paid-up wallet and
a breached $100 cap couldn't trigger any new run via the comment
path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded —
surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token`
is now the sole enforcement point, refusing only the next auto-reload
that would push past. Wallet credit always drains. Dropped the now-dead
`router_limit` arm in `buildPaywallCommentBody`, the dead
`routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`,
and the post-paywall email-fire David added — all unreachable.

**Major — split `manual_topup` from `router_topup`.** Manual on-session
top-ups at `/billing-top-up/<owner>` were landing as
`Invoice.kind = "router_topup"` and counting toward the cap. The cap
exists to brake *passive* runaway (auto-reload loops); a manual top-up
is a deliberate click-through that the user owns. Added
`InvoiceKind.manual_topup`, flipped the manual write site +
`createTopUpCheckoutSession` metadata, broadened wallet /
reconcile / billing-report reads to `kind IN (router_topup,
manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap
aggregate) to `router_topup` only. Worked example: cap=$300,
reload=$100 → exactly three reloads succeed; a fourth is blocked.
Historical rows stay labelled `router_topup` (no backfill); the
asymmetry is small and accepted since the manual flow only existed
alongside auto-reload for a brief window. Extended the
`invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows
the same shape as `router_topup` (PaymentIntent-backed, no
stripeInvoiceId); split into a second migration because PG forbids
using a freshly-added enum value in the same transaction.

**Major — email reframed around the triggering reload event.** The
`alert_only` body was reporting a pre-eager-write `spentCents` while
the dashboard reads the post-commit value, so email and dashboard
disagreed by exactly one reload. Both flavors now say "Your most
recent $50 auto-reload brought you over your $300 monthly limit"
instead of a running spent-of-cap total — no reconciliation needed,
no more "you've hit your monthly cap" copy firing for partial breaches
(spent=$80 of $100, reload=$30 would have triggered that wording).

**Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded
"You've used your 30 free runs this month. Add a card to continue at
7¢/run." regardless of `detail.reason`. Branched on `cap` vs
`delinquent` so each paywall surfaces actionable copy with the right
CTA. `router_limit` no longer flows through here (per F4 above).

**Major — RouterLimitBanner.** Added an `isAlertBreached` visual
state (amber palette) so an `alert_only` account at $240 of $200 no
longer renders in the same neutral zinc chrome as a healthy under-cap
account. Updated popover copy to reflect the auto-reload-only scope.

**Medium — paywall log line.** Added `detail.reason` to the
`[Installation X] paywall:` log so on-call grepping for "why was this
paused" can distinguish `cap` from `delinquent`.

**Cleanup.** Dropped dead `utcMonthKey` import + re-export in
`maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*`
fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*`
since they now handle both kinds. Updated wiki/billing.md +
docs/billing.mdx + schema doc comments throughout.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key

The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt`
sitting next to it — a single-purpose state column on `Account` that
encoded a date as a string and required a custom CAS predicate to
read/write race-safely. Plus it had real holes: Resend send failure
left the sentinel stamped and the account silently un-emailed for the
month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered
edge cases never fired at all.

Replace it with: fire `maybeNotifyRouterLimit` on every breaching
reload, let the Resend `Idempotency-Key`
`router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside
Resend's 24h dedup window. Continuously-breaching accounts get ~1
reminder per day; brief Resend outages self-heal because the next
breaching reload re-attempts the send. Mode is in the dedup key so
`alert_only → hard_cap` mid-month re-arms a fresh email with the
appropriate copy.

Drops `Account.routerLimitNotifiedMonth` and
`claimRouterLimitNotificationSlot`; simplifies the proxy-token
phase-1 branch significantly. Net diff is negative LOC and the data
model loses a single-purpose sentinel.

Migration was branch-local — never deployed — so I edited the original
add-cap migration in place to drop the column from the ALTER TABLE
rather than chain a drop-column migration on top. Preview Neon
branches reset automatically on history rewrite per wiki/migrations.md.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): hide RouterLimitBanner when no cap is configured

The banner was unconditionally rendered for every billing-enabled
account, including pure-BYOK admins who never touch Router. They got
"No monthly spend limit / Router has spent $0.00" + a divider as
visual noise on the model access page — basically nagging them to set
a feature they may not want. Running without a cap is valid; we don't
nag.

ModelAccessCard now gates the banner block (banner + dividers) on
`routerMonthlyLimitCents !== null`. RouterLimitBanner drops the
no-limit visual state, the "Set monthly limit" CTA text, and the dead
`hasLimit` branching. Cleaner three-state shape (under cap / amber
breached / brick breached).

Discoverability: no-cap users no longer see a UI affordance to set
one. That's deliberate — the cap is a power-user feature documented
in docs/billing.mdx. If discoverability becomes an ask, we can add a
small inline link inside RouterWalletSection without bringing back
the always-visible banner.

Resolves the only outstanding finding from cursor bugbot's review of
ff5328c (banner-visible-for-byok thread).

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(billing): docs/wiki match new "no banner without a cap" reality

Pullfrog bot review of f7672ca pointed out the customer docs still
told users to "Set the cap from the **Monthly spend limit** banner in
the **Model costs** card" — but after hiding the banner for no-cap
accounts there is no such banner to use until you already have a cap.
Catch-22 for first-time setup.

Rewrote docs/billing.mdx to be self-contained: explain what the cap
is, what the two modes do, what the banner shows *once configured*,
and direct admins to PATCH the billing settings endpoint (or reach
out to support) for first-time setup. Cap is positioned as optional;
running without one is the documented default.

Wiki paragraph in wiki/billing.md updated to match — banner is only
rendered when a cap exists, three visual states (under / amber / red),
no first-time-setup UI nag by design.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely

The standalone `RouterLimitBanner` was the wrong shape. It only
rendered when a cap was already configured (so there was no UI to
discover the feature in the first place — first-time setup required
hitting the API directly), and it occupied prominent real estate above
the tabs to surface state that already lives in the row's own input
when the form moves down where it belongs.

New shape: monthly cap is just a third row inside `RouterWalletSection`
sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated
the same way (card on file + auto-reload enabled — the only state
where the cap actually means anything). Empty input → no cap, with
placeholder "No limit". Setting a number reveals a **Behavior at
limit** toggle built on the same `Tabs` slider component used for the
Router/BYOK tab switch, so the look matches the rest of the card.

Deletes:
- `RouterLimitBanner` component (212 lines)
- banner mount + conditional + spacers in `ModelAccessCard`
- `AlertTriangle` is still imported (used by `DelinquencyBanner`)

Adds:
- one settings row in `RouterWalletSection` with the cap input + mode tabs
- `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the
  existing `saveSettings` helper (widened to accept `string | null`)
- `Tabs` / `TabsList` / `TabsTrigger` import

Docs + wiki updated to match the new shape; the customer doc no
longer points at a banner that won't appear.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between

Previously bundled both into one row block. Restructure: cap input is
its own row; Behavior-at-limit Tabs gets a sibling row with the
standard `h-5 + hr + h-5` separator between (matching the rhythm of
auto-reload amount → threshold → monthly cap). Mode-toggle row is
gated on `routerMonthlyLimitCents !== null` so the hr + tabs only
appear once a number is in the cap input.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row

Same `flex items-center justify-between gap-3` layout as the
Auto-reload row: label group on the left, control on the right.
Drops the vertical stack in favour of the horizontal one — looks
identical to the toggle row directly above.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>

* drop italic TL;DR commit-range line from review body

the metadata (sha range, commit list, timestamps) is already in the html
comment for downstream agents. the visible italic line was clutter and
the ellipsis form broke the second sha's auto-link on github anyway.

* add agent-browser fallback rule for unreachable chrome devtools mcp

* onboarding: gated org-console wizard (#762)

* onboarding: gated org-console wizard

Replaces the org console's `/console/[owner]` page with a single-card,
"growing" stepper when the account has zero `Repo` rows. Walks first-time
users through billing mode, BYOK provider+key (if applicable), repo pick,
workflow file creation, and a celebratory redeem-credit moment before
landing them back on the now-populated org console.

## What's new

- New: `components/OnboardingStepper.tsx` — the wizard. Six steps, each
  derived from real persisted state (Account.modelAccessMode,
  AccountSecret, Repo). Step state ladder with progressive disclosure and
  click-to-edit collapsed summaries.
- New: `app/console/[owner]/OnboardingView.tsx` — page-chrome wrapper
  that hosts the stepper inside the same header/sidebar shell as the
  member view.
- Modified: `app/console/[owner]/page.tsx` — adds a `prisma.repo.count`
  gate alongside existing parallel queries; renders OnboardingView when
  count === 0, else falls through to the existing repo grid.

## Schema

- Flipped `Account.modelAccessMode` default from `byok` to `router`.
  Router is the lower-friction default (signup credit funds first ~150
  runs without a card; users can flip to BYOK explicitly via the wizard
  or the existing `<ModelAccessCard>` switch). Existing rows keep their
  current explicit value — Postgres column-default change doesn't
  backfill, by design.
- Migration: `20260516014601_modelaccessmode_default_router`.

## Credit-claim semantics

Killed the historical mount-time auto-claim on `<SignupCreditModal>`.
All claims are now explicit clicks, fired from one of two surfaces:

1. Wizard step 6 "Redeem $10 credit" CTA (Router branch, eligible).
2. New explicit "Redeem $10 credit" button on `<BillingCard>`'s Router
   wallet section, visible only when the new server-derived
   `signupCreditEligible` flag is true (promo active + no prior signup
   or welcome grant). Covers existing users who'd otherwise lose the
   auto-claim entry point.

`<SignupCreditModal>` is now a controlled component (`open` /
`onOpenChange` / `amountCents` props) with a sibling
`useClaimSignupCredit(owner)` hook for explicit invocation. The Sparkles
celebration dialog rendering is unchanged.

## Other touched surfaces

- `app/api/create-workflow/route.ts`: optional `model` body field. When
  present, the route updates `Repo.model` on the row that
  `createWorkflowForRepo` just created/surfaced — wizard threads the
  picked provider's `preferred` model alias through here so a fresh repo
  doesn't sit on null/auto.
- `app/api/account/[owner]/billing/route.ts`: surfaces
  `signupCreditEligible: boolean` (derived from
  `SIGNUP_CREDIT_PROMO_ACTIVE` + grant scan). Drives the new explicit
  redeem button.
- `components/AgentSettings.tsx`: fixes the Router-no-billing copy lie
  ("Runs will draw from your signup credit until exhausted" was false —
  `isInfraCovered` gates Router minting on `hasCardOnFile`, not balance,
  so credit-only-no-card users can't actually spend the grant on Router
  runs). New copy: "Add a card to use Pullfrog Router. Your $10 signup
  credit (if claimed) applies on top."

## Resume-tomorrow detection

Every step's expansion is derived from persisted state (no new column,
no localStorage). With the Router default flip, `modelAccessMode ===
"byok"` is now a reliable signal of explicit user pick, eliminating the
heuristic that the byok-default schema would have required. The only
ambiguous case is "Router-bailed-before-redeem" (looks identical to a
default-Router fresh visit since neither card nor grant exists yet) —
acceptable 1-click cost on revisit.

## Testing

- `pnpm lint`: clean
- `pnpm format`: clean
- `pnpm typecheck`: clean
- `pnpm -C action test`: 596/596 passing
- Visual verification: blocked — Chrome DevTools MCP returned "Not
  connected" across both available servers. Manual walkthrough needed
  before merge to confirm step transitions, going-back UX, and the
  celebration modal redirect destinations match the plan in
  `.cursor/plans/org_onboarding_stepper_4fdfebbb.plan.md`.

* onboarding: drop accordion, multi-repo bulk-onboard, full-width radio rows

Three rounds of UX feedback rolled in:

1. **Drop the accordion.** Steps no longer collapse to a one-line summary
   when "done" — the wizard literally grows by appending steps below as
   the user progresses, and earlier steps stay fully interactive
   (re-flip Router→BYOK, re-pick provider, toggle a repo) without any
   "edit" affordance. `StepShell` now always renders its body for any
   step the user has reached; the only state distinction is the number
   circle (filled = active, check = done).

2. **Step 1 is full-width radio rows, not narrow tabs with side-by-side
   info tiles.** Two rows, each with the option title, an inline
   "Recommended" badge on Router, and a description sentence inside the
   row. The persisted `Account.modelAccessMode` (default `router`)
   drives the initial selection, so step 1 always has one row picked
   on first paint — no "neither selected" empty state.

3. **Multi-repo bulk-onboard.** Step 4 now uses checkboxes; copy reads
   "Select the repos you'd like to install Pullfrog into. We'll create
   a pullfrog.yml GitHub Actions workflow file in each." Step 5 fans
   out N parallel `POST /api/create-workflow` calls (concurrency
   capped at 4) and renders per-repo status inline (running →
   committed / PR #N / already configured / error). Step 6 celebrates
   with a multi-result headline ("Pullfrog is set up across N repos")
   and a sub-line breaking down `committed · PRs awaiting merge ·
   failed` plus a per-repo PR list when any PRs were opened. Single-
   repo path renders the same control surface but with singular copy.

Other bits:
- Per-step description sentences below every title.
- Repo picker shows totalCount inline with the pagination controls and
  "N repos selected" summary below the table.
- Dropped the `userPickedBillingMode` and `editingStep` state machinery
  + the `isFreshDefault` heuristic — all simplified out by the
  no-accordion design (we just trust `billingMode` directly).
- `createWorkflowPR` PR body already links back to
  `pullfrog.com/console/<owner>/<repo>` with a "Verify workflow" CTA;
  no change needed there.

* fix(onboarding): provider tile labels — getProviderDisplayName expects slug

`getProviderDisplayName` from `pullfrog/internal` parses its argument as a
`provider/model` slug. Step 2 was passing bare provider keys (e.g.
"anthropic"), which made the helper throw "invalid model slug 'anthropic'
— expected 'provider/model'" and crashed the BYOK branch with the
page-level error boundary.

Replace with a local `providerDisplayName` that reads the registry
directly (`providers[key].displayName`). Drops the unused
`getProviderDisplayName` import.

Caught by Chrome DevTools end-to-end: clicking Bring-your-own-key on the
fresh wizard renders the page-level error. Re-verified post-fix: BYOK
flow shows step 2 with all 9 provider tiles correctly labeled
(Anthropic / OpenAI / Google / xAI / DeepSeek / Moonshot AI / Amazon
Bedrock / OpenRouter / OpenCode), step 3 reveals on tile click.

Also adds a guardrail to AGENTS.md: don't silently abandon visual
verification when DevTools breaks. Recovery is always possible
(pkill -9 chrome-devtools-mcp + pkill puppeteer + rm Singleton locks +
retry several times); if it genuinely won't recover, abort and tell the
user — never mask as "verified by code review".

* agents.md: never give up on Chrome DevTools MCP failures

Recovery is always possible (pkill chrome-devtools-mcp, remove Singleton
locks, retry several times). If genuinely unrecoverable, abort and tell
the user explicitly — never silently mask as "verified by code review".
Visual verification is non-negotiable for UI changes.

* onboarding: polish — checkbox color, redundant labels, copy

Caught during chrome-devtools verification of the BYOK + cross-page
selection flows:

- **Checkbox color**: native browser pink/red replaced with
  evergreen via `accent-evergreen-600`. Visually consistent with the
  rest of the wizard's selection states.
- **Bedrock provider tile**: was rendering "Amazon Bedrock" twice
  (provider name + recommended-model name both resolve to "Amazon
  Bedrock" because Bedrock has no `preferred` model under
  `providers.bedrock.models` — its single routing entry IS the
  recommended pick). Suppress the recommended subtitle when it
  duplicates the provider name.
- **Step 6 description**: tightened from a clunky two-clause sentence
  about workflow file landing to a single direct call: "Mention
  @pullfrog in any PR or issue to dispatch a run. (Branch-protected
  repos: merge the PR first.)"
- **Wizard intro**: was "Set up Pullfrog for your first repo" —
  outdated since multi-repo. Now: "Connect Pullfrog to your repos.
  Each step unlocks the next as you go."

Cross-page multi-select also verified: selections from page 1 persist
when navigating to page 2 and back. "N repos selected" counter
reflects total across all pages.

BYOK secret-add flow verified end-to-end: AddSecretModal opens with
the env var pre-filled, save triggers secrets refetch, step 3 flips
to "✓ ANTHROPIC_API_KEY configured", step 4 reveals automatically.

* onboarding: serial install, inline secrets, explicit credit redeem

- step 3: replace modal-based secret entry with inline password fields per
  provider, with deep links to provider dashboards. claude code OAuth
  surfaces as a distinct group when anthropic is picked. bedrock gets
  three-field form. github actions secrets path is collapsible with
  org/personal-aware urls + self-certify.
- step 4: merge repo-pick + workflow-create into one step. install is now
  serial (visible slow-reveal) instead of concurrent. continue button
  renders immediately on submit, disabled until every repo reaches a
  terminal state. errored rows render a single soft amber 'failed' label.
  pagination uses chevron buttons + keepPreviousData (no layout shift).
- step 6: explicit 'redeem $10 credit' for router+eligible, 'complete
  setup' otherwise. final redirect is a hard refresh so the repo grid
  picks up.
- signup credit: drop the mount-time auto-claim modal in favor of explicit
  user clicks. new useClaimSignupCredit hook + RedeemSignupCreditCallout
  banner inside RouterWalletSection so a BYOK→Router flip surfaces a
  one-click redeem affordance.
- billing mode is now optimistic (local state + background PATCH) and
  initialBillingMode + signupCreditEligible eager-load via server props
  to kill the multi-second click latency.
- skip onboarding: header button sets pullfrog_skip_onboarding cookie;
  server reads it in page.tsx and falls through to the regular grid.
- demo mode: NEXT_PUBLIC_ONBOARDING_DEMO=1 cycles the install progress
  list through pending/running/committed/PR/existing/failed states.
- createWorkflowForRepo: PULLFROG_FORCE_PR_CREATION=1 skips direct commit
  to exercise the PR fallback locally.

* onboarding: review feedback — focused eligibility query, best-effort model pre-fill, claim error toast

- billing/route.ts + console/[owner]/page.tsx: replace top-N
  recentGrants scan for signup-credit eligibility with a focused
  findFirst({ reason: { in: [SIGNUP, WELCOME] } }). the prior query
  could return any 5/10 rows (no orderBy on page.tsx) and miss a prior
  signup/welcome grant if a future grant reason (refund/referral/etc.)
  ever ships. recentGrants stays for the billing-history list.
- create-workflow/route.ts: gate Repo.model updateMany on result.type
  === "created" so an existing user-set model isn't clobbered when the
  workflow file already exists. wrap in try/catch: GitHub side effect
  already succeeded, so a transient DB blip shouldn't 500 the route
  and have the UI report failure on a partially-completed setup.
- SignupCreditModal: add onError toast to useClaimSignupCredit so
  transient redeem failures surface ("Couldn't redeem your credit. Try
  again in a moment."). callers .catch(() => null) the rejection so it
  doesn't propagate as an unhandled rejection in the React handler.
- OnboardingStepper: trim stale "per-row try again button" wording
  from progressRef + processRepo comments — that button was removed in
  the prior commit per design feedback.

* router: fix unspendable signup credit on no-card private repos (#791)

The bug
-------
`run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`,
which is `oss || hasCard`. So a no-card account with positive wallet
balance (signup credit, top-up, etc.) on a private repo would never
get a `proxyModel` set on the run context. The action runtime then
fell through to whatever provider keys happened to be in the workflow
env — using the user's BYOK keys without their knowledge if any were
configured, or failing the run entirely otherwise.

Meanwhile `proxy-token/route.ts` already gated correctly on
`oss || hasCard || balance > 0`. The two routes disagreed, with
run-context being strictly more restrictive, so the agent never even
attempted to call proxy-token for these accounts. The wiki at
`billing.md:1052` documented the *intended* behavior ("a Router usage
row can debit a wallet with no card on file"), aspirational against
the actual code.

The action side had a parallel bug at `action/utils/proxy.ts:151` —
it re-derived `isInfraCovered({ isOss, plan })` and short-circuited
mint even when the server set `proxyModel`. Belt-and-suspenders that
was strictly more restrictive than the server.

Production impact
-----------------
Queried 55 router-mode no-card accounts holding signup credit:
- ALL have wallet balance = exactly $10.00 (untouched)
- ALL have 0 router proxy keys ever minted, 0 hwm usage
- ~25 have successful runs (using BYOK env vars from their workflow,
  unaware their credit isn't being touched)
- The rest have zero successes; some accumulated 25+ failures
  (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit).

The fix
-------
- `run-context/route.ts`: widen `useRouter` to match proxy-token's
  gate. OSS short-circuits as before. Otherwise: router mode + card
  on file → mint; router mode + no card + positive balance → fetch
  balance, mint if > 0. Skip the balance read when a card is on file
  (auto-reload covers it without needing pre-flight balance — keeps
  the hot path single-query).
- `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check.
  `ctx.proxyModel` IS the signal — the server is the authority on
  funding decisions; the action just trusts and mints.
- `wiki/pricing.md`: correct the Router proxy key minting gate row
  + add a paragraph explaining why this gate diverges from
  `isInfraCovered`.
- `wiki/billing.md`: rewrite the misleading "proxy-token returns 402"
  paragraph to describe what actually happens at both routes.

`isInfraCovered` is unchanged. It still gates Pullfrog-paid features
(learnings writes, indexing). The bug was in conflating "Pullfrog
pays for marginal infra" with "user can fund a Router run via wallet"
— different concerns, now untangled.

* revert: extract router-gate fix into its own PR

The router fix at a14bcdd4 is being shipped as a standalone PR so it
can be reviewed and merged independently of the onboarding-wizard
work. Reverting here keeps #762 focused on the wizard. The fix itself
landed at https://github.com/pullfrog/app/pull/792.

* router: fix unspendable signup credit on no-card private repos (#792)

* router: fix unspendable signup credit on no-card private repos (#791)

The bug
-------
`run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`,
which is `oss || hasCard`. So a no-card account with positive wallet
balance (signup credit, top-up, etc.) on a private repo would never
get a `proxyModel` set on the run context. The action runtime then
fell through to whatever provider keys happened to be in the workflow
env — using the user's BYOK keys without their knowledge if any were
configured, or failing the run entirely otherwise.

Meanwhile `proxy-token/route.ts` already gated correctly on
`oss || hasCard || balance > 0`. The two routes disagreed, with
run-context being strictly more restrictive, so the agent never even
attempted to call proxy-token for these accounts. The wiki at
`billing.md:1052` documented the *intended* behavior ("a Router usage
row can debit a wallet with no card on file"), aspirational against
the actual code.

The action side had a parallel bug at `action/utils/proxy.ts:151` —
it re-derived `isInfraCovered({ isOss, plan })` and short-circuited
mint even when the server set `proxyModel`. Belt-and-suspenders that
was strictly more restrictive than the server.

Production impact
-----------------
Queried 55 router-mode no-card accounts holding signup credit:
- ALL have wallet balance = exactly $10.00 (untouched)
- ALL have 0 router proxy keys ever minted, 0 hwm usage
- ~25 have successful runs (using BYOK env vars from their workflow,
  unaware their credit isn't being touched)
- The rest have zero successes; some accumulated 25+ failures
  (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit).

The fix
-------
- `run-context/route.ts`: widen `useRouter` to match proxy-token's
  gate. OSS short-circuits as before. Otherwise: router mode + card
  on file → mint; router mode + no card + positive balance → fetch
  balance, mint if > 0. Skip the balance read when a card is on file
  (auto-reload covers it without needing pre-flight balance — keeps
  the hot path single-query).
- `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check.
  `ctx.proxyModel` IS the signal — the server is the authority on
  funding decisions; the action just trusts and mints.
- `wiki/pricing.md`: correct the Router proxy key minting gate row
  + add a paragraph explaining why this gate diverges from
  `isInfraCovered`.
- `wiki/billing.md`: rewrite the misleading "proxy-token returns 402"
  paragraph to describe what actually happens at both routes.

`isInfraCovered` is unchanged. It still gates Pullfrog-paid features
(learnings writes, indexing). The bug was in conflating "Pullfrog
pays for marginal infra" with "user can fund a Router run via wallet"
— different concerns, now untangled.

* action: drop dead isInfraCovered + plan param post-fix

Cleanup the action-side dead code introduced by the previous commit's
removal of the redundant `isInfraCovered` re-derivation in proxy.ts:

- delete `isInfraCovered` from action/utils/runContext.ts (was the only
  callsite; mirror in server's utils/billing.ts is unchanged and still
  load-bearing for learnings/indexing)
- drop unused `plan: AccountPlan` param from `resolveProxyModel` /
  `runProxyResolution` (and the corresponding `AccountPlan` import +
  the `plan: runContext.plan` arg at the main.ts call site)
- update the action/mcp/server.ts comment that pointed at the now-gone
  action mirror to reference the server-side `utils/billing.ts` instead

`AccountPlan` itself is still load-bearing (mcp/server, runContextData,
run-context fetch), only `isInfraCovered` and the dead `plan` parameter
go away.

* eager signup credit + free-OpenCode fallback when BYOK has no key (#789)

* eager signup credit + free-OpenCode fallback when BYOK has no key

addresses the silent-churn pattern that took out 15 first-run-failure
accounts post-launch: GH Actions secret references resolved to empty
strings (because the secrets didn't exist on the repo), the action
launched Claude Code with no key, the LLM provider 401'd, and the run
died in seconds with a synthetic "Invalid API key" message. those
accounts had no Router credits to fall back to because the lazy claim
required a dashboard visit they never made.

three changes, one PR:

1. Eager $10 signup credit at account creation. Both account-creation
   sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo`
   for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }`
   in the same transaction as the `accounts` row. CLI installers who
   never sign in get the credit. The dashboard `/signup-credit/claim`
   POST stays as an idempotent backstop for accounts created before
   this shipped.

2. Free-OpenCode fallback in the action. When the configured BYOK slug
   needs a provider key the runner doesn't have, swap to
   `opencode/minimax-m2.5-free` before agent selection so the run still
   succeeds. Surfaced via a `» fell back from <slug> to <free>` warning
   in the action log. Skipped on Router runs (Pullfrog mints the key)
   and when no model is configured (auto-select-with-throw still fires
   for the genuinely-misconfigured case).

3. New action-test fixture `byok-no-keys-fallback` that empty-strings
   every known provider key (matching how GH Actions handles missing
   secrets) and asserts the run succeeds with the fallback log line
   present. plus a unit test for the helper covering each skip case.

skipping the schema flip from `byok` to `router` — that's coming via
the onboarding-stepper PR (#762).

* fallback: skip Bedrock + surface in PR-comment footer

addresses copilot review on #789 (real bug — parseModel throws on
Bedrock raw IDs that have no slash, would crash before
validateBedrockSetup could surface its own error) and the user-side
ask to make the fallback visible in PR comments.

- selectFallbackModelIfNeeded skips when resolvedModel has no '/' so
  Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash
  inside hasProviderKey -> parseModel. unit test covers it.
- toolState.modelFallback records the configured slug we fell back
  from. set in main.ts when fallback engages.
- buildPullfrogFooter accepts fallbackFrom and renders
  "Using `MiniMax M2.5` (free) (credentials for Claude Opus not
  configured)" so the substitution is visible in PR comments,
  reviews, PR bodies, and error reports.
- threaded through all four action-side footer call sites
  (mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side
  call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts
  fire pre-action and don't have toolState — left as-is.

* fallback footer: use provider display name + document email asymmetry

addresses pullfrog reviewer findings on #789:

- footer now shows 'credentials for Anthropic not configured' (provider
  display name from `providers.anthropic.displayName`) instead of the
  per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY
  covers all Anthropic models), so this matches what the user actually
  needs to fix.

- document the intentional asymmetry between eager and lazy signup
  credit paths: eager skips both the signupCreditClaimedEmail and the
  per-grant team@ alert. comment explains why (the 'new account
  created' alert already covers it on the eager path; the user-facing
  email assumes a user-initiated action that hasn't happened yet for
  CLI/GH-App-only signups).

- skipping the backfill for the 15 historical accounts per user's
  earlier decision — they all uninstalled, so the cohort self-selected
  out of being reachable.

* fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap

local agnostic fixture run surfaced two real bugs the unit tests didn't
catch:

1. fallback gate was on configuredSlug (=payload.model) but the test
   uses PULLFROG_MODEL to set the model, which is read by resolveModel
   AFTER its slug arg. configuredSlug stayed undefined → fallback never
   fired. drop configuredSlug from the helper signature; gate purely on
   resolvedModel since that's the same value regardless of how the
   model was specified (DB config vs PULLFROG_MODEL env).

2. when fallback engaged, the post-swap resolveModel({slug: fallback.to})
   call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback
   target back to the unkeyed model. validateAgentApiKey then threw
   "no API key found" against the original model. fix: skip the
   re-resolve. fallback.to is already a CLI-ready specifier.

unit tests updated for the new helper signature (8 tests, all pass).
fallback log line confirmed emitted in the local run pre-second-fix;
the second fix unblocks the validation that previously threw.

* models-bump: harden CI and bot prompt against catalog hallucinations

PR #790 (the first bot-authored models-bump PR) shipped a broken bump for
openrouter/gemini-flash: the bot pattern-matched the parallel google bump
and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the
OpenRouter API but not on models.dev's openrouter section — the catalog
OpenCode actually reads. The slug failed at runtime with `Model not found`.
Two CI gaps let it through:

1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*`
   alias as a "passthrough", smoke-testing only one canary per routing
   layer. But those aren't passthroughs — each is a distinct models.dev
   catalog entry that can drift independently of the direct-provider
   mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25).
   Only `bedrock/byok` stays pruned (sentinel resolve).

2. `models-catalog` test (the integrity gate that asserts every resolve
   exists on models.dev) was main-only by design — to keep upstream catalog
   churn from blocking unrelated PRs. But it's exactly the test we want
   running on the bot's own catalog edits. Add `pullfrog/models-bump`
   head-ref to its trigger.

Also tighten the bot prompt in models-bump.yml: new rule 0 requires every
new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the
alias's own `candidates[]` in models-bump-context.json — the deterministic
preprocessor only emits candidates sourced from models.dev's mirror, so
this gates against the cross-alias pattern-matching that broke PR #790.
For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter
API), which is necessary but not sufficient; the `models-catalog` job is
the authoritative models.dev check.

Verified locally:
- baseline `pnpm -C action test:catalog` passes 133 tests
- simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash`
  into action/models.ts) and the catalog test fails with the right
  assertion: `model "google/gemini-3.5-flash" not found under openrouter
  on models.dev`
- `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every
  openrouter/* alias and every keyed opencode/* alias now smoked

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-20 03:58:18 +00:00
David Blass 2f1f136da8 modes: actually call resolve_review_thread on addressed PR feedback (#749)
* modes: instruct IncrementalReview + AddressReviews to actually call resolve_review_thread

empirically (per #672 audit on 195 prod runs across 4 repos) only 1 in 195 runs
called resolve_review_thread. IncrementalReview's prompt only mentioned prior
reviews as a dedup filter — the agent had the data in hand but no instruction
to retire addressed threads, so it almost never did. AddressReviews mentioned
resolve as a one-line bullet at the end of step 6, which the agent treated as
optional (the one observed AddressReviews run replied to 3 comments and
resolved 0).

IncrementalReview step 4 now: fetch prior reviews → for each open Pf-originated
thread, decide if the new commits addressed it (anchor moved, isOutdated, or
substantive concern resolved on a re-read), reply + resolve when addressed,
leave open when uncertain. Conservative scope: only Pf-originated threads —
human-reviewer threads stay theirs to mediate.

AddressReviews step 6 now pairs reply + resolve in the same beat with explicit
rules: resolve when you made the change OR replied substantively; do NOT
resolve when you pushed back and the disagreement is unresolved.

addresses #672.

Co-authored-by: Cursor <cursoragent@cursor.com>

* anneal: tighten auto-resolve decision rules

correctness fixes from /anneal pass on IncrementalReview step 4 + AddressReviews step 6:

- `[OUTDATED]` no longer "strong signal of address" — it just means GitHub moved the anchor (line shift / reformat / force-push); agent must re-read code at new location
- explicit Pf-origin detection rule (first `comment author=pullfrog[bot]` tag), clarifies `*` marker is unrelated to thread root
- explicit dual-ID separation: numeric `id=` for `reply_to_review_comment.comment_id`, GraphQL `thread=` for `resolve_review_thread.thread_id` (was silently 422-ing)
- reformatter / partial-fix loophole closed: lines being modified isn't enough; all concerns in multi-concern comments must be addressed
- AddressReviews push-failure path explicit: STOP and report_progress, do NOT reply or resolve when fix isn't live
- step 4 → step 8 wiring rationale corrected (step 8 dedups by line range, not thread state)

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-20 03:57:52 +00:00
Colin McDonnell cb0dbcd371 feat(action): make prepush hook non-blocking after one failure (#777)
* feat(action): make prepush hook non-blocking after one failure

push_branch now treats the repository's prepush hook as best-effort: it
runs at most once per run, surfaces the failure output if the script
exits non-zero, and every subsequent push_branch call this run skips
the hook so the agent isn't blocked by failures unrelated to its
change. The agent can iterate by running the hook command itself via
the shell tool when shell access is available; push_branch will not
re-run the hook automatically after a failure.

Why: a one-line OSS-allowlist change took 9 minutes (#776) because the
agent retried push_branch six times against a prepush hook that was
failing for env-leak and missing-build-artifact reasons unrelated to
the change. CI catches the same checks on the GitHub side; the local
prepush gate was duplicating work and blocking unrelated fixes.

- ToolState: new prepushFailureCount counter (per-run, never resets)
- executeLifecycleHook: returns structured failure (kind/output/exitCode)
  so prepush can compose its own agent-facing message instead of
  inheriting the generic retry/no-retry advice meant for setup
- push_branch: composes a shell-mode-aware error message; surfaces
  prepushSkipped on the success payload + appends a note to the message
- instructions.ts + wiki/prompt.md + docs/comparisons.mdx: updated to
  reflect best-effort semantics

* fix(action): clarify prepush latch semantics + soften static guidance

review fixes from PR #777:

- toolState comment, instructions, success message, tool description:
  replace "runs at most once per run" / "first call only" wording with
  the actual semantic — successful prepush keeps running on later
  push_branch calls; only a hook FAILURE latches the bypass.
- tool description: drop hardcoded "via the shell tool" guidance so
  the static description doesn't mislead in shell:disabled runs (the
  dynamic agent prompt in instructions.ts already does shell-conditional
  messaging).
- LifecycleHookFailure.output JSDoc: match the implementation
  (stderr-preferred fallback to stdout, empty for timeout/spawn).

* fix(action): shorten prepush-skip log to terse operator telemetry

the previous log line tried to address the agent ("re-run the hook
command yourself via shell"), but log.info writes to the action
runtime's stdout — the agent never sees it. agent-facing skip
guidance already lives in the error message from
buildPrepushFailureMessage, the success message when bypassed, and
the system prompt in instructions.ts. log line is now just operator
telemetry.

* refactor(action): drop slop from prepush soft-fail

self-audit pass after the previous review-fix round. removed
duplication between code-level comment and the five other places that
already explain the same behavior, tightened verbose JSDoc, and
collapsed redundant clauses in agent-facing strings.

- LifecycleHookFailure → discriminated union. drops the optional
  exitCode/spawnError fields (and the empty-output sentinel for
  timeout/spawn) plus the corresponding ?? fallbacks in the helper.
- PushBranchTool: 7-line code comment above the latch removed
  (toolState field comment + tool description + error message +
  success message + system prompt all already cover it). tool
  description third sentence dropped (restated the second). success
  message tightened to a parenthetical.
- buildPrepushFailureMessage: 4-line JSDoc → 1 line. shared "if you
  think the failure could indicate a real bug in your code" prefix
  factored out across the shell-conditional branches.
- ToolState.prepushFailureCount comment: 8 lines → 3. the "what" is
  in git.ts; comment now only documents the invariant (never
  decremented within a run).
- instructions.ts prepush guidance: collapsed nested bullets + ternary
  into one paragraph; dropped the "so re-running via shell is the only
  way…" tail that restated "push_branch will NOT re-run it".

* fix(action): hint prepush bypass on dirty tree after hook failure

When push_branch blocks on a dirty working tree and the prepush latch
is already set, tell the agent the hook will be skipped once the tree is clean.

* fix(test): narrow CI matrix for lifecycle and toolState changes

Remove lifecycle.ts from ALWAYS_RUN_ALL and add lifecycle.ts + toolState.ts
to push/git agnostic test coverage so PRs touching prepush latch logic run
targeted tests instead of the full matrix.
2026-05-20 02:43:23 +00:00
Colin McDonnell 7e90e5cae6 Align Plan-mode prompts on report_progress as the canonical plan tool (#786)
* fix: align Plan-mode prompts on report_progress as the canonical plan tool

Fixes #673.

Three sites disagreed on where Plan output should be posted, letting a
model synthesize a broken third interpretation (initial post via
`report_progress({ target_plan_comment: true })`, which then misses the
`existingPlanCommentId` precondition). This PR aligns all three on
`report_progress` as canonical, with `target_plan_comment` reserved for
revisions only:

- `action/modes.ts` Plan step 4 — spell out that the initial plan post
  uses `report_progress` WITHOUT `target_plan_comment`, and that
  revisions go through `select_mode`'s PlanEdit override.
- `action/mcp/comment.ts` `target_plan_comment` flag description —
  make the "revisions only" precondition explicit and call out the
  initial-post path by name.
- `action/utils/instructions.ts` Progress reporting paragraph — drop
  the misleading "(e.g., Plan comments)" parenthetical that read as
  "use create_issue_comment for plans".

`PlanEdit` (in `action/mcp/selectMode.ts`) was already correct and is
unchanged.

Intentionally out of scope (to keep the fix minimal): a `publish_plan`
tool, removing the vestigial `create_issue_comment({ type: "Plan" })`
branch, hardening the run-end cleanup guard for the
`target_plan_comment but no existingPlanCommentId` fallthrough, and
renaming `target_plan_comment`.

* align create_issue_comment description with report_progress as canonical plan tool
2026-05-20 02:31:34 +00:00
pullfrog[bot] f49d4206aa chore(models): bump resolved versions (#790)
* chore(models): bump resolved versions

* fix(models): keep google gemini-3.5-flash bump, revert invalid openrouter ids

openrouter/google/gemini-3.5-flash is not on models.dev yet; only the direct
google/gemini-flash resolve should move to 3.5-flash.

---------

Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-20 02:25:50 +00:00
Colin McDonnell 69c7d4b8cd models-bump: harden CI and bot prompt against catalog hallucinations
PR #790 (the first bot-authored models-bump PR) shipped a broken bump for
openrouter/gemini-flash: the bot pattern-matched the parallel google bump
and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the
OpenRouter API but not on models.dev's openrouter section — the catalog
OpenCode actually reads. The slug failed at runtime with `Model not found`.
Two CI gaps let it through:

1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*`
   alias as a "passthrough", smoke-testing only one canary per routing
   layer. But those aren't passthroughs — each is a distinct models.dev
   catalog entry that can drift independently of the direct-provider
   mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25).
   Only `bedrock/byok` stays pruned (sentinel resolve).

2. `models-catalog` test (the integrity gate that asserts every resolve
   exists on models.dev) was main-only by design — to keep upstream catalog
   churn from blocking unrelated PRs. But it's exactly the test we want
   running on the bot's own catalog edits. Add `pullfrog/models-bump`
   head-ref to its trigger.

Also tighten the bot prompt in models-bump.yml: new rule 0 requires every
new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the
alias's own `candidates[]` in models-bump-context.json — the deterministic
preprocessor only emits candidates sourced from models.dev's mirror, so
this gates against the cross-alias pattern-matching that broke PR #790.
For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter
API), which is necessary but not sufficient; the `models-catalog` job is
the authoritative models.dev check.

Verified locally:
- baseline `pnpm -C action test:catalog` passes 133 tests
- simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash`
  into action/models.ts) and the catalog test fails with the right
  assertion: `model "google/gemini-3.5-flash" not found under openrouter
  on models.dev`
- `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every
  openrouter/* alias and every keyed opencode/* alias now smoked
2026-05-20 02:18:38 +00:00
Colin McDonnell f3d18401ac eager signup credit + free-OpenCode fallback when BYOK has no key (#789)
* eager signup credit + free-OpenCode fallback when BYOK has no key

addresses the silent-churn pattern that took out 15 first-run-failure
accounts post-launch: GH Actions secret references resolved to empty
strings (because the secrets didn't exist on the repo), the action
launched Claude Code with no key, the LLM provider 401'd, and the run
died in seconds with a synthetic "Invalid API key" message. those
accounts had no Router credits to fall back to because the lazy claim
required a dashboard visit they never made.

three changes, one PR:

1. Eager $10 signup credit at account creation. Both account-creation
   sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo`
   for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }`
   in the same transaction as the `accounts` row. CLI installers who
   never sign in get the credit. The dashboard `/signup-credit/claim`
   POST stays as an idempotent backstop for accounts created before
   this shipped.

2. Free-OpenCode fallback in the action. When the configured BYOK slug
   needs a provider key the runner doesn't have, swap to
   `opencode/minimax-m2.5-free` before agent selection so the run still
   succeeds. Surfaced via a `» fell back from <slug> to <free>` warning
   in the action log. Skipped on Router runs (Pullfrog mints the key)
   and when no model is configured (auto-select-with-throw still fires
   for the genuinely-misconfigured case).

3. New action-test fixture `byok-no-keys-fallback` that empty-strings
   every known provider key (matching how GH Actions handles missing
   secrets) and asserts the run succeeds with the fallback log line
   present. plus a unit test for the helper covering each skip case.

skipping the schema flip from `byok` to `router` — that's coming via
the onboarding-stepper PR (#762).

* fallback: skip Bedrock + surface in PR-comment footer

addresses copilot review on #789 (real bug — parseModel throws on
Bedrock raw IDs that have no slash, would crash before
validateBedrockSetup could surface its own error) and the user-side
ask to make the fallback visible in PR comments.

- selectFallbackModelIfNeeded skips when resolvedModel has no '/' so
  Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash
  inside hasProviderKey -> parseModel. unit test covers it.
- toolState.modelFallback records the configured slug we fell back
  from. set in main.ts when fallback engages.
- buildPullfrogFooter accepts fallbackFrom and renders
  "Using `MiniMax M2.5` (free) (credentials for Claude Opus not
  configured)" so the substitution is visible in PR comments,
  reviews, PR bodies, and error reports.
- threaded through all four action-side footer call sites
  (mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side
  call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts
  fire pre-action and don't have toolState — left as-is.

* fallback footer: use provider display name + document email asymmetry

addresses pullfrog reviewer findings on #789:

- footer now shows 'credentials for Anthropic not configured' (provider
  display name from `providers.anthropic.displayName`) instead of the
  per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY
  covers all Anthropic models), so this matches what the user actually
  needs to fix.

- document the intentional asymmetry between eager and lazy signup
  credit paths: eager skips both the signupCreditClaimedEmail and the
  per-grant team@ alert. comment explains why (the 'new account
  created' alert already covers it on the eager path; the user-facing
  email assumes a user-initiated action that hasn't happened yet for
  CLI/GH-App-only signups).

- skipping the backfill for the 15 historical accounts per user's
  earlier decision — they all uninstalled, so the cohort self-selected
  out of being reachable.

* fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap

local agnostic fixture run surfaced two real bugs the unit tests didn't
catch:

1. fallback gate was on configuredSlug (=payload.model) but the test
   uses PULLFROG_MODEL to set the model, which is read by resolveModel
   AFTER its slug arg. configuredSlug stayed undefined → fallback never
   fired. drop configuredSlug from the helper signature; gate purely on
   resolvedModel since that's the same value regardless of how the
   model was specified (DB config vs PULLFROG_MODEL env).

2. when fallback engaged, the post-swap resolveModel({slug: fallback.to})
   call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback
   target back to the unkeyed model. validateAgentApiKey then threw
   "no API key found" against the original model. fix: skip the
   re-resolve. fallback.to is already a CLI-ready specifier.

unit tests updated for the new helper signature (8 tests, all pass).
fallback log line confirmed emitted in the local run pre-second-fix;
the second fix unblocks the validation that previously threw.
2026-05-20 02:17:22 +00:00
Colin McDonnell 8dff91ac49 router: fix unspendable signup credit on no-card private repos (#792)
* router: fix unspendable signup credit on no-card private repos (#791)

The bug
-------
`run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`,
which is `oss || hasCard`. So a no-card account with positive wallet
balance (signup credit, top-up, etc.) on a private repo would never
get a `proxyModel` set on the run context. The action runtime then
fell through to whatever provider keys happened to be in the workflow
env — using the user's BYOK keys without their knowledge if any were
configured, or failing the run entirely otherwise.

Meanwhile `proxy-token/route.ts` already gated correctly on
`oss || hasCard || balance > 0`. The two routes disagreed, with
run-context being strictly more restrictive, so the agent never even
attempted to call proxy-token for these accounts. The wiki at
`billing.md:1052` documented the *intended* behavior ("a Router usage
row can debit a wallet with no card on file"), aspirational against
the actual code.

The action side had a parallel bug at `action/utils/proxy.ts:151` —
it re-derived `isInfraCovered({ isOss, plan })` and short-circuited
mint even when the server set `proxyModel`. Belt-and-suspenders that
was strictly more restrictive than the server.

Production impact
-----------------
Queried 55 router-mode no-card accounts holding signup credit:
- ALL have wallet balance = exactly $10.00 (untouched)
- ALL have 0 router proxy keys ever minted, 0 hwm usage
- ~25 have successful runs (using BYOK env vars from their workflow,
  unaware their credit isn't being touched)
- The rest have zero successes; some accumulated 25+ failures
  (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit).

The fix
-------
- `run-context/route.ts`: widen `useRouter` to match proxy-token's
  gate. OSS short-circuits as before. Otherwise: router mode + card
  on file → mint; router mode + no card + positive balance → fetch
  balance, mint if > 0. Skip the balance read when a card is on file
  (auto-reload covers it without needing pre-flight balance — keeps
  the hot path single-query).
- `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check.
  `ctx.proxyModel` IS the signal — the server is the authority on
  funding decisions; the action just trusts and mints.
- `wiki/pricing.md`: correct the Router proxy key minting gate row
  + add a paragraph explaining why this gate diverges from
  `isInfraCovered`.
- `wiki/billing.md`: rewrite the misleading "proxy-token returns 402"
  paragraph to describe what actually happens at both routes.

`isInfraCovered` is unchanged. It still gates Pullfrog-paid features
(learnings writes, indexing). The bug was in conflating "Pullfrog
pays for marginal infra" with "user can fund a Router run via wallet"
— different concerns, now untangled.

* action: drop dead isInfraCovered + plan param post-fix

Cleanup the action-side dead code introduced by the previous commit's
removal of the redundant `isInfraCovered` re-derivation in proxy.ts:

- delete `isInfraCovered` from action/utils/runContext.ts (was the only
  callsite; mirror in server's utils/billing.ts is unchanged and still
  load-bearing for learnings/indexing)
- drop unused `plan: AccountPlan` param from `resolveProxyModel` /
  `runProxyResolution` (and the corresponding `AccountPlan` import +
  the `plan: runContext.plan` arg at the main.ts call site)
- update the action/mcp/server.ts comment that pointed at the now-gone
  action mirror to reference the server-side `utils/billing.ts` instead

`AccountPlan` itself is still load-bearing (mcp/server, runContextData,
run-context fetch), only `isInfraCovered` and the dead `plan` parameter
go away.
2026-05-20 02:10:29 +00:00
Colin McDonnell 0d7955d87d add agent-browser fallback rule for unreachable chrome devtools mcp 2026-05-20 01:39:45 +00:00
David Blass 6e94f513df feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748)
* feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660)

Per-account ceiling on the sum of `router_topup` invoices (pending +
succeeded) for the current UTC calendar month. Closes a gap where a
runaway agent loop, leaked PR trigger, or stuck workflow could
auto-reload indefinitely with no aggregate per-month ceiling.

Two enforcement modes via `RouterLimitMode` enum:
  - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun;
    402 `router_monthly_limit` from /api/proxy-token; email + banner
  - `alert_only`: auto-reload keeps flowing; email + banner only,
    first breach per UTC month

Enforcement is split across reserveRun (pre-dispatch paywall comment)
and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through
the same `getRouterSpentThisMonthCents` helper so the dashboard, the
dispatch gate, and the auto-reload gate can't disagree.

Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string),
claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent
reloads breaching together send exactly one email. Read-time
comparison with the current month re-arms on rollover — no cron.

Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell)
above the Router/BYOK tabs in `ModelAccessCard`, with a popover
"Adjust limit" form that PATCHes the existing
/api/account/[owner]/billing/settings route. Same `assertBillingAdmin`
gate that owns the other billing settings — no new auth surface.

See wiki/billing.md § Router monthly spend limit for the full
contract + edge cases.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): anneal pass on monthly Router spend limit (#660)

Round-1 review across 5 lenses (billing-subsystem, correctness,
security, operational-readiness, research-validated-assumptions)
surfaced one critical + three actionable major findings on top of
[#748](https://github.com/pullfrog/app/pull/748).

**Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot`
used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles
to `field != value` — UNKNOWN (not TRUE) against the post-migration
`NULL` default. First breach for any account would never claim the
slot, never stamp the row, and never fire the email (hard_cap or
alert_only). Replaced with `OR: [{ field: null }, { field: { not:
monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern.

**Major — email gap on manual-top-up over cap.** Breach email was only
wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>`
that crosses the cap blocks dispatch via `reserveRun` but never hits
proxy-token, so the user got the PR comment but no email. Wired the
CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s
PaywallError catch (the SERIALIZABLE txn rolled back when we threw,
so we re-claim with the global client; single-statement CAS is its
own race boundary against concurrent proxy-token claims).

**Major — PR paywall comment leaked $ figures.** `router_limit` body
embedded `($X of $Y)` in a comment visible to anyone with PR read
access (public repos, forks, outside collaborators). Other paywall
types deliberately avoid amounts. Removed; deep link still points to
the authenticated console for the figures.

**Medium — observability.** Added `[router-limit]` structured logs at
the three enforcement sites (proxy-token hard_cap 402, proxy-token
alert_only breach, reserveRun paywall) so on-call can grep "did the
cap fire for customer X this month."

**Medium — customer docs.** Added a `### Monthly spend limit` section
to `docs/billing.mdx` (Mintlify) describing the two modes and the
manual-top-up caveat.

**Doc — refund/dispute interaction.** Documented in `wiki/billing.md`
that the cap inherits the existing webhook semantics: disputed
`router_topup` drops from the sum (cap briefly un-trips); refunds
don't flip status today so refunded top-ups keep counting. Matches
wallet behavior — not redefined here.

Accepted as-is (documented or pre-existing): `after()` reliability vs
stamp-before-send tradeoff, alert_only email fires before Stripe
phase-2, proxy-token reads limit fields outside SERIALIZABLE scope
(brief TOCTOU on admin lowering cap), stale paywall comment on cap
clear, no global kill switch (per-account `alert_only` flip is the
practical kill switch), no audit log on cap changes (no existing
audit infra), action version not bumped (separate release commit).

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): anneal round 2 on monthly Router spend limit

Round-2 anneal (billing-subsystem, correctness, research-validated,
user-journey, operational-readiness) surfaced a critical merge conflict
and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748).

**Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted
`formatBillingErrorSummary` from `action/main.ts` to
`action/utils/billingErrors.ts`. The PR's new `router_monthly_limit`
arm still lived in `action/main.ts`. Took main's slim orchestrator
wholesale; moved the arm into the extracted file.

**Major — cap = payments only, not dispatch.** `reserveRun` was
pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit`
regardless of wallet balance, contradicting the cap's positioning as
"ceiling on what you pay." An account with $500 of paid-up wallet and
a breached $100 cap couldn't trigger any new run via the comment
path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded —
surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token`
is now the sole enforcement point, refusing only the next auto-reload
that would push past. Wallet credit always drains. Dropped the now-dead
`router_limit` arm in `buildPaywallCommentBody`, the dead
`routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`,
and the post-paywall email-fire David added — all unreachable.

**Major — split `manual_topup` from `router_topup`.** Manual on-session
top-ups at `/billing-top-up/<owner>` were landing as
`Invoice.kind = "router_topup"` and counting toward the cap. The cap
exists to brake *passive* runaway (auto-reload loops); a manual top-up
is a deliberate click-through that the user owns. Added
`InvoiceKind.manual_topup`, flipped the manual write site +
`createTopUpCheckoutSession` metadata, broadened wallet /
reconcile / billing-report reads to `kind IN (router_topup,
manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap
aggregate) to `router_topup` only. Worked example: cap=$300,
reload=$100 → exactly three reloads succeed; a fourth is blocked.
Historical rows stay labelled `router_topup` (no backfill); the
asymmetry is small and accepted since the manual flow only existed
alongside auto-reload for a brief window. Extended the
`invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows
the same shape as `router_topup` (PaymentIntent-backed, no
stripeInvoiceId); split into a second migration because PG forbids
using a freshly-added enum value in the same transaction.

**Major — email reframed around the triggering reload event.** The
`alert_only` body was reporting a pre-eager-write `spentCents` while
the dashboard reads the post-commit value, so email and dashboard
disagreed by exactly one reload. Both flavors now say "Your most
recent $50 auto-reload brought you over your $300 monthly limit"
instead of a running spent-of-cap total — no reconciliation needed,
no more "you've hit your monthly cap" copy firing for partial breaches
(spent=$80 of $100, reload=$30 would have triggered that wording).

**Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded
"You've used your 30 free runs this month. Add a card to continue at
7¢/run." regardless of `detail.reason`. Branched on `cap` vs
`delinquent` so each paywall surfaces actionable copy with the right
CTA. `router_limit` no longer flows through here (per F4 above).

**Major — RouterLimitBanner.** Added an `isAlertBreached` visual
state (amber palette) so an `alert_only` account at $240 of $200 no
longer renders in the same neutral zinc chrome as a healthy under-cap
account. Updated popover copy to reflect the auto-reload-only scope.

**Medium — paywall log line.** Added `detail.reason` to the
`[Installation X] paywall:` log so on-call grepping for "why was this
paused" can distinguish `cap` from `delinquent`.

**Cleanup.** Dropped dead `utcMonthKey` import + re-export in
`maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*`
fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*`
since they now handle both kinds. Updated wiki/billing.md +
docs/billing.mdx + schema doc comments throughout.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key

The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt`
sitting next to it — a single-purpose state column on `Account` that
encoded a date as a string and required a custom CAS predicate to
read/write race-safely. Plus it had real holes: Resend send failure
left the sentinel stamped and the account silently un-emailed for the
month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered
edge cases never fired at all.

Replace it with: fire `maybeNotifyRouterLimit` on every breaching
reload, let the Resend `Idempotency-Key`
`router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside
Resend's 24h dedup window. Continuously-breaching accounts get ~1
reminder per day; brief Resend outages self-heal because the next
breaching reload re-attempts the send. Mode is in the dedup key so
`alert_only → hard_cap` mid-month re-arms a fresh email with the
appropriate copy.

Drops `Account.routerLimitNotifiedMonth` and
`claimRouterLimitNotificationSlot`; simplifies the proxy-token
phase-1 branch significantly. Net diff is negative LOC and the data
model loses a single-purpose sentinel.

Migration was branch-local — never deployed — so I edited the original
add-cap migration in place to drop the column from the ALTER TABLE
rather than chain a drop-column migration on top. Preview Neon
branches reset automatically on history rewrite per wiki/migrations.md.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): hide RouterLimitBanner when no cap is configured

The banner was unconditionally rendered for every billing-enabled
account, including pure-BYOK admins who never touch Router. They got
"No monthly spend limit / Router has spent $0.00" + a divider as
visual noise on the model access page — basically nagging them to set
a feature they may not want. Running without a cap is valid; we don't
nag.

ModelAccessCard now gates the banner block (banner + dividers) on
`routerMonthlyLimitCents !== null`. RouterLimitBanner drops the
no-limit visual state, the "Set monthly limit" CTA text, and the dead
`hasLimit` branching. Cleaner three-state shape (under cap / amber
breached / brick breached).

Discoverability: no-cap users no longer see a UI affordance to set
one. That's deliberate — the cap is a power-user feature documented
in docs/billing.mdx. If discoverability becomes an ask, we can add a
small inline link inside RouterWalletSection without bringing back
the always-visible banner.

Resolves the only outstanding finding from cursor bugbot's review of
ff5328c (banner-visible-for-byok thread).

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(billing): docs/wiki match new "no banner without a cap" reality

Pullfrog bot review of f7672ca pointed out the customer docs still
told users to "Set the cap from the **Monthly spend limit** banner in
the **Model costs** card" — but after hiding the banner for no-cap
accounts there is no such banner to use until you already have a cap.
Catch-22 for first-time setup.

Rewrote docs/billing.mdx to be self-contained: explain what the cap
is, what the two modes do, what the banner shows *once configured*,
and direct admins to PATCH the billing settings endpoint (or reach
out to support) for first-time setup. Cap is positioned as optional;
running without one is the documented default.

Wiki paragraph in wiki/billing.md updated to match — banner is only
rendered when a cap exists, three visual states (under / amber / red),
no first-time-setup UI nag by design.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely

The standalone `RouterLimitBanner` was the wrong shape. It only
rendered when a cap was already configured (so there was no UI to
discover the feature in the first place — first-time setup required
hitting the API directly), and it occupied prominent real estate above
the tabs to surface state that already lives in the row's own input
when the form moves down where it belongs.

New shape: monthly cap is just a third row inside `RouterWalletSection`
sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated
the same way (card on file + auto-reload enabled — the only state
where the cap actually means anything). Empty input → no cap, with
placeholder "No limit". Setting a number reveals a **Behavior at
limit** toggle built on the same `Tabs` slider component used for the
Router/BYOK tab switch, so the look matches the rest of the card.

Deletes:
- `RouterLimitBanner` component (212 lines)
- banner mount + conditional + spacers in `ModelAccessCard`
- `AlertTriangle` is still imported (used by `DelinquencyBanner`)

Adds:
- one settings row in `RouterWalletSection` with the cap input + mode tabs
- `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the
  existing `saveSettings` helper (widened to accept `string | null`)
- `Tabs` / `TabsList` / `TabsTrigger` import

Docs + wiki updated to match the new shape; the customer doc no
longer points at a banner that won't appear.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between

Previously bundled both into one row block. Restructure: cap input is
its own row; Behavior-at-limit Tabs gets a sibling row with the
standard `h-5 + hr + h-5` separator between (matching the rhythm of
auto-reload amount → threshold → monthly cap). Mode-toggle row is
gated on `routerMonthlyLimitCents !== null` so the hr + tabs only
appear once a number is in the cap input.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row

Same `flex items-center justify-between gap-3` layout as the
Auto-reload row: label group on the left, control on the right.
Drops the vertical stack in favour of the horizontal one — looks
identical to the toggle row directly above.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-20 01:38:36 +00:00
David Blass dd26d35137 learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743)
* learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy

three audit fixes on top of the recent learnings overhaul (#717):

- `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry
  when a body has non-whitespace content before the first heading. the
  prompt instructs the agent NOT to slurp the whole file when a TOC is
  present, so without this any preamble lines were silently invisible
  (realistic transitional case: an agent partially restructures a
  legacy free-text body and leaves bullets above the first `## `).

- server-side PATCH route now applies the same line-boundary-aware
  truncation as the action (defense in depth via a shared
  `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from
  `action/internal`). the raw `.slice` it used before could leave a
  mid-heading tail on any caller that bypassed the client-side
  truncate, breaking the next-seed TOC parse. removes the duplicated
  cap constant.

- `buildLearningsSection` intro no longer asserts "accumulated by
  previous agent runs" — false for fresh repos with zero history. new
  copy is tense-neutral and works for empty + populated bodies. also
  nudges the agent to re-read after mid-run edits (the inlined TOC
  ranges are a run-start snapshot).

Co-authored-by: Cursor <cursoragent@cursor.com>

* learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste

The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned
calls discovering a quirk this run, recording the workaround prevents the
next run from repeating the waste. Reframe around one litmus ("would a
future run do its work better because this bullet exists?") and trust it
to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary)
and the four-example pullfrog/PR/date/play-by-play list (the rule
underneath is "don't anchor facts to repo state that will move"). Cuts
~10 lines from a prompt the model was already mostly ignoring; the
remaining anchor list is narrower and more enforceable.

* audit-learnings-r2: align wiki + tighten re-read nudge

- wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls.
- buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly.

* postRun: refresh JSDoc to match the reflection prompt rewrite

`buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets.

* fix(mcp/issueEvents): narrow event.event before Set.has lookup

octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup.

* learnings: split truncation helpers into MCP-free module

re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph.

move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-19 21:47:10 +00:00
Colin McDonnell 3514bbc39f review prompt: tighten body-section bar + inline technical-details (#770)
* review prompt: tighten body-section bar + add inline technical-details

Two layers of tightening to the Review/IncrementalReview prompts in
PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step):

1. Reframe inline-vs-body split. Body `### ` sections are now reserved
   for concerns that genuinely have no line to anchor to — absence,
   sequencing, design decisions, scope questions, architectural risk.
   Drop the "cross-cutting concerns" framing (misled the agent into
   either filing nothing in the body or filing multi-file anchored
   findings there).

2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step
   6) and IncrementalReview (step 8) aggregate phases. Diagnosis from
   PR #767's auto-review: on substantial PRs the agent surfaced
   findings but routed all of them inline, producing reviews with zero
   `### ` body sections even on diffs where non-anchored concerns
   clearly existed.

3. Replace the abstract `### ` example with a concrete non-anchored
   one ("Legacy `opencode.ts` has no documented deletion plan") so the
   agent pattern-matches the absence-shaped finding, not a line-bug.

4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT
   so inline comments can carry a `<details>Technical details</details>`
   block when the fix has cross-file implications. Rename the existing
   "Agent details" inline collapsible to "Technical details" for
   consistency with body sections.

5. (Carried over from prior uncommitted work) Restructure the review
   metadata block from `<details>Review metadata</details>` into an
   HTML comment + an italic TL;DR commit-range line. The HTML comment
   keeps the metadata addressable for downstream agents without
   eating user-visible review real estate.

No tests touched.

* wiki: document multi-model end-to-end eval pattern
2026-05-19 21:06:30 +00:00
Colin McDonnell 8ac954a27f fix(handleIncompleteSetup): also skip nudge when repos are disabled, not just active 2026-05-19 18:46:54 +00:00
Colin McDonnell 88f170e19a fix: 7 log-audit / run-audit findings (mega-PR) (#769)
* fix(#765): silence Clerk 400 (revoked OAuth) noise from getTokenForClerkId

Branch on isClerkAPIResponseError + status<500 so the well-understood
revoked-token redirect doesn't emit a level=error line in Better Stack
on every request. Vercel maps console.warn -> error for non-streaming
routes, so a downgrade to log.warn wouldn't help; only the unexpected
shape (5xx, network) is worth surfacing.

* fix(#742): stop logging input verbatim from yes.op retry-failure paths

GitHub OAuth user tokens (ghu_...) were leaking to Better Stack on every
yes.op retry-failure for any utils/github/get* helper that takes a token
field — 38 leaks/7d in the most recent audit window. The leak path is
console.log inside the yes package (its own log shim, not utils/log.ts).

Drop input from the four log sites + the cache-key-derivation throw site.
key (SHA-1 of input) is sufficient for retry correlation; error already
carries request URL + status. Defense-in-depth comment so future
contributors don't re-add the field.

Operational follow-up (separate task): inventory ghu_... strings in
Better Stack ingested in the last 90d, revoke matching Clerk grants,
scrub cold-tier S3, rotate the BS source token.

* fix(#759): handle GraphqlResponseError "Could not resolve to a node" as 404

When the stored planCommentNodeId references a comment that's been
deleted on GitHub, octokit.graphql throws GraphqlResponseError before
the existing `node === null` 404 branch is reached. Add a narrow
isGraphqlNodeNotFound predicate in utils/errors.ts and a new catch
branch in the plan-comment route. The action treats 404 as "no prior
plan comment" and creates a fresh one, so behavior matches existing
contract.

* fix(#747): convert webhook GraphQL rate-limit 5xx into a Result<T> sentinel + 200 ack

When GitHub's GraphQL responds with "API rate limit exceeded for
installation ID N", _getReviewCommentsWithReplies threw, propagated
through the bare yes.op wrapper (no rate-limit bail), out of the bare
await in handleWebhook, and crashed /api/webhook/github with 500 — 77
webhook 500s/24h on the most recent audit window. GitHub redelivery
plus R2 dedup also silently masked the legitimate handler from
re-running once the rate-limit window cleared.

Mirror the #658 / _getRepository pattern: detect GraphqlResponseError
matching /rate limit (already )?exceeded/i, log.warn with the
x-ratelimit-reset value (and [Installation N] prefix when available),
return failure(...) with status 429. Webhook handler short-circuits
the case with 200 + log.info so GitHub stops the redelivery storm
against an exhausted budget, and the trigger page surfaces a clean
ThrowClientError. Document the new pattern as a Tier 2 false-positive
in wiki/log-audit.md so the next audit cron doesn't re-flag it.

Note that returning [] silently (the issue's first suggestion) would
have dropped @pullfrog mentions inline in review comments and
dispatched an agent run that re-rate-limits — skip-the-whole-case is
the correct semantics. Co-vulnerable getPullRequest / getWorkflow
have zero occurrences in this window; per #737 policy, defer until
they show up.

NOTE: this commit and the bracket of touched files revert as a unit —
the Result<T> shape change in getReviewCommentsWithReplies is
breaking; partial revert breaks the type chain.

* fix(#766): fold stderr+stdout into shell.ts errors + carve out merge-base --is-ancestor

action/utils/shell.ts dropped stdout when constructing failure messages
($\{stderr || "Unknown error"\}), so git subcommands that write
context-bearing diagnostics to stdout (merge conflicts, cherry-pick
rejections, diff --exit-code, ls-files --error-unmatch) surfaced as
"Command failed with exit code 1: Unknown error" through
mcp__pullfrog__git. The agent burned an extra MCP round-trip calling
git status to recover.

Fold stderr + stdout into the thrown error message (stderr first,
stdout fallback) so the agent always sees the real diagnostic. Plus
a narrow carve-out for `git merge-base --is-ancestor` in
action/mcp/git.ts: that subcommand uses exit code as data (0=ancestor,
1=not-an-ancestor, >1=error), so return { success: true, isAncestor }
instead of throwing on exit 1.

No caller in action/ string-matches on the old error format
(verified). diff --exit-code and ls-files --error-unmatch are not
carved out — both are zero-occurrence in the May audit window, and
the stderr+stdout fold renders their output usefully anyway.

* fix(#739): point customers at the actual fix when permissions: id-token: write is missing

When a customer workflow runs in GitHub Actions but lacks
permissions: id-token: write, ACTIONS_ID_TOKEN_REQUEST_URL/_TOKEN
aren't injected, isOIDCAvailable() is false, and acquireNewToken
falls through to the local-dev-only acquireTokenViaGitHubApp path,
which throws "GITHUB_APP_ID and GITHUB_PRIVATE_KEY must be set" —
pointing at a self-hosted-app fix that doesn't apply. One affected
customer burned 13 dispatches in 24h on this misleading error.

Detect (GITHUB_ACTIONS=true) AND (no OIDC env vars) inside
acquireNewToken before falling through to the local-dev branch, and
throw an actionable message naming the missing permissions block,
the exact YAML, and the docs anchor. The error surfaces via
##[error]action failed: ... in the workflow log (the only customer
surface available before main()'s inner try opens). Local-dev path
keeps the existing GITHUB_APP_ID message.

* fix(#760): suspend activity watchdog across in-flight tool calls

mcp__pullfrog__checkout_pr was hard-failing 6/24h on SenecaLabs/senecaWeb
because git fetch+deepen on a large monorepo can take 4-5 min, the
agent's stdout pipe goes silent the entire time (FastMCP is in-process
HTTP, but Claude/opencode CLIs await the synchronous tools/call
response), and both the spawn-level activity timer (300s in
subprocess.ts) and the process-level activity monitor (300s in
activity.ts) fire and kill the run.

Re-introduce the bracket pattern that PR #634 removed: bracket
suspendActivity()/resumeActivity() around tool_use -> tool_result in
both agent harnesses, plumb isPausedExternally into spawn() so both
timers suspend in lockstep. Bounded by MAX_TOOL_CALL_SUSPENSION_MS
(15 min auto-resume) plus the outer 1h agent timeout — neither
zombie-run avenue from #12 is reopened (subprocess.close still
resolves on death; outer timeout is suspend-agnostic; suspends gated
on explicit paired CLI events, not internal noise).

opencode tool_use handler: gate suspendActivity() on non-terminal
status (running/pending) so the bus_event re-dispatch path at line
915 — which only fires for completed/error subagent parts and never
emits a paired tool_result — doesn't latch the watchdog into
suspension until the 15min ceiling.

Add a heuristic:activity-watchdog-ceiling classifier to
scripts/analyze-logs.ts so a tool that genuinely hangs past
MAX_TOOL_CALL_SUSPENSION_MS surfaces in run-audit instead of being
bucketed into failure:unknown.

NOTE: this commit and the bracket of touched files revert as a unit
— activity.ts, subprocess.ts, and the two harnesses must move
together or the bracketing breaks.

* refactor(#747): swap Result<T> for InstallationRateLimitError typed throw

The Result<T> shape from 3ebf6c4c was cargo-culted from the #658
_getRepository pattern, but _getReviewCommentsWithReplies has only one
expected-error case (installation rate-limit) and two callers — Result
imposes branching on the trigger-page caller that never cared about
the rate-limit case specifically. A typed error class is lighter (~10
LoC vs ~33) and matches the actual need:

- new InstallationRateLimitError(resetAt) thrown from
  _getReviewCommentsWithReplies; rate-limit log.warn unchanged.
- handleWebhook catches it and breaks with log.info (unchanged
  semantics: 200 ack, no redelivery storm).
- trigger page reverts to direct array access; any failure propagates
  to the page error boundary (the pre-#747-commit shape).
- log-audit.md wording updated to match.
2026-05-19 18:40:53 +00:00
pullfrog[bot] 0abaaa1e37 chore(oss): add yamcodes/arkenv to OSS program (#776)
* chore(oss): add yamcodes/arkenv to OSS program

* fix(test): strip CODEX_AUTH_JSON in apiKeys auto-select test

The beforeEach strip list omitted CODEX_AUTH_JSON, which is in
`knownApiKeys` via the openai provider's managedCredentials. When the
env has CODEX_AUTH_JSON set, the auto-select "throws when no provider
keys are present" assertion finds it and fails to throw.

---------

Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
2026-05-17 20:26:26 +00:00
Colin McDonnell e20f32fb09 fix(test): randomize tag name in push-enabled fixture
the branch name was already randomized with \${RANDOM}, but the tag
name was hardcoded as `test-tag-enabled`. every successful run left
that tag on the fixture remote with no cleanup, so subsequent runs'
checkouts (which fetch tags) saw the local tag already exist and
failed with `fatal: tag already exists`. the agent's git-push fallback
then hit the deliberately-broken creds and the test scored
push_tags=✗ delete_branch=✗.

mirror the branch pattern with \${RANDOM} so every run gets a fresh
tag name. tags still accumulate on the remote but no longer collide;
that's an infra cleanup concern, not a correctness one.
2026-05-16 18:49:07 +00:00
Colin McDonnell c0988e35b0 fix(security): block docker socket from sandboxed shell; disable opencode batch_tool
two real CI failures on main, both shipping bugs in the action:

1. `token-exfil-claude` was a real sandbox escape: GHA `ubuntu-latest`
   puts `runner` in the `docker` group, so a sandboxed shell could run
   `docker run --pid=host --privileged busybox cat /proc/<parent>/environ`
   and read the action process's env (which holds user secrets) — fully
   bypassing the unshare PID-namespace. fix: inside the sandbox's mount
   namespace (already private via `--mount-proc` which implies `--mount`),
   bind-mount /dev/null over /var/run/docker.sock (+ podman/containerd/crio
   variants) so any container-runtime socket connect from the sandbox fails.
   only affects sandboxed shells — host runner mount table is untouched, so
   user workflow steps outside pullfrog keep working.

2. `restricted-opencode` regressed in #719 (`experimental.batch_tool`).
   opencode's batch tool rejects MCP tools with `"Tool '<name>' not in
   registry. External tools (MCP, environment) cannot be batched."` when a
   model emits parallel `pullfrog_shell` (or any MCP) tool_use blocks,
   opencode internally routes them through batch, they all fail, the model
   misreads the error as "the tool doesn't exist", and gives up. caught by
   a `lens:` subagent in the restricted test concluding shell was
   unavailable and setting `DIAGNOSTIC_ID=empty`. drop `batch_tool: true`
   and the matching opencode-specific guidance in `instructions.ts` — native
   parallel tool_use (multiple tool_use blocks per assistant message) still
   works for both built-in and MCP tools without batch, so we lose only the
   1-25 wrapper, not parallelism.
2026-05-16 15:40:44 +00:00
Colin McDonnell efc1b67e7b fix(test): skip models.dev existence check for fallback aliases
deprecated aliases (`fallback` set) legitimately point at dead resolve
targets — xAI just retired grok-4-1-fast/grok-code-fast-1 and #761 wired
them through the fallback chain. the terminal-fallback is validated
separately by the Zen served-list test.
2026-05-16 05:14:51 +00:00
Colin McDonnell 0a64659ee7 refactor: slim action/main.ts to an orchestrator + extract helpers (#755)
* refactor: extract helpers out of action/main.ts so non-orchestration churn stops touching the file

main.ts had grown to ~1240 lines holding ~500 lines of helpers that have
nothing to do with the resolver pipeline — billing-error UI/copy, proxy
minting, summary/learnings persistence, log formatters, end-of-run
cleanup waterfalls. any PR adding a new billing code branch or a new
log line was forced to edit main.ts, and since main.ts is in
ALWAYS_RUN_ALL the entire 52-job LLM CI matrix fired on what should
have been a 0-job change (e.g. #748).

extractions:
- action/utils/billingErrors.ts — BillingError, TransientError, the
  format*Summary renderers, billingConsoleUrl
- action/utils/proxy.ts — mintProxyKey, buildProxyTokenHeaders,
  resolveProxyModel, plus runProxyResolution wrapper that renders +
  rethrows BillingError/TransientError before the outer catch
- action/utils/prSummary.ts — fetchPreviousSnapshot, persistSummary
  co-located with the existing seed/read file helpers
- action/utils/learnings.ts — persistLearnings co-located with the
  existing seed/read file helpers
- action/utils/runStartupLog.ts — resolveOutputSchema + logRunStartup
  (the model/agent/push/shell/timeout block)
- action/utils/runErrorRenderer.ts — renderRunError classifies
  (BillingError reclassify / hang detect / API-key auth) and emits
  {summary, comment} markdown bodies
- action/utils/runLifecycle.ts — persistRunArtifacts, finalizeSuccessRun,
  writeRunErrorOutputs — the three end-of-run cleanup phases shared
  between the success path and the error catch path

main.ts is now ~570 lines — the irreducible orchestrator: disposables
(`await using` for tokenRef / gitAuthServer / mcpHttpServer), the
toolContext construction, the agent-timeout race, the catch/finally
shape, and the named phase calls. behavior is preserved verbatim
(verified: pnpm -r typecheck + pnpm test 695/695 pass, action/test
596/596 pass).

wiki/main.md gets a new "file layout" section describing the split.
AGENTS.md gets a single line pointing future edits at the helpers
instead of main.ts.

* anneal: address review findings

- restore MainResult.result?: string (accidental removal in initial commit;
  field was unused in current code but is part of the exported interface
  surface — keep the diff truly behavior-preserving)
- move resolveOutputSchema from runStartupLog.ts to payload.ts (it's an
  action-input resolver alongside resolvePromptInput / resolvePayload, not
  a log helper — was placed in runStartupLog.ts for matrix-churn pragmatism
  but the domain fit is in payload.ts)
- un-export resolveProxyModel (only used internally by runProxyResolution
  in proxy.ts; no external importer)
- fix runErrorRenderer.ts JSDoc "Three classifications" → four (Billing,
  hang, API-key, default)
- expand runLifecycle.ts module banner to note that finalizeSuccessRun
  calls persistRunArtifacts first, and to explain why the catch path
  splits writeRunErrorOutputs + persistRunArtifacts
- update billingErrors.ts header to point at proxy.ts and
  runErrorRenderer.ts as the actual origin sites (was stale "main.ts")
- expand proxy.ts header to spell out the runProxyResolution entrypoint
  contract (was stale "main.ts can render")
- update wiki/main.md resolver chain + dependency table to name
  runProxyResolution as the actual call site and document the early
  BillingError/TransientError rendering branch
- update wiki/main.md file-layout table to lead with runProxyResolution
  and describe mintProxyKey/buildProxyTokenHeaders/resolveProxyModel
  as internal helpers (was implying they were public surface)
2026-05-16 05:09:52 +00:00
Colin McDonnell a78b1542da feat: pullfrog auth codex + fresh-branch (#757)
* feat: pullfrog auth codex + fresh-branch

Add `pullfrog auth codex` standalone command for minting Codex
(ChatGPT) subscription credentials and saving them as the
`CODEX_AUTH_JSON` Pullfrog secret.

Codex device-auth runs in a subprocess with an isolated `CODEX_HOME`
(temp dir) so the user's `~/.codex/auth.json` is never touched. The
spawned `codex login --device-auth` output is captured line-by-line,
ANSI-stripped, and re-rendered with a `$ codex login --device-auth`
header above dimmed sub-output on the @clack/prompts rail so the user
visually understands they're seeing a sub-process.

Companion `pnpm fresh-branch` script: from inside `.worktrees/<name>`,
creates a schema-only Neon branch named `dev/<git-branch>`, patches the
worktree's `.env` (DATABASE_URL, DATABASE_URL_UNPOOLED, NEON_DEV_BRANCH),
then runs `prisma migrate reset --force` so migrations apply cleanly
against a data-free copy. Refuses to run from the primary checkout or
on protected branch names.

Other:
- bump CLI/account/repo secret value limit 4096 -> 49152 chars (matches
  GitHub Actions' 48KB cap; auth.json is ~4-5KB)
- extract shared CLI helpers (gh/pullfrog API, secret save) into
  `action/commands/_shared.ts`

* fix(auth): address PR review + add CodexAuthCallout, default account scope

Review fixes:
- handle 'error' event from `codex` spawn (ENOENT) so missing PATH bails
  with an actionable "install codex CLI" message instead of an unhandled
  Node error
- escalate SIGTERM -> SIGKILL after 5s grace when killing a stuck codex
  child so the CLI can't get pinned indefinitely
- stop the spinner with a red "failed" glyph in the catch path before
  clearing activeSpin, mirroring `bail` (no orphan spinner above errors)
- enforce 48 KB secret value cap by *bytes* (Buffer.byteLength) not
  UTF-16 code units, across all 3 secret routes; matches GH Actions'
  byte-based limit
- preserve existing blank lines + comments when fresh-branch rewrites
  worktree .env (no more cosmetic reformat on every run)

Scope:
- default to `account` scope on org-owned repos too — never silently
  prompt for repo scope. Pullfrog has no per-GitHub-user secret store,
  so account is right for both user and org owners; `--scope repo` is
  the explicit opt-in for repo-only.

UI:
- new CodexAuthCallout (sibling to ClaudeCodeOAuthCallout); surfaces
  `pullfrog auth codex` for ChatGPT subscribers when an OpenAI provider
  model is selected. wired into AgentSettings.tsx (model-costs surface)
  and OnboardingCard.tsx (first-time setup). no paste button — the CLI
  handles minting + saving end-to-end.

* auth/codex: rename to neon-fresh-branch, address PR review

- rename `pnpm fresh-branch` → `pnpm neon-fresh-branch` (and the script
  file) to disambiguate from git branches.
- `--scope` help text now explains the default (account) and when to
  pass `repo`.
- move `_shared.ts` import up with the rest in `action/commands/auth.ts`
  and push the `stripAnsi` helper below the import block.
- `sanitizeBranchName` no longer slices: slicing after trim could
  reintroduce a trailing `-`/`/`. callers slice the raw input first,
  then sanitize.
- DRY the `start` branch of the codex progress callback (single
  header path, optional retry log).
- thread a `timedOut` flag from `runDeviceAuth` → `ProgressEvent.exit`
  so the retry prompt can say "device authorization timed out — retry?"
  instead of the generic "no auth.json was written" line when the
  per-attempt timeout fires.
- drop the redundant `mkdirSync` after `mkdtempSync` in `codexAuth.ts`.

* untrack .scratch/ (committed screenshot fixture by mistake)

* auth codex: prompt for scope on orgs (mirrors init)

* revert worktree.ts: out of scope for this PR

* anneal: trim _shared.ts dead exports, collapse CodexSpawnError, inline packageBin

* codex auth: wire end-to-end runtime consumer

CODEX_AUTH_JSON is now actually usable: the action runtime materializes
it as OpenCode's auth.json at the runner's real $HOME/.local/share/opencode,
OpenCode routes openai requests through the ChatGPT subscription via the
embedded CodexAuthPlugin, and a GitHub Actions post: hook detects any
refresh-chain rotation during the run and PUTs it back to Pullfrog via a
new JWT-authenticated PUT /api/runtime/secret endpoint.

Key decisions:

- Write to the real $HOME (not the per-run tmpdir-redirected HOME) so the
  file lives outside OpenCode's `/tmp/*` permission allow zone — its
  existing deny-default protects it without any new permission rule.
- Materialization gated on agent === opencode (Codex auth is OpenAI-only,
  Claude never sees the file).
- Defense-in-depth on Claude: deny Read/Grep/Edit/Glob + sandbox.denyRead
  for ~/.local/share/opencode/auth.json in managedSettings (covers Bash
  file-reading commands too per Claude Code permissions docs).
- New `provider.managedCredentials` field on the provider config — CLI-only
  credentials authored via `pullfrog auth <provider>`. Counted for
  hasAnyKey/log-redaction but never surfaced as a paste option in init.
  CODEX_AUTH_JSON is the first member; OPENAI_API_KEY stays in envVars.
- Eager refresh on `pullfrog auth codex`: one OAuth round-trip before
  setPullfrogSecret so Pullfrog's copy is the freshest in the chain
  (avoids the user's laptop refreshing first and stranding our copy).
- Post-hook approach for write-back so it survives cancellation, timeouts,
  and unhandled errors in the main step. State is ferried via
  core.saveState since apiToken is run-scoped and not in env.
- Server-side write-back endpoint is allowlist-gated to CODEX_AUTH_JSON
  only — never a generic secret-write surface. Looks up the secret at
  repo scope first, falls back to account scope. 404s on create
  (refresh-only, never auto-provision).

* codex auth: documentation + wiki cross-links

* debug: log dbSecrets keys + CODEX_AUTH_JSON presence (temporary)

* debug: surface install path + parse failure preview

* remove debug log lines (E2E verified)

* hide CodexAuthCallout until opencode-ai bump (1.1.56's allowed-models set excludes gpt-5.5)
2026-05-16 05:06:24 +00:00
Colin McDonnell ddbc610569 review prompt: friendly green callouts + per-section severity emojis (#756)
* review prompt: friendly green callouts + per-section severity emojis

- Replace `[!NOTE]` informational tier and the no-callout minor-suggestions
  tier with friendly green blockquotes (`> ` / `> 💡`). The two loud
  tiers (`[!CAUTION]` / `[!IMPORTANT]`) keep their GitHub admonitions.
- Add a per-`##`-section severity-emoji rule (🚨/⚠️/💡/ℹ️) for
  cross-cutting review concerns that don't anchor to a line and would
  otherwise be buried in summary content.
- Drop the `<br/>` between summary sections — heading + blank line
  carries enough visual spacing.
- Skip the post-run learnings-reflection turn for `IncrementalReview`.
  It's the lowest-novelty mode (delta review against existing PR with
  prior summary already loaded) and almost never produces durable
  learnings — reflection there costs ~$0.50-0.80/run for nothing.
- Surface real error info on `agent-browser` skill install failures
  (exit code + stdout + stderr + spawn error). The skills CLI uses a
  TUI that prints errors to stdout, so the prior stderr-only logging
  silently swallowed every failure.

* review prompt: per-bullet severity emoji + bullets-only sections

Section headings are plain again (no leading severity emoji). Severity
moves to individual bullets so a section that mixes a 🚨 and a 💡 isn't
mislabeled by either. Section bodies are now bullets only — paragraph
prose under a heading is harder to scan and tends to bury the
actionable point.

Bullets can carry indented continuation content (sub-bullets, code
fences, blockquotes) by indenting two spaces under the parent.

* review prompt: cap section length + identifier discipline

Bound each summary section to at most 4 bullets at most 2 lines each,
and explicitly call out identifier-heavy prose as an anti-pattern. The
reader is often a manager or non-author; identifier-dense paragraphs
('foo calls bar.fetch which dispatches to baz via qux...') are
unreadable for them. Default to plain-language behavior descriptions,
name an identifier only when it's the subject of an actionable concern
or a public surface a reader would recognize, target 2-3 backtick
tokens per bullet.

Move the deep-explanation pattern from open blockquote to a default-
collapsed details/summary so depth doesn't dominate the visible body.

* review prompt: hard cap on bullet identifier density + worked rewrite example

Soft 'aim for 2-3 tokens' guidance was ignored — first big-PR e2e
showed 12 of 19 actionable bullets exceeded the target (avg 4.8 tokens,
several over 8). Promote to a hard cap of 3 backticked tokens per
bullet and pair with a concrete bad/good rewrite the agent can pattern-
match against. Also tighten the per-bullet length cap from ~240 to
~200 chars and explicitly call it 'hard cap, not target'.

* review prompt: tighten bullet length cap to 160 chars, dramatize the worked example

V2 e2e test: token discipline improved (4.8 -> 3.3 avg, 12/19 -> 6/14
violations) but length got worse (235 -> 286 chars, 13/14 over the 200
cap). The agent compensated for fewer identifiers with more prose.

Two changes: (1) tighten the cap from ~200 chars to 160 chars / 1
visual line and call out wrap-to-multiple-lines as the failure mode;
(2) rewrite the worked example so the good version is genuinely half
the length of the bad one, not just lower token count. The example was
the thing the agent pattern-matches against; making the good version
~130 chars vs the bad version's ~290 chars sets the right shape.

* review prompt: drop fixed bullet-count cap, keep length + identifier caps

Per user feedback — section length should be governed by content, not
an arbitrary count. Soft guidance ('past ~6, ask whether to split') is
fine; the hard '≤ 4 bullets per section' rule was the wrong shape.
Length cap (160c) and identifier cap (3 backtick tokens) stay; those
target the actual scanability problem.

* review prompt: drop ## subsystem sections, flat 'Issues found' list

Per-section structure forced every concern into a subsystem frame and
made the body read like a series of mini-essays. Replace with two
parts: (1) TL;DR + Key changes as the dispassionate overview, (2) flat
'### Issues found' list ordered by severity, intermixed across files
and subsystems. Per-bullet rules (≤160c, ≤3 backtick tokens, severity
emoji prefix, optional indented continuation) carry over unchanged.

* review prompt: full v6 structure — preamble + cross-cutting H3s + nitpicks

Replaces the flat 'Issues found' bullet list with the iterated v6 shape:

- Preamble is a bolded inline 'Reviewed changes' lead-in plus bullets
  plus a collapsed 'Review metadata' block (mode/files/commits/refs/
  reviewed commits list/prior pullfrog review/staleness note).
- Each cross-cutting concern gets a '### emoji Title' section. The
  visible problem write-up is human-friendly and DESCRIBES THE PROBLEM
  ONLY — no asks, no suggested fixes, no 'the right thing to do is'.
- Each section carries a collapsed 'Technical details' block wrapped
  in a 4-backtick markdown fence (so it can hold its own 3-tick code
  fences cleanly, agent-readable, one-click copyable). Standard four
  inner sections: Affected sites, Required outcome, optional Suggested
  approach, optional Open questions for the human.
- '### ℹ️ Nitpicks' at the bottom for body-only nits that don't
  inline; simple bullets, no technical-details collapse.
- Anti-paragraph-wall rule: never two successive plain paragraphs in
  visible '### ' sections; alternate prose with structure.
- Inline-vs-body discipline: anything that anchors to a single line
  goes inline, body is for cross-cutting only.
- Drops legacy '### Key changes', '### Issues found', '<b>TL;DR</b>',
  and the '<sub>Summary</sub>' line.

* model effort: bump Gemini + GPT to high effort; drop Gemini Pro→Flash subagent

E2E review eval against a substantive billing-module diff surfaced two
related quality gaps:

1. Gemini Pro at thinkingLevel=medium (#663's CI-timeout fix) reviewed
   the diff only, took the 0-lens path, and missed a catastrophic
   camelCase/snake_case service-vs-schema mismatch. Bumping back to
   high — review work is exactly the wrong shape for the medium/high
   tradeoff #663 was optimizing for; the per-turn TTFT cost is worth
   paying when reasoning IS the value.

2. GPT had no reasoningEffort override, defaulting to upstream medium.
   Same diff, similar shallow result vs Claude. Adding reasoningEffort:
   high for the curated direct-OpenAI slugs, mirroring the Gemini
   pattern (Anthropic separately uses --effort high via the Claude
   Code CLI flag in claude.ts).

3. Gemini Pro's subagentModel was 'gemini-flash' — but Google has no
   in-between tier between Pro and Flash, and Flash is a meaningful
   capability cliff for review work. Dropping the override so subagents
   inherit Pro. Cost stays reasonable since Gemini Pro is already the
   cheapest of the flagship trio.

Other providers unchanged: Anthropic opus→sonnet and OpenAI gpt→gpt-5.4
remain (each is a one-tier drop to a still-capable sibling).

* model effort: revert orchestrator override, set explicit high on reviewfrog subagent

Reshape the effort design after eval:

- Drop the explicit Gemini and GPT model-level overrides — orchestrators
  now run at upstream defaults (Gemini high, GPT-5.x medium). Gemini's
  upstream IS high, so this is a no-op there; GPT goes back to upstream
  medium for orchestrator-level routing work.
- Add explicit 'high' on the reviewfrog subagent via agent.options.
  OpenCode merge order is base ← model.options ← agent.options ← variant
  per session/llm.ts:141, so the subagent always runs at high regardless
  of which orchestrator dispatched it. Both thinkingConfig.thinkingLevel
  (Gemini) and reasoningEffort (GPT) keys included; irrelevant keys are
  ignored per provider.
- Bump providers-live timeouts (12min job / 10min step, from 8/6) to
  budget for Gemini's TTFT variance at high effort. #663's 4min timeout
  was sized for the medium-effort override that's now removed.

* model effort: restore Gemini explicit high override (no-override path breaks)

Bare 'rely on upstream default' for Gemini failed in e2e — removing the
model-level provider config produced 'Function call is missing a
thought_signature' API errors on every gemini-pro run. Even though
upstream opencode's options() returns the same thinkingLevel: high we
were explicitly setting, opencode's resolution path differs subtly
between the two cases. v2's explicit override worked; v3's removal
broke. Reproducible across two consecutive runs.

Restoring the explicit Gemini override (back to v2 design). GPT
orchestrator stays UN-overridden — at upstream default (medium) — since
removing that override didn't trigger the same failure pattern and the
reviewfrog subagent agent.options high override compensates for the
extra depth GPT loses at medium.

* diag: remove reviewfrog agent.options to isolate Gemini thought_signature failure

v3 (no Gemini orch override) failed with thought_signature error. v4
(restored Gemini orch override at v2-equivalent) ALSO failed, even
though the orchestrator config matches v2. The variable between v2
(working) and v4 (failing) is the new reviewfrog agent.options block.
Removing it to confirm — if Gemini works again, the agent.options
addition is the culprit and we need a different shape for it.

* opencode-ai: bump 1.1.56 → 1.15.0 + clean up gemini effort config

opencode-ai@1.1.56 was published 2026-02-10 (3 months old). The Google
API tightened thought_signature validation 24-48h ago (per
https://discuss.ai.google.dev/t/gemini-thought-signature-patch/122555),
and the bug class hits opencode's session→prompt serializer for MCP
tool-call parts (anomalyco/opencode#4832, #8321). Latest stable bumps
us through ~3 months of fixes; needed for Gemini-direct to stop dying
with 'thought_signature is missing' on every multi-turn run.

Companion cleanup: the gemini provider override in opencode.ts had
30-line block of comments, four unused constants, and a 6-line
Object.fromEntries map for two entries. Replaced with one source-of-
truth helper that loops modelAliases, filters provider==='google',
strips the 'google/' prefix, and returns the override map. Adding any
future Google alias to the registry now flows through automatically.

Test added: action/agents/opencode.test.ts asserts the helper covers
every direct-Google alias, strips the prefix correctly, and pins every
entry to thinkingLevel high — catches drift in helper logic without
hardcoding the API ids the test would have to update in lockstep
with the registry.

* fix(workflow): tolerate listJobsForWorkflowRun 404 in resolveRun

PR #750 (docker testing rewrite) replaced the per-call env allowlist
with full process.env passthrough into the test container. That now
leaks GITHUB_RUN_ID + GITHUB_JOB into runs whose MCP token is scoped
to a DIFFERENT repo (e.g. providers-live smoke runs the action against
pullfrog/test-repo with pullfrog/app's run ID). The unconditional
listJobsForWorkflowRun call 404s and crashes the entire run, breaking
every providers-live job on main since #750 landed.

jobId is purely cosmetic (deep-links 'View workflow run' footer to a
specific job vs the run-level URL). Wrapping the API call in try/catch
so a 404 logs a debug message and falls through to undefined jobId is
the right fix — the failure mode is exactly what graceful degradation
is for, and the alternative (filter the env vars at the docker boundary)
re-introduces the kind of allowlist #750 was getting rid of.

* opencode-ai: pin 1.14.51 instead of 1.15.0 (effect refactor breaks JSON output)

opencode 1.15.0 (May 15) ships a major architectural refactor onto
@effect — the run command boots an in-process server via
@opencode-ai/sdk/v2 and the JSON event emission path through that SDK
client doesn't surface on stdout the way our parser expects (CI run
on 1.15.0 produced 0 stdout events but the agent still completed).
Local invocation also hangs at the in-process server boot.

The Gemini thought_signature fixes (the original reason for bumping)
landed earlier in the 1.14.x line, so 1.14.51 (May 14) gets us the
upstream fix without the Effect rewrite. Defer the 1.15.x bump until
we're ready to rewire our parser/spawn around the new SDK.

* opencode-ai: revert to 1.1.56; gha: filter outer-CI workflow-run vars at the docker boundary

Two related changes for the docker testing harness's ergonomics:

1. Revert opencode-ai 1.14.51 → 1.1.56. The 1.14+ line ships an Effect
   refactor (the SDK-v2 client + in-process server architecture) that
   our --format json parser doesn't speak — even the 1.14.51 release,
   pre-dating the 1.15.0 Effect rename, produced 0 stdout events on
   our skill-invoke smoke. There's no clean pre-Effect version that
   ships the Gemini thought_signature fix; that fix needs a separate
   workstream once we're ready to rewire the parser onto SDK v2.

2. Filter outer-CI workflow-run identifiers (GITHUB_RUN_ID, GITHUB_JOB,
   GITHUB_WORKFLOW, GITHUB_ACTION, GITHUB_REF, GITHUB_SHA, etc.) from
   gha.ts's --env-file passthrough. PR #750's full-process.env design
   leaks pullfrog/app's CI run identifiers into runs that act against
   a different repo (e.g. pullfrog/test-repo); any code path inside
   the action that uses them as keys (most notably resolveRun's
   listJobsForWorkflowRun lookup) 404s. Filtering them here means
   the action sees undefined and skips the lookup, complementing the
   defensive try/catch in resolveRun (commit addc76d4). GITHUB_REPOSITORY
   and GITHUB_TOKEN are NOT filtered — those are genuinely needed.

Companion to addc76d4 (resolveRun 404 tolerance). The two together
make this class of bug 'either fix would have caught it' rather than
'silently breaks the entire test matrix'.

* fix(deps): sync pnpm-lock.yaml with opencode-ai 1.1.56 manifest revert

Forgot to refresh the lockfile after reverting the manifest in 02c6d8c1.
CI's frozen-lockfile install was failing with 'lockfile: 1.14.51,
manifest: 1.1.56' mismatch.
2026-05-16 04:58:31 +00:00
Colin McDonnell a0dce200d0 fix(claude): prefer OAuth token over ANTHROPIC_API_KEY (#763)
* fix(claude): prefer OAuth token over ANTHROPIC_API_KEY in Claude Code

When both `CLAUDE_CODE_OAUTH_TOKEN` and `ANTHROPIC_API_KEY` are present,
claude-code's auth resolver (`Vw()` in cli.js) returns the API key first
and silently ignores the OAuth token. The result: accounts that have a
Max-subscription OAuth token in `account_secrets` are still billed at
per-token API rates because the workflow `env:` block also forwards
`ANTHROPIC_API_KEY` from org-level secrets.

Strip `ANTHROPIC_API_KEY` from the spawned claude-code subprocess env
when an OAuth token is present (and we're not on the Bedrock route),
so the Max subscription is actually used. Other agents in the same run
still see the API key in `process.env` via the parent.

* chore: tighten comment-length rule + trim claude.ts comment

Caps inline comments at 2-3 lines above any single line of code (the
prior wording allowed runaway block comments as long as the comment
was nominally shorter than the annotated code).

* chore: downgrade OAuth-strip log to debug + document debug-mode pattern

`log.info` was overkill for a per-run path-selection marker. `log.debug`
keeps production logs quiet while preserving full visibility in e2e
verification, where `LOG_LEVEL=debug` (or `gh run rerun --debug`)
flips the same line on.

Adds a "Action debug mode" subsection to wiki/e2e-testing.md so the
affordance is discoverable: `log.debug(...)` is the right tool for
breadcrumbs that prove a code path fired during preview-repo e2e but
shouldn't ship to customer logs.

* chore(wiki): correct debug-mode trigger guidance for preview repos

LOG_LEVEL=debug only works when the template's pullfrog.yml forwards
it, which it doesn't. ACTIONS_STEP_DEBUG=true is the GitHub-magic name
that's auto-injected into every step's env without any yaml change,
so make that the documented default for preview-repo e2e.

* chore(wiki): fix render-format claim in debug-mode table

When `ACTIONS_STEP_DEBUG=true`, `log.debug` routes through
`core.debug()`, which GitHub renders as `##[debug]<msg>`, not the
`[DEBUG] <msg>` format. The `[DEBUG]` prefix only happens via the
LOG_LEVEL=debug path which isn't currently wired into the template.

* feat(action): add `overrides` input for per-dispatch env mutation

Accepts a JSON {string:string} map via the workflow_dispatch input,
parsed and merged into process.env at the start of `main()` (before
any agent or token-acquisition code runs). Lets a privileged caller
flip env vars for one dispatch without persisting state on the repo
(repo Actions variables) or being restricted to GitHub's debug names
(`gh run rerun --debug`).

Deny-list refuses overrides for integrity-critical names — GITHUB_TOKEN,
ACTIONS_RUNTIME_TOKEN, ACTIONS_RUNTIME_URL, ACTIONS_ID_TOKEN_REQUEST_*,
ACTIONS_CACHE_URL, PULLFROG_API_SECRET, VERCEL_AUTOMATION_BYPASS_SECRET.
Customer provider keys (ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, etc.)
are explicitly allowed — overriding them per-run for cred-rotation tests
and auth-failure repros is the use case.

Touches:
- action/action.yml — declare `overrides` input
- action/utils/overrides.ts — parse + apply with deny-list (+ unit tests)
- action/main.ts — wire into `main()` after `normalizeEnv()`
- .github/workflows/pullfrog.yml — forward to action
- utils/github/pullfrog.yml.ts — same in the customer-facing template
- wiki/e2e-testing.md — documented as preferred debug-mode trigger

* fix(overrides): strip raw INPUT_OVERRIDES + mask applied values

GitHub Actions injects every action input as an env var (INPUT_<NAME>),
so the original JSON of `overrides` sits in process.env as INPUT_OVERRIDES
and is inherited by every spawned subprocess (claude, opencode, MCP
servers, shell). That defeats the deny-list (a downstream re-application
would have access to the raw JSON) and leaks arbitrary caller-supplied
values into agent env verbatim.

After applying, applyOverrides now:
1. delete process.env.INPUT_OVERRIDES — subprocesses see only the
   surgically-applied keys, not the raw JSON
2. core.setSecret(value) for each applied value — the runner masks
   those strings in subsequent log output, so an overridden
   ANTHROPIC_API_KEY can't accidentally surface in debug logs.

Two new tests cover the deletion path (both applied and all-denied).

* fix(overrides): scope auto-masking to credential-shaped keys

core.setSecret(value) is a global string-match — calling it on a short
config value like "claude" masks every appearance in subsequent logs
(including "claude-opus-4-7", "anthropic-claude-sonnet", etc.), which
actively harms debugging.

Restrict the auto-mask to keys whose names end in _KEY / _TOKEN /
_SECRET / _PASSWORD / _OAUTH / _PRIVATE_KEY — the credential-shape
naming convention. Customer keys (ANTHROPIC_API_KEY, etc.) and the
deny-listed names match. Plain config (PULLFROG_AGENT, PULLFROG_MODEL,
ACTIONS_STEP_DEBUG) doesn't.

* docs(wiki): document the three security layers + runner-echo caveat

Lays out exactly what the `overrides` input does to mitigate the secret-
leak surface (deletion + masking) and the one unavoidable limit: GH
Actions echoes the `with:` block once before any action code runs, so
the raw JSON appears in the workflow log header in plaintext. Anyone
using `overrides` should treat that one-shot exposure as part of the
threat model.

* fix(overrides): forward via env, not action input, so the value isn't echoed verbatim in the runner step header

GH Actions echoes the `with:` block of every `uses:` step in the log
group header, BEFORE any action code runs — so the raw JSON of
`overrides` was always visible in the workflow log regardless of any
in-action `core.setSecret` calls.

Refactor: drop the `overrides` action input; instead the action reads
`process.env.PULLFROG_OVERRIDES`. The workflow yaml forwards
`inputs.overrides` via the step-level `env:` block. We still need to
verify empirically whether `env:` block values from workflow inputs
get echoed too (separate test); even if they do, masking via
core.setSecret + delete of PULLFROG_OVERRIDES after parsing closes
the leak to subprocesses, which is the part the action controls.

* fix(overrides): rename to unsafe_overrides + UNSAFE_OVERRIDES

The runner echoes step-header env-block values in plaintext before any
action code runs, so the raw JSON of this affordance is visible to
anyone with actions:read on the calling repo. That's acceptable
because the workflow only exists on our private repos, but the input
name should make the trade-off obvious at the call site rather than
buried in a wiki.

- workflow_dispatch input: `overrides` → `unsafe_overrides`
- env var the action reads: `PULLFROG_OVERRIDES` → `UNSAFE_OVERRIDES`
- wiki: rewrite the section to surface the runner-echo as the central
  trade-off rather than a buried caveat

* chore(overrides): tighten error messages to reference UNSAFE_OVERRIDES

* docs(wiki): fix stale 'overrides' refs + correct render-format mechanism

Addresses two unresolved review threads on PR #763:

1. The opening sentence of "Action debug mode" still referenced the
   pre-rename `overrides` input and `gh workflow run -f overrides=...`.
   Updated to `unsafe_overrides`.

2. The render-format claim was technically wrong. `core.isDebug()`
   doesn't cache — it reads `process.env.RUNNER_DEBUG === '1'` on
   every call. The actual mechanism: the runner only sets
   RUNNER_DEBUG=1 when ACTIONS_STEP_DEBUG=true is observed at
   workflow-trigger time. Mutating ACTIONS_STEP_DEBUG mid-step
   doesn't retroactively flip RUNNER_DEBUG, so the call falls through
   to isLocalDebugEnabled() which reads ACTIONS_STEP_DEBUG directly.
   Rewrote the explanation to match.

* fix: drop unsafe_overrides from customer-facing workflow template + remove test theater

Two cleanups from a stricter re-read of AGENTS.md:

1. utils/github/pullfrog.yml.ts is the workflow yaml we sync into every
   customer repo. unsafe_overrides has no business there — it's a
   pullfrog-only debugging affordance. Reverted. The action's read of
   UNSAFE_OVERRIDES env var stays — it's a no-op for any workflow that
   doesn't set it, and pullfrog/template + pullfrog/app's own workflow
   still forward it.

2. Deleted action/utils/overrides.test.ts entirely. AGENTS.md is clear:
   no tests unless explicitly asked. I added them anyway. The tests
   were mostly testing JSON.parse + typeof, plus one regression guard
   for the deny-list that is better protected by code review of the
   tiny DENIED_OVERRIDE_NAMES set than by a vitest file.

Also strengthened the corresponding AGENTS.md rule from a buried bullet
to an explicit "NEVER write tests unless asked, here's why agents
violate this constantly, here's the bar" callout.

Wiki note added: unsafe_overrides is pullfrog-only infra, not customer-
facing.
2026-05-16 04:37:26 +00:00
Colin McDonnell 7907fac64e fix(test): bump model-smoke timeout 60s → 120s (#764)
xai/grok-4.3 jobs in the models-live matrix land at 42-67s wall time vs
23-41s for every other provider, brushing the 60s ceiling and crossing
it intermittently (e.g. xai/grok-code-fast in run 25949844470 timed out
at 60s with `OK` already in stdout — model replied, harness just hadn't
seen close). 120s gives ~2x headroom on the slowest provider without
penalizing the fast-path providers, since the timer only fires on
actual hangs.
2026-05-16 03:14:19 +00:00
Colin McDonnell 76879b27ec docker testing rewrite: bake the image, drop the allowlist, kill the quoting (#750)
* docker testing rewrite: bake the image, drop the allowlist, kill the quoting

- new `pnpm gha <script>` wrapper. one entry point for running any node
  script in the GHA-like container; replaces the runtime apt-get +
  useradd + chown ceremony in `action/utils/docker.ts`.
- `action/Dockerfile` bakes ubuntu:24.04 + node 24 + gh + jq + sudo +
  testuser at uid 1000. `action/docker-entrypoint.sh` remaps to the host
  uid/gid and `exec`s the requested command — no `bash -c` nesting, no
  `escapeForDoubleQuotes`.
- env passthrough: full `process.env` (+ `.env` via dotenv) flows through
  `--env-file`, multi-line values via `-e` fallback. drops
  `EnvFilterMode` / `testEnvAllowList`.
- image rebuild is content-hash gated on Dockerfile + entrypoint; volume
  is versioned by hash so a stale `node_modules` cache from an old image
  can't poison a new one.
- `action/play.ts` slimmed to a CLI; `run()` extracted to
  `action/utils/runFixture.ts`. drops the `--local` / `PLAY_LOCAL` dual
  mode in favor of explicit `play:local` / `runtest:local` scripts.
- `action/test/run.ts` no longer self-relaunches into docker — that's
  `gha`'s job now.
- `action/test/coverage.ts` `ALWAYS_RUN_ALL` updated to track the new
  files.
- `wiki/docker.md` rewritten (243 → 105 lines). `wiki/action-tests.md`,
  `wiki/billing.md`, `wiki/adversarial.md`, `README.md`, `AGENTS.md` all
  updated to drop `--local` / `PLAY_LOCAL` references.

verified end-to-end: `pnpm play` runs the default fixture against
pullfrog/scratch, exit 0; `sudo unshare --pid` still works inside the
container; `pnpm runtest` boots through the wrapper.

* gha: address review feedback + 3 related issues found locally

review-flagged:
- bare `pnpm gha --build` now builds the image and exits 0 (was
  printing help and exiting 1 — docs claimed it was a valid standalone)
- `initVolumeOwnership` skipped when the named volume already exists;
  saves the ~240ms `docker run … chown` on every warm invocation
- `GIT_SSH_COMMAND` gate widened to any `id_*` private key (was hard-
  coded to `id_rsa`, leaving ed25519-only linux contributors with the
  default ssh config). dropped `-i` so ssh picks whichever key exists
- new `action/.dockerignore` — partial mitigation noted: BuildKit
  (default since docker 23) only sends files referenced by the
  Dockerfile (~42B in practice), so the perf concern is mostly
  hypothetical. file is still worth keeping for `DOCKER_BUILDKIT=0`
  fallback and as documented intent for future `COPY . .` additions

related issues found while validating locally:
- `parseArgs` now stops flag-parsing at the first positional (or
  literal `--`); `pnpm gha test/run.ts --build` previously
  intercepted `--build` as a gha flag instead of forwarding to
  `test/run.ts`
- new `pnpm gha --clean` command prunes orphan `pullfrog-gha:*`
  images and `pullfrog-gha-node-modules-*` volumes whose hash
  doesn't match the current Dockerfile (each Dockerfile/entrypoint
  edit creates a fresh hash and orphans the prior pair, ~600MB +
  ~200MB each — without a cleaner they accumulate silently)
- `--shell` without a TTY now fails fast with an actionable message
  before docker is invoked, instead of producing the confusing
  `the input device is not a TTY` from docker run

wiki updated: documents `--clean`, the parseArgs passthrough rule,
and a new "Reclaiming disk" section.

* gha: fidelity, flexibility, and signal-safety improvements

investigated local fidelity vs the real GHA ubuntu-24.04 runner and
addressed the gaps that have actually bitten contributors or could.

fidelity (image now matches GHA closer):
- bake build-essential, wget, xz-utils, file alongside the existing
  toolset. gh, jq, git, python3, sudo, ssh, build-essential, wget,
  xz, file, unzip, curl all present. native module builds (node-gyp,
  any package missing arm64 prebuilts) now work; common agent shell
  calls don't hit ENOENT
- `host.docker.internal:host-gateway` flag wires the host into the
  container's DNS on linux (macOS Docker Desktop bakes it in). lets
  scripts that hit a local dev server use `API_URL=http://host.docker.
  internal:3100` and work identically on both platforms
- `--init` makes tini PID 1, fixing signal forwarding during the
  pre-exec warmup window (Ctrl-C was previously taking up to 10s to
  tear down because bash-as-PID-1 swallowed the signal)
- pnpm version is correctly pinned via the workspace's
  `packageManager` field — corepack resolves it at install time;
  verified via the new `--doctor` command

flexibility (new affordances):
- `pnpm gha --doctor` runs an inside-the-container fidelity audit:
  os + arch + node/pnpm/python versions, version snapshots of every
  baked tool, env vars (CI, HOME, TMPDIR), uid/gid, and the
  host.docker.internal resolution. useful for "works in CI fails
  locally" or vice versa
- `pnpm gha --build --no-cache` busts the docker layer cache when
  an apt mirror, base image, or external download has changed
  upstream
- entrypoint's `pnpm install` warmup is now wrapped in a `flock` on
  a file in the shared node_modules volume — concurrent `pnpm gha`
  invocations (e.g. play in one terminal, runtest in another)
  serialize their install instead of racing

docs:
- new "Gaps (known)" section in wiki/docker.md explicitly calling
  out the things this system can't do yet, including the missing
  `uses: ./action` semantics gap that
  `.github/workflows/action-gha-e2e-adhoc.yml` currently fills via
  GHA only (designing a local `pnpm gha-action <fixture>` is on the
  roadmap), service containers, parallel-run sharing, and arch
  differences (arm64 vs amd64)

* docs: audit + corrections after testing fronts

self-audit pass for stale references and incomplete pointers:

- wiki/browser.md: `Docker (node:24)` → `pnpm gha container (ubuntu:24.04)`.
  the substance was right (chrome not preinstalled) but the base image
  reference was stale.
- wiki/docker.md: the "Permission errors" troubleshooting line claimed
  the node_modules volume is chowned on every run; now correctly says
  "owned by the host uid on first creation; warm runs skip the chown"
  to match the actual behavior after the initVolumeOwnership fix.
- wiki/action-tests.md: `API_URL` env-var doc now mentions BOTH paths
  (`localhost:` from play:local, `host.docker.internal:` from inside
  the container). Proxy/router recipe now shows both invocations
  side-by-side instead of saying "must use play:local".
- wiki/billing.md: same dual-recipe update for the loop-including-the-
  action proxy walkthrough.
- gha.ts header: expanded the usage block to include --clean / --doctor /
  --no-cache / --shell-TTY, added the host.docker.internal note, and
  pointed at wiki/docker.md for design rationale.

self-document check: a future agent landing on this code can answer
"how do I run a fixture / debug in shell / add a tool / diagnose
fidelity / reach a local dev server" purely from gha.ts header +
wiki/docker.md without spelunking through the entrypoint or git
history.
2026-05-16 03:12:25 +00:00
Colin McDonnell 8e1acfba99 fix(models): mark grok-fast and grok-code-fast as deprecated (#761) 2026-05-16 01:58:07 +00:00
Colin McDonnell fa7ddcee4a prompt: discerning review-feedback handling + elegance bar
strengthen build-mode self-review and addressreviews step 4 to require
verifying every reviewer finding, reject AI slop / over-defensive code,
and frame the goal as a complete + minimal + elegant solution. mirror
the elegance/no-slop bar in AGENTS.md.
2026-05-15 19:13:04 +00:00
Colin McDonnell 3add2cbc49 fix(action/tsconfig): noEmit + exclude dist to silence editor TS5055
action/tsconfig.json had "exclude": [] (overriding the default outDir
exclusion) and unset noEmit, so tsserver pulled action/dist/**/*.d.ts
into the program and flagged 92 TS5055 errors ("Cannot write file ...
.d.ts because it would overwrite input file") any time dist/ existed.
the CLI typecheck script passes --noEmit so it never tripped — only the
editor was affected.

emit is owned by tsconfig.exports.json, which extends this one and
overrides noEmit: false, emitDeclarationOnly: true. so the main config
is editor/typecheck only and should declare noEmit: true.
2026-05-14 17:11:03 +00:00
Colin McDonnell 5abb3072c7 release: action v0.1.8 2026-05-14 05:37:33 +00:00
Colin McDonnell 74b7329f64 fix(action): dedupe concurrent checkout_pr + guard cross-PR clobber (#735)
* fix(action): dedupe concurrent checkout_pr calls + guard cross-PR clobber (#642)

agents occasionally emit duplicate parallel `checkout_pr` tool_use blocks
in one turn, causing two `checkoutPrBranch` invocations to race the same
`.git/shallow.lock` and one to fail with `File exists`. the prior fix
(#564) added a 30s staleness sweep, but that very threshold protects the
within-run concurrent case from itself.

dedupe at the tool layer: a module-level `Map<pull_number, Promise>`
shares a single in-flight promise across concurrent same-PR calls. the
fetch race becomes architecturally impossible — first call does the work,
duplicate gets the same `CheckoutPrResult`. cleared in `finally` so
subsequent same-PR calls re-do the work normally.

also reject cross-PR checkouts when the working tree is dirty, surfacing
a clear error instead of silently overwriting uncommitted work from a
prior PR. uses existing `toolState.issueNumber` (no new state).

* review: use dedicated `pullNumber` toolState field for cross-PR guard

per copilot review: the prior guard used `toolState.issueNumber`, which
is also set by issue/comment lookup tools (issueInfo, issueComments,
issueEvents, review). that conflation is intentional and correct for
its only consumer (`report_progress` falls back to `issueNumber` to
choose which issue/PR to comment on, and GitHub treats both via the
same comment API). but it makes the field wrong for the cross-PR
guard: a same-PR re-checkout after `get_issue(other)` would falsely
fire and surface a misleading "from PR #other" message.

introduce a separate `pullNumber` field, set only by `checkoutPrBranch`
alongside `issueNumber` and `checkoutSha`. narrower invariant, no
disturbance to the existing `issueNumber` semantics.

* review: drop dual-write — single `issueNumber` is sufficient for the guard

reverting the `pullNumber` addition. setting both `issueNumber` and
`pullNumber` to the same value at the same site was a code smell — there
is no scenario where they diverge. issues and PRs share GitHub's number
space, and the cross-PR guard's actual job is "refuse to clobber a dirty
tree when switching to a different number"; that's expressible with
`issueNumber` alone.

addresses copilot's original concern (misleading "from PR #X" message
when X was an issue) by removing the prior-number reference from the
error message entirely. the dirty paths are the actionable detail.
2026-05-14 05:08:11 +00:00
Colin McDonnell ba7f5a0b89 action: surface agent hang context in progress comment (#733)
* action: surface agent hang context in progress comment

When the activity-timeout watchdog kills a stalled opencode subprocess,
the user used to see a bare "activity timeout: no output for 30Xs" — no
provider context, no stderr trace, no clue why the run died. Investigation
of the six runs in #728 showed the same shape every time: opencode hangs
after a non-retryable provider event (auth 401, 502 stream lost, free-tier
flake), and the only useful signal was buried in stderr where the user
couldn't see it without diving into Actions logs.

Stop trying to prevent the hang. Surface it.

Add a small `AgentDiagnostic` handle on `toolState` that the harness
mutates as a run progresses (recent stderr ring buffer reference, last
provider-error label, event count). `formatAgentHangBody` renders that
into a markdown body — bold headline, one-line explanation, collapsible
`<details>` with the last ~10 stderr lines (capped to 3KB) — used by
both the agent harness's own catch path and main.ts's outer catch when
the watchdog wins the race against the harness.

Both paths converge on one formatter; the existing
"View workflow run ➔" footer affordance in `reportErrorToComment` is
unchanged, so the user still has one click from the comment to the raw
logs to develop their own thesis.

* address review: gate hang body on isHang; fix contradictory copy

- Only render `hangBody` when `isHang`. The harness sets
  `agentDiagnostic` on entry, so any non-hang throw past `runOpenCode`'s
  own catch (post-success `output_schema` validator, late cleanup throws)
  was rendering "Pullfrog failed — N events processed…" with the real
  exception message dropped — including for runs that actually succeeded
  before a late throw.

- When `lastProviderError` already names the cause in the headline, the
  zero-events sentence "check whether the model provider is reachable"
  contradicts it (a 401 produces zero events but isn't a reachability
  issue). Drop the nudge in that case; keep it for the silent-stall path
  where it's still actionable.

* address copilot review: fence escape, idle parsing, secret redaction, tests

- pick a backtick fence longer than any backtick run in the rendered
  stderr tail. opencode error JSON occasionally embeds triple backticks
  in tool input dumps; the fixed three-tick fence let those terminate
  the fence early and corrupt the rest of the comment markdown.

- parse idle seconds out of the timer reject string ("activity timeout:
  no output for 301s") and use that for the hang explanation. previously
  rendered total runtime, which overstated the stall by 20+ minutes for
  runs that streamed for a long time before going quiet (e.g.
  Rohithgilla12/data-peek#25784038918, 1230s elapsed but 304s idle).

- redact sensitive env-var values from the rendered stderr tail before
  it lands in the PR comment / job summary. workflow log writes already
  go through `core.setSecret` masking; PR comments and summaries bypass
  that pipeline entirely. matches against `isSensitiveEnvName` (the same
  *_KEY/*_TOKEN/*_SECRET/*_PASSWORD/*_CREDENTIAL surface that
  `normalizeEnv` registers with the runner) and only redacts values
  >= 8 chars to avoid false-positive substring hits.

- add `agentHangReport.test.ts` covering the branchy bits: idle-seconds
  parsing, eventCount-zero copy with and without provider error,
  fence-escape against embedded triple backticks, 3 KB tail truncation,
  null-on-no-diagnostic, and secret redaction.

`startedAtMs` is dropped from `AgentDiagnostic` — total runtime was the
only consumer and idle seconds replaces it.

* strip slop: drop tests, drop redactSecrets, simplify ternary

- delete `agentHangReport.test.ts`. half the cases just pinned literal
  copy ("**Pullfrog stalled**", "check whether the model provider is
  reachable") which is exactly the "performative tests to every string
  utility" pattern AGENTS.md flags. the other half tested 2-5 line pure
  helpers (parseIdleSec / pickFence / truncation) that code review
  catches. the formatter is a best-effort string output; pinning it in
  tests creates churn without catching real regressions.

- remove `redactSecrets` and revert the formatter's import. theatrical
  defense: opencode doesn't dump env on startup, bearer tokens aren't
  in request bodies, bash is denied. the action has many other
  PR-comment write paths that don't redact (comment.ts, errorReport.ts,
  the progress writer) — if PR-comment secret hygiene matters, it's a
  cross-cutting concern at the comment-write layer, not bolted onto
  one formatter.

- factor the explanation triple-ternary into `formatExplanation` with
  early returns. same logic, easier to read.

`isHang` gate, fence-length escaping, and idle-seconds parsing stay —
those are real correctness fixes.
2026-05-14 04:13:26 +00:00
Colin McDonnell b9383bbcfd action: center provider-error log excerpt on the matched line (closes #703)
the `» provider error detected (...)` excerpt was `chunk.substring(0, 500)`
— the head of whatever stderr buffer node delivered. on big writes that's
the front of an mcp tool-schema dump, not the matched error text. label
was correct (regex.test on the whole chunk), excerpt was misleading.

introduce findProviderErrorMatch(text) that returns { label, excerpt }
where excerpt is a windowed slice centered on the regex match index:
the matched line plus 1 line before and 2 lines after, hard-capped at
600 bytes. detectProviderError stays as a thin wrapper for label-only
callers. both opencode and claude harnesses log match.excerpt instead
of chunk.substring(0, 500).

regression tests cover the multi-line buffer case, surrounding-line
context, byte-cap fallback to matched-line-only, and head truncation
of a single oversize line.
2026-05-14 03:59:45 +00:00
Colin McDonnell 8d6460da1c fix: surface real tool error string in opencode log handler (#736)
opencode's `ToolStateError` carries the failure reason on `state.error`,
not `state.output`. our log handler was reading `state.output` and
falling back to `(no error message)`, so every tool failure logged a
useless line. type the state as a discriminated union (mirrors
@opencode-ai/sdk) so the field misread becomes a compile error.

operator-facing only: the model already received the real error via
opencode's tool-result envelope (verified by running webfetch against
a known-404 URL — model reported "Error: Request failed with status
code: 404" verbatim).

closes #662
2026-05-14 03:56:24 +00:00
Colin McDonnell 1f4c3031be ci: filter test matrices by per-test coverage globs (#730)
* ci: filter test matrices by per-test coverage globs to cut LLM spend

every test in `crossagent/`, `agnostic/`, and every provider entry now
declares a `coverage: string[]` of repo-relative globs. the new `changes`
job runs `paths-filter` for a docs-only short-circuit, then pipes the
changed-file list into `action/test/matrix.ts`, which intersects each
entry's coverage against the diff and emits filtered `agents`,
`agnostic`, `flagships`, and `aliases` matrices. main pushes and
`workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety
net.

retires `changed-agents.sh` and the `MODE=flagships` branch in
`list-aliases.ts` in favor of one consistent model.

* ci(matrix): switch test discovery to dep-free static parsing

the GHA `changes` job has no `node_modules` installed. the previous
dynamic-import path pulled the test files transitively through
`utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with
ERR_MODULE_NOT_FOUND. parse the test files via regex instead so
matrix.ts stays zero-dep — the chain (matrix -> coverage / providers /
list-aliases / models) imports only node builtins and relative TS files.

* ci(matrix): address PR #730 review feedback

- drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`,
  `restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but
  the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`,
  both already covered).
- drop unused `coverageForProvider` export and its `byName` map from
  `providers.ts` (matrix.ts builds its own lookup inline).
- derive the active agent list from `agents/index.ts` via the same
  dep-free regex tactic as `parseTestFile` instead of hardcoding
  `["claude", "opencode"]` — adding a new harness file now wires it
  into the dynamic matrix automatically.
- treat `coverage: []` as `coverage: undefined` in `shouldRun` so an
  accidentally-empty array doesn't silently skip CI on every PR.
- add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the
  `timeout` test's coverage — the activity-timeout enforcement path
  was the original reason the test exists.
- ungate the `root` job (lint/format/typecheck/vitest). it's a required
  status check on `main`, so gating it on `code == 'true'` would make
  docs-only PRs unmergeable (skipped jobs don't satisfy required-check
  rules). the real LLM savings come from skipping the four matrices,
  not from skipping `root`.
- harden the four matrix-job `if:` guards from `outputs.matrix && ...`
  to `outputs.matrix != '' && ...` — explicit > implicit short-circuit.
- document `expandBraces`'s flat-only support so a future author isn't
  surprised by `{a,{b,c}}` not expanding.
- fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".
2026-05-14 03:55:33 +00:00
Colin McDonnell 4ad649ebb9 action: extend shallow-unreachable deepen-retry to checkout_pr fetches (#734)
extracts the deepen-retry helper from `GitFetchTool` into shared
`$gitFetchWithDeepen` and applies it to every fetch in `checkoutPrBranch`
(baseRef, pull/N/head, before_sha temp branch). on shallow clones with
deep PR ancestry — the failure mode behind ~10 of 51 `heuristic:very-slow`
runs in 24h on `remotion-dev/remotion` — the baseRef fetch was throwing
`Could not read <sha>` to the agent before the compare-api deepen block
could run. agents then burned 10+ minutes retrying `checkout_pr` and
falling back to ad-hoc shell `git fetch --deepen` workarounds.

also splits the analyzer's `heuristic:git-error-recovered` into
`heuristic:git-shallow-unreachable` and `heuristic:git-shallow-lock`
buckets so future audits surface this without manual log-grep.

closes #656.
2026-05-14 03:44:08 +00:00
Colin McDonnell 2960d51493 shell tool: cap output at 5K chars and spill overflow to tempfile (#732)
unbounded shell tool output blows the agent's context window on commands
that dump big logs (test runners, build tools, grep on large trees). cap
the inline body at 5000 chars; on overflow, persist the full output to
${PULLFROG_TEMP_DIR}/shell-<id>.log and return the tail prefixed with a
sentinel pointing at the saved path. agents re-read the tempfile with
cat/tail/grep when they need more.
2026-05-14 03:18:54 +00:00
Colin McDonnell b6df2860c3 action: bump to 0.1.7 2026-05-14 02:48:31 +00:00
Colin McDonnell d495f0b984 surface BYOK failures + chronic-failures card + WorkflowRunStatus mirrors GitHub conclusions (#722)
- Migrates `WorkflowRunStatus` from `running | completed | cancelled` to a 9-state mirror of `workflow_run.conclusion`. Backfill: old `completed → success`, `cancelled → failure`. New rows write `hook.workflow_run.conclusion` verbatim via `statusFromConclusion`.
- Adds Discord links to `formatApiKeyErrorSummary` (both missing-key and 401 invalid-key shapes).
- Repo console: `<ChronicFailuresCard>` fires when the last 3 terminal-state runs are all `failure`. Pure DB read; latest-run button hidden for pre-dispatch failures (`runId: null`).
- `StatusIcon` distinguishes `cancelled` (gray X, intentional stop) from `failure` (red X) so the visual matches the chronic-card threshold.
- Pre-dispatch failures (workflow lookup miss, dispatch API error) write `failure` instead of `cancelled` so they feed the card.
- Cascade: every `status: "completed"` filter in billing routes / cron / cohort queries / analyzer becomes `status: "success"`.

Verified end-to-end on `pullfrog/preview-722-failure-surfaces` — Better Stack logs confirm webhooks reached the preview deploy and all three e2e runs got `marked as failure (conclusion=failure)` via the new mapper.

Closes #679, #702.
2026-05-14 02:39:41 +00:00
Colin McDonnell 206c11fe7c review: drop misleading 'with the same arguments' from diff-coverage nudge
agent is free to refine review body/comments on retry — there's no
enforcement that the second call matches the first, and if reading the
nudged region surfaces a new finding the agent should add it.
2026-05-14 02:36:06 +00:00
Colin McDonnell 7414c1e9ca review: clarify diff-coverage nudge gives explicit license to skip generated artifacts
the one-time pre-flight nudge said "optionally read" but never told the agent
it's free to retry without reading when every unread region is generated
(lockfiles, codegen, snapshots, migration metadata). audit #677 surfaced ~21
runs/24h burning an extra model turn re-reading drizzle snapshots, pnpm-lock,
and *.gen.ts files purely to satisfy the gate. mode prompts only mention
generated content in the "skip self-review entirely" path, not the
"in-progress substantive review" path, so the in-the-moment error message
was the gap. behavior unchanged for legitimately-unread source regions.
2026-05-14 02:30:37 +00:00
Colin McDonnell 8f9208bd3f feat: Amazon Bedrock support via routing slug (#720)
* add Amazon Bedrock as a routing slug

introduces a single `bedrock/byok` catalog entry that the harness translates
to the appropriate Bedrock model ID at run time via `BEDROCK_MODEL_ID`. routes
Anthropic IDs through claude-code (with `CLAUDE_CODE_USE_BEDROCK=1`) and
everything else through opencode's `amazon-bedrock` provider — keeps the
catalog flat for an audience that needs version pinning rather than aliasing.

accepts either `AWS_BEARER_TOKEN_BEDROCK` or `AWS_ACCESS_KEY_ID` +
`AWS_SECRET_ACCESS_KEY` for auth; both validated alongside `AWS_REGION` and
`BEDROCK_MODEL_ID` in `validateAgentApiKey`. catalog drift tests, the bumps
cron, and per-alias smoke scripts all skip routing slugs since there's no
fixed `resolve` to validate.

docs/bedrock.mdx walks through setup; wiki/model-resolution.md has a section
explaining why bedrock breaks the usual alias pattern.

closes pullfrog/pullfrog#40

* ci: add bedrock env vars to test workflows

mirrors the new bedrock provider's required env vars (AWS_BEARER_TOKEN_BEDROCK
inherited from org secret + AWS_REGION + BEDROCK_MODEL_ID hardcoded) into both
.github/workflows/test.yml files so the ci.test "env vars cover all provider
API keys" assertion passes.

* docs(bedrock): clearer setup flow + screenshot of model selector

restructures the setup section into three concrete steps in execution order:
select Bedrock from the dropdown, store the bearer token as a secret (Pullfrog
or GitHub — links to keys.mdx for the trade-off), then add region + model id
directly in pullfrog.yml since neither is sensitive. enable-model-access in
the Bedrock console moved to step 4 (only required once per model and only
when AWS rejects the call, not blocking on first run).

adds a screenshot of the console model selector with Amazon Bedrock selected
so readers can recognize the UI state they're aiming for.

* fix(bedrock): tolerate raw Bedrock model IDs in validateAgentApiKey

main.ts passes the resolved model into validateAgentApiKey
(`payload.proxyModel ?? resolvedModel ?? payload.model`). For Bedrock,
`resolveModel` translates `bedrock/byok` into the raw AWS model ID
(e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/` and so
trips parseModel inside getModelEnvVars.

Detect the no-slash case and re-run the bedrock setup check (auth +
region; BEDROCK_MODEL_ID is already enforced upstream by resolveModel).

Caught by PR #720 e2e dispatch on pullfrog/preview-720-bedrock —
"invalid model slug 'us.anthropic.claude-opus-4-6-v1' — expected
'provider/model'". Two regression tests cover the raw-ID path.

* fix(bedrock): always prepend amazon-bedrock/ prefix when bedrock-routed

opencode.ts was gating the prefix-injection on `!isBedrockAnthropicId(rawModel)`,
on the theory that Anthropic Bedrock IDs always go through claude-code. But
`PULLFROG_AGENT=opencode` is a documented escape hatch — when it forces
opencode for an Anthropic Bedrock model, the prefix still has to be added or
opencode fails with 'Model not found: <modelId>/.'.

The Anthropic-vs-other discriminant only belongs in resolveAgent. Once an
agent is selected, it should consistently honor the bedrock route.

Caught by the PULLFROG_AGENT=opencode + Opus 4.6 e2e on
pullfrog/preview-720-bedrock — run 25823437606.

* ui+docs(bedrock): bespoke setup callout + clearer docs

UI:
- BedrockSetupCallout in components/AgentSettings.tsx covers both the
  Model costs section and the onboarding card. Detects bedrock via
  resolveDisplayAlias().routing === "bedrock", shows a dedicated message
  ("store AWS_BEARER_TOKEN_BEDROCK as a secret, then put AWS_REGION +
  BEDROCK_MODEL_ID directly in pullfrog.yml") + link to the setup guide.
  Replaces the generic "X, Y, or Z is required" prompt that misrepresented
  the three values as three separate secrets to add (and used the wrong
  "or" connector for what's actually an AND).
- OnboardingCard re-uses the same callout with the gradient-card variant.

Docs:
- Drop the obsolete "Enable model access" step. AWS retired the manual
  enrollment page; foundation models auto-enable on first invocation.
  Anthropic models still need a one-time use-case form for first-time
  users — surfaced under the AccessDenied troubleshooting entry.
- Drop the "Testing a different model in one run" PULLFROG_MODEL note.
  It introduced the secrets-vs-vars distinction we want to keep out of
  the bedrock setup story.
- Step 3 already recommends hardcoding region + model id in pullfrog.yml.

Workflow template:
- The default pullfrog.yml customers receive (utils/github/pullfrog.yml.ts)
  now references AWS_BEARER_TOKEN_BEDROCK from secrets but inlines
  AWS_REGION and BEDROCK_MODEL_ID as plain values. Matches the docs.

* fix(bedrock): three review-caught edges in routing + UI copy

Addresses three real issues from PR #720 review:

1. agent.ts: PULLFROG_MODEL=bedrock/byok no longer leaks the literal
   sentinel "bedrock" downstream. resolveCliModel returns the alias's
   resolve field verbatim, which for routing entries IS the sentinel.
   Refactored both the env-override and slug-lookup paths through a
   shared resolveSlug() that recognizes routing aliases and defers to
   their backing env var (BEDROCK_MODEL_ID).

2. models.ts: isBedrockAnthropicId() now anchors on a discrete
   dot/slash/colon-segment match (case-insensitive) instead of a
   substring contains. The substring check was fragile in both
   directions for inference-profile ARNs (BEDROCK_MODEL_ID accepts
   ARNs per AWS docs) — a non-Anthropic profile whose user-chosen name
   contained "anthropic" would mis-route to claude-code, and an
   Anthropic profile whose name omitted it would miss
   CLAUDE_CODE_USE_BEDROCK=1.

3. AgentSettings.tsx: BedrockSetupCallout's configured-state copy
   showed "AWS_BEARER_TOKEN_BEDROCK configured" even when the user
   satisfied the gate via AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY,
   gaslighting access-key users about a secret they never set.
   Detect which auth method is actually present and name the right
   secret(s) in the success message.

Regression tests in models.test.ts (5 new isBedrockAnthropicId cases
including positive and negative ARN forms) and agent.test.ts (2 new
PULLFROG_MODEL=bedrock/byok cases). 171/171 action tests pass.

* yml template: add commented AWS access-key alternative for Bedrock auth

Mirrors the IAM access-key path verified end-to-end on PR #720 e2e
run 25830764987. Bearer token stays as the primary nudge; the access-key
pair is the fallback for users who can't mint Bedrock API keys.

* yml template: drop redundant 'or, alternatively' annotation

* ui+docs(bedrock): rewrite callout copy + refresh screenshot

Reframes the BedrockSetupCallout away from generic BYOK language to a
Bedrock-specific message: leads with "Amazon Bedrock is configured
entirely via environment variables", lists all four (auth, region,
model id), and ends with the requested CTA sentence ("click below to
learn more about Bedrock support in Pullfrog").

Promotes the "Bedrock setup guide" docs link from an inline anchor to
a prominent button (always visible, regardless of auth state). The
"Add AWS_BEARER_TOKEN_BEDROCK" affordance is now a secondary chip
shown only when no auth secret is configured.

Refreshes docs/images/model-selector-bedrock.png to capture the new
callout — the prior screenshot still showed the old generic
"BYOK / X, Y, or Z required" wording.
2026-05-14 02:12:38 +00:00
Colin McDonnell 1a9d3c1f82 fix bootstrap ETARGET when customer has npm min-release-age policy (#725)
* fix bootstrap ETARGET when customer has npm min-release-age policy

set npm_config_min_release_age=0 in the action runtime env so
`npx --yes pullfrog@<spec>` doesn't get rejected by a customer-side
release-age gate (npm 11.5+'s min-release-age / pnpm's
minimumReleaseAge). env vars beat .npmrc in npm config precedence,
so this neutralises the policy regardless of where it's defined.

pullfrog's npm version is server-stamped from a SHA-pinned action
ref customers already vet at the action layer — it isn't a
customer-vetted dep, so the release-age policy is the wrong
affordance for our bootstrap and would otherwise hard-fail every
run while the latest publish ages into the customer's window.

closes #713

* also cover pnpm's minimumReleaseAge key for corepack fallback path

* correct pnpm env var (pnpm v11+ uses pnpm_config_*, not npm_config_*)

the prior commit set `npm_config_minimum_release_age=0` to cover the
pnpm corepack-dlx fallback path, but pnpm v11+ only reads env vars
prefixed `pnpm_config_*` / `PNPM_CONFIG_*` (the v10→v11 migration
explicitly renamed the prefix). swap to the correct env var so the
fallback path actually neutralises pnpm's `minimumReleaseAge`.

also tighten the comment block, and add an AGENTS.md rule reminding
us to fetch top-level reviews AND inline review comments together —
they live on different endpoints and the inline set is easy to miss
with `gh pr view --json reviews,comments` alone.

* add scripts/pr-reviews.ts for one-shot review evaluation

dumps top-level reviews + inline review threads (with resolved/outdated
state) + PR-level conversation in a single GraphQL round trip, so agents
don't miss inline-comment feedback. fixes the trap where
`gh pr view --json reviews,comments` silently omits the inline
`pulls/{n}/comments` set.

borrows `gh auth token` so no env vars are required. registered in
`wiki/scripts.md`; AGENTS.md rule updated to point at the script
instead of the two-step gh-CLI workaround.

* pr-reviews: dump raw JSON for jq piping
2026-05-14 01:48:14 +00:00
Colin McDonnell 951745ec89 disable stop hook (runtime + dashboard) (#727) 2026-05-14 01:44:32 +00:00
Colin McDonnell 56793d4a81 claude: prefer non-JSON stdout over NDJSON tail in exit-1 fallback (#643) (#726)
Claude CLI under CLAUDE_CODE_OAUTH_TOKEN exits 1 without setting `is_error`
when the OAuth subscription's quota is exhausted. The existing fallback
chain (`lastResultError || stderr || tailLines(stdout)`) had nothing
structured to grab and dumped ~2KB of `system/init` NDJSON into the
progress comment, hiding the actionable quota notice the CLI had already
printed as plain text.

Capture non-JSON stdout lines into a 20-line ring buffer (mirroring the
existing `recentStderr` pattern) and prefer it over the raw NDJSON tail.
Generic — no regex on bubble text — so any human-readable line the CLI
emits surfaces instead of the event stream.

Also adds a `failure:claude-oauth-quota` bucket to `analyze-logs.ts`,
ordered before the SIGTERM check so the NDJSON tail's `cancelled` /
`cancel_url` substrings (from learnings content) stop shadowing it.
2026-05-14 01:26:10 +00:00
Colin McDonnell d857e06731 postrun: tighten unsubmitted-review gate to require create_pull_request_review for Review mode (#724)
The gate at `getUnsubmittedReview` accepted `toolState.finalSummaryWritten`
as a valid Review exit, contradicting the post-failure error message which
already says Review's only valid exit is `create_pull_request_review`.
This let any caller that flipped `finalSummaryWritten` — including a
`task`-dispatched `reviewfrog` subagent calling `pullfrog_report_progress`
in violation of its prose-only read-only contract — silence the gate even
when the orchestrator never submitted a review.

Split per-mode: Review requires `toolState.review`, IncrementalReview keeps
the existing `||` (its post-failure message explicitly accepts
`report_progress` as a "no review warranted" exit). Test split mirrors the
new semantics.

closes #648
2026-05-14 00:01:15 +00:00
Colin McDonnell b9f0938405 mcp: restore operational guidance dropped in #723
#723's revision pass cut four substantive strings along with the
negative anchors. those strings address real, audit-observed failure
modes and the positive examples don't carry them.

restored:
- push_branch: "if the response reports a timeout, the underlying
  push may have actually succeeded — verify with git log
  origin/<branch> before retrying" (was on the tool description)
- create_pull_request_review commit_id .describe(): "must be the FULL
  40-character SHA — abbreviated SHAs are rejected by GitHub with 422"
- create_pull_request_review comments[].line .describe(): "must sit
  inside a `@@` hunk... dropped entries are reported under
  droppedComments in the response"
- create_pull_request_review comments[].start_line .describe(): "both
  start_line and line must sit inside the same @@ hunk"

also: get_commit_info example used a 31-character SHA (non-standard
truncation). swapped to a 7-char short form, which is what git
log --oneline emits and what agents see in practice. note that this
tool accepts either full or abbreviated, unlike create_pull_request_review
which requires full.
2026-05-13 22:49:06 +00:00
Colin McDonnell b8ac42e875 mcp: embed example calls in top-level tool descriptions (#723)
* mcp: embed example calls in top-level tool descriptions

agents (esp. claude sonnet) hallucinate param names from training-data
priors — `pr_number` instead of `pull_number`, `summary` instead of
`body`, full subcommand strings jammed into `git({command})` like it
were `shell({command})`. each error burns a tool round-trip plus a
follow-up ToolSearch, ~40+ events / 24h, no observable recovery cost
to us but visible to users in agent logs.

cheapest fix: add a sample formatted function call to every affected
tool's top-level description. example anchors are more reliable than
schema descriptions alone because the model treats descriptions as
narrative but call examples as canonical structure. for `git` and
`shell` (whose `command` fields collide), include explicit
counter-examples disambiguating which tool owns which shape.

no schema aliases / coercion yet — try the cheap thing first; if the
next audit window still shows the same hallucination rate, layer
aliases on top per #585's recommendation.

closes #585, closes #701

* mcp: drop negative anchors from tool descriptions

negation is a footgun in tool descriptions — telling the model "NOT
pr_number" makes pr_number more salient, not less. let the positive
example carry the schema and trust the model to read it.

removes:
- "the parameter is pull_number (a number), NOT pr_number" and
  similar across checkout_pr, get_pull_request, list_pull_request_reviews,
  get_review_comments, create_pull_request_review
- "NOT summary, message, or content" on report_progress
- "WRONG: git({ command: 'log --oneline' })" counter-example on git
- redundant param-type restatements after the example (e.g. "depth is a
  number, not a string" on git_fetch, "description is required" on shell)

keeps a single positive example per tool. for tools with multiple call
shapes (git, git_fetch, push_branch), two positive examples instead of
one + a counter-example.
2026-05-13 22:45:08 +00:00
Colin McDonnell 868576a474 audit: format byok auth errors actionably + tighten audit prompt
- `action/utils/apiKeys.ts`: rewrite the missing-key body as Markdown with
  linked CTAs (repo secrets / model settings / docs). add
  `isApiKeyAuthError` + `formatApiKeyErrorSummary` covering both shapes:
  missing key (#679) and revoked/invalid 401 key (#702).
- `action/main.ts`: reclassify in the result-failure branch and the catch
  block so the PR progress comment surfaces the actionable CTA instead of
  the raw `Invalid API key · Fix external API key` / numbered-list dump.
- `scripts/analyze-logs.ts`: split `failure:user-misconfig` into
  `:no-key` and `:invalid-key` so both buckets are visible separately
  and the audit can ignore them as user-correctable.
- `.github/workflows/run-audit.yml`: add three explicit prompt rules —
  cross-customer signal required (≥3 distinct accounts; single-customer
  concentration is not enough), recovered failures are not actionable,
  user misconfig is out of scope. closes the loop on #679 / #702 being
  filed in the first place.
2026-05-13 21:59:47 +00:00
Colin McDonnell b2b1e588e7 biome: exclude .scripts/ — gitignored operator scratchpad
Mirrors the gitignore. Same shape as the existing !**/logs / !**/.logs
/ !.worktrees exclusions in files.includes. Matches the upstream
.gitignore policy for the .scripts/ directory.

Without this, .scripts/ scripts (`.scripts/kyle-*.ts`,
`.scripts/check-comment.ts`, etc.) get scanned by `pnpm lint` and
`pnpm format` from the repo root and routinely fail husky pre-push
even though they're explicitly intended to be local-only / personal.
The companion to .gitignore — both are operator-owned scratchpads;
neither participates in repo-wide hygiene.
2026-05-13 21:25:43 +00:00
Colin McDonnell 5caeb75344 review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models (#710)
* review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models

PR review wall-time was dominated by two failure modes: orchestrator
serial-dispatching subagents (despite prompt asking for parallel) and
running every lens on the same Opus tier as the orchestrator. Sample of
recent runs showed 25-60min reviews on small PRs, with 8-10min idle
gaps between subagent dispatches.

Three changes:

1. `action/modes.ts` — replace the soft "1 trivial / 2-3 typical /
   4-5 high-stakes" lens calibration with a binary 0-or-2+ rule. Default
   is 0 lenses (orchestrator handles review solo with optional cheap
   tracerfrog dispatches). 2+ parallel lenses only fire for substantive
   PRs (>5 files AND >200 lines) or high-stakes-subsystem touches. Never
   exactly one. Both Review and IncrementalReview prompts get loud
   ALL-CAPS framing on parallel dispatch — emit ALL Task tool_use blocks
   in a single assistant turn before reading any result. Drop the
   "do NOT lens-review the diff yourself" advice; orchestrator pulls
   context aggressively, in parallel with the lens fan-out.

2. New `tracerfrog` subagent for mechanical code tracing ("where is X
   used / who calls Y / what depends on Z"). Pure read+grep+report with
   no judgment — orchestrator can dispatch many tracers cheaply in
   parallel. Defined in `action/agents/reviewer.ts`. Wired into both
   claude.ts (`--agents` JSON) and opencode.ts (`agent` config block).

3. Per-subagent model downshifts via `deriveSubagentModels`:
   - Anthropic: reviewfrog → Sonnet, tracerfrog → Haiku
   - OpenAI: both → gpt-5.4-mini
   - other providers (xai, deepseek, gemini, etc.): inherit (no
     standard tier triplet to downshift to)

Claude Code path always runs Anthropic so the downshift is hardcoded
inline in claude.ts. OpenCode uses the helper since orchestrator
provider varies.

Both runtimes' subagent-definition formats verified directly against
their source: `--agents` JSON `model` field (claude-code's
`AgentJsonSchema` accepts model+effort+maxTurns+more) and OpenCode's
`agent.{name}.model` config field (parsed via Provider.parseModel,
applied per-task in tool/task.ts line 92). Parallel dispatch is
infra-supported in both — only the orchestrator model's tool_use
emission pattern was the bottleneck.

Tests: subagentModels.test.ts (14 tests covering provider matrix),
subagentRegistration.test.ts (6 source-asserts catching shape
regressions in buildAgentsJson / buildReviewerAgentConfig).

* subagentModels: add openrouter routes (proxy/router mode)

Initial helper missed the openrouter prefix used by Pullfrog's router
proxy. preview-710 e2e showed the OpenCode + openrouter path receiving
no downshift — orchestrator and lenses both ran on opus-4.7 because
'openrouter/anthropic/claude-opus-4.7' didn't match any of the
anthropic/openai prefixes the helper checked.

Add explicit branches for 'openrouter/anthropic/...' (uses dot notation:
claude-sonnet-4.6 / claude-haiku-4.5) and 'openrouter/openai/...'
(gpt-5.4-mini for both reviewer and tracer). Same opus->sonnet,
sonnet->keep-but-haiku-tracer, haiku->no-op semantics as the direct
anthropic path.

* opencode: log resolved subagent models at startup

So we can verify per-subagent model overrides actually take effect at
runtime. Prints once per run alongside the existing model/effort log
lines.

* drop tracerfrog: keep reviewfrog only, LSP-powered tracer planned later

Removes the cheap-haiku-tracer subagent (TRACER_AGENT_NAME +
TRACER_SYSTEM_PROMPT, registrations in claude.ts/opencode.ts, dispatch
guidance in modes.ts). The mechanical-tracing use case will be served
better by an LSP-powered tool than by a separately-prompted subagent.

deriveSubagentModels collapses to a single { reviewer } shape; the
reviewfrog-on-Sonnet downshift stays. Same source-assert + provider-
matrix tests, minus the tracer-specific cases.

modes.ts wording: drop the 'subagent type cheat sheet' bullet, drop
the parenthetical 'often better served by tracerfrog than reviewfrog'
on the impact lens, drop tracerfrog from the same-turn-context-pulling
hint. The 0-or-2+ rule and ALL-CAPS parallel emphasis are unchanged.

* subagentModels: broader downshift coverage (gpt-pro, gemini-pro, grok); drop gpt-mini target

Scanned every resolved orchestrator slug in action/models.ts against
models.dev pricing data. Identified five clear cases where the
orchestrator is meaningfully expensive AND has a cheaper sibling that
remains capable enough for review-style judgment work.

Changes:
- Anthropic: opus → sonnet  (kept; -40%)
- OpenAI: gpt → gpt-5.4 (was: gpt-mini; -54% instead of -85% but
  preserves review-quality judgment — gpt-mini was too dumb)
- OpenAI: gpt-pro → gpt   (NEW; -93%, biggest single unlock —
  gpt-5.5-pro is $30/Mtok in vs gpt-5.5 at $5)
- Google: gemini-pro → gemini-flash  (NEW; -75%)
- xAI: grok-4.3 → grok-4-1-fast  (NEW; -80%)

Every branch handles the three routes in use: direct provider slug,
opencode-vendored, and openrouter-proxied. Variants below the downshift
target (mini/nano/flash/fast/sonnet/haiku) inherit (no further drop).

Skipped:
- DeepSeek: v4-flash ($0.14/Mtok) is too far below review judgment
  threshold; v4-pro orchestrator already cheap ($0.55 blended).
- Moonshot: kimi-k2-thinking would only save 32% and slug stability on
  OpenRouter is uncertain; revisit if cost matters.
- o3: already mid-tier in OpenAI's reasoning family; no clean target.

* models: hoist subagent downshift into the registry, add hidden flag

The downshift relationship now lives next to each alias's resolve /
openRouterResolve as a sibling field. Two new ModelDef fields:

- subagentModel?: string — alias key (within same provider) of the
  cheaper sibling reviewfrog should use as a lens-fanout subagent.
  e.g. claude-opus → 'claude-sonnet'.
- hidden?: boolean — exclude from selectable lists (UI dropdown,
  CLI init picker). Does NOT affect resolution; for that use
  fallback. Used so internal-only subagent targets like openai/gpt-5.4
  exist in the registry but never appear as a user-facing pick.

Wiring:
- anthropic.claude-opus → claude-sonnet (-40%)
- openai.gpt-pro → gpt (-93%, biggest unlock)
- openai.gpt → gpt-5.4 (-54%); gpt-5.4 added with hidden:true
- google.gemini-pro → gemini-flash (-75%)
- mirrored across opencode + openrouter providers (each provider
  declares its own three-route data so the downshift declaration
  is colocated with the rest of the alias definition).

deriveSubagentModels collapses from ~85 lines of prefix-matching to
a ~15-line registry reverse-lookup: find the alias whose resolve OR
openRouterResolve matches the orchestrator's spec, follow its
subagentModel pointer, return the matching field of the target alias.

Filter sites updated:
- components/ModelSelector.tsx: !a.fallback && !a.hidden
- action/commands/init.ts:       same

Tests rewritten to exercise the registry through the public surface;
the matrix collapses to one assertion per (provider × route) pair.

* TEMP: log per-step cost+tokens for subagent model verification (PR #710)

* TEMP: also log SUBAGENT step_finish from bus envelope handler

* remove temporary per-step diagnostic logs (verification done)

Verified subagent model downshift takes effect end-to-end on the OpenCode
+ openrouter path. PR #8 in pullfrog/preview-710-review-perf dispatched
3 lenses (billing-subsystem / security / correctness) on the orchestrator's
opus-4.7 session, and per-subagent step_finish events showed actual cost
exactly matching Sonnet pricing rates (60% of what Opus would have cost):

  session       n  actual    if-Opus   if-Sonnet  match
  T3VrUuF...    5  $0.2425   $0.4042   $0.2425    Sonnet ✓
  93ZZR7E...    4  $0.2253   $0.3754   $0.2253    Sonnet ✓
  Fb1Kr7b...    4  $0.2495   $0.4158   $0.2495    Sonnet ✓

The startup '» subagent models: reviewfrog=...' line stays — useful
permanent diagnostic showing the resolved subagent model per-run.

* TEMP: log per-event model from claude.ts assistant handler

* remove temporary per-event model log (claude.ts verification done)

Verified subagent model downshift takes effect end-to-end on the Claude
Code path. PR #9 in pullfrog/preview-710-review-perf dispatched 2 lenses
on an opus-4-7 orchestrator. Per-assistant-event model field from the
SDK's stream-json output, partitioned by parent_tool_use_id:

  ORCH (parent_tool_use_id=null):  17 events all model=claude-opus-4-7
  SUBAGENT lens:billing-subsystem: 17 events all model=claude-sonnet-4-6
  SUBAGENT lens:security:          21 events all model=claude-sonnet-4-6

Zero leakage to opus from either subagent session. The per-subagent
'model' field in --agents JSON is honored by claude-code at the SDK
level, identical to the OpenCode path verified earlier.

* opencode: bump per-call output cap 5K → 16K to unblock large reviews

The 5K cap (added in #616 to lower OpenRouter upfront budget reservation
for low-wallet runs) was capping the entire response of a single LLM call,
not just the budget reservation. A single tool_use response — like a
`create_pull_request_review` with many inline comments — would truncate
mid-stream past 5K output tokens, leave the JSON unparseable, and the tool
would never actually invoke. We hit this on PR #710's verify-downshift PR:
review aggregated from 3 lenses had 11 inline comments + a long body,
truncated at out=5000 on every retry attempt, action exited with 'Review
mode finished without calling create_pull_request_review after 3 retry
attempts'.

Investigated whether OpenCode (or OpenAI/Anthropic/OpenRouter directly)
exposes a separate budget-reservation parameter that could stay small
while letting the response exceed it. They don't — `max_tokens` /
`max_completion_tokens` is the single value all four use for both the
upfront reservation and the hard output ceiling. No way to decouple them
at the API surface.

Bumped to 16K as a middle ground: 8× the prior cap (handles every review
shape we've observed plus headroom), still half of OpenCode's 32K default
so the wallet-burn benefit for low-balance accounts is preserved, just
smaller. For Opus 4.7 a typical ~50K-input call now reserves roughly
$0.65 instead of the prior $0.38.

Updated the constant comment to spell out the trade-off clearly so this
doesn't happen again.

* opencode: drop OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX override entirely

Verified the original rationale for the override is obsolete. From #616
the cap shrunk OpenRouter's per-call upfront budget reservation so a
single call's reservation wouldn't exceed the per-run key cap
(`ROUTER_PER_RUN_LIMIT_USD = 25`) and lock low-balance accounts out of
starting a run.

That per-run gate is gone. `app/api/proxy-token/route.ts` ~line 422
explicitly says: 'No upper cap (the old ROUTER_PER_RUN_LIMIT_USD = 25 is
gone). The natural ceiling is whatever the user has + their buffer.'
Router now mints keys with `keyLimitCents = balance + buffer` ($50 for
autoreload+card, $5 for card-only, $0 for no-card). A single call's
upfront reservation fits comfortably within that — no separate per-call
gate to fail past.

The cap had a real downside as a hard per-call output truncation. A
single `create_pull_request_review` tool_use with many inline comments
would truncate mid-stream past 5K output tokens, the JSON would be
unparseable, and the tool never invoked. Hit on PR #710's
verify-downshift PR.

Removing the override entirely; OpenCode falls back to its 32K default.
Left an explanatory note above the env-var assignment site so the next
person doesn't unknowingly re-add it.
2026-05-13 21:05:52 +00:00
David Blass 5518890b18 learnings: TOC + section taxonomy + 100k cap, hygiene rules, tool-quirk descriptions (#717)
* audit learnings: reshape reflection prompt + bake tool quirks into descriptions (#619)

Cross-repo audit of the 48 repos with non-null learnings turned up two
recurring failure modes:

1. ~25-30% of bullets across the most-active repos are pullfrog-tool
   quirks ("shell timeout is in milliseconds", "git args must be a JSON
   array", "create_pull_request_review drops out-of-hunk comments",
   "push_branch may report timeout when push succeeded", "checkout_pr
   shallow.lock retries", "commit_id needs full 40-char SHA"). These are
   universal across repos and should live in tool descriptions, not be
   rediscovered and stored 48 times. Tool descriptions now surface them.

2. Bullets are routinely 200-1000 chars (paragraph-length), and 12 of 48
   repos are at the 10k cap. The reflection prompt now: caps bullets at
   ~240 chars (one specific fact), bans PR/review/commit/date-anchored
   facts that decay within weeks, bans tool-quirk learnings, and tells
   the agent that cap pressure means compress+prune existing bullets,
   not skip new findings.

Co-authored-by: Cursor <cursoragent@cursor.com>

* learnings: add server-generated TOC, fixed section taxonomy, raise cap to 100k (#707)

Cap goes 10k → 100k. Reads stay bounded because the seeded file now
opens with a server-generated table of contents listing every `## `
section's line range — agents read the TOC, then `read_file offset/limit`
just the sections relevant to the current task instead of slurping the
whole file.

## Section taxonomy (fixed)

`## Build & test`, `## CI`, `## Conventions`, `## Architecture`,
`## Gotchas`. Free-form `### ` sub-headings inside a section are fine.
Pre-taxonomy free-text rows get wrapped in a `## Legacy` carve-out on
first seed so they remain visible while the agent gradually re-curates
them during reflection turns.

## Storage shape unchanged

`Repo.learnings` still holds raw markdown (no schema migration). The TOC
is a pure read-side affordance: prepended at seed time, stripped from
the agent-edited file before persist. Markers
`<!-- pullfrog-learnings-toc:* -->` delimit the strip region. Agent
edits inside the markers are discarded.

## Round-trip semantics

`seedLearningsFile` now returns `{ path, canonicalSeed }` where
`canonicalSeed` is the post-TOC body — same shape `readLearningsFile`
returns at end-of-run, so `persistLearnings` byte-compares them
directly to skip the no-op PATCH. Empty-repo first runs end up with the
section scaffold both as seed and as read-back, so untouched runs still
short-circuit cleanly.

## Reflection prompt

Adds explicit section-placement guidance (place each new bullet under
the most relevant `## `; do NOT add new top-level headings; do NOT
edit anything between the TOC markers). Carries forward the bullet
hygiene from the previous commit: ≤240 chars per bullet, no
pullfrog-tool quirks (those belong in tool descriptions), no
PR/review/commit/date references. The "near cap" framing is replaced
with "compress and prune within a section when it grows noisy" since
the cap pressure that drove cramming is gone.

Co-authored-by: Cursor <cursoragent@cursor.com>

* anneal round 1: line-anchored taxonomy detect, partial-merge, line-boundary truncation, scaffold-empty UI

Multi-lens review of the TOC + taxonomy diff surfaced a cluster of
correctness and operational bugs. Fixes:

- `hasAnyTaxonomyHeading` used `String.includes("## X")` which
  false-positives on `### X` (the `## ` substring sits inside `### `),
  prose containing `## CI`, fenced code documenting markdown, etc.
  Replaced with a line-anchored predicate that reuses `parseHeadings`
  so detection and TOC construction stay consistent.

- The "any heading present → pass through verbatim" rule meant a body
  with one taxonomy heading would seed without the other four. Worse,
  requiring all five would flip a body back into Legacy when the agent
  legitimately pruned a section to empty. New `partial` kind: keep
  existing content in place, append missing sections in canonical order
  so the agent always has the full scaffold without losing pruning
  intent.

- `stripLearningsToc` collapsed `\n{3,}` globally; `canonicalSeed`
  doesn't, so an untouched body with intentional triple-newline spacing
  would compare unequal and burn a spurious LearningsRevision row each
  run. Drop the global collapse — only the leading newlines that the
  strip itself introduces are normalized.

- 100k truncation via `slice(0, 100_000)` could cut mid-line, breaking
  `parseHeadings` (whole-line `^## `) on the next seed and flipping a
  cut body back into Legacy. New `truncateAtLineBoundary` cuts at the
  last newline before the cap.

- `LearningsSection.tsx` rendered a scaffold-only body as "has
  learnings" instead of the empty placeholder. Added a
  `hasOnlyEmptyScaffold` guard so the console behaves the same as
  pre-PR for the empty case.

- Seed log line distinguishes `kind=structured/partial/legacy-wrapped/
  empty` instead of `existing=yes/no`, so operators can spot legacy
  migration activity in logs.

- New tests cover: substring false-positive (`### Build & test`,
  in-prose mentions), partial-taxonomy merge (no Legacy wrap),
  full-taxonomy structured pass-through, last-newline truncation,
  triple-newline preservation.

Deferred (documented in PR body): deploy-ordering footgun (action
before API), rollback for rows >10k, Gemini sanitizer dropping
`description` on `anyOf` branches, reflection-on-failed-runs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* anneal r2: hard-truncate fallback when line boundary discards >4k

Round-2 review caught a regression in `truncateAtLineBoundary`: when the
only newline within the first 100k chars sits near the start (e.g. one
heading + 100k+ char single line — pathological pasted log dumps), the
line-boundary cut discards almost all of the body. losing one partial
line is preferable to losing kilobytes; threshold the fallback at 4k.

Co-authored-by: Cursor <cursoragent@cursor.com>

* move TOC out of file: prompt-side rendering, server-parsed headings

drops the in-file TOC + fixed taxonomy in favor of:
- file on disk = verbatim Repo.learnings (no markers, no scaffold)
- server parses headings (mdast-util-from-markdown) at run-context time
  and returns them as RepoSettings.learningsHeadings
- action renders heading TOC into the LEARNINGS prompt section as
  parenthesized line ranges like `Build & test (L1-L42)` with hierarchy
  via 2-space indent off the shallowest depth
- reflection prompt teaches agent-curated structure with a soft 300-line
  per-section cap and explicit guidance to restructure flat legacy lists

cuts 8 helpers (ensureSections, stripLearningsToc, assembleFile,
buildTocBlock, parseHeadings, buildSectionScaffold, hasAnyTaxonomyHeading,
LEARNINGS_SECTIONS) and the canonicalSeed round-trip dance.

action seedLearningsFile is now { path } only; main.ts byte-compares the
trimmed read-back against (current ?? "").trim() to gate the persist
PATCH. truncateAtLineBoundary kept for safety.

new tests:
- test/learningsToc.test.ts (11 parser cases incl. fenced-code, blockquote,
  arbitrary h1-h6 nesting, startLine-points-at-heading invariant)
- action/utils/learningsTocRender.test.ts (7 renderer cases)

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-13 20:14:26 +00:00
Colin McDonnell d04c1ca3da action: bump to 0.1.6 2026-05-13 18:23:45 +00:00
Colin McDonnell ae976e7159 parallel tool execution: enable opencode batch + nudge agents to parallelize (#719)
opencode: opt into `experimental.batch_tool` (anomalyco/opencode#2983) so the
`batch` tool registers and the model can bundle 1-25 independent calls into one
round trip. edit calls are excluded upstream.

instructions.ts: add a "Parallel tool execution" section to the SYSTEM Workflow
block, agent-specialized via ctx.agentId. uses Anthropic's canonical wording
("invoke all relevant tools simultaneously...") so Claude reliably emits multiple
tool_use blocks per message; tells OpenCode about the new `batch` affordance.

verified end-to-end against haiku-class models (sonnet for claude, default for
opencode) with a "read 3 files and report first lines" fixture. results:
- opencode used `batch` with 3 nested reads AND emitted 3 native parallel
  read calls in the same assistant turn
- claude went from 3 serial turns (1 read each) to 1 message with 3 parallel
  Read tool_use blocks
2026-05-13 18:05:39 +00:00
Colin McDonnell 5aabd1e4a9 fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680) (#715)
* fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680)

unbounded `stdoutBuffer += chunk` / `stderrBuffer += chunk` in
`action/utils/subprocess.ts` previously crashed the wrapper with
`RangeError: Invalid string length` once V8's ~1 GiB kMaxLength was
breached on long-lived agent runs. multi-lens opencode Reviews on large
monorepos (e.g. tambo-ai/buildy) hit this consistently — 23 runs in the
last 24h, 100% of Review-mode hard failures on that repo.

- add `retain: "tail" | "none"` to SpawnOptions, defaulting to "tail"
  with an 8 MiB cap. tail-mode prepends a `... [N MiB truncated] ...`
  sentinel so downstream consumers can detect truncation.
- export `TailBuffer` helper for callers that need the same bounded
  accumulator semantics at their own layer.
- wrap stream `data` listeners in try/catch as defense in depth — any
  synchronous throw inside a stream handler is otherwise fatal.
- opencode + claude pass `retain: "none"` (they drain via onStdout /
  onStderr) and switch their own `output` accumulators to TailBuffer.
  their error paths read the agent-layer bounded mirrors instead of
  the now-empty `result.stdout` / `result.stderr`.
- add `failure:string-length-overflow` heuristic to scripts/analyze-logs.ts
  so post-fix recurrences are visible at a glance instead of bucketing
  into `failure:unknown`.
- regression tests cover >1 MiB stderr without crash, retain:"none"
  contract, and TailBuffer truncation semantics.

* fix: avoid TS parameter property syntax in TailBuffer for strip-only node loader

* address review: clarify try/catch scope + lock retain default to "tail"

- the original comment claimed the try/catch caught "any synchronous throw"
  in the data listener, but `options.onStdout?.(chunk)` returns a Promise
  in the agent callers (claude.ts:569, opencode.ts:933) — a throw inside
  an async user callback surfaces as an unhandled Promise rejection, not
  a synchronous exception. reword to describe the actual protection:
  defense-in-depth for synchronous throws in the listener body, which is
  exactly the shape of the original RangeError on `+= chunk`.
- add a test that locks `retain` default to "tail" by spawning without
  the option and asserting `result.stderr` is non-empty. a future refactor
  that flipped the default to "none" would silently break gitAuth,
  package installs, and lifecycle hooks that read result.stderr for
  failure messages, and the rest of the suite wouldn't catch it.
2026-05-13 17:54:28 +00:00
Colin McDonnell 60cc8772a6 fix(log-audit): kill 404 noise from /api/github/installation-token at source (#693) (#708)
* fix(log-audit): kill 404 noise from `/api/github/installation-token` at source (#693)

Closes #693. Issue diagnosed a surface symptom (`log.error` on expected
404s) but missed the actual root causes. Investigation revealed two
distinct populations producing identical 3-call 404 bursts:

1. **Fork-CI on `pullfrog/pullfrog`**: `test-token.yml` and
   `trigger-sync.yml` ship with `on: push: main`, so every fork inherits
   them and 404s our token endpoint on first push. Self-inflicted noise
   that scales with fork count.
2. **Real users hitting the full action without installing the App**:
   `/api/repo/.../run-context` uses the caller's `GITHUB_TOKEN` to read
   the repo from GitHub and then unconditionally lazy-provisions
   Account+Repo rows via `fetchOrCreateRepo`, even when the App isn't
   installed. Generates phantom DB rows and false `new account created`
   team@ alerts. (Confirmed via Prisma: `ezcorp-org` has an Account row
   with `installerLogin: null`, never installed our App.)

Both populations then trip the client retry loop in
`acquireTokenViaOIDC`, which matched `"Token exchange failed"` and
retried 3× on terminal 4xx — tripling log volume and wasting CI time.

## Changes

- `action/.github/workflows/{test-token,trigger-sync}.yml`: gate jobs
  with `if: github.repository == 'pullfrog/pullfrog'`. Forks inherit
  the files but the jobs no-op.
- `app/api/repo/[owner]/[repo]/run-context/route.ts`: call
  `getRepoInstallation` first; return 404 with install URL if the App
  isn't installed, before any DB writes or GitHub repo fetch.
- `action/utils/github.ts`: introduce `TokenExchangeError` for non-2xx
  server responses; `acquireNewToken` no longer retries it. Retry now
  fires only on genuine network/timeout failures. 404 surfaces a
  user-actionable error pointing at the install URL.
- `app/api/github/installation-token/route.ts`: move `log.error` inside
  the 500 branch only. 404 branch is silent (expected user-state) and
  returns the same install URL message for consistency.

## Effect

- Better Stack `level=error` lines from this path: 6/day → 0.
- Failed user-trial CI time: 3 wasted token requests → 1.
- User-facing error: opaque `Token exchange failed: 404` → actionable
  install URL.
- No more phantom Account rows from never-installed callers.

Skipped per design discussion: phantom-account cleanup (conservative —
stop the bleed, leave history), `AGENTS.md` rule (overgeneralized).

* review: address oracle leak + per-env install URL + retryable 5xx

Addresses pullfrog[bot] (IMPORTANT) and Copilot review findings on #708:

- **Install-status oracle in `run-context`** [pullfrog, Copilot]:
  `getRepoInstallation` runs with our App's JWT, *before* the caller's
  bearer token is validated against the repo. Pre-PR the route was
  uniformly bad-token-shaped; the new install-specific 404 turned it
  into an unauthenticated oracle distinguishing "Pullfrog installed
  here" from "not installed". Collapsed the 404 message to match the
  outer catch's ambiguous "repository not found or token lacks access".
  Legit runners still get the actionable install URL from
  `/api/github/installation-token`, which IS gated by OIDC.

- **Hardcoded `github.com/apps/pullfrog`** [Copilot]: server-side
  `installation-token` now uses `GITHUB_APP_INSTALL_URL` from
  `app/globals.ts`, so dev/staging deployments with a different
  `GITHUB_APP_SLUG` direct users to the correct app. Action-side
  echoes the server's `error` body when present (single source of
  truth) and falls back to a generic message only if the body isn't
  JSON.

- **Transient 5xx/429 made terminal** [Copilot]: `shouldRetry` now
  returns `true` for `TokenExchangeError` with `status >= 500` or
  `status === 429`. 4xx remains terminal (the actual #693 fix). Real
  outages no longer fail the workflow immediately.

- **Stale comment** [pullfrog, Copilot]: reworded the comment at
  `installation-token/route.ts:141` to reflect the new retry policy
  ("the action surfaces this once (no retry)" instead of "the action
  retries on this").

* review: restore caller-token-first auth in run-context

Pre-PR, `getEnrichedRepo({owner, repo, token})` used the caller's
token as the auth boundary — `getRepo({token})` succeeding was the
proof-of-access check. My initial install-gate inverted the order
and ran the App-credentialed `getRepoInstallation` first, which is
how it became:

- an install-status oracle (pullfrog bot, addressed previously by
  matching the outer-catch wording), and
- an outbound amplifier against our App JWT for arbitrary `owner/repo`
  (pullfrog bot, this commit).

Reordered so `getRepo({token})` runs first. Garbage / unauthorized
bearers get rejected by github (mapped to 403 by the outer catch)
before any App-credentialed call fires. `getRepo` is cached 5min,
so `getEnrichedRepo` below remains a free re-hit.
2026-05-13 17:47:13 +00:00
Colin McDonnell 4260984257 attribute claude subagent log lines + per-session thinking timer; tighten lens calibration (#700)
* attribute claude subagent log lines + per-session thinking timer; tighten lens calibration

three orthogonal fixes diagnosed from the 10m PR-699 review run:

1. wire SessionLabeler into the Claude Code harness. claude-agent-sdk
   stamps every Assistant/User/System message with session_id and a
   non-null parent_tool_use_id when emitted from a subagent context, so
   the same FIFO labeler the OpenCode harness uses works here too.
   parallel reviewfrog dispatches now log with [lens:correctness] /
   [lens:operational-readiness] / etc. prefixes instead of being
   indistinguishable from the orchestrator. matches both "Task" and
   "Agent" tool names per the v2.1.63 rename.

2. one ThinkingTimer per session. the global timer treated cross-session
   interleaving (parent thinks → child tool_call, child returns →
   parent dispatches next) as parent thinking time, so individual
   "thought for Xs" numbers were untrustworthy. each session now owns
   its own timer and prefixes its own log line.

3. tighten the Review/IncrementalReview lens-add discipline. PR-699
   triggered 4 lenses on a typical refactor (no auth/billing/schema)
   when the prompt's own calibration says 2-3 is typical; the
   research-validated lens went deep on Resend idempotency window +
   prisma updateMany lost-updates without either being load-bearing.
   adds an explicit "name the failure mode this lens would catch
   that the diff plausibly introduces" bar, and tightens
   research-validated specifically: only when correctness depends on
   the third-party contract, not when the API is merely used.

side benefits from #1: subagents' TodoWrite events no longer clobber
the orchestrator's progress comment; subagent text no longer overwrites
finalOutput; system-event handler safely routes through eventLabel even
though SDK only emits system:init for the top-level query today.

* fix node strip-only mode: declare formatLine as field, not parameter property

* key claude subagent labels by parent_tool_use_id, not session_id

claude-agent-sdk runs subagents inside the orchestrator's session — they
share session_id — and stamps subagent messages with parent_tool_use_id
pointing at the Agent tool_use that spawned them. e2e on PR-700 with
preview-700-claude-labeling#1 confirmed the original session_id-keyed
wiring never differentiated subagent activity (only the dispatch line
got [lens:correctness] in the log; the subagent's reads, writes, and
todos all rendered as orchestrator).

extend SessionLabeler so labelFor accepts an optional parent_tool_use_id
and short-circuits to a direct map keyed by Agent tool_use id when set.
recordTaskDispatch optionally takes the Agent tool_use id (block.id at
dispatch time) and binds it. orchestrator events keep flowing through
the sessionID/FIFO path unchanged so opencode wiring is untouched.

* drop weak timer test that asserted only field isolation

per pullfrog review on PR-700: the 'two timers do not bleed timestamps'
test only verified that two ThinkingTimer instances have separate
private fields, which has always been true. doesn't earn its keep —
the per-session behavior is exercised by integration through claude.ts
+ opencode.ts.
2026-05-13 15:28:08 +00:00
Colin McDonnell d5f881e9fc action: trim sensitive env values before GitHub Actions log masking (#698)
* action: trim sensitive env values before GitHub Actions log masking

GitHub Actions' log masking is line-based: a secret value containing a
newline only registers the first line as a mask, leaving the remainder
exposed verbatim in logs. A trailing newline copied from a terminal into
a GitHub Actions secret (e.g. ANTHROPIC_API_KEY) was enough to leak
"a large part of the key" in run logs (pullfrog/pullfrog#41).

normalizeEnv now trims leading/trailing whitespace from any value whose
key matches the sensitive name pattern, masks the cleaned value, and
warns when whitespace was stripped so the user notices the source.
sanitizeSecret is reused for dbSecrets injection in main.ts. The three
secret-store PUT/POST routes also trim values defensively, matching the
existing name.trim() pattern.

Real multi-line secrets are not used in practice — even GITHUB_PRIVATE_KEY
PEMs are stored single-line with escaped \n and unescaped at the point of
use — so a straight trim() is safe.

* action: address review — use core.setSecret for masking, don't zero whitespace-only

Pullfrog's review of #698 caught two real issues in the original fix:

1. `console.log(\`::add-mask::\${trimmed}\`)` doesn't escape \r/\n. If a
   value survives trim with an embedded newline (PEMs, kubeconfigs, JSON),
   the runner only registers the first line as a mask and the rest leaks.
   `core.setSecret(trimmed)` routes through @actions/core which
   percent-encodes \r/\n so the runner V2 parser decodes back to the full
   value and registers every non-empty line as a separate mask. Removes
   the load-bearing "no embedded newlines" invariant from the fix.

2. Whitespace-only sensitive values silently became "". Downstream
   truthy checks would flip from "set" to "missing" with no log. Now
   sanitizeSecret returns null in that case and callers skip the
   process.env write, surfacing a clear missing-key error instead.

Tests rewritten to assert process.env state directly — no stdout spies.
Masking correctness is delegated to @actions/core (trusted dependency).
2026-05-13 15:27:13 +00:00
Colin McDonnell 1dc53043a6 chore: bump action to 0.1.5 2026-05-13 04:56:01 +00:00
Colin McDonnell 076e5a17b5 default Claude Code effort to high
max effort burns roughly 2x the wall time per turn for marginal quality
gain. high is the model's tuned default ('equivalent to not setting the
parameter' per Anthropic docs). full-send can be reintroduced as an
opt-in per-run override later if needed.
2026-05-13 04:49:07 +00:00
Colin McDonnell d5d8a0d7ac fix(#691): drop opencode/gpt-5-nano + opencode/mimo-v2-pro-free (not actually keyless on Zen) (#695)
* remove opencode/gpt-5-nano and opencode/mimo-v2-pro-free from catalog

#7 delete aliases. both were listed as `isFree: true, envVars: []` but
neither is keyless on opencode zen, producing a hard-fail
`UnknownError: Model not found: opencode/<id>` on every run without an
opencode_api_key. fixes pullfrog/app#691 (5 runs across 3 repos, 100%
failure rate in the last 24h).

root cause: opencode's provider gate
(`packages/opencode/src/provider/provider.ts` `opencode:` loader) keeps
a zen model only when models.dev reports `cost.input === 0` for it,
then signs requests with `apiKey: "public"`. paid zen models get
deleted from the autoloaded set and opencode surfaces the deletion as
"model not found".

- `opencode/gpt-5-nano`: models.dev reports `cost: {input: 0.05, output:
  0.4, cache_read: 0.005}`. paid → requires `OPENCODE_API_KEY`.
- `opencode/mimo-v2-pro-free`: free on models.dev but not in
  `https://opencode.ai/zen/v1/models` — zen never served it, so even
  the public-key path fails.

remaining free aliases (`opencode/big-pickle`,
`opencode/minimax-m2.5-free`) both pass both checks (cost.input === 0
in models.dev AND present in zen's served list) and continue to work
without a key — verified against the opencode source.

callers swept: `action/utils/apiKeys.test.ts`, `action/models.test.ts`,
`action/test/list-aliases.ts`, `action/test/model-smoke.ts`,
`components/ModelSelector.tsx` (`modelIdToUpstream`),
`wiki/model-resolution.md`, `wiki/models-catalog.md`. wrote up the
free-zen verification rule in models-catalog so the next maintainer
can sanity-check both conditions before adding any `isFree` alias.

users with a stored `opencode/gpt-5-nano` or `opencode/mimo-v2-pro-free`
will now fall through `resolveCliModel → undefined` into the auto-select
path — a strict improvement over today's hard fail. no DB migration
needed; the slugs are simply unknown and treated like any other
unrecognized stored value.

* rework: keep mimo deprecated, demote gpt-5-nano to paid, add free-zen invariants

revised approach after the first commit over-corrected. mimo was never
broken at runtime — `fallback: "opencode/big-pickle"` already routes
stored values through to a real free model before any zen call. the
literal `opencode/mimo-v2-pro-free` being absent from zen's served list
is irrelevant because `resolveCliModel` walks the chain first. restoring
it as-is.

the actual bug was `opencode/gpt-5-nano`: marked `isFree: true,
envVars: []` but `models.dev` reports `cost: {input: 0.05, output: 0.4}`
on the opencode provider, so opencode's keyless gate
(`packages/opencode/src/provider/provider.ts` `opencode:`) deletes it
when `OPENCODE_API_KEY` is missing and the run hard-fails with
`UnknownError: Model not found: opencode/gpt-5-nano`. demoting it to a
regular paid zen alias (drop `isFree`/`envVars: []`, add
`openRouterResolve: "openrouter/openai/gpt-5-nano"` — verified to exist
on openrouter at the same price). users without `OPENCODE_API_KEY` now
get our explicit "no API key found" error pointing at the secrets page
instead of opencode's cryptic upstream error. confirmed via
`https://opencode.ai/zen/v1/models` that zen serves no free GPT
variants, so there's no cheaper-than-`gpt-mini` free option to suggest
in its place.

CI gap analysis (why this slipped through):

- `models-catalog.main.test.ts` only checked existence + `status !==
  "deprecated"` on models.dev. paid-model-marked-free regressions and
  zen-served-list drift both passed.
- `models-live` (`model-smoke.ts`) runs with `OPENCODE_API_KEY` in env,
  so the keyless deletion gate never fires. `gpt-5-nano` returned "OK"
  in CI even though end users hit a hard fail.
- `model-smoke.ts` walks the fallback chain, so mimo would have been
  smoked as big-pickle anyway — the dead resolve target was never
  exercised directly. (this is the right design; the gap is at the
  catalog layer, not the smoke layer.)

new tests:

- PR-blocking, static (`action/test/models.test.ts`, `isFree
  invariants`): every `isFree` alias must live under `opencode`, have
  `envVars: []`, omit `openRouterResolve`, AND have a fallback chain
  whose terminal alias is also `isFree` (catches "deprecate a free
  alias to a paid target" — the worst silent-charge regression).
- main-only, network (`action/test/models-catalog.main.test.ts`,
  `opencode Zen served list`): every alias whose terminal-fallback
  resolve is `opencode/*` must appear in
  `https://opencode.ai/zen/v1/models`. catches zen dropping a model
  from its served list.
- main-only, network (same file, `isFree models.dev cost`): every
  `isFree` alias's terminal-fallback resolve must have `cost.input ===
  0` in the `opencode` provider block on `models.dev`. would have
  caught `gpt-5-nano` at the next models-bump run.

both network tests dedupe on terminal resolve, so deprecated aliases
sharing a target aren't double-counted. `pnpm vitest run`: 113 static
tests pass. `pnpm test:catalog`: 142 network tests pass against the
live `models.dev`, `openrouter.ai`, and `opencode.ai/zen/v1/models`
endpoints.

wiki/models-catalog.md: rewrote the new "Free-Zen aliases need Zen-side
verification" section to (a) describe the two conditions, (b) note
that a fallback to an isFree alias is the legitimate escape hatch
(mimo's pattern), and (c) point at the three tests by name so the next
maintainer can find the enforcement surface. wiki/model-resolution.md
points at the new section.

* make gpt-5-nano a deprecated free alias falling back to big-pickle

revising the previous "demote to paid" approach. the user-facing
ergonomics are cleaner: anyone who picked gpt-5-nano under the "Free"
badge gets transparent-upgraded to a real free model (big-pickle)
instead of suddenly being asked to set OPENCODE_API_KEY. matches the
existing mimo pattern exactly. the dropdown already filters
`!a.fallback`, so the slug disappears from the picker on its own and
the trigger renders it as "Big Pickle" via `resolveDisplayAlias`.

no other catalog or test surface changes — the isFree invariants and
the main-only zen/cost checks still pass (gpt-5-nano's terminal is
now big-pickle, which is both isFree and zero-cost on models.dev,
deduping with big-pickle's own row in both network tests).

* revise: keep gpt-5-nano as paid alias, backfill affected DB rows instead

dropping the deprecated-alias approach. `opencode/gpt-5-nano` is a
legitimate cheap paid model people may want with BYOK
(`OPENCODE_API_KEY`) — giving it `fallback: "opencode/big-pickle"`
would foreclose that for everyone going forward. correct fix is two
parts:

(a) reclassify in the catalog as a regular paid OpenCode alias:
  - drop `isFree: true` and `envVars: []` so the local validator
    demands `OPENCODE_API_KEY`
  - add `openRouterResolve: "openrouter/openai/gpt-5-nano"` to satisfy
    the completeness test and route BYOK-via-OpenRouter users
  - no `fallback` — slug stays visible in the picker as a paid option

(b) one-shot DB backfill of provably-affected repos
(`scripts/backfill-gpt5-nano-affected.ts`). scope:
  - `Repo.model = "opencode/gpt-5-nano"`
  - AND at least one `WorkflowRun` with `inputTokens IS NULL` (evidence
    of an attempted run that didn't get past the model-init gate)

skipped intentionally:
  - repos whose runs have `inputTokens > 0` — they have a key, gpt-5-
    nano works for them
  - repos with zero WorkflowRun rows — never dispatched; touching them
    would be presumptuous
  - `LearningsRevision.model` — audit trail of which model authored a
    revision, rewriting it would falsify history

ran against .env.prod: 2 repos stored the slug; 1 was provably
affected (sodown4thecause/seobot, 5/5 zero-token runs — matches #691's
3 failed runs from this repo plus 2 outside the 24h audit window).
1 was an internal test account that never dispatched (left as-is).
applied: 1 row updated. confirmed idempotent on re-run.

the other two repos in #691 (Nantiee/ALTA-breast-pump-tool,
keksiqc/ansible-setup-linux) don't store the slug in `Repo.model`;
their failed dispatches passed the model inline in the
`workflow_dispatch` `prompt` payload, so the catalog fix alone (no
longer offering it as free) is what helps them.

tests:
  - models.test.ts: `getModelEnvVars("opencode/gpt-5-nano")` now
    returns `["OPENCODE_API_KEY"]`, moved into the keyed-model group
  - apiKeys.test.ts: added "throws without OPENCODE_API_KEY" case
  - isFree invariants from the previous commit still pass — gpt-5-nano
    no longer triggers them since it's no longer isFree
  - main-only catalog tests still pass (gpt-5-nano served by Zen, just
    paid; no isFree cost check applies)

* docs: drop stale GPT Nano + MiMo V2 Pro from free-tier lists

addressing pullfrog auto-review feedback on #695. three mintlify pages
still advertised both as keyless after the catalog pivot, which now
makes the docs affirmatively wrong rather than merely stale:

- gpt nano is paid in the catalog (no `isFree`, inherits
  `OPENCODE_API_KEY`); a user following the docs would hit the same
  "missing API key" failure that's described 4 lines below in
  `docs/keys.mdx`.
- mimo v2 pro is hidden from the picker (`fallback` triggers
  `ModelSelector`'s `!a.fallback` filter); the alias only exists for
  legacy stored-value resolution. a user reading the docs cannot
  actually pick it.

surviving picker-visible free set: Big Pickle and MiniMax M2.5.

- `docs/keys.mdx`: drop both bullets from the "Free models" list
- `docs/billing.mdx`: drop both bullets from the "Free models" list
- `docs/getting-started.mdx`: collapse the inline mention from a
  4-model list to "Big Pickle and MiniMax M2.5"

* address third review: picker grouping + backfill classifier honesty

i had not pulled the third pullfrog review (`02:17:28Z`) when i declared
reviews triaged after the docs sweep — the fourth review flagged that
three findings remained pending. addressing them now.

1. picker grouping for now-selectable paid gpt-5-nano. when i removed
   `"gpt-5-nano": "OpenAI"` from `modelIdToUpstream` in the previous
   pivot-to-paid commit, i mistook it for dead code. it's not — the map
   IS consulted for paid opencode aliases via `groupByUpstream →
   getUpstreamLabel` inside the OpenCode submenu's
   `renderSubContent`. without the entry, `gpt-5-nano` falls back to
   `getProviderDisplayName("opencode")` = "OpenCode" and gets dropped
   into its own sub-header instead of joining opencode/gpt,
   opencode/gpt-pro, opencode/gpt-mini under the "OpenAI" upstream
   group. re-added with an explanatory comment so the next refactor
   doesn't make the same mistake.

2. JSDoc / code mismatch in `scripts/backfill-gpt5-nano-affected.ts`.
   the JSDoc said "at least one `WorkflowRun` with `inputTokens IS
   NULL`" but the code is `no WorkflowRun has inputTokens > 0` — a
   strictly broader filter (catches `null` AND `0`). rewrote the scope
   block to describe what the code actually does, with the operative
   classifier spelled out: "a billable run with `inputTokens > 0` is
   proof the agent successfully reached and called the model".

3. classifier breadth (raised in the same review). honest answer: the
   "no positive-token run" filter IS a heuristic — a repo whose only
   dispatches happened to fail or cancel for unrelated reasons would
   get false-positive-classified A. for THIS one-shot population (2
   repos, 1 with 5/5 zero-token runs — strong systematic-failure
   signal) the heuristic was good enough and the dry-run inspection
   confirmed before APPLY. for any larger reuse of this pattern, you
   need to cross-reference the runtime error string (`UnknownError:
   Model not found: opencode/gpt-5-nano`) from GitHub Actions logs or
   Better Stack — that error doesn't live on `WorkflowRun` rows. added
   a "Classifier limitations" section to the JSDoc making this
   explicit.

nothing about the actual applied backfill changes — the prod write
(1 repo: sodown4thecause/seobot → opencode/big-pickle) is unchanged
and re-running the script remains idempotent.
2026-05-13 02:43:08 +00:00
Colin McDonnell 159389fad2 fix(mcp): sanitize for gemini when model is unresolved (#697)
* fix(mcp): sanitize for gemini when model is unresolved

isGeminiRouted() previously required the effective model string to
contain "gemini" — but when payload.model="auto" (or any unresolved
slug) reaches addTools(), `effective` is the literal "auto", which
doesn't match. opencode then auto-selects gemini *after* the MCP
server has registered raw arktype schemas, and every tool turn dies
on `function_declarations[*].properties[*].any_of[*].enum: only
allowed for STRING type`.

widen the gate: any unresolved specifier (undefined / "auto" / a
slug without a `provider/` prefix) is treated as gemini-routed and
sanitized. the transforms are universally compatible normalizations
so the false-positive cost is negligible. tighten case 3 to preserve
`description` so the only lossy path no longer drops operator-facing
context.

fixes #676.

* revert case-3 description preservation

per pullfrog review on #697: keeping `description` as a peer of
`anyOf`/`oneOf` directly contradicts the file's own header (lines
19-21) and the upstream opencode #14659 rationale that gates this
sanitizer — gemini requires anyOf to be the ONLY field on a schema
node, sibling keywords trigger
`anyOf must be the only field in a schema node`. the change was
speculative scope creep with no evidence, and would silently
re-introduce a different gemini failure for any future schema using
`.describe().or(...)`. the bug fix for #676 doesn't need it (arktype
doesn't emit non-collapsible anyOf for current tool schemas).
2026-05-13 02:31:59 +00:00
Colin McDonnell 43bb14bf87 action: strip Content-Type on body-less apiFetch requests (#692) (#694)
* action: strip Content-Type on body-less apiFetch requests (#692)

Vercel's Next.js lambda adapter (Next 16.1.x) attempts to decode a
request body when Content-Type is set and throws
`SyntaxError: Unexpected end of data` before delegating to the route
handler, returning a 500. Hit /run-context exclusively because it was
the only body-less GET that sent `Content-Type: application/json`.

- Drop `Content-Type: application/json` from the GET in
  `action/utils/runContext.ts` (meaningless on a body-less request).
- Defensively strip any `content-type` header in `action/utils/apiFetch.ts`
  when no body is present so future callers can't reintroduce this.

* apiFetch: soften comment — empirical observation, RFC 9110 §8.3 framing
2026-05-13 02:03:24 +00:00
Colin McDonnell d8f825034f billing: $10 signup credit + lazy claim modal; disable welcome credit promo (#674)
* billing: $10 signup credit + lazy claim modal; disable welcome credit promo

Adds a per-Account $10 Router signup credit granted on first Router-tab
mount via a new admin-gated POST /api/account/[owner]/signup-credit/claim.
The endpoint is idempotent — the inserted CreditGrant row IS the dedup
state, so subsequent calls return granted:false. Client SignupCreditModal
fires the POST on mount (only when modelAccessMode === "router") and
opens a celebratory dialog when granted:true.

Disables the legacy welcome credit ($10 on first card add) via a new
WELCOME_CREDIT_PROMO_ACTIVE = false flag in utils/stripe.ts. Code path
stays intact — flip the flag to revive. Strips the now-untruthful
"$10 on enabling billing" copy from BillingCard, EnableRouterPrompt,
triggerWorkflow paywall comment, action router_requires_card summary,
email snippet, billing/pricing docs and wiki.

Cuts WELCOME_CREDIT_CENTS from 2000 to 1000 to reflect the lower amount
that would land if the flag is ever re-enabled. Adds "signup" reason
mapping to BillingCard wallet history.

Verified end-to-end against dev: admin+Router fires modal, admin+BYOK
gate-blocks mount, BYOK→Router transition fires modal on click, member
and collaborator paths skip the mount entirely, reload after grant is
idempotent. Wallet history shows "Router signup credit +$10.00".

* billing: address PR review (race fix, copy sweep, modal retry)

Correctness:
- Add @@unique([accountId, reason]) on CreditGrant + migration. The prior
  check-then-insert pattern in /signup-credit/claim and finalizeCheckoutSession
  raced at READ COMMITTED — two concurrent admin tabs could land two grants of
  the same reason on a fresh account ($10 each). Both write sites now rely on
  the unique index for dedup (P2002 = "already granted") and route updated to
  catch P2002 cleanly. Verified zero existing duplicates in prod before
  migration.
- Add log.info on signup grant insert so a successful grant has any chance of
  being caught by ops monitoring.
- Add retry: 2 with backoff to the claim mutation. Endpoint is idempotent so
  a server-side success that lost its response cleanly returns granted:false
  on retry.

Public copy that still advertised the (now-deleted) $20 welcome credit:
- app/page.tsx landing pricing card
- emails/announceBilling.ts broadcast template
- docs/keys.mdx BYOK note
- components/AgentSettings.tsx Router-without-billing warning
- utils/stripe.ts finalizeCheckoutSession JSDoc
- utils/email/snippets.ts ROUTER_CREDIT_PS_HTML JSDoc

Wiki staleness sweep:
- wiki/billing.md TOC, mermaid diagram (signup edge added; welcome marked
  dormant), test coverage list, key modules section, no-card wallet narrative
- wiki/pricing.md welcome-credit drawdown reference
- Rewrote my own internally-inconsistent dormancy paragraph to be honest
  about the $20-historical / $10-on-revival framing.

Trivia:
- ModelAccessCard JSX comment had a literal \\u2192 instead of →.

* billing: address PR review round 2

- Replace try/catch P2002 inside finalizeCheckoutSession's prisma.$transaction
  with createMany skipDuplicates. The previous form is broken on Postgres: a
  unique-violation poisons the surrounding TX, so the catch block returns
  cleanly but the outer commit fails and the account.update (stripeCustomerId)
  silently rolls back too. Currently armed only behind the dormant welcome-
  credit flag, but would have broken billing enablement the moment the flag
  flipped. createMany skipDuplicates yields a single ON CONFLICT DO NOTHING
  statement that returns count: 0 cleanly without aborting the TX.
- Apply the same createMany skipDuplicates pattern to the signup-credit route
  too — drops the exception-as-control-flow Prisma namespace import and is
  more uniform with the welcome path.
- Drop the now-orphaned credit_grants_accountId_idx in the same migration.
  The schema removed @@index([accountId]) when @@unique([accountId, reason])
  was added (covered by the leftmost prefix), but the migration only added
  the unique index, leaving prod drifted.

* billing: fix stale finalizeCheckoutSession JSDoc

The function-level JSDoc still described the abandoned try/catch P2002
mechanism after switching to createMany skipDuplicates. The inline
comment + code now agree on the new ON CONFLICT DO NOTHING shape.

* billing: decouple first-card alert, drop vestigial billing field, fix modal cents; sync copy

* docs+homepage: align Router credit copy with signup claim (no card-on-add carrot)

* homepage: add pricing screenshot and pay-as-you-go promo line

* billing: fix once-per-lifetime misframe on first-card alert

* billing: suppress signup credit for prior welcome-credit recipients

* billing: drop bogus '1000 users' cap; invalidate billing on signup-credit settle
2026-05-12 23:47:52 +00:00
Colin McDonnell f0805b78f5 learnings: surface persist failures as warnings, not debug
`persistLearnings` only emitted `log.info("» learnings updated")` on
success; every failure path (non-2xx, fetch throw, 10s timeout) was
`log.debug`, which is hidden unless `ACTIONS_RUNNER_DEBUG=true`. Survey
of recent runs caught at least one case where the agent definitively
edited the tmpfile but no DB row was written and no warning surfaced.

Promote both failure paths to `log.warning` so dropped agent work is
visible in CI logs. The unchanged-from-seed short-circuit stays at
debug — that's a genuine no-op.
2026-05-11 23:51:46 +00:00
127 changed files with 9369 additions and 2270 deletions
+31
View File
@@ -0,0 +1,31 @@
# the Dockerfile only `COPY`s docker-entrypoint.sh, so most of this is
# defense-in-depth — modern docker BuildKit (default since docker 23)
# already prunes unreferenced files from the build context. but:
# - documents intent for future maintainers who add `COPY . .`
# - resurfaces the bytes-saved win if someone disables BuildKit
# (DOCKER_BUILDKIT=0) or adopts a builder that doesn't prune
# - keeps `docker build` snappy even on cold builders that DO send
# everything
# pnpm-managed workspace deps — large and never needed at build time
node_modules/
# secrets — must never enter an image, even by accident
.env
.env.*
!.env.example
# build outputs
dist/
build/
*.log
# editor / VCS noise
.DS_Store
.idea/
.vscode/
# tests + fixtures we don't need at build time
coverage/
test/
.scripts/
+4
View File
@@ -11,6 +11,10 @@ permissions:
jobs:
test-token:
# only run in the upstream publish target. forks inherit this file but
# haven't installed the pullfrog github app — running it there 404s our
# token endpoint and pollutes our error logs (see #693).
if: github.repository == 'pullfrog/pullfrog'
runs-on: ubuntu-latest
steps:
- name: Get installation token
+14
View File
@@ -30,6 +30,7 @@ jobs:
agent: [claude, opencode]
test:
[
codex-auth,
mcpmerge,
nobash,
restricted,
@@ -41,6 +42,8 @@ jobs:
exclude:
- agent: claude
test: skill-invoke-opencode
- agent: claude
test: codex-auth
- agent: opencode
test: skill-invoke-claude
env:
@@ -55,7 +58,17 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
OPENCODE_API_KEY: ${{ secrets.OPENCODE_API_KEY }}
AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
AWS_REGION: us-east-1
BEDROCK_MODEL_ID: us.anthropic.claude-opus-4-6-v1
PULLFROG_MODEL: ${{ vars.PULLFROG_MODEL }}
# CI smoke-testing shortcut only — production stores this in Pullfrog's
# per-org secret store (Postgres), set via `pullfrog auth codex`. GH
# Actions secrets are immutable at runtime so the post-hook can't write
# back the rotated refresh token; CI accepts the staleness and we
# manually re-provision when smoke tests start failing. Do not copy this
# pattern into user-facing workflows. See wiki/codex-auth.md.
CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
steps:
- uses: actions/checkout@v6
- uses: pnpm/action-setup@v4
@@ -78,6 +91,7 @@ jobs:
matrix:
test:
[
byok-no-keys-fallback,
git-permissions,
githooks,
pkg-json-scripts,
+4 -2
View File
@@ -10,8 +10,10 @@ permissions:
jobs:
trigger:
# skip if pushed by our bot (breaks the loop)
if: github.actor != 'pullfrog[bot]'
# only run in the upstream publish target (forks inherit this file but
# can't dispatch into pullfrog/app), and skip if pushed by our bot (breaks
# the loop).
if: github.repository == 'pullfrog/pullfrog' && github.actor != 'pullfrog[bot]'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
+78
View File
@@ -0,0 +1,78 @@
# pullfrog GHA-like test container.
#
# baked once at image build time, used by `pnpm docker`. all runtime cost
# (apt-get, useradd, sudoers wiring) is paid here so each `docker` invocation
# is a single `docker run` with no in-container setup.
#
# rebuild is content-hash gated by docker.ts (Dockerfile + docker-entrypoint.sh).
# bump anything in this file or the entrypoint and the next `pnpm docker` rebuilds.
FROM ubuntu:24.04
ENV DEBIAN_FRONTEND=noninteractive
# core toolset matching what GHA `ubuntu-24.04` runners ship: gh, jq, git,
# python3, ssh client, plus the compression + build-essential surface that
# `pnpm install` / `node-gyp` / agent shell calls regularly need. keeps
# test-time invocations of these tools honest (no "works on the runner,
# breaks in the local container").
RUN apt-get update -qq \
&& apt-get install -qq -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
file \
git \
gnupg \
jq \
openssh-client \
python3 \
sudo \
unzip \
wget \
xz-utils \
&& rm -rf /var/lib/apt/lists/*
# node 24 from nodesource + corepack (provides pnpm without a global install).
RUN curl -fsSL https://deb.nodesource.com/setup_24.x | bash - \
&& apt-get install -y nodejs \
&& rm -rf /var/lib/apt/lists/* \
&& corepack enable
# gh cli (matches GHA pre-installed tooling).
RUN mkdir -p /etc/apt/keyrings \
&& curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg \
| gpg --dearmor -o /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" \
> /etc/apt/sources.list.d/github-cli.list \
&& apt-get update -qq \
&& apt-get install -qq -y gh \
&& rm -rf /var/lib/apt/lists/*
# ubuntu:24.04 ships a default `ubuntu` user at uid 1000 — remove it so we
# can place `testuser` at 1000 (the typical macOS dev uid). the entrypoint
# remaps to the host uid/gid at runtime if they differ.
RUN userdel -r ubuntu 2>/dev/null || true \
&& groupadd -g 1000 testuser \
&& useradd -u 1000 -g 1000 -m -s /bin/bash testuser \
&& echo "testuser ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/testuser \
&& chmod 0440 /etc/sudoers.d/testuser
# layout matching the bind mount + named volume targets in docker.ts.
RUN mkdir -p /app/action /app/action/node_modules /tmp/home/.config /tmp/home/.cache \
&& chown -R testuser:testuser /app /tmp/home
# CI=true is critical: `shell.ts` PID-namespace sandbox keys off it. baking
# it ensures security tests can't pass vacuously because someone forgot the
# flag.
ENV HOME=/tmp/home \
TMPDIR=/tmp \
CI=true \
COREPACK_ENABLE_DOWNLOAD_PROMPT=0
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
WORKDIR /app/action
ENTRYPOINT ["/entrypoint.sh"]
+5
View File
@@ -36,6 +36,11 @@ outputs:
runs:
using: "node24"
main: "entry.ts"
# Always-run post step persists best-effort state that must survive
# cancellation, timeouts, and unhandled errors in the main step. Today's
# only consumer: Codex auth.json refresh write-back. See wiki/codex-auth.md.
post: "entryPost.ts"
post-if: "always()"
branding:
icon: "code"
+253 -54
View File
@@ -16,19 +16,36 @@ import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { performance } from "node:perf_hooks";
import { pullfrogMcpName } from "../external.ts";
import { BEDROCK_MODEL_ID_ENV, isBedrockAnthropicId } from "../models.ts";
import { getIdleMs, markActivity } from "../utils/activity.ts";
import { log } from "../utils/cli.ts";
import {
getIdleMs,
isActivitySuspended,
markActivity,
resumeActivity,
suspendActivity,
} from "../utils/activity.ts";
import { formatJsonValue, log } from "../utils/cli.ts";
import { installFromNpmTarball } from "../utils/install.ts";
import { detectProviderError } from "../utils/providerErrors.ts";
import { findProviderErrorMatch } from "../utils/providerErrors.ts";
import { addSkill, installBundledSkills } from "../utils/skills.ts";
import { SPAWN_ACTIVITY_TIMEOUT_CODE, SpawnTimeoutError, spawn } from "../utils/subprocess.ts";
import {
DEFAULT_MAX_RETAINED_BYTES,
SPAWN_ACTIVITY_TIMEOUT_CODE,
SpawnTimeoutError,
spawn,
TailBuffer,
} from "../utils/subprocess.ts";
import { ThinkingTimer } from "../utils/timer.ts";
import type { TodoTracker } from "../utils/todoTracking.ts";
import { getDevDependencyVersion } from "../utils/version.ts";
import { buildLearningsReflectionPrompt, runPostRunRetryLoop } from "./postRun.ts";
import {
buildLearningsReflectionPrompt,
runPostRunRetryLoop,
shouldRunReflection,
} from "./postRun.ts";
import { REVIEWER_AGENT_NAME, REVIEWER_SYSTEM_PROMPT } from "./reviewer.ts";
import { deriveLabelFromTaskInput } from "./sessionLabeler.ts";
import { formatWithLabel, ORCHESTRATOR_LABEL, SessionLabeler } from "./sessionLabeler.ts";
import {
type AgentResult,
type AgentRunContext,
@@ -66,17 +83,27 @@ function writeMcpConfig(ctx: AgentRunContext): string {
/**
* Build the `--agents` JSON definition for the `reviewfrog` subagent.
*
* The Claude Code path always runs against an Anthropic model (see
* resolveAgent), so we hardcode the cheaper-sibling downshift: lenses run
* on Sonnet, the orchestrator stays on whatever model `--model` was passed.
*
* Per-call model override is also possible (Task tool's `model` arg accepts
* 'sonnet' | 'opus' | 'haiku') and takes precedence over what's set here —
* we don't pass it; the per-subagent `model` field is the right default.
*
* The non-mutative + non-recursive contract is enforced by the prose system
* prompt baked into the agent — see action/agents/reviewer.ts for why we no
* longer wire per-agent `disallowedTools` here.
* prompt baked into the agent — see action/agents/reviewer.ts for why we
* no longer wire per-agent `disallowedTools` here.
*/
function buildAgentsJson(): string {
const agents = {
[REVIEWER_AGENT_NAME]: {
description:
"Read-only review subagent for self-review and lens-based code review. " +
"Read-only review subagent for lens-based code review (correctness, security, billing-subsystem, etc.). " +
"Reads only — no writes, no state-changing shell or MCP calls, no nested subagent dispatch.",
prompt: REVIEWER_SYSTEM_PROMPT,
model: "claude-sonnet-4-6",
},
};
return JSON.stringify(agents);
@@ -90,10 +117,13 @@ function stripProviderPrefix(specifier: string): string {
return slashIndex > 0 ? specifier.slice(slashIndex + 1) : specifier;
}
// `max` effort is supported on Opus 4.6 / 4.7; other models fall back to `high`.
// claude-code deny-lists older opus/sonnet generations from `max` at invocation time.
function resolveEffort(model: string | undefined): "max" | "high" {
if (model?.includes("opus")) return "max";
// `high` is the model's tuned default ("equivalent to not setting the parameter"
// per Anthropic docs). `max` is "absolute maximum capability with no constraints
// on token spending" — meaningfully slower and burns more thinking budget per
// turn. We default everyone to `high`; PRs that genuinely need full-send can
// opt in via a future per-run override rather than paying the wall-time cost on
// every Opus run.
function resolveEffort(_model: string | undefined): "high" {
return "high";
}
@@ -111,13 +141,21 @@ interface ContentBlock {
[key: string]: unknown;
}
// SDK schema (per claude-agent-sdk docs) puts `session_id` and
// `parent_tool_use_id` at the top level of every Assistant/User/System/Result
// message, not inside `message`. Subagent events carry a non-null
// `parent_tool_use_id` pointing at the orchestrator's Task/Agent tool_use id.
interface ClaudeSystemEvent {
type: "system";
session_id?: string;
parent_tool_use_id?: string | null;
[key: string]: unknown;
}
interface ClaudeAssistantEvent {
type: "assistant";
session_id?: string;
parent_tool_use_id?: string | null;
message?: {
role?: string;
content?: ContentBlock[];
@@ -135,6 +173,8 @@ interface ClaudeAssistantEvent {
interface ClaudeUserEvent {
type: "user";
session_id?: string;
parent_tool_use_id?: string | null;
message?: {
role?: string;
content?: ContentBlock[];
@@ -233,7 +273,37 @@ function tailLines(text: string, maxCodeUnits: number): string {
export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
const startTime = performance.now();
let eventCount = 0;
const thinkingTimer = new ThinkingTimer();
// per-session labeler so parallel subagent log lines can be differentiated.
// claude-agent-sdk runs subagents inside the orchestrator's session — they
// share `session_id` — and stamps every subagent message with a non-null
// `parent_tool_use_id` pointing at the Agent tool_use that spawned them.
// we bind each Agent tool_use id to its dispatched label up front, then
// labelFor short-circuits to the direct mapping when parent_tool_use_id is
// set. orchestrator events (parent_tool_use_id === null) flow through the
// sessionID path and bind to ORCHESTRATOR_LABEL on first sighting.
const labeler = new SessionLabeler();
function eventLabel(event: { session_id?: string; parent_tool_use_id?: string | null }): string {
return labeler.labelFor(event.session_id ?? null, event.parent_tool_use_id ?? null);
}
function withLabel(label: string, message: string): string {
return label === ORCHESTRATOR_LABEL ? message : formatWithLabel(label, message);
}
// one ThinkingTimer per session — sharing a single timer across sessions
// conflated cross-session interleaving as parent thinking time. each timer
// formats its log lines through the session label so attribution is visible.
const thinkingTimers = new Map<string, ThinkingTimer>();
function timerFor(label: string): ThinkingTimer {
let t = thinkingTimers.get(label);
if (!t) {
const formatLine = (line: string) =>
label === ORCHESTRATOR_LABEL ? line : formatWithLabel(label, line);
t = new ThinkingTimer(formatLine);
thinkingTimers.set(label, t);
}
return t;
}
let finalOutput = "";
let sessionId: string | undefined;
@@ -277,43 +347,73 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
}
const handlers = {
system: (_event: ClaudeSystemEvent) => {
log.debug(`» ${params.label} system event`);
system: (event: ClaudeSystemEvent) => {
// claude-agent-sdk only emits system:init for the top-level query, so
// this binds the orchestrator label and never appears in subagent flow.
// we still route through eventLabel so a subagent system event (if the
// SDK ever adds one) wouldn't go silently misattributed.
const label = eventLabel(event);
log.debug(withLabel(label, `» ${params.label} system event`));
},
assistant: (event: ClaudeAssistantEvent) => {
const content = event.message?.content;
if (!content) return;
const label = eventLabel(event);
const boxTitle = label === ORCHESTRATOR_LABEL ? params.label : `${params.label} [${label}]`;
for (const block of content) {
if (block.type === "text" && block.text?.trim()) {
const message = block.text.trim();
log.box(message, { title: params.label });
finalOutput = message;
log.box(message, { title: boxTitle });
// only the orchestrator's text becomes the run's "output" — subagent
// report-back text would otherwise clobber the parent's final answer.
if (label === ORCHESTRATOR_LABEL) {
finalOutput = message;
}
} else if (block.type === "tool_use") {
const toolName = block.name || "unknown";
// suspend the activity watchdog across the tool call. claude's
// stdout pipe goes silent while it awaits the synchronous MCP
// tools/call HTTP response; without this, long fetches/deepens
// (issue #760) trip the spawn-level idle timer at 300s. paired
// with resumeActivity() in tool_result below; bounded by the
// MAX_TOOL_CALL_SUSPENSION_MS auto-resume in activity.ts.
suspendActivity();
if (params.onToolUse) {
params.onToolUse({
toolName,
input: block.input,
});
}
thinkingTimer.markToolCall();
log.toolCall({ toolName, input: block.input || {} });
timerFor(label).markToolCall();
const inputFormatted = formatJsonValue(block.input || {});
const toolCallLine =
inputFormatted !== "{}" ? `» ${toolName}(${inputFormatted})` : `» ${toolName}()`;
log.info(withLabel(label, toolCallLine));
// surface the subagent identity when the orchestrator dispatches a
// Task — claude rolls subagent activity up into a single tool_result
// (no per-event session_id in its stream), so this log line is the
// only attribution available before the subagent's report-back.
if (toolName === "Task" && block.input && typeof block.input === "object") {
// when the orchestrator dispatches a subagent, bind the Agent
// tool_use id to the dispatched label so future events carrying
// `parent_tool_use_id === block.id` resolve directly to the right
// lens. v2.1.63+ renamed the tool to "Agent"; older versions
// emitted "Task". match both for forward-compat.
if (
(toolName === "Task" || toolName === "Agent") &&
block.input &&
typeof block.input === "object"
) {
const taskInput = block.input as {
description?: string;
subagent_type?: string;
prompt?: string;
};
const label = deriveLabelFromTaskInput(taskInput);
const dispatchedLabel = labeler.recordTaskDispatch(taskInput, block.id ?? null);
log.info(
`» dispatching subagent: ${label}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
withLabel(
label,
`» dispatching subagent: ${dispatchedLabel}` +
(taskInput.subagent_type ? ` (subagent_type=${taskInput.subagent_type})` : "")
)
);
}
@@ -323,8 +423,14 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
params.todoTracker.cancel();
}
// parse TodoWrite events for live progress tracking
if (toolName === "TodoWrite" && params.todoTracker?.enabled) {
// parse TodoWrite events for live progress tracking. only honor the
// orchestrator's todos — subagents emit their own todo lists which
// would otherwise clobber the visible progress comment.
if (
toolName === "TodoWrite" &&
params.todoTracker?.enabled &&
label === ORCHESTRATOR_LABEL
) {
params.todoTracker.update(block.input);
}
}
@@ -345,10 +451,13 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
const content = event.message?.content;
if (!content) return;
const label = eventLabel(event);
for (const block of content) {
if (typeof block === "string") continue;
if (block.type === "tool_result") {
thinkingTimer.markToolResult();
resumeActivity();
timerFor(label).markToolResult();
const outputContent =
typeof block.content === "string"
@@ -366,9 +475,9 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
: String(block.content);
if (block.is_error) {
log.info(`» tool error: ${outputContent}`);
log.info(withLabel(label, `» tool error: ${outputContent}`));
} else {
log.debug(`» tool output: ${outputContent}`);
log.debug(withLabel(label, `» tool output: ${outputContent}`));
}
}
}
@@ -459,10 +568,18 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
};
const recentStderr: string[] = [];
// ring buffer of recent non-JSON stdout lines. Claude CLI prints
// human-readable TTY chrome (status bubbles, quota notices, etc.)
// alongside the NDJSON event stream. when the CLI exits non-zero without
// emitting a structured error event, these lines are the only actionable
// signal — preferring them over the NDJSON tail keeps progress comments
// readable. issue #643.
const recentNonJsonStdout: string[] = [];
let lastProviderError: string | null = null;
let output = "";
// capped accumulator — see opencode.ts for rationale (issue #680).
const output = new TailBuffer(DEFAULT_MAX_RETAINED_BYTES);
let stdoutBuffer = "";
try {
@@ -473,6 +590,7 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
env: params.env,
activityTimeout: 300_000,
onActivityTimeout: params.onActivityTimeout,
isPausedExternally: isActivitySuspended,
stdio: ["ignore", "pipe", "pipe"],
// run claude in its own process group so SIGKILL on activity timeout /
// outer cancellation reaches any subprocesses it spawns (rg, file
@@ -480,9 +598,14 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
// there's no shim-orphan issue like opencode-ai/bin/opencode, but
// detached + killGroup is the right default for any agent runtime.
killGroup: true,
// claude already drains every chunk via onStdout (NDJSON parsing) and
// onStderr (recentStderr ring buffer). retaining a second copy in the
// spawn wrapper would grow unbounded for long sessions and previously
// crashed the wrapper with RangeError. see issue #680.
retain: "none",
onStdout: async (chunk) => {
const text = chunk.toString();
output += text;
output.append(text);
markActivity();
stdoutBuffer += text;
@@ -498,6 +621,8 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
event = JSON.parse(trimmed) as ClaudeEvent;
} catch {
log.debug(`» non-JSON stdout line: ${trimmed.substring(0, 200)}`);
recentNonJsonStdout.push(trimmed);
if (recentNonJsonStdout.length > MAX_STDERR_LINES) recentNonJsonStdout.shift();
continue;
}
@@ -533,10 +658,10 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
recentStderr.push(trimmed);
if (recentStderr.length > MAX_STDERR_LINES) recentStderr.shift();
const providerError = detectProviderError(trimmed);
if (providerError) {
lastProviderError = providerError;
log.info(`» provider error detected (${providerError}): ${trimmed.substring(0, 500)}`);
const match = findProviderErrorMatch(trimmed);
if (match) {
lastProviderError = match.label;
log.info(`» provider error detected (${match.label}): ${match.excerpt}`);
} else {
log.debug(trimmed);
}
@@ -592,20 +717,33 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
// hides the actionable provider message and pollutes the run log. cap
// the stdout fallback to the last 2KB so it stays readable when neither
// a structured error nor stderr is available.
const truncatedStdout = result.stdout ? tailLines(result.stdout, 2048) : "";
//
// result.stdout / result.stderr are empty because we pass retain:"none"
// to spawn (see issue #680); the agent layer keeps its own bounded
// mirrors via `output` (TailBuffer) and `recentStderr` (ring buffer).
const stdoutSnapshot = output.toString();
const stderrSnapshot = recentStderr.join("\n");
const truncatedStdout = stdoutSnapshot ? tailLines(stdoutSnapshot, 2048) : "";
// prefer non-JSON stdout (human-readable TTY chrome the CLI prints,
// including status bubbles and quota notices) over the raw NDJSON
// tail. when the CLI exits 1 without emitting `is_error` (issue #643),
// the NDJSON fallback would otherwise dump 2KB of `system/init` events
// into the progress comment with no mention of the actual cause.
const nonJsonStdoutSnapshot = recentNonJsonStdout.join("\n");
const errorMessage =
lastResultError ||
result.stderr ||
stderrSnapshot ||
nonJsonStdoutSnapshot ||
truncatedStdout ||
`unknown error - no output from Claude CLI${errorContext}`;
log.error(
`${params.label} exited with code ${result.exitCode}${errorContext}: ${errorMessage}`
);
log.debug(`stdout: ${result.stdout?.substring(0, 500)}`);
log.debug(`stderr: ${result.stderr?.substring(0, 500)}`);
log.debug(`stdout: ${stdoutSnapshot.substring(0, 500)}`);
log.debug(`stderr: ${stderrSnapshot.substring(0, 500)}`);
return {
success: false,
output: finalOutput || output,
output: finalOutput || stdoutSnapshot,
error: errorMessage,
usage,
sessionId,
@@ -615,7 +753,7 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
if (eventCount === 0 && lastProviderError) {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `provider error: ${lastProviderError}`,
usage,
sessionId,
@@ -625,14 +763,14 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
if (resultErrorSubtype) {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: lastResultError || `result subtype: ${resultErrorSubtype}`,
usage,
sessionId,
};
}
return { success: true, output: finalOutput || output, usage, sessionId };
return { success: true, output: finalOutput || output.toString(), usage, sessionId };
} catch (error) {
params.todoTracker?.cancel();
const duration = performance.now() - startTime;
@@ -658,7 +796,7 @@ export async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `${errorMessage} [${diagnosis}]`,
usage: buildUsage(),
sessionId,
@@ -679,6 +817,14 @@ const MANAGED_SETTINGS_PATH = `${MANAGED_SETTINGS_DIR}/managed-settings.json`;
// allowManagedPermissionRulesOnly prevents malicious PRs from adding allow rules that override
// our deny rules — safe in CI because --dangerously-skip-permissions makes allow/ask irrelevant.
// allowManagedHooksOnly prevents malicious project hooks from bypassing deny rules.
// Codex auth.json (Pullfrog-stored ChatGPT subscription credential) lives at
// `~/.local/share/opencode/auth.json` when the opencode harness materialized
// it. Claude shouldn't be running OpenAI models — they route to opencode —
// but defense-in-depth: deny the file regardless. Per Claude Code permissions
// docs, Read(...) deny ALSO blocks file-reading Bash commands (cat, head,
// tail, sed) and survives bypassPermissions mode. See wiki/codex-auth.md.
const CODEX_AUTH_DENY_PATH = "~/.local/share/opencode/auth.json";
const managedSettings = {
allowManagedPermissionRulesOnly: true,
allowManagedHooksOnly: true,
@@ -692,11 +838,15 @@ const managedSettings = {
"Edit(//sys/**)",
"Glob(//proc/**)",
"Glob(//sys/**)",
`Read(${CODEX_AUTH_DENY_PATH})`,
`Grep(${CODEX_AUTH_DENY_PATH})`,
`Edit(${CODEX_AUTH_DENY_PATH})`,
`Glob(${CODEX_AUTH_DENY_PATH})`,
],
},
sandbox: {
filesystem: {
denyRead: ["/proc", "/sys"],
denyRead: ["/proc", "/sys", CODEX_AUTH_DENY_PATH],
},
},
};
@@ -726,7 +876,22 @@ export const claude = agent({
const cliPath = await installClaudeCli();
const specifier = ctx.payload.proxyModel ?? ctx.resolvedModel;
const model = specifier ? stripProviderPrefix(specifier) : undefined;
// claude-code on Bedrock takes the bare AWS model ID — no provider prefix
// to strip, since the ID is already in `provider.model` form (e.g.
// `us.anthropic.claude-opus-4-7`). detect via the env-var sentinel: if
// BEDROCK_MODEL_ID is set and matches the resolved specifier, this is a
// bedrock route. see `wiki/model-resolution.md` for the routing pattern.
const bedrockModelId = process.env[BEDROCK_MODEL_ID_ENV]?.trim();
const isBedrockRoute =
specifier !== undefined &&
bedrockModelId !== undefined &&
bedrockModelId === specifier &&
isBedrockAnthropicId(specifier);
const model = !specifier
? undefined
: isBedrockRoute
? specifier
: stripProviderPrefix(specifier);
const homeEnv = {
HOME: ctx.tmpdir,
@@ -773,12 +938,45 @@ export const claude = agent({
// agent process gets full env — needs LLM API keys, PATH, locale, etc.
// security is enforced via managed-settings.json, --disallowedTools (Bash), and MCP tool filtering.
//
// bedrock route: claude-code reads `CLAUDE_CODE_USE_BEDROCK=1` to switch
// its provider implementation from the direct Anthropic API to Bedrock.
// AWS_BEARER_TOKEN_BEDROCK / AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY +
// AWS_REGION are already in process.env from the workflow's `env:` block.
// see https://docs.claude.com/en/docs/claude-code/amazon-bedrock.
//
// we only force CLAUDE_CODE_USE_BEDROCK=1 when this is a Pullfrog-routed
// bedrock run; if the user has set the env var manually for some other
// reason (e.g. always-Bedrock org policy), `...process.env` already
// carries it through and we don't disturb it.
const repoDir = process.cwd();
// PWD must match the spawn cwd (see opencode_v2.ts for the analogous fix).
// claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
// additional-working-directory when it differs from `process.cwd()` (per
// the bundled cli.js — `let H=process.env.PWD; if(H && H !== Y7() && ...)
// j.set(H, {path: H, source: "session"})`). Inheriting harness PWD via
// `...process.env` ends up adding the wrong dir to the agent's allowed
// working set under `pnpm runtest` / `pnpm play`, which silently confuses
// path-relative tools.
const env: Record<string, string | undefined> = {
...process.env,
...homeEnv,
PWD: repoDir,
};
if (isBedrockRoute) {
env.CLAUDE_CODE_USE_BEDROCK = "1";
}
const repoDir = process.cwd();
// claude-code's `Vw()` resolver prefers ANTHROPIC_API_KEY over the OAuth
// token when both are set, so we strip the API key to fall through to the
// Max-subscription path. bedrock route uses AWS creds and is excluded.
if (env.CLAUDE_CODE_OAUTH_TOKEN && !isBedrockRoute && env.ANTHROPIC_API_KEY) {
log.debug(
"» CLAUDE_CODE_OAUTH_TOKEN present — stripping ANTHROPIC_API_KEY from Claude Code env so the OAuth subscription is used"
);
delete env.ANTHROPIC_API_KEY;
}
log.info(`» effort: ${effort}`);
log.debug(`» starting Pullfrog (Claude Code): node ${baseArgs.join(" ")}`);
@@ -808,9 +1006,10 @@ export const claude = agent({
ctx,
initialResult: result,
initialUsage: result.usage,
reflectionPrompt: ctx.toolState.learningsFilePath
? buildLearningsReflectionPrompt(ctx.toolState.learningsFilePath)
: undefined,
reflectionPrompt:
ctx.toolState.learningsFilePath && shouldRunReflection(ctx.toolState.selectedMode)
? buildLearningsReflectionPrompt(ctx.toolState.learningsFilePath)
: undefined,
canResume: (r) => Boolean(r.sessionId),
resume: async (c) => {
const sessionId = c.previousResult.sessionId;
+4 -1
View File
@@ -1,5 +1,8 @@
import { claude } from "./claude.ts";
import { opencode } from "./opencode.ts";
// v2 harness — adapted to opencode-ai >=1.14.x SDK-v2 / Effect-ts CLI rewrite.
// The legacy v1 module (`./opencode.ts`) is kept around for reference + fast
// revert; the active runner is the v2 module below.
import { opencode } from "./opencode_v2.ts";
import type { Agent } from "./shared.ts";
export type { Agent, AgentUsage } from "./shared.ts";
+34
View File
@@ -0,0 +1,34 @@
import { describe, expect, it } from "vitest";
import { modelAliases } from "../models.ts";
import { geminiHighThinkingOverrides } from "./opencode.ts";
describe("geminiHighThinkingOverrides", () => {
// Expected truth pulled the same way the helper does — both must derive from
// the registry so the test exercises the wiring, not a hand-maintained list.
const expectedApiIds = modelAliases
.filter((a) => a.provider === "google")
.map((a) => a.resolve.replace(/^google\//, ""));
const overrides = geminiHighThinkingOverrides();
it("covers every direct-Google alias in the registry", () => {
expect(Object.keys(overrides).sort()).toEqual([...expectedApiIds].sort());
});
it("is non-empty (catches accidental whole-provider removal)", () => {
expect(Object.keys(overrides).length).toBeGreaterThan(0);
});
it("strips the `google/` prefix from each resolve to get the bare API id", () => {
for (const id of Object.keys(overrides)) {
expect(id).not.toMatch(/^google\//);
}
});
it("pins every entry to thinkingLevel: high", () => {
for (const [id, value] of Object.entries(overrides)) {
expect(value, `entry for ${id}`).toEqual({
options: { thinkingConfig: { thinkingLevel: "high" } },
});
}
});
});
+263 -202
View File
@@ -11,18 +11,32 @@
* the agent process itself gets full env (needs LLM API keys, PATH, etc.).
* security is enforced at the tool layer, not the process layer.
*/
import { execFileSync } from "node:child_process";
import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { performance } from "node:perf_hooks";
import * as core from "@actions/core";
import { pullfrogMcpName } from "../external.ts";
import { modelAliases } from "../models.ts";
import { getIdleMs, markActivity } from "../utils/activity.ts";
import { BEDROCK_MODEL_ID_ENV } from "../models.ts";
import type { ToolState } from "../toolState.ts";
import {
getIdleMs,
isActivitySuspended,
markActivity,
resumeActivity,
suspendActivity,
} from "../utils/activity.ts";
import { type AgentDiagnostic, formatAgentHangBody } from "../utils/agentHangReport.ts";
import { formatJsonValue, log } from "../utils/cli.ts";
import { installFromNpmTarball } from "../utils/install.ts";
import { detectProviderError } from "../utils/providerErrors.ts";
import { installCodexAuth } from "../utils/codexHome.ts";
import { findProviderErrorMatch } from "../utils/providerErrors.ts";
import { addSkill, installBundledSkills } from "../utils/skills.ts";
import { SPAWN_ACTIVITY_TIMEOUT_CODE, SpawnTimeoutError, spawn } from "../utils/subprocess.ts";
import {
DEFAULT_MAX_RETAINED_BYTES,
SPAWN_ACTIVITY_TIMEOUT_CODE,
SpawnTimeoutError,
spawn,
TailBuffer,
} from "../utils/subprocess.ts";
import { ThinkingTimer } from "../utils/timer.ts";
import type { TodoTracker } from "../utils/todoTracking.ts";
import { getDevDependencyVersion } from "../utils/version.ts";
@@ -31,8 +45,19 @@ import {
PULLFROG_OPENCODE_PLUGIN_FILENAME,
PULLFROG_OPENCODE_PLUGIN_SOURCE,
} from "./opencodePlugin.ts";
import { buildLearningsReflectionPrompt, runPostRunRetryLoop } from "./postRun.ts";
import { REVIEWER_AGENT_NAME, REVIEWER_SYSTEM_PROMPT } from "./reviewer.ts";
import {
autoSelectModel,
buildReviewerAgentConfig,
geminiHighThinkingOverrides,
installOpencodeCli,
type OpenCodeConfig,
} from "./opencodeShared.ts";
import {
buildLearningsReflectionPrompt,
runPostRunRetryLoop,
shouldRunReflection,
} from "./postRun.ts";
import { REVIEWER_AGENT_NAME } from "./reviewer.ts";
import { formatWithLabel, ORCHESTRATOR_LABEL, SessionLabeler } from "./sessionLabeler.ts";
import {
type AgentResult,
@@ -43,62 +68,32 @@ import {
MAX_STDERR_LINES,
} from "./shared.ts";
async function installOpencodeCli(): Promise<string> {
return await installFromNpmTarball({
packageName: "opencode-ai",
version: getDevDependencyVersion("opencode-ai"),
executablePath: "bin/opencode",
installDependencies: true,
});
}
// re-export for the existing test (`./opencode.test.ts`) — once v1 is
// retired this module collapses and the test imports from opencodeShared.
export { geminiHighThinkingOverrides } from "./opencodeShared.ts";
// ── config ─────────────────────────────────────────────────────────────────────
// v1.4-era npm package shipped a per-platform binary directly at this path.
const installCli = () => installOpencodeCli({ binPath: "bin/opencode" });
type OpenCodeConfig = {
mcp?: Record<string, unknown>;
permission?: Record<string, unknown>;
provider?: Record<string, unknown>;
agent?: Record<string, unknown>;
model?: string;
enabled_providers?: string[];
[key: string]: unknown;
};
/**
* Per-inference `max_tokens` reservation the agent sends to the upstream
* model. OpenCode's default is 32_000 (sized for long-running TUI sessions
* where a human user might want big outputs). Pullfrog runs are headless and
* short — typical outputs are 1-3K tokens — so we cap at 5_000. This
* drastically reduces the upfront budget reservation OpenRouter requires per
* call (~$0.38 vs ~$2.40 for Opus), which is what lets low-wallet runs
* actually start.
*
* Plumbed via `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX` env var rather than the
* config JSON. OpenCode's `OUTPUT_TOKEN_MAX` (session/llm.ts) is sourced
* exclusively from this env var; top-level `limit.output` in the config
* has no read site and is silently dropped on merge.
*/
const PULLFROG_OPENCODE_OUTPUT_LIMIT = 5000;
/**
* upstream opencode hardcodes `thinkingLevel: "high"` as the default for every
* gemini-3 model on the direct google SDK (`provider/transform.ts` `options()`).
* that adds 30-60s of pre-tool-call TTFT and 5-46s of post-tool jabber per turn,
* which is overkill for agentic loops where most steps are tool-routing
* decisions. we override to "medium" for the curated slugs we ship in
* `action/models.ts`; users who want max quality can still pick the `-high`
* variant explicitly. flash stays at "medium" too — low-effort flash is
* visibly worse on harder tasks and the latency savings aren't meaningful
* (flash is already fast). other gemini-3 ids that exist in models.dev but
* aren't in our curated alias map keep the upstream `"high"` default.
*
* keyed by upstream api id (matches the slugs in `action/models.ts`). the
* merge order in opencode `session/llm.ts` is `base ← model.options ← agent.options ← variant`,
* deep-merged — so an explicit `--variant high` still wins, and explicit
* model.options in a user-provided opencode config would also win.
*/
const GEMINI_3_DIRECT_THINKING_LEVEL = "medium";
const GEMINI_3_DIRECT_API_IDS = ["gemini-3.1-pro-preview", "gemini-3-flash-preview"];
// NOTE: OpenCode's per-call `max_tokens` defaults to 32_000. We previously
// overrode this via `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX = 5000` in #616
// to lower OpenRouter's per-call upfront budget reservation — back when the
// `ROUTER_PER_RUN_LIMIT_USD = 25` per-run key cap meant that reservation was
// a hard gate that could lock low-balance accounts out of starting a run.
//
// That gate is gone (see `app/api/proxy-token/route.ts` ~line 422 — "Per-run
// key budget … is decoupled from wallet balance"); the router now mints
// keys with `keyLimitCents = balance + buffer` ($50 / $5 / $0). The override
// no longer materially helps, and as a hard per-call output truncation it
// actively hurt: a single `create_pull_request_review` tool_use with many
// inline comments would truncate mid-stream past 5K output tokens, the JSON
// was unparseable, and the tool never invoked. We hit this on PR #710's
// verify-downshift PR. Removed in #710 — using OpenCode's 32K default.
//
// If you need to re-cap output for some reason, set
// `OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX` in the action env. OpenCode's
// top-level `limit.output` config field has no read site (silently dropped
// on merge in session/llm.ts), so the env var is the only working knob.
function buildSecurityConfig(ctx: AgentRunContext, model: string | undefined): string {
const config: OpenCodeConfig = {
@@ -113,21 +108,28 @@ function buildSecurityConfig(ctx: AgentRunContext, model: string | undefined): s
mcp: {
[pullfrogMcpName]: { type: "remote", url: ctx.mcpServerUrl },
},
agent: buildReviewerAgentConfig(),
provider: {
google: {
models: Object.fromEntries(
GEMINI_3_DIRECT_API_IDS.map((id) => [
id,
{
options: {
thinkingConfig: { thinkingLevel: GEMINI_3_DIRECT_THINKING_LEVEL },
},
},
])
),
},
},
agent: (() => {
const cfg = buildReviewerAgentConfig(model);
const reviewerModel = (cfg[REVIEWER_AGENT_NAME] as { model?: string })?.model ?? "(inherit)";
log.info(`» subagent models: reviewfrog=${reviewerModel}`);
return cfg;
})(),
// NOTE: `experimental.batch_tool` was enabled in #719 to bundle 1-25
// independent tool calls into one round trip, but the batch tool rejects
// MCP/"external" tools with `"Tool '<name>' not in registry. External
// tools (MCP, environment) cannot be batched - call them directly."`
// (anomalyco/opencode PR #2983 design). when a model emits parallel
// tool_use blocks containing `pullfrog_*` calls, opencode internally
// routes them through batch — they all fail, the model misreads the
// error as "the tool doesn't exist", and gives up. caught in CI by
// `restricted-opencode` after a `lens:` subagent dispatched parallel
// `pullfrog_shell` calls and concluded shell was unavailable.
// native parallel tool_use (multiple tool_use blocks per assistant
// message) still works without batch_tool for both built-in and MCP
// tools, so we lose only the batch wrapper, not parallelism.
// gemini-3 thinking pinned to high for review depth; gpt and anthropic
// effort set elsewhere (gpt: upstream default, anthropic: --effort flag in claude.ts).
provider: { google: { models: geminiHighThinkingOverrides() } },
};
if (model) {
@@ -142,76 +144,6 @@ function buildSecurityConfig(ctx: AgentRunContext, model: string | undefined): s
return JSON.stringify(config);
}
/**
* Read-only subagent for self-review and /anneal lens dispatch. The
* non-mutative + non-recursive contract is enforced by the prose system
* prompt — see action/agents/reviewer.ts for why we no longer wire per-agent
* tool/permission denies here.
*/
function buildReviewerAgentConfig(): Record<string, unknown> {
return {
[REVIEWER_AGENT_NAME]: {
description:
"Read-only review subagent for self-review and lens-based code review. " +
"Reads only — no writes, no state-changing shell or MCP calls, no nested subagent dispatch.",
mode: "subagent",
prompt: REVIEWER_SYSTEM_PROMPT,
},
};
}
// ── model auto-select fallback ──────────────────────────────────────────────────
//
// steps 12 of model resolution (PULLFROG_MODEL env, slug resolution) are handled
// by resolveModel() in utils/agent.ts before the agent runs. this fallback only
// handles step 3: auto-select via `opencode models`.
function getOpenCodeModels(cliPath: string): string[] {
try {
const output = execFileSync(cliPath, ["models"], {
encoding: "utf-8",
timeout: 30_000,
env: process.env,
});
return output
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
} catch (error) {
log.debug(
`» failed to run \`opencode models\`: ${error instanceof Error ? error.message : String(error)}`
);
return [];
}
}
const AUTO_SELECT_WARNING =
"select a model explicitly in the Pullfrog console (https://pullfrog.com/console) to avoid this.";
function autoSelectModel(cliPath: string): string | undefined {
const availableModels = getOpenCodeModels(cliPath);
const availableSet = new Set(availableModels);
if (availableSet.size > 0) {
log.debug(`» opencode models (${availableSet.size}): ${availableModels.join(", ")}`);
const match =
modelAliases.find((a) => a.preferred && availableSet.has(a.resolve)) ??
modelAliases.find((a) => availableSet.has(a.resolve));
if (match) {
log.info(
`» model: ${match.resolve} (auto-selected${match.preferred ? " — preferred" : ""} curated match)`
);
log.warning(`» model auto-selected. ${AUTO_SELECT_WARNING}`);
return match.resolve;
}
log.info(
`» opencode has ${availableSet.size} models but none match curated aliases — letting OpenCode auto-select`
);
}
log.warning(`» no model resolved. letting OpenCode auto-select. ${AUTO_SELECT_WARNING}`);
return undefined;
}
// ── NDJSON event types ─────────────────────────────────────────────────────────
interface OpenCodeInitEvent {
@@ -267,6 +199,20 @@ interface OpenCodeStepFinishEvent {
[key: string]: unknown;
}
/**
* tool-part state, mirroring opencode's `ToolState` (anomalyco/opencode
* `session/message-v2.ts`). error parts carry the reason on `error`,
* completed parts on `output` — reading the wrong field is what caused
* the silent `(no error message)` log in #662.
*
* Named `ToolPartState` locally (not `ToolState`) so it doesn't shadow the
* action-wide `ToolState` imported above.
*/
type ToolPartState =
| { status: "pending" | "running"; input?: unknown }
| { status: "completed"; input?: unknown; output: string }
| { status: "error"; input?: unknown; error: string };
interface OpenCodeToolUseEvent {
type: "tool_use";
timestamp?: number;
@@ -275,7 +221,7 @@ interface OpenCodeToolUseEvent {
id?: string;
callID?: string;
tool?: string;
state?: { status?: string; input?: unknown; output?: string };
state?: ToolPartState;
};
[key: string]: unknown;
}
@@ -284,7 +230,7 @@ interface OpenCodeToolResultEvent {
type: "tool_result";
timestamp?: number;
sessionID?: string;
part?: { callID?: string; state?: { status?: string; output?: string } };
part?: { callID?: string; state?: ToolPartState };
tool_id?: string;
status?: "success" | "error";
output?: string;
@@ -369,6 +315,7 @@ type RunParams = {
args: string[];
cwd: string;
env: Record<string, string | undefined>;
toolState: ToolState;
todoTracker?: TodoTracker | undefined;
onActivityTimeout?: (() => void) | undefined;
onToolUse?: ((event: { toolName: string; input: unknown }) => void) | undefined;
@@ -377,7 +324,6 @@ type RunParams = {
async function runOpenCode(params: RunParams): Promise<AgentResult> {
const startTime = performance.now();
let eventCount = 0;
const thinkingTimer = new ThinkingTimer();
let finalOutput = "";
let accumulatedTokens = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
@@ -409,6 +355,23 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
return label === ORCHESTRATOR_LABEL ? message : formatWithLabel(label, message);
}
// one ThinkingTimer per session — sharing a single timer across sessions
// conflated cross-session interleaving (parent thinks → child tool_call,
// or child returns → parent dispatches next) as parent thinking time. each
// timer formats its log lines through the session label so the "thought
// for X" attribution is visible in the merged stream.
const thinkingTimers = new Map<string, ThinkingTimer>();
function timerFor(label: string): ThinkingTimer {
let t = thinkingTimers.get(label);
if (!t) {
const formatLine = (line: string) =>
label === ORCHESTRATOR_LABEL ? line : formatWithLabel(label, line);
t = new ThinkingTimer(formatLine);
thinkingTimers.set(label, t);
}
return t;
}
// tracks per-task dispatch metadata so the matching tool_result can log a
// labeled "» subagent finished: lens=X duration=Ys" line. this is the most
// useful per-lens observability available given that subagent-internal
@@ -583,6 +546,21 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
return;
}
// suspend the activity watchdog across the tool call (issue #760).
// for `task` tool dispatches the injected plugin already reverbs
// child.stdout chunks, so this is mostly defense-in-depth there;
// for non-task MCP tools (checkout_pr, etc.) the suspend is the
// only thing keeping a multi-minute fetch from tripping the 300s
// spawn-level idle timer. gate by part status: bus-envelope
// re-dispatches at line 915 fire only on terminal statuses
// (`completed`/`error`) and never produce a paired `tool_result`,
// so suspending on those would leak the watchdog open until the
// 15min auto-resume — exactly the issue #12 zombie-run window.
const status = event.part?.state?.status;
if (status !== "completed" && status !== "error") {
suspendActivity();
}
// when the orchestrator dispatches a subagent via the `task` tool, push
// a label for the upcoming child session so its events are attributable.
// record BEFORE label lookup: this event's session is the parent (whose
@@ -634,7 +612,7 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
});
}
thinkingTimer.markToolCall();
timerFor(label).markToolCall();
const inputFormatted = formatJsonValue(event.part?.state?.input || {});
const toolCallLine =
inputFormatted !== "{}" ? `» ${toolName}(${inputFormatted})` : `» ${toolName}()`;
@@ -647,11 +625,9 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
// status="error" through the same `tool_use` event the CLI's run-loop
// (and our injected plugin for subagent parts) emits — without this
// branch the only signal in the user's logs is `» <tool>(...)` with
// no indication the call failed. error info lives in `state.output`
// (an error string set by the tool layer).
// no indication the call failed.
if (event.part?.state?.status === "error") {
const errorMsg = event.part.state.output ?? "(no error message)";
log.info(withLabel(label, `» tool call failed: ${errorMsg}`));
log.info(withLabel(label, `» tool call failed: ${event.part.state.error}`));
}
// agent's explicit MCP report_progress takes priority over todo tracking
@@ -666,12 +642,19 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
}
},
tool_result: (event: OpenCodeToolResultEvent) => {
resumeActivity();
const toolId = event.part?.callID || event.tool_id;
const status = event.part?.state?.status || event.status || "unknown";
const output = event.part?.state?.output || event.output;
const state = event.part?.state;
const status = state?.status ?? event.status ?? "unknown";
const payload =
state?.status === "completed"
? state.output
: state?.status === "error"
? state.error
: event.output;
const label = eventLabel(event);
thinkingTimer.markToolResult();
timerFor(label).markToolResult();
// surface subagent completion at info level — opencode otherwise hides
// per-task timing in debug-only logs, so a parallel multi-lens fan-out
@@ -687,12 +670,12 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
if (taskDispatchByCallID.size > 0 || pendingTaskDispatches.length > 0) {
if (toolId && taskDispatchByCallID.has(toolId)) {
const dispatch = taskDispatchByCallID.get(toolId);
if (dispatch) emitSubagentFinished(dispatch, status, output, "exact");
if (dispatch) emitSubagentFinished(dispatch, status, payload, "exact");
} else {
const callIDIsKnownNonTask = toolId ? knownNonTaskCallIDs.has(toolId) : false;
if (!callIDIsKnownNonTask && pendingTaskDispatches.length > 0) {
const dispatch = pendingTaskDispatches[0]!;
emitSubagentFinished(dispatch, status, output, "fifo");
emitSubagentFinished(dispatch, status, payload, "fifo");
}
}
}
@@ -709,13 +692,8 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
`» ${params.label} tool_result${stepContext}: id=${toolId}, status=${status}, duration=${Math.round(toolDuration)}ms`
)
);
if (output) {
log.debug(
withLabel(
label,
` output: ${typeof output === "string" ? output : JSON.stringify(output)}`
)
);
if (payload) {
log.debug(withLabel(label, ` output: ${payload}`));
}
if (toolDuration > 5000) {
log.info(
@@ -728,11 +706,9 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
}
}
if (status === "error") {
const errorMsg = typeof output === "string" ? output : JSON.stringify(output);
log.info(withLabel(label, `» tool call failed: ${errorMsg}`));
} else if (output) {
const outputStr = typeof output === "string" ? output : JSON.stringify(output);
log.debug(withLabel(label, `tool output: ${outputStr}`));
log.info(withLabel(label, `» tool call failed: ${payload ?? "(no error message)"}`));
} else if (payload) {
log.debug(withLabel(label, `tool output: ${payload}`));
}
},
error: (event: OpenCodeErrorEvent) => {
@@ -872,7 +848,25 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
let lastProviderError: string | null = null;
let agentErrorEvent: OpenCodeErrorEvent | null = null;
let output = "";
// shared with main.ts via toolState. updated in place as events stream and
// stderr accumulates so the outer activity-timeout catch sees the same
// context the harness's own catch path uses to format `result.error`.
// recentStderr is shared by reference; the scalar fields are mirrored on
// each update below.
const diagnostic: AgentDiagnostic = {
label: params.label,
recentStderr,
lastProviderError: undefined,
eventCount: 0,
};
params.toolState.agentDiagnostic = diagnostic;
// capped accumulator for the agent's narration. used as a post-run fallback
// when `finalOutput` (the orchestrator's final assistant message) is empty.
// unbounded `output += text` previously grew to ~1 GiB on multi-lens Reviews
// and contributed to the wrapper-level RangeError. retain:"none" on spawn
// skips the duplicate buffer there; this TailBuffer caps the agent layer.
const output = new TailBuffer(DEFAULT_MAX_RETAINED_BYTES);
let stdoutBuffer = "";
try {
@@ -891,16 +885,23 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
// never fires — producing zombie runs. detached + killGroup nukes the
// whole tree.
killGroup: true,
// NB: we used to pass `isPausedExternally: isSubagentInFlight` to suspend
// the activity timer during subagent dispatches. unnecessary now that
// our injected plugin (action/agents/opencodePlugin.ts) re-emits
// subagent `message.part.updated` events on opencode's stdout — those
// arrive at child.stdout here, fire updateActivity(), and reset
// lastActivityTime naturally. verified empirically in PR #634
// (~3.3 plugin events/sec during a typical subagent run).
// we already drain every chunk via onStdout/onStderr (NDJSON parsing
// + recentStderr ring buffer). retaining a second copy in the spawn
// wrapper would grow unbounded for multi-lens Reviews and previously
// crashed the wrapper with RangeError at ~1 GiB. see issue #680.
retain: "none",
// suspend the spawn-level idle watchdog across MCP tool calls (issue
// #760). bracketed by suspendActivity()/resumeActivity() in the
// tool_use/tool_result handlers above, bounded by
// MAX_TOOL_CALL_SUSPENSION_MS in activity.ts. the injected plugin
// (action/agents/opencodePlugin.ts) re-emits subagent
// `message.part.updated` events on opencode's stdout, so subagent
// dispatches keep marking child.stdout activity as well — defense
// in depth (verified empirically in PR #634, ~3.3 plugin events/sec).
isPausedExternally: isActivitySuspended,
onStdout: async (chunk) => {
const text = chunk.toString();
output += text;
output.append(text);
markActivity();
stdoutBuffer += text;
@@ -920,6 +921,7 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
}
eventCount++;
diagnostic.eventCount = eventCount;
log.debug(JSON.stringify(event, null, 2));
const timeSinceLastActivity = getIdleMs();
@@ -958,10 +960,11 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
recentStderr.push(trimmed);
if (recentStderr.length > MAX_STDERR_LINES) recentStderr.shift();
const providerError = detectProviderError(trimmed);
if (providerError) {
lastProviderError = providerError;
log.info(`» provider error detected (${providerError}): ${trimmed.substring(0, 500)}`);
const match = findProviderErrorMatch(trimmed);
if (match) {
lastProviderError = match.label;
diagnostic.lastProviderError = match.label;
log.info(`» provider error detected (${match.label}): ${match.excerpt}`);
} else {
log.debug(trimmed);
}
@@ -1025,22 +1028,31 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
if (result.exitCode !== 0) {
const errorContext = lastProviderError ? ` (${lastProviderError})` : "";
// result.stdout / result.stderr are empty because we pass retain:"none"
// to spawn (see issue #680); use the agent's bounded mirrors instead.
const stdoutSnapshot = output.toString();
const stderrSnapshot = recentStderr.join("\n");
const errorMessage =
result.stderr ||
result.stdout ||
stderrSnapshot ||
stdoutSnapshot ||
`unknown error - no output from OpenCode CLI${errorContext}`;
log.error(
`${params.label} exited with code ${result.exitCode}${errorContext}: ${errorMessage}`
);
log.debug(`stdout: ${result.stdout?.substring(0, 500)}`);
log.debug(`stderr: ${result.stderr?.substring(0, 500)}`);
return { success: false, output: finalOutput || output, error: errorMessage, usage };
log.debug(`stdout: ${stdoutSnapshot.substring(0, 500)}`);
log.debug(`stderr: ${stderrSnapshot.substring(0, 500)}`);
return {
success: false,
output: finalOutput || stdoutSnapshot,
error: errorMessage,
usage,
};
}
if (eventCount === 0 && lastProviderError) {
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `provider error: ${lastProviderError}`,
usage,
};
@@ -1053,13 +1065,13 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
errorEvent.error?.data?.message || errorEvent.error?.name || JSON.stringify(errorEvent);
return {
success: false,
output: finalOutput || output,
output: finalOutput || output.toString(),
error: `${errorName}: ${errorMessage}`,
usage,
};
}
return { success: true, output: finalOutput || output, usage };
return { success: true, output: finalOutput || output.toString(), usage };
} catch (error) {
params.todoTracker?.cancel();
const duration = performance.now() - startTime;
@@ -1083,10 +1095,11 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
`» recent stderr (last ${Math.min(recentStderr.length, 10)} lines):\n${stderrContext}`
);
const body = formatAgentHangBody({ diagnostic, isHang: isActivityTimeout, errorMessage });
return {
success: false,
output: finalOutput || output,
error: `${errorMessage} [${diagnosis}]`,
output: finalOutput || output.toString(),
error: body ?? `${errorMessage} [${diagnosis}]`,
usage: buildUsage(),
};
}
@@ -1096,11 +1109,28 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
export const opencode = agent({
name: "opencode",
install: installOpencodeCli,
install: installCli,
run: async (ctx) => {
const cliPath = await installOpencodeCli();
const cliPath = await installCli();
const model = ctx.payload.proxyModel ?? ctx.resolvedModel ?? autoSelectModel(cliPath);
const rawModel = ctx.payload.proxyModel ?? ctx.resolvedModel ?? autoSelectModel(cliPath);
// bedrock route: opencode's `amazon-bedrock` provider expects the model
// string in `amazon-bedrock/<bedrock-id>` form. the bare AWS model ID
// (what the user puts in `BEDROCK_MODEL_ID`) needs the prefix added.
// detect via env-var sentinel — same pattern as claude.ts.
//
// we deliberately do NOT gate on `!isBedrockAnthropicId(rawModel)` here:
// Anthropic-on-Bedrock normally routes to claude-code (per `resolveAgent`),
// but `PULLFROG_AGENT=opencode` is the documented escape hatch for forcing
// opencode regardless. when that override fires, opencode still needs the
// `amazon-bedrock/` prefix or the provider lookup fails with
// "Model not found: <modelId>/.". the Anthropic-vs-other discriminant
// only belongs in `resolveAgent`.
const bedrockModelId = process.env[BEDROCK_MODEL_ID_ENV]?.trim();
const isBedrockRoute =
rawModel !== undefined && bedrockModelId !== undefined && bedrockModelId === rawModel;
const model = isBedrockRoute ? `amazon-bedrock/${rawModel}` : rawModel;
const homeEnv = {
HOME: ctx.tmpdir,
@@ -1133,12 +1163,20 @@ export const opencode = agent({
installBundledSkills({ home: homeEnv.HOME });
// materialize CODEX_AUTH_JSON (Pullfrog-stored Codex subscription
// credential) into the runner's REAL $HOME/.local/share/opencode/auth.json
// so OpenCode's CodexAuthPlugin picks it up and routes openai requests
// through the ChatGPT subscription instead of needing OPENAI_API_KEY.
// see action/utils/codexHome.ts and wiki/codex-auth.md.
const codexAuth = installCodexAuth();
// base args shared between initial run and continue runs
const baseArgs = ["run", "--format", "json", "--print-logs"];
// OPENCODE_PERMISSION has absolute highest precedence (merged after managed/MDM configs).
// external_directory gates ALL native filesystem tools (Read, Write, Edit, Glob, Grep, etc.)
// for paths outside the project root. last-match-wins: deny everything, then allow /tmp.
// auth.json sits under real $HOME (outside /tmp/*), so deny-default protects it.
const permissionOverride = JSON.stringify({
external_directory: { "*": "deny", "/tmp/*": "allow" },
});
@@ -1148,11 +1186,32 @@ export const opencode = agent({
...homeEnv,
OPENCODE_CONFIG_CONTENT: buildSecurityConfig(ctx, model),
OPENCODE_PERMISSION: permissionOverride,
OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX: PULLFROG_OPENCODE_OUTPUT_LIMIT.toString(),
GOOGLE_GENERATIVE_AI_API_KEY:
process.env.GOOGLE_GENERATIVE_AI_API_KEY || process.env.GEMINI_API_KEY,
};
if (codexAuth) {
// point OpenCode at the real-home XDG dir so it reads auth.json from
// where we wrote it (not the tmpdir-redirected default).
env.XDG_DATA_HOME = codexAuth.xdgDataHome;
// remove OPENAI_API_KEY so OpenCode's provider merge unambiguously
// picks the OAuth path. with both set, the merge order in opencode
// makes the effective key ambiguous.
delete env.OPENAI_API_KEY;
// hand the post-hook everything it needs to detect + persist refresh.
// post-hook runs in a fresh node process, so we have to ferry apiToken
// explicitly — env is preserved across main/post but our run-context
// JWT is computed at runtime and not put in env. see action/entryPost.ts.
core.saveState(
"codex_writeback",
JSON.stringify({
apiToken: ctx.apiToken,
authPath: codexAuth.authPath,
originalRefresh: codexAuth.originalRefresh,
})
);
}
const repoDir = process.cwd();
log.debug(`» starting Pullfrog (OpenCode): ${cliPath} ${baseArgs.join(" ")}`);
@@ -1163,6 +1222,7 @@ export const opencode = agent({
cliPath,
cwd: repoDir,
env,
toolState: ctx.toolState,
todoTracker: ctx.todoTracker,
onActivityTimeout: ctx.onActivityTimeout,
onToolUse: ctx.onToolUse,
@@ -1182,9 +1242,10 @@ export const opencode = agent({
ctx,
initialResult: result,
initialUsage: result.usage,
reflectionPrompt: ctx.toolState.learningsFilePath
? buildLearningsReflectionPrompt(ctx.toolState.learningsFilePath)
: undefined,
reflectionPrompt:
ctx.toolState.learningsFilePath && shouldRunReflection(ctx.toolState.selectedMode)
? buildLearningsReflectionPrompt(ctx.toolState.learningsFilePath)
: undefined,
resume: async (c) =>
runOpenCode({
...runParams,
+144
View File
@@ -0,0 +1,144 @@
// Shared helpers for the OpenCode agent harnesses (`./opencode.ts` v1 and
// `./opencode_v2.ts` v2). Pure config / model-registry / install glue —
// nothing here touches the NDJSON event loop, which differs between v1 and v2.
//
// Once v1 is deleted post-burn-in this module collapses back into v2; until
// then it keeps both runners synchronized so a config drift can't make v1 a
// silently-broken fallback.
import { execFileSync } from "node:child_process";
import { modelAliases } from "../models.ts";
import { log } from "../utils/cli.ts";
import { installFromNpmTarball } from "../utils/install.ts";
import { getDevDependencyVersion } from "../utils/version.ts";
import { REVIEWER_AGENT_NAME, REVIEWER_SYSTEM_PROMPT } from "./reviewer.ts";
import { deriveSubagentModels } from "./subagentModels.ts";
// ── config ─────────────────────────────────────────────────────────────────────
export type OpenCodeConfig = {
mcp?: Record<string, unknown>;
permission?: Record<string, unknown>;
provider?: Record<string, unknown>;
agent?: Record<string, unknown>;
experimental?: Record<string, unknown>;
model?: string;
enabled_providers?: string[];
[key: string]: unknown;
};
/**
* Build the `provider.google.models[id].options` map that pins every direct-Google
* Gemini alias to `thinkingLevel: "high"`. Sourced from the model registry so
* adding/renaming a Google alias in `action/models.ts` flows through automatically.
*/
export function geminiHighThinkingOverrides(): Record<string, { options: object }> {
return Object.fromEntries(
modelAliases
.filter((a) => a.provider === "google")
.map((a) => [
a.resolve.replace(/^google\//, ""),
{ options: { thinkingConfig: { thinkingLevel: "high" } } },
])
);
}
/**
* Read-only `reviewfrog` subagent for lens-based review. Non-mutative +
* non-recursive — enforced by the system prompt in reviewer.ts.
*
* Per-subagent `model:` override is driven by the registry in
* `action/models.ts` via each alias's `subagentModel` field. Currently wired:
* Anthropic opus → sonnet, OpenAI gpt-pro → gpt and gpt → gpt-5.4, Google
* gemini-pro → gemini-flash. Other providers inherit (no override).
*/
export function buildReviewerAgentConfig(
orchestratorModel: string | undefined
): Record<string, unknown> {
const overrides = deriveSubagentModels(orchestratorModel);
return {
[REVIEWER_AGENT_NAME]: {
description:
"Read-only review subagent for lens-based code review (correctness, security, billing-subsystem, etc.). " +
"Reads only — no writes, no state-changing shell or MCP calls, no nested subagent dispatch.",
mode: "subagent",
prompt: REVIEWER_SYSTEM_PROMPT,
...(overrides.reviewer !== undefined ? { model: overrides.reviewer } : {}),
},
};
}
// ── install ────────────────────────────────────────────────────────────────────
/**
* Install the opencode-ai npm tarball and return the path to the executable.
*
* The bin path differs by version: v1.4.x and earlier shipped `bin/opencode`;
* v1.14+ renames the platform-specific binary to `bin/opencode.exe` for every
* OS via the postinstall script. Callers pass the binPath that matches their
* pinned version so a v1↔v2 swap can't silently install the wrong file.
*/
export async function installOpencodeCli(params: { binPath: string }): Promise<string> {
return await installFromNpmTarball({
packageName: "opencode-ai",
version: getDevDependencyVersion("opencode-ai"),
executablePath: params.binPath,
installDependencies: true,
});
}
// ── model auto-select fallback ──────────────────────────────────────────────────
//
// steps 12 of model resolution (PULLFROG_MODEL env, slug resolution) happen
// in resolveModel() in utils/agent.ts before the agent runs. this is step 3:
// auto-select via `opencode models`.
const AUTO_SELECT_WARNING =
"select a model explicitly in the Pullfrog console (https://pullfrog.com/console) to avoid this.";
function getOpenCodeModels(cliPath: string): string[] {
try {
const output = execFileSync(cliPath, ["models"], {
encoding: "utf-8",
timeout: 30_000,
env: process.env,
});
return output
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
} catch (error) {
log.debug(
`» failed to run \`opencode models\`: ${error instanceof Error ? error.message : String(error)}`
);
return [];
}
}
export function autoSelectModel(cliPath: string): string | undefined {
const availableModels = getOpenCodeModels(cliPath);
const availableSet = new Set(availableModels);
if (availableSet.size > 0) {
log.debug(`» opencode models (${availableSet.size}): ${availableModels.join(", ")}`);
// skip hidden aliases (internal subagent-tier targets like
// opencode/gpt-5.4) — they should never surface as a user-facing
// orchestrator pick. mirrors the selectable-list filter in
// components/ModelSelector.tsx and action/commands/init.ts.
const match =
modelAliases.find((a) => !a.hidden && a.preferred && availableSet.has(a.resolve)) ??
modelAliases.find((a) => !a.hidden && availableSet.has(a.resolve));
if (match) {
log.info(
`» model: ${match.resolve} (auto-selected${match.preferred ? " — preferred" : ""} curated match)`
);
log.warning(`» model auto-selected. ${AUTO_SELECT_WARNING}`);
return match.resolve;
}
log.info(
`» opencode has ${availableSet.size} models but none match curated aliases — letting OpenCode auto-select`
);
}
log.warning(`» no model resolved. letting OpenCode auto-select. ${AUTO_SELECT_WARNING}`);
return undefined;
}
File diff suppressed because it is too large Load Diff
+15 -1
View File
@@ -6,6 +6,7 @@ function makeToolState(overrides: Partial<ToolState> = {}): ToolState {
return {
progressComment: undefined,
hadProgressComment: true,
prepushFailureCount: 0,
backgroundProcesses: new Map(),
usageEntries: [],
...overrides,
@@ -29,9 +30,22 @@ describe("getUnsubmittedReview", () => {
).toBeNull();
});
it("returns null when report_progress wrote a final summary", () => {
it("fires for Review even when report_progress wrote a final summary", () => {
// Review's only valid exit is `create_pull_request_review`. a summary
// comment is not a substitute, and accepting it here previously let
// subagent-flipped `finalSummaryWritten` silence the gate.
expect(
getUnsubmittedReview(makeToolState({ selectedMode: "Review", finalSummaryWritten: true }))
).toBe("Review");
});
it("returns null for IncrementalReview when report_progress wrote a final summary", () => {
// IncrementalReview treats `report_progress` as a legitimate
// "no review warranted" exit, matching the post-failure error message.
expect(
getUnsubmittedReview(
makeToolState({ selectedMode: "IncrementalReview", finalSummaryWritten: true })
)
).toBeNull();
});
+79 -13
View File
@@ -30,13 +30,26 @@ import {
*
* the gate is anchored to `hadProgressComment` so silent runs (non-issue
* events, dispatcher skipped seeding) don't fire a nudge there's no UI for.
*
* `Review` and `IncrementalReview` have different valid exits:
* - Review: only `create_pull_request_review` counts. `report_progress` is
* not a substitute — a Review run that exits with just a summary comment
* has produced nothing reviewable on the PR. matches the hard-fail
* message at `expected = "create_pull_request_review"` below.
* - IncrementalReview: `report_progress` is a legitimate "no review
* warranted" exit, so either toolState flag short-circuits.
* splitting per mode also closes the bypass where a subagent (e.g. a
* `task`-dispatched `reviewfrog` lens) calls `report_progress` and silences
* the gate even though the orchestrator never submitted a review.
*/
export function getUnsubmittedReview(toolState: ToolState): "Review" | "IncrementalReview" | null {
const mode = toolState.selectedMode;
if (mode !== "Review" && mode !== "IncrementalReview") return null;
if (toolState.review || toolState.finalSummaryWritten) return null;
if (!toolState.hadProgressComment) return null;
return mode;
if (mode === "Review") return toolState.review ? null : "Review";
if (mode === "IncrementalReview") {
return toolState.review || toolState.finalSummaryWritten ? null : "IncrementalReview";
}
return null;
}
/**
@@ -145,7 +158,7 @@ export function buildUnsubmittedReviewPrompt(mode: "Review" | "IncrementalReview
return [
`MISSING REVIEW OUTPUT — you selected Review mode but stopped without calling \`create_pull_request_review\`. the user has no visible signal that this run produced anything; the progress comment will be deleted on exit and no review will appear on the PR.`,
"",
"call `create_pull_request_review` now with your aggregated review (body + inline comments). pick the tier per the mode prompt — Review mode has no no-submit exit, so even informational `> [!NOTE]` reviews and `No new issues found.` reviews must be submitted (both use `approved: true`). the first call may error once with a diff-coverage nudge — retry the same call to proceed.",
"call `create_pull_request_review` now with your aggregated review (body + inline comments). pick the tier per the mode prompt — Review mode has no no-submit exit, so even informational `> No new issues found.` reviews must be submitted (with `approved: true`). the first call may error once with a diff-coverage nudge — retry the same call to proceed.",
"",
"do NOT stop again until `create_pull_request_review` has been called successfully.",
].join("\n");
@@ -178,10 +191,16 @@ export async function collectPostRunIssues(
options: { skipSummaryStale?: boolean } = {}
): Promise<PostRunIssues> {
const issues: PostRunIssues = {};
if (ctx.stopScript) {
const failure = await executeStopHook(ctx.stopScript);
if (failure) issues.stopHook = failure;
}
// stop hook is disabled — production audit (May 2026) showed 8/9 configured
// scripts are foot-guns (duplicates of prepushScript, run on non-committing
// modes against unchanged trees) burning the retry budget on un-fixable
// gates. re-enable here + the dashboard block in `AgentSettings.tsx` once
// we've decided on the right semantics (mode-gating vs. HEAD-changed gating
// vs. deletion). see issue #714.
// if (ctx.stopScript) {
// const failure = await executeStopHook(ctx.stopScript);
// if (failure) issues.stopHook = failure;
// }
// dirty-tree gate fires only in modes that legitimately commit. Review /
// IncrementalReview / Plan complete via review submission or a Plan
// comment, not by touching files — any tree dirt is incidental (e.g. a
@@ -224,6 +243,24 @@ export function buildPostRunPrompt(issues: PostRunIssues): string {
return parts.join("\n\n---\n\n");
}
/**
* modes for which the post-run reflection turn is skipped. reflection costs a
* full resume turn (~$0.50-0.80 per run on Opus, mostly cache-write) and only
* pays for itself when the run actually produced novel, durable findings.
*
* `IncrementalReview` is the lowest-novelty mode — it's a tight delta review
* against an existing PR with the prior summary already loaded as context.
* the agent rarely discovers anything generalizable to next runs, so the
* reflection turn is dead weight. initial `Review` still touches fresh PR
* territory and benefits; `Build` / `Fix` / `AddressReviews` definitely do.
*/
const REFLECTION_SKIP_MODES: ReadonlySet<string> = new Set(["IncrementalReview"]);
export function shouldRunReflection(mode: string | undefined): boolean {
if (!mode) return true;
return !REFLECTION_SKIP_MODES.has(mode);
}
/**
* prompt for a dedicated post-run reflection turn nudging the agent to edit
* the rolling learnings file if it discovered anything worth persisting.
@@ -237,6 +274,23 @@ export function buildPostRunPrompt(issues: PostRunIssues): string {
* the file is the single source of truth — there is no separate MCP tool
* call. the server reads the file at end-of-run and persists any edits to
* `Repo.learnings`.
*
* the prompt copy is shaped by repo-wide audits of the actual content the
* agent has been writing (issue #619 in pullfrog/app). recurring failure
* modes the framing pushes back on:
* - massive multi-paragraph "bullets" that are really mini-articles
* - facts anchored to moving repo state (PR / review / commit / branch
* refs, dates, version pins, line numbers) that decay within weeks
* - sections growing into giant flat lists with no internal structure,
* forcing future runs to read kilobytes to find one fact
*
* single litmus delivered in the prompt: "would a future run on this repo
* do its work better because this bullet exists?". tool-quirk workarounds
* are explicitly allowed when the agent burned calls discovering the
* quirk this run — recording the workaround prevents next run from
* repeating the waste. tradeoff: the same quirk gets duplicated across
* repos, so when a quirk is fixed upstream in tool descriptions the
* per-repo bullets go stale and we have no batch-invalidation path.
*/
export function buildLearningsReflectionPrompt(filePath: string): string {
return [
@@ -244,11 +298,23 @@ export function buildLearningsReflectionPrompt(filePath: string): string {
"",
`the rolling learnings file is at \`${filePath}\`. read it first if you haven't already, then edit it in place using your native file tools. the server reads this file at end-of-run and persists any changes — there is no tool to call.`,
"",
`keep the file healthy:`,
`- only add bullets when the finding is high-confidence AND broadly useful. skip speculative, one-off, or "maybe" findings.`,
`- prune bullets that are clearly wrong, no longer relevant, or low-signal (rarely useful). a focused, accurate file beats a long stale one.`,
`- format: flat bullet list, one fact per line starting with \`- \`. deduplicate against existing entries — if a bullet covers the same fact, update it in place instead of adding a duplicate.`,
`- leave the file alone if you have nothing substantively new to add and the existing entries still look healthy. silence is a valid outcome — just reply "done" and stop.`,
`structure:`,
`- markdown hierarchy: \`## \` for top-level themes, \`### \` and deeper for sub-themes when a section grows. there is no fixed taxonomy — choose headings that fit THIS repo (e.g. for one repo \`## Migrations\` / \`## Local dev\` may make sense; for another, \`## API quirks\` / \`## Failure modes\`).`,
`- **no section over ~300 lines.** when a section is approaching that, split it: introduce \`### \` subsections grouping related bullets, or hoist a coherent group into a new top-level \`## \` section. granular sections mean future runs read targeted line ranges instead of slurping the whole file. this is the most important hygiene rule on long-lived repos.`,
`- if you find a flat unstructured list (legacy content from before this format), restructure it: read it, group related bullets, rewrite the file with \`## \` / \`### \` headings around them. don't preserve bad structure — fix it.`,
"",
`the only test: would a future run on this repo do its work better because this bullet exists? useful for future runs in this repo — prevent wasted tool calls, rabbit holes, and mistakes.`,
"",
`bullet hygiene:`,
`- one fact per line starting with \`- \`, ≤ 240 chars.`,
`- only add when high-confidence, broadly useful, evergreen.`,
`- prune wrong or low-signal bullets; merge overlaps; dedupe across sections.`,
"",
`don't anchor facts to repo state that will move: PR / review / commit / branch refs, dates, version pins, line numbers. state the rule directly. if it needs the anchor to be load-bearing, it isn't evergreen.`,
"",
`tool-quirk bullets are fine when you burned calls discovering the quirk and a future run would repeat them. write the workaround, not the war story.`,
"",
`if you have nothing substantively new to add AND the existing entries still look healthy and well-structured, leave the file alone — just reply "done" and stop. silence is a valid outcome.`,
].join("\n");
}
+34
View File
@@ -146,6 +146,40 @@ describe("SessionLabeler", () => {
]);
});
test("Claude path: parent_tool_use_id resolves directly without consuming FIFO", () => {
// Claude runs subagents inside the orchestrator's session — they share
// session_id — and stamps subagent messages with parent_tool_use_id.
// recording dispatch with the Agent tool_use id binds it directly so
// future events resolve regardless of session_id.
const labeler = new SessionLabeler();
expect(labeler.labelFor("shared-session", null)).toBe(ORCHESTRATOR_LABEL);
labeler.recordTaskDispatch({ description: "correctness" }, "toolu_01");
labeler.recordTaskDispatch({ description: "security" }, "toolu_02");
// subagent events come through with shared session_id but distinct
// parent_tool_use_id — direct mapping wins
expect(labeler.labelFor("shared-session", "toolu_01")).toBe("lens:correctness");
expect(labeler.labelFor("shared-session", "toolu_02")).toBe("lens:security");
// orchestrator events on the same session still resolve correctly
expect(labeler.labelFor("shared-session", null)).toBe(ORCHESTRATOR_LABEL);
// pendingLabels is unused on the Claude path — FIFO never consumed
expect(labeler.pendingDispatchCount()).toBe(2);
expect(labeler.size()).toBe(1);
});
test("Claude path: unknown parent_tool_use_id falls through to sessionID/FIFO logic", () => {
// defensive: if a subagent event arrives with a parent_tool_use_id we
// never recorded (e.g. orchestrator dispatched off-stream, or a tool we
// didn't track), the labeler shouldn't crash — it should fall through
// to the sessionID-keyed path.
const labeler = new SessionLabeler();
labeler.labelFor("shared", null);
expect(labeler.labelFor("shared", "unknown-tool-id")).toBe(ORCHESTRATOR_LABEL);
});
test("realistic four-lens parallel fan-out — interleaved tool_use stream", () => {
// simulates the event order we'd see when the orchestrator dispatches
// 4 lens subagents in a single assistant turn and they all start emitting
+48 -18
View File
@@ -67,38 +67,68 @@ export function deriveLabelFromTaskInput(input: TaskDispatchInput): string {
}
/**
* Stateful tracker mapping sessionIDs to human labels.
* Stateful tracker mapping subagent activity back to human-readable labels.
*
* Lifecycle:
* - First call to `labelFor()` returns ORCHESTRATOR_LABEL and binds that
* sessionID to it. Every subsequent event from that session gets the
* same label.
* - When the orchestrator emits a Task tool_use, the harness calls
* `recordTaskDispatch()` to push the dispatch's derived label onto a
* pending FIFO queue.
* - The next previously-unseen sessionID consumes the head of the queue.
* - If `labelFor()` is called for a new session with an empty queue
* (e.g. a subagent emitted events before the parent's tool_use was
* parsed, or the runtime spawned a session we didn't expect), the
* labeler falls back to `subagent#N` so log lines remain attributable.
* Two attribution channels are supported because the runtimes differ:
*
* - **OpenCode** spawns each subagent as its own opencode `Session` with
* a distinct `sessionID`. The harness records each Task dispatch into a
* pending FIFO queue; the next previously-unseen sessionID consumes the
* head of the queue and binds it to that label.
*
* - **Claude Code** runs subagents inside the orchestrator's session — they
* all share `session_id` — and instead stamps every subagent message with
* `parent_tool_use_id` pointing at the Agent tool_use id that spawned them.
* The harness binds each Agent tool_use id to its dispatched label up
* front, then `labelFor` looks the label up directly when an event arrives
* carrying that `parent_tool_use_id`.
*
* `labelFor(sessionID, parentToolUseId?)` accepts both: when
* `parentToolUseId` is set and known it short-circuits to the direct mapping;
* otherwise it falls through to the FIFO/sessionID path.
*/
export class SessionLabeler {
private readonly labels = new Map<string, string>();
private readonly labelsByToolUseId = new Map<string, string>();
private readonly pendingLabels: string[] = [];
private fallbackCounter = 0;
recordTaskDispatch(input: TaskDispatchInput): string {
/**
* Record a Task/Agent tool dispatch.
*
* @param input Task tool input — used to derive the lens label.
* @param toolUseId Optional Agent tool_use id. When provided, future events
* carrying `parent_tool_use_id === toolUseId` resolve
* directly to this label without consuming the FIFO queue
* (Claude path). Always also pushed to the FIFO queue so
* the OpenCode path still works when toolUseId is absent.
*/
recordTaskDispatch(input: TaskDispatchInput, toolUseId?: string | null): string {
const label = deriveLabelFromTaskInput(input);
this.pendingLabels.push(label);
if (toolUseId) this.labelsByToolUseId.set(toolUseId, label);
return label;
}
/**
* Return a label for the given sessionID. Binds on first call.
* Pass undefined/empty for events that lack a session id — the caller
* gets ORCHESTRATOR_LABEL so the line is still attributable.
* Return a label for the given event.
*
* @param sessionID Session id from the event (OpenCode: per-session;
* Claude: shared across orchestrator + subagents).
* @param parentToolUseId Claude's `parent_tool_use_id` — non-null on
* subagent messages. When set and known, takes
* priority over the FIFO/sessionID path.
*/
labelFor(sessionID: string | undefined | null): string {
labelFor(sessionID: string | undefined | null, parentToolUseId?: string | null): string {
// Claude path: subagent messages carry parent_tool_use_id pointing at
// the Agent tool_use that spawned them. resolve directly without
// touching the sessionID-keyed map (which is bound to the orchestrator
// for the shared session_id and would otherwise misattribute).
if (parentToolUseId) {
const direct = this.labelsByToolUseId.get(parentToolUseId);
if (direct) return direct;
}
if (!sessionID) return ORCHESTRATOR_LABEL;
const existing = this.labels.get(sessionID);
if (existing) return existing;
+7
View File
@@ -154,6 +154,13 @@ export interface AgentRunContext {
*/
onActivityTimeout?: (() => void) | undefined;
onToolUse?: ((event: AgentToolUseEvent) => void) | undefined;
/**
* Pullfrog API JWT scoped to this run. agents only need this when they
* have to write state back to Pullfrog mid-run (today: opencode.ts uses
* it to seed the post-hook's writeback envelope for Codex auth refresh).
* empty string when the run wasn't context-resolved (e.g. local dry-runs).
*/
apiToken: string;
}
export interface Agent {
+113
View File
@@ -0,0 +1,113 @@
import { describe, expect, it } from "vitest";
import { deriveSubagentModels } from "./subagentModels.ts";
describe("deriveSubagentModels", () => {
it("returns no override when orchestrator is undefined", () => {
expect(deriveSubagentModels(undefined)).toEqual({ reviewer: undefined });
});
it("returns no override when orchestrator slug isn't registered", () => {
expect(deriveSubagentModels("nonexistent/model")).toEqual({ reviewer: undefined });
});
describe("anthropic family — opus → sonnet", () => {
it("direct anthropic opus", () => {
expect(deriveSubagentModels("anthropic/claude-opus-4-7")).toEqual({
reviewer: "anthropic/claude-sonnet-4-6",
});
});
it("opencode-vendored opus stays on opencode prefix", () => {
expect(deriveSubagentModels("opencode/claude-opus-4-7")).toEqual({
reviewer: "opencode/claude-sonnet-4-6",
});
});
it("openrouter-anthropic-opus-via-anthropic-direct hits anthropic alias's openRouterResolve", () => {
// both the anthropic alias and the opencode alias have the same
// openRouterResolve. first-match-wins by alias declaration order
// (anthropic declared first in providers).
expect(deriveSubagentModels("openrouter/anthropic/claude-opus-4.7")).toEqual({
reviewer: "openrouter/anthropic/claude-sonnet-4.6",
});
});
it("sonnet has no further downshift", () => {
expect(deriveSubagentModels("anthropic/claude-sonnet-4-6")).toEqual({ reviewer: undefined });
expect(deriveSubagentModels("opencode/claude-sonnet-4-6")).toEqual({ reviewer: undefined });
});
it("haiku has no downshift", () => {
expect(deriveSubagentModels("anthropic/claude-haiku-4-5")).toEqual({ reviewer: undefined });
});
});
describe("openai family", () => {
it("gpt-pro → gpt (direct)", () => {
expect(deriveSubagentModels("openai/gpt-5.5-pro")).toEqual({ reviewer: "openai/gpt-5.5" });
});
it("gpt → gpt-5.4 (direct)", () => {
expect(deriveSubagentModels("openai/gpt-5.5")).toEqual({ reviewer: "openai/gpt-5.4" });
});
it("gpt → gpt-5.4 (opencode-vendored)", () => {
expect(deriveSubagentModels("opencode/gpt-5.5")).toEqual({ reviewer: "opencode/gpt-5.4" });
});
it("gpt-pro → gpt (openrouter)", () => {
expect(deriveSubagentModels("openrouter/openai/gpt-5.5-pro")).toEqual({
reviewer: "openrouter/openai/gpt-5.5",
});
});
it("gpt → gpt-5.4 (openrouter)", () => {
expect(deriveSubagentModels("openrouter/openai/gpt-5.5")).toEqual({
reviewer: "openrouter/openai/gpt-5.4",
});
});
it("gpt-5.4 itself (the hidden subagent target) has no further downshift", () => {
expect(deriveSubagentModels("openai/gpt-5.4")).toEqual({ reviewer: undefined });
});
it("gpt-mini has no downshift", () => {
expect(deriveSubagentModels("openai/gpt-5.4-mini")).toEqual({ reviewer: undefined });
});
});
describe("google (gemini) — inherit (Pro for both orchestrator and lenses)", () => {
// pro → flash was a meaningful capability cliff (Flash missed catastrophic
// cross-file bugs the v4 e2e test surfaced); Pro is cost-effective enough
// to keep on for lenses too. Google has no in-between tier.
it("direct google pro inherits", () => {
expect(deriveSubagentModels("google/gemini-3.1-pro-preview")).toEqual({
reviewer: undefined,
});
});
it("opencode-vendored gemini-pro inherits", () => {
expect(deriveSubagentModels("opencode/gemini-3.1-pro")).toEqual({
reviewer: undefined,
});
});
it("openrouter gemini-pro inherits", () => {
expect(deriveSubagentModels("openrouter/google/gemini-3.1-pro-preview")).toEqual({
reviewer: undefined,
});
});
it("flash has no downshift", () => {
expect(deriveSubagentModels("google/gemini-3-flash-preview")).toEqual({
reviewer: undefined,
});
});
});
describe("providers / models without a subagentModel — inherit", () => {
it("xai grok (already cheap flagship)", () => {
expect(deriveSubagentModels("xai/grok-4.3")).toEqual({ reviewer: undefined });
});
it("deepseek", () => {
expect(deriveSubagentModels("deepseek/deepseek-v4-pro")).toEqual({ reviewer: undefined });
});
it("moonshot kimi", () => {
expect(deriveSubagentModels("moonshotai/kimi-k2.6")).toEqual({ reviewer: undefined });
});
it("opencode big-pickle", () => {
expect(deriveSubagentModels("opencode/big-pickle")).toEqual({ reviewer: undefined });
});
it("legacy fallback aliases (gpt-codex, deepseek-reasoner)", () => {
expect(deriveSubagentModels("openai/gpt-5.3-codex")).toEqual({ reviewer: undefined });
expect(deriveSubagentModels("deepseek/deepseek-reasoner")).toEqual({ reviewer: undefined });
});
});
});
+40
View File
@@ -0,0 +1,40 @@
import { modelAliases } from "../models.ts";
/**
* Derive a cheaper subagent model override from the orchestrator's resolved
* model spec.
*
* This is a pure registry lookup: every alias in `action/models.ts` declares
* its own `subagentModel` (alias key in the same provider). At runtime we
* reverse-lookup the orchestrator's resolved slug to find the alias that
* produced it, follow the `subagentModel` pointer, and return the target
* alias's resolve / openRouterResolve depending on which route the
* orchestrator was using.
*
* Returns `{ reviewer: undefined }` when the orchestrator's alias has no
* `subagentModel` (e.g. it's already at a sufficiently cheap tier, or its
* provider doesn't have a clean cheaper-but-capable sibling). See models.ts
* for the wiring + per-provider rationale.
*/
export function deriveSubagentModels(orchestratorSpec: string | undefined): {
reviewer: string | undefined;
} {
if (!orchestratorSpec) return { reviewer: undefined };
// Reverse-lookup. The same resolve string appears in only one alias
// (within its provider), so first match wins. We track which field
// matched (resolve vs openRouterResolve) so we can pick the same field
// off the subagent target — keeping the orchestrator's route consistent.
for (const source of modelAliases) {
const matchedDirect = source.resolve === orchestratorSpec;
const matchedOR = source.openRouterResolve === orchestratorSpec;
if (!matchedDirect && !matchedOR) continue;
if (!source.subagentModel) return { reviewer: undefined };
const target = modelAliases.find((a) => a.slug === source.subagentModel);
if (!target) return { reviewer: undefined };
const reviewer = matchedOR ? target.openRouterResolve : target.resolve;
return { reviewer };
}
return { reviewer: undefined };
}
+41
View File
@@ -0,0 +1,41 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
const claudeSource = readFileSync(join(__dirname, "claude.ts"), "utf-8");
const opencodeSharedSource = readFileSync(join(__dirname, "opencodeShared.ts"), "utf-8");
const opencodeV2Source = readFileSync(join(__dirname, "opencode_v2.ts"), "utf-8");
/**
* The Claude Code `--agents` JSON and OpenCode `agent` config block are the
* only places where per-subagent model overrides take effect. They're built
* by string-only helpers we don't export, so this test reads the source and
* asserts the literal model strings + agent names are wired in. A regression
* here means the next review run silently runs lenses on Opus instead of
* Sonnet.
*/
describe("subagent registration source asserts", () => {
describe("claude.ts buildAgentsJson", () => {
it("registers reviewfrog with sonnet model", () => {
expect(claudeSource).toMatch(
/\[REVIEWER_AGENT_NAME\]:\s*\{[^}]*model:\s*"claude-sonnet-4-6"/s
);
});
it("imports the reviewer name constant", () => {
expect(claudeSource).toMatch(/REVIEWER_AGENT_NAME/);
});
});
describe("opencodeShared.ts buildReviewerAgentConfig", () => {
it("registers reviewfrog with mode: subagent", () => {
expect(opencodeSharedSource).toMatch(/\[REVIEWER_AGENT_NAME\]:[^}]*mode:\s*"subagent"/s);
});
it("uses deriveSubagentModels for the reviewer model override", () => {
expect(opencodeSharedSource).toMatch(/deriveSubagentModels\(/);
expect(opencodeSharedSource).toMatch(/overrides\.reviewer/);
});
it("v2 runner passes orchestrator model to buildReviewerAgentConfig", () => {
expect(opencodeV2Source).toMatch(/buildReviewerAgentConfig\(model\)/);
});
});
});
+11
View File
@@ -1,6 +1,7 @@
import { basename } from "node:path";
import arg from "arg";
import pc from "picocolors";
import { runCli as runAuthCli } from "./commands/auth.ts";
import { runCli as runGhaCli } from "./commands/gha.ts";
import { runCli as runInitCli } from "./commands/init.ts";
@@ -13,6 +14,7 @@ function printMainUsage(stream: typeof console.log): void {
stream(`usage: ${PROG} <command>\n`);
stream("commands:");
stream(" init set up pullfrog on the current repository");
stream(" auth manage provider credentials for the current repository");
stream("");
stream("global options:");
stream(" -h, --help show help");
@@ -85,6 +87,15 @@ async function run(): Promise<void> {
return;
}
if (command === "auth") {
await runAuthCli({
args: commandArgs,
prog: PROG,
showHelp: globalParsed["--help"] === true,
});
return;
}
if (globalParsed["--help"]) {
printMainUsage(console.log);
process.exit(0);
+229
View File
@@ -0,0 +1,229 @@
// shared helpers used by `init` and `auth` subcommands. these were originally
// inlined in `init.ts`; pulled out so `auth.ts` can reuse them without
// duplicating gh-auth/pullfrog-api/secret-save logic.
import { execFileSync } from "node:child_process";
import * as p from "@clack/prompts";
import pc from "picocolors";
export const PULLFROG_API_URL = (process.env.PULLFROG_API_URL || "https://pullfrog.com").replace(
/\/+$/,
""
);
// active spinner reference so bail/cancel can stop it before exiting. shared
// across init/auth subcommands via this module's singleton scope; whichever
// command starts a spinner sets this so handleCancel/bail can clean up.
let activeSpin: ReturnType<typeof p.spinner> | null = null;
export function setActiveSpin(spin: ReturnType<typeof p.spinner> | null): void {
activeSpin = spin;
}
export function bail(msg: string): never {
if (activeSpin) {
activeSpin.stop(pc.red("failed"));
activeSpin = null;
}
p.cancel(msg);
process.exit(1);
}
export function handleCancel<T>(value: T | symbol): asserts value is T {
if (p.isCancel(value)) {
if (activeSpin) {
activeSpin.stop(pc.red("canceled."));
activeSpin = null;
}
p.cancel("canceled.");
process.exit(0);
}
}
export function getGhToken(): string {
let token: string;
try {
token = execFileSync("gh", ["auth", "token"], { encoding: "utf-8" }).trim();
} catch {
bail(
`gh cli not found or not authenticated.\n` +
` ${pc.dim("install:")} https://cli.github.com\n` +
` ${pc.dim("then:")} gh auth login`
);
}
if (!token) {
bail(
`gh cli returned an empty token. try re-authenticating:\n` +
` ${pc.dim("run:")} gh auth login`
);
}
return token;
}
export function parseGitRemote(): { owner: string; repo: string } {
let url: string;
try {
url = execFileSync("git", ["remote", "get-url", "origin"], { encoding: "utf-8" }).trim();
} catch {
bail("not a git repository or no 'origin' remote found.");
}
const match = url.match(/github\.com(?::\d+)?[:/]+([^/]+)\/(.+?)(?:\.git)?(?:\/)?$/);
if (!match) bail(`could not parse github owner/repo from remote: ${url}`);
return { owner: match[1], repo: match[2] };
}
// ── Pullfrog API ──
type SecretsApiData = {
error?: string;
appSlug?: string;
installationId?: number | null;
repositorySelection?: string | null;
isOrg?: boolean;
accessible?: boolean;
repoSecrets?: string[];
orgSecrets?: string[];
pullfrogSecrets?: string[];
repoStatus?: string | null;
repoModel?: string | null;
hasRuns?: boolean;
};
type SecretsInfo = {
isOrg: boolean;
installationId: number | null;
secretsAccessible: boolean;
repoSecrets: string[];
orgSecrets: string[];
pullfrogSecrets: string[];
model: string | null;
hasRuns: boolean;
};
type InstallationNotFound = {
appSlug: string;
installationId: number | null;
repositorySelection: "all" | "selected" | null;
isOrg: boolean;
};
type StatusResult =
| ({ installed: true } & SecretsInfo)
| ({ installed: false } & InstallationNotFound);
type ApiResult<T = Record<string, unknown>> = {
ok: boolean;
status: number;
data: T;
};
async function pullfrogApi<T = Record<string, unknown>>(ctx: {
path: string;
token: string;
method?: string;
body?: Record<string, unknown>;
}): Promise<ApiResult<T>> {
const headers: Record<string, string> = { authorization: `Bearer ${ctx.token}` };
if (ctx.body) headers["content-type"] = "application/json";
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 30_000);
try {
const response = await fetch(`${PULLFROG_API_URL}${ctx.path}`, {
method: ctx.method || "GET",
headers,
body: ctx.body ? JSON.stringify(ctx.body) : null,
signal: controller.signal,
});
const data = (await response.json().catch(() => ({}))) as T;
return { ok: response.ok, status: response.status, data };
} finally {
clearTimeout(timeout);
}
}
export async function fetchStatus(ctx: {
token: string;
owner: string;
repo: string;
}): Promise<StatusResult> {
const result = await pullfrogApi<SecretsApiData>({
path: `/api/cli/secrets?owner=${encodeURIComponent(ctx.owner)}&repo=${encodeURIComponent(ctx.repo)}`,
token: ctx.token,
});
if (!result.ok) {
const errorMsg = result.data.error || "";
if (result.status === 401) bail("invalid or expired github token.");
if (result.status === 404) {
const sel = result.data.repositorySelection;
if (!result.data.appSlug) bail("server did not return appSlug");
return {
installed: false,
appSlug: result.data.appSlug,
installationId:
typeof result.data.installationId === "number" ? result.data.installationId : null,
repositorySelection: sel === "all" || sel === "selected" ? sel : null,
isOrg: result.data.isOrg === true,
};
}
bail(errorMsg || `secrets check failed (${result.status})`);
}
return {
installed: true,
isOrg: result.data.isOrg === true,
installationId:
typeof result.data.installationId === "number" ? result.data.installationId : null,
secretsAccessible: result.data.accessible !== false,
repoSecrets: result.data.repoSecrets || [],
orgSecrets: result.data.orgSecrets || [],
pullfrogSecrets: result.data.pullfrogSecrets || [],
model: result.data.repoModel ?? null,
hasRuns: result.data.hasRuns === true,
};
}
// ── secret save ──
export type SecretScope = "account" | "repo";
type PullfrogSecretResult = { saved: boolean; error: string };
export async function setPullfrogSecret(ctx: {
token: string;
owner: string;
repo: string;
name: string;
value: string;
scope: SecretScope;
}): Promise<PullfrogSecretResult> {
const result = await pullfrogApi<{ success?: boolean; error?: string }>({
path: "/api/cli/secrets",
token: ctx.token,
method: "POST",
body: {
owner: ctx.owner,
repo: ctx.repo,
name: ctx.name,
value: ctx.value,
scope: ctx.scope,
},
});
if (result.ok && result.data.success === true) {
return { saved: true, error: "" };
}
return { saved: false, error: result.data.error || `api returned ${result.status}` };
}
export async function promptScope(ctx: { owner: string; repo: string }): Promise<SecretScope> {
const scope = await p.select<SecretScope>({
message: "secret scope",
options: [
{ value: "account", label: `${ctx.owner} organization`, hint: "shared across repos" },
{ value: "repo", label: `${ctx.owner}/${ctx.repo} only` },
],
});
handleCancel(scope);
return scope;
}
+324
View File
@@ -0,0 +1,324 @@
// `pullfrog auth <provider>` — manage credentials for a configured repo
// without going through the full `init` flow. currently supports:
//
// pullfrog auth codex mint a Codex subscription credential and save it
// as the `CODEX_AUTH_JSON` Pullfrog secret
//
// the `codex` subcommand runs `codex login --device-auth` against an
// isolated `CODEX_HOME` (so the user's existing ~/.codex/auth.json is never
// touched), validates the resulting auth.json, and posts it to the Pullfrog
// secrets API. used both for first-time setup of a Codex subscription on a
// repo and for rotating a stale credential.
import { spawn } from "node:child_process";
import * as p from "@clack/prompts";
import arg from "arg";
import pc from "picocolors";
import { mintCodexAuth, refreshCodexAuth } from "../utils/codexAuth.ts";
import {
bail,
fetchStatus,
getGhToken,
handleCancel,
PULLFROG_API_URL,
parseGitRemote,
promptScope,
setActiveSpin,
setPullfrogSecret,
} from "./_shared.ts";
const CODEX_AUTH_SECRET = "CODEX_AUTH_JSON";
/** strip CSI ANSI escapes (color, cursor) from a string so callers can re-style
* the visible text without inheriting the source's formatting. covers what
* Codex emits during device auth (mostly `\x1b[<digits>m` color codes).
*/
function stripAnsi(s: string): string {
// biome-ignore lint/suspicious/noControlCharactersInRegex: ANSI escapes are control chars by design
return s.replace(/\x1b\[[0-9;]*[a-zA-Z]/g, "");
}
/** matches the Codex device-auth verification URL printed by `codex login
* --device-auth`. captures the full URL (with query string) up to whitespace.
*/
const CODEX_DEVICE_URL_RE = /https:\/\/auth\.openai\.com\/codex\/device\S*/;
/** best-effort cross-platform "open URL in default browser". swallows
* spawn errors and non-zero exits — the user can always copy-paste the URL
* Codex already printed. on Linux, falls back to `wslview` when `xdg-open`
* is missing (covers WSL where xdg-open isn't installed by default).
*/
function openInBrowser(url: string): void {
const platform = process.platform;
let cmd: string;
let args: string[];
if (platform === "darwin") {
cmd = "open";
args = [url];
} else if (platform === "win32") {
// `start` is a cmd.exe builtin. the empty "" is the window title
// (required when the next argument is quoted, which happens for
// URLs with `&`).
cmd = "cmd.exe";
args = ["/c", "start", "", url];
} else {
cmd = "xdg-open";
args = [url];
}
const child = spawn(cmd, args, { stdio: "ignore", detached: true });
child.on("error", () => {
if (platform !== "linux") return;
const fallback = spawn("wslview", [url], { stdio: "ignore", detached: true });
fallback.on("error", () => {});
fallback.unref();
});
child.unref();
}
interface AuthCliParams {
args: string[];
prog: string;
showHelp?: boolean;
}
function printAuthUsage(params: { stream: typeof console.log; prog: string }): void {
params.stream(`usage: ${params.prog} auth <provider>\n`);
params.stream("manage provider credentials for the current repository.");
params.stream("");
params.stream("providers:");
params.stream(" codex mint a Codex (ChatGPT) subscription credential");
params.stream("");
params.stream("options:");
params.stream(" -h, --help show help");
}
function printCodexUsage(params: { stream: typeof console.log; prog: string }): void {
params.stream(`usage: ${params.prog} auth codex [options]\n`);
params.stream("mint a Codex subscription credential and save it as CODEX_AUTH_JSON.");
params.stream("");
params.stream("options:");
params.stream(" -h, --help show help");
}
export async function runCli(params: AuthCliParams): Promise<void> {
// route `auth --help` (no subcommand) to top-level usage. when the user
// passes `auth codex --help`, we leave the flag in the rest args so the
// subcommand's own parser handles it.
const firstArg = params.args[0];
const helpAtTopLevel =
params.showHelp ||
params.args.length === 0 ||
(params.args.length === 1 && (firstArg === "--help" || firstArg === "-h"));
if (helpAtTopLevel) {
printAuthUsage({ stream: console.log, prog: params.prog });
return;
}
const subcommand = firstArg;
const rest = params.args.slice(1);
if (subcommand === "codex") {
await runCodex({ args: rest, prog: params.prog });
return;
}
console.error(`unknown auth provider: ${pc.bold(subcommand)}\n`);
printAuthUsage({ stream: console.error, prog: params.prog });
process.exit(1);
}
interface CodexCliParams {
args: string[];
prog: string;
}
function parseCodexArgs(args: string[]) {
return arg(
{
"--help": Boolean,
"-h": "--help",
},
{ argv: args }
);
}
async function runCodex(params: CodexCliParams): Promise<void> {
let parsed: ReturnType<typeof parseCodexArgs>;
try {
parsed = parseCodexArgs(params.args);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.error(`${message}\n`);
printCodexUsage({ stream: console.error, prog: params.prog });
process.exit(1);
}
if (parsed["--help"]) {
printCodexUsage({ stream: console.log, prog: params.prog });
return;
}
await runCodexAuth();
}
async function runCodexAuth(): Promise<void> {
p.intro(pc.bgGreen(pc.black(" pullfrog auth codex ")));
const spin = p.spinner();
setActiveSpin(spin);
try {
spin.start("authenticating with github");
const token = getGhToken();
spin.stop("github authenticated");
spin.start("detecting repository");
const remote = parseGitRemote();
spin.stop(`detected repo ${pc.cyan(`${remote.owner}/${remote.repo}`)}`);
spin.start("checking pullfrog app installation");
const status = await fetchStatus({ token, owner: remote.owner, repo: remote.repo });
if (!status.installed) {
spin.stop(pc.red("pullfrog app not installed on this repo"));
bail(
`install pullfrog on ${pc.bold(`${remote.owner}/${remote.repo}`)} before configuring auth.\n` +
` ${pc.dim("run:")} ${pc.cyan(`npx pullfrog init`)}`
);
}
spin.stop(`pullfrog app is installed on ${pc.cyan(`@${remote.owner}`)}`);
if (status.pullfrogSecrets.includes(CODEX_AUTH_SECRET)) {
const overwrite = await p.select({
message: `${pc.cyan(CODEX_AUTH_SECRET)} is already configured — overwrite?`,
options: [
{ value: true, label: "overwrite", hint: "rotate to a freshly minted credential" },
{ value: false, label: "cancel" },
],
});
handleCancel(overwrite);
if (!overwrite) {
p.cancel("canceled.");
return;
}
}
// user-owned repos can only ever be "account" (Pullfrog has no per-repo
// store for user accounts), so we never bother prompting. on org-owned
// repos, prompt interactively — matches `init`'s behavior.
const scope = status.isOrg
? await promptScope({ owner: remote.owner, repo: remote.repo })
: "account";
p.log.info(
[
`signing in via Codex device authorization. open the URL Codex prints`,
`below, enter the one-time code, and approve in your browser.`,
``,
`${pc.dim("note:")} if your ChatGPT account doesn't have device-code auth enabled,`,
`Codex will exit early. enable it at ${pc.cyan(`https://chatgpt.com/#settings/Security`)}`,
`then re-run ${pc.cyan(`${process.env.PULLFROG_BIN_NAME || "pullfrog"} auth codex`)}.`,
].join("\n")
);
// tracks the most recent exit so the retry prompt can tell the user
// *why* no auth.json was written (timeout vs. early-exit).
let lastTimedOut = false;
// gate so we don't re-launch the browser if Codex prints the URL
// more than once (e.g. on a retry attempt within the same flow).
let hasOpenedDeviceUrl = false;
const auth = await mintCodexAuth({
childStdio: "pipe",
onChildLine: (line) => {
// dim Codex's own colored output (URL/code in cyan, boilerplate in
// gray) so the user reads it as sub-process noise, not Pullfrog's
// own prompts. the rail char matches @clack/prompts so the column
// reads as one continuous flow.
const stripped = stripAnsi(line);
process.stdout.write(`${pc.gray(p.S_BAR)} ${pc.dim(stripped)}\n`);
if (hasOpenedDeviceUrl) return;
const match = stripped.match(CODEX_DEVICE_URL_RE);
if (!match) return;
hasOpenedDeviceUrl = true;
const url = match[0];
openInBrowser(url);
process.stdout.write(
`${pc.gray(p.S_BAR)} ${pc.dim(`» opened ${url} in browser (paste manually if it didn't open)`)}\n`
);
},
onProgress: (event) => {
if (event.kind === "start") {
lastTimedOut = false;
if (event.attempt > 1) p.log.info(`retry attempt ${event.attempt}`);
// shell-prompt style header so the user sees what Pullfrog is
// about to spawn, with the rail to keep the visual column.
process.stdout.write(`${pc.gray(p.S_BAR)}\n`);
process.stdout.write(`${pc.gray(p.S_BAR)} $ codex login --device-auth\n`);
}
if (event.kind === "exit") {
if (event.timedOut) lastTimedOut = true;
// trailing blank rail so the next clack prompt isn't crammed
// against the last codex output line.
process.stdout.write(`${pc.gray(p.S_BAR)}\n`);
}
},
shouldRetry: async () => {
const message = lastTimedOut
? "device authorization timed out — retry?"
: "no auth.json was written — retry?";
const retry = await p.select({
message,
options: [
{ value: true, label: "retry", hint: "after enabling device-code auth" },
{ value: false, label: "cancel" },
],
});
handleCancel(retry);
return retry;
},
});
// eager refresh: bump the OAuth chain once before persisting so the
// saved token is one Pullfrog has used. otherwise the user's laptop's
// codex CLI could refresh first and strand our copy.
spin.start("refreshing token");
let savable: typeof auth;
try {
savable = await refreshCodexAuth(auth);
spin.stop("refreshed");
} catch (err) {
spin.stop(pc.yellow("refresh failed — saving minted token as-is"));
p.log.warn(err instanceof Error ? err.message : String(err));
savable = auth;
}
spin.start(`saving ${pc.cyan(CODEX_AUTH_SECRET)} to Pullfrog`);
const result = await setPullfrogSecret({
token,
owner: remote.owner,
repo: remote.repo,
name: CODEX_AUTH_SECRET,
value: savable.json,
scope,
});
if (!result.saved) {
spin.stop(pc.red("could not save secret"));
p.log.warn(
`${result.error}\n ${pc.dim("set it manually at:")} ${PULLFROG_API_URL}/console/${remote.owner}`
);
process.exit(1);
}
spin.stop(`saved ${pc.cyan(CODEX_AUTH_SECRET)} to Pullfrog (${scope})`);
setActiveSpin(null);
p.outro("done.");
} catch (error) {
// mirror what `bail` does: stop the spinner with a red "failed" glyph
// before clearing it, otherwise an in-flight spinner keeps animating
// above the error message we're about to print.
spin.stop(pc.red("failed"));
setActiveSpin(null);
const message = error instanceof Error ? error.message : String(error);
p.log.error(message);
process.exit(1);
}
}
+9 -2
View File
@@ -22,9 +22,16 @@ type CliProvider = {
function buildProviders(): CliProvider[] {
return Object.entries(providers)
.filter(([key]) => key !== "opencode" && key !== "openrouter")
.filter(([key]) => key !== "opencode" && key !== "openrouter" && key !== "bedrock")
.map(([key, config]: [string, ProviderConfig]) => {
const aliases = modelAliases.filter((a) => a.provider === key && !a.fallback);
// bedrock requires multi-secret setup (auth + region + model id) that
// doesn't fit the single-paste flow below — direct users to
// https://docs.pullfrog.com/bedrock instead. revisit once the init flow
// supports multi-value setup. `hidden` excludes internal-only subagent
// targets (e.g. openai/gpt-5.4) per #710.
const aliases = modelAliases.filter(
(a) => a.provider === key && !a.fallback && !a.routing && !a.hidden
);
const recommended = aliases.find((a) => a.preferred);
const sorted = [...aliases].sort((a, b) => {
if (a.preferred && !b.preferred) return -1;
+63
View File
@@ -0,0 +1,63 @@
#!/bin/bash
# entrypoint for the pullfrog GHA-like container (see Dockerfile).
#
# - remaps `testuser` to the host uid/gid so bind-mounted files keep correct
# ownership after writes inside the container
# - on linux hosts, copies host ssh keys into testuser's $HOME (darwin hosts
# forward the ssh-agent socket instead, no copy needed)
# - installs action workspace deps (volume-cached, ~1.5s warm)
# - exec's the requested command as testuser; argv is preserved (no nested
# `bash -c`, no shell quoting hazards)
set -euo pipefail
HOST_UID="${HOST_UID:-1000}"
HOST_GID="${HOST_GID:-1000}"
if [ "$HOST_UID" != "1000" ] || [ "$HOST_GID" != "1000" ]; then
groupmod -g "$HOST_GID" testuser 2>/dev/null || true
usermod -u "$HOST_UID" -g "$HOST_GID" testuser 2>/dev/null || true
# chown top-level dirs only — recursive chown would fail on `:ro` bind
# mounts (e.g. macOS known_hosts mounted directly into /tmp/home/.ssh).
chown "$HOST_UID:$HOST_GID" /tmp/home /tmp/home/.config /tmp/home/.cache 2>/dev/null || true
chown "$HOST_UID:$HOST_GID" /app /app/action /app/action/node_modules 2>/dev/null || true
fi
# linux hosts: copy host ssh keys into testuser's $HOME (we own this dir,
# safe to chown). darwin hosts forward the ssh-agent socket instead and
# bind-mount known_hosts read-only — nothing to do here.
if [ -d /tmp/.ssh-host ]; then
mkdir -p /tmp/home/.ssh
cp /tmp/.ssh-host/id_* /tmp/home/.ssh/ 2>/dev/null || true
chmod 600 /tmp/home/.ssh/id_* 2>/dev/null || true
ssh-keyscan -t ed25519,rsa github.com >> /tmp/home/.ssh/known_hosts 2>/dev/null || true
chmod 644 /tmp/home/.ssh/known_hosts 2>/dev/null || true
chown -R "$HOST_UID:$HOST_GID" /tmp/home/.ssh 2>/dev/null || true
# set GIT_SSH_COMMAND if any private key got copied. don't pin a
# specific key with -i — let ssh pick whatever's in /tmp/home/.ssh
# (covers id_rsa, id_ed25519, id_ecdsa, etc.).
if ls /tmp/home/.ssh/id_* 2>/dev/null | grep -qv '\.pub$'; then
export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/tmp/home/.ssh/known_hosts -o StrictHostKeyChecking=no"
fi
fi
# warm the volume-cached node_modules. frozen-lockfile + ignore-scripts keeps
# this idempotent and fast (~1.5s when nothing changed).
#
# the lockfile lives IN the shared node_modules volume so concurrent
# `pnpm docker` invocations (e.g. `pnpm play:docker` in one terminal and
# `pnpm runtest:docker` in another) serialize their install instead of racing.
# `flock -w 120` waits up to 2min before giving up — well under any
# real-world install time but short enough to surface true deadlocks.
mkdir -p /app/action/node_modules
flock -w 120 /app/action/node_modules/.gha-install.lock \
sudo -u testuser -E env HOME=/tmp/home \
corepack pnpm install --frozen-lockfile --ignore-scripts >/dev/null
# `--shell` drops into an interactive bash for debugging the container.
if [ "${1:-}" = "--shell" ]; then
exec sudo -u testuser -E env HOME=/tmp/home bash
fi
# exec the command as testuser, preserving env. argv passes through unchanged
# — no `bash -c` nesting, no quoting required by callers.
exec sudo -u testuser -E env HOME=/tmp/home "$@"
+532
View File
@@ -0,0 +1,532 @@
// run any node script inside the pullfrog local docker container that
// mocks the GHA `ubuntu-24.04` runner environment. NOT a real GitHub
// Actions runner — for the real thing, see `.github/workflows/*.yml`
// and `action/commands/gha.ts` (the action's GHA entry point).
//
// usage:
// pnpm docker <script> [args…] # run script in container
// pnpm docker --shell # interactive bash (requires TTY)
// pnpm docker --build [--no-cache] # force-rebuild image
// pnpm docker --clean # prune orphan images/volumes
// pnpm docker --doctor # versions of every baked tool
//
// the action's two main entrypoints default to the host (fast iteration).
// `:docker` suffix wraps this script:
// pnpm play [args…] # host (this is the fast default)
// pnpm play:docker [args…] # === pnpm docker play.ts [args…]
// pnpm runtest [filters…] # host
// pnpm runtest:docker [filters…] # === pnpm docker test/run.ts [filters…]
//
// the container is a baked ubuntu:24.04 image (see Dockerfile) with the
// same toolset as GHA `ubuntu-24.04` runners. host env passes through
// verbatim — no allowlist. multi-line values (RSA keys) handled via -e
// fallback; everything else flows through `--env-file` for cleanliness.
//
// host services are reachable at `host.docker.internal:<port>` (works on
// both linux and macOS — see --add-host below).
//
// rebuild is content-hash gated on Dockerfile + docker-entrypoint.sh.
//
// design rationale + gaps: wiki/docker.md.
import { spawnSync } from "node:child_process";
import { createHash } from "node:crypto";
import { existsSync, mkdirSync, readFileSync, unlinkSync, writeFileSync } from "node:fs";
import { platform, tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { config } from "dotenv";
const __dirname = dirname(fileURLToPath(import.meta.url));
const actionDir = __dirname;
const repoRoot = join(actionDir, "..");
config({ path: join(actionDir, ".env") });
config({ path: join(repoRoot, ".env") });
// host env vars that would actively conflict with the container's own
// configuration (paths, identity, shell, and outer-CI workflow-run identifiers
// that don't apply to whatever repo the harness is acting against). everything
// else passes through.
const HOST_ONLY_VARS = new Set([
// paths / identity / shell — would clobber the container's testuser setup
"PATH",
"HOME",
"USER",
"LOGNAME",
"SHELL",
"PWD",
"OLDPWD",
"TMPDIR",
"TMP",
"TEMP",
"DOCKER_HOST",
"DOCKER_CONFIG",
"_",
"SHLVL",
"PS1",
"PS2",
"TERM_PROGRAM",
"TERM_PROGRAM_VERSION",
"TERM_SESSION_ID",
"__CF_USER_TEXT_ENCODING",
"XPC_SERVICE_NAME",
"XPC_FLAGS",
"Apple_PubSub_Socket_Render",
"COMMAND_MODE",
"COLORTERM",
"ITERM_PROFILE",
"ITERM_SESSION_ID",
// outer-CI workflow-run identifiers — when the test suite runs inside
// pullfrog/app's CI, these refer to pullfrog/app's run, NOT the test repo
// the harness is acting against (e.g. pullfrog/test-repo). Anything inside
// the action that uses them as keys to look up state on the test repo (most
// notably `resolveRun()`'s `actions.listJobsForWorkflowRun(...)` call) will
// 404. Filtering them here means the action sees them as undefined and
// skips the lookup, instead of misdirecting it. `GITHUB_REPOSITORY` and
// `GITHUB_TOKEN` are NOT filtered — those are genuinely needed inside.
"GITHUB_RUN_ID",
"GITHUB_RUN_NUMBER",
"GITHUB_RUN_ATTEMPT",
"GITHUB_JOB",
"GITHUB_WORKFLOW",
"GITHUB_ACTION",
"GITHUB_REF",
"GITHUB_SHA",
"GITHUB_HEAD_REF",
"GITHUB_BASE_REF",
"GITHUB_TRIGGERING_ACTOR",
]);
type Args = {
forceBuild: boolean;
noCache: boolean;
shell: boolean;
clean: boolean;
doctor: boolean;
passthrough: string[];
};
/**
* parses docker-level flags up to (but not including) the first positional
* argument. anything after the first positional, or after a literal `--`,
* passes through verbatim to the inner script. this prevents
* `pnpm docker test/run.ts --build` from intercepting `--build` as a
* docker flag.
*/
function parseArgs(argv: string[]): Args {
const out: Args = {
forceBuild: false,
noCache: false,
shell: false,
clean: false,
doctor: false,
passthrough: [],
};
let i = 0;
while (i < argv.length) {
const a = argv[i];
if (a === "--") {
out.passthrough.push(...argv.slice(i + 1));
return out;
}
if (a === "--build") out.forceBuild = true;
else if (a === "--no-cache") {
out.forceBuild = true;
out.noCache = true;
} else if (a === "--shell") out.shell = true;
else if (a === "--clean") out.clean = true;
else if (a === "--doctor") out.doctor = true;
else if (a === "--help" || a === "-h") {
showHelp();
process.exit(0);
} else {
// first positional — script name and everything after passes through.
out.passthrough.push(...argv.slice(i));
return out;
}
i++;
}
return out;
}
function showHelp(): void {
process.stdout.write(`Usage: pnpm docker <script> [args…]
pnpm docker --shell
pnpm docker --build [--no-cache]
pnpm docker --clean
pnpm docker --doctor
Run a node script inside the pullfrog local docker container that mocks
the GHA ubuntu-24.04 runner toolset (gh, jq, python3, sudo, +
build-essential / wget / xz / file). Host env passes through verbatim.
The host is reachable from inside the container at host.docker.internal
(useful for scripts that hit your local dev server).
The action's two main entrypoints have host (fast) and docker variants:
pnpm play [args…] # host — the fast default
pnpm play:docker [args…] # === pnpm docker play.ts [args…]
pnpm runtest [filters…] # host
pnpm runtest:docker [filters…] # === pnpm docker test/run.ts [filters…]
Options:
--build rebuild the current image (otherwise rebuilt automatically
when Dockerfile or docker-entrypoint.sh content changes).
on its own, builds and exits.
--no-cache pair with --build to also bust docker's layer cache;
useful when an apt mirror or base image changed.
--shell drop into an interactive bash inside the container.
requires a TTY.
--clean prune orphaned pullfrog-docker:* images and node_modules
volumes whose hash doesn't match the current Dockerfile.
--doctor print version info for tools inside the container (node,
pnpm, gh, jq, git, python3, ssh, …). useful for diagnosing
"works in CI fails locally" or vice versa.
-h, --help show this message.
Pass-through:
Anything after the first positional argument (or after a literal \`--\`)
goes to the inner script verbatim. so \`pnpm docker test/run.ts --build\`
passes \`--build\` to test/run.ts, not to docker.
Examples:
pnpm docker play.ts
pnpm docker play.ts --raw '{"prompt":"hi"}'
pnpm docker test/run.ts smoke
pnpm docker --shell
pnpm docker --build # build image, then exit
pnpm docker --build --no-cache # rebuild from scratch
pnpm docker --clean # reclaim disk from old image hashes
pnpm docker --doctor # fidelity audit
`);
}
function ensureDocker(): void {
if (platform() === "win32") {
fail("pnpm docker is not supported on native windows. use wsl2.");
}
const probe = spawnSync("docker", ["info"], { stdio: "ignore" });
if (probe.status !== 0) {
fail("docker is not running. start docker desktop and retry.");
}
}
function fail(msg: string): never {
process.stderr.write(`error: ${msg}\n`);
process.exit(1);
}
type ImageRef = { tag: string; volumeName: string };
function imageRefFor(ctx: { dockerfile: string; entrypoint: string }): ImageRef {
const hash = createHash("sha256")
.update(readFileSync(ctx.dockerfile))
.update(readFileSync(ctx.entrypoint))
.digest("hex")
.slice(0, 12);
return {
tag: `pullfrog-docker:${hash}`,
// version the volume by image hash so a stale node_modules cache from
// an old image (e.g. different node major) can't poison a new image.
volumeName: `pullfrog-docker-node-modules-${hash}`,
};
}
/**
* remove pullfrog-docker:* images and pullfrog-docker-node-modules-* volumes
* whose hash doesn't match the current Dockerfile + entrypoint. each
* Dockerfile/entrypoint edit creates a fresh hash and orphans the prior
* pair; without periodic cleanup these accumulate (~600MB image + ~200MB
* node_modules each).
*/
function cleanOrphans(currentRef: ImageRef): void {
const imgList = spawnSync("docker", ["image", "ls", "--format", "{{.Repository}}:{{.Tag}}"], {
encoding: "utf8",
});
const images = (imgList.stdout ?? "")
.split("\n")
.filter((s) => s.startsWith("pullfrog-docker:") && s !== currentRef.tag);
if (images.length > 0) {
process.stderr.write(`» removing ${images.length} orphan image(s): ${images.join(", ")}\n`);
spawnSync("docker", ["image", "rm", "-f", ...images], { stdio: "inherit" });
}
const volList = spawnSync("docker", ["volume", "ls", "-q"], { encoding: "utf8" });
const volumes = (volList.stdout ?? "")
.split("\n")
.filter((s) => s.startsWith("pullfrog-docker-node-modules-") && s !== currentRef.volumeName);
if (volumes.length > 0) {
process.stderr.write(`» removing ${volumes.length} orphan volume(s): ${volumes.join(", ")}\n`);
spawnSync("docker", ["volume", "rm", ...volumes], { stdio: "inherit" });
}
if (images.length === 0 && volumes.length === 0) {
process.stderr.write("» no orphans to clean (all matching current image hash)\n");
}
}
function buildImageIfNeeded(ctx: {
ref: ImageRef;
force: boolean;
noCache: boolean;
dockerfile: string;
}): void {
if (!ctx.force) {
const inspect = spawnSync("docker", ["image", "inspect", ctx.ref.tag], { stdio: "ignore" });
if (inspect.status === 0) return;
}
process.stderr.write(
`» building ${ctx.ref.tag}${ctx.noCache ? " (--no-cache)" : ""} (one-time, ~30-60s)…\n`
);
const buildArgs = ["build", "-t", ctx.ref.tag, "-f", ctx.dockerfile];
if (ctx.noCache) buildArgs.push("--no-cache");
buildArgs.push(actionDir);
const build = spawnSync("docker", buildArgs, { stdio: "inherit" });
if (build.status !== 0) {
fail("image build failed");
}
}
/**
* print versions of every tool we expect to be available, so contributors
* can sanity-check fidelity with the GHA `ubuntu-24.04` runner when a test
* passes locally but fails in CI (or vice versa).
*/
function runDoctor(ref: ImageRef): void {
// multi-line bash script; spawnSync passes the whole thing as one argv
// entry so there's no nested-shell quoting to worry about, and `do` is
// not followed by a stray semicolon.
const script = `set +e
echo '--- container ---'
grep -E '^(NAME|VERSION)=' /etc/os-release
echo "arch=$(uname -m)"
echo
echo '--- runtimes ---'
echo "node $(node --version)"
if cd /app/action 2>/dev/null; then
echo "pnpm $(corepack pnpm --version) (corepack-resolved from packageManager)"
else
echo "pnpm $(pnpm --version) (system fallback — /app/action not mounted?)"
fi
python3 --version
echo
echo '--- tools ---'
for t in gh jq git ssh curl wget tar gzip xz unzip file make gcc g++ sudo unshare awk sed grep find xargs; do
if ! command -v "$t" >/dev/null 2>&1; then
printf ' %-10s MISSING\\n' "$t"
continue
fi
case "$t" in
ssh|unzip) v=$("$t" -V 2>&1 | head -1) ;;
*) v=$("$t" --version 2>&1 | head -1) ;;
esac
printf ' %-10s %s\\n' "$t" "$v"
done
echo
echo '--- env ---'
echo "CI=$CI HOME=$HOME TMPDIR=$TMPDIR"
echo "doctor runs as: $(whoami) (uid=$(id -u) gid=$(id -g))"
echo "tests run as: testuser (uid remapped to host uid at entrypoint)"
echo "host.docker.internal -> $(getent hosts host.docker.internal | awk '{print $1}' || echo UNRESOLVED)"
`;
const result = spawnSync(
"docker",
[
"run",
"--rm",
"-v",
`${actionDir}:/app/action:cached`,
"--add-host=host.docker.internal:host-gateway",
"--entrypoint",
"/bin/bash",
ref.tag,
"-c",
script,
],
{ stdio: "inherit" }
);
process.exit(result.status ?? 1);
}
function volumeExists(name: string): boolean {
return spawnSync("docker", ["volume", "inspect", name], { stdio: "ignore" }).status === 0;
}
function initVolumeOwnership(ctx: { ref: ImageRef; uid: number; gid: number }): void {
// a fresh named volume is owned by root; chown once on creation. on warm
// runs the volume already has the right ownership and `docker run … chown`
// is sub-second pure overhead — skip it.
if (volumeExists(ctx.ref.volumeName)) return;
spawnSync(
"docker",
[
"run",
"--rm",
"--entrypoint",
"chown",
"-v",
`${ctx.ref.volumeName}:/app/action/node_modules`,
ctx.ref.tag,
"-R",
`${ctx.uid}:${ctx.gid}`,
"/app/action/node_modules",
],
{ stdio: "ignore" }
);
}
type EnvParts = { envFile: string; multiLineFlags: string[] };
function buildEnvParts(env: NodeJS.ProcessEnv): EnvParts {
const dir = join(tmpdir(), "pullfrog-docker");
mkdirSync(dir, { recursive: true });
const envFile = join(dir, `env-${process.pid}-${Date.now()}.list`);
const lines: string[] = [];
const multiLineFlags: string[] = [];
for (const key of Object.keys(env)) {
if (HOST_ONLY_VARS.has(key)) continue;
const value = env[key];
if (value === undefined) continue;
// docker --env-file is line-oriented and does not support multi-line
// values. fall back to -e for those (RSA keys, multi-line PEMs, etc.).
if (value.includes("\n") || value.includes("\r")) {
multiLineFlags.push("-e", `${key}=${value}`);
} else {
lines.push(`${key}=${value}`);
}
}
writeFileSync(envFile, `${lines.join("\n")}\n`, { mode: 0o600 });
return { envFile, multiLineFlags };
}
function buildSshFlags(home: string | undefined): string[] {
const flags: string[] = [];
if (!home) return flags;
if (platform() === "darwin") {
const knownHosts = join(home, ".ssh", "known_hosts");
if (existsSync(knownHosts)) {
flags.push("-v", `${knownHosts}:/tmp/home/.ssh/known_hosts:ro`);
}
flags.push(
"-v",
"/run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock",
"-e",
"SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock"
);
} else {
const sshDir = join(home, ".ssh");
if (existsSync(sshDir)) {
flags.push("-v", `${sshDir}:/tmp/.ssh-host:ro`);
}
}
return flags;
}
function main(): void {
const args = parseArgs(process.argv.slice(2));
ensureDocker();
const dockerfile = join(actionDir, "Dockerfile");
const entrypoint = join(actionDir, "docker-entrypoint.sh");
const ref = imageRefFor({ dockerfile, entrypoint });
if (args.clean) {
cleanOrphans(ref);
if (!args.shell && !args.doctor && args.passthrough.length === 0 && !args.forceBuild) {
process.exit(0);
}
}
buildImageIfNeeded({ ref, force: args.forceBuild, noCache: args.noCache, dockerfile });
if (args.doctor) {
runDoctor(ref);
// runDoctor exits; unreachable.
}
// standalone `--build`: image's done, nothing to run.
if (!args.shell && args.passthrough.length === 0) {
if (!args.forceBuild) {
showHelp();
process.exit(1);
}
process.exit(0);
}
// node sets isTTY to `true` for a terminal stdin, `undefined` otherwise
// (never `false`). check truthiness, not equality.
if (args.shell && !process.stdin.isTTY) {
fail("--shell needs a TTY (stdin is not a terminal). run from an interactive shell.");
}
const uid = process.getuid?.() ?? 1000;
const gid = process.getgid?.() ?? 1000;
initVolumeOwnership({ ref, uid, gid });
const envParts = buildEnvParts(process.env);
const sshFlags = buildSshFlags(process.env.HOME);
const runArgs: string[] = [
"run",
"--rm",
// `--init` uses tini as PID 1, which forwards signals (SIGINT/SIGTERM)
// to our entrypoint and reaps zombies. Without it, bash-as-PID-1
// swallows Ctrl-C during the pre-exec warmup phase.
"--init",
args.shell ? "-it" : "-t",
"--privileged",
// make the host reachable from inside the container at a stable name
// (macOS Docker Desktop bakes this in; the flag makes Linux match,
// matters when scripts hit local dev servers like API_URL=
// http://host.docker.internal:3100).
"--add-host=host.docker.internal:host-gateway",
"-v",
`${actionDir}:/app/action:cached`,
"-v",
`${ref.volumeName}:/app/action/node_modules`,
"-w",
"/app/action",
"--env-file",
envParts.envFile,
"-e",
`HOST_UID=${uid}`,
"-e",
`HOST_GID=${gid}`,
...envParts.multiLineFlags,
...sshFlags,
ref.tag,
];
if (args.shell) {
runArgs.push("--shell");
} else {
// resolve script paths relative to actionDir (matches `pnpm -C action`
// mental model). absolute paths and bare flags pass through unchanged.
const [script, ...rest] = args.passthrough;
if (script === undefined) {
fail("internal: passthrough empty");
}
runArgs.push("node", script, ...rest);
}
let exitCode = 1;
try {
const result = spawnSync("docker", runArgs, { stdio: "inherit" });
exitCode = result.status ?? 1;
} finally {
try {
unlinkSync(envParts.envFile);
} catch {
// best-effort; tmpdir is GC'd by the OS regardless.
}
}
process.exit(exitCode);
}
const isDirectExecution = process.argv[1]
? import.meta.url === pathToFileURL(resolve(process.argv[1])).href
: false;
if (isDirectExecution) {
main();
}
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env node
//
// GitHub Actions `post:` entry point. Runs after the main step regardless of
// exit status (cancellation, timeout, unhandled error) — that's the contract
// we need for credential persistence: if OpenCode refreshed the Codex
// auth.json during the run, the refreshed token must land back in Pullfrog
// even when the main step died unexpectedly.
//
// THIS IS WHY `CODEX_AUTH_JSON` HAS TO LIVE IN PULLFROG'S OWN SECRET STORE,
// NOT IN GITHUB ACTIONS SECRETS. The refresh chain rotates on every use; this
// hook PUTs the rotated chain back to Pullfrog Postgres so the next run starts
// from a fresh token. GH Actions secrets are read-only at runtime — there is
// no API to write them back from inside a job — so a token stashed there
// silently goes stale on the first refresh and the next run fails. See
// wiki/codex-auth.md.
//
// Today's only job: detect a Codex auth refresh by diffing the on-disk
// auth.json against the original refresh token (saved to GH Actions state
// by action/agents/opencode_v2.ts — see also the legacy v1 file kept as
// reference at action/agents/opencode.ts), convert OpenCode's auth shape
// back to Codex CLI shape, and PUT it to /api/runtime/secret.
//
// Silent no-op when the main step didn't materialize Codex auth (no state
// saved). Best-effort: failures are logged but never throw — the workflow
// is already done, and a missed refresh write-back means the user re-runs
// `pullfrog auth codex` next time the chain breaks.
import { existsSync, readFileSync } from "node:fs";
import * as core from "@actions/core";
import { apiFetch } from "./utils/apiFetch.ts";
import { detectCodexRefresh } from "./utils/codexHome.ts";
async function main(): Promise<void> {
const raw = core.getState("codex_writeback");
if (!raw) {
core.info("codex post-hook: no writeback state — skipping");
return;
}
let state: { apiToken: string; authPath: string; originalRefresh: string };
try {
state = JSON.parse(raw) as typeof state;
} catch (err) {
core.warning(`codex post-hook: malformed writeback state — ${err}`);
return;
}
if (!state.apiToken || !state.authPath || !state.originalRefresh) {
core.warning("codex post-hook: incomplete writeback state — skipping");
return;
}
if (!existsSync(state.authPath)) {
core.info(`codex post-hook: ${state.authPath} not found — nothing to write back`);
return;
}
let authFileContent: string;
try {
authFileContent = readFileSync(state.authPath, "utf8");
} catch (err) {
core.warning(`codex post-hook: cannot read ${state.authPath}${err}`);
return;
}
const refreshedCodexJson = detectCodexRefresh({
authFileContent,
originalRefresh: state.originalRefresh,
});
if (!refreshedCodexJson) {
core.info("codex post-hook: refresh chain unchanged — no writeback needed");
return;
}
try {
// route through apiFetch so the Vercel preview-deployment SSO gate gets
// the `x-vercel-protection-bypass` header/query (raw fetch silently 401s
// against preview envs — production is unaffected but every preview-run
// refresh would be lost). see action/utils/apiFetch.ts.
const response = await apiFetch({
path: "/api/runtime/secret",
method: "PUT",
headers: {
authorization: `Bearer ${state.apiToken}`,
"content-type": "application/json",
},
body: JSON.stringify({ name: "CODEX_AUTH_JSON", value: refreshedCodexJson }),
});
if (!response.ok) {
const body = await response.text().catch(() => "");
core.warning(`codex post-hook: writeback returned ${response.status}: ${body}`);
return;
}
core.info("codex post-hook: refreshed CODEX_AUTH_JSON persisted to Pullfrog");
} catch (err) {
core.warning(`codex post-hook: writeback failed — ${err}`);
}
}
main().catch((err) => {
// never throw — post-hook failure must not fail the workflow
core.warning(`codex post-hook: unexpected error — ${err}`);
});
+1
View File
@@ -30,6 +30,7 @@ export function formatMcpToolRef(agentId: AgentId, toolName: string): string {
export type { ModelAlias, ModelProvider, ProviderConfig } from "./models.ts";
export {
getModelEnvVars,
getModelManagedCredentials,
getModelProvider,
getProviderDisplayName,
modelAliases,
+2
View File
@@ -17,6 +17,7 @@ export type {
} from "../external.ts";
export {
getModelEnvVars,
getModelManagedCredentials,
getModelProvider,
getProviderDisplayName,
modelAliases,
@@ -44,6 +45,7 @@ export {
isLeapingIntoActionCommentBody,
LEAPING_INTO_ACTION_PREFIX,
} from "../utils/leapingComment.ts";
export { MAX_LEARNINGS_LENGTH, truncateAtLineBoundary } from "../utils/learningsTruncate.ts";
export type {
CreateProgressCommentTarget,
ProgressComment,
+113 -677
View File
@@ -3,12 +3,11 @@
import { existsSync, readdirSync } from "node:fs";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import * as core from "@actions/core";
import { deleteProgressComment, reportProgress } from "./mcp/comment.ts";
import { reportProgress } from "./mcp/comment.ts";
import { startInstallation } from "./mcp/dependencies.ts";
import { startMcpHttpServer, type ToolContext } from "./mcp/server.ts";
import { computeModes } from "./modes.ts";
import { initToolState, type ToolState } from "./toolState.ts";
import { initToolState } from "./toolState.ts";
import {
type ActivityTimeout,
createProcessOutputActivityTimeout,
@@ -16,29 +15,33 @@ import {
DEFAULT_ACTIVITY_TIMEOUT_MS,
} from "./utils/activity.ts";
import { resolveAgent, resolveModel } from "./utils/agent.ts";
import { apiFetch } from "./utils/apiFetch.ts";
import { validateAgentApiKey } from "./utils/apiKeys.ts";
import { isLocalApiUrl } from "./utils/apiUrl.ts";
import { resolveBody } from "./utils/body.ts";
import { formatUsageSummary, log, writeSummary } from "./utils/cli.ts";
import { selectFallbackModelIfNeeded } from "./utils/byokFallback.ts";
import { log } from "./utils/cli.ts";
import { recordDiffReadFromToolUse } from "./utils/diffCoverage.ts";
import { reportErrorToComment } from "./utils/errorReport.ts";
import { onExitSignal } from "./utils/exitHandler.ts";
import { resolveGit, setGitAuthServer } from "./utils/gitAuth.ts";
import { startGitAuthServer } from "./utils/gitAuthServer.ts";
import { createOctokit, writeGitHubUsageSummaryToFile } from "./utils/github.ts";
import { resolveInstructions } from "./utils/instructions.ts";
import { readLearningsFile, seedLearningsFile } from "./utils/learnings.ts";
import { persistLearnings, seedLearningsFile } from "./utils/learnings.ts";
import { executeLifecycleHook } from "./utils/lifecycle.ts";
import { normalizeEnv } from "./utils/normalizeEnv.ts";
import { normalizeEnv, sanitizeSecret } from "./utils/normalizeEnv.ts";
import { applyOverrides } from "./utils/overrides.ts";
import { aggregateUsage, patchWorkflowRunFields } from "./utils/patchWorkflowRunFields.ts";
import { resolvePayload, resolvePromptInput } from "./utils/payload.ts";
import { isRouterKeylimitExhaustedError } from "./utils/providerErrors.ts";
import { readSummaryFile, seedSummaryFile } from "./utils/prSummary.ts";
import { postReviewCleanup } from "./utils/reviewCleanup.ts";
import { resolveOutputSchema, resolvePayload, resolvePromptInput } from "./utils/payload.ts";
import { type OidcCredentials, runProxyResolution } from "./utils/proxy.ts";
import { fetchPreviousSnapshot, persistSummary, seedSummaryFile } from "./utils/prSummary.ts";
import { handleAgentResult } from "./utils/run.ts";
import { type AccountPlan, isInfraCovered } from "./utils/runContext.ts";
import { resolveRunContextData } from "./utils/runContextData.ts";
import { renderRunError } from "./utils/runErrorRenderer.ts";
import {
finalizeSuccessRun,
persistRunArtifacts,
writeRunErrorOutputs,
} from "./utils/runLifecycle.ts";
import { logRunStartup } from "./utils/runStartupLog.ts";
import { setEnvAllowlist } from "./utils/secrets.ts";
import { createTempDirectory, setupGit } from "./utils/setup.ts";
import { killTrackedChildren } from "./utils/subprocess.ts";
@@ -57,476 +60,29 @@ export interface MainResult {
result?: string | undefined;
}
function resolveOutputSchema(): Record<string, unknown> | undefined {
const raw = core.getInput("output_schema");
if (!raw) return undefined;
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
throw new Error(`invalid output_schema: not valid JSON`);
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
throw new Error(`invalid output_schema: must be a JSON object`);
}
log.info("» structured output schema provided — output will be required");
return parsed as Record<string, unknown>;
}
function resolveTimeoutForLog(timeout: string | undefined): string {
if (!timeout) return "1h (default)";
if (timeout === TIMEOUT_DISABLED) return "none (disabled)";
return timeout;
}
function resolveModelForLog(ctx: {
payload: ResolvedPayload;
resolvedModel: string | undefined;
}): string {
const envModel = process.env.PULLFROG_MODEL?.trim();
if (envModel) return `${envModel} (override via PULLFROG_MODEL)`;
if (ctx.payload.proxyModel) return `${ctx.payload.proxyModel} (proxy)`;
if (ctx.resolvedModel && ctx.payload.model && ctx.payload.model !== ctx.resolvedModel) {
return `${ctx.resolvedModel} (resolved from ${ctx.payload.model})`;
}
if (ctx.resolvedModel) return ctx.resolvedModel;
if (ctx.payload.model) return `${ctx.payload.model} (unresolved)`;
return "auto";
}
function resolveAgentForLog(ctx: { agentName: string; resolvedModel: string | undefined }): string {
const envAgent = process.env.PULLFROG_AGENT?.trim();
if (envAgent && envAgent === ctx.agentName) {
return `${ctx.agentName} (override via PULLFROG_AGENT)`;
}
if (ctx.agentName === "claude" && ctx.resolvedModel) {
return `${ctx.agentName} (auto-selected for ${ctx.resolvedModel})`;
}
return ctx.agentName;
}
import type { ResolvedPayload } from "./utils/payload.ts";
interface OidcCredentials {
requestUrl: string;
requestToken: string;
}
/**
* Billing-layer error surfaced from `/api/proxy-token` as a 402. User-actionable
* — distinct from TransientError (503 / transient sync issue) so the job
* summary + PR comment can use affirmative "you need to do X" copy rather than
* the ambiguous "billing error" label that makes transient outages look like
* the user's fault.
*
* `code` is a server-side discriminator: `router_requires_card` (no card + no
* wallet balance on Router), or null for unclassified. `declineCode` is
* Stripe's more specific sub-reason on `card_declined` (e.g.
* `insufficient_funds`, `lost_card`). `needsReauthentication` is the 3DS case
* broken out for convenience.
*/
class BillingError extends Error {
code: string | null;
declineCode: string | null;
needsReauthentication: boolean;
constructor(
message: string,
opts: {
code?: string | null;
declineCode?: string | null;
needsReauthentication?: boolean;
} = {}
) {
super(message);
this.name = "BillingError";
this.code = opts.code ?? null;
this.declineCode = opts.declineCode ?? null;
this.needsReauthentication = opts.needsReauthentication ?? false;
}
}
/**
* Transient service failures from `/api/proxy-token` (503: partial OpenRouter
* usage sync, DB flake, in-flight payment intent). Not the user's fault — the
* summary uses "temporarily unavailable" framing, and the non-zero exit lets
* GH Actions apply whatever retry policy the workflow has configured.
*/
class TransientError extends Error {
constructor(message: string) {
super(message);
this.name = "TransientError";
}
}
/**
* Deep link into the right console section for the failing account. Anchors
* are defined in `app/console/[owner]/page.tsx` (`#billing`, `#model-access`).
* `owner` is the GitHub login of the repo's account — i.e. the org or user
* that pays for this repo's runs, which is the right scope for billing.
*/
function billingConsoleUrl(owner: string, anchor: "billing" | "model-access"): string {
return `https://pullfrog.com/console/${encodeURIComponent(owner)}#${anchor}`;
}
/**
* Render a BillingError as user-facing markdown (shared between GH job summary
* and the PR progress comment). Goals:
*
* - quiet, not alarmist — bold first line instead of an `### ❌` H3, since
* the comment already has Pullfrog branding in the footer
* - actionable — every branch ends in a single CTA deep-linked to the
* correct section of the owner's console
* - honest — say what actually went wrong (card declined vs. balance
* empty vs. 3DS required), don't lump them under "billing error"
*
* Branches:
* - `router_requires_card`: user is on Router mode with no card AND no
* wallet balance. Lead with the carrot ($20 free credit), link to
* `#model-access` where the Add Card flow lives.
* - `router_balance_exhausted`: user has a card on file but auto-reload is
* disabled and they've spent past their $5 overdraft buffer. Frame as
* "balance ran out" and surface both remediation paths (top up, or flip
* on auto-reload).
* - `router_keylimit_exhausted`: OpenRouter rejected mid-run because the
* per-run key budget was exhausted while the agent was working. The
* wallet is now negative; same remediation as `router_balance_exhausted`
* but framed for the after-the-fact case ("this run was cut short").
* - `needsReauthentication`: issuer requires 3DS on every off-session
* charge. Re-adding the card won't help — the only escape is a manual
* top-up where 3DS runs interactively in Stripe Checkout.
* - `declineCode` set: Stripe declined a real charge. Show the sub-code
* so support can act on it; tell the user we'll retry on next dispatch.
* - default: balance hit zero with no in-flight charge (auto-reload off
* or amount below threshold). Direct them to top up or enable auto-reload.
*/
function formatBillingErrorSummary(error: BillingError, owner: string): string {
if (error.code === "router_requires_card") {
return [
"**Add a card to start using Pullfrog Router.**",
"",
"Router proxies OpenRouter at raw cost — no platform markup, and your first $20 of usage is on us.",
"",
`[Add a card →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.code === "router_balance_exhausted") {
return [
"**Your Pullfrog Router balance is exhausted.**",
"",
"You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.",
"",
`[Top up balance →](${billingConsoleUrl(owner, "billing")}) · [Enable auto-reload →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.code === "router_keylimit_exhausted") {
return [
"**This run was cut short — your Pullfrog Router balance ran out mid-run.**",
"",
"OpenRouter stopped the agent because the per-run budget was exhausted. Your wallet is now negative; top up or enable auto-reload to keep runs flowing.",
"",
`[Top up balance →](${billingConsoleUrl(owner, "billing")}) · [Enable auto-reload →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.needsReauthentication) {
const code = error.declineCode ?? "authentication_required";
return [
`**Your card issuer requires 3D Secure on every charge** (\`${code}\`).`,
"",
"Pullfrog can't complete a 3DS challenge from inside a workflow. Top up your Router balance once in Stripe Checkout — subsequent runs draw from the prepaid balance without re-triggering 3DS.",
"",
`[Top up balance →](${billingConsoleUrl(owner, "billing")})`,
].join("\n");
}
if (error.declineCode) {
return [
`**Your card was declined** (\`${error.declineCode}\`).`,
"",
"Update your payment method and Pullfrog will retry on the next run.",
"",
`[Update payment method →](${billingConsoleUrl(owner, "billing")})`,
].join("\n");
}
return [
"**Your Pullfrog balance is empty.**",
"",
"Top up your balance or enable auto-reload to keep runs flowing.",
"",
`[Manage billing →](${billingConsoleUrl(owner, "billing")})`,
].join("\n");
}
/**
* Render a TransientError as user-facing markdown. Distinct framing from
* BillingError so the user doesn't read an alarm and assume their card
* failed — this branch is "our fault, retry shortly", not theirs.
*/
function formatTransientErrorSummary(error: TransientError, owner: string): string {
return [
"**Pullfrog billing is temporarily unavailable.**",
"",
error.message,
"",
`Usually transient — the next dispatch should succeed. If it persists, check [status.pullfrog.com](https://status.pullfrog.com) or [your console](${billingConsoleUrl(owner, "billing")}).`,
].join("\n");
}
async function mintProxyKey(ctx: {
oidcCredentials: OidcCredentials | null;
repo: { owner: string; name: string };
}): Promise<string | null> {
try {
const headers = await buildProxyTokenHeaders(ctx);
if (!headers) return null;
const response = await apiFetch({
path: "/api/proxy-token",
method: "POST",
headers,
});
if (response.status === 402) {
const body = (await response.json().catch(() => null)) as {
error?: string;
code?: string;
declineCode?: string;
needsReauthentication?: boolean;
} | null;
throw new BillingError(body?.error ?? "insufficient balance", {
code: body?.code ?? null,
declineCode: body?.declineCode ?? null,
needsReauthentication: body?.needsReauthentication ?? false,
});
}
// 503 = transient sync issue (partial OpenRouter failure, DB flake,
// in-flight top-up). Not the user's fault — TransientError renders a
// "temporarily unavailable" summary instead of the "billing error"
// label that BillingError uses.
if (response.status === 503) {
const body = (await response.json().catch(() => null)) as { error?: string } | null;
throw new TransientError(
body?.error ?? "billing service temporarily unavailable — retry shortly"
);
}
if (!response.ok) {
log.warning(`proxy key mint failed (${response.status})`);
return null;
}
const data = (await response.json()) as { key: string };
return data.key;
} catch (error) {
if (error instanceof BillingError) throw error;
if (error instanceof TransientError) throw error;
log.warning(`proxy key mint error: ${error instanceof Error ? error.message : String(error)}`);
return null;
} finally {
delete process.env.ACTIONS_ID_TOKEN_REQUEST_URL;
delete process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN;
}
}
/**
* choose how to authenticate the `/api/proxy-token` request:
*
* - production: mint a fresh OIDC token via `core.getIDToken` and send as
* `Authorization: Bearer …` (the server verifies it cryptographically).
* - local dev (no OIDC + `API_URL` is localhost): send `x-dev-repo:
* owner/repo` instead. the server-side route only honors this header
* when `NODE_ENV === "development"`, so prod is never reachable through
* this branch even if the action is misconfigured.
*
* returns null when neither path is available — caller treats as soft skip.
*/
async function buildProxyTokenHeaders(ctx: {
oidcCredentials: OidcCredentials | null;
repo: { owner: string; name: string };
}): Promise<Record<string, string> | null> {
if (ctx.oidcCredentials) {
process.env.ACTIONS_ID_TOKEN_REQUEST_URL = ctx.oidcCredentials.requestUrl;
process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN = ctx.oidcCredentials.requestToken;
const oidcToken = await core.getIDToken("pullfrog-api");
delete process.env.ACTIONS_ID_TOKEN_REQUEST_URL;
delete process.env.ACTIONS_ID_TOKEN_REQUEST_TOKEN;
return { Authorization: `Bearer ${oidcToken}` };
}
if (isLocalApiUrl()) {
log.info(`» proxy: dev bypass (x-dev-repo) for ${ctx.repo.owner}/${ctx.repo.name}`);
return { "x-dev-repo": `${ctx.repo.owner}/${ctx.repo.name}` };
}
return null;
}
async function resolveProxyModel(ctx: {
payload: ResolvedPayload;
oss: boolean;
plan: AccountPlan;
proxyModel?: string | undefined;
oidcCredentials: OidcCredentials | null;
repo: { owner: string; name: string };
}): Promise<void> {
// env override = BYOK escape hatch, don't proxy
if (process.env.PULLFROG_MODEL?.trim()) return;
const needsProxy = isInfraCovered({ isOss: ctx.oss, plan: ctx.plan }) && ctx.proxyModel;
if (!needsProxy) return;
// dev affordance: when talking to a localhost API, the server-side
// x-dev-repo bypass replaces OIDC verification, so a play run can
// exercise the proxy/router/oss path without GitHub Actions OIDC.
if (!ctx.oidcCredentials && !isLocalApiUrl()) {
log.warning("» proxy requested but no OIDC credentials available — skipping");
return;
}
const key = await mintProxyKey({ oidcCredentials: ctx.oidcCredentials, repo: ctx.repo });
if (!key) return;
process.env.OPENROUTER_API_KEY = key;
core.setSecret(key);
ctx.payload.proxyModel = ctx.proxyModel;
const label = ctx.oss ? "oss" : "router";
log.info(`» proxy: ${label}${ctx.proxyModel}`);
}
/**
* Fetch the most recent persisted PR summary snapshot for this PR.
* Returns null on first-time PRs, when summary is disabled, or on any error.
* Best-effort: a transient API failure should not block the run.
*/
async function fetchPreviousSnapshot(ctx: ToolContext, prNumber: number): Promise<string | null> {
if (!ctx.githubInstallationToken) return null;
try {
const response = await apiFetch({
path: `/api/repo/${ctx.repo.owner}/${ctx.repo.name}/pr/${prNumber}/summary-comment`,
method: "GET",
headers: { authorization: `Bearer ${ctx.githubInstallationToken}` },
signal: AbortSignal.timeout(10_000),
});
if (!response.ok) return null;
const data = (await response.json()) as { snapshot?: string | null };
return typeof data.snapshot === "string" && data.snapshot.length > 0 ? data.snapshot : null;
} catch {
return null;
}
}
/**
* Read the agent-edited PR summary tmpfile and persist to `WorkflowRun.summarySnapshot`.
*
* Best-effort: any failure is logged and does not affect the run's success
* status. Skips the PATCH when the file is byte-identical to its seed —
* persisting the seed verbatim would either re-write what the DB already has
* (on incremental runs) or serialize the placeholder scaffold (on first
* runs), neither of which is useful.
*/
/**
* Read the agent-edited repo-level learnings tmpfile and PATCH it to
* `Repo.learnings`.
*
* Best-effort: any failure is logged and does not affect the run's success
* status. Skips the PATCH when the file is byte-trim-identical to its seed —
* the agent didn't touch it, so writing the same content back would just
* burn a `LearningsRevision` row and an API round-trip.
*
* `model` is forwarded so `LearningsRevision.model` keeps populating; it
* powers the per-revision attribution badge in the UI history view.
*/
async function persistLearnings(ctx: ToolContext): Promise<void> {
const filePath = ctx.toolState.learningsFilePath;
if (!filePath) return;
if (ctx.toolState.learningsPersistAttempted) return;
ctx.toolState.learningsPersistAttempted = true;
const current = await readLearningsFile(filePath);
if (current === null) {
log.debug(`learnings tmpfile missing or unreadable at ${filePath} — skipping persist`);
return;
}
const seed = ctx.toolState.learningsSeed?.trim() ?? "";
if (current === seed) {
log.debug("learnings tmpfile unchanged from seed — skipping persist");
return;
}
try {
const response = await apiFetch({
path: `/api/repo/${ctx.repo.owner}/${ctx.repo.name}/learnings`,
method: "PATCH",
headers: {
authorization: `Bearer ${ctx.apiToken}`,
"content-type": "application/json",
},
body: JSON.stringify({
learnings: current,
model: ctx.toolState.model,
}),
signal: AbortSignal.timeout(10_000),
});
if (!response.ok) {
const error = await response.text().catch(() => "(no body)");
log.debug(`learnings persist failed (${response.status}): ${error}`);
return;
}
log.info("» learnings updated");
} catch (err) {
log.debug(`learnings persist failed: ${err instanceof Error ? err.message : String(err)}`);
}
}
async function persistSummary(ctx: ToolContext): Promise<void> {
const filePath = ctx.toolState.summaryFilePath;
if (!filePath) return;
// already-completed guard: the error-path call (success path persisted,
// then a late step threw) and the SIGINT/SIGTERM handler all funnel
// through here; the first one to arrive wins.
if (ctx.toolState.summaryPersistAttempted) return;
ctx.toolState.summaryPersistAttempted = true;
const snapshot = await readSummaryFile(filePath);
if (!snapshot) {
log.debug(`pr summary tmpfile missing or invalid at ${filePath} — skipping persist`);
return;
}
// soft gate: agent never touched the seeded file. saving the seed back
// is a no-op at best (incremental run — DB already has it) and a bug at
// worst (first run — serializes the placeholder italics). log a warning
// so the failure mode is visible in CI without flipping the run to
// failed.
const seed = ctx.toolState.summarySeed?.trim();
if (seed !== undefined && snapshot === seed) {
log.warning(
"» pr summary tmpfile unchanged from seed — skipping persist (agent did not edit it)"
);
return;
}
await patchWorkflowRunFields(ctx, { summarySnapshot: snapshot }).catch((err) => {
log.debug(`pr summary persist failed: ${err instanceof Error ? err.message : String(err)}`);
});
}
// fall back to the agent's final assistant message when the agent never
// called report_progress (e.g. schedule/workflow_dispatch runs that have no
// PR/issue context to comment on). lastProgressBody wins when present so we
// don't double up the progress comment body in the job summary.
async function writeJobSummary(toolState: ToolState, finalOutput?: string): Promise<void> {
const usageSummary = formatUsageSummary(toolState.usageEntries);
const body = toolState.lastProgressBody || finalOutput;
const summaryParts = [body, usageSummary].filter(Boolean);
if (summaryParts.length > 0) {
await writeSummary(summaryParts.join("\n\n"));
}
}
export async function main(): Promise<MainResult> {
// normalize env var names to uppercase (handles case-insensitive workflow files)
normalizeEnv();
// apply caller-supplied env overrides — JSON object forwarded as the
// UNSAFE_OVERRIDES env var (NOT a `with:` input). gated by `actions:write`
// on the repo and refuses integrity-critical names; see utils/overrides.ts
// for the deny-list and wiki/e2e-testing.md for usage + threat model.
// the `unsafe` prefix is intentional: GH echoes the env-block value in the
// step-header log, so the raw JSON is visible to anyone with `actions:read`.
const overridesRaw = process.env.UNSAFE_OVERRIDES ?? "";
if (overridesRaw.trim()) {
const result = applyOverrides({ raw: overridesRaw, env: process.env });
if (result.applied.length > 0) {
log.info(`» applied ${result.applied.length} env override(s): ${result.applied.join(", ")}`);
}
if (result.denied.length > 0) {
log.warning(
`» refused to override ${result.denied.length} protected env var(s): ${result.denied.join(", ")}`
);
}
}
// write usage summary on SIGINT/SIGTERM so the worker can read it after sandbox.exec
const usageSummaryPath = process.env.PULLFROG_USAGE_SUMMARY_PATH;
if (usageSummaryPath) {
@@ -554,12 +110,15 @@ export async function main(): Promise<MainResult> {
const runContext = await resolveRunContextData({ octokit: initialOctokit, token: jobToken });
timer.checkpoint("runContextData");
// inject account-level secrets into process.env (YAML secrets take precedence)
// inject account-level secrets into process.env (YAML secrets take precedence).
// sanitizeSecret trims + masks so accidental trailing whitespace doesn't leak
// through GitHub Actions' line-based log masking. whitespace-only values
// return null and skip injection so the user sees a clear missing-key error.
if (runContext.dbSecrets) {
for (const [key, value] of Object.entries(runContext.dbSecrets)) {
if (!process.env[key]) {
process.env[key] = value;
core.setSecret(value);
const sanitized = sanitizeSecret(key, value);
if (sanitized !== null) process.env[key] = sanitized;
}
}
const count = Object.keys(runContext.dbSecrets).length;
@@ -598,39 +157,18 @@ export async function main(): Promise<MainResult> {
}
// Proxy decision: mint an OpenRouter key for OSS repos or managed billing
// accounts. BillingError (402) and TransientError (503) both surface here.
// Handle explicitly so the user sees an actionable message (job summary +
// PR progress comment when one exists) — otherwise the error unwinds past
// the main try/catch (which needs toolState) and lands in runMain with only
// a generic core.setFailed.
try {
await resolveProxyModel({
payload,
oss: runContext.oss,
plan: runContext.plan,
proxyModel: runContext.proxyModel,
oidcCredentials,
repo: runContext.repo,
});
} catch (error) {
if (error instanceof BillingError) {
const summary = formatBillingErrorSummary(error, runContext.repo.owner);
await writeSummary(summary).catch(() => {});
// Mirror to the PR progress comment if the trigger created one
// (mention / PR event). Without this, auto-reload declines are only
// visible in the job summary — users rarely open that, so the agent
// just appears to silently stop mid-run.
await reportErrorToComment({ toolState, error: summary }).catch(() => {});
throw error;
}
if (error instanceof TransientError) {
const summary = formatTransientErrorSummary(error, runContext.repo.owner);
await writeSummary(summary).catch(() => {});
await reportErrorToComment({ toolState, error: summary }).catch(() => {});
throw error;
}
throw error;
}
// accounts. BillingError (402) and TransientError (503) get rendered inside
// `runProxyResolution` before being rethrown — handled here (not in the
// outer catch) because the outer catch needs `toolContext` (not yet built)
// for its general-purpose error path.
await runProxyResolution({
payload,
oss: runContext.oss,
proxyModel: runContext.proxyModel,
oidcCredentials,
repo: runContext.repo,
toolState,
});
// create octokit with MCP token for GitHub API calls
const octokit = createOctokit(tokenRef.mcpToken);
@@ -666,18 +204,46 @@ export async function main(): Promise<MainResult> {
await using gitAuthServer = await startGitAuthServer(tmpdir);
setGitAuthServer(gitAuthServer);
const resolvedModel = payload.proxyModel ? undefined : resolveModel({ slug: payload.model });
const initialResolvedModel = payload.proxyModel
? undefined
: resolveModel({ slug: payload.model });
// BYOK fallback: if the configured model needs a key the runner doesn't
// have, swap to a free OpenCode model so the run can still produce
// value. Without this, the agent launches with no key, the LLM provider
// 401s, and the run dies in seconds with a synthetic "Invalid API key"
// — exactly the silent-churn pattern that took out 15 accounts before
// this landed. Router/proxy runs are skipped (Pullfrog mints the key);
// see `selectFallbackModelIfNeeded` for the full skip set.
const fallback = selectFallbackModelIfNeeded({
resolvedModel: initialResolvedModel,
proxyModel: payload.proxyModel,
});
// when fallback engages we bypass `resolveModel` for the new slug —
// `PULLFROG_MODEL` has higher priority than the slug arg inside that
// helper and would otherwise re-override back to the unkeyed model.
// the free fallback slug is already a CLI-ready specifier, so using
// it verbatim is correct and avoids the override.
const effectiveSlug = fallback.fallback ? fallback.to : payload.model;
const resolvedModel = fallback.fallback ? fallback.to : initialResolvedModel;
if (fallback.fallback) {
log.warning(
`» fell back from ${fallback.from} to ${fallback.to} — no BYOK key present in runner env. add a provider key in repo secrets to use ${fallback.from} instead.`
);
toolState.modelFallback = { from: fallback.from };
}
const agent = resolveAgent({ model: resolvedModel });
// surface the effective model in comment/review footers. payload.model is
// just the stored slug (often undefined for router/oss runs that derive
// the target from proxyModel). matching priority with resolveModelForLog
// so the "Using `…`" badge reflects what actually ran.
toolState.model = payload.proxyModel ?? resolvedModel ?? payload.model;
toolState.model = payload.proxyModel ?? resolvedModel ?? effectiveSlug;
validateAgentApiKey({
agent,
model: payload.proxyModel ?? resolvedModel ?? payload.model,
model: payload.proxyModel ?? resolvedModel ?? effectiveSlug,
owner: runContext.repo.owner,
name: runContext.repo.name,
});
@@ -751,23 +317,19 @@ export async function main(): Promise<MainResult> {
// and an unwrapped filesystem failure (ENOSPC, EACCES, hostile sandbox)
// would unwind into the outer main() catch and flip an otherwise-
// successful run to "❌ Pullfrog failed" before the agent even starts.
// matches `persistLearnings`'s own best-effort contract — learnings
// are a peripheral artifact, not a load-bearing capability. on failure
// toolState.learningsFilePath stays unset, and downstream consumers
// (`persistLearnings`, agent harnesses, `resolveInstructions`) all
// treat undefined as "no learnings affordance this run".
// on failure toolState.learningsFilePath stays unset, and downstream
// consumers (`persistLearnings`, agent harnesses, `resolveInstructions`)
// all treat undefined as "no learnings affordance this run".
try {
const learningsPath = await seedLearningsFile({
tmpdir,
current: runContext.repoSettings.learnings,
});
toolState.learningsFilePath = learningsPath;
try {
toolState.learningsSeed = await readFile(learningsPath, "utf8");
} catch {
// intentionally empty — learningsSeed stays undefined, persistLearnings
// will treat seed as "" and persist any non-empty content
}
// file on disk is the verbatim DB body, so the seed used for
// change-detection is just `current ?? ""` (trimmed). persistLearnings
// byte-compares against the trimmed read-back to skip no-op PATCHes.
toolState.learningsSeed = (runContext.repoSettings.learnings ?? "").trim();
log.info(
`» learnings seeded at ${learningsPath} (existing=${runContext.repoSettings.learnings ? "yes" : "no"})`
);
@@ -791,9 +353,6 @@ export async function main(): Promise<MainResult> {
// the post-run retry loop to detect the agent forgetting to edit
// the file (byte-identical to seed → nudge once via resume turn)
// and by persistSummary to skip the DB write when nothing changed.
// we just wrote the file, so the read shouldn't fail; the catch
// leaves summarySeed unset (its default), in which case the unchanged
// checks downstream are simply skipped.
try {
toolState.summarySeed = await readFile(filePath, "utf8");
} catch {
@@ -815,14 +374,7 @@ export async function main(): Promise<MainResult> {
startInstallation(toolContext);
const modelForLog = resolveModelForLog({ payload, resolvedModel });
const agentForLog = resolveAgentForLog({ agentName: agent.name, resolvedModel });
const timeoutForLog = resolveTimeoutForLog(payload.timeout);
log.info(`» model: ${modelForLog}`);
log.info(`» agent: ${agentForLog}`);
log.info(`» push: ${payload.push}`);
log.info(`» shell: ${payload.shell}`);
log.info(`» timeout: ${timeoutForLog}`);
logRunStartup({ payload, resolvedModel, agentName: agent.name });
const instructions = resolveInstructions({
payload,
@@ -831,6 +383,7 @@ export async function main(): Promise<MainResult> {
agentId,
outputSchema,
learningsFilePath: toolState.learningsFilePath ?? null,
learningsHeadings: runContext.repoSettings.learningsHeadings,
});
const logParts = [
instructions.eventInstructions
@@ -926,6 +479,7 @@ export async function main(): Promise<MainResult> {
todoTracker,
stopScript: runContext.repoSettings.stopScript,
toolState,
apiToken: runContext.apiToken,
onActivityTimeout: onInnerActivityTimeout,
onToolUse: (event) => {
const wasTracked = recordDiffReadFromToolUse({
@@ -993,96 +547,13 @@ export async function main(): Promise<MainResult> {
);
}
// post-agent review cleanup: reportReviewNodeId → follow-up re-review dispatch.
// runs after the agent exits so ordering is architecturally guaranteed (no LLM involvement).
// best-effort: cleanup failures must not turn a successful agent run into a failure.
//
// note: progress-comment deletion on review submission is owned by
// create_pull_request_review (action/mcp/review.ts) and runs atomically
// with the submission, so it survives any path out of main (success,
// timeout, crash) without relying on cleanup ordering here.
if (toolContext) {
await postReviewCleanup(toolContext).catch((error) => {
log.debug(`post-review cleanup failed: ${error}`);
});
}
// read the agent-edited summary tmpfile and persist to the DB. happens
// after the agent exits so the file is in its final state.
if (toolContext) {
await persistSummary(toolContext);
}
// same for the rolling repo-level learnings tmpfile. always seeded, so
// always read back; persistLearnings short-circuits when the file is
// unchanged from its seed.
if (toolContext) {
await persistLearnings(toolContext);
}
// when the agent harness returns success=false (e.g. unsubmitted-review
// gate exhausted retries, stop-hook persistently failing), surface the
// error in the progress comment so the user sees it instead of a
// deleted-comment void. mirrors the catch-block error reporting for
// thrown errors. runs before the stranded-comment cleanup below so
// the comment is still around to update; reportErrorToComment sets
// wasUpdated=true and the !result.success guard skips deletion.
if (!result.success && toolContext && toolState.progressComment) {
await reportErrorToComment({
toolState,
error: result.error || "agent run failed",
}).catch((error) => {
log.debug(`failure error report failed: ${error}`);
});
}
// clean up stranded progress comments. the comment is stale unless
// report_progress wrote a final summary to it — three sub-cases all reduce
// to !finalSummaryWritten:
// 1. nothing wrote to the comment ("Leaping into action" orphan)
// 2. tracker published a checklist but the agent never finalized it
// 3. the agent produced a substantive artifact via another MCP write tool
// (create_issue_comment, update_pull_request_body, reply_to_review_comment)
// and skipped report_progress — wasUpdated is true, but the progress
// comment itself was never touched.
// create_pull_request_review owns its own deletion (see action/mcp/review.ts),
// so progressComment is already null by the time we get here for that path.
// uses finalSummaryWritten (not todoTracker.enabled or wasUpdated) so cleanup
// survives API failures in report_progress where cancel() ran but the write
// didn't succeed, and isn't fooled by writes to *other* artifacts. skipped
// entirely on result.success===false: the error message just written above
// is the user's only signal that the run happened — deleting it would
// restore the same empty-void UX this commit fixes.
if (
toolContext &&
result.success &&
toolState.progressComment &&
!toolState.finalSummaryWritten
) {
await deleteProgressComment(toolContext).catch((error) => {
log.debug(`stranded progress comment cleanup failed: ${error}`);
});
}
// best-effort: failures writing the actions step summary must not throw
// past this point. on the result.success===false branch above we already
// wrote `result.error` to the progress comment, and a throw here would
// jump to the outer catch which calls reportErrorToComment again with
// the (less actionable) writeJobSummary error — silently overwriting the
// gate's failure message in the progress comment. the step-summary write
// is informational; let it fail silently rather than corrupt user-facing
// output.
try {
await writeJobSummary(toolState, result.output);
} catch (error) {
log.debug(`job summary write failed: ${error}`);
}
// emit structured output marker for test validation
if (toolState.output) {
log.info(`::pullfrog-output::${Buffer.from(toolState.output).toString("base64")}`);
core.setOutput("result", toolState.output);
}
// success-path cleanup: postReview → persistSummary → persistLearnings →
// failure-error-report → stranded-comment cleanup → job summary → output
// marker. each step is best-effort; see `finalizeSuccessRun` for ordering
// rationale (notably: progress-comment deletion lives in
// create_pull_request_review for review-mode runs, so deletion here
// covers the non-review success paths).
await finalizeSuccessRun({ toolContext, toolState, result, repo: runContext.repo });
return await handleAgentResult({
result,
@@ -1096,54 +567,19 @@ export async function main(): Promise<MainResult> {
killTrackedChildren();
log.error(errorMessage);
// Reclassify OpenRouter "key budget exhausted" mid-run errors as
// BillingError. The agent runtime surfaces this as a generic APIError,
// but it's a Pullfrog billing concern — the user's Router wallet ran
// out partway through the run. Route through the same formatBillingErrorSummary
// path as proxy-token 402s so the user gets actionable copy + a top-up
// CTA on both the job summary and the PR progress comment, instead of
// a generic "❌ Pullfrog failed" stack-trace dump.
const billingError = isRouterKeylimitExhaustedError(errorMessage)
? new BillingError(errorMessage, { code: "router_keylimit_exhausted" })
: null;
// classify (BillingError reclassification + hang detection + API-key auth
// detection) and render to {summary, comment} markdown bodies.
const rendered = renderRunError({
errorMessage,
repo: runContext.repo,
agentDiagnostic: toolState.agentDiagnostic,
});
await writeRunErrorOutputs({ rendered, toolState });
// best-effort summary — write the error so it's visible in the Actions summary tab
try {
const errorSummary = billingError
? formatBillingErrorSummary(billingError, runContext.repo.owner)
: `### ❌ Pullfrog failed\n\n\`\`\`\n${errorMessage}\n\`\`\``;
const usageSummary = formatUsageSummary(toolState.usageEntries);
const parts = [errorSummary, toolState.lastProgressBody, usageSummary].filter(Boolean);
await writeSummary(parts.join("\n\n"));
} catch {}
try {
const commentBody = billingError
? formatBillingErrorSummary(billingError, runContext.repo.owner)
: errorMessage;
await reportErrorToComment({ toolState, error: commentBody });
} catch {
// error reporting failed, but don't let it mask the original error
}
// best-effort review cleanup (e.g., agent timed out after submitting a review)
// best-effort cleanup: review dispatch, summary persist, learnings persist.
// a partial edit before the crash is still worth keeping.
if (toolContext) {
await postReviewCleanup(toolContext).catch((error) => {
log.debug(`post-review cleanup failed: ${error}`);
});
}
// best-effort summary persist on the error path: if the agent successfully
// edited the summary file before timing out / crashing, those edits are
// worth keeping for the next incremental run.
if (toolContext) {
await persistSummary(toolContext);
}
// same rationale for learnings: a partial edit before a crash is still
// worth keeping. persistLearnings is idempotent via learningsPersistAttempted.
if (toolContext) {
await persistLearnings(toolContext);
await persistRunArtifacts(toolContext);
}
return {
+236 -175
View File
@@ -5,7 +5,7 @@ import type { Octokit, RestEndpointMethodTypes } from "@octokit/rest";
import { type } from "arktype";
import { log } from "../utils/cli.ts";
import { countLines, createDiffCoverageState } from "../utils/diffCoverage.ts";
import { $git } from "../utils/gitAuth.ts";
import { $git, $gitFetchWithDeepen } from "../utils/gitAuth.ts";
import { executeLifecycleHook } from "../utils/lifecycle.ts";
import { computeIncrementalDiff } from "../utils/rangeDiff.ts";
import { retry } from "../utils/retry.ts";
@@ -259,10 +259,10 @@ async function ensureBeforeShaReachable(params: EnsureBeforeShaParams): Promise<
sha: params.sha,
ref: tempBranch,
});
await $git(
"fetch",
await $gitFetchWithDeepen(
["--no-tags", ...(params.isShallow ? ["--depth=1"] : []), "origin", tempBranch],
{ token: params.gitToken }
{ token: params.gitToken },
`before_sha temp branch ${tempBranch}`
);
log.debug(`» fetched before_sha via temp branch ${tempBranch}`);
return true;
@@ -410,9 +410,17 @@ export async function checkoutPrBranch(
toolState.checkoutSha = $("git", ["rev-parse", "HEAD"], { log: false }).trim();
const alreadyOnBranch = toolState.checkoutSha === pr.headSha;
// fetch base branch so origin/<base> exists for diff operations
// fetch base branch so origin/<base> exists for diff operations.
// wrap with deepen-retry: on shallow clones (the actions/checkout default
// is depth=1), repos with deep PR ancestry can't reach the baseRef tip in
// a single round trip, surfacing as `Could not read <sha>` / `remote did
// not send all necessary objects` (issue #656).
log.debug(`» fetching base branch (${pr.baseRef})...`);
await $git("fetch", ["--no-tags", "origin", pr.baseRef], { token: gitToken });
await $gitFetchWithDeepen(
["--no-tags", "origin", pr.baseRef],
{ token: gitToken },
`base branch ${pr.baseRef}`
);
// alreadyOnBranch only matches for repeated checkout_pr calls for the same PR in one session
// (without the tip moving), or if an external setup already checked out the PR head.
@@ -426,14 +434,21 @@ export async function checkoutPrBranch(
// -B creates or resets the branch to match origin/baseBranch
$("git", ["checkout", "-B", pr.baseRef, `origin/${pr.baseRef}`], { log: false });
// fetch PR branch using pull/{n}/head refspec (works for both fork and same-repo PRs)
// fetch PR branch using pull/{n}/head refspec (works for both fork and same-repo PRs).
// two transient classes wrap this fetch:
// - shallow-unreachable (`Could not read <sha>` etc.) — handled by the
// inner `$gitFetchWithDeepen` deepen-retry (one shot, see issue #656)
// - pull/N/head webhook race (`couldn't find remote ref pull/N/head`) —
// handled by the outer retry below (see issue #591)
log.debug(`» fetching PR #${pr.number} (${localBranch})...`);
await retry(
async () => {
try {
await $git("fetch", ["--no-tags", "origin", `+pull/${pr.number}/head:${localBranch}`], {
token: gitToken,
});
await $gitFetchWithDeepen(
["--no-tags", "origin", `+pull/${pr.number}/head:${localBranch}`],
{ token: gitToken },
`PR #${pr.number}`
);
} catch (e) {
// on the webhook race, check whether the PR still matches what we
// dispatched on. if it's been closed/merged or the head SHA moved,
@@ -588,186 +603,232 @@ export async function checkoutPrBranch(
return { hookWarning: postCheckoutHook.warning };
}
/**
* dedupes concurrent `checkout_pr` calls for the same PR. agents (notably
* Sonnet/Claude) occasionally emit duplicate parallel tool_use blocks for the
* same args in one turn; without this, both invocations race
* `checkoutPrBranch` against the same `.git/shallow.lock` and one fails with
* `File exists` (issue #642). cleared in `finally` so subsequent same-PR
* calls re-do the work normally.
*/
const inFlightCheckouts = new Map<number, Promise<CheckoutPrResult>>();
export function CheckoutPrTool(ctx: ToolContext) {
const runCheckout = async (pull_number: number): Promise<CheckoutPrResult> => {
const prResponse = await ctx.octokit.rest.pulls.get({
owner: ctx.repo.owner,
repo: ctx.repo.name,
pull_number,
});
const headRepo = prResponse.data.head.repo;
if (!headRepo) {
throw new Error(`PR #${pull_number} source repository was deleted`);
}
const pr: PrData = {
number: pull_number,
headSha: prResponse.data.head.sha,
headRef: prResponse.data.head.ref,
headRepoFullName: headRepo.full_name,
baseRef: prResponse.data.base.ref,
baseRepoFullName: prResponse.data.base.repo.full_name,
maintainerCanModify: prResponse.data.maintainer_can_modify,
};
const checkoutResult = await checkoutPrBranch(pr, {
octokit: ctx.octokit,
owner: ctx.repo.owner,
name: ctx.repo.name,
gitToken: ctx.gitToken,
toolState: ctx.toolState,
shell: ctx.payload.shell,
postCheckoutScript: ctx.postCheckoutScript,
beforeSha: ctx.toolState.beforeSha,
});
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) {
throw new Error(
"PULLFROG_TEMP_DIR not set - checkout_pr must run in pullfrog action context"
);
}
const headShort = ctx.toolState.checkoutSha!.slice(0, 7);
// compute incremental diff if we have a beforeSha to compare against
let incrementalDiffPath: string | undefined;
if (ctx.toolState.beforeSha && ctx.toolState.checkoutSha) {
const beforeShort = ctx.toolState.beforeSha.slice(0, 7);
const incremental = computeIncrementalDiff({
baseBranch: pr.baseRef,
beforeSha: ctx.toolState.beforeSha,
headSha: ctx.toolState.checkoutSha,
});
if (incremental) {
incrementalDiffPath = join(
tempDir,
`pr-${pull_number}-${beforeShort}-${headShort}-incremental.diff`
);
writeFileSync(incrementalDiffPath, incremental);
log.info(
`» incremental diff computed (${incremental.length} bytes) → ${incrementalDiffPath}`
);
}
}
// fetch PR files and format with line numbers
const formatResult = await fetchAndFormatPrDiff(ctx, pull_number);
const diffPreview = formatResult.content.split("\n").slice(0, 100).join("\n");
log.debug(`formatted diff preview (first 100 lines):\n${diffPreview}`);
const diffPath = join(tempDir, `pr-${pull_number}-${headShort}.diff`);
writeFileSync(diffPath, formatResult.content);
log.debug(`wrote diff to ${diffPath} (${formatResult.content.length} bytes)`);
ctx.toolState.diffCoverage = createDiffCoverageState({
diffPath,
totalLines: countLines({ content: formatResult.content }),
toc: formatResult.toc,
previous: ctx.toolState.diffCoverage,
});
log.debug(
`» diff coverage initialized: diffPath=${diffPath}, totalLines=${ctx.toolState.diffCoverage.totalLines}, tocEntries=${ctx.toolState.diffCoverage.tocEntries.length}`
);
// cache commentable-lines snapshot so review-time validation matches what
// GitHub will anchor to (commit_id=checkoutSha), even if the PR is updated
// between checkout and review.
const cached = new Map<string, ReturnType<typeof commentableLinesForFile>>();
for (const file of formatResult.files) {
cached.set(file.filename, commentableLinesForFile(file.patch));
}
ctx.toolState.commentableLinesByFile = cached;
ctx.toolState.commentableLinesPullNumber = pull_number;
ctx.toolState.commentableLinesCheckoutSha = ctx.toolState.checkoutSha;
const incrementalInstructions = incrementalDiffPath
? ` IMPORTANT: incrementalDiffPath contains ONLY the changes since the last reviewed version ` +
`(computed via range-diff). you MUST read incrementalDiffPath FIRST to understand what changed, ` +
`then use diffPath for full PR context. do NOT skip the incremental diff.`
: "";
// commit metadata relative to the PR base (e.g. main). use origin/<base>
// because the local base ref may not exist after a shallow fetch. cap
// the log so a PR with thousands of commits doesn't blow up the tool
// response. if the base ref can't be resolved (e.g. shallow fetch that
// didn't pull down origin/<base>), degrade gracefully rather than
// failing the whole checkout_pr call over metadata.
const COMMIT_LOG_MAX = 200;
const baseRange = `origin/${pr.baseRef}..HEAD`;
let commitCount = 0;
let commitLog = "";
let commitLogUnavailable = false;
try {
commitCount = parseInt(
$("git", ["rev-list", "--count", baseRange], { log: false }).trim() || "0",
10
);
commitLog = $("git", ["log", "--oneline", `--max-count=${COMMIT_LOG_MAX}`, baseRange], {
log: false,
});
} catch (err) {
commitLogUnavailable = true;
log.debug(
`» unable to compute commit metadata for ${baseRange}: ${err instanceof Error ? err.message : String(err)}`
);
}
const commitLogTruncated = commitCount > COMMIT_LOG_MAX;
const hookWarningInstructions = checkoutResult.hookWarning
? ` HOOK WARNING: the post-checkout lifecycle hook reported a non-fatal failure (see hookWarning). ` +
`decide whether to retry based on the guidance in that field before proceeding.`
: "";
const commitLogInstructions = commitLogUnavailable
? ` NOTE: commit metadata is partial (base ref unreachable, likely a shallow fetch). ` +
`commitCount/commitLog may be 0/empty or incomplete; treat them as "unknown" rather than "no commits", ` +
`and use \`git log\` directly if you need the full history.`
: commitLogTruncated
? ` NOTE: commitLog was capped at ${COMMIT_LOG_MAX} entries out of ${commitCount} commits; ` +
`use \`git log\` directly if you need the full history.`
: "";
return {
success: true,
number: prResponse.data.number,
title: prResponse.data.title,
body: prResponse.data.body,
base: pr.baseRef,
localBranch: `pr-${pull_number}`,
remoteBranch: `refs/heads/${pr.headRef}`,
isFork: pr.headRepoFullName !== pr.baseRepoFullName,
maintainerCanModify: pr.maintainerCanModify,
url: prResponse.data.html_url,
headRepo: pr.headRepoFullName,
diffPath,
incrementalDiffPath,
toc: formatResult.toc,
commitCount,
commitLog,
commitLogTruncated,
commitLogUnavailable,
hookWarning: checkoutResult.hookWarning,
instructions:
`the diff file at diffPath contains a table of contents (TOC) at the top listing every changed file with its line range. ` +
`use the TOC line ranges as your checklist and read specific files from the diff instead of reading the entire file. ` +
`for example, if the TOC says "src/foo.ts → lines 5-42", read lines 5-42 from diffPath to see that file's changes. ` +
`review files selectively based on relevance rather than reading everything sequentially. ` +
`to inspect the PR's changed files, use diffPath — do NOT run \`git diff <base>..<head>\` to re-derive what's already in diffPath. the formatted diff with line numbers is authoritative. ` +
`\`git log\` and \`git diff --stat\` are fine for commit-range overview, and \`git diff\` / \`git diff --cached\` are fine for inspecting *your own* uncommitted changes — but PR review content MUST come from diffPath. ` +
`before your review is submitted, a one-time coverage pre-flight may error listing unread TOC regions. ` +
`retry the same create_pull_request_review call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session. ` +
`the local branch is 'localBranch' (pr-{number}), not the remote branch name. ` +
`when pushing, omit branchName to use the current branch. do not use remoteBranch as a local branch name.` +
incrementalInstructions +
hookWarningInstructions +
commitLogInstructions,
} satisfies CheckoutPrResult;
};
return tool({
name: "checkout_pr",
description:
"Checkout a pull request branch locally. This fetches the PR branch and sets up push configuration for fork PRs. " +
"Returns diffPath pointing to the formatted diff file.",
"Returns diffPath pointing to the formatted diff file. " +
"Example: `checkout_pr({ pull_number: 1234 })`. " +
"Transient fetch timeouts are common — retry the same call up to a few times before treating the failure as terminal. " +
"If the error mentions `.git/shallow.lock: File exists` or `.git/index.lock: File exists`, that's a stale lock from a prior timed-out fetch — remove it via the shell tool (`rm -f .git/shallow.lock .git/index.lock`) and retry.",
parameters: CheckoutPr,
execute: execute(async ({ pull_number }) => {
const prResponse = await ctx.octokit.rest.pulls.get({
owner: ctx.repo.owner,
repo: ctx.repo.name,
pull_number,
});
const headRepo = prResponse.data.head.repo;
if (!headRepo) {
throw new Error(`PR #${pull_number} source repository was deleted`);
const inFlight = inFlightCheckouts.get(pull_number);
if (inFlight) {
log.info(`» checkout_pr({pull_number:${pull_number}}) already in flight — sharing result`);
return inFlight;
}
const pr: PrData = {
number: pull_number,
headSha: prResponse.data.head.sha,
headRef: prResponse.data.head.ref,
headRepoFullName: headRepo.full_name,
baseRef: prResponse.data.base.ref,
baseRepoFullName: prResponse.data.base.repo.full_name,
maintainerCanModify: prResponse.data.maintainer_can_modify,
};
const checkoutResult = await checkoutPrBranch(pr, {
octokit: ctx.octokit,
owner: ctx.repo.owner,
name: ctx.repo.name,
gitToken: ctx.gitToken,
toolState: ctx.toolState,
shell: ctx.payload.shell,
postCheckoutScript: ctx.postCheckoutScript,
beforeSha: ctx.toolState.beforeSha,
});
const tempDir = process.env.PULLFROG_TEMP_DIR;
if (!tempDir) {
throw new Error(
"PULLFROG_TEMP_DIR not set - checkout_pr must run in pullfrog action context"
);
}
const headShort = ctx.toolState.checkoutSha!.slice(0, 7);
// compute incremental diff if we have a beforeSha to compare against
let incrementalDiffPath: string | undefined;
if (ctx.toolState.beforeSha && ctx.toolState.checkoutSha) {
const beforeShort = ctx.toolState.beforeSha.slice(0, 7);
const incremental = computeIncrementalDiff({
baseBranch: pr.baseRef,
beforeSha: ctx.toolState.beforeSha,
headSha: ctx.toolState.checkoutSha,
});
if (incremental) {
incrementalDiffPath = join(
tempDir,
`pr-${pull_number}-${beforeShort}-${headShort}-incremental.diff`
);
writeFileSync(incrementalDiffPath, incremental);
log.info(
`» incremental diff computed (${incremental.length} bytes) → ${incrementalDiffPath}`
// refuse to clobber an uncommitted tree whenever this call would move
// HEAD away from the target pr-N branch. keyed off the live current
// branch (not toolState.issueNumber, which is also written by
// get_issue / get_issue_comments / get_issue_events and so doesn't
// mean "currently checked out"). catches the subagent-sharing-cwd
// case from zed-industries/cloud (2026-05-18).
const currentBranch = $("git", ["rev-parse", "--abbrev-ref", "HEAD"], { log: false }).trim();
if (currentBranch !== `pr-${pull_number}`) {
const dirty = $("git", ["status", "--porcelain"], { log: false }).trim();
if (dirty) {
throw new Error(
`cannot checkout PR #${pull_number} while the working tree has uncommitted changes. ` +
`commit, push, or discard them before switching. dirty paths:\n${dirty}`
);
}
}
// fetch PR files and format with line numbers
const formatResult = await fetchAndFormatPrDiff(ctx, pull_number);
const diffPreview = formatResult.content.split("\n").slice(0, 100).join("\n");
log.debug(`formatted diff preview (first 100 lines):\n${diffPreview}`);
const diffPath = join(tempDir, `pr-${pull_number}-${headShort}.diff`);
writeFileSync(diffPath, formatResult.content);
log.debug(`wrote diff to ${diffPath} (${formatResult.content.length} bytes)`);
ctx.toolState.diffCoverage = createDiffCoverageState({
diffPath,
totalLines: countLines({ content: formatResult.content }),
toc: formatResult.toc,
previous: ctx.toolState.diffCoverage,
});
log.debug(
`» diff coverage initialized: diffPath=${diffPath}, totalLines=${ctx.toolState.diffCoverage.totalLines}, tocEntries=${ctx.toolState.diffCoverage.tocEntries.length}`
);
// cache commentable-lines snapshot so review-time validation matches what
// GitHub will anchor to (commit_id=checkoutSha), even if the PR is updated
// between checkout and review.
const cached = new Map<string, ReturnType<typeof commentableLinesForFile>>();
for (const file of formatResult.files) {
cached.set(file.filename, commentableLinesForFile(file.patch));
}
ctx.toolState.commentableLinesByFile = cached;
ctx.toolState.commentableLinesPullNumber = pull_number;
ctx.toolState.commentableLinesCheckoutSha = ctx.toolState.checkoutSha;
const incrementalInstructions = incrementalDiffPath
? ` IMPORTANT: incrementalDiffPath contains ONLY the changes since the last reviewed version ` +
`(computed via range-diff). you MUST read incrementalDiffPath FIRST to understand what changed, ` +
`then use diffPath for full PR context. do NOT skip the incremental diff.`
: "";
// commit metadata relative to the PR base (e.g. main). use origin/<base>
// because the local base ref may not exist after a shallow fetch. cap
// the log so a PR with thousands of commits doesn't blow up the tool
// response. if the base ref can't be resolved (e.g. shallow fetch that
// didn't pull down origin/<base>), degrade gracefully rather than
// failing the whole checkout_pr call over metadata.
const COMMIT_LOG_MAX = 200;
const baseRange = `origin/${pr.baseRef}..HEAD`;
let commitCount = 0;
let commitLog = "";
let commitLogUnavailable = false;
const promise = runCheckout(pull_number);
inFlightCheckouts.set(pull_number, promise);
try {
commitCount = parseInt(
$("git", ["rev-list", "--count", baseRange], { log: false }).trim() || "0",
10
);
commitLog = $("git", ["log", "--oneline", `--max-count=${COMMIT_LOG_MAX}`, baseRange], {
log: false,
});
} catch (err) {
commitLogUnavailable = true;
log.debug(
`» unable to compute commit metadata for ${baseRange}: ${err instanceof Error ? err.message : String(err)}`
);
return await promise;
} finally {
inFlightCheckouts.delete(pull_number);
}
const commitLogTruncated = commitCount > COMMIT_LOG_MAX;
const hookWarningInstructions = checkoutResult.hookWarning
? ` HOOK WARNING: the post-checkout lifecycle hook reported a non-fatal failure (see hookWarning). ` +
`decide whether to retry based on the guidance in that field before proceeding.`
: "";
const commitLogInstructions = commitLogUnavailable
? ` NOTE: commit metadata is partial (base ref unreachable, likely a shallow fetch). ` +
`commitCount/commitLog may be 0/empty or incomplete; treat them as "unknown" rather than "no commits", ` +
`and use \`git log\` directly if you need the full history.`
: commitLogTruncated
? ` NOTE: commitLog was capped at ${COMMIT_LOG_MAX} entries out of ${commitCount} commits; ` +
`use \`git log\` directly if you need the full history.`
: "";
return {
success: true,
number: prResponse.data.number,
title: prResponse.data.title,
body: prResponse.data.body,
base: pr.baseRef,
localBranch: `pr-${pull_number}`,
remoteBranch: `refs/heads/${pr.headRef}`,
isFork: pr.headRepoFullName !== pr.baseRepoFullName,
maintainerCanModify: pr.maintainerCanModify,
url: prResponse.data.html_url,
headRepo: pr.headRepoFullName,
diffPath,
incrementalDiffPath,
toc: formatResult.toc,
commitCount,
commitLog,
commitLogTruncated,
commitLogUnavailable,
hookWarning: checkoutResult.hookWarning,
instructions:
`the diff file at diffPath contains a table of contents (TOC) at the top listing every changed file with its line range. ` +
`use the TOC line ranges as your checklist and read specific files from the diff instead of reading the entire file. ` +
`for example, if the TOC says "src/foo.ts → lines 5-42", read lines 5-42 from diffPath to see that file's changes. ` +
`review files selectively based on relevance rather than reading everything sequentially. ` +
`to inspect the PR's changed files, use diffPath — do NOT run \`git diff <base>..<head>\` to re-derive what's already in diffPath. the formatted diff with line numbers is authoritative. ` +
`\`git log\` and \`git diff --stat\` are fine for commit-range overview, and \`git diff\` / \`git diff --cached\` are fine for inspecting *your own* uncommitted changes — but PR review content MUST come from diffPath. ` +
`before your review is submitted, a one-time coverage pre-flight may error listing unread TOC regions. ` +
`retry the same create_pull_request_review call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session. ` +
`the local branch is 'localBranch' (pr-{number}), not the remote branch name. ` +
`when pushing, omit branchName to use the current branch. do not use remoteBranch as a local branch name.` +
incrementalInstructions +
hookWarningInstructions +
commitLogInstructions,
} satisfies CheckoutPrResult;
}),
});
}
+11 -4
View File
@@ -33,6 +33,7 @@ function buildCommentFooter(ctx: ToolContext, customParts?: string[]): string {
: undefined,
customParts,
model: ctx.toolState.model,
fallbackFrom: ctx.toolState.modelFallback?.from,
});
}
@@ -65,7 +66,9 @@ export function CreateCommentTool(ctx: ToolContext) {
return tool({
name: "create_issue_comment",
description:
"Create a comment on a GitHub issue or PR. For progress/plan updates on the current run use report_progress instead. Use type: 'Plan' for plan comments.",
"Create a comment on a GitHub issue or PR. " +
'Example: `create_issue_comment({ issueNumber: 1234, body: "Thanks for the report." })`. ' +
"For progress/plan updates on the current run use report_progress instead — plan output (initial post AND revisions) is always posted via report_progress, never via this tool.",
parameters: Comment,
execute: execute(async ({ issueNumber, body, type: commentType }) => {
const bodyWithFooter = addFooter(ctx, body);
@@ -150,7 +153,7 @@ export function EditCommentTool(ctx: ToolContext) {
export const ReportProgress = type({
body: type.string.describe("the progress update content to share"),
"target_plan_comment?": type("boolean").describe(
"when true, update the existing plan comment (from select_mode lookup) instead of the progress comment; use when editing an existing plan"
"for revising an existing plan comment ONLY. set to true only when the PlanEdit checklist from select_mode tells you to (i.e. a prior plan comment was found for this issue). NEVER set on the initial plan post — the initial plan reuses the run's progress comment and is posted by calling report_progress without this flag."
),
});
@@ -310,7 +313,9 @@ export function ReportProgressTool(ctx: ToolContext) {
return tool({
name: "report_progress",
description:
"Share progress on the associated GitHub issue/PR. The first call creates a comment; subsequent calls update it in place. Call this at the end of every run with a brief final summary (1-3 sentences) unless the mode guidance instructs otherwise. The current task list is automatically appended in a collapsible section — do not restate individual steps.",
"Share progress on the associated GitHub issue/PR. The first call creates a comment; subsequent calls update it in place. " +
'Example: `report_progress({ body: "Implemented the auth check and added tests." })`. ' +
"Call this at the end of every run with a brief final summary (1-3 sentences) unless the mode guidance instructs otherwise. The current task list is automatically appended in a collapsible section — do not restate individual steps.",
parameters: ReportProgress,
execute: execute(async (params) => {
let body = params.body;
@@ -445,7 +450,9 @@ export function ReplyToReviewCommentTool(ctx: ToolContext) {
return tool({
name: "reply_to_review_comment",
description:
"Reply to a PR review comment thread (NOT issue comments — this only works for inline review comments on PR diffs). Call exactly ONCE per parent comment you address in AddressReviews mode — duplicate calls with the same body are a no-op. Keep replies extremely brief (1 sentence max).",
"Reply to a PR review comment thread (NOT issue comments — this only works for inline review comments on PR diffs). " +
'Example: `reply_to_review_comment({ pull_number: 1234, comment_id: 567890, body: "Fixed by adding a null check." })`. ' +
"Call exactly ONCE per parent comment you address in AddressReviews mode — duplicate calls with the same body are a no-op. Keep replies extremely brief (1 sentence max).",
parameters: ReplyToReviewComment,
execute: execute(async ({ pull_number, comment_id, body }) => {
const bodyWithFooter = addFooter(ctx, body);
+2 -1
View File
@@ -15,7 +15,8 @@ export function CommitInfoTool(ctx: ToolContext) {
name: "get_commit_info",
description:
"Retrieve commit metadata and diff via GitHub API. Use this instead of git show for reviewing commits - " +
"it works with shallow clones and shows the actual changes in the commit. Returns diffPath pointing to formatted diff file.",
"it works with shallow clones and shows the actual changes in the commit. Returns diffPath pointing to formatted diff file. " +
'Example: `get_commit_info({ sha: "2a6ab5d" })`.',
parameters: CommitInfo,
execute: execute(async ({ sha }) => {
const response = await ctx.octokit.rest.repos.getCommit({
+25 -6
View File
@@ -176,13 +176,32 @@ export function sanitizeToolForGemini<T extends Tool<any, any>>(tool: T): T {
}
/**
* true when the effective upstream model is served by google's generative
* language API — directly (`google/*`), via opencode (`opencode/gemini-*`),
* or via openrouter (`openrouter/google/gemini-*`). slug-substring match
* works because every gemini route's model id contains "gemini".
* true when the effective upstream model is — or might become — google
* generative language API traffic. matches:
* - direct `google/*`, opencode `opencode/gemini-*`, openrouter
* `openrouter/google/gemini-*` (slug substring "gemini" wins).
* - any unresolved specifier: `undefined`, `"auto"`, or a slug that
* didn't map through the alias registry (no `provider/` prefix).
* these flow through the agent's own auto-select, which may land
* on gemini *after* the MCP server has already registered tools —
* at which point sanitization is too late to apply. erring on the
* side of sanitizing is safe: cases 1 + 2 are universally
* compatible JSON-Schema normalizations (enum-only → typed string,
* collapsible const-unions → string enum); case 3 is gemini-
* specific but only fires on non-collapsible unions, which arktype
* does not emit for our current tool schemas. see issue #676 for
* the prod failure that motivated this widening.
*/
export function isGeminiRouted(ctx: ToolContext): boolean {
const effective = ctx.payload.proxyModel ?? ctx.resolvedModel ?? ctx.payload.model;
if (!effective) return false;
return effective.toLowerCase().includes("gemini");
if (!effective) return true;
const normalized = effective.toLowerCase();
if (normalized.includes("gemini")) return true;
// every concrete model resolved through the registry carries a
// `provider/` prefix (e.g. "anthropic/claude-opus-4-7"). anything
// without a slash is either the literal `"auto"` alias or an
// unrecognized slug that resolveModel logged a warning for — both
// route through the agent's late auto-select, which may pick gemini.
if (!normalized.includes("/")) return true;
return false;
}
+121 -59
View File
@@ -2,8 +2,8 @@ import { regex } from "arkregex";
import { type } from "arktype";
import type { StoredPushDest } from "../toolState.ts";
import { log } from "../utils/cli.ts";
import { $git } from "../utils/gitAuth.ts";
import { executeLifecycleHook } from "../utils/lifecycle.ts";
import { $git, $gitFetchWithDeepen } from "../utils/gitAuth.ts";
import { executeLifecycleHook, type LifecycleHookFailure } from "../utils/lifecycle.ts";
import { $ } from "../utils/shell.ts";
import type { ToolContext } from "./server.ts";
import { execute, tool } from "./shared.ts";
@@ -218,10 +218,12 @@ export function PushBranchTool(ctx: ToolContext) {
name: "push_branch",
description:
"Push the current branch to the remote repository. Omit branchName to push the current branch (recommended). " +
'Example: `push_branch({})` to push the current branch. Example: `push_branch({ branchName: "pr-1" })` to push a specific local branch. ' +
"If specifying branchName, use the LOCAL branch name (e.g., 'pr-1'), not the remote branch name. " +
"The correct remote and remote branch are determined automatically from branch config set by checkout_pr. " +
"Requires a clean working tree. Runs the repository prepush hook (if configured) before the network push — hook failure means tests/lint or similar in that script failed, not necessarily a Pullfrog timeout. " +
"Never force push unless explicitly requested. Pushes to the default branch are blocked in restricted mode.",
"Requires a clean working tree. Runs the repository prepush hook (if configured) — best-effort. If the hook fails, the tool returns the failure output and every subsequent call this run skips the hook. " +
"Never force push unless explicitly requested. Pushes to the default branch are blocked in restricted mode. " +
"If the response reports a timeout, the underlying push may have actually succeeded — verify with `git log origin/<branch>` (or this tool with command 'log') before retrying, otherwise you'll push a duplicate.",
parameters: PushBranch,
execute: execute(async ({ branchName, force }) => {
// permission check
@@ -241,13 +243,38 @@ export function PushBranchTool(ctx: ToolContext) {
if (status) {
throw new Error(
`push blocked: working tree is not clean (tracked changes and/or untracked files). commit, discard, or remove stray artifacts before pushing.\n\n` +
`git status:\n${status}`
`git status:\n${status}` +
(ctx.toolState.prepushFailureCount > 0
? "\n\nnote: the prepush hook failed earlier this run — once the working tree is clean, push_branch will skip the hook."
: "")
);
}
// validate push destination matches expected URL
const pushDest = validatePushDestination(ctx, branch);
// backstop against subagent-induced cross-PR clobbers: a subagent
// shares cwd + toolState with the orchestrator, so its `checkout_pr(N)`
// moves HEAD to pr-N and persists pushDest pointing at the foreign
// PR's remote branch. refuse pr-N → origin/<other> pushes unless this
// run is itself scoped to PR N (zed-industries/cloud, 2026-05-18).
const prBranchMatch = branch.match(/^pr-(\d+)$/);
if (prBranchMatch && pushDest.remoteBranch !== branch) {
const prNumber = Number(prBranchMatch[1]);
const event = ctx.payload.event;
const runScoped = event.is_pr === true && event.issue_number === prNumber;
if (!runScoped) {
throw new Error(
`push blocked: local branch '${branch}' would push to '${pushDest.remoteName}/${pushDest.remoteBranch}', ` +
`but this run is not scoped to PR #${prNumber}. ` +
`the 'pr-${prNumber}' branch was created by a prior checkout_pr call (likely from a subagent — subagents share the working tree and toolState with the orchestrator). ` +
`you have probably landed your commit on the wrong branch. ` +
`switch to your own feature branch first (e.g. 'git checkout <feature-branch>') and then push. ` +
`if the push to PR #${prNumber} is intentional, this run needs to be triggered against that PR.`
);
}
}
// block pushes to default branch in restricted mode
if (pushPermission === "restricted" && pushDest.remoteBranch === defaultBranch) {
throw new Error(
@@ -263,27 +290,31 @@ export function PushBranchTool(ctx: ToolContext) {
? ["--force", "-u", pushDest.remoteName, refspec]
: ["-u", pushDest.remoteName, refspec];
// prepush failure should block the push — a passing hook is the gate
// that protects main from bad pushes.
const prepushHook = await executeLifecycleHook({
event: "prepush",
script: ctx.prepushScript,
});
if (prepushHook.warning) {
throw new Error(prepushHook.warning);
}
const prepushSkipped = ctx.toolState.prepushFailureCount > 0;
if (prepushSkipped) {
log.info(`» skipping prepush hook (failed earlier this run)`);
} else if (ctx.prepushScript) {
const prepushHook = await executeLifecycleHook({
event: "prepush",
script: ctx.prepushScript,
});
if (prepushHook.failure) {
ctx.toolState.prepushFailureCount += 1;
throw new Error(buildPrepushFailureMessage(prepushHook.failure, ctx.payload.shell));
}
// re-verify clean working tree after prepush. a hook that writes tracked
// files (formatter, type generator, build artifacts) would leave those
// changes uncommitted — pushing now would silently drop them, and the
// agent would report a "successful push" of code the hook had expected
// to be included.
const postHookStatus = $("git", ["status", "--porcelain"], { log: false });
if (postHookStatus) {
throw new Error(
`push blocked: the prepush hook modified the working tree. those changes are not included in the push. commit or discard them (or change the hook to not mutate tracked files) before retrying.\n\n` +
`git status:\n${postHookStatus}`
);
// re-verify clean working tree after prepush. a hook that writes tracked
// files (formatter, type generator, build artifacts) would leave those
// changes uncommitted — pushing now would silently drop them, and the
// agent would report a "successful push" of code the hook had expected
// to be included.
const postHookStatus = $("git", ["status", "--porcelain"], { log: false });
if (postHookStatus) {
throw new Error(
`push blocked: the prepush hook modified the working tree. those changes are not included in the push. commit or discard them (or change the hook to not mutate tracked files) before retrying.\n\n` +
`git status:\n${postHookStatus}`
);
}
}
log.debug(`pushing ${branch} to ${pushDest.remoteName}/${pushDest.remoteBranch}`);
@@ -357,18 +388,50 @@ export function PushBranchTool(ctx: ToolContext) {
`» pushed branch ${branch} to ${pushDest.remoteName}/${pushDest.remoteBranch} (sha ${pushedSha})`
);
const baseMsg = `successfully pushed ${branch} to ${pushDest.remoteName}/${pushDest.remoteBranch}`;
const message = prepushSkipped
? `${baseMsg} (prepush hook skipped — failed earlier this run).`
: baseMsg;
return {
success: true,
branch,
remoteBranch: pushDest.remoteBranch,
remote: pushDest.remoteName,
force,
message: `successfully pushed ${branch} to ${pushDest.remoteName}/${pushDest.remoteBranch}`,
prepushSkipped,
message,
};
}),
});
}
/** agent-facing prepush failure message: script output + bypass guidance,
* with no generic lifecycle retry advice (which would conflict). */
function buildPrepushFailureMessage(
failure: LifecycleHookFailure,
shell: ToolContext["payload"]["shell"]
): string {
const header =
failure.kind === "exit"
? `prepush hook failed with exit code ${failure.exitCode}.\n\nscript output:\n${failure.output || "(empty)"}`
: failure.kind === "timeout"
? `prepush hook timed out — the script is hung or doing too much work.`
: `prepush hook failed to spawn: ${failure.spawnError}.`;
const ifRealBug =
shell === "disabled"
? `fix it before pushing again — shell access is disabled in this run, so you can't re-run the hook command yourself.`
: `run the hook command yourself via the shell tool to iterate (push_branch will NOT re-run it).`;
return (
`${header}\n\n` +
`this repo's prepush hook is best-effort: the next push_branch call will SKIP the hook and proceed. ` +
`if the failure is unrelated to your changes (pre-existing breakage, flaky check), just call push_branch again. ` +
`if it could be a real bug in your code, ${ifRealBug}`
);
}
// commands that require authentication - redirect to dedicated tools.
// exported so tests can exercise the same table the runtime uses.
//
@@ -451,7 +514,10 @@ export function GitTool(ctx: ToolContext) {
return tool({
name: "git",
description:
"Run git commands. For push/fetch, use the dedicated MCP tools (push_branch, git_fetch). " +
"Run a git subcommand. `command` is a single subcommand; flags and positional args go in `args`. " +
'Example: `git({ command: "log", args: ["--oneline", "-n", "20"] })`. ' +
'Example: `git({ command: "diff", args: ["origin/main..HEAD"] })`. ' +
"For push/fetch, use the dedicated MCP tools (push_branch, git_fetch). " +
"git pull is not available — use git_fetch then this tool with command 'merge'.",
parameters: Git,
execute: execute(async (params) => {
@@ -485,6 +551,30 @@ export function GitTool(ctx: ToolContext) {
}
}
// `git merge-base --is-ancestor` uses exit codes as data: 0 = ancestor,
// 1 = not-an-ancestor, >1 = real error. Surface the binary answer
// instead of throwing on exit 1. see #766.
if (command === "merge-base" && args.includes("--is-ancestor")) {
let isAncestor = true;
$("git", [command, ...args], {
log: false,
onError: (r) => {
if (r.status === 1) {
isAncestor = false;
return;
}
const detail = [r.stderr, r.stdout]
.map((s) => s.trim())
.filter(Boolean)
.join("\n");
throw new Error(
`git merge-base --is-ancestor failed (exit ${r.status}): ${detail || "Unknown error"}`
);
},
});
return { success: true, isAncestor };
}
const output = $("git", [command, ...args], { log: false });
const lineCount = output.split("\n").length;
if (lineCount > COLLAPSE_THRESHOLD) {
@@ -505,25 +595,12 @@ const GitFetch = type({
depth: type.number.describe("Fetch depth (for shallow clones)").optional(),
});
// when an agent-supplied depth is too shallow to reach the merge base, git
// surfaces "Could not read <sha>" and "remote did not send all necessary
// objects". detect both wordings so a single deepen retry can recover before
// the error reaches the agent (issue #564). git emits the full OID via
// oid_to_hex, so the bound is 40 (SHA-1) or 64 (SHA-256).
const SHALLOW_UNREACHABLE_PATTERNS: RegExp[] = [
/Could not read [a-f0-9]{40,64}/,
/remote did not send all necessary objects/,
];
// large enough to clear the merge base on most real-world PRs without
// downloading the full history; matches the fallback used by checkoutPrBranch
// when the compare API is unavailable.
const DEEPEN_RETRY_DEPTH = 1000;
export function GitFetchTool(ctx: ToolContext) {
return tool({
name: "git_fetch",
description: "Fetch refs from remote repository. Use this instead of git fetch directly.",
description:
"Fetch refs from remote repository. Use this instead of git fetch directly. " +
'Example: `git_fetch({ ref: "main" })`. With depth: `git_fetch({ ref: "pull/1234/head", depth: 1 })`.',
parameters: GitFetch,
execute: execute(async (params) => {
rejectIfLeadingDash(params.ref, "ref");
@@ -531,22 +608,7 @@ export function GitFetchTool(ctx: ToolContext) {
if (params.depth !== undefined) {
fetchArgs.push(`--depth=${params.depth}`);
}
try {
await $git("fetch", fetchArgs, { token: ctx.gitToken });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const isShallowUnreachable = SHALLOW_UNREACHABLE_PATTERNS.some((p) => p.test(msg));
const isShallow =
isShallowUnreachable &&
$("git", ["rev-parse", "--is-shallow-repository"], { log: false }).trim() === "true";
if (!isShallow) throw err;
log.info(
`» git_fetch hit shallow-unreachable error, retrying with --deepen=${DEEPEN_RETRY_DEPTH}`
);
await $git("fetch", [`--deepen=${DEEPEN_RETRY_DEPTH}`, "--no-tags", "origin", params.ref], {
token: ctx.gitToken,
});
}
await $gitFetchWithDeepen(fetchArgs, { token: ctx.gitToken }, "git_fetch");
return { success: true, ref: params.ref };
}),
});
+2 -1
View File
@@ -10,7 +10,8 @@ export function GetIssueCommentsTool(ctx: ToolContext) {
return tool({
name: "get_issue_comments",
description:
"Get all comments for a GitHub issue. Returns all comments including the issue body and all subsequent discussion comments.",
"Get all comments for a GitHub issue. Returns all comments including the issue body and all subsequent discussion comments. " +
"Example: `get_issue_comments({ issue_number: 1234 })`.",
parameters: GetIssueComments,
execute: execute(async ({ issue_number }) => {
// set issue context
+5 -4
View File
@@ -27,10 +27,11 @@ export function GetIssueEventsTool(ctx: ToolContext) {
const relevantEventTypes = new Set(["cross_referenced", "referenced"]);
const parsedEvents = events.flatMap((event) => {
// Filter to only events with an 'event' property and relevant types
if (!("event" in event) || !relevantEventTypes.has(event.event)) {
return [];
}
// octokit's timeline-event union includes members with `event?:
// string`, so `"event" in event` does not narrow it to defined.
// require a string before the Set.has() check.
if (!("event" in event) || typeof event.event !== "string") return [];
if (!relevantEventTypes.has(event.event)) return [];
const baseEvent: Record<string, any> = {
event: event.event,
+3 -1
View File
@@ -9,7 +9,9 @@ export const IssueInfo = type({
export function IssueInfoTool(ctx: ToolContext) {
return tool({
name: "get_issue",
description: "Retrieve GitHub issue information by issue number",
description:
"Retrieve GitHub issue information by issue number. " +
"Example: `get_issue({ issue_number: 1234 })`.",
parameters: IssueInfo,
execute: execute(async ({ issue_number }) => {
const issue = await ctx.octokit.rest.issues.get({
+1
View File
@@ -23,6 +23,7 @@ function buildPrBodyWithFooter(ctx: ToolContext, body: string): string {
? { owner: ctx.repo.owner, repo: ctx.repo.name, runId: ctx.runId, jobId: ctx.jobId }
: undefined,
model: ctx.toolState.model,
fallbackFrom: ctx.toolState.modelFallback?.from,
});
const bodyWithoutFooter = stripExistingFooter(fixDoubleEscapedString(body));
+3 -1
View File
@@ -30,7 +30,9 @@ export function PullRequestInfoTool(ctx: ToolContext) {
return tool({
name: "get_pull_request",
description:
"Retrieve PR metadata (title, body, state, branches, author, labels, linked issues). To checkout a PR branch locally, use checkout_pr instead.",
"Retrieve PR metadata (title, body, state, branches, author, labels, linked issues). " +
"Example: `get_pull_request({ pull_number: 1234 })`. " +
"To checkout a PR branch locally, use checkout_pr instead.",
parameters: PullRequestInfo,
execute: execute(async ({ pull_number }) => {
// fetch REST and GraphQL in parallel
+13 -7
View File
@@ -316,18 +316,20 @@ export const CreatePullRequestReview = type({
.optional(),
approved: type.boolean
.describe(
"Set to true to submit as an approval. Use for both 'no issues found' and informational `> [!NOTE]` reviews where the PR is mergeable as-is and nothing in the body warrants code changes — approving also suppresses the Fix-button footer affordance so users don't dispatch a fix run on non-actionable feedback. Reserve approved: false for `> [!IMPORTANT]` (recommended changes) and `> [!CAUTION]` (critical) reviews. Defaults to false (comment-only review). Rejections are not supported."
"Set to true to submit as an approval. Use for `> ✅ No new issues found.` reviews where the PR is mergeable as-is and nothing in the body warrants code changes — approving also suppresses the Fix-button footer affordance so users don't dispatch a fix run on non-actionable feedback. Reserve approved: false for `> ️ ...` (minor suggestions inline), `> [!IMPORTANT]` (recommended changes), and `> [!CAUTION]` (critical) reviews. Defaults to false (comment-only review). Rejections are not supported."
)
.optional(),
commit_id: type.string
.describe("Optional SHA of the commit being reviewed. Defaults to latest.")
.describe(
"Optional SHA of the commit being reviewed. Defaults to latest. Must be the FULL 40-character SHA — abbreviated SHAs are rejected by GitHub with `422 Unprocessable Entity`. The PR-synchronize event payload's `head_sha` is already full-length."
)
.optional(),
comments: type({
path: type.string.describe(
"The file path to comment on (relative to repo root). Must be a file that appears in the PR diff."
),
line: type.number.describe(
"Line number to comment on. For multi-line ranges, this is the end line. Use NEW column from diff format."
"Line number to comment on. For multi-line ranges, this is the end line. Use NEW column from diff format. Must sit inside a `@@` hunk in the PR diff — anchors on context-only or untouched lines are dropped silently (the rest of the review still posts; dropped entries are reported under `droppedComments` in the response)."
),
side: type
.enumerated("LEFT", "RIGHT")
@@ -345,7 +347,7 @@ export const CreatePullRequestReview = type({
.optional(),
start_line: type.number
.describe(
"Start line for multi-line comment ranges. Omit for single-line comments. The range [start_line, line] defines which lines a suggestion replaces."
"Start line for multi-line comment ranges. Omit for single-line comments. The range [start_line, line] defines which lines a suggestion replaces. Both `start_line` and `line` must sit inside the same `@@` hunk — a `start_line` outside the hunk causes the whole comment to be dropped even when `line` is valid. If you need to comment on context just above/below a hunk, shrink the range to a single line that is provably modified."
)
.optional(),
})
@@ -361,6 +363,7 @@ export function CreatePullRequestReviewTool(ctx: ToolContext) {
name: "create_pull_request_review",
description:
"Submit a review for an existing pull request. " +
'Example: `create_pull_request_review({ pull_number: 1234, body: "LGTM", approved: true, comments: [{ path: "src/api.ts", line: 42, body: "nit: rename" }] })`. ' +
"Each call creates a permanent, visible review on the PR — NEVER submit test or diagnostic reviews. " +
"Reviews with no body AND no comments are silently skipped (nothing to post). " +
"IMPORTANT: 95%+ of feedback should be in 'comments' array with file paths and line numbers. " +
@@ -709,7 +712,9 @@ function runDiffCoveragePreflight(params: { ctx: ToolContext }): void {
.join("\n");
throw new Error(
`diff coverage pre-flight: some TOC regions were not read before review submission. ` +
`this is a one-time nudge — optionally read the ranges below from ${coverageState.diffPath}, then call create_pull_request_review again with the same arguments. ` +
`this is a one-time nudge — read the ranges below from ${coverageState.diffPath} on a best-effort basis, then call create_pull_request_review again. ` +
`you are NOT obligated to read generated artifacts (lockfiles like pnpm-lock.yaml / package-lock.json / yarn.lock / Cargo.lock; codegen output like *.gen.*, *.pb.go, *.generated.*; snapshot/fixture dirs like __snapshots__/; migration metadata like drizzle/meta/, prisma migration SQL). ` +
`if every unread region is generated, retry immediately without reading. ` +
`this pre-flight will not block again in this review session.\n\n` +
`unread TOC regions:\n${unreadText}\n\n` +
`${coverageState.lastBreakdown}`
@@ -845,8 +850,8 @@ async function createAndSubmitWithFooter(
// introduce new throw paths. keep the whole body wrapped.
try {
// Fix buttons are suppressed on approving reviews — those are mergeable
// by definition (either "no issues found" or `> [!NOTE]` informational
// observations), so dispatching a fix run would be a UX trap.
// by definition (the `> ✅ No new issues found.` tier, with no inline
// comments), so dispatching a fix run would be a UX trap.
const customParts: string[] = [];
if (!opts.approved) {
const apiUrl = getApiUrl();
@@ -866,6 +871,7 @@ async function createAndSubmitWithFooter(
: undefined,
customParts,
model: ctx.toolState.model,
fallbackFrom: ctx.toolState.modelFallback?.from,
});
return await ctx.octokit.rest.pulls.submitReview({
+5 -1
View File
@@ -602,6 +602,7 @@ export function GetReviewCommentsTool(ctx: ToolContext) {
name: "get_review_comments",
description:
"Get review comments for a pull request review with full thread context. " +
"Example: `get_review_comments({ pull_number: 1234, review_id: 567890 })`. " +
"Automatically filters to approved comments when applicable. " +
"Returns a TOC and commentsPath pointing to a markdown file with full comment details.",
parameters: GetReviewComments,
@@ -673,7 +674,8 @@ export function ListPullRequestReviewsTool(ctx: ToolContext) {
return tool({
name: "list_pull_request_reviews",
description:
"List all reviews for a pull request. Returns all reviews including approvals, request changes, and comments.",
"List all reviews for a pull request. Returns all reviews including approvals, request changes, and comments. " +
"Example: `list_pull_request_reviews({ pull_number: 1234 })`.",
parameters: ListPullRequestReviews,
execute: execute(async (params) => {
const reviews = await ctx.octokit.paginate(ctx.octokit.rest.pulls.listReviews, {
@@ -691,6 +693,8 @@ export function ListPullRequestReviewsTool(ctx: ToolContext) {
state: review.state,
user: review.user?.login,
submitted_at: review.submitted_at,
commit_id: review.commit_id,
html_url: review.html_url,
})),
count: reviews.length,
};
+2 -1
View File
@@ -121,7 +121,8 @@ export function SelectModeTool(ctx: ToolContext) {
return tool({
name: "select_mode",
description:
"Select a mode and receive step-by-step guidance on how to handle the task. Call this to understand the best workflow for the current mode.",
"Select a mode and receive step-by-step guidance on how to handle the task. Call this to understand the best workflow for the current mode. " +
'Example: `select_mode({ mode: "Review" })` or `select_mode({ mode: "Plan", issue_number: 1234 })`.',
parameters: SelectModeParams,
execute: execute(async (params) => {
if (ctx.toolState.selectedMode) {
+4 -4
View File
@@ -63,10 +63,10 @@ export interface ToolContext {
mcpServerUrl: string;
tmpdir: string;
// repo-level OSS flag + account-level billing plan. together they decide
// whether pullfrog is paying for marginal infra — see isInfraCovered in
// utils/runContext.ts. plan gating for endpoints like the learnings PATCH
// is enforced server-side via 402, so we pass plan along mostly for future
// use / observability. see wiki/pricing.md.
// whether pullfrog is paying for marginal infra — see `isInfraCovered` in
// the server's `utils/billing.ts`. plan gating for endpoints like the
// learnings PATCH is enforced server-side via 402, so we pass plan along
// mostly for future use / observability. see wiki/pricing.md.
oss: boolean;
plan: AccountPlan;
// resolved upstream model specifier (e.g. "google/gemini-3.1-pro-preview").
+57 -5
View File
@@ -15,7 +15,9 @@ import { execute, tool } from "./shared.ts";
export const ShellParams = type({
command: "string",
description: "string",
"timeout?": "number",
"timeout?": type.number.describe(
"Timeout in MILLISECONDS (not seconds). Default 30000 (30s), max 120000 (2m). e.g. timeout: 180000 for 3 minutes; timeout: 180 means 180ms and will kill the process almost immediately."
),
"working_directory?": "string",
"background?": "boolean",
});
@@ -94,6 +96,27 @@ function detectSandboxMethod(): SandboxMethod {
const PROC_CLEANUP =
"umount /proc 2>/dev/null; umount /proc 2>/dev/null; mount -t proc proc /proc 2>/dev/null;";
// block container-runtime sockets that would otherwise grant a PID-namespace
// escape: `docker run --pid=host --privileged busybox cat /proc/<pid>/environ`
// reads the parent action process's env (which contains user secrets) even
// though the sandbox itself is unsharing PIDs. GHA `ubuntu-latest` puts the
// `runner` user in the `docker` group by default, so the socket is reachable
// without sudo. bind-mounting /dev/null on top inside the sandbox's mount
// namespace makes the socket unreachable from sandboxed shells without
// touching the host runner (so it doesn't break user workflow steps that
// run before/after pullfrog and legitimately need docker). same trick for
// podman/containerd/cri-o sockets — all silent-fail if the path is missing.
const SOCKET_CLEANUP = [
"/var/run/docker.sock",
"/run/docker.sock",
"/var/run/podman/podman.sock",
"/run/podman/podman.sock",
"/run/containerd/containerd.sock",
"/var/run/crio/crio.sock",
]
.map((path) => `mount --bind /dev/null ${path} 2>/dev/null;`)
.join(" ");
function spawnShell(params: SpawnParams): ChildProcess {
const spawnOpts = { env: params.env, cwd: params.cwd, stdio: params.stdio, detached: true };
const sandboxMethod = detectSandboxMethod();
@@ -108,7 +131,14 @@ function spawnShell(params: SpawnParams): ChildProcess {
if (sandboxMethod === "unshare") {
return spawn(
"unshare",
["--pid", "--fork", "--mount-proc", "bash", "-c", `${PROC_CLEANUP} ${params.command}`],
[
"--pid",
"--fork",
"--mount-proc",
"bash",
"-c",
`${PROC_CLEANUP} ${SOCKET_CLEANUP} ${params.command}`,
],
spawnOpts
);
}
@@ -141,7 +171,7 @@ function spawnShell(params: SpawnParams): ChildProcess {
"--mount-proc",
"bash",
"-c",
`${PROC_CLEANUP} exec su -p -s /bin/bash ${username} -c '${escaped}'`,
`${PROC_CLEANUP} ${SOCKET_CLEANUP} exec su -p -s /bin/bash ${username} -c '${escaped}'`,
],
{ ...spawnOpts, env: {} }
);
@@ -174,6 +204,23 @@ function getTempDir(): string {
return tempDir;
}
/** chars of shell output kept inline in the agent reply. anything past this
* blows the agent's context budget on commands that dump big logs (test
* runners, build tools, grep on large trees), so the overflow is spilled
* to a tempfile the agent can re-read selectively (cat/tail/grep). */
export const MAX_OUTPUT_CHARS = 5000;
/** if `output` exceeds `MAX_OUTPUT_CHARS`, persist the full body to a
* tempfile and return the last `MAX_OUTPUT_CHARS` prefixed with a sentinel
* pointing at the saved path. otherwise return as-is. */
function capOutput(output: string): string {
if (output.length <= MAX_OUTPUT_CHARS) return output;
const fullPath = join(getTempDir(), `shell-${randomUUID().slice(0, 8)}.log`);
writeFileSync(fullPath, output);
const elided = output.length - MAX_OUTPUT_CHARS;
return `... [${elided} chars truncated; full output saved to ${fullPath}] ...\n${output.slice(-MAX_OUTPUT_CHARS)}`;
}
/** detect git as a command invocation (not as part of another word like .gitignore) */
function isGitCommand(command: string): boolean {
const trimmed = command.trim();
@@ -187,11 +234,15 @@ export function ShellTool(ctx: ToolContext) {
name: "shell",
description: `Execute shell commands securely. Environment is filtered to remove API keys and secrets.
Example: \`shell({ command: "pnpm test", description: "run the test suite" })\`.
Use this tool to:
- Run shell commands (ls, cat, grep, find, etc.)
- Execute build tools (npm, pnpm, cargo, make, etc.)
- Run tests and linters
Output is capped at ${MAX_OUTPUT_CHARS} chars: if exceeded, only the tail is returned and the full body is saved to a tempfile (path included in the response). Re-read the tempfile with cat/tail/grep when you need more.
Do NOT use this tool for git commands — use the dedicated git tools instead.`,
parameters: ShellParams,
execute: execute(async (params) => {
@@ -297,13 +348,14 @@ Do NOT use this tool for git commands — use the dedicated git tools instead.`,
: `[timed out after ${timeout}ms]`;
const finalExitCode = exitCode ?? (timedOut ? 124 : -1);
const trimmed = output.trim();
if (finalExitCode !== 0) {
log.info(`shell command failed with exit code ${finalExitCode}: ${params.command}`);
if (output) log.info(`output: ${output.trim()}`);
if (trimmed) log.info(`output: ${trimmed}`);
}
return {
output: output.trim(),
output: capOutput(trimmed),
exit_code: finalExitCode,
timed_out: timedOut,
};
+60 -2
View File
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import {
getModelEnvVars,
getModelProvider,
isBedrockAnthropicId,
modelAliases,
parseModel,
providers,
@@ -55,13 +56,13 @@ describe("getModelEnvVars", () => {
it("returns empty env vars for free opencode models", () => {
expect(getModelEnvVars("opencode/big-pickle")).toEqual([]);
expect(getModelEnvVars("opencode/gpt-5-nano")).toEqual([]);
expect(getModelEnvVars("opencode/mimo-v2-pro-free")).toEqual([]);
expect(getModelEnvVars("opencode/minimax-m2.5-free")).toEqual([]);
});
it("still requires OPENCODE_API_KEY for non-free opencode models", () => {
expect(getModelEnvVars("opencode/claude-opus")).toEqual(["OPENCODE_API_KEY"]);
expect(getModelEnvVars("opencode/gpt-5-nano")).toEqual(["OPENCODE_API_KEY"]);
});
});
@@ -175,7 +176,12 @@ describe("modelAliases registry", () => {
it("has exactly one preferred model per provider", () => {
for (const providerKey of Object.keys(providers)) {
const preferred = modelAliases.filter((a) => a.provider === providerKey && a.preferred);
// routing-only providers (bedrock) deliberately have no preferred
// model — the user picks the actual model via a per-run env var, so
// there's no "preferred default" to surface to auto-select.
const aliases = modelAliases.filter((a) => a.provider === providerKey);
if (aliases.every((a) => a.routing)) continue;
const preferred = aliases.filter((a) => a.preferred);
expect(preferred.length, `${providerKey} should have exactly 1 preferred model`).toBe(1);
}
});
@@ -190,6 +196,10 @@ describe("modelAliases registry", () => {
it("all resolve values follow provider/model format", () => {
for (const alias of modelAliases) {
// routing slugs use a sentinel `resolve` (e.g. "bedrock") that's never
// passed to a CLI directly — the harness reads a separate env var to
// get the real model ID. format check doesn't apply.
if (alias.routing) continue;
expect(alias.resolve).toContain("/");
}
});
@@ -200,6 +210,54 @@ describe("modelAliases registry", () => {
});
});
describe("isBedrockAnthropicId", () => {
it("matches geo-prefixed Anthropic foundation IDs", () => {
expect(isBedrockAnthropicId("us.anthropic.claude-opus-4-7")).toBe(true);
expect(isBedrockAnthropicId("eu.anthropic.claude-sonnet-4-6")).toBe(true);
expect(isBedrockAnthropicId("global.anthropic.claude-haiku-4-5-20251001-v1:0")).toBe(true);
});
it("matches in-region Anthropic foundation IDs", () => {
expect(isBedrockAnthropicId("anthropic.claude-opus-4-7")).toBe(true);
});
it("rejects non-Anthropic foundation IDs", () => {
expect(isBedrockAnthropicId("amazon.nova-pro-v1:0")).toBe(false);
expect(isBedrockAnthropicId("us.meta.llama4-scout-17b-instruct-v1:0")).toBe(false);
expect(isBedrockAnthropicId("deepseek.v3.2")).toBe(false);
});
// regression: PR #720 review caught that a substring-only match was
// fragile for inference-profile ARNs (which BEDROCK_MODEL_ID accepts per
// the AWS docs). ARN names are user-chosen — both directions of the
// heuristic could break depending on what name the operator picked.
// We anchor on a discrete dot-segment match (case-insensitive) instead.
it("ignores 'anthropic' substrings inside non-segment text", () => {
// ARN whose user-chosen profile name happens to contain "anthropic" as
// part of a longer word — would route to claude-code under naive
// includes("anthropic") even though the backing model is unknown.
expect(
isBedrockAnthropicId(
"arn:aws:bedrock:us-east-2:123456789012:application-inference-profile/my-anthropicish-profile"
)
).toBe(false);
});
it("matches when 'anthropic' appears as its own dot-segment in ARN", () => {
// ARN whose profile name embeds the foundation segment correctly —
// operator chose to surface the backing model in the name.
expect(
isBedrockAnthropicId(
"arn:aws:bedrock:us-east-2:123456789012:application-inference-profile/anthropic.claude-opus-4-7"
)
).toBe(true);
});
it("is case-insensitive", () => {
expect(isBedrockAnthropicId("US.ANTHROPIC.CLAUDE-OPUS-4-7")).toBe(true);
});
});
describe("providers registry", () => {
it("every provider has envVars", () => {
for (const [key, config] of Object.entries(providers)) {
+158 -8
View File
@@ -7,6 +7,22 @@
// ── types ──────────────────────────────────────────────────────────────────────
/**
* routing discriminant for entries whose `resolve` is dynamic — looked up
* from a separate env var at run time rather than fixed in the catalog.
*
* `"bedrock"` means the actual model ID comes from `BEDROCK_MODEL_ID`
* (an AWS-canonical Bedrock model ID like `us.anthropic.claude-opus-4-7`
* or `amazon.nova-pro-v1:0`). enterprise Bedrock customers self-select for
* version control — silent alias bumps would break compliance review,
* model-access enrollment, and provisioned-throughput contracts. so the
* single `bedrock/byok` entry is a routing slug, not a model alias: the
* harness reads `BEDROCK_MODEL_ID` and routes to claude-code (when the ID
* contains "anthropic") or opencode (everything else, with an
* `amazon-bedrock/` prefix).
*/
export type ModelRouting = "bedrock";
export interface ModelAlias {
/** stable alias stored in DB, e.g. "anthropic/claude-opus" */
slug: string;
@@ -14,9 +30,9 @@ export interface ModelAlias {
provider: string;
/** human-readable name shown in dropdowns */
displayName: string;
/** concrete models.dev specifier, e.g. "anthropic/claude-opus-4-6" */
/** concrete models.dev specifier, e.g. "anthropic/claude-opus-4-6". sentinel for routing entries — never passed to a CLI directly. */
resolve: string;
/** full models.dev specifier for the OpenRouter equivalent (undefined for free models) */
/** full models.dev specifier for the OpenRouter equivalent (undefined for free models and routing entries) */
openRouterResolve: string | undefined;
/** top-tier pick for this provider — preferred during auto-select */
preferred: boolean;
@@ -24,6 +40,15 @@ export interface ModelAlias {
isFree: boolean;
/** slug of a replacement model — presence implies this model is deprecated */
fallback: string | undefined;
/** dynamic-resolution discriminant — see ModelRouting docs */
routing: ModelRouting | undefined;
/** alias key (within same provider) of the cheaper sibling reviewfrog should
* use as its lens-fanout subagent. e.g. claude-opus → "claude-sonnet". */
subagentModel: string | undefined;
/** hide from selectable lists (UI dropdowns, CLI pickers). does NOT affect
* resolution — for that use `fallback`. used for internal-only tier targets
* (e.g. gpt-5.4 as a subagent target without exposing it to users). */
hidden: boolean;
}
interface ModelDef {
@@ -37,11 +62,23 @@ interface ModelDef {
isFree?: boolean;
/** slug of a replacement model — presence implies this model is deprecated */
fallback?: string;
/** dynamic-resolution discriminant — see ModelRouting docs */
routing?: ModelRouting;
/** alias key (within same provider) of the cheaper sibling reviewfrog should
* use as its lens-fanout subagent (e.g. claude-opus → "claude-sonnet"). */
subagentModel?: string;
/** hide from selectable lists. does NOT affect resolution; for that use `fallback`. */
hidden?: boolean;
}
export interface ProviderConfig {
displayName: string;
envVars: readonly string[];
/** credentials authored only via `pullfrog auth <provider>` — never
* user-facing in `init`, never documented as a manual GHA secret. counted
* for hasAnyKey / log-redaction purposes but excluded from any prompt /
* paste flow. CLI-managed magic. see wiki/codex-auth.md. */
managedCredentials?: readonly string[];
models: Record<string, ModelDef>;
}
@@ -61,6 +98,7 @@ export const providers = {
resolve: "anthropic/claude-opus-4-7",
openRouterResolve: "openrouter/anthropic/claude-opus-4.7",
preferred: true,
subagentModel: "claude-sonnet",
},
"claude-sonnet": {
displayName: "Claude Sonnet",
@@ -77,17 +115,29 @@ export const providers = {
openai: provider({
displayName: "OpenAI",
envVars: ["OPENAI_API_KEY"],
managedCredentials: ["CODEX_AUTH_JSON"],
models: {
gpt: {
displayName: "GPT",
resolve: "openai/gpt-5.5",
openRouterResolve: "openrouter/openai/gpt-5.5",
preferred: true,
subagentModel: "gpt-5.4",
},
"gpt-pro": {
displayName: "GPT Pro",
resolve: "openai/gpt-5.5-pro",
openRouterResolve: "openrouter/openai/gpt-5.5-pro",
subagentModel: "gpt",
},
// hidden subagent target — `gpt` lenses run against this. surfacing
// it in the picker would just confuse users (it's the prior-flagship,
// and they already have `gpt` and `gpt-mini` to choose from).
"gpt-5.4": {
displayName: "GPT 5.4",
resolve: "openai/gpt-5.4",
openRouterResolve: "openrouter/openai/gpt-5.4",
hidden: true,
},
"gpt-mini": {
displayName: "GPT Mini",
@@ -126,10 +176,15 @@ export const providers = {
resolve: "google/gemini-3.1-pro-preview",
openRouterResolve: "openrouter/google/gemini-3.1-pro-preview",
preferred: true,
// Inherit (subagents stay on Pro). Google has no in-between tier;
// dropping to Flash for review work was a meaningful capability cliff
// (Flash missed the catastrophic camelCase/snake_case mismatch in
// the v4 e2e test). Pro is cost-effective enough to use for both
// orchestrator and lenses.
},
"gemini-flash": {
displayName: "Gemini Flash",
resolve: "google/gemini-3-flash-preview",
resolve: "google/gemini-3.5-flash",
openRouterResolve: "openrouter/google/gemini-3-flash-preview",
},
},
@@ -144,15 +199,22 @@ export const providers = {
openRouterResolve: "openrouter/x-ai/grok-4.3",
preferred: true,
},
// legacy aliases — xAI retired the entire fast/code-fast line on
// 2026-05-15 (https://docs.x.ai/developers/migration/may-15-deprecation)
// and now redirects every deprecated text-model slug to grok-4.3 at
// standard pricing. fall back to the live `xai/grok` so the alias
// chain resolves to grok-4.3 for both direct-key and OpenRouter users.
"grok-fast": {
displayName: "Grok Fast",
resolve: "xai/grok-4-1-fast",
openRouterResolve: "openrouter/x-ai/grok-4.1-fast",
openRouterResolve: "openrouter/x-ai/grok-4.3",
fallback: "xai/grok",
},
"grok-code-fast": {
displayName: "Grok Code Fast",
resolve: "xai/grok-code-fast-1",
openRouterResolve: "openrouter/x-ai/grok-code-fast-1",
openRouterResolve: "openrouter/x-ai/grok-4.3",
fallback: "xai/grok",
},
},
}),
@@ -214,6 +276,7 @@ export const providers = {
displayName: "Claude Opus",
resolve: "opencode/claude-opus-4-7",
openRouterResolve: "openrouter/anthropic/claude-opus-4.7",
subagentModel: "claude-sonnet",
},
"claude-sonnet": {
displayName: "Claude Sonnet",
@@ -229,11 +292,20 @@ export const providers = {
displayName: "GPT",
resolve: "opencode/gpt-5.5",
openRouterResolve: "openrouter/openai/gpt-5.5",
subagentModel: "gpt-5.4",
},
"gpt-pro": {
displayName: "GPT Pro",
resolve: "opencode/gpt-5.5-pro",
openRouterResolve: "openrouter/openai/gpt-5.5-pro",
subagentModel: "gpt",
},
// hidden subagent target — see openai provider above for context.
"gpt-5.4": {
displayName: "GPT 5.4",
resolve: "opencode/gpt-5.4",
openRouterResolve: "openrouter/openai/gpt-5.4",
hidden: true,
},
"gpt-mini": {
displayName: "GPT Mini",
@@ -257,6 +329,7 @@ export const providers = {
displayName: "Gemini Pro",
resolve: "opencode/gemini-3.1-pro",
openRouterResolve: "openrouter/google/gemini-3.1-pro-preview",
// Inherit — see google/gemini-pro for rationale.
},
"gemini-flash": {
displayName: "Gemini Flash",
@@ -271,8 +344,7 @@ export const providers = {
"gpt-5-nano": {
displayName: "GPT Nano",
resolve: "opencode/gpt-5-nano",
envVars: [],
isFree: true,
openRouterResolve: "openrouter/openai/gpt-5-nano",
},
"mimo-v2-pro-free": {
displayName: "MiMo V2 Pro",
@@ -289,6 +361,20 @@ export const providers = {
},
},
}),
bedrock: provider({
displayName: "Amazon Bedrock",
envVars: ["AWS_BEARER_TOKEN_BEDROCK", "AWS_REGION", "BEDROCK_MODEL_ID"],
models: {
// single routing entry — the actual Bedrock model ID is read from
// BEDROCK_MODEL_ID at run time. see ModelRouting docs for why we
// don't catalog individual Bedrock models.
byok: {
displayName: "Amazon Bedrock",
resolve: "bedrock",
routing: "bedrock",
},
},
}),
openrouter: provider({
displayName: "OpenRouter",
envVars: ["OPENROUTER_API_KEY"],
@@ -298,6 +384,7 @@ export const providers = {
resolve: "openrouter/anthropic/claude-opus-4.7",
openRouterResolve: "openrouter/anthropic/claude-opus-4.7",
preferred: true,
subagentModel: "claude-sonnet",
},
"claude-sonnet": {
displayName: "Claude Sonnet",
@@ -313,11 +400,20 @@ export const providers = {
displayName: "GPT",
resolve: "openrouter/openai/gpt-5.5",
openRouterResolve: "openrouter/openai/gpt-5.5",
subagentModel: "gpt-5.4",
},
"gpt-pro": {
displayName: "GPT Pro",
resolve: "openrouter/openai/gpt-5.5-pro",
openRouterResolve: "openrouter/openai/gpt-5.5-pro",
subagentModel: "gpt",
},
// hidden subagent target — see openai provider above for context.
"gpt-5.4": {
displayName: "GPT 5.4",
resolve: "openrouter/openai/gpt-5.4",
openRouterResolve: "openrouter/openai/gpt-5.4",
hidden: true,
},
"gpt-mini": {
displayName: "GPT Mini",
@@ -346,6 +442,7 @@ export const providers = {
displayName: "Gemini Pro",
resolve: "openrouter/google/gemini-3.1-pro-preview",
openRouterResolve: "openrouter/google/gemini-3.1-pro-preview",
// Inherit — see google/gemini-pro for rationale.
},
"gemini-flash": {
displayName: "Gemini Flash",
@@ -420,6 +517,16 @@ export function getModelEnvVars(slug: string): string[] {
return providerConfig.envVars.slice();
}
/** managed credentials are authored only via `pullfrog auth <provider>` — they
* count as "configured" for hasAnyKey-style UI checks but are never offered as
* a manual-paste option in `init` or the AgentSettings env-var button row.
* see `provider.managedCredentials` and wiki/codex-auth.md. */
export function getModelManagedCredentials(slug: string): string[] {
const parsed = parseModel(slug);
const providerConfig = (providers as Record<string, ProviderConfig>)[parsed.provider];
return providerConfig?.managedCredentials?.slice() ?? [];
}
// ── derived flat list ──────────────────────────────────────────────────────────
export const modelAliases: ModelAlias[] = Object.entries(providers).flatMap(
@@ -433,6 +540,12 @@ export const modelAliases: ModelAlias[] = Object.entries(providers).flatMap(
preferred: def.preferred ?? false,
isFree: def.isFree ?? false,
fallback: def.fallback,
routing: def.routing,
// subagentModel is stored as an alias key local to the provider; expand
// here to a fully-qualified slug so callers can look up the target alias
// directly without re-deriving the provider.
subagentModel: def.subagentModel ? `${providerKey}/${def.subagentModel}` : undefined,
hidden: def.hidden ?? false,
}))
);
@@ -452,7 +565,7 @@ const MAX_FALLBACK_DEPTH = 10;
* use this in UI display sites (dropdown trigger labels, PR-comment footers,
* etc.) so a deprecated stored slug renders as the model the user actually
* runs against — not the historical name. selectable lists should still hide
* deprecated aliases by filtering on `!a.fallback`.
* deprecated and internal-only aliases by filtering on `!a.fallback && !a.hidden`.
*/
export function resolveDisplayAlias(slug: string): ModelAlias | undefined {
let current = slug;
@@ -486,3 +599,40 @@ export function resolveCliModel(slug: string): string | undefined {
export function resolveOpenRouterModel(slug: string): string | undefined {
return resolveDisplayAlias(slug)?.openRouterResolve;
}
// ── bedrock routing ────────────────────────────────────────────────────────────
/** env var that supplies the Bedrock model ID for the `bedrock/byok` slug. */
export const BEDROCK_MODEL_ID_ENV = "BEDROCK_MODEL_ID";
/**
* the Bedrock model ID passed to claude-code or opencode is whatever the
* user set in `BEDROCK_MODEL_ID` — Pullfrog never resolves or upgrades it.
* we route by checking whether the ID names an Anthropic model: claude-code
* handles Anthropic-on-Bedrock natively (with `CLAUDE_CODE_USE_BEDROCK=1`),
* everything else goes through opencode's `amazon-bedrock` provider.
*
* AWS Bedrock IDs come in two shapes:
* - dotted foundation IDs: `us.anthropic.claude-opus-4-7`,
* `anthropic.claude-haiku-4-5-20251001-v1:0`, `amazon.nova-pro-v1:0`,
* `meta.llama4-scout-17b-instruct-v1:0`. AWS-published, lowercase, the
* foundation provider always appears as a discrete dot-segment.
* - inference-profile ARNs: `arn:aws:bedrock:us-east-2:<acct>:application-inference-profile/<user-name>`.
* `<user-name>` is operator-chosen, so a naive substring check is fragile
* in both directions (Anthropic profile named without "anthropic" → routes
* to opencode and misses CLAUDE_CODE_USE_BEDROCK; non-Anthropic profile
* whose name happens to contain "anthropic" → routes to claude-code).
*
* we anchor on a discrete dot-segment match (case-insensitive). this catches
* every published foundation ID and is conservative for ARN-form IDs: ARN
* names that don't include "anthropic" as their own dot-segment route to
* opencode by default. operators using ARN-form IDs whose backing model is
* Anthropic should set `PULLFROG_AGENT=claude` to force the right route, or
* include the foundation segment in the profile name.
*/
export function isBedrockAnthropicId(bedrockModelId: string): boolean {
// split on `.`, `/`, and `:` so the check works for both dotted foundation
// IDs (anthropic.* / us.anthropic.*) and ARN-form IDs (where the relevant
// foundation segment sits between `/` and `.` inside the resource name).
return bedrockModelId.toLowerCase().split(/[./:]/).includes("anthropic");
}
+265 -112
View File
@@ -10,60 +10,154 @@ export interface Mode {
prompt?: string | undefined;
}
// Default user-facing summary format embedded in Review mode review bodies.
// Deliberately scoped to Review (initial PR review). IncrementalReview keeps
// its own terser bullet-list "Reviewed changes" shape since re-review bodies
// are deltas, not introductions. Distinct from the agent-internal snapshot
// (action/utils/prSummary.ts) which has its own stable scaffold and is never
// shaped by user instructions — see selectMode.ts for the firewall.
// Default user-facing summary format embedded in BOTH Review and
// IncrementalReview review bodies. The two modes share the preamble +
// cross-cutting + nitpicks shape; the only difference is scope (full PR for
// Review vs delta against the prior pullfrog review for IncrementalReview).
// Distinct from the agent-internal snapshot (action/utils/prSummary.ts) which
// has its own stable scaffold and is never shaped by user instructions — see
// selectMode.ts for the firewall.
export const PR_SUMMARY_FORMAT = `### Default format
Follow this structure exactly:
The body has at most three parts in this exact order:
<b>TL;DR</b> — 1-3 sentences on what the PR does and why. Focus on intent, not mechanics.
NOTE: use HTML bold <b>TL;DR</b>, NOT markdown bold **TL;DR**.
1. **Reviewed changes preamble** — one bolded inline lead-in describing what was reviewed in this run, a bullet list of the substantive changes, and an HTML comment carrying review metadata for downstream agents.
2. **Cross-cutting issue sections** (zero or more) — one \`### \` heading per concern, with a human-readable problem write-up and a collapsed \`<details>Technical details</details>\` block underneath.
3. **\`### ️ Nitpicks\`** at the very bottom (only if there are nits worth surfacing in the body) — a flat bullet list, no technical-details block.
### Key changes
Inline-vs-body split: concerns that anchor to a specific line go inline (use the \`comments\` parameter). Body \`### \` sections are reserved for concerns that **have no line to anchor to** — typically because the concern is about *absence* (something the diff should have done but didn't), *sequencing* (rollout / deletion / migration order), *design decisions only the human can make*, or *scope questions the diff implicitly raises but doesn't address*. A concern that anchors to a line but has broad implications still goes inline (use the technical-details block there to capture the implications — see Inline technical details below). If you found no non-anchorable concerns, the body has zero \`### \` issue sections — just the preamble + metadata.
- **Short human-readable title** — 1 sentence per change. Write a short prose phrase (title case or sentence case); when you name a file, type, or function, put that name in backticks (e.g. **Add \`TodoTracker\` for live checklists**). A reviewer should understand the full PR from this list alone.
## 1. Reviewed changes preamble
<sub><b>Summary</b> {file_count} files {commit_count} commits base: \`{base}\`\`{head}\`</sub>
NOTE: the metadata line goes AFTER the bullet list, not before it.
Open with a single bolded inline lead-in followed immediately by the bullet list (no \`### Key changes\` heading, no \`<b>TL;DR</b>\`):
Then for each key change, a ## section with a short descriptive title that reads like a documentation heading (e.g. ## Live todo checklist tracking).
\`\`\`
**Reviewed changes** — one sentence on what was reviewed in this run. For Review (initial), this is what the PR does and why. For IncrementalReview, this is what changed since the prior pullfrog review. Focus on intent, not mechanics.
<br/>
- **Short human-readable title** — 1 sentence per substantive change. Write a short prose phrase; when you name a file, type, or function, put that name in backticks (e.g. **Add \\\`TodoTracker\\\` for live checklists**). A reviewer should understand the full reviewed scope from this list alone — this IS the dispassionate "what was reviewed and what changed" overview, so cover the substantive changes, not just the loudest ones.
## Example readable section title
<!--
Pullfrog review metadata — for any agent (or human-with-agent) reading this
review. Incorporate the fields below into your understanding of the context
this review was made in. The findings below were written against
{head_sha_short}; if new commits have landed on {head_ref} since this review
was submitted, treat any specific bug, file, or line callout as POTENTIALLY
STALE — re-diff against {head_sha_short} (or trigger a fresh review) and
factor commits past {head_sha_short} into your understanding of the current
state before acting on findings.
> **Before:** [old behavior/state]<br/>**After:** [new behavior/state]
IMPORTANT: Before and After MUST be on a SINGLE blockquote line with an inline <br/> between them. Two separate \`>\` lines creates a double line break.
- Mode: Review (initial) or IncrementalReview (delta against prior pullfrog review)
- Files reviewed: {file_count}
- Commits reviewed: {commit_count}
- Base: {base_ref} ({base_sha_short})
- Head: {head_ref} ({head_sha_short})
- Reviewed commits:
- {sha_short} — {commit_subject}
- ...
- Prior pullfrog review: none or {prior_sha_short} ({prior_review_html_url})
- Submitted at: {iso_timestamp}
-->
\`\`\`
1-2 sentences of explanation. Break up text with tables, blockquotes, or lists — NEVER 3+ plain paragraphs in a row.
Pull every metadata field from the \`checkout_pr\` tool's response — file count, commit count, base/head ref + SHA, the commit list. For \`IncrementalReview\` runs, populate \`Prior pullfrog review\` with the prior review's commit_id (short SHA) and \`html_url\` from \`list_pull_request_reviews\`.
If a change warrants deeper explanation, use a blockquoted details/summary framed as a question:
> <details><summary>How does X work?</summary>
> Extended explanation here.
> </details>
## 2. Cross-cutting issue sections (zero or more)
End each section with a file links trail (3-4 key files max):
[\`file.ts\`](https://github.com/{owner}/{repo}/pull/{number}/files#diff-{sha256hex_of_filepath}) · ...
For each cross-cutting concern, one \`### \` section. Use this exact shape:
Single-feature PRs: skip the ## sections. Fold before/after and explanation into the header after key changes.
\`\`\`
### {emoji} {short, descriptive title — what's wrong, not what to do}
CRITICAL — GitHub markdown rendering rule:
GitHub's markdown parser requires a blank line between ALL block-level elements. This includes transitions between: HTML tags (<br/>, <sub>, <details>, <b>, etc.) and markdown syntax (headings, lists, blockquotes, paragraphs). Without a blank line, GitHub treats the following content as a continuation of the HTML block and renders markdown syntax as literal text. ALWAYS separate block-level elements with a blank line.
{Human-readable problem write-up. Describes the PROBLEM only — what's broken, what the symptom is, what the blast radius is. NO asks, NO suggested fixes, NO "the right thing to do is...". Asks and fixes live in the technical-details block below; the visible part is for the human to *understand* the problem, not to implement it.}
Rules:
- \`##\` titles and key-change bullet lead-ins are plain-language summaries; backtick only actual code tokens (files, types, functions) where they appear in the title
- ALL variable names, identifiers, and file names in body text must be in backticks
- ALL file references MUST link to the PR Files Changed view. Use the \`diff-<hex>\` anchor precomputed next to each filename in the \`checkout_pr\` TOC — do NOT run \`sha256sum\` or any other shell command to compute anchors. NEVER fabricate hex strings. If a file is not in the TOC, omit the \`#diff-\` anchor rather than guessing.
- Add <br/> before each ## heading for visual spacing. Do NOT use horizontal rules (---)
- Do NOT include raw diff stats like '+123 / -45' or line counts
- Do NOT include code blocks or repeat diff contents
- Do NOT include a changelog section — the key changes list serves this purpose
- Focus on *intent*, not *what* — the diff already shows what changed
- Get the file count and commit count from the checkout_pr metadata, not by counting manually`;
<details><summary>Technical details</summary>
\\\`\\\`\\\`\\\`markdown
# {title repeated}
## Affected sites
- {file path:line} — {what's wrong there}
- ...
## Required outcome
- {what the fix needs to achieve, not how to achieve it}
- ...
## Suggested approach (optional)
{When the fix shape is non-obvious, sketch one or more reasonable directions. Skip when the outcome alone makes the fix obvious.}
## Open questions for the human (optional)
- {Any decision an implementing agent shouldn't make unilaterally — pricing thresholds, breaking-change policy, naming, scope of follow-up.}
\\\`\\\`\\\`\\\`
</details>
\`\`\`
Concrete example of the visible part of a non-anchored section (technical-details block unchanged from the template above):
\`\`\`
### ️ Legacy \`opencode.ts\` has no documented deletion plan
The v2 harness lands alongside the v1 file and imports one helper from it. Worth a follow-up issue or a TODO so the next maintainer doesn't have to re-derive the cleanup plan.
\`\`\`
The example's value is its *shape*: a finding about absence (no deletion plan), not a line-anchored bug. Body sections live or die on whether the concern genuinely doesn't fit on a line.
**Heading severity emoji** — every \`### \` heading carries one:
- 🚨 critical — blocks merge (data loss, security, broken core flow)
- ⚠️ important — must address before merging (regression, missing validation, incorrect behavior)
- ️ informational — surfaced for awareness; mergeable as-is
**Visible problem write-up rules:**
- **No asks, no suggested fixes** in the visible part. The visible portion describes the problem; the technical-details block describes the fix shape and any open questions. The exception: a fix so self-evident that NOT stating it would be weird (e.g. "the typo is missing an 'r'") — in that case, fold it into the problem statement and skip the suggested-approach block in technical details too.
- **Never two successive plain paragraphs.** Every transition between block-level elements must alternate prose with structure: paragraph → bullet list → paragraph; paragraph → code fence → bullet list; paragraph → table → paragraph. Two consecutive paragraphs in a row create a wall of text that's impossible to digest. If you catch yourself writing one, find a way to split it: pull a list out of it, drop a 2-3 line code fence between them, or merge them into a single tighter paragraph.
- **Per-paragraph budget:** ~3 sentences max. Past that, you're explaining where you should be structuring.
- **Identifier discipline still applies** in the visible part. Lead with behavior in plain English; name an identifier only when it's the subject of the concern or a public surface a reader would recognize. The technical-details block is where dense identifier references belong.
**Technical-details block rules:**
- Wrapped in a 4-backtick markdown fence (\`\\\`\\\`\\\`\\\`markdown ... \\\`\\\`\\\`\\\`\`) so it's visually distinct, one-click copyable, and can contain its own 3-backtick code fences without escape gymnastics. The contents are agent-readable — a fix-agent will pull the body down and use this block as the brief.
- File paths and \`file:line\` refs are encouraged (and necessary) — the next agent uses these to navigate. Identifier density is fine here.
- Slightly more verbose than the absolute minimum is OK when it materially helps the next agent: a small code snippet showing the symptom, a short table of mismatched key/column pairs, a one-paragraph "why CI doesn't catch it" note. Skip massive regression-test scaffolding or full route rewrites — the implementing agent writes those.
- Use the four standard sections (\`Affected sites\`, \`Required outcome\`, optional \`Suggested approach\`, optional \`Open questions for the human\`). Skip the optional sections when they wouldn't add anything.
## Inline technical details
Inline comments are short (~2-3 sentences) by default. When an inline finding has broader implications worth recording for a fix-agent — e.g. a localized bug whose proper fix requires touching several files, or where the right fix depends on a design decision the human needs to make — append a collapsed \`<details><summary>Technical details</summary>\` block to the inline comment's body. Same shape as the body-section technical-details block (4-backtick fenced markdown, \`## Affected sites\` / \`## Required outcome\` / optional \`## Suggested approach\` / optional \`## Open questions for the human\`).
GitHub renders the same markdown parser in inline comments as in the review body, so the collapsed-details affordance works the same way. The visible part of the inline comment stays scannable; the depth is one click away for any agent that needs it.
## 3. \`### ️ Nitpicks\` (optional, last section)
Only when there are nits that for some reason can't be inlined. Filepaths in nit text are fine — these are simple enough that a human or agent reads once and acts. No technical-details block.
\`\`\`
### ️ Nitpicks
- {nit, with file path inline if useful, ≤ ~200 chars}
- ...
\`\`\`
## Inline comment shape
Inline comments use the same severity framing as body \`### \` sections, scaled down for line-anchored use:
- **Lead with a 1-2 sentence problem statement.** The reader is looking at the line in question, so don't restate what the line says — describe what's wrong with it. Optionally prefix the visible line with a severity emoji (🚨 / ⚠️ / ️) when severity isn't obvious from context.
- **Optional \`<details><summary>Technical details</summary>...</details>\` collapsible** for findings whose technical context (longer file:line references, related-code snippets, suggested approach, regression-risk notes) would overwhelm the human-readable lead-in. Same agent-readable purpose, same 4-backtick fence shape, and same 4-section structure as the body's technical-details block — see *Inline technical details* above. Encouraged whenever the depth helps a downstream fix-agent; don't force one when the inline lead-in already says everything.
- **Visible portion ≤ 2-3 sentences.** If you find yourself writing more, that's the cue to split the depth into the \`Technical details\` collapsible.
## Body-wide rules
- **Inline-vs-body discipline (repeated for emphasis):** anything that anchors to a specific line goes inline (with a \`<details>Technical details</details>\` block when the implications are broad). The body is for non-anchorable concerns only — absence, sequencing, design decisions, scope questions, architectural risk.
- **No \`### Issues found\` heading** above the issue sections — each \`### \` heading IS the issue.
- **Severity emoji on every \`### \` heading** (🚨 / ⚠️ / ️). No emoji on the preamble lead-in or anywhere else.
- **GitHub block-level rendering**: GitHub's markdown parser requires a blank line between ALL block-level elements (HTML tags like \`<br/>\`, \`<sub>\`, \`<details>\`, \`<b>\` and markdown syntax like headings, lists, blockquotes, code fences, paragraphs). Without a blank line, GitHub treats following content as a continuation of the HTML block and renders markdown syntax as literal text. ALWAYS separate block-level elements with a blank line.
- **Backtick-wrap** every variable, identifier, or file name when you mention one (in either visible or technical-details portions).
- **Don't repeat diff content**, don't include raw \`+123 / -45\` stats, don't include a changelog section, don't use horizontal rules (\`---\`).
- **Pull file/commit counts from \`checkout_pr\` metadata** — never count manually.
- **Legacy headings REMOVED.** Do not use \`### Key changes\`, \`### Issues found\`, \`<b>TL;DR</b>\`, or \`<sub><b>Summary</b>\`. The new structure subsumes them.`;
export function computeModes(agentId: AgentId): Mode[] {
const t = (toolName: string) => formatMcpToolRef(agentId, toolName);
@@ -107,7 +201,7 @@ export function computeModes(agentId: AgentId): Mode[] {
Otherwise delegate the \`${REVIEWER_AGENT_NAME}\` subagent to review your diff with fresh eyes against YOUR TASK. The subagent's baked-in system prompt enforces a non-mutative + non-recursive contract: read-only file/search/web tools and read-only MCP queries only; no writes, shell side effects, state-changing MCP calls, or nested subagent dispatch. Enforcement is prose-only — restate the constraint in your dispatch instructions and do not relax it.
Provide the subagent with YOUR TASK, the output of \`git diff\`, and a tight summary (not raw output) of any lint/typecheck/test failures you fixed during build — what broke, root cause, the fix — so it can check that fixes addressed root causes rather than suppressed symptoms; say "no build-phase failures" if the build path was clean. Instruct it to flag bugs, logic errors, missing edge cases, gaps between request and diff, and unintended changes.
Provide the subagent with YOUR TASK, the output of \`git diff origin/<base-branch>\` (single-rev form, no \`HEAD\` — this compares the working tree against the remote base and captures committed + staged + unstaged work; \`main...HEAD\` and \`--cached\` both miss the uncommitted edits Build self-review runs on, since self-review happens BEFORE the commit), and a tight summary (not raw output) of any lint/typecheck/test failures you fixed during build — what broke, root cause, the fix — so it can check that fixes addressed root causes rather than suppressed symptoms; say "no build-phase failures" if the build path was clean. Instruct it to flag bugs, logic errors, missing edge cases, gaps between request and diff, and unintended changes.
Delegation + research discipline (distilled from \`/anneal\` canonical — these are codified learnings from many review rounds, not theoretical best practices):
- Do NOT summarize what you implemented — that biases the subagent toward validating the shape of your solution rather than questioning it.
@@ -116,7 +210,7 @@ export function computeModes(agentId: AgentId): Mode[] {
- Do NOT defect-hunt the diff yourself in parallel with the subagent. Your role is dispatch + evaluation; doing the review yourself reintroduces the implementation bias the subagent is meant to mitigate.
- For diffs that rely on third-party API contracts, SDK semantics, framework directives, or DB engine specifics, instruct the subagent to verify load-bearing claims via web search and quote source URLs rather than trust training data — this is the single most common review-quality failure mode.
Review the findings, address valid points, and discard nitpicks or false positives. The reviewer is fallible — it biases toward *recommending additions* (defensive checks for impossible cases, extra logging, new abstractions used once, comments restating code, tests asserting tautologies, "just-in-case" guards). For each finding, ask: would applying it leave the code more sound, correct, AND elegant? Two-out-of-three is usually a signal to look harder for a fix that gets all three before settling for one that trades elegance for correctness. Reject bloat-shaped findings without applying them, and after applying the rest re-read your diff and be discerning about what *you just changed*: if any fix turned out to be bloat in context, revert it. The goal is code that is sound and correct *while remaining elegant*; the smallest diff that fixes the real defect almost always wins. Then verify only intended changes are present, no debug artifacts or commented-out code remain, no unrelated files were modified. Commit locally via shell (\`git add . && git commit -m "..."\`).
Be **discerning** about what comes back. The reviewer is an AI subagent and is fallible — treat every finding as a hypothesis, not a directive, and **verify each one yourself** against the diff and the code before deciding whether to apply. You are searching for a solution that is **complete, minimal, and elegant** — you may need to think hard to find it. Do not over-engineer, do not be over-defensive, **do not write AI slop**. Reviewers bias toward *recommending additions*, and that bias has a recognizable slop texture: defensive checks for cases that cannot happen, extra logging, new abstractions used once, comments restating code, tests asserting tautologies, "just-in-case" guards, error handlers for cases the type system already rules out. Reject those. For each surviving finding, ask: would applying it leave the code more sound, correct, AND elegant? Two-out-of-three means look harder for a fix that gets all three before settling. After applying the fixes you accept, re-read your diff and be discerning about what *you just changed*: if any fix turned out to be bloat in context, revert it. Then verify only intended changes are present, no debug artifacts or commented-out code remain, no unrelated files were modified. Commit locally via shell (\`git add . && git commit -m "..."\`).
6. **finalize**:
- confirm a clean working tree, then push via \`${t("push_branch")}\` (see *SYSTEM* Git rules if this fails — prepush errors are usually the repo's tests/lint, not infra timeouts)
@@ -141,7 +235,8 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
4. For each comment:
- understand the feedback
- evaluate whether applying it would leave the code more **sound, correct, AND elegant**. reviewers are fallible and bias toward *recommending additions* (defensive checks for impossible cases, extra abstractions, comments restating obvious code, tests asserting tautologies, "just-in-case" guards). if a request would add bloat — ceremony without commensurate correctness benefit — push back in your reply rather than mechanically applying it. two-out-of-three is usually a signal to look harder for a fix that gets all three before settling.
- **verify the finding yourself** against the actual code before deciding whether to apply — every comment (human or agent) is a hypothesis, not a directive. agent reviewers especially are fallible.
- you are searching for a solution that is **complete, minimal, and elegant** — you may need to think hard to find it. do not over-engineer, do not be over-defensive, **do not write AI slop**. reviewers bias toward *recommending additions*, and that bias has a recognizable slop texture: defensive checks for impossible cases, extra abstractions used once, comments restating obvious code, tests asserting tautologies, "just-in-case" guards, error handlers for cases the type system already rules out. reject those. evaluate whether applying the finding would leave the code more **sound, correct, AND elegant**; two-out-of-three is a signal to look harder for a fix that gets all three. if a request would add bloat — ceremony without commensurate correctness benefit — push back in your reply rather than mechanically applying it.
- if the request stands, make the code change using your native tools; otherwise reply explaining why
- record what was done (or why nothing was done)
@@ -149,24 +244,35 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
- test changes, then review the diff before committing — verify only intended changes are present, no debug artifacts remain, no fix turned out to be bloat in context (revert any that did), and the changes are clean enough that a senior engineer would approve without hesitation
- commit locally via shell (\`git add . && git commit -m "..."\`)
6. Finalize:
6. Finalize. Reply + resolve are paired write actions: do BOTH or NEITHER for each thread.
- confirm a clean working tree, then push via \`${t("push_branch")}\` (same push/prepush guidance as Build mode in *SYSTEM*)
- reply to each comment **exactly once** using \`${t("reply_to_review_comment")}\` — do not re-emit the same call (the runtime dedupes identical bodies and the second call is wasted)
- resolve addressed threads via \`${t("resolve_review_thread")}\`
- call \`${t("report_progress")}\` with a brief summary (or the exact push error if push failed)`,
- **if push fails**, call \`${t("report_progress")}\` with the exact error and STOP — do NOT reply or resolve any thread until the fix is live on the remote. Resolving a thread without the fix landing misleads the reviewer.
- **on push success**, for each thread you acted on:
- reply ONCE via \`${t("reply_to_review_comment")}\`. The \`comment_id\` parameter takes the root comment's numeric \`id=\` (from the first \`comment author=...\` tag in the \`${t("get_review_comments")}\` output) — NOT the \`thread=\` value; that's a separate GraphQL ID used by resolve. The runtime dedupes identical bodies within a session.
- **immediately** call \`${t("resolve_review_thread")}\` with that thread's \`thread=\` value as \`thread_id\`. Resolve every thread where you (a) made the requested code change in full — partial fixes leave the thread open — OR (b) replied with a substantive answer the user explicitly asked for. Do NOT resolve threads where you pushed back on the request and the disagreement is unresolved; leave those open for the human to mediate.
- call \`${t("report_progress")}\` with a brief summary`,
},
// Review and IncrementalReview use the multi-lens orchestrator pattern
// (canonical source: .claude/commands/anneal.md). The orchestrator does
// triage → parallel read-only subagent fan-out → aggregate → draft comments
// → submit. For someone else's PR, parallel lenses (correctness, security,
// research-validated claims, user-journey, etc.) provide breadth across
// angles that a single subagent can't carry coherently. Build mode keeps
// a single fresh-eyes subagent (different problem shape — orchestrator
// wrote the code and bias-mitigation comes from delegating to one
// subagent that doesn't share the implementation context).
// Deliberate omission vs canonical /anneal: severity categorization in the
// final message (the review body has its own CAUTION/IMPORTANT framing
// instead of a severity table).
// Review and IncrementalReview use a 0-or-2+ lens pattern. The default is
// 0 lenses (orchestrator handles the review solo). Multi-lens (2+
// reviewfrog subagents in parallel) only fires for substantive PRs or
// high-stakes-subsystem touches — and when it fires, ALL lenses must
// dispatch in a single assistant turn or the parallelism win disappears.
// We never dispatch exactly one lens: a single lens is just a worse,
// slower version of doing the work yourself.
//
// Build mode self-review is a different problem shape: the orchestrator
// wrote the code, so bias-mitigation comes from delegating to one
// fresh-eyes subagent that doesn't share the implementation context. A
// single subagent there is appropriate; the 0-or-2+ rule applies only to
// the Review/IncrementalReview lens fan-out where independence between
// perspectives is what's being purchased.
//
// Severity categorization is split across two surfaces: the opening
// callout (CAUTION/IMPORTANT/️/✅) sets the review's overall tier, and
// per-bullet emoji prefixes (🚨/⚠️/️ in PR_SUMMARY_FORMAT) tag
// individual points inside summary sections — scoping severity to the
// specific bullet rather than the whole section keeps a section that
// mixes a 🚨 and an ️ from being mislabeled by either of them.
{
name: "Review",
description:
@@ -177,9 +283,9 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
2. **checkout**: call \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. read the diff TOC end-to-end and treat its file line ranges as your coverage checklist.
3. **triage**: orient yourself on the PR — identify *what kind of thing this is* (domain it touches, seams it crosses, external contracts it depends on, user-facing surfaces it changes). orientation only — defer specific defect-hunting to the subagents; pre-reviewing biases the lenses you pick. use \`${t("get_pull_request")}\` and other read-only GitHub tools for additional context if needed.
3. **triage**: orient yourself on the PR — identify *what kind of thing this is* (domain it touches, seams it crosses, external contracts it depends on, user-facing surfaces it changes). pull as much context as you need to render a confident, well-grounded review: read related files, grep for callers of changed symbols, check tests that exercise the touched paths, fetch related GitHub state. **you are the synthesizer** — never delegate understanding to subagents.
if the PR is **genuinely trivial**, skip steps 45 entirely and submit a \`No new issues found.\` review per step 6. there's no value in dispatching even one lens for a typo.
if the PR is **genuinely trivial**, skip the fan-out entirely and submit a \`No new issues found.\` review per step 7.
"Genuinely trivial" (skip):
- single-word doc typo, whitespace/format-only, comment-only across any number of files
@@ -198,23 +304,25 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
- any "typo fix" in user-facing copy that changes meaning ("approved" → "denied")
- mixed diffs where a semantic 1-liner is buried in whitespace/formatting changes
When unsure, treat as non-trivial. The cost of one extra subagent is cents; the cost of a missed billing/auth/data bug is much more.
4. **lens decision — 0 or 2+, NEVER 1**.
otherwise pick lenses by where the PR concentrates risk — **there's no fixed count**. lens count is judgment, not a formula. concrete shapes to anchor against:
The default is **0 lenses**: handle the review yourself end-to-end. Most PRs land here.
- **1 lens** — pure refactor / mechanical rename across many files (impact); new test file with no source change (test-integrity); small isolated bug fix (correctness); doc-only PR with non-trivial technical content (research-validated or holistic)
- **23 lenses (most PRs land here)** — new CRUD endpoint (correctness + security + test-integrity); new UI flow (user-journey + correctness); a single bug fix in a non-critical subsystem (correctness + test-integrity); design doc covering one domain (research-validated + correctness or holistic)
- **45 lenses (high-stakes subsystem touches)** — any billing/payments change (billing-subsystem + correctness + security + operational-readiness); new auth flow (auth-subsystem + correctness + security + test-integrity); schema migration (schema-migration-subsystem + correctness + operational-readiness + impact); cross-subsystem PR that touches billing AND auth AND schema (one subsystem lens per domain + correctness)
- **6+ lenses** — almost always a smell; you're either covering overlapping ground or this PR should have been split. push back via the review body rather than expanding lens count.
Dispatch **2+ \`${REVIEWER_AGENT_NAME}\` lenses in parallel** ONLY when ALL of the following are true:
- the PR is substantive (>5 files changed AND >200 net lines), OR touches a high-stakes subsystem (auth, billing, payments, schema migration, webhooks, secrets, RBAC, multi-tenant isolation, cron/scheduling)
- you can name 2+ distinct concrete failure modes that warrant independent lenses (one lens per failure mode; orthogonal, not overlapping)
- parallel-orchestrated independent perspectives meaningfully outperform what you'd find solo
lenses come in two flavors, and you can mix them:
**NEVER dispatch exactly one lens.** A single lens is just a more expensive version of doing the work yourself with a worse model — it adds wall time and a context-handoff for no orthogonality benefit. Either you have at least two genuinely independent failure-mode hypotheses (dispatch all in one turn), or you don't (do the review yourself).
When you do go multi-lens, lens framings come in two flavors:
- **themed lenses** — a perspective applied across the whole diff (correctness, security, user-journey, performance, etc.).
- **subsystem lenses** — a domain-scoped frame for high-stakes subsystems the PR touches (e.g. "the auth lens", "the billing lens", "the schema-migration lens"). a subsystem lens is "review the PR specifically for what could go wrong in this subsystem" and naturally combines theme + scope. **for high-stakes domains, lead with the subsystem lens rather than the generic themed equivalent** — "billing-subsystem" outperforms "correctness on billing code" because the framing primes the subagent to remember domain-specific failure modes (double-charges, refund races, currency rounding, dispute flows) the generic lens misses.
- **subsystem lenses** — a domain-scoped frame for high-stakes subsystems the PR touches (e.g. "the auth lens", "the billing lens", "the schema-migration lens"). **for high-stakes domains, lead with the subsystem lens rather than the generic themed equivalent** — "billing-subsystem" outperforms "correctness on billing code" because the framing primes the subagent to remember domain-specific failure modes (double-charges, refund races, currency rounding, dispute flows) the generic lens misses.
starter menu (combine, omit, or invent your own):
- **correctness & invariants** — bugs, races, error handling, edge cases, state-machine boundaries
- **impact** — when the PR removes features, deletes exports, renames identifiers, or changes architectural patterns: stale references in code, tests, docs (\`docs/\`, \`wiki/\`), comments, configs, UI
- **research-validated assumptions** — third-party API contracts, SDK semantics, framework directives, version-gated behavior. the subagent must verify load-bearing claims via web search and quote source URLs.
- **impact** — stale references in code/tests/docs/configs/UI after rename/remove
- **research-validated assumptions** — third-party API contracts, SDK semantics, framework directives, version-gated behavior. **only pick when the PR's correctness depends on the contract behaving a specific way** — not when the API is merely used. The bar is "if the third-party contract differs from what the diff assumes, the PR is incorrect." When dispatched, the subagent must verify load-bearing claims via web search and quote source URLs.
- **security** — new endpoints, authZ, input validation, secrets handling, replay/CSRF/injection, cross-tenant isolation
- **user-journey** — UX-touching flows: walk through happy path and failure modes as a user
- **operational readiness** — observability, alerting, migrations (forward + rollback), feature flags, on-call burden
@@ -224,37 +332,49 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
- **holistic** — does the PR make sense as a whole? symmetric flows (delete for every create, rollback for every migration)?
- **subsystem lenses** (invent as the PR demands) — auth, billing, payments, schema migration, webhooks, secrets, RBAC, multi-tenant isolation, cron/scheduling, etc.
4. **fan out**: dispatch one \`${REVIEWER_AGENT_NAME}\` subagent per lens — its baked-in system prompt enforces the non-mutative + non-recursive contract (read-only file/search/web tools and read-only MCP queries; no writes, shell side effects, state-changing MCP calls, or nested subagent dispatch). when picking 2+ lenses, dispatch them in a **single assistant turn with multiple parallel subagent calls**; issuing one and awaiting reply before the next collapses the fan-out into a serial review. if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body — do not skip step 4 entirely on a single subagent failure. each subagent gets:
The only subagent type is \`${REVIEWER_AGENT_NAME}\` — used for lens judgment work ("is this safe / correct / well-tested?"), runs on a mid-tier model.
5. **fan out (only if step 4 said 2+ lenses)**: dispatch every \`${REVIEWER_AGENT_NAME}\` subagent for this run **IN A SINGLE ASSISTANT TURN, AS MULTIPLE PARALLEL TASK TOOL_USE BLOCKS IN ONE MESSAGE.**
⚠️ CRITICAL — PARALLELISM IS THE ONLY REASON LENSES EXIST. ⚠️
The default tool-call behavior of Claude Code (and most agent runtimes) is **serial dispatch**: emit one Task call, await result, emit next, await, etc. This collapses your fan-out into a sequential review where each lens adds N × (orchestrator-think-time + lens-execution-time) to wall time. **YOU MUST OVERRIDE THIS DEFAULT.** Emit ALL of your Task tool_use blocks in the SAME assistant message, BEFORE you read ANY result from ANY of them. If you find yourself emitting one Task call, then thinking about the result, then emitting another — STOP and re-issue them all together. The whole point of going multi-lens is the wall-clock speedup from parallel execution; serial dispatch defeats it entirely.
✅ Right pattern: one assistant turn with N Task tool_use blocks → wait → N results arrive together → aggregate.
❌ Wrong pattern: turn 1 = Task(lens A) → turn 2 (after A's result) = Task(lens B) → turn 3 (after B's result) = Task(lens C). This is the failure mode. Do not do this.
You can also include your own \`read\` / \`grep\` / \`webfetch\` calls in the SAME turn as the parallel \`${REVIEWER_AGENT_NAME}\` dispatches — concurrent context-pulling on the orchestrator side runs in parallel with the lens fan-out and costs zero extra wall time.
if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body — do not skip the fan-out entirely on a single subagent failure. each subagent gets:
- the diff path / target — reading the diff and the codebase is its job
- **only one lens** — never a multi-section "review for X, Y, and Z" prompt
- **a Task \`description\` set to the lens name** (e.g. \`"security"\`, \`"correctness"\`, \`"billing-subsystem"\`) — the harness reads this field to label the subagent's log lines so parallel runs can be told apart in CI output. without it, every subagent shows up as \`subagent#N\`.
- the read-only contract restated in your dispatch instructions so the rule is present twice (the subagent's system prompt also enforces it). The test: would this call still be a no-op if reverted? If not (PR comments, branch pushes, issue updates, set_output, label changes, dependency installs, etc.), don't make it.
- if the lens touches external contracts, instruct the subagent to verify load-bearing claims via web search rather than trust training data, and to quote source URLs in its reasoning. action runs are non-interactive — there's no human in the loop to catch "I'm pretty sure Stripe does X."
- ask the subagent to report findings with file paths and NEW line numbers from the diff so you can anchor inline comments without re-reading the entire diff.
delegation discipline:
- do NOT lens-review the diff yourself in parallel with the subagents (your job is dispatch + comment-drafting; doing the lens work yourself reintroduces the bias the fan-out avoids)
- do NOT summarize the PR for them (biases toward a validation frame)
- do NOT hand them a curated reading list (let them discover scope)
- do NOT pre-shape their output with a finding schema
- do NOT mention the other lenses (independence is the point — overlapping findings are a strong signal)
5. **aggregate & draft**: merge findings; de-dup overlaps (two lenses catching the same issue = higher-confidence signal); trace each finding yourself before accepting it. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the PR (heuristic: if the finding's root cause lives in lines this PR added or modified, it's in scope; otherwise drop unless the PR plausibly introduced or amplified the regression), and anything not actionable. also drop **bloat-shaped findings** — proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or worse, degrades elegance to nominally improve correctness) makes the codebase worse, not better.
6. **aggregate & draft**: when the fan-out lands, merge findings; de-dup overlaps (two lenses catching the same issue = higher-confidence signal); trace each finding yourself before accepting it. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the PR (heuristic: if the finding's root cause lives in lines this PR added or modified, it's in scope; otherwise drop unless the PR plausibly introduced or amplified the regression), and anything not actionable. also drop **bloat-shaped findings** — proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or worse, degrades elegance to nominally improve correctness) makes the codebase worse, not better.
for surviving findings, draft inline comments with NEW line numbers from the diff. every comment must be actionable, 2-3 sentences max. use GitHub permalink format for code references. for impact-analysis findings (stale references after rename/remove), report them in the review body ordered by severity (runtime breakage > incorrect docs > stale comments) rather than as inline comments unless they're anchored to a specific line.
**Hunt for non-anchored concerns before drafting.** After collecting your anchored findings, deliberately scan for concerns that have no specific line to point at — typically: deletion / cleanup plans for code the diff replaces or shadows; rollout sequencing (what happens to in-flight state during deploy / revert?); coverage gaps the diff implies but doesn't add; scope questions that only the human can answer (e.g. is the legacy path going away or is this a long-term dual track?); architectural risks the diff opens up that aren't a single-line bug. On substantial PRs (migrations, refactors, multi-file rewrites, version bumps that change runtime semantics), at least one such concern almost always exists; if you can't think of any, your bar is probably too high.
6. **submit**: ALWAYS submit exactly one review via \`${t("create_pull_request_review")}\`. Do NOT call \`report_progress\` — the review is the final record and the progress comment will be cleaned up automatically.
for surviving findings, draft inline comments with NEW line numbers from the diff — attach a \`<details>Technical details</details>\` block to any inline comment whose fix is non-trivial or has cross-file implications (see Inline technical details in the format below). every comment must be actionable, 2-3 sentences max in the visible part. use GitHub permalink format for code references. for impact-analysis findings (stale references after rename/remove), report them in the review body ordered by severity (runtime breakage > incorrect docs > stale comments) rather than as inline comments unless they're anchored to a specific line.
7. **submit**: ALWAYS submit exactly one review via \`${t("create_pull_request_review")}\`. Do NOT call \`report_progress\` — the review is the final record and the progress comment will be cleaned up automatically.
note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session.
The review body is structured as: \`[optional alert blockquote]\`\`[PR summary using the default format below]\`. Inline comments are passed via the \`comments\` parameter, not in the body.
GitHub alert blockquotes render at four visual intensities — the callout is what the author sees first, so pick the one that matches what you want them to do:
The opening callout is what the author sees first pick the one that matches what you want them to do. Five tiers, from loudest to friendliest:
- \`[!CAUTION]\` — large red banner. Reads as "this will break something."
- \`[!IMPORTANT]\` — large purple banner. Reads as "you need to look at this before merging."
- \`[!NOTE]\`small blue inline callout. Reads as "FYI, here's something worth noting."
- no callout — plain text. Reads as routine review output.
- \`> ️ ...\`informational blockquote. Reads as "minor suggestions, nothing blocking."
- \`> ✅ ...\` — green friendly blockquote. Reads as "no concerns, mergeable."
Two reinforcing levers: callout intensity (above) and \`approved\` (which gates the footer Fix-button affordance — Fix renders on every non-approving review, so \`approved: true\` suppresses it). Wrapping mergeable feedback in \`[!IMPORTANT]\` trains users to click Fix on reviews that don't need fixing. Pick the tier the author's actual next action justifies.
@@ -263,25 +383,25 @@ For simple, well-defined tasks, skip the plan phase and go straight to build.`,
- **must-address non-critical findings** (real consequences if shipped — incorrect behavior in non-critical paths, missing validation on user input, regressions the author should fix before merge):
\`approved: false\`. Body opens with \`> [!IMPORTANT]\\n> ...\`, followed by the PR summary. Reserve this tier for findings with concrete fallout — do NOT use \`[!IMPORTANT]\` for nits, style preferences, or "consider also" suggestions. Include all inline comments via \`comments\`.
- **minor suggestions only** (single-line nits, doc/comment polish, defer-able observations, "rough edges"):
\`approved: false\`. NO alert blockquote. Body opens directly with the PR summary. Include all inline comments via \`comments\`.
\`approved: false\`. Body opens with \`> ️ No critical issues — minor suggestions inline.\\n\\n\` followed by the PR summary. Include all inline comments via \`comments\`. Vary the wording after the emoji to fit the review (e.g. "Minor suggestions only.", "Two rough edges worth a look."), but always keep the ️ prefix and keep it short.
- **informational observations** (mergeable as-is, nothing actionable — e.g. prior feedback addressed cleanly, surfacing a minor stale doc reference, calling out something noteworthy without recommending a change):
\`approved: true\`. Body opens with \`> [!NOTE]\\n> ...\`, followed by the PR summary. Do NOT include inline \`comments\`\`[!NOTE]\` signals "no action needed", which contradicts an actionable anchor; if a point is concrete enough to anchor to a line, downgrade the whole review to "minor suggestions only" (\`approved: false\`) instead.
\`approved: true\`. Body opens with \`> ✅ No new issues found.\\n\\n\` followed by the PR summary. Do NOT include inline \`comments\`the ✅ signals "no action needed", which contradicts an actionable anchor; if a point is concrete enough to anchor to a line, downgrade the whole review to "minor suggestions only" (\`approved: false\`) instead.
- **no actionable issues**:
\`approved: true\`. Body opens with \`No new issues found.\` followed by the PR summary.
\`approved: true\`. Body opens with \`> ✅ No new issues found.\\n\\n\` followed by the PR summary.
${PR_SUMMARY_FORMAT}`,
},
// IncrementalReview shares Review's multi-lens orchestrator pattern but
// scopes the target to the incremental diff. The "issues must be NEW
// since the last Pullfrog review" filter lives at aggregation time
// (step 6), NOT in the subagent prompt — pushing the filter into
// subagents matches the canonical anneal anti-pattern of "list known
// pre-existing failures — don't flag these" and suppresses signal on
// regressions the new commits amplified. The review body is just
// "Reviewed changes" — a separate "Prior review feedback" checklist
// would duplicate the rolling PR summary snapshot's record of what
// earlier runs already addressed and add noise to the user-facing
// body. Same severity-table omission as Review.
// IncrementalReview shares Review's 0-or-2+ lens pattern AND its body
// format (PR_SUMMARY_FORMAT), scoped to the incremental delta against the
// prior pullfrog review. The "issues must be NEW since the last Pullfrog
// review" filter lives at aggregation time (step 8), NOT in the subagent
// prompt — pushing the filter into subagents matches the canonical anneal
// anti-pattern of "list known pre-existing failures — don't flag these"
// and suppresses signal on regressions the new commits amplified. A
// separate "Prior review feedback" checklist would duplicate the rolling
// PR summary snapshot's record of what earlier runs already addressed and
// add noise to the user-facing body. Same opening-callout + per-bullet
// emoji severity split as Review.
{
name: "IncrementalReview",
description:
@@ -294,49 +414,82 @@ ${PR_SUMMARY_FORMAT}`,
3. **incremental scope**: if \`incrementalDiffPath\` is present, read it to see what changed since the last review. this is a range-diff that isolates the net changes, filtering out base branch noise. if not present, fall back to reviewing the full PR diff and determine what changed since Pullfrog's most recent review.
4. **prior feedback**: fetch previous reviews via \`${t("list_pull_request_reviews")}\`. for the most recent Pullfrog review, call \`${t("get_review_comments")}\` with the review ID to retrieve specific prior line-level feedback. you'll use this to filter your aggregation in step 6 — anything already flagged in a prior review and not changed by the new commits should not be re-raised. you do NOT need to render this in the review body; the rolling PR summary snapshot is the durable record of what's been addressed.
4. **prior feedback — read AND retire it**: fetch previous reviews via \`${t("list_pull_request_reviews")}\`, then call \`${t("get_review_comments")}\` on each prior Pullfrog review. Each thread renders as a section whose first line is a fenced tag \`comment author=<login> id=<fullDatabaseId> review=<reviewId> thread=<graphqlId>\`; section headers carry \`[RESOLVED]\` / \`[OUTDATED]\` when relevant. For every **open, Pullfrog-originated** thread, decide and act:
5. **triage & fan out**: orient on the *incremental* changes — domain, seams, external contracts, user-facing surfaces.
- **Pullfrog-originated** means the FIRST \`comment author=...\` tag in the section is \`author=pullfrog[bot]\`. The \`*\` marker on individual comments is unrelated — it flags whether a comment belongs to the queried review, not whether it is the thread root.
- **addressed?** read the file at the thread's anchor and judge whether the substantive concern is now resolved by the new commits. Lines being modified isn't enough: reformatting, renaming, or moving the same code elsewhere doesn't address a concern. If the comment raised multiple distinct concerns, ALL must be addressed. The \`[OUTDATED]\` tag means GitHub moved the anchor (line shift, force-push, rename) — it does NOT mean the concern was addressed; re-read the code at its new location before deciding.
- **if addressed**: call \`${t("reply_to_review_comment")}\` with the root tag's numeric \`id=\` as \`comment_id\` (NOT the \`thread=\` value — that's a separate GraphQL ID used only by resolve) and a one-line body (e.g. \`Addressed in <short-sha>.\`), then call \`${t("resolve_review_thread")}\` with the root tag's \`thread=\` value as \`thread_id\`. Do this BEFORE drafting the new review so the GitHub thread state aligns with the new review by the time it lands.
- **if uncertain or partially addressed**: leave open. False-positive resolutions erode trust faster than false negatives.
- **scope**: only retire Pullfrog-originated threads. Threads from human reviewers belong to those humans to resolve, even if the commit happened to address them.
if the incremental changes are **genuinely trivial**, skip the fan-out entirely and jump to step 8's non-substantive path (do NOT submit a review).
The remaining open threads feed step 8's dedup filter — anything already flagged and unchanged by the new commits should not be re-raised. The rolling PR summary snapshot is the durable record of retire activity; you don't need to surface it in the review body.
5. **triage**: orient on the *incremental* changes — domain, seams, external contracts, user-facing surfaces. pull as much context as you need to render a confident review: read related files, grep for callers of changed symbols, check tests that exercise the touched paths. **you are the synthesizer.**
if the incremental changes are **genuinely trivial**, skip the fan-out entirely and jump to step 10's non-substantive path (do NOT submit a review).
"Genuinely trivial" (skip): formatting/comment tweaks, import reordering, lockfile regen, mechanical rename of import paths, whitespace-only.
"Looks trivial but isn't" (do NOT skip — same anti-patterns as Review mode): 1-line changes to SQL/regex/auth/billing/permissions/signature-verification code; flipping feature-flag defaults or retry/timeout constants; money/tax/HTTP-method/redirect changes; tightening or loosening a comparison operator; mixed diffs with a semantic line buried in formatting.
When unsure, treat as non-trivial.
otherwise pick lenses by where the new commits concentrate risk — **there's no fixed count**, same calibration as Review mode (1 lens for pure refactor / isolated fix; 23 for typical features; 45 for high-stakes subsystem touches; 6+ is a smell). lens framing follows Review mode: themed lenses (correctness & invariants, impact when new commits remove/rename/deprecate things, research-validated assumptions, security, user-journey, operational readiness, integration & cross-cutting, test integrity, performance, holistic) and subsystem lenses (auth, billing, schema migration, etc.) — for high-stakes domains lead with the subsystem lens rather than the generic themed equivalent.
6. **lens decision — 0 or 2+, NEVER 1**.
dispatch one \`${REVIEWER_AGENT_NAME}\` subagent per lens — its baked-in system prompt enforces the non-mutative + non-recursive contract (read-only file/search/web tools and read-only MCP queries; no writes, shell side effects, state-changing MCP calls, or nested subagent dispatch). dispatch them in a **single assistant turn with multiple parallel subagent calls** (serial dispatch collapses the fan-out). if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body — do not skip step 5 entirely on a single subagent failure. each subagent gets:
- the diff scope (incremental diff path if available, full diff otherwise). do NOT tell them to skip pre-existing issues — that suppresses regressions the new commits amplified; the "issues must be NEW" filter lives at aggregation time (step 6), not in the subagent prompt
The default is **0 lenses**: handle the re-review yourself end-to-end. Most incremental reviews land here — especially thread-reply re-reviews where the user is asking "did you address X?" rather than "review the diff again."
Dispatch **2+ \`${REVIEWER_AGENT_NAME}\` lenses in parallel** ONLY when ALL of the following are true:
- the incremental changes are substantive (>5 files changed AND >200 net new lines), OR touch a high-stakes subsystem (auth, billing, payments, schema migration, webhooks, secrets, RBAC, multi-tenant isolation, cron/scheduling)
- you can name 2+ distinct concrete failure modes the new commits plausibly introduce that warrant independent lenses
- parallel-orchestrated independent perspectives meaningfully outperform what you'd find solo
**NEVER dispatch exactly one lens.** Single-lens dispatch adds wall time and cost for no orthogonality benefit. Either go multi-lens (≥2 in parallel) or do the re-review yourself.
Lens framing follows Review mode: themed lenses (correctness, security, etc.) and subsystem lenses (auth, billing, schema-migration, etc.) — for high-stakes domains lead with the subsystem lens.
7. **fan out (only if step 6 said 2+ lenses)**: dispatch every \`${REVIEWER_AGENT_NAME}\` subagent for this run **IN A SINGLE ASSISTANT TURN, AS MULTIPLE PARALLEL TASK TOOL_USE BLOCKS IN ONE MESSAGE.**
⚠️ CRITICAL — PARALLELISM IS THE ONLY REASON LENSES EXIST. ⚠️
Default tool-call behavior is **serial dispatch**: emit one Task call, await result, emit next, await, etc. This collapses your fan-out into a sequential review where each lens adds N × (orchestrator-think-time + lens-execution-time) to wall time. **YOU MUST OVERRIDE THIS DEFAULT.** Emit ALL of your Task tool_use blocks in the SAME assistant message, BEFORE you read ANY result from ANY of them.
✅ Right pattern: one assistant turn with N Task tool_use blocks → wait → N results arrive together → aggregate.
❌ Wrong pattern: turn 1 = Task(lens A) → turn 2 (after A's result) = Task(lens B). This is the failure mode.
You can also include your own \`read\` / \`grep\` / \`webfetch\` calls in the SAME turn as the parallel \`${REVIEWER_AGENT_NAME}\` dispatches.
if a subagent errors out, times out, or returns nothing usable, retry once with the same lens; if it still fails, proceed with partial coverage and note the missing lens in the review body. each subagent gets:
- the diff scope (incremental diff path if available, full diff otherwise). do NOT tell them to skip pre-existing issues — that suppresses regressions the new commits amplified; the "issues must be NEW" filter lives at aggregation time (step 8), not in the subagent prompt
- **only one lens** — never a multi-section "review for X, Y, and Z" prompt
- **a Task \`description\` set to the lens name** (e.g. \`"security"\`, \`"correctness"\`, \`"billing-subsystem"\`) — the harness reads this field to label the subagent's log lines so parallel runs can be told apart in CI output. without it, every subagent shows up as \`subagent#N\`.
- the read-only contract restated in your dispatch instructions so the rule is present twice (the subagent's system prompt also enforces it). The test: would this call still be a no-op if reverted? If not (PR comments, branch pushes, issue updates, set_output, label changes, dependency installs, etc.), don't make it.
- if the lens touches external contracts, instruct the subagent to verify load-bearing claims via web search and quote source URLs. action runs are non-interactive — there's no human to catch "I'm pretty sure Stripe does X."
- **a Task \`description\` set to the lens name** — the harness reads this field to label log lines so parallel runs can be told apart.
- if the lens touches external contracts, instruct the subagent to verify load-bearing claims via web search and quote source URLs.
- ask the subagent to report findings with file paths and NEW line numbers from the full PR diff so you can anchor inline comments.
delegation discipline:
- do NOT lens-review the diff yourself in parallel with the subagents
- do NOT summarize the changes for them (biases toward validation frame)
- do NOT hand them a curated reading list (let them discover scope)
- do NOT pre-shape their output with a finding schema
- do NOT mention the other lenses (independence is the point)
6. **aggregate, draft, self-critique**: merge findings; de-dup overlaps; trace each finding yourself. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the new commits, anything not actionable, and anything that re-states prior review feedback (heuristic: if the finding's root cause lives in lines the *new commits* added or modified, it's in scope; otherwise drop). also drop **bloat-shaped findings** — proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or degrades elegance to nominally improve correctness) makes the codebase worse, not better. To compute "lines the new commits added or modified": if \`incrementalDiffPath\` from step 2 is present, use it directly. Otherwise, take the prior Pullfrog review's \`commit_id\` (returned alongside each entry from \`${t("list_pull_request_reviews")}\` in step 4) and run \`git diff <prior-review-sha>..HEAD\` to isolate the lines added since that review. draft inline comments with NEW line numbers from the full PR diff — every comment must be actionable, 2-3 sentences max.
8. **aggregate, draft, self-critique**: merge findings (yours + any subagent output if you went multi-lens); de-dup overlaps; trace each finding yourself. drop praise, style preferences, speculative/unverified claims, findings about pre-existing code unrelated to the new commits, anything not actionable, and anything that re-states prior review feedback (heuristic: if the finding's root cause lives in lines the *new commits* added or modified, it's in scope; otherwise drop). also drop **bloat-shaped findings** — proposed fixes that would add defensive checks for cases that can't happen, abstractions used once, comments restating obvious code, tests asserting tautologies, or "just-in-case" guards. subagents are fallible and bias toward recommending changes; the bar for an actionable inline comment is sound + correct + elegant. recommending a change that improves only one of the three (or degrades elegance to nominally improve correctness) makes the codebase worse, not better. To compute "lines the new commits added or modified": if \`incrementalDiffPath\` from step 2 is present, use it directly. Otherwise, take the prior Pullfrog review's \`commit_id\` (returned alongside each entry from \`${t("list_pull_request_reviews")}\` in step 4) and run \`git diff <prior-review-sha>..HEAD\` to isolate the lines added since that review.
7. **build the review body** — a single "Reviewed changes" section: summarize at the logical-change level, not per-file. each bullet starts with a past-tense verb (e.g. \`- Extracted shared CLI runtime into a single module\`, \`- Renamed package to pullfrog\`). avoid file paths unless they add clarity. if the changes can be described in one sentence, use one sentence — no bullets needed. do NOT include a separate "Prior review feedback" checklist; that's tracked in the rolling PR summary snapshot for the next agent run, and surfacing it in the user-facing body is noise (changes that addressed prior feedback are already covered by the Reviewed-changes bullets). in some cases you may receive a complete diff for the whole pull request instead of an incremental one — when this happens, you will need to determine what changes have happened since Pullfrog's most recent review.
**Hunt for non-anchored concerns before drafting.** After collecting your anchored findings, deliberately scan for concerns that have no specific line to point at — typically: deletion / cleanup plans for code the new commits replace or shadow; rollout sequencing (what happens to in-flight state during deploy / revert?); coverage gaps the new commits imply but don't add; scope questions that only the human can answer (e.g. is the legacy path going away or is this a long-term dual track?); architectural risks the new commits open up that aren't a single-line bug. On substantial incremental diffs (migrations, refactors, multi-file rewrites, version bumps that change runtime semantics), at least one such concern almost always exists; if you can't think of any, your bar is probably too high.
8. Submit — every run must end with EXACTLY ONE of \`${t("create_pull_request_review")}\` (substantive review) or \`${t("report_progress")}\` (no-review acknowledgement). do NOT call \`create_issue_comment\` for review output.
draft inline comments with NEW line numbers from the full PR diff — attach a \`<details>Technical details</details>\` block to any inline comment whose fix is non-trivial or has cross-file implications (see Inline technical details in the format below). every comment must be actionable, 2-3 sentences max in the visible part.
Same callout-intensity ladder as Review mode — \`[!CAUTION]\` (large red, "will break") → \`[!IMPORTANT]\` (large purple, "must address before merging") → \`[!NOTE]\` (small blue, "FYI") → no callout (plain text). And the same Fix-button lever: the footer renders a Fix button on every non-approving review, so \`approved: true\` suppresses it. Wrapping mergeable feedback in \`[!IMPORTANT]\` trains users to click Fix on reviews that don't need fixing — pick the tier the author's actual next action justifies.
9. **build the review body**: use the same default format as Review mode (preamble + optional cross-cutting \`### \` sections + optional \`### ️ Nitpicks\`) — scoped to the **incremental delta**, not the full PR. The "Reviewed changes" bullets describe what changed since the prior pullfrog review (each bullet starts with a past-tense verb, e.g. \`- Extracted shared CLI runtime into a single module\`). Do NOT include a separate "Prior review feedback" checklist — that's tracked in the rolling PR summary snapshot for the next agent run, and surfacing it in the user-facing body is noise (changes that addressed prior feedback are already covered by the Reviewed-changes bullets). In some cases you may receive a complete diff for the whole PR instead of an incremental one; when this happens, determine what changed since Pullfrog's most recent review yourself before drafting bullets.
10. Submit — every run must end with EXACTLY ONE of \`${t("create_pull_request_review")}\` (substantive review) or \`${t("report_progress")}\` (no-review acknowledgement). do NOT call \`create_issue_comment\` for review output.
Same callout ladder as Review mode — \`[!CAUTION]\` (red, "will break") → \`[!IMPORTANT]\` (purple, "must address before merging") → \`> ️ ...\` (informational, "minor suggestions only") → \`> ✅ ...\` (green friendly, "no concerns"). Same Fix-button lever: the footer renders a Fix button on every non-approving review, so \`approved: true\` suppresses it. Wrapping mergeable feedback in \`[!IMPORTANT]\` trains users to click Fix on reviews that don't need fixing — pick the tier the author's actual next action justifies.
Follow these rules:
- note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session.
- IF NO NEW ISSUES, NON-SUBSTANTIVE CHANGES ONLY (trivial formatting, import reordering, comment tweaks): do NOT submit a review. Instead call \`${t("report_progress")}\` with a 1-2 sentence note explaining no review was warranted (e.g. "No new issues. Changes since last review are formatting-only."). this leaves a visible signal that the run completed.
- ELSE IF NEW CRITICAL ISSUES (blocks merge — bugs, security, data loss, broken core flows): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!CAUTION]\\n> This PR introduces ...\`, then the Reviewed-changes summary.
- ELSE IF NEW MUST-ADDRESS NON-CRITICAL FINDINGS (real consequences if shipped — incorrect behavior, missing validation, regressions the author should fix before merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\`, then the Reviewed-changes summary. Do NOT use this tier for nits, style preferences, or "consider also" suggestions.
- ELSE IF NEW MINOR SUGGESTIONS ONLY (single-line nits, doc/comment polish, defer-able observations, "rough edges"): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens directly with \`Reviewed the following changes:\\n\` (NO alert blockquote), then the Reviewed-changes summary.
- ELSE IF INFORMATIONAL OBSERVATIONS (mergeable as-is, but worth surfacing — e.g. prior feedback addressed cleanly with one minor stale doc reference, or a noteworthy positive observation): call \`${t("create_pull_request_review")}\` with \`approved: true\`, NO inline comments, and the review body. body opens with \`> [!NOTE]\\n> ...\` alert, then the Reviewed-changes summary. If a point is concrete enough to anchor to a line, downgrade the whole review to "minor suggestions only" (\`approved: false\`) instead — \`[!NOTE]\` and inline comments don't mix.
- ELSE IF NO NEW ISSUES, SUBSTANTIVE CHANGES (new functionality, behavior changes, or fixes to prior review feedback): call \`${t("create_pull_request_review")}\` to create a PR review. If all previous reviews have been properly addressed and no new issues were discovered, you can set \`approved: true\`. body opens with \`No new issues. Reviewed the following changes:\\n\`, then the Reviewed-changes summary.`,
- ELSE IF NEW CRITICAL ISSUES (blocks merge — bugs, security, data loss, broken core flows): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!CAUTION]\\n> This PR introduces ...\`, followed by the PR summary using the default format below.
- ELSE IF NEW MUST-ADDRESS NON-CRITICAL FINDINGS (real consequences if shipped — incorrect behavior, missing validation, regressions the author should fix before merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\`, followed by the PR summary using the default format below. Do NOT use this tier for nits, style preferences, or "consider also" suggestions.
- ELSE IF NEW MINOR SUGGESTIONS ONLY (single-line nits, doc/comment polish, defer-able observations, "rough edges"): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> ️ No critical issues — minor suggestions inline.\\n\\n\` (vary the wording after ️ to fit the review), followed by the PR summary using the default format below.
- ELSE IF INFORMATIONAL OBSERVATIONS (mergeable as-is, but worth surfacing — e.g. prior feedback addressed cleanly with one minor stale doc reference, or a noteworthy positive observation): call \`${t("create_pull_request_review")}\` with \`approved: true\`, NO inline comments, and the review body. body opens with \`> ✅ No new issues found.\\n\\n\` (or similar friendly green opener), followed by the PR summary using the default format below. If a point is concrete enough to anchor to a line, downgrade the whole review to "minor suggestions only" (\`approved: false\`) instead — the ✅ signals "no action needed", which contradicts an actionable anchor.
- ELSE IF NO NEW ISSUES, SUBSTANTIVE CHANGES (new functionality, behavior changes, or fixes to prior review feedback): call \`${t("create_pull_request_review")}\` to create a PR review. If all previous reviews have been properly addressed and no new issues were discovered, set \`approved: true\`. body opens with \`> ✅ No new issues found.\\n\\n\`, followed by the PR summary using the default format below.
${PR_SUMMARY_FORMAT}`,
},
{
name: "Plan",
@@ -352,7 +505,7 @@ ${PR_SUMMARY_FORMAT}`,
3. Produce a structured, actionable plan with clear milestones.
4. Call \`${t("report_progress")}\` with the plan.`,
4. Call \`${t("report_progress")}\` with the plan body. Do NOT set \`target_plan_comment\` — that flag is exclusively for revising an existing plan, and \`${t("select_mode")}\` will route you to a separate PlanEdit checklist when a prior plan comment exists for this issue.`,
},
{
name: "Fix",
+3 -2
View File
@@ -1,6 +1,6 @@
{
"name": "pullfrog",
"version": "0.1.4",
"version": "0.1.10",
"type": "module",
"bin": {
"pullfrog": "dist/cli.mjs",
@@ -16,6 +16,7 @@
"typecheck": "tsc --noEmit",
"build": "node esbuild.config.js && tsc -p tsconfig.exports.json",
"check:entrypoints": "node scripts/check-entrypoint-imports.ts",
"docker": "node docker.ts",
"play": "node play.ts",
"runtest": "node test/run.ts",
"scratch": "node scratch.ts",
@@ -49,7 +50,7 @@
"fastmcp": "^3.34.0",
"file-type": "^21.3.0",
"husky": "^9.0.0",
"opencode-ai": "1.1.56",
"opencode-ai": "1.15.1",
"package-manager-detector": "^1.6.0",
"picocolors": "^1.1.1",
"semver": "^7.7.3",
+26 -130
View File
@@ -1,22 +1,28 @@
import { execSync } from "node:child_process";
import { mkdtemp } from "node:fs/promises";
import { devNull, tmpdir } from "node:os";
// thin CLI for ad-hoc fixture runs against the Pullfrog action.
//
// invoke from the repo root:
// pnpm play [args…] # host, in-process — fast iteration (default)
// pnpm play:docker [args…] # local docker container that mocks GHA
// pnpm docker play.ts [args…] # explicit container form (equivalent to `pnpm play:docker`)
//
// see wiki/docker.md for when host vs container matters.
import { dirname, join, resolve } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import arg from "arg";
import { config } from "dotenv";
import type { AgentResult } from "./agents/shared.ts";
import { type Inputs, main } from "./main.ts";
import type { Inputs } from "./main.ts";
import { defineFixture } from "./test/utils.ts";
import { log } from "./utils/cli.ts";
import { runInDocker } from "./utils/docker.ts";
import { ensureGitHubToken } from "./utils/github.ts";
import { isInsideDocker } from "./utils/globals.ts";
import { setupTestRepo } from "./utils/setup.ts";
import { run } from "./utils/runFixture.ts";
const __dirname = dirname(fileURLToPath(import.meta.url));
config();
config({ path: join(__dirname, "..", ".env") });
/**
* default play fixture for ad-hoc testing.
* change this freely without affecting any tests.
* default fixture for ad-hoc `pnpm play` runs. change this freely without
* affecting any tests it's only consumed by this script's no-arg path.
*/
export const playFixture = defineFixture(
{
@@ -25,85 +31,6 @@ export const playFixture = defineFixture(
{ localOnly: true }
);
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// load action's .env file in case it exists for local dev
config();
// also load .env from repo root (for monorepo structure)
config({ path: join(__dirname, "..", ".env") });
export async function run(inputsOrPrompt: Inputs | string): Promise<AgentResult> {
await ensureGitHubToken();
// play.ts is a CI-emulator — isolate it from the developer's user- and
// system-scope gitconfig so checks like `validatePushDestination` see the
// raw stored remote URL instead of values mutated by `url.*.insteadOf`
// rewrites (a common SSH-auth convenience on dev boxes). CI runners have
// empty gitconfigs so this is a no-op there; locally it makes `pnpm play`
// and real runs produce identical git state. `os.devNull` canonicalizes
// the null device across Unix (`/dev/null`) and Windows (`\\.\nul`).
process.env.GIT_CONFIG_GLOBAL = devNull;
process.env.GIT_CONFIG_SYSTEM = devNull;
// create unique temp directory path in OS temp location for parallel execution
// use a parent dir from mkdtemp, then clone into a 'repo' subdirectory
const tempParent = await mkdtemp(join(tmpdir(), "pullfrog-play-"));
const tempDir = join(tempParent, "repo");
const originalCwd = process.cwd();
try {
setupTestRepo({ tempDir });
process.chdir(tempDir);
// run repo setup commands if provided (for pre-planting test state like symlinks).
// this runs AFTER clone but BEFORE the agent, simulating pre-existing repo content.
if (process.env.PULLFROG_TEST_REPO_SETUP) {
log.info("» running repo setup commands...");
execSync(process.env.PULLFROG_TEST_REPO_SETUP, { cwd: tempDir, stdio: "pipe" });
}
// set GITHUB_WORKSPACE to tempDir so main() doesn't try to chdir to the CI checkout path
process.env.GITHUB_WORKSPACE = tempDir;
// allow passing full Inputs object or just a prompt string
const inputs: Inputs =
typeof inputsOrPrompt === "string" ? { prompt: inputsOrPrompt } : inputsOrPrompt;
// set INPUT_* env vars for @actions/core.getInput()
for (const [key, value] of Object.entries(inputs)) {
if (value !== undefined && value !== null) {
process.env[`INPUT_${key.toUpperCase()}`] = String(value);
}
}
const result: AgentResult = await main();
process.chdir(originalCwd);
if (result.success) {
log.success("Action completed successfully");
return { success: true, output: result.output || undefined, error: undefined };
} else {
log.error(`Action failed: ${result.error || "Unknown error"}`);
return { success: false, error: result.error || undefined, output: undefined };
}
} catch (err) {
const errorMessage = (err as Error).message;
log.error(`Error: ${errorMessage}`);
return { success: false, error: errorMessage, output: undefined };
} finally {
// cleanup temp directory - use sudo rm because sandbox isolation may create
// files with different ownership that rmSync can't delete
process.chdir(originalCwd);
try {
execSync(`sudo rm -rf "${tempParent}"`, { stdio: "ignore" });
} catch {
// ignore - cleanup failure is not critical
}
}
}
const isDirectExecution = process.argv[1]
? import.meta.url === pathToFileURL(resolve(process.argv[1])).href
: false;
@@ -112,71 +39,40 @@ if (isDirectExecution) {
const args = arg({
"--help": Boolean,
"--raw": String,
"--local": Boolean,
"-h": "--help",
"-l": "--local",
});
if (args["--help"]) {
log.info(`
Usage: node play.ts [options]
Usage: pnpm play [--raw <input>] (host, in-process; this entry)
pnpm play:docker [--raw <input>] (local docker container that mocks GHA)
Test the Pullfrog action with the inline playFixture.
Run the Pullfrog action against an inline fixture.
Options:
--raw [input] Use raw string as prompt, or JSON object as full fixture
--local, -l Run locally (default: runs in Docker)
-h, --help Show this help message
Environment:
PLAY_LOCAL=1 Same as --local
--raw <input> raw string used as the prompt, or JSON object as full fixture
-h, --help show this message
Examples:
node play.ts # Run inline playFixture
node play.ts --raw "Hello world" # Use raw string as prompt
node play.ts --raw '{"prompt":"Hello","timeout":"5s"}' # Use JSON fixture
pnpm play
pnpm play --raw "Hello world"
pnpm play --raw '{"prompt":"Hi","timeout":"5s"}'
`);
process.exit(0);
}
// default: run in Docker (unless --local, PLAY_LOCAL=1, or already inside Docker)
const useLocal = args["--local"] || process.env.PLAY_LOCAL === "1" || isInsideDocker;
if (!useLocal) {
const passArgs = process.argv
.slice(2)
.map((a) => `'${a.replace(/'/g, "'\\''")}'`)
.join(" ");
const nodeCmd = `node play.ts ${passArgs}`;
const volumeName = "pullfrog-action-node-modules";
const result = runInDocker({
actionDir: __dirname,
args: process.argv.slice(2),
nodeCmd,
volumeName,
envFilterMode: "passthrough",
onStart: () => log.info("» running in Docker container..."),
});
process.exit(result.status ?? 1);
}
if (args["--raw"]) {
const raw = args["--raw"];
// try to parse as JSON, otherwise treat as prompt string
let input: Inputs | string = raw;
try {
input = JSON.parse(raw) as Inputs;
} catch {
// not valid JSON, use as prompt string
// not valid JSON — treat as a literal prompt string.
}
const result = await run(input);
process.exit(result.success ? 0 : 1);
}
// no args - use inline playFixture
const result = await run(playFixture);
process.exit(result.success ? 0 : 1);
}
+60 -49
View File
@@ -84,8 +84,8 @@ importers:
specifier: ^9.0.0
version: 9.1.7
opencode-ai:
specifier: 1.1.56
version: 1.1.56
specifier: 1.15.1
version: 1.15.1
package-manager-detector:
specifier: ^1.6.0
version: 1.6.0
@@ -1444,62 +1444,69 @@ packages:
once@1.4.0:
resolution: {integrity: sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==}
opencode-ai@1.1.56:
resolution: {integrity: sha512-OAF0G/1jVXpOrCh++M5gFEZ0bRLiXOnbxSYMFx5TOoD0OhCjHJS1JlARzaLDAx461qBnn+jocI9BBxD0wwFH3Q==}
opencode-ai@1.15.1:
resolution: {integrity: sha512-xLb1NuYZcMJ1p33hC/kgTMcJAueACVTfX6ps91a54GOFTM/wFp7br0t2cqHopEU9paqItbAnQwZB573qmPKH6w==}
cpu: [arm64, x64]
os: [darwin, linux, win32]
hasBin: true
opencode-darwin-arm64@1.1.56:
resolution: {integrity: sha512-0HqvLm7tcYZr4VJgEzi3Wicia5M9yCX66O7Cv470Qu4+GGbCC2sTTmzQu6pCehSleYgkZsiSBNRcFcW/6F7v0Q==}
opencode-darwin-arm64@1.15.1:
resolution: {integrity: sha512-eNgIfATsnHcud4Pr58OIR+TJGSsDvWmyNlfSDVVgP92qdnHFdZ5YsHKjcUGmeuuUN+oZwPb/z5nZSrkf+CCB2g==}
cpu: [arm64]
os: [darwin]
opencode-darwin-x64-baseline@1.1.56:
resolution: {integrity: sha512-Z9QO9cTC9TnlUxTfEtDbvZbRFxc5Je8rs2Ei0cLH46W9gHGhh1fW4oX64tcrlhF5NUDKJzr/qwdoMfmKkeu53A==}
opencode-darwin-x64-baseline@1.15.1:
resolution: {integrity: sha512-XDx90Hhj+SPUxu0rqewsNR10JTny7+VE4C5pjWB04I6eoiEuBWy2EMvPXPg2FUA5Suz1PXXJ6yThfRtOxXNHuw==}
cpu: [x64]
os: [darwin]
opencode-darwin-x64@1.1.56:
resolution: {integrity: sha512-8ZhNd4sFbDviT1OJs42C35Bx2/z6mkcb7uNhPHX3KKm27KOm7cYjFQa1UTF5or3ZpjkjPQpR/cpo7TELvnnmRA==}
opencode-darwin-x64@1.15.1:
resolution: {integrity: sha512-tNbzF6n+TczILEqo0adtup1ZXBgAcqftQd11+eQohGxtNAjmD7Z/gCTVpEzh9GlHUPzUEuREZ4gRAbJmPpafBQ==}
cpu: [x64]
os: [darwin]
opencode-linux-arm64-musl@1.1.56:
resolution: {integrity: sha512-f30SmYX4xE2fUsnNl66dDX++8iTMI9PLXz7BRHhgXL4XAdMpUUJbOKIsw3ZUB0KWHwjR8cjkEJQOZxR3HPbrZg==}
opencode-linux-arm64-musl@1.15.1:
resolution: {integrity: sha512-UuoizYN32eTWmQT494bw70Sq4AByS0pGk46Mo/z+KzV+KTQlsDXRQrnKATKYFDqE2T3c1VsPM1KqRV/DqvnXxw==}
cpu: [arm64]
os: [linux]
opencode-linux-arm64@1.1.56:
resolution: {integrity: sha512-wEmIEXiEKghurQYgKY5yFUFOmnT/QIlAbYnVeH73gSqOwlYFUUlowXLDJnF+3OGd6m000qalhCxjExL7qKINWw==}
opencode-linux-arm64@1.15.1:
resolution: {integrity: sha512-MG6tuLZqzDjHGeaotejhYuuv2USR0y3v8N+6g5gWPHScX/iJWkJDMFBeT6+KOV/CWawrGRqZfBDfdJSKirX2LQ==}
cpu: [arm64]
os: [linux]
opencode-linux-x64-baseline-musl@1.1.56:
resolution: {integrity: sha512-EC8EGTJCgTZgGIMZdQjOKGXw02+igo0am1Ry6wrPrB0li7XBlYbY6dz8tL3FlknRVMLamtGtb3tXQ6tqbwjX0g==}
opencode-linux-x64-baseline-musl@1.15.1:
resolution: {integrity: sha512-Is50zWUqa9fIJ+tiDOpxENcgn2XBk0QKNEocbu/x9aOdpfFsHhtxe33zi/+9CNdSr+O/6y9jRAMGn7AirJyZlg==}
cpu: [x64]
os: [linux]
opencode-linux-x64-baseline@1.1.56:
resolution: {integrity: sha512-jKOvhkNLcn0h6zmKX+hDdkODijSVDxjAGTZp8BPbJCCJVxnKll9dYfDMTqxi3YN6yXp6sBvyaJT1mcpf8knkeA==}
opencode-linux-x64-baseline@1.15.1:
resolution: {integrity: sha512-ExKWMk/6ULM9HBda2KKZJNE5Ejzaa51QWpr7+Ljv1AlazxQQZKwJfqcZcSNfk0YsgXDESw2w2dwBmOcMaxQZKA==}
cpu: [x64]
os: [linux]
opencode-linux-x64-musl@1.1.56:
resolution: {integrity: sha512-9BFYBPgpY2RrUd7/Ul0VNh+6B9l0FmCNRUV4yYmeFGee9ZDgPem5YGicsKAMQqS/5X0jW3ZWje+KBbBQz4RMwQ==}
opencode-linux-x64-musl@1.15.1:
resolution: {integrity: sha512-feNjVo7XGjqFHf5lejxuyZIkNi9Yi4B2H3w+p2SF9vcyUdPaJnta2/6Os7Pf8kwElRs6EnWRyUO2JVg4hjAjjg==}
cpu: [x64]
os: [linux]
opencode-linux-x64@1.1.56:
resolution: {integrity: sha512-0FP3BzLjn+a9naTtm9hpHibJ+eV1RkI+tItcDQXKwHtvzoBUvytDRP8v6TotEHNtNF0ZLAb3OGurwDIIdtLarg==}
opencode-linux-x64@1.15.1:
resolution: {integrity: sha512-mKRg+iHdwEYNDS+DYa9VQnN903zlw8FInCQRGpY155aR/AF1r3hIn+7IopOTDAwqkutL9vJWMXELxmNpPdaTQg==}
cpu: [x64]
os: [linux]
opencode-windows-x64-baseline@1.1.56:
resolution: {integrity: sha512-zEokdohfoDjWzwULvlSq0Y/tRoVmz2/6GofYO936buPQf09cBcPqpAgLUjZV+pQI0Atyd05YG3BCO4PQxmXzdw==}
opencode-windows-arm64@1.15.1:
resolution: {integrity: sha512-M3Wz4U+hF8paqrBpOWPqOM16MhDDZsnb0EZc1fFdKMfu1a8g2oR3gtq1heQgUOKd8FHaeDQHvBYOqaaJdoaCmA==}
cpu: [arm64]
os: [win32]
opencode-windows-x64-baseline@1.15.1:
resolution: {integrity: sha512-sFvI5sY4kijrkIt9qry34aqZASRo9jJKBLm6PH/zZbGdRtvFM32/n+A26Z/NDbowya8fOtj7MX2Ih5DvR9Md1A==}
cpu: [x64]
os: [win32]
opencode-windows-x64@1.1.56:
resolution: {integrity: sha512-zm/oaWT5uGrW3DumKRHiqv2L2pKwrTvhsT4XUSfIPLQn5EvrYNU6bh3WFO4v71ZBjSIKx5Q7rEvTEQvFiFEZFQ==}
opencode-windows-x64@1.15.1:
resolution: {integrity: sha512-MdCBncbhpcImw3zjYBuoI+ZqfMR1uI4mc8KCltwIgI2DrxuOZNe66A/3feOhWd9MQQ2c2PSdyyJfW9PE0FA/Ow==}
cpu: [x64]
os: [win32]
@@ -3154,51 +3161,55 @@ snapshots:
dependencies:
wrappy: 1.0.2
opencode-ai@1.1.56:
opencode-ai@1.15.1:
optionalDependencies:
opencode-darwin-arm64: 1.1.56
opencode-darwin-x64: 1.1.56
opencode-darwin-x64-baseline: 1.1.56
opencode-linux-arm64: 1.1.56
opencode-linux-arm64-musl: 1.1.56
opencode-linux-x64: 1.1.56
opencode-linux-x64-baseline: 1.1.56
opencode-linux-x64-baseline-musl: 1.1.56
opencode-linux-x64-musl: 1.1.56
opencode-windows-x64: 1.1.56
opencode-windows-x64-baseline: 1.1.56
opencode-darwin-arm64: 1.15.1
opencode-darwin-x64: 1.15.1
opencode-darwin-x64-baseline: 1.15.1
opencode-linux-arm64: 1.15.1
opencode-linux-arm64-musl: 1.15.1
opencode-linux-x64: 1.15.1
opencode-linux-x64-baseline: 1.15.1
opencode-linux-x64-baseline-musl: 1.15.1
opencode-linux-x64-musl: 1.15.1
opencode-windows-arm64: 1.15.1
opencode-windows-x64: 1.15.1
opencode-windows-x64-baseline: 1.15.1
opencode-darwin-arm64@1.1.56:
opencode-darwin-arm64@1.15.1:
optional: true
opencode-darwin-x64-baseline@1.1.56:
opencode-darwin-x64-baseline@1.15.1:
optional: true
opencode-darwin-x64@1.1.56:
opencode-darwin-x64@1.15.1:
optional: true
opencode-linux-arm64-musl@1.1.56:
opencode-linux-arm64-musl@1.15.1:
optional: true
opencode-linux-arm64@1.1.56:
opencode-linux-arm64@1.15.1:
optional: true
opencode-linux-x64-baseline-musl@1.1.56:
opencode-linux-x64-baseline-musl@1.15.1:
optional: true
opencode-linux-x64-baseline@1.1.56:
opencode-linux-x64-baseline@1.15.1:
optional: true
opencode-linux-x64-musl@1.1.56:
opencode-linux-x64-musl@1.15.1:
optional: true
opencode-linux-x64@1.1.56:
opencode-linux-x64@1.15.1:
optional: true
opencode-windows-x64-baseline@1.1.56:
opencode-windows-arm64@1.15.1:
optional: true
opencode-windows-x64@1.1.56:
opencode-windows-x64-baseline@1.15.1:
optional: true
opencode-windows-x64@1.15.1:
optional: true
package-manager-detector@1.6.0: {}
+9
View File
@@ -104,6 +104,15 @@ function createRuntimeContext(): RuntimeContext {
const env: NodeJS.ProcessEnv = { ...process.env };
env.npm_config_registry = NPM_REGISTRY;
env.COREPACK_NPM_REGISTRY = NPM_REGISTRY;
// bypass customer-side release-age gates (npm's `min-release-age`, pnpm's
// `minimumReleaseAge`) so our bootstrap can resolve the latest publish.
// pullfrog's npm version is server-stamped from a SHA-pinned action ref the
// customer already vets at the action layer — not a customer-vetted dep, so
// the gate is the wrong affordance here. env beats .npmrc in both tools.
// npm uses `npm_config_*`; pnpm v11+ requires `pnpm_config_*` (the v10→v11
// migration renamed the prefix). tracked: #713
env.npm_config_min_release_age = "0";
env.pnpm_config_minimum_release_age = "0";
const currentPath = process.env.PATH ?? "";
env.PATH = currentPath ? `${nodeBinDir}${delimiter}${currentPath}` : nodeBinDir;
+75
View File
@@ -0,0 +1,75 @@
import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
import { defineFixture, getAgentOutput } from "../utils.ts";
/**
* BYOK-no-keys fallback test proves that an account configured for a
* BYOK model (here: `anthropic/claude-opus`) but with no provider API
* keys present in the runner env still gets a successful run by falling
* back to a free OpenCode model.
*
* This was the structural failure that took out 15 accounts post-launch
* before the fallback shipped: GH Actions secret references resolved to
* empty strings (because the secrets didn't exist), the action launched
* Claude Code with no key, the LLM provider 401'd, and the run died in
* 20s with a synthesized "Invalid API key" message.
*
* The env block below empty-strings every known provider key that's
* exactly what GitHub Actions does when a `${{ secrets.X }}` reference
* resolves to a missing secret. We verify:
* 1. the run succeeded
* 2. the fallback log line was emitted (proves the swap happened)
*/
const fixture = defineFixture(
{
prompt: "Reply with exactly the single character: 4",
timeout: "5m",
},
{ localOnly: true }
);
function validator(result: AgentResult): ValidationCheck[] {
const output = getAgentOutput(result);
const fellBack = /fell back from .* to opencode\/minimax-m2\.5-free/.test(output);
return [
{ name: "run_succeeded", passed: result.success },
{ name: "fallback_logged", passed: fellBack },
];
}
export const test: TestRunnerOptions = {
name: "byok-no-keys-fallback",
fixture,
validator,
env: {
// simulate every BYOK provider's secret being absent — same shape as
// a fresh-install account whose user never configured any keys.
ANTHROPIC_API_KEY: "",
CLAUDE_CODE_OAUTH_TOKEN: "",
OPENAI_API_KEY: "",
OPENROUTER_API_KEY: "",
GEMINI_API_KEY: "",
GOOGLE_GENERATIVE_AI_API_KEY: "",
XAI_API_KEY: "",
DEEPSEEK_API_KEY: "",
MOONSHOT_API_KEY: "",
OPENCODE_API_KEY: "",
AWS_BEARER_TOKEN_BEDROCK: "",
AWS_ACCESS_KEY_ID: "",
AWS_SECRET_ACCESS_KEY: "",
BEDROCK_MODEL_ID: "",
// configure a model that requires a BYOK key — the fallback only
// engages when there's a configured model whose provider key is
// absent, so we have to pin one. anthropic/claude-opus is the
// most common first-run choice (it's the catalog "preferred" for
// the anthropic provider).
PULLFROG_MODEL: "anthropic/claude-opus",
},
tags: ["agnostic"],
coverage: [
"action/utils/byokFallback.ts",
"action/utils/apiKeys.ts",
"action/utils/agent.ts",
"action/main.ts",
"action/models.ts",
],
};
+6
View File
@@ -96,4 +96,10 @@ export const test: TestRunnerOptions = {
repoSetup,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic", "security"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+8
View File
@@ -104,4 +104,12 @@ export const test: TestRunnerOptions = {
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/utils/lifecycle.ts",
"action/toolState.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+1
View File
@@ -92,4 +92,5 @@ export const test: TestRunnerOptions = {
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic", "security"],
coverage: ["action/mcp/dependencies.ts", "action/utils/install.ts"],
};
+8
View File
@@ -62,4 +62,12 @@ export const test: TestRunnerOptions = {
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/utils/lifecycle.ts",
"action/toolState.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+10 -2
View File
@@ -18,8 +18,8 @@ const fixture = defineFixture(
3. Report if it succeeded
## Test 2: Tag Operations
1. Create a local tag using the git MCP tool: git tag -a test-tag-enabled -m "test tag"
2. Try push_tags tool with tag "test-tag-enabled"
1. Create a local tag using the git MCP tool: git tag -a test-tag-enabled-\${RANDOM} -m "test tag"
2. Try push_tags tool with the tag you just created
3. Report if tag push succeeded
## Test 3: Branch Deletion (cleanup)
@@ -74,4 +74,12 @@ export const test: TestRunnerOptions = {
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/utils/lifecycle.ts",
"action/toolState.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+8
View File
@@ -67,4 +67,12 @@ export const test: TestRunnerOptions = {
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/gitAuth.ts",
"action/utils/gitAuthServer.ts",
"action/utils/lifecycle.ts",
"action/toolState.ts",
"action/mcp/git.ts",
"action/mcp/checkout.ts",
],
};
+7
View File
@@ -29,4 +29,11 @@ export const test: TestRunnerOptions = {
expectFailure: true,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
tags: ["agnostic"],
coverage: [
"action/utils/timer.ts",
"action/utils/subprocess.ts",
"action/utils/exitHandler.ts",
"action/utils/activity.ts",
"action/mcp/selectMode.ts",
],
};
-70
View File
@@ -1,70 +0,0 @@
#!/usr/bin/env bash
# determines which agents need testing based on changed files.
# reads changed file paths from stdin (JSON array or newline-delimited).
# outputs a JSON array of agent names to stdout.
#
# only agents whose harness file changed AND are exported from index.ts are included.
# shared.ts/index.ts/postRun.ts and other non-harness action changes fall back to opencode as a canary.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
AGENTS_INDEX="$SCRIPT_DIR/../agents/index.ts"
# build the set of active agents from index.ts imports (portable, no -P)
active_agents=()
while IFS= read -r line; do
[[ -n "$line" ]] && active_agents+=("$line")
done < <(sed -n 's/.*from "\.\/\([^"]*\)\.ts".*/\1/p' "$AGENTS_INDEX" | grep -v shared)
# read stdin - auto-detect JSON array vs newline-delimited
input=$(cat)
if echo "$input" | jq -e 'type == "array"' > /dev/null 2>&1; then
files=$(echo "$input" | jq -r '.[]')
else
files="$input"
fi
is_active_agent() {
local name="$1"
for a in "${active_agents[@]}"; do
[[ "$a" == "$name" ]] && return 0
done
return 1
}
# find which agent harness files changed
changed_agents=()
has_non_agent_change=false
while IFS= read -r file; do
[[ -z "$file" ]] && continue
case "$file" in
action/agents/shared.ts|action/agents/index.ts|action/agents/postRun.ts)
has_non_agent_change=true
;;
action/agents/*.ts)
agent_name="$(basename "$file" .ts)"
if is_active_agent "$agent_name"; then
changed_agents+=("$agent_name")
else
# legacy/inactive agent file changed — treat as non-agent change
has_non_agent_change=true
fi
;;
action/*)
has_non_agent_change=true
;;
esac
done <<< "$files"
# output agents based on change type.
# non-agent action changes always include opencode as a canary.
if $has_non_agent_change; then
changed_agents+=("opencode")
fi
if [[ ${#changed_agents[@]} -gt 0 ]]; then
printf '%s\n' "${changed_agents[@]}" | sort -u | jq -R . | jq -sc .
else
echo '[]'
fi
+25 -53
View File
@@ -1,4 +1,3 @@
import { execFileSync } from "node:child_process";
import { readdirSync, readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -16,7 +15,7 @@ type WorkflowJob = {
"runs-on": string;
"timeout-minutes"?: number;
permissions?: WorkflowPermissions;
strategy?: { "fail-fast": boolean; matrix: Record<string, string[]> };
strategy?: { "fail-fast": boolean; matrix: Record<string, unknown> };
env?: Record<string, string>;
steps?: unknown[];
};
@@ -57,12 +56,14 @@ const expectedAgents = Object.keys(agents).sort();
const crossagentTests = getTestNamesFromDir("crossagent");
const agnosticTests = getTestNamesFromDir("agnostic");
const adhocTests = getTestNamesFromDir("adhoc");
const dynamicAgentsExpression = "$" + "{{ fromJSON(needs.changes.outputs.agents) }}";
// all provider API key names + GITHUB_TOKEN + model overrides
// all provider API key names + managed credentials (e.g. Codex auth blob)
// + GITHUB_TOKEN + model overrides
const expectedAgentEnvVars = [
"GITHUB_TOKEN",
...new Set(Object.values(providers).flatMap((p) => [...p.envVars])),
...new Set(
Object.values(providers).flatMap((p) => [...p.envVars, ...(p.managedCredentials ?? [])])
),
"PULLFROG_MODEL",
].sort();
@@ -83,53 +84,22 @@ describe("ci workflow consistency", () => {
const rootJob = rootWorkflow.jobs["action-agents"];
const actionJob = actionWorkflow.jobs.agents;
it("root agent matrix uses dynamic output from changes job", () => {
expect(rootJob.strategy!.matrix.agent).toBe(dynamicAgentsExpression);
});
it("changed-agents.sh falls back to opencode when shared agent code changed", () => {
const input = JSON.stringify(["action/agents/shared.ts"]);
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input,
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
});
it("changed-agents.sh falls back to opencode for non-agent action changes", () => {
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input: JSON.stringify(["action/mcp/server.ts"]),
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
});
it("changed-agents.sh includes opencode canary alongside changed agents", () => {
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input: JSON.stringify(["action/agents/opencode.ts", "action/mcp/server.ts"]),
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
});
it("changed-agents.sh treats legacy agent files as non-agent changes", () => {
const output = execFileSync("bash", [join(__dirname, "changed-agents.sh")], {
input: JSON.stringify(["action/agents/codex.ts", "action/agents/gemini.ts"]),
encoding: "utf-8",
});
expect(JSON.parse(output)).toEqual(["opencode"]);
it("root agents matrix is wired to the dynamic matrix output", () => {
const include = rootJob.strategy?.matrix.include;
expect(typeof include).toBe("string");
expect(include as string).toContain("fromJSON(needs.changes.outputs.matrix).agents");
});
it("action agent matrix matches agents map", () => {
expect([...actionJob.strategy!.matrix.agent].sort()).toEqual(expectedAgents);
});
it("root test matrix matches crossagent/ directory", () => {
expect([...rootJob.strategy!.matrix.test].sort()).toEqual(crossagentTests);
expect((actionJob.strategy?.matrix.agent as string[])?.slice().sort()).toEqual(
expectedAgents
);
});
it("action test matrix matches crossagent/ directory", () => {
expect([...actionJob.strategy!.matrix.test].sort()).toEqual(crossagentTests);
expect((actionJob.strategy?.matrix.test as string[])?.slice().sort()).toEqual(
crossagentTests
);
});
it("permissions match between root and action", () => {
@@ -149,8 +119,8 @@ describe("ci workflow consistency", () => {
});
it("fail-fast is enabled in both", () => {
expect(rootJob.strategy!["fail-fast"]).toBe(true);
expect(actionJob.strategy!["fail-fast"]).toBe(true);
expect(rootJob.strategy?.["fail-fast"]).toBe(true);
expect(actionJob.strategy?.["fail-fast"]).toBe(true);
});
});
@@ -158,12 +128,14 @@ describe("ci workflow consistency", () => {
const rootJob = rootWorkflow.jobs["action-agnostic"];
const actionJob = actionWorkflow.jobs.agnostic;
it("root test matrix matches agnostic/ directory", () => {
expect([...rootJob.strategy!.matrix.test].sort()).toEqual(agnosticTests);
it("root agnostic matrix is wired to the dynamic matrix output", () => {
const include = rootJob.strategy?.matrix.include;
expect(typeof include).toBe("string");
expect(include as string).toContain("fromJSON(needs.changes.outputs.matrix).agnostic");
});
it("action test matrix matches agnostic/ directory", () => {
expect([...actionJob.strategy!.matrix.test].sort()).toEqual(agnosticTests);
expect((actionJob.strategy?.matrix.test as string[])?.slice().sort()).toEqual(agnosticTests);
});
it("permissions match between root and action", () => {
@@ -183,8 +155,8 @@ describe("ci workflow consistency", () => {
});
it("fail-fast is enabled in both", () => {
expect(rootJob.strategy!["fail-fast"]).toBe(true);
expect(actionJob.strategy!["fail-fast"]).toBe(true);
expect(rootJob.strategy?.["fail-fast"]).toBe(true);
expect(actionJob.strategy?.["fail-fast"]).toBe(true);
});
});
});
+116
View File
@@ -0,0 +1,116 @@
/**
* shared coverage / glob plumbing for the matrix builder.
*
* every test (`crossagent/`, `agnostic/`) and every provider entry
* (`providers.ts`) declares a `coverage` array of repo-relative globs. on a PR
* push, the `changes` job feeds the changed-file list into `matrix.ts`, which
* intersects each entry's globs against the diff and emits only the entries
* that need to run.
*
* `ALWAYS_RUN_ALL` is the escape hatch: any change to a file matched here
* forces the full matrix (every test, every flagship, every alias). it
* captures cross-cutting infrastructure where fan-out is unpredictable
* agent loader, MCP server boot, test runner itself. if a per-test glob
* goes stale, this list and the on-`main`-full-matrix policy are the safety
* nets there's no completeness lint.
*
* `coverage` is optional on tests/providers; missing = always run (treat as
* "any code change touches me"). default to defensive opt into precision
* by adding globs.
*/
/** patterns that, when matched by any changed file, force the full matrix. */
export const ALWAYS_RUN_ALL: string[] = [
// agent loader + cross-agent shared code
"action/agents/shared.ts",
"action/agents/index.ts",
"action/agents/postRun.ts",
// test harness — changing these can affect every test
"action/test/run.ts",
"action/test/utils.ts",
"action/test/matrix.ts",
"action/test/list-aliases.ts",
"action/test/coverage.ts",
"action/test/providers.ts",
// boot + lifecycle
"action/main.ts",
"action/index.ts",
"action/cli.ts",
"action/utils/setup.ts",
"action/utils/install.ts",
"action/utils/runFixture.ts",
"action/utils/globals.ts",
// local docker container plumbing (changes invalidate every test's environment)
"action/Dockerfile",
"action/docker-entrypoint.sh",
"action/docker.ts",
// MCP orchestrator (every test runs through it)
"action/mcp/server.ts",
"action/mcp/shared.ts",
// dependency graph
"action/package.json",
"action/pnpm-lock.yaml",
// workflow itself
".github/workflows/test.yml",
];
/**
* expand a single brace group like `{a,b,c}` into an array of patterns.
*
* intentionally minimal: nested braces (`{a,{b,c}}`) and escaped braces are
* NOT supported coverage globs in this repo only need flat brace groups
* (`{claude,opencode}.ts`). add complexity if a real use case emerges.
*/
function expandBraces(pattern: string): string[] {
const m = pattern.match(/\{([^{}]+)\}/);
if (!m || m.index === undefined) return [pattern];
const before = pattern.slice(0, m.index);
const after = pattern.slice(m.index + m[0].length);
const opts = m[1].split(",");
return opts.flatMap((opt) => expandBraces(`${before}${opt}${after}`));
}
/** convert a glob pattern to a regex anchored at start + end. */
function globToRegex(pattern: string): RegExp {
const DSTAR = "\u0000DSTAR\u0000";
let s = pattern.replace(/\*\*/g, DSTAR);
s = s.replace(/[.+^$()|[\]\\]/g, "\\$&");
s = s.replace(/\*/g, "[^/]*");
s = s.replace(/\?/g, "[^/]");
s = s.replaceAll(DSTAR, ".*");
return new RegExp(`^${s}$`);
}
/** does any path in `paths` match any glob in `patterns`? */
export function anyMatch(paths: string[], patterns: string[]): boolean {
if (patterns.length === 0) return false;
const regexes = patterns.flatMap((p) => expandBraces(p)).map(globToRegex);
return paths.some((path) => regexes.some((r) => r.test(path)));
}
/**
* decide whether an entry runs given changed files + its coverage globs.
*
* three short-circuits:
* 1. `full` flag (e.g. main pushes, workflow_dispatch) always run
* 2. any changed file matches `ALWAYS_RUN_ALL` run everything
* 3. coverage missing or empty on the entry run (defensive default)
*
* otherwise: run iff any changed file matches the entry's coverage globs.
*
* `coverage: []` is treated identically to `coverage: undefined` to avoid the
* footgun where a future test author intends "skip on PRs" by passing an
* empty array silently skipping CI on every PR is worse than always running.
*/
export type ShouldRunInput = {
changedFiles: string[];
coverage: string[] | undefined;
full: boolean;
};
export function shouldRun(input: ShouldRunInput): boolean {
if (input.full) return true;
if (anyMatch(input.changedFiles, ALWAYS_RUN_ALL)) return true;
if (input.coverage === undefined || input.coverage.length === 0) return true;
return anyMatch(input.changedFiles, input.coverage);
}
+110
View File
@@ -0,0 +1,110 @@
import { randomUUID } from "node:crypto";
import { readFileSync } from "node:fs";
import { detectCodexRefresh } from "../../utils/codexHome.ts";
import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.ts";
import { defineFixture } from "../utils.ts";
/**
* codex-auth test end-to-end Codex ChatGPT-subscription auth smoke.
*
* Pins openai/gpt-5.5 (in upstream opencode's Codex `ALLOWED_MODELS` allow
* list) and runs the full opencode harness against the developer's / CI's
* `CODEX_AUTH_JSON`. Exercises:
*
* - installCodexAuth() materializes auth.json at $HOME/.local/share/opencode/
* with `expires: 0` (forces refresh on first request).
* - opencode's CodexAuthPlugin routes openai requests through the ChatGPT
* subscription instead of needing OPENAI_API_KEY.
* - the refresh chain advances during the run (proving the refresh path
* works end-to-end against live Codex auth servers).
* - detectCodexRefresh() would surface the rotation to entryPost.ts for
* write-back to Pullfrog's secret store.
*
* the post-hook itself runs in a separate GHA `post:` step and is not
* invoked by `pnpm runtest`. instead, this test asserts the on-disk auth.json
* state that the post-hook would consume, which is the genuine integration
* boundary (everything past `detectCodexRefresh` is a single fetch + unit-
* tested in codexHome.test.ts).
*
* requires `CODEX_AUTH_JSON` in the environment. dev-local: put it in
* `.env`. CI: provisioned as `secrets.CODEX_AUTH_JSON` and forwarded by the
* `action-agents` job env block in `.github/workflows/test.yml`.
*/
const token = randomUUID();
const fixture = defineFixture(
{
prompt: `Call set_output with exactly this token and nothing else: ${token}`,
shell: "restricted",
push: "disabled",
timeout: "4m",
},
{ localOnly: true }
);
function parseOriginalRefresh(): string | null {
const raw = process.env.CODEX_AUTH_JSON;
if (!raw) return null;
try {
const parsed = JSON.parse(raw) as { tokens?: { refresh_token?: unknown } };
const rt = parsed?.tokens?.refresh_token;
return typeof rt === "string" && rt.length > 0 ? rt : null;
} catch {
return null;
}
}
function validator(result: AgentResult): ValidationCheck[] {
const setOutputCalled = result.structuredOutput !== null;
const tokenMatches = result.structuredOutput === token;
// installCodexAuth() emits this log line with the absolute path; we use it
// to find the per-test HOME (randomized inside runAgentStreaming).
const pathMatch = result.output.match(/installed Codex auth at (\S+)/);
const authPath = pathMatch?.[1];
let authMaterialized = false;
let refreshRotated = false;
if (authPath) {
try {
const content = readFileSync(authPath, "utf8");
authMaterialized = true;
const originalRefresh = parseOriginalRefresh();
if (originalRefresh) {
refreshRotated = detectCodexRefresh({ authFileContent: content, originalRefresh }) !== null;
}
} catch {
// authMaterialized stays false
}
}
return [
{ name: "set_output", passed: setOutputCalled },
{ name: "token_matches", passed: tokenMatches },
{ name: "auth_materialized", passed: authMaterialized },
{ name: "refresh_rotated", passed: refreshRotated },
];
}
export const test: TestRunnerOptions = {
name: "codex-auth",
fixture,
validator,
agents: ["opencode"],
env: {
PULLFROG_MODEL: "openai/gpt",
PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1",
},
coverage: [
"action/utils/codexHome.ts",
"action/entryPost.ts",
"action/agents/{opencode,opencode_v2}.ts",
],
// forks + contributors without the Codex secret skip cleanly rather than
// failing on `auth_materialized=✗` and (with fail-fast: true) cascading
// cancellation across the rest of the matrix. CI on `pullfrog/app` and
// dev-local with `.env` both have the secret and run the test as normal.
skipIf: () => (process.env.CODEX_AUTH_JSON ? null : "CODEX_AUTH_JSON unset"),
};
+2
View File
@@ -43,4 +43,6 @@ export const test: TestRunnerOptions = {
},
repoSetup:
'mkdir -p /tmp/pullfrog-mcp-secret && printf "%s" "$PULLFROG_MCP_SECRET" > /tmp/pullfrog-mcp-secret/secret.txt',
// any MCP-layer change can affect repo-MCP merging; agents own MCP wiring.
coverage: ["action/mcp/**", "action/agents/{claude,opencode,opencode_v2}.ts"],
};
+1
View File
@@ -43,4 +43,5 @@ export const test: TestRunnerOptions = {
validator,
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
coverage: ["action/mcp/shell.ts", "action/agents/{claude,opencode,opencode_v2}.ts"],
};
+5
View File
@@ -52,4 +52,9 @@ export const test: TestRunnerOptions = {
validator,
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
coverage: [
"action/utils/normalizeEnv.ts",
"action/mcp/shell.ts",
"action/agents/{claude,opencode,opencode_v2}.ts",
],
};
+1
View File
@@ -44,4 +44,5 @@ export const test: TestRunnerOptions = {
PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1",
PULLFROG_MODEL: "anthropic/claude-sonnet-4-6",
},
coverage: ["action/agents/claude.ts"],
};
+5
View File
@@ -44,4 +44,9 @@ export const test: TestRunnerOptions = {
PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1",
PULLFROG_MODEL: "anthropic/claude-sonnet-4-6",
},
coverage: [
"action/agents/opencode.ts",
"action/agents/opencode_v2.ts",
"action/agents/opencodePlugin.ts",
],
};
+10 -4
View File
@@ -2,13 +2,16 @@ import type { AgentResult, TestRunnerOptions, ValidationCheck } from "../utils.t
import { defineFixture } from "../utils.ts";
/**
* smoke test - validates agent can connect to API and call MCP tools.
* verifies set_output tool is called with correct value.
* smoke test validates agent can connect to the API and call MCP tools.
*
* two tool calls (not one) on purpose: this is the canary that exercises the
* 2nd modelagent round-trip across every providers-live flagship. bugs like
* the Gemini `thought_signature` echo only fire after the first tool result
* comes back. do not collapse to a single tool call.
*/
const fixture = defineFixture(
{
prompt: `Call set_output with "SMOKE TEST PASSED".`,
prompt: `First call the git tool with command "status" to confirm the repo is reachable. Then call set_output with exactly the literal string "SMOKE TEST PASSED".`,
},
{ localOnly: true }
);
@@ -29,4 +32,7 @@ export const test: TestRunnerOptions = {
fixture,
validator,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
// canary: any agent harness change runs the smoke. shared MCP set_output
// surface is also captured.
coverage: ["action/agents/{claude,opencode,opencode_v2}.ts", "action/mcp/output.ts"],
};
+5
View File
@@ -58,4 +58,9 @@ export const test: TestRunnerOptions = {
validator,
agentEnv,
env: { PULLFROG_DISABLE_SECURITY_INSTRUCTIONS: "1" },
coverage: [
"action/utils/normalizeEnv.ts",
"action/mcp/shell.ts",
"action/agents/{claude,opencode,opencode_v2}.ts",
],
};
+53 -60
View File
@@ -1,61 +1,45 @@
/**
* emits a JSON array of { slug, agent, name } entries for one of two CI matrix
* jobs. `agent` mirrors the harness the runtime would pick in production
* (anthropic/* claude-code, everything else opencode).
* (anthropic/* claude, everything else opencode).
*
* MODE=aliases (default) every alias minus pruned passthroughs. consumed by
* `models-live`, which runs the cheap top-level CLI smoke per alias
* (`action/test/model-smoke.ts`) to validate resolution + auth.
* MODE=aliases (default) every alias. consumed by `models-live`, which runs
* the cheap top-level CLI smoke per alias (`action/test/model-smoke.ts`) to
* validate resolution + auth.
*
* MODE=flagships one standard-tier model per provider. consumed by
* `providers-live`, which runs the full harness smoke
* (`pnpm runtest smoke <agent>`) to validate provider-class tool-calling
* (e.g. Gemini schema sanitizer, OpenAI tool-call format).
* (e.g. Gemini schema sanitizer, OpenAI tool-call format). flagship slugs
* live in `providers.ts` alongside their per-provider coverage globs.
*
* passthrough pruning (aliases mode): openrouter/* aliases and keyed opencode/*
* aliases are routing-layer wrappers around models we already smoke-test
* directly. running every passthrough burns CI minutes without catching
* anything new slug-drift is covered by the `models-catalog` job. one canary
* per routing layer proves the routing surface (auth, tool-call translation)
* is alive; set INCLUDE_PASSTHROUGHS=1 to bypass for full validation.
* Every keyed alias is smoked including `openrouter/*` and keyed `opencode/*`
* passthroughs. They look like routing-layer wrappers but each one is a
* distinct catalog entry on models.dev (under the `openrouter` / `opencode`
* provider sections) that can drift independently of the upstream provider
* mirror testing the direct google entry tells you nothing about whether
* the openrouter mirror has the same model id. The only entries pruned are
* routing slugs (bedrock/byok) whose `resolve` is a sentinel that picks the
* actual model id from a per-run env var.
*
* usage:
* node action/test/list-aliases.ts
* MODE=flagships node action/test/list-aliases.ts
* MATRIX_FILTER=gemini node action/test/list-aliases.ts
* INCLUDE_PASSTHROUGHS=1 node action/test/list-aliases.ts
*
* NOTE: the per-PR-precision matrix lives in `matrix.ts`, which calls into
* this file. raw invocation here emits the unfiltered matrix.
*/
import { modelAliases } from "../models.ts";
import { providers } from "./providers.ts";
const ROUTING_CANARIES = new Set(["openrouter/claude-sonnet", "opencode/claude-sonnet"]);
export type MatrixEntry = {
slug: string;
agent: string;
name: string;
};
// hand-picked "standard good model" per provider — not the pro/opus tier (too
// expensive for per-push) and not the free/experimental tier (too flaky). these
// aliases anchor the harness smoke job that catches provider-class regressions
// like Gemini schema sanitization or OpenAI tool-call format drift. the
// assertion below catches slug-drift loudly, but adding a NEW provider without
// an entry here silently omits it from `providers-live` — see
// wiki/models-catalog.md "To add a provider".
const FLAGSHIPS = [
"anthropic/claude-sonnet",
"openai/gpt",
"google/gemini-pro",
"xai/grok",
"deepseek/deepseek-pro",
"moonshotai/kimi-k2",
"opencode/big-pickle",
"openrouter/claude-sonnet",
];
function isPrunablePassthrough(alias: (typeof modelAliases)[number]): boolean {
if (ROUTING_CANARIES.has(alias.slug)) return false;
if (alias.provider === "openrouter") return true;
// opencode FREE models (big-pickle, mimo, minimax, gpt-5-nano) are unique
// to opencode and used in prod — keep them. only prune the keyed mirrors.
return alias.provider === "opencode" && !alias.isFree;
}
function toMatrixEntry(alias: (typeof modelAliases)[number]) {
function toMatrixEntry(alias: (typeof modelAliases)[number]): MatrixEntry {
return {
slug: alias.slug,
agent: alias.slug.startsWith("anthropic/") ? "claude" : "opencode",
@@ -64,32 +48,41 @@ function toMatrixEntry(alias: (typeof modelAliases)[number]) {
};
}
const mode = process.env.MODE === "flagships" ? "flagships" : "aliases";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const includePassthroughs = process.env.INCLUDE_PASSTHROUGHS === "1";
const aliasBySlug = new Map(modelAliases.map((a) => [a.slug, a]));
const matrix = (() => {
if (mode === "flagships") {
return FLAGSHIPS.map((slug) => {
const alias = aliasBySlug.get(slug);
export function buildAliasMatrix(opts: { filter?: string }): MatrixEntry[] {
const filter = opts.filter ?? "";
return modelAliases
.filter((alias) => {
if (filter && !alias.slug.toLowerCase().includes(filter)) return false;
// routing slugs (bedrock/byok) need a per-run env var to pick the actual
// model — there's no generic smoke test.
if (alias.routing) return false;
return true;
})
.map(toMatrixEntry);
}
export function buildFlagshipMatrix(opts: { filter?: string }): MatrixEntry[] {
const filter = opts.filter ?? "";
return providers
.map((p) => {
const alias = aliasBySlug.get(p.flagship);
if (!alias) {
throw new Error(
`list-aliases: flagship "${slug}" missing from modelAliases — update FLAGSHIPS`
`list-aliases: flagship "${p.flagship}" missing from modelAliases — update providers.ts`
);
}
return alias;
})
.filter((alias) => !filter || alias.slug.toLowerCase().includes(filter))
.map(toMatrixEntry);
}
return modelAliases
.filter((alias) => {
if (filter && !alias.slug.toLowerCase().includes(filter)) return false;
if (!includePassthroughs && isPrunablePassthrough(alias)) return false;
return true;
})
.filter((alias) => !filter || alias.slug.toLowerCase().includes(filter))
.map(toMatrixEntry);
})();
}
process.stdout.write(JSON.stringify(matrix));
if (import.meta.url === `file://${process.argv[1]}`) {
const mode = process.env.MODE === "flagships" ? "flagships" : "aliases";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const matrix =
mode === "flagships" ? buildFlagshipMatrix({ filter }) : buildAliasMatrix({ filter });
process.stdout.write(JSON.stringify(matrix));
}
+227
View File
@@ -0,0 +1,227 @@
/**
* unified CI matrix builder. emits the four matrices consumed by
* `.github/workflows/test.yml`:
*
* - agents: crossagent tests × eligible agents (fan-out)
* - agnostic: agnostic infrastructure tests (run with opencode)
* - flagships: one harness smoke per provider (providers-live)
* - aliases: one CLI smoke per model alias (models-live)
*
* input: a JSON array of repo-relative changed paths on stdin (the
* `paths-filter` action's `*_files` output). PR pushes pass the diff;
* `main` pushes and `workflow_dispatch` set FULL=1 to skip filtering and
* emit every entry.
*
* each test/provider declares its own `coverage` globs colocated with the
* test (`crossagent/`, `agnostic/`) or provider (`providers.ts`). the matrix
* builder intersects coverage against the diff. a top-level `ALWAYS_RUN_ALL`
* (see `coverage.ts`) bypasses filtering when test-harness or cross-cutting
* agent code changes keeps stale globs from silently skipping critical
* tests on test runner / shared.ts churn.
*
* usage:
* echo '["action/agents/opencode.ts"]' | node action/test/matrix.ts
* FULL=1 node action/test/matrix.ts < /dev/null
* MATRIX_FILTER=gemini FULL=1 node action/test/matrix.ts < /dev/null
*/
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { shouldRun } from "./coverage.ts";
import { buildAliasMatrix, buildFlagshipMatrix } from "./list-aliases.ts";
import { providers } from "./providers.ts";
const __dirname = dirname(fileURLToPath(import.meta.url));
type AgentEntry = { agent: string; test: string; name: string };
type AgnosticEntry = { test: string; name: string };
type SlugEntry = { slug: string; agent: string; name: string };
type MatrixOutput = {
agents: AgentEntry[];
agnostic: AgnosticEntry[];
flagships: SlugEntry[];
aliases: SlugEntry[];
};
/**
* extracted test metadata. parsed via regex from the test source see
* `parseTestFile`. dynamic-import is intentionally avoided: the GHA `changes`
* job runs without `pnpm install`, and the real test modules transitively
* import `@actions/core` etc. parsing keeps `matrix.ts` zero-dep.
*/
type ParsedTest = {
name: string;
agents: string[] | undefined;
coverage: string[] | undefined;
};
const STRING_LITERAL = /"((?:\\.|[^"\\])*)"/g;
function extractStringLiterals(source: string): string[] {
const out: string[] = [];
STRING_LITERAL.lastIndex = 0;
let m: RegExpExecArray | null;
// biome-ignore lint/suspicious/noAssignInExpressions: idiomatic regex iteration
while ((m = STRING_LITERAL.exec(source))) {
out.push(m[1]);
}
return out;
}
/**
* extract a `key: [...]` array literal of strings from a test object. matches
* line-leading indented `key:` to avoid colliding with the same word inside
* prompts / template literals.
*/
function extractStringArray(source: string, key: string): string[] | undefined {
const re = new RegExp(`^\\s+${key}:\\s*\\[([\\s\\S]*?)\\]`, "m");
const m = source.match(re);
if (!m) return undefined;
return extractStringLiterals(m[1]);
}
function parseTestFile(source: string): ParsedTest | null {
// strip line comments — `//` inside string literals is rare in test files,
// and the static parser doesn't need to be perfect (defensive default of
// "missing coverage = always run" covers parse misses).
const stripped = source.replace(/\/\/[^\n]*$/gm, "");
const nameMatch = stripped.match(/^\s+name:\s*"([^"]+)"/m);
if (!nameMatch) return null;
return {
name: nameMatch[1],
agents: extractStringArray(stripped, "agents"),
coverage: extractStringArray(stripped, "coverage"),
};
}
function loadDir(dir: string): ParsedTest[] {
const dirPath = join(__dirname, dir);
if (!existsSync(dirPath)) return [];
const files = readdirSync(dirPath).filter((f) => f.endsWith(".ts"));
const out: ParsedTest[] = [];
for (const file of files) {
const source = readFileSync(join(dirPath, file), "utf8");
const parsed = parseTestFile(source);
if (parsed) out.push(parsed);
}
return out;
}
/**
* derive the active agent list from `agents/index.ts` so adding a new harness
* file automatically wires it into the matrix. avoids dynamic-import
* (transitively pulls `@actions/core` etc. would explode in the no-install
* `changes` job) by regex-parsing the imports the same way `parseTestFile`
* handles tests.
*/
function loadAgents(): string[] {
const indexPath = join(__dirname, "..", "agents", "index.ts");
const source = readFileSync(indexPath, "utf8");
const out: string[] = [];
const re = /^\s*import\s+\{\s*(\w+)\s*\}\s+from\s+"\.\/(\w+)\.ts"/gm;
let m: RegExpExecArray | null;
// biome-ignore lint/suspicious/noAssignInExpressions: idiomatic regex iteration
while ((m = re.exec(source))) {
if (m[2] === "shared") continue;
out.push(m[1]);
}
return out.sort();
}
function readChangedFiles(): string[] {
const raw = readFileSync(0, "utf8").trim();
if (!raw) return [];
const parsed: unknown = JSON.parse(raw);
if (!Array.isArray(parsed)) {
throw new Error("matrix: stdin must be a JSON array of changed paths");
}
return parsed.map((p) => {
if (typeof p !== "string") {
throw new Error(`matrix: non-string entry in changed paths: ${JSON.stringify(p)}`);
}
return p;
});
}
function buildAgentsMatrix(input: { changedFiles: string[]; full: boolean }): AgentEntry[] {
const tests = loadDir("crossagent");
const allAgents = loadAgents();
const out: AgentEntry[] = [];
for (const t of tests) {
if (!shouldRun({ changedFiles: input.changedFiles, coverage: t.coverage, full: input.full })) {
continue;
}
const agents = t.agents ?? allAgents;
for (const agent of agents) {
out.push({ agent, test: t.name, name: `${t.name}-${agent}` });
}
}
return out;
}
function buildAgnosticMatrix(input: { changedFiles: string[]; full: boolean }): AgnosticEntry[] {
const tests = loadDir("agnostic");
const out: AgnosticEntry[] = [];
for (const t of tests) {
if (!shouldRun({ changedFiles: input.changedFiles, coverage: t.coverage, full: input.full })) {
continue;
}
out.push({ test: t.name, name: t.name });
}
return out;
}
function buildFlagshipsMatrix(input: {
changedFiles: string[];
full: boolean;
filter: string;
}): SlugEntry[] {
const all = buildFlagshipMatrix({ filter: input.filter });
const byName = new Map(providers.map((p) => [p.flagship, p]));
return all.filter((entry) => {
const provider = byName.get(entry.slug);
return shouldRun({
changedFiles: input.changedFiles,
coverage: provider?.coverage,
full: input.full,
});
});
}
function buildAliasesMatrix(input: {
changedFiles: string[];
full: boolean;
filter: string;
}): SlugEntry[] {
const all = buildAliasMatrix({ filter: input.filter });
const coverageByProvider = new Map(providers.map((p) => [p.name, p.coverage]));
return all.filter((entry) => {
const provider = entry.slug.split("/")[0];
return shouldRun({
changedFiles: input.changedFiles,
coverage: coverageByProvider.get(provider),
full: input.full,
});
});
}
function main(): void {
const full = process.env.FULL === "1";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const changedFiles = full ? [] : readChangedFiles();
const output: MatrixOutput = {
agents: buildAgentsMatrix({ changedFiles, full }),
agnostic: buildAgnosticMatrix({ changedFiles, full }),
flagships: buildFlagshipsMatrix({ changedFiles, full, filter }),
aliases: buildAliasesMatrix({ changedFiles, full, filter }),
};
process.stdout.write(JSON.stringify(output));
}
if (import.meta.url === `file://${process.argv[1]}`) {
main();
}
+14 -2
View File
@@ -28,7 +28,12 @@ config({ path: join(import.meta.dirname, "..", "..", ".env") });
const PROMPT = "Reply with exactly OK and nothing else.";
const MATCH = /\bOK\b/i;
const TIMEOUT_MS = 60_000;
// xai is the slowest provider in the matrix — winning xai/grok-4.3 jobs land
// at 42-67s wall time (vs 23-41s for every other provider), brushing a 60s
// ceiling and intermittently crossing it. 120s gives ~2x headroom on the
// slowest provider observed in CI, with no downside on the fast-path
// providers since the timer only fires on actual hangs.
const TIMEOUT_MS = 120_000;
function parseSlug(): string {
const argIdx = process.argv.indexOf("--slug");
@@ -44,6 +49,11 @@ type Plan =
async function plan(slug: string): Promise<Plan> {
const alias = modelAliases.find((a) => a.slug === slug);
if (!alias) throw new Error(`model-smoke: unknown alias "${slug}"`);
if (alias.routing) {
throw new Error(
`model-smoke: ${slug} is a routing slug (no fixed model). pass an explicit Bedrock model ID via PULLFROG_MODEL or the workflow env block.`
);
}
// walk the fallback chain so deprecated aliases (those with `fallback` set,
// e.g. opencode/mimo-v2-pro-free → opencode/big-pickle) hit their replacement
@@ -73,7 +83,9 @@ async function plan(slug: string): Promise<Plan> {
const cliPath = await installFromNpmTarball({
packageName: "opencode-ai",
version: getDevDependencyVersion("opencode-ai"),
executablePath: "bin/opencode",
// v1.14+: postinstall.mjs renames the platform-specific binary to
// `bin/opencode.exe` for every OS — see action/agents/opencode_v2.ts.
executablePath: "bin/opencode.exe",
installDependencies: true,
});
return {
+91 -11
View File
@@ -1,12 +1,16 @@
import { describe, expect, it } from "vitest";
import { modelAliases } from "../models.ts";
import { modelAliases, resolveDisplayAlias } from "../models.ts";
// ── catalog drift tests — main-only ─────────────────────────────────────────────
// ── catalog drift tests ─────────────────────────────────────────────────────
//
// these tests fetch models.dev and openrouter.ai to verify that every alias in
// models.ts still corresponds to a live, non-deprecated upstream model. upstream
// catalog drift (new model ships, old model deprecated, etc.) causes failures
// that are unrelated to any code change in the PR — so these run only on main.
// that are unrelated to any code change in a typical PR — so these are gated
// off for normal PRs and run only on main pushes plus PRs from the
// `pullfrog/models-bump` branch (the bot-authored bump PR — this test IS the
// integrity gate for its edits, so it has to run on the PR itself, not just
// post-merge).
//
// the registry is kept in sync with upstreams by the `models-bump` cron
// (`.github/workflows/models-bump.yml`), which scans models.dev every 12h and
@@ -15,12 +19,12 @@ import { modelAliases } from "../models.ts";
// for that PR — they catch typos, removed models, and openrouter mismatches.
//
// run locally with `pnpm test:catalog`.
// in CI, gated to push events on main.
type ModelsDevModel = {
name: string;
status?: string;
release_date?: string;
cost?: { input?: number; output?: number };
};
type ModelsDevProvider = {
@@ -41,6 +45,17 @@ describe("models.dev validity", async () => {
const data = await api;
for (const alias of modelAliases) {
// routing slugs (e.g. bedrock/byok) have no fixed `resolve` — the actual
// model ID is read from a separate env var at run time. skip drift checks
// since there's no models.dev entry to validate against.
if (alias.routing) continue;
// aliases with a `fallback` are deprecated entries that legitimately point
// at dead resolve targets — the fallback chain redirects callers to a live
// model. skip both existence and deprecation checks; the terminal-fallback
// is validated separately by the Zen served-list test below.
if (alias.fallback) continue;
const parsed = parseResolve(alias.resolve);
it(`${alias.resolve} exists on models.dev`, () => {
@@ -53,13 +68,11 @@ describe("models.dev validity", async () => {
).toBeDefined();
});
if (!alias.fallback) {
it(`${alias.resolve} is not deprecated`, () => {
const model = data[parsed.provider]?.models[parsed.modelId];
if (!model) return; // covered by existence test above
expect(model.status, `${alias.resolve} is deprecated on models.dev`).not.toBe("deprecated");
});
}
it(`${alias.resolve} is not deprecated`, () => {
const model = data[parsed.provider]?.models[parsed.modelId];
if (!model) return; // covered by existence test above
expect(model.status, `${alias.resolve} is deprecated on models.dev`).not.toBe("deprecated");
});
}
});
@@ -112,3 +125,70 @@ describe("openRouterResolve OpenRouter API validity", async () => {
});
}
});
// ── OpenCode Zen served-list + free-cost checks ────────────────────────────────
//
// these enforce the two dynamic conditions for "this opencode alias works for a
// user without OPENCODE_API_KEY" — the gap that let issue #691 ship:
// 1. the alias's terminal-fallback resolve appears in Zen's /v1/models (Zen
// actually serves it). caught nothing in #691 because mimo had a fallback
// to big-pickle which IS served, but would catch any future alias that
// points at a Zen-removed model without a fallback.
// 2. for isFree aliases, the terminal-fallback's models.dev `cost.input` is
// zero. caught the gpt-5-nano regression: $0.05/M input on models.dev,
// marked isFree in our catalog.
//
// we check the terminal-fallback (via resolveDisplayAlias) because deprecated
// aliases legitimately point at dead resolve targets — the terminal is what
// actually runs at the agent CLI.
type ZenModel = { id: string };
type ZenModelsResponse = { data: ZenModel[] };
const zenApi = fetch("https://opencode.ai/zen/v1/models").then(
(r) => r.json() as Promise<ZenModelsResponse>
);
describe("opencode Zen served list", async () => {
const zenData = await zenApi;
const zenIds = new Set(zenData.data.map((m) => m.id));
const seen = new Set<string>();
for (const alias of modelAliases) {
const terminal = resolveDisplayAlias(alias.slug);
if (!terminal) continue;
const parsed = parseResolve(terminal.resolve);
if (parsed.provider !== "opencode") continue;
if (seen.has(terminal.resolve)) continue;
seen.add(terminal.resolve);
it(`${alias.slug} terminal resolve ${terminal.resolve} is served by Zen`, () => {
expect(
zenIds.has(parsed.modelId),
`terminal resolve "${terminal.resolve}" for alias "${alias.slug}" is not in https://opencode.ai/zen/v1/models — Zen no longer serves it. either point a fallback at a Zen-served alias or remove the entry.`
).toBe(true);
});
}
});
describe("isFree models.dev cost", async () => {
const data = await api;
const seen = new Set<string>();
for (const alias of modelAliases.filter((a) => a.isFree)) {
const terminal = resolveDisplayAlias(alias.slug);
if (!terminal) continue;
const parsed = parseResolve(terminal.resolve);
if (seen.has(terminal.resolve)) continue;
seen.add(terminal.resolve);
it(`${alias.slug} terminal resolve ${terminal.resolve} has cost.input === 0`, () => {
const model = data[parsed.provider]?.models[parsed.modelId];
expect(model, `terminal resolve "${terminal.resolve}" missing on models.dev`).toBeDefined();
expect(
model?.cost?.input,
`isFree alias "${alias.slug}" walks to "${terminal.resolve}" which reports cost.input=${model?.cost?.input} on models.dev — either repoint the fallback or drop \`isFree\``
).toBe(0);
});
}
});
+60 -2
View File
@@ -1,11 +1,12 @@
import { describe, expect, it } from "vitest";
import { modelAliases, resolveCliModel } from "../models.ts";
import { getModelEnvVars, modelAliases, resolveCliModel, resolveDisplayAlias } from "../models.ts";
// ── pure alias-registry invariants ──────────────────────────────────────────────
//
// these tests validate our alias data structure without hitting external APIs.
// network-dependent checks (models.dev / OpenRouter catalog drift, latest-model
// snapshot) live in models-catalog.main.test.ts and run only on main.
// snapshot) live in models-catalog.main.test.ts and run on main pushes plus
// `pullfrog/models-bump` PRs (the bot's bump branch, gated in test.yml).
// models that have no OpenRouter equivalent and require BYOK.
// add a model here ONLY when it genuinely doesn't exist on both models.dev and OpenRouter.
@@ -14,6 +15,10 @@ const BYOK_ONLY_MODELS = new Set(["openai/o3"]);
describe("openRouterResolve completeness", () => {
for (const alias of modelAliases) {
if (alias.isFree) continue;
// routing slugs (e.g. bedrock/byok) are inherently BYOK — there's no
// single model to map to OpenRouter because the actual model ID is read
// from a per-run env var.
if (alias.routing) continue;
if (BYOK_ONLY_MODELS.has(alias.slug)) continue;
it(`${alias.slug} has openRouterResolve`, () => {
expect(
@@ -29,6 +34,13 @@ describe("openRouterResolve completeness", () => {
expect(alias.openRouterResolve).toBeUndefined();
});
}
for (const alias of modelAliases) {
if (!alias.routing) continue;
it(`${alias.slug} (routing slug) has no openRouterResolve`, () => {
expect(alias.openRouterResolve).toBeUndefined();
});
}
});
describe("fallback chain resolution", () => {
@@ -42,3 +54,49 @@ describe("fallback chain resolution", () => {
});
}
});
// ── isFree invariants — sanity-check the catalog data shape ─────────────────────
//
// these catch the latent regressions that produced issue #691:
// - opencode/gpt-5-nano was marked `isFree` despite costing $0.05/M
// (no static check existed; demoted to paid in the same PR adding these tests)
// - opencode/mimo-v2-pro-free was free + fallback to big-pickle (correct shape),
// but nothing enforced that the terminal of an isFree fallback chain is itself
// free. if someone repointed big-pickle's fallback at a paid model, all of mimo
// and big-pickle's users would silently start hitting a paid endpoint.
//
// the cost.input check itself is network-dependent (lives in
// models-catalog.main.test.ts); these are the static sibling that runs on every PR.
describe("isFree invariants", () => {
for (const alias of modelAliases.filter((a) => a.isFree)) {
it(`${alias.slug} lives under the opencode provider`, () => {
expect(
alias.provider,
`isFree alias "${alias.slug}" must be under "opencode" (Zen's keyless gate is opencode-only)`
).toBe("opencode");
});
it(`${alias.slug} has empty envVars`, () => {
expect(
getModelEnvVars(alias.slug),
`isFree alias "${alias.slug}" must declare \`envVars: []\` so validateAgentApiKey doesn't demand OPENCODE_API_KEY`
).toEqual([]);
});
it(`${alias.slug} has no openRouterResolve`, () => {
expect(
alias.openRouterResolve,
`isFree alias "${alias.slug}" must omit \`openRouterResolve\` — free Zen models don't exist on OpenRouter`
).toBeUndefined();
});
it(`${alias.slug} fallback chain terminates at an isFree alias`, () => {
const terminal = resolveDisplayAlias(alias.slug);
expect(terminal, `fallback chain for "${alias.slug}" is broken`).toBeDefined();
expect(
terminal?.isFree,
`isFree alias "${alias.slug}" walks to "${terminal?.slug}" which is NOT isFree — users would silently start paying`
).toBe(true);
});
}
});
+89
View File
@@ -0,0 +1,89 @@
/**
* provider catalog the source of truth for `providers-live` (full harness
* smoke per provider) and the per-provider coverage globs that scope `models-live`
* (per-alias CLI smoke).
*
* each entry pins one standard-tier flagship slug per provider not the
* pro/opus tier (too expensive for per-push) and not the free/experimental
* tier (too flaky). these flagships catch provider-class regressions like
* Gemini schema sanitization or OpenAI tool-call format drift that the cheap
* per-alias CLI smoke can't see.
*
* `coverage` lists the source files that, when changed, should rerun this
* provider's flagship + every alias of this provider. `action/models.ts` is
* included on every entry touching the resolution table reruns all model
* tests (simple model; matches the per-PR-precision answer from planning).
*
* adding a new provider:
* 1. add an entry here with the flagship slug, agent harness, coverage globs
* 2. add a row to wiki/models-catalog.md "To add a provider"
* 3. CI picks it up automatically no workflow change
*/
export type ProviderEntry = {
name: string;
/** flagship slug for `providers-live` full-harness smoke. */
flagship: string;
/** harness used by the runtime for this provider's models. */
agent: "claude" | "opencode";
/** repo-relative globs that invalidate this provider's matrix entries. */
coverage: string[];
};
const SHARED_OPENCODE_COVERAGE = [
"action/models.ts",
"action/agents/opencode.ts",
"action/agents/opencode_v2.ts",
"action/agents/opencodePlugin.ts",
];
export const providers: ProviderEntry[] = [
{
name: "anthropic",
flagship: "anthropic/claude-sonnet",
agent: "claude",
coverage: ["action/models.ts", "action/agents/claude.ts"],
},
{
name: "openai",
flagship: "openai/gpt",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "google",
flagship: "google/gemini-pro",
agent: "opencode",
coverage: [...SHARED_OPENCODE_COVERAGE, "action/mcp/geminiSanitizer.ts"],
},
{
name: "xai",
flagship: "xai/grok",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "deepseek",
flagship: "deepseek/deepseek-pro",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "moonshotai",
flagship: "moonshotai/kimi-k2",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "opencode",
flagship: "opencode/big-pickle",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
{
name: "openrouter",
flagship: "openrouter/claude-sonnet",
agent: "opencode",
coverage: SHARED_OPENCODE_COVERAGE,
},
];
+34 -41
View File
@@ -2,9 +2,6 @@ import { existsSync, readdirSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { config } from "dotenv";
import { runInDocker } from "../utils/docker.ts";
import { ensureGitHubToken } from "../utils/github.ts";
import { isInsideDocker } from "../utils/globals.ts";
import { killTrackedChildren, setSignalHandler } from "../utils/subprocess.ts";
import {
type AgentResult,
@@ -22,32 +19,33 @@ import {
/**
* unified test runner for all agent tests.
*
* usage: node test/run.ts [filters...]
* invoke from the repo root:
* pnpm runtest [filters] # host, in-process fast iteration (default)
* pnpm runtest:docker [filters] # local docker container that mocks GHA
* pnpm docker test/run.ts [filters] # explicit container form (equivalent to `pnpm runtest:docker`)
*
* filters can be test names, tags, or agent names:
* node test/run.ts # run all tests (excludes adhoc-tagged tests)
* node test/run.ts smoke # run tests named "smoke" or tagged "smoke"
* node test/run.ts opencode # run all tests for opencode only
* node test/run.ts security # run all tests tagged "security"
* node test/run.ts agnostic # run all agnostic-tagged tests (with opencode)
* node test/run.ts adhoc # run all adhoc-tagged tests
* node test/run.ts smoke opencode # run smoke tests for opencode only
* pnpm runtest # run all tests (excludes adhoc-tagged tests)
* pnpm runtest smoke # run tests named "smoke" or tagged "smoke"
* pnpm runtest opencode # run all tests for opencode only
* pnpm runtest security # run all tests tagged "security"
* pnpm runtest agnostic # run all agnostic-tagged tests (with opencode)
* pnpm runtest adhoc # run all adhoc-tagged tests
* pnpm runtest smoke opencode # run smoke tests for opencode only
*
* special tags:
* - "agnostic": runs with opencode only, excluded when filtering by agent
* - "adhoc": excluded from default runs, must be explicitly requested
*
* by default, runs in a Docker container for isolation.
* see wiki/docker.md for when host vs container matters.
*/
const __dirname = dirname(fileURLToPath(import.meta.url));
export const actionDir = join(__dirname, "..");
// load .env files
config({ path: join(actionDir, ".env") });
config({ path: join(actionDir, "..", ".env") });
const nodeModulesVolume = "pullfrog-action-test-node-modules";
const mcpPortBase = 49000;
let nextMcpPort = mcpPortBase;
@@ -57,25 +55,6 @@ function allocateMcpPort(): number {
return port;
}
function buildNodeCmd(args: string[]): string {
const passArgs = args.map((arg) => `'${arg.replace(/'/g, "'\\''")}'`).join(" ");
return `node test/run.ts ${passArgs}`;
}
// run the test runner inside docker
function runTestsInDocker(args: string[]): never {
const result = runInDocker({
actionDir,
args,
nodeCmd: buildNodeCmd(args),
volumeName: nodeModulesVolume,
envFilterMode: "allowlist",
onStart: () => console.log("» running tests in docker container...\n"),
});
process.exit(result.status ?? 1);
}
type TestInfo = {
name: string;
config: TestRunnerOptions;
@@ -282,6 +261,28 @@ function shouldRetry(result: AgentResult, validation: ValidationResult): RetryDe
async function runTestForAgent(ctx: RunContext): Promise<ValidationResult> {
const testConfig = ctx.testInfo.config;
// runtime-evaluated skip: gate on env (e.g. CODEX_AUTH_JSON for codex-auth).
// skipped runs short-circuit before any agent spawn AND count as passing so
// a missing optional secret doesn't fail-fast cancel the rest of the matrix.
const skipReason = testConfig.skipIf?.();
if (skipReason) {
const prefix = getPrefix({ test: ctx.testInfo.name, agent: ctx.agent });
console.log(`${prefix} ⏭ skipped: ${skipReason}`);
const skipped: ValidationResult = {
test: ctx.testInfo.name,
agent: ctx.agent,
passed: true,
canceled: false,
checks: [],
output: `skipped: ${skipReason}`,
skipped: true,
skipReason,
};
ctx.results.set(getRunKey(ctx.testInfo.name, ctx.agent), skipped);
return skipped;
}
const env: Record<string, string> = {};
if (testConfig.env) {
const entries = Object.entries(testConfig.env);
@@ -393,14 +394,6 @@ async function runTestForAgent(ctx: RunContext): Promise<ValidationResult> {
async function main(): Promise<void> {
const args = process.argv.slice(2);
// run in Docker unless already inside
if (!isInsideDocker) {
// acquire token for docker if needed
await ensureGitHubToken();
runTestsInDocker(args);
}
// load all tests
const allTests = await loadAllTests();
const parsed = parseArgs(args, allTests);
+8
View File
@@ -28,6 +28,14 @@ const results: { model: string; status: "pass" | "fail" | "skip"; detail?: strin
const seen = new Set<string>();
for (const alias of modelAliases) {
// routing slugs (bedrock/byok) have no fixed `resolve` to test against —
// the model ID is supplied at run time via a per-run env var. skipping
// here matches the bumps cron + catalog drift test.
if (alias.routing) {
results.push({ model: alias.slug, status: "skip", detail: "routing slug (no fixed resolve)" });
continue;
}
if (seen.has(alias.resolve)) {
results.push({ model: alias.resolve, status: "skip", detail: "duplicate resolve" });
continue;
+27 -4
View File
@@ -152,6 +152,8 @@ export interface ValidationResult {
canceled: boolean;
checks: ValidationCheck[];
output: string;
skipped?: boolean;
skipReason?: string;
}
export type ValidatorFn = (result: AgentResult) => ValidationCheck[];
@@ -328,6 +330,16 @@ export interface TestRunnerOptions {
// - "agnostic": runs with opencode only, excluded when filtering by agent
// - "adhoc": excluded from default runs, must be explicitly requested
tags?: TestTag[];
// repo-relative globs of source files that, when changed in a PR, should
// trigger this test in CI. omit to opt out of filtering (test always runs
// — the defensive default). see action/test/coverage.ts.
coverage?: string[];
/** evaluated at test-runtime (after `pnpm install`, before agent spawn).
* return a non-empty reason string to skip the test entirely the runner
* records a passing-with-skipped result so the matrix doesn't fail-fast
* cancel the rest of the jobs. used to gate tests on optional secrets
* (e.g. codex-auth needs `CODEX_AUTH_JSON`, which forks won't have). */
skipIf?: () => string | null;
}
export type TestTag = "adhoc" | "agnostic" | "security";
@@ -336,8 +348,9 @@ export function printSingleValidation(validation: ValidationResult): void {
const checksStr = validation.checks.map((c) => `${c.name}=${c.passed ? "✓" : "✗"}`).join(" ");
const color = AGENT_COLORS[validation.agent] ?? "";
const canceledNote = validation.canceled ? " (canceled)" : "";
const skippedNote = validation.skipped ? ` (skipped: ${validation.skipReason ?? ""})` : "";
console.log(
`\n${color}[${validation.test}][${validation.agent}]${RESET} ${checksStr}${canceledNote}`
`\n${color}[${validation.test}][${validation.agent}]${RESET} ${checksStr}${canceledNote}${skippedNote}`
);
}
@@ -349,8 +362,16 @@ export function printResults(validations: ValidationResult[]): void {
for (const v of validations) {
const color = AGENT_COLORS[v.agent] ?? "";
const status = v.canceled ? "❌ canceled" : v.passed ? "✅ pass" : "❌ fail";
const checkCols = v.checks.map((c) => `${c.name}=${c.passed ? "✓" : "✗"}`).join(" ");
const status = v.canceled
? "❌ canceled"
: v.skipped
? "⏭ skipped"
: v.passed
? "✅ pass"
: "❌ fail";
const checkCols = v.skipped
? `(skipped: ${v.skipReason ?? ""})`
: v.checks.map((c) => `${c.name}=${c.passed ? "✓" : "✗"}`).join(" ");
console.log(
`${status} ${v.test.padEnd(12)} ${color}${v.agent.padEnd(10)}${RESET} ${checkCols}`
);
@@ -358,5 +379,7 @@ export function printResults(validations: ValidationResult[]): void {
console.log("-".repeat(70));
const passed = validations.filter((v) => v.passed);
console.log(`\n${passed.length}/${validations.length} passed`);
const skipped = validations.filter((v) => v.skipped).length;
const skippedNote = skipped > 0 ? ` (${skipped} skipped)` : "";
console.log(`\n${passed.length}/${validations.length} passed${skippedNote}`);
}
+18
View File
@@ -1,5 +1,6 @@
import type { AgentUsage } from "./agents/shared.ts";
import type { PrepResult } from "./prep/types.ts";
import type { AgentDiagnostic } from "./utils/agentHangReport.ts";
import { log } from "./utils/cli.ts";
import type { DiffCoverageState } from "./utils/diffCoverage.ts";
import {
@@ -88,6 +89,10 @@ export interface ToolState {
// then from checkoutSha when review.ts detects new commits mid-review
beforeSha?: string;
selectedMode?: string;
// number of prepush hook failures this run. push_branch runs the hook
// while this is 0 and skips it once non-zero; never decremented within
// a run.
prepushFailureCount: number;
backgroundProcesses: Map<string, BackgroundProcess>;
browserDaemon?: BrowserDaemon | undefined;
review?: {
@@ -155,8 +160,20 @@ export interface ToolState {
output?: string;
usageEntries: AgentUsage[];
model?: string | undefined;
// set by main.ts when the BYOK fallback engaged (configured model needed
// a provider key the runner didn't have). carried into PR-comment footers
// so users can see "Using <free model> (credentials for <configured> not
// configured)" rather than just being silently downgraded. literal record
// of an event that happened — matches the ToolState design rule.
modelFallback?: { from: string } | undefined;
todoTracker?: TodoTracker | undefined;
diffCoverage?: DiffCoverageState | undefined;
// mutable handle the agent harness writes to as a run progresses (recent
// stderr ring buffer reference, last provider-error label, event count).
// read by main.ts's outer catch so a watchdog-fired activity timeout still
// surfaces the same agent-side context the harness's own catch path returns
// via `result.error`. see `utils/agentHangReport.ts`.
agentDiagnostic?: AgentDiagnostic | undefined;
}
interface InitToolStateParams {
@@ -173,6 +190,7 @@ export function initToolState(params: InitToolStateParams): ToolState {
return {
progressComment: resolved,
hadProgressComment: !!resolved,
prepushFailureCount: 0,
backgroundProcesses: new Map(),
usageEntries: [],
};
+3 -2
View File
@@ -19,7 +19,8 @@
"forceConsistentCasingInFileNames": true,
"stripInternal": true,
"moduleDetection": "force",
"useUnknownInCatchVariables": true
"useUnknownInCatchVariables": true,
"noEmit": true
},
"exclude": []
"exclude": ["dist"]
}
+58 -1
View File
@@ -1,4 +1,5 @@
import { performance } from "node:perf_hooks";
import { log } from "./log.ts";
function isMonitorDebugEnabled(): boolean {
return (
@@ -79,6 +80,19 @@ type WriteFunction = {
// module-level activity tracking - allows agents to mark activity on any event
let _lastActivity = performance.now();
/**
* upper bound on how long a single tool call can suspend the activity
* watchdog. matched against the typical worst-case `checkout_pr`
* fetch+deepen on a large monorepo (issue #760: 4-5min) plus generous
* headroom for slower MCP tools, while still bounding the worst case if
* a tool genuinely hangs and `tool_result` never arrives auto-resume
* fires here and the normal idle clock takes over from a fresh baseline.
*/
export const MAX_TOOL_CALL_SUSPENSION_MS = 15 * 60 * 1000;
let _suspendedAt: number | null = null;
let _suspensionTimer: NodeJS.Timeout | null = null;
/**
* mark activity to reset the no-output timeout.
* call this whenever the agent emits any event, even if it isn't logged to stdout.
@@ -88,12 +102,55 @@ export function markActivity(): void {
}
/**
* get the time since last activity in milliseconds
* get the time since last activity in milliseconds.
* returns 0 while the watchdog is suspended (issue #760).
*/
export function getIdleMs(): number {
if (_suspendedAt !== null) return 0;
return Math.round(performance.now() - _lastActivity);
}
/**
* suspend the activity watchdog while a long-running, in-flight unit of
* work is happening (e.g. an MCP `tools/call` that synchronously awaits
* a multi-minute git fetch). bracket calls with `resumeActivity()` from
* the agent harness's `tool_use` / `tool_result` event handlers.
*
* - idempotent: nested suspends are no-ops; the first resume wins.
* - bounded: auto-resumes after `maxMs` so a buggy tool that never
* produces a `tool_result` can't pin the watchdog open forever.
* - safe: only the *agent harness* (claude.ts / opencode.ts) on explicit,
* paired CLI events should call this. NEVER blanket-suspend on internal
* noise that would resurrect issue #12 zombie runs.
*/
export function suspendActivity(maxMs: number = MAX_TOOL_CALL_SUSPENSION_MS): void {
if (_suspendedAt !== null) return;
_suspendedAt = performance.now();
_suspensionTimer = setTimeout(() => {
log.warning(`activity watchdog suspended >${Math.round(maxMs / 1000)}s — auto-resuming`);
resumeActivity();
}, maxMs);
_suspensionTimer.unref?.();
}
/**
* resume the activity watchdog. resets the idle baseline so a stale
* idle window before the suspend can't immediately re-fire.
*/
export function resumeActivity(): void {
if (_suspendedAt === null) return;
_suspendedAt = null;
if (_suspensionTimer) {
clearTimeout(_suspensionTimer);
_suspensionTimer = null;
}
_lastActivity = performance.now();
}
export function isActivitySuspended(): boolean {
return _suspendedAt !== null;
}
function wrapWrite(original: WriteFunction, onActivity: () => void): WriteFunction {
const wrapped: WriteFunction = (
chunk: string | Uint8Array,
+127 -5
View File
@@ -1,9 +1,131 @@
import { describe, expect, it } from "vitest";
import { resolveAgent } from "./agent.ts";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { resolveAgent, resolveModel } from "./agent.ts";
const savedEnv = { ...process.env };
const STRIPPED = [
/_API_KEY$/,
/^CLAUDE_CODE_OAUTH_TOKEN$/,
/^AWS_BEARER_TOKEN_BEDROCK$/,
/^AWS_ACCESS_KEY_ID$/,
/^AWS_SECRET_ACCESS_KEY$/,
/^AWS_SESSION_TOKEN$/,
/^AWS_REGION$/,
/^BEDROCK_MODEL_ID$/,
/^PULLFROG_MODEL$/,
/^PULLFROG_AGENT$/,
];
beforeEach(() => {
for (const key of Object.keys(process.env)) {
if (STRIPPED.some((re) => re.test(key))) delete process.env[key];
}
});
afterEach(() => {
process.env = { ...savedEnv };
});
describe("resolveAgent", () => {
it("returns opencode", () => {
const agent = resolveAgent({});
expect(agent.name).toBe("opencode");
it("returns opencode by default", () => {
expect(resolveAgent({}).name).toBe("opencode");
});
it("routes anthropic/* to claude when ANTHROPIC_API_KEY is set", () => {
process.env.ANTHROPIC_API_KEY = "sk-test";
expect(resolveAgent({ model: "anthropic/claude-opus-4-7" }).name).toBe("claude");
});
it("falls back to opencode for anthropic/* without claude-code creds", () => {
expect(resolveAgent({ model: "anthropic/claude-opus-4-7" }).name).toBe("opencode");
});
describe("bedrock routing", () => {
it("routes Anthropic Bedrock IDs to claude", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveAgent({ model: "us.anthropic.claude-opus-4-7" }).name).toBe("claude");
});
it("routes Anthropic Bedrock IDs (no region prefix) to claude", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "anthropic.claude-haiku-4-5-20251001-v1:0";
expect(resolveAgent({ model: "anthropic.claude-haiku-4-5-20251001-v1:0" }).name).toBe(
"claude"
);
});
it("routes non-Anthropic Bedrock IDs to opencode", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "amazon.nova-pro-v1:0";
expect(resolveAgent({ model: "amazon.nova-pro-v1:0" }).name).toBe("opencode");
});
it("routes Llama IDs to opencode", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.meta.llama4-scout-17b-instruct-v1:0";
expect(resolveAgent({ model: "us.meta.llama4-scout-17b-instruct-v1:0" }).name).toBe(
"opencode"
);
});
it("accepts AWS access keys as auth", () => {
process.env.AWS_ACCESS_KEY_ID = "AKIA-test";
process.env.AWS_SECRET_ACCESS_KEY = "secret-test";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveAgent({ model: "us.anthropic.claude-opus-4-7" }).name).toBe("claude");
});
it("PULLFROG_AGENT override wins over Anthropic auto-routing", () => {
process.env.PULLFROG_AGENT = "opencode";
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveAgent({ model: "us.anthropic.claude-opus-4-7" }).name).toBe("opencode");
});
});
});
describe("resolveModel", () => {
it("PULLFROG_MODEL override wins", () => {
process.env.PULLFROG_MODEL = "anthropic/claude-opus";
expect(resolveModel({ slug: "openai/gpt" })).toBe("anthropic/claude-opus-4-7");
});
it("PULLFROG_MODEL bypasses bedrock routing entirely", () => {
process.env.PULLFROG_MODEL = "openai/gpt";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveModel({ slug: "bedrock/byok" })).toBe("openai/gpt-5.5");
});
it("resolves bedrock/byok to BEDROCK_MODEL_ID", () => {
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveModel({ slug: "bedrock/byok" })).toBe("us.anthropic.claude-opus-4-7");
});
it("throws when bedrock/byok is selected without BEDROCK_MODEL_ID", () => {
expect(() => resolveModel({ slug: "bedrock/byok" })).toThrow("BEDROCK_MODEL_ID");
});
it("returns the alias resolve for normal slugs", () => {
expect(resolveModel({ slug: "openai/gpt" })).toBe("openai/gpt-5.5");
});
it("returns undefined for no slug + no PULLFROG_MODEL", () => {
expect(resolveModel({})).toBeUndefined();
});
// regression: PR #720 review caught that `resolveCliModel("bedrock/byok")`
// returns the literal sentinel `"bedrock"` from the alias's `resolve`
// field. Without routing-aware handling, PULLFROG_MODEL=bedrock/byok would
// leak that sentinel downstream and break agent dispatch.
it("PULLFROG_MODEL=bedrock/byok defers to BEDROCK_MODEL_ID, not the sentinel", () => {
process.env.PULLFROG_MODEL = "bedrock/byok";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(resolveModel({ slug: "openai/gpt" })).toBe("us.anthropic.claude-opus-4-7");
});
it("PULLFROG_MODEL=bedrock/byok throws if BEDROCK_MODEL_ID is missing", () => {
process.env.PULLFROG_MODEL = "bedrock/byok";
expect(() => resolveModel({ slug: "openai/gpt" })).toThrow("BEDROCK_MODEL_ID");
});
});
+55 -7
View File
@@ -1,6 +1,12 @@
import type { Agent } from "../agents/index.ts";
import { agents } from "../agents/index.ts";
import { getModelProvider, resolveCliModel } from "../models.ts";
import {
BEDROCK_MODEL_ID_ENV,
getModelProvider,
isBedrockAnthropicId,
resolveCliModel,
resolveDisplayAlias,
} from "../models.ts";
import { log } from "./cli.ts";
function hasEnvVar(name: string): boolean {
@@ -12,6 +18,37 @@ function hasClaudeCodeAuth(): boolean {
return hasEnvVar("CLAUDE_CODE_OAUTH_TOKEN") || hasEnvVar("ANTHROPIC_API_KEY");
}
function hasBedrockAuth(): boolean {
return (
hasEnvVar("AWS_BEARER_TOKEN_BEDROCK") ||
(hasEnvVar("AWS_ACCESS_KEY_ID") && hasEnvVar("AWS_SECRET_ACCESS_KEY"))
);
}
/**
* resolve a single slug to its CLI-ready model string. routing aliases
* (e.g. `bedrock/byok`) defer to their backing env var instead of the
* sentinel stored in `resolve`. shared between PULLFROG_MODEL override
* and repo-config slug resolution so both paths get the same routing
* semantics without this helper, `PULLFROG_MODEL=bedrock/byok` would
* leak the literal sentinel string `"bedrock"` downstream.
*/
function resolveSlug(slug: string): string | undefined {
const alias = resolveDisplayAlias(slug);
if (alias?.routing === "bedrock") {
const bedrockId = process.env[BEDROCK_MODEL_ID_ENV]?.trim();
if (!bedrockId) {
throw new Error(
`${BEDROCK_MODEL_ID_ENV} env var is required when the model is set to "${slug}". ` +
`set it to an AWS Bedrock model ID (e.g. "us.anthropic.claude-opus-4-7", "amazon.nova-pro-v1:0"). ` +
`see https://docs.pullfrog.com/bedrock for setup.`
);
}
return bedrockId;
}
return resolveCliModel(slug);
}
/**
* resolve the effective model for this run.
*
@@ -19,17 +56,20 @@ function hasClaudeCodeAuth(): boolean {
* 1. PULLFROG_MODEL env var resolved through the alias registry first,
* so values like "anthropic/claude-opus" become "anthropic/claude-opus-4-7".
* raw specifiers (e.g. "anthropic/claude-opus-4-6") pass through unchanged.
* 2. slug from repo config / payload alias registry
* 3. undefined agent will auto-select
* always wins bypasses Bedrock routing entirely. to test a different
* Bedrock model, change `BEDROCK_MODEL_ID`, not `PULLFROG_MODEL`.
* 2. slug from repo config / payload alias registry. routing slugs
* (e.g. `bedrock/byok`) defer to a separate env var (`BEDROCK_MODEL_ID`).
* 3. undefined agent will auto-select.
*/
export function resolveModel(ctx: { slug?: string | undefined }): string | undefined {
const envModel = process.env.PULLFROG_MODEL?.trim();
if (envModel) {
return resolveCliModel(envModel) ?? envModel;
return resolveSlug(envModel) ?? envModel;
}
if (ctx.slug) {
const resolved = resolveCliModel(ctx.slug);
const resolved = resolveSlug(ctx.slug);
if (resolved) {
return resolved;
}
@@ -49,7 +89,15 @@ export function resolveAgent(ctx: { model?: string | undefined }): Agent {
log.warning(`» unknown PULLFROG_AGENT="${envAgent}" — falling through to auto-select`);
}
// 2. if model is Anthropic and Claude Code credentials are available, use Claude Code
// 2. Bedrock routing: when BEDROCK_MODEL_ID is the resolved model, route
// Anthropic IDs through claude-code (which supports Bedrock natively
// once CLAUDE_CODE_USE_BEDROCK=1) and everything else through opencode's
// `amazon-bedrock` provider.
if (ctx.model && hasBedrockAuth() && process.env[BEDROCK_MODEL_ID_ENV]?.trim() === ctx.model) {
return isBedrockAnthropicId(ctx.model) ? agents.claude : agents.opencode;
}
// 3. if model is Anthropic and Claude Code credentials are available, use Claude Code
if (ctx.model) {
try {
const provider = getModelProvider(ctx.model);
@@ -61,6 +109,6 @@ export function resolveAgent(ctx: { model?: string | undefined }): Agent {
}
}
// 3. default: OpenCode (universal, supports all providers)
// 4. default: OpenCode (universal, supports all providers)
return agents.opencode;
}
+170
View File
@@ -0,0 +1,170 @@
const MAX_STDERR_BYTES = 3000;
/**
* mutable per-run handle the agent harness writes to as a run progresses.
* the action's outer try/catch in `main.ts` reads this off `toolState` when
* the activity-timeout watchdog wins the race against the harness's own
* catch the bare timer reject reason ("activity timeout: no output for
* 302s") tells the user nothing actionable, but `recentStderr` +
* `lastProviderError` together usually point straight at the upstream cause.
*
* `recentStderr` is shared by reference with the harness's bounded ring
* buffer, so the diagnostic always reflects the latest captured tail.
*/
export type AgentDiagnostic = {
/** display label for the agent, e.g. "Pullfrog". used in the headline. */
label: string;
/** shared reference to the harness's bounded stderr ring buffer. */
recentStderr: string[];
/** most-recent provider-error label from `detectProviderError`, if any. */
lastProviderError: string | undefined;
/** count of stdout events successfully parsed before the failure. */
eventCount: number;
};
/**
* Build a user-facing markdown body for an agent hang or failure.
*
* Rendered into both the PR progress comment and the GitHub Actions job
* summary. Returns `null` when no diagnostic is available, which signals to
* the caller to fall back to its bare-error rendering.
*
* `errorMessage` is the underlying timer / spawn reject string (e.g.
* `activity timeout: no output for 301s`). The idle seconds are parsed out
* of it for the hang explanation total runtime would overstate the stall
* for runs that streamed for a long time before going quiet.
*/
export function formatAgentHangBody(input: {
diagnostic: AgentDiagnostic | undefined;
isHang: boolean;
errorMessage: string;
}): string | null {
if (!input.diagnostic) return null;
// billing exhaustion (CreditsError / FreeUsageLimitError / spending cap /
// Insufficient balance) is mis-classified as transient by upstream harnesses
// and the run only ends when the activity-timeout watchdog fires (see #778).
// when we recognise the billing label, replace the generic "stalled — auth
// error" headline with a billing-specific CTA that names the actual remedy.
if (input.diagnostic.lastProviderError === "provider billing exhausted") {
return formatBillingExhaustedBody(input.diagnostic);
}
const verb = input.isHang ? "stalled" : "failed";
const cause = input.diagnostic.lastProviderError
? ` — likely cause: \`${input.diagnostic.lastProviderError}\``
: "";
const headline = `**${input.diagnostic.label} ${verb}**${cause}`;
const explanation = formatExplanation({
isHang: input.isHang,
errorMessage: input.errorMessage,
});
const parts = [headline, "", `${explanation} ${formatEventsPart(input.diagnostic)}`];
const tail = renderStderrTail(input.diagnostic.recentStderr);
if (tail) {
// pick a fence longer than any backtick run in the body so a stderr line
// containing ``` (provider error JSON occasionally embeds it) can't
// terminate the fence early and corrupt the rest of the markdown.
const fence = pickFence(tail);
parts.push(
"",
"<details><summary>Recent agent stderr</summary>",
"",
fence,
tail,
fence,
"",
"</details>"
);
}
return parts.join("\n");
}
function formatExplanation(input: { isHang: boolean; errorMessage: string }): string {
if (!input.isHang) return `The agent exited unexpectedly: ${input.errorMessage}`;
const idleSec = parseIdleSec(input.errorMessage);
if (idleSec === undefined) {
return "The agent stopped emitting events and was killed by the activity-timeout watchdog.";
}
return `The agent stopped emitting events for ${idleSec}s and was killed by the activity-timeout watchdog.`;
}
function parseIdleSec(message: string): number | undefined {
const match = /no output for (\d+)s/.exec(message);
return match ? Number(match[1]) : undefined;
}
function formatEventsPart(diagnostic: AgentDiagnostic): string {
if (diagnostic.eventCount > 0) {
return `${diagnostic.eventCount} events were processed before the failure.`;
}
// when the provider-error label already names the cause in the headline,
// the reachability nudge below contradicts it (e.g. an immediate 401 also
// produces zero events but isn't a reachability problem). suppress it.
if (diagnostic.lastProviderError) return "No events were emitted before the failure.";
return "No events were emitted — check whether the model provider is reachable.";
}
function renderStderrTail(lines: readonly string[]): string {
if (lines.length === 0) return "";
const joined = lines.join("\n");
if (joined.length <= MAX_STDERR_BYTES) return joined;
return `... (older lines truncated)\n${joined.slice(-MAX_STDERR_BYTES)}`;
}
function pickFence(content: string): string {
let max = 0;
for (const match of content.matchAll(/`+/g)) {
if (match[0].length > max) max = match[0].length;
}
return "`".repeat(Math.max(3, max + 1));
}
/**
* Pull a billing URL out of the captured stderr if the provider helpfully
* embedded one (OpenCode Zen does Anthropic and Gemini do not). Restricted
* to known billing/console hosts so a stray URL elsewhere in the buffer
* can't masquerade as the remedy link.
*/
function extractBillingUrl(lines: readonly string[]): string | undefined {
const urlPattern =
/https:\/\/(?:opencode\.ai\/[^\s"]*billing[^\s"]*|console\.anthropic\.com[^\s"]*|console\.cloud\.google\.com[^\s"]*billing[^\s"]*)/i;
for (let i = lines.length - 1; i >= 0; i--) {
const m = urlPattern.exec(lines[i] ?? "");
if (m) return m[0];
}
return undefined;
}
function formatBillingExhaustedBody(diagnostic: AgentDiagnostic): string {
const headline = `**${diagnostic.label} stopped** — your model provider returned a billing-exhausted response.`;
const billingUrl = extractBillingUrl(diagnostic.recentStderr);
const cta = billingUrl
? `Top up your provider balance, then re-run: [${billingUrl}](${billingUrl})`
: "Top up your model-provider balance (or rotate to a key with remaining credits) and re-run.";
const explanation =
"The agent kept retrying the request because the provider marked the failure as transient. Pullfrog's activity-timeout watchdog ended the run after no further events were emitted.";
const parts = [headline, "", explanation, "", cta];
const tail = renderStderrTail(diagnostic.recentStderr);
if (tail) {
const fence = pickFence(tail);
parts.push(
"",
"<details><summary>Recent agent stderr</summary>",
"",
fence,
tail,
fence,
"",
"</details>"
);
}
return parts.join("\n");
}
+12
View File
@@ -37,6 +37,18 @@ export async function apiFetch(options: ApiFetchOptions): Promise<Response> {
headers["x-vercel-protection-bypass"] = bypassSecret;
}
// never send Content-Type on body-less requests. empirically, Vercel's
// Next.js lambda adapter (Next 16.1.x) throws `SyntaxError: Unexpected
// end of data` before delegating to the route handler — returning a 500 —
// when Content-Type is set but no body is present. exact mechanism is
// unverified (minified runtime frame), but Content-Type on a body-less
// request has no defined semantics per RFC 9110 §8.3 anyway. see #692.
if (!options.body) {
for (const key of Object.keys(headers)) {
if (key.toLowerCase() === "content-type") delete headers[key];
}
}
log.debug(`api fetch: ${options.method ?? "GET"} ${url.pathname}`);
const init: RequestInit = {
+155 -8
View File
@@ -1,5 +1,5 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { validateAgentApiKey } from "./apiKeys.ts";
import { formatApiKeyErrorSummary, isApiKeyAuthError, validateAgentApiKey } from "./apiKeys.ts";
const base = {
agent: { name: "opencode" },
@@ -9,10 +9,24 @@ const base = {
const savedEnv = { ...process.env };
// keys that count as provider auth in `knownApiKeys` and would let the
// auto-select path pass without our intent. strip all of them at test setup
// so each `it` starts from a clean slate regardless of what's in the dev `.env`.
const STRIPPED_PREFIXES_OR_NAMES = [
/_API_KEY$/,
/^CLAUDE_CODE_OAUTH_TOKEN$/,
/^CODEX_AUTH_JSON$/,
/^AWS_BEARER_TOKEN_BEDROCK$/,
/^AWS_ACCESS_KEY_ID$/,
/^AWS_SECRET_ACCESS_KEY$/,
/^AWS_SESSION_TOKEN$/,
/^AWS_REGION$/,
/^BEDROCK_MODEL_ID$/,
];
beforeEach(() => {
// strip all known provider keys so tests start clean
for (const key of Object.keys(process.env)) {
if (key.endsWith("_API_KEY") || key === "CLAUDE_CODE_OAUTH_TOKEN") delete process.env[key];
if (STRIPPED_PREFIXES_OR_NAMES.some((re) => re.test(key))) delete process.env[key];
}
});
@@ -27,11 +41,7 @@ describe("validateAgentApiKey", () => {
});
it("passes for other free opencode models", () => {
for (const slug of [
"opencode/gpt-5-nano",
"opencode/mimo-v2-pro-free",
"opencode/minimax-m2.5-free",
]) {
for (const slug of ["opencode/mimo-v2-pro-free", "opencode/minimax-m2.5-free"]) {
expect(() => validateAgentApiKey({ ...base, model: slug })).not.toThrow();
}
});
@@ -59,6 +69,12 @@ describe("validateAgentApiKey", () => {
"no API key found"
);
});
it("throws for opencode/gpt-5-nano without OPENCODE_API_KEY (paid Zen alias)", () => {
expect(() => validateAgentApiKey({ ...base, model: "opencode/gpt-5-nano" })).toThrow(
"no API key found"
);
});
});
describe("no model (auto-select)", () => {
@@ -71,4 +87,135 @@ describe("validateAgentApiKey", () => {
expect(() => validateAgentApiKey({ ...base, model: undefined })).toThrow("no API key found");
});
});
describe("bedrock routing slug", () => {
it("passes with AWS_BEARER_TOKEN_BEDROCK + AWS_REGION + BEDROCK_MODEL_ID", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).not.toThrow();
});
it("passes with AWS access keys + region + model id", () => {
process.env.AWS_ACCESS_KEY_ID = "AKIA-test";
process.env.AWS_SECRET_ACCESS_KEY = "secret-test";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "amazon.nova-pro-v1:0";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).not.toThrow();
});
it("throws when BEDROCK_MODEL_ID is missing", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.AWS_REGION = "us-east-1";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow(
"BEDROCK_MODEL_ID"
);
});
it("throws when AWS_REGION is missing", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow("AWS_REGION");
});
it("throws when no auth is set", () => {
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow(
"AWS_BEARER_TOKEN_BEDROCK"
);
});
it("throws when only AWS_ACCESS_KEY_ID is set (missing secret)", () => {
process.env.AWS_ACCESS_KEY_ID = "AKIA-test";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-7";
expect(() => validateAgentApiKey({ ...base, model: "bedrock/byok" })).toThrow(
"AWS_BEARER_TOKEN_BEDROCK"
);
});
// regression: main.ts passes the resolved model into validateAgentApiKey
// (`payload.proxyModel ?? resolvedModel ?? payload.model`), which for
// bedrock is the raw AWS model ID and has no `/`. parseModel would throw.
// see PR #720 e2e run 25821218139 for the original failure mode.
it("accepts a raw Bedrock model ID (post-resolveModel) without throwing", () => {
process.env.AWS_BEARER_TOKEN_BEDROCK = "bedrock-token";
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-6-v1";
expect(() =>
validateAgentApiKey({ ...base, model: "us.anthropic.claude-opus-4-6-v1" })
).not.toThrow();
});
it("throws on raw Bedrock model ID when AWS auth is missing", () => {
process.env.AWS_REGION = "us-east-1";
process.env.BEDROCK_MODEL_ID = "us.anthropic.claude-opus-4-6-v1";
expect(() =>
validateAgentApiKey({ ...base, model: "us.anthropic.claude-opus-4-6-v1" })
).toThrow("AWS_BEARER_TOKEN_BEDROCK");
});
});
});
describe("isApiKeyAuthError", () => {
it("matches the missing-key marker thrown by validateAgentApiKey", () => {
expect(isApiKeyAuthError("no API key found. Pullfrog needs ...")).toBe(true);
});
it("matches Claude CLI 401 strings", () => {
expect(isApiKeyAuthError("Invalid API key · Fix external API key")).toBe(true);
});
it("matches OpenAI / OpenRouter 401 phrasings", () => {
expect(isApiKeyAuthError("ProviderAuthError: User not found")).toBe(true);
expect(isApiKeyAuthError("401 Invalid authentication")).toBe(true);
});
// see #782 — direct-Anthropic 401 shape (revoked / mistyped / rotated
// ANTHROPIC_API_KEY) reaches us via Claude CLI as a JSON dump, not as
// any of the canonical "Invalid API key" strings. these matchers ensure
// the formatted CTA fires instead of the raw 401 JSON blob.
it("matches direct-Anthropic 401 shapes", () => {
expect(
isApiKeyAuthError(
'Failed to authenticate. API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"Invalid bearer token"}}'
)
).toBe(true);
expect(
isApiKeyAuthError(
"» Pullfrog result error: subtype=success, api_error_status=401, message=Failed to authenticate."
)
).toBe(true);
});
it("ignores unrelated errors", () => {
expect(isApiKeyAuthError("git fetch failed")).toBe(false);
expect(isApiKeyAuthError("")).toBe(false);
});
});
describe("formatApiKeyErrorSummary", () => {
it("renders the missing-key body when the raw error contains the marker", () => {
const msg = formatApiKeyErrorSummary({
owner: "acme",
name: "repo",
raw: "no API key found in this run",
});
expect(msg).toContain("no API key found");
expect(msg).toContain("https://github.com/acme/repo/settings/secrets/actions");
expect(msg).toContain("/console/acme/repo");
expect(msg).toContain("https://discord.gg/8y96raFg8e");
});
it("renders the invalid-key body for any other auth error", () => {
const msg = formatApiKeyErrorSummary({
owner: "acme",
name: "repo",
raw: "Invalid API key · Fix external API key",
});
expect(msg).toContain("rejected (401)");
expect(msg).toContain("https://github.com/acme/repo/settings/secrets/actions");
expect(msg).toContain("https://discord.gg/8y96raFg8e");
});
});
+122 -15
View File
@@ -1,28 +1,48 @@
import { getModelEnvVars, providers } from "../models.ts";
import {
BEDROCK_MODEL_ID_ENV,
getModelEnvVars,
providers,
resolveDisplayAlias,
} from "../models.ts";
import { getApiUrl } from "./apiUrl.ts";
const knownApiKeys: Set<string> = new Set(Object.values(providers).flatMap((p) => [...p.envVars]));
const knownApiKeys: Set<string> = new Set(
Object.values(providers).flatMap((p) => [...p.envVars, ...(p.managedCredentials ?? [])])
);
/** marker prefix on the throw message for the catch-side reclassification path */
const MISSING_KEY_MARKER = "no API key found";
/** Markdown body used for both the thrown error and the formatted PR comment summary. */
function buildMissingApiKeyError(params: { owner: string; name: string }): string {
const apiUrl = getApiUrl();
const settingsUrl = `${apiUrl}/console/${params.owner}/${params.name}`;
const githubSecretsUrl = `https://github.com/${params.owner}/${params.name}/settings/secrets/actions`;
const settingsUrl = `${getApiUrl()}/console/${params.owner}/${params.name}`;
const githubRepoUrl = `https://github.com/${params.owner}/${params.name}`;
const githubSecretsUrl = `${githubRepoUrl}/settings/secrets/actions`;
return [
`**${MISSING_KEY_MARKER}** — Pullfrog needs at least one LLM provider API key (e.g. \`ANTHROPIC_API_KEY\`, \`OPENAI_API_KEY\`, \`GEMINI_API_KEY\`) configured as a GitHub Actions secret.`,
"",
`[Open repo secrets →](${githubSecretsUrl}) · [Configure model →](${settingsUrl}) · [Setup docs →](https://docs.pullfrog.com/keys) · [Ask in Discord →](https://discord.gg/8y96raFg8e)`,
].join("\n");
}
return `no API key found. Pullfrog requires at least one LLM provider API key.
function buildBedrockSetupError(params: {
owner: string;
name: string;
missing: string[];
}): string {
const githubSecretsUrl = `https://github.com/${params.owner}/${params.name}/settings/secrets/actions`;
to fix this, add the required secret to your GitHub repository:
return `Bedrock model selected but required configuration is missing: ${params.missing.join(", ")}.
1. go to: ${githubSecretsUrl}
2. click "New repository secret"
3. set the name to your provider's key (e.g., \`ANTHROPIC_API_KEY\`, \`OPENAI_API_KEY\`, \`GEMINI_API_KEY\`)
4. set the value to your API key
5. click "Add secret"
add the missing secret(s) to your GitHub repository at ${githubSecretsUrl}, then reference them in your workflow's \`env:\` block:
configure your model at ${settingsUrl}
AWS_BEARER_TOKEN_BEDROCK: \${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
AWS_REGION: \${{ secrets.AWS_REGION }}
${BEDROCK_MODEL_ID_ENV}: \${{ secrets.${BEDROCK_MODEL_ID_ENV} }}
for full setup instructions, see https://docs.pullfrog.com/keys`;
\`AWS_BEARER_TOKEN_BEDROCK\` may be substituted with \`AWS_ACCESS_KEY_ID\` + \`AWS_SECRET_ACCESS_KEY\` (and optional \`AWS_SESSION_TOKEN\`) if you prefer access keys.
for full setup instructions, see https://docs.pullfrog.com/bedrock`;
}
function hasEnvVar(name: string): boolean {
@@ -37,6 +57,22 @@ export function hasProviderKey(model: string): boolean {
return requiredVars.some((v) => hasEnvVar(v));
}
function validateBedrockSetup(params: { owner: string; name: string }): void {
const hasAuth =
hasEnvVar("AWS_BEARER_TOKEN_BEDROCK") ||
(hasEnvVar("AWS_ACCESS_KEY_ID") && hasEnvVar("AWS_SECRET_ACCESS_KEY"));
const missing: string[] = [];
if (!hasAuth)
missing.push("AWS_BEARER_TOKEN_BEDROCK (or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY)");
if (!hasEnvVar("AWS_REGION")) missing.push("AWS_REGION");
if (!hasEnvVar(BEDROCK_MODEL_ID_ENV)) missing.push(BEDROCK_MODEL_ID_ENV);
if (missing.length > 0) {
throw new Error(buildBedrockSetupError({ owner: params.owner, name: params.name, missing }));
}
}
export function validateAgentApiKey(params: {
agent: { name: string };
model: string | undefined;
@@ -45,6 +81,28 @@ export function validateAgentApiKey(params: {
}): void {
// if a specific model is configured, only check that model's required env vars
if (params.model) {
// routing slugs (e.g. bedrock) get a tailored validation path because
// their auth shape doesn't match the standard "any one envVar present"
// rule (Bedrock needs auth + region + model-id, with auth being either
// a bearer token OR an access-key pair).
const alias = resolveDisplayAlias(params.model);
if (alias?.routing === "bedrock") {
validateBedrockSetup({ owner: params.owner, name: params.name });
return;
}
// upstream `resolveModel` translates `bedrock/byok` into the raw Bedrock
// model ID (e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/`
// and so isn't parseable as `provider/model`. these IDs only reach this
// function via routing aliases, so re-run the bedrock setup check rather
// than falling through to `getModelEnvVars` (which would throw inside
// parseModel). resolveModel itself already enforced BEDROCK_MODEL_ID,
// but auth + region are still validated here.
if (!params.model.includes("/")) {
validateBedrockSetup({ owner: params.owner, name: params.name });
return;
}
const requiredVars = getModelEnvVars(params.model);
// free models have no required env vars — skip validation entirely
if (requiredVars.length === 0) return;
@@ -59,3 +117,52 @@ export function validateAgentApiKey(params: {
throw new Error(buildMissingApiKeyError({ owner: params.owner, name: params.name }));
}
}
/**
* Detect agent-runtime auth failures that should be reformatted as an actionable
* key-fix CTA before being shown to the user. Covers the shapes we see:
* - missing key (validateAgentApiKey throw): contains MISSING_KEY_MARKER
* - revoked / invalid key (Claude CLI 401 surfaced via api_error_status):
* "Invalid API key · Fix external API key" + similar provider variants
* - direct-Anthropic 401 (`Failed to authenticate. API Error: 401 ...
* {"type":"error","error":{"type":"authentication_error", ...
* "Invalid bearer token"}}`) emitted by the Claude CLI for revoked /
* mistyped / rotated `ANTHROPIC_API_KEY`. see #782.
*/
export function isApiKeyAuthError(text: string): boolean {
if (!text) return false;
return (
text.includes(MISSING_KEY_MARKER) ||
/Invalid API key/i.test(text) ||
/\bUser not found\b/i.test(text) ||
/\bInvalid authentication\b/i.test(text) ||
/authentication_error/i.test(text) ||
/Invalid bearer token/i.test(text) ||
/api_error_status\s*=\s*401/i.test(text) ||
/API Error:\s*401/i.test(text)
);
}
/**
* Friendly Markdown summary for both the missing-key and invalid-key cases.
* Used in the catch / result-failure paths in `main.ts` to overwrite the raw
* agent error before it's posted to the PR progress comment.
*/
export function formatApiKeyErrorSummary(params: {
owner: string;
name: string;
raw: string;
}): string {
if (params.raw.includes(MISSING_KEY_MARKER)) {
return buildMissingApiKeyError({ owner: params.owner, name: params.name });
}
const githubSecretsUrl = `https://github.com/${params.owner}/${params.name}/settings/secrets/actions`;
const settingsUrl = `${getApiUrl()}/console/${params.owner}/${params.name}`;
return [
`**Your LLM provider API key was rejected (401).** Rotate the key in your provider dashboard, then update the matching GitHub Actions secret.`,
"",
`[Update repo secret →](${githubSecretsUrl}) · [Model settings →](${settingsUrl}) · [Setup docs →](https://docs.pullfrog.com/keys) · [Ask in Discord →](https://discord.gg/8y96raFg8e)`,
].join("\n");
}
+189
View File
@@ -0,0 +1,189 @@
/**
* Billing-error classification + user-facing copy for `/api/proxy-token`
* failures and OpenRouter mid-run exhaustion. Two error classes (Billing vs.
* Transient) keep the framing honest: a card decline is *not* the same UX as
* a 503 from the proxy service. Both originate in `utils/proxy.ts` (mint
* failures) and `utils/runErrorRenderer.ts` (mid-run keylimit reclassify).
*
* Renderers return markdown bodies that are written into both the GitHub
* Actions job summary and the PR progress comment.
*
* Lives outside `main.ts` so adding a new error `code` branch is a one-file
* edit that does not retrigger the full LLM CI matrix (`action/main.ts` is
* in `action/test/coverage.ts::ALWAYS_RUN_ALL`).
*/
/**
* Billing-layer error surfaced from `/api/proxy-token` as a 402. User-actionable
* distinct from TransientError (503 / transient sync issue) so the job
* summary + PR comment can use affirmative "you need to do X" copy rather than
* the ambiguous "billing error" label that makes transient outages look like
* the user's fault.
*
* `code` is a server-side discriminator: `router_requires_card` (no card + no
* wallet balance on Router), or null for unclassified. `declineCode` is
* Stripe's more specific sub-reason on `card_declined` (e.g.
* `insufficient_funds`, `lost_card`). `needsReauthentication` is the 3DS case
* broken out for convenience.
*/
export class BillingError extends Error {
code: string | null;
declineCode: string | null;
needsReauthentication: boolean;
constructor(
message: string,
opts: {
code?: string | null;
declineCode?: string | null;
needsReauthentication?: boolean;
} = {}
) {
super(message);
this.name = "BillingError";
this.code = opts.code ?? null;
this.declineCode = opts.declineCode ?? null;
this.needsReauthentication = opts.needsReauthentication ?? false;
}
}
/**
* Transient service failures from `/api/proxy-token` (503: partial OpenRouter
* usage sync, DB flake, in-flight payment intent). Not the user's fault the
* summary uses "temporarily unavailable" framing, and the non-zero exit lets
* GH Actions apply whatever retry policy the workflow has configured.
*/
export class TransientError extends Error {
constructor(message: string) {
super(message);
this.name = "TransientError";
}
}
/**
* Deep link into the right console section for the failing account. Anchors
* are defined in `app/console/[owner]/page.tsx` (`#billing`, `#model-access`).
* `owner` is the GitHub login of the repo's account i.e. the org or user
* that pays for this repo's runs, which is the right scope for billing.
*/
function billingConsoleUrl(owner: string, anchor: "billing" | "model-access"): string {
return `https://pullfrog.com/console/${encodeURIComponent(owner)}#${anchor}`;
}
/**
* Render a BillingError as user-facing markdown (shared between GH job summary
* and the PR progress comment). Goals:
*
* - quiet, not alarmist bold first line instead of an `### ❌` H3, since
* the comment already has Pullfrog branding in the footer
* - actionable every branch ends in a single CTA deep-linked to the
* correct section of the owner's console
* - honest say what actually went wrong (card declined vs. balance
* empty vs. 3DS required), don't lump them under "billing error"
*
* Branches:
* - `router_requires_card`: user is on Router mode with no card AND no
* wallet balance (signup credit exhausted or not granted). Frame as
* "add a card to continue", link to `#model-access` where the Add
* Card flow lives.
* - `router_balance_exhausted`: user has a card on file but auto-reload is
* disabled and they've spent past their $5 overdraft buffer. Frame as
* "balance ran out" and surface both remediation paths (top up, or flip
* on auto-reload).
* - `router_keylimit_exhausted`: OpenRouter rejected mid-run because the
* per-run key budget was exhausted while the agent was working. The
* wallet is now negative; same remediation as `router_balance_exhausted`
* but framed for the after-the-fact case ("this run was cut short").
* - `needsReauthentication`: issuer requires 3DS on every off-session
* charge. Re-adding the card won't help the only escape is a manual
* top-up where 3DS runs interactively in Stripe Checkout.
* - `declineCode` set: Stripe declined a real charge. Show the sub-code
* so support can act on it; tell the user we'll retry on next dispatch.
* - default: balance hit zero with no in-flight charge (auto-reload off
* or amount below threshold). Direct them to top up or enable auto-reload.
*/
export function formatBillingErrorSummary(error: BillingError, owner: string): string {
if (error.code === "router_requires_card") {
return [
"**Add a card to start using Pullfrog Router.**",
"",
"Router proxies OpenRouter at raw cost — no platform markup. Add a card and we'll auto-reload your wallet so runs keep flowing.",
"",
`[Add a card →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.code === "router_balance_exhausted") {
return [
"**Your Pullfrog Router balance is exhausted.**",
"",
"You have a card on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.",
"",
`[Top up balance →](${billingConsoleUrl(owner, "billing")}) · [Enable auto-reload →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.code === "router_keylimit_exhausted") {
return [
"**This run was cut short — your Pullfrog Router balance ran out mid-run.**",
"",
"OpenRouter stopped the agent because the per-run budget was exhausted. Your wallet is now negative; top up or enable auto-reload to keep runs flowing.",
"",
`[Top up balance →](${billingConsoleUrl(owner, "billing")}) · [Enable auto-reload →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.code === "router_monthly_limit") {
return [
"**Pullfrog Router hit its monthly spend limit.**",
"",
"Auto-reloads are paused for the rest of this UTC month. Ask your admin to raise the cap, or wait for it to reset at 00:00 UTC on the 1st.",
"",
`[Adjust limit →](${billingConsoleUrl(owner, "model-access")})`,
].join("\n");
}
if (error.needsReauthentication) {
const code = error.declineCode ?? "authentication_required";
return [
`**Your card issuer requires 3D Secure on every charge** (\`${code}\`).`,
"",
"Pullfrog can't complete a 3DS challenge from inside a workflow. Top up your Router balance once in Stripe Checkout — subsequent runs draw from the prepaid balance without re-triggering 3DS.",
"",
`[Top up balance →](${billingConsoleUrl(owner, "billing")})`,
].join("\n");
}
if (error.declineCode) {
return [
`**Your card was declined** (\`${error.declineCode}\`).`,
"",
"Update your payment method and Pullfrog will retry on the next run.",
"",
`[Update payment method →](${billingConsoleUrl(owner, "billing")})`,
].join("\n");
}
return [
"**Your Pullfrog balance is empty.**",
"",
"Top up your balance or enable auto-reload to keep runs flowing.",
"",
`[Manage billing →](${billingConsoleUrl(owner, "billing")})`,
].join("\n");
}
/**
* Render a TransientError as user-facing markdown. Distinct framing from
* BillingError so the user doesn't read an alarm and assume their card
* failed this branch is "our fault, retry shortly", not theirs.
*/
export function formatTransientErrorSummary(error: TransientError, owner: string): string {
return [
"**Pullfrog billing is temporarily unavailable.**",
"",
error.message,
"",
`Usually transient — the next dispatch should succeed. If it persists, check [status.pullfrog.com](https://status.pullfrog.com) or [your console](${billingConsoleUrl(owner, "billing")}).`,
].join("\n");
}
+38
View File
@@ -0,0 +1,38 @@
import { describe, expect, it } from "vitest";
import { buildPullfrogFooter } from "./buildPullfrogFooter.ts";
describe("buildPullfrogFooter — fallbackFrom annotation", () => {
it("renders the provider display name when fallbackFrom is set", () => {
const footer = buildPullfrogFooter({
model: "opencode/minimax-m2.5-free",
fallbackFrom: "anthropic/claude-opus",
});
expect(footer).toContain(
"Using `MiniMax M2.5` (free) (credentials for Anthropic not configured)"
);
});
it("works for OpenAI's display name too", () => {
const footer = buildPullfrogFooter({
model: "opencode/minimax-m2.5-free",
fallbackFrom: "openai/gpt",
});
expect(footer).toContain("(credentials for OpenAI not configured)");
});
it("falls back to the raw provider key when the slug provider is unknown to the catalog", () => {
const footer = buildPullfrogFooter({
model: "opencode/minimax-m2.5-free",
fallbackFrom: "some-unknown/model",
});
expect(footer).toContain("(credentials for some-unknown not configured)");
});
it("omits the annotation when fallbackFrom is not set", () => {
const footer = buildPullfrogFooter({
model: "anthropic/claude-opus",
});
expect(footer).toContain("Using `Claude Opus`");
expect(footer).not.toContain("not configured");
});
});
+32 -9
View File
@@ -1,4 +1,4 @@
import { modelAliases, resolveDisplayAlias } from "../models.ts";
import { getModelProvider, modelAliases, providers, resolveDisplayAlias } from "../models.ts";
export const PULLFROG_DIVIDER = "<!-- PULLFROG_DIVIDER_DO_NOT_REMOVE_PLZ -->";
@@ -23,20 +23,41 @@ export interface BuildPullfrogFooterParams {
customParts?: string[] | undefined;
/** model slug from payload (e.g., "anthropic/claude-opus"). shown in footer as "Using `Model Name`" */
model?: string | undefined;
/**
* When the action engaged the BYOK fallback, this is the slug the user
* had configured (e.g. "anthropic/claude-opus") the footer renders
* `Using <free model> (credentials for <configured> not configured)`
* so the substitution is visible in PR comments + reviews.
*/
fallbackFrom?: string | undefined;
}
function formatModelLabel(slug: string): string {
// walk the fallback chain so a deprecated stored slug shows the model the
// run actually executed against (e.g. "GPT", not "GPT Codex").
/** Provider display name (e.g. "Anthropic") for the slug, or the raw provider segment as a fallback. */
function providerDisplayName(slug: string): string {
try {
const key = getModelProvider(slug);
const meta = providers[key as keyof typeof providers];
return meta?.displayName ?? key;
} catch {
// raw IDs without a `/` (Bedrock model IDs) — never reach this function
// in practice because the BYOK fallback skips Bedrock, but defensively
// return the slug itself rather than throw if it ever does.
return slug;
}
}
function formatModelLabel(params: { model: string; fallbackFrom?: string | undefined }): string {
const alias =
resolveDisplayAlias(slug) ??
resolveDisplayAlias(params.model) ??
// reverse-lookup: when the caller passes an effective model (proxy or
// resolved target like "openrouter/anthropic/claude-opus-4.7") instead of
// a stored alias slug, find the alias whose resolve target matches so we
// still render a friendly display name.
modelAliases.find((a) => a.resolve === slug || a.openRouterResolve === slug);
if (!alias) return `\`${slug}\``;
return alias.isFree ? `\`${alias.displayName}\` (free)` : `\`${alias.displayName}\``;
modelAliases.find((a) => a.resolve === params.model || a.openRouterResolve === params.model);
const displayName = alias?.displayName ?? params.model;
const base = alias?.isFree ? `\`${displayName}\` (free)` : `\`${displayName}\``;
if (!params.fallbackFrom) return base;
return `${base} (credentials for ${providerDisplayName(params.fallbackFrom)} not configured)`;
}
/**
@@ -64,7 +85,9 @@ export function buildPullfrogFooter(params: BuildPullfrogFooterParams): string {
}
if (params.model) {
parts.push(`Using ${formatModelLabel(params.model)}`);
parts.push(
`Using ${formatModelLabel({ model: params.model, fallbackFrom: params.fallbackFrom })}`
);
}
const allParts = [...parts, "[𝕏](https://x.com/pullfrogai)"];
+102
View File
@@ -0,0 +1,102 @@
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { FREE_FALLBACK_SLUG, selectFallbackModelIfNeeded } from "./byokFallback.ts";
describe("selectFallbackModelIfNeeded", () => {
const originalEnv = { ...process.env };
const KEYS = [
"ANTHROPIC_API_KEY",
"CLAUDE_CODE_OAUTH_TOKEN",
"OPENAI_API_KEY",
"OPENROUTER_API_KEY",
"GEMINI_API_KEY",
"GOOGLE_GENERATIVE_AI_API_KEY",
"XAI_API_KEY",
"DEEPSEEK_API_KEY",
"MOONSHOT_API_KEY",
"OPENCODE_API_KEY",
] as const;
beforeEach(() => {
for (const k of KEYS) delete process.env[k];
});
afterEach(() => {
for (const k of KEYS) {
if (originalEnv[k] === undefined) delete process.env[k];
else process.env[k] = originalEnv[k];
}
});
it("falls back when the resolved model needs a key that isn't set", () => {
const result = selectFallbackModelIfNeeded({
resolvedModel: "anthropic/claude-opus-4-7",
proxyModel: undefined,
});
expect(result).toEqual({
fallback: true,
from: "anthropic/claude-opus-4-7",
to: FREE_FALLBACK_SLUG,
});
});
it("does not fall back when the resolved model's key IS set", () => {
process.env.ANTHROPIC_API_KEY = "sk-test";
const result = selectFallbackModelIfNeeded({
resolvedModel: "anthropic/claude-opus-4-7",
proxyModel: undefined,
});
expect(result.fallback).toBe(false);
});
it("does not fall back on Router runs (proxyModel set)", () => {
const result = selectFallbackModelIfNeeded({
resolvedModel: undefined,
proxyModel: "openrouter/anthropic/claude-opus-4.7",
});
expect(result.fallback).toBe(false);
});
it("does not fall back when no model is resolved (auto-select path)", () => {
const result = selectFallbackModelIfNeeded({
resolvedModel: undefined,
proxyModel: undefined,
});
expect(result.fallback).toBe(false);
});
it("does not fall back when the resolved model is itself the free fallback", () => {
const result = selectFallbackModelIfNeeded({
resolvedModel: FREE_FALLBACK_SLUG,
proxyModel: undefined,
});
expect(result.fallback).toBe(false);
});
it("does not fall back for Bedrock routing (raw model ID has no slash)", () => {
// resolveModel({slug:"bedrock/byok"}) returns the raw BEDROCK_MODEL_ID
// value (e.g. "us.anthropic.claude-opus-4-7"), which has no `/`. without
// a guard, hasProviderKey → parseModel would throw and crash the action
// before validateBedrockSetup can surface its tailored error.
const result = selectFallbackModelIfNeeded({
resolvedModel: "us.anthropic.claude-opus-4-7",
proxyModel: undefined,
});
expect(result.fallback).toBe(false);
});
it("does not fall back for free models that need no key", () => {
const result = selectFallbackModelIfNeeded({
resolvedModel: "opencode/mimo-v2-pro-free",
proxyModel: undefined,
});
expect(result.fallback).toBe(false);
});
it("treats empty-string env vars as missing (matches GH Actions secret-not-found behavior)", () => {
process.env.ANTHROPIC_API_KEY = "";
const result = selectFallbackModelIfNeeded({
resolvedModel: "anthropic/claude-opus-4-7",
proxyModel: undefined,
});
expect(result.fallback).toBe(true);
});
});
+59
View File
@@ -0,0 +1,59 @@
import { hasProviderKey } from "./apiKeys.ts";
/**
* Slug we fall back to when a BYOK-required model is configured but the
* runner has no provider key in env. Picked because it's free
* (`isFree: true`, `envVars: []` see `action/models.ts`), stable, and
* currently the strongest free OpenCode model in the catalog. If a
* smarter free model is added later, update this single constant.
*
* The slug is intentionally hard-coded and not a config knob the
* fallback is a safety net, not a user-facing preference, and adding a
* config surface here would just push the same "what to fall back to"
* decision into another setting that goes stale the same way.
*/
export const FREE_FALLBACK_SLUG = "opencode/minimax-m2.5-free";
export type FallbackDecision = { fallback: false } | { fallback: true; from: string; to: string };
/**
* If the resolved model requires a BYOK key but no provider key is
* available in env, return `fallback: true` with a free OpenCode slug
* so the run can still succeed. Caller is responsible for swapping the
* model state and surfacing the fallback (log line + run summary).
*
* Gates on `resolvedModel` directly (not the configured slug) so the
* decision matches both code paths that reach this point: payload-based
* config (`repo.model` from DB) and `PULLFROG_MODEL` env var. Both end
* up in `resolvedModel` after `resolveModel()` runs upstream.
*
* Skip cases:
* - Router / proxy runs (`proxyModel` set): Pullfrog mints the key,
* no BYOK in play never fall back.
* - No resolved model: keeps the existing auto-select-with-throw
* behavior in `validateAgentApiKey` for the "neither model nor
* key" case (genuine misconfig the user should see).
* - Resolved model is itself the free fallback: avoid suggesting we
* fell back to the model we're already running.
* - Resolved model is a Bedrock raw ID (no `/`): Bedrock has its own
* auth shape (`AWS_BEARER_TOKEN_BEDROCK` + region + model ID), and
* `validateBedrockSetup` already surfaces a tailored error. Skipping
* here also avoids `parseModel`'s slash requirement crashing inside
* `hasProviderKey`.
* - Resolved model has its provider key present: no fallback needed.
*/
export function selectFallbackModelIfNeeded(input: {
resolvedModel: string | undefined;
proxyModel: string | undefined;
}): FallbackDecision {
if (input.proxyModel) return { fallback: false };
if (!input.resolvedModel) return { fallback: false };
if (input.resolvedModel === FREE_FALLBACK_SLUG) return { fallback: false };
if (!input.resolvedModel.includes("/")) return { fallback: false };
if (hasProviderKey(input.resolvedModel)) return { fallback: false };
return {
fallback: true,
from: input.resolvedModel,
to: FREE_FALLBACK_SLUG,
};
}
+283
View File
@@ -0,0 +1,283 @@
import { spawn } from "node:child_process";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
/**
* minted Codex subscription credential. raw `auth.json` body that Codex CLI /
* OpenCode plugins consume. validated to be `auth_mode: "chatgpt"` with a
* refresh token before being returned. caller is responsible for storing it
* (typically as the `CODEX_AUTH_JSON` Pullfrog secret).
*/
export interface CodexAuth {
/** raw JSON body of the minted `auth.json`; safe to persist verbatim. */
json: string;
/** parsed for caller convenience; mirrors the shape Codex CLI writes. */
parsed: CodexAuthJson;
}
export interface CodexAuthJson {
auth_mode: "chatgpt";
tokens: {
access_token: string;
id_token?: string;
refresh_token: string;
account_id?: string;
};
last_refresh?: string;
}
/** OAuth client id Codex CLI and OpenCode both use against `auth.openai.com`.
* Same chain a refresh token minted via `codex login --device-auth` can be
* refreshed against this client_id. */
const CODEX_OAUTH_CLIENT_ID = "app_EMoamEEZ73f0CkXaXp7hrann";
const CODEX_OAUTH_TOKEN_URL = "https://auth.openai.com/oauth/token";
interface OAuthTokenResponse {
access_token: string;
refresh_token: string;
id_token?: string;
expires_in?: number;
}
/** force one refresh round-trip against the OAuth provider so the saved
* credential carries the freshest refresh token. used right after `codex
* login --device-auth` and again any time we want to bump the chain before
* persisting (avoids the user's laptop refreshing first and burning ours). */
export async function refreshCodexAuth(auth: CodexAuth): Promise<CodexAuth> {
const response = await fetch(CODEX_OAUTH_TOKEN_URL, {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
grant_type: "refresh_token",
refresh_token: auth.parsed.tokens.refresh_token,
client_id: CODEX_OAUTH_CLIENT_ID,
}).toString(),
});
if (!response.ok) {
const body = await response.text().catch(() => "");
throw new Error(`Codex token refresh failed: ${response.status} ${body}`);
}
const tokens = (await response.json()) as OAuthTokenResponse;
const idToken = tokens.id_token ?? auth.parsed.tokens.id_token;
const accountId = auth.parsed.tokens.account_id;
const refreshed: CodexAuthJson = {
auth_mode: "chatgpt",
tokens: {
access_token: tokens.access_token,
refresh_token: tokens.refresh_token,
...(idToken ? { id_token: idToken } : {}),
...(accountId ? { account_id: accountId } : {}),
},
last_refresh: new Date().toISOString(),
};
return { json: `${JSON.stringify(refreshed, null, 2)}\n`, parsed: refreshed };
}
export type ProgressEvent =
| { kind: "start"; attempt: number }
| { kind: "exit"; exitCode: number; signal: NodeJS.Signals | null; timedOut: boolean }
| { kind: "retry"; reason: "user-request" | "no-auth-written" }
| { kind: "cancel" };
interface RunOptions {
/** abort the whole flow when true is returned. polled before each retry. */
shouldRetry: () => Promise<boolean>;
/** observe progress for UI rendering. */
onProgress?: (event: ProgressEvent) => void;
/**
* pass-through control over the child's stdio. `inherit` streams Codex's
* own UI directly to the user's terminal. `pipe` is what `pullfrog auth
* codex` uses so it can re-render each line with a Pullfrog-styled rail
* + dim formatting via `onChildLine`.
*/
childStdio?: "inherit" | "pipe";
/**
* called once per line of Codex's stdout/stderr when `childStdio` is
* "pipe". raw line text is passed through unmodified (including any ANSI
* escapes Codex emitted); the caller is responsible for stripping/styling.
*/
onChildLine?: (line: string, stream: "stdout" | "stderr") => void;
/** how long a single device-auth attempt is allowed to run. */
perAttemptTimeoutMs?: number;
}
/**
* mint a fresh Codex subscription credential by running `codex login
* --device-auth` against an isolated `CODEX_HOME`. the user's global
* `~/.codex/auth.json` is never touched; on success or failure, the
* temporary home is cleaned up.
*
* the caller controls retry behavior via `shouldRetry`: when device auth
* exits without writing `auth.json` (most commonly because the user needed
* to enable device-code auth on their ChatGPT account first), the function
* invokes `shouldRetry()` to decide whether to spin up another attempt.
*/
export async function mintCodexAuth(options: RunOptions): Promise<CodexAuth> {
// mkdtempSync already creates the dir with the default 0o700 perms on
// posix; an extra mkdirSync would just be ceremony.
const codexHome = mkdtempSync(join(tmpdir(), "pullfrog-codex-"));
try {
// device auth requires file-backed credentials; otherwise Codex routes the
// refresh token into the OS keyring and we can't observe / persist it.
writeFileSync(join(codexHome, "config.toml"), 'cli_auth_credentials_store = "file"\n', {
mode: 0o600,
});
const authPath = join(codexHome, "auth.json");
let attempt = 1;
while (true) {
options.onProgress?.({ kind: "start", attempt });
const result = await runDeviceAuth({
codexHome,
timeoutMs: options.perAttemptTimeoutMs ?? 15 * 60 * 1000,
childStdio: options.childStdio ?? "inherit",
onChildLine: options.onChildLine,
});
options.onProgress?.({
kind: "exit",
exitCode: result.exitCode,
signal: result.signal,
timedOut: result.timedOut,
});
const auth = readAuthIfPresent(authPath);
if (auth) return auth;
if (!(await options.shouldRetry())) {
options.onProgress?.({ kind: "cancel" });
throw new Error("Codex login did not produce auth.json (no retry requested)");
}
options.onProgress?.({ kind: "retry", reason: "no-auth-written" });
attempt += 1;
}
} finally {
rmSync(codexHome, { recursive: true, force: true });
}
}
interface DeviceAuthResult {
exitCode: number;
signal: NodeJS.Signals | null;
/** true if the attempt was killed by our per-attempt timeout (vs. exited
* naturally or was interrupted by the user). lets callers distinguish
* "user walked away" from "user closed the device flow early". */
timedOut: boolean;
}
interface DeviceAuthInput {
codexHome: string;
timeoutMs: number;
childStdio: "inherit" | "pipe";
onChildLine?: ((line: string, stream: "stdout" | "stderr") => void) | undefined;
}
/** how long to wait between SIGTERM and SIGKILL when killing a stuck `codex`
* subprocess. Codex usually exits cleanly on SIGTERM, but if it ignores it we
* don't want the CLI pinned forever. */
const SIGTERM_GRACE_MS = 5_000;
/** spawn `codex login --device-auth` with stdin closed so Codex doesn't hang
* waiting for input. by default inherits stdout/stderr so the user sees the
* device URL + one-time code Codex prints; when `pipe`d, lines are forwarded
* to `onChildLine` so the caller can re-style them. on per-attempt timeout,
* sends SIGTERM and escalates to SIGKILL after a short grace.
*/
function runDeviceAuth(input: DeviceAuthInput): Promise<DeviceAuthResult> {
return new Promise((resolve, reject) => {
const child = spawn("codex", ["login", "--device-auth"], {
env: { ...process.env, CODEX_HOME: input.codexHome },
stdio: ["ignore", input.childStdio, input.childStdio],
});
if (input.childStdio === "pipe") {
const onLine = input.onChildLine ?? (() => {});
if (child.stdout) pipeLines(child.stdout, (line) => onLine(line, "stdout"));
if (child.stderr) pipeLines(child.stderr, (line) => onLine(line, "stderr"));
}
let killTimer: NodeJS.Timeout | null = null;
let timedOut = false;
const timeoutTimer = setTimeout(() => {
timedOut = true;
child.kill("SIGTERM");
// give Codex a grace window to exit cleanly on SIGTERM. if it ignores
// it, force SIGKILL so we don't pin the CLI on a stuck child.
killTimer = setTimeout(() => child.kill("SIGKILL"), SIGTERM_GRACE_MS);
}, input.timeoutMs);
// `spawn` emits 'error' (not 'close') when the binary can't be found
// (ENOENT) or otherwise fails to start. without a listener, Node crashes
// the process with an unhandled 'error' event.
child.on("error", (err) => {
clearTimeout(timeoutTimer);
if (killTimer) clearTimeout(killTimer);
const errno = err as NodeJS.ErrnoException;
const message =
errno.code === "ENOENT"
? "codex CLI not found on PATH. install it with `npm i -g @openai/codex` or see https://developers.openai.com/codex/cli for other install options."
: `failed to spawn codex: ${errno.message}`;
reject(new Error(message));
});
child.on("close", (code, signal) => {
clearTimeout(timeoutTimer);
if (killTimer) clearTimeout(killTimer);
resolve({ exitCode: code ?? 1, signal, timedOut });
});
});
}
/** byte-stream newline-delimited line callback. emits any final partial
* line on stream end so trailing content (e.g. a prompt with no newline)
* still surfaces to the renderer.
*/
function pipeLines(stream: NodeJS.ReadableStream, onLine: (line: string) => void): void {
let buffer = "";
stream.on("data", (chunk: Buffer | string) => {
buffer += typeof chunk === "string" ? chunk : chunk.toString("utf8");
let idx = buffer.indexOf("\n");
while (idx !== -1) {
const line = buffer.slice(0, idx).replace(/\r$/, "");
buffer = buffer.slice(idx + 1);
onLine(line);
idx = buffer.indexOf("\n");
}
});
stream.on("end", () => {
if (buffer.length > 0) {
onLine(buffer);
buffer = "";
}
});
}
function readAuthIfPresent(authPath: string): CodexAuth | null {
let raw: string;
try {
raw = readFileSync(authPath, "utf8");
} catch {
return null;
}
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
return null;
}
if (!isCodexAuthJson(parsed)) return null;
return { json: raw, parsed };
}
function isCodexAuthJson(value: unknown): value is CodexAuthJson {
if (!value || typeof value !== "object") return false;
const v = value as Record<string, unknown>;
if (v.auth_mode !== "chatgpt") return false;
const tokens = v.tokens;
if (!tokens || typeof tokens !== "object") return false;
const t = tokens as Record<string, unknown>;
if (typeof t.access_token !== "string" || t.access_token.length === 0) return false;
if (typeof t.refresh_token !== "string" || t.refresh_token.length === 0) return false;
return true;
}
+85
View File
@@ -0,0 +1,85 @@
import { describe, expect, it } from "vitest";
import { detectCodexRefresh } from "./codexHome.ts";
// installCodexAuth touches the filesystem (mkdir + writeFile) — leaving it
// untested here per AGENTS.md guidance ("be highly dubious of any test that
// relies on mocks"). The conversion math is what we actually want to
// protect; the disk write is one writeFileSync call.
describe("detectCodexRefresh", () => {
const original = "rt_original_chain";
it("returns Codex-shape JSON when openai.refresh advanced", () => {
const authFileContent = JSON.stringify({
openai: {
type: "oauth",
refresh: "rt_new_chain",
access: "at_new",
expires: 9_999_999_999_999,
accountId: "acc_123",
},
});
const result = detectCodexRefresh({ authFileContent, originalRefresh: original });
expect(result).not.toBeNull();
const parsed = JSON.parse(result ?? "{}");
expect(parsed.auth_mode).toBe("chatgpt");
expect(parsed.tokens.refresh_token).toBe("rt_new_chain");
expect(parsed.tokens.access_token).toBe("at_new");
expect(parsed.tokens.account_id).toBe("acc_123");
expect(typeof parsed.last_refresh).toBe("string");
});
it("omits account_id when accountId is absent from OpenCode shape", () => {
const authFileContent = JSON.stringify({
openai: {
type: "oauth",
refresh: "rt_new",
access: "at_new",
expires: 0,
},
});
const result = detectCodexRefresh({ authFileContent, originalRefresh: original });
const parsed = JSON.parse(result ?? "{}");
expect("account_id" in parsed.tokens).toBe(false);
});
it("returns null when refresh token unchanged (no rotation happened)", () => {
const authFileContent = JSON.stringify({
openai: { type: "oauth", refresh: original, access: "at_same", expires: 0 },
});
expect(detectCodexRefresh({ authFileContent, originalRefresh: original })).toBeNull();
});
it("returns null when openai entry is missing", () => {
const authFileContent = JSON.stringify({
anthropic: { type: "oauth", refresh: "rt_other", access: "at_other", expires: 0 },
});
expect(detectCodexRefresh({ authFileContent, originalRefresh: original })).toBeNull();
});
it("returns null when openai is api-key type (no refresh chain)", () => {
const authFileContent = JSON.stringify({
openai: { type: "api", key: "sk-something" },
});
expect(detectCodexRefresh({ authFileContent, originalRefresh: original })).toBeNull();
});
it("returns null for malformed JSON", () => {
expect(
detectCodexRefresh({ authFileContent: "{not json", originalRefresh: original })
).toBeNull();
});
it("returns null for non-object content", () => {
expect(
detectCodexRefresh({ authFileContent: '"a string"', originalRefresh: original })
).toBeNull();
});
it("returns null when refresh field is missing", () => {
const authFileContent = JSON.stringify({
openai: { type: "oauth", access: "at_new", expires: 0 },
});
expect(detectCodexRefresh({ authFileContent, originalRefresh: original })).toBeNull();
});
});
+174
View File
@@ -0,0 +1,174 @@
// Codex-to-OpenCode auth bridging for the action runtime.
//
// `pullfrog auth codex` stores a Codex CLI `auth.json` blob in the Pullfrog
// per-org secret store (production Postgres) — NOT a GitHub Actions secret.
// This is non-negotiable: the OAuth refresh chain rotates on every use, and
// `entryPost.ts` writes the rotated chain back via `PUT /api/runtime/secret`
// after each run. GH Actions secrets are immutable at runtime, so a token
// stashed there silently expires on the first refresh (~1h). See
// wiki/codex-auth.md for the full constraint.
//
// At runtime, `CODEX_AUTH_JSON` lands in process.env via `runContext.dbSecrets`
// merged in main.ts — sourced from Pullfrog Postgres through the OIDC-validated
// run-context endpoint, never from `${{ secrets.CODEX_AUTH_JSON }}` in
// workflow yaml. This utility:
//
// 1. parses + validates that env value
// 2. converts Codex's shape `{ auth_mode, tokens: { access_token, refresh_token, ... } }`
// into OpenCode's shape `{ openai: { type: "oauth", refresh, access, expires, accountId } }`
// 3. materializes it to disk at the runner's REAL `$HOME/.local/share/opencode/auth.json`
// (NOT the per-run tmpdir's HOME)
// 4. returns the path + the original refresh token so the post-run hook
// can detect a refresh and write back to Pullfrog
//
// Why real $HOME and not ctx.tmpdir-redirected HOME: the broad
// `external_directory: { "/tmp/*": "allow" }` rule on OpenCode would expose
// auth.json to the agent's filesystem tools if the file lived under
// `ctx.tmpdir` = `/tmp/pullfrog-*`. Real `$HOME/.local/share/opencode/...`
// falls outside that allow zone, so OpenCode's deny-default protects it
// without any new permission rules.
//
// `expires: 0` forces OpenCode to refresh on first request (we don't trust
// the in-blob freshness — the saved token was eager-refreshed once at
// `auth codex` time but may have aged since).
//
// See [wiki/codex-auth.md] for the full data-flow picture.
import { mkdirSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
import { log } from "./cli.ts";
const CODEX_AUTH_ENV = "CODEX_AUTH_JSON";
interface CodexAuthBlob {
auth_mode: "chatgpt";
tokens: {
access_token: string;
refresh_token: string;
id_token?: string;
account_id?: string;
};
last_refresh?: string;
}
interface OpenCodeAuthFile {
openai: {
type: "oauth";
refresh: string;
access: string;
expires: number;
accountId?: string;
};
}
export interface InstalledCodexAuth {
/** absolute path of the auth.json we wrote caller passes this to the
* post-hook via core.saveState for refresh-detection later. */
authPath: string;
/** value to set as XDG_DATA_HOME for the OpenCode subprocess. */
xdgDataHome: string;
/** refresh_token from the env at materialization time. post-hook compares
* against the on-disk file after the run to detect whether OpenCode
* refreshed during the session. */
originalRefresh: string;
}
/** materialize CODEX_AUTH_JSON from env into a disk path OpenCode reads from.
* returns null when the env var is absent, malformed, or wrong auth mode
* caller treats null as "no codex auth, fall through to API key flow". */
export function installCodexAuth(): InstalledCodexAuth | null {
const raw = process.env[CODEX_AUTH_ENV];
if (!raw) return null;
const blob = parseCodexBlob(raw);
if (!blob) {
log.warning(`» ${CODEX_AUTH_ENV} present but malformed; ignoring`);
return null;
}
const xdgDataHome = join(homedir(), ".local", "share");
const opencodeDir = join(xdgDataHome, "opencode");
const authPath = join(opencodeDir, "auth.json");
const opencodeAuth: OpenCodeAuthFile = {
openai: {
type: "oauth",
refresh: blob.tokens.refresh_token,
access: blob.tokens.access_token,
// expires: 0 forces OpenCode's CodexAuthPlugin to refresh on first
// request (it checks `expires < Date.now()`). safest default — we
// don't carry an `expires_in` from the Codex blob.
expires: 0,
...(blob.tokens.account_id ? { accountId: blob.tokens.account_id } : {}),
},
};
mkdirSync(opencodeDir, { recursive: true });
writeFileSync(authPath, `${JSON.stringify(opencodeAuth, null, 2)}\n`, { mode: 0o600 });
log.info(`» installed Codex auth at ${authPath}`);
return { authPath, xdgDataHome, originalRefresh: blob.tokens.refresh_token };
}
function parseCodexBlob(raw: string): CodexAuthBlob | null {
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch {
return null;
}
if (!parsed || typeof parsed !== "object") return null;
const v = parsed as Record<string, unknown>;
if (v.auth_mode !== "chatgpt") return null;
const tokens = v.tokens;
if (!tokens || typeof tokens !== "object") return null;
const t = tokens as Record<string, unknown>;
if (typeof t.access_token !== "string" || t.access_token.length === 0) return null;
if (typeof t.refresh_token !== "string" || t.refresh_token.length === 0) return null;
return {
auth_mode: "chatgpt",
tokens: {
access_token: t.access_token,
refresh_token: t.refresh_token,
...(typeof t.id_token === "string" ? { id_token: t.id_token } : {}),
...(typeof t.account_id === "string" ? { account_id: t.account_id } : {}),
},
...(typeof v.last_refresh === "string" ? { last_refresh: v.last_refresh } : {}),
};
}
/** convert an on-disk OpenCode auth.json back to the Codex CLI shape so the
* post-hook can write it to the Pullfrog secret store. returns null when the
* file's `openai` entry is missing, has the wrong type, or hasn't actually
* refreshed (refresh token unchanged from `originalRefresh`). */
export function detectCodexRefresh(params: {
authFileContent: string;
originalRefresh: string;
}): string | null {
let parsed: unknown;
try {
parsed = JSON.parse(params.authFileContent);
} catch {
return null;
}
if (!parsed || typeof parsed !== "object") return null;
const oauth = (parsed as Record<string, unknown>).openai;
if (!oauth || typeof oauth !== "object") return null;
const o = oauth as Record<string, unknown>;
if (o.type !== "oauth") return null;
if (typeof o.refresh !== "string" || typeof o.access !== "string") return null;
if (o.refresh === params.originalRefresh) return null;
const codexShape: CodexAuthBlob = {
auth_mode: "chatgpt",
tokens: {
access_token: o.access,
refresh_token: o.refresh,
...(typeof o.accountId === "string" ? { account_id: o.accountId } : {}),
},
last_refresh: new Date().toISOString(),
};
return `${JSON.stringify(codexShape, null, 2)}\n`;
}
-286
View File
@@ -1,286 +0,0 @@
/**
* shared docker utilities for running commands in containers.
* used by both play.ts (dev) and test/run.ts (CI).
*/
import { type SpawnSyncReturns, spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import { platform } from "node:os";
import { join } from "node:path";
export type DockerRunContext = {
actionDir: string;
args: string[];
platformName: NodeJS.Platform;
home: string | undefined;
env: NodeJS.ProcessEnv;
uid: number;
gid: number;
};
export type SshSetup = {
sshFlags: string[];
sshSetupCmd: string;
};
export type DockerRunArgsContext = {
ctx: DockerRunContext;
envFlags: string[];
nodeCmd: string;
sshSetup: SshSetup;
volumeName: string;
};
export type VolumeInitContext = {
actionDir: string;
volumeName: string;
uid: number;
gid: number;
};
export function buildDockerRunContext(ctx: {
actionDir: string;
args: string[];
}): DockerRunContext {
return {
actionDir: ctx.actionDir,
args: ctx.args,
platformName: platform(),
home: process.env.HOME,
env: process.env,
uid: process.getuid?.() ?? 1000,
gid: process.getgid?.() ?? 1000,
};
}
export function assertDockerSupported(ctx: DockerRunContext): void {
if (ctx.platformName === "win32") {
throw new Error("docker mode is not supported on native windows. use wsl2.");
}
}
function buildDarwinSshSetup(ctx: DockerRunContext): SshSetup {
const sshFlags: string[] = [];
const sshSetupCmd = "";
if (ctx.home) {
const knownHostsPath = join(ctx.home, ".ssh", "known_hosts");
if (existsSync(knownHostsPath)) {
sshFlags.push("-v", `${knownHostsPath}:/root/.ssh/known_hosts:ro`);
}
}
sshFlags.push(
"-v",
"/run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock",
"-e",
"SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock"
);
return { sshFlags, sshSetupCmd };
}
function buildLinuxSshSetup(ctx: DockerRunContext): SshSetup {
const sshFlags: string[] = [];
let sshSetupCmd = "";
if (ctx.home) {
const sshDir = join(ctx.home, ".ssh");
if (existsSync(sshDir)) {
sshFlags.push("-v", `${sshDir}:/tmp/.ssh-host:ro`);
sshSetupCmd =
"mkdir -p /tmp/home/.ssh && cp /tmp/.ssh-host/id_* /tmp/home/.ssh/ 2>/dev/null; chmod 600 /tmp/home/.ssh/id_* 2>/dev/null; " +
"ssh-keyscan -t ed25519,rsa github.com >> /tmp/home/.ssh/known_hosts 2>/dev/null; chmod 644 /tmp/home/.ssh/known_hosts; " +
"export GIT_SSH_COMMAND='ssh -i /tmp/home/.ssh/id_rsa -o UserKnownHostsFile=/tmp/home/.ssh/known_hosts -o StrictHostKeyChecking=no'; ";
}
}
return { sshFlags, sshSetupCmd };
}
export function buildSshSetup(ctx: DockerRunContext): SshSetup {
if (ctx.platformName === "darwin") {
return buildDarwinSshSetup(ctx);
}
return buildLinuxSshSetup(ctx);
}
// allowlist of env vars to pass through to the container for `pnpm runtest`.
// NOTE: `pnpm play` uses "passthrough" mode and passes ALL env vars.
// if your env var isn't working with `pnpm runtest`, add it here!
// see wiki/adversarial.md for documentation.
const testEnvAllowList = new Set([
"CI",
"GITHUB_ACTIONS",
"PULLFROG_DISABLE_SECURITY_INSTRUCTIONS", // disables security messaging for pentest
"GITHUB_TOKEN",
"GH_TOKEN",
"GITHUB_REPOSITORY",
"GITHUB_APP_ID",
"GITHUB_PRIVATE_KEY",
"OPENAI_API_KEY",
"ANTHROPIC_API_KEY",
"CLAUDE_CODE_OAUTH_TOKEN",
"GEMINI_API_KEY",
"GOOGLE_GENERATIVE_AI_API_KEY",
"XAI_API_KEY",
"DEEPSEEK_API_KEY",
"OPENROUTER_API_KEY",
"MOONSHOT_API_KEY",
"OPENCODE_API_KEY",
"PULLFROG_MODEL",
"LOG_LEVEL",
"DEBUG",
"NODE_ENV",
"PLAY_LOCAL",
"HOME",
"USER",
"SSH_AUTH_SOCK",
"ACTIONS_ID_TOKEN_REQUEST_URL",
"ACTIONS_ID_TOKEN_REQUEST_TOKEN",
"GITHUB_API_URL",
"GITHUB_SERVER_URL",
"GITHUB_GRAPHQL_URL",
"GITHUB_OUTPUT",
]);
export type EnvFilterMode = "allowlist" | "passthrough";
export function buildEnvFlags(ctx: DockerRunContext, mode: EnvFilterMode): string[] {
const envFlags: string[] = [];
const entries = Object.entries(ctx.env);
for (const entry of entries) {
const key = entry[0];
const value = entry[1];
if (value === undefined) continue;
if (mode === "passthrough" || testEnvAllowList.has(key)) {
envFlags.push("-e", `${key}=${value}`);
}
}
return envFlags;
}
export function initializeNodeModulesVolume(ctx: VolumeInitContext): void {
spawnSync(
"docker",
[
"run",
"--rm",
"-v",
`${ctx.volumeName}:/app/action/node_modules`,
"node:24",
"chown",
"-R",
`${ctx.uid}:${ctx.gid}`,
"/app/action/node_modules",
],
{ stdio: "ignore", cwd: ctx.actionDir }
);
}
/**
* escape a string for embedding in a double-quoted shell context.
* handles: backslash, double quote, dollar sign, backtick.
*/
function escapeForDoubleQuotes(str: string): string {
return str.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\$/g, "\\$").replace(/`/g, "\\`");
}
export function buildDockerRunArgs(config: DockerRunArgsContext): string[] {
const args: string[] = [
"run",
"--rm",
"-t",
"--privileged", // needed for PID namespace isolation (unshare --pid)
"-v",
`${config.ctx.actionDir}:/app/action:cached`,
"-v",
`${config.volumeName}:/app/action/node_modules`,
"-w",
"/app/action",
];
args.push(...config.envFlags);
args.push(...config.sshSetup.sshFlags);
// escape nodeCmd for embedding in su -c "..." context
const escapedNodeCmd = escapeForDoubleQuotes(config.nodeCmd);
// run as root initially, setup sudo for a test user, then run tests as that user
// this simulates GHA environment where sudo is available
const setupCmd = [
// install sudo (node:24 is Debian-based) - check if already installed first
`which sudo > /dev/null 2>&1 || (apt-get update -qq && apt-get install -qq -y sudo > /dev/null 2>&1)`,
// remove any existing user/group with the same uid/gid (e.g. node:24 has "node" at 1000:1000)
`existing_user=$(getent passwd ${config.ctx.uid} | cut -d: -f1) && [ -n "$existing_user" ] && [ "$existing_user" != "testuser" ] && userdel "$existing_user" 2>/dev/null || true`,
`existing_group=$(getent group ${config.ctx.gid} | cut -d: -f1) && [ -n "$existing_group" ] && [ "$existing_group" != "testuser" ] && groupdel "$existing_group" 2>/dev/null || true`,
// create user matching host uid/gid for file permissions
`id testuser > /dev/null 2>&1 || (groupadd -g ${config.ctx.gid} testuser 2>/dev/null || true; useradd -u ${config.ctx.uid} -g ${config.ctx.gid} -m -s /bin/bash testuser 2>/dev/null || true)`,
// configure passwordless sudo (like GHA runners) - check if already configured
`grep -q "testuser ALL" /etc/sudoers 2>/dev/null || echo "testuser ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers`,
// setup directories
`mkdir -p /tmp/home/.config /tmp/home/.cache`,
`chown -R ${config.ctx.uid}:${config.ctx.gid} /tmp/home /app/action/node_modules`,
// install deps as user
`su testuser -c "corepack pnpm install --frozen-lockfile --ignore-scripts"`,
// run test as user - nodeCmd is escaped for double-quote context
`su testuser -c "${escapedNodeCmd}"`,
].join(" && ");
args.push(
"-e",
"COREPACK_ENABLE_DOWNLOAD_PROMPT=0",
"-e",
"HOME=/tmp/home",
"-e",
"TMPDIR=/tmp",
// always set CI=true in docker to enable sandbox - this is critical for security tests
// without this, PID namespace isolation is skipped and tests may pass vacuously
"-e",
"CI=true",
"node:24",
"bash",
"-c",
`${config.sshSetup.sshSetupCmd}${setupCmd}`
);
return args;
}
export type RunInDockerOptions = {
actionDir: string;
args: string[];
nodeCmd: string;
volumeName: string;
envFilterMode: EnvFilterMode;
onStart?: () => void;
};
export function runInDocker(options: RunInDockerOptions): SpawnSyncReturns<Buffer> {
const ctx = buildDockerRunContext({
actionDir: options.actionDir,
args: options.args,
});
assertDockerSupported(ctx);
const sshSetup = buildSshSetup(ctx);
const envFlags = buildEnvFlags(ctx, options.envFilterMode);
initializeNodeModulesVolume({
actionDir: ctx.actionDir,
volumeName: options.volumeName,
uid: ctx.uid,
gid: ctx.gid,
});
if (options.onStart) {
options.onStart();
}
return spawnSync(
"docker",
buildDockerRunArgs({
ctx,
envFlags,
nodeCmd: options.nodeCmd,
sshSetup,
volumeName: options.volumeName,
}),
{ stdio: "inherit", cwd: ctx.actionDir }
);
}
+45 -12
View File
@@ -1,6 +1,7 @@
import type { ToolState } from "../toolState.ts";
import { getApiUrl } from "./apiUrl.ts";
import { buildPullfrogFooter } from "./buildPullfrogFooter.ts";
import { log } from "./cli.ts";
import { createOctokit, parseRepoContext } from "./github.ts";
import { updateProgressComment } from "./progressComment.ts";
import { getGitHubInstallationToken } from "./token.ts";
@@ -9,16 +10,20 @@ interface ReportErrorParams {
toolState: ToolState;
error: string;
title?: string;
/**
* When the run has no pre-existing progress comment to update (silent
* IncrementalReview / pull_request_synchronize, mode-less polls), create
* a fresh issue comment on `toolState.issueNumber` instead of returning
* silently. Used for terminal errors (BillingError, TransientError) where
* the GH job summary is the only other surface and most users never open
* it. see #775.
*/
createIfMissing?: boolean;
}
export async function reportErrorToComment(ctx: ReportErrorParams): Promise<void> {
const formattedError = ctx.title ? `${ctx.title}\n\n${ctx.error}` : ctx.error;
const comment = ctx.toolState.progressComment;
if (!comment) {
return;
}
const repoContext = parseRepoContext();
const octokit = createOctokit(getGitHubInstallationToken());
const runId = process.env.GITHUB_RUN_ID
@@ -38,14 +43,42 @@ export async function reportErrorToComment(ctx: ReportErrorParams): Promise<void
workflowRun: runId ? { owner: repoContext.owner, repo: repoContext.name, runId } : undefined,
customParts,
model: ctx.toolState.model,
fallbackFrom: ctx.toolState.modelFallback?.from,
});
await updateProgressComment(
{ octokit, owner: repoContext.owner, repo: repoContext.name },
comment,
`${formattedError}${footer}`
);
const body = `${formattedError}${footer}`;
// mark as updated so exit handler doesn't try to update again
ctx.toolState.wasUpdated = true;
const comment = ctx.toolState.progressComment;
if (comment) {
await updateProgressComment(
{ octokit, owner: repoContext.owner, repo: repoContext.name },
comment,
body
);
ctx.toolState.wasUpdated = true;
return;
}
// silent triggers (pull_request_synchronize IncrementalReview, etc.)
// intentionally have no progress comment. for terminal errors that need
// user action — billing exhaustion, transient billing-service outage —
// surface a fresh issue comment instead of leaving the GH job summary as
// the only signal. see #775.
if (!ctx.createIfMissing) return;
if (!ctx.toolState.issueNumber) return;
try {
const created = await octokit.rest.issues.createComment({
owner: repoContext.owner,
repo: repoContext.name,
issue_number: ctx.toolState.issueNumber,
body,
});
ctx.toolState.progressComment = { id: created.data.id, type: "issue" };
ctx.toolState.wasUpdated = true;
} catch (error) {
log.warning(
`[errorReport] fallback comment create failed: ${error instanceof Error ? error.message : String(error)}`
);
}
}
+54
View File
@@ -14,6 +14,7 @@ import { readFileSync, realpathSync, unlinkSync } from "node:fs";
import { log } from "./cli.ts";
import type { GitAuthServer } from "./gitAuthServer.ts";
import { filterEnv } from "./secrets.ts";
import { $ } from "./shell.ts";
import { spawn } from "./subprocess.ts";
type SafeGitSubcommand = "fetch" | "push";
@@ -181,3 +182,56 @@ export async function $git(
}
}
}
/**
* shallow-clone unreachable: when an existing local depth is too shallow for
* git to traverse to the requested ref's ancestry, the remote walk fails with
* one of these wordings (git emits the full OID via oid_to_hex, so the bound
* is 40 for SHA-1 or 64 for SHA-256). detecting both lets a single deepen
* retry recover before the error reaches the agent see issue #564 for the
* original `git_fetch` precedent and #656 for the `checkout_pr` follow-up.
*/
export const SHALLOW_UNREACHABLE_PATTERNS: RegExp[] = [
/Could not read [a-f0-9]{40,64}/,
/remote did not send all necessary objects/,
];
/**
* large enough to clear the merge base on most real-world PRs without
* downloading the full history; matches the fallback used by
* `checkoutPrBranch` when the GitHub compare API is unavailable.
*/
export const DEEPEN_RETRY_DEPTH = 1000;
/**
* authenticated `git fetch` that recovers from shallow-unreachable errors
* by retrying once with `--deepen=1000`. callers pass the same args they
* would to `$git("fetch", ...)`; on shallow-unreachable failures in a
* shallow repo, the second attempt prepends `--deepen=N` and strips any
* caller-supplied `--depth=` (the two flags are mutually exclusive, and
* the caller's depth is what got us into this mess).
*
* non-shallow-unreachable errors and non-shallow repos rethrow unchanged,
* so this is safe to wrap any fetch without changing fast-path behavior.
*/
export async function $gitFetchWithDeepen(
args: string[],
options: GitAuthOptions,
label?: string
): Promise<GitResult> {
try {
return await $git("fetch", args, options);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const isShallowUnreachable = SHALLOW_UNREACHABLE_PATTERNS.some((p) => p.test(msg));
if (!isShallowUnreachable) throw err;
const isShallow =
$("git", ["rev-parse", "--is-shallow-repository"], { log: false }).trim() === "true";
if (!isShallow) throw err;
log.info(
`» ${label ?? "git fetch"} hit shallow-unreachable error, retrying with --deepen=${DEEPEN_RETRY_DEPTH}`
);
const retryArgs = args.filter((a) => !a.startsWith("--depth="));
return await $git("fetch", [`--deepen=${DEEPEN_RETRY_DEPTH}`, ...retryArgs], options);
}
}

Some files were not shown because too many files have changed in this diff Show More