max effort burns roughly 2x the wall time per turn for marginal quality
gain. high is the model's tuned default ('equivalent to not setting the
parameter' per Anthropic docs). full-send can be reintroduced as an
opt-in per-run override later if needed.
* remove opencode/gpt-5-nano and opencode/mimo-v2-pro-free from catalog
#7 delete aliases. both were listed as `isFree: true, envVars: []` but
neither is keyless on opencode zen, producing a hard-fail
`UnknownError: Model not found: opencode/<id>` on every run without an
opencode_api_key. fixespullfrog/app#691 (5 runs across 3 repos, 100%
failure rate in the last 24h).
root cause: opencode's provider gate
(`packages/opencode/src/provider/provider.ts` `opencode:` loader) keeps
a zen model only when models.dev reports `cost.input === 0` for it,
then signs requests with `apiKey: "public"`. paid zen models get
deleted from the autoloaded set and opencode surfaces the deletion as
"model not found".
- `opencode/gpt-5-nano`: models.dev reports `cost: {input: 0.05, output:
0.4, cache_read: 0.005}`. paid → requires `OPENCODE_API_KEY`.
- `opencode/mimo-v2-pro-free`: free on models.dev but not in
`https://opencode.ai/zen/v1/models` — zen never served it, so even
the public-key path fails.
remaining free aliases (`opencode/big-pickle`,
`opencode/minimax-m2.5-free`) both pass both checks (cost.input === 0
in models.dev AND present in zen's served list) and continue to work
without a key — verified against the opencode source.
callers swept: `action/utils/apiKeys.test.ts`, `action/models.test.ts`,
`action/test/list-aliases.ts`, `action/test/model-smoke.ts`,
`components/ModelSelector.tsx` (`modelIdToUpstream`),
`wiki/model-resolution.md`, `wiki/models-catalog.md`. wrote up the
free-zen verification rule in models-catalog so the next maintainer
can sanity-check both conditions before adding any `isFree` alias.
users with a stored `opencode/gpt-5-nano` or `opencode/mimo-v2-pro-free`
will now fall through `resolveCliModel → undefined` into the auto-select
path — a strict improvement over today's hard fail. no DB migration
needed; the slugs are simply unknown and treated like any other
unrecognized stored value.
* rework: keep mimo deprecated, demote gpt-5-nano to paid, add free-zen invariants
revised approach after the first commit over-corrected. mimo was never
broken at runtime — `fallback: "opencode/big-pickle"` already routes
stored values through to a real free model before any zen call. the
literal `opencode/mimo-v2-pro-free` being absent from zen's served list
is irrelevant because `resolveCliModel` walks the chain first. restoring
it as-is.
the actual bug was `opencode/gpt-5-nano`: marked `isFree: true,
envVars: []` but `models.dev` reports `cost: {input: 0.05, output: 0.4}`
on the opencode provider, so opencode's keyless gate
(`packages/opencode/src/provider/provider.ts` `opencode:`) deletes it
when `OPENCODE_API_KEY` is missing and the run hard-fails with
`UnknownError: Model not found: opencode/gpt-5-nano`. demoting it to a
regular paid zen alias (drop `isFree`/`envVars: []`, add
`openRouterResolve: "openrouter/openai/gpt-5-nano"` — verified to exist
on openrouter at the same price). users without `OPENCODE_API_KEY` now
get our explicit "no API key found" error pointing at the secrets page
instead of opencode's cryptic upstream error. confirmed via
`https://opencode.ai/zen/v1/models` that zen serves no free GPT
variants, so there's no cheaper-than-`gpt-mini` free option to suggest
in its place.
CI gap analysis (why this slipped through):
- `models-catalog.main.test.ts` only checked existence + `status !==
"deprecated"` on models.dev. paid-model-marked-free regressions and
zen-served-list drift both passed.
- `models-live` (`model-smoke.ts`) runs with `OPENCODE_API_KEY` in env,
so the keyless deletion gate never fires. `gpt-5-nano` returned "OK"
in CI even though end users hit a hard fail.
- `model-smoke.ts` walks the fallback chain, so mimo would have been
smoked as big-pickle anyway — the dead resolve target was never
exercised directly. (this is the right design; the gap is at the
catalog layer, not the smoke layer.)
new tests:
- PR-blocking, static (`action/test/models.test.ts`, `isFree
invariants`): every `isFree` alias must live under `opencode`, have
`envVars: []`, omit `openRouterResolve`, AND have a fallback chain
whose terminal alias is also `isFree` (catches "deprecate a free
alias to a paid target" — the worst silent-charge regression).
- main-only, network (`action/test/models-catalog.main.test.ts`,
`opencode Zen served list`): every alias whose terminal-fallback
resolve is `opencode/*` must appear in
`https://opencode.ai/zen/v1/models`. catches zen dropping a model
from its served list.
- main-only, network (same file, `isFree models.dev cost`): every
`isFree` alias's terminal-fallback resolve must have `cost.input ===
0` in the `opencode` provider block on `models.dev`. would have
caught `gpt-5-nano` at the next models-bump run.
both network tests dedupe on terminal resolve, so deprecated aliases
sharing a target aren't double-counted. `pnpm vitest run`: 113 static
tests pass. `pnpm test:catalog`: 142 network tests pass against the
live `models.dev`, `openrouter.ai`, and `opencode.ai/zen/v1/models`
endpoints.
wiki/models-catalog.md: rewrote the new "Free-Zen aliases need Zen-side
verification" section to (a) describe the two conditions, (b) note
that a fallback to an isFree alias is the legitimate escape hatch
(mimo's pattern), and (c) point at the three tests by name so the next
maintainer can find the enforcement surface. wiki/model-resolution.md
points at the new section.
* make gpt-5-nano a deprecated free alias falling back to big-pickle
revising the previous "demote to paid" approach. the user-facing
ergonomics are cleaner: anyone who picked gpt-5-nano under the "Free"
badge gets transparent-upgraded to a real free model (big-pickle)
instead of suddenly being asked to set OPENCODE_API_KEY. matches the
existing mimo pattern exactly. the dropdown already filters
`!a.fallback`, so the slug disappears from the picker on its own and
the trigger renders it as "Big Pickle" via `resolveDisplayAlias`.
no other catalog or test surface changes — the isFree invariants and
the main-only zen/cost checks still pass (gpt-5-nano's terminal is
now big-pickle, which is both isFree and zero-cost on models.dev,
deduping with big-pickle's own row in both network tests).
* revise: keep gpt-5-nano as paid alias, backfill affected DB rows instead
dropping the deprecated-alias approach. `opencode/gpt-5-nano` is a
legitimate cheap paid model people may want with BYOK
(`OPENCODE_API_KEY`) — giving it `fallback: "opencode/big-pickle"`
would foreclose that for everyone going forward. correct fix is two
parts:
(a) reclassify in the catalog as a regular paid OpenCode alias:
- drop `isFree: true` and `envVars: []` so the local validator
demands `OPENCODE_API_KEY`
- add `openRouterResolve: "openrouter/openai/gpt-5-nano"` to satisfy
the completeness test and route BYOK-via-OpenRouter users
- no `fallback` — slug stays visible in the picker as a paid option
(b) one-shot DB backfill of provably-affected repos
(`scripts/backfill-gpt5-nano-affected.ts`). scope:
- `Repo.model = "opencode/gpt-5-nano"`
- AND at least one `WorkflowRun` with `inputTokens IS NULL` (evidence
of an attempted run that didn't get past the model-init gate)
skipped intentionally:
- repos whose runs have `inputTokens > 0` — they have a key, gpt-5-
nano works for them
- repos with zero WorkflowRun rows — never dispatched; touching them
would be presumptuous
- `LearningsRevision.model` — audit trail of which model authored a
revision, rewriting it would falsify history
ran against .env.prod: 2 repos stored the slug; 1 was provably
affected (sodown4thecause/seobot, 5/5 zero-token runs — matches #691's
3 failed runs from this repo plus 2 outside the 24h audit window).
1 was an internal test account that never dispatched (left as-is).
applied: 1 row updated. confirmed idempotent on re-run.
the other two repos in #691 (Nantiee/ALTA-breast-pump-tool,
keksiqc/ansible-setup-linux) don't store the slug in `Repo.model`;
their failed dispatches passed the model inline in the
`workflow_dispatch` `prompt` payload, so the catalog fix alone (no
longer offering it as free) is what helps them.
tests:
- models.test.ts: `getModelEnvVars("opencode/gpt-5-nano")` now
returns `["OPENCODE_API_KEY"]`, moved into the keyed-model group
- apiKeys.test.ts: added "throws without OPENCODE_API_KEY" case
- isFree invariants from the previous commit still pass — gpt-5-nano
no longer triggers them since it's no longer isFree
- main-only catalog tests still pass (gpt-5-nano served by Zen, just
paid; no isFree cost check applies)
* docs: drop stale GPT Nano + MiMo V2 Pro from free-tier lists
addressing pullfrog auto-review feedback on #695. three mintlify pages
still advertised both as keyless after the catalog pivot, which now
makes the docs affirmatively wrong rather than merely stale:
- gpt nano is paid in the catalog (no `isFree`, inherits
`OPENCODE_API_KEY`); a user following the docs would hit the same
"missing API key" failure that's described 4 lines below in
`docs/keys.mdx`.
- mimo v2 pro is hidden from the picker (`fallback` triggers
`ModelSelector`'s `!a.fallback` filter); the alias only exists for
legacy stored-value resolution. a user reading the docs cannot
actually pick it.
surviving picker-visible free set: Big Pickle and MiniMax M2.5.
- `docs/keys.mdx`: drop both bullets from the "Free models" list
- `docs/billing.mdx`: drop both bullets from the "Free models" list
- `docs/getting-started.mdx`: collapse the inline mention from a
4-model list to "Big Pickle and MiniMax M2.5"
* address third review: picker grouping + backfill classifier honesty
i had not pulled the third pullfrog review (`02:17:28Z`) when i declared
reviews triaged after the docs sweep — the fourth review flagged that
three findings remained pending. addressing them now.
1. picker grouping for now-selectable paid gpt-5-nano. when i removed
`"gpt-5-nano": "OpenAI"` from `modelIdToUpstream` in the previous
pivot-to-paid commit, i mistook it for dead code. it's not — the map
IS consulted for paid opencode aliases via `groupByUpstream →
getUpstreamLabel` inside the OpenCode submenu's
`renderSubContent`. without the entry, `gpt-5-nano` falls back to
`getProviderDisplayName("opencode")` = "OpenCode" and gets dropped
into its own sub-header instead of joining opencode/gpt,
opencode/gpt-pro, opencode/gpt-mini under the "OpenAI" upstream
group. re-added with an explanatory comment so the next refactor
doesn't make the same mistake.
2. JSDoc / code mismatch in `scripts/backfill-gpt5-nano-affected.ts`.
the JSDoc said "at least one `WorkflowRun` with `inputTokens IS
NULL`" but the code is `no WorkflowRun has inputTokens > 0` — a
strictly broader filter (catches `null` AND `0`). rewrote the scope
block to describe what the code actually does, with the operative
classifier spelled out: "a billable run with `inputTokens > 0` is
proof the agent successfully reached and called the model".
3. classifier breadth (raised in the same review). honest answer: the
"no positive-token run" filter IS a heuristic — a repo whose only
dispatches happened to fail or cancel for unrelated reasons would
get false-positive-classified A. for THIS one-shot population (2
repos, 1 with 5/5 zero-token runs — strong systematic-failure
signal) the heuristic was good enough and the dry-run inspection
confirmed before APPLY. for any larger reuse of this pattern, you
need to cross-reference the runtime error string (`UnknownError:
Model not found: opencode/gpt-5-nano`) from GitHub Actions logs or
Better Stack — that error doesn't live on `WorkflowRun` rows. added
a "Classifier limitations" section to the JSDoc making this
explicit.
nothing about the actual applied backfill changes — the prod write
(1 repo: sodown4thecause/seobot → opencode/big-pickle) is unchanged
and re-running the script remains idempotent.
* fix(mcp): sanitize for gemini when model is unresolved
isGeminiRouted() previously required the effective model string to
contain "gemini" — but when payload.model="auto" (or any unresolved
slug) reaches addTools(), `effective` is the literal "auto", which
doesn't match. opencode then auto-selects gemini *after* the MCP
server has registered raw arktype schemas, and every tool turn dies
on `function_declarations[*].properties[*].any_of[*].enum: only
allowed for STRING type`.
widen the gate: any unresolved specifier (undefined / "auto" / a
slug without a `provider/` prefix) is treated as gemini-routed and
sanitized. the transforms are universally compatible normalizations
so the false-positive cost is negligible. tighten case 3 to preserve
`description` so the only lossy path no longer drops operator-facing
context.
fixes#676.
* revert case-3 description preservation
per pullfrog review on #697: keeping `description` as a peer of
`anyOf`/`oneOf` directly contradicts the file's own header (lines
19-21) and the upstream opencode #14659 rationale that gates this
sanitizer — gemini requires anyOf to be the ONLY field on a schema
node, sibling keywords trigger
`anyOf must be the only field in a schema node`. the change was
speculative scope creep with no evidence, and would silently
re-introduce a different gemini failure for any future schema using
`.describe().or(...)`. the bug fix for #676 doesn't need it (arktype
doesn't emit non-collapsible anyOf for current tool schemas).
* action: strip Content-Type on body-less apiFetch requests (#692)
Vercel's Next.js lambda adapter (Next 16.1.x) attempts to decode a
request body when Content-Type is set and throws
`SyntaxError: Unexpected end of data` before delegating to the route
handler, returning a 500. Hit /run-context exclusively because it was
the only body-less GET that sent `Content-Type: application/json`.
- Drop `Content-Type: application/json` from the GET in
`action/utils/runContext.ts` (meaningless on a body-less request).
- Defensively strip any `content-type` header in `action/utils/apiFetch.ts`
when no body is present so future callers can't reintroduce this.
* apiFetch: soften comment — empirical observation, RFC 9110 §8.3 framing
* billing: $10 signup credit + lazy claim modal; disable welcome credit promo
Adds a per-Account $10 Router signup credit granted on first Router-tab
mount via a new admin-gated POST /api/account/[owner]/signup-credit/claim.
The endpoint is idempotent — the inserted CreditGrant row IS the dedup
state, so subsequent calls return granted:false. Client SignupCreditModal
fires the POST on mount (only when modelAccessMode === "router") and
opens a celebratory dialog when granted:true.
Disables the legacy welcome credit ($10 on first card add) via a new
WELCOME_CREDIT_PROMO_ACTIVE = false flag in utils/stripe.ts. Code path
stays intact — flip the flag to revive. Strips the now-untruthful
"$10 on enabling billing" copy from BillingCard, EnableRouterPrompt,
triggerWorkflow paywall comment, action router_requires_card summary,
email snippet, billing/pricing docs and wiki.
Cuts WELCOME_CREDIT_CENTS from 2000 to 1000 to reflect the lower amount
that would land if the flag is ever re-enabled. Adds "signup" reason
mapping to BillingCard wallet history.
Verified end-to-end against dev: admin+Router fires modal, admin+BYOK
gate-blocks mount, BYOK→Router transition fires modal on click, member
and collaborator paths skip the mount entirely, reload after grant is
idempotent. Wallet history shows "Router signup credit +$10.00".
* billing: address PR review (race fix, copy sweep, modal retry)
Correctness:
- Add @@unique([accountId, reason]) on CreditGrant + migration. The prior
check-then-insert pattern in /signup-credit/claim and finalizeCheckoutSession
raced at READ COMMITTED — two concurrent admin tabs could land two grants of
the same reason on a fresh account ($10 each). Both write sites now rely on
the unique index for dedup (P2002 = "already granted") and route updated to
catch P2002 cleanly. Verified zero existing duplicates in prod before
migration.
- Add log.info on signup grant insert so a successful grant has any chance of
being caught by ops monitoring.
- Add retry: 2 with backoff to the claim mutation. Endpoint is idempotent so
a server-side success that lost its response cleanly returns granted:false
on retry.
Public copy that still advertised the (now-deleted) $20 welcome credit:
- app/page.tsx landing pricing card
- emails/announceBilling.ts broadcast template
- docs/keys.mdx BYOK note
- components/AgentSettings.tsx Router-without-billing warning
- utils/stripe.ts finalizeCheckoutSession JSDoc
- utils/email/snippets.ts ROUTER_CREDIT_PS_HTML JSDoc
Wiki staleness sweep:
- wiki/billing.md TOC, mermaid diagram (signup edge added; welcome marked
dormant), test coverage list, key modules section, no-card wallet narrative
- wiki/pricing.md welcome-credit drawdown reference
- Rewrote my own internally-inconsistent dormancy paragraph to be honest
about the $20-historical / $10-on-revival framing.
Trivia:
- ModelAccessCard JSX comment had a literal \\u2192 instead of →.
* billing: address PR review round 2
- Replace try/catch P2002 inside finalizeCheckoutSession's prisma.$transaction
with createMany skipDuplicates. The previous form is broken on Postgres: a
unique-violation poisons the surrounding TX, so the catch block returns
cleanly but the outer commit fails and the account.update (stripeCustomerId)
silently rolls back too. Currently armed only behind the dormant welcome-
credit flag, but would have broken billing enablement the moment the flag
flipped. createMany skipDuplicates yields a single ON CONFLICT DO NOTHING
statement that returns count: 0 cleanly without aborting the TX.
- Apply the same createMany skipDuplicates pattern to the signup-credit route
too — drops the exception-as-control-flow Prisma namespace import and is
more uniform with the welcome path.
- Drop the now-orphaned credit_grants_accountId_idx in the same migration.
The schema removed @@index([accountId]) when @@unique([accountId, reason])
was added (covered by the leftmost prefix), but the migration only added
the unique index, leaving prod drifted.
* billing: fix stale finalizeCheckoutSession JSDoc
The function-level JSDoc still described the abandoned try/catch P2002
mechanism after switching to createMany skipDuplicates. The inline
comment + code now agree on the new ON CONFLICT DO NOTHING shape.
* billing: decouple first-card alert, drop vestigial billing field, fix modal cents; sync copy
* docs+homepage: align Router credit copy with signup claim (no card-on-add carrot)
* homepage: add pricing screenshot and pay-as-you-go promo line
* billing: fix once-per-lifetime misframe on first-card alert
* billing: suppress signup credit for prior welcome-credit recipients
* billing: drop bogus '1000 users' cap; invalidate billing on signup-credit settle
`persistLearnings` only emitted `log.info("» learnings updated")` on
success; every failure path (non-2xx, fetch throw, 10s timeout) was
`log.debug`, which is hidden unless `ACTIONS_RUNNER_DEBUG=true`. Survey
of recent runs caught at least one case where the agent definitively
edited the tmpfile but no DB row was written and no warning surfaced.
Promote both failure paths to `log.warning` so dropped agent work is
visible in CI logs. The unchanged-from-seed short-circuit stays at
debug — that's a genuine no-op.
* cancel + restart workflow run when @pullfrog mention is edited
- add `WorkflowRun.triggeringCommentId` (BigInt?, indexed) so the webhook
handler can find the run that was fired by a given comment
- thread `triggeringCommentId` through `reserveRun` / `triggerWorkflow`
- factor `dispatchMentionRun` out of `issue_comment_created` so the same
shape is reused on edit
- replace the `issue_comment_edited` stub: re-evaluates the trigger gate,
cancels prior runs (`octokit.rest.actions.cancelWorkflowRun` + DB
status='cancelled'), then re-dispatches with a `previousRunsNote`
appended to `eventInstructions` so the agent acknowledges the prior
run/PR/artifacts in its summary
- if the edit removes `@pullfrog`, cancel only (no restart)
Co-authored-by: Cursor <cursoragent@cursor.com>
* thread previousRunsNote via dedicated payload field
user prompt has precedence over eventInstructions, so stuffing the
prior-runs note into eventInstructions made it vanish whenever the
trigger comment contained an @pullfrog mention (which is always for the
edit path). pass it as its own payload field and render it alongside the
user's task so the agent actually sees it.
* delete cancelled run's progress comment on edit-restart
so the issue thread doesn't accumulate "This run was cancelled" stubs
on every edit. only deletes for runs we actively cancel; runs that were
already terminal (e.g. completed before the edit) keep their summary
comment in the thread, and `previousRunsNote` links to it so the new
agent can reference prior work.
post-cleanup is race-safe: the action's `validateStuckProgressComment`
swallows the 404 from the deleted comment and exits cleanly, so the
old run's post step cannot clobber the new run's leaping comment.
Co-authored-by: Cursor <cursoragent@cursor.com>
* also cancel + delete progress comment when triggering comment is deleted
mirrors the edit-removes-@pullfrog path: when an @pullfrog comment that
fired a run is hard-deleted, look up any prior runs by triggeringCommentId,
GH-cancel running ones, and delete their leaping progress comments.
skips trigger-gate re-eval (we're tearing down a run, not firing one) and
performs no restart. reuses the existing cancelRunsForTriggeringComment
helper; the returned previousRunsNote is discarded since no dispatch
follows.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: move cancellation before trigger gate in issue_comment_edited
cancelRunsForTriggeringComment now runs before the triggerEnabled check,
so edits that remove @pullfrog still cancel in-flight runs even when the
repo mention trigger is currently disabled (e.g. for non-collaborators).
* anneal: scope cancel updates per-row + simplify edit gate
- replace blanket updateMany on (triggeringCommentId, repoId) with per-row, status-guarded updates so a parallel handler's freshly-reserved run cannot be clobbered into cancelled by a racing edit delivery.
- drop wasMention/isMention early-break in issue_comment_edited; always run cancelRunsForTriggeringComment (DB is the canonical "did this comment ever trigger a run" source). closes the missing-changes.body.from edge and lets us tear down a still-running prior run even if the admin disabled the mention trigger mid-flight.
- buildPreviousRunsNote returns undefined (not "") when no link lines materialize.
- doc cleanups + wiki/modes.md addendum noting issue_comment_edited / _deleted now drive cancel + restart.
Co-authored-by: Cursor <cursoragent@cursor.com>
* address review feedback on cancel/restart semantics
- guard workflow_run.completed update against status='cancelled' so a
successful-but-uncancellable GH Actions job can't resurrect a cancelled
row (and re-bill it) via the completed webhook.
- bucket only status='completed' runs into `preserved` in
cancelRunsForTriggeringComment; cancelled/failed prior runs have stubs
as their progress comment, not summaries worth referencing.
- emit previousRunsNote for the runId-null cancel case so the restarted
agent always knows when it's superseding a prior dispatch.
- drop the agent-forbidden `gh pr list` hint and soften 'was cancelled'
to 'was signalled to cancel' in the note body.
- post a fallback comment when the edit-path dispatch fails (prior run
already torn down and progress comment already deleted).
- symmetrize the delete-handler's pullfrog guard with the edit handler
(key off hook.comment.user, not hook.sender).
- trim misleading comments on the per-row DB update guard.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-11 23:20:44 +00:00
15 changed files with 203 additions and 33 deletions
it(`${alias.slug} terminal resolve ${terminal.resolve} is served by Zen`,()=>{
expect(
zenIds.has(parsed.modelId),
`terminal resolve "${terminal.resolve}" for alias "${alias.slug}" is not in https://opencode.ai/zen/v1/models — Zen no longer serves it. either point a fallback at a Zen-served alias or remove the entry.`
expect(model,`terminal resolve "${terminal.resolve}" missing on models.dev`).toBeDefined();
expect(
model?.cost?.input,
`isFree alias "${alias.slug}" walks to "${terminal.resolve}" which reports cost.input=${model?.cost?.input} on models.dev — either repoint the fallback or drop \`isFree\``
@@ -88,7 +88,6 @@ export async function fetchRunContext(params: {
try{
constheaders: Record<string,string>={
Authorization:`Bearer ${params.token}`,
"Content-Type":"application/json",
};
if(params.oidcToken){
headers["X-GitHub-OIDC-Token"]=params.oidcToken;
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.