* remove opencode/gpt-5-nano and opencode/mimo-v2-pro-free from catalog
#7 delete aliases. both were listed as `isFree: true, envVars: []` but
neither is keyless on opencode zen, producing a hard-fail
`UnknownError: Model not found: opencode/<id>` on every run without an
opencode_api_key. fixespullfrog/app#691 (5 runs across 3 repos, 100%
failure rate in the last 24h).
root cause: opencode's provider gate
(`packages/opencode/src/provider/provider.ts` `opencode:` loader) keeps
a zen model only when models.dev reports `cost.input === 0` for it,
then signs requests with `apiKey: "public"`. paid zen models get
deleted from the autoloaded set and opencode surfaces the deletion as
"model not found".
- `opencode/gpt-5-nano`: models.dev reports `cost: {input: 0.05, output:
0.4, cache_read: 0.005}`. paid → requires `OPENCODE_API_KEY`.
- `opencode/mimo-v2-pro-free`: free on models.dev but not in
`https://opencode.ai/zen/v1/models` — zen never served it, so even
the public-key path fails.
remaining free aliases (`opencode/big-pickle`,
`opencode/minimax-m2.5-free`) both pass both checks (cost.input === 0
in models.dev AND present in zen's served list) and continue to work
without a key — verified against the opencode source.
callers swept: `action/utils/apiKeys.test.ts`, `action/models.test.ts`,
`action/test/list-aliases.ts`, `action/test/model-smoke.ts`,
`components/ModelSelector.tsx` (`modelIdToUpstream`),
`wiki/model-resolution.md`, `wiki/models-catalog.md`. wrote up the
free-zen verification rule in models-catalog so the next maintainer
can sanity-check both conditions before adding any `isFree` alias.
users with a stored `opencode/gpt-5-nano` or `opencode/mimo-v2-pro-free`
will now fall through `resolveCliModel → undefined` into the auto-select
path — a strict improvement over today's hard fail. no DB migration
needed; the slugs are simply unknown and treated like any other
unrecognized stored value.
* rework: keep mimo deprecated, demote gpt-5-nano to paid, add free-zen invariants
revised approach after the first commit over-corrected. mimo was never
broken at runtime — `fallback: "opencode/big-pickle"` already routes
stored values through to a real free model before any zen call. the
literal `opencode/mimo-v2-pro-free` being absent from zen's served list
is irrelevant because `resolveCliModel` walks the chain first. restoring
it as-is.
the actual bug was `opencode/gpt-5-nano`: marked `isFree: true,
envVars: []` but `models.dev` reports `cost: {input: 0.05, output: 0.4}`
on the opencode provider, so opencode's keyless gate
(`packages/opencode/src/provider/provider.ts` `opencode:`) deletes it
when `OPENCODE_API_KEY` is missing and the run hard-fails with
`UnknownError: Model not found: opencode/gpt-5-nano`. demoting it to a
regular paid zen alias (drop `isFree`/`envVars: []`, add
`openRouterResolve: "openrouter/openai/gpt-5-nano"` — verified to exist
on openrouter at the same price). users without `OPENCODE_API_KEY` now
get our explicit "no API key found" error pointing at the secrets page
instead of opencode's cryptic upstream error. confirmed via
`https://opencode.ai/zen/v1/models` that zen serves no free GPT
variants, so there's no cheaper-than-`gpt-mini` free option to suggest
in its place.
CI gap analysis (why this slipped through):
- `models-catalog.main.test.ts` only checked existence + `status !==
"deprecated"` on models.dev. paid-model-marked-free regressions and
zen-served-list drift both passed.
- `models-live` (`model-smoke.ts`) runs with `OPENCODE_API_KEY` in env,
so the keyless deletion gate never fires. `gpt-5-nano` returned "OK"
in CI even though end users hit a hard fail.
- `model-smoke.ts` walks the fallback chain, so mimo would have been
smoked as big-pickle anyway — the dead resolve target was never
exercised directly. (this is the right design; the gap is at the
catalog layer, not the smoke layer.)
new tests:
- PR-blocking, static (`action/test/models.test.ts`, `isFree
invariants`): every `isFree` alias must live under `opencode`, have
`envVars: []`, omit `openRouterResolve`, AND have a fallback chain
whose terminal alias is also `isFree` (catches "deprecate a free
alias to a paid target" — the worst silent-charge regression).
- main-only, network (`action/test/models-catalog.main.test.ts`,
`opencode Zen served list`): every alias whose terminal-fallback
resolve is `opencode/*` must appear in
`https://opencode.ai/zen/v1/models`. catches zen dropping a model
from its served list.
- main-only, network (same file, `isFree models.dev cost`): every
`isFree` alias's terminal-fallback resolve must have `cost.input ===
0` in the `opencode` provider block on `models.dev`. would have
caught `gpt-5-nano` at the next models-bump run.
both network tests dedupe on terminal resolve, so deprecated aliases
sharing a target aren't double-counted. `pnpm vitest run`: 113 static
tests pass. `pnpm test:catalog`: 142 network tests pass against the
live `models.dev`, `openrouter.ai`, and `opencode.ai/zen/v1/models`
endpoints.
wiki/models-catalog.md: rewrote the new "Free-Zen aliases need Zen-side
verification" section to (a) describe the two conditions, (b) note
that a fallback to an isFree alias is the legitimate escape hatch
(mimo's pattern), and (c) point at the three tests by name so the next
maintainer can find the enforcement surface. wiki/model-resolution.md
points at the new section.
* make gpt-5-nano a deprecated free alias falling back to big-pickle
revising the previous "demote to paid" approach. the user-facing
ergonomics are cleaner: anyone who picked gpt-5-nano under the "Free"
badge gets transparent-upgraded to a real free model (big-pickle)
instead of suddenly being asked to set OPENCODE_API_KEY. matches the
existing mimo pattern exactly. the dropdown already filters
`!a.fallback`, so the slug disappears from the picker on its own and
the trigger renders it as "Big Pickle" via `resolveDisplayAlias`.
no other catalog or test surface changes — the isFree invariants and
the main-only zen/cost checks still pass (gpt-5-nano's terminal is
now big-pickle, which is both isFree and zero-cost on models.dev,
deduping with big-pickle's own row in both network tests).
* revise: keep gpt-5-nano as paid alias, backfill affected DB rows instead
dropping the deprecated-alias approach. `opencode/gpt-5-nano` is a
legitimate cheap paid model people may want with BYOK
(`OPENCODE_API_KEY`) — giving it `fallback: "opencode/big-pickle"`
would foreclose that for everyone going forward. correct fix is two
parts:
(a) reclassify in the catalog as a regular paid OpenCode alias:
- drop `isFree: true` and `envVars: []` so the local validator
demands `OPENCODE_API_KEY`
- add `openRouterResolve: "openrouter/openai/gpt-5-nano"` to satisfy
the completeness test and route BYOK-via-OpenRouter users
- no `fallback` — slug stays visible in the picker as a paid option
(b) one-shot DB backfill of provably-affected repos
(`scripts/backfill-gpt5-nano-affected.ts`). scope:
- `Repo.model = "opencode/gpt-5-nano"`
- AND at least one `WorkflowRun` with `inputTokens IS NULL` (evidence
of an attempted run that didn't get past the model-init gate)
skipped intentionally:
- repos whose runs have `inputTokens > 0` — they have a key, gpt-5-
nano works for them
- repos with zero WorkflowRun rows — never dispatched; touching them
would be presumptuous
- `LearningsRevision.model` — audit trail of which model authored a
revision, rewriting it would falsify history
ran against .env.prod: 2 repos stored the slug; 1 was provably
affected (sodown4thecause/seobot, 5/5 zero-token runs — matches #691's
3 failed runs from this repo plus 2 outside the 24h audit window).
1 was an internal test account that never dispatched (left as-is).
applied: 1 row updated. confirmed idempotent on re-run.
the other two repos in #691 (Nantiee/ALTA-breast-pump-tool,
keksiqc/ansible-setup-linux) don't store the slug in `Repo.model`;
their failed dispatches passed the model inline in the
`workflow_dispatch` `prompt` payload, so the catalog fix alone (no
longer offering it as free) is what helps them.
tests:
- models.test.ts: `getModelEnvVars("opencode/gpt-5-nano")` now
returns `["OPENCODE_API_KEY"]`, moved into the keyed-model group
- apiKeys.test.ts: added "throws without OPENCODE_API_KEY" case
- isFree invariants from the previous commit still pass — gpt-5-nano
no longer triggers them since it's no longer isFree
- main-only catalog tests still pass (gpt-5-nano served by Zen, just
paid; no isFree cost check applies)
* docs: drop stale GPT Nano + MiMo V2 Pro from free-tier lists
addressing pullfrog auto-review feedback on #695. three mintlify pages
still advertised both as keyless after the catalog pivot, which now
makes the docs affirmatively wrong rather than merely stale:
- gpt nano is paid in the catalog (no `isFree`, inherits
`OPENCODE_API_KEY`); a user following the docs would hit the same
"missing API key" failure that's described 4 lines below in
`docs/keys.mdx`.
- mimo v2 pro is hidden from the picker (`fallback` triggers
`ModelSelector`'s `!a.fallback` filter); the alias only exists for
legacy stored-value resolution. a user reading the docs cannot
actually pick it.
surviving picker-visible free set: Big Pickle and MiniMax M2.5.
- `docs/keys.mdx`: drop both bullets from the "Free models" list
- `docs/billing.mdx`: drop both bullets from the "Free models" list
- `docs/getting-started.mdx`: collapse the inline mention from a
4-model list to "Big Pickle and MiniMax M2.5"
* address third review: picker grouping + backfill classifier honesty
i had not pulled the third pullfrog review (`02:17:28Z`) when i declared
reviews triaged after the docs sweep — the fourth review flagged that
three findings remained pending. addressing them now.
1. picker grouping for now-selectable paid gpt-5-nano. when i removed
`"gpt-5-nano": "OpenAI"` from `modelIdToUpstream` in the previous
pivot-to-paid commit, i mistook it for dead code. it's not — the map
IS consulted for paid opencode aliases via `groupByUpstream →
getUpstreamLabel` inside the OpenCode submenu's
`renderSubContent`. without the entry, `gpt-5-nano` falls back to
`getProviderDisplayName("opencode")` = "OpenCode" and gets dropped
into its own sub-header instead of joining opencode/gpt,
opencode/gpt-pro, opencode/gpt-mini under the "OpenAI" upstream
group. re-added with an explanatory comment so the next refactor
doesn't make the same mistake.
2. JSDoc / code mismatch in `scripts/backfill-gpt5-nano-affected.ts`.
the JSDoc said "at least one `WorkflowRun` with `inputTokens IS
NULL`" but the code is `no WorkflowRun has inputTokens > 0` — a
strictly broader filter (catches `null` AND `0`). rewrote the scope
block to describe what the code actually does, with the operative
classifier spelled out: "a billable run with `inputTokens > 0` is
proof the agent successfully reached and called the model".
3. classifier breadth (raised in the same review). honest answer: the
"no positive-token run" filter IS a heuristic — a repo whose only
dispatches happened to fail or cancel for unrelated reasons would
get false-positive-classified A. for THIS one-shot population (2
repos, 1 with 5/5 zero-token runs — strong systematic-failure
signal) the heuristic was good enough and the dry-run inspection
confirmed before APPLY. for any larger reuse of this pattern, you
need to cross-reference the runtime error string (`UnknownError:
Model not found: opencode/gpt-5-nano`) from GitHub Actions logs or
Better Stack — that error doesn't live on `WorkflowRun` rows. added
a "Classifier limitations" section to the JSDoc making this
explicit.
nothing about the actual applied backfill changes — the prod write
(1 repo: sodown4thecause/seobot → opencode/big-pickle) is unchanged
and re-running the script remains idempotent.
Colin McDonnell
d5d8a0d7ac