PR #790 (the first bot-authored models-bump PR) shipped a broken bump for
openrouter/gemini-flash: the bot pattern-matched the parallel google bump
and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the
OpenRouter API but not on models.dev's openrouter section — the catalog
OpenCode actually reads. The slug failed at runtime with `Model not found`.
Two CI gaps let it through:
1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*`
alias as a "passthrough", smoke-testing only one canary per routing
layer. But those aren't passthroughs — each is a distinct models.dev
catalog entry that can drift independently of the direct-provider
mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25).
Only `bedrock/byok` stays pruned (sentinel resolve).
2. `models-catalog` test (the integrity gate that asserts every resolve
exists on models.dev) was main-only by design — to keep upstream catalog
churn from blocking unrelated PRs. But it's exactly the test we want
running on the bot's own catalog edits. Add `pullfrog/models-bump`
head-ref to its trigger.
Also tighten the bot prompt in models-bump.yml: new rule 0 requires every
new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the
alias's own `candidates[]` in models-bump-context.json — the deterministic
preprocessor only emits candidates sourced from models.dev's mirror, so
this gates against the cross-alias pattern-matching that broke PR #790.
For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter
API), which is necessary but not sufficient; the `models-catalog` job is
the authoritative models.dev check.
Verified locally:
- baseline `pnpm -C action test:catalog` passes 133 tests
- simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash`
into action/models.ts) and the catalog test fails with the right
assertion: `model "google/gemini-3.5-flash" not found under openrouter
on models.dev`
- `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every
openrouter/* alias and every keyed opencode/* alias now smoked
Colin McDonnell
69c7d4b8cd