models-bump: harden CI and bot prompt against catalog hallucinations

PR #790 (the first bot-authored models-bump PR) shipped a broken bump for
openrouter/gemini-flash: the bot pattern-matched the parallel google bump
and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the
OpenRouter API but not on models.dev's openrouter section — the catalog
OpenCode actually reads. The slug failed at runtime with `Model not found`.
Two CI gaps let it through:

1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*`
   alias as a "passthrough", smoke-testing only one canary per routing
   layer. But those aren't passthroughs — each is a distinct models.dev
   catalog entry that can drift independently of the direct-provider
   mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25).
   Only `bedrock/byok` stays pruned (sentinel resolve).

2. `models-catalog` test (the integrity gate that asserts every resolve
   exists on models.dev) was main-only by design — to keep upstream catalog
   churn from blocking unrelated PRs. But it's exactly the test we want
   running on the bot's own catalog edits. Add `pullfrog/models-bump`
   head-ref to its trigger.

Also tighten the bot prompt in models-bump.yml: new rule 0 requires every
new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the
alias's own `candidates[]` in models-bump-context.json — the deterministic
preprocessor only emits candidates sourced from models.dev's mirror, so
this gates against the cross-alias pattern-matching that broke PR #790.
For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter
API), which is necessary but not sufficient; the `models-catalog` job is
the authoritative models.dev check.

Verified locally:
- baseline `pnpm -C action test:catalog` passes 133 tests
- simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash`
  into action/models.ts) and the catalog test fails with the right
  assertion: `model "google/gemini-3.5-flash" not found under openrouter
  on models.dev`
- `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every
  openrouter/* alias and every keyed opencode/* alias now smoked
This commit is contained in:
Colin McDonnell
2026-05-20 02:18:38 +00:00
committed by pullfrog[bot]
parent f3d18401ac
commit 69c7d4b8cd
5 changed files with 27 additions and 45 deletions
+16 -34
View File
@@ -3,9 +3,9 @@
* jobs. `agent` mirrors the harness the runtime would pick in production
* (anthropic/* → claude, everything else → opencode).
*
* MODE=aliases (default) — every alias minus pruned passthroughs. consumed by
* `models-live`, which runs the cheap top-level CLI smoke per alias
* (`action/test/model-smoke.ts`) to validate resolution + auth.
* MODE=aliases (default) — every alias. consumed by `models-live`, which runs
* the cheap top-level CLI smoke per alias (`action/test/model-smoke.ts`) to
* validate resolution + auth.
*
* MODE=flagships — one standard-tier model per provider. consumed by
* `providers-live`, which runs the full harness smoke
@@ -13,18 +13,19 @@
* (e.g. Gemini schema sanitizer, OpenAI tool-call format). flagship slugs
* live in `providers.ts` alongside their per-provider coverage globs.
*
* passthrough pruning (aliases mode): openrouter/* aliases and keyed opencode/*
* aliases are routing-layer wrappers around models we already smoke-test
* directly. running every passthrough burns CI minutes without catching
* anything new — slug-drift is covered by the `models-catalog` job. one canary
* per routing layer proves the routing surface (auth, tool-call translation)
* is alive; set INCLUDE_PASSTHROUGHS=1 to bypass for full validation.
* Every keyed alias is smoked — including `openrouter/*` and keyed `opencode/*`
* passthroughs. They look like routing-layer wrappers but each one is a
* distinct catalog entry on models.dev (under the `openrouter` / `opencode`
* provider sections) that can drift independently of the upstream provider
* mirror — testing the direct google entry tells you nothing about whether
* the openrouter mirror has the same model id. The only entries pruned are
* routing slugs (bedrock/byok) whose `resolve` is a sentinel that picks the
* actual model id from a per-run env var.
*
* usage:
* node action/test/list-aliases.ts
* MODE=flagships node action/test/list-aliases.ts
* MATRIX_FILTER=gemini node action/test/list-aliases.ts
* INCLUDE_PASSTHROUGHS=1 node action/test/list-aliases.ts
*
* NOTE: the per-PR-precision matrix lives in `matrix.ts`, which calls into
* this file. raw invocation here emits the unfiltered matrix.
@@ -32,20 +33,6 @@
import { modelAliases } from "../models.ts";
import { providers } from "./providers.ts";
const ROUTING_CANARIES = new Set(["openrouter/claude-sonnet", "opencode/claude-sonnet"]);
function isPrunablePassthrough(alias: (typeof modelAliases)[number]): boolean {
if (ROUTING_CANARIES.has(alias.slug)) return false;
if (alias.provider === "openrouter") return true;
// routing slugs (bedrock/byok) need a per-run env var to pick the actual
// model — there's no generic smoke test, so prune from both matrices.
if (alias.routing) return true;
// opencode FREE models (big-pickle, mimo-v2-pro-free, minimax-m2.5-free)
// are unique to opencode and used in prod — keep them. only prune the keyed
// mirrors.
return alias.provider === "opencode" && !alias.isFree;
}
export type MatrixEntry = {
slug: string;
agent: string;
@@ -63,16 +50,14 @@ function toMatrixEntry(alias: (typeof modelAliases)[number]): MatrixEntry {
const aliasBySlug = new Map(modelAliases.map((a) => [a.slug, a]));
export function buildAliasMatrix(opts: {
filter?: string;
includePassthroughs?: boolean;
}): MatrixEntry[] {
export function buildAliasMatrix(opts: { filter?: string }): MatrixEntry[] {
const filter = opts.filter ?? "";
const includePassthroughs = opts.includePassthroughs ?? false;
return modelAliases
.filter((alias) => {
if (filter && !alias.slug.toLowerCase().includes(filter)) return false;
if (!includePassthroughs && isPrunablePassthrough(alias)) return false;
// routing slugs (bedrock/byok) need a per-run env var to pick the actual
// model — there's no generic smoke test.
if (alias.routing) return false;
return true;
})
.map(toMatrixEntry);
@@ -97,10 +82,7 @@ export function buildFlagshipMatrix(opts: { filter?: string }): MatrixEntry[] {
if (import.meta.url === `file://${process.argv[1]}`) {
const mode = process.env.MODE === "flagships" ? "flagships" : "aliases";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const includePassthroughs = process.env.INCLUDE_PASSTHROUGHS === "1";
const matrix =
mode === "flagships"
? buildFlagshipMatrix({ filter })
: buildAliasMatrix({ filter, includePassthroughs });
mode === "flagships" ? buildFlagshipMatrix({ filter }) : buildAliasMatrix({ filter });
process.stdout.write(JSON.stringify(matrix));
}