models-bump: harden CI and bot prompt against catalog hallucinations
PR #790 (the first bot-authored models-bump PR) shipped a broken bump for openrouter/gemini-flash: the bot pattern-matched the parallel google bump and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the OpenRouter API but not on models.dev's openrouter section — the catalog OpenCode actually reads. The slug failed at runtime with `Model not found`. Two CI gaps let it through: 1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*` alias as a "passthrough", smoke-testing only one canary per routing layer. But those aren't passthroughs — each is a distinct models.dev catalog entry that can drift independently of the direct-provider mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25). Only `bedrock/byok` stays pruned (sentinel resolve). 2. `models-catalog` test (the integrity gate that asserts every resolve exists on models.dev) was main-only by design — to keep upstream catalog churn from blocking unrelated PRs. But it's exactly the test we want running on the bot's own catalog edits. Add `pullfrog/models-bump` head-ref to its trigger. Also tighten the bot prompt in models-bump.yml: new rule 0 requires every new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the alias's own `candidates[]` in models-bump-context.json — the deterministic preprocessor only emits candidates sourced from models.dev's mirror, so this gates against the cross-alias pattern-matching that broke PR #790. For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter API), which is necessary but not sufficient; the `models-catalog` job is the authoritative models.dev check. Verified locally: - baseline `pnpm -C action test:catalog` passes 133 tests - simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash` into action/models.ts) and the catalog test fails with the right assertion: `model "google/gemini-3.5-flash" not found under openrouter on models.dev` - `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every openrouter/* alias and every keyed opencode/* alias now smoked
This commit is contained in:
committed by
pullfrog[bot]
parent
f3d18401ac
commit
69c7d4b8cd
+2
-7
@@ -194,12 +194,8 @@ function buildAliasesMatrix(input: {
|
||||
changedFiles: string[];
|
||||
full: boolean;
|
||||
filter: string;
|
||||
includePassthroughs: boolean;
|
||||
}): SlugEntry[] {
|
||||
const all = buildAliasMatrix({
|
||||
filter: input.filter,
|
||||
includePassthroughs: input.includePassthroughs,
|
||||
});
|
||||
const all = buildAliasMatrix({ filter: input.filter });
|
||||
const coverageByProvider = new Map(providers.map((p) => [p.name, p.coverage]));
|
||||
return all.filter((entry) => {
|
||||
const provider = entry.slug.split("/")[0];
|
||||
@@ -214,14 +210,13 @@ function buildAliasesMatrix(input: {
|
||||
function main(): void {
|
||||
const full = process.env.FULL === "1";
|
||||
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
|
||||
const includePassthroughs = process.env.INCLUDE_PASSTHROUGHS === "1";
|
||||
const changedFiles = full ? [] : readChangedFiles();
|
||||
|
||||
const output: MatrixOutput = {
|
||||
agents: buildAgentsMatrix({ changedFiles, full }),
|
||||
agnostic: buildAgnosticMatrix({ changedFiles, full }),
|
||||
flagships: buildFlagshipsMatrix({ changedFiles, full, filter }),
|
||||
aliases: buildAliasesMatrix({ changedFiles, full, filter, includePassthroughs }),
|
||||
aliases: buildAliasesMatrix({ changedFiles, full, filter }),
|
||||
};
|
||||
|
||||
process.stdout.write(JSON.stringify(output));
|
||||
|
||||
Reference in New Issue
Block a user