ShockVPN/shockbot
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ci: split per-alias resolution smoke from per-provider harness smoke (#650)

Browse Source 
* ci: split per-alias resolution smoke from per-provider harness smoke

`models-live` previously ran the full Pullfrog harness (Docker + MCP +
agent + structured-output validation) once per alias on every PR that
touched `models.ts` or `agents/**`. That cost minutes and dollars per
alias and re-validated tool-calling for every routing wrapper.

The per-alias signal we actually need from `models.ts` changes is just
"does this alias resolve and authenticate." Tool-calling correctness is
a property of the underlying model, not the alias, and it doesn't change
when someone adds a row to the catalog. Splitting the two concerns:

- `models-live` now runs `action/test/model-smoke.ts` per alias — a
  top-level CLI invocation (`opencode run -m <resolve> "reply OK"` or
  `claude -p "reply OK" --model <bare>`) with no Docker, MCP, or
  Pullfrog harness. Validates resolution + auth in seconds at fractions
  of a cent. Lets us drop the `EXPENSIVE_RESOLVE_SUBSTRINGS` carve-out
  for `gpt-pro` since the cheap smoke covers it for free.

- `providers-live` (new) runs the full harness smoke once per provider
  against a hand-curated standard-tier model (`anthropic/claude-sonnet`,
  `openai/gpt`, `google/gemini-pro`, `xai/grok`,
  `deepseek/deepseek-pro`, `moonshotai/kimi-k2`,
  `opencode/big-pickle`, `openrouter/claude-sonnet`). Catches
  provider-class regressions like the Gemini schema sanitizer or
  OpenAI tool-call format drift. ~8 jobs, ~$0.40/push, ~4min critical
  path in parallel.

Net change per push that touches `models.ts`: ~$20 → ~$0.40.

`list-aliases.ts` now branches on `MODE` to emit either matrix; the
flagship list asserts each slug exists in `modelAliases` so renames
break CI loudly. Wiki updated to reflect the new two-tier coverage and
the operational rule for new Gemini aliases (cheap smoke covers
auth, manual harness run still needed for sanitizer compatibility on
non-flagship Gemini additions).

* fix(model-smoke): walk fallback chain; address pr review comments

- model-smoke now uses `resolveCliModel(slug)` instead of `alias.resolve`
  so deprecated aliases (those with `fallback` set, e.g.
  `opencode/mimo-v2-pro-free` → `opencode/big-pickle`) hit the
  replacement model the way production does. mimo-v2-pro-free was
  failing CI because the underlying opencode model is dead — the
  fallback chain is the whole point of marking it deprecated.

- tighten stale `agentForSlug()` reference in model-smoke.ts comment
  (function was deleted in this same PR; classification is now inline
  in `list-aliases.ts toMatrixEntry`).

- tighten `FLAGSHIPS` drift comment to call out that the assertion is
  one-way (catches slug-rename, but silently omits new providers).
  Update wiki step 4 of "To add a provider" to require adding the
  standard-tier slug to `FLAGSHIPS` for harness coverage.

* docs: scrub stale env-knob refs in models-catalog parity section

`wiki/models-catalog.md` cross-provider parity paragraph still pointed
at `INCLUDE_ALL_PASSTHROUGHS` / `INCLUDE_EXPENSIVE` and the implicit
filter→expensive-gate coupling — all removed in this PR. Aligned the
copy with Step 9 (which was already updated): `INCLUDE_PASSTHROUGHS`,
no expensive gate, `MATRIX_FILTER` applies to both aliases and
flagships modes.
This commit is contained in:
Colin McDonnell
2026-05-11 16:36:14 +00:00
committed by pullfrog[bot]
parent 10590993f4
commit 4cc6d95a91
2 changed files with 240 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/list-aliases.ts
+70 -42
View File
@@ -1,67 +1,95 @@
/**
* emits a JSON array of { slug, agent, name } entries for the `models-live`
* matrix job. `agent` is auto-derived from the alias provider and matches the
* harness the runtime would pick in production.
* emits a JSON array of { slug, agent, name } entries for one of two CI matrix
* jobs. `agent` mirrors the harness the runtime would pick in production
* (anthropic/* → claude-code, everything else → opencode).
*
* set MATRIX_FILTER to a substring to restrict the matrix to matching aliases
* — useful for iterating on a single provider without paying for every model.
* MODE=aliases (default) — every alias minus pruned passthroughs. consumed by
* `models-live`, which runs the cheap top-level CLI smoke per alias
* (`action/test/model-smoke.ts`) to validate resolution + auth.
*
* passthrough pruning: openrouter/* aliases and keyed opencode/* aliases are
* just routing-layer wrappers around models we already smoke-test directly
* (anthropic/*, openai/*, google/*, etc). running every passthrough burns CI
* minutes without catching anything the direct smoke doesn't. we keep one
* canary per routing layer to validate the routing layer itself is alive;
* slug-drift is caught separately by the `models-catalog` job. set
* INCLUDE_ALL_PASSTHROUGHS=1 to bypass this for full validation.
* MODE=flagships — one standard-tier model per provider. consumed by
* `providers-live`, which runs the full harness smoke
* (`pnpm runtest smoke <agent>`) to validate provider-class tool-calling
* (e.g. Gemini schema sanitizer, OpenAI tool-call format).
*
* passthrough pruning (aliases mode): openrouter/* aliases and keyed opencode/*
* aliases are routing-layer wrappers around models we already smoke-test
* directly. running every passthrough burns CI minutes without catching
* anything new — slug-drift is covered by the `models-catalog` job. one canary
* per routing layer proves the routing surface (auth, tool-call translation)
* is alive; set INCLUDE_PASSTHROUGHS=1 to bypass for full validation.
*
* usage:
* node action/test/list-aliases.ts
* MODE=flagships node action/test/list-aliases.ts
* MATRIX_FILTER=gemini node action/test/list-aliases.ts
* INCLUDE_ALL_PASSTHROUGHS=1 node action/test/list-aliases.ts
* INCLUDE_EXPENSIVE=1 node action/test/list-aliases.ts
* INCLUDE_PASSTHROUGHS=1 node action/test/list-aliases.ts
*/
import { modelAliases } from "../models.ts";
function agentForSlug(slug: string): "claude" | "opencode" {
return slug.startsWith("anthropic/") ? "claude" : "opencode";
}
// one canary per routing layer — proves the routing surface (auth, tool-call
// translation) is alive without re-testing every underlying model.
const ROUTING_CANARIES = new Set(["openrouter/claude-sonnet", "opencode/claude-sonnet"]);
// pruned by default; opt back in with INCLUDE_EXPENSIVE=1 or MATRIX_FILTER.
// matched against `alias.resolve` so every routing layer (openai/, opencode/,
// openrouter/) is covered without enumerating each slug separately.
// gpt-5.5-pro burns ~$2.40/run on this fixture — too expensive per-push.
const EXPENSIVE_RESOLVE_SUBSTRINGS = ["gpt-5.5-pro"];
function isExpensive(alias: (typeof modelAliases)[number]): boolean {
return EXPENSIVE_RESOLVE_SUBSTRINGS.some((s) => alias.resolve.includes(s));
}
// hand-picked "standard good model" per provider — not the pro/opus tier (too
// expensive for per-push) and not the free/experimental tier (too flaky). these
// aliases anchor the harness smoke job that catches provider-class regressions
// like Gemini schema sanitization or OpenAI tool-call format drift. the
// assertion below catches slug-drift loudly, but adding a NEW provider without
// an entry here silently omits it from `providers-live` — see
// wiki/models-catalog.md "To add a provider".
const FLAGSHIPS = [
"anthropic/claude-sonnet",
"openai/gpt",
"google/gemini-pro",
"xai/grok",
"deepseek/deepseek-pro",
"moonshotai/kimi-k2",
"opencode/big-pickle",
"openrouter/claude-sonnet",
];
function isPrunablePassthrough(alias: (typeof modelAliases)[number]): boolean {
if (ROUTING_CANARIES.has(alias.slug)) return false;
if (alias.provider === "openrouter") return true;
// opencode FREE models (big-pickle, mimo, minimax, gpt-5-nano) are unique
// to opencode and used in prod — keep them. only prune the keyed mirrors.
if (alias.provider === "opencode" && !alias.isFree) return true;
return false;
return alias.provider === "opencode" && !alias.isFree;
}
const filter = process.env.MATRIX_FILTER?.trim() ?? "";
const includeAllPassthroughs = process.env.INCLUDE_ALL_PASSTHROUGHS === "1";
const includeExpensive = process.env.INCLUDE_EXPENSIVE === "1" || filter !== "";
const matrix = modelAliases
.filter((alias) => (filter ? alias.slug.toLowerCase().includes(filter.toLowerCase()) : true))
.filter((alias) => includeAllPassthroughs || !isPrunablePassthrough(alias))
.filter((alias) => includeExpensive || !isExpensive(alias))
.map((alias) => ({
function toMatrixEntry(alias: (typeof modelAliases)[number]) {
return {
slug: alias.slug,
agent: agentForSlug(alias.slug),
agent: alias.slug.startsWith("anthropic/") ? "claude" : "opencode",
// readable display name (GHA renders slashes awkwardly in matrix job titles)
name: alias.slug.replace("/", "-"),
}));
};
}
const mode = process.env.MODE === "flagships" ? "flagships" : "aliases";
const filter = process.env.MATRIX_FILTER?.trim().toLowerCase() ?? "";
const includePassthroughs = process.env.INCLUDE_PASSTHROUGHS === "1";
const aliasBySlug = new Map(modelAliases.map((a) => [a.slug, a]));
const matrix = (() => {
if (mode === "flagships") {
return FLAGSHIPS.map((slug) => {
const alias = aliasBySlug.get(slug);
if (!alias) {
throw new Error(
`list-aliases: flagship "${slug}" missing from modelAliases — update FLAGSHIPS`
);
}
return alias;
})
.filter((alias) => !filter || alias.slug.toLowerCase().includes(filter))
.map(toMatrixEntry);
}
return modelAliases
.filter((alias) => {
if (filter && !alias.slug.toLowerCase().includes(filter)) return false;
if (!includePassthroughs && isPrunablePassthrough(alias)) return false;
return true;
})
.map(toMatrixEntry);
})();
process.stdout.write(JSON.stringify(matrix));