Colin McDonnell
5e6ff67623
PR CI kept breaking on upstream catalog drift (new model ships on models.dev, OpenRouter renames an id, etc.) — failures unrelated to the PR's contents. split the model-alias test suite so PRs only see pure-logic checks, and push the external-state drift + end-to-end coverage to main. test organization: - action/test/models.test.ts keeps pure invariants: openRouterResolve completeness and fallback-chain resolution. runs on every PR. - action/test/models-catalog.main.test.ts gets the 4 network-dependent describes (models.dev validity x2, OpenRouter API validity, latest-model snapshot). runs only on main push via a dedicated vitest config (vitest.main.config.ts + `pnpm test:catalog`). new CI jobs in .github/workflows/test.yml: - models-catalog: `pnpm test:catalog` on every main push. detects upstream catalog drift so we can react at the next convenient window. - models-live: 38-entry matrix that invokes the agent harness end-to-end against the real provider for each alias in models.ts. generated from action/test/list-aliases.ts. runs only on main push AND only when resolution-affecting files changed (action/models.ts, action/package.json, action/agents/**) — the exact shape of the opus 4.7 incident. test/run.ts: PULLFROG_MODEL now flows through from process.env so the live matrix can pin an alias per job without the per-agent default clobbering it. Made-with: Cursor
39 lines
851 B
Plaintext
39 lines
851 B
Plaintext
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
|
|
|
|
exports[`latest model per provider snapshot > matches snapshot 1`] = `
|
|
{
|
|
"anthropic": {
|
|
"modelId": "claude-opus-4-7",
|
|
"releaseDate": "2026-04-16",
|
|
},
|
|
"deepseek": {
|
|
"modelId": "deepseek-reasoner",
|
|
"releaseDate": "2025-12-01",
|
|
},
|
|
"google": {
|
|
"modelId": "gemma-4-31b-it",
|
|
"releaseDate": "2026-04-02",
|
|
},
|
|
"moonshotai": {
|
|
"modelId": "kimi-k2.5",
|
|
"releaseDate": "2026-01",
|
|
},
|
|
"openai": {
|
|
"modelId": "gpt-5.4-nano",
|
|
"releaseDate": "2026-03-17",
|
|
},
|
|
"opencode": {
|
|
"modelId": "claude-opus-4-7",
|
|
"releaseDate": "2026-04-16",
|
|
},
|
|
"openrouter": {
|
|
"modelId": "openrouter/elephant-alpha",
|
|
"releaseDate": "2026-04-13",
|
|
},
|
|
"xai": {
|
|
"modelId": "grok-4.20-multi-agent-0309",
|
|
"releaseDate": "2026-03-09",
|
|
},
|
|
}
|
|
`;
|