move models.dev drift tests to main-only; add per-alias live smoke matrix

PR CI kept breaking on upstream catalog drift (new model ships on models.dev,
OpenRouter renames an id, etc.) — failures unrelated to the PR's contents.
split the model-alias test suite so PRs only see pure-logic checks, and push
the external-state drift + end-to-end coverage to main.

test organization:
- action/test/models.test.ts keeps pure invariants: openRouterResolve
  completeness and fallback-chain resolution. runs on every PR.
- action/test/models-catalog.main.test.ts gets the 4 network-dependent
  describes (models.dev validity x2, OpenRouter API validity, latest-model
  snapshot). runs only on main push via a dedicated vitest config
  (vitest.main.config.ts + `pnpm test:catalog`).

new CI jobs in .github/workflows/test.yml:
- models-catalog: `pnpm test:catalog` on every main push. detects upstream
  catalog drift so we can react at the next convenient window.
- models-live: 38-entry matrix that invokes the agent harness end-to-end
  against the real provider for each alias in models.ts. generated from
  action/test/list-aliases.ts. runs only on main push AND only when
  resolution-affecting files changed (action/models.ts, action/package.json,
  action/agents/**) — the exact shape of the opus 4.7 incident.

test/run.ts: PULLFROG_MODEL now flows through from process.env so the live
matrix can pin an alias per job without the per-agent default clobbering it.

Made-with: Cursor
This commit is contained in:
Colin McDonnell
2026-04-16 21:10:15 +00:00
committed by pullfrog[bot]
parent 74b313e612
commit 5e6ff67623
10 changed files with 225 additions and 159 deletions
+2 -2
View File
@@ -71,8 +71,8 @@ function stripProviderPrefix(specifier: string): string {
return slashIndex > 0 ? specifier.slice(slashIndex + 1) : specifier;
}
// `max` effort is Opus 4.6 only — errors on other models.
// use `max` when the resolved model is Opus, `high` otherwise.
// `max` effort is supported on Opus 4.6 / 4.7; other models fall back to `high`.
// claude-code deny-lists older opus/sonnet generations from `max` at invocation time.
function resolveEffort(model: string | undefined): "max" | "high" {
if (model?.includes("opus")) return "max";
return "high";