move models.dev drift tests to main-only; add per-alias live smoke matrix

PR CI kept breaking on upstream catalog drift (new model ships on models.dev,
OpenRouter renames an id, etc.) — failures unrelated to the PR's contents.
split the model-alias test suite so PRs only see pure-logic checks, and push
the external-state drift + end-to-end coverage to main.

test organization:
- action/test/models.test.ts keeps pure invariants: openRouterResolve
  completeness and fallback-chain resolution. runs on every PR.
- action/test/models-catalog.main.test.ts gets the 4 network-dependent
  describes (models.dev validity x2, OpenRouter API validity, latest-model
  snapshot). runs only on main push via a dedicated vitest config
  (vitest.main.config.ts + `pnpm test:catalog`).

new CI jobs in .github/workflows/test.yml:
- models-catalog: `pnpm test:catalog` on every main push. detects upstream
  catalog drift so we can react at the next convenient window.
- models-live: 38-entry matrix that invokes the agent harness end-to-end
  against the real provider for each alias in models.ts. generated from
  action/test/list-aliases.ts. runs only on main push AND only when
  resolution-affecting files changed (action/models.ts, action/package.json,
  action/agents/**) — the exact shape of the opus 4.7 incident.

test/run.ts: PULLFROG_MODEL now flows through from process.env so the live
matrix can pin an alias per job without the per-agent default clobbering it.

Made-with: Cursor
This commit is contained in:
Colin McDonnell
2026-04-16 21:10:15 +00:00
committed by pullfrog[bot]
parent 74b313e612
commit 5e6ff67623
10 changed files with 225 additions and 159 deletions
+5 -5
View File
@@ -12,8 +12,8 @@ importers:
specifier: ^1.11.1
version: 1.11.1
'@anthropic-ai/claude-code':
specifier: 2.1.85
version: 2.1.85
specifier: 2.1.112
version: 2.1.112
'@ark/fs':
specifier: 0.56.0
version: 0.56.0
@@ -128,8 +128,8 @@ packages:
'@actions/io@1.1.3':
resolution: {integrity: sha512-wi9JjgKLYS7U/z8PPbco+PvTb/nRWjeoFlJ1Qer83k/3C5PHQi28hiVdeE2kHXmIL99mQFawx8qt/JPjZilJ8Q==}
'@anthropic-ai/claude-code@2.1.85':
resolution: {integrity: sha512-3/q3xTpk9EnBfQ/XsHGkOZniOgQx4sqD95CDKw1mvN1Qw5+9IZTp6ILdds02d7vOM6YuLL0G0zhqsMSAFVse4w==}
'@anthropic-ai/claude-code@2.1.112':
resolution: {integrity: sha512-9FUgJ0EOvILyhIqxFKNVliebiUjL68dwpEW3eGSSe0vkVDJ1c5qMDNWc22gW3zkD7zRAqtfQPSGv0t4vMM2DPA==}
engines: {node: '>=18.0.0'}
hasBin: true
@@ -1960,7 +1960,7 @@ snapshots:
'@actions/io@1.1.3': {}
'@anthropic-ai/claude-code@2.1.85':
'@anthropic-ai/claude-code@2.1.112':
optionalDependencies:
'@img/sharp-darwin-arm64': 0.34.5
'@img/sharp-darwin-x64': 0.34.5