move models.dev drift tests to main-only; add per-alias live smoke matrix
PR CI kept breaking on upstream catalog drift (new model ships on models.dev, OpenRouter renames an id, etc.) — failures unrelated to the PR's contents. split the model-alias test suite so PRs only see pure-logic checks, and push the external-state drift + end-to-end coverage to main. test organization: - action/test/models.test.ts keeps pure invariants: openRouterResolve completeness and fallback-chain resolution. runs on every PR. - action/test/models-catalog.main.test.ts gets the 4 network-dependent describes (models.dev validity x2, OpenRouter API validity, latest-model snapshot). runs only on main push via a dedicated vitest config (vitest.main.config.ts + `pnpm test:catalog`). new CI jobs in .github/workflows/test.yml: - models-catalog: `pnpm test:catalog` on every main push. detects upstream catalog drift so we can react at the next convenient window. - models-live: 38-entry matrix that invokes the agent harness end-to-end against the real provider for each alias in models.ts. generated from action/test/list-aliases.ts. runs only on main push AND only when resolution-affecting files changed (action/models.ts, action/package.json, action/agents/**) — the exact shape of the opus 4.7 incident. test/run.ts: PULLFROG_MODEL now flows through from process.env so the live matrix can pin an alias per job without the per-agent default clobbering it. Made-with: Cursor
This commit is contained in:
committed by
pullfrog[bot]
parent
74b313e612
commit
5e6ff67623
Generated
+5
-5
@@ -12,8 +12,8 @@ importers:
|
||||
specifier: ^1.11.1
|
||||
version: 1.11.1
|
||||
'@anthropic-ai/claude-code':
|
||||
specifier: 2.1.85
|
||||
version: 2.1.85
|
||||
specifier: 2.1.112
|
||||
version: 2.1.112
|
||||
'@ark/fs':
|
||||
specifier: 0.56.0
|
||||
version: 0.56.0
|
||||
@@ -128,8 +128,8 @@ packages:
|
||||
'@actions/io@1.1.3':
|
||||
resolution: {integrity: sha512-wi9JjgKLYS7U/z8PPbco+PvTb/nRWjeoFlJ1Qer83k/3C5PHQi28hiVdeE2kHXmIL99mQFawx8qt/JPjZilJ8Q==}
|
||||
|
||||
'@anthropic-ai/claude-code@2.1.85':
|
||||
resolution: {integrity: sha512-3/q3xTpk9EnBfQ/XsHGkOZniOgQx4sqD95CDKw1mvN1Qw5+9IZTp6ILdds02d7vOM6YuLL0G0zhqsMSAFVse4w==}
|
||||
'@anthropic-ai/claude-code@2.1.112':
|
||||
resolution: {integrity: sha512-9FUgJ0EOvILyhIqxFKNVliebiUjL68dwpEW3eGSSe0vkVDJ1c5qMDNWc22gW3zkD7zRAqtfQPSGv0t4vMM2DPA==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
hasBin: true
|
||||
|
||||
@@ -1960,7 +1960,7 @@ snapshots:
|
||||
|
||||
'@actions/io@1.1.3': {}
|
||||
|
||||
'@anthropic-ai/claude-code@2.1.85':
|
||||
'@anthropic-ai/claude-code@2.1.112':
|
||||
optionalDependencies:
|
||||
'@img/sharp-darwin-arm64': 0.34.5
|
||||
'@img/sharp-darwin-x64': 0.34.5
|
||||
|
||||
Reference in New Issue
Block a user