move models.dev drift tests to main-only; add per-alias live smoke matrix
PR CI kept breaking on upstream catalog drift (new model ships on models.dev, OpenRouter renames an id, etc.) — failures unrelated to the PR's contents. split the model-alias test suite so PRs only see pure-logic checks, and push the external-state drift + end-to-end coverage to main. test organization: - action/test/models.test.ts keeps pure invariants: openRouterResolve completeness and fallback-chain resolution. runs on every PR. - action/test/models-catalog.main.test.ts gets the 4 network-dependent describes (models.dev validity x2, OpenRouter API validity, latest-model snapshot). runs only on main push via a dedicated vitest config (vitest.main.config.ts + `pnpm test:catalog`). new CI jobs in .github/workflows/test.yml: - models-catalog: `pnpm test:catalog` on every main push. detects upstream catalog drift so we can react at the next convenient window. - models-live: 38-entry matrix that invokes the agent harness end-to-end against the real provider for each alias in models.ts. generated from action/test/list-aliases.ts. runs only on main push AND only when resolution-affecting files changed (action/models.ts, action/package.json, action/agents/**) — the exact shape of the opus 4.7 incident. test/run.ts: PULLFROG_MODEL now flows through from process.env so the live matrix can pin an alias per job without the per-agent default clobbering it. Made-with: Cursor
This commit is contained in:
committed by
pullfrog[bot]
parent
74b313e612
commit
5e6ff67623
+2
-1
@@ -12,6 +12,7 @@
|
||||
],
|
||||
"scripts": {
|
||||
"test": "vitest",
|
||||
"test:catalog": "vitest run --config vitest.main.config.ts",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"build": "node esbuild.config.js && tsc -p tsconfig.exports.json",
|
||||
"check:entrypoints": "node scripts/check-entrypoint-imports.ts",
|
||||
@@ -24,7 +25,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@actions/core": "^1.11.1",
|
||||
"@anthropic-ai/claude-code": "2.1.85",
|
||||
"@anthropic-ai/claude-code": "2.1.112",
|
||||
"@ark/fs": "0.56.0",
|
||||
"@ark/util": "0.56.0",
|
||||
"@clack/prompts": "^1.2.0",
|
||||
|
||||
Reference in New Issue
Block a user