ShockVPN/shockbot
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

move models.dev drift tests to main-only; add per-alias live smoke matrix

Browse Source 
PR CI kept breaking on upstream catalog drift (new model ships on models.dev,
OpenRouter renames an id, etc.) — failures unrelated to the PR's contents.
split the model-alias test suite so PRs only see pure-logic checks, and push
the external-state drift + end-to-end coverage to main.

test organization:
- action/test/models.test.ts keeps pure invariants: openRouterResolve
  completeness and fallback-chain resolution. runs on every PR.
- action/test/models-catalog.main.test.ts gets the 4 network-dependent
  describes (models.dev validity x2, OpenRouter API validity, latest-model
  snapshot). runs only on main push via a dedicated vitest config
  (vitest.main.config.ts + `pnpm test:catalog`).

new CI jobs in .github/workflows/test.yml:
- models-catalog: `pnpm test:catalog` on every main push. detects upstream
  catalog drift so we can react at the next convenient window.
- models-live: 38-entry matrix that invokes the agent harness end-to-end
  against the real provider for each alias in models.ts. generated from
  action/test/list-aliases.ts. runs only on main push AND only when
  resolution-affecting files changed (action/models.ts, action/package.json,
  action/agents/**) — the exact shape of the opus 4.7 incident.

test/run.ts: PULLFROG_MODEL now flows through from process.env so the live
matrix can pin an alias per job without the per-agent default clobbering it.

Made-with: Cursor
This commit is contained in:
Colin McDonnell
2026-04-16 21:10:15 +00:00
committed by pullfrog[bot]
parent 74b313e612
commit 5e6ff67623
10 changed files with 225 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/run.ts
+14 -8
View File
@@ -299,15 +299,21 @@ async function runTestForAgent(ctx: RunContext): Promise<ValidationResult> {
env.PULLFROG_AGENT = ctx.agent;
// override DB model to avoid mismatch when PULLFROG_AGENT forces a specific agent
// (DB model may belong to a different provider than the forced agent supports)
// (DB model may belong to a different provider than the forced agent supports).
// precedence: testConfig.env > process.env.PULLFROG_MODEL > per-agent default.
// the process.env pass-through lets CI (models-live matrix) pin an alias per job.
if (!Object.hasOwn(env, "PULLFROG_MODEL")) {
const defaultModels: Record<string, string> = {
claude: "anthropic/claude-sonnet-4-6",
opencode: "anthropic/claude-sonnet-4-6",
};
const model = defaultModels[ctx.agent];
if (model) {
env.PULLFROG_MODEL = model;
if (process.env.PULLFROG_MODEL) {
env.PULLFROG_MODEL = process.env.PULLFROG_MODEL;
} else {
const defaultModels: Record<string, string> = {
claude: "anthropic/claude-sonnet-4-6",
opencode: "anthropic/claude-sonnet-4-6",
};
const model = defaultModels[ctx.agent];
if (model) {
env.PULLFROG_MODEL = model;
}
}
}