PR #790 (the first bot-authored models-bump PR) shipped a broken bump for
openrouter/gemini-flash: the bot pattern-matched the parallel google bump
and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the
OpenRouter API but not on models.dev's openrouter section — the catalog
OpenCode actually reads. The slug failed at runtime with `Model not found`.
Two CI gaps let it through:
1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*`
alias as a "passthrough", smoke-testing only one canary per routing
layer. But those aren't passthroughs — each is a distinct models.dev
catalog entry that can drift independently of the direct-provider
mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25).
Only `bedrock/byok` stays pruned (sentinel resolve).
2. `models-catalog` test (the integrity gate that asserts every resolve
exists on models.dev) was main-only by design — to keep upstream catalog
churn from blocking unrelated PRs. But it's exactly the test we want
running on the bot's own catalog edits. Add `pullfrog/models-bump`
head-ref to its trigger.
Also tighten the bot prompt in models-bump.yml: new rule 0 requires every
new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the
alias's own `candidates[]` in models-bump-context.json — the deterministic
preprocessor only emits candidates sourced from models.dev's mirror, so
this gates against the cross-alias pattern-matching that broke PR #790.
For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter
API), which is necessary but not sufficient; the `models-catalog` job is
the authoritative models.dev check.
Verified locally:
- baseline `pnpm -C action test:catalog` passes 133 tests
- simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash`
into action/models.ts) and the catalog test fails with the right
assertion: `model "google/gemini-3.5-flash" not found under openrouter
on models.dev`
- `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every
openrouter/* alias and every keyed opencode/* alias now smoked
* ci: filter test matrices by per-test coverage globs to cut LLM spend
every test in `crossagent/`, `agnostic/`, and every provider entry now
declares a `coverage: string[]` of repo-relative globs. the new `changes`
job runs `paths-filter` for a docs-only short-circuit, then pipes the
changed-file list into `action/test/matrix.ts`, which intersects each
entry's coverage against the diff and emits filtered `agents`,
`agnostic`, `flagships`, and `aliases` matrices. main pushes and
`workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety
net.
retires `changed-agents.sh` and the `MODE=flagships` branch in
`list-aliases.ts` in favor of one consistent model.
* ci(matrix): switch test discovery to dep-free static parsing
the GHA `changes` job has no `node_modules` installed. the previous
dynamic-import path pulled the test files transitively through
`utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with
ERR_MODULE_NOT_FOUND. parse the test files via regex instead so
matrix.ts stays zero-dep — the chain (matrix -> coverage / providers /
list-aliases / models) imports only node builtins and relative TS files.
* ci(matrix): address PR #730 review feedback
- drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`,
`restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but
the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`,
both already covered).
- drop unused `coverageForProvider` export and its `byName` map from
`providers.ts` (matrix.ts builds its own lookup inline).
- derive the active agent list from `agents/index.ts` via the same
dep-free regex tactic as `parseTestFile` instead of hardcoding
`["claude", "opencode"]` — adding a new harness file now wires it
into the dynamic matrix automatically.
- treat `coverage: []` as `coverage: undefined` in `shouldRun` so an
accidentally-empty array doesn't silently skip CI on every PR.
- add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the
`timeout` test's coverage — the activity-timeout enforcement path
was the original reason the test exists.
- ungate the `root` job (lint/format/typecheck/vitest). it's a required
status check on `main`, so gating it on `code == 'true'` would make
docs-only PRs unmergeable (skipped jobs don't satisfy required-check
rules). the real LLM savings come from skipping the four matrices,
not from skipping `root`.
- harden the four matrix-job `if:` guards from `outputs.matrix && ...`
to `outputs.matrix != '' && ...` — explicit > implicit short-circuit.
- document `expandBraces`'s flat-only support so a future author isn't
surprised by `{a,{b,c}}` not expanding.
- fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".
Colin McDonnell