* ci: split per-alias resolution smoke from per-provider harness smoke
`models-live` previously ran the full Pullfrog harness (Docker + MCP +
agent + structured-output validation) once per alias on every PR that
touched `models.ts` or `agents/**`. That cost minutes and dollars per
alias and re-validated tool-calling for every routing wrapper.
The per-alias signal we actually need from `models.ts` changes is just
"does this alias resolve and authenticate." Tool-calling correctness is
a property of the underlying model, not the alias, and it doesn't change
when someone adds a row to the catalog. Splitting the two concerns:
- `models-live` now runs `action/test/model-smoke.ts` per alias — a
top-level CLI invocation (`opencode run -m <resolve> "reply OK"` or
`claude -p "reply OK" --model <bare>`) with no Docker, MCP, or
Pullfrog harness. Validates resolution + auth in seconds at fractions
of a cent. Lets us drop the `EXPENSIVE_RESOLVE_SUBSTRINGS` carve-out
for `gpt-pro` since the cheap smoke covers it for free.
- `providers-live` (new) runs the full harness smoke once per provider
against a hand-curated standard-tier model (`anthropic/claude-sonnet`,
`openai/gpt`, `google/gemini-pro`, `xai/grok`,
`deepseek/deepseek-pro`, `moonshotai/kimi-k2`,
`opencode/big-pickle`, `openrouter/claude-sonnet`). Catches
provider-class regressions like the Gemini schema sanitizer or
OpenAI tool-call format drift. ~8 jobs, ~$0.40/push, ~4min critical
path in parallel.
Net change per push that touches `models.ts`: ~$20 → ~$0.40.
`list-aliases.ts` now branches on `MODE` to emit either matrix; the
flagship list asserts each slug exists in `modelAliases` so renames
break CI loudly. Wiki updated to reflect the new two-tier coverage and
the operational rule for new Gemini aliases (cheap smoke covers
auth, manual harness run still needed for sanitizer compatibility on
non-flagship Gemini additions).
* fix(model-smoke): walk fallback chain; address pr review comments
- model-smoke now uses `resolveCliModel(slug)` instead of `alias.resolve`
so deprecated aliases (those with `fallback` set, e.g.
`opencode/mimo-v2-pro-free` → `opencode/big-pickle`) hit the
replacement model the way production does. mimo-v2-pro-free was
failing CI because the underlying opencode model is dead — the
fallback chain is the whole point of marking it deprecated.
- tighten stale `agentForSlug()` reference in model-smoke.ts comment
(function was deleted in this same PR; classification is now inline
in `list-aliases.ts toMatrixEntry`).
- tighten `FLAGSHIPS` drift comment to call out that the assertion is
one-way (catches slug-rename, but silently omits new providers).
Update wiki step 4 of "To add a provider" to require adding the
standard-tier slug to `FLAGSHIPS` for harness coverage.
* docs: scrub stale env-knob refs in models-catalog parity section
`wiki/models-catalog.md` cross-provider parity paragraph still pointed
at `INCLUDE_ALL_PASSTHROUGHS` / `INCLUDE_EXPENSIVE` and the implicit
filter→expensive-gate coupling — all removed in this PR. Aligned the
copy with Step 9 (which was already updated): `INCLUDE_PASSTHROUGHS`,
no expensive gate, `MATRIX_FILTER` applies to both aliases and
flagships modes.
Colin McDonnell
4cc6d95a91