Colin McDonnell 4cc6d95a91 ci: split per-alias resolution smoke from per-provider harness smoke (#650)
* ci: split per-alias resolution smoke from per-provider harness smoke

`models-live` previously ran the full Pullfrog harness (Docker + MCP +
agent + structured-output validation) once per alias on every PR that
touched `models.ts` or `agents/**`. That cost minutes and dollars per
alias and re-validated tool-calling for every routing wrapper.

The per-alias signal we actually need from `models.ts` changes is just
"does this alias resolve and authenticate." Tool-calling correctness is
a property of the underlying model, not the alias, and it doesn't change
when someone adds a row to the catalog. Splitting the two concerns:

- `models-live` now runs `action/test/model-smoke.ts` per alias — a
  top-level CLI invocation (`opencode run -m <resolve> "reply OK"` or
  `claude -p "reply OK" --model <bare>`) with no Docker, MCP, or
  Pullfrog harness. Validates resolution + auth in seconds at fractions
  of a cent. Lets us drop the `EXPENSIVE_RESOLVE_SUBSTRINGS` carve-out
  for `gpt-pro` since the cheap smoke covers it for free.

- `providers-live` (new) runs the full harness smoke once per provider
  against a hand-curated standard-tier model (`anthropic/claude-sonnet`,
  `openai/gpt`, `google/gemini-pro`, `xai/grok`,
  `deepseek/deepseek-pro`, `moonshotai/kimi-k2`,
  `opencode/big-pickle`, `openrouter/claude-sonnet`). Catches
  provider-class regressions like the Gemini schema sanitizer or
  OpenAI tool-call format drift. ~8 jobs, ~$0.40/push, ~4min critical
  path in parallel.

Net change per push that touches `models.ts`: ~$20 → ~$0.40.

`list-aliases.ts` now branches on `MODE` to emit either matrix; the
flagship list asserts each slug exists in `modelAliases` so renames
break CI loudly. Wiki updated to reflect the new two-tier coverage and
the operational rule for new Gemini aliases (cheap smoke covers
auth, manual harness run still needed for sanitizer compatibility on
non-flagship Gemini additions).

* fix(model-smoke): walk fallback chain; address pr review comments

- model-smoke now uses `resolveCliModel(slug)` instead of `alias.resolve`
  so deprecated aliases (those with `fallback` set, e.g.
  `opencode/mimo-v2-pro-free` → `opencode/big-pickle`) hit the
  replacement model the way production does. mimo-v2-pro-free was
  failing CI because the underlying opencode model is dead — the
  fallback chain is the whole point of marking it deprecated.

- tighten stale `agentForSlug()` reference in model-smoke.ts comment
  (function was deleted in this same PR; classification is now inline
  in `list-aliases.ts toMatrixEntry`).

- tighten `FLAGSHIPS` drift comment to call out that the assertion is
  one-way (catches slug-rename, but silently omits new providers).
  Update wiki step 4 of "To add a provider" to require adding the
  standard-tier slug to `FLAGSHIPS` for harness coverage.

* docs: scrub stale env-knob refs in models-catalog parity section

`wiki/models-catalog.md` cross-provider parity paragraph still pointed
at `INCLUDE_ALL_PASSTHROUGHS` / `INCLUDE_EXPENSIVE` and the implicit
filter→expensive-gate coupling — all removed in this PR. Aligned the
copy with Step 9 (which was already updated): `INCLUDE_PASSTHROUGHS`,
no expensive gate, `MATRIX_FILTER` applies to both aliases and
flagships modes.
2026-05-11 16:36:14 +00:00
2026-01-16 08:00:16 +00:00
2026-03-12 05:22:51 +00:00
2025-08-27 16:53:48 -07:00
2026-01-19 08:41:56 +00:00
2026-05-08 23:37:52 +00:00
2026-03-12 05:22:51 +00:00

Green Pullfrog logo
Pullfrog

Bring your favorite coding agent into GitHub


🚀 Pullfrog is in beta! We're onboarding users in waves. Get on the waitlist →


What is Pullfrog?

Pullfrog is a GitHub bot that brings the full power of your favorite coding agents into GitHub. It's open source and powered by GitHub Actions.

  • Tag @pullfrog — Tag @pullfrog in a comment anywhere in your repo. It will pull in any relevant context using the action's internal MCP server and perform the appropriate task.
  • Prompt from the web — Trigger arbitrary tasks from the Pullfrog dashboard
  • Automated triggers — Configure Pullfrog to trigger agent runs in response to specific events. Each of these triggers can be associated with custom prompt instructions.
    • issue created
    • issue labeled
    • PR created
    • PR review created
    • PR review requested
    • and more...

Pullfrog is the bridge between your preferred coding agents and GitHub. Use it for:

  • 🤖 Coding tasks — Tell @pullfrog to implement something and it'll spin up a PR. If CI fails, it'll read the logs and attempt a fix automatically. It'll automatically address any PR reviews too.
  • 🔍 PR review — Coding agents are great at reviewing PRs. Using the "PR created" trigger, you can configure Pullfrog to auto-review new PRs.
  • 🤙 Issue management — Via the "issue created" trigger, Pullfrog can automatically respond to common questions, create implementation plans, and link to related issues/PRs. Or (if you're feeling lucky) you can prompt it to immediately attempt a PR addressing new issues.
  • Literally whatever — Want to have the agent automatically add docs to all new PRs? Cut a new release with agent-written notes on every commit to main? Pullfrog lets you do it.

Standalone Usage

You can also use pullfrog/pullfrog as a step in your own workflows. The action exposes a result output that can be consumed by subsequent steps.

Example: Auto-generate release notes on new tags

name: Release
on:
  push:
    tags: ['v*']

permissions:
  contents: write

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate release notes
        id: notes
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Generate release notes for ${{ github.ref_name }}.
            Compare commits between this tag and the previous tag.
            Format as markdown: summary paragraph, then ### Features, ### Fixes, ### Breaking Changes sections.
            Omit empty sections. Be concise.
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      # write to file to avoid shell escaping issues with special characters
      - name: Create GitHub release
        run: |
          notesfile="$RUNNER_TEMP/release-notes-$GITHUB_RUN_ID.md"
          printf '%s' "$NOTES" > "$notesfile"
          gh release create ${{ github.ref_name }} --title "${{ github.ref_name }}" --notes-file "$notesfile"
        env:
          GH_TOKEN: ${{ github.token }}
          NOTES: ${{ steps.notes.outputs.result }}

Example: Structured Output with Zod Schema

You can force the agent to return structured JSON output by providing a JSON schema. This allows you to reliably parse and use the agent's response in subsequent workflow steps.

You can define your JSON schema directly or uou can use any validation library that converts to JSON Schema. Here's an example using Zod:

name: Release Check
on:
  pull_request:
    types: [closed]

jobs:
  check-release:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm install --no-save --no-package-lock zod @actions/core

      - name: Generate Schema
        id: schema
        run: |
          node -e '
            import { z } from "zod";
            import { setOutput } from "@actions/core";
            const schema = z.object({
              version: z.string().describe("Semantic version number (e.g. 1.0.0)"),
              isBreaking: z.boolean().describe("Whether this release contains breaking changes"),
              changelog: z.array(z.string()).describe("List of changes in this release"),
            });
            setOutput("schema", JSON.stringify(z.toJSONSchema(schema)));
          '

      - name: Analyze PR
        id: analysis
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Analyze this PR and determine semantic versioning impact.
            Return a JSON object matching the provided schema.
          output_schema: ${{ steps.schema.outputs.schema }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Process Result
        run: |
          # Parse the JSON result using fromJSON()
          echo "Version: ${{ fromJSON(steps.analysis.outputs.result).version }}"
          echo "Breaking: ${{ fromJSON(steps.analysis.outputs.result).isBreaking }}"
S
Description
Self-hosted Ollama-powered code review bot for Gitea Actions based on pullfrog
Readme MIT 27 MiB
Languages
TypeScript 97.9%
Shell 0.7%
Dockerfile 0.7%
JavaScript 0.7%