Colin McDonnell 1f4c3031be ci: filter test matrices by per-test coverage globs (#730)
* ci: filter test matrices by per-test coverage globs to cut LLM spend

every test in `crossagent/`, `agnostic/`, and every provider entry now
declares a `coverage: string[]` of repo-relative globs. the new `changes`
job runs `paths-filter` for a docs-only short-circuit, then pipes the
changed-file list into `action/test/matrix.ts`, which intersects each
entry's coverage against the diff and emits filtered `agents`,
`agnostic`, `flagships`, and `aliases` matrices. main pushes and
`workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety
net.

retires `changed-agents.sh` and the `MODE=flagships` branch in
`list-aliases.ts` in favor of one consistent model.

* ci(matrix): switch test discovery to dep-free static parsing

the GHA `changes` job has no `node_modules` installed. the previous
dynamic-import path pulled the test files transitively through
`utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with
ERR_MODULE_NOT_FOUND. parse the test files via regex instead so
matrix.ts stays zero-dep — the chain (matrix -> coverage / providers /
list-aliases / models) imports only node builtins and relative TS files.

* ci(matrix): address PR #730 review feedback

- drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`,
  `restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but
  the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`,
  both already covered).
- drop unused `coverageForProvider` export and its `byName` map from
  `providers.ts` (matrix.ts builds its own lookup inline).
- derive the active agent list from `agents/index.ts` via the same
  dep-free regex tactic as `parseTestFile` instead of hardcoding
  `["claude", "opencode"]` — adding a new harness file now wires it
  into the dynamic matrix automatically.
- treat `coverage: []` as `coverage: undefined` in `shouldRun` so an
  accidentally-empty array doesn't silently skip CI on every PR.
- add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the
  `timeout` test's coverage — the activity-timeout enforcement path
  was the original reason the test exists.
- ungate the `root` job (lint/format/typecheck/vitest). it's a required
  status check on `main`, so gating it on `code == 'true'` would make
  docs-only PRs unmergeable (skipped jobs don't satisfy required-check
  rules). the real LLM savings come from skipping the four matrices,
  not from skipping `root`.
- harden the four matrix-job `if:` guards from `outputs.matrix && ...`
  to `outputs.matrix != '' && ...` — explicit > implicit short-circuit.
- document `expandBraces`'s flat-only support so a future author isn't
  surprised by `{a,{b,c}}` not expanding.
- fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".
2026-05-14 03:55:33 +00:00
2026-01-16 08:00:16 +00:00
2026-03-12 05:22:51 +00:00
2025-08-27 16:53:48 -07:00
2026-01-19 08:41:56 +00:00
2026-05-14 02:48:31 +00:00
2026-03-12 05:22:51 +00:00

Green Pullfrog logo
Pullfrog

Bring your favorite coding agent into GitHub


🚀 Pullfrog is in beta! We're onboarding users in waves. Get on the waitlist →


What is Pullfrog?

Pullfrog is a GitHub bot that brings the full power of your favorite coding agents into GitHub. It's open source and powered by GitHub Actions.

  • Tag @pullfrog — Tag @pullfrog in a comment anywhere in your repo. It will pull in any relevant context using the action's internal MCP server and perform the appropriate task.
  • Prompt from the web — Trigger arbitrary tasks from the Pullfrog dashboard
  • Automated triggers — Configure Pullfrog to trigger agent runs in response to specific events. Each of these triggers can be associated with custom prompt instructions.
    • issue created
    • issue labeled
    • PR created
    • PR review created
    • PR review requested
    • and more...

Pullfrog is the bridge between your preferred coding agents and GitHub. Use it for:

  • 🤖 Coding tasks — Tell @pullfrog to implement something and it'll spin up a PR. If CI fails, it'll read the logs and attempt a fix automatically. It'll automatically address any PR reviews too.
  • 🔍 PR review — Coding agents are great at reviewing PRs. Using the "PR created" trigger, you can configure Pullfrog to auto-review new PRs.
  • 🤙 Issue management — Via the "issue created" trigger, Pullfrog can automatically respond to common questions, create implementation plans, and link to related issues/PRs. Or (if you're feeling lucky) you can prompt it to immediately attempt a PR addressing new issues.
  • Literally whatever — Want to have the agent automatically add docs to all new PRs? Cut a new release with agent-written notes on every commit to main? Pullfrog lets you do it.

Standalone Usage

You can also use pullfrog/pullfrog as a step in your own workflows. The action exposes a result output that can be consumed by subsequent steps.

Example: Auto-generate release notes on new tags

name: Release
on:
  push:
    tags: ['v*']

permissions:
  contents: write

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate release notes
        id: notes
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Generate release notes for ${{ github.ref_name }}.
            Compare commits between this tag and the previous tag.
            Format as markdown: summary paragraph, then ### Features, ### Fixes, ### Breaking Changes sections.
            Omit empty sections. Be concise.
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      # write to file to avoid shell escaping issues with special characters
      - name: Create GitHub release
        run: |
          notesfile="$RUNNER_TEMP/release-notes-$GITHUB_RUN_ID.md"
          printf '%s' "$NOTES" > "$notesfile"
          gh release create ${{ github.ref_name }} --title "${{ github.ref_name }}" --notes-file "$notesfile"
        env:
          GH_TOKEN: ${{ github.token }}
          NOTES: ${{ steps.notes.outputs.result }}

Example: Structured Output with Zod Schema

You can force the agent to return structured JSON output by providing a JSON schema. This allows you to reliably parse and use the agent's response in subsequent workflow steps.

You can define your JSON schema directly or uou can use any validation library that converts to JSON Schema. Here's an example using Zod:

name: Release Check
on:
  pull_request:
    types: [closed]

jobs:
  check-release:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm install --no-save --no-package-lock zod @actions/core

      - name: Generate Schema
        id: schema
        run: |
          node -e '
            import { z } from "zod";
            import { setOutput } from "@actions/core";
            const schema = z.object({
              version: z.string().describe("Semantic version number (e.g. 1.0.0)"),
              isBreaking: z.boolean().describe("Whether this release contains breaking changes"),
              changelog: z.array(z.string()).describe("List of changes in this release"),
            });
            setOutput("schema", JSON.stringify(z.toJSONSchema(schema)));
          '

      - name: Analyze PR
        id: analysis
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Analyze this PR and determine semantic versioning impact.
            Return a JSON object matching the provided schema.
          output_schema: ${{ steps.schema.outputs.schema }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Process Result
        run: |
          # Parse the JSON result using fromJSON()
          echo "Version: ${{ fromJSON(steps.analysis.outputs.result).version }}"
          echo "Breaking: ${{ fromJSON(steps.analysis.outputs.result).isBreaking }}"
S
Description
Self-hosted Ollama-powered code review bot for Gitea Actions based on pullfrog
Readme MIT 27 MiB
Languages
TypeScript 97.9%
Shell 0.7%
Dockerfile 0.7%
JavaScript 0.7%