Colin McDonnell 5caeb75344 review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models (#710)
* review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models

PR review wall-time was dominated by two failure modes: orchestrator
serial-dispatching subagents (despite prompt asking for parallel) and
running every lens on the same Opus tier as the orchestrator. Sample of
recent runs showed 25-60min reviews on small PRs, with 8-10min idle
gaps between subagent dispatches.

Three changes:

1. `action/modes.ts` — replace the soft "1 trivial / 2-3 typical /
   4-5 high-stakes" lens calibration with a binary 0-or-2+ rule. Default
   is 0 lenses (orchestrator handles review solo with optional cheap
   tracerfrog dispatches). 2+ parallel lenses only fire for substantive
   PRs (>5 files AND >200 lines) or high-stakes-subsystem touches. Never
   exactly one. Both Review and IncrementalReview prompts get loud
   ALL-CAPS framing on parallel dispatch — emit ALL Task tool_use blocks
   in a single assistant turn before reading any result. Drop the
   "do NOT lens-review the diff yourself" advice; orchestrator pulls
   context aggressively, in parallel with the lens fan-out.

2. New `tracerfrog` subagent for mechanical code tracing ("where is X
   used / who calls Y / what depends on Z"). Pure read+grep+report with
   no judgment — orchestrator can dispatch many tracers cheaply in
   parallel. Defined in `action/agents/reviewer.ts`. Wired into both
   claude.ts (`--agents` JSON) and opencode.ts (`agent` config block).

3. Per-subagent model downshifts via `deriveSubagentModels`:
   - Anthropic: reviewfrog → Sonnet, tracerfrog → Haiku
   - OpenAI: both → gpt-5.4-mini
   - other providers (xai, deepseek, gemini, etc.): inherit (no
     standard tier triplet to downshift to)

Claude Code path always runs Anthropic so the downshift is hardcoded
inline in claude.ts. OpenCode uses the helper since orchestrator
provider varies.

Both runtimes' subagent-definition formats verified directly against
their source: `--agents` JSON `model` field (claude-code's
`AgentJsonSchema` accepts model+effort+maxTurns+more) and OpenCode's
`agent.{name}.model` config field (parsed via Provider.parseModel,
applied per-task in tool/task.ts line 92). Parallel dispatch is
infra-supported in both — only the orchestrator model's tool_use
emission pattern was the bottleneck.

Tests: subagentModels.test.ts (14 tests covering provider matrix),
subagentRegistration.test.ts (6 source-asserts catching shape
regressions in buildAgentsJson / buildReviewerAgentConfig).

* subagentModels: add openrouter routes (proxy/router mode)

Initial helper missed the openrouter prefix used by Pullfrog's router
proxy. preview-710 e2e showed the OpenCode + openrouter path receiving
no downshift — orchestrator and lenses both ran on opus-4.7 because
'openrouter/anthropic/claude-opus-4.7' didn't match any of the
anthropic/openai prefixes the helper checked.

Add explicit branches for 'openrouter/anthropic/...' (uses dot notation:
claude-sonnet-4.6 / claude-haiku-4.5) and 'openrouter/openai/...'
(gpt-5.4-mini for both reviewer and tracer). Same opus->sonnet,
sonnet->keep-but-haiku-tracer, haiku->no-op semantics as the direct
anthropic path.

* opencode: log resolved subagent models at startup

So we can verify per-subagent model overrides actually take effect at
runtime. Prints once per run alongside the existing model/effort log
lines.

* drop tracerfrog: keep reviewfrog only, LSP-powered tracer planned later

Removes the cheap-haiku-tracer subagent (TRACER_AGENT_NAME +
TRACER_SYSTEM_PROMPT, registrations in claude.ts/opencode.ts, dispatch
guidance in modes.ts). The mechanical-tracing use case will be served
better by an LSP-powered tool than by a separately-prompted subagent.

deriveSubagentModels collapses to a single { reviewer } shape; the
reviewfrog-on-Sonnet downshift stays. Same source-assert + provider-
matrix tests, minus the tracer-specific cases.

modes.ts wording: drop the 'subagent type cheat sheet' bullet, drop
the parenthetical 'often better served by tracerfrog than reviewfrog'
on the impact lens, drop tracerfrog from the same-turn-context-pulling
hint. The 0-or-2+ rule and ALL-CAPS parallel emphasis are unchanged.

* subagentModels: broader downshift coverage (gpt-pro, gemini-pro, grok); drop gpt-mini target

Scanned every resolved orchestrator slug in action/models.ts against
models.dev pricing data. Identified five clear cases where the
orchestrator is meaningfully expensive AND has a cheaper sibling that
remains capable enough for review-style judgment work.

Changes:
- Anthropic: opus → sonnet  (kept; -40%)
- OpenAI: gpt → gpt-5.4 (was: gpt-mini; -54% instead of -85% but
  preserves review-quality judgment — gpt-mini was too dumb)
- OpenAI: gpt-pro → gpt   (NEW; -93%, biggest single unlock —
  gpt-5.5-pro is $30/Mtok in vs gpt-5.5 at $5)
- Google: gemini-pro → gemini-flash  (NEW; -75%)
- xAI: grok-4.3 → grok-4-1-fast  (NEW; -80%)

Every branch handles the three routes in use: direct provider slug,
opencode-vendored, and openrouter-proxied. Variants below the downshift
target (mini/nano/flash/fast/sonnet/haiku) inherit (no further drop).

Skipped:
- DeepSeek: v4-flash ($0.14/Mtok) is too far below review judgment
  threshold; v4-pro orchestrator already cheap ($0.55 blended).
- Moonshot: kimi-k2-thinking would only save 32% and slug stability on
  OpenRouter is uncertain; revisit if cost matters.
- o3: already mid-tier in OpenAI's reasoning family; no clean target.

* models: hoist subagent downshift into the registry, add hidden flag

The downshift relationship now lives next to each alias's resolve /
openRouterResolve as a sibling field. Two new ModelDef fields:

- subagentModel?: string — alias key (within same provider) of the
  cheaper sibling reviewfrog should use as a lens-fanout subagent.
  e.g. claude-opus → 'claude-sonnet'.
- hidden?: boolean — exclude from selectable lists (UI dropdown,
  CLI init picker). Does NOT affect resolution; for that use
  fallback. Used so internal-only subagent targets like openai/gpt-5.4
  exist in the registry but never appear as a user-facing pick.

Wiring:
- anthropic.claude-opus → claude-sonnet (-40%)
- openai.gpt-pro → gpt (-93%, biggest unlock)
- openai.gpt → gpt-5.4 (-54%); gpt-5.4 added with hidden:true
- google.gemini-pro → gemini-flash (-75%)
- mirrored across opencode + openrouter providers (each provider
  declares its own three-route data so the downshift declaration
  is colocated with the rest of the alias definition).

deriveSubagentModels collapses from ~85 lines of prefix-matching to
a ~15-line registry reverse-lookup: find the alias whose resolve OR
openRouterResolve matches the orchestrator's spec, follow its
subagentModel pointer, return the matching field of the target alias.

Filter sites updated:
- components/ModelSelector.tsx: !a.fallback && !a.hidden
- action/commands/init.ts:       same

Tests rewritten to exercise the registry through the public surface;
the matrix collapses to one assertion per (provider × route) pair.

* TEMP: log per-step cost+tokens for subagent model verification (PR #710)

* TEMP: also log SUBAGENT step_finish from bus envelope handler

* remove temporary per-step diagnostic logs (verification done)

Verified subagent model downshift takes effect end-to-end on the OpenCode
+ openrouter path. PR #8 in pullfrog/preview-710-review-perf dispatched
3 lenses (billing-subsystem / security / correctness) on the orchestrator's
opus-4.7 session, and per-subagent step_finish events showed actual cost
exactly matching Sonnet pricing rates (60% of what Opus would have cost):

  session       n  actual    if-Opus   if-Sonnet  match
  T3VrUuF...    5  $0.2425   $0.4042   $0.2425    Sonnet ✓
  93ZZR7E...    4  $0.2253   $0.3754   $0.2253    Sonnet ✓
  Fb1Kr7b...    4  $0.2495   $0.4158   $0.2495    Sonnet ✓

The startup '» subagent models: reviewfrog=...' line stays — useful
permanent diagnostic showing the resolved subagent model per-run.

* TEMP: log per-event model from claude.ts assistant handler

* remove temporary per-event model log (claude.ts verification done)

Verified subagent model downshift takes effect end-to-end on the Claude
Code path. PR #9 in pullfrog/preview-710-review-perf dispatched 2 lenses
on an opus-4-7 orchestrator. Per-assistant-event model field from the
SDK's stream-json output, partitioned by parent_tool_use_id:

  ORCH (parent_tool_use_id=null):  17 events all model=claude-opus-4-7
  SUBAGENT lens:billing-subsystem: 17 events all model=claude-sonnet-4-6
  SUBAGENT lens:security:          21 events all model=claude-sonnet-4-6

Zero leakage to opus from either subagent session. The per-subagent
'model' field in --agents JSON is honored by claude-code at the SDK
level, identical to the OpenCode path verified earlier.

* opencode: bump per-call output cap 5K → 16K to unblock large reviews

The 5K cap (added in #616 to lower OpenRouter upfront budget reservation
for low-wallet runs) was capping the entire response of a single LLM call,
not just the budget reservation. A single tool_use response — like a
`create_pull_request_review` with many inline comments — would truncate
mid-stream past 5K output tokens, leave the JSON unparseable, and the tool
would never actually invoke. We hit this on PR #710's verify-downshift PR:
review aggregated from 3 lenses had 11 inline comments + a long body,
truncated at out=5000 on every retry attempt, action exited with 'Review
mode finished without calling create_pull_request_review after 3 retry
attempts'.

Investigated whether OpenCode (or OpenAI/Anthropic/OpenRouter directly)
exposes a separate budget-reservation parameter that could stay small
while letting the response exceed it. They don't — `max_tokens` /
`max_completion_tokens` is the single value all four use for both the
upfront reservation and the hard output ceiling. No way to decouple them
at the API surface.

Bumped to 16K as a middle ground: 8× the prior cap (handles every review
shape we've observed plus headroom), still half of OpenCode's 32K default
so the wallet-burn benefit for low-balance accounts is preserved, just
smaller. For Opus 4.7 a typical ~50K-input call now reserves roughly
$0.65 instead of the prior $0.38.

Updated the constant comment to spell out the trade-off clearly so this
doesn't happen again.

* opencode: drop OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX override entirely

Verified the original rationale for the override is obsolete. From #616
the cap shrunk OpenRouter's per-call upfront budget reservation so a
single call's reservation wouldn't exceed the per-run key cap
(`ROUTER_PER_RUN_LIMIT_USD = 25`) and lock low-balance accounts out of
starting a run.

That per-run gate is gone. `app/api/proxy-token/route.ts` ~line 422
explicitly says: 'No upper cap (the old ROUTER_PER_RUN_LIMIT_USD = 25 is
gone). The natural ceiling is whatever the user has + their buffer.'
Router now mints keys with `keyLimitCents = balance + buffer` ($50 for
autoreload+card, $5 for card-only, $0 for no-card). A single call's
upfront reservation fits comfortably within that — no separate per-call
gate to fail past.

The cap had a real downside as a hard per-call output truncation. A
single `create_pull_request_review` tool_use with many inline comments
would truncate mid-stream past 5K output tokens, the JSON would be
unparseable, and the tool never invoked. Hit on PR #710's
verify-downshift PR.

Removing the override entirely; OpenCode falls back to its 32K default.
Left an explanatory note above the env-var assignment site so the next
person doesn't unknowingly re-add it.
2026-05-13 21:05:52 +00:00
2026-01-16 08:00:16 +00:00
2026-03-12 05:22:51 +00:00
2025-08-27 16:53:48 -07:00
2026-01-19 08:41:56 +00:00
2026-05-13 18:23:45 +00:00
2026-03-12 05:22:51 +00:00

Green Pullfrog logo
Pullfrog

Bring your favorite coding agent into GitHub


🚀 Pullfrog is in beta! We're onboarding users in waves. Get on the waitlist →


What is Pullfrog?

Pullfrog is a GitHub bot that brings the full power of your favorite coding agents into GitHub. It's open source and powered by GitHub Actions.

  • Tag @pullfrog — Tag @pullfrog in a comment anywhere in your repo. It will pull in any relevant context using the action's internal MCP server and perform the appropriate task.
  • Prompt from the web — Trigger arbitrary tasks from the Pullfrog dashboard
  • Automated triggers — Configure Pullfrog to trigger agent runs in response to specific events. Each of these triggers can be associated with custom prompt instructions.
    • issue created
    • issue labeled
    • PR created
    • PR review created
    • PR review requested
    • and more...

Pullfrog is the bridge between your preferred coding agents and GitHub. Use it for:

  • 🤖 Coding tasks — Tell @pullfrog to implement something and it'll spin up a PR. If CI fails, it'll read the logs and attempt a fix automatically. It'll automatically address any PR reviews too.
  • 🔍 PR review — Coding agents are great at reviewing PRs. Using the "PR created" trigger, you can configure Pullfrog to auto-review new PRs.
  • 🤙 Issue management — Via the "issue created" trigger, Pullfrog can automatically respond to common questions, create implementation plans, and link to related issues/PRs. Or (if you're feeling lucky) you can prompt it to immediately attempt a PR addressing new issues.
  • Literally whatever — Want to have the agent automatically add docs to all new PRs? Cut a new release with agent-written notes on every commit to main? Pullfrog lets you do it.

Standalone Usage

You can also use pullfrog/pullfrog as a step in your own workflows. The action exposes a result output that can be consumed by subsequent steps.

Example: Auto-generate release notes on new tags

name: Release
on:
  push:
    tags: ['v*']

permissions:
  contents: write

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate release notes
        id: notes
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Generate release notes for ${{ github.ref_name }}.
            Compare commits between this tag and the previous tag.
            Format as markdown: summary paragraph, then ### Features, ### Fixes, ### Breaking Changes sections.
            Omit empty sections. Be concise.
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      # write to file to avoid shell escaping issues with special characters
      - name: Create GitHub release
        run: |
          notesfile="$RUNNER_TEMP/release-notes-$GITHUB_RUN_ID.md"
          printf '%s' "$NOTES" > "$notesfile"
          gh release create ${{ github.ref_name }} --title "${{ github.ref_name }}" --notes-file "$notesfile"
        env:
          GH_TOKEN: ${{ github.token }}
          NOTES: ${{ steps.notes.outputs.result }}

Example: Structured Output with Zod Schema

You can force the agent to return structured JSON output by providing a JSON schema. This allows you to reliably parse and use the agent's response in subsequent workflow steps.

You can define your JSON schema directly or uou can use any validation library that converts to JSON Schema. Here's an example using Zod:

name: Release Check
on:
  pull_request:
    types: [closed]

jobs:
  check-release:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm install --no-save --no-package-lock zod @actions/core

      - name: Generate Schema
        id: schema
        run: |
          node -e '
            import { z } from "zod";
            import { setOutput } from "@actions/core";
            const schema = z.object({
              version: z.string().describe("Semantic version number (e.g. 1.0.0)"),
              isBreaking: z.boolean().describe("Whether this release contains breaking changes"),
              changelog: z.array(z.string()).describe("List of changes in this release"),
            });
            setOutput("schema", JSON.stringify(z.toJSONSchema(schema)));
          '

      - name: Analyze PR
        id: analysis
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Analyze this PR and determine semantic versioning impact.
            Return a JSON object matching the provided schema.
          output_schema: ${{ steps.schema.outputs.schema }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Process Result
        run: |
          # Parse the JSON result using fromJSON()
          echo "Version: ${{ fromJSON(steps.analysis.outputs.result).version }}"
          echo "Breaking: ${{ fromJSON(steps.analysis.outputs.result).isBreaking }}"
S
Description
Self-hosted Ollama-powered code review bot for Gitea Actions based on pullfrog
Readme MIT 27 MiB
Languages
TypeScript 97.9%
Shell 0.7%
Dockerfile 0.7%
JavaScript 0.7%