Colin McDonnell f662b1a0c8 unify per-run token + cost accounting + persist to WorkflowRun (#547)
* unify per-run token + cost accounting across agents

every agent harness now logs the same 5-column (or 6 with cost) table and
populates the same AgentUsage contract, regardless of agent or upstream
provider. previously OpenCode and the Claude fallback path emitted a 3-col
table whose "Input Tokens" was actually only the non-cached delta, silently
dropping cache read/write — real runs were being reported at ~0.4% of their
true input (e.g. one baseline showed Input=30 while step_finish events
summed to cache_read=724,753).

changes:
- add logTokenTable helper in action/agents/shared.ts with stable columns:
  Input | Cache Read | Cache Write | Output | Total | Cost ($). cost
  column renders only when a value is known.
- action/agents/opencode.ts: accumulate step_finish.part.tokens AND
  step_finish.part.cost (sourced from models.dev inside opencode —
  confirmed working across Anthropic, OpenAI, Google, xAI, DeepSeek,
  Moonshot, and OpenRouter). drop the event.stats.total_tokens fallback
  since that payload has no cache breakdown.
- action/agents/claude.ts: success-path now treats input_tokens as the
  non-cached field (matching OpenCode semantics), carries
  cache_read_input_tokens / cache_creation_input_tokens separately, and
  captures total_cost_usd from the final result event. the per-message
  fallback accumulator now captures cache fields too so it's no longer
  lossy when the result event never fires.
- formatUsageSummary gains a Cost ($) column that matches the stdout
  table row-for-row; missing values render as "—".
- scripts/token-usage.ts parses all three historical formats (new 5-col,
  legacy 4-col Claude success, legacy 3-col lossy) and explicitly flags
  the lossy runs instead of averaging misleading values.

validation (pnpm play --local, identical "say hello" prompt):

  agent+model                           Input  CacheR  CacheW  Output  Total   Cost
  OpenCode + Anthropic Sonnet 4.6           4  41,177  20,735     129  62,045  $0.0921
  Claude CLI  + Anthropic Sonnet 4.6        9  80,133  11,611     389  92,142  $0.0766
  OpenCode + OpenAI codex-mini         10,893  46,976       0     606  58,475  $0.0059
  OpenCode + Google Gemini 3 Flash         —       —       —       —       —  $0.0114
  OpenCode + xAI Grok 4 Fast                —       —       —       —       —  $0.0035
  OpenCode + DeepSeek Chat             18,854       0       0       1  18,855  $0.0053
  OpenCode + Moonshot Kimi K2.5             —       —       —       —       —  $0.0106
  OpenCode + OpenRouter→Anthropic           —       —       —       —       —  $0.0617
  OpenCode + OpenRouter→OpenAI              —       —       —       —       —  $0.0038

* isolate play.ts from developer gitconfig

play.ts is a CI-emulator but inherits the developer's user- and system-scope
gitconfig. a common local convenience — url."git@github.com:".insteadOf
"https://github.com/" to force SSH auth — gets applied at read time on every
git call inside the temp repo, causing `git remote get-url --push origin`
to return an SSH URL instead of the stored HTTPS one. pullfrog_push_branch's
validatePushDestination (correctly) treats that as tampering and blocks the
push. the agent then burns the full MAX_COMMIT_RETRIES budget trying
workarounds that can't beat a user-scope insteadOf rule, turning a trivial
"say hello" run into a 1.35M-token session.

point GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null inside run() so
the play process and its spawned agent see the same empty gitconfig that
a real CI runner would. CI has no rewrites, so this is a no-op there; dev
machines get CI-identical git state. SSH client config (~/.ssh/config and
keys) is separate from gitconfig and is unaffected, so setupTestRepo's SSH
clone still works locally. setupGit only writes --local scope, so nothing
downstream depends on user-scope values.

verification: with the scratch repo cleaned up and this isolation in place,
OpenCode + Anthropic on the same "say hello" prompt goes from 1,349,654
tokens / $2.00+ to 62,045 tokens / $0.0921 — no retry loop, no push blocks.

* persist aggregated token + cost usage to WorkflowRun

AgentUsage has been memory-only — rendered into the GitHub step summary
and then discarded when the runner tears down. that made questions like
"avg cost per customer per day" require log-spelunking. persist it:

- add Int? columns for inputTokens / outputTokens / cacheReadTokens /
  cacheWriteTokens and a Decimal? costUsd column on workflow_runs.
  Int4's 2.1B ceiling is ~200x larger than any realistic run so BigInt
  would be overkill. costUsd uses the same default Decimal precision
  as existing money columns (accounts.usageUsd, proxy_keys.hwmUsage).

- extend PATCH /api/workflow-run/[runId] to accept the new numeric
  fields alongside the existing artifact strings. per-field type
  validation ensures the allowlist stays scalar-safe and rejects
  negative / non-finite values.

- generalize patchWorkflowRunFields in the action so it accepts a
  mixed string/number payload, and add an aggregateUsage(entries)
  helper that sums per-agent AgentUsage records into a single patch.

- call the reporter from main.ts's outer finally block, gated on
  toolContext. this is the shared cleanup path that every agent
  implementation flows through — claude.ts, opencode.ts, and any
  future harness all push their AgentUsage into toolState.usageEntries
  via the same line 468, so one finally-block call covers them all.
  running in finally also means partial usage gets persisted even
  when the agent errored out mid-run.

* anneal token + cost accounting

follow-up polish from a review pass:

- aggregate usage across commit-retry iterations inside each agent harness.
  previously runClaude / runOpenCode returned only the final retry's usage,
  so any run that hit the dirty-tree retry loop under-counted tokens and
  cost in both the stdout table and the WorkflowRun row. added a shared
  mergeAgentUsage helper in agents/shared.ts; both harnesses now fold each
  iteration's usage into a running total and return the sum.

- scripts/token-usage.ts now handles the unified format with or without
  the Cost ($) column. previously the int-only number regex rejected
  decimals and the 5-cell length check rejected 6-cell rows, so logs
  from post-cost-tracking runs fell through to "no token table". the
  parser now accepts both 5- and 6-cell unified rows, splits int vs
  decimal cells, and averages reported Cost alongside the tokens.

- PATCH /api/workflow-run/[runId] now rejects INT field values above
  INT4_MAX (2_147_483_647) so a malformed payload gets a clean 400
  instead of propagating a Prisma error. also defends against a
  compromised runner sending a deliberately huge value.

- clarifying comments: opencode.ts documents that step_finish.part.cost
  is a per-step delta (empirically verified), main.ts explains that
  toolState.usageEntries already carries merged per-retry usage so
  aggregateUsage just sums entries (one per agent.run()).

- tests for aggregateUsage and mergeAgentUsage — 12 new cases covering
  empty / partial / multi-agent inputs and the "keep undefined" semantic
  that prevents spurious zeros from being persisted.

- drop `as number` cast in logTokenTable — narrow via const instead.

* anneal: clamp INT overflow + guarantee mergeAgentUsage immutability

second review pass surfaced two defensive gaps:

- a single token field exceeding INT4_MAX would pass the client but be
  rejected by the server's per-field validator, writing a partial row
  with some NULLs where sums belonged. clamp in aggregateUsage so the
  wire payload is always self-consistent across all numeric columns,
  with a loud warning so the clamp doesn't silently swallow weirdness.

- mergeAgentUsage's single-sided branches returned the input reference.
  callers treat AgentUsage as immutable but future callers might not;
  always return a fresh shallow copy instead. two new tests guarantee
  the no-mutation-leak property.

no behavior change in the happy path — INT4_MAX is ~200x the largest
realistic per-run token count.

* anneal: resilient usage persistence + cross-platform null device

third review pass surfaced three small issues:

- main.ts finally block: writeGitHubUsageSummaryToFile throwing would
  skip the WorkflowRun usage PATCH. both are independent best-effort
  cleanup tasks — wrap the former in catch so a filesystem failure
  doesn't block DB persistence.

- AgentUsage.inputTokens had no jsdoc explaining that it's the full
  billable input (cached + non-cached). the same word "Input" means
  "non-cached only" in the stdout/markdown tables (derived by
  subtraction). document the semantic so dashboards querying
  WorkflowRun.inputTokens don't misinterpret it.

- play.ts gitconfig isolation was hard-coded to "/dev/null" which
  doesn't exist on Windows. use `os.devNull` for cross-platform
  parity (resolves to `\\.\nul` on win32). the project is Linux-only
  in CI so this only helps local Windows contributors, but it's a
  zero-cost swap.

also updated the finally-block caveat comment: usage is only pushed
to toolState.usageEntries when agent.run() returns an AgentResult,
not when the timeout race rejects — so timed-out runs don't
persist partial usage. documented instead of trying to thread state
through Promise.race.

* anneal: NaN-guard cost accumulators + clarify inputTokens docs

final polish from review round 4:

- guard both cost accumulators (opencode step_finish.part.cost and claude
  result.total_cost_usd) with Number.isFinite. `typeof x === "number"`
  accepts NaN, and one NaN `+=` would poison the running total for the
  whole session.

- reword prisma schema comment on WorkflowRun usage fields to call out
  that cacheReadTokens / cacheWriteTokens are SUB-totals within
  inputTokens (not additional tokens on top). prevents future dashboards
  from double-counting by ~2x when summing "total tokens used".
2026-04-20 21:27:54 +00:00
2026-01-16 08:00:16 +00:00
2026-03-12 05:22:51 +00:00
2025-08-27 16:53:48 -07:00
2026-01-19 08:41:56 +00:00
2026-04-16 16:33:49 +00:00
2026-03-12 05:22:51 +00:00
2026-03-12 05:22:51 +00:00

Green Pullfrog logo
Pullfrog

Bring your favorite coding agent into GitHub


🚀 Pullfrog is in beta! We're onboarding users in waves. Get on the waitlist →


What is Pullfrog?

Pullfrog is a GitHub bot that brings the full power of your favorite coding agents into GitHub. It's open source and powered by GitHub Actions.

  • Tag @pullfrog — Tag @pullfrog in a comment anywhere in your repo. It will pull in any relevant context using the action's internal MCP server and perform the appropriate task.
  • Prompt from the web — Trigger arbitrary tasks from the Pullfrog dashboard
  • Automated triggers — Configure Pullfrog to trigger agent runs in response to specific events. Each of these triggers can be associated with custom prompt instructions.
    • issue created
    • issue labeled
    • PR created
    • PR review created
    • PR review requested
    • and more...

Pullfrog is the bridge between your preferred coding agents and GitHub. Use it for:

  • 🤖 Coding tasks — Tell @pullfrog to implement something and it'll spin up a PR. If CI fails, it'll read the logs and attempt a fix automatically. It'll automatically address any PR reviews too.
  • 🔍 PR review — Coding agents are great at reviewing PRs. Using the "PR created" trigger, you can configure Pullfrog to auto-review new PRs.
  • 🤙 Issue management — Via the "issue created" trigger, Pullfrog can automatically respond to common questions, create implementation plans, and link to related issues/PRs. Or (if you're feeling lucky) you can prompt it to immediately attempt a PR addressing new issues.
  • Literally whatever — Want to have the agent automatically add docs to all new PRs? Cut a new release with agent-written notes on every commit to main? Pullfrog lets you do it.

Standalone Usage

You can also use pullfrog/pullfrog as a step in your own workflows. The action exposes a result output that can be consumed by subsequent steps.

Example: Auto-generate release notes on new tags

name: Release
on:
  push:
    tags: ['v*']

permissions:
  contents: write

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate release notes
        id: notes
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Generate release notes for ${{ github.ref_name }}.
            Compare commits between this tag and the previous tag.
            Format as markdown: summary paragraph, then ### Features, ### Fixes, ### Breaking Changes sections.
            Omit empty sections. Be concise.
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      # write to file to avoid shell escaping issues with special characters
      - name: Create GitHub release
        run: |
          notesfile="$RUNNER_TEMP/release-notes-$GITHUB_RUN_ID.md"
          printf '%s' "$NOTES" > "$notesfile"
          gh release create ${{ github.ref_name }} --title "${{ github.ref_name }}" --notes-file "$notesfile"
        env:
          GH_TOKEN: ${{ github.token }}
          NOTES: ${{ steps.notes.outputs.result }}

Example: Structured Output with Zod Schema

You can force the agent to return structured JSON output by providing a JSON schema. This allows you to reliably parse and use the agent's response in subsequent workflow steps.

You can define your JSON schema directly or uou can use any validation library that converts to JSON Schema. Here's an example using Zod:

name: Release Check
on:
  pull_request:
    types: [closed]

jobs:
  check-release:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm install --no-save --no-package-lock zod @actions/core

      - name: Generate Schema
        id: schema
        run: |
          node -e '
            import { z } from "zod";
            import { setOutput } from "@actions/core";
            const schema = z.object({
              version: z.string().describe("Semantic version number (e.g. 1.0.0)"),
              isBreaking: z.boolean().describe("Whether this release contains breaking changes"),
              changelog: z.array(z.string()).describe("List of changes in this release"),
            });
            setOutput("schema", JSON.stringify(z.toJSONSchema(schema)));
          '

      - name: Analyze PR
        id: analysis
        uses: pullfrog/pullfrog@v0
        with:
          prompt: |
            Analyze this PR and determine semantic versioning impact.
            Return a JSON object matching the provided schema.
          output_schema: ${{ steps.schema.outputs.schema }}
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - name: Process Result
        run: |
          # Parse the JSON result using fromJSON()
          echo "Version: ${{ fromJSON(steps.analysis.outputs.result).version }}"
          echo "Breaking: ${{ fromJSON(steps.analysis.outputs.result).isBreaking }}"
S
Description
Self-hosted Ollama-powered code review bot for Gitea Actions based on pullfrog
Readme MIT 27 MiB
Languages
TypeScript 97.9%
Shell 0.7%
Dockerfile 0.7%
JavaScript 0.7%