unify per-run token + cost accounting + persist to WorkflowRun (#547)
* unify per-run token + cost accounting across agents every agent harness now logs the same 5-column (or 6 with cost) table and populates the same AgentUsage contract, regardless of agent or upstream provider. previously OpenCode and the Claude fallback path emitted a 3-col table whose "Input Tokens" was actually only the non-cached delta, silently dropping cache read/write — real runs were being reported at ~0.4% of their true input (e.g. one baseline showed Input=30 while step_finish events summed to cache_read=724,753). changes: - add logTokenTable helper in action/agents/shared.ts with stable columns: Input | Cache Read | Cache Write | Output | Total | Cost ($). cost column renders only when a value is known. - action/agents/opencode.ts: accumulate step_finish.part.tokens AND step_finish.part.cost (sourced from models.dev inside opencode — confirmed working across Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, and OpenRouter). drop the event.stats.total_tokens fallback since that payload has no cache breakdown. - action/agents/claude.ts: success-path now treats input_tokens as the non-cached field (matching OpenCode semantics), carries cache_read_input_tokens / cache_creation_input_tokens separately, and captures total_cost_usd from the final result event. the per-message fallback accumulator now captures cache fields too so it's no longer lossy when the result event never fires. - formatUsageSummary gains a Cost ($) column that matches the stdout table row-for-row; missing values render as "—". - scripts/token-usage.ts parses all three historical formats (new 5-col, legacy 4-col Claude success, legacy 3-col lossy) and explicitly flags the lossy runs instead of averaging misleading values. validation (pnpm play --local, identical "say hello" prompt): agent+model Input CacheR CacheW Output Total Cost OpenCode + Anthropic Sonnet 4.6 4 41,177 20,735 129 62,045 $0.0921 Claude CLI + Anthropic Sonnet 4.6 9 80,133 11,611 389 92,142 $0.0766 OpenCode + OpenAI codex-mini 10,893 46,976 0 606 58,475 $0.0059 OpenCode + Google Gemini 3 Flash — — — — — $0.0114 OpenCode + xAI Grok 4 Fast — — — — — $0.0035 OpenCode + DeepSeek Chat 18,854 0 0 1 18,855 $0.0053 OpenCode + Moonshot Kimi K2.5 — — — — — $0.0106 OpenCode + OpenRouter→Anthropic — — — — — $0.0617 OpenCode + OpenRouter→OpenAI — — — — — $0.0038 * isolate play.ts from developer gitconfig play.ts is a CI-emulator but inherits the developer's user- and system-scope gitconfig. a common local convenience — url."git@github.com:".insteadOf "https://github.com/" to force SSH auth — gets applied at read time on every git call inside the temp repo, causing `git remote get-url --push origin` to return an SSH URL instead of the stored HTTPS one. pullfrog_push_branch's validatePushDestination (correctly) treats that as tampering and blocks the push. the agent then burns the full MAX_COMMIT_RETRIES budget trying workarounds that can't beat a user-scope insteadOf rule, turning a trivial "say hello" run into a 1.35M-token session. point GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null inside run() so the play process and its spawned agent see the same empty gitconfig that a real CI runner would. CI has no rewrites, so this is a no-op there; dev machines get CI-identical git state. SSH client config (~/.ssh/config and keys) is separate from gitconfig and is unaffected, so setupTestRepo's SSH clone still works locally. setupGit only writes --local scope, so nothing downstream depends on user-scope values. verification: with the scratch repo cleaned up and this isolation in place, OpenCode + Anthropic on the same "say hello" prompt goes from 1,349,654 tokens / $2.00+ to 62,045 tokens / $0.0921 — no retry loop, no push blocks. * persist aggregated token + cost usage to WorkflowRun AgentUsage has been memory-only — rendered into the GitHub step summary and then discarded when the runner tears down. that made questions like "avg cost per customer per day" require log-spelunking. persist it: - add Int? columns for inputTokens / outputTokens / cacheReadTokens / cacheWriteTokens and a Decimal? costUsd column on workflow_runs. Int4's 2.1B ceiling is ~200x larger than any realistic run so BigInt would be overkill. costUsd uses the same default Decimal precision as existing money columns (accounts.usageUsd, proxy_keys.hwmUsage). - extend PATCH /api/workflow-run/[runId] to accept the new numeric fields alongside the existing artifact strings. per-field type validation ensures the allowlist stays scalar-safe and rejects negative / non-finite values. - generalize patchWorkflowRunFields in the action so it accepts a mixed string/number payload, and add an aggregateUsage(entries) helper that sums per-agent AgentUsage records into a single patch. - call the reporter from main.ts's outer finally block, gated on toolContext. this is the shared cleanup path that every agent implementation flows through — claude.ts, opencode.ts, and any future harness all push their AgentUsage into toolState.usageEntries via the same line 468, so one finally-block call covers them all. running in finally also means partial usage gets persisted even when the agent errored out mid-run. * anneal token + cost accounting follow-up polish from a review pass: - aggregate usage across commit-retry iterations inside each agent harness. previously runClaude / runOpenCode returned only the final retry's usage, so any run that hit the dirty-tree retry loop under-counted tokens and cost in both the stdout table and the WorkflowRun row. added a shared mergeAgentUsage helper in agents/shared.ts; both harnesses now fold each iteration's usage into a running total and return the sum. - scripts/token-usage.ts now handles the unified format with or without the Cost ($) column. previously the int-only number regex rejected decimals and the 5-cell length check rejected 6-cell rows, so logs from post-cost-tracking runs fell through to "no token table". the parser now accepts both 5- and 6-cell unified rows, splits int vs decimal cells, and averages reported Cost alongside the tokens. - PATCH /api/workflow-run/[runId] now rejects INT field values above INT4_MAX (2_147_483_647) so a malformed payload gets a clean 400 instead of propagating a Prisma error. also defends against a compromised runner sending a deliberately huge value. - clarifying comments: opencode.ts documents that step_finish.part.cost is a per-step delta (empirically verified), main.ts explains that toolState.usageEntries already carries merged per-retry usage so aggregateUsage just sums entries (one per agent.run()). - tests for aggregateUsage and mergeAgentUsage — 12 new cases covering empty / partial / multi-agent inputs and the "keep undefined" semantic that prevents spurious zeros from being persisted. - drop `as number` cast in logTokenTable — narrow via const instead. * anneal: clamp INT overflow + guarantee mergeAgentUsage immutability second review pass surfaced two defensive gaps: - a single token field exceeding INT4_MAX would pass the client but be rejected by the server's per-field validator, writing a partial row with some NULLs where sums belonged. clamp in aggregateUsage so the wire payload is always self-consistent across all numeric columns, with a loud warning so the clamp doesn't silently swallow weirdness. - mergeAgentUsage's single-sided branches returned the input reference. callers treat AgentUsage as immutable but future callers might not; always return a fresh shallow copy instead. two new tests guarantee the no-mutation-leak property. no behavior change in the happy path — INT4_MAX is ~200x the largest realistic per-run token count. * anneal: resilient usage persistence + cross-platform null device third review pass surfaced three small issues: - main.ts finally block: writeGitHubUsageSummaryToFile throwing would skip the WorkflowRun usage PATCH. both are independent best-effort cleanup tasks — wrap the former in catch so a filesystem failure doesn't block DB persistence. - AgentUsage.inputTokens had no jsdoc explaining that it's the full billable input (cached + non-cached). the same word "Input" means "non-cached only" in the stdout/markdown tables (derived by subtraction). document the semantic so dashboards querying WorkflowRun.inputTokens don't misinterpret it. - play.ts gitconfig isolation was hard-coded to "/dev/null" which doesn't exist on Windows. use `os.devNull` for cross-platform parity (resolves to `\\.\nul` on win32). the project is Linux-only in CI so this only helps local Windows contributors, but it's a zero-cost swap. also updated the finally-block caveat comment: usage is only pushed to toolState.usageEntries when agent.run() returns an AgentResult, not when the timeout race rejects — so timed-out runs don't persist partial usage. documented instead of trying to thread state through Promise.race. * anneal: NaN-guard cost accumulators + clarify inputTokens docs final polish from review round 4: - guard both cost accumulators (opencode step_finish.part.cost and claude result.total_cost_usd) with Number.isFinite. `typeof x === "number"` accepts NaN, and one NaN `+=` would poison the running total for the whole session. - reword prisma schema comment on WorkflowRun usage fields to call out that cacheReadTokens / cacheWriteTokens are SUB-totals within inputTokens (not additional tokens on top). prevents future dashboards from double-counting by ~2x when summing "total tokens used".
This commit is contained in:
committed by
pullfrog[bot]
parent
57bd10d6dd
commit
f662b1a0c8
+45
-23
@@ -33,8 +33,10 @@ import {
|
||||
agent,
|
||||
buildCommitPrompt,
|
||||
getGitStatus,
|
||||
logTokenTable,
|
||||
MAX_COMMIT_RETRIES,
|
||||
MAX_STDERR_LINES,
|
||||
mergeAgentUsage,
|
||||
} from "./shared.ts";
|
||||
|
||||
async function installClaudeCli(): Promise<string> {
|
||||
@@ -192,6 +194,10 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
|
||||
let finalOutput = "";
|
||||
let sessionId: string | undefined;
|
||||
let accumulatedTokens = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
|
||||
// Claude CLI reports a single end-of-run `total_cost_usd` on the result
|
||||
// event. per-message events don't carry cost, so there's nothing to sum —
|
||||
// we just capture the final value when it arrives.
|
||||
let accumulatedCostUsd = 0;
|
||||
let tokensLogged = false;
|
||||
|
||||
function buildUsage(): AgentUsage | undefined {
|
||||
@@ -204,6 +210,7 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
|
||||
outputTokens: accumulatedTokens.output,
|
||||
cacheReadTokens: accumulatedTokens.cacheRead || undefined,
|
||||
cacheWriteTokens: accumulatedTokens.cacheWrite || undefined,
|
||||
costUsd: accumulatedCostUsd > 0 ? accumulatedCostUsd : undefined,
|
||||
}
|
||||
: undefined;
|
||||
}
|
||||
@@ -245,11 +252,15 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
|
||||
}
|
||||
}
|
||||
|
||||
// accumulate per-message usage if available
|
||||
// accumulate per-message usage if available. capture cache fields too
|
||||
// so the fallback token table (used when no final `result` event fires)
|
||||
// still reports the full breakdown instead of silently dropping cache.
|
||||
const msgUsage = event.message?.usage;
|
||||
if (msgUsage) {
|
||||
accumulatedTokens.input += msgUsage.input_tokens || 0;
|
||||
accumulatedTokens.output += msgUsage.output_tokens || 0;
|
||||
accumulatedTokens.cacheRead += msgUsage.cache_read_input_tokens || 0;
|
||||
accumulatedTokens.cacheWrite += msgUsage.cache_creation_input_tokens || 0;
|
||||
}
|
||||
},
|
||||
user: (event: ClaudeUserEvent) => {
|
||||
@@ -290,28 +301,35 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
|
||||
const numTurns = event.num_turns || 0;
|
||||
|
||||
if (subtype === "success") {
|
||||
// extract detailed usage from result event (most accurate source)
|
||||
// extract detailed usage from result event (most accurate source).
|
||||
// note: `input` here is non-cached input tokens only, matching the
|
||||
// semantics of OpenCode's step_finish.tokens.input — the logTokenTable
|
||||
// helper sums Input + Cache Read + Cache Write + Output into the Total
|
||||
// column so consumers get the real billable figure.
|
||||
const usage = event.usage;
|
||||
const inputTokens = usage?.input_tokens || 0;
|
||||
const cacheRead = usage?.cache_read_input_tokens || 0;
|
||||
const cacheWrite = usage?.cache_creation_input_tokens || 0;
|
||||
const outputTokens = usage?.output_tokens || 0;
|
||||
const totalInput = inputTokens + cacheRead + cacheWrite;
|
||||
// guard against NaN/Infinity from malformed CLI output poisoning the total
|
||||
const costUsd =
|
||||
typeof event.total_cost_usd === "number" && Number.isFinite(event.total_cost_usd)
|
||||
? event.total_cost_usd
|
||||
: 0;
|
||||
|
||||
accumulatedTokens = { input: inputTokens, output: outputTokens, cacheRead, cacheWrite };
|
||||
accumulatedCostUsd = costUsd;
|
||||
|
||||
log.info(`» ${params.label} result: subtype=${subtype}, turns=${numTurns}`);
|
||||
|
||||
if (!tokensLogged) {
|
||||
log.table([
|
||||
[
|
||||
{ data: "Input", header: true },
|
||||
{ data: "Cache Read", header: true },
|
||||
{ data: "Cache Write", header: true },
|
||||
{ data: "Output", header: true },
|
||||
],
|
||||
[String(totalInput), String(cacheRead), String(cacheWrite), String(outputTokens)],
|
||||
]);
|
||||
logTokenTable({
|
||||
input: inputTokens,
|
||||
cacheRead,
|
||||
cacheWrite,
|
||||
output: outputTokens,
|
||||
costUsd,
|
||||
});
|
||||
tokensLogged = true;
|
||||
}
|
||||
} else if (subtype === "error_max_turns") {
|
||||
@@ -432,16 +450,15 @@ async function runClaude(params: RunParams): Promise<ClaudeRunResult> {
|
||||
if (stderrContext) log.info(`» last stderr output:\n${stderrContext}`);
|
||||
}
|
||||
|
||||
if (!tokensLogged && (accumulatedTokens.input > 0 || accumulatedTokens.output > 0)) {
|
||||
const totalTokens = accumulatedTokens.input + accumulatedTokens.output;
|
||||
log.table([
|
||||
[
|
||||
{ data: "Input Tokens", header: true },
|
||||
{ data: "Output Tokens", header: true },
|
||||
{ data: "Total Tokens", header: true },
|
||||
],
|
||||
[String(accumulatedTokens.input), String(accumulatedTokens.output), String(totalTokens)],
|
||||
]);
|
||||
if (
|
||||
!tokensLogged &&
|
||||
(accumulatedTokens.input > 0 ||
|
||||
accumulatedTokens.output > 0 ||
|
||||
accumulatedTokens.cacheRead > 0 ||
|
||||
accumulatedTokens.cacheWrite > 0)
|
||||
) {
|
||||
logTokenTable({ ...accumulatedTokens, costUsd: accumulatedCostUsd });
|
||||
tokensLogged = true;
|
||||
}
|
||||
|
||||
const usage = buildUsage();
|
||||
@@ -637,6 +654,10 @@ export const claude = agent({
|
||||
...runParams,
|
||||
args: [...baseArgs, "-p", ctx.instructions.full],
|
||||
});
|
||||
// usage needs to aggregate across the initial run + every commit retry.
|
||||
// each runClaude() returns only its own iteration's usage, so without
|
||||
// merging the caller sees only the final retry's slice and undercounts.
|
||||
let aggregatedUsage = result.usage;
|
||||
|
||||
// post-run: if the working tree is dirty, resume the session and ask the agent to commit
|
||||
for (let attempt = 0; attempt < MAX_COMMIT_RETRIES; attempt++) {
|
||||
@@ -655,8 +676,9 @@ export const claude = agent({
|
||||
result.sessionId,
|
||||
],
|
||||
});
|
||||
aggregatedUsage = mergeAgentUsage(aggregatedUsage, result.usage);
|
||||
}
|
||||
|
||||
return result;
|
||||
return { ...result, usage: aggregatedUsage };
|
||||
},
|
||||
});
|
||||
|
||||
+43
-23
@@ -33,8 +33,10 @@ import {
|
||||
agent,
|
||||
buildCommitPrompt,
|
||||
getGitStatus,
|
||||
logTokenTable,
|
||||
MAX_COMMIT_RETRIES,
|
||||
MAX_STDERR_LINES,
|
||||
mergeAgentUsage,
|
||||
} from "./shared.ts";
|
||||
|
||||
async function installOpencodeCli(): Promise<string> {
|
||||
@@ -268,6 +270,10 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
|
||||
|
||||
let finalOutput = "";
|
||||
let accumulatedTokens = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
|
||||
// per-step `part.cost` sums across the whole session. sourced from models.dev
|
||||
// inside opencode — present for every supported provider (Anthropic, OpenAI,
|
||||
// Google, xAI, DeepSeek, Moonshot, OpenRouter sub-providers, etc.).
|
||||
let accumulatedCostUsd = 0;
|
||||
let tokensLogged = false;
|
||||
const toolCallTimings = new Map<string, number>();
|
||||
let currentStepId: string | null = null;
|
||||
@@ -284,6 +290,7 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
|
||||
outputTokens: accumulatedTokens.output,
|
||||
cacheReadTokens: accumulatedTokens.cacheRead || undefined,
|
||||
cacheWriteTokens: accumulatedTokens.cacheWrite || undefined,
|
||||
costUsd: accumulatedCostUsd > 0 ? accumulatedCostUsd : undefined,
|
||||
}
|
||||
: undefined;
|
||||
}
|
||||
@@ -296,6 +303,7 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
|
||||
log.debug(`» ${params.label} init event (full): ${JSON.stringify(event)}`);
|
||||
finalOutput = "";
|
||||
accumulatedTokens = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
|
||||
accumulatedCostUsd = 0;
|
||||
tokensLogged = false;
|
||||
},
|
||||
message: (event: OpenCodeMessageEvent) => {
|
||||
@@ -340,6 +348,15 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
|
||||
accumulatedTokens.cacheRead += eventTokens.cache?.read || 0;
|
||||
accumulatedTokens.cacheWrite += eventTokens.cache?.write || 0;
|
||||
}
|
||||
// step_finish.part.cost is a per-step delta (not a running total) —
|
||||
// OpenCode emits varying per-event values that sum to the session cost.
|
||||
// verified empirically across Anthropic, OpenAI, Gemini, xAI, DeepSeek,
|
||||
// Moonshot, and OpenRouter (see pullfrog-baseline/opencode-*.log).
|
||||
// guard against NaN/Infinity — a single poison value would make the
|
||||
// running total un-recoverable for the rest of the session.
|
||||
if (typeof event.part?.cost === "number" && Number.isFinite(event.part.cost)) {
|
||||
accumulatedCostUsd += event.part.cost;
|
||||
}
|
||||
if (currentStepId === stepId) {
|
||||
currentStepId = null;
|
||||
currentStepType = null;
|
||||
@@ -429,20 +446,19 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
|
||||
if (event.status === "error") {
|
||||
log.info(`» ${params.label} failed: ${JSON.stringify(event)}`);
|
||||
} else {
|
||||
const inputTokens = event.stats?.input_tokens || accumulatedTokens.input || 0;
|
||||
const outputTokens = event.stats?.output_tokens || accumulatedTokens.output || 0;
|
||||
const totalTokens = event.stats?.total_tokens || inputTokens + outputTokens;
|
||||
// the final `result` event only carries input_tokens/output_tokens and
|
||||
// no cache breakdown — accumulatedTokens (summed across step_finish
|
||||
// events) is strictly more accurate, so we prefer it unconditionally.
|
||||
log.info(`» run complete: tool_calls=${toolCalls}, duration=${duration}ms`);
|
||||
|
||||
if ((inputTokens > 0 || outputTokens > 0) && !tokensLogged) {
|
||||
log.table([
|
||||
[
|
||||
{ data: "Input Tokens", header: true },
|
||||
{ data: "Output Tokens", header: true },
|
||||
{ data: "Total Tokens", header: true },
|
||||
],
|
||||
[String(inputTokens), String(outputTokens), String(totalTokens)],
|
||||
]);
|
||||
if (
|
||||
(accumulatedTokens.input > 0 ||
|
||||
accumulatedTokens.output > 0 ||
|
||||
accumulatedTokens.cacheRead > 0 ||
|
||||
accumulatedTokens.cacheWrite > 0) &&
|
||||
!tokensLogged
|
||||
) {
|
||||
logTokenTable({ ...accumulatedTokens, costUsd: accumulatedCostUsd });
|
||||
tokensLogged = true;
|
||||
}
|
||||
}
|
||||
@@ -555,16 +571,15 @@ async function runOpenCode(params: RunParams): Promise<AgentResult> {
|
||||
if (stderrContext) log.info(`» last stderr output:\n${stderrContext}`);
|
||||
}
|
||||
|
||||
if (!tokensLogged && (accumulatedTokens.input > 0 || accumulatedTokens.output > 0)) {
|
||||
const totalTokens = accumulatedTokens.input + accumulatedTokens.output;
|
||||
log.table([
|
||||
[
|
||||
{ data: "Input Tokens", header: true },
|
||||
{ data: "Output Tokens", header: true },
|
||||
{ data: "Total Tokens", header: true },
|
||||
],
|
||||
[String(accumulatedTokens.input), String(accumulatedTokens.output), String(totalTokens)],
|
||||
]);
|
||||
if (
|
||||
!tokensLogged &&
|
||||
(accumulatedTokens.input > 0 ||
|
||||
accumulatedTokens.output > 0 ||
|
||||
accumulatedTokens.cacheRead > 0 ||
|
||||
accumulatedTokens.cacheWrite > 0)
|
||||
) {
|
||||
logTokenTable({ ...accumulatedTokens, costUsd: accumulatedCostUsd });
|
||||
tokensLogged = true;
|
||||
}
|
||||
|
||||
const usage = buildUsage();
|
||||
@@ -688,6 +703,10 @@ export const opencode = agent({
|
||||
...runParams,
|
||||
args: [...baseArgs, ctx.instructions.full],
|
||||
});
|
||||
// usage needs to aggregate across the initial run + every commit retry.
|
||||
// each runOpenCode() returns only its own iteration's usage, so without
|
||||
// merging the caller sees only the final retry's slice and undercounts.
|
||||
let aggregatedUsage = result.usage;
|
||||
|
||||
// post-run: if the working tree is dirty, continue the session and ask the agent to commit
|
||||
for (let attempt = 0; attempt < MAX_COMMIT_RETRIES; attempt++) {
|
||||
@@ -700,8 +719,9 @@ export const opencode = agent({
|
||||
...runParams,
|
||||
args: [...baseArgs, "--continue", buildCommitPrompt("opencode", status)],
|
||||
});
|
||||
aggregatedUsage = mergeAgentUsage(aggregatedUsage, result.usage);
|
||||
}
|
||||
|
||||
return result;
|
||||
return { ...result, usage: aggregatedUsage };
|
||||
},
|
||||
});
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { type AgentUsage, mergeAgentUsage } from "./shared.ts";
|
||||
|
||||
const entry = (overrides: Partial<AgentUsage>): AgentUsage => ({
|
||||
agent: "pullfrog",
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe("mergeAgentUsage", () => {
|
||||
it("returns undefined when both sides are undefined", () => {
|
||||
expect(mergeAgentUsage(undefined, undefined)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("returns a copy of b when a is undefined", () => {
|
||||
const b = entry({ inputTokens: 10 });
|
||||
expect(mergeAgentUsage(undefined, b)).toEqual(b);
|
||||
});
|
||||
|
||||
it("returns a copy of a when b is undefined", () => {
|
||||
const a = entry({ inputTokens: 10 });
|
||||
expect(mergeAgentUsage(a, undefined)).toEqual(a);
|
||||
});
|
||||
|
||||
it("sums inputTokens and outputTokens unconditionally", () => {
|
||||
const merged = mergeAgentUsage(
|
||||
entry({ inputTokens: 10, outputTokens: 5 }),
|
||||
entry({ inputTokens: 20, outputTokens: 7 })
|
||||
);
|
||||
expect(merged?.inputTokens).toBe(30);
|
||||
expect(merged?.outputTokens).toBe(12);
|
||||
});
|
||||
|
||||
it("keeps cache/cost fields undefined when both sides lack them", () => {
|
||||
// this matters so downstream aggregateUsage doesn't persist spurious 0s into the DB
|
||||
const merged = mergeAgentUsage(entry({ inputTokens: 10 }), entry({ inputTokens: 20 }));
|
||||
expect(merged?.cacheReadTokens).toBeUndefined();
|
||||
expect(merged?.cacheWriteTokens).toBeUndefined();
|
||||
expect(merged?.costUsd).toBeUndefined();
|
||||
});
|
||||
|
||||
it("sums cache and cost fields when either side reports them", () => {
|
||||
const merged = mergeAgentUsage(
|
||||
entry({ inputTokens: 10, cacheReadTokens: 100, costUsd: 0.01 }),
|
||||
entry({ inputTokens: 20, cacheWriteTokens: 50, costUsd: 0.02 })
|
||||
);
|
||||
expect(merged?.cacheReadTokens).toBe(100);
|
||||
expect(merged?.cacheWriteTokens).toBe(50);
|
||||
expect(merged?.costUsd).toBeCloseTo(0.03, 10);
|
||||
});
|
||||
|
||||
it("preserves the agent id of the left operand", () => {
|
||||
// the aggregator is called inside a single agent's run() — the agent label
|
||||
// is a fixed property of the harness, not something that can flip mid-run
|
||||
const merged = mergeAgentUsage(
|
||||
entry({ agent: "claude", inputTokens: 10 }),
|
||||
entry({ agent: "something-else", inputTokens: 20 })
|
||||
);
|
||||
expect(merged?.agent).toBe("claude");
|
||||
});
|
||||
|
||||
it("returns a fresh object rather than the input reference", () => {
|
||||
// callers treat AgentUsage as immutable; returning the input itself would
|
||||
// leak that invariant. mutating the returned value must not affect inputs.
|
||||
const a = entry({ inputTokens: 10 });
|
||||
const mergedWithUndef = mergeAgentUsage(a, undefined);
|
||||
expect(mergedWithUndef).not.toBe(a);
|
||||
expect(mergedWithUndef).toEqual(a);
|
||||
|
||||
const b = entry({ inputTokens: 20 });
|
||||
const mergedFromUndef = mergeAgentUsage(undefined, b);
|
||||
expect(mergedFromUndef).not.toBe(b);
|
||||
expect(mergedFromUndef).toEqual(b);
|
||||
});
|
||||
});
|
||||
+105
-1
@@ -34,10 +34,21 @@ export function buildCommitPrompt(_agentId: AgentId, status: string): string {
|
||||
}
|
||||
|
||||
/**
|
||||
* token/cost usage data from a single agent run
|
||||
* token/cost usage data from a single agent run.
|
||||
*
|
||||
* NOTE on semantics: `inputTokens` here is the *total* billable input for the
|
||||
* run — non-cached input + cache read + cache write — matching the per-agent
|
||||
* SDK conventions. This is what gets persisted to `WorkflowRun.inputTokens`.
|
||||
*
|
||||
* The stdout token table and markdown step summary display a different "Input"
|
||||
* column that shows only the non-cached portion (derivable as
|
||||
* `inputTokens - cacheReadTokens - cacheWriteTokens`) so humans can see the
|
||||
* cache hit ratio at a glance. Dashboards that query `WorkflowRun.inputTokens`
|
||||
* directly are seeing the full total, not the log column.
|
||||
*/
|
||||
export interface AgentUsage {
|
||||
agent: string;
|
||||
/** full billable input: non-cached + cache read + cache write */
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
cacheReadTokens?: number | undefined;
|
||||
@@ -95,3 +106,96 @@ export const agent = (input: Agent): Agent => {
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
/** format a USD cost to 4 decimal places, always showing the leading zero */
|
||||
export function formatCostUsd(costUsd: number): string {
|
||||
return costUsd.toFixed(4);
|
||||
}
|
||||
|
||||
/**
|
||||
* merge two AgentUsage snapshots into one running total.
|
||||
*
|
||||
* both agent harnesses invoke their runner multiple times per `run()` when the
|
||||
* post-run dirty-tree loop kicks in (MAX_COMMIT_RETRIES). each invocation
|
||||
* produces its own AgentUsage; we sum them so downstream callers (usage
|
||||
* summary, WorkflowRun persistence) see the whole session — not just the
|
||||
* final retry's slice.
|
||||
*
|
||||
* returns `undefined` when both sides are empty so callers can short-circuit
|
||||
* without a special case. zero-valued cache / cost fields are dropped to
|
||||
* `undefined` for symmetry with each harness's `buildUsage`.
|
||||
*/
|
||||
export function mergeAgentUsage(
|
||||
a: AgentUsage | undefined,
|
||||
b: AgentUsage | undefined
|
||||
): AgentUsage | undefined {
|
||||
// always return a fresh object — callers treat AgentUsage as immutable, and
|
||||
// returning `a` / `b` directly would leak that invariant to future callers
|
||||
if (!a && !b) return undefined;
|
||||
if (!a) return { ...(b as AgentUsage) };
|
||||
if (!b) return { ...a };
|
||||
const cacheRead = (a.cacheReadTokens ?? 0) + (b.cacheReadTokens ?? 0);
|
||||
const cacheWrite = (a.cacheWriteTokens ?? 0) + (b.cacheWriteTokens ?? 0);
|
||||
const cost = (a.costUsd ?? 0) + (b.costUsd ?? 0);
|
||||
return {
|
||||
agent: a.agent,
|
||||
inputTokens: a.inputTokens + b.inputTokens,
|
||||
outputTokens: a.outputTokens + b.outputTokens,
|
||||
cacheReadTokens: cacheRead > 0 ? cacheRead : undefined,
|
||||
cacheWriteTokens: cacheWrite > 0 ? cacheWrite : undefined,
|
||||
costUsd: cost > 0 ? cost : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* unified per-run token table used by every agent harness.
|
||||
*
|
||||
* columns are kept stable across agents and models so downstream log parsers
|
||||
* (scripts/token-usage.ts, cost dashboards) only have to understand one format:
|
||||
*
|
||||
* Input non-cached input tokens sent this run
|
||||
* Cache Read input tokens served from prompt cache (Anthropic, etc.)
|
||||
* Cache Write input tokens written to prompt cache this run
|
||||
* Output assistant output tokens
|
||||
* Total sum of the four columns — the real billable quantity
|
||||
* Cost ($) USD cost reported by the provider (only rendered when known)
|
||||
*
|
||||
* models that don't report prompt caching leave Cache Read / Write at 0.
|
||||
* OpenCode emits per-step `part.cost` sourced from models.dev (works across
|
||||
* Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, OpenRouter, etc.);
|
||||
* Claude CLI emits `total_cost_usd` on its final `result` event. pass the
|
||||
* accumulated value via `costUsd` to render the Cost column.
|
||||
*/
|
||||
export function logTokenTable(t: {
|
||||
input: number;
|
||||
cacheRead: number;
|
||||
cacheWrite: number;
|
||||
output: number;
|
||||
costUsd?: number | undefined;
|
||||
}): void {
|
||||
const total = t.input + t.cacheRead + t.cacheWrite + t.output;
|
||||
// narrow costUsd to a concrete number so the render path doesn't need a cast
|
||||
const costUsd = typeof t.costUsd === "number" && t.costUsd > 0 ? t.costUsd : undefined;
|
||||
|
||||
const headerRow: Array<{ data: string; header: true }> = [
|
||||
{ data: "Input", header: true },
|
||||
{ data: "Cache Read", header: true },
|
||||
{ data: "Cache Write", header: true },
|
||||
{ data: "Output", header: true },
|
||||
{ data: "Total", header: true },
|
||||
];
|
||||
const dataRow: string[] = [
|
||||
String(t.input),
|
||||
String(t.cacheRead),
|
||||
String(t.cacheWrite),
|
||||
String(t.output),
|
||||
String(total),
|
||||
];
|
||||
|
||||
if (costUsd !== undefined) {
|
||||
headerRow.push({ data: "Cost ($)", header: true });
|
||||
dataRow.push(formatCostUsd(costUsd));
|
||||
}
|
||||
|
||||
log.table([headerRow, dataRow]);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user