f662b1a0c8
* unify per-run token + cost accounting across agents every agent harness now logs the same 5-column (or 6 with cost) table and populates the same AgentUsage contract, regardless of agent or upstream provider. previously OpenCode and the Claude fallback path emitted a 3-col table whose "Input Tokens" was actually only the non-cached delta, silently dropping cache read/write — real runs were being reported at ~0.4% of their true input (e.g. one baseline showed Input=30 while step_finish events summed to cache_read=724,753). changes: - add logTokenTable helper in action/agents/shared.ts with stable columns: Input | Cache Read | Cache Write | Output | Total | Cost ($). cost column renders only when a value is known. - action/agents/opencode.ts: accumulate step_finish.part.tokens AND step_finish.part.cost (sourced from models.dev inside opencode — confirmed working across Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, and OpenRouter). drop the event.stats.total_tokens fallback since that payload has no cache breakdown. - action/agents/claude.ts: success-path now treats input_tokens as the non-cached field (matching OpenCode semantics), carries cache_read_input_tokens / cache_creation_input_tokens separately, and captures total_cost_usd from the final result event. the per-message fallback accumulator now captures cache fields too so it's no longer lossy when the result event never fires. - formatUsageSummary gains a Cost ($) column that matches the stdout table row-for-row; missing values render as "—". - scripts/token-usage.ts parses all three historical formats (new 5-col, legacy 4-col Claude success, legacy 3-col lossy) and explicitly flags the lossy runs instead of averaging misleading values. validation (pnpm play --local, identical "say hello" prompt): agent+model Input CacheR CacheW Output Total Cost OpenCode + Anthropic Sonnet 4.6 4 41,177 20,735 129 62,045 $0.0921 Claude CLI + Anthropic Sonnet 4.6 9 80,133 11,611 389 92,142 $0.0766 OpenCode + OpenAI codex-mini 10,893 46,976 0 606 58,475 $0.0059 OpenCode + Google Gemini 3 Flash — — — — — $0.0114 OpenCode + xAI Grok 4 Fast — — — — — $0.0035 OpenCode + DeepSeek Chat 18,854 0 0 1 18,855 $0.0053 OpenCode + Moonshot Kimi K2.5 — — — — — $0.0106 OpenCode + OpenRouter→Anthropic — — — — — $0.0617 OpenCode + OpenRouter→OpenAI — — — — — $0.0038 * isolate play.ts from developer gitconfig play.ts is a CI-emulator but inherits the developer's user- and system-scope gitconfig. a common local convenience — url."git@github.com:".insteadOf "https://github.com/" to force SSH auth — gets applied at read time on every git call inside the temp repo, causing `git remote get-url --push origin` to return an SSH URL instead of the stored HTTPS one. pullfrog_push_branch's validatePushDestination (correctly) treats that as tampering and blocks the push. the agent then burns the full MAX_COMMIT_RETRIES budget trying workarounds that can't beat a user-scope insteadOf rule, turning a trivial "say hello" run into a 1.35M-token session. point GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null inside run() so the play process and its spawned agent see the same empty gitconfig that a real CI runner would. CI has no rewrites, so this is a no-op there; dev machines get CI-identical git state. SSH client config (~/.ssh/config and keys) is separate from gitconfig and is unaffected, so setupTestRepo's SSH clone still works locally. setupGit only writes --local scope, so nothing downstream depends on user-scope values. verification: with the scratch repo cleaned up and this isolation in place, OpenCode + Anthropic on the same "say hello" prompt goes from 1,349,654 tokens / $2.00+ to 62,045 tokens / $0.0921 — no retry loop, no push blocks. * persist aggregated token + cost usage to WorkflowRun AgentUsage has been memory-only — rendered into the GitHub step summary and then discarded when the runner tears down. that made questions like "avg cost per customer per day" require log-spelunking. persist it: - add Int? columns for inputTokens / outputTokens / cacheReadTokens / cacheWriteTokens and a Decimal? costUsd column on workflow_runs. Int4's 2.1B ceiling is ~200x larger than any realistic run so BigInt would be overkill. costUsd uses the same default Decimal precision as existing money columns (accounts.usageUsd, proxy_keys.hwmUsage). - extend PATCH /api/workflow-run/[runId] to accept the new numeric fields alongside the existing artifact strings. per-field type validation ensures the allowlist stays scalar-safe and rejects negative / non-finite values. - generalize patchWorkflowRunFields in the action so it accepts a mixed string/number payload, and add an aggregateUsage(entries) helper that sums per-agent AgentUsage records into a single patch. - call the reporter from main.ts's outer finally block, gated on toolContext. this is the shared cleanup path that every agent implementation flows through — claude.ts, opencode.ts, and any future harness all push their AgentUsage into toolState.usageEntries via the same line 468, so one finally-block call covers them all. running in finally also means partial usage gets persisted even when the agent errored out mid-run. * anneal token + cost accounting follow-up polish from a review pass: - aggregate usage across commit-retry iterations inside each agent harness. previously runClaude / runOpenCode returned only the final retry's usage, so any run that hit the dirty-tree retry loop under-counted tokens and cost in both the stdout table and the WorkflowRun row. added a shared mergeAgentUsage helper in agents/shared.ts; both harnesses now fold each iteration's usage into a running total and return the sum. - scripts/token-usage.ts now handles the unified format with or without the Cost ($) column. previously the int-only number regex rejected decimals and the 5-cell length check rejected 6-cell rows, so logs from post-cost-tracking runs fell through to "no token table". the parser now accepts both 5- and 6-cell unified rows, splits int vs decimal cells, and averages reported Cost alongside the tokens. - PATCH /api/workflow-run/[runId] now rejects INT field values above INT4_MAX (2_147_483_647) so a malformed payload gets a clean 400 instead of propagating a Prisma error. also defends against a compromised runner sending a deliberately huge value. - clarifying comments: opencode.ts documents that step_finish.part.cost is a per-step delta (empirically verified), main.ts explains that toolState.usageEntries already carries merged per-retry usage so aggregateUsage just sums entries (one per agent.run()). - tests for aggregateUsage and mergeAgentUsage — 12 new cases covering empty / partial / multi-agent inputs and the "keep undefined" semantic that prevents spurious zeros from being persisted. - drop `as number` cast in logTokenTable — narrow via const instead. * anneal: clamp INT overflow + guarantee mergeAgentUsage immutability second review pass surfaced two defensive gaps: - a single token field exceeding INT4_MAX would pass the client but be rejected by the server's per-field validator, writing a partial row with some NULLs where sums belonged. clamp in aggregateUsage so the wire payload is always self-consistent across all numeric columns, with a loud warning so the clamp doesn't silently swallow weirdness. - mergeAgentUsage's single-sided branches returned the input reference. callers treat AgentUsage as immutable but future callers might not; always return a fresh shallow copy instead. two new tests guarantee the no-mutation-leak property. no behavior change in the happy path — INT4_MAX is ~200x the largest realistic per-run token count. * anneal: resilient usage persistence + cross-platform null device third review pass surfaced three small issues: - main.ts finally block: writeGitHubUsageSummaryToFile throwing would skip the WorkflowRun usage PATCH. both are independent best-effort cleanup tasks — wrap the former in catch so a filesystem failure doesn't block DB persistence. - AgentUsage.inputTokens had no jsdoc explaining that it's the full billable input (cached + non-cached). the same word "Input" means "non-cached only" in the stdout/markdown tables (derived by subtraction). document the semantic so dashboards querying WorkflowRun.inputTokens don't misinterpret it. - play.ts gitconfig isolation was hard-coded to "/dev/null" which doesn't exist on Windows. use `os.devNull` for cross-platform parity (resolves to `\\.\nul` on win32). the project is Linux-only in CI so this only helps local Windows contributors, but it's a zero-cost swap. also updated the finally-block caveat comment: usage is only pushed to toolState.usageEntries when agent.run() returns an AgentResult, not when the timeout race rejects — so timed-out runs don't persist partial usage. documented instead of trying to thread state through Promise.race. * anneal: NaN-guard cost accumulators + clarify inputTokens docs final polish from review round 4: - guard both cost accumulators (opencode step_finish.part.cost and claude result.total_cost_usd) with Number.isFinite. `typeof x === "number"` accepts NaN, and one NaN `+=` would poison the running total for the whole session. - reword prisma schema comment on WorkflowRun usage fields to call out that cacheReadTokens / cacheWriteTokens are SUB-totals within inputTokens (not additional tokens on top). prevents future dashboards from double-counting by ~2x when summing "total tokens used".
190 lines
6.5 KiB
TypeScript
190 lines
6.5 KiB
TypeScript
import { execSync } from "node:child_process";
|
|
import { mkdtemp } from "node:fs/promises";
|
|
import { devNull, tmpdir } from "node:os";
|
|
import { dirname, join, resolve } from "node:path";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import arg from "arg";
|
|
import { config } from "dotenv";
|
|
import type { AgentResult } from "./agents/shared.ts";
|
|
import { type Inputs, main } from "./main.ts";
|
|
import { defineFixture } from "./test/utils.ts";
|
|
import { log } from "./utils/cli.ts";
|
|
import { runInDocker } from "./utils/docker.ts";
|
|
import { ensureGitHubToken } from "./utils/github.ts";
|
|
import { isInsideDocker } from "./utils/globals.ts";
|
|
import { runPostCleanup } from "./utils/postCleanup.ts";
|
|
import { setupTestRepo } from "./utils/setup.ts";
|
|
|
|
/**
|
|
* default play fixture for ad-hoc testing.
|
|
* change this freely without affecting any tests.
|
|
*/
|
|
export const playFixture = defineFixture(
|
|
{
|
|
prompt: `List every MCP tool you have access to. Call set_output with a JSON array of all tool names you can see.`,
|
|
},
|
|
{ localOnly: true }
|
|
);
|
|
|
|
const __filename = fileURLToPath(import.meta.url);
|
|
const __dirname = dirname(__filename);
|
|
|
|
// load action's .env file in case it exists for local dev
|
|
config();
|
|
// also load .env from repo root (for monorepo structure)
|
|
config({ path: join(__dirname, "..", ".env") });
|
|
|
|
export async function run(inputsOrPrompt: Inputs | string): Promise<AgentResult> {
|
|
await ensureGitHubToken();
|
|
|
|
// play.ts is a CI-emulator — isolate it from the developer's user- and
|
|
// system-scope gitconfig so checks like `validatePushDestination` see the
|
|
// raw stored remote URL instead of values mutated by `url.*.insteadOf`
|
|
// rewrites (a common SSH-auth convenience on dev boxes). CI runners have
|
|
// empty gitconfigs so this is a no-op there; locally it makes `pnpm play`
|
|
// and real runs produce identical git state. `os.devNull` canonicalizes
|
|
// the null device across Unix (`/dev/null`) and Windows (`\\.\nul`).
|
|
process.env.GIT_CONFIG_GLOBAL = devNull;
|
|
process.env.GIT_CONFIG_SYSTEM = devNull;
|
|
|
|
// create unique temp directory path in OS temp location for parallel execution
|
|
// use a parent dir from mkdtemp, then clone into a 'repo' subdirectory
|
|
const tempParent = await mkdtemp(join(tmpdir(), "pullfrog-play-"));
|
|
const tempDir = join(tempParent, "repo");
|
|
const originalCwd = process.cwd();
|
|
|
|
try {
|
|
setupTestRepo({ tempDir });
|
|
process.chdir(tempDir);
|
|
|
|
// run repo setup commands if provided (for pre-planting test state like symlinks).
|
|
// this runs AFTER clone but BEFORE the agent, simulating pre-existing repo content.
|
|
if (process.env.PULLFROG_TEST_REPO_SETUP) {
|
|
log.info("» running repo setup commands...");
|
|
execSync(process.env.PULLFROG_TEST_REPO_SETUP, { cwd: tempDir, stdio: "pipe" });
|
|
}
|
|
|
|
// set GITHUB_WORKSPACE to tempDir so main() doesn't try to chdir to the CI checkout path
|
|
process.env.GITHUB_WORKSPACE = tempDir;
|
|
|
|
// allow passing full Inputs object or just a prompt string
|
|
const inputs: Inputs =
|
|
typeof inputsOrPrompt === "string" ? { prompt: inputsOrPrompt } : inputsOrPrompt;
|
|
|
|
// set INPUT_* env vars for @actions/core.getInput()
|
|
for (const [key, value] of Object.entries(inputs)) {
|
|
if (value !== undefined && value !== null) {
|
|
process.env[`INPUT_${key.toUpperCase()}`] = String(value);
|
|
}
|
|
}
|
|
|
|
// wrap main() so post cleanup runs even on failure (mirrors action.yml post-if: "failure() || cancelled()")
|
|
let result: AgentResult;
|
|
try {
|
|
result = await main();
|
|
} finally {
|
|
await runPostCleanup();
|
|
}
|
|
|
|
process.chdir(originalCwd);
|
|
|
|
if (result.success) {
|
|
log.success("Action completed successfully");
|
|
return { success: true, output: result.output || undefined, error: undefined };
|
|
} else {
|
|
log.error(`Action failed: ${result.error || "Unknown error"}`);
|
|
return { success: false, error: result.error || undefined, output: undefined };
|
|
}
|
|
} catch (err) {
|
|
const errorMessage = (err as Error).message;
|
|
log.error(`Error: ${errorMessage}`);
|
|
return { success: false, error: errorMessage, output: undefined };
|
|
} finally {
|
|
// cleanup temp directory - use sudo rm because sandbox isolation may create
|
|
// files with different ownership that rmSync can't delete
|
|
process.chdir(originalCwd);
|
|
try {
|
|
execSync(`sudo rm -rf "${tempParent}"`, { stdio: "ignore" });
|
|
} catch {
|
|
// ignore - cleanup failure is not critical
|
|
}
|
|
}
|
|
}
|
|
|
|
const isDirectExecution = process.argv[1]
|
|
? import.meta.url === pathToFileURL(resolve(process.argv[1])).href
|
|
: false;
|
|
|
|
if (isDirectExecution) {
|
|
const args = arg({
|
|
"--help": Boolean,
|
|
"--raw": String,
|
|
"--local": Boolean,
|
|
"-h": "--help",
|
|
"-l": "--local",
|
|
});
|
|
|
|
if (args["--help"]) {
|
|
log.info(`
|
|
Usage: node play.ts [options]
|
|
|
|
Test the Pullfrog action with the inline playFixture.
|
|
|
|
Options:
|
|
--raw [input] Use raw string as prompt, or JSON object as full fixture
|
|
--local, -l Run locally (default: runs in Docker)
|
|
-h, --help Show this help message
|
|
|
|
Environment:
|
|
PLAY_LOCAL=1 Same as --local
|
|
|
|
Examples:
|
|
node play.ts # Run inline playFixture
|
|
node play.ts --raw "Hello world" # Use raw string as prompt
|
|
node play.ts --raw '{"prompt":"Hello","timeout":"5s"}' # Use JSON fixture
|
|
`);
|
|
process.exit(0);
|
|
}
|
|
|
|
// default: run in Docker (unless --local, PLAY_LOCAL=1, or already inside Docker)
|
|
const useLocal = args["--local"] || process.env.PLAY_LOCAL === "1" || isInsideDocker;
|
|
|
|
if (!useLocal) {
|
|
const passArgs = process.argv
|
|
.slice(2)
|
|
.map((a) => `'${a.replace(/'/g, "'\\''")}'`)
|
|
.join(" ");
|
|
const nodeCmd = `node play.ts ${passArgs}`;
|
|
|
|
const volumeName = "pullfrog-action-node-modules";
|
|
|
|
const result = runInDocker({
|
|
actionDir: __dirname,
|
|
args: process.argv.slice(2),
|
|
nodeCmd,
|
|
volumeName,
|
|
envFilterMode: "passthrough",
|
|
onStart: () => log.info("» running in Docker container..."),
|
|
});
|
|
|
|
process.exit(result.status ?? 1);
|
|
}
|
|
|
|
if (args["--raw"]) {
|
|
const raw = args["--raw"];
|
|
// try to parse as JSON, otherwise treat as prompt string
|
|
let input: Inputs | string = raw;
|
|
try {
|
|
input = JSON.parse(raw) as Inputs;
|
|
} catch {
|
|
// not valid JSON, use as prompt string
|
|
}
|
|
const result = await run(input);
|
|
process.exit(result.success ? 0 : 1);
|
|
}
|
|
|
|
// no args - use inline playFixture
|
|
const result = await run(playFixture);
|
|
process.exit(result.success ? 0 : 1);
|
|
}
|