* docker testing rewrite: bake the image, drop the allowlist, kill the quoting
- new `pnpm gha <script>` wrapper. one entry point for running any node
script in the GHA-like container; replaces the runtime apt-get +
useradd + chown ceremony in `action/utils/docker.ts`.
- `action/Dockerfile` bakes ubuntu:24.04 + node 24 + gh + jq + sudo +
testuser at uid 1000. `action/docker-entrypoint.sh` remaps to the host
uid/gid and `exec`s the requested command — no `bash -c` nesting, no
`escapeForDoubleQuotes`.
- env passthrough: full `process.env` (+ `.env` via dotenv) flows through
`--env-file`, multi-line values via `-e` fallback. drops
`EnvFilterMode` / `testEnvAllowList`.
- image rebuild is content-hash gated on Dockerfile + entrypoint; volume
is versioned by hash so a stale `node_modules` cache from an old image
can't poison a new one.
- `action/play.ts` slimmed to a CLI; `run()` extracted to
`action/utils/runFixture.ts`. drops the `--local` / `PLAY_LOCAL` dual
mode in favor of explicit `play:local` / `runtest:local` scripts.
- `action/test/run.ts` no longer self-relaunches into docker — that's
`gha`'s job now.
- `action/test/coverage.ts` `ALWAYS_RUN_ALL` updated to track the new
files.
- `wiki/docker.md` rewritten (243 → 105 lines). `wiki/action-tests.md`,
`wiki/billing.md`, `wiki/adversarial.md`, `README.md`, `AGENTS.md` all
updated to drop `--local` / `PLAY_LOCAL` references.
verified end-to-end: `pnpm play` runs the default fixture against
pullfrog/scratch, exit 0; `sudo unshare --pid` still works inside the
container; `pnpm runtest` boots through the wrapper.
* gha: address review feedback + 3 related issues found locally
review-flagged:
- bare `pnpm gha --build` now builds the image and exits 0 (was
printing help and exiting 1 — docs claimed it was a valid standalone)
- `initVolumeOwnership` skipped when the named volume already exists;
saves the ~240ms `docker run … chown` on every warm invocation
- `GIT_SSH_COMMAND` gate widened to any `id_*` private key (was hard-
coded to `id_rsa`, leaving ed25519-only linux contributors with the
default ssh config). dropped `-i` so ssh picks whichever key exists
- new `action/.dockerignore` — partial mitigation noted: BuildKit
(default since docker 23) only sends files referenced by the
Dockerfile (~42B in practice), so the perf concern is mostly
hypothetical. file is still worth keeping for `DOCKER_BUILDKIT=0`
fallback and as documented intent for future `COPY . .` additions
related issues found while validating locally:
- `parseArgs` now stops flag-parsing at the first positional (or
literal `--`); `pnpm gha test/run.ts --build` previously
intercepted `--build` as a gha flag instead of forwarding to
`test/run.ts`
- new `pnpm gha --clean` command prunes orphan `pullfrog-gha:*`
images and `pullfrog-gha-node-modules-*` volumes whose hash
doesn't match the current Dockerfile (each Dockerfile/entrypoint
edit creates a fresh hash and orphans the prior pair, ~600MB +
~200MB each — without a cleaner they accumulate silently)
- `--shell` without a TTY now fails fast with an actionable message
before docker is invoked, instead of producing the confusing
`the input device is not a TTY` from docker run
wiki updated: documents `--clean`, the parseArgs passthrough rule,
and a new "Reclaiming disk" section.
* gha: fidelity, flexibility, and signal-safety improvements
investigated local fidelity vs the real GHA ubuntu-24.04 runner and
addressed the gaps that have actually bitten contributors or could.
fidelity (image now matches GHA closer):
- bake build-essential, wget, xz-utils, file alongside the existing
toolset. gh, jq, git, python3, sudo, ssh, build-essential, wget,
xz, file, unzip, curl all present. native module builds (node-gyp,
any package missing arm64 prebuilts) now work; common agent shell
calls don't hit ENOENT
- `host.docker.internal:host-gateway` flag wires the host into the
container's DNS on linux (macOS Docker Desktop bakes it in). lets
scripts that hit a local dev server use `API_URL=http://host.docker.
internal:3100` and work identically on both platforms
- `--init` makes tini PID 1, fixing signal forwarding during the
pre-exec warmup window (Ctrl-C was previously taking up to 10s to
tear down because bash-as-PID-1 swallowed the signal)
- pnpm version is correctly pinned via the workspace's
`packageManager` field — corepack resolves it at install time;
verified via the new `--doctor` command
flexibility (new affordances):
- `pnpm gha --doctor` runs an inside-the-container fidelity audit:
os + arch + node/pnpm/python versions, version snapshots of every
baked tool, env vars (CI, HOME, TMPDIR), uid/gid, and the
host.docker.internal resolution. useful for "works in CI fails
locally" or vice versa
- `pnpm gha --build --no-cache` busts the docker layer cache when
an apt mirror, base image, or external download has changed
upstream
- entrypoint's `pnpm install` warmup is now wrapped in a `flock` on
a file in the shared node_modules volume — concurrent `pnpm gha`
invocations (e.g. play in one terminal, runtest in another)
serialize their install instead of racing
docs:
- new "Gaps (known)" section in wiki/docker.md explicitly calling
out the things this system can't do yet, including the missing
`uses: ./action` semantics gap that
`.github/workflows/action-gha-e2e-adhoc.yml` currently fills via
GHA only (designing a local `pnpm gha-action <fixture>` is on the
roadmap), service containers, parallel-run sharing, and arch
differences (arm64 vs amd64)
* docs: audit + corrections after testing fronts
self-audit pass for stale references and incomplete pointers:
- wiki/browser.md: `Docker (node:24)` → `pnpm gha container (ubuntu:24.04)`.
the substance was right (chrome not preinstalled) but the base image
reference was stale.
- wiki/docker.md: the "Permission errors" troubleshooting line claimed
the node_modules volume is chowned on every run; now correctly says
"owned by the host uid on first creation; warm runs skip the chown"
to match the actual behavior after the initVolumeOwnership fix.
- wiki/action-tests.md: `API_URL` env-var doc now mentions BOTH paths
(`localhost:` from play:local, `host.docker.internal:` from inside
the container). Proxy/router recipe now shows both invocations
side-by-side instead of saying "must use play:local".
- wiki/billing.md: same dual-recipe update for the loop-including-the-
action proxy walkthrough.
- gha.ts header: expanded the usage block to include --clean / --doctor /
--no-cache / --shell-TTY, added the host.docker.internal note, and
pointed at wiki/docker.md for design rationale.
self-document check: a future agent landing on this code can answer
"how do I run a fixture / debug in shell / add a tool / diagnose
fidelity / reach a local dev server" purely from gha.ts header +
wiki/docker.md without spelunking through the entrypoint or git
history.
* action: minimize pullfrog.yml permissions and drop actions:read
The recommended pullfrog.yml workflow asked for a permissions block that's
broader than what the action actually uses with the workflow GITHUB_TOKEN —
all real work (git push, PR comments, reviews) goes through installation
tokens that the action mints via OIDC. Customer security scanners flagged
the workflow-level block as too permissive.
- Move permissions to the job level and reduce to id-token: write,
pull-requests: write, issues: write. contents:read is the implicit default
and covers actions/checkout; contents:write, checks:read are unused by
any GITHUB_TOKEN consumer; actions:read was only used by post-cleanup's
listJobsForWorkflowRun call.
- Replace listJobsForWorkflowRun with a SIGTERM/SIGINT handler in main.ts
that calls core.saveState("cancelled", "true"); post-cleanup reads it
back via core.getState. Same cancel-vs-failure UX, no extra scope needed.
- Sync the docs (headless-action, getting-started, action/README) and the
two dogfood pullfrog.yml workflows to the new minimal block. Update the
post-cleanup wiki to describe the saveState approach.
* action: drop pull-requests/issues from required workflow scopes
Switch postCleanup.ts to mint its own short-lived installation token via OIDC
(acquireNewToken with issues:write + pull_requests:write) instead of using the
workflow GITHUB_TOKEN. Same comment-update behavior, but the workflow no longer
needs those scopes — the only permissions Pullfrog ever asks for are id-token:write
(OIDC exchange) and contents:read (actions/checkout).
Also fixes a bug from the previous commit: setting an explicit permissions block
drops every unlisted scope to none (with metadata as the only exception), so
omitting contents would have broken actions/checkout. Restored at both workflow
and job level.
* action: scope id-token:write to pullfrog job, not workflow level
id-token:write is the powerful one — it lets a job mint OIDC tokens that can
be exchanged for cloud credentials or our installation tokens. Keeping it at
workflow level means any future job added to this file silently inherits it.
Move it to the job level where it's actually used; leave only contents:read
at workflow level as a safe baseline for any future jobs.
* action: move stuck-comment cleanup server-side, drop write perms entirely
The action's post-cleanup step lived inside the runner and used the workflow
GITHUB_TOKEN to update the "Leaping into action…" progress comment when a run
failed/cancelled, requiring pull-requests:write + issues:write at the workflow
level. Move that responsibility to the workflow_run.completed webhook handler:
it already has installation-token access via the GitHub App, runs server-side
(no Pullfrog API dependency loop on failure), and lets us drop both write perms.
Recommended workflow permissions block is now truly minimal:
permissions:
contents: read
jobs:
pullfrog:
permissions:
id-token: write
contents: read
Server side
- handleWorkflowRunCompleted: when conclusion != "success" and the WorkflowRun
has progressCommentId, mint installation octokit and update the stuck comment
in place. Try issues.getComment first, fall back to pulls.getReviewComment on
404 (we don't store comment type — one wasted GET on the rarer review case).
- Reuses buildPullfrogFooter and updateProgressComment from pullfrog/internal,
matching the wording the action used to write client-side.
Client side
- Delete action/utils/postCleanup.ts and action/post.ts.
- Remove post: + post-if: from action/action.yml.
- Drop runPostCleanup wiring from action/commands/gha.ts and action/play.ts.
- Remove the SIGTERM/saveState handler I added in main.ts in the previous commit
(no longer needed; cancel/fail signal comes from the webhook hook payload).
Plumbing
- Extract isLeapingIntoActionCommentBody into action/utils/leapingComment.ts so
the predicate can be re-exported via pullfrog/internal without dragging the
MCP server's transitive type graph into the Next.js app's typecheck.
- mcp/comment.ts re-exports from the new location for backward compat.
Wiki
- Delete wiki/post-cleanup.md (obsolete; cleanup is now a one-liner branch in
the workflow_run webhook handler).
* chore: ignore .worktrees in biome config
Recently-added pnpm worktree feature creates nested git worktrees under
.worktrees/, each with their own biome.jsonc declaring root. Biome's
recursive scan trips on the nested config and fails pnpm lint. Excluding
the directory matches the existing .gitignore entry.
* fix: address PR #594 review findings
Two real bugs caught by code review:
1. handleWorkflowRunWebhook.ts:323 — drop the /m flag on the stuck-comment
detection regex. With /m, ^ matches any line start, so any finalized
progress comment that embeds a task list (report_progress writes
`- [x]`/`- [ ]` lines via todoTracking.ts) would be flagged as "stuck"
and silently overwritten with the "This run croaked" boilerplate
whenever the workflow concluded non-success after the agent's final
summary already landed. Restores the body-start anchoring the original
in-process postCleanup.ts:90 had.
2. action/scripts/check-entrypoint-imports.ts — drop ../post.ts from the
esbuild entry-point list (the file was deleted in aa43b9af). The
`pnpm check:entrypoints` step in test.yml would have failed on every
run with an unresolvable-entry-point error.
Plus three small follow-ups:
- main.ts:580 — comment said "post-cleanup has its own verify-retry loop"
but post-cleanup is gone. Updated to describe the new server-side path.
- mcp/comment.ts:443 — comment said "so post script doesn't think the run
failed". Updated to describe the actual current consumers of wasUpdated.
- commands/gha.ts:84 — `--post` help text said "run post-cleanup flow" but
with the post-cleanup path removed, --post is only valid alongside the
`token` subcommand for installation-token revocation. Updated wording.
* fix(action): scope --post help text to gha token subcommand
Root gha help text was documenting --post, but --post only makes sense
paired with the token subcommand (it's how the post step revokes the
installation token previously acquired in the main step). Move it to a
dedicated gha token help section and add a parser layer that rejects
--post on the bare gha command.
$ pullfrog gha --help
usage: pullfrog gha [subcommand]
...
options:
-h, --help show help
$ pullfrog gha token --help
usage: pullfrog gha token [--post]
...
options:
-h, --help show help
--post revoke the previously-acquired token (post-step usage only)
* webhook: artifact-aware cleanup of stranded leaping comments on success
Previously the workflow_run.completed cleanup only handled non-success
conclusions. Extend it to also catch the rare case where a successful
run leaves a "Leaping into action…" comment stuck (in-process cleanup at
action/main.ts:723 normally handles this, but can be skipped on SIGKILL,
runner host crash, or any exit path that bypasses main()'s finally block).
New behavior in cleanupStuckProgressComment:
- cancelled → update with "cancelled 🛑" body (unchanged)
- failure (other) → update with "croaked 😵" body (unchanged)
- success + artifact recorded → delete the comment (the artifact is the
user-facing surface; the leaping comment
is just stale UI noise at this point)
- success + no artifact recorded → delete the comment AND alert
team@pullfrog.com via emailAlert
The "success + no artifact" path is "should never happen" territory: the
run claims success but produced no review, PR, issue, plan, or summary
comment. The team alert helps us catch in-process cleanup regressions or
artifact-tracking gaps. hasRecordedArtifact reads {review,pr,issue,
planComment,summaryComment}NodeId off the WorkflowRun row to make the call.
* webhook: narrow stuck-comment detection to leaping prefix only
Drop the stranded-todo-pattern branch from cleanupStuckProgressComment.
The leaping prefix is highly specific and impossible to confuse with a
legitimate summary; a leading todo line is not — the agent's
error-reporting paths can produce useful explanatory comments whose
body leads with a checklist (e.g. "here's what I was working on" + the
incomplete todo list), and we don't want to silently overwrite those
with the generic "croaked" boilerplate.
In-process cleanup at action/main.ts:723 still handles the stranded-todo
case in the common path (gated on !finalSummaryWritten with full access
to the in-memory tool state). Missing the rare runner-died-mid-todo case
server-side is a worthwhile trade vs. the false-positive risk on real
explanatory comments.
* unify per-run token + cost accounting across agents
every agent harness now logs the same 5-column (or 6 with cost) table and
populates the same AgentUsage contract, regardless of agent or upstream
provider. previously OpenCode and the Claude fallback path emitted a 3-col
table whose "Input Tokens" was actually only the non-cached delta, silently
dropping cache read/write — real runs were being reported at ~0.4% of their
true input (e.g. one baseline showed Input=30 while step_finish events
summed to cache_read=724,753).
changes:
- add logTokenTable helper in action/agents/shared.ts with stable columns:
Input | Cache Read | Cache Write | Output | Total | Cost ($). cost
column renders only when a value is known.
- action/agents/opencode.ts: accumulate step_finish.part.tokens AND
step_finish.part.cost (sourced from models.dev inside opencode —
confirmed working across Anthropic, OpenAI, Google, xAI, DeepSeek,
Moonshot, and OpenRouter). drop the event.stats.total_tokens fallback
since that payload has no cache breakdown.
- action/agents/claude.ts: success-path now treats input_tokens as the
non-cached field (matching OpenCode semantics), carries
cache_read_input_tokens / cache_creation_input_tokens separately, and
captures total_cost_usd from the final result event. the per-message
fallback accumulator now captures cache fields too so it's no longer
lossy when the result event never fires.
- formatUsageSummary gains a Cost ($) column that matches the stdout
table row-for-row; missing values render as "—".
- scripts/token-usage.ts parses all three historical formats (new 5-col,
legacy 4-col Claude success, legacy 3-col lossy) and explicitly flags
the lossy runs instead of averaging misleading values.
validation (pnpm play --local, identical "say hello" prompt):
agent+model Input CacheR CacheW Output Total Cost
OpenCode + Anthropic Sonnet 4.6 4 41,177 20,735 129 62,045 $0.0921
Claude CLI + Anthropic Sonnet 4.6 9 80,133 11,611 389 92,142 $0.0766
OpenCode + OpenAI codex-mini 10,893 46,976 0 606 58,475 $0.0059
OpenCode + Google Gemini 3 Flash — — — — — $0.0114
OpenCode + xAI Grok 4 Fast — — — — — $0.0035
OpenCode + DeepSeek Chat 18,854 0 0 1 18,855 $0.0053
OpenCode + Moonshot Kimi K2.5 — — — — — $0.0106
OpenCode + OpenRouter→Anthropic — — — — — $0.0617
OpenCode + OpenRouter→OpenAI — — — — — $0.0038
* isolate play.ts from developer gitconfig
play.ts is a CI-emulator but inherits the developer's user- and system-scope
gitconfig. a common local convenience — url."git@github.com:".insteadOf
"https://github.com/" to force SSH auth — gets applied at read time on every
git call inside the temp repo, causing `git remote get-url --push origin`
to return an SSH URL instead of the stored HTTPS one. pullfrog_push_branch's
validatePushDestination (correctly) treats that as tampering and blocks the
push. the agent then burns the full MAX_COMMIT_RETRIES budget trying
workarounds that can't beat a user-scope insteadOf rule, turning a trivial
"say hello" run into a 1.35M-token session.
point GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null inside run() so
the play process and its spawned agent see the same empty gitconfig that
a real CI runner would. CI has no rewrites, so this is a no-op there; dev
machines get CI-identical git state. SSH client config (~/.ssh/config and
keys) is separate from gitconfig and is unaffected, so setupTestRepo's SSH
clone still works locally. setupGit only writes --local scope, so nothing
downstream depends on user-scope values.
verification: with the scratch repo cleaned up and this isolation in place,
OpenCode + Anthropic on the same "say hello" prompt goes from 1,349,654
tokens / $2.00+ to 62,045 tokens / $0.0921 — no retry loop, no push blocks.
* persist aggregated token + cost usage to WorkflowRun
AgentUsage has been memory-only — rendered into the GitHub step summary
and then discarded when the runner tears down. that made questions like
"avg cost per customer per day" require log-spelunking. persist it:
- add Int? columns for inputTokens / outputTokens / cacheReadTokens /
cacheWriteTokens and a Decimal? costUsd column on workflow_runs.
Int4's 2.1B ceiling is ~200x larger than any realistic run so BigInt
would be overkill. costUsd uses the same default Decimal precision
as existing money columns (accounts.usageUsd, proxy_keys.hwmUsage).
- extend PATCH /api/workflow-run/[runId] to accept the new numeric
fields alongside the existing artifact strings. per-field type
validation ensures the allowlist stays scalar-safe and rejects
negative / non-finite values.
- generalize patchWorkflowRunFields in the action so it accepts a
mixed string/number payload, and add an aggregateUsage(entries)
helper that sums per-agent AgentUsage records into a single patch.
- call the reporter from main.ts's outer finally block, gated on
toolContext. this is the shared cleanup path that every agent
implementation flows through — claude.ts, opencode.ts, and any
future harness all push their AgentUsage into toolState.usageEntries
via the same line 468, so one finally-block call covers them all.
running in finally also means partial usage gets persisted even
when the agent errored out mid-run.
* anneal token + cost accounting
follow-up polish from a review pass:
- aggregate usage across commit-retry iterations inside each agent harness.
previously runClaude / runOpenCode returned only the final retry's usage,
so any run that hit the dirty-tree retry loop under-counted tokens and
cost in both the stdout table and the WorkflowRun row. added a shared
mergeAgentUsage helper in agents/shared.ts; both harnesses now fold each
iteration's usage into a running total and return the sum.
- scripts/token-usage.ts now handles the unified format with or without
the Cost ($) column. previously the int-only number regex rejected
decimals and the 5-cell length check rejected 6-cell rows, so logs
from post-cost-tracking runs fell through to "no token table". the
parser now accepts both 5- and 6-cell unified rows, splits int vs
decimal cells, and averages reported Cost alongside the tokens.
- PATCH /api/workflow-run/[runId] now rejects INT field values above
INT4_MAX (2_147_483_647) so a malformed payload gets a clean 400
instead of propagating a Prisma error. also defends against a
compromised runner sending a deliberately huge value.
- clarifying comments: opencode.ts documents that step_finish.part.cost
is a per-step delta (empirically verified), main.ts explains that
toolState.usageEntries already carries merged per-retry usage so
aggregateUsage just sums entries (one per agent.run()).
- tests for aggregateUsage and mergeAgentUsage — 12 new cases covering
empty / partial / multi-agent inputs and the "keep undefined" semantic
that prevents spurious zeros from being persisted.
- drop `as number` cast in logTokenTable — narrow via const instead.
* anneal: clamp INT overflow + guarantee mergeAgentUsage immutability
second review pass surfaced two defensive gaps:
- a single token field exceeding INT4_MAX would pass the client but be
rejected by the server's per-field validator, writing a partial row
with some NULLs where sums belonged. clamp in aggregateUsage so the
wire payload is always self-consistent across all numeric columns,
with a loud warning so the clamp doesn't silently swallow weirdness.
- mergeAgentUsage's single-sided branches returned the input reference.
callers treat AgentUsage as immutable but future callers might not;
always return a fresh shallow copy instead. two new tests guarantee
the no-mutation-leak property.
no behavior change in the happy path — INT4_MAX is ~200x the largest
realistic per-run token count.
* anneal: resilient usage persistence + cross-platform null device
third review pass surfaced three small issues:
- main.ts finally block: writeGitHubUsageSummaryToFile throwing would
skip the WorkflowRun usage PATCH. both are independent best-effort
cleanup tasks — wrap the former in catch so a filesystem failure
doesn't block DB persistence.
- AgentUsage.inputTokens had no jsdoc explaining that it's the full
billable input (cached + non-cached). the same word "Input" means
"non-cached only" in the stdout/markdown tables (derived by
subtraction). document the semantic so dashboards querying
WorkflowRun.inputTokens don't misinterpret it.
- play.ts gitconfig isolation was hard-coded to "/dev/null" which
doesn't exist on Windows. use `os.devNull` for cross-platform
parity (resolves to `\\.\nul` on win32). the project is Linux-only
in CI so this only helps local Windows contributors, but it's a
zero-cost swap.
also updated the finally-block caveat comment: usage is only pushed
to toolState.usageEntries when agent.run() returns an AgentResult,
not when the timeout race rejects — so timed-out runs don't
persist partial usage. documented instead of trying to thread state
through Promise.race.
* anneal: NaN-guard cost accumulators + clarify inputTokens docs
final polish from review round 4:
- guard both cost accumulators (opencode step_finish.part.cost and claude
result.total_cost_usd) with Number.isFinite. `typeof x === "number"`
accepts NaN, and one NaN `+=` would poison the running total for the
whole session.
- reword prisma schema comment on WorkflowRun usage fields to call out
that cacheReadTokens / cacheWriteTokens are SUB-totals within
inputTokens (not additional tokens on top). prevents future dashboards
from double-counting by ~2x when summing "total tokens used".
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7
Made-with: Cursor
* fix stale agent/effort refs, add tests for askpass + model resolution
- reviewCleanup.ts: payload.agent -> payload.model, remove effort
- selectMode.ts PlanEdit: remove delegation/subagent/effort references
- pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY,
add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY)
- FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy
- new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen)
- new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override)
- new: models.test.ts (19 tests — parseModel, resolution, registry invariants)
- update models.dev snapshot
Made-with: Cursor
* fix changed-agents.sh to filter legacy agent files from CI matrix
legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not
exported from index.ts. changed-agents.sh now reads index.ts imports to
build the active agent set and treats changes to inactive files as
non-agent changes (opentoad canary only).
Made-with: Cursor
* remove MCP file tools, old agent harnesses, and obsolete security tests
ASKPASS-based git auth makes the old MCP file tool security layer unnecessary:
- token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it
- agents now use native file tools (OpenCode builtin read/edit)
deleted:
- action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory)
- action/mcp/index.ts (dead re-export)
- agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts
- opencode-runner.ts (inlined into opentoad.ts)
- security tests that validated MCP file tool restrictions
- commented-out three-step review flow (~300 lines)
- sanitizeSchema/wrapSchema dead code from mcp/shared.ts
- OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed)
updated test prompts to use generic file ops instead of MCP tool names.
restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense).
Made-with: Cursor
* bump actions/checkout v4 → v6 (node 24)
node 20 actions deprecated june 2, 2026.
Made-with: Cursor
* temporarily disable fail-fast on agnostic tests to debug checkout@v6
Made-with: Cursor
* re-enable fail-fast on agnostic tests
Made-with: Cursor
* fix test token mismatch: mint OIDC tokens scoped to target repo
CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit
the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on
every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so
ensureGitHubToken() mints a properly scoped token via OIDC.
Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming
instead of repeating it in every test file, and fixes preview-cleanup
to remove workers from all queues (not just name-matching ones).
Made-with: Cursor
* fix ensureGitHubToken to try OIDC when app credentials are absent
ensureGitHubToken only attempted token minting when GITHUB_APP_ID and
GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds
aren't exposed — so the guard prevented minting entirely.
Made-with: Cursor
* dead code cleanup: remove remnants of deleted agents, file tools, effort system
remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps,
orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale
opencode-runner wiki refs, deleted test file references, and MCP file tool
docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to
globalSetup (runs once before forks instead of per-file, 19s → 200ms).
Made-with: Cursor
* address review feedback: remove dead code, update stale references
- remove AGENT_OVERRIDE (only opentoad exists)
- remove shellToolName plumbing (always restricted shell)
- bump action version to 0.0.179
- remove CURSOR_API_KEY from all workflows/configs
- remove OPENCODE_MODEL_MINI/MAX from workflows/docs
- delete wiki/effort.md, rewrite docs/effort.mdx as "Models"
- rewrite wiki/modes.md: orchestrator/subagent → single agent
- simplify flag system: drop builtin flag extraction (debug, effort,
timeout, agent), keep custom flag replacement only
- reserve all legacy flag names to prevent custom flag conflicts
Made-with: Cursor
* regenerate lockfile after removing claude-agent-sdk and codex-sdk
Made-with: Cursor
* fix import ordering, add lockfile check to pre-push hook
Made-with: Cursor
* remove dead debug payload field, stale packageExtensions
Made-with: Cursor
* merge proc-sandbox and token-exfil into a single test
proc-sandbox and token-exfil were duplicative — both tested that
SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into
token-exfil with shell:restricted (which actually exercises filterEnv)
and the /proc attack vector hints from proc-sandbox.
Made-with: Cursor
* fix wiki adversarial.md to match actual tokenExfil validator
Made-with: Cursor
* run post action cleanup in play script after main completes
* clarify that GITHUB_RUN_ID is the actual bail-out gate in play context
* treat GITHUB_RUN_ID as optional in post cleanup
* replace dynamic import with static import of `runPostCleanup`
Export `runPostCleanup` from post.ts and guard the top-level
execution with `import.meta.url` so it only auto-runs as an
entry point. play.ts now statically imports and calls it.
* move runPostCleanup into finally block and let failures propagate
* refactor post cleanup into utility module
move post cleanup logic into a dedicated utility and keep post.ts as a pure script entrypoint. update play.ts to import the shared utility directly and normalize direct-execution detection.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Scope installation token permissions in restricted mode
In restricted/disabled bash mode, the installation token is now scoped
to match the workflow's permissions block. This preserves fork push
capability while limiting what the agent can do with the token.
- Read workflow permissions from pullfrog.yml at runtime
- Pass permissions to API when acquiring installation token
- Clear OIDC env vars in restricted mode to prevent token minting
- Simplify setupGit by moving token resolution to main.ts
* Address review feedback: fail closed with default permissions
- Add restrictive default permissions (contents:read, pull_requests:read,
issues:read) as fallback when workflow permissions can't be read
- Add support for job-level permissions via GITHUB_JOB env var
- Fix misleading comment about token resolution in restricted mode
- Add documentation about fork PR checkout behavior
* Simplify to separate git/MCP tokens without workflow permission scoping
- gitToken: minimal contents:write only (assumed exfiltratable)
- mcpToken: full installation token (not exfiltratable via MCP tools)
- Remove workflowPermissions.ts - security-conscious users can pass
their own token via GH_TOKEN or inputs.token
- Add type-safe InstallationTokenPermissions to github.ts and API route
* Rename `write` permission to `push` and remove vestigial tool blocking
The `write` permission was previously used to block local file write tools
in agents. This was security theater since bash can write files anyway.
Now `push` only controls the git token scope:
- push: enabled → contents:write (can push commits)
- push: disabled → contents:read (read-only, can't push)
Changes:
- Rename `write` to `push` in action.yml, Prisma schema, and all TS types
- Remove vestigial write tool blocking from all agents (claude, cursor,
gemini, opencode, codex)
- Add data-preserving Prisma migration using RENAME COLUMN
- Update UI: "Write files" → "Git push" with updated description
* add PID namespace isolation for bash sandbox
when running in CI, attempts to use unshare --pid to create a new PID
namespace for bash subprocesses. this prevents the /proc/$PPID/environ
attack where a malicious command could read secrets from the parent
process's environment.
the protection works by:
1. creating a new PID namespace (subprocess becomes PID 1)
2. mounting fresh /proc showing only sandbox PIDs
3. parent PIDs become invisible (PPID = 0, /proc/0 doesn't exist)
combined with filterEnv(), this provides complete protection against
/proc-based secret theft. falls back gracefully if namespaces aren't
available.
includes test script to verify the protection works.
* add PID namespace test to CI workflow
tests whether unshare --pid works on GHA runners out of the box,
and if not, whether enabling via sysctl helps. also runs the
pidNamespace.ts test to verify the full protection.
* fix pnpm setup and add procIsolation agent test
- fix pnpm/action-setup by specifying package_json_file path
- add procIsolation crossagent test that has agent attempt to
read secrets via /proc/$PPID/environ
- add procIsolation to CI test matrix
* add pid-namespace test job to main workflow
this job tests unshare --pid capabilities on GHA runners and runs
the pidNamespace.ts adhoc test to verify /proc isolation works
* test bubblewrap's sysctl approach for enabling namespaces
- write to /etc/sysctl.d/99-userns.conf and run sysctl --system
- try aa-complain on unshare binary
- more detailed diagnostics
* fix pidNamespace test and add sudo-unshare fallback for GHA
- fix reference error in pidNamespace.ts (renamed function but didn't update calls)
- add sudo-unshare as fallback method for GHA runners where unprivileged
namespaces are blocked but sudo is available
- update bash.ts to detect and use sudo unshare when unprivileged fails
* consolidate security docs and document PID namespace isolation
- update security.md with current implementation details
- document sudo unshare fallback for GHA runners
- add testing instructions for local Docker and CI
- add "Further Exploration" section with Landlock and path validation ideas
- delete bash-sandbox.md and landlock.md (consolidated into security.md)
* move procIsolation test to adhoc folder
the procIsolation test requires PID namespace capabilities that aren't
available in the Docker test environment. moved to adhoc/ so it's excluded
from default test runs and can be run explicitly when needed (e.g. via
the pid-namespace CI job or locally with --privileged docker).
* fix Docker test environment for PID namespace isolation
- add CI and GITHUB_ACTIONS to testEnvAllowList so sandbox detection runs
- add --privileged to Docker run for PID namespace support (unshare)
this fixes the test environment to properly test the sandbox. in production,
the action runs directly on GHA runner where sudo unshare works.
* fix getJobToken() to work in test environment
add fallback to GH_TOKEN and GITHUB_TOKEN when INPUT_TOKEN is not set.
this allows tests to run without requiring workflow-level token input.
the token resolution order is:
1. INPUT_TOKEN (from workflow `with: token:`)
2. GH_TOKEN (external token override)
3. GITHUB_TOKEN (pre-acquired in tests or from GHA env)
* security: filter secrets from all subprocess environments
- extract filterEnv() to shared utils/secrets.ts
- make $() utility filter secrets by default (git, npm, etc. don't need them)
- disable git hooks via core.hooksPath to prevent hook-based exfiltration
- git auth uses token embedded in URL, not env vars
this prevents malicious git hooks, npm postinstall scripts, and other
code execution vectors from exfiltrating GITHUB_TOKEN and API keys.
* docs: clarify defense-in-depth security model
update security.md to explain why BOTH layers are required:
- filterEnv(): cleans child's own /proc/self/environ
- PID namespace: hides parent's /proc entries
PID namespace alone isn't sufficient - with --mount-proc, the child
becomes PID 1, so /proc/1/environ is the child's OWN environment.
without filterEnv(), secrets would still be accessible there.
* add procSandbox crossagent test for PID namespace security
- add crossagent/procSandbox.ts: security test that instructs agent to try
various /proc attack vectors and validates sandbox blocks them
- update wiki/security.md: document PID namespace isolation details, add
verification commands, explain why sudo inside sandbox doesn't break security
- update docker.ts: use node:24 with sudo for GHA-like test environment
- update instructions.ts: allow disabling security messaging for pentests
- clean up adhoc test files (procIsolation.ts, securityAudit.ts)
the procSandbox test sets SANDBOX_TEST_TOKEN (matches *_TOKEN filter), gives
the agent explicit hints about attack vectors (/proc, sudo, nsenter, etc.),
disables prompt-based mitigations, and validates the secret is never leaked.
* move procSandbox test to agnostic/ (runs with one agent)
* WIP
* docs: add agent testing guide (pnpm play, Docker, pentesting)
* docs: add CI details to agent testing guide
* docs: add interesting findings and gotchas from pentesting
* improve test fidelity: auto-set CI=true, verify sandbox active
- docker.ts: always set CI=true in container so sandbox activates
- docker.ts: skip sudo/user setup if already done (faster reruns)
- bash.ts: export getSandboxMethod() for diagnostics
- bash.ts: add debug log when sandbox disabled
- procSandbox.ts: add sandbox_was_active check to catch vacuous passes
the CI=true change is critical: without it, PID namespace isolation
is skipped and security tests pass without actually testing anything.
* docs: update agent-testing.md with CI=true auto-set note
* docs: clarify log format is agent-specific
* fix git auth, simplify MCP tools, add adversarial tests
- fix gitWithAuth to use Basic auth format (Bearer doesn't work with git's http.extraheader)
- fix token.ts: push:restricted now correctly gets contents:write
- fix github.ts: pass permissions when acquiring installation tokens locally
- remove commit_files and create_branch MCP tools (redundant, don't require credentials)
- remove containsSecrets() - trivially bypassable, not sound security
- add agnostic adversarial tests: pushDisabled, pushRestricted, tokenExfil
- update instructions.ts to clearly list available git MCP tools
- add wiki/git.md documenting credential isolation and push permission tiers
- update wiki/docker.md with custom image considerations
- update wiki/agent-testing.md with adversarial testing patterns
* fix type errors after rebase
- change ResolveTokensParams.push from ToolPermission to PushPermission
- use tags: ["agnostic"] instead of agnostic: true in test files
* fix cleanup permission error in sandbox tests
when sandbox isolation is enabled (CI=true), files created by the unshare
subprocess may have different ownership, causing rmSync to fail with EACCES.
this error in the finally block was overriding the test's success result.
fix: wrap cleanup in try-catch and fall back to sudo rm if rmSync fails.
* Add adhoc
* Handle git config/remote bypasses
* add git hooks protection and simplify ToolState
- disable git hooks in restricted mode via -c core.hooksPath=/dev/null
- add gitHooks adversarial test to verify hook protection works
- unify prNumber/issueNumber into single issueNumber field
- add pushUrl to ToolState for push validation
- add generateTestMarker() for simpler single-agent test markers
- export SENSITIVE_PATTERNS and isSensitiveEnvName from secrets.ts
- remove redundant pidNamespace.ts (duplicated by procSandbox.ts)
- update documentation
* harden $git() auth: subcommand whitelist, binary tamper detection
- rename gitWithAuth() to $git() with explicit subcommand first arg
- restrict to "fetch" | "push" at type level (filters don't run for these)
- resolve git binary path at startup via resolveGit(), sha256 fingerprint
- verify hash before each $git() call to detect binary replacement
- rename disableHooks to restricted for cleaner semantics
- document filter exfiltration attack and empirical verification in wiki
* remove redundant pid-namespace CI job
the PID namespace isolation testing is now handled by
action/test/agnostic/procSandbox.ts via pnpm runtest agnostic
* fix push_branch for new branches and improve token leak detection
- getPushDestination now falls back to origin/<branch> when @{push}
is not configured (happens for new branches created locally)
- gitPerms validator now checks for actual token patterns instead
of matching "x-access-token" string in test instructions
* use kebab-case for test names
* simplify shell env API: "restricted" | "inherit" | object
replace passFullEnv boolean with cleaner env option that accepts:
- "restricted" (default): filterEnv() to prevent secret leakage
- "inherit": full process.env
- object: custom env merged with restricted base
* share EnvMode and resolveEnv between shell.ts and bash.ts
move shared env resolution logic to secrets.ts
* add env option to bash tool (default: restricted)
* delete agent-testing.md (renamed to adversarial.md)
* Add checkout tests
* reframe githooks test prompt to avoid claude safety refusal
claude was refusing to execute the test because the prompt used words
like "malicious" and "security testing". reframed as a debugging task
with innocuous env var name (TESTING_DEBUG_TAG) per adversarial.md guidance.
Co-authored-by: Cursor <cursoragent@cursor.com>
* clean up verbose token acquisition logs
move logging responsibility to call sites which have better context
(git token vs MCP token). remove redundant intermediate OIDC logs
and unused "(permission-scoped)" suffix.
Co-authored-by: Cursor <cursoragent@cursor.com>
* isolate agnostic tests with matrix strategy, fix .pullfrog-env secret leak
- split action-agnostic into per-test matrix jobs for isolated logs and filesystems
- only write explicitly opted-in env vars to .pullfrog-env via fileAgentEnv
(fixes token-exfil test where claude found SANDBOX_TEST_TOKEN on disk)
- mcpmerge test opts in via fileAgentEnv for cursor's repo-level MCP fallback
Co-authored-by: Cursor <cursoragent@cursor.com>
* remove env parameter from bash tool to prevent agents bypassing filterEnv
the bash tool exposed an `env` parameter accepting "restricted" | "inherit"
which allowed agents to pass env: "inherit" and see all secrets including
SANDBOX_TEST_TOKEN, bypassing the restricted environment filtering entirely.
env mode is now determined internally (always restricted).
Co-authored-by: Cursor <cursoragent@cursor.com>
* use pullfrog/test-repo for push tests to stop polluting main repo
push tests were creating branches and tags on pullfrog/app directly.
now all push tests (push-restricted, push-disabled, push-enabled,
git-permissions) target pullfrog/test-repo instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* use pullfrog/test-repo for all tests, not just push tests
no test should clone or operate on pullfrog/app directly.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix token scoping for test-repo and bash timeout defaults
- acquireTokenViaOIDC now includes GITHUB_REPOSITORY repo in token
scope so push tests work against pullfrog/test-repo
- bash tool default timeout: 120s -> 30s, cap: 600s -> 120s
- activity timeout: 30s -> 60s
- prevents hung bash commands (e.g. find /) from killing the agent
via activity timeout
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Colin McDonnell