* add one-time diff coverage preflight for PR reviews
track diff read coverage from agent tool-use events and run a one-time pre-flight before review submission, with explicit coverage skip reasons for low-value files like lockfiles.
Made-with: Cursor
* add manual dispatch fallback for preview deploy workflow
allow preview repo and preview sync jobs to be run via workflow_dispatch with explicit PR number and branch inputs, so preview provisioning can be retriggered when pull_request events fail to fire.
Made-with: Cursor
* fix manual preview dispatch PR input wiring
use normalized PR number and branch env values for comment creation and script env wiring so workflow_dispatch preview runs can create and update PR-specific preview resources.
Made-with: Cursor
* remove obsolete snapshots invalidated by checkout instructions change
* fix diff coverage read offset handling and add local sanity-check guidance
normalize read offset semantics for diff coverage tracking, reuse shared range counting in review preflight, add focused diff coverage unit tests, and document the local play.ts testing workflow in AGENTS.md.
Made-with: Cursor
* add regenerated mcp test snapshots
capture snapshot files generated by the review comment and checkout formatting tests during pre-push validation so the branch remains clean and reproducible.
Made-with: Cursor
* add diff coverage preflight instrumentation logs
log diff coverage initialization in checkout_pr and emit preflight state/breakdown diagnostics in create_pull_request_review to debug missing coverage enforcement in preview e2e runs.
Made-with: Cursor
* add env override to force local cli execution in action runtime
support explicit local-cli execution via PULLFROG_FORCE_LOCAL_CLI so preview workflows can run branch action code instead of the npm fallback package during e2e debugging.
Made-with: Cursor
* add preview e2e debugging learnings for action runtime validation
capture the preview execution-path gotchas and one-time preflight verification pattern in AGENTS.md so future investigations validate the real runtime and avoid npm fallback confusion.
Made-with: Cursor
* reduce diff coverage log noise while preserving failure visibility
downgrade verbose diff coverage lifecycle diagnostics to debug, keep a concise info-level pre-flight failure signal, and document preview runtime debugging learnings in AGENTS.md.
Made-with: Cursor
* WIP
* tune sync.md: ff override + softer overlap verification
Made-with: Cursor
* chore: bump models snapshot for claude-opus-4-7
Made-with: Cursor
* rip out coverage_skips waiver from diff coverage pre-flight
Made-with: Cursor
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
PR CI kept breaking on upstream catalog drift (new model ships on models.dev,
OpenRouter renames an id, etc.) — failures unrelated to the PR's contents.
split the model-alias test suite so PRs only see pure-logic checks, and push
the external-state drift + end-to-end coverage to main.
test organization:
- action/test/models.test.ts keeps pure invariants: openRouterResolve
completeness and fallback-chain resolution. runs on every PR.
- action/test/models-catalog.main.test.ts gets the 4 network-dependent
describes (models.dev validity x2, OpenRouter API validity, latest-model
snapshot). runs only on main push via a dedicated vitest config
(vitest.main.config.ts + `pnpm test:catalog`).
new CI jobs in .github/workflows/test.yml:
- models-catalog: `pnpm test:catalog` on every main push. detects upstream
catalog drift so we can react at the next convenient window.
- models-live: 38-entry matrix that invokes the agent harness end-to-end
against the real provider for each alias in models.ts. generated from
action/test/list-aliases.ts. runs only on main push AND only when
resolution-affecting files changed (action/models.ts, action/package.json,
action/agents/**) — the exact shape of the opus 4.7 incident.
test/run.ts: PULLFROG_MODEL now flows through from process.env so the live
matrix can pin an alias per job without the per-agent default clobbering it.
Made-with: Cursor
* rename agent key to opencode and add skill invocation coverage.
add skill-invoke tests for claude and opencode with local play-based validation signals, update CI matrices, and include the current tracked refactors in this branch for review.
Made-with: Cursor
* exclude agent-specific skill-invoke tests from wrong agent in CI matrix
* address review follow-up and preserve workflow run UI tweak.
switch changed-agents ci coverage to exercise the opentoad implementation path while keeping the opencode expectation, and include the local workflow run client interaction updates requested on this branch.
Made-with: Cursor
* remove opentoad agent filename from runtime.
rename the opencode harness implementation file from opentoad.ts to opencode.ts and update ci coverage input accordingly so action code no longer carries the old filename.
Made-with: Cursor
* ensure security prompt bypass is set on every test fixture.
this keeps adversarial and permissions harnesses from being blocked by the default prompt-injection refusal path during CI security tests.
Made-with: Cursor
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
* reorder prompt sections: task-first with dynamic TOC
put the actual task at the top of the prompt for primacy, add a
dynamic table of contents, and push system/runtime metadata to the end.
new section order: TOC → YOUR TASK → PROCEDURE → EVENT CONTEXT →
SYSTEM → LEARNINGS → RUNTIME
Made-with: Cursor
* enforce clean working tree: continue session if agent leaves uncommitted changes
after each agent run, check `git status --porcelain`. if dirty, resume
the same session with instructions to commit on a new branch, push, and
open a PR. retries up to 3 times before giving up.
- claude code: capture session_id from result event, use --resume <id>
- opencode: use --continue to resume the last session
- remove --no-session-persistence from claude (needed for --resume)
- update Task mode to clarify branch/push/PR is the default finalize step
Made-with: Cursor
* log full prompt in collapsible group for debugging
Made-with: Cursor
* fix: format tool refs in buildCommitPrompt via formatMcpToolRef
* enforce clean git status: general instructions, stop hook, and Task mode
Made-with: Cursor
* fix: rename stale titleBody references after body leak fix
Made-with: Cursor
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
models can now be marked `deprecated: true` with a `fallback` slug
pointing to a replacement. `resolveCliModel` follows the chain
recursively (with cycle detection) until it finds a non-deprecated
model. this keeps deprecated models in the registry for backward
compatibility instead of removing them.
marks opencode/mimo-v2-pro-free as deprecated with fallback to
opencode/nemotron-3-super-free.
Made-with: Cursor
* sandbox native filesystem tools to prevent /proc/self/environ exfiltration
the agent's native Read/Grep/Edit tools can bypass the shell sandbox by
reading /proc/self/environ directly. this adds agent-native filesystem
restrictions using the highest-precedence, non-overridable config for each CLI:
OpenCode: OPENCODE_PERMISSION env var with external_directory deny-all + /tmp allow,
plus deletion of untrusted .opencode/plugins/ and .opencode/tools/ before launch.
Claude Code: managed-settings.json at /etc/claude-code/ with denyRead, permissions.deny,
allowManagedPermissionRulesOnly, allowManagedHooksOnly. also --setting-sources user and
--disallowedTools path patterns as belt-and-suspenders.
Made-with: Cursor
* add Glob to Claude Code /proc and /sys deny lists
closes gap identified in review — Glob can enumerate /proc entries.
added to both managed-settings.json permissions.deny and --disallowedTools.
Made-with: Cursor
* run token-exfil test with both agents, hint at native /proc reads
changed tag from "agnostic" (opentoad-only) to "security" so the test
runs with both opentoad and claude. updated prompt to explicitly instruct
the agent to try reading /proc/self/environ via native Read tool.
added API keys to action-agnostic CI job for claude support.
Made-with: Cursor
* move token-exfil to crossagent matrix, remove redundant permissions.deny
- moved token-exfil from agnostic/ to crossagent/ so it runs via the
agent matrix (claude + opentoad in parallel) instead of sequentially
- removed permissions.deny per-tool rules from managed-settings.json;
sandbox.filesystem.denyRead is the single enforcement mechanism
- reverted action-agnostic env vars to minimal set
- updated wiki to match
Made-with: Cursor
* document post-spawn API key deletion analysis in security wiki
evaluated whether API key env vars can be deleted from agent processes
after spawn. OpenCode snapshots env at startup (safe to delete), but
Claude Code re-reads process.env per request (not viable). documented
as further exploration item with per-agent breakdown and caveats.
Made-with: Cursor
* fix stale tokenExfil path references in wiki docs
moved from test/agnostic/ to test/crossagent/ in directory tree
and adversarial test example.
Made-with: Cursor
* revert accidental prisma.config.ts changes
Made-with: Cursor
* hardcode PULLFROG_MODEL per agent in test runner to avoid DB model mismatch
when PULLFROG_AGENT forces a specific agent, the DB-configured model may
belong to a different provider (e.g. openai model with claude agent).
PULLFROG_MODEL short-circuits the DB slug resolution entirely.
Made-with: Cursor
* Update waitlist, run ralph experiments
* fix PR files pagination: use octokit.paginate() for >100 files
* fix garbled FAQ answer on landing page
* track cache read/write tokens in OpenCode agent usage
* wrap dispatch() calls in try/catch to prevent webhook retries on transient failures
* replace raw error messages with generic responses in API routes
* guard request.json() calls with try-catch returning 400 on malformed bodies
* log warning when GraphQL review thread/comment counts hit pagination limits
* reduce review comment cache TTL from 24 hours to 10 minutes
* use select instead of include for proxyKey in workflow run queries
* align Claude agent activity timeout to 5 minutes to match OpenCode agent
* add in-memory dedup for PR close webhooks to prevent duplicate indexing
* extract isPullfrogLogin() helper for shared Pullfrog detection logic
* check response.ok on log fetch in checkSuite.ts
* add 10s timeouts to checkSuite API calls and log fetch
* parallelize proxy key usage API calls with Promise.allSettled
* fix three typos on landing page: colleage, dectects, reponse
* move MAX_STDERR_LINES constant to shared.ts
* add indexes on Repo.accountId and PFUser.accountId FK columns
* remove unused Permission enum from Prisma schema
* populate author and keywords in action/package.json
* use crypto.timingSafeEqual for all secret comparisons
* add missing env vars to globals.ts: R2, webhook, and API secrets
* remove commented-out UserRepo model from Prisma schema
* replace console.log/error with log utility in production API routes
* replace catch(error: any) with proper type guards in getUserRole
* remove stale TODO comment on console page
* handle repository_transferred webhook to update owner
* show toast.error instead of console.error on mode/workflow mutation failures
* add Space key handler for keyboard navigation on workflow run links
* replace role=link spans with button elements for proper accessibility
* add root 404 page with Pullfrog branding
* update ISSUES.md: mark completed items
* mark remaining low-priority UX items as addressed
* add error logging alongside toasts, add check script, update ralph commands
* address review feedback: squash migrations, fix try/catch scope, wire up globals consumers
- squash drop_permission_enum migration into add_indexes migration (one migration per PR)
- move getPullRequest() outside try/catch in mention handler so errors aren't mislogged as "dispatch failed"
- restore key ID in proxyKeys.ts Promise.allSettled error log
- remove accidental asdf.txt and ralph.md files
- wire up globals.ts exports to consumers (r2-uploads, r2-private, verifyHookdeckSignature, sync-usage, forwardPreviewWebhook, dispatch-workflow)
Made-with: Cursor
* update model snapshot (qwen3.6-plus-preview renamed to qwen3.6-plus)
Made-with: Cursor
* Update waitlist, run ralph experiments
* improve review quality: add --effort flag, subagent guidance, remove dead prompts
- add --effort high/max to Claude Code CLI (max for Opus, high for Sonnet/Haiku).
default was silently dropped from high to medium in March 2026.
- add subagent guidance to Review/IncrementalReview modeGuidance for parallel
investigation of large cross-cutting PRs (read-only, no side effects).
- remove "THINK HARDER" from mode prompts (vestigial, no longer controls thinking).
- remove redundant mode.prompt bodies from modes.ts — the actual guidance lives in
modeGuidance (selectMode.ts) and mode.prompt was dead code for all built-in modes
since the delegation system was removed in March.
Made-with: Cursor
* make Mode.prompt optional, remove ModeSchema dead code
prompt is only needed by custom user-defined modes (validated by Zod
modeSchema in utils/schemas/modes.ts). built-in modes get their guidance
from modeGuidance in selectMode.ts. the arktype ModeSchema was never
imported anywhere.
Made-with: Cursor
* make modes.ts the single source of truth for mode guidance
move all mode guidance from modeGuidance in selectMode.ts into
mode.prompt in modes.ts. selectMode.ts now only contains the runtime
tool logic (resolving modes, merging user instructions, handling
PlanEdit/SummaryUpdate overrides). this eliminates the confusing
fallback chain where someone editing mode instructions had to know
to look in selectMode.ts rather than modes.ts.
Made-with: Cursor
* add self-review subagent step to Build mode, update wiki
Build mode now delegates a read-only subagent to review the diff
before committing, catching bugs/logic errors/edge cases that the
builder might miss. Also updates wiki/modes.md to reflect the
single-source-of-truth architecture (modes.ts owns all guidance,
selectMode.ts is pure runtime logic).
Made-with: Cursor
* update model snapshot (openrouter qwen3.6-plus rename)
Made-with: Cursor
model resolution was duplicated inside each agent (opentoad, claude) and
PULLFROG_MODEL override was not considered when choosing the agent. now
resolveModel() runs first in main.ts, its result feeds into resolveAgent()
for agent selection, and the resolved model is passed to the agent via
ctx.resolvedModel. agents only handle their own fallback (opentoad: auto-select
via opencode models, claude: strip provider prefix).
also removes the hardcoded anthropic/claude-sonnet test runner default since
ANTHROPIC_API_KEY is no longer in CI.
Made-with: Cursor
- stop capturing/displaying total_cost_usd from Claude CLI (theoretical cost is misleading for subscription users)
- remove Cost column from action logs table and GitHub Job Summary
- extract SecretsCard into its own sidebar tab with KeyRound icon
- remove children prop from AgentSettingsSection
Made-with: Cursor
* feat: add Claude Code agent for Anthropic model users
Re-adds Claude Code support (removed in #478) so users with Anthropic API
keys or Claude Code OAuth tokens can use their Claude subscriptions directly.
When an Anthropic model is selected and Claude Code credentials are available,
the system auto-selects the Claude agent instead of OpenCode. The harness
mirrors opentoad's security model: native Bash blocked via --disallowedTools,
MCP ShellTool for restricted shell, ASKPASS for git auth. Includes NDJSON
streaming, provider error detection, cache/cost tracking, browser skill,
and todo progress tracking.
Key changes:
- action/agents/claude.ts: full Claude Code harness
- action/utils/agent.ts: auto-select Claude for anthropic/* models
- action/utils/providerErrors.ts: extracted shared provider error detection
- action/utils/skills.ts: extracted shared skill installation (agent-aware)
- action/models.ts: add CLAUDE_CODE_OAUTH_TOKEN to anthropic envVars
- action/utils/docker.ts: add CLAUDE_CODE_OAUTH_TOKEN to test env allowlist
- CI: add claude to test matrix, pass CLAUDE_CODE_OAUTH_TOKEN secret
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove unused toolId variable, fix apiKeys test env cleanup
The apiKeys test cleanup stripped *_API_KEY vars but missed
CLAUDE_CODE_OAUTH_TOKEN which doesn't match that pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: strip provider prefix from PULLFROG_MODEL in Claude agent
the env override path was returning the raw value (e.g.
"anthropic/claude-sonnet-4-5") without stripping the provider prefix,
causing the Claude CLI to receive an invalid model ID.
Made-with: Cursor
* fix: remove dead cliPath field, add CLAUDE_CODE_OAUTH_TOKEN to workflows
remove unused cliPath from Claude agent RunParams, and pass
CLAUDE_CODE_OAUTH_TOKEN through all pullfrog.yml workflow templates
so users with Claude Pro/Team subscriptions can use their membership.
Made-with: Cursor
* fix: block Bash subagent in Claude Code disallowedTools
Made-with: Cursor
* chore: update model snapshot (opencode/openrouter latest → qwen3.6-plus-free)
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix proxy key usage tracking and add OSS spend reporting
replace ProxyKey.disabled with finalizedAt to fix a race where keys
were disabled before their usage was synced, causing all HWM values to
be zero. retireKey now fetches final usage from OpenRouter and records
it atomically with optimistic concurrency. syncAccountUsage skips keys
that fail to fetch rather than recording false zeros.
other fixes:
- wrap OpenRouter API calls in retry logic (exponential backoff)
- reconcileStaleWorkflowRuns now retires proxy keys for completed runs
- subprocess activity timeout only tracks stdout (stderr retry loops
no longer prevent hung agent detection)
- add oss-spend script (single bulk fetch from OpenRouter) and
backfill-proxy-key-usage script
Made-with: Cursor
* address review: move isActiveKey check inside transaction, remove redundant guard
Made-with: Cursor
* fix false "without reporting progress" error + live todo tracking
clean up orphaned progress comments when review is skipped or only
set_output is used, preventing the false positive in handleAgentResult.
parse todowrite events from OpenCode's NDJSON stream and render a
live markdown checklist in the PR progress comment (2s debounce).
agent's explicit report_progress always takes priority.
Made-with: Cursor
* fix contradictory review/progress prompting
align Review and IncrementalReview mode prompts with their guidance —
mode prompts said "always submit" while guidance said "skip if clean."
remove the empty-approval submission that was silently dropped by the
tool. make progress comment lifecycle explicit: created on first call,
updated in place, removed after review submission.
Made-with: Cursor
* centralize todo tracking into shared TodoTracker module
extract inline todo tracking logic (~95 lines) from opentoad.ts into
action/utils/todoTracking.ts. the tracker is created once in main.ts
and passed to agents via AgentRunContext.todoTracker, making it
agent-agnostic and reusable for future agent implementations.
Made-with: Cursor
* fix todoTracker optional type to match file convention
add | undefined to todoTracker in AgentRunContext, matching every
other optional property in the same interface.
Made-with: Cursor
* instruct agents to always maintain a task list for live progress
system prompt now tells agents to create an internal task list at
the start of every run. the tracker renders it to the progress
comment automatically. report_progress is reserved for final
results only — no more intermediate "Checking..." messages that
cancel the tracker and leave stale text on the comment.
Made-with: Cursor
* require report_progress summary at end of every run
agents must always call report_progress with a final summary —
the completed task list should never be the end state of the
progress comment. updated all review mode prompts to call
report_progress after submitting (or not submitting) a review.
Made-with: Cursor
* keep progress comment after review with final summary
stop deleting the progress comment after review submission —
the agent now always calls report_progress with a summary at
the end, and that summary should persist as a record of what
was done.
Made-with: Cursor
* harden stranded progress comment cleanup
- main.ts: detect when tracker was last writer (agent never called
report_progress) and delete the stranded checklist instead of
leaving it as the final comment state
- postCleanup.ts: expand stuck-comment detection to also catch
stranded todo checklists (regex match for checklist patterns)
when the process is killed before normal cleanup runs
- modes.ts + selectMode.ts: add report_progress step to Summarize
and SummaryUpdate modes (only modes that were missing it)
Made-with: Cursor
* fix stale comments, typo, and build mode redundancy
- comment.ts: update deleteProgressComment docstring and inline comment
to reflect current usage (stranded-comment cleanup, not post-review)
- modes.ts: merge duplicate report_progress steps (8 + 10) into single
step 9, fix "optimizatfixons" typo
- wiki/post-cleanup.md: document checklist detection regex
Made-with: Cursor
* collapsible completed todos in final progress, hide set_output outside standalone mode
- add renderCollapsible() to TodoTracker, append completed task list as
<details> section when agent calls report_progress
- cancel tracker after agent's final report_progress so it doesn't
overwrite with raw checklist
- conditionally register SetOutputTool only in standalone mode or when
output_schema is provided
- remove unconditional set_output instruction from orchestrator task section
- update Summarize/SummaryUpdate mode guidance to not reference set_output
Made-with: Cursor
* show completion count in collapsible task list summary
Made-with: Cursor
* only count completed (not cancelled) in collapsible task list summary
Made-with: Cursor
* reinforce concise summary prompting across system prompt, modes, and tool description
Made-with: Cursor
* address review feedback: wasUpdated bypass, tracker false-positive, race condition
- remove wasUpdated=true from cleanup paths so handleAgentResult correctly
detects genuinely silent runs
- add hadProgressComment to ToolState as immutable snapshot for the safety check
- use todoTracker.hasPublished instead of enabled for stranded-comment cleanup
- serialize onUpdate calls via inflightPromise chain with post-cancel guard
- add settled() to wait for in-flight updates before writing final summary
Made-with: Cursor
* address round-2 review: hasPublished after success, finalSummaryWritten flag
- set hasPublished only after onUpdate resolves (not before) so failed
writes are not counted as published
- add finalSummaryWritten flag to ToolState, set after successful
non-plan reportProgress; decouple cleanup detection from
todoTracker.enabled so it survives API failures where cancel() ran
but the write didn't succeed
Made-with: Cursor
OpenRouter is a separate model specifier, not an alternative key for
direct providers. Also skip the "pass it through in pullfrog.yml"
instruction when the key is already in the default workflow template.
Made-with: Cursor
* add OpenRouter proxy for managed model routing and OSS program
proxy layer that mints ephemeral OpenRouter keys for users without BYOK
API keys. two paths: pro plan users get their selected model proxied via
OpenRouter; OSS program repos (hard-coded allowlist in config.ts) get
free Claude Opus. BYOK env vars (PULLFROG_MODEL/OPENCODE_MODEL) always
take precedence.
frontend: OSS repos see a static "Opus (Free)" badge with the model
dropdown disabled and no API key requirement. all models now carry
openRouterResolve metadata for proxy target resolution.
Made-with: Cursor
* implement OSS program: proxy infrastructure for free model credits
server-side OSS allowlist determines eligible repos. action mints
ephemeral OpenRouter keys via OIDC-authenticated /api/proxy-token
endpoint (idempotent on runId, $10 per-key safety limit). keys are
disabled on workflow completion when no running refs remain. HWM-based
usage sync tracks cumulative spend per account.
schema: ProxyKey model, Account.usageUsd/activeKeyId, WorkflowRun.proxyKeyId
action: OIDC credential stashing, resolveProxyModel uses server oss flag
frontend: isOss flows from server page to components (no client allowlist)
Made-with: Cursor
* address PR review: repo cross-check, key retirement lifecycle, dead code removal
- proxy-token: verify runId belongs to OIDC-authenticated repo via repo relation
- add retireKey() shared primitive: disable in OpenRouter first, then mark disabled in DB
- rotateKey: retire old active key after swap to prevent orphans
- webhook: replace inline cleanupProxyKey with retireKey calls
- syncAccountUsage: skip disabled keys
- remove vestigial AccountPlan/plan field from action types
- add disabled field to ProxyKey schema + migration
Made-with: Cursor
* replace deprecated opencode/mimo-v2-flash-free with mimo-v2-pro-free
Made-with: Cursor
* fix migration ordering: rename disabled migration to sort after table creation
Made-with: Cursor
* squash proxy key migrations into single migration
Made-with: Cursor
* add preview repo to OSS allowlist for testing
Made-with: Cursor
* populate OSS allowlist from oss-program-invitees.json
Made-with: Cursor
* format oss-program-invitees.json
Made-with: Cursor
* add installed public repos to OSS allowlist
split ossRepos into three provenance-tracked lists:
- internalRepos (pullfrog, colinhacks, RobinTail)
- installedPublicRepos (external public non-fork repos with active installs)
- invitees (from oss-program-invitees.json)
also adds scripts/list-oss-candidates.ts to regenerate the installed list
Made-with: Cursor
* fix: resolve tokens before clearing OIDC env vars
resolveTokens → acquireNewToken → isOIDCAvailable() checks
ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN env vars. The new OIDC
stashing code was deleting them in restricted shell mode before
resolveTokens ran, causing it to fall through to the GitHub App
path which requires GITHUB_APP_ID/GITHUB_PRIVATE_KEY.
Made-with: Cursor
* derive proxy-token auth from OIDC claims, add ensureWorkflowRun upsert
- proxy-token no longer requires body.runId; uses claims.run_id + claims.repository
- shared ensureWorkflowRun upsert called from both webhook and proxy-token
- workflow_run_requested handler now eagerly creates WorkflowRun records
- eliminates race condition between webhook and action proxy-token call
Made-with: Cursor
* hardcode PULLFROG_ACCOUNT_ID, document preview debugging lessons
GitHub node IDs are constant — no reason for this to be an env var.
Removes the trailing-newline bug that caused P2025 errors.
Adds wiki docs on workflow testing, Vercel env gotchas, and Neon
preview branch discovery.
Made-with: Cursor
* fix: parse OpenRouter create-key response correctly
the API returns `key` at the top level, not inside `data`
Made-with: Cursor
* onboarding cards, unlock OSS model selection, simplify console
- add OnboardingCard component with two states: workflow install
and model+test (dispatches "Tell me a joke" for test run)
- replace PromptBox overlay gates with dedicated onboarding cards;
PromptBox is now just the form, always enabled
- use hasWorkflowRuns DB check to decide onboarding vs promptbox
- unlock ModelSelector for OSS repos (was locked to Opus badge);
resolve proxyModel from repo's selected model alias in run-context
- rename "API key" row to "Model costs" with pure client-side states:
OSS covered, auto-resolve, free model, BYOK with env var names
- add "(Recommended)" badge to model aliases with recommended: true
- remove OSS_MODEL_DISPLAY_NAME constant and secrets-fetching logic
Made-with: Cursor
* update stale xai model snapshot
Made-with: Cursor
* rename Permissions to Security, hide git push toggle, add shell isolation toggle with disabled state for public repos, remove opentoad agent name from logs
Made-with: Cursor
* chevron hover states, sidebar hooks/security entries
Made-with: Cursor
* address PR review feedback: rename recommended→preferred, fix dispatch orphan, update wiki docs
- rename `recommended` to `preferred` in model alias registry to distinguish
from the UI "Recommended" badge (which is hardcoded for opus + codex only)
- cancel precreated WorkflowRun when workflow lookup fails in dispatch-workflow
- replace run_sql/vercel env pull in wiki docs with neonctl + Prisma pattern
- extend scripts/neon-branch.ts to output DATABASE_URL via neonctl
- add Recommended badge to GPT Codex alongside Claude Opus in ModelSelector
Made-with: Cursor
- split triggers.mdx into direct-prompting, pr-reviews, issue-enrichment, coding-tasks
- rename manual-setup.mdx to headless-action.mdx (CI integration)
- reorganize sidebar into Getting started / Usage / Reference groups
- add redirects for /triggers and /manual-setup
- add PULLFROG_MODEL env var support across action, workflows, and docs
- rewrite models.mdx with aliases, free models, resolution chain, routers
- update all cross-references in app, components, and docs
Made-with: Cursor
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7
Made-with: Cursor
* fix stale agent/effort refs, add tests for askpass + model resolution
- reviewCleanup.ts: payload.agent -> payload.model, remove effort
- selectMode.ts PlanEdit: remove delegation/subagent/effort references
- pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY,
add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY)
- FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy
- new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen)
- new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override)
- new: models.test.ts (19 tests — parseModel, resolution, registry invariants)
- update models.dev snapshot
Made-with: Cursor
* fix changed-agents.sh to filter legacy agent files from CI matrix
legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not
exported from index.ts. changed-agents.sh now reads index.ts imports to
build the active agent set and treats changes to inactive files as
non-agent changes (opentoad canary only).
Made-with: Cursor
* remove MCP file tools, old agent harnesses, and obsolete security tests
ASKPASS-based git auth makes the old MCP file tool security layer unnecessary:
- token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it
- agents now use native file tools (OpenCode builtin read/edit)
deleted:
- action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory)
- action/mcp/index.ts (dead re-export)
- agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts
- opencode-runner.ts (inlined into opentoad.ts)
- security tests that validated MCP file tool restrictions
- commented-out three-step review flow (~300 lines)
- sanitizeSchema/wrapSchema dead code from mcp/shared.ts
- OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed)
updated test prompts to use generic file ops instead of MCP tool names.
restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense).
Made-with: Cursor
* bump actions/checkout v4 → v6 (node 24)
node 20 actions deprecated june 2, 2026.
Made-with: Cursor
* temporarily disable fail-fast on agnostic tests to debug checkout@v6
Made-with: Cursor
* re-enable fail-fast on agnostic tests
Made-with: Cursor
* fix test token mismatch: mint OIDC tokens scoped to target repo
CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit
the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on
every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so
ensureGitHubToken() mints a properly scoped token via OIDC.
Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming
instead of repeating it in every test file, and fixes preview-cleanup
to remove workers from all queues (not just name-matching ones).
Made-with: Cursor
* fix ensureGitHubToken to try OIDC when app credentials are absent
ensureGitHubToken only attempted token minting when GITHUB_APP_ID and
GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds
aren't exposed — so the guard prevented minting entirely.
Made-with: Cursor
* dead code cleanup: remove remnants of deleted agents, file tools, effort system
remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps,
orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale
opencode-runner wiki refs, deleted test file references, and MCP file tool
docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to
globalSetup (runs once before forks instead of per-file, 19s → 200ms).
Made-with: Cursor
* address review feedback: remove dead code, update stale references
- remove AGENT_OVERRIDE (only opentoad exists)
- remove shellToolName plumbing (always restricted shell)
- bump action version to 0.0.179
- remove CURSOR_API_KEY from all workflows/configs
- remove OPENCODE_MODEL_MINI/MAX from workflows/docs
- delete wiki/effort.md, rewrite docs/effort.mdx as "Models"
- rewrite wiki/modes.md: orchestrator/subagent → single agent
- simplify flag system: drop builtin flag extraction (debug, effort,
timeout, agent), keep custom flag replacement only
- reserve all legacy flag names to prevent custom flag conflicts
Made-with: Cursor
* regenerate lockfile after removing claude-agent-sdk and codex-sdk
Made-with: Cursor
* fix import ordering, add lockfile check to pre-push hook
Made-with: Cursor
* remove dead debug payload field, stale packageExtensions
Made-with: Cursor
* merge proc-sandbox and token-exfil into a single test
proc-sandbox and token-exfil were duplicative — both tested that
SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into
token-exfil with shell:restricted (which actually exercises filterEnv)
and the /proc attack vector hints from proc-sandbox.
Made-with: Cursor
* fix wiki adversarial.md to match actual tokenExfil validator
Made-with: Cursor
* fix: report errors to progress comment when agent fails without throwing
when an agent returns success: false without throwing (e.g., opencode exits
with code 0 despite provider errors), the catch block in main.ts is never
reached, leaving the progress comment stuck on "leaping into action."
two fixes:
- opencode: return success: false when 0 events processed and a provider
error was detected (converts silent failures to explicit failures)
- main.ts: after handleAgentResult, check if the progress comment was ever
updated — if not, report the error to the comment as a safety net
Co-authored-by: Cursor <cursoragent@cursor.com>
* address review: use result directly and force failure on unreported progress
* refactor: move safety-net logic into handleAgentResult
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Anna Bocharova <robin_tail@me.com>
* Stop using command-based logs for warnings and errors
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert
* tweak
* de-noise
* Remove redundant ts() timestamp prefix from log calls
* Restore timestamped logging and refine debug output routing.
Bring back timestamp prefixes for standard logs and make log.debug emit via core.debug when runner debug is enabled, while still surfacing debug lines for --debug runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Mateusz Burzyński <mateuszburzynski@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Log the instructions box immediately after instruction resolution in main, and standardize agent permission summaries to debug-level "disallowed built-ins" output to reduce confusion with pullfrog MCP tools.
Co-authored-by: Cursor <cursoragent@cursor.com>
* add workflows permission to git token and waitlist improvements
- add `workflows` to `InstallationTokenPermissions` type in both action and API token routes
- include `workflows: write` in the git token so agents can push workflow file changes
- add `githubFollowers` field to WaitlistSignup schema with migration
- add script to populate waitlist followers from GitHub API
- add frog-green-square-border logo asset
Co-authored-by: Cursor <cursoragent@cursor.com>
* improve CI, agent logging, token permissions, and delegation guardrails
- add format check and build step to root CI job
- standardize agent model/effort log lines across all agents
- fix GitHub App permissions types to match OpenAPI schema (workflows is write-only)
- improve delegation error message to prevent subagent recursion
- demote noisy OpenCode stderr to debug level
- add subagent delegation rules to resolved instructions
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix graphql partial error handling, update delegation message, add workflow_run fixtures
Co-authored-by: Cursor <cursoragent@cursor.com>
* remove module-level env var throws that break CI build
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix logging bug and type hole from PR review
- use batch-local notFound counter so per-batch log doesn't undercount
- add workflows to WorkflowTokenPermissions so wire type matches what action sends
Co-authored-by: Cursor <cursoragent@cursor.com>
* lazy-init appOctokit to fix next build without env vars
Co-authored-by: Cursor <cursoragent@cursor.com>
* drop pnpm build from CI test workflow
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix delegate-effort test regex to match actual log format, disable fail-fast for agnostic tests
the test was matching `running \w+ with effort=auto` but the actual log
line from shared.ts is `» effort: auto`. also temporarily set
fail-fast: false on action-agnostic so all failures surface at once.
Co-authored-by: Cursor <cursoragent@cursor.com>
* disable fail-fast in action workflow too, relax ci.test.ts to match
both workflow files now use fail-fast: false for agnostic tests so all
matrix jobs run to completion. the ci consistency test now checks that
the two workflows agree rather than requiring true.
Co-authored-by: Cursor <cursoragent@cursor.com>
* restore fail-fast: true now that all agnostic tests pass
Co-authored-by: Cursor <cursoragent@cursor.com>
* skip agent tests in CI when agent harness file didn't change
adds action/test/changed-agents.sh which reads the PR diff (via
dorny/paths-filter) and outputs only agents whose harness file was
modified. the action-agents matrix now uses this dynamic list instead
of a hardcoded array, so e.g. a PR touching only cursor.ts runs 6
jobs instead of 30.
Co-authored-by: Cursor <cursoragent@cursor.com>
* update ci.test.ts to validate dynamic agent matrix
the test now checks that the matrix references the changes job output
and that changed-agents.sh correctly discovers all agents.
Co-authored-by: Cursor <cursoragent@cursor.com>
* parallelize action jobs and use claude canary fallback for shared changes
runs action-agents in parallel with action-agnostic after root/changes, and updates changed-agents logic so shared or non-harness action runtime changes run only claude while harness-specific edits run only those changed agents.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* pin all CLI installations to explicit versions and use pro models by default
- codex: pin @openai/codex to 0.101.0 (was "latest")
- opencode: pin opencode-ai to 1.1.56 (was "latest")
- gemini: pin gemini-cli to v0.28.2 via new tag param on installFromGithub
- cursor: pin to 2026.01.28-fd13201 via direct tarball download (replaces curl install script)
- add installFromDirectTarball to install.ts for versioned tarball URLs
- gemini auto effort now uses pro-preview instead of flash-preview
- test runner model overrides updated to use pro-preview consistently
Co-authored-by: Cursor <cursoragent@cursor.com>
* increase activity timeout
* fix delegation timeout
* fix delegate timeouts
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* pin all CLI installations to explicit versions and use pro models by default
- codex: pin @openai/codex to 0.101.0 (was "latest")
- opencode: pin opencode-ai to 1.1.56 (was "latest")
- gemini: pin gemini-cli to v0.28.2 via new tag param on installFromGithub
- cursor: pin to 2026.01.28-fd13201 via direct tarball download (replaces curl install script)
- add installFromDirectTarball to install.ts for versioned tarball URLs
- gemini auto effort now uses pro-preview instead of flash-preview
- test runner model overrides updated to use pro-preview consistently
Co-authored-by: Cursor <cursoragent@cursor.com>
* increase activity timeout
* fix delegation timeout
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Replace Date.now() with performance.now() for duration measurements
- Import performance from node:perf_hooks in all affected files
- Update Timer and ThinkingTimer classes to use performance.now()
- Update activity tracking (markActivity, getIdleMs) to use performance.now()
- Update cache duration measurements to use performance.now()
- Update agent execution timing (cursor, opencode) to use performance.now()
- Update subprocess execution timing to use performance.now()
- Update API performance monitoring to use performance.now()
- Update prep phase timing to use performance.now()
- Update timer.test.ts to mock performance.now() instead of Date.now()
Benefits:
- Monotonic clock immune to system clock adjustments
- Higher precision (microsecond vs millisecond resolution)
- Purpose-built for performance measurement
Fixes#245
* fix lint.
* Round float durations to integers in logging
Preserve original behavior by rounding performance.now() float values
to integers when displaying/logging millisecond durations.
* fix lint.
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Robin Tail <robin_tail@me.com>
* upgrade Claude to Opus 4.6 with --effort max for --max mode
- mini: haiku → sonnet
- auto: opusplan → opus (Opus 4.6)
- max: opus → opus + --effort max (Opus 4.6 max effort)
- bump @anthropic-ai/claude-agent-sdk 0.2.7 → 0.2.39
Co-authored-by: Cursor <cursoragent@cursor.com>
* update action lockfile for claude-agent-sdk 0.2.39
Co-authored-by: Cursor <cursoragent@cursor.com>
* add tool_use_summary handler for SDK 0.2.39
Co-authored-by: Cursor <cursoragent@cursor.com>
* integrate gpt-5.3-codex with runtime model availability detection
checks GET /v1/models at agent start to determine if gpt-5.3-codex is
available for the API key, falling back to gpt-5.2-codex when it isn't.
model resolution runs concurrently with CLI install for zero added latency.
also bumps @openai/codex-sdk from 0.80.0 to 0.98.0.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(agents): add thinking time logging between tool calls
Adds a ThinkingTimer utility that tracks the gap between tool results and
the next tool call. When the gap exceeds 3 seconds, it logs the duration
with a stopwatch emoji (⏱️ 4.2s).
Uses performance.now() for high-resolution timing and Intl.NumberFormat
for rendering duration in seconds with optional fraction digits.
Integrated across all 5 agents: Claude, Codex, Cursor, Gemini, OpenCode.
Closes#127
* fix: adjusting tests for mocking performance.now.
* fix: reducing diff for claude.
* fix: rm unused args for claude.
* rm unused args for codex.
* fix: rm unused args for gemini.
* fix: rm unused args for opencode.
* mv THINKING_THRESHOLD.
* rev: I decided to pospone node:perf_hooks integration since it requires more comprehensive refactoring.
* fix: using Intl unit formatting.
* tests for ThinkingTimer.
* fix: narrow unit.
* fix: making durationFormatter a class instance property since using one agent per run.
* fix: inverting condition in markToolCall.
* thinking timer improvements and fix actions/checkout v6 auth
- thinking timer: use » chevron and "thought for X seconds" format
- thinking timer: add debug timestamps for sanity checking
- demote PID namespace isolation logs to debug
- remove redundant "setting up git authentication" log
- fix duplicate Authorization header with actions/checkout v6: clean up
includeIf credential entries that v6 persists via external config files
Co-authored-by: Cursor <cursoragent@cursor.com>
* standardize tool call log prefix to » double chevron
Co-authored-by: Cursor <cursoragent@cursor.com>
* update timer tests for new thinking log format
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Robin Tail <robin_tail@me.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Add `lint`, `lint:fix`, `format`, and `format:fix` package.json scripts
backed by biome. Add AGENTS.md rule for agents to run them after changes.
Fix all existing lint and format violations across the codebase.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Scope installation token permissions in restricted mode
In restricted/disabled bash mode, the installation token is now scoped
to match the workflow's permissions block. This preserves fork push
capability while limiting what the agent can do with the token.
- Read workflow permissions from pullfrog.yml at runtime
- Pass permissions to API when acquiring installation token
- Clear OIDC env vars in restricted mode to prevent token minting
- Simplify setupGit by moving token resolution to main.ts
* Address review feedback: fail closed with default permissions
- Add restrictive default permissions (contents:read, pull_requests:read,
issues:read) as fallback when workflow permissions can't be read
- Add support for job-level permissions via GITHUB_JOB env var
- Fix misleading comment about token resolution in restricted mode
- Add documentation about fork PR checkout behavior
* Simplify to separate git/MCP tokens without workflow permission scoping
- gitToken: minimal contents:write only (assumed exfiltratable)
- mcpToken: full installation token (not exfiltratable via MCP tools)
- Remove workflowPermissions.ts - security-conscious users can pass
their own token via GH_TOKEN or inputs.token
- Add type-safe InstallationTokenPermissions to github.ts and API route
* Rename `write` permission to `push` and remove vestigial tool blocking
The `write` permission was previously used to block local file write tools
in agents. This was security theater since bash can write files anyway.
Now `push` only controls the git token scope:
- push: enabled → contents:write (can push commits)
- push: disabled → contents:read (read-only, can't push)
Changes:
- Rename `write` to `push` in action.yml, Prisma schema, and all TS types
- Remove vestigial write tool blocking from all agents (claude, cursor,
gemini, opencode, codex)
- Add data-preserving Prisma migration using RENAME COLUMN
- Update UI: "Write files" → "Git push" with updated description
* add PID namespace isolation for bash sandbox
when running in CI, attempts to use unshare --pid to create a new PID
namespace for bash subprocesses. this prevents the /proc/$PPID/environ
attack where a malicious command could read secrets from the parent
process's environment.
the protection works by:
1. creating a new PID namespace (subprocess becomes PID 1)
2. mounting fresh /proc showing only sandbox PIDs
3. parent PIDs become invisible (PPID = 0, /proc/0 doesn't exist)
combined with filterEnv(), this provides complete protection against
/proc-based secret theft. falls back gracefully if namespaces aren't
available.
includes test script to verify the protection works.
* add PID namespace test to CI workflow
tests whether unshare --pid works on GHA runners out of the box,
and if not, whether enabling via sysctl helps. also runs the
pidNamespace.ts test to verify the full protection.
* fix pnpm setup and add procIsolation agent test
- fix pnpm/action-setup by specifying package_json_file path
- add procIsolation crossagent test that has agent attempt to
read secrets via /proc/$PPID/environ
- add procIsolation to CI test matrix
* add pid-namespace test job to main workflow
this job tests unshare --pid capabilities on GHA runners and runs
the pidNamespace.ts adhoc test to verify /proc isolation works
* test bubblewrap's sysctl approach for enabling namespaces
- write to /etc/sysctl.d/99-userns.conf and run sysctl --system
- try aa-complain on unshare binary
- more detailed diagnostics
* fix pidNamespace test and add sudo-unshare fallback for GHA
- fix reference error in pidNamespace.ts (renamed function but didn't update calls)
- add sudo-unshare as fallback method for GHA runners where unprivileged
namespaces are blocked but sudo is available
- update bash.ts to detect and use sudo unshare when unprivileged fails
* consolidate security docs and document PID namespace isolation
- update security.md with current implementation details
- document sudo unshare fallback for GHA runners
- add testing instructions for local Docker and CI
- add "Further Exploration" section with Landlock and path validation ideas
- delete bash-sandbox.md and landlock.md (consolidated into security.md)
* move procIsolation test to adhoc folder
the procIsolation test requires PID namespace capabilities that aren't
available in the Docker test environment. moved to adhoc/ so it's excluded
from default test runs and can be run explicitly when needed (e.g. via
the pid-namespace CI job or locally with --privileged docker).
* fix Docker test environment for PID namespace isolation
- add CI and GITHUB_ACTIONS to testEnvAllowList so sandbox detection runs
- add --privileged to Docker run for PID namespace support (unshare)
this fixes the test environment to properly test the sandbox. in production,
the action runs directly on GHA runner where sudo unshare works.
* fix getJobToken() to work in test environment
add fallback to GH_TOKEN and GITHUB_TOKEN when INPUT_TOKEN is not set.
this allows tests to run without requiring workflow-level token input.
the token resolution order is:
1. INPUT_TOKEN (from workflow `with: token:`)
2. GH_TOKEN (external token override)
3. GITHUB_TOKEN (pre-acquired in tests or from GHA env)
* security: filter secrets from all subprocess environments
- extract filterEnv() to shared utils/secrets.ts
- make $() utility filter secrets by default (git, npm, etc. don't need them)
- disable git hooks via core.hooksPath to prevent hook-based exfiltration
- git auth uses token embedded in URL, not env vars
this prevents malicious git hooks, npm postinstall scripts, and other
code execution vectors from exfiltrating GITHUB_TOKEN and API keys.
* docs: clarify defense-in-depth security model
update security.md to explain why BOTH layers are required:
- filterEnv(): cleans child's own /proc/self/environ
- PID namespace: hides parent's /proc entries
PID namespace alone isn't sufficient - with --mount-proc, the child
becomes PID 1, so /proc/1/environ is the child's OWN environment.
without filterEnv(), secrets would still be accessible there.
* add procSandbox crossagent test for PID namespace security
- add crossagent/procSandbox.ts: security test that instructs agent to try
various /proc attack vectors and validates sandbox blocks them
- update wiki/security.md: document PID namespace isolation details, add
verification commands, explain why sudo inside sandbox doesn't break security
- update docker.ts: use node:24 with sudo for GHA-like test environment
- update instructions.ts: allow disabling security messaging for pentests
- clean up adhoc test files (procIsolation.ts, securityAudit.ts)
the procSandbox test sets SANDBOX_TEST_TOKEN (matches *_TOKEN filter), gives
the agent explicit hints about attack vectors (/proc, sudo, nsenter, etc.),
disables prompt-based mitigations, and validates the secret is never leaked.
* move procSandbox test to agnostic/ (runs with one agent)
* WIP
* docs: add agent testing guide (pnpm play, Docker, pentesting)
* docs: add CI details to agent testing guide
* docs: add interesting findings and gotchas from pentesting
* improve test fidelity: auto-set CI=true, verify sandbox active
- docker.ts: always set CI=true in container so sandbox activates
- docker.ts: skip sudo/user setup if already done (faster reruns)
- bash.ts: export getSandboxMethod() for diagnostics
- bash.ts: add debug log when sandbox disabled
- procSandbox.ts: add sandbox_was_active check to catch vacuous passes
the CI=true change is critical: without it, PID namespace isolation
is skipped and security tests pass without actually testing anything.
* docs: update agent-testing.md with CI=true auto-set note
* docs: clarify log format is agent-specific
* fix git auth, simplify MCP tools, add adversarial tests
- fix gitWithAuth to use Basic auth format (Bearer doesn't work with git's http.extraheader)
- fix token.ts: push:restricted now correctly gets contents:write
- fix github.ts: pass permissions when acquiring installation tokens locally
- remove commit_files and create_branch MCP tools (redundant, don't require credentials)
- remove containsSecrets() - trivially bypassable, not sound security
- add agnostic adversarial tests: pushDisabled, pushRestricted, tokenExfil
- update instructions.ts to clearly list available git MCP tools
- add wiki/git.md documenting credential isolation and push permission tiers
- update wiki/docker.md with custom image considerations
- update wiki/agent-testing.md with adversarial testing patterns
* fix type errors after rebase
- change ResolveTokensParams.push from ToolPermission to PushPermission
- use tags: ["agnostic"] instead of agnostic: true in test files
* fix cleanup permission error in sandbox tests
when sandbox isolation is enabled (CI=true), files created by the unshare
subprocess may have different ownership, causing rmSync to fail with EACCES.
this error in the finally block was overriding the test's success result.
fix: wrap cleanup in try-catch and fall back to sudo rm if rmSync fails.
* Add adhoc
* Handle git config/remote bypasses
* add git hooks protection and simplify ToolState
- disable git hooks in restricted mode via -c core.hooksPath=/dev/null
- add gitHooks adversarial test to verify hook protection works
- unify prNumber/issueNumber into single issueNumber field
- add pushUrl to ToolState for push validation
- add generateTestMarker() for simpler single-agent test markers
- export SENSITIVE_PATTERNS and isSensitiveEnvName from secrets.ts
- remove redundant pidNamespace.ts (duplicated by procSandbox.ts)
- update documentation
* harden $git() auth: subcommand whitelist, binary tamper detection
- rename gitWithAuth() to $git() with explicit subcommand first arg
- restrict to "fetch" | "push" at type level (filters don't run for these)
- resolve git binary path at startup via resolveGit(), sha256 fingerprint
- verify hash before each $git() call to detect binary replacement
- rename disableHooks to restricted for cleaner semantics
- document filter exfiltration attack and empirical verification in wiki
* remove redundant pid-namespace CI job
the PID namespace isolation testing is now handled by
action/test/agnostic/procSandbox.ts via pnpm runtest agnostic
* fix push_branch for new branches and improve token leak detection
- getPushDestination now falls back to origin/<branch> when @{push}
is not configured (happens for new branches created locally)
- gitPerms validator now checks for actual token patterns instead
of matching "x-access-token" string in test instructions
* use kebab-case for test names
* simplify shell env API: "restricted" | "inherit" | object
replace passFullEnv boolean with cleaner env option that accepts:
- "restricted" (default): filterEnv() to prevent secret leakage
- "inherit": full process.env
- object: custom env merged with restricted base
* share EnvMode and resolveEnv between shell.ts and bash.ts
move shared env resolution logic to secrets.ts
* add env option to bash tool (default: restricted)
* delete agent-testing.md (renamed to adversarial.md)
* Add checkout tests
* reframe githooks test prompt to avoid claude safety refusal
claude was refusing to execute the test because the prompt used words
like "malicious" and "security testing". reframed as a debugging task
with innocuous env var name (TESTING_DEBUG_TAG) per adversarial.md guidance.
Co-authored-by: Cursor <cursoragent@cursor.com>
* clean up verbose token acquisition logs
move logging responsibility to call sites which have better context
(git token vs MCP token). remove redundant intermediate OIDC logs
and unused "(permission-scoped)" suffix.
Co-authored-by: Cursor <cursoragent@cursor.com>
* isolate agnostic tests with matrix strategy, fix .pullfrog-env secret leak
- split action-agnostic into per-test matrix jobs for isolated logs and filesystems
- only write explicitly opted-in env vars to .pullfrog-env via fileAgentEnv
(fixes token-exfil test where claude found SANDBOX_TEST_TOKEN on disk)
- mcpmerge test opts in via fileAgentEnv for cursor's repo-level MCP fallback
Co-authored-by: Cursor <cursoragent@cursor.com>
* remove env parameter from bash tool to prevent agents bypassing filterEnv
the bash tool exposed an `env` parameter accepting "restricted" | "inherit"
which allowed agents to pass env: "inherit" and see all secrets including
SANDBOX_TEST_TOKEN, bypassing the restricted environment filtering entirely.
env mode is now determined internally (always restricted).
Co-authored-by: Cursor <cursoragent@cursor.com>
* use pullfrog/test-repo for push tests to stop polluting main repo
push tests were creating branches and tags on pullfrog/app directly.
now all push tests (push-restricted, push-disabled, push-enabled,
git-permissions) target pullfrog/test-repo instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* use pullfrog/test-repo for all tests, not just push tests
no test should clone or operate on pullfrog/app directly.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix token scoping for test-repo and bash timeout defaults
- acquireTokenViaOIDC now includes GITHUB_REPOSITORY repo in token
scope so push tests work against pullfrog/test-repo
- bash tool default timeout: 120s -> 30s, cap: 600s -> 120s
- activity timeout: 30s -> 60s
- prevents hung bash commands (e.g. find /) from killing the agent
via activity timeout
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Colin McDonnell