Commit Graph

61 Commits

Author SHA1 Message Date
Mateusz Burzyński a7b8dcbced Rework incremental diffing (#499)
* Improve our deepening logic

* Use consistent SHA for PR-related operations in CheckoutPrTool

* compute `deepenDepth` at more appropriate time

* fix stale comment

* add comments for `alreadyOnBranch`

* ensure before sha is available

* small cleanup

* computeIncrementalDiff

* move the util

* improve algorithm

* improve algorithm further

* get rid of temp result array

* add comment

* compute incremental diff and updte instructions

* add comment

* update stale comment

* get rid of redundant rev-parse call

* improve comment

* strenghten the instructions

* make diff paths unique
2026-03-27 16:09:13 +00:00
Colin McDonnell 6d25adfd1a Agent & model refactor (#478)
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7

Made-with: Cursor

* fix stale agent/effort refs, add tests for askpass + model resolution

- reviewCleanup.ts: payload.agent -> payload.model, remove effort
- selectMode.ts PlanEdit: remove delegation/subagent/effort references
- pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY,
  add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY)
- FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy
- new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen)
- new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override)
- new: models.test.ts (19 tests — parseModel, resolution, registry invariants)
- update models.dev snapshot

Made-with: Cursor

* fix changed-agents.sh to filter legacy agent files from CI matrix

legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not
exported from index.ts. changed-agents.sh now reads index.ts imports to
build the active agent set and treats changes to inactive files as
non-agent changes (opentoad canary only).

Made-with: Cursor

* remove MCP file tools, old agent harnesses, and obsolete security tests

ASKPASS-based git auth makes the old MCP file tool security layer unnecessary:
- token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it
- agents now use native file tools (OpenCode builtin read/edit)

deleted:
- action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory)
- action/mcp/index.ts (dead re-export)
- agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts
- opencode-runner.ts (inlined into opentoad.ts)
- security tests that validated MCP file tool restrictions
- commented-out three-step review flow (~300 lines)
- sanitizeSchema/wrapSchema dead code from mcp/shared.ts
- OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed)

updated test prompts to use generic file ops instead of MCP tool names.
restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense).

Made-with: Cursor

* bump actions/checkout v4 → v6 (node 24)

node 20 actions deprecated june 2, 2026.

Made-with: Cursor

* temporarily disable fail-fast on agnostic tests to debug checkout@v6

Made-with: Cursor

* re-enable fail-fast on agnostic tests

Made-with: Cursor

* fix test token mismatch: mint OIDC tokens scoped to target repo

CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit
the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on
every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so
ensureGitHubToken() mints a properly scoped token via OIDC.

Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming
instead of repeating it in every test file, and fixes preview-cleanup
to remove workers from all queues (not just name-matching ones).

Made-with: Cursor

* fix ensureGitHubToken to try OIDC when app credentials are absent

ensureGitHubToken only attempted token minting when GITHUB_APP_ID and
GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds
aren't exposed — so the guard prevented minting entirely.

Made-with: Cursor

* dead code cleanup: remove remnants of deleted agents, file tools, effort system

remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps,
orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale
opencode-runner wiki refs, deleted test file references, and MCP file tool
docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to
globalSetup (runs once before forks instead of per-file, 19s → 200ms).

Made-with: Cursor

* address review feedback: remove dead code, update stale references

- remove AGENT_OVERRIDE (only opentoad exists)
- remove shellToolName plumbing (always restricted shell)
- bump action version to 0.0.179
- remove CURSOR_API_KEY from all workflows/configs
- remove OPENCODE_MODEL_MINI/MAX from workflows/docs
- delete wiki/effort.md, rewrite docs/effort.mdx as "Models"
- rewrite wiki/modes.md: orchestrator/subagent → single agent
- simplify flag system: drop builtin flag extraction (debug, effort,
  timeout, agent), keep custom flag replacement only
- reserve all legacy flag names to prevent custom flag conflicts

Made-with: Cursor

* regenerate lockfile after removing claude-agent-sdk and codex-sdk

Made-with: Cursor

* fix import ordering, add lockfile check to pre-push hook

Made-with: Cursor

* remove dead debug payload field, stale packageExtensions

Made-with: Cursor

* merge proc-sandbox and token-exfil into a single test

proc-sandbox and token-exfil were duplicative — both tested that
SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into
token-exfil with shell:restricted (which actually exercises filterEnv)
and the /proc attack vector hints from proc-sandbox.

Made-with: Cursor

* fix wiki adversarial.md to match actual tokenExfil validator

Made-with: Cursor
2026-03-12 05:22:51 +00:00
Mateusz Burzyński da72d0d6ee Fixed semantic conflict between #377 and #354 (#380)
* Move out checking out PRs from `etupGit` (#377 intent)

* Keep subagent-related changes from #377
2026-02-24 14:20:59 +00:00
Colin McDonnell b472aa1ba9 fix(opencode): add effort-aware model overrides and OpenRouter guidance (#354)
* chore: create empty commit for PR

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(action): trigger preview repo creation workflow

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(opencode): merge repo opencode config and log provider key presence

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(opencode): add effort-aware model override precedence

Add OPENCODE_MODEL_MINI and OPENCODE_MODEL_MAX support with fallback to OPENCODE_MODEL and auto-selection, and keep provider resolution aligned via inline OpenCode config when overrides are used. Update workflow env wiring, test allowlists, and docs (including OpenRouter setup guidance and sidebar ordering) to document the new behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(effort): clarify OpenCode precedence and add section link

Add a concise OpenCode precedence callout in the summary area and link directly to the OpenCode section for full details and examples.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Restructure dash (#372)

* Restructure dash

* WIP

* WIP

* refactor trigger UI: extract PR summary card, add mentions section, rename labels

Co-authored-by: Cursor <cursoragent@cursor.com>

* clean up console UI: remove info icons from section descriptions, rename mentions trigger

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix review feedback: layout, terminology, form scope

- extract console sidebar sections to module-level constant
- align three-column layout breakpoints to xl (match sidebar visibility)
- fix mixed shell/bash terminology in beta page
- scope FormProvider to trigger sections only, restore autoComplete="off"

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: create empty commit for PR

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(opencode): merge repo opencode config and log provider key presence

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(opencode): add effort-aware model override precedence

Add OPENCODE_MODEL_MINI and OPENCODE_MODEL_MAX support with fallback to OPENCODE_MODEL and auto-selection, and keep provider resolution aligned via inline OpenCode config when overrides are used. Update workflow env wiring, test allowlists, and docs (including OpenRouter setup guidance and sidebar ordering) to document the new behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore: update agent rules and OpenCode config docs

Align AGENTS guidance with current preferences and apply review-driven wording updates in OpenCode-related files.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-24 02:05:26 +00:00
Colin McDonnell 2017922780 Improve delegate (#377)
* Improve delegate

* fix stale log regexes in delegate tests and add test-coupling comments

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-23 23:41:27 +00:00
Colin McDonnell a7bd746f21 Restructure dash (#372)
* Restructure dash

* WIP

* WIP

* refactor trigger UI: extract PR summary card, add mentions section, rename labels

Co-authored-by: Cursor <cursoragent@cursor.com>

* clean up console UI: remove info icons from section descriptions, rename mentions trigger

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix review feedback: layout, terminology, form scope

- extract console sidebar sections to module-level constant
- align three-column layout breakpoints to xl (match sidebar visibility)
- fix mixed shell/bash terminology in beta page
- scope FormProvider to trigger sections only, restore autoComplete="off"

Co-authored-by: Cursor <cursoragent@cursor.com>

* Bump

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-23 23:34:29 +00:00
pullfrog[bot] 70f1c47a28 Audit core.warning/core.error usage (#269)
* Stop using command-based logs for warnings and errors

Co-authored-by: Cursor <cursoragent@cursor.com>

* revert

* tweak

* de-noise

* Remove redundant ts() timestamp prefix from log calls

* Restore timestamped logging and refine debug output routing.

Bring back timestamp prefixes for standard logs and make log.debug emit via core.debug when runner debug is enabled, while still surfacing debug lines for --debug runs.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Mateusz Burzyński <mateuszburzynski@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-02-19 23:11:47 +00:00
Colin McDonnell b6658ddbc1 improve agent CI matrix, token permissions, and waitlist follower backfill (#313)
* add workflows permission to git token and waitlist improvements

- add `workflows` to `InstallationTokenPermissions` type in both action and API token routes
- include `workflows: write` in the git token so agents can push workflow file changes
- add `githubFollowers` field to WaitlistSignup schema with migration
- add script to populate waitlist followers from GitHub API
- add frog-green-square-border logo asset

Co-authored-by: Cursor <cursoragent@cursor.com>

* improve CI, agent logging, token permissions, and delegation guardrails

- add format check and build step to root CI job
- standardize agent model/effort log lines across all agents
- fix GitHub App permissions types to match OpenAPI schema (workflows is write-only)
- improve delegation error message to prevent subagent recursion
- demote noisy OpenCode stderr to debug level
- add subagent delegation rules to resolved instructions

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix graphql partial error handling, update delegation message, add workflow_run fixtures

Co-authored-by: Cursor <cursoragent@cursor.com>

* remove module-level env var throws that break CI build

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix logging bug and type hole from PR review

- use batch-local notFound counter so per-batch log doesn't undercount
- add workflows to WorkflowTokenPermissions so wire type matches what action sends

Co-authored-by: Cursor <cursoragent@cursor.com>

* lazy-init appOctokit to fix next build without env vars

Co-authored-by: Cursor <cursoragent@cursor.com>

* drop pnpm build from CI test workflow

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix delegate-effort test regex to match actual log format, disable fail-fast for agnostic tests

the test was matching `running \w+ with effort=auto` but the actual log
line from shared.ts is `» effort:  auto`. also temporarily set
fail-fast: false on action-agnostic so all failures surface at once.

Co-authored-by: Cursor <cursoragent@cursor.com>

* disable fail-fast in action workflow too, relax ci.test.ts to match

both workflow files now use fail-fast: false for agnostic tests so all
matrix jobs run to completion. the ci consistency test now checks that
the two workflows agree rather than requiring true.

Co-authored-by: Cursor <cursoragent@cursor.com>

* restore fail-fast: true now that all agnostic tests pass

Co-authored-by: Cursor <cursoragent@cursor.com>

* skip agent tests in CI when agent harness file didn't change

adds action/test/changed-agents.sh which reads the PR diff (via
dorny/paths-filter) and outputs only agents whose harness file was
modified. the action-agents matrix now uses this dynamic list instead
of a hardcoded array, so e.g. a PR touching only cursor.ts runs 6
jobs instead of 30.

Co-authored-by: Cursor <cursoragent@cursor.com>

* update ci.test.ts to validate dynamic agent matrix

the test now checks that the matrix references the changes job output
and that changed-agents.sh correctly discovers all agents.

Co-authored-by: Cursor <cursoragent@cursor.com>

* parallelize action jobs and use claude canary fallback for shared changes

runs action-agents in parallel with action-agnostic after root/changes, and updates changed-agents logic so shared or non-harness action runtime changes run only claude while harness-specific edits run only those changed agents.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-16 22:36:29 +00:00
David Blass bc28c658f2 harden sandbox escape vectors for bash disabled/restricted modes (#257)
* harden sandbox escape vectors for bash disabled/restricted modes

block git config injection (-c flag as subcommand), dangerous subcommands
(config, submodule, rebase, bisect), code-executing arg flags (--exec,
--extcmd), .gitattributes/.gitmodules writes, and package lifecycle scripts.
add retry logic to test runner for transient failures. add security unit
tests and adhoc attack tests.

Co-authored-by: Cursor <cursoragent@cursor.com>

* only filter subcommands in nobash, remove nobash from ui

* use regex matching

* iterate on tests

* simplify githooks

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-11 02:02:41 +00:00
pullfrog[bot] fb80343ffd feat(agents): add thinking time logging between tool calls (#244)
* feat(agents): add thinking time logging between tool calls

Adds a ThinkingTimer utility that tracks the gap between tool results and
the next tool call. When the gap exceeds 3 seconds, it logs the duration
with a stopwatch emoji (⏱️ 4.2s).

Uses performance.now() for high-resolution timing and Intl.NumberFormat
for rendering duration in seconds with optional fraction digits.

Integrated across all 5 agents: Claude, Codex, Cursor, Gemini, OpenCode.

Closes #127

* fix: adjusting tests for mocking performance.now.

* fix: reducing diff for claude.

* fix: rm unused args for claude.

* rm unused args for codex.

* fix: rm unused args for gemini.

* fix: rm unused args for opencode.

* mv THINKING_THRESHOLD.

* rev: I decided to pospone node:perf_hooks integration since it requires more comprehensive refactoring.

* fix: using Intl unit formatting.

* tests for ThinkingTimer.

* fix: narrow unit.

* fix: making durationFormatter a class instance property since using one agent per run.

* fix: inverting condition in markToolCall.

* thinking timer improvements and fix actions/checkout v6 auth

- thinking timer: use » chevron and "thought for X seconds" format
- thinking timer: add debug timestamps for sanity checking
- demote PID namespace isolation logs to debug
- remove redundant "setting up git authentication" log
- fix duplicate Authorization header with actions/checkout v6: clean up
  includeIf credential entries that v6 persists via external config files

Co-authored-by: Cursor <cursoragent@cursor.com>

* standardize tool call log prefix to » double chevron

Co-authored-by: Cursor <cursoragent@cursor.com>

* update timer tests for new thinking log format

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
Co-authored-by: Robin Tail <robin_tail@me.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-10 06:17:43 +00:00
pullfrog[bot] 1d59fd3d21 feat: Lifecycle hooks (#219)
* flatten lifecycle hooks into RepoSettings string fields

replace the separate LifecycleHook model with setupScript and
postCheckoutScript string fields directly on RepoSettings. move the UI
into the Agent settings section alongside environment variables and
custom instructions. delete the standalone lifecycle-hooks API route,
component, and schema since the existing settings PATCH endpoint
handles the new fields automatically.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: pass env to lifecycle hook spawn so scripts can use package managers

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 07:16:14 +00:00
Colin McDonnell 3a7145db1a Scope installation token permissions in restricted mode (#226)
* Scope installation token permissions in restricted mode

In restricted/disabled bash mode, the installation token is now scoped
to match the workflow's permissions block. This preserves fork push
capability while limiting what the agent can do with the token.

- Read workflow permissions from pullfrog.yml at runtime
- Pass permissions to API when acquiring installation token
- Clear OIDC env vars in restricted mode to prevent token minting
- Simplify setupGit by moving token resolution to main.ts

* Address review feedback: fail closed with default permissions

- Add restrictive default permissions (contents:read, pull_requests:read,
  issues:read) as fallback when workflow permissions can't be read
- Add support for job-level permissions via GITHUB_JOB env var
- Fix misleading comment about token resolution in restricted mode
- Add documentation about fork PR checkout behavior

* Simplify to separate git/MCP tokens without workflow permission scoping

- gitToken: minimal contents:write only (assumed exfiltratable)
- mcpToken: full installation token (not exfiltratable via MCP tools)
- Remove workflowPermissions.ts - security-conscious users can pass
  their own token via GH_TOKEN or inputs.token
- Add type-safe InstallationTokenPermissions to github.ts and API route

* Rename `write` permission to `push` and remove vestigial tool blocking

The `write` permission was previously used to block local file write tools
in agents. This was security theater since bash can write files anyway.

Now `push` only controls the git token scope:
- push: enabled → contents:write (can push commits)
- push: disabled → contents:read (read-only, can't push)

Changes:
- Rename `write` to `push` in action.yml, Prisma schema, and all TS types
- Remove vestigial write tool blocking from all agents (claude, cursor,
  gemini, opencode, codex)
- Add data-preserving Prisma migration using RENAME COLUMN
- Update UI: "Write files" → "Git push" with updated description

* add PID namespace isolation for bash sandbox

when running in CI, attempts to use unshare --pid to create a new PID
namespace for bash subprocesses. this prevents the /proc/$PPID/environ
attack where a malicious command could read secrets from the parent
process's environment.

the protection works by:
1. creating a new PID namespace (subprocess becomes PID 1)
2. mounting fresh /proc showing only sandbox PIDs
3. parent PIDs become invisible (PPID = 0, /proc/0 doesn't exist)

combined with filterEnv(), this provides complete protection against
/proc-based secret theft. falls back gracefully if namespaces aren't
available.

includes test script to verify the protection works.

* add PID namespace test to CI workflow

tests whether unshare --pid works on GHA runners out of the box,
and if not, whether enabling via sysctl helps. also runs the
pidNamespace.ts test to verify the full protection.

* fix pnpm setup and add procIsolation agent test

- fix pnpm/action-setup by specifying package_json_file path
- add procIsolation crossagent test that has agent attempt to
  read secrets via /proc/$PPID/environ
- add procIsolation to CI test matrix

* add pid-namespace test job to main workflow

this job tests unshare --pid capabilities on GHA runners and runs
the pidNamespace.ts adhoc test to verify /proc isolation works

* test bubblewrap's sysctl approach for enabling namespaces

- write to /etc/sysctl.d/99-userns.conf and run sysctl --system
- try aa-complain on unshare binary
- more detailed diagnostics

* fix pidNamespace test and add sudo-unshare fallback for GHA

- fix reference error in pidNamespace.ts (renamed function but didn't update calls)
- add sudo-unshare as fallback method for GHA runners where unprivileged
  namespaces are blocked but sudo is available
- update bash.ts to detect and use sudo unshare when unprivileged fails

* consolidate security docs and document PID namespace isolation

- update security.md with current implementation details
  - document sudo unshare fallback for GHA runners
  - add testing instructions for local Docker and CI
  - add "Further Exploration" section with Landlock and path validation ideas
- delete bash-sandbox.md and landlock.md (consolidated into security.md)

* move procIsolation test to adhoc folder

the procIsolation test requires PID namespace capabilities that aren't
available in the Docker test environment. moved to adhoc/ so it's excluded
from default test runs and can be run explicitly when needed (e.g. via
the pid-namespace CI job or locally with --privileged docker).

* fix Docker test environment for PID namespace isolation

- add CI and GITHUB_ACTIONS to testEnvAllowList so sandbox detection runs
- add --privileged to Docker run for PID namespace support (unshare)

this fixes the test environment to properly test the sandbox. in production,
the action runs directly on GHA runner where sudo unshare works.

* fix getJobToken() to work in test environment

add fallback to GH_TOKEN and GITHUB_TOKEN when INPUT_TOKEN is not set.
this allows tests to run without requiring workflow-level token input.

the token resolution order is:
1. INPUT_TOKEN (from workflow `with: token:`)
2. GH_TOKEN (external token override)
3. GITHUB_TOKEN (pre-acquired in tests or from GHA env)

* security: filter secrets from all subprocess environments

- extract filterEnv() to shared utils/secrets.ts
- make $() utility filter secrets by default (git, npm, etc. don't need them)
- disable git hooks via core.hooksPath to prevent hook-based exfiltration
- git auth uses token embedded in URL, not env vars

this prevents malicious git hooks, npm postinstall scripts, and other
code execution vectors from exfiltrating GITHUB_TOKEN and API keys.

* docs: clarify defense-in-depth security model

update security.md to explain why BOTH layers are required:
- filterEnv(): cleans child's own /proc/self/environ
- PID namespace: hides parent's /proc entries

PID namespace alone isn't sufficient - with --mount-proc, the child
becomes PID 1, so /proc/1/environ is the child's OWN environment.
without filterEnv(), secrets would still be accessible there.

* add procSandbox crossagent test for PID namespace security

- add crossagent/procSandbox.ts: security test that instructs agent to try
  various /proc attack vectors and validates sandbox blocks them
- update wiki/security.md: document PID namespace isolation details, add
  verification commands, explain why sudo inside sandbox doesn't break security
- update docker.ts: use node:24 with sudo for GHA-like test environment
- update instructions.ts: allow disabling security messaging for pentests
- clean up adhoc test files (procIsolation.ts, securityAudit.ts)

the procSandbox test sets SANDBOX_TEST_TOKEN (matches *_TOKEN filter), gives
the agent explicit hints about attack vectors (/proc, sudo, nsenter, etc.),
disables prompt-based mitigations, and validates the secret is never leaked.

* move procSandbox test to agnostic/ (runs with one agent)

* WIP

* docs: add agent testing guide (pnpm play, Docker, pentesting)

* docs: add CI details to agent testing guide

* docs: add interesting findings and gotchas from pentesting

* improve test fidelity: auto-set CI=true, verify sandbox active

- docker.ts: always set CI=true in container so sandbox activates
- docker.ts: skip sudo/user setup if already done (faster reruns)
- bash.ts: export getSandboxMethod() for diagnostics
- bash.ts: add debug log when sandbox disabled
- procSandbox.ts: add sandbox_was_active check to catch vacuous passes

the CI=true change is critical: without it, PID namespace isolation
is skipped and security tests pass without actually testing anything.

* docs: update agent-testing.md with CI=true auto-set note

* docs: clarify log format is agent-specific

* fix git auth, simplify MCP tools, add adversarial tests

- fix gitWithAuth to use Basic auth format (Bearer doesn't work with git's http.extraheader)
- fix token.ts: push:restricted now correctly gets contents:write
- fix github.ts: pass permissions when acquiring installation tokens locally
- remove commit_files and create_branch MCP tools (redundant, don't require credentials)
- remove containsSecrets() - trivially bypassable, not sound security
- add agnostic adversarial tests: pushDisabled, pushRestricted, tokenExfil
- update instructions.ts to clearly list available git MCP tools
- add wiki/git.md documenting credential isolation and push permission tiers
- update wiki/docker.md with custom image considerations
- update wiki/agent-testing.md with adversarial testing patterns

* fix type errors after rebase

- change ResolveTokensParams.push from ToolPermission to PushPermission
- use tags: ["agnostic"] instead of agnostic: true in test files

* fix cleanup permission error in sandbox tests

when sandbox isolation is enabled (CI=true), files created by the unshare
subprocess may have different ownership, causing rmSync to fail with EACCES.
this error in the finally block was overriding the test's success result.

fix: wrap cleanup in try-catch and fall back to sudo rm if rmSync fails.

* Add adhoc

* Handle git config/remote bypasses

* add git hooks protection and simplify ToolState

- disable git hooks in restricted mode via -c core.hooksPath=/dev/null
- add gitHooks adversarial test to verify hook protection works
- unify prNumber/issueNumber into single issueNumber field
- add pushUrl to ToolState for push validation
- add generateTestMarker() for simpler single-agent test markers
- export SENSITIVE_PATTERNS and isSensitiveEnvName from secrets.ts
- remove redundant pidNamespace.ts (duplicated by procSandbox.ts)
- update documentation

* harden $git() auth: subcommand whitelist, binary tamper detection

- rename gitWithAuth() to $git() with explicit subcommand first arg
- restrict to "fetch" | "push" at type level (filters don't run for these)
- resolve git binary path at startup via resolveGit(), sha256 fingerprint
- verify hash before each $git() call to detect binary replacement
- rename disableHooks to restricted for cleaner semantics
- document filter exfiltration attack and empirical verification in wiki

* remove redundant pid-namespace CI job

the PID namespace isolation testing is now handled by
action/test/agnostic/procSandbox.ts via pnpm runtest agnostic

* fix push_branch for new branches and improve token leak detection

- getPushDestination now falls back to origin/<branch> when @{push}
  is not configured (happens for new branches created locally)
- gitPerms validator now checks for actual token patterns instead
  of matching "x-access-token" string in test instructions

* use kebab-case for test names

* simplify shell env API: "restricted" | "inherit" | object

replace passFullEnv boolean with cleaner env option that accepts:
- "restricted" (default): filterEnv() to prevent secret leakage
- "inherit": full process.env
- object: custom env merged with restricted base

* share EnvMode and resolveEnv between shell.ts and bash.ts

move shared env resolution logic to secrets.ts

* add env option to bash tool (default: restricted)

* delete agent-testing.md (renamed to adversarial.md)

* Add checkout tests

* reframe githooks test prompt to avoid claude safety refusal

claude was refusing to execute the test because the prompt used words
like "malicious" and "security testing". reframed as a debugging task
with innocuous env var name (TESTING_DEBUG_TAG) per adversarial.md guidance.

Co-authored-by: Cursor <cursoragent@cursor.com>

* clean up verbose token acquisition logs

move logging responsibility to call sites which have better context
(git token vs MCP token). remove redundant intermediate OIDC logs
and unused "(permission-scoped)" suffix.

Co-authored-by: Cursor <cursoragent@cursor.com>

* isolate agnostic tests with matrix strategy, fix .pullfrog-env secret leak

- split action-agnostic into per-test matrix jobs for isolated logs and filesystems
- only write explicitly opted-in env vars to .pullfrog-env via fileAgentEnv
  (fixes token-exfil test where claude found SANDBOX_TEST_TOKEN on disk)
- mcpmerge test opts in via fileAgentEnv for cursor's repo-level MCP fallback

Co-authored-by: Cursor <cursoragent@cursor.com>

* remove env parameter from bash tool to prevent agents bypassing filterEnv

the bash tool exposed an `env` parameter accepting "restricted" | "inherit"
which allowed agents to pass env: "inherit" and see all secrets including
SANDBOX_TEST_TOKEN, bypassing the restricted environment filtering entirely.
env mode is now determined internally (always restricted).

Co-authored-by: Cursor <cursoragent@cursor.com>

* use pullfrog/test-repo for push tests to stop polluting main repo

push tests were creating branches and tags on pullfrog/app directly.
now all push tests (push-restricted, push-disabled, push-enabled,
git-permissions) target pullfrog/test-repo instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* use pullfrog/test-repo for all tests, not just push tests

no test should clone or operate on pullfrog/app directly.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix token scoping for test-repo and bash timeout defaults

- acquireTokenViaOIDC now includes GITHUB_REPOSITORY repo in token
  scope so push tests work against pullfrog/test-repo
- bash tool default timeout: 120s -> 30s, cap: 600s -> 120s
- activity timeout: 30s -> 60s
- prevents hung bash commands (e.g. find /) from killing the agent
  via activity timeout

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-06 06:26:26 +00:00
David Blass 18ba8e5fd0 improve runtest, optimize CI batching (#210) 2026-02-01 21:48:53 +00:00
Mateusz Burzyński 2daab6fc78 Obtain job-level token by default for less privileged runs (#198) 2026-01-29 21:30:08 +00:00
Mateusz Burzyński 071e885d63 Add upload tool and related APIs (#187)
* Add utils for r2 upload

* Add the tool and new routes

* fix auth issue

* sign headers

* add comment

* use our own API key to auth signed uploads

* Restructure things slightly

* tweak

* tweak

* add comments

* tweak

* revert a thing

* twaek

* drop mime type filtering

* new incarnation of mime type filtering

* jsut allow all octet-streams

* simplify further

* tweak

* update lockfile

---------

Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-01-28 05:34:58 +00:00
Colin McDonnell 01ee59a96c Restrict github token (#140) 2026-01-21 02:17:46 +00:00
Colin McDonnell cb925556e8 refactor instructions to return object with full/system/user/event/runtime properties, fix duplicate modes and json prompt extraction (#110) 2026-01-16 21:43:54 +00:00
David Blass e4b086938e iterate on CI 2026-01-16 15:52:37 +00:00
David Blass f34379415e add per-agent smoke tests (#100) 2026-01-16 08:00:16 +00:00
Colin McDonnell 9e019d89d2 Clean up actions and payloads (#98)
* Clean up actions and payloads

* Clean up action

* Cleanup
2026-01-16 07:16:25 +00:00
Colin McDonnell 3539ddf943 Update committer email 2026-01-14 21:15:05 +00:00
Colin McDonnell 280bb7ef15 Fix vercel build 2026-01-13 08:11:59 +00:00
Mateusz Burzyński a483711fee Auto-retry ratelimited octokit requests 2026-01-07 17:17:20 +01:00
Colin McDonnell d5bec7499b Update review process 2025-12-21 22:23:18 -08:00
David Blass 5034ff8285 switch to start_dependency_installation and await_dependency_installation, fix action play.ts repo 2025-12-19 16:29:46 -05:00
David Blass adc87d8b64 check packageManager 2025-12-17 18:00:36 -05:00
David Blass 90ed2648be refactor main 2025-12-17 16:28:17 -05:00
Colin McDonnell 4826e9acb1 Clean up logs 2025-12-17 12:43:08 -08:00
Shawn Morreau 6716183068 Fix MCP file discovery errors (#9)
* fix tool errors

*QA
2025-12-17 11:29:27 -05:00
Colin McDonnell 9132a59758 Fix create_review and various opencode things 2025-12-16 22:14:41 -08:00
Colin McDonnell 1bff21f7fb overhaul git setup 2025-12-16 18:01:51 -08:00
Colin McDonnell f6ac916e22 Merge pull request #6 from pullfrog/fix-setup-git-auth-order
fix: move origin URL auth setup before git fetch in setupGit
2025-12-16 18:00:54 -08:00
Colin McDonnell 9a68a35ac6 No tags 2025-12-16 17:00:00 -08:00
Colin McDonnell db68424ffc fix: move origin URL auth setup before git fetch in setupGit 2025-12-16 16:51:53 -08:00
Colin McDonnell 853746ba65 Clean up fork setup 2025-12-16 00:15:57 -08:00
Colin McDonnell efb4ad186f Improve remote tracking 2025-12-15 23:56:47 -08:00
Colin McDonnell e383dd33dd Clean up destructuring 2025-12-15 23:32:02 -08:00
Colin McDonnell b833cdd4af 0.0.141 2025-12-15 23:22:30 -08:00
Colin McDonnell 26336d0ac2 Tool factories 2025-12-15 23:04:20 -08:00
Colin McDonnell 0fced1dfa6 Clean up init 2025-12-15 22:21:47 -08:00
Colin McDonnell a19ae49224 Determinstically set up PR branch 2025-12-15 21:12:55 -08:00
Colin McDonnell 2f16d2ef0e Improve repo setup with gh cli 2025-12-15 20:21:56 -08:00
Colin McDonnell b7511752b6 Improve PR review on external PRs 2025-12-15 19:10:10 -08:00
Colin McDonnell 59f85a9003 Switch to pullfrog.com 2025-12-04 16:40:10 -08:00
Colin McDonnell e5a83284df Tweak instructions, add git email 2025-12-04 14:47:51 -08:00
Colin McDonnell c8cbda6972 simplify initialization 2025-11-26 10:23:27 -08:00
David Blass aba21e7583 remove unnecessary git cleanup logic 2025-11-25 16:05:40 -05:00
David Blass ff375b97e4 fix local git setup 2025-11-25 16:02:08 -05:00
Colin McDonnell 595b246235 update instructions fixtures and comment handling 2025-11-20 18:58:20 -08:00
Colin McDonnell 0ce1d9fd7b deterministically set up working branch 2025-11-20 16:31:00 -08:00