110 Commits

Author	SHA1	Message	Date
Colin McDonnell	57f54e37c5	add bundled git-archaeology skill, auto-installed for opencode and claude (#565) ... * add bundled git-archaeology skill, auto-installed for opencode and claude ships a SKILL.md teaching agents the underused git history primitives (pickaxe -S/-G, -L for function/line ranges, --reverse blame, deleted-file recovery) so they stop scrolling git log -p when blame comes up empty. introduces a lightweight bundled-skill path alongside the existing addSkill (npx skills add) flow used for external skills like agent-browser. SKILL.md is inlined into dist/cli.mjs via esbuild's text loader and written to <home>/.agents/skills/<name>/SKILL.md at runtime — no network, no version drift, no per-run install cost. * fix: register vitest plugin to load .md as text for bundled-skill tests * fix: drop vite type import from vitest plugin (vite isn't a direct dep) * fix: load bundled skills via readFileSync so source mode works esbuild's text loader only applies to the npm-bundled dist/cli.mjs path. the preview / oss path runs cli.ts directly with node (PULLFROG_FORCE_LOCAL_CLI=1 in runCli.ts#runLocalCli), where node has no idea how to import .md files — ERR_UNKNOWN_FILE_EXTENSION crashes the action before any agent starts. switch to runtime readFileSync that checks both candidate locations: - source mode: <actionRoot>/skills/<name>/SKILL.md (relative to utils/skills.ts) - bundled mode: <distDir>/skills/<name>/SKILL.md (esbuild copies the tree) drops the no-longer-needed esbuild text loader, vitest .md plugin, and ambient *.md type declaration. wiki/skills.md updated with the why. * fix: write bundled skills to per-agent dirs so claude actually registers them	2026-05-04 18:49:50 +00:00
Colin McDonnell	f662b1a0c8	unify per-run token + cost accounting + persist to WorkflowRun (#547) ... * unify per-run token + cost accounting across agents every agent harness now logs the same 5-column (or 6 with cost) table and populates the same AgentUsage contract, regardless of agent or upstream provider. previously OpenCode and the Claude fallback path emitted a 3-col table whose "Input Tokens" was actually only the non-cached delta, silently dropping cache read/write — real runs were being reported at ~0.4% of their true input (e.g. one baseline showed Input=30 while step_finish events summed to cache_read=724,753). changes: - add logTokenTable helper in action/agents/shared.ts with stable columns: Input | Cache Read | Cache Write | Output | Total | Cost ($). cost column renders only when a value is known. - action/agents/opencode.ts: accumulate step_finish.part.tokens AND step_finish.part.cost (sourced from models.dev inside opencode — confirmed working across Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, and OpenRouter). drop the event.stats.total_tokens fallback since that payload has no cache breakdown. - action/agents/claude.ts: success-path now treats input_tokens as the non-cached field (matching OpenCode semantics), carries cache_read_input_tokens / cache_creation_input_tokens separately, and captures total_cost_usd from the final result event. the per-message fallback accumulator now captures cache fields too so it's no longer lossy when the result event never fires. - formatUsageSummary gains a Cost ($) column that matches the stdout table row-for-row; missing values render as "—". - scripts/token-usage.ts parses all three historical formats (new 5-col, legacy 4-col Claude success, legacy 3-col lossy) and explicitly flags the lossy runs instead of averaging misleading values. validation (pnpm play --local, identical "say hello" prompt): agent+model Input CacheR CacheW Output Total Cost OpenCode + Anthropic Sonnet 4.6 4 41,177 20,735 129 62,045 $0.0921 Claude CLI + Anthropic Sonnet 4.6 9 80,133 11,611 389 92,142 $0.0766 OpenCode + OpenAI codex-mini 10,893 46,976 0 606 58,475 $0.0059 OpenCode + Google Gemini 3 Flash — — — — — $0.0114 OpenCode + xAI Grok 4 Fast — — — — — $0.0035 OpenCode + DeepSeek Chat 18,854 0 0 1 18,855 $0.0053 OpenCode + Moonshot Kimi K2.5 — — — — — $0.0106 OpenCode + OpenRouter→Anthropic — — — — — $0.0617 OpenCode + OpenRouter→OpenAI — — — — — $0.0038 * isolate play.ts from developer gitconfig play.ts is a CI-emulator but inherits the developer's user- and system-scope gitconfig. a common local convenience — url."git@github.com:".insteadOf "https://github.com/" to force SSH auth — gets applied at read time on every git call inside the temp repo, causing `git remote get-url --push origin` to return an SSH URL instead of the stored HTTPS one. pullfrog_push_branch's validatePushDestination (correctly) treats that as tampering and blocks the push. the agent then burns the full MAX_COMMIT_RETRIES budget trying workarounds that can't beat a user-scope insteadOf rule, turning a trivial "say hello" run into a 1.35M-token session. point GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null inside run() so the play process and its spawned agent see the same empty gitconfig that a real CI runner would. CI has no rewrites, so this is a no-op there; dev machines get CI-identical git state. SSH client config (~/.ssh/config and keys) is separate from gitconfig and is unaffected, so setupTestRepo's SSH clone still works locally. setupGit only writes --local scope, so nothing downstream depends on user-scope values. verification: with the scratch repo cleaned up and this isolation in place, OpenCode + Anthropic on the same "say hello" prompt goes from 1,349,654 tokens / $2.00+ to 62,045 tokens / $0.0921 — no retry loop, no push blocks. * persist aggregated token + cost usage to WorkflowRun AgentUsage has been memory-only — rendered into the GitHub step summary and then discarded when the runner tears down. that made questions like "avg cost per customer per day" require log-spelunking. persist it: - add Int? columns for inputTokens / outputTokens / cacheReadTokens / cacheWriteTokens and a Decimal? costUsd column on workflow_runs. Int4's 2.1B ceiling is ~200x larger than any realistic run so BigInt would be overkill. costUsd uses the same default Decimal precision as existing money columns (accounts.usageUsd, proxy_keys.hwmUsage). - extend PATCH /api/workflow-run/[runId] to accept the new numeric fields alongside the existing artifact strings. per-field type validation ensures the allowlist stays scalar-safe and rejects negative / non-finite values. - generalize patchWorkflowRunFields in the action so it accepts a mixed string/number payload, and add an aggregateUsage(entries) helper that sums per-agent AgentUsage records into a single patch. - call the reporter from main.ts's outer finally block, gated on toolContext. this is the shared cleanup path that every agent implementation flows through — claude.ts, opencode.ts, and any future harness all push their AgentUsage into toolState.usageEntries via the same line 468, so one finally-block call covers them all. running in finally also means partial usage gets persisted even when the agent errored out mid-run. * anneal token + cost accounting follow-up polish from a review pass: - aggregate usage across commit-retry iterations inside each agent harness. previously runClaude / runOpenCode returned only the final retry's usage, so any run that hit the dirty-tree retry loop under-counted tokens and cost in both the stdout table and the WorkflowRun row. added a shared mergeAgentUsage helper in agents/shared.ts; both harnesses now fold each iteration's usage into a running total and return the sum. - scripts/token-usage.ts now handles the unified format with or without the Cost ($) column. previously the int-only number regex rejected decimals and the 5-cell length check rejected 6-cell rows, so logs from post-cost-tracking runs fell through to "no token table". the parser now accepts both 5- and 6-cell unified rows, splits int vs decimal cells, and averages reported Cost alongside the tokens. - PATCH /api/workflow-run/[runId] now rejects INT field values above INT4_MAX (2_147_483_647) so a malformed payload gets a clean 400 instead of propagating a Prisma error. also defends against a compromised runner sending a deliberately huge value. - clarifying comments: opencode.ts documents that step_finish.part.cost is a per-step delta (empirically verified), main.ts explains that toolState.usageEntries already carries merged per-retry usage so aggregateUsage just sums entries (one per agent.run()). - tests for aggregateUsage and mergeAgentUsage — 12 new cases covering empty / partial / multi-agent inputs and the "keep undefined" semantic that prevents spurious zeros from being persisted. - drop `as number` cast in logTokenTable — narrow via const instead. * anneal: clamp INT overflow + guarantee mergeAgentUsage immutability second review pass surfaced two defensive gaps: - a single token field exceeding INT4_MAX would pass the client but be rejected by the server's per-field validator, writing a partial row with some NULLs where sums belonged. clamp in aggregateUsage so the wire payload is always self-consistent across all numeric columns, with a loud warning so the clamp doesn't silently swallow weirdness. - mergeAgentUsage's single-sided branches returned the input reference. callers treat AgentUsage as immutable but future callers might not; always return a fresh shallow copy instead. two new tests guarantee the no-mutation-leak property. no behavior change in the happy path — INT4_MAX is ~200x the largest realistic per-run token count. * anneal: resilient usage persistence + cross-platform null device third review pass surfaced three small issues: - main.ts finally block: writeGitHubUsageSummaryToFile throwing would skip the WorkflowRun usage PATCH. both are independent best-effort cleanup tasks — wrap the former in catch so a filesystem failure doesn't block DB persistence. - AgentUsage.inputTokens had no jsdoc explaining that it's the full billable input (cached + non-cached). the same word "Input" means "non-cached only" in the stdout/markdown tables (derived by subtraction). document the semantic so dashboards querying WorkflowRun.inputTokens don't misinterpret it. - play.ts gitconfig isolation was hard-coded to "/dev/null" which doesn't exist on Windows. use `os.devNull` for cross-platform parity (resolves to `\\.\nul` on win32). the project is Linux-only in CI so this only helps local Windows contributors, but it's a zero-cost swap. also updated the finally-block caveat comment: usage is only pushed to toolState.usageEntries when agent.run() returns an AgentResult, not when the timeout race rejects — so timed-out runs don't persist partial usage. documented instead of trying to thread state through Promise.race. * anneal: NaN-guard cost accumulators + clarify inputTokens docs final polish from review round 4: - guard both cost accumulators (opencode step_finish.part.cost and claude result.total_cost_usd) with Number.isFinite. `typeof x === "number"` accepts NaN, and one NaN `+=` would poison the running total for the whole session. - reword prisma schema comment on WorkflowRun usage fields to call out that cacheReadTokens / cacheWriteTokens are SUB-totals within inputTokens (not additional tokens on top). prevents future dashboards from double-counting by ~2x when summing "total tokens used".	2026-04-20 21:27:54 +00:00
David Blass	57bd10d6dd	run-issues fixes: #5, #11, #12, #15, #16/#25, #20, #21, #22, #31 (#546) ... * fix(#15): precompute diff anchors in checkout_pr TOC * test(#15): update TOC snapshot for precomputed diff anchors * chore(tests): skip codex-mini-latest models.dev check + refresh latest-by-provider snapshot * fix(#22): add commitCount and commitLog to checkout_pr return * fix(#21): include PR body in checkout_pr return * fix(#5): force-fetch PR refspec to overwrite stale local branch * fix(#31): rename git tool parameter from subcommand to command * fix(#11): soft-fail post-checkout hook, bump timeout to 10min * fix(#16): strengthen diff file usage guidance Agent was bypassing diffPath and running `git diff` instead. Tighten instructions in `checkout_pr` result and remove the mixed-signal "log, diff" listing in the global Git guidance. `git log` and `git diff --stat` remain allowed for commit-range overview. * fix(#20): drop invalid inline review comments instead of failing review Previously, a single inline comment anchored outside a diff hunk would 422 the entire review submission. Pre-validate comments against the PR file patches via listFiles, drop the invalid ones, and append a note to the review body listing what was skipped. Include the dropped list in the tool response so the agent can retry targeted fixes. * fix(#12): stop MCP server on inner activity kill + filter reconnect noise Inner-activity-kill zombies were burning multi-hour runner time because mcp-proxy's SSE reconnect and provider-error retry lines kept the outer activity timer alive long after the agent subprocess was killed. - Filter [mcp-proxy] / "provider error detected" chunks so they don't count as outer-timer activity. - Add onActivityTimeout callback to spawn + thread through agent runs. - main.ts wires that callback to stop the MCP HTTP server (so reconnects finally fail instead of looping) and arms a 5min safety-net timer that force-rejects the outer timer if the agent promise is still pending. * audit: harden #12 lifecycle + cover #20/#12 with unit tests Bugs found during Ralph audit of the prior run-issues fixes: - main.ts's 5min safety-net setTimeout was never cleared on the happy path; also activityTimeout.stop() didn't null the internal rejectFn, so a late forceReject from the safety-net could still reject a long-resolved promise. Timer now cleared in finally; stop() now disarms forceReject. - mcp server disposal was non-idempotent, so the inner-kill path ran server.stop() twice once the outer `await using` block exited. Made the returned disposer idempotent. Tests: - action/mcp/review.test.ts: 14 tests for commentableLinesForFile (multi-hunk, no-count hunks, no-newline marker, empty) and validateInlineComments (file not in diff, wrong side, out-of-range line and start_line, partitioning batches, default side). - action/utils/activity.test.ts: 6 tests for isActivityNoise covering mcp-proxy lines, provider-error lines, mixed chunks, Buffer input. * audit(#22): cap commitLog at 200 + scope git-diff restriction to PR review - cap git log --oneline at 200 entries so a PR with thousands of commits cannot blow up the MCP tool response; expose commitLogTruncated so callers can warn the agent when the log was clipped - tighten instruction wording so `git diff` / `git diff --cached` remain available for inspecting an agent's own uncommitted changes, while PR review content must still come from diffPath Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#11,#22,#31): surface hook/commit warnings in instructions + polish git tool - append hookWarning + commitLogTruncated advisories to checkout_pr instructions so the agent actually sees the warning inline, not just as a field it may skip - fix stale 'subcommand' wording in git tool redirect for `pull` and in the `command` parameter description; the MCP parameter is named `command` now, and that's what the agent binds to Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(#20): reassign params.comments even when all inline comments dropped if every inline comment fails pre-validation, the earlier guard skipped reassigning params.comments, so the submission still carried the bad comments and GitHub 422'd on the whole review. always reassign to validation.valid so the downstream 'nothing left to post' skip fires and an otherwise-empty review is no-oped cleanly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#22): degrade gracefully when base ref isn't resolvable checkout_pr used to assume \`origin/<base>\` is always reachable, but it isn't guaranteed after a shallow fetch that only pulled down the PR head. Failing the whole checkout over metadata we added for ergonomics would be a regression, so wrap the rev-list / log in a try/catch and return empty commit metadata instead. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#12): anchor noise patterns to line start to avoid false positives before this, a line like "agent said: [mcp-proxy] was there" or "context: provider error detected in log" in real agent output would have been treated as noise and failed to reset the outer activity timer. both patterns now anchor at the start of the (optionally debug-timestamped) line, matching only lines mcp-proxy or our own log.info actually emit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#20): export and unit-test formatDroppedCommentsNote covers single-line `path:N`, multi-line `path:start-end`, and startLine==line fallback so changes to the dropped-comments note format surface in test diffs instead of only in GitHub UI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#20): cap dropped-comment note to stay under GitHub body limit a pathological run (agent emits hundreds of invalid inline comments on a huge PR and they all get dropped) would push the review body past GitHub's ~65KB limit and fail the whole submission with a body-too-long 422 — the exact all-or-nothing failure #20 was meant to prevent. cap the detail list at 50 entries with a "…and N more" line so the note stays bounded. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#20): distinguish binary/no-patch files in dropped-comment reason previously a comment on a binary file (or pure rename / mode-only change) was dropped with "line X is not inside a diff hunk", which misleads the agent into retrying with different line numbers. call out the no-textual-diff case explicitly so the agent knows to move that feedback to the review body instead. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#11): replace lifecycle timeout string-match with typed sentinel spawn() now rejects with SpawnTimeoutError (code === SPAWN_TIMEOUT_CODE or SPAWN_ACTIVITY_TIMEOUT_CODE) instead of a plain Error. executeLifecycleHook now branches on that code so rewording the error message in subprocess.ts can no longer silently misroute timeouts into the "transient — retry" warning. * audit(#12): route agent hung-vs-failed via typed SpawnTimeoutError claude.ts and opentoad.ts decide between "hung" and "failed" log wording based on the subprocess error. move them off the literal "activity timeout" substring match onto the same SPAWN_ACTIVITY_TIMEOUT_CODE sentinel used by lifecycle.ts so all three call sites agree on the source of truth. * audit(#20): delete leftover pending review when submit fails Why: `createAndSubmitWithFooter` creates a PENDING review first so we can mint Fix-links with the review ID, then submits. If submitReview fails (e.g. 422 from a race where the diff moved between pre-validation and submission), the draft was left on the PR. GitHub only allows one pending review per user, so the agent's retry would then fail with "already has a pending review" — an error the agent has no tools to clean up from. Best-effort cleanup: delete the pending draft on submit failure before re-throwing the original error, so retries start from a clean slate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#31): point agent to concrete alternative when rebase/bisect blocked Why: in disabled-shell mode, `git rebase` and `git bisect` are blocked as arbitrary-code-execution escape hatches. Previous error messages explained *why* but left the agent without a next step — especially painful right after the `pull` redirect, which suggested "merge or rebase locally." The agent would follow that advice, hit the rebase block, and loop without knowing what to try next. Now: rebase block explicitly says "use 'merge' instead"; bisect block notes that manual bisect is also unavailable through this tool; pull redirect no longer recommends rebase in shell-disabled contexts. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: import security tables into security.test to prevent drift Why: the security tests re-declared AUTH_REQUIRED_REDIRECT, NOSHELL_BLOCKED_SUBCOMMANDS, and NOSHELL_BLOCKED_ARGS inline with hand-copied message strings. When the runtime messages in git.ts were tightened (recent rebase/bisect guidance updates), the test copies drifted and tests validated a stale version of the logic while passing clean. A missing or mistyped entry in git.ts could therefore slip through. Now: export the tables from git.ts and import them into the test file. If a runtime message changes, the tests exercise the new string automatically; if an entry is added or removed, tests covering that command see the change without manual sync. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: widen pending-review cleanup to cover pre-submit throws getApiUrl() (invoked in footer build) can throw if API_URL is misconfigured, which would leak a pending draft between createReview and the previous submitReview try/catch. Move the try/catch to wrap the entire post-create body so any throw routes through deletePendingReview cleanup. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: reject leading-dash refs/branch names to block flag injection git's parseopt accepts options intermixed with positional args, so a ref like "--upload-pack=evil" passed to git_fetch could be parsed as a flag rather than a refspec. Add a narrow rejectIfLeadingDash helper to git_fetch (ref), delete_branch (branchName), and push_branch (branchName). HTTPS remotes ignore --upload-pack server-side, but the hygiene matters for defense in depth (ssh remotes, future code paths). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: validate the resolved branch in push_branch too When branchName is omitted, rev-parse surfaces the current branch name, which could start with '-' if git state was tampered with. Move the leading-dash check to after the branch is resolved so both the explicit and derived paths go through validation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: cache commentable-lines snapshot at checkout to match review anchor Review comments are anchored to checkoutSha (commit_id), but validation was hitting pulls.listFiles at review time — latest HEAD, not the SHA the agent actually reviewed. If the PR was updated mid-run, valid comments could be silently dropped (or invalid ones admitted). Snapshot the commentable lines during checkout_pr so review-time validation matches the anchor exactly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#12): route activity monitor's own debug output around the write wrap startProcessOutputMonitor monkey-patches process.stdout.write to mark activity, then called log.debug(...) every 5s to report idle time — which landed right back in its own wrapper, failed isActivityNoise, and called markActivity. with ACTIONS_STEP_DEBUG=true (common on reruns) the idle counter reset every interval and the timeout could never fire, re-creating the #12 zombie-run bug for any debug-enabled run. Fix: capture the original stdout.write and use it directly for the monitor's own diagnostics so they bypass the feedback loop. Added a tight-timeout regression test that asserts the timeout still rejects in debug mode. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#12): noise-filter subprocess.ts monitor logs so outer timer survives debug activity.ts's own monitor output already bypasses the wrap (c35cd3fb), but subprocess.ts's spawn activity timer uses log.debug — which goes straight through process.stdout.write and would still mark activity on every interval when debug logging is enabled. Pattern-filter those '(spawn|process) activity (check|timer|monitor)' lines in both local ([DEBUG] ...) and GH-runner (::debug::...) formats so they don't reset the outer agent-hang timer. Kept scoped to those specific monitor messages — a blanket [DEBUG] filter would silently classify any coincidentally-debug-prefixed agent output as idle, which is a worse failure mode than the one we're fixing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#11): surface spawn ENOENT-style errors in stderr buffer spawn() resolved with exitCode=1 and an empty stderr when the command itself couldn't start (missing binary, bad permissions). lifecycle.ts then reported 'output: (empty)' to the user, who was explicitly told 'retry if the failure looks flaky' — so every run hit the same wall with no diagnostic trail. Append the '[spawn] <cmd>: <node error>' line to stderrBuffer before resolving so the real cause (ENOENT, EACCES, …) flows through to the hook-warning message. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit(#11,#12): cover executeLifecycleHook typed-timeout routing the typed SpawnTimeoutError + sentinel-code branching introduced in d7ee7fd2 / ea8dd2c4 classifies hung vs failed lifecycle hooks — critical for whether agents retry — but had no unit coverage. add tests for all four branches (no script, exit 0, non-zero exit with retry-if-flaky guidance, timeout with do-NOT-retry guidance, transient spawn failure). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: re-verify clean tree after prepush hook the pre-prepush check guarantees we enter the hook with a clean tree, but if the hook writes tracked files (formatter, type generator, build artifacts), the push still only sends the pre-hook commit — the hook's edits silently disappear from the upstream branch while the tool reports "successfully pushed". add a post-hook status check so the agent sees the dropped mutations and can commit or discard them before retrying. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: reject push_tags refspec injection via ':' in tag name without tag validation, a tag like "foo:refs/heads/main" concatenated into "refs/tags/${tag}" becomes a valid <src>:<dst> refspec — git pushes the local refs/tags/foo's commit to remote main, bypassing push_branch's default-branch guard. same shape blocks leading '-' (flag injection) and other refspec metacharacters (~ ^ ? * [ \) via an allow-list regex. only reachable in push=enabled today, so this is defense-in-depth, but hardens the tool in case push_tags is ever exposed in restricted mode. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: stop pointing agents at an internal constant they can't change the lifecycle-hook timeout warning told agents to "bump LIFECYCLE_HOOK_TIMEOUT_MS" — but that's a hard-coded constant in the action, not something the agent or repo owner can tune. the agent would plausibly loop hunting for where to change it. redirect to the actual lever they control: ask the repo owner to simplify the hook. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: drop inverted inline-comment ranges locally with precise reason validateInlineComments only checked that both line and start_line anchor inside a hunk, not that start_line <= line. an inverted range (e.g. start=44, line=42) would pass local validation and GitHub would 422 with "invalid line numbers" — opaque to the agent and unfixable without reading docs. reject locally with a reason that names the constraint. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: don't let usage-summary write error mask main's outcome writeGitHubUsageSummaryToFile is called in main's finally block. it can throw on ENOSPC / EACCES / missing parent dir. a throw here propagates past the try's successful return or the catch's error return, hiding the actual run outcome behind an I/O failure on a purely informational file. swallow the write error (debug-logged) — the summary is nice-to-have, not load-bearing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: don't mislabel agent handler errors as JSON parse failures the onStdout event loop wrapped both JSON.parse and the handler call in one try/catch that logged every caught error as 'non-JSON stdout line'. if a handler threw (e.g. todowrite state shape drift), the error was silently classified as a parse error, making diagnosis impossible. split the try blocks so JSON errors and handler errors get distinct, identifying log lines. * audit: reject leading-dash PR refs before they reach git commands PR head/base refs come from GitHub and are attacker-controlled on fork PRs (the PR author picks headRef freely). they flow straight into `git fetch origin <ref>`, `git checkout -B <ref>`, and config writes. without a leading-dash check, a ref named like '-upload-pack=evil' could be parsed as a flag instead of a refspec. validate both refs at the top of checkoutPrBranch (before any async work) and cover the two attack shapes with unit tests. * audit: cover ActivityTimeout.stop()'s forceReject disarming main.ts's safety-net-timer path depends on ActivityTimeout.stop() nulling out rejectFn so a late safety-net fire after a successful agent run is a no-op. that behavior had no direct coverage — removing the \`rejectFn = null\` in stop() would silently break the happy path (unhandled rejection / spurious failure) without failing any test. add three tests covering: forceReject rejects with the reason, stop() disarms forceReject, and forceReject after timer rejection is an idempotent no-op. * audit: stabilize activity-timeout idleSec against late stdout race * audit: reject 0ms timeout parses to avoid insta-fail from '0m' * audit: surface raw GitHub error on review 422 instead of assuming anchor cause * audit: key commentable-lines cache by PR number to prevent cross-PR drift * audit: enumerate concrete 422 causes and name checkout_pr in review error * audit: stop shipping ralph-loop runtime state in PR history .claude/ralph-loop.local.md and .claude/ralph-loop-prompt.md were accidentally staged in an earlier audit commit. the .local.md suffix is conventional for gitignored runtime state, and the prompt file is per-run harness config — neither should merge to main. ignore the pattern and untrack the existing entries (files remain on disk so the active loop keeps working). * audit: pin commentable-lines cache to checkoutSha, not just PR number a second checkout_pr(N) call advances toolState.checkoutSha at line 305 or 334, then runs fetchAndFormatPrDiff + cache population at line 549. any throw between those two points (rate limit, 5xx, network blip) left the old snapshot keyed to (pullNumber=N) while checkoutSha now points at a different sha. review_pr(N) would reuse the stale snapshot, silently validating comments against the wrong anchor — the original failure this cache was meant to prevent. track commentableLinesCheckoutSha alongside the pull number and require both to match before returning the cache. if either has moved, fall back to listFiles like any other miss. * audit: auto-clear leftover pending review from killed prior runs a workflow timeout or OOM between createReview PENDING and submitReview leaves GitHub holding a pending draft. the next run hits GitHub's one-pending-per-user-per-PR limit and 422s at pending-create, with no way to recover short of a human cleaning up manually. catch 422 at pending-create, list the PR's reviews (GitHub only exposes our own pending to us, so the filter is safe), delete the leftover, and retry once. 404/422 on the cleanup are treated as no-ops (race with another concurrent cleanup or the draft was submitted); any other cleanup error rethrows so the real cause reaches the caller. * audit: extract + unit-test stranded-pending-review cleanup the recovery branch inside createAndSubmitWithFooter had no direct test coverage. a regression in any of its guards (status check, message match, listReviews filter, 404/422 tolerance, non-retryable rethrow) would silently cause either destructive deletes of unrelated reviews or the old failure mode where a stranded pending draft blocks every retry. extract to clearStrandedPendingReview so the cases can be exercised with a mocked octokit, and add tests for each branch — including the load-bearing negative cases (non-422 passthrough, non-pending-review 422 passthrough, no-leftover-found passthrough, non-retryable cleanup error passthrough). no behavior change at the call site. * audit: document concurrent-run race in clearStrandedPendingReview two runs on the same PR using the same GitHub App installation token would both see each other's PENDING draft via listReviews (GitHub exposes PENDING only to the author, and both runs share authorship). the loser's recovery path would delete the winner's active draft, causing the winner's submitReview to 404. no reliable in-request signal distinguishes a genuinely-stranded prior-run draft from an active peer's draft — PENDING reviews have no created_at, and the user field is the same bot in both cases. the correct fix is workflow-level concurrency (a per-PR concurrency key), not a heuristic here. document the limitation so future readers don't try to bolt on a broken heuristic. * audit: report signal-killed subprocesses as failures, not exit code 0 node's close event delivers (code=null, signal=<name>) when a child is killed by signal (OOM killer, segfault, external SIGTERM). the close handler captured only exitCode and coerced null to 0 via `exitCode || 0`, so lifecycle hooks killed by signal were silently reported as successful — lifecycle.ts's `if (result.exitCode !== 0)` check skipped the warning and callers proceeded as if setup/post-checkout/prepush had completed. now capture signal, append "killed by signal <name>" to stderr, and resolve with exitCode=1 when code is null but signal is set. adds a regression test that spawns `kill -KILL \$\$` and asserts a non-zero exit plus the signal-kill marker in stderr. * audit: untrack RUN_ISSUES*.md ralph-loop working docs same pattern called out in 4f14dbf1: these files are per-run harness state and analysis scratch, not merge-to-main deliverables. the TODO literally opens with "Ralph loop instructions:", so it's unambiguously in the same category as .claude/ralph-loop-prompt.md was. files stay on disk so the active loop keeps working. * audit: block refs/... + symbolic-ref bypass of default-branch guard push_branch's restricted-mode guard compared the resolved remoteBranch against defaultBranch with exact-string equality. an agent passing branchName "refs/heads/main" flowed through: rejectIfLeadingDash passed, getPushDestination's fallback preserved the refs/heads/main string as remoteBranch, so "refs/heads/main" !== "main" and the block was skipped, yet git push happily resolved refs/heads/main to the local main commit and pushed to the remote main branch. symbolic refs (HEAD / FETCH_HEAD / ORIG_HEAD / MERGE_HEAD) are the same class of bypass — they resolve to whatever commit they point at, unconstrained by the name-based guard. add rejectSpecialRef to enforce bare branch names at the tool entry, use it in push_branch and delete_branch. checkout_pr only ever assigns pr-<number> as the local branch, so nothing legitimate relied on the refs/... form here. * audit: keep original 422 visible when listReviews fails during pending-review cleanup if listReviews threw (e.g. transient 502, rate limit) during the stranded pending-review recovery path, the listing failure replaced the original 422 "pending review" error when it propagated up through the tool's outer catch. agents then saw a generic server error with no mention of the real blocker and stopped retrying the cleanup. now the listing failure is logged at debug but does not mask the original 422. the caller's retry re-attempts cleanup, which succeeds if the listing failure was transient. * audit: block default-branch deletion even under push: enabled delete_branch required push: enabled, but within that mode the agent could delete the default branch with no local guard. GitHub branch protection usually catches this at the remote, but not every repo has protection configured — and even when it does, relying on remote config for local safety is wrong. pushing to main is reversible (revert, force-push old HEAD); deleting main is not (reflog recovery only, 30-day window). block deletion of the resolved default_branch in DeleteBranchTool regardless of push permission. push: enabled authorizes pushes, not wholesale removal of the repository's primary branch. * audit: attach no-op catch to agentPromise so a late rejection can't crash cleanup agentPromise raced against activityTimeout.promise (and the --timeout timeoutPromise), both of which had .catch(() => {}) handlers. agentPromise did not. if a timeout won the race, agentPromise became stranded and its subsequent rejection was an unhandled rejection — under node 15+'s default unhandled-rejection policy that terminates the process, which would kill main() mid-cleanup and lose the error-reporting and usage-summary work queued in the catch/finally blocks. the race still sees the rejection (the original promise is shared); this catch only prevents node from treating a post-race rejection as unobserved. * audit: close push_branch refspec-injection via ':' / '+' in branchName rejectSpecialRef only forbade leading-dash, `refs/` prefix, and symbolic refs. git push accepts `[+]src[:dst]` refspec syntax, so an agent under push:restricted could smuggle a full refspec through branchName and bypass the downstream exact-string default-branch guard: "evil:refs/heads/main" → push local 'evil' to remote main ":refs/heads/main" → delete remote main ":other" → delete arbitrary branches (outside grant) "+main" → force-push refspec prefix reject ':', '+', '^', '~', '?', '*', '[', '\\', and whitespace — git's own check-ref-format forbids all of them in branch names, so the allow-list cannot false-positive against a legitimate branch. add regression tests. * audit: stop suggesting blocked 'rebase' in push_rejected advice under shell=disabled Why: when push fails with non-fast-forward, the advice told the agent to run 'git rebase origin/...'. In shell=disabled mode the git MCP tool blocks rebase (as an arbitrary-code-execution escape hatch), so the agent's only path forward was to hit the block, read the fallback message, and try merge — one wasted round trip. Now: under shell=disabled we directly suggest 'git merge origin/...', which always works. Under other modes the advice keeps the rebase/merge choice but leads with merge so the example is copy-pastable either way. * audit: harden includeIf cleanup against shell-injection via subsection names setupGit read `includeif.*` keys via `git config --get-regexp`, split on the first space, and fed the result into `execSync(\`git config --unset "${key}"\`)`. git config subsection values preserve arbitrary characters, so a crafted `[includeIf "gitdir:$(touch${IFS}/tmp/pwn)safe"]` entry round-trips through `--get-regexp` with its `$(...)` command substitution intact, survives the split-on-space filter (IFS-bypass leaves the payload space-free), and gets evaluated when interpolated into the shell command. Confirmed reachable as an RCE sink in local repro. Switch to `--get-regexp -z` (null-terminated, no ambiguity on whitespace) and call `$("git", ["config", "--unset-all", key])` which uses spawn-array and never hands the key to a shell. Extract the logic into `removeIncludeIfEntries` and add regression tests covering the injection payload, whitespace-in-subsection keys, benign entries, and the no-op case. * audit: clear SIGKILL escalator on clean SIGTERM exit the overall-timeout path scheduled a 5s SIGKILL follow-up without capturing the timer id. if the child cooperated with SIGTERM and `close` fired promptly, the escalator stayed pending in the event loop for up to 5s — delaying any subsequent clean shutdown (e.g. the main action exiting after an agent timeout) by that long. capture sigkillEscalatorId alongside timeoutId and clear it in both close and error handlers. regression test asserts the active-timer count does not grow past the pre-spawn baseline after a timed-out child exits on SIGTERM. * audit: correct rebase-availability hints to reflect shell=restricted the MCP git tool only blocks rebase when shell=disabled (NOSHELL_BLOCKED_SUBCOMMANDS check in GitTool). under shell=restricted, git({command: "rebase"}) works fine through the tool — NOSHELL_BLOCKED_SUBCOMMANDS doesn't apply. but two agent-facing messages implied rebase is only available with shell=enabled: - AUTH_REQUIRED_REDIRECT["pull"] said "rebase is only available when shell is enabled" - push-rejected integrateStep (non-disabled branch) said "(or 'rebase' if shell is enabled)" under shell=restricted, agents reading these would wrongly think they had to pick merge — pushing them toward merge commits when rebase would have been cleaner. the push-rejected branch is already ternary-gated on shell !== "disabled", so the qualifier there was just redundant noise. * audit: block difftool/mergetool under shell=disabled git difftool -x <cmd> is the short form of --extcmd. the args blocklist only matches --extcmd / --extcmd=*, so -x slipped through and let an agent run arbitrary commands even when shell=disabled. globally blocking -x would false-positive on git cherry-pick -x, which only appends metadata, so block difftool (and mergetool, same shape via mergetool.<name>.cmd) at the subcommand level instead. agents have no legitimate need for either — diffs go through diff/show and merges are resolved by file edits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * audit: recover stranded PENDING drafts on no-body createReview too The body path already clears a stranded PENDING draft from a prior crashed run via createAndSubmitWithFooter's own try/catch. The no-body path (approve-with-no-feedback or comments-only) called createReview directly — so a PR whose previous body-path run crashed between createReview(PENDING) and submitReview would permanently 422 any subsequent no-body review with "already has a pending review" until a body-path run happened to clear it. Factored out createReviewWithStrandedRecovery so both paths get the same recovery treatment, and added regression tests covering the no-stranded / stranded-and-retry / non-stranded-422-no-retry cases. * audit: reject timeouts past node's setTimeout ceiling a user-supplied timeout like "999h" parses fine (parseTimeString has no upper cap) but falls off the 2^31-1 ms limit setTimeout clamps to 1ms. the agent run would reject with "timed out after 999h" in a single tick. extract a resolveTimeoutMs helper that centralizes the zero/overflow/ unparseable checks (previously scattered behind inline boolean logic in main.ts) and cover the behavior with unit tests including the boundary value. * fix(#22): replace parameter property in SpawnTimeoutError node --experimental-strip-types rejects readonly/public/private param properties in constructors. tests run via node directly (no tsc), so CI was hitting ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX on every action-agents / action-agnostic job before any test code ran. declare the field and assign in the body instead. * audit: tighten git tool description and delete_branch refspec - `git` tool description previously implied `pull` had a dedicated MCP tool alongside `push_branch`/`git_fetch`. it doesn't — the redirect sends the agent back to the same git tool with `command: "merge"` (or `rebase`). update the description to teach this directly instead of letting agents discover it through the redirect error. - `delete_branch` now passes `refs/heads/${branchName}` to `git push --delete` so a same-named tag can't be silently deleted when both exist on the remote. `rejectSpecialRef` already guarantees the bare-name invariant, so the template construction stays injection-safe. Made-with: Cursor * audit: polish review.ts per anneal findings - drop `as "LEFT" | "RIGHT"` cast in `validateInlineComments` — octokit types `side?: string` at the createReview endpoint, so narrow via `c.side === "LEFT" ? "LEFT" : "RIGHT"`. no cast, no redundant annotation — TS infers the literal union from the ternary. - consolidate `clearStrandedPendingReview` from 3 params to 2 by folding `originalErr` into `params`, per AGENTS.md "max 2 parameters" rule. updates both call sites (`createReviewWithStrandedRecovery`, `createAndSubmitWithFooter`) and all 7 test paths. - upgrade `listReviews`-during-cleanup failure log from `log.debug` to `log.info` so operators not running at debug still see that recovery was attempted before the original 422 bubbles up. message now reads "surfacing original 422" to make the intent unambiguous. Made-with: Cursor * audit: signal partial commit metadata in checkout_pr previously a rev-list/log failure (e.g. shallow fetch where `origin/<base>` isn't reachable) silently returned `commitCount: 0, commitLog: ""` — indistinguishable from "this PR has no commits past base", which could mislead review reasoning about scope. add a `commitLogUnavailable: boolean` field to `CheckoutPrResult`, set when the rev-list/log calls throw. instructions footer now tells the agent to treat the values as "unknown" rather than "no commits" in that case. message phrased to cover the rare case where rev-list succeeds but git log throws (partial, not strictly zero) metadata. Made-with: Cursor * audit: fix parseDiffTocEntries to match production ' · diff-<sha>' TOC suffix the regex required $ right after the line range, but formatFilesWithLineNumbers in checkout.ts appends ` · diff-<sha256>` so agents have the GitHub "Files Changed" anchor precomputed. result: tocEntries was always empty on real PR reviews, breakdown.files was empty, and runDiffCoveragePreflight never fired its one-time "read the diff" nudge. add an optional suffix to the regex and a regression test that uses the exact production TOC shape. Made-with: Cursor * audit(#20): skip empty downgraded-APPROVE reviews before they 422 GitHub rejects `event: "COMMENT"` reviews with no body and no inline comments (HTTP 422 "Unprocessable Entity", verified empirically on repos/pullfrog/preview-546-run-issues-fixes/pulls/1). the runtime `prApproveEnabled` downgrade folds approved=true into event=COMMENT when the repo flag is off, so an agent asking to APPROVE a PR with no other feedback produces exactly that rejected shape — but the existing empty-review skip only fired for !approved cases, so the tool POSTed the doomed COMMENT, octokit returned what looked like a success-with- no-persisted-review shape, and agents reported a phantom reviewId that 404s on any subsequent GET. extract the skip decision into `reviewSkipDecision` and add a second branch for approved + !prApproveEnabled + empty. the function returns null when the review should be submitted, so a real bare APPROVE (approved + prApproveEnabled + empty) still goes through unchanged — GitHub accepts empty APPROVE reviews because the stamp itself is the content. surfaced in the PR #546 preview e2e run 24678139563 (reviewId 4141786854 reported by the agent but absent from every reviews listing). TC13 run 24680349445 re-ran the same scenario with prApproveEnabled=enabled and the review persisted correctly, isolating the cause to the downgrade + empty interaction. * audit(#31): drop misleading rebase mention from pull redirect AUTH_REQUIRED_REDIRECT["pull"] and the git tool's top-level description both said "use git_fetch then this tool with command 'merge' (or 'rebase' unless shell is disabled)". the "(or 'rebase' unless shell is disabled)" qualifier is active misinformation when the agent is already running under shell=disabled: rebase is blocked there by NOSHELL_BLOCKED_SUBCOMMANDS, so the suggestion sends the agent into a second block on the next tool call. 3b83ee97 already fixed this pattern for the push-rejected advice at line 248, but the pull redirect at line 280 and the tool description at line 351 were missed. the right copy isn't a conditional qualifier that agents have to parse against their own shell mode — it's just naming the one alternative that works everywhere (merge). agents under shell=restricted/enabled who want rebase can invoke it directly; the redirect doesn't need to advertise it. verified in preview e2e run 24679728733 (TC8 probe 6) where the agent correctly captured the verbatim redirect message under shell=disabled and explicitly flagged the "(or 'rebase' unless shell is disabled)" clause as confusing — the new test in security.test.ts asserts the message names merge and never rebase in every shell mode. * audit: drop vestigial entry/post references + add preview-546 settings util followup to d79860c6 "refactor: flatten action entrypoints" (Apr 10), which moved action.yml from built `entry`/`post` files to source `entry.ts`/`post.ts` but left three stale references lying around: - .gitignore: `action/run/entry` / `action/dispatch/entry` paths no longer exist anywhere in the build. - .github/workflows/pull-from-action.yml: agent instruction told the upstream sync agent to "Ignore `entry` files (they are built artifacts and .gitignored in this repo)". there are no built entry artifacts anymore — entry.ts is source. - .cursor/settings.json: search.exclude pattern "**/entry" excluded the old built files that no longer exist. none of these were load-bearing on their own, but the same drift had already broken preview e2e end-to-end: the pullfrog/template workflow's three-file copy step (cp .../entry, cp .../post) silently failed with cp: no such file on every preview PR since Apr 10. that template fix went to pullfrog/template@7ec7c8d and the preview-546 mirror at @17ab585, which is what unblocked this PR's full e2e validation. also adds scripts/preview-546-settings.ts, the helper used during the e2e validation to show/set/reset DB-level repo settings on the Neon preview branch (push, shell, prApproveEnabled, hook scripts). scoped to this preview repo ID so it can't accidentally mutate prod. * audit(#11): scope removeIncludeIfEntries to repoDir under inherited GIT_* the function takes `repoDir` as the target, but plain execSync / $(...) inherit GIT_DIR, GIT_WORK_TREE, and GIT_INDEX_FILE from the parent process — and `git config --local` honors GIT_DIR over cwd. when this runs as a child of another git invocation (notably the pre-push hook, but also any future caller embedded inside a git subcommand), the cleanup silently targets the outer repo instead of repoDir. latent today because the real caller is ASKPASS setup, which runs before any git-subcommand ancestor exists, but the function's contract still promised the wrong thing — and the test suite hit exactly this bug when invoked through `git push`. - envScopedToRepo() strips GIT_* before both the get-regexp and unset calls, so cwd wins. - swap the $(...) shell helper for execFileSync on the unset call. $() would merge our scoped env with a "restricted" base that's tuned for hook execution (no tokens) — overkill here and it re-introduces the shell-vs-argv distinction this function was explicitly hardened against in a9aa3b2b. execFileSync with argv is the right tool for a call where the key can contain arbitrary characters. - setup.test.ts also strips GIT_* in its own execSync harness so the suite passes identically under `pnpm vitest run`, `pnpm -r test`, and `git push`'s pre-push hook. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-04-20 21:12:17 +00:00
Colin McDonnell	6d0254c7b8	pass --disallowedTools as a single comma-separated arg ... claude-code's commander parser treats --disallowedTools as variadic <tools...>, which silently absorbs extra tokens but may not enforce them as reliably as a single comma-separated value. switch to the form the CLI help documents ("Bash,Agent(Bash)") to make the deny list unambiguous.	2026-04-16 23:38:42 +00:00
Colin McDonnell	56a5d29598	add diff coverage preflight for PR review submissions (#544) ... * add one-time diff coverage preflight for PR reviews track diff read coverage from agent tool-use events and run a one-time pre-flight before review submission, with explicit coverage skip reasons for low-value files like lockfiles. Made-with: Cursor * add manual dispatch fallback for preview deploy workflow allow preview repo and preview sync jobs to be run via workflow_dispatch with explicit PR number and branch inputs, so preview provisioning can be retriggered when pull_request events fail to fire. Made-with: Cursor * fix manual preview dispatch PR input wiring use normalized PR number and branch env values for comment creation and script env wiring so workflow_dispatch preview runs can create and update PR-specific preview resources. Made-with: Cursor * remove obsolete snapshots invalidated by checkout instructions change * fix diff coverage read offset handling and add local sanity-check guidance normalize read offset semantics for diff coverage tracking, reuse shared range counting in review preflight, add focused diff coverage unit tests, and document the local play.ts testing workflow in AGENTS.md. Made-with: Cursor * add regenerated mcp test snapshots capture snapshot files generated by the review comment and checkout formatting tests during pre-push validation so the branch remains clean and reproducible. Made-with: Cursor * add diff coverage preflight instrumentation logs log diff coverage initialization in checkout_pr and emit preflight state/breakdown diagnostics in create_pull_request_review to debug missing coverage enforcement in preview e2e runs. Made-with: Cursor * add env override to force local cli execution in action runtime support explicit local-cli execution via PULLFROG_FORCE_LOCAL_CLI so preview workflows can run branch action code instead of the npm fallback package during e2e debugging. Made-with: Cursor * add preview e2e debugging learnings for action runtime validation capture the preview execution-path gotchas and one-time preflight verification pattern in AGENTS.md so future investigations validate the real runtime and avoid npm fallback confusion. Made-with: Cursor * reduce diff coverage log noise while preserving failure visibility downgrade verbose diff coverage lifecycle diagnostics to debug, keep a concise info-level pre-flight failure signal, and document preview runtime debugging learnings in AGENTS.md. Made-with: Cursor * WIP * tune sync.md: ff override + softer overlap verification Made-with: Cursor * chore: bump models snapshot for claude-opus-4-7 Made-with: Cursor * rip out coverage_skips waiver from diff coverage pre-flight Made-with: Cursor --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>	2026-04-16 21:51:44 +00:00
Colin McDonnell	5e6ff67623	move models.dev drift tests to main-only; add per-alias live smoke matrix ... PR CI kept breaking on upstream catalog drift (new model ships on models.dev, OpenRouter renames an id, etc.) — failures unrelated to the PR's contents. split the model-alias test suite so PRs only see pure-logic checks, and push the external-state drift + end-to-end coverage to main. test organization: - action/test/models.test.ts keeps pure invariants: openRouterResolve completeness and fallback-chain resolution. runs on every PR. - action/test/models-catalog.main.test.ts gets the 4 network-dependent describes (models.dev validity x2, OpenRouter API validity, latest-model snapshot). runs only on main push via a dedicated vitest config (vitest.main.config.ts + `pnpm test:catalog`). new CI jobs in .github/workflows/test.yml: - models-catalog: `pnpm test:catalog` on every main push. detects upstream catalog drift so we can react at the next convenient window. - models-live: 38-entry matrix that invokes the agent harness end-to-end against the real provider for each alias in models.ts. generated from action/test/list-aliases.ts. runs only on main push AND only when resolution-affecting files changed (action/models.ts, action/package.json, action/agents/**) — the exact shape of the opus 4.7 incident. test/run.ts: PULLFROG_MODEL now flows through from process.env so the live matrix can pin an alias per job without the per-agent default clobbering it. Made-with: Cursor	2026-04-16 21:10:15 +00:00
Colin McDonnell	4b3c5ca905	rename agent key to opencode and add skill invocation coverage (#541) ... * rename agent key to opencode and add skill invocation coverage. add skill-invoke tests for claude and opencode with local play-based validation signals, update CI matrices, and include the current tracked refactors in this branch for review. Made-with: Cursor * exclude agent-specific skill-invoke tests from wrong agent in CI matrix * address review follow-up and preserve workflow run UI tweak. switch changed-agents ci coverage to exercise the opentoad implementation path while keeping the opencode expectation, and include the local workflow run client interaction updates requested on this branch. Made-with: Cursor * remove opentoad agent filename from runtime. rename the opencode harness implementation file from opentoad.ts to opencode.ts and update ci coverage input accordingly so action code no longer carries the old filename. Made-with: Cursor * ensure security prompt bypass is set on every test fixture. this keeps adversarial and permissions harnesses from being blocked by the default prompt-injection refusal path during CI security tests. Made-with: Cursor --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>	2026-04-15 19:38:36 +00:00
Colin McDonnell	b9b6503315	reorder prompt sections: task-first with dynamic TOC (#513) ... * reorder prompt sections: task-first with dynamic TOC put the actual task at the top of the prompt for primacy, add a dynamic table of contents, and push system/runtime metadata to the end. new section order: TOC → YOUR TASK → PROCEDURE → EVENT CONTEXT → SYSTEM → LEARNINGS → RUNTIME Made-with: Cursor * enforce clean working tree: continue session if agent leaves uncommitted changes after each agent run, check `git status --porcelain`. if dirty, resume the same session with instructions to commit on a new branch, push, and open a PR. retries up to 3 times before giving up. - claude code: capture session_id from result event, use --resume <id> - opencode: use --continue to resume the last session - remove --no-session-persistence from claude (needed for --resume) - update Task mode to clarify branch/push/PR is the default finalize step Made-with: Cursor * log full prompt in collapsible group for debugging Made-with: Cursor * fix: format tool refs in buildCommitPrompt via formatMcpToolRef * enforce clean git status: general instructions, stop hook, and Task mode Made-with: Cursor * fix: rename stale titleBody references after body leak fix Made-with: Cursor --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>	2026-04-04 19:37:44 +00:00
Colin McDonnell	8cd36d221a	sandbox native filesystem tools to prevent /proc/self/environ exfiltration (#509) ... * sandbox native filesystem tools to prevent /proc/self/environ exfiltration the agent's native Read/Grep/Edit tools can bypass the shell sandbox by reading /proc/self/environ directly. this adds agent-native filesystem restrictions using the highest-precedence, non-overridable config for each CLI: OpenCode: OPENCODE_PERMISSION env var with external_directory deny-all + /tmp allow, plus deletion of untrusted .opencode/plugins/ and .opencode/tools/ before launch. Claude Code: managed-settings.json at /etc/claude-code/ with denyRead, permissions.deny, allowManagedPermissionRulesOnly, allowManagedHooksOnly. also --setting-sources user and --disallowedTools path patterns as belt-and-suspenders. Made-with: Cursor * add Glob to Claude Code /proc and /sys deny lists closes gap identified in review — Glob can enumerate /proc entries. added to both managed-settings.json permissions.deny and --disallowedTools. Made-with: Cursor * run token-exfil test with both agents, hint at native /proc reads changed tag from "agnostic" (opentoad-only) to "security" so the test runs with both opentoad and claude. updated prompt to explicitly instruct the agent to try reading /proc/self/environ via native Read tool. added API keys to action-agnostic CI job for claude support. Made-with: Cursor * move token-exfil to crossagent matrix, remove redundant permissions.deny - moved token-exfil from agnostic/ to crossagent/ so it runs via the agent matrix (claude + opentoad in parallel) instead of sequentially - removed permissions.deny per-tool rules from managed-settings.json; sandbox.filesystem.denyRead is the single enforcement mechanism - reverted action-agnostic env vars to minimal set - updated wiki to match Made-with: Cursor * document post-spawn API key deletion analysis in security wiki evaluated whether API key env vars can be deleted from agent processes after spawn. OpenCode snapshots env at startup (safe to delete), but Claude Code re-reads process.env per request (not viable). documented as further exploration item with per-agent breakdown and caveats. Made-with: Cursor * fix stale tokenExfil path references in wiki docs moved from test/agnostic/ to test/crossagent/ in directory tree and adversarial test example. Made-with: Cursor * revert accidental prisma.config.ts changes Made-with: Cursor * hardcode PULLFROG_MODEL per agent in test runner to avoid DB model mismatch when PULLFROG_AGENT forces a specific agent, the DB-configured model may belong to a different provider (e.g. openai model with claude agent). PULLFROG_MODEL short-circuits the DB slug resolution entirely. Made-with: Cursor	2026-04-02 22:31:41 +00:00
Colin McDonnell	36cc5cde14	Code quality sweep: 30+ bug fixes, security hardening, and UX improvements (#507) ... * Update waitlist, run ralph experiments * fix PR files pagination: use octokit.paginate() for >100 files * fix garbled FAQ answer on landing page * track cache read/write tokens in OpenCode agent usage * wrap dispatch() calls in try/catch to prevent webhook retries on transient failures * replace raw error messages with generic responses in API routes * guard request.json() calls with try-catch returning 400 on malformed bodies * log warning when GraphQL review thread/comment counts hit pagination limits * reduce review comment cache TTL from 24 hours to 10 minutes * use select instead of include for proxyKey in workflow run queries * align Claude agent activity timeout to 5 minutes to match OpenCode agent * add in-memory dedup for PR close webhooks to prevent duplicate indexing * extract isPullfrogLogin() helper for shared Pullfrog detection logic * check response.ok on log fetch in checkSuite.ts * add 10s timeouts to checkSuite API calls and log fetch * parallelize proxy key usage API calls with Promise.allSettled * fix three typos on landing page: colleage, dectects, reponse * move MAX_STDERR_LINES constant to shared.ts * add indexes on Repo.accountId and PFUser.accountId FK columns * remove unused Permission enum from Prisma schema * populate author and keywords in action/package.json * use crypto.timingSafeEqual for all secret comparisons * add missing env vars to globals.ts: R2, webhook, and API secrets * remove commented-out UserRepo model from Prisma schema * replace console.log/error with log utility in production API routes * replace catch(error: any) with proper type guards in getUserRole * remove stale TODO comment on console page * handle repository_transferred webhook to update owner * show toast.error instead of console.error on mode/workflow mutation failures * add Space key handler for keyboard navigation on workflow run links * replace role=link spans with button elements for proper accessibility * add root 404 page with Pullfrog branding * update ISSUES.md: mark completed items * mark remaining low-priority UX items as addressed * add error logging alongside toasts, add check script, update ralph commands * address review feedback: squash migrations, fix try/catch scope, wire up globals consumers - squash drop_permission_enum migration into add_indexes migration (one migration per PR) - move getPullRequest() outside try/catch in mention handler so errors aren't mislogged as "dispatch failed" - restore key ID in proxyKeys.ts Promise.allSettled error log - remove accidental asdf.txt and ralph.md files - wire up globals.ts exports to consumers (r2-uploads, r2-private, verifyHookdeckSignature, sync-usage, forwardPreviewWebhook, dispatch-workflow) Made-with: Cursor * update model snapshot (qwen3.6-plus-preview renamed to qwen3.6-plus) Made-with: Cursor	2026-04-02 21:02:38 +00:00
Colin McDonnell	70d56ebc89	improve review quality: add --effort flag, subagent guidance, remove dead prompts (#508) ... * Update waitlist, run ralph experiments * improve review quality: add --effort flag, subagent guidance, remove dead prompts - add --effort high/max to Claude Code CLI (max for Opus, high for Sonnet/Haiku). default was silently dropped from high to medium in March 2026. - add subagent guidance to Review/IncrementalReview modeGuidance for parallel investigation of large cross-cutting PRs (read-only, no side effects). - remove "THINK HARDER" from mode prompts (vestigial, no longer controls thinking). - remove redundant mode.prompt bodies from modes.ts — the actual guidance lives in modeGuidance (selectMode.ts) and mode.prompt was dead code for all built-in modes since the delegation system was removed in March. Made-with: Cursor * make Mode.prompt optional, remove ModeSchema dead code prompt is only needed by custom user-defined modes (validated by Zod modeSchema in utils/schemas/modes.ts). built-in modes get their guidance from modeGuidance in selectMode.ts. the arktype ModeSchema was never imported anywhere. Made-with: Cursor * make modes.ts the single source of truth for mode guidance move all mode guidance from modeGuidance in selectMode.ts into mode.prompt in modes.ts. selectMode.ts now only contains the runtime tool logic (resolving modes, merging user instructions, handling PlanEdit/SummaryUpdate overrides). this eliminates the confusing fallback chain where someone editing mode instructions had to know to look in selectMode.ts rather than modes.ts. Made-with: Cursor * add self-review subagent step to Build mode, update wiki Build mode now delegates a read-only subagent to review the diff before committing, catching bugs/logic errors/edge cases that the builder might miss. Also updates wiki/modes.md to reflect the single-source-of-truth architecture (modes.ts owns all guidance, selectMode.ts is pure runtime logic). Made-with: Cursor * update model snapshot (openrouter qwen3.6-plus rename) Made-with: Cursor	2026-04-02 20:04:09 +00:00
Colin McDonnell	b1f9878877	extract resolveModel() to run before agent selection ... model resolution was duplicated inside each agent (opentoad, claude) and PULLFROG_MODEL override was not considered when choosing the agent. now resolveModel() runs first in main.ts, its result feeds into resolveAgent() for agent selection, and the resolved model is passed to the agent via ctx.resolvedModel. agents only handle their own fallback (opentoad: auto-select via opencode models, claude: strip provider prefix). also removes the hardcoded anthropic/claude-sonnet test runner default since ANTHROPIC_API_KEY is no longer in CI. Made-with: Cursor	2026-04-01 06:51:18 +00:00
Colin McDonnell	f1400ffb7c	remove cost logging from agent runs; extract secrets into own tab ... - stop capturing/displaying total_cost_usd from Claude CLI (theoretical cost is misleading for subscription users) - remove Cost column from action logs table and GitHub Job Summary - extract SecretsCard into its own sidebar tab with KeyRound icon - remove children prop from AgentSettingsSection Made-with: Cursor	2026-03-31 06:14:04 +00:00
Colin McDonnell	0055aef618	feat: add Claude Code agent for Anthropic model users (#502) ... * feat: add Claude Code agent for Anthropic model users Re-adds Claude Code support (removed in #478) so users with Anthropic API keys or Claude Code OAuth tokens can use their Claude subscriptions directly. When an Anthropic model is selected and Claude Code credentials are available, the system auto-selects the Claude agent instead of OpenCode. The harness mirrors opentoad's security model: native Bash blocked via --disallowedTools, MCP ShellTool for restricted shell, ASKPASS for git auth. Includes NDJSON streaming, provider error detection, cache/cost tracking, browser skill, and todo progress tracking. Key changes: - action/agents/claude.ts: full Claude Code harness - action/utils/agent.ts: auto-select Claude for anthropic/* models - action/utils/providerErrors.ts: extracted shared provider error detection - action/utils/skills.ts: extracted shared skill installation (agent-aware) - action/models.ts: add CLAUDE_CODE_OAUTH_TOKEN to anthropic envVars - action/utils/docker.ts: add CLAUDE_CODE_OAUTH_TOKEN to test env allowlist - CI: add claude to test matrix, pass CLAUDE_CODE_OAUTH_TOKEN secret Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove unused toolId variable, fix apiKeys test env cleanup The apiKeys test cleanup stripped *_API_KEY vars but missed CLAUDE_CODE_OAUTH_TOKEN which doesn't match that pattern. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: strip provider prefix from PULLFROG_MODEL in Claude agent the env override path was returning the raw value (e.g. "anthropic/claude-sonnet-4-5") without stripping the provider prefix, causing the Claude CLI to receive an invalid model ID. Made-with: Cursor * fix: remove dead cliPath field, add CLAUDE_CODE_OAUTH_TOKEN to workflows remove unused cliPath from Claude agent RunParams, and pass CLAUDE_CODE_OAUTH_TOKEN through all pullfrog.yml workflow templates so users with Claude Pro/Team subscriptions can use their membership. Made-with: Cursor * fix: block Bash subagent in Claude Code disallowedTools Made-with: Cursor * chore: update model snapshot (opencode/openrouter latest → qwen3.6-plus-free) Made-with: Cursor --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-31 03:29:54 +00:00
Colin McDonnell	6d25adfd1a	Agent & model refactor (#478) ... * agent & model refactor with ASKPASS git auth, UI restructure, clerk v7 Made-with: Cursor * fix stale agent/effort refs, add tests for askpass + model resolution - reviewCleanup.ts: payload.agent -> payload.model, remove effort - selectMode.ts PlanEdit: remove delegation/subagent/effort references - pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY, add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY) - FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy - new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen) - new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override) - new: models.test.ts (19 tests — parseModel, resolution, registry invariants) - update models.dev snapshot Made-with: Cursor * fix changed-agents.sh to filter legacy agent files from CI matrix legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not exported from index.ts. changed-agents.sh now reads index.ts imports to build the active agent set and treats changes to inactive files as non-agent changes (opentoad canary only). Made-with: Cursor * remove MCP file tools, old agent harnesses, and obsolete security tests ASKPASS-based git auth makes the old MCP file tool security layer unnecessary: - token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it - agents now use native file tools (OpenCode builtin read/edit) deleted: - action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory) - action/mcp/index.ts (dead re-export) - agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts - opencode-runner.ts (inlined into opentoad.ts) - security tests that validated MCP file tool restrictions - commented-out three-step review flow (~300 lines) - sanitizeSchema/wrapSchema dead code from mcp/shared.ts - OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed) updated test prompts to use generic file ops instead of MCP tool names. restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense). Made-with: Cursor * bump actions/checkout v4 → v6 (node 24) node 20 actions deprecated june 2, 2026. Made-with: Cursor * temporarily disable fail-fast on agnostic tests to debug checkout@v6 Made-with: Cursor * re-enable fail-fast on agnostic tests Made-with: Cursor * fix test token mismatch: mint OIDC tokens scoped to target repo CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so ensureGitHubToken() mints a properly scoped token via OIDC. Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming instead of repeating it in every test file, and fixes preview-cleanup to remove workers from all queues (not just name-matching ones). Made-with: Cursor * fix ensureGitHubToken to try OIDC when app credentials are absent ensureGitHubToken only attempted token minting when GITHUB_APP_ID and GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds aren't exposed — so the guard prevented minting entirely. Made-with: Cursor * dead code cleanup: remove remnants of deleted agents, file tools, effort system remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps, orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale opencode-runner wiki refs, deleted test file references, and MCP file tool docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to globalSetup (runs once before forks instead of per-file, 19s → 200ms). Made-with: Cursor * address review feedback: remove dead code, update stale references - remove AGENT_OVERRIDE (only opentoad exists) - remove shellToolName plumbing (always restricted shell) - bump action version to 0.0.179 - remove CURSOR_API_KEY from all workflows/configs - remove OPENCODE_MODEL_MINI/MAX from workflows/docs - delete wiki/effort.md, rewrite docs/effort.mdx as "Models" - rewrite wiki/modes.md: orchestrator/subagent → single agent - simplify flag system: drop builtin flag extraction (debug, effort, timeout, agent), keep custom flag replacement only - reserve all legacy flag names to prevent custom flag conflicts Made-with: Cursor * regenerate lockfile after removing claude-agent-sdk and codex-sdk Made-with: Cursor * fix import ordering, add lockfile check to pre-push hook Made-with: Cursor * remove dead debug payload field, stale packageExtensions Made-with: Cursor * merge proc-sandbox and token-exfil into a single test proc-sandbox and token-exfil were duplicative — both tested that SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into token-exfil with shell:restricted (which actually exercises filterEnv) and the /proc attack vector hints from proc-sandbox. Made-with: Cursor * fix wiki adversarial.md to match actual tokenExfil validator Made-with: Cursor	2026-03-12 05:22:51 +00:00
Colin McDonnell	a7bd746f21	Restructure dash (#372) ... * Restructure dash * WIP * WIP * refactor trigger UI: extract PR summary card, add mentions section, rename labels Co-authored-by: Cursor <cursoragent@cursor.com> * clean up console UI: remove info icons from section descriptions, rename mentions trigger Co-authored-by: Cursor <cursoragent@cursor.com> * fix review feedback: layout, terminology, form scope - extract console sidebar sections to module-level constant - align three-column layout breakpoints to xl (match sidebar visibility) - fix mixed shell/bash terminology in beta page - scope FormProvider to trigger sections only, restore autoComplete="off" Co-authored-by: Cursor <cursoragent@cursor.com> * Bump --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-23 23:34:29 +00:00
David Blass	cfd38d82fc	refactor delegation system, add PR summary comments, and improve code quality (#334) ... * refactor delegation system and add PR summary comments Delegation system: - replace mode-based delegation with select_mode → delegate two-step flow - orchestrator crafts self-contained subagent prompts (clean context — no system/repo/event instructions leak) - add role-based tool filtering via FastMCP authenticate hook (?role=subagent hides orchestrator-only tools) - add select_mode tool for orchestrator guidance per mode - add ask_question tool for lightweight research subagents - extract shared subagent lifecycle into subagent.ts (create, complete, stdout, instructions) - route set_output to per-subagent state when activeSubagentId is set - track per-subagent state (SubagentState Map) replacing boolean delegationActive flag - capture and aggregate AgentUsage across all agents (claude, codex, gemini, opencode) - write usage summary table to GitHub job summary - block built-in subagent spawning (Task for Claude, Task(*) for Cursor) - increase activity timeout from 60s to 300s (subagent thinking phases) - fix gh CLI misguidance in system prompt — explicitly forbid usage PR summary comments: - add prSummaryComment trigger (DB schema + migrations + Zod + UI toggle) - dispatch mini-effort summary job alongside PR review on pr.created - add update_pull_request_body MCP tool - add defaultEffort option to webhook dispatch Hardening: - rewrite delegate/selectMode tests with simulated state management - add toolFiltering.test.ts for role extraction, canAccess, set_output routing - remove non-null assertions for PULLFROG_TEMP_DIR (proper error throws) - use fetchWithRetry for direct tarball downloads - DRY fix for rate limit check in test runner Co-authored-by: Cursor <cursoragent@cursor.com> * fix: add type keyword to Effort import in handleWebhook.ts Co-authored-by: Cursor <cursoragent@cursor.com> * clean up delegation system, improve code quality across the codebase - simplify delegate tool to instructions + effort params with subagent lifecycle in subagent.ts - add select_mode and ask_question orchestrator-only tools with canAccess filtering - replace delegate.test.ts/selectMode.test.ts with toolFiltering.test.ts (live MCP integration) - add set_output routing for subagent context and AgentUsage tracking across all agents - add PR summary comment trigger (schema, UI, webhook dispatch with silent flag) - add update_pull_request_body MCP tool - fix changed-agents.sh to always include claude canary for non-agent action changes - fix cursor pagination bug in getSelectedInstallationReposPage - remove destructuring patterns, inline type definitions, and unsafe type casts - replace non-null assertions with explicit checks in install.ts - convert multi-param functions to single param objects (postCleanup, runActionLocal, etc.) - use isHttpError helper in API routes instead of catch-any patterns - add adhoc test fixtures for delegation scenarios (context isolation, error handling, synthesis, etc.) Co-authored-by: Cursor <cursoragent@cursor.com> * no subagent mutation, one mcp per subagent * address review feedback: parallel-safe usage tracking, subagent isolation, minor improvements * fix subagent state isolation: replace Object.freeze with shallow copy Object.freeze throws TypeErrors when subagent tools (checkout_pr, report_progress) write scalar properties to toolState. A shallow copy achieves the same isolation for scalar fields while allowing tools to work normally. Shared references (subagents Map, usageEntries array) remain shared for coordination. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>	2026-02-22 14:12:43 +00:00
pullfrog[bot]	70f1c47a28	Audit core.warning/core.error usage (#269) ... * Stop using command-based logs for warnings and errors Co-authored-by: Cursor <cursoragent@cursor.com> * revert * tweak * de-noise * Remove redundant ts() timestamp prefix from log calls * Restore timestamped logging and refine debug output routing. Bring back timestamp prefixes for standard logs and make log.debug emit via core.debug when runner debug is enabled, while still surfacing debug lines for --debug runs. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Mateusz Burzyński <mateuszburzynski@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-02-19 23:11:47 +00:00
Colin McDonnell	57537d1a95	move instructions logging earlier and clarify built-in tool logs ... Log the instructions box immediately after instruction resolution in main, and standardize agent permission summaries to debug-level "disallowed built-ins" output to reduce confusion with pullfrog MCP tools. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-18 19:01:08 +00:00
Mateusz Burzyński	510f2c96f9	Fix the availability of some @anthropic-ai/claude-agent-sdk types (#322) ... * Fix the availability of some `@anthropic-ai/claude-agent-sdk` types * update it in the action too * fix types	2026-02-18 12:12:40 +00:00
Colin McDonnell	b6658ddbc1	improve agent CI matrix, token permissions, and waitlist follower backfill (#313) ... * add workflows permission to git token and waitlist improvements - add `workflows` to `InstallationTokenPermissions` type in both action and API token routes - include `workflows: write` in the git token so agents can push workflow file changes - add `githubFollowers` field to WaitlistSignup schema with migration - add script to populate waitlist followers from GitHub API - add frog-green-square-border logo asset Co-authored-by: Cursor <cursoragent@cursor.com> * improve CI, agent logging, token permissions, and delegation guardrails - add format check and build step to root CI job - standardize agent model/effort log lines across all agents - fix GitHub App permissions types to match OpenAPI schema (workflows is write-only) - improve delegation error message to prevent subagent recursion - demote noisy OpenCode stderr to debug level - add subagent delegation rules to resolved instructions Co-authored-by: Cursor <cursoragent@cursor.com> * fix graphql partial error handling, update delegation message, add workflow_run fixtures Co-authored-by: Cursor <cursoragent@cursor.com> * remove module-level env var throws that break CI build Co-authored-by: Cursor <cursoragent@cursor.com> * fix logging bug and type hole from PR review - use batch-local notFound counter so per-batch log doesn't undercount - add workflows to WorkflowTokenPermissions so wire type matches what action sends Co-authored-by: Cursor <cursoragent@cursor.com> * lazy-init appOctokit to fix next build without env vars Co-authored-by: Cursor <cursoragent@cursor.com> * drop pnpm build from CI test workflow Co-authored-by: Cursor <cursoragent@cursor.com> * fix delegate-effort test regex to match actual log format, disable fail-fast for agnostic tests the test was matching `running \w+ with effort=auto` but the actual log line from shared.ts is `» effort: auto`. also temporarily set fail-fast: false on action-agnostic so all failures surface at once. Co-authored-by: Cursor <cursoragent@cursor.com> * disable fail-fast in action workflow too, relax ci.test.ts to match both workflow files now use fail-fast: false for agnostic tests so all matrix jobs run to completion. the ci consistency test now checks that the two workflows agree rather than requiring true. Co-authored-by: Cursor <cursoragent@cursor.com> * restore fail-fast: true now that all agnostic tests pass Co-authored-by: Cursor <cursoragent@cursor.com> * skip agent tests in CI when agent harness file didn't change adds action/test/changed-agents.sh which reads the PR diff (via dorny/paths-filter) and outputs only agents whose harness file was modified. the action-agents matrix now uses this dynamic list instead of a hardcoded array, so e.g. a PR touching only cursor.ts runs 6 jobs instead of 30. Co-authored-by: Cursor <cursoragent@cursor.com> * update ci.test.ts to validate dynamic agent matrix the test now checks that the matrix references the changes job output and that changed-agents.sh correctly discovers all agents. Co-authored-by: Cursor <cursoragent@cursor.com> * parallelize action jobs and use claude canary fallback for shared changes runs action-agents in parallel with action-agnostic after root/changes, and updates changed-agents logic so shared or non-harness action runtime changes run only claude while harness-specific edits run only those changed agents. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-16 22:36:29 +00:00
David Blass	a8dde34531	fix delegate timeout (#284) ... * pin all CLI installations to explicit versions and use pro models by default - codex: pin @openai/codex to 0.101.0 (was "latest") - opencode: pin opencode-ai to 1.1.56 (was "latest") - gemini: pin gemini-cli to v0.28.2 via new tag param on installFromGithub - cursor: pin to 2026.01.28-fd13201 via direct tarball download (replaces curl install script) - add installFromDirectTarball to install.ts for versioned tarball URLs - gemini auto effort now uses pro-preview instead of flash-preview - test runner model overrides updated to use pro-preview consistently Co-authored-by: Cursor <cursoragent@cursor.com> * increase activity timeout * fix delegation timeout * fix delegate timeouts --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-12 23:46:28 +00:00
David Blass	ceadb3120a	pin all CLI installations to explicit versions and use pro models by default (#283) ... * pin all CLI installations to explicit versions and use pro models by default - codex: pin @openai/codex to 0.101.0 (was "latest") - opencode: pin opencode-ai to 1.1.56 (was "latest") - gemini: pin gemini-cli to v0.28.2 via new tag param on installFromGithub - cursor: pin to 2026.01.28-fd13201 via direct tarball download (replaces curl install script) - add installFromDirectTarball to install.ts for versioned tarball URLs - gemini auto effort now uses pro-preview instead of flash-preview - test runner model overrides updated to use pro-preview consistently Co-authored-by: Cursor <cursoragent@cursor.com> * increase activity timeout * fix delegation timeout --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-12 22:48:29 +00:00
Colin McDonnell	f37d02b292	upgrade Claude to Opus 4.6 with effort levels (#256) ... * upgrade Claude to Opus 4.6 with --effort max for --max mode - mini: haiku → sonnet - auto: opusplan → opus (Opus 4.6) - max: opus → opus + --effort max (Opus 4.6 max effort) - bump @anthropic-ai/claude-agent-sdk 0.2.7 → 0.2.39 Co-authored-by: Cursor <cursoragent@cursor.com> * update action lockfile for claude-agent-sdk 0.2.39 Co-authored-by: Cursor <cursoragent@cursor.com> * add tool_use_summary handler for SDK 0.2.39 Co-authored-by: Cursor <cursoragent@cursor.com> * integrate gpt-5.3-codex with runtime model availability detection checks GET /v1/models at agent start to determine if gpt-5.3-codex is available for the API key, falling back to gpt-5.2-codex when it isn't. model resolution runs concurrently with CLI install for zero added latency. also bumps @openai/codex-sdk from 0.80.0 to 0.98.0. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-10 23:34:13 +00:00
David Blass	19df8372cd	add file_read/file_write tools, sandbox tests, CI improvements (#239) ... * migrate to flags * init * iterate on file write lockdown tests * improve ci * fix lockfile * fix typecheck * fix lint * improve pushRestricted * ok * fix more * ok * remove process.env spreading rule Co-authored-by: Cursor <cursoragent@cursor.com> * enhanced fs rw tools --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-02-10 06:35:47 +00:00
pullfrog[bot]	fb80343ffd	feat(agents): add thinking time logging between tool calls (#244) ... * feat(agents): add thinking time logging between tool calls Adds a ThinkingTimer utility that tracks the gap between tool results and the next tool call. When the gap exceeds 3 seconds, it logs the duration with a stopwatch emoji (⏱️ 4.2s). Uses performance.now() for high-resolution timing and Intl.NumberFormat for rendering duration in seconds with optional fraction digits. Integrated across all 5 agents: Claude, Codex, Cursor, Gemini, OpenCode. Closes #127 * fix: adjusting tests for mocking performance.now. * fix: reducing diff for claude. * fix: rm unused args for claude. * rm unused args for codex. * fix: rm unused args for gemini. * fix: rm unused args for opencode. * mv THINKING_THRESHOLD. * rev: I decided to pospone node:perf_hooks integration since it requires more comprehensive refactoring. * fix: using Intl unit formatting. * tests for ThinkingTimer. * fix: narrow unit. * fix: making durationFormatter a class instance property since using one agent per run. * fix: inverting condition in markToolCall. * thinking timer improvements and fix actions/checkout v6 auth - thinking timer: use » chevron and "thought for X seconds" format - thinking timer: add debug timestamps for sanity checking - demote PID namespace isolation logs to debug - remove redundant "setting up git authentication" log - fix duplicate Authorization header with actions/checkout v6: clean up includeIf credential entries that v6 persists via external config files Co-authored-by: Cursor <cursoragent@cursor.com> * standardize tool call log prefix to » double chevron Co-authored-by: Cursor <cursoragent@cursor.com> * update timer tests for new thinking log format Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> Co-authored-by: Robin Tail <robin_tail@me.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-10 06:17:43 +00:00
David Blass	51205b3d0a	update codex to 5.2 (#235)	2026-02-06 15:55:57 +00:00
Colin McDonnell	6deeea7032	add lint/format scripts and fix all biome errors (#233) ... Add `lint`, `lint:fix`, `format`, and `format:fix` package.json scripts backed by biome. Add AGENTS.md rule for agents to run them after changes. Fix all existing lint and format violations across the codebase. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-06 07:18:00 +00:00
Colin McDonnell	3a7145db1a	Scope installation token permissions in restricted mode (#226) ... * Scope installation token permissions in restricted mode In restricted/disabled bash mode, the installation token is now scoped to match the workflow's permissions block. This preserves fork push capability while limiting what the agent can do with the token. - Read workflow permissions from pullfrog.yml at runtime - Pass permissions to API when acquiring installation token - Clear OIDC env vars in restricted mode to prevent token minting - Simplify setupGit by moving token resolution to main.ts * Address review feedback: fail closed with default permissions - Add restrictive default permissions (contents:read, pull_requests:read, issues:read) as fallback when workflow permissions can't be read - Add support for job-level permissions via GITHUB_JOB env var - Fix misleading comment about token resolution in restricted mode - Add documentation about fork PR checkout behavior * Simplify to separate git/MCP tokens without workflow permission scoping - gitToken: minimal contents:write only (assumed exfiltratable) - mcpToken: full installation token (not exfiltratable via MCP tools) - Remove workflowPermissions.ts - security-conscious users can pass their own token via GH_TOKEN or inputs.token - Add type-safe InstallationTokenPermissions to github.ts and API route * Rename `write` permission to `push` and remove vestigial tool blocking The `write` permission was previously used to block local file write tools in agents. This was security theater since bash can write files anyway. Now `push` only controls the git token scope: - push: enabled → contents:write (can push commits) - push: disabled → contents:read (read-only, can't push) Changes: - Rename `write` to `push` in action.yml, Prisma schema, and all TS types - Remove vestigial write tool blocking from all agents (claude, cursor, gemini, opencode, codex) - Add data-preserving Prisma migration using RENAME COLUMN - Update UI: "Write files" → "Git push" with updated description * add PID namespace isolation for bash sandbox when running in CI, attempts to use unshare --pid to create a new PID namespace for bash subprocesses. this prevents the /proc/$PPID/environ attack where a malicious command could read secrets from the parent process's environment. the protection works by: 1. creating a new PID namespace (subprocess becomes PID 1) 2. mounting fresh /proc showing only sandbox PIDs 3. parent PIDs become invisible (PPID = 0, /proc/0 doesn't exist) combined with filterEnv(), this provides complete protection against /proc-based secret theft. falls back gracefully if namespaces aren't available. includes test script to verify the protection works. * add PID namespace test to CI workflow tests whether unshare --pid works on GHA runners out of the box, and if not, whether enabling via sysctl helps. also runs the pidNamespace.ts test to verify the full protection. * fix pnpm setup and add procIsolation agent test - fix pnpm/action-setup by specifying package_json_file path - add procIsolation crossagent test that has agent attempt to read secrets via /proc/$PPID/environ - add procIsolation to CI test matrix * add pid-namespace test job to main workflow this job tests unshare --pid capabilities on GHA runners and runs the pidNamespace.ts adhoc test to verify /proc isolation works * test bubblewrap's sysctl approach for enabling namespaces - write to /etc/sysctl.d/99-userns.conf and run sysctl --system - try aa-complain on unshare binary - more detailed diagnostics * fix pidNamespace test and add sudo-unshare fallback for GHA - fix reference error in pidNamespace.ts (renamed function but didn't update calls) - add sudo-unshare as fallback method for GHA runners where unprivileged namespaces are blocked but sudo is available - update bash.ts to detect and use sudo unshare when unprivileged fails * consolidate security docs and document PID namespace isolation - update security.md with current implementation details - document sudo unshare fallback for GHA runners - add testing instructions for local Docker and CI - add "Further Exploration" section with Landlock and path validation ideas - delete bash-sandbox.md and landlock.md (consolidated into security.md) * move procIsolation test to adhoc folder the procIsolation test requires PID namespace capabilities that aren't available in the Docker test environment. moved to adhoc/ so it's excluded from default test runs and can be run explicitly when needed (e.g. via the pid-namespace CI job or locally with --privileged docker). * fix Docker test environment for PID namespace isolation - add CI and GITHUB_ACTIONS to testEnvAllowList so sandbox detection runs - add --privileged to Docker run for PID namespace support (unshare) this fixes the test environment to properly test the sandbox. in production, the action runs directly on GHA runner where sudo unshare works. * fix getJobToken() to work in test environment add fallback to GH_TOKEN and GITHUB_TOKEN when INPUT_TOKEN is not set. this allows tests to run without requiring workflow-level token input. the token resolution order is: 1. INPUT_TOKEN (from workflow `with: token:`) 2. GH_TOKEN (external token override) 3. GITHUB_TOKEN (pre-acquired in tests or from GHA env) * security: filter secrets from all subprocess environments - extract filterEnv() to shared utils/secrets.ts - make $() utility filter secrets by default (git, npm, etc. don't need them) - disable git hooks via core.hooksPath to prevent hook-based exfiltration - git auth uses token embedded in URL, not env vars this prevents malicious git hooks, npm postinstall scripts, and other code execution vectors from exfiltrating GITHUB_TOKEN and API keys. * docs: clarify defense-in-depth security model update security.md to explain why BOTH layers are required: - filterEnv(): cleans child's own /proc/self/environ - PID namespace: hides parent's /proc entries PID namespace alone isn't sufficient - with --mount-proc, the child becomes PID 1, so /proc/1/environ is the child's OWN environment. without filterEnv(), secrets would still be accessible there. * add procSandbox crossagent test for PID namespace security - add crossagent/procSandbox.ts: security test that instructs agent to try various /proc attack vectors and validates sandbox blocks them - update wiki/security.md: document PID namespace isolation details, add verification commands, explain why sudo inside sandbox doesn't break security - update docker.ts: use node:24 with sudo for GHA-like test environment - update instructions.ts: allow disabling security messaging for pentests - clean up adhoc test files (procIsolation.ts, securityAudit.ts) the procSandbox test sets SANDBOX_TEST_TOKEN (matches *_TOKEN filter), gives the agent explicit hints about attack vectors (/proc, sudo, nsenter, etc.), disables prompt-based mitigations, and validates the secret is never leaked. * move procSandbox test to agnostic/ (runs with one agent) * WIP * docs: add agent testing guide (pnpm play, Docker, pentesting) * docs: add CI details to agent testing guide * docs: add interesting findings and gotchas from pentesting * improve test fidelity: auto-set CI=true, verify sandbox active - docker.ts: always set CI=true in container so sandbox activates - docker.ts: skip sudo/user setup if already done (faster reruns) - bash.ts: export getSandboxMethod() for diagnostics - bash.ts: add debug log when sandbox disabled - procSandbox.ts: add sandbox_was_active check to catch vacuous passes the CI=true change is critical: without it, PID namespace isolation is skipped and security tests pass without actually testing anything. * docs: update agent-testing.md with CI=true auto-set note * docs: clarify log format is agent-specific * fix git auth, simplify MCP tools, add adversarial tests - fix gitWithAuth to use Basic auth format (Bearer doesn't work with git's http.extraheader) - fix token.ts: push:restricted now correctly gets contents:write - fix github.ts: pass permissions when acquiring installation tokens locally - remove commit_files and create_branch MCP tools (redundant, don't require credentials) - remove containsSecrets() - trivially bypassable, not sound security - add agnostic adversarial tests: pushDisabled, pushRestricted, tokenExfil - update instructions.ts to clearly list available git MCP tools - add wiki/git.md documenting credential isolation and push permission tiers - update wiki/docker.md with custom image considerations - update wiki/agent-testing.md with adversarial testing patterns * fix type errors after rebase - change ResolveTokensParams.push from ToolPermission to PushPermission - use tags: ["agnostic"] instead of agnostic: true in test files * fix cleanup permission error in sandbox tests when sandbox isolation is enabled (CI=true), files created by the unshare subprocess may have different ownership, causing rmSync to fail with EACCES. this error in the finally block was overriding the test's success result. fix: wrap cleanup in try-catch and fall back to sudo rm if rmSync fails. * Add adhoc * Handle git config/remote bypasses * add git hooks protection and simplify ToolState - disable git hooks in restricted mode via -c core.hooksPath=/dev/null - add gitHooks adversarial test to verify hook protection works - unify prNumber/issueNumber into single issueNumber field - add pushUrl to ToolState for push validation - add generateTestMarker() for simpler single-agent test markers - export SENSITIVE_PATTERNS and isSensitiveEnvName from secrets.ts - remove redundant pidNamespace.ts (duplicated by procSandbox.ts) - update documentation * harden $git() auth: subcommand whitelist, binary tamper detection - rename gitWithAuth() to $git() with explicit subcommand first arg - restrict to "fetch" | "push" at type level (filters don't run for these) - resolve git binary path at startup via resolveGit(), sha256 fingerprint - verify hash before each $git() call to detect binary replacement - rename disableHooks to restricted for cleaner semantics - document filter exfiltration attack and empirical verification in wiki * remove redundant pid-namespace CI job the PID namespace isolation testing is now handled by action/test/agnostic/procSandbox.ts via pnpm runtest agnostic * fix push_branch for new branches and improve token leak detection - getPushDestination now falls back to origin/<branch> when @{push} is not configured (happens for new branches created locally) - gitPerms validator now checks for actual token patterns instead of matching "x-access-token" string in test instructions * use kebab-case for test names * simplify shell env API: "restricted" | "inherit" | object replace passFullEnv boolean with cleaner env option that accepts: - "restricted" (default): filterEnv() to prevent secret leakage - "inherit": full process.env - object: custom env merged with restricted base * share EnvMode and resolveEnv between shell.ts and bash.ts move shared env resolution logic to secrets.ts * add env option to bash tool (default: restricted) * delete agent-testing.md (renamed to adversarial.md) * Add checkout tests * reframe githooks test prompt to avoid claude safety refusal claude was refusing to execute the test because the prompt used words like "malicious" and "security testing". reframed as a debugging task with innocuous env var name (TESTING_DEBUG_TAG) per adversarial.md guidance. Co-authored-by: Cursor <cursoragent@cursor.com> * clean up verbose token acquisition logs move logging responsibility to call sites which have better context (git token vs MCP token). remove redundant intermediate OIDC logs and unused "(permission-scoped)" suffix. Co-authored-by: Cursor <cursoragent@cursor.com> * isolate agnostic tests with matrix strategy, fix .pullfrog-env secret leak - split action-agnostic into per-test matrix jobs for isolated logs and filesystems - only write explicitly opted-in env vars to .pullfrog-env via fileAgentEnv (fixes token-exfil test where claude found SANDBOX_TEST_TOKEN on disk) - mcpmerge test opts in via fileAgentEnv for cursor's repo-level MCP fallback Co-authored-by: Cursor <cursoragent@cursor.com> * remove env parameter from bash tool to prevent agents bypassing filterEnv the bash tool exposed an `env` parameter accepting "restricted" | "inherit" which allowed agents to pass env: "inherit" and see all secrets including SANDBOX_TEST_TOKEN, bypassing the restricted environment filtering entirely. env mode is now determined internally (always restricted). Co-authored-by: Cursor <cursoragent@cursor.com> * use pullfrog/test-repo for push tests to stop polluting main repo push tests were creating branches and tags on pullfrog/app directly. now all push tests (push-restricted, push-disabled, push-enabled, git-permissions) target pullfrog/test-repo instead. Co-authored-by: Cursor <cursoragent@cursor.com> * use pullfrog/test-repo for all tests, not just push tests no test should clone or operate on pullfrog/app directly. Co-authored-by: Cursor <cursoragent@cursor.com> * fix token scoping for test-repo and bash timeout defaults - acquireTokenViaOIDC now includes GITHUB_REPOSITORY repo in token scope so push tests work against pullfrog/test-repo - bash tool default timeout: 120s -> 30s, cap: 600s -> 120s - activity timeout: 30s -> 60s - prevents hung bash commands (e.g. find /) from killing the agent via activity timeout Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-02-06 06:26:26 +00:00
David Blass	6fbff21fca	add agent and debug macros, improve activity timeouts, migrate claude and codex to cli (#224)	2026-02-04 22:29:45 +00:00
Colin McDonnell	3a975cc384	Add md <> code comments	2026-01-24 18:56:51 +00:00
David Blass	4e19178c81	fix CI (#111)	2026-01-20 17:25:06 +00:00
Colin McDonnell	cb925556e8	refactor instructions to return object with full/system/user/event/runtime properties, fix duplicate modes and json prompt extraction (#110)	2026-01-16 21:43:54 +00:00
Colin McDonnell	69b9b96ddd	Refactor (#109)	2026-01-16 18:43:09 +00:00
Colin McDonnell	1f2f671be0	Fix claude	2026-01-16 16:25:49 +00:00
Colin McDonnell	9e019d89d2	Clean up actions and payloads (#98) ... * Clean up actions and payloads * Clean up action * Cleanup	2026-01-16 07:16:25 +00:00
Colin McDonnell	2d2d31adfa	Code style (#97) ... * Cleanup * fix: populate deny array before assigning to config, add CursorCliConfig type * Fix deny array ordering and add CursorCliConfig type Move deny array population before config declaration to avoid relying on reference semantics. Add proper type interface for the CLI config object. --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>	2026-01-15 22:06:53 +00:00
Colin McDonnell	97dce099c1	Implement granular tool permissions (#82) ... * Granular tool permissions * Fix build * Start on UI * Fixes * Fmt * Go ham on UI * Update migrations * Considate wiki files * Clean up * More tweaks. Docs. * Consolidate collab and noncollab * Fix build * Restrict for non-collaborators	2026-01-15 08:05:30 +00:00
pullfrog[bot]	71feba0a76	fix: prevent log.writeSummary from overwriting reportProgress content (#87) ... * fix: prevent log.writeSummary from overwriting reportProgress content The run summary was showing logs instead of the final reportProgress content because log.writeSummary() was called after reportProgress. Now log.writeSummary() checks if the summary was already overwritten by reportProgress and skips if so. Fixes #86 * refactor: replace dynamic import with static import in cli.ts Replace unnecessary dynamic import of wasSummaryOverwritten with static import. No circular dependency exists since comment.ts doesn't import from cli.ts. * Fix run summary writing --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-01-15 00:55:42 +00:00
David Blass	1daf1571cf	add macros (#68)	2026-01-14 22:52:54 +00:00
Colin McDonnell	edb7603587	Update claude impl	2026-01-12 14:12:16 -08:00
Colin McDonnell	45f837cedb	Fixes	2026-01-12 14:12:16 -08:00
pullfrog	c6572f0987	Address review feedback: use effort params, fix model names, add safety checks	2026-01-12 14:12:16 -08:00
David Blass	c335032c37	init	2026-01-12 14:12:11 -08:00
Colin McDonnell	fe7ce4af11	Updates	2026-01-08 14:11:43 -08:00
Colin McDonnell	879d33403c	Switch to custom Bash tool. Mask secrets from Bashsubprocs. Simplify security handling.	2026-01-08 14:09:08 -08:00
Colin McDonnell	573c473dc1	Drop opus flag	2025-12-30 13:52:54 -08:00
Colin McDonnell	cc59a16472	Clean up review mode	2025-12-22 18:31:27 -08:00
Colin McDonnell	8db0c40487	Do not return diff. Stick with opus	2025-12-22 18:22:35 -08:00
Colin McDonnell	7fb788a883	Token efficiency	2025-12-22 18:16:55 -08:00