8f36eca62ab7d3ce5697c07e86f81386cef60e8e
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 David Blass
|
c6a757424c |
Stop hook + learnings reflection via post-run loop (#515) (#548)
* add stop hook + learnings reflection to post-run loop (#515) stop hook (#515): repo-configured script that runs after the agent finishes. non-zero exit resumes the agent with the hook output as guidance; persistent failure (3 attempts) marks the run failed. the dirty-tree and stop-hook gates share a single retry loop so a fix + push happen in one turn. learnings reflection: per Colin, the learnings step baked into mode checklists rarely fires — the agent stays focused on the task and the meta-ask falls through. the post-run loop now delivers a dedicated one-shot --continue turn asking the agent to call update_learnings if relevant, nothing else competing for attention. reflection doesn't consume the gate-retry budget; if it dirties the tree, the next loop iteration catches it via the dirty-tree gate. plumbing: Repo.stopScript column + migration, zod schema, run-context api, AgentSettings UI. RepoSettings.stopScript threads through to AgentRunContext and into each agent harness. subprocess-dependent logic lives in action/agents/postRun.ts to keep action/agents/shared.ts lean — shared.ts is reachable from pullfrog/internal, and pulling node:child_process through it leaks into root tsc (which uses bundler resolution, not NodeNext). * fix: preserve successful run when reflection turn fails The post-run reflection turn (update_learnings nudge) is a best-effort one-shot; its failure must not flip a successful run to failed. Prior code overwrote `result` with the reflection's return value, so a model API error during reflection caused the whole run to be reported as failed even though the gated work had already completed cleanly. Now: save the pre-reflection result, and if reflection returns `success: false`, log a warning, restore the prior success, and exit without re-invoking the gates (re-running a freshly-green stop hook risks a flaky false-positive failure). Adds action/agents/postRun.test.ts covering the reflection path — previously uncovered. * fix: surface both stop-hook stdout and stderr to the agent The `(stderr || stdout)` heuristic in executeStopHook dropped stdout entirely whenever stderr had any content. Scripts that emit a benign warning to stderr and the actionable error to stdout (common for wrapper scripts) starved the agent of the information it needed to fix the issue. Now concatenate both streams (stderr first, stdout second, skipping empty ones) before truncation. This keeps stdout's tail — usually where summaries and totals live — intact under the 4096-char cap. * test: lock in the core post-run retry + reflection invariants PR #548's test plan ships four manual verification scenarios. Convert three to vitest coverage, catching regressions on the hottest code paths: - persistent stop hook failure exhausts MAX_POST_RUN_RETRIES and surfaces as AgentResult.error with both the retry count and the verbatim hook output (so the GitHub-comment rendering stays actionable). - every gate retry is fed the hook output as the resume prompt. - usage aggregates across the initial run plus every retry (billing relies on this). - reflection turn still fires when no stop hook is configured and the tree is clean. Manual item remaining is the full UI round-trip of the settings form, which is out of scope for unit tests. * test: cover executeStopHook soft-fail and truncation invariants Three paths the PR documents but previously had no regression gates: - timeout (SPAWN_TIMEOUT_CODE) and activity-timeout (SPAWN_ACTIVITY_TIMEOUT_CODE) must return null, not a failure. a hook that times out is an infra problem; retrying with an agent turn risks an infinite loop. - spawn errors (ENOENT from a typoed binary, etc.) take the same soft-fail path for the same reason. - oversize hook output is truncated to the last 4096 chars with a "truncated" marker, keeping the tail (where summaries live) and protecting the 65535-char GitHub-comment budget downstream. Regression targets — a refactor that accidentally surfaces an infra failure as a gate failure, or blows the comment budget, will now fail loudly in CI. * test: cover soft-fail, no-resume, and short-circuit invariants Three more documented behaviors that previously had no regression gates: - dirty-tree-only is a soft-fail: persistent uncommitted changes log and warn but DO NOT flip the run to failed. a regression that started surfacing this as AgentResult.error would break every run that leaves a test fixture untracked. - canResume=false + stop hook failure still surfaces the hook failure as AgentResult.error. the retry budget is zero so "N retry attempts" is correctly omitted from the message, but the run still reports WHY it failed rather than silently reporting success. - initial result with success=false short-circuits the loop: no gate checks, no reflection, no resume calls. the original agent error flows through verbatim for clean triage. Also reset mockedSpawn in beforeEach so test state doesn't leak between cases. * test: lock in the reflection-dirties-tree → dirty-tree-gate path The PR description claims: "if the reflection turn dirties the tree, the loop picks that up on the next iteration via the normal dirty-tree gate." There was no regression gate on this invariant. Without it, a refactor that moved the reflection out of the retry loop (e.g., into a one-shot post-loop call) would silently bypass the commit-before-you-finish contract whenever the agent misbehaves during reflection — uncommitted changes would ship as part of the run's "success" state. The test sequences three getGitStatus returns (clean → dirty → clean) and asserts two resume calls: REFLECTION first, then UNCOMMITTED CHANGES with the dirtying file in the prompt. * fix: preserve pre-reflection task output when reflection succeeds the reflection turn's reply ("done" or "updated learnings with N bullets") is a meta-ask, not a task summary. before this fix, result = reflectionResult clobbered the original task's output on the returned AgentResult, so downstream consumers (handleAgentResult's fallback path when toolState is empty, programmatic callers of main()) saw the reflection's trivial reply instead of the real summary. spread reflectionResult to inherit fields subsequent gate retries need (e.g. the new sessionId claude emits per --resume invocation), but keep the pre-reflection output verbatim. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: fall back to reflection's output when pre-reflection output is empty the prior fix used `??` which only fell through on null/undefined. runs that communicate exclusively through MCP tools (e.g. report_progress) and emit no plain text leave result.output = "", which `??` preserved as-is — dropping the reflection's reply and leaving handleAgentResult's fallback path with nothing to show. switch to `||` so empty-string pre-reflection output yields the reflection's output instead of ""; non-empty task output still wins as intended. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: drop reflection-failure-skips-hook test (over-specified control flow) the test pinned the literal `break` in the post-reflection failure branch with stopScript=null, asserting only that getGitStatus was called once. that's not a behavior contract — a reasonable refactor (e.g. `continue` to re-check gates with explicit flake guards) would fail this test even though the new behavior would be fine. the "does not flip a successful run to failed" test already covers the only thing callers depend on. * test: drop low-value mock-driven tests from postRun - "fires the reflection turn when no stop hook is configured" — fully subsumed by the output-preservation test (asserts task output survives, which is only possible if reflection fired). - "uses stdout alone" / "uses stderr alone" — pin format trivia (`filter(Boolean).join`) that LLMs ignore. - "returns empty output (not undefined) when both streams are empty" — guards a TS-impossible case; every consumer uses `output || "(no output)"`. - "returns null on activity-timeout" — duplicate of the timeout test; same `return null` branch with a different constant. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |
||
|
Colin McDonnell
|
4b3c5ca905 |
rename agent key to opencode and add skill invocation coverage (#541)
* rename agent key to opencode and add skill invocation coverage. add skill-invoke tests for claude and opencode with local play-based validation signals, update CI matrices, and include the current tracked refactors in this branch for review. Made-with: Cursor * exclude agent-specific skill-invoke tests from wrong agent in CI matrix * address review follow-up and preserve workflow run UI tweak. switch changed-agents ci coverage to exercise the opentoad implementation path while keeping the opencode expectation, and include the local workflow run client interaction updates requested on this branch. Made-with: Cursor * remove opentoad agent filename from runtime. rename the opencode harness implementation file from opentoad.ts to opencode.ts and update ci coverage input accordingly so action code no longer carries the old filename. Made-with: Cursor * ensure security prompt bypass is set on every test fixture. this keeps adversarial and permissions harnesses from being blocked by the default prompt-injection refusal path during CI security tests. Made-with: Cursor --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> |
||
|
Colin McDonnell
|
6d25adfd1a |
Agent & model refactor (#478)
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7 Made-with: Cursor * fix stale agent/effort refs, add tests for askpass + model resolution - reviewCleanup.ts: payload.agent -> payload.model, remove effort - selectMode.ts PlanEdit: remove delegation/subagent/effort references - pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY, add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY) - FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy - new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen) - new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override) - new: models.test.ts (19 tests — parseModel, resolution, registry invariants) - update models.dev snapshot Made-with: Cursor * fix changed-agents.sh to filter legacy agent files from CI matrix legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not exported from index.ts. changed-agents.sh now reads index.ts imports to build the active agent set and treats changes to inactive files as non-agent changes (opentoad canary only). Made-with: Cursor * remove MCP file tools, old agent harnesses, and obsolete security tests ASKPASS-based git auth makes the old MCP file tool security layer unnecessary: - token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it - agents now use native file tools (OpenCode builtin read/edit) deleted: - action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory) - action/mcp/index.ts (dead re-export) - agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts - opencode-runner.ts (inlined into opentoad.ts) - security tests that validated MCP file tool restrictions - commented-out three-step review flow (~300 lines) - sanitizeSchema/wrapSchema dead code from mcp/shared.ts - OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed) updated test prompts to use generic file ops instead of MCP tool names. restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense). Made-with: Cursor * bump actions/checkout v4 → v6 (node 24) node 20 actions deprecated june 2, 2026. Made-with: Cursor * temporarily disable fail-fast on agnostic tests to debug checkout@v6 Made-with: Cursor * re-enable fail-fast on agnostic tests Made-with: Cursor * fix test token mismatch: mint OIDC tokens scoped to target repo CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so ensureGitHubToken() mints a properly scoped token via OIDC. Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming instead of repeating it in every test file, and fixes preview-cleanup to remove workers from all queues (not just name-matching ones). Made-with: Cursor * fix ensureGitHubToken to try OIDC when app credentials are absent ensureGitHubToken only attempted token minting when GITHUB_APP_ID and GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds aren't exposed — so the guard prevented minting entirely. Made-with: Cursor * dead code cleanup: remove remnants of deleted agents, file tools, effort system remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps, orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale opencode-runner wiki refs, deleted test file references, and MCP file tool docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to globalSetup (runs once before forks instead of per-file, 19s → 200ms). Made-with: Cursor * address review feedback: remove dead code, update stale references - remove AGENT_OVERRIDE (only opentoad exists) - remove shellToolName plumbing (always restricted shell) - bump action version to 0.0.179 - remove CURSOR_API_KEY from all workflows/configs - remove OPENCODE_MODEL_MINI/MAX from workflows/docs - delete wiki/effort.md, rewrite docs/effort.mdx as "Models" - rewrite wiki/modes.md: orchestrator/subagent → single agent - simplify flag system: drop builtin flag extraction (debug, effort, timeout, agent), keep custom flag replacement only - reserve all legacy flag names to prevent custom flag conflicts Made-with: Cursor * regenerate lockfile after removing claude-agent-sdk and codex-sdk Made-with: Cursor * fix import ordering, add lockfile check to pre-push hook Made-with: Cursor * remove dead debug payload field, stale packageExtensions Made-with: Cursor * merge proc-sandbox and token-exfil into a single test proc-sandbox and token-exfil were duplicative — both tested that SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into token-exfil with shell:restricted (which actually exercises filterEnv) and the /proc attack vector hints from proc-sandbox. Made-with: Cursor * fix wiki adversarial.md to match actual tokenExfil validator Made-with: Cursor |
||
|
David Blass
|
cfd38d82fc |
refactor delegation system, add PR summary comments, and improve code quality (#334)
* refactor delegation system and add PR summary comments Delegation system: - replace mode-based delegation with select_mode → delegate two-step flow - orchestrator crafts self-contained subagent prompts (clean context — no system/repo/event instructions leak) - add role-based tool filtering via FastMCP authenticate hook (?role=subagent hides orchestrator-only tools) - add select_mode tool for orchestrator guidance per mode - add ask_question tool for lightweight research subagents - extract shared subagent lifecycle into subagent.ts (create, complete, stdout, instructions) - route set_output to per-subagent state when activeSubagentId is set - track per-subagent state (SubagentState Map) replacing boolean delegationActive flag - capture and aggregate AgentUsage across all agents (claude, codex, gemini, opencode) - write usage summary table to GitHub job summary - block built-in subagent spawning (Task for Claude, Task(*) for Cursor) - increase activity timeout from 60s to 300s (subagent thinking phases) - fix gh CLI misguidance in system prompt — explicitly forbid usage PR summary comments: - add prSummaryComment trigger (DB schema + migrations + Zod + UI toggle) - dispatch mini-effort summary job alongside PR review on pr.created - add update_pull_request_body MCP tool - add defaultEffort option to webhook dispatch Hardening: - rewrite delegate/selectMode tests with simulated state management - add toolFiltering.test.ts for role extraction, canAccess, set_output routing - remove non-null assertions for PULLFROG_TEMP_DIR (proper error throws) - use fetchWithRetry for direct tarball downloads - DRY fix for rate limit check in test runner Co-authored-by: Cursor <cursoragent@cursor.com> * fix: add type keyword to Effort import in handleWebhook.ts Co-authored-by: Cursor <cursoragent@cursor.com> * clean up delegation system, improve code quality across the codebase - simplify delegate tool to instructions + effort params with subagent lifecycle in subagent.ts - add select_mode and ask_question orchestrator-only tools with canAccess filtering - replace delegate.test.ts/selectMode.test.ts with toolFiltering.test.ts (live MCP integration) - add set_output routing for subagent context and AgentUsage tracking across all agents - add PR summary comment trigger (schema, UI, webhook dispatch with silent flag) - add update_pull_request_body MCP tool - fix changed-agents.sh to always include claude canary for non-agent action changes - fix cursor pagination bug in getSelectedInstallationReposPage - remove destructuring patterns, inline type definitions, and unsafe type casts - replace non-null assertions with explicit checks in install.ts - convert multi-param functions to single param objects (postCleanup, runActionLocal, etc.) - use isHttpError helper in API routes instead of catch-any patterns - add adhoc test fixtures for delegation scenarios (context isolation, error handling, synthesis, etc.) Co-authored-by: Cursor <cursoragent@cursor.com> * no subagent mutation, one mcp per subagent * address review feedback: parallel-safe usage tracking, subagent isolation, minor improvements * fix subagent state isolation: replace Object.freeze with shallow copy Object.freeze throws TypeErrors when subagent tools (checkout_pr, report_progress) write scalar properties to toolState. A shallow copy achieves the same isolation for scalar fields while allowing tools to work normally. Shared references (subagents Map, usageEntries array) remain shared for coordination. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> |
||
|
Colin McDonnell
|
b6658ddbc1 |
improve agent CI matrix, token permissions, and waitlist follower backfill (#313)
* add workflows permission to git token and waitlist improvements - add `workflows` to `InstallationTokenPermissions` type in both action and API token routes - include `workflows: write` in the git token so agents can push workflow file changes - add `githubFollowers` field to WaitlistSignup schema with migration - add script to populate waitlist followers from GitHub API - add frog-green-square-border logo asset Co-authored-by: Cursor <cursoragent@cursor.com> * improve CI, agent logging, token permissions, and delegation guardrails - add format check and build step to root CI job - standardize agent model/effort log lines across all agents - fix GitHub App permissions types to match OpenAPI schema (workflows is write-only) - improve delegation error message to prevent subagent recursion - demote noisy OpenCode stderr to debug level - add subagent delegation rules to resolved instructions Co-authored-by: Cursor <cursoragent@cursor.com> * fix graphql partial error handling, update delegation message, add workflow_run fixtures Co-authored-by: Cursor <cursoragent@cursor.com> * remove module-level env var throws that break CI build Co-authored-by: Cursor <cursoragent@cursor.com> * fix logging bug and type hole from PR review - use batch-local notFound counter so per-batch log doesn't undercount - add workflows to WorkflowTokenPermissions so wire type matches what action sends Co-authored-by: Cursor <cursoragent@cursor.com> * lazy-init appOctokit to fix next build without env vars Co-authored-by: Cursor <cursoragent@cursor.com> * drop pnpm build from CI test workflow Co-authored-by: Cursor <cursoragent@cursor.com> * fix delegate-effort test regex to match actual log format, disable fail-fast for agnostic tests the test was matching `running \w+ with effort=auto` but the actual log line from shared.ts is `» effort: auto`. also temporarily set fail-fast: false on action-agnostic so all failures surface at once. Co-authored-by: Cursor <cursoragent@cursor.com> * disable fail-fast in action workflow too, relax ci.test.ts to match both workflow files now use fail-fast: false for agnostic tests so all matrix jobs run to completion. the ci consistency test now checks that the two workflows agree rather than requiring true. Co-authored-by: Cursor <cursoragent@cursor.com> * restore fail-fast: true now that all agnostic tests pass Co-authored-by: Cursor <cursoragent@cursor.com> * skip agent tests in CI when agent harness file didn't change adds action/test/changed-agents.sh which reads the PR diff (via dorny/paths-filter) and outputs only agents whose harness file was modified. the action-agents matrix now uses this dynamic list instead of a hardcoded array, so e.g. a PR touching only cursor.ts runs 6 jobs instead of 30. Co-authored-by: Cursor <cursoragent@cursor.com> * update ci.test.ts to validate dynamic agent matrix the test now checks that the matrix references the changes job output and that changed-agents.sh correctly discovers all agents. Co-authored-by: Cursor <cursoragent@cursor.com> * parallelize action jobs and use claude canary fallback for shared changes runs action-agents in parallel with action-agnostic after root/changes, and updates changed-agents logic so shared or non-harness action runtime changes run only claude while harness-specific edits run only those changed agents. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |