Agent & model refactor (#478)
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7 Made-with: Cursor * fix stale agent/effort refs, add tests for askpass + model resolution - reviewCleanup.ts: payload.agent -> payload.model, remove effort - selectMode.ts PlanEdit: remove delegation/subagent/effort references - pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY, add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY) - FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy - new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen) - new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override) - new: models.test.ts (19 tests — parseModel, resolution, registry invariants) - update models.dev snapshot Made-with: Cursor * fix changed-agents.sh to filter legacy agent files from CI matrix legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not exported from index.ts. changed-agents.sh now reads index.ts imports to build the active agent set and treats changes to inactive files as non-agent changes (opentoad canary only). Made-with: Cursor * remove MCP file tools, old agent harnesses, and obsolete security tests ASKPASS-based git auth makes the old MCP file tool security layer unnecessary: - token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it - agents now use native file tools (OpenCode builtin read/edit) deleted: - action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory) - action/mcp/index.ts (dead re-export) - agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts - opencode-runner.ts (inlined into opentoad.ts) - security tests that validated MCP file tool restrictions - commented-out three-step review flow (~300 lines) - sanitizeSchema/wrapSchema dead code from mcp/shared.ts - OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed) updated test prompts to use generic file ops instead of MCP tool names. restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense). Made-with: Cursor * bump actions/checkout v4 → v6 (node 24) node 20 actions deprecated june 2, 2026. Made-with: Cursor * temporarily disable fail-fast on agnostic tests to debug checkout@v6 Made-with: Cursor * re-enable fail-fast on agnostic tests Made-with: Cursor * fix test token mismatch: mint OIDC tokens scoped to target repo CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so ensureGitHubToken() mints a properly scoped token via OIDC. Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming instead of repeating it in every test file, and fixes preview-cleanup to remove workers from all queues (not just name-matching ones). Made-with: Cursor * fix ensureGitHubToken to try OIDC when app credentials are absent ensureGitHubToken only attempted token minting when GITHUB_APP_ID and GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds aren't exposed — so the guard prevented minting entirely. Made-with: Cursor * dead code cleanup: remove remnants of deleted agents, file tools, effort system remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps, orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale opencode-runner wiki refs, deleted test file references, and MCP file tool docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to globalSetup (runs once before forks instead of per-file, 19s → 200ms). Made-with: Cursor * address review feedback: remove dead code, update stale references - remove AGENT_OVERRIDE (only opentoad exists) - remove shellToolName plumbing (always restricted shell) - bump action version to 0.0.179 - remove CURSOR_API_KEY from all workflows/configs - remove OPENCODE_MODEL_MINI/MAX from workflows/docs - delete wiki/effort.md, rewrite docs/effort.mdx as "Models" - rewrite wiki/modes.md: orchestrator/subagent → single agent - simplify flag system: drop builtin flag extraction (debug, effort, timeout, agent), keep custom flag replacement only - reserve all legacy flag names to prevent custom flag conflicts Made-with: Cursor * regenerate lockfile after removing claude-agent-sdk and codex-sdk Made-with: Cursor * fix import ordering, add lockfile check to pre-push hook Made-with: Cursor * remove dead debug payload field, stale packageExtensions Made-with: Cursor * merge proc-sandbox and token-exfil into a single test proc-sandbox and token-exfil were duplicative — both tested that SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into token-exfil with shell:restricted (which actually exercises filterEnv) and the /proc attack vector hints from proc-sandbox. Made-with: Cursor * fix wiki adversarial.md to match actual tokenExfil validator Made-with: Cursor
This commit is contained in:
committed by
pullfrog[bot]
parent
5bcfae990a
commit
6d25adfd1a
+11
-16
@@ -27,14 +27,14 @@ import {
|
||||
* filters can be test names, tags, or agent names:
|
||||
* node test/run.ts # run all tests (excludes adhoc-tagged tests)
|
||||
* node test/run.ts smoke # run tests named "smoke" or tagged "smoke"
|
||||
* node test/run.ts claude # run all tests for claude only
|
||||
* node test/run.ts fs # run all tests tagged "fs"
|
||||
* node test/run.ts agnostic # run all agnostic-tagged tests (with claude)
|
||||
* node test/run.ts opentoad # run all tests for opentoad only
|
||||
* node test/run.ts security # run all tests tagged "security"
|
||||
* node test/run.ts agnostic # run all agnostic-tagged tests (with opentoad)
|
||||
* node test/run.ts adhoc # run all adhoc-tagged tests
|
||||
* node test/run.ts smoke claude # run smoke tests for claude only
|
||||
* node test/run.ts smoke opentoad # run smoke tests for opentoad only
|
||||
*
|
||||
* special tags:
|
||||
* - "agnostic": runs with claude only, excluded when filtering by agent
|
||||
* - "agnostic": runs with opentoad only, excluded when filtering by agent
|
||||
* - "adhoc": excluded from default runs, must be explicitly requested
|
||||
*
|
||||
* by default, runs in a Docker container for isolation.
|
||||
@@ -249,11 +249,11 @@ function shouldRetry(result: AgentResult, validation: ValidationResult): RetryDe
|
||||
// security-relevant checks (like no_leak_filtered, native_blocked) are designed
|
||||
// to PASS when set_output wasn't called (defensive coding). so cascade failures
|
||||
// are never genuine security findings — they're transient instruction-following
|
||||
// issues (MCP connection drop, agent confusion, low effort level, etc.).
|
||||
// issues (MCP connection drop, agent confusion, etc.).
|
||||
const setOutputCheck = validation.checks.find((c) => c.name === "set_output");
|
||||
if (setOutputCheck && !setOutputCheck.passed) {
|
||||
// if the output contains rate limit indicators, use the longer backoff
|
||||
// (the agent process may have succeeded but the subagent hit quota limits)
|
||||
// (the agent process may have succeeded but hit quota limits mid-run)
|
||||
const rateLimited = isRateLimited(result.output);
|
||||
return {
|
||||
retry: true,
|
||||
@@ -305,16 +305,11 @@ async function runTestForAgent(ctx: RunContext): Promise<ValidationResult> {
|
||||
env.PULLFROG_TEST_REPO_SETUP = testConfig.repoSetup;
|
||||
}
|
||||
|
||||
// opencode: use anthropic sonnet to avoid google quota issues and gemini doom-looping
|
||||
if (ctx.agent === "opencode") {
|
||||
// use anthropic sonnet to avoid google quota issues and gemini doom-looping
|
||||
if (ctx.agent === "opentoad") {
|
||||
env.OPENCODE_MODEL ??= "anthropic/claude-sonnet-4-5";
|
||||
}
|
||||
|
||||
// gemini: use 2.5 pro for testing
|
||||
if (ctx.agent === "gemini") {
|
||||
env.GEMINI_MODEL ??= "gemini-2.5-pro";
|
||||
}
|
||||
|
||||
// build file-based env vars for MCP servers that don't inherit parent env
|
||||
let fileEnv: Record<string, string> | undefined;
|
||||
if (testConfig.fileAgentEnv) {
|
||||
@@ -409,11 +404,11 @@ async function main(): Promise<void> {
|
||||
const isAgnostic = hasTag(testInfo, "agnostic");
|
||||
|
||||
if (isAgnostic) {
|
||||
// agnostic tests: skip if only filtering by agent, otherwise run with claude
|
||||
// agnostic tests: skip if only filtering by agent, otherwise run with opentoad
|
||||
if (parsed.filters.length === 0 && parsed.agentFilters.length > 0) {
|
||||
continue;
|
||||
}
|
||||
runs.push({ testInfo, agent: "claude" });
|
||||
runs.push({ testInfo, agent: "opentoad" });
|
||||
} else {
|
||||
// determine which agents to run for this test
|
||||
const testAgents = testInfo.config.agents ?? agents;
|
||||
|
||||
Reference in New Issue
Block a user