Agent & model refactor (#478)

* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7

Made-with: Cursor

* fix stale agent/effort refs, add tests for askpass + model resolution

- reviewCleanup.ts: payload.agent -> payload.model, remove effort
- selectMode.ts PlanEdit: remove delegation/subagent/effort references
- pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY,
  add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY)
- FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy
- new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen)
- new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override)
- new: models.test.ts (19 tests — parseModel, resolution, registry invariants)
- update models.dev snapshot

Made-with: Cursor

* fix changed-agents.sh to filter legacy agent files from CI matrix

legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not
exported from index.ts. changed-agents.sh now reads index.ts imports to
build the active agent set and treats changes to inactive files as
non-agent changes (opentoad canary only).

Made-with: Cursor

* remove MCP file tools, old agent harnesses, and obsolete security tests

ASKPASS-based git auth makes the old MCP file tool security layer unnecessary:
- token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it
- agents now use native file tools (OpenCode builtin read/edit)

deleted:
- action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory)
- action/mcp/index.ts (dead re-export)
- agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts
- opencode-runner.ts (inlined into opentoad.ts)
- security tests that validated MCP file tool restrictions
- commented-out three-step review flow (~300 lines)
- sanitizeSchema/wrapSchema dead code from mcp/shared.ts
- OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed)

updated test prompts to use generic file ops instead of MCP tool names.
restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense).

Made-with: Cursor

* bump actions/checkout v4 → v6 (node 24)

node 20 actions deprecated june 2, 2026.

Made-with: Cursor

* temporarily disable fail-fast on agnostic tests to debug checkout@v6

Made-with: Cursor

* re-enable fail-fast on agnostic tests

Made-with: Cursor

* fix test token mismatch: mint OIDC tokens scoped to target repo

CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit
the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on
every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so
ensureGitHubToken() mints a properly scoped token via OIDC.

Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming
instead of repeating it in every test file, and fixes preview-cleanup
to remove workers from all queues (not just name-matching ones).

Made-with: Cursor

* fix ensureGitHubToken to try OIDC when app credentials are absent

ensureGitHubToken only attempted token minting when GITHUB_APP_ID and
GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds
aren't exposed — so the guard prevented minting entirely.

Made-with: Cursor

* dead code cleanup: remove remnants of deleted agents, file tools, effort system

remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps,
orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale
opencode-runner wiki refs, deleted test file references, and MCP file tool
docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to
globalSetup (runs once before forks instead of per-file, 19s → 200ms).

Made-with: Cursor

* address review feedback: remove dead code, update stale references

- remove AGENT_OVERRIDE (only opentoad exists)
- remove shellToolName plumbing (always restricted shell)
- bump action version to 0.0.179
- remove CURSOR_API_KEY from all workflows/configs
- remove OPENCODE_MODEL_MINI/MAX from workflows/docs
- delete wiki/effort.md, rewrite docs/effort.mdx as "Models"
- rewrite wiki/modes.md: orchestrator/subagent → single agent
- simplify flag system: drop builtin flag extraction (debug, effort,
  timeout, agent), keep custom flag replacement only
- reserve all legacy flag names to prevent custom flag conflicts

Made-with: Cursor

* regenerate lockfile after removing claude-agent-sdk and codex-sdk

Made-with: Cursor

* fix import ordering, add lockfile check to pre-push hook

Made-with: Cursor

* remove dead debug payload field, stale packageExtensions

Made-with: Cursor

* merge proc-sandbox and token-exfil into a single test

proc-sandbox and token-exfil were duplicative — both tested that
SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into
token-exfil with shell:restricted (which actually exercises filterEnv)
and the /proc attack vector hints from proc-sandbox.

Made-with: Cursor

* fix wiki adversarial.md to match actual tokenExfil validator

Made-with: Cursor
This commit is contained in:
Colin McDonnell
2026-03-12 05:22:51 +00:00
committed by pullfrog[bot]
parent 5bcfae990a
commit 6d25adfd1a
111 changed files with 4196 additions and 10202 deletions
+21 -87
View File
@@ -5,30 +5,6 @@ import { type } from "arktype";
import { FastMCP } from "fastmcp";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { execute, tool } from "./shared.ts";
import { buildSubagentInstructions } from "./subagent.ts";
describe("buildSubagentInstructions", () => {
it("includes system preamble, resolved context, and orchestrator prompt", () => {
const prompt = "Read file.ts and fix the type error.";
const ctx = {
repo: { owner: "test-owner", name: "test-repo" },
} as any;
const instructions = buildSubagentInstructions({
ctx,
label: "test-task",
instructions: prompt,
});
expect(instructions.user).toBe(prompt);
expect(instructions.full).toContain("[CONTEXT]");
expect(instructions.full).toContain("test-owner/test-repo");
expect(instructions.full).toContain("subagent_label: test-task");
expect(instructions.full).toContain("set_output");
expect(instructions.full).toContain(prompt);
});
});
// ─── per-server tool isolation integration test ─────────────────────────
// demonstrates the architecture: orchestrator and subagent get separate servers
function getRandomPort(): Promise<number> {
return new Promise((resolve, reject) => {
@@ -59,44 +35,27 @@ function mockTool(name: string, description: string) {
});
}
describe("per-server tool isolation - integration", () => {
let orchestratorServer: FastMCP;
let subagentServer: FastMCP;
let orchestratorUrl: string;
let subagentUrl: string;
describe("MCP server tool registration - integration", () => {
let server: FastMCP;
let serverUrl: string;
const clients: Client[] = [];
beforeAll(async () => {
const [orchestratorPort, subagentPort] = await Promise.all([getRandomPort(), getRandomPort()]);
orchestratorUrl = `http://127.0.0.1:${orchestratorPort}/mcp`;
subagentUrl = `http://127.0.0.1:${subagentPort}/mcp`;
const port = await getRandomPort();
serverUrl = `http://127.0.0.1:${port}/mcp`;
// orchestrator gets ALL tools (common + delegation + remote mutation)
orchestratorServer = new FastMCP({ name: "orchestrator", version: "0.0.1" });
orchestratorServer.addTool(mockTool("file_read", "read a file"));
orchestratorServer.addTool(mockTool("git", "run git commands"));
orchestratorServer.addTool(mockTool("set_output", "set output"));
orchestratorServer.addTool(mockTool("select_mode", "select a mode"));
orchestratorServer.addTool(mockTool("delegate", "delegate a task"));
orchestratorServer.addTool(mockTool("ask_question", "ask a question"));
orchestratorServer.addTool(mockTool("push_branch", "push branch"));
orchestratorServer.addTool(mockTool("create_pull_request", "create PR"));
server = new FastMCP({ name: "test-server", version: "0.0.1" });
server.addTool(mockTool("shell", "run shell commands"));
server.addTool(mockTool("git", "run git commands"));
server.addTool(mockTool("set_output", "set output"));
server.addTool(mockTool("select_mode", "select a mode"));
server.addTool(mockTool("push_branch", "push branch"));
server.addTool(mockTool("create_pull_request", "create PR"));
// subagent gets ONLY file ops, shell, read-only GitHub, upload, set_output
subagentServer = new FastMCP({ name: "subagent", version: "0.0.1" });
subagentServer.addTool(mockTool("file_read", "read a file"));
subagentServer.addTool(mockTool("set_output", "set output"));
await Promise.all([
orchestratorServer.start({
transportType: "httpStream",
httpStream: { port: orchestratorPort, host: "127.0.0.1", endpoint: "/mcp" },
}),
subagentServer.start({
transportType: "httpStream",
httpStream: { port: subagentPort, host: "127.0.0.1", endpoint: "/mcp" },
}),
]);
await server.start({
transportType: "httpStream",
httpStream: { port, host: "127.0.0.1", endpoint: "/mcp" },
});
});
afterAll(async () => {
@@ -107,45 +66,20 @@ describe("per-server tool isolation - integration", () => {
// best-effort cleanup
}
}
await Promise.all([orchestratorServer.stop(), subagentServer.stop()]);
await server.stop();
});
it("orchestrator sees all tools including delegation and mutation", async () => {
const client = await connectMcpClient(orchestratorUrl);
it("server exposes all registered tools", async () => {
const client = await connectMcpClient(serverUrl);
clients.push(client);
const result = await client.listTools();
const names = result.tools.map((t) => t.name);
expect(names).toContain("select_mode");
expect(names).toContain("delegate");
expect(names).toContain("ask_question");
expect(names).toContain("push_branch");
expect(names).toContain("create_pull_request");
expect(names).toContain("file_read");
expect(names).toContain("shell");
expect(names).toContain("git");
expect(names).toContain("set_output");
expect(names.length).toBe(8);
});
it("subagent cannot see orchestrator-only tools", async () => {
const client = await connectMcpClient(subagentUrl);
clients.push(client);
const result = await client.listTools();
const names = result.tools.map((t) => t.name);
expect(names).not.toContain("select_mode");
expect(names).not.toContain("delegate");
expect(names).not.toContain("ask_question");
expect(names).not.toContain("push_branch");
expect(names).not.toContain("create_pull_request");
expect(names).not.toContain("git");
});
it("subagent sees only file ops, read-only tools, and set_output", async () => {
const client = await connectMcpClient(subagentUrl);
clients.push(client);
const result = await client.listTools();
const names = result.tools.map((t) => t.name);
expect(names).toContain("file_read");
expect(names).toContain("set_output");
expect(names.length).toBe(2);
expect(names.length).toBe(6);
});
});