Agent & model refactor (#478)
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7 Made-with: Cursor * fix stale agent/effort refs, add tests for askpass + model resolution - reviewCleanup.ts: payload.agent -> payload.model, remove effort - selectMode.ts PlanEdit: remove delegation/subagent/effort references - pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY, add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY) - FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy - new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen) - new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override) - new: models.test.ts (19 tests — parseModel, resolution, registry invariants) - update models.dev snapshot Made-with: Cursor * fix changed-agents.sh to filter legacy agent files from CI matrix legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not exported from index.ts. changed-agents.sh now reads index.ts imports to build the active agent set and treats changes to inactive files as non-agent changes (opentoad canary only). Made-with: Cursor * remove MCP file tools, old agent harnesses, and obsolete security tests ASKPASS-based git auth makes the old MCP file tool security layer unnecessary: - token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it - agents now use native file tools (OpenCode builtin read/edit) deleted: - action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory) - action/mcp/index.ts (dead re-export) - agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts - opencode-runner.ts (inlined into opentoad.ts) - security tests that validated MCP file tool restrictions - commented-out three-step review flow (~300 lines) - sanitizeSchema/wrapSchema dead code from mcp/shared.ts - OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed) updated test prompts to use generic file ops instead of MCP tool names. restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense). Made-with: Cursor * bump actions/checkout v4 → v6 (node 24) node 20 actions deprecated june 2, 2026. Made-with: Cursor * temporarily disable fail-fast on agnostic tests to debug checkout@v6 Made-with: Cursor * re-enable fail-fast on agnostic tests Made-with: Cursor * fix test token mismatch: mint OIDC tokens scoped to target repo CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so ensureGitHubToken() mints a properly scoped token via OIDC. Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming instead of repeating it in every test file, and fixes preview-cleanup to remove workers from all queues (not just name-matching ones). Made-with: Cursor * fix ensureGitHubToken to try OIDC when app credentials are absent ensureGitHubToken only attempted token minting when GITHUB_APP_ID and GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds aren't exposed — so the guard prevented minting entirely. Made-with: Cursor * dead code cleanup: remove remnants of deleted agents, file tools, effort system remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps, orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale opencode-runner wiki refs, deleted test file references, and MCP file tool docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to globalSetup (runs once before forks instead of per-file, 19s → 200ms). Made-with: Cursor * address review feedback: remove dead code, update stale references - remove AGENT_OVERRIDE (only opentoad exists) - remove shellToolName plumbing (always restricted shell) - bump action version to 0.0.179 - remove CURSOR_API_KEY from all workflows/configs - remove OPENCODE_MODEL_MINI/MAX from workflows/docs - delete wiki/effort.md, rewrite docs/effort.mdx as "Models" - rewrite wiki/modes.md: orchestrator/subagent → single agent - simplify flag system: drop builtin flag extraction (debug, effort, timeout, agent), keep custom flag replacement only - reserve all legacy flag names to prevent custom flag conflicts Made-with: Cursor * regenerate lockfile after removing claude-agent-sdk and codex-sdk Made-with: Cursor * fix import ordering, add lockfile check to pre-push hook Made-with: Cursor * remove dead debug payload field, stale packageExtensions Made-with: Cursor * merge proc-sandbox and token-exfil into a single test proc-sandbox and token-exfil were duplicative — both tested that SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into token-exfil with shell:restricted (which actually exercises filterEnv) and the /proc attack vector hints from proc-sandbox. Made-with: Cursor * fix wiki adversarial.md to match actual tokenExfil validator Made-with: Cursor
This commit is contained in:
Colin McDonnell
committed by
pullfrog[bot]
pullfrog[bot]
parent
5bcfae990a
commit
6d25adfd1a
+24
-17
@@ -3,7 +3,7 @@ import { randomUUID } from "node:crypto";
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { agentsManifest } from "../external.ts";
|
||||
import { agents as agentMap } from "../agents/index.ts";
|
||||
import type { Inputs } from "../main.ts";
|
||||
import { trackChild, untrackChild } from "../utils/subprocess.ts";
|
||||
|
||||
@@ -38,7 +38,7 @@ export function defineFixture(inputs: Inputs, options?: FixtureOptions): Inputs
|
||||
return inputs;
|
||||
}
|
||||
|
||||
export const agents = Object.keys(agentsManifest) as (keyof typeof agentsManifest)[];
|
||||
export const agents = Object.keys(agentMap) as (keyof typeof agentMap)[];
|
||||
|
||||
export type AgentUuids<T extends string> = {
|
||||
// get marker value for a specific agent and env var
|
||||
@@ -90,11 +90,7 @@ export function generateAgentUuids<T extends string>(envVarNames: T[]): AgentUui
|
||||
|
||||
// assign consistent colors to agents (using ANSI codes)
|
||||
const AGENT_COLORS: Record<string, string> = {
|
||||
claude: "\x1b[35m", // magenta
|
||||
codex: "\x1b[32m", // green
|
||||
cursor: "\x1b[36m", // cyan
|
||||
gemini: "\x1b[33m", // yellow
|
||||
opencode: "\x1b[34m", // blue
|
||||
opentoad: "\x1b[32m", // green
|
||||
};
|
||||
const RESET = "\x1b[0m";
|
||||
|
||||
@@ -124,6 +120,11 @@ export function getAgentOutput(result: AgentResult): string {
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
// extract structured output from test result.
|
||||
export function getStructuredOutput(result: AgentResult): string | null {
|
||||
return result.structuredOutput;
|
||||
}
|
||||
|
||||
// parse GITHUB_OUTPUT file format to extract a key's value.
|
||||
// format: key<<ghadelimiter_<uuid>\n<value>\nghadelimiter_<uuid>
|
||||
function parseGitHubOutputFile(filePath: string, key: string): string | null {
|
||||
@@ -187,7 +188,7 @@ export async function runAgentStreaming(options: RunStreamingOptions): Promise<A
|
||||
};
|
||||
|
||||
// create unique HOME directory per test to avoid config file conflicts
|
||||
// when multiple tests run in parallel (e.g., cursor writes ~/.cursor/mcp.json)
|
||||
// when multiple tests run in parallel
|
||||
const mcpPort = options.env?.PULLFROG_MCP_PORT ?? "default";
|
||||
const testHome = `/tmp/home-${mcpPort}-${Date.now()}`;
|
||||
mkdirSync(testHome, { recursive: true });
|
||||
@@ -207,15 +208,21 @@ export async function runAgentStreaming(options: RunStreamingOptions): Promise<A
|
||||
}
|
||||
}
|
||||
|
||||
const subEnv: Record<string, string | undefined> = {
|
||||
...process.env,
|
||||
GITHUB_REPOSITORY: "pullfrog/test-repo", // default
|
||||
...options.env,
|
||||
HOME: testHome,
|
||||
GITHUB_OUTPUT: githubOutputFile,
|
||||
};
|
||||
|
||||
// clear CI runner's GITHUB_TOKEN so ensureGitHubToken() mints a
|
||||
// properly scoped token for the target GITHUB_REPOSITORY via OIDC
|
||||
delete subEnv.GITHUB_TOKEN;
|
||||
|
||||
const child = spawn("node", ["play.ts", "--raw", JSON.stringify(fixture)], {
|
||||
cwd: actionDir,
|
||||
env: {
|
||||
...process.env,
|
||||
AGENT_OVERRIDE: options.agent,
|
||||
...options.env,
|
||||
HOME: testHome,
|
||||
GITHUB_OUTPUT: githubOutputFile,
|
||||
},
|
||||
env: subEnv as Record<string, string>,
|
||||
stdio: "pipe",
|
||||
detached: true,
|
||||
});
|
||||
@@ -322,12 +329,12 @@ export interface TestRunnerOptions {
|
||||
repoSetup?: string;
|
||||
// tags for grouping tests (e.g., ["agnostic"], ["fs"])
|
||||
// special tags:
|
||||
// - "agnostic": runs with claude only, excluded when filtering by agent
|
||||
// - "agnostic": runs with opentoad only, excluded when filtering by agent
|
||||
// - "adhoc": excluded from default runs, must be explicitly requested
|
||||
tags?: TestTag[];
|
||||
}
|
||||
|
||||
export type TestTag = "adhoc" | "agnostic" | "fs" | "security";
|
||||
export type TestTag = "adhoc" | "agnostic" | "security";
|
||||
|
||||
export function printSingleValidation(validation: ValidationResult): void {
|
||||
const checksStr = validation.checks.map((c) => `${c.name}=${c.passed ? "✓" : "✗"}`).join(" ");
|
||||
|
||||
Reference in New Issue
Block a user