5e6ff676236bc1c9d7921ab0b103b7aa37b9a21e
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colin McDonnell
|
5e6ff67623 |
move models.dev drift tests to main-only; add per-alias live smoke matrix
PR CI kept breaking on upstream catalog drift (new model ships on models.dev, OpenRouter renames an id, etc.) — failures unrelated to the PR's contents. split the model-alias test suite so PRs only see pure-logic checks, and push the external-state drift + end-to-end coverage to main. test organization: - action/test/models.test.ts keeps pure invariants: openRouterResolve completeness and fallback-chain resolution. runs on every PR. - action/test/models-catalog.main.test.ts gets the 4 network-dependent describes (models.dev validity x2, OpenRouter API validity, latest-model snapshot). runs only on main push via a dedicated vitest config (vitest.main.config.ts + `pnpm test:catalog`). new CI jobs in .github/workflows/test.yml: - models-catalog: `pnpm test:catalog` on every main push. detects upstream catalog drift so we can react at the next convenient window. - models-live: 38-entry matrix that invokes the agent harness end-to-end against the real provider for each alias in models.ts. generated from action/test/list-aliases.ts. runs only on main push AND only when resolution-affecting files changed (action/models.ts, action/package.json, action/agents/**) — the exact shape of the opus 4.7 incident. test/run.ts: PULLFROG_MODEL now flows through from process.env so the live matrix can pin an alias per job without the per-agent default clobbering it. Made-with: Cursor |
||
|
Colin McDonnell
|
74b313e612 |
bump claude-opus alias to 4-7
anthropic shipped claude-opus-4-7 today; opencode also republished it. point the "claude-opus" alias at the new version for both providers so existing users get the upgrade automatically. openrouter hasn't published 4.7 yet, so leave openRouterResolve at 4.6 as the BYOR fallback. also clarify the latest-model snapshot comment: new model drops usually just mean bumping the `resolve` on an existing alias, not adding a new one. Made-with: Cursor |
||
|
Colin McDonnell
|
536fae692a |
update snapshot for google/gemma-4-31b release
Made-with: Cursor |
||
|
Colin McDonnell
|
b8c4d5b716 |
add deprecated model fallback chain resolution
models can now be marked `deprecated: true` with a `fallback` slug pointing to a replacement. `resolveCliModel` follows the chain recursively (with cycle detection) until it finds a non-deprecated model. this keeps deprecated models in the registry for backward compatibility instead of removing them. marks opencode/mimo-v2-pro-free as deprecated with fallback to opencode/nemotron-3-super-free. Made-with: Cursor |
||
|
Colin McDonnell
|
e6d34ee01b |
add OpenRouter proxy for managed model routing and OSS program (#488)
* add OpenRouter proxy for managed model routing and OSS program proxy layer that mints ephemeral OpenRouter keys for users without BYOK API keys. two paths: pro plan users get their selected model proxied via OpenRouter; OSS program repos (hard-coded allowlist in config.ts) get free Claude Opus. BYOK env vars (PULLFROG_MODEL/OPENCODE_MODEL) always take precedence. frontend: OSS repos see a static "Opus (Free)" badge with the model dropdown disabled and no API key requirement. all models now carry openRouterResolve metadata for proxy target resolution. Made-with: Cursor * implement OSS program: proxy infrastructure for free model credits server-side OSS allowlist determines eligible repos. action mints ephemeral OpenRouter keys via OIDC-authenticated /api/proxy-token endpoint (idempotent on runId, $10 per-key safety limit). keys are disabled on workflow completion when no running refs remain. HWM-based usage sync tracks cumulative spend per account. schema: ProxyKey model, Account.usageUsd/activeKeyId, WorkflowRun.proxyKeyId action: OIDC credential stashing, resolveProxyModel uses server oss flag frontend: isOss flows from server page to components (no client allowlist) Made-with: Cursor * address PR review: repo cross-check, key retirement lifecycle, dead code removal - proxy-token: verify runId belongs to OIDC-authenticated repo via repo relation - add retireKey() shared primitive: disable in OpenRouter first, then mark disabled in DB - rotateKey: retire old active key after swap to prevent orphans - webhook: replace inline cleanupProxyKey with retireKey calls - syncAccountUsage: skip disabled keys - remove vestigial AccountPlan/plan field from action types - add disabled field to ProxyKey schema + migration Made-with: Cursor * replace deprecated opencode/mimo-v2-flash-free with mimo-v2-pro-free Made-with: Cursor * fix migration ordering: rename disabled migration to sort after table creation Made-with: Cursor * squash proxy key migrations into single migration Made-with: Cursor * add preview repo to OSS allowlist for testing Made-with: Cursor * populate OSS allowlist from oss-program-invitees.json Made-with: Cursor * format oss-program-invitees.json Made-with: Cursor * add installed public repos to OSS allowlist split ossRepos into three provenance-tracked lists: - internalRepos (pullfrog, colinhacks, RobinTail) - installedPublicRepos (external public non-fork repos with active installs) - invitees (from oss-program-invitees.json) also adds scripts/list-oss-candidates.ts to regenerate the installed list Made-with: Cursor * fix: resolve tokens before clearing OIDC env vars resolveTokens → acquireNewToken → isOIDCAvailable() checks ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN env vars. The new OIDC stashing code was deleting them in restricted shell mode before resolveTokens ran, causing it to fall through to the GitHub App path which requires GITHUB_APP_ID/GITHUB_PRIVATE_KEY. Made-with: Cursor * derive proxy-token auth from OIDC claims, add ensureWorkflowRun upsert - proxy-token no longer requires body.runId; uses claims.run_id + claims.repository - shared ensureWorkflowRun upsert called from both webhook and proxy-token - workflow_run_requested handler now eagerly creates WorkflowRun records - eliminates race condition between webhook and action proxy-token call Made-with: Cursor * hardcode PULLFROG_ACCOUNT_ID, document preview debugging lessons GitHub node IDs are constant — no reason for this to be an env var. Removes the trailing-newline bug that caused P2025 errors. Adds wiki docs on workflow testing, Vercel env gotchas, and Neon preview branch discovery. Made-with: Cursor * fix: parse OpenRouter create-key response correctly the API returns `key` at the top level, not inside `data` Made-with: Cursor * onboarding cards, unlock OSS model selection, simplify console - add OnboardingCard component with two states: workflow install and model+test (dispatches "Tell me a joke" for test run) - replace PromptBox overlay gates with dedicated onboarding cards; PromptBox is now just the form, always enabled - use hasWorkflowRuns DB check to decide onboarding vs promptbox - unlock ModelSelector for OSS repos (was locked to Opus badge); resolve proxyModel from repo's selected model alias in run-context - rename "API key" row to "Model costs" with pure client-side states: OSS covered, auto-resolve, free model, BYOK with env var names - add "(Recommended)" badge to model aliases with recommended: true - remove OSS_MODEL_DISPLAY_NAME constant and secrets-fetching logic Made-with: Cursor * update stale xai model snapshot Made-with: Cursor * rename Permissions to Security, hide git push toggle, add shell isolation toggle with disabled state for public repos, remove opentoad agent name from logs Made-with: Cursor * chevron hover states, sidebar hooks/security entries Made-with: Cursor * address PR review feedback: rename recommended→preferred, fix dispatch orphan, update wiki docs - rename `recommended` to `preferred` in model alias registry to distinguish from the UI "Recommended" badge (which is hardcoded for opus + codex only) - cancel precreated WorkflowRun when workflow lookup fails in dispatch-workflow - replace run_sql/vercel env pull in wiki docs with neonctl + Prisma pattern - extend scripts/neon-branch.ts to output DATABASE_URL via neonctl - add Recommended badge to GPT Codex alongside Claude Opus in ModelSelector Made-with: Cursor |
||
|
Colin McDonnell
|
c6a3ee0e9a |
show model name in footer, drop pullfrog.com link (#484)
* show model name in footer, drop pullfrog.com link add model slug to buildPullfrogFooter so every Pullfrog comment displays the active model (e.g. "Using `Big Pickle` (free)" or "Using `Claude Opus`"). remove the pullfrog.com link from all footers. Made-with: Cursor * reject <br/> tags in comment bodies, add prompt guidance add runtime validation in addFooter that throws if <br/> is followed by a non-blank line (breaks GitHub heading rendering). the agent sees the error and retries with clean markdown. also update Summarize mode prompt to explicitly forbid <br/> tags. Made-with: Cursor * fix <br/> guidance: move to event instructions, clarify blank line rule the formatting rule belongs in DEFAULT_PR_SUMMARY_INSTRUCTIONS (event instructions), not the Summarize mode prompt. clarify that <br/> must always be followed by a blank line before headings. Made-with: Cursor * generalize block-level HTML rule in summary instructions add a prominent top-level rule about requiring blank lines between ALL block-level HTML elements and markdown syntax, not just <br/>. Made-with: Cursor * move model to toolState instead of threading through params model is set once at startup and read everywhere — it belongs on toolState, not threaded as a separate param through 8 call sites. postCleanup runs without toolState so it just omits the model label. Made-with: Cursor * update models.dev snapshot (openai latest changed) Made-with: Cursor * add comment to models snapshot test explaining its purpose Made-with: Cursor |
||
|
Colin McDonnell
|
8a734c32f4 |
fix workflow detection, duplicate summaries, review resilience (#482)
* fix workflow detection when repos have many workflows Switch workflow lookup to GitHub's direct workflow-by-filename API so pullfrog.yml is found even when list endpoints paginate, and paginate installation scans in maintenance scripts to avoid partial coverage. Made-with: Cursor * fix review comment line resolution: pre-validate against diff hunks + auto-bisect fallback when submitting a review with inline comments, the tool now: 1. fetches the PR diff and validates each comment's line range against the actual hunk boundaries 2. moves invalid comments to the review body with a clear explanation 3. on 422 from GitHub (rare API quirks where valid-looking lines are rejected), bisects comments using disposable pending reviews to isolate failures 4. retries with only the comments GitHub accepts also fixes getHttpStatus (previously isStatusError) which wasn't recognizing Octokit errors, and warns in the tool description that each call creates a permanent visible review. Made-with: Cursor * remove bisect fallback, anchor review to checkout sha, make start_line optional - drop the auto-bisect-on-422 logic entirely; pre-validation catches the real issues and the 422 catch now just throws a clear actionable error - anchor review submission to checkoutSha so line numbers match the diff the agent actually analyzed (avoids stale-line 422s from new pushes) - make start_line optional and only set start_line/start_side when it differs from line (single-line comments don't need the range fields) - improve headMovedDuringReview detection to use latestHeadSha directly Made-with: Cursor * drop review comment pre-validation in favor of pinned commit_id The pre-validation (listFiles + hunk parsing) was checking comments against the current PR diff, but the review is now anchored to checkoutSha. When HEAD moves, pre-validation checks the wrong diff and can false-reject valid comments. GitHub's own commit_id-anchored validation is the correct source of truth. Made-with: Cursor * add logging to fetchExistingSummaryComment for duplicate summary debug Made-with: Cursor * fix duplicate summary comments: guard create_issue_comment for existing summaries When select_mode finds an existing summary comment (existingSummaryCommentId), create_issue_comment with type: "Summary" now auto-redirects to update instead of creating a new comment. Belt-and-suspenders for the token fix in selectMode.ts. Made-with: Cursor * document api auth patterns to prevent token misuse add wiki/api-auth.md explaining the two auth patterns (GitHub token vs Pullfrog JWT) and when to use each. add auth comments to all action-facing routes and their callers so the correct token is obvious. Made-with: Cursor * fix models.dev snapshot: filter beta models, add tiebreaker Skip models with any status (beta, deprecated) so nightly/experimental releases don't cause snapshot churn. Add lexicographic tiebreaker for stable ordering when release dates match. Made-with: Cursor * add tests to pre-push hook Made-with: Cursor * fix: decouple summary dispatch from re-review gate on pull_request_synchronize The summary workflow was never dispatched on new commits because the pull_request_synchronize handler broke early when prReReview was disabled, before reaching the prSummaryComment check. Now re-review and summary are dispatched independently. Made-with: Cursor * resolve merge conflicts in rebase.md and checkout.ts Made-with: Cursor * fix: restore checkout.ts and rebase.md from remote Made-with: Cursor |
||
|
Colin McDonnell
|
6d25adfd1a |
Agent & model refactor (#478)
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7 Made-with: Cursor * fix stale agent/effort refs, add tests for askpass + model resolution - reviewCleanup.ts: payload.agent -> payload.model, remove effort - selectMode.ts PlanEdit: remove delegation/subagent/effort references - pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY, add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY) - FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy - new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen) - new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override) - new: models.test.ts (19 tests — parseModel, resolution, registry invariants) - update models.dev snapshot Made-with: Cursor * fix changed-agents.sh to filter legacy agent files from CI matrix legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not exported from index.ts. changed-agents.sh now reads index.ts imports to build the active agent set and treats changes to inactive files as non-agent changes (opentoad canary only). Made-with: Cursor * remove MCP file tools, old agent harnesses, and obsolete security tests ASKPASS-based git auth makes the old MCP file tool security layer unnecessary: - token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it - agents now use native file tools (OpenCode builtin read/edit) deleted: - action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory) - action/mcp/index.ts (dead re-export) - agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts - opencode-runner.ts (inlined into opentoad.ts) - security tests that validated MCP file tool restrictions - commented-out three-step review flow (~300 lines) - sanitizeSchema/wrapSchema dead code from mcp/shared.ts - OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed) updated test prompts to use generic file ops instead of MCP tool names. restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense). Made-with: Cursor * bump actions/checkout v4 → v6 (node 24) node 20 actions deprecated june 2, 2026. Made-with: Cursor * temporarily disable fail-fast on agnostic tests to debug checkout@v6 Made-with: Cursor * re-enable fail-fast on agnostic tests Made-with: Cursor * fix test token mismatch: mint OIDC tokens scoped to target repo CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so ensureGitHubToken() mints a properly scoped token via OIDC. Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming instead of repeating it in every test file, and fixes preview-cleanup to remove workers from all queues (not just name-matching ones). Made-with: Cursor * fix ensureGitHubToken to try OIDC when app credentials are absent ensureGitHubToken only attempted token minting when GITHUB_APP_ID and GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds aren't exposed — so the guard prevented minting entirely. Made-with: Cursor * dead code cleanup: remove remnants of deleted agents, file tools, effort system remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps, orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale opencode-runner wiki refs, deleted test file references, and MCP file tool docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to globalSetup (runs once before forks instead of per-file, 19s → 200ms). Made-with: Cursor * address review feedback: remove dead code, update stale references - remove AGENT_OVERRIDE (only opentoad exists) - remove shellToolName plumbing (always restricted shell) - bump action version to 0.0.179 - remove CURSOR_API_KEY from all workflows/configs - remove OPENCODE_MODEL_MINI/MAX from workflows/docs - delete wiki/effort.md, rewrite docs/effort.mdx as "Models" - rewrite wiki/modes.md: orchestrator/subagent → single agent - simplify flag system: drop builtin flag extraction (debug, effort, timeout, agent), keep custom flag replacement only - reserve all legacy flag names to prevent custom flag conflicts Made-with: Cursor * regenerate lockfile after removing claude-agent-sdk and codex-sdk Made-with: Cursor * fix import ordering, add lockfile check to pre-push hook Made-with: Cursor * remove dead debug payload field, stale packageExtensions Made-with: Cursor * merge proc-sandbox and token-exfil into a single test proc-sandbox and token-exfil were duplicative — both tested that SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into token-exfil with shell:restricted (which actually exercises filterEnv) and the /proc attack vector hints from proc-sandbox. Made-with: Cursor * fix wiki adversarial.md to match actual tokenExfil validator Made-with: Cursor |