* add Amazon Bedrock as a routing slug
introduces a single `bedrock/byok` catalog entry that the harness translates
to the appropriate Bedrock model ID at run time via `BEDROCK_MODEL_ID`. routes
Anthropic IDs through claude-code (with `CLAUDE_CODE_USE_BEDROCK=1`) and
everything else through opencode's `amazon-bedrock` provider — keeps the
catalog flat for an audience that needs version pinning rather than aliasing.
accepts either `AWS_BEARER_TOKEN_BEDROCK` or `AWS_ACCESS_KEY_ID` +
`AWS_SECRET_ACCESS_KEY` for auth; both validated alongside `AWS_REGION` and
`BEDROCK_MODEL_ID` in `validateAgentApiKey`. catalog drift tests, the bumps
cron, and per-alias smoke scripts all skip routing slugs since there's no
fixed `resolve` to validate.
docs/bedrock.mdx walks through setup; wiki/model-resolution.md has a section
explaining why bedrock breaks the usual alias pattern.
closes pullfrog/pullfrog#40
* ci: add bedrock env vars to test workflows
mirrors the new bedrock provider's required env vars (AWS_BEARER_TOKEN_BEDROCK
inherited from org secret + AWS_REGION + BEDROCK_MODEL_ID hardcoded) into both
.github/workflows/test.yml files so the ci.test "env vars cover all provider
API keys" assertion passes.
* docs(bedrock): clearer setup flow + screenshot of model selector
restructures the setup section into three concrete steps in execution order:
select Bedrock from the dropdown, store the bearer token as a secret (Pullfrog
or GitHub — links to keys.mdx for the trade-off), then add region + model id
directly in pullfrog.yml since neither is sensitive. enable-model-access in
the Bedrock console moved to step 4 (only required once per model and only
when AWS rejects the call, not blocking on first run).
adds a screenshot of the console model selector with Amazon Bedrock selected
so readers can recognize the UI state they're aiming for.
* fix(bedrock): tolerate raw Bedrock model IDs in validateAgentApiKey
main.ts passes the resolved model into validateAgentApiKey
(`payload.proxyModel ?? resolvedModel ?? payload.model`). For Bedrock,
`resolveModel` translates `bedrock/byok` into the raw AWS model ID
(e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/` and so
trips parseModel inside getModelEnvVars.
Detect the no-slash case and re-run the bedrock setup check (auth +
region; BEDROCK_MODEL_ID is already enforced upstream by resolveModel).
Caught by PR #720 e2e dispatch on pullfrog/preview-720-bedrock —
"invalid model slug 'us.anthropic.claude-opus-4-6-v1' — expected
'provider/model'". Two regression tests cover the raw-ID path.
* fix(bedrock): always prepend amazon-bedrock/ prefix when bedrock-routed
opencode.ts was gating the prefix-injection on `!isBedrockAnthropicId(rawModel)`,
on the theory that Anthropic Bedrock IDs always go through claude-code. But
`PULLFROG_AGENT=opencode` is a documented escape hatch — when it forces
opencode for an Anthropic Bedrock model, the prefix still has to be added or
opencode fails with 'Model not found: <modelId>/.'.
The Anthropic-vs-other discriminant only belongs in resolveAgent. Once an
agent is selected, it should consistently honor the bedrock route.
Caught by the PULLFROG_AGENT=opencode + Opus 4.6 e2e on
pullfrog/preview-720-bedrock — run 25823437606.
* ui+docs(bedrock): bespoke setup callout + clearer docs
UI:
- BedrockSetupCallout in components/AgentSettings.tsx covers both the
Model costs section and the onboarding card. Detects bedrock via
resolveDisplayAlias().routing === "bedrock", shows a dedicated message
("store AWS_BEARER_TOKEN_BEDROCK as a secret, then put AWS_REGION +
BEDROCK_MODEL_ID directly in pullfrog.yml") + link to the setup guide.
Replaces the generic "X, Y, or Z is required" prompt that misrepresented
the three values as three separate secrets to add (and used the wrong
"or" connector for what's actually an AND).
- OnboardingCard re-uses the same callout with the gradient-card variant.
Docs:
- Drop the obsolete "Enable model access" step. AWS retired the manual
enrollment page; foundation models auto-enable on first invocation.
Anthropic models still need a one-time use-case form for first-time
users — surfaced under the AccessDenied troubleshooting entry.
- Drop the "Testing a different model in one run" PULLFROG_MODEL note.
It introduced the secrets-vs-vars distinction we want to keep out of
the bedrock setup story.
- Step 3 already recommends hardcoding region + model id in pullfrog.yml.
Workflow template:
- The default pullfrog.yml customers receive (utils/github/pullfrog.yml.ts)
now references AWS_BEARER_TOKEN_BEDROCK from secrets but inlines
AWS_REGION and BEDROCK_MODEL_ID as plain values. Matches the docs.
* fix(bedrock): three review-caught edges in routing + UI copy
Addresses three real issues from PR #720 review:
1. agent.ts: PULLFROG_MODEL=bedrock/byok no longer leaks the literal
sentinel "bedrock" downstream. resolveCliModel returns the alias's
resolve field verbatim, which for routing entries IS the sentinel.
Refactored both the env-override and slug-lookup paths through a
shared resolveSlug() that recognizes routing aliases and defers to
their backing env var (BEDROCK_MODEL_ID).
2. models.ts: isBedrockAnthropicId() now anchors on a discrete
dot/slash/colon-segment match (case-insensitive) instead of a
substring contains. The substring check was fragile in both
directions for inference-profile ARNs (BEDROCK_MODEL_ID accepts
ARNs per AWS docs) — a non-Anthropic profile whose user-chosen name
contained "anthropic" would mis-route to claude-code, and an
Anthropic profile whose name omitted it would miss
CLAUDE_CODE_USE_BEDROCK=1.
3. AgentSettings.tsx: BedrockSetupCallout's configured-state copy
showed "AWS_BEARER_TOKEN_BEDROCK configured" even when the user
satisfied the gate via AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY,
gaslighting access-key users about a secret they never set.
Detect which auth method is actually present and name the right
secret(s) in the success message.
Regression tests in models.test.ts (5 new isBedrockAnthropicId cases
including positive and negative ARN forms) and agent.test.ts (2 new
PULLFROG_MODEL=bedrock/byok cases). 171/171 action tests pass.
* yml template: add commented AWS access-key alternative for Bedrock auth
Mirrors the IAM access-key path verified end-to-end on PR #720 e2e
run 25830764987. Bearer token stays as the primary nudge; the access-key
pair is the fallback for users who can't mint Bedrock API keys.
* yml template: drop redundant 'or, alternatively' annotation
* ui+docs(bedrock): rewrite callout copy + refresh screenshot
Reframes the BedrockSetupCallout away from generic BYOK language to a
Bedrock-specific message: leads with "Amazon Bedrock is configured
entirely via environment variables", lists all four (auth, region,
model id), and ends with the requested CTA sentence ("click below to
learn more about Bedrock support in Pullfrog").
Promotes the "Bedrock setup guide" docs link from an inline anchor to
a prominent button (always visible, regardless of auth state). The
"Add AWS_BEARER_TOKEN_BEDROCK" affordance is now a secondary chip
shown only when no auth secret is configured.
Refreshes docs/images/model-selector-bedrock.png to capture the new
callout — the prior screenshot still showed the old generic
"BYOK / X, Y, or Z required" wording.
two bugs blocked the live matrix from reaching real APIs:
1. resolveModel returned PULLFROG_MODEL raw without passing it through the
alias registry. when CI set PULLFROG_MODEL=anthropic/claude-opus (alias),
the bare alias slug was forwarded to the Anthropic API as a model id and
404'd. now resolves via resolveCliModel first, with raw specifiers
(anthropic/claude-opus-4-6) still passing through unchanged.
2. the testEnvAllowList in docker.ts only forwarded Anthropic/OpenAI/Google
keys into the test container. XAI/DeepSeek/OpenRouter/Moonshot/OpenCode
keys got stripped, so every non-big-3 alias failed with "no API key found"
even when the secret existed. add all five to the allowlist.
Made-with: Cursor
remove duplicate model and agent log emitters, then print model, agent, push, shell, and timeout on separate startup lines so run settings stay concise and easy to scan.
Made-with: Cursor
* rename agent key to opencode and add skill invocation coverage.
add skill-invoke tests for claude and opencode with local play-based validation signals, update CI matrices, and include the current tracked refactors in this branch for review.
Made-with: Cursor
* exclude agent-specific skill-invoke tests from wrong agent in CI matrix
* address review follow-up and preserve workflow run UI tweak.
switch changed-agents ci coverage to exercise the opentoad implementation path while keeping the opencode expectation, and include the local workflow run client interaction updates requested on this branch.
Made-with: Cursor
* remove opentoad agent filename from runtime.
rename the opencode harness implementation file from opentoad.ts to opencode.ts and update ci coverage input accordingly so action code no longer carries the old filename.
Made-with: Cursor
* ensure security prompt bypass is set on every test fixture.
this keeps adversarial and permissions harnesses from being blocked by the default prompt-injection refusal path during CI security tests.
Made-with: Cursor
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
model resolution was duplicated inside each agent (opentoad, claude) and
PULLFROG_MODEL override was not considered when choosing the agent. now
resolveModel() runs first in main.ts, its result feeds into resolveAgent()
for agent selection, and the resolved model is passed to the agent via
ctx.resolvedModel. agents only handle their own fallback (opentoad: auto-select
via opencode models, claude: strip provider prefix).
also removes the hardcoded anthropic/claude-sonnet test runner default since
ANTHROPIC_API_KEY is no longer in CI.
Made-with: Cursor
* feat: add Claude Code agent for Anthropic model users
Re-adds Claude Code support (removed in #478) so users with Anthropic API
keys or Claude Code OAuth tokens can use their Claude subscriptions directly.
When an Anthropic model is selected and Claude Code credentials are available,
the system auto-selects the Claude agent instead of OpenCode. The harness
mirrors opentoad's security model: native Bash blocked via --disallowedTools,
MCP ShellTool for restricted shell, ASKPASS for git auth. Includes NDJSON
streaming, provider error detection, cache/cost tracking, browser skill,
and todo progress tracking.
Key changes:
- action/agents/claude.ts: full Claude Code harness
- action/utils/agent.ts: auto-select Claude for anthropic/* models
- action/utils/providerErrors.ts: extracted shared provider error detection
- action/utils/skills.ts: extracted shared skill installation (agent-aware)
- action/models.ts: add CLAUDE_CODE_OAUTH_TOKEN to anthropic envVars
- action/utils/docker.ts: add CLAUDE_CODE_OAUTH_TOKEN to test env allowlist
- CI: add claude to test matrix, pass CLAUDE_CODE_OAUTH_TOKEN secret
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove unused toolId variable, fix apiKeys test env cleanup
The apiKeys test cleanup stripped *_API_KEY vars but missed
CLAUDE_CODE_OAUTH_TOKEN which doesn't match that pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: strip provider prefix from PULLFROG_MODEL in Claude agent
the env override path was returning the raw value (e.g.
"anthropic/claude-sonnet-4-5") without stripping the provider prefix,
causing the Claude CLI to receive an invalid model ID.
Made-with: Cursor
* fix: remove dead cliPath field, add CLAUDE_CODE_OAUTH_TOKEN to workflows
remove unused cliPath from Claude agent RunParams, and pass
CLAUDE_CODE_OAUTH_TOKEN through all pullfrog.yml workflow templates
so users with Claude Pro/Team subscriptions can use their membership.
Made-with: Cursor
* fix: block Bash subagent in Claude Code disallowedTools
Made-with: Cursor
* chore: update model snapshot (opencode/openrouter latest → qwen3.6-plus-free)
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* agent & model refactor with ASKPASS git auth, UI restructure, clerk v7
Made-with: Cursor
* fix stale agent/effort refs, add tests for askpass + model resolution
- reviewCleanup.ts: payload.agent -> payload.model, remove effort
- selectMode.ts PlanEdit: remove delegation/subagent/effort references
- pullfrog.yml.ts: update env vars (drop GOOGLE_API_KEY/CURSOR_API_KEY,
add GOOGLE_GENERATIVE_AI_API_KEY/XAI_API_KEY/MOONSHOT_API_KEY/OPENCODE_API_KEY)
- FlagsSettings/RepoInstructionsSection: remove stale effort/timeout copy
- new: gitAuthServer.test.ts (10 tests — lifecycle, token delivery, tamper detection, script gen)
- new: agent.test.ts (4 tests — default opentoad, AGENT_OVERRIDE, invalid override)
- new: models.test.ts (19 tests — parseModel, resolution, registry invariants)
- update models.dev snapshot
Made-with: Cursor
* fix changed-agents.sh to filter legacy agent files from CI matrix
legacy agent files (claude.ts, codex.ts, etc.) are @ts-nocheck and not
exported from index.ts. changed-agents.sh now reads index.ts imports to
build the active agent set and treats changes to inactive files as
non-agent changes (opentoad canary only).
Made-with: Cursor
* remove MCP file tools, old agent harnesses, and obsolete security tests
ASKPASS-based git auth makes the old MCP file tool security layer unnecessary:
- token never in subprocess env, so symlink/gitattributes/hook attacks can't exfiltrate it
- agents now use native file tools (OpenCode builtin read/edit)
deleted:
- action/mcp/file.ts (file_read, file_write, file_edit, file_delete, list_directory)
- action/mcp/index.ts (dead re-export)
- agent harnesses: claude.ts, codex.ts, cursor.ts, gemini.ts, opencode.ts
- opencode-runner.ts (inlined into opentoad.ts)
- security tests that validated MCP file tool restrictions
- commented-out three-step review flow (~300 lines)
- sanitizeSchema/wrapSchema dead code from mcp/shared.ts
- OPENCODE_MODEL_MINI/MAX env vars (effort-level model overrides removed)
updated test prompts to use generic file ops instead of MCP tool names.
restored pkg-json-scripts + requirements-txt-attack (test --ignore-scripts defense).
Made-with: Cursor
* bump actions/checkout v4 → v6 (node 24)
node 20 actions deprecated june 2, 2026.
Made-with: Cursor
* temporarily disable fail-fast on agnostic tests to debug checkout@v6
Made-with: Cursor
* re-enable fail-fast on agnostic tests
Made-with: Cursor
* fix test token mismatch: mint OIDC tokens scoped to target repo
CI tests override GITHUB_REPOSITORY to pullfrog/test-repo but inherit
the runner's GITHUB_TOKEN (scoped to pullfrog/app), causing 401s on
every run-context fetch. Clear GITHUB_TOKEN in the test subprocess so
ensureGitHubToken() mints a properly scoped token via OIDC.
Also centralizes the default GITHUB_REPOSITORY in runAgentStreaming
instead of repeating it in every test file, and fixes preview-cleanup
to remove workers from all queues (not just name-matching ones).
Made-with: Cursor
* fix ensureGitHubToken to try OIDC when app credentials are absent
ensureGitHubToken only attempted token minting when GITHUB_APP_ID and
GITHUB_PRIVATE_KEY were set. In CI, OIDC is available but app creds
aren't exposed — so the guard prevented minting entirely.
Made-with: Cursor
* dead code cleanup: remove remnants of deleted agents, file tools, effort system
remove unused @anthropic-ai/claude-agent-sdk and @openai/codex-sdk deps,
orphaned file-tool security tests, dead GEMINI_MODEL passthrough, stale
opencode-runner wiki refs, deleted test file references, and MCP file tool
docs. rename docs/effort → docs/models. fix vitest setup: move dotenv to
globalSetup (runs once before forks instead of per-file, 19s → 200ms).
Made-with: Cursor
* address review feedback: remove dead code, update stale references
- remove AGENT_OVERRIDE (only opentoad exists)
- remove shellToolName plumbing (always restricted shell)
- bump action version to 0.0.179
- remove CURSOR_API_KEY from all workflows/configs
- remove OPENCODE_MODEL_MINI/MAX from workflows/docs
- delete wiki/effort.md, rewrite docs/effort.mdx as "Models"
- rewrite wiki/modes.md: orchestrator/subagent → single agent
- simplify flag system: drop builtin flag extraction (debug, effort,
timeout, agent), keep custom flag replacement only
- reserve all legacy flag names to prevent custom flag conflicts
Made-with: Cursor
* regenerate lockfile after removing claude-agent-sdk and codex-sdk
Made-with: Cursor
* fix import ordering, add lockfile check to pre-push hook
Made-with: Cursor
* remove dead debug payload field, stale packageExtensions
Made-with: Cursor
* merge proc-sandbox and token-exfil into a single test
proc-sandbox and token-exfil were duplicative — both tested that
SANDBOX_TEST_TOKEN couldn't be exfiltrated. consolidated into
token-exfil with shell:restricted (which actually exercises filterEnv)
and the /proc attack vector hints from proc-sandbox.
Made-with: Cursor
* fix wiki adversarial.md to match actual tokenExfil validator
Made-with: Cursor
* Add utils for r2 upload
* Add the tool and new routes
* fix auth issue
* sign headers
* add comment
* use our own API key to auth signed uploads
* Restructure things slightly
* tweak
* tweak
* add comments
* tweak
* revert a thing
* twaek
* drop mime type filtering
* new incarnation of mime type filtering
* jsut allow all octet-streams
* simplify further
* tweak
* update lockfile
---------
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Colin McDonnell