972 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Colin McDonnell	01e4daa0b5	checkout_pr: refuse unconditionally on dirty working tree (#808) ... * checkout_pr: refuse unconditionally on dirty working tree drop the live-HEAD comparison from the guard introduced in #796. any checkout_pr call with staged or unstaged changes now throws, even when HEAD is already on pr-N. no stashing, no idempotent escape hatch. motivation is the zed-industries/cloud (2026-05-18) incident: shared-cwd subagents make "carry edits along" semantics dangerous, and the HEAD-equality predicate let a re-checkout silently inherit working-tree state from a sibling agent. forcing commit/discard before any PR-context operation eliminates the entire carry-forward failure class. error names the PR number, lists dirty paths, and tells the agent to commit/push/restore/clean before retrying. * improve dirty-tree error: precise discard commands copilot caught two sloppy bits in the error string: - "push" alone does not clean a dirty tree (needs commit first) - bare `git clean` is a no-op without `-fd` reword to "commit (then push if needed), or discard with `git restore --staged --worktree .` / `git clean -fd`" so the guidance is actually actionable. * checkout_pr: initial-branch invariant setupGit captures `toolState.initialBranch` at run start via live `git rev-parse --abbrev-ref HEAD`. checkout_pr refuses unless current HEAD matches the run-entry branch or the target `pr-N` (idempotent same-PR re-checkout). uses live rev-parse, not toolState.issueNumber (poisonable per the PR #796 review). refusal error names the current branch, target PR, recovery path (`git checkout <initialBranch>` with the literal branch name), and explicitly states routing around via the `git` tool is not sanctioned. closes the zed-industries/cloud (2026-05-18) shape where a subagent parked HEAD on someone else's `pr-X` and the orchestrator's next checkout_pr inherited that position. * reviewfrog: enforce canonical diff + pre-commit halt; align Build dispatch extend REVIEWER_SYSTEM_PROMPT with two prepended HARD CONSTRAINTS: - first action MUST be `git diff origin/<base>` (single-rev, captures uncommitted). no other diff first; no checkout_pr; no alt-ref fetches; no branch listing; no `gh pr list`. - empty canonical diff + claimed-changes dispatch ⇒ reply exactly with `no changes detected — likely pre-commit Build self-review; orchestrator should commit then re-dispatch` and stop. do not guess PR numbers (the zed thrash that ended in `checkout_pr({2582})`). reshape Build mode reviewfrog dispatch step around a verbatim template that names: (a) the situation is pre-commit, (b) canonical diff command, (c) halt-on-empty-diff rule. orchestrator side now says the same thing as the reviewer's baked-in prompt. delegation-discipline bullets and orchestrator-evaluation guidance kept intact. * checkout_pr: handle detached-HEAD entry in initial-branch invariant pullfrog incremental review caught a defense-in-depth gap: `git rev-parse --abbrev-ref HEAD` returns the sentinel string `"HEAD"` on detached entry, which is the default `actions/checkout` state for `pull_request` events. with the previous string-typed `initialBranch`, both the captured value and the live probe would equal `"HEAD"` on any detached state, trivially satisfying the invariant — including a subagent doing `git checkout --detach <sha>`. discriminate the captured HEAD: probe `git symbolic-ref --short HEAD` first (works on named branches), fall back to `git rev-parse HEAD` (SHA) on detached entry. store as `{ kind: "branch"; name } | { kind: "detached"; sha }`. checkout_pr runs the identical probe at call time and compares like-with-like (branch name vs branch name, SHA vs SHA). refusal error renders both heads via a small `describeHead` helper and chooses the right `git checkout` recovery target (branch name or SHA). no inline-discriminant `as` casts — uses a top-level `headsEqual` that narrows via the discriminator.	2026-05-22 22:38:57 +00:00
David Blass	d3b5340583	fix: audit batch — MCP timeouts, entryPost, vip_audit 404s, and 6 more (#824) ... * fix: 9 unaddressed log-audit / run-audit findings Co-authored-by: Cursor <cursoragent@cursor.com> #815 entryPost stdlib-only imports; #823 MCP timeoutMs on checkout_pr/shell; #816 FREE_FALLBACK → opencode/big-pickle; #822 chunk GraphQL nodes ≤100; #817/#821 vip_audit 404 skip paths; #813 longer serializable retries; #818 run-context handler-entered log; #805 audit severity template. * fix: update footer test for big-pickle fallback slug Co-authored-by: Cursor <cursoragent@cursor.com> * fix: anneal round 1 — ghaCore getState casing, post-hook timeout Match @actions/core STATE_ key semantics (no uppercasing), cap postApiFetch at 30s, trim serializable retries to stay under GitHub's 10s webhook window, log Clerk failures in getUserTokenByGithubLogin. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: drop run-context handler log (#818 deferred) The #692 client-side fix is already on main; residual SyntaxError hits are old action pins. Per-request log added noise without fixing anything. Co-authored-by: Cursor <cursoragent@cursor.com> * document per-issue Closes syntax for audit PRs GitHub only auto-closes the first issue when numbers are comma-separated; /audits and AGENTS.md now require Closes before each issue number. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: anneal round 2 — outreach privacy, alert resilience, vertex cleanup Filter private repos from VIP authority output, harden console alert lines against DB failures, drop spoofable changesets body check, and unset GOOGLE_APPLICATION_CREDENTIALS after vertex credential cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: drop codexHome re-export of detectCodexRefresh Import detectCodexRefresh directly from codexRefreshDetect.ts everywhere; rename the unit test file to match. codexHome.ts stays install-only. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: drop deprecated minimax-m2.5-free; add paid minimax-m2.5 Remove the deprecated free MiniMax promo from the catalog, docs, and tests. BYOK fallback and picker copy stay on opencode/big-pickle. Add opencode/minimax-m2.5 and openrouter/minimax-m2.5 for Zen BYOK and Router. Pin #816 regressions with freeFallbackCatalog and runErrorRenderer unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: hidden minimax-m2.5-free fallback for stored slugs Re-add opencode/minimax-m2.5-free as a hidden fallback alias to big-pickle so repos with the legacy slug still resolve as free. Drop live Zen API experiment tests in freeFallbackCatalog.test.ts. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-22 16:48:25 +00:00
Colin McDonnell	e65dbe420c	Use Vertex Claude Opus in vertex-claude CI smoke — Haiku quota exhausted.	2026-05-22 16:46:23 +00:00
Colin McDonnell	58e5b74cb8	Update footer test for big-pickle BYOK fallback.	2026-05-22 16:06:50 +00:00
Colin McDonnell	7c5ed7add0	Rename OSS_PROXY_MODEL to DEFAULT_PROXY_MODEL; default Router proxy to Kimi K2.6. ... Derive the platform default from moonshotai/kimi-k2 openRouterResolve, add models.dev drift coverage, and promote repos on default-branch workflow pushes when still needs_setup.	2026-05-22 15:52:20 +00:00
Colin McDonnell	fb22cb3ae3	release: bump action to 0.1.11 v0.1.11	2026-05-20 16:49:37 +00:00
Colin McDonnell	c43ed65c3b	Add Vertex AI routing support (#753) ... * add Vertex AI routing support * include Vertex smokes in action CI	2026-05-20 16:49:08 +00:00
Colin McDonnell	09344a9ec9	release: bump action to 0.1.10 v0.1.10	2026-05-20 15:01:35 +00:00
Colin McDonnell	1b201352b5	release: bump action to 0.1.9 v0.1.9	2026-05-20 14:09:52 +00:00
Colin McDonnell	6c166ac1cc	fix: prevent cross-PR push from subagent-induced branch switch (#796) ... * fix: prevent cross-PR push from subagent-induced branch switch A workflow_dispatch run for zed-industries/cloud (workflow run 26036155393) force-pushed the orchestrator's work onto an unrelated engineer's PR branch (origin/reactivate-pro-plan, PR #2582). The orchestrator's reviewfrog subagent called checkout_pr({pull_number: 2582}), which (1) moved the shared working tree to pr-2582 and (2) persisted pushDest pointing at reactivate-pro-plan. The orchestrator's subsequent commit + push_branch then clobbered the victim PR. Recovery + disclosure in PR #2584. Three compounding bugs closed here: 1. checkout_pr dirty-tree guard had a first-call hole: the previous condition required ctx.toolState.issueNumber to already be set, so on workflow_dispatch runs the first checkout_pr (commonly from a subagent) bypassed the guard entirely. Now any PR switch with a dirty tree is refused, including the first switch of a run. Idempotent same-PR re-checkouts are still absorbed by alreadyOnBranch inside checkoutPrBranch. 2. push_branch trusted sticky pushDest blindly. Added a backstop: refuse pushes where the local branch matches /^pr-(\d+)$/ AND pushDest.remoteBranch differs from it AND the current run is not scoped to PR N (event.is_pr === true && event.issue_number === N). This catches subagent-induced silent branch switches even if a future bug reintroduces a first-call hole in fix #1. 3. Build-mode self-review prompt told the orchestrator to ship "the output of git diff" to the reviewer. The model in this run synthesized git diff main...HEAD, which excludes uncommitted work — and Build self-review runs BEFORE the commit, so the reviewer saw an empty diff and thrashed, eventually calling checkout_pr on a random PR to find something to look at. Prompt now specifies git diff origin/<base-branch> (two-dot, no HEAD), which compares the working tree against the remote base. Refs: zed-industries/cloud workflow run 26036155393 zed-industries/cloud#2582 (victim) zed-industries/cloud#2584 (disclosure) * review: key dirty-tree guard on current branch + drop 'two-dot' misnomer Address review feedback on PR #796. 1. checkout_pr dirty-tree guard now keys off the live current branch (git rev-parse --abbrev-ref HEAD), not ctx.toolState.issueNumber. issueNumber is ALSO set by get_issue / get_issue_comments / get_issue_events, so a subagent doing get_issue(N) followed by checkout_pr(N) on a dirty tree would have bypassed the original guard (issueNumber === pull_number). The current branch is the actual primitive for "would this call move HEAD" — querying it directly avoids correlating on toolState that other tools write to. 2. modes.ts: drop the wrong "two-dot" label on git diff origin/<base>. That's the single-rev form, not two-dot. Copilot was right that the label was confusing/contradictory with the actually-shown command.	2026-05-20 05:24:24 +00:00
Colin McDonnell	a0576a702a	opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767) ... * opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference; `opencode_v2.ts` is the active runner via `agents/index.ts`. Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish tool call. The fix only exists in the `1.14.x`+ line, which also ships the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory. Also unblocks the Codex ChatGPT-subscription auth path. Surface area: - drop `init` / `message` / `result` / `tool_result` event types and handlers (no longer emitted at v1.14+ per upstream `cli/cmd/run.ts:588-601`). - `tool_use` is now a single event covering both `state.status: "completed"` and `"error"`. duration / subagent-finish bookkeeping moves from the v1 `tool_result` handler into the consolidated `tool_use` handler. - new `reasoning` event handler — gated on `--thinking`, surfaces Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to `baseArgs`. - drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at v1.15 the `task` tool callID is stable across the whole `tool-input-* → tool-call → tool-result/tool-error` chain (`session/processor.ts:282-330`). exact-match map is sufficient. - drop `experimental.batch_tool: true` from injected config — declared but inert at v1.15. re-add once upstream wires it back. - bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames the platform-specific binary into `opencode.exe` for every OS now). Validated locally: - `pnpm test` 610/610 ✓ - `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓ - `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls, multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓ (this is the headline `thought_signature` fix) - runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓ - runtest opencode: skill-invoke and mcpmerge fail (model-behavior drift on the new system prompt; wiring confirmed intact via direct repro showing both `robinMCP` and `pullfrog` MCP tools exposed). Tracked for follow-up; does not gate the migration. Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at v1.15 — verified upstream and reused as-is. Bus subscription via `bus.subscribeAll()` and the `event` hook still fan out every payload. * model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename) The v1.14+ postinstall.mjs renames the platform-specific binary to `bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows. Mirrors the fix in action/agents/opencode_v2.ts. * opencode v2: set PWD env explicitly to fix skill / project-config discovery Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15 reads `process.env.PWD` first (with `process.cwd()` as fallback) when resolving the SDK client's `directory` parameter — see upstream `cli/cmd/run.ts:282`: const root = Filesystem.resolve(process.env.PWD ?? process.cwd()) We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the `action/` directory, not the cloned test repo. Result: opencode creates two instances per session — one at `process.cwd()` (correct) and one at `PWD` (wrong) — and the agent's session runs in the PWD-derived one, which can't see the project's `.opencode/skills/` or `.claude/skills/`. Empirically traced via the full opencode stderr trace under the runtest harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a second `service=default directory=<harness-pwd> creating instance` line per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})` exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value` becomes accessible too. Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓, restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration match, passes on retry; unrelated to PWD). * opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end` on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block "thought for X s" duration straight from the runtime. The v1 ThinkingTimer heuristic — measuring wall-clock between markToolResult and the next markToolCall — was an approximation when no native source existed; with v2 it's redundant and noisy (it would log alongside the real reasoning event, and conflated network latency with model thinking). Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()` helper, both `markToolCall` / `markToolResult` call sites in `tool_use`. The `reasoning` handler now reads `part.time.start/end` directly and prefixes the visible preview with `(X.Ys)`. Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate) Output now: `» thinking (4.0s): <preview>` (one line, sourced) For models that don't emit reasoning (Sonnet without extended thinking, GPT-4o, etc.), there's just no thinking line — which matches reality better than the gap-heuristic, which would fire on any pause >3s including provider-side latency that wasn't actual model reasoning. Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations. * claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment claude-code 2.1.x reads `process.env.PWD` and registers it as a "session" additional-working-directory when it differs from `process.cwd()` (per the bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...) j.set(H, { path: H, source: "session" })`). Without overriding PWD on the spawn env, claude inherits the harness's PWD via `...process.env` — under `pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo — and adds the wrong dir to the agent's allowed working set. Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of "agent's session sees the wrong cwd" failures on the claude side. Also refresh the stale `action/agents/opencode.ts` reference in entryPost.ts to point at opencode_v2.ts (the active runner), with the v1 file noted as kept-for-reference. * opencode: extract shared helpers into opencodeShared.ts; v2 cleanup Code-quality pass on the v2 work: 1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers between v1 and v2: - `OpenCodeConfig` type - `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin) - `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1 and re-imported by v2 via a back-reference) - `installOpencodeCli({ binPath })` (parameterized — v1 passes `bin/opencode`, v2 passes `bin/opencode.exe` via a per-version `installCli` lambda; matches each pinned version's npm shape) - `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1 keeps a one-line `export { geminiHighThinkingOverrides }` re-export so `opencode.test.ts` keeps working unchanged. Once v1 is retired (post burn-in) opencodeShared collapses back into v2. 2. `opencode_v2.ts` cleanup: - drop dead state (`currentStepId`, `stepHistory` were write-only — their reader was the v1 `tool_result` handler we deleted) - hoist `state` in `tool_use` handler; replace nested-ternary payload extraction with a `terminalPayload(state)` helper - extract `formatPartDuration(time)` for the reasoning-block "(X.Ys)" suffix - tighten `OpenCodeBusEnvelopeEvent` type to include `tool` / `callID` fields directly, drop the `partWithToolFields` cast - trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning handler, tool_use handler, bus envelope handler all shortened - `step_start` becomes an explicit `() => {}` no-op so the dispatcher doesn't log "unhandled event" for every step 3. `subagentRegistration.test.ts` retargeted at the new file split — reads opencodeShared.ts for the buildReviewerAgentConfig assertions and opencode_v2.ts for the orchestrator-model wire-through. Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format + typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified against the refactored v2 runtime. * opencode v2: address PR review feedback Three fixes from the inline review threads on #767: 1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk- level `markActivity()` resets the module-level idle counter, so the per-event `getIdleMs()` sample inside the dispatch loop was always ~0ms — the "no activity for Xs" diagnostic never fired. Replaced with a runner-local `lastEventAt` so we measure real event-to-event silence instead of chunk-arrival latency. Drop the unused `getIdleMs` import. 2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`, `lastProviderError`, and `recentStderr` are closed over by the `handlers` const but were declared after it. No current bug because handlers only fire inside the awaited `spawn()`, but a future refactor that triggers a handler synchronously during setup would surface a TDZ. Hoisted above `handlers`. 3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at L566): leave a `TODO` comment marking the gap until `AgentUsage` grows a `reasoningTokens` field — separate PR with schema work. Cost totals stay correct because `part.cost` is summed independently. Other thread states for the record: - Copilot L63 (geminiHighThinkingOverrides import from legacy): already fixed by the opencodeShared.ts extraction in 83a7cab. - Copilot L672 (ThinkingTimer over-reports on terminal events): already fixed by dropping ThinkingTimer in a1e536b — we use opencode's own `reasoning.part.time.{start,end}` for thinking durations now. - Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked the bus-envelope flow; the plugin filters orchestrator events except for status=running task dispatches, and bus-envelope returns before calling handlers.tool_use on those. No double-fire under current code. Validated: 610/610 unit tests, lint + format + typecheck clean, skill-invoke-opencode ✓. * DX: flip pnpm play / pnpm runtest to docker-by-default Restores the script shape wiki/docker.md has documented since the docker rewrite (#750). PR #756 inadvertently reverted action/package.json's gha/play/runtest scripts to host-only and dropped the :local variants; the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2 migration agent ran `pnpm play --raw …` host-side throughout because the host entry was the only thing that existed. scripts (root → action): - pnpm play → pnpm -C action gha play.ts (docker, default) - pnpm play:local → pnpm -C action play:local (host) - pnpm runtest → pnpm -C action gha test/run.ts (docker, default) - pnpm runtest:local → pnpm -C action runtest:local (host) - pnpm gha is restored in action/package.json (re-adds `node gha.ts`) action/package.json deliberately ships only the :local variants — bare `pnpm -C action play` now errors instead of silently bypassing docker. This is a tradeoff per the user prompt's "consider whether NAMES should change" hint: the explicit error is worth the small CI churn. CI workflows: `.github/workflows/test.yml` and `action/.github/workflows/test.yml` flipped from `pnpm runtest …` to `pnpm runtest:local …`. Semantics unchanged — they still execute `node test/run.ts` directly on the GHA Linux runner; nesting docker on GHA is unnecessary overhead. Only the script name changed to match the new package.json. Webhook tester: the existing root `pnpm play` was actually a webhook handler smoke harness (root play.ts), unrelated to the action runtime. Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so `pnpm play` can carry the docker-by-default action shortcut without collision. README updated. File headers updated: - action/play.ts: invocation block now points at `pnpm play` / `pnpm play:local` - action/test/run.ts: same - action/gha.ts: usage block calls out the new shortcut wrappers AGENTS.md: extended the existing "local sanity checks of action tool logic" rule with the play / play:local / runtest / runtest:local selection guidance and the `cd action; pnpm play` footgun note. wiki/docker.md unchanged — already described the now-real shape. * test/crossagent: add codex-auth smoke Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies: - installCodexAuth() materializes auth.json under the test HOME - opencode routes openai requests through ChatGPT subscription auth (no OPENAI_API_KEY in env, AT path forced via expires: 0) - the refresh chain advances during the run (refresh_token rotates) - detectCodexRefresh() would surface the rotation to entryPost.ts The post-hook write-back fetch isn't reachable from `pnpm runtest` (it's a separate GHA `post:` step). The integration boundary that matters end-to-end is "did the on-disk auth.json change in a way detectCodexRefresh recognizes" — that's exactly what this test asserts. CI wiring (already committed in a1c1fd4f as part of the DX flip): - .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in action-agents env block - action/.github/workflows/test.yml: same; codex-auth in the hardcoded test matrix with a claude exclude The provisioning step on the user's side is `gh secret set CODEX_AUTH_JSON --repo pullfrog/app < auth.json`. ci.test.ts: expectedAgentEnvVars now includes provider `managedCredentials` so the "env vars cover all provider API keys" invariant stays self-correcting as more managed credentials land. * docs(codex-auth): make storage requirement unmissable A previous reviewing agent on this branch came away thinking `CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't — `entryPost.ts` rewrites the rotated refresh token after every run, and GH Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres storage breaks the refresh chain on the first rotation (~1h silent expiry). - wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold, with the words "GitHub Actions secrets DO NOT WORK" verbatim and an enumeration of broken alternatives. - action/utils/codexHome.ts + action/entryPost.ts: header comments now loudly contrast Pullfrog secret store vs GH Actions and explain the writeback constraint. - AGENTS.md: terse one-bullet rule next to the model-resolution rule so future agents don't repeat the mistake. - .github/workflows/test.yml + action/.github/workflows/test.yml: added a comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a CI smoke-testing shortcut, not the canonical pattern. CI wiring itself unchanged per scope. * auth codex: auto-open device URL, drop --scope flag - detect `https://auth.openai.com/codex/device...` from codex CLI output and best-effort launch it in the user's default browser (open / xdg-open / cmd start, wslview fallback on linux). gated so we only open once per flow; failures are swallowed so manual copy-paste still works. - drop the `--scope` flag entirely. the device-code flow is fundamentally interactive (browser approval), so a "skip-the-prompt" flag for just one of the prompts was dead weight. collapses scope selection to "always prompt on org-owned, always account on user-owned". * rename gha→docker, flip play/runtest defaults to host the previous shape conflated "real GitHub Actions" with the local docker container that mocks it, and made the slow docker path the default for fast-iteration scripts. - `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`, tmpdir, error messages) - `pnpm play` / `pnpm runtest` now default to host (fast iteration); `pnpm play:docker` / `pnpm runtest:docker` run inside the container - `pnpm gha` → `pnpm docker` (the container runner shortcut) - `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the bare name implied a webhook server, which hookdeck-cli already is) - update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`, `README.md`, `AGENTS.md`), CI workflows (`.github/workflows/test.yml`, `action/.github/workflows/test.yml`), and code headers (`action/{play,test/run,utils/runFixture}.ts`, `webhook.ts`, `action/test/coverage.ts`) `action/commands/gha.ts` keeps its name — it's the real GitHub Actions entry point for the `pullfrog gha` CLI command (not the docker mock). * fix(codex): route post-hook writeback through apiFetch + conditional skip Three threads addressing PR #767 followups. action/entryPost.ts: replace raw fetch() with apiFetch() so the PUT /api/runtime/secret call carries the x-vercel-protection-bypass header/query when targeting a preview deployment. raw fetch silently 401s against the Vercel SSO gate, so every preview-env Codex run was losing its rotated refresh token. production is unaffected (no SSO). action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON via new TestRunnerOptions.skipIf hook. when the secret is absent (forks, contributors without it), runTestForAgent short-circuits to a passing-with-skipped ValidationResult before any agent spawn — so the matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI on pullfrog/app and dev-local with .env both still run the test for real. printSingleValidation/printResults now render skipped entries distinctly. doc/comment drift: - docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag mention (removed in 10be96db, scope is now always interactively prompted or implicit). - wiki/codex-auth.md: tighten Claude-defense wording — materialization is agent-gated (opencode/opencode_v2 harness), not model-gated; opencode runs with non-OpenAI models still materialize the file, it's just not read. - action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts → pnpm docker / docker.ts (renamed in a2a63929). - app/api/runtime/secret/route.ts: refer to the save-time scope prompt instead of the dropped --scope flag. * smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS upgrade crossagent/smoke prompt to call pullfrog_git status before set_output. this exercises the 2nd model→agent round-trip across every providers-live flagship, catching bugs like the Gemini thought_signature echo that single-tool-call tests can't see. also adds the "bar for adding new LLM-driven tests" section to wiki/action-tests.md and an extension to the existing AGENTS.md no-tests rule pointing at it — prefer upgrading existing matrix entries over adding new ones. local: pnpm runtest smoke opencode passes against both anthropic/claude-sonnet-4-6 and google/gemini-pro. --------- Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>	2026-05-20 04:05:16 +00:00
David Blass	4d1fd5ea1a	fix: 4 unaddressed log-audit / run-audit findings + close 10 already-resolved issues (#785) ... * fix: 4 unaddressed log-audit / run-audit findings closes 4 issues with code changes; 7 issues are already addressed by #769 and 3 are deferred — see PR description. #782 Anthropic 401 → `isApiKeyAuthError` now matches the direct-Anthropic 401 shape (`Failed to authenticate. API Error: 401 ...`, `authentication_error`, `Invalid bearer token`, `api_error_status=401`) so revoked / mistyped / rotated `ANTHROPIC_API_KEY` users see the formatted rotate-key CTA instead of a raw 401 JSON dump. #778 billing-class provider errors → `providerErrors.ts` now classifies `CreditsError` / `FreeUsageLimitError` / `Insufficient balance` / `spending cap` as `provider billing exhausted` *before* status-code patterns can win and tag them as transient `auth error (401)` / `rate limited (429)`. `agentHangReport.ts` swaps the bare "Pullfrog stalled — auth error" headline for a billing-specific CTA (extracts the provider's billing URL when present). #775 silent IncrementalReview swallows `BillingError` → `reportErrorToComment` now optionally falls through to creating a fresh issue comment on `toolState.issueNumber` when no progress comment exists. Wired with `createIfMissing: true` from the `BillingError` / `TransientError` paths in `proxy.ts` so silent triggers (`pull_request_synchronize`) finally surface the router-balance signal on the PR instead of only in the GH job summary. #773 `currentUser()` inside `after()` → `fillInstallerIdentityIfMissing` is split into `resolveInstallerIdentity` (must run inside the request body) and `fillInstallerIdentity` (DB-only, safe in `after()`). The `/console/[owner]` caller now resolves Clerk identity up-front and defers only the prisma write, fixing the broken installer-identity backfill on org-console first-admin visits. Co-authored-by: Cursor <cursoragent@cursor.com> * add /audits cursor command for triaging run-audit + log-audit issues Co-authored-by: Cursor <cursoragent@cursor.com> * review prompt: tighten body-section bar + inline technical-details (#770) * review prompt: tighten body-section bar + add inline technical-details Two layers of tightening to the Review/IncrementalReview prompts in PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step): 1. Reframe inline-vs-body split. Body `### ` sections are now reserved for concerns that genuinely have no line to anchor to — absence, sequencing, design decisions, scope questions, architectural risk. Drop the "cross-cutting concerns" framing (misled the agent into either filing nothing in the body or filing multi-file anchored findings there). 2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step 6) and IncrementalReview (step 8) aggregate phases. Diagnosis from PR #767's auto-review: on substantial PRs the agent surfaced findings but routed all of them inline, producing reviews with zero `### ` body sections even on diffs where non-anchored concerns clearly existed. 3. Replace the abstract `### ` example with a concrete non-anchored one ("Legacy `opencode.ts` has no documented deletion plan") so the agent pattern-matches the absence-shaped finding, not a line-bug. 4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT so inline comments can carry a `<details>Technical details</details>` block when the fix has cross-file implications. Rename the existing "Agent details" inline collapsible to "Technical details" for consistency with body sections. 5. (Carried over from prior uncommitted work) Restructure the review metadata block from `<details>Review metadata</details>` into an HTML comment + an italic TL;DR commit-range line. The HTML comment keeps the metadata addressable for downstream agents without eating user-visible review real estate. No tests touched. * wiki: document multi-model end-to-end eval pattern * feat(promo): cookie-stashed promo codes for onboarding rewards (#771) * feat(promo): cookie-stashed promo codes for onboarding rewards Operator hands out a link like https://pullfrog.com/start?promo=FROGGY; middleware validates the code against an in-code registry, stashes it in an HttpOnly cookie, and the install callback applies the reward once the GH-side account exists. v1 reward: unlimited_runs (lifts the monthly free-runs cap to 1M, same convention prod-grandfathered accounts use). No schema changes. Idempotent across reinstalls via the lte: 100 gate. * fix(promo): integrate handler into existing proxy.ts (Next 16 rename) * docs(promo): clarify sentinel + sync plan doc with renamed paths * feat(promo): add FOUNDATIONS code * feat(promo): show applied promo code in console * refactor(promo): move cookie set to client-side * docs(promo): point JSDocs at PromoCookieSetter, not proxy.ts * billing: cap counts only successful runs (#787) * billing: cap counts only successful runs `reserveRun` was counting `WorkflowRun` rows regardless of status against `Account.includedMonthlyRuns`. Failed / cancelled / skipped / timed-out runs consumed cap slots even though their `billableCents` got zeroed on the completion webhook — pushing paying users into billable territory earlier than the contract implies. `inthhq` paid for 2 extra runs this month because 2 failed runs ate 2 of their 100 free slots. Cap query now filters on `CAP_CONSUMING_STATUS = "success"`. Only runs that actually deliver value consume slots; in-flight (`running`) runs hold no slot until they terminate as success (burst-bypass risk is theoretical given GH Actions concurrency limits). Shared constant lives in `utils/billing.ts` and is used in lockstep by three call sites: `reserveRun` (live cap gate), the billing API's `runsThisMonth` (dashboard progress bar), and the billing-report script's `cap` column. Script's `cap` cell was also broken independently — it compared `monthBillableRuns` (overage count) against `includedMonthlyRuns` (free cap), so `inthhq` rendered as `125/100 (over)` when the meaningful ratio is `223/100 (over)`. Fixed to use `mRuns/cap`, which is the same predicate the live billing path uses. * move CAP_CONSUMING_STATUS to workflowRunStatus.ts + wire script through it Per copilot review: the JSDoc claimed the billing-report script used the constant in lockstep, but the script kept `status: "success"` inline. The script imports from raw-node ESM and can't pull in `next/server`, so it couldn't import from `utils/billing.ts`. Moved the constant to `utils/workflowRunStatus.ts` (already Next-free, already the home of `CONCLUSION_VALUES`) and updated all three call sites to import from there. Script's `mRuns` query now uses `CAP_CONSUMING_STATUS` directly, making drift impossible. * learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743) * learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy three audit fixes on top of the recent learnings overhaul (#717): - `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry when a body has non-whitespace content before the first heading. the prompt instructs the agent NOT to slurp the whole file when a TOC is present, so without this any preamble lines were silently invisible (realistic transitional case: an agent partially restructures a legacy free-text body and leaves bullets above the first `## `). - server-side PATCH route now applies the same line-boundary-aware truncation as the action (defense in depth via a shared `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from `action/internal`). the raw `.slice` it used before could leave a mid-heading tail on any caller that bypassed the client-side truncate, breaking the next-seed TOC parse. removes the duplicated cap constant. - `buildLearningsSection` intro no longer asserts "accumulated by previous agent runs" — false for fresh repos with zero history. new copy is tense-neutral and works for empty + populated bodies. also nudges the agent to re-read after mid-run edits (the inlined TOC ranges are a run-start snapshot). Co-authored-by: Cursor <cursoragent@cursor.com> * learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned calls discovering a quirk this run, recording the workaround prevents the next run from repeating the waste. Reframe around one litmus ("would a future run do its work better because this bullet exists?") and trust it to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary) and the four-example pullfrog/PR/date/play-by-play list (the rule underneath is "don't anchor facts to repo state that will move"). Cuts ~10 lines from a prompt the model was already mostly ignoring; the remaining anchor list is narrower and more enforceable. * audit-learnings-r2: align wiki + tighten re-read nudge - wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls. - buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly. * postRun: refresh JSDoc to match the reflection prompt rewrite `buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets. * fix(mcp/issueEvents): narrow event.event before Set.has lookup octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup. * learnings: split truncation helpers into MCP-free module re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph. move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> * trim first-run celebration email to short personal note drops the feature-dump bullet list (custom review instructions, github iteration walkthrough, security model) — wrong moment to teach. keeps the congrats, the reply CTA, adds discord/x links, keeps the router credit P.S. handler no longer needs the workflowRun→repo lookup. * signup-report: per-bucket histogram Adds a UTC-aligned signups-per-bucket histogram between the overview block and the company-email list. Empty buckets are pre-filled with 0 so dry spells render as gaps. New `BUCKET=hour|day` env flag with a smart default (hour if window ≤ 48h, else day). Histogram is also included in the JSON payload under `histogram: [{key, count}, ...]`. * signup-report: drop hourly bucket, day-only histogram * feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748) * feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660) Per-account ceiling on the sum of `router_topup` invoices (pending + succeeded) for the current UTC calendar month. Closes a gap where a runaway agent loop, leaked PR trigger, or stuck workflow could auto-reload indefinitely with no aggregate per-month ceiling. Two enforcement modes via `RouterLimitMode` enum: - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun; 402 `router_monthly_limit` from /api/proxy-token; email + banner - `alert_only`: auto-reload keeps flowing; email + banner only, first breach per UTC month Enforcement is split across reserveRun (pre-dispatch paywall comment) and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through the same `getRouterSpentThisMonthCents` helper so the dashboard, the dispatch gate, and the auto-reload gate can't disagree. Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string), claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent reloads breaching together send exactly one email. Read-time comparison with the current month re-arms on rollover — no cron. Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell) above the Router/BYOK tabs in `ModelAccessCard`, with a popover "Adjust limit" form that PATCHes the existing /api/account/[owner]/billing/settings route. Same `assertBillingAdmin` gate that owns the other billing settings — no new auth surface. See wiki/billing.md § Router monthly spend limit for the full contract + edge cases. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal pass on monthly Router spend limit (#660) Round-1 review across 5 lenses (billing-subsystem, correctness, security, operational-readiness, research-validated-assumptions) surfaced one critical + three actionable major findings on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot` used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles to `field != value` — UNKNOWN (not TRUE) against the post-migration `NULL` default. First breach for any account would never claim the slot, never stamp the row, and never fire the email (hard_cap or alert_only). Replaced with `OR: [{ field: null }, { field: { not: monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern. **Major — email gap on manual-top-up over cap.** Breach email was only wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>` that crosses the cap blocks dispatch via `reserveRun` but never hits proxy-token, so the user got the PR comment but no email. Wired the CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s PaywallError catch (the SERIALIZABLE txn rolled back when we threw, so we re-claim with the global client; single-statement CAS is its own race boundary against concurrent proxy-token claims). **Major — PR paywall comment leaked $ figures.** `router_limit` body embedded `($X of $Y)` in a comment visible to anyone with PR read access (public repos, forks, outside collaborators). Other paywall types deliberately avoid amounts. Removed; deep link still points to the authenticated console for the figures. **Medium — observability.** Added `[router-limit]` structured logs at the three enforcement sites (proxy-token hard_cap 402, proxy-token alert_only breach, reserveRun paywall) so on-call can grep "did the cap fire for customer X this month." **Medium — customer docs.** Added a `### Monthly spend limit` section to `docs/billing.mdx` (Mintlify) describing the two modes and the manual-top-up caveat. **Doc — refund/dispute interaction.** Documented in `wiki/billing.md` that the cap inherits the existing webhook semantics: disputed `router_topup` drops from the sum (cap briefly un-trips); refunds don't flip status today so refunded top-ups keep counting. Matches wallet behavior — not redefined here. Accepted as-is (documented or pre-existing): `after()` reliability vs stamp-before-send tradeoff, alert_only email fires before Stripe phase-2, proxy-token reads limit fields outside SERIALIZABLE scope (brief TOCTOU on admin lowering cap), stale paywall comment on cap clear, no global kill switch (per-account `alert_only` flip is the practical kill switch), no audit log on cap changes (no existing audit infra), action version not bumped (separate release commit). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal round 2 on monthly Router spend limit Round-2 anneal (billing-subsystem, correctness, research-validated, user-journey, operational-readiness) surfaced a critical merge conflict and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted `formatBillingErrorSummary` from `action/main.ts` to `action/utils/billingErrors.ts`. The PR's new `router_monthly_limit` arm still lived in `action/main.ts`. Took main's slim orchestrator wholesale; moved the arm into the extracted file. **Major — cap = payments only, not dispatch.** `reserveRun` was pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit` regardless of wallet balance, contradicting the cap's positioning as "ceiling on what you pay." An account with $500 of paid-up wallet and a breached $100 cap couldn't trigger any new run via the comment path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded — surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token` is now the sole enforcement point, refusing only the next auto-reload that would push past. Wallet credit always drains. Dropped the now-dead `router_limit` arm in `buildPaywallCommentBody`, the dead `routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`, and the post-paywall email-fire David added — all unreachable. **Major — split `manual_topup` from `router_topup`.** Manual on-session top-ups at `/billing-top-up/<owner>` were landing as `Invoice.kind = "router_topup"` and counting toward the cap. The cap exists to brake *passive* runaway (auto-reload loops); a manual top-up is a deliberate click-through that the user owns. Added `InvoiceKind.manual_topup`, flipped the manual write site + `createTopUpCheckoutSession` metadata, broadened wallet / reconcile / billing-report reads to `kind IN (router_topup, manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap aggregate) to `router_topup` only. Worked example: cap=$300, reload=$100 → exactly three reloads succeed; a fourth is blocked. Historical rows stay labelled `router_topup` (no backfill); the asymmetry is small and accepted since the manual flow only existed alongside auto-reload for a brief window. Extended the `invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows the same shape as `router_topup` (PaymentIntent-backed, no stripeInvoiceId); split into a second migration because PG forbids using a freshly-added enum value in the same transaction. **Major — email reframed around the triggering reload event.** The `alert_only` body was reporting a pre-eager-write `spentCents` while the dashboard reads the post-commit value, so email and dashboard disagreed by exactly one reload. Both flavors now say "Your most recent $50 auto-reload brought you over your $300 monthly limit" instead of a running spent-of-cap total — no reconciliation needed, no more "you've hit your monthly cap" copy firing for partial breaches (spent=$80 of $100, reload=$30 would have triggered that wording). **Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded "You've used your 30 free runs this month. Add a card to continue at 7¢/run." regardless of `detail.reason`. Branched on `cap` vs `delinquent` so each paywall surfaces actionable copy with the right CTA. `router_limit` no longer flows through here (per F4 above). **Major — RouterLimitBanner.** Added an `isAlertBreached` visual state (amber palette) so an `alert_only` account at $240 of $200 no longer renders in the same neutral zinc chrome as a healthy under-cap account. Updated popover copy to reflect the auto-reload-only scope. **Medium — paywall log line.** Added `detail.reason` to the `[Installation X] paywall:` log so on-call grepping for "why was this paused" can distinguish `cap` from `delinquent`. **Cleanup.** Dropped dead `utcMonthKey` import + re-export in `maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*` fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*` since they now handle both kinds. Updated wiki/billing.md + docs/billing.mdx + schema doc comments throughout. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt` sitting next to it — a single-purpose state column on `Account` that encoded a date as a string and required a custom CAS predicate to read/write race-safely. Plus it had real holes: Resend send failure left the sentinel stamped and the account silently un-emailed for the month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered edge cases never fired at all. Replace it with: fire `maybeNotifyRouterLimit` on every breaching reload, let the Resend `Idempotency-Key` `router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside Resend's 24h dedup window. Continuously-breaching accounts get ~1 reminder per day; brief Resend outages self-heal because the next breaching reload re-attempts the send. Mode is in the dedup key so `alert_only → hard_cap` mid-month re-arms a fresh email with the appropriate copy. Drops `Account.routerLimitNotifiedMonth` and `claimRouterLimitNotificationSlot`; simplifies the proxy-token phase-1 branch significantly. Net diff is negative LOC and the data model loses a single-purpose sentinel. Migration was branch-local — never deployed — so I edited the original add-cap migration in place to drop the column from the ALTER TABLE rather than chain a drop-column migration on top. Preview Neon branches reset automatically on history rewrite per wiki/migrations.md. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): hide RouterLimitBanner when no cap is configured The banner was unconditionally rendered for every billing-enabled account, including pure-BYOK admins who never touch Router. They got "No monthly spend limit / Router has spent $0.00" + a divider as visual noise on the model access page — basically nagging them to set a feature they may not want. Running without a cap is valid; we don't nag. ModelAccessCard now gates the banner block (banner + dividers) on `routerMonthlyLimitCents !== null`. RouterLimitBanner drops the no-limit visual state, the "Set monthly limit" CTA text, and the dead `hasLimit` branching. Cleaner three-state shape (under cap / amber breached / brick breached). Discoverability: no-cap users no longer see a UI affordance to set one. That's deliberate — the cap is a power-user feature documented in docs/billing.mdx. If discoverability becomes an ask, we can add a small inline link inside RouterWalletSection without bringing back the always-visible banner. Resolves the only outstanding finding from cursor bugbot's review of ff5328c (banner-visible-for-byok thread). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(billing): docs/wiki match new "no banner without a cap" reality Pullfrog bot review of f7672ca pointed out the customer docs still told users to "Set the cap from the **Monthly spend limit** banner in the **Model costs** card" — but after hiding the banner for no-cap accounts there is no such banner to use until you already have a cap. Catch-22 for first-time setup. Rewrote docs/billing.mdx to be self-contained: explain what the cap is, what the two modes do, what the banner shows *once configured*, and direct admins to PATCH the billing settings endpoint (or reach out to support) for first-time setup. Cap is positioned as optional; running without one is the documented default. Wiki paragraph in wiki/billing.md updated to match — banner is only rendered when a cap exists, three visual states (under / amber / red), no first-time-setup UI nag by design. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely The standalone `RouterLimitBanner` was the wrong shape. It only rendered when a cap was already configured (so there was no UI to discover the feature in the first place — first-time setup required hitting the API directly), and it occupied prominent real estate above the tabs to surface state that already lives in the row's own input when the form moves down where it belongs. New shape: monthly cap is just a third row inside `RouterWalletSection` sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated the same way (card on file + auto-reload enabled — the only state where the cap actually means anything). Empty input → no cap, with placeholder "No limit". Setting a number reveals a **Behavior at limit** toggle built on the same `Tabs` slider component used for the Router/BYOK tab switch, so the look matches the rest of the card. Deletes: - `RouterLimitBanner` component (212 lines) - banner mount + conditional + spacers in `ModelAccessCard` - `AlertTriangle` is still imported (used by `DelinquencyBanner`) Adds: - one settings row in `RouterWalletSection` with the cap input + mode tabs - `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the existing `saveSettings` helper (widened to accept `string | null`) - `Tabs` / `TabsList` / `TabsTrigger` import Docs + wiki updated to match the new shape; the customer doc no longer points at a banner that won't appear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between Previously bundled both into one row block. Restructure: cap input is its own row; Behavior-at-limit Tabs gets a sibling row with the standard `h-5 + hr + h-5` separator between (matching the rhythm of auto-reload amount → threshold → monthly cap). Mode-toggle row is gated on `routerMonthlyLimitCents !== null` so the hr + tabs only appear once a number is in the cap input. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row Same `flex items-center justify-between gap-3` layout as the Auto-reload row: label group on the left, control on the right. Drops the vertical stack in favour of the horizontal one — looks identical to the toggle row directly above. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> * drop italic TL;DR commit-range line from review body the metadata (sha range, commit list, timestamps) is already in the html comment for downstream agents. the visible italic line was clutter and the ellipsis form broke the second sha's auto-link on github anyway. * add agent-browser fallback rule for unreachable chrome devtools mcp * onboarding: gated org-console wizard (#762) * onboarding: gated org-console wizard Replaces the org console's `/console/[owner]` page with a single-card, "growing" stepper when the account has zero `Repo` rows. Walks first-time users through billing mode, BYOK provider+key (if applicable), repo pick, workflow file creation, and a celebratory redeem-credit moment before landing them back on the now-populated org console. ## What's new - New: `components/OnboardingStepper.tsx` — the wizard. Six steps, each derived from real persisted state (Account.modelAccessMode, AccountSecret, Repo). Step state ladder with progressive disclosure and click-to-edit collapsed summaries. - New: `app/console/[owner]/OnboardingView.tsx` — page-chrome wrapper that hosts the stepper inside the same header/sidebar shell as the member view. - Modified: `app/console/[owner]/page.tsx` — adds a `prisma.repo.count` gate alongside existing parallel queries; renders OnboardingView when count === 0, else falls through to the existing repo grid. ## Schema - Flipped `Account.modelAccessMode` default from `byok` to `router`. Router is the lower-friction default (signup credit funds first ~150 runs without a card; users can flip to BYOK explicitly via the wizard or the existing `<ModelAccessCard>` switch). Existing rows keep their current explicit value — Postgres column-default change doesn't backfill, by design. - Migration: `20260516014601_modelaccessmode_default_router`. ## Credit-claim semantics Killed the historical mount-time auto-claim on `<SignupCreditModal>`. All claims are now explicit clicks, fired from one of two surfaces: 1. Wizard step 6 "Redeem $10 credit" CTA (Router branch, eligible). 2. New explicit "Redeem $10 credit" button on `<BillingCard>`'s Router wallet section, visible only when the new server-derived `signupCreditEligible` flag is true (promo active + no prior signup or welcome grant). Covers existing users who'd otherwise lose the auto-claim entry point. `<SignupCreditModal>` is now a controlled component (`open` / `onOpenChange` / `amountCents` props) with a sibling `useClaimSignupCredit(owner)` hook for explicit invocation. The Sparkles celebration dialog rendering is unchanged. ## Other touched surfaces - `app/api/create-workflow/route.ts`: optional `model` body field. When present, the route updates `Repo.model` on the row that `createWorkflowForRepo` just created/surfaced — wizard threads the picked provider's `preferred` model alias through here so a fresh repo doesn't sit on null/auto. - `app/api/account/[owner]/billing/route.ts`: surfaces `signupCreditEligible: boolean` (derived from `SIGNUP_CREDIT_PROMO_ACTIVE` + grant scan). Drives the new explicit redeem button. - `components/AgentSettings.tsx`: fixes the Router-no-billing copy lie ("Runs will draw from your signup credit until exhausted" was false — `isInfraCovered` gates Router minting on `hasCardOnFile`, not balance, so credit-only-no-card users can't actually spend the grant on Router runs). New copy: "Add a card to use Pullfrog Router. Your $10 signup credit (if claimed) applies on top." ## Resume-tomorrow detection Every step's expansion is derived from persisted state (no new column, no localStorage). With the Router default flip, `modelAccessMode === "byok"` is now a reliable signal of explicit user pick, eliminating the heuristic that the byok-default schema would have required. The only ambiguous case is "Router-bailed-before-redeem" (looks identical to a default-Router fresh visit since neither card nor grant exists yet) — acceptable 1-click cost on revisit. ## Testing - `pnpm lint`: clean - `pnpm format`: clean - `pnpm typecheck`: clean - `pnpm -C action test`: 596/596 passing - Visual verification: blocked — Chrome DevTools MCP returned "Not connected" across both available servers. Manual walkthrough needed before merge to confirm step transitions, going-back UX, and the celebration modal redirect destinations match the plan in `.cursor/plans/org_onboarding_stepper_4fdfebbb.plan.md`. * onboarding: drop accordion, multi-repo bulk-onboard, full-width radio rows Three rounds of UX feedback rolled in: 1. **Drop the accordion.** Steps no longer collapse to a one-line summary when "done" — the wizard literally grows by appending steps below as the user progresses, and earlier steps stay fully interactive (re-flip Router→BYOK, re-pick provider, toggle a repo) without any "edit" affordance. `StepShell` now always renders its body for any step the user has reached; the only state distinction is the number circle (filled = active, check = done). 2. **Step 1 is full-width radio rows, not narrow tabs with side-by-side info tiles.** Two rows, each with the option title, an inline "Recommended" badge on Router, and a description sentence inside the row. The persisted `Account.modelAccessMode` (default `router`) drives the initial selection, so step 1 always has one row picked on first paint — no "neither selected" empty state. 3. **Multi-repo bulk-onboard.** Step 4 now uses checkboxes; copy reads "Select the repos you'd like to install Pullfrog into. We'll create a pullfrog.yml GitHub Actions workflow file in each." Step 5 fans out N parallel `POST /api/create-workflow` calls (concurrency capped at 4) and renders per-repo status inline (running → committed / PR #N / already configured / error). Step 6 celebrates with a multi-result headline ("Pullfrog is set up across N repos") and a sub-line breaking down `committed · PRs awaiting merge · failed` plus a per-repo PR list when any PRs were opened. Single- repo path renders the same control surface but with singular copy. Other bits: - Per-step description sentences below every title. - Repo picker shows totalCount inline with the pagination controls and "N repos selected" summary below the table. - Dropped the `userPickedBillingMode` and `editingStep` state machinery + the `isFreshDefault` heuristic — all simplified out by the no-accordion design (we just trust `billingMode` directly). - `createWorkflowPR` PR body already links back to `pullfrog.com/console/<owner>/<repo>` with a "Verify workflow" CTA; no change needed there. * fix(onboarding): provider tile labels — getProviderDisplayName expects slug `getProviderDisplayName` from `pullfrog/internal` parses its argument as a `provider/model` slug. Step 2 was passing bare provider keys (e.g. "anthropic"), which made the helper throw "invalid model slug 'anthropic' — expected 'provider/model'" and crashed the BYOK branch with the page-level error boundary. Replace with a local `providerDisplayName` that reads the registry directly (`providers[key].displayName`). Drops the unused `getProviderDisplayName` import. Caught by Chrome DevTools end-to-end: clicking Bring-your-own-key on the fresh wizard renders the page-level error. Re-verified post-fix: BYOK flow shows step 2 with all 9 provider tiles correctly labeled (Anthropic / OpenAI / Google / xAI / DeepSeek / Moonshot AI / Amazon Bedrock / OpenRouter / OpenCode), step 3 reveals on tile click. Also adds a guardrail to AGENTS.md: don't silently abandon visual verification when DevTools breaks. Recovery is always possible (pkill -9 chrome-devtools-mcp + pkill puppeteer + rm Singleton locks + retry several times); if it genuinely won't recover, abort and tell the user — never mask as "verified by code review". * agents.md: never give up on Chrome DevTools MCP failures Recovery is always possible (pkill chrome-devtools-mcp, remove Singleton locks, retry several times). If genuinely unrecoverable, abort and tell the user explicitly — never silently mask as "verified by code review". Visual verification is non-negotiable for UI changes. * onboarding: polish — checkbox color, redundant labels, copy Caught during chrome-devtools verification of the BYOK + cross-page selection flows: - **Checkbox color**: native browser pink/red replaced with evergreen via `accent-evergreen-600`. Visually consistent with the rest of the wizard's selection states. - **Bedrock provider tile**: was rendering "Amazon Bedrock" twice (provider name + recommended-model name both resolve to "Amazon Bedrock" because Bedrock has no `preferred` model under `providers.bedrock.models` — its single routing entry IS the recommended pick). Suppress the recommended subtitle when it duplicates the provider name. - **Step 6 description**: tightened from a clunky two-clause sentence about workflow file landing to a single direct call: "Mention @pullfrog in any PR or issue to dispatch a run. (Branch-protected repos: merge the PR first.)" - **Wizard intro**: was "Set up Pullfrog for your first repo" — outdated since multi-repo. Now: "Connect Pullfrog to your repos. Each step unlocks the next as you go." Cross-page multi-select also verified: selections from page 1 persist when navigating to page 2 and back. "N repos selected" counter reflects total across all pages. BYOK secret-add flow verified end-to-end: AddSecretModal opens with the env var pre-filled, save triggers secrets refetch, step 3 flips to "✓ ANTHROPIC_API_KEY configured", step 4 reveals automatically. * onboarding: serial install, inline secrets, explicit credit redeem - step 3: replace modal-based secret entry with inline password fields per provider, with deep links to provider dashboards. claude code OAuth surfaces as a distinct group when anthropic is picked. bedrock gets three-field form. github actions secrets path is collapsible with org/personal-aware urls + self-certify. - step 4: merge repo-pick + workflow-create into one step. install is now serial (visible slow-reveal) instead of concurrent. continue button renders immediately on submit, disabled until every repo reaches a terminal state. errored rows render a single soft amber 'failed' label. pagination uses chevron buttons + keepPreviousData (no layout shift). - step 6: explicit 'redeem $10 credit' for router+eligible, 'complete setup' otherwise. final redirect is a hard refresh so the repo grid picks up. - signup credit: drop the mount-time auto-claim modal in favor of explicit user clicks. new useClaimSignupCredit hook + RedeemSignupCreditCallout banner inside RouterWalletSection so a BYOK→Router flip surfaces a one-click redeem affordance. - billing mode is now optimistic (local state + background PATCH) and initialBillingMode + signupCreditEligible eager-load via server props to kill the multi-second click latency. - skip onboarding: header button sets pullfrog_skip_onboarding cookie; server reads it in page.tsx and falls through to the regular grid. - demo mode: NEXT_PUBLIC_ONBOARDING_DEMO=1 cycles the install progress list through pending/running/committed/PR/existing/failed states. - createWorkflowForRepo: PULLFROG_FORCE_PR_CREATION=1 skips direct commit to exercise the PR fallback locally. * onboarding: review feedback — focused eligibility query, best-effort model pre-fill, claim error toast - billing/route.ts + console/[owner]/page.tsx: replace top-N recentGrants scan for signup-credit eligibility with a focused findFirst({ reason: { in: [SIGNUP, WELCOME] } }). the prior query could return any 5/10 rows (no orderBy on page.tsx) and miss a prior signup/welcome grant if a future grant reason (refund/referral/etc.) ever ships. recentGrants stays for the billing-history list. - create-workflow/route.ts: gate Repo.model updateMany on result.type === "created" so an existing user-set model isn't clobbered when the workflow file already exists. wrap in try/catch: GitHub side effect already succeeded, so a transient DB blip shouldn't 500 the route and have the UI report failure on a partially-completed setup. - SignupCreditModal: add onError toast to useClaimSignupCredit so transient redeem failures surface ("Couldn't redeem your credit. Try again in a moment."). callers .catch(() => null) the rejection so it doesn't propagate as an unhandled rejection in the React handler. - OnboardingStepper: trim stale "per-row try again button" wording from progressRef + processRepo comments — that button was removed in the prior commit per design feedback. * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * revert: extract router-gate fix into its own PR The router fix at a14bcdd4 is being shipped as a standalone PR so it can be reviewed and merged independently of the onboarding-wizard work. Reverting here keeps #762 focused on the wizard. The fix itself landed at https://github.com/pullfrog/app/pull/792. * router: fix unspendable signup credit on no-card private repos (#792) * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * action: drop dead isInfraCovered + plan param post-fix Cleanup the action-side dead code introduced by the previous commit's removal of the redundant `isInfraCovered` re-derivation in proxy.ts: - delete `isInfraCovered` from action/utils/runContext.ts (was the only callsite; mirror in server's utils/billing.ts is unchanged and still load-bearing for learnings/indexing) - drop unused `plan: AccountPlan` param from `resolveProxyModel` / `runProxyResolution` (and the corresponding `AccountPlan` import + the `plan: runContext.plan` arg at the main.ts call site) - update the action/mcp/server.ts comment that pointed at the now-gone action mirror to reference the server-side `utils/billing.ts` instead `AccountPlan` itself is still load-bearing (mcp/server, runContextData, run-context fetch), only `isInfraCovered` and the dead `plan` parameter go away. * eager signup credit + free-OpenCode fallback when BYOK has no key (#789) * eager signup credit + free-OpenCode fallback when BYOK has no key addresses the silent-churn pattern that took out 15 first-run-failure accounts post-launch: GH Actions secret references resolved to empty strings (because the secrets didn't exist on the repo), the action launched Claude Code with no key, the LLM provider 401'd, and the run died in seconds with a synthetic "Invalid API key" message. those accounts had no Router credits to fall back to because the lazy claim required a dashboard visit they never made. three changes, one PR: 1. Eager $10 signup credit at account creation. Both account-creation sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo` for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }` in the same transaction as the `accounts` row. CLI installers who never sign in get the credit. The dashboard `/signup-credit/claim` POST stays as an idempotent backstop for accounts created before this shipped. 2. Free-OpenCode fallback in the action. When the configured BYOK slug needs a provider key the runner doesn't have, swap to `opencode/minimax-m2.5-free` before agent selection so the run still succeeds. Surfaced via a `» fell back from <slug> to <free>` warning in the action log. Skipped on Router runs (Pullfrog mints the key) and when no model is configured (auto-select-with-throw still fires for the genuinely-misconfigured case). 3. New action-test fixture `byok-no-keys-fallback` that empty-strings every known provider key (matching how GH Actions handles missing secrets) and asserts the run succeeds with the fallback log line present. plus a unit test for the helper covering each skip case. skipping the schema flip from `byok` to `router` — that's coming via the onboarding-stepper PR (#762). * fallback: skip Bedrock + surface in PR-comment footer addresses copilot review on #789 (real bug — parseModel throws on Bedrock raw IDs that have no slash, would crash before validateBedrockSetup could surface its own error) and the user-side ask to make the fallback visible in PR comments. - selectFallbackModelIfNeeded skips when resolvedModel has no '/' so Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash inside hasProviderKey -> parseModel. unit test covers it. - toolState.modelFallback records the configured slug we fell back from. set in main.ts when fallback engages. - buildPullfrogFooter accepts fallbackFrom and renders "Using `MiniMax M2.5` (free) (credentials for Claude Opus not configured)" so the substitution is visible in PR comments, reviews, PR bodies, and error reports. - threaded through all four action-side footer call sites (mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts fire pre-action and don't have toolState — left as-is. * fallback footer: use provider display name + document email asymmetry addresses pullfrog reviewer findings on #789: - footer now shows 'credentials for Anthropic not configured' (provider display name from `providers.anthropic.displayName`) instead of the per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY covers all Anthropic models), so this matches what the user actually needs to fix. - document the intentional asymmetry between eager and lazy signup credit paths: eager skips both the signupCreditClaimedEmail and the per-grant team@ alert. comment explains why (the 'new account created' alert already covers it on the eager path; the user-facing email assumes a user-initiated action that hasn't happened yet for CLI/GH-App-only signups). - skipping the backfill for the 15 historical accounts per user's earlier decision — they all uninstalled, so the cohort self-selected out of being reachable. * fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap local agnostic fixture run surfaced two real bugs the unit tests didn't catch: 1. fallback gate was on configuredSlug (=payload.model) but the test uses PULLFROG_MODEL to set the model, which is read by resolveModel AFTER its slug arg. configuredSlug stayed undefined → fallback never fired. drop configuredSlug from the helper signature; gate purely on resolvedModel since that's the same value regardless of how the model was specified (DB config vs PULLFROG_MODEL env). 2. when fallback engaged, the post-swap resolveModel({slug: fallback.to}) call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback target back to the unkeyed model. validateAgentApiKey then threw "no API key found" against the original model. fix: skip the re-resolve. fallback.to is already a CLI-ready specifier. unit tests updated for the new helper signature (8 tests, all pass). fallback log line confirmed emitted in the local run pre-second-fix; the second fix unblocks the validation that previously threw. * models-bump: harden CI and bot prompt against catalog hallucinations PR #790 (the first bot-authored models-bump PR) shipped a broken bump for openrouter/gemini-flash: the bot pattern-matched the parallel google bump and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the OpenRouter API but not on models.dev's openrouter section — the catalog OpenCode actually reads. The slug failed at runtime with `Model not found`. Two CI gaps let it through: 1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*` alias as a "passthrough", smoke-testing only one canary per routing layer. But those aren't passthroughs — each is a distinct models.dev catalog entry that can drift independently of the direct-provider mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25). Only `bedrock/byok` stays pruned (sentinel resolve). 2. `models-catalog` test (the integrity gate that asserts every resolve exists on models.dev) was main-only by design — to keep upstream catalog churn from blocking unrelated PRs. But it's exactly the test we want running on the bot's own catalog edits. Add `pullfrog/models-bump` head-ref to its trigger. Also tighten the bot prompt in models-bump.yml: new rule 0 requires every new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the alias's own `candidates[]` in models-bump-context.json — the deterministic preprocessor only emits candidates sourced from models.dev's mirror, so this gates against the cross-alias pattern-matching that broke PR #790. For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter API), which is necessary but not sufficient; the `models-catalog` job is the authoritative models.dev check. Verified locally: - baseline `pnpm -C action test:catalog` passes 133 tests - simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash` into action/models.ts) and the catalog test fails with the right assertion: `model "google/gemini-3.5-flash" not found under openrouter on models.dev` - `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every openrouter/* alias and every keyed opencode/* alias now smoked --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-05-20 03:58:18 +00:00
David Blass	2f1f136da8	modes: actually call resolve_review_thread on addressed PR feedback (#749) ... * modes: instruct IncrementalReview + AddressReviews to actually call resolve_review_thread empirically (per #672 audit on 195 prod runs across 4 repos) only 1 in 195 runs called resolve_review_thread. IncrementalReview's prompt only mentioned prior reviews as a dedup filter — the agent had the data in hand but no instruction to retire addressed threads, so it almost never did. AddressReviews mentioned resolve as a one-line bullet at the end of step 6, which the agent treated as optional (the one observed AddressReviews run replied to 3 comments and resolved 0). IncrementalReview step 4 now: fetch prior reviews → for each open Pf-originated thread, decide if the new commits addressed it (anchor moved, isOutdated, or substantive concern resolved on a re-read), reply + resolve when addressed, leave open when uncertain. Conservative scope: only Pf-originated threads — human-reviewer threads stay theirs to mediate. AddressReviews step 6 now pairs reply + resolve in the same beat with explicit rules: resolve when you made the change OR replied substantively; do NOT resolve when you pushed back and the disagreement is unresolved. addresses #672. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal: tighten auto-resolve decision rules correctness fixes from /anneal pass on IncrementalReview step 4 + AddressReviews step 6: - `[OUTDATED]` no longer "strong signal of address" — it just means GitHub moved the anchor (line shift / reformat / force-push); agent must re-read code at new location - explicit Pf-origin detection rule (first `comment author=pullfrog[bot]` tag), clarifies `*` marker is unrelated to thread root - explicit dual-ID separation: numeric `id=` for `reply_to_review_comment.comment_id`, GraphQL `thread=` for `resolve_review_thread.thread_id` (was silently 422-ing) - reformatter / partial-fix loophole closed: lines being modified isn't enough; all concerns in multi-concern comments must be addressed - AddressReviews push-failure path explicit: STOP and report_progress, do NOT reply or resolve when fix isn't live - step 4 → step 8 wiring rationale corrected (step 8 dedups by line range, not thread state) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-05-20 03:57:52 +00:00
Colin McDonnell	cb0dbcd371	feat(action): make prepush hook non-blocking after one failure (#777) ... * feat(action): make prepush hook non-blocking after one failure push_branch now treats the repository's prepush hook as best-effort: it runs at most once per run, surfaces the failure output if the script exits non-zero, and every subsequent push_branch call this run skips the hook so the agent isn't blocked by failures unrelated to its change. The agent can iterate by running the hook command itself via the shell tool when shell access is available; push_branch will not re-run the hook automatically after a failure. Why: a one-line OSS-allowlist change took 9 minutes (#776) because the agent retried push_branch six times against a prepush hook that was failing for env-leak and missing-build-artifact reasons unrelated to the change. CI catches the same checks on the GitHub side; the local prepush gate was duplicating work and blocking unrelated fixes. - ToolState: new prepushFailureCount counter (per-run, never resets) - executeLifecycleHook: returns structured failure (kind/output/exitCode) so prepush can compose its own agent-facing message instead of inheriting the generic retry/no-retry advice meant for setup - push_branch: composes a shell-mode-aware error message; surfaces prepushSkipped on the success payload + appends a note to the message - instructions.ts + wiki/prompt.md + docs/comparisons.mdx: updated to reflect best-effort semantics * fix(action): clarify prepush latch semantics + soften static guidance review fixes from PR #777: - toolState comment, instructions, success message, tool description: replace "runs at most once per run" / "first call only" wording with the actual semantic — successful prepush keeps running on later push_branch calls; only a hook FAILURE latches the bypass. - tool description: drop hardcoded "via the shell tool" guidance so the static description doesn't mislead in shell:disabled runs (the dynamic agent prompt in instructions.ts already does shell-conditional messaging). - LifecycleHookFailure.output JSDoc: match the implementation (stderr-preferred fallback to stdout, empty for timeout/spawn). * fix(action): shorten prepush-skip log to terse operator telemetry the previous log line tried to address the agent ("re-run the hook command yourself via shell"), but log.info writes to the action runtime's stdout — the agent never sees it. agent-facing skip guidance already lives in the error message from buildPrepushFailureMessage, the success message when bypassed, and the system prompt in instructions.ts. log line is now just operator telemetry. * refactor(action): drop slop from prepush soft-fail self-audit pass after the previous review-fix round. removed duplication between code-level comment and the five other places that already explain the same behavior, tightened verbose JSDoc, and collapsed redundant clauses in agent-facing strings. - LifecycleHookFailure → discriminated union. drops the optional exitCode/spawnError fields (and the empty-output sentinel for timeout/spawn) plus the corresponding ?? fallbacks in the helper. - PushBranchTool: 7-line code comment above the latch removed (toolState field comment + tool description + error message + success message + system prompt all already cover it). tool description third sentence dropped (restated the second). success message tightened to a parenthetical. - buildPrepushFailureMessage: 4-line JSDoc → 1 line. shared "if you think the failure could indicate a real bug in your code" prefix factored out across the shell-conditional branches. - ToolState.prepushFailureCount comment: 8 lines → 3. the "what" is in git.ts; comment now only documents the invariant (never decremented within a run). - instructions.ts prepush guidance: collapsed nested bullets + ternary into one paragraph; dropped the "so re-running via shell is the only way…" tail that restated "push_branch will NOT re-run it". * fix(action): hint prepush bypass on dirty tree after hook failure When push_branch blocks on a dirty working tree and the prepush latch is already set, tell the agent the hook will be skipped once the tree is clean. * fix(test): narrow CI matrix for lifecycle and toolState changes Remove lifecycle.ts from ALWAYS_RUN_ALL and add lifecycle.ts + toolState.ts to push/git agnostic test coverage so PRs touching prepush latch logic run targeted tests instead of the full matrix.	2026-05-20 02:43:23 +00:00
Colin McDonnell	7e90e5cae6	Align Plan-mode prompts on report_progress as the canonical plan tool (#786) ... * fix: align Plan-mode prompts on report_progress as the canonical plan tool Fixes #673. Three sites disagreed on where Plan output should be posted, letting a model synthesize a broken third interpretation (initial post via `report_progress({ target_plan_comment: true })`, which then misses the `existingPlanCommentId` precondition). This PR aligns all three on `report_progress` as canonical, with `target_plan_comment` reserved for revisions only: - `action/modes.ts` Plan step 4 — spell out that the initial plan post uses `report_progress` WITHOUT `target_plan_comment`, and that revisions go through `select_mode`'s PlanEdit override. - `action/mcp/comment.ts` `target_plan_comment` flag description — make the "revisions only" precondition explicit and call out the initial-post path by name. - `action/utils/instructions.ts` Progress reporting paragraph — drop the misleading "(e.g., Plan comments)" parenthetical that read as "use create_issue_comment for plans". `PlanEdit` (in `action/mcp/selectMode.ts`) was already correct and is unchanged. Intentionally out of scope (to keep the fix minimal): a `publish_plan` tool, removing the vestigial `create_issue_comment({ type: "Plan" })` branch, hardening the run-end cleanup guard for the `target_plan_comment but no existingPlanCommentId` fallthrough, and renaming `target_plan_comment`. * align create_issue_comment description with report_progress as canonical plan tool	2026-05-20 02:31:34 +00:00
pullfrog[bot]	f49d4206aa	chore(models): bump resolved versions (#790) ... * chore(models): bump resolved versions * fix(models): keep google gemini-3.5-flash bump, revert invalid openrouter ids openrouter/google/gemini-3.5-flash is not on models.dev yet; only the direct google/gemini-flash resolve should move to 3.5-flash. --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-05-20 02:25:50 +00:00
Colin McDonnell	69c7d4b8cd	models-bump: harden CI and bot prompt against catalog hallucinations ... PR #790 (the first bot-authored models-bump PR) shipped a broken bump for openrouter/gemini-flash: the bot pattern-matched the parallel google bump and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the OpenRouter API but not on models.dev's openrouter section — the catalog OpenCode actually reads. The slug failed at runtime with `Model not found`. Two CI gaps let it through: 1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*` alias as a "passthrough", smoke-testing only one canary per routing layer. But those aren't passthroughs — each is a distinct models.dev catalog entry that can drift independently of the direct-provider mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25). Only `bedrock/byok` stays pruned (sentinel resolve). 2. `models-catalog` test (the integrity gate that asserts every resolve exists on models.dev) was main-only by design — to keep upstream catalog churn from blocking unrelated PRs. But it's exactly the test we want running on the bot's own catalog edits. Add `pullfrog/models-bump` head-ref to its trigger. Also tighten the bot prompt in models-bump.yml: new rule 0 requires every new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the alias's own `candidates[]` in models-bump-context.json — the deterministic preprocessor only emits candidates sourced from models.dev's mirror, so this gates against the cross-alias pattern-matching that broke PR #790. For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter API), which is necessary but not sufficient; the `models-catalog` job is the authoritative models.dev check. Verified locally: - baseline `pnpm -C action test:catalog` passes 133 tests - simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash` into action/models.ts) and the catalog test fails with the right assertion: `model "google/gemini-3.5-flash" not found under openrouter on models.dev` - `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every openrouter/* alias and every keyed opencode/* alias now smoked	2026-05-20 02:18:38 +00:00
Colin McDonnell	f3d18401ac	eager signup credit + free-OpenCode fallback when BYOK has no key (#789) ... * eager signup credit + free-OpenCode fallback when BYOK has no key addresses the silent-churn pattern that took out 15 first-run-failure accounts post-launch: GH Actions secret references resolved to empty strings (because the secrets didn't exist on the repo), the action launched Claude Code with no key, the LLM provider 401'd, and the run died in seconds with a synthetic "Invalid API key" message. those accounts had no Router credits to fall back to because the lazy claim required a dashboard visit they never made. three changes, one PR: 1. Eager $10 signup credit at account creation. Both account-creation sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo` for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }` in the same transaction as the `accounts` row. CLI installers who never sign in get the credit. The dashboard `/signup-credit/claim` POST stays as an idempotent backstop for accounts created before this shipped. 2. Free-OpenCode fallback in the action. When the configured BYOK slug needs a provider key the runner doesn't have, swap to `opencode/minimax-m2.5-free` before agent selection so the run still succeeds. Surfaced via a `» fell back from <slug> to <free>` warning in the action log. Skipped on Router runs (Pullfrog mints the key) and when no model is configured (auto-select-with-throw still fires for the genuinely-misconfigured case). 3. New action-test fixture `byok-no-keys-fallback` that empty-strings every known provider key (matching how GH Actions handles missing secrets) and asserts the run succeeds with the fallback log line present. plus a unit test for the helper covering each skip case. skipping the schema flip from `byok` to `router` — that's coming via the onboarding-stepper PR (#762). * fallback: skip Bedrock + surface in PR-comment footer addresses copilot review on #789 (real bug — parseModel throws on Bedrock raw IDs that have no slash, would crash before validateBedrockSetup could surface its own error) and the user-side ask to make the fallback visible in PR comments. - selectFallbackModelIfNeeded skips when resolvedModel has no '/' so Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash inside hasProviderKey -> parseModel. unit test covers it. - toolState.modelFallback records the configured slug we fell back from. set in main.ts when fallback engages. - buildPullfrogFooter accepts fallbackFrom and renders "Using `MiniMax M2.5` (free) (credentials for Claude Opus not configured)" so the substitution is visible in PR comments, reviews, PR bodies, and error reports. - threaded through all four action-side footer call sites (mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts fire pre-action and don't have toolState — left as-is. * fallback footer: use provider display name + document email asymmetry addresses pullfrog reviewer findings on #789: - footer now shows 'credentials for Anthropic not configured' (provider display name from `providers.anthropic.displayName`) instead of the per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY covers all Anthropic models), so this matches what the user actually needs to fix. - document the intentional asymmetry between eager and lazy signup credit paths: eager skips both the signupCreditClaimedEmail and the per-grant team@ alert. comment explains why (the 'new account created' alert already covers it on the eager path; the user-facing email assumes a user-initiated action that hasn't happened yet for CLI/GH-App-only signups). - skipping the backfill for the 15 historical accounts per user's earlier decision — they all uninstalled, so the cohort self-selected out of being reachable. * fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap local agnostic fixture run surfaced two real bugs the unit tests didn't catch: 1. fallback gate was on configuredSlug (=payload.model) but the test uses PULLFROG_MODEL to set the model, which is read by resolveModel AFTER its slug arg. configuredSlug stayed undefined → fallback never fired. drop configuredSlug from the helper signature; gate purely on resolvedModel since that's the same value regardless of how the model was specified (DB config vs PULLFROG_MODEL env). 2. when fallback engaged, the post-swap resolveModel({slug: fallback.to}) call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback target back to the unkeyed model. validateAgentApiKey then threw "no API key found" against the original model. fix: skip the re-resolve. fallback.to is already a CLI-ready specifier. unit tests updated for the new helper signature (8 tests, all pass). fallback log line confirmed emitted in the local run pre-second-fix; the second fix unblocks the validation that previously threw.	2026-05-20 02:17:22 +00:00
Colin McDonnell	8dff91ac49	router: fix unspendable signup credit on no-card private repos (#792) ... * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * action: drop dead isInfraCovered + plan param post-fix Cleanup the action-side dead code introduced by the previous commit's removal of the redundant `isInfraCovered` re-derivation in proxy.ts: - delete `isInfraCovered` from action/utils/runContext.ts (was the only callsite; mirror in server's utils/billing.ts is unchanged and still load-bearing for learnings/indexing) - drop unused `plan: AccountPlan` param from `resolveProxyModel` / `runProxyResolution` (and the corresponding `AccountPlan` import + the `plan: runContext.plan` arg at the main.ts call site) - update the action/mcp/server.ts comment that pointed at the now-gone action mirror to reference the server-side `utils/billing.ts` instead `AccountPlan` itself is still load-bearing (mcp/server, runContextData, run-context fetch), only `isInfraCovered` and the dead `plan` parameter go away.	2026-05-20 02:10:29 +00:00
Colin McDonnell	0d7955d87d	add agent-browser fallback rule for unreachable chrome devtools mcp	2026-05-20 01:39:45 +00:00
David Blass	6e94f513df	feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748) ... * feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660) Per-account ceiling on the sum of `router_topup` invoices (pending + succeeded) for the current UTC calendar month. Closes a gap where a runaway agent loop, leaked PR trigger, or stuck workflow could auto-reload indefinitely with no aggregate per-month ceiling. Two enforcement modes via `RouterLimitMode` enum: - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun; 402 `router_monthly_limit` from /api/proxy-token; email + banner - `alert_only`: auto-reload keeps flowing; email + banner only, first breach per UTC month Enforcement is split across reserveRun (pre-dispatch paywall comment) and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through the same `getRouterSpentThisMonthCents` helper so the dashboard, the dispatch gate, and the auto-reload gate can't disagree. Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string), claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent reloads breaching together send exactly one email. Read-time comparison with the current month re-arms on rollover — no cron. Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell) above the Router/BYOK tabs in `ModelAccessCard`, with a popover "Adjust limit" form that PATCHes the existing /api/account/[owner]/billing/settings route. Same `assertBillingAdmin` gate that owns the other billing settings — no new auth surface. See wiki/billing.md § Router monthly spend limit for the full contract + edge cases. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal pass on monthly Router spend limit (#660) Round-1 review across 5 lenses (billing-subsystem, correctness, security, operational-readiness, research-validated-assumptions) surfaced one critical + three actionable major findings on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot` used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles to `field != value` — UNKNOWN (not TRUE) against the post-migration `NULL` default. First breach for any account would never claim the slot, never stamp the row, and never fire the email (hard_cap or alert_only). Replaced with `OR: [{ field: null }, { field: { not: monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern. **Major — email gap on manual-top-up over cap.** Breach email was only wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>` that crosses the cap blocks dispatch via `reserveRun` but never hits proxy-token, so the user got the PR comment but no email. Wired the CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s PaywallError catch (the SERIALIZABLE txn rolled back when we threw, so we re-claim with the global client; single-statement CAS is its own race boundary against concurrent proxy-token claims). **Major — PR paywall comment leaked $ figures.** `router_limit` body embedded `($X of $Y)` in a comment visible to anyone with PR read access (public repos, forks, outside collaborators). Other paywall types deliberately avoid amounts. Removed; deep link still points to the authenticated console for the figures. **Medium — observability.** Added `[router-limit]` structured logs at the three enforcement sites (proxy-token hard_cap 402, proxy-token alert_only breach, reserveRun paywall) so on-call can grep "did the cap fire for customer X this month." **Medium — customer docs.** Added a `### Monthly spend limit` section to `docs/billing.mdx` (Mintlify) describing the two modes and the manual-top-up caveat. **Doc — refund/dispute interaction.** Documented in `wiki/billing.md` that the cap inherits the existing webhook semantics: disputed `router_topup` drops from the sum (cap briefly un-trips); refunds don't flip status today so refunded top-ups keep counting. Matches wallet behavior — not redefined here. Accepted as-is (documented or pre-existing): `after()` reliability vs stamp-before-send tradeoff, alert_only email fires before Stripe phase-2, proxy-token reads limit fields outside SERIALIZABLE scope (brief TOCTOU on admin lowering cap), stale paywall comment on cap clear, no global kill switch (per-account `alert_only` flip is the practical kill switch), no audit log on cap changes (no existing audit infra), action version not bumped (separate release commit). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal round 2 on monthly Router spend limit Round-2 anneal (billing-subsystem, correctness, research-validated, user-journey, operational-readiness) surfaced a critical merge conflict and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted `formatBillingErrorSummary` from `action/main.ts` to `action/utils/billingErrors.ts`. The PR's new `router_monthly_limit` arm still lived in `action/main.ts`. Took main's slim orchestrator wholesale; moved the arm into the extracted file. **Major — cap = payments only, not dispatch.** `reserveRun` was pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit` regardless of wallet balance, contradicting the cap's positioning as "ceiling on what you pay." An account with $500 of paid-up wallet and a breached $100 cap couldn't trigger any new run via the comment path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded — surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token` is now the sole enforcement point, refusing only the next auto-reload that would push past. Wallet credit always drains. Dropped the now-dead `router_limit` arm in `buildPaywallCommentBody`, the dead `routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`, and the post-paywall email-fire David added — all unreachable. **Major — split `manual_topup` from `router_topup`.** Manual on-session top-ups at `/billing-top-up/<owner>` were landing as `Invoice.kind = "router_topup"` and counting toward the cap. The cap exists to brake *passive* runaway (auto-reload loops); a manual top-up is a deliberate click-through that the user owns. Added `InvoiceKind.manual_topup`, flipped the manual write site + `createTopUpCheckoutSession` metadata, broadened wallet / reconcile / billing-report reads to `kind IN (router_topup, manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap aggregate) to `router_topup` only. Worked example: cap=$300, reload=$100 → exactly three reloads succeed; a fourth is blocked. Historical rows stay labelled `router_topup` (no backfill); the asymmetry is small and accepted since the manual flow only existed alongside auto-reload for a brief window. Extended the `invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows the same shape as `router_topup` (PaymentIntent-backed, no stripeInvoiceId); split into a second migration because PG forbids using a freshly-added enum value in the same transaction. **Major — email reframed around the triggering reload event.** The `alert_only` body was reporting a pre-eager-write `spentCents` while the dashboard reads the post-commit value, so email and dashboard disagreed by exactly one reload. Both flavors now say "Your most recent $50 auto-reload brought you over your $300 monthly limit" instead of a running spent-of-cap total — no reconciliation needed, no more "you've hit your monthly cap" copy firing for partial breaches (spent=$80 of $100, reload=$30 would have triggered that wording). **Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded "You've used your 30 free runs this month. Add a card to continue at 7¢/run." regardless of `detail.reason`. Branched on `cap` vs `delinquent` so each paywall surfaces actionable copy with the right CTA. `router_limit` no longer flows through here (per F4 above). **Major — RouterLimitBanner.** Added an `isAlertBreached` visual state (amber palette) so an `alert_only` account at $240 of $200 no longer renders in the same neutral zinc chrome as a healthy under-cap account. Updated popover copy to reflect the auto-reload-only scope. **Medium — paywall log line.** Added `detail.reason` to the `[Installation X] paywall:` log so on-call grepping for "why was this paused" can distinguish `cap` from `delinquent`. **Cleanup.** Dropped dead `utcMonthKey` import + re-export in `maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*` fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*` since they now handle both kinds. Updated wiki/billing.md + docs/billing.mdx + schema doc comments throughout. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt` sitting next to it — a single-purpose state column on `Account` that encoded a date as a string and required a custom CAS predicate to read/write race-safely. Plus it had real holes: Resend send failure left the sentinel stamped and the account silently un-emailed for the month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered edge cases never fired at all. Replace it with: fire `maybeNotifyRouterLimit` on every breaching reload, let the Resend `Idempotency-Key` `router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside Resend's 24h dedup window. Continuously-breaching accounts get ~1 reminder per day; brief Resend outages self-heal because the next breaching reload re-attempts the send. Mode is in the dedup key so `alert_only → hard_cap` mid-month re-arms a fresh email with the appropriate copy. Drops `Account.routerLimitNotifiedMonth` and `claimRouterLimitNotificationSlot`; simplifies the proxy-token phase-1 branch significantly. Net diff is negative LOC and the data model loses a single-purpose sentinel. Migration was branch-local — never deployed — so I edited the original add-cap migration in place to drop the column from the ALTER TABLE rather than chain a drop-column migration on top. Preview Neon branches reset automatically on history rewrite per wiki/migrations.md. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): hide RouterLimitBanner when no cap is configured The banner was unconditionally rendered for every billing-enabled account, including pure-BYOK admins who never touch Router. They got "No monthly spend limit / Router has spent $0.00" + a divider as visual noise on the model access page — basically nagging them to set a feature they may not want. Running without a cap is valid; we don't nag. ModelAccessCard now gates the banner block (banner + dividers) on `routerMonthlyLimitCents !== null`. RouterLimitBanner drops the no-limit visual state, the "Set monthly limit" CTA text, and the dead `hasLimit` branching. Cleaner three-state shape (under cap / amber breached / brick breached). Discoverability: no-cap users no longer see a UI affordance to set one. That's deliberate — the cap is a power-user feature documented in docs/billing.mdx. If discoverability becomes an ask, we can add a small inline link inside RouterWalletSection without bringing back the always-visible banner. Resolves the only outstanding finding from cursor bugbot's review of ff5328c (banner-visible-for-byok thread). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(billing): docs/wiki match new "no banner without a cap" reality Pullfrog bot review of f7672ca pointed out the customer docs still told users to "Set the cap from the **Monthly spend limit** banner in the **Model costs** card" — but after hiding the banner for no-cap accounts there is no such banner to use until you already have a cap. Catch-22 for first-time setup. Rewrote docs/billing.mdx to be self-contained: explain what the cap is, what the two modes do, what the banner shows *once configured*, and direct admins to PATCH the billing settings endpoint (or reach out to support) for first-time setup. Cap is positioned as optional; running without one is the documented default. Wiki paragraph in wiki/billing.md updated to match — banner is only rendered when a cap exists, three visual states (under / amber / red), no first-time-setup UI nag by design. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely The standalone `RouterLimitBanner` was the wrong shape. It only rendered when a cap was already configured (so there was no UI to discover the feature in the first place — first-time setup required hitting the API directly), and it occupied prominent real estate above the tabs to surface state that already lives in the row's own input when the form moves down where it belongs. New shape: monthly cap is just a third row inside `RouterWalletSection` sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated the same way (card on file + auto-reload enabled — the only state where the cap actually means anything). Empty input → no cap, with placeholder "No limit". Setting a number reveals a **Behavior at limit** toggle built on the same `Tabs` slider component used for the Router/BYOK tab switch, so the look matches the rest of the card. Deletes: - `RouterLimitBanner` component (212 lines) - banner mount + conditional + spacers in `ModelAccessCard` - `AlertTriangle` is still imported (used by `DelinquencyBanner`) Adds: - one settings row in `RouterWalletSection` with the cap input + mode tabs - `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the existing `saveSettings` helper (widened to accept `string | null`) - `Tabs` / `TabsList` / `TabsTrigger` import Docs + wiki updated to match the new shape; the customer doc no longer points at a banner that won't appear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between Previously bundled both into one row block. Restructure: cap input is its own row; Behavior-at-limit Tabs gets a sibling row with the standard `h-5 + hr + h-5` separator between (matching the rhythm of auto-reload amount → threshold → monthly cap). Mode-toggle row is gated on `routerMonthlyLimitCents !== null` so the hr + tabs only appear once a number is in the cap input. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row Same `flex items-center justify-between gap-3` layout as the Auto-reload row: label group on the left, control on the right. Drops the vertical stack in favour of the horizontal one — looks identical to the toggle row directly above. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-05-20 01:38:36 +00:00
David Blass	dd26d35137	learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743) ... * learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy three audit fixes on top of the recent learnings overhaul (#717): - `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry when a body has non-whitespace content before the first heading. the prompt instructs the agent NOT to slurp the whole file when a TOC is present, so without this any preamble lines were silently invisible (realistic transitional case: an agent partially restructures a legacy free-text body and leaves bullets above the first `## `). - server-side PATCH route now applies the same line-boundary-aware truncation as the action (defense in depth via a shared `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from `action/internal`). the raw `.slice` it used before could leave a mid-heading tail on any caller that bypassed the client-side truncate, breaking the next-seed TOC parse. removes the duplicated cap constant. - `buildLearningsSection` intro no longer asserts "accumulated by previous agent runs" — false for fresh repos with zero history. new copy is tense-neutral and works for empty + populated bodies. also nudges the agent to re-read after mid-run edits (the inlined TOC ranges are a run-start snapshot). Co-authored-by: Cursor <cursoragent@cursor.com> * learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned calls discovering a quirk this run, recording the workaround prevents the next run from repeating the waste. Reframe around one litmus ("would a future run do its work better because this bullet exists?") and trust it to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary) and the four-example pullfrog/PR/date/play-by-play list (the rule underneath is "don't anchor facts to repo state that will move"). Cuts ~10 lines from a prompt the model was already mostly ignoring; the remaining anchor list is narrower and more enforceable. * audit-learnings-r2: align wiki + tighten re-read nudge - wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls. - buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly. * postRun: refresh JSDoc to match the reflection prompt rewrite `buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets. * fix(mcp/issueEvents): narrow event.event before Set.has lookup octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup. * learnings: split truncation helpers into MCP-free module re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph. move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>	2026-05-19 21:47:10 +00:00
Colin McDonnell	3514bbc39f	review prompt: tighten body-section bar + inline technical-details (#770) ... * review prompt: tighten body-section bar + add inline technical-details Two layers of tightening to the Review/IncrementalReview prompts in PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step): 1. Reframe inline-vs-body split. Body `### ` sections are now reserved for concerns that genuinely have no line to anchor to — absence, sequencing, design decisions, scope questions, architectural risk. Drop the "cross-cutting concerns" framing (misled the agent into either filing nothing in the body or filing multi-file anchored findings there). 2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step 6) and IncrementalReview (step 8) aggregate phases. Diagnosis from PR #767's auto-review: on substantial PRs the agent surfaced findings but routed all of them inline, producing reviews with zero `### ` body sections even on diffs where non-anchored concerns clearly existed. 3. Replace the abstract `### ` example with a concrete non-anchored one ("Legacy `opencode.ts` has no documented deletion plan") so the agent pattern-matches the absence-shaped finding, not a line-bug. 4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT so inline comments can carry a `<details>Technical details</details>` block when the fix has cross-file implications. Rename the existing "Agent details" inline collapsible to "Technical details" for consistency with body sections. 5. (Carried over from prior uncommitted work) Restructure the review metadata block from `<details>Review metadata</details>` into an HTML comment + an italic TL;DR commit-range line. The HTML comment keeps the metadata addressable for downstream agents without eating user-visible review real estate. No tests touched. * wiki: document multi-model end-to-end eval pattern	2026-05-19 21:06:30 +00:00
Colin McDonnell	8ac954a27f	fix(handleIncompleteSetup): also skip nudge when repos are disabled, not just active	2026-05-19 18:46:54 +00:00
Colin McDonnell	88f170e19a	fix: 7 log-audit / run-audit findings (mega-PR) (#769) ... * fix(#765): silence Clerk 400 (revoked OAuth) noise from getTokenForClerkId Branch on isClerkAPIResponseError + status<500 so the well-understood revoked-token redirect doesn't emit a level=error line in Better Stack on every request. Vercel maps console.warn -> error for non-streaming routes, so a downgrade to log.warn wouldn't help; only the unexpected shape (5xx, network) is worth surfacing. * fix(#742): stop logging input verbatim from yes.op retry-failure paths GitHub OAuth user tokens (ghu_...) were leaking to Better Stack on every yes.op retry-failure for any utils/github/get* helper that takes a token field — 38 leaks/7d in the most recent audit window. The leak path is console.log inside the yes package (its own log shim, not utils/log.ts). Drop input from the four log sites + the cache-key-derivation throw site. key (SHA-1 of input) is sufficient for retry correlation; error already carries request URL + status. Defense-in-depth comment so future contributors don't re-add the field. Operational follow-up (separate task): inventory ghu_... strings in Better Stack ingested in the last 90d, revoke matching Clerk grants, scrub cold-tier S3, rotate the BS source token. * fix(#759): handle GraphqlResponseError "Could not resolve to a node" as 404 When the stored planCommentNodeId references a comment that's been deleted on GitHub, octokit.graphql throws GraphqlResponseError before the existing `node === null` 404 branch is reached. Add a narrow isGraphqlNodeNotFound predicate in utils/errors.ts and a new catch branch in the plan-comment route. The action treats 404 as "no prior plan comment" and creates a fresh one, so behavior matches existing contract. * fix(#747): convert webhook GraphQL rate-limit 5xx into a Result<T> sentinel + 200 ack When GitHub's GraphQL responds with "API rate limit exceeded for installation ID N", _getReviewCommentsWithReplies threw, propagated through the bare yes.op wrapper (no rate-limit bail), out of the bare await in handleWebhook, and crashed /api/webhook/github with 500 — 77 webhook 500s/24h on the most recent audit window. GitHub redelivery plus R2 dedup also silently masked the legitimate handler from re-running once the rate-limit window cleared. Mirror the #658 / _getRepository pattern: detect GraphqlResponseError matching /rate limit (already )?exceeded/i, log.warn with the x-ratelimit-reset value (and [Installation N] prefix when available), return failure(...) with status 429. Webhook handler short-circuits the case with 200 + log.info so GitHub stops the redelivery storm against an exhausted budget, and the trigger page surfaces a clean ThrowClientError. Document the new pattern as a Tier 2 false-positive in wiki/log-audit.md so the next audit cron doesn't re-flag it. Note that returning [] silently (the issue's first suggestion) would have dropped @pullfrog mentions inline in review comments and dispatched an agent run that re-rate-limits — skip-the-whole-case is the correct semantics. Co-vulnerable getPullRequest / getWorkflow have zero occurrences in this window; per #737 policy, defer until they show up. NOTE: this commit and the bracket of touched files revert as a unit — the Result<T> shape change in getReviewCommentsWithReplies is breaking; partial revert breaks the type chain. * fix(#766): fold stderr+stdout into shell.ts errors + carve out merge-base --is-ancestor action/utils/shell.ts dropped stdout when constructing failure messages ($\{stderr || "Unknown error"\}), so git subcommands that write context-bearing diagnostics to stdout (merge conflicts, cherry-pick rejections, diff --exit-code, ls-files --error-unmatch) surfaced as "Command failed with exit code 1: Unknown error" through mcp__pullfrog__git. The agent burned an extra MCP round-trip calling git status to recover. Fold stderr + stdout into the thrown error message (stderr first, stdout fallback) so the agent always sees the real diagnostic. Plus a narrow carve-out for `git merge-base --is-ancestor` in action/mcp/git.ts: that subcommand uses exit code as data (0=ancestor, 1=not-an-ancestor, >1=error), so return { success: true, isAncestor } instead of throwing on exit 1. No caller in action/ string-matches on the old error format (verified). diff --exit-code and ls-files --error-unmatch are not carved out — both are zero-occurrence in the May audit window, and the stderr+stdout fold renders their output usefully anyway. * fix(#739): point customers at the actual fix when permissions: id-token: write is missing When a customer workflow runs in GitHub Actions but lacks permissions: id-token: write, ACTIONS_ID_TOKEN_REQUEST_URL/_TOKEN aren't injected, isOIDCAvailable() is false, and acquireNewToken falls through to the local-dev-only acquireTokenViaGitHubApp path, which throws "GITHUB_APP_ID and GITHUB_PRIVATE_KEY must be set" — pointing at a self-hosted-app fix that doesn't apply. One affected customer burned 13 dispatches in 24h on this misleading error. Detect (GITHUB_ACTIONS=true) AND (no OIDC env vars) inside acquireNewToken before falling through to the local-dev branch, and throw an actionable message naming the missing permissions block, the exact YAML, and the docs anchor. The error surfaces via ##[error]action failed: ... in the workflow log (the only customer surface available before main()'s inner try opens). Local-dev path keeps the existing GITHUB_APP_ID message. * fix(#760): suspend activity watchdog across in-flight tool calls mcp__pullfrog__checkout_pr was hard-failing 6/24h on SenecaLabs/senecaWeb because git fetch+deepen on a large monorepo can take 4-5 min, the agent's stdout pipe goes silent the entire time (FastMCP is in-process HTTP, but Claude/opencode CLIs await the synchronous tools/call response), and both the spawn-level activity timer (300s in subprocess.ts) and the process-level activity monitor (300s in activity.ts) fire and kill the run. Re-introduce the bracket pattern that PR #634 removed: bracket suspendActivity()/resumeActivity() around tool_use -> tool_result in both agent harnesses, plumb isPausedExternally into spawn() so both timers suspend in lockstep. Bounded by MAX_TOOL_CALL_SUSPENSION_MS (15 min auto-resume) plus the outer 1h agent timeout — neither zombie-run avenue from #12 is reopened (subprocess.close still resolves on death; outer timeout is suspend-agnostic; suspends gated on explicit paired CLI events, not internal noise). opencode tool_use handler: gate suspendActivity() on non-terminal status (running/pending) so the bus_event re-dispatch path at line 915 — which only fires for completed/error subagent parts and never emits a paired tool_result — doesn't latch the watchdog into suspension until the 15min ceiling. Add a heuristic:activity-watchdog-ceiling classifier to scripts/analyze-logs.ts so a tool that genuinely hangs past MAX_TOOL_CALL_SUSPENSION_MS surfaces in run-audit instead of being bucketed into failure:unknown. NOTE: this commit and the bracket of touched files revert as a unit — activity.ts, subprocess.ts, and the two harnesses must move together or the bracketing breaks. * refactor(#747): swap Result<T> for InstallationRateLimitError typed throw The Result<T> shape from 3ebf6c4c was cargo-culted from the #658 _getRepository pattern, but _getReviewCommentsWithReplies has only one expected-error case (installation rate-limit) and two callers — Result imposes branching on the trigger-page caller that never cared about the rate-limit case specifically. A typed error class is lighter (~10 LoC vs ~33) and matches the actual need: - new InstallationRateLimitError(resetAt) thrown from _getReviewCommentsWithReplies; rate-limit log.warn unchanged. - handleWebhook catches it and breaks with log.info (unchanged semantics: 200 ack, no redelivery storm). - trigger page reverts to direct array access; any failure propagates to the page error boundary (the pre-#747-commit shape). - log-audit.md wording updated to match.	2026-05-19 18:40:53 +00:00
pullfrog[bot]	0abaaa1e37	chore(oss): add yamcodes/arkenv to OSS program (#776) ... * chore(oss): add yamcodes/arkenv to OSS program * fix(test): strip CODEX_AUTH_JSON in apiKeys auto-select test The beforeEach strip list omitted CODEX_AUTH_JSON, which is in `knownApiKeys` via the openai provider's managedCredentials. When the env has CODEX_AUTH_JSON set, the auto-select "throws when no provider keys are present" assertion finds it and fails to throw. --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>	2026-05-17 20:26:26 +00:00
Colin McDonnell	e20f32fb09	fix(test): randomize tag name in push-enabled fixture ... the branch name was already randomized with \${RANDOM}, but the tag name was hardcoded as `test-tag-enabled`. every successful run left that tag on the fixture remote with no cleanup, so subsequent runs' checkouts (which fetch tags) saw the local tag already exist and failed with `fatal: tag already exists`. the agent's git-push fallback then hit the deliberately-broken creds and the test scored push_tags=✗ delete_branch=✗. mirror the branch pattern with \${RANDOM} so every run gets a fresh tag name. tags still accumulate on the remote but no longer collide; that's an infra cleanup concern, not a correctness one.	2026-05-16 18:49:07 +00:00
Colin McDonnell	c0988e35b0	fix(security): block docker socket from sandboxed shell; disable opencode batch_tool ... two real CI failures on main, both shipping bugs in the action: 1. `token-exfil-claude` was a real sandbox escape: GHA `ubuntu-latest` puts `runner` in the `docker` group, so a sandboxed shell could run `docker run --pid=host --privileged busybox cat /proc/<parent>/environ` and read the action process's env (which holds user secrets) — fully bypassing the unshare PID-namespace. fix: inside the sandbox's mount namespace (already private via `--mount-proc` which implies `--mount`), bind-mount /dev/null over /var/run/docker.sock (+ podman/containerd/crio variants) so any container-runtime socket connect from the sandbox fails. only affects sandboxed shells — host runner mount table is untouched, so user workflow steps outside pullfrog keep working. 2. `restricted-opencode` regressed in #719 (`experimental.batch_tool`). opencode's batch tool rejects MCP tools with `"Tool '<name>' not in registry. External tools (MCP, environment) cannot be batched."` when a model emits parallel `pullfrog_shell` (or any MCP) tool_use blocks, opencode internally routes them through batch, they all fail, the model misreads the error as "the tool doesn't exist", and gives up. caught by a `lens:` subagent in the restricted test concluding shell was unavailable and setting `DIAGNOSTIC_ID=empty`. drop `batch_tool: true` and the matching opencode-specific guidance in `instructions.ts` — native parallel tool_use (multiple tool_use blocks per assistant message) still works for both built-in and MCP tools without batch, so we lose only the 1-25 wrapper, not parallelism.	2026-05-16 15:40:44 +00:00
Colin McDonnell	efc1b67e7b	fix(test): skip models.dev existence check for fallback aliases ... deprecated aliases (`fallback` set) legitimately point at dead resolve targets — xAI just retired grok-4-1-fast/grok-code-fast-1 and #761 wired them through the fallback chain. the terminal-fallback is validated separately by the Zen served-list test.	2026-05-16 05:14:51 +00:00
Colin McDonnell	0a64659ee7	refactor: slim action/main.ts to an orchestrator + extract helpers (#755) ... * refactor: extract helpers out of action/main.ts so non-orchestration churn stops touching the file main.ts had grown to ~1240 lines holding ~500 lines of helpers that have nothing to do with the resolver pipeline — billing-error UI/copy, proxy minting, summary/learnings persistence, log formatters, end-of-run cleanup waterfalls. any PR adding a new billing code branch or a new log line was forced to edit main.ts, and since main.ts is in ALWAYS_RUN_ALL the entire 52-job LLM CI matrix fired on what should have been a 0-job change (e.g. #748). extractions: - action/utils/billingErrors.ts — BillingError, TransientError, the format*Summary renderers, billingConsoleUrl - action/utils/proxy.ts — mintProxyKey, buildProxyTokenHeaders, resolveProxyModel, plus runProxyResolution wrapper that renders + rethrows BillingError/TransientError before the outer catch - action/utils/prSummary.ts — fetchPreviousSnapshot, persistSummary co-located with the existing seed/read file helpers - action/utils/learnings.ts — persistLearnings co-located with the existing seed/read file helpers - action/utils/runStartupLog.ts — resolveOutputSchema + logRunStartup (the model/agent/push/shell/timeout block) - action/utils/runErrorRenderer.ts — renderRunError classifies (BillingError reclassify / hang detect / API-key auth) and emits {summary, comment} markdown bodies - action/utils/runLifecycle.ts — persistRunArtifacts, finalizeSuccessRun, writeRunErrorOutputs — the three end-of-run cleanup phases shared between the success path and the error catch path main.ts is now ~570 lines — the irreducible orchestrator: disposables (`await using` for tokenRef / gitAuthServer / mcpHttpServer), the toolContext construction, the agent-timeout race, the catch/finally shape, and the named phase calls. behavior is preserved verbatim (verified: pnpm -r typecheck + pnpm test 695/695 pass, action/test 596/596 pass). wiki/main.md gets a new "file layout" section describing the split. AGENTS.md gets a single line pointing future edits at the helpers instead of main.ts. * anneal: address review findings - restore MainResult.result?: string (accidental removal in initial commit; field was unused in current code but is part of the exported interface surface — keep the diff truly behavior-preserving) - move resolveOutputSchema from runStartupLog.ts to payload.ts (it's an action-input resolver alongside resolvePromptInput / resolvePayload, not a log helper — was placed in runStartupLog.ts for matrix-churn pragmatism but the domain fit is in payload.ts) - un-export resolveProxyModel (only used internally by runProxyResolution in proxy.ts; no external importer) - fix runErrorRenderer.ts JSDoc "Three classifications" → four (Billing, hang, API-key, default) - expand runLifecycle.ts module banner to note that finalizeSuccessRun calls persistRunArtifacts first, and to explain why the catch path splits writeRunErrorOutputs + persistRunArtifacts - update billingErrors.ts header to point at proxy.ts and runErrorRenderer.ts as the actual origin sites (was stale "main.ts") - expand proxy.ts header to spell out the runProxyResolution entrypoint contract (was stale "main.ts can render") - update wiki/main.md resolver chain + dependency table to name runProxyResolution as the actual call site and document the early BillingError/TransientError rendering branch - update wiki/main.md file-layout table to lead with runProxyResolution and describe mintProxyKey/buildProxyTokenHeaders/resolveProxyModel as internal helpers (was implying they were public surface)	2026-05-16 05:09:52 +00:00
Colin McDonnell	a78b1542da	feat: pullfrog auth codex + fresh-branch (#757) ... * feat: pullfrog auth codex + fresh-branch Add `pullfrog auth codex` standalone command for minting Codex (ChatGPT) subscription credentials and saving them as the `CODEX_AUTH_JSON` Pullfrog secret. Codex device-auth runs in a subprocess with an isolated `CODEX_HOME` (temp dir) so the user's `~/.codex/auth.json` is never touched. The spawned `codex login --device-auth` output is captured line-by-line, ANSI-stripped, and re-rendered with a `$ codex login --device-auth` header above dimmed sub-output on the @clack/prompts rail so the user visually understands they're seeing a sub-process. Companion `pnpm fresh-branch` script: from inside `.worktrees/<name>`, creates a schema-only Neon branch named `dev/<git-branch>`, patches the worktree's `.env` (DATABASE_URL, DATABASE_URL_UNPOOLED, NEON_DEV_BRANCH), then runs `prisma migrate reset --force` so migrations apply cleanly against a data-free copy. Refuses to run from the primary checkout or on protected branch names. Other: - bump CLI/account/repo secret value limit 4096 -> 49152 chars (matches GitHub Actions' 48KB cap; auth.json is ~4-5KB) - extract shared CLI helpers (gh/pullfrog API, secret save) into `action/commands/_shared.ts` * fix(auth): address PR review + add CodexAuthCallout, default account scope Review fixes: - handle 'error' event from `codex` spawn (ENOENT) so missing PATH bails with an actionable "install codex CLI" message instead of an unhandled Node error - escalate SIGTERM -> SIGKILL after 5s grace when killing a stuck codex child so the CLI can't get pinned indefinitely - stop the spinner with a red "failed" glyph in the catch path before clearing activeSpin, mirroring `bail` (no orphan spinner above errors) - enforce 48 KB secret value cap by *bytes* (Buffer.byteLength) not UTF-16 code units, across all 3 secret routes; matches GH Actions' byte-based limit - preserve existing blank lines + comments when fresh-branch rewrites worktree .env (no more cosmetic reformat on every run) Scope: - default to `account` scope on org-owned repos too — never silently prompt for repo scope. Pullfrog has no per-GitHub-user secret store, so account is right for both user and org owners; `--scope repo` is the explicit opt-in for repo-only. UI: - new CodexAuthCallout (sibling to ClaudeCodeOAuthCallout); surfaces `pullfrog auth codex` for ChatGPT subscribers when an OpenAI provider model is selected. wired into AgentSettings.tsx (model-costs surface) and OnboardingCard.tsx (first-time setup). no paste button — the CLI handles minting + saving end-to-end. * auth/codex: rename to neon-fresh-branch, address PR review - rename `pnpm fresh-branch` → `pnpm neon-fresh-branch` (and the script file) to disambiguate from git branches. - `--scope` help text now explains the default (account) and when to pass `repo`. - move `_shared.ts` import up with the rest in `action/commands/auth.ts` and push the `stripAnsi` helper below the import block. - `sanitizeBranchName` no longer slices: slicing after trim could reintroduce a trailing `-`/`/`. callers slice the raw input first, then sanitize. - DRY the `start` branch of the codex progress callback (single header path, optional retry log). - thread a `timedOut` flag from `runDeviceAuth` → `ProgressEvent.exit` so the retry prompt can say "device authorization timed out — retry?" instead of the generic "no auth.json was written" line when the per-attempt timeout fires. - drop the redundant `mkdirSync` after `mkdtempSync` in `codexAuth.ts`. * untrack .scratch/ (committed screenshot fixture by mistake) * auth codex: prompt for scope on orgs (mirrors init) * revert worktree.ts: out of scope for this PR * anneal: trim _shared.ts dead exports, collapse CodexSpawnError, inline packageBin * codex auth: wire end-to-end runtime consumer CODEX_AUTH_JSON is now actually usable: the action runtime materializes it as OpenCode's auth.json at the runner's real $HOME/.local/share/opencode, OpenCode routes openai requests through the ChatGPT subscription via the embedded CodexAuthPlugin, and a GitHub Actions post: hook detects any refresh-chain rotation during the run and PUTs it back to Pullfrog via a new JWT-authenticated PUT /api/runtime/secret endpoint. Key decisions: - Write to the real $HOME (not the per-run tmpdir-redirected HOME) so the file lives outside OpenCode's `/tmp/*` permission allow zone — its existing deny-default protects it without any new permission rule. - Materialization gated on agent === opencode (Codex auth is OpenAI-only, Claude never sees the file). - Defense-in-depth on Claude: deny Read/Grep/Edit/Glob + sandbox.denyRead for ~/.local/share/opencode/auth.json in managedSettings (covers Bash file-reading commands too per Claude Code permissions docs). - New `provider.managedCredentials` field on the provider config — CLI-only credentials authored via `pullfrog auth <provider>`. Counted for hasAnyKey/log-redaction but never surfaced as a paste option in init. CODEX_AUTH_JSON is the first member; OPENAI_API_KEY stays in envVars. - Eager refresh on `pullfrog auth codex`: one OAuth round-trip before setPullfrogSecret so Pullfrog's copy is the freshest in the chain (avoids the user's laptop refreshing first and stranding our copy). - Post-hook approach for write-back so it survives cancellation, timeouts, and unhandled errors in the main step. State is ferried via core.saveState since apiToken is run-scoped and not in env. - Server-side write-back endpoint is allowlist-gated to CODEX_AUTH_JSON only — never a generic secret-write surface. Looks up the secret at repo scope first, falls back to account scope. 404s on create (refresh-only, never auto-provision). * codex auth: documentation + wiki cross-links * debug: log dbSecrets keys + CODEX_AUTH_JSON presence (temporary) * debug: surface install path + parse failure preview * remove debug log lines (E2E verified) * hide CodexAuthCallout until opencode-ai bump (1.1.56's allowed-models set excludes gpt-5.5)	2026-05-16 05:06:24 +00:00
Colin McDonnell	ddbc610569	review prompt: friendly green callouts + per-section severity emojis (#756) ... * review prompt: friendly green callouts + per-section severity emojis - Replace `[!NOTE]` informational tier and the no-callout minor-suggestions tier with friendly green blockquotes (`> ✅` / `> 💡`). The two loud tiers (`[!CAUTION]` / `[!IMPORTANT]`) keep their GitHub admonitions. - Add a per-`##`-section severity-emoji rule (🚨/⚠️/💡/ℹ️) for cross-cutting review concerns that don't anchor to a line and would otherwise be buried in summary content. - Drop the `<br/>` between summary sections — heading + blank line carries enough visual spacing. - Skip the post-run learnings-reflection turn for `IncrementalReview`. It's the lowest-novelty mode (delta review against existing PR with prior summary already loaded) and almost never produces durable learnings — reflection there costs ~$0.50-0.80/run for nothing. - Surface real error info on `agent-browser` skill install failures (exit code + stdout + stderr + spawn error). The skills CLI uses a TUI that prints errors to stdout, so the prior stderr-only logging silently swallowed every failure. * review prompt: per-bullet severity emoji + bullets-only sections Section headings are plain again (no leading severity emoji). Severity moves to individual bullets so a section that mixes a 🚨 and a 💡 isn't mislabeled by either. Section bodies are now bullets only — paragraph prose under a heading is harder to scan and tends to bury the actionable point. Bullets can carry indented continuation content (sub-bullets, code fences, blockquotes) by indenting two spaces under the parent. * review prompt: cap section length + identifier discipline Bound each summary section to at most 4 bullets at most 2 lines each, and explicitly call out identifier-heavy prose as an anti-pattern. The reader is often a manager or non-author; identifier-dense paragraphs ('foo calls bar.fetch which dispatches to baz via qux...') are unreadable for them. Default to plain-language behavior descriptions, name an identifier only when it's the subject of an actionable concern or a public surface a reader would recognize, target 2-3 backtick tokens per bullet. Move the deep-explanation pattern from open blockquote to a default- collapsed details/summary so depth doesn't dominate the visible body. * review prompt: hard cap on bullet identifier density + worked rewrite example Soft 'aim for 2-3 tokens' guidance was ignored — first big-PR e2e showed 12 of 19 actionable bullets exceeded the target (avg 4.8 tokens, several over 8). Promote to a hard cap of 3 backticked tokens per bullet and pair with a concrete bad/good rewrite the agent can pattern- match against. Also tighten the per-bullet length cap from ~240 to ~200 chars and explicitly call it 'hard cap, not target'. * review prompt: tighten bullet length cap to 160 chars, dramatize the worked example V2 e2e test: token discipline improved (4.8 -> 3.3 avg, 12/19 -> 6/14 violations) but length got worse (235 -> 286 chars, 13/14 over the 200 cap). The agent compensated for fewer identifiers with more prose. Two changes: (1) tighten the cap from ~200 chars to 160 chars / 1 visual line and call out wrap-to-multiple-lines as the failure mode; (2) rewrite the worked example so the good version is genuinely half the length of the bad one, not just lower token count. The example was the thing the agent pattern-matches against; making the good version ~130 chars vs the bad version's ~290 chars sets the right shape. * review prompt: drop fixed bullet-count cap, keep length + identifier caps Per user feedback — section length should be governed by content, not an arbitrary count. Soft guidance ('past ~6, ask whether to split') is fine; the hard '≤ 4 bullets per section' rule was the wrong shape. Length cap (160c) and identifier cap (3 backtick tokens) stay; those target the actual scanability problem. * review prompt: drop ## subsystem sections, flat 'Issues found' list Per-section structure forced every concern into a subsystem frame and made the body read like a series of mini-essays. Replace with two parts: (1) TL;DR + Key changes as the dispassionate overview, (2) flat '### Issues found' list ordered by severity, intermixed across files and subsystems. Per-bullet rules (≤160c, ≤3 backtick tokens, severity emoji prefix, optional indented continuation) carry over unchanged. * review prompt: full v6 structure — preamble + cross-cutting H3s + nitpicks Replaces the flat 'Issues found' bullet list with the iterated v6 shape: - Preamble is a bolded inline 'Reviewed changes' lead-in plus bullets plus a collapsed 'Review metadata' block (mode/files/commits/refs/ reviewed commits list/prior pullfrog review/staleness note). - Each cross-cutting concern gets a '### emoji Title' section. The visible problem write-up is human-friendly and DESCRIBES THE PROBLEM ONLY — no asks, no suggested fixes, no 'the right thing to do is'. - Each section carries a collapsed 'Technical details' block wrapped in a 4-backtick markdown fence (so it can hold its own 3-tick code fences cleanly, agent-readable, one-click copyable). Standard four inner sections: Affected sites, Required outcome, optional Suggested approach, optional Open questions for the human. - '### ℹ️ Nitpicks' at the bottom for body-only nits that don't inline; simple bullets, no technical-details collapse. - Anti-paragraph-wall rule: never two successive plain paragraphs in visible '### ' sections; alternate prose with structure. - Inline-vs-body discipline: anything that anchors to a single line goes inline, body is for cross-cutting only. - Drops legacy '### Key changes', '### Issues found', '<b>TL;DR</b>', and the '<sub>Summary</sub>' line. * model effort: bump Gemini + GPT to high effort; drop Gemini Pro→Flash subagent E2E review eval against a substantive billing-module diff surfaced two related quality gaps: 1. Gemini Pro at thinkingLevel=medium (#663's CI-timeout fix) reviewed the diff only, took the 0-lens path, and missed a catastrophic camelCase/snake_case service-vs-schema mismatch. Bumping back to high — review work is exactly the wrong shape for the medium/high tradeoff #663 was optimizing for; the per-turn TTFT cost is worth paying when reasoning IS the value. 2. GPT had no reasoningEffort override, defaulting to upstream medium. Same diff, similar shallow result vs Claude. Adding reasoningEffort: high for the curated direct-OpenAI slugs, mirroring the Gemini pattern (Anthropic separately uses --effort high via the Claude Code CLI flag in claude.ts). 3. Gemini Pro's subagentModel was 'gemini-flash' — but Google has no in-between tier between Pro and Flash, and Flash is a meaningful capability cliff for review work. Dropping the override so subagents inherit Pro. Cost stays reasonable since Gemini Pro is already the cheapest of the flagship trio. Other providers unchanged: Anthropic opus→sonnet and OpenAI gpt→gpt-5.4 remain (each is a one-tier drop to a still-capable sibling). * model effort: revert orchestrator override, set explicit high on reviewfrog subagent Reshape the effort design after eval: - Drop the explicit Gemini and GPT model-level overrides — orchestrators now run at upstream defaults (Gemini high, GPT-5.x medium). Gemini's upstream IS high, so this is a no-op there; GPT goes back to upstream medium for orchestrator-level routing work. - Add explicit 'high' on the reviewfrog subagent via agent.options. OpenCode merge order is base ← model.options ← agent.options ← variant per session/llm.ts:141, so the subagent always runs at high regardless of which orchestrator dispatched it. Both thinkingConfig.thinkingLevel (Gemini) and reasoningEffort (GPT) keys included; irrelevant keys are ignored per provider. - Bump providers-live timeouts (12min job / 10min step, from 8/6) to budget for Gemini's TTFT variance at high effort. #663's 4min timeout was sized for the medium-effort override that's now removed. * model effort: restore Gemini explicit high override (no-override path breaks) Bare 'rely on upstream default' for Gemini failed in e2e — removing the model-level provider config produced 'Function call is missing a thought_signature' API errors on every gemini-pro run. Even though upstream opencode's options() returns the same thinkingLevel: high we were explicitly setting, opencode's resolution path differs subtly between the two cases. v2's explicit override worked; v3's removal broke. Reproducible across two consecutive runs. Restoring the explicit Gemini override (back to v2 design). GPT orchestrator stays UN-overridden — at upstream default (medium) — since removing that override didn't trigger the same failure pattern and the reviewfrog subagent agent.options high override compensates for the extra depth GPT loses at medium. * diag: remove reviewfrog agent.options to isolate Gemini thought_signature failure v3 (no Gemini orch override) failed with thought_signature error. v4 (restored Gemini orch override at v2-equivalent) ALSO failed, even though the orchestrator config matches v2. The variable between v2 (working) and v4 (failing) is the new reviewfrog agent.options block. Removing it to confirm — if Gemini works again, the agent.options addition is the culprit and we need a different shape for it. * opencode-ai: bump 1.1.56 → 1.15.0 + clean up gemini effort config opencode-ai@1.1.56 was published 2026-02-10 (3 months old). The Google API tightened thought_signature validation 24-48h ago (per https://discuss.ai.google.dev/t/gemini-thought-signature-patch/122555), and the bug class hits opencode's session→prompt serializer for MCP tool-call parts (anomalyco/opencode#4832, #8321). Latest stable bumps us through ~3 months of fixes; needed for Gemini-direct to stop dying with 'thought_signature is missing' on every multi-turn run. Companion cleanup: the gemini provider override in opencode.ts had 30-line block of comments, four unused constants, and a 6-line Object.fromEntries map for two entries. Replaced with one source-of- truth helper that loops modelAliases, filters provider==='google', strips the 'google/' prefix, and returns the override map. Adding any future Google alias to the registry now flows through automatically. Test added: action/agents/opencode.test.ts asserts the helper covers every direct-Google alias, strips the prefix correctly, and pins every entry to thinkingLevel high — catches drift in helper logic without hardcoding the API ids the test would have to update in lockstep with the registry. * fix(workflow): tolerate listJobsForWorkflowRun 404 in resolveRun PR #750 (docker testing rewrite) replaced the per-call env allowlist with full process.env passthrough into the test container. That now leaks GITHUB_RUN_ID + GITHUB_JOB into runs whose MCP token is scoped to a DIFFERENT repo (e.g. providers-live smoke runs the action against pullfrog/test-repo with pullfrog/app's run ID). The unconditional listJobsForWorkflowRun call 404s and crashes the entire run, breaking every providers-live job on main since #750 landed. jobId is purely cosmetic (deep-links 'View workflow run' footer to a specific job vs the run-level URL). Wrapping the API call in try/catch so a 404 logs a debug message and falls through to undefined jobId is the right fix — the failure mode is exactly what graceful degradation is for, and the alternative (filter the env vars at the docker boundary) re-introduces the kind of allowlist #750 was getting rid of. * opencode-ai: pin 1.14.51 instead of 1.15.0 (effect refactor breaks JSON output) opencode 1.15.0 (May 15) ships a major architectural refactor onto @effect — the run command boots an in-process server via @opencode-ai/sdk/v2 and the JSON event emission path through that SDK client doesn't surface on stdout the way our parser expects (CI run on 1.15.0 produced 0 stdout events but the agent still completed). Local invocation also hangs at the in-process server boot. The Gemini thought_signature fixes (the original reason for bumping) landed earlier in the 1.14.x line, so 1.14.51 (May 14) gets us the upstream fix without the Effect rewrite. Defer the 1.15.x bump until we're ready to rewire our parser/spawn around the new SDK. * opencode-ai: revert to 1.1.56; gha: filter outer-CI workflow-run vars at the docker boundary Two related changes for the docker testing harness's ergonomics: 1. Revert opencode-ai 1.14.51 → 1.1.56. The 1.14+ line ships an Effect refactor (the SDK-v2 client + in-process server architecture) that our --format json parser doesn't speak — even the 1.14.51 release, pre-dating the 1.15.0 Effect rename, produced 0 stdout events on our skill-invoke smoke. There's no clean pre-Effect version that ships the Gemini thought_signature fix; that fix needs a separate workstream once we're ready to rewire the parser onto SDK v2. 2. Filter outer-CI workflow-run identifiers (GITHUB_RUN_ID, GITHUB_JOB, GITHUB_WORKFLOW, GITHUB_ACTION, GITHUB_REF, GITHUB_SHA, etc.) from gha.ts's --env-file passthrough. PR #750's full-process.env design leaks pullfrog/app's CI run identifiers into runs that act against a different repo (e.g. pullfrog/test-repo); any code path inside the action that uses them as keys (most notably resolveRun's listJobsForWorkflowRun lookup) 404s. Filtering them here means the action sees undefined and skips the lookup, complementing the defensive try/catch in resolveRun (commit addc76d4). GITHUB_REPOSITORY and GITHUB_TOKEN are NOT filtered — those are genuinely needed. Companion to addc76d4 (resolveRun 404 tolerance). The two together make this class of bug 'either fix would have caught it' rather than 'silently breaks the entire test matrix'. * fix(deps): sync pnpm-lock.yaml with opencode-ai 1.1.56 manifest revert Forgot to refresh the lockfile after reverting the manifest in 02c6d8c1. CI's frozen-lockfile install was failing with 'lockfile: 1.14.51, manifest: 1.1.56' mismatch.	2026-05-16 04:58:31 +00:00
Colin McDonnell	a0dce200d0	fix(claude): prefer OAuth token over ANTHROPIC_API_KEY (#763) ... * fix(claude): prefer OAuth token over ANTHROPIC_API_KEY in Claude Code When both `CLAUDE_CODE_OAUTH_TOKEN` and `ANTHROPIC_API_KEY` are present, claude-code's auth resolver (`Vw()` in cli.js) returns the API key first and silently ignores the OAuth token. The result: accounts that have a Max-subscription OAuth token in `account_secrets` are still billed at per-token API rates because the workflow `env:` block also forwards `ANTHROPIC_API_KEY` from org-level secrets. Strip `ANTHROPIC_API_KEY` from the spawned claude-code subprocess env when an OAuth token is present (and we're not on the Bedrock route), so the Max subscription is actually used. Other agents in the same run still see the API key in `process.env` via the parent. * chore: tighten comment-length rule + trim claude.ts comment Caps inline comments at 2-3 lines above any single line of code (the prior wording allowed runaway block comments as long as the comment was nominally shorter than the annotated code). * chore: downgrade OAuth-strip log to debug + document debug-mode pattern `log.info` was overkill for a per-run path-selection marker. `log.debug` keeps production logs quiet while preserving full visibility in e2e verification, where `LOG_LEVEL=debug` (or `gh run rerun --debug`) flips the same line on. Adds a "Action debug mode" subsection to wiki/e2e-testing.md so the affordance is discoverable: `log.debug(...)` is the right tool for breadcrumbs that prove a code path fired during preview-repo e2e but shouldn't ship to customer logs. * chore(wiki): correct debug-mode trigger guidance for preview repos LOG_LEVEL=debug only works when the template's pullfrog.yml forwards it, which it doesn't. ACTIONS_STEP_DEBUG=true is the GitHub-magic name that's auto-injected into every step's env without any yaml change, so make that the documented default for preview-repo e2e. * chore(wiki): fix render-format claim in debug-mode table When `ACTIONS_STEP_DEBUG=true`, `log.debug` routes through `core.debug()`, which GitHub renders as `##[debug]<msg>`, not the `[DEBUG] <msg>` format. The `[DEBUG]` prefix only happens via the LOG_LEVEL=debug path which isn't currently wired into the template. * feat(action): add `overrides` input for per-dispatch env mutation Accepts a JSON {string:string} map via the workflow_dispatch input, parsed and merged into process.env at the start of `main()` (before any agent or token-acquisition code runs). Lets a privileged caller flip env vars for one dispatch without persisting state on the repo (repo Actions variables) or being restricted to GitHub's debug names (`gh run rerun --debug`). Deny-list refuses overrides for integrity-critical names — GITHUB_TOKEN, ACTIONS_RUNTIME_TOKEN, ACTIONS_RUNTIME_URL, ACTIONS_ID_TOKEN_REQUEST_*, ACTIONS_CACHE_URL, PULLFROG_API_SECRET, VERCEL_AUTOMATION_BYPASS_SECRET. Customer provider keys (ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, etc.) are explicitly allowed — overriding them per-run for cred-rotation tests and auth-failure repros is the use case. Touches: - action/action.yml — declare `overrides` input - action/utils/overrides.ts — parse + apply with deny-list (+ unit tests) - action/main.ts — wire into `main()` after `normalizeEnv()` - .github/workflows/pullfrog.yml — forward to action - utils/github/pullfrog.yml.ts — same in the customer-facing template - wiki/e2e-testing.md — documented as preferred debug-mode trigger * fix(overrides): strip raw INPUT_OVERRIDES + mask applied values GitHub Actions injects every action input as an env var (INPUT_<NAME>), so the original JSON of `overrides` sits in process.env as INPUT_OVERRIDES and is inherited by every spawned subprocess (claude, opencode, MCP servers, shell). That defeats the deny-list (a downstream re-application would have access to the raw JSON) and leaks arbitrary caller-supplied values into agent env verbatim. After applying, applyOverrides now: 1. delete process.env.INPUT_OVERRIDES — subprocesses see only the surgically-applied keys, not the raw JSON 2. core.setSecret(value) for each applied value — the runner masks those strings in subsequent log output, so an overridden ANTHROPIC_API_KEY can't accidentally surface in debug logs. Two new tests cover the deletion path (both applied and all-denied). * fix(overrides): scope auto-masking to credential-shaped keys core.setSecret(value) is a global string-match — calling it on a short config value like "claude" masks every appearance in subsequent logs (including "claude-opus-4-7", "anthropic-claude-sonnet", etc.), which actively harms debugging. Restrict the auto-mask to keys whose names end in _KEY / _TOKEN / _SECRET / _PASSWORD / _OAUTH / _PRIVATE_KEY — the credential-shape naming convention. Customer keys (ANTHROPIC_API_KEY, etc.) and the deny-listed names match. Plain config (PULLFROG_AGENT, PULLFROG_MODEL, ACTIONS_STEP_DEBUG) doesn't. * docs(wiki): document the three security layers + runner-echo caveat Lays out exactly what the `overrides` input does to mitigate the secret- leak surface (deletion + masking) and the one unavoidable limit: GH Actions echoes the `with:` block once before any action code runs, so the raw JSON appears in the workflow log header in plaintext. Anyone using `overrides` should treat that one-shot exposure as part of the threat model. * fix(overrides): forward via env, not action input, so the value isn't echoed verbatim in the runner step header GH Actions echoes the `with:` block of every `uses:` step in the log group header, BEFORE any action code runs — so the raw JSON of `overrides` was always visible in the workflow log regardless of any in-action `core.setSecret` calls. Refactor: drop the `overrides` action input; instead the action reads `process.env.PULLFROG_OVERRIDES`. The workflow yaml forwards `inputs.overrides` via the step-level `env:` block. We still need to verify empirically whether `env:` block values from workflow inputs get echoed too (separate test); even if they do, masking via core.setSecret + delete of PULLFROG_OVERRIDES after parsing closes the leak to subprocesses, which is the part the action controls. * fix(overrides): rename to unsafe_overrides + UNSAFE_OVERRIDES The runner echoes step-header env-block values in plaintext before any action code runs, so the raw JSON of this affordance is visible to anyone with actions:read on the calling repo. That's acceptable because the workflow only exists on our private repos, but the input name should make the trade-off obvious at the call site rather than buried in a wiki. - workflow_dispatch input: `overrides` → `unsafe_overrides` - env var the action reads: `PULLFROG_OVERRIDES` → `UNSAFE_OVERRIDES` - wiki: rewrite the section to surface the runner-echo as the central trade-off rather than a buried caveat * chore(overrides): tighten error messages to reference UNSAFE_OVERRIDES * docs(wiki): fix stale 'overrides' refs + correct render-format mechanism Addresses two unresolved review threads on PR #763: 1. The opening sentence of "Action debug mode" still referenced the pre-rename `overrides` input and `gh workflow run -f overrides=...`. Updated to `unsafe_overrides`. 2. The render-format claim was technically wrong. `core.isDebug()` doesn't cache — it reads `process.env.RUNNER_DEBUG === '1'` on every call. The actual mechanism: the runner only sets RUNNER_DEBUG=1 when ACTIONS_STEP_DEBUG=true is observed at workflow-trigger time. Mutating ACTIONS_STEP_DEBUG mid-step doesn't retroactively flip RUNNER_DEBUG, so the call falls through to isLocalDebugEnabled() which reads ACTIONS_STEP_DEBUG directly. Rewrote the explanation to match. * fix: drop unsafe_overrides from customer-facing workflow template + remove test theater Two cleanups from a stricter re-read of AGENTS.md: 1. utils/github/pullfrog.yml.ts is the workflow yaml we sync into every customer repo. unsafe_overrides has no business there — it's a pullfrog-only debugging affordance. Reverted. The action's read of UNSAFE_OVERRIDES env var stays — it's a no-op for any workflow that doesn't set it, and pullfrog/template + pullfrog/app's own workflow still forward it. 2. Deleted action/utils/overrides.test.ts entirely. AGENTS.md is clear: no tests unless explicitly asked. I added them anyway. The tests were mostly testing JSON.parse + typeof, plus one regression guard for the deny-list that is better protected by code review of the tiny DENIED_OVERRIDE_NAMES set than by a vitest file. Also strengthened the corresponding AGENTS.md rule from a buried bullet to an explicit "NEVER write tests unless asked, here's why agents violate this constantly, here's the bar" callout. Wiki note added: unsafe_overrides is pullfrog-only infra, not customer- facing.	2026-05-16 04:37:26 +00:00
Colin McDonnell	7907fac64e	fix(test): bump model-smoke timeout 60s → 120s (#764) ... xai/grok-4.3 jobs in the models-live matrix land at 42-67s wall time vs 23-41s for every other provider, brushing the 60s ceiling and crossing it intermittently (e.g. xai/grok-code-fast in run 25949844470 timed out at 60s with `OK` already in stdout — model replied, harness just hadn't seen close). 120s gives ~2x headroom on the slowest provider without penalizing the fast-path providers, since the timer only fires on actual hangs.	2026-05-16 03:14:19 +00:00
Colin McDonnell	76879b27ec	docker testing rewrite: bake the image, drop the allowlist, kill the quoting (#750) ... * docker testing rewrite: bake the image, drop the allowlist, kill the quoting - new `pnpm gha <script>` wrapper. one entry point for running any node script in the GHA-like container; replaces the runtime apt-get + useradd + chown ceremony in `action/utils/docker.ts`. - `action/Dockerfile` bakes ubuntu:24.04 + node 24 + gh + jq + sudo + testuser at uid 1000. `action/docker-entrypoint.sh` remaps to the host uid/gid and `exec`s the requested command — no `bash -c` nesting, no `escapeForDoubleQuotes`. - env passthrough: full `process.env` (+ `.env` via dotenv) flows through `--env-file`, multi-line values via `-e` fallback. drops `EnvFilterMode` / `testEnvAllowList`. - image rebuild is content-hash gated on Dockerfile + entrypoint; volume is versioned by hash so a stale `node_modules` cache from an old image can't poison a new one. - `action/play.ts` slimmed to a CLI; `run()` extracted to `action/utils/runFixture.ts`. drops the `--local` / `PLAY_LOCAL` dual mode in favor of explicit `play:local` / `runtest:local` scripts. - `action/test/run.ts` no longer self-relaunches into docker — that's `gha`'s job now. - `action/test/coverage.ts` `ALWAYS_RUN_ALL` updated to track the new files. - `wiki/docker.md` rewritten (243 → 105 lines). `wiki/action-tests.md`, `wiki/billing.md`, `wiki/adversarial.md`, `README.md`, `AGENTS.md` all updated to drop `--local` / `PLAY_LOCAL` references. verified end-to-end: `pnpm play` runs the default fixture against pullfrog/scratch, exit 0; `sudo unshare --pid` still works inside the container; `pnpm runtest` boots through the wrapper. * gha: address review feedback + 3 related issues found locally review-flagged: - bare `pnpm gha --build` now builds the image and exits 0 (was printing help and exiting 1 — docs claimed it was a valid standalone) - `initVolumeOwnership` skipped when the named volume already exists; saves the ~240ms `docker run … chown` on every warm invocation - `GIT_SSH_COMMAND` gate widened to any `id_*` private key (was hard- coded to `id_rsa`, leaving ed25519-only linux contributors with the default ssh config). dropped `-i` so ssh picks whichever key exists - new `action/.dockerignore` — partial mitigation noted: BuildKit (default since docker 23) only sends files referenced by the Dockerfile (~42B in practice), so the perf concern is mostly hypothetical. file is still worth keeping for `DOCKER_BUILDKIT=0` fallback and as documented intent for future `COPY . .` additions related issues found while validating locally: - `parseArgs` now stops flag-parsing at the first positional (or literal `--`); `pnpm gha test/run.ts --build` previously intercepted `--build` as a gha flag instead of forwarding to `test/run.ts` - new `pnpm gha --clean` command prunes orphan `pullfrog-gha:*` images and `pullfrog-gha-node-modules-*` volumes whose hash doesn't match the current Dockerfile (each Dockerfile/entrypoint edit creates a fresh hash and orphans the prior pair, ~600MB + ~200MB each — without a cleaner they accumulate silently) - `--shell` without a TTY now fails fast with an actionable message before docker is invoked, instead of producing the confusing `the input device is not a TTY` from docker run wiki updated: documents `--clean`, the parseArgs passthrough rule, and a new "Reclaiming disk" section. * gha: fidelity, flexibility, and signal-safety improvements investigated local fidelity vs the real GHA ubuntu-24.04 runner and addressed the gaps that have actually bitten contributors or could. fidelity (image now matches GHA closer): - bake build-essential, wget, xz-utils, file alongside the existing toolset. gh, jq, git, python3, sudo, ssh, build-essential, wget, xz, file, unzip, curl all present. native module builds (node-gyp, any package missing arm64 prebuilts) now work; common agent shell calls don't hit ENOENT - `host.docker.internal:host-gateway` flag wires the host into the container's DNS on linux (macOS Docker Desktop bakes it in). lets scripts that hit a local dev server use `API_URL=http://host.docker. internal:3100` and work identically on both platforms - `--init` makes tini PID 1, fixing signal forwarding during the pre-exec warmup window (Ctrl-C was previously taking up to 10s to tear down because bash-as-PID-1 swallowed the signal) - pnpm version is correctly pinned via the workspace's `packageManager` field — corepack resolves it at install time; verified via the new `--doctor` command flexibility (new affordances): - `pnpm gha --doctor` runs an inside-the-container fidelity audit: os + arch + node/pnpm/python versions, version snapshots of every baked tool, env vars (CI, HOME, TMPDIR), uid/gid, and the host.docker.internal resolution. useful for "works in CI fails locally" or vice versa - `pnpm gha --build --no-cache` busts the docker layer cache when an apt mirror, base image, or external download has changed upstream - entrypoint's `pnpm install` warmup is now wrapped in a `flock` on a file in the shared node_modules volume — concurrent `pnpm gha` invocations (e.g. play in one terminal, runtest in another) serialize their install instead of racing docs: - new "Gaps (known)" section in wiki/docker.md explicitly calling out the things this system can't do yet, including the missing `uses: ./action` semantics gap that `.github/workflows/action-gha-e2e-adhoc.yml` currently fills via GHA only (designing a local `pnpm gha-action <fixture>` is on the roadmap), service containers, parallel-run sharing, and arch differences (arm64 vs amd64) * docs: audit + corrections after testing fronts self-audit pass for stale references and incomplete pointers: - wiki/browser.md: `Docker (node:24)` → `pnpm gha container (ubuntu:24.04)`. the substance was right (chrome not preinstalled) but the base image reference was stale. - wiki/docker.md: the "Permission errors" troubleshooting line claimed the node_modules volume is chowned on every run; now correctly says "owned by the host uid on first creation; warm runs skip the chown" to match the actual behavior after the initVolumeOwnership fix. - wiki/action-tests.md: `API_URL` env-var doc now mentions BOTH paths (`localhost:` from play:local, `host.docker.internal:` from inside the container). Proxy/router recipe now shows both invocations side-by-side instead of saying "must use play:local". - wiki/billing.md: same dual-recipe update for the loop-including-the- action proxy walkthrough. - gha.ts header: expanded the usage block to include --clean / --doctor / --no-cache / --shell-TTY, added the host.docker.internal note, and pointed at wiki/docker.md for design rationale. self-document check: a future agent landing on this code can answer "how do I run a fixture / debug in shell / add a tool / diagnose fidelity / reach a local dev server" purely from gha.ts header + wiki/docker.md without spelunking through the entrypoint or git history.	2026-05-16 03:12:25 +00:00
Colin McDonnell	8e1acfba99	fix(models): mark grok-fast and grok-code-fast as deprecated (#761)	2026-05-16 01:58:07 +00:00
Colin McDonnell	fa7ddcee4a	prompt: discerning review-feedback handling + elegance bar ... strengthen build-mode self-review and addressreviews step 4 to require verifying every reviewer finding, reject AI slop / over-defensive code, and frame the goal as a complete + minimal + elegant solution. mirror the elegance/no-slop bar in AGENTS.md.	2026-05-15 19:13:04 +00:00
Colin McDonnell	3add2cbc49	fix(action/tsconfig): noEmit + exclude dist to silence editor TS5055 ... action/tsconfig.json had "exclude": [] (overriding the default outDir exclusion) and unset noEmit, so tsserver pulled action/dist/**/*.d.ts into the program and flagged 92 TS5055 errors ("Cannot write file ... .d.ts because it would overwrite input file") any time dist/ existed. the CLI typecheck script passes --noEmit so it never tripped — only the editor was affected. emit is owned by tsconfig.exports.json, which extends this one and overrides noEmit: false, emitDeclarationOnly: true. so the main config is editor/typecheck only and should declare noEmit: true.	2026-05-14 17:11:03 +00:00
Colin McDonnell	5abb3072c7	release: action v0.1.8 v0.1.8	2026-05-14 05:37:33 +00:00
Colin McDonnell	74b7329f64	fix(action): dedupe concurrent checkout_pr + guard cross-PR clobber (#735) ... * fix(action): dedupe concurrent checkout_pr calls + guard cross-PR clobber (#642) agents occasionally emit duplicate parallel `checkout_pr` tool_use blocks in one turn, causing two `checkoutPrBranch` invocations to race the same `.git/shallow.lock` and one to fail with `File exists`. the prior fix (#564) added a 30s staleness sweep, but that very threshold protects the within-run concurrent case from itself. dedupe at the tool layer: a module-level `Map<pull_number, Promise>` shares a single in-flight promise across concurrent same-PR calls. the fetch race becomes architecturally impossible — first call does the work, duplicate gets the same `CheckoutPrResult`. cleared in `finally` so subsequent same-PR calls re-do the work normally. also reject cross-PR checkouts when the working tree is dirty, surfacing a clear error instead of silently overwriting uncommitted work from a prior PR. uses existing `toolState.issueNumber` (no new state). * review: use dedicated `pullNumber` toolState field for cross-PR guard per copilot review: the prior guard used `toolState.issueNumber`, which is also set by issue/comment lookup tools (issueInfo, issueComments, issueEvents, review). that conflation is intentional and correct for its only consumer (`report_progress` falls back to `issueNumber` to choose which issue/PR to comment on, and GitHub treats both via the same comment API). but it makes the field wrong for the cross-PR guard: a same-PR re-checkout after `get_issue(other)` would falsely fire and surface a misleading "from PR #other" message. introduce a separate `pullNumber` field, set only by `checkoutPrBranch` alongside `issueNumber` and `checkoutSha`. narrower invariant, no disturbance to the existing `issueNumber` semantics. * review: drop dual-write — single `issueNumber` is sufficient for the guard reverting the `pullNumber` addition. setting both `issueNumber` and `pullNumber` to the same value at the same site was a code smell — there is no scenario where they diverge. issues and PRs share GitHub's number space, and the cross-PR guard's actual job is "refuse to clobber a dirty tree when switching to a different number"; that's expressible with `issueNumber` alone. addresses copilot's original concern (misleading "from PR #X" message when X was an issue) by removing the prior-number reference from the error message entirely. the dirty paths are the actionable detail.	2026-05-14 05:08:11 +00:00
Colin McDonnell	ba7f5a0b89	action: surface agent hang context in progress comment (#733) ... * action: surface agent hang context in progress comment When the activity-timeout watchdog kills a stalled opencode subprocess, the user used to see a bare "activity timeout: no output for 30Xs" — no provider context, no stderr trace, no clue why the run died. Investigation of the six runs in #728 showed the same shape every time: opencode hangs after a non-retryable provider event (auth 401, 502 stream lost, free-tier flake), and the only useful signal was buried in stderr where the user couldn't see it without diving into Actions logs. Stop trying to prevent the hang. Surface it. Add a small `AgentDiagnostic` handle on `toolState` that the harness mutates as a run progresses (recent stderr ring buffer reference, last provider-error label, event count). `formatAgentHangBody` renders that into a markdown body — bold headline, one-line explanation, collapsible `<details>` with the last ~10 stderr lines (capped to 3KB) — used by both the agent harness's own catch path and main.ts's outer catch when the watchdog wins the race against the harness. Both paths converge on one formatter; the existing "View workflow run ➔" footer affordance in `reportErrorToComment` is unchanged, so the user still has one click from the comment to the raw logs to develop their own thesis. * address review: gate hang body on isHang; fix contradictory copy - Only render `hangBody` when `isHang`. The harness sets `agentDiagnostic` on entry, so any non-hang throw past `runOpenCode`'s own catch (post-success `output_schema` validator, late cleanup throws) was rendering "Pullfrog failed — N events processed…" with the real exception message dropped — including for runs that actually succeeded before a late throw. - When `lastProviderError` already names the cause in the headline, the zero-events sentence "check whether the model provider is reachable" contradicts it (a 401 produces zero events but isn't a reachability issue). Drop the nudge in that case; keep it for the silent-stall path where it's still actionable. * address copilot review: fence escape, idle parsing, secret redaction, tests - pick a backtick fence longer than any backtick run in the rendered stderr tail. opencode error JSON occasionally embeds triple backticks in tool input dumps; the fixed three-tick fence let those terminate the fence early and corrupt the rest of the comment markdown. - parse idle seconds out of the timer reject string ("activity timeout: no output for 301s") and use that for the hang explanation. previously rendered total runtime, which overstated the stall by 20+ minutes for runs that streamed for a long time before going quiet (e.g. Rohithgilla12/data-peek#25784038918, 1230s elapsed but 304s idle). - redact sensitive env-var values from the rendered stderr tail before it lands in the PR comment / job summary. workflow log writes already go through `core.setSecret` masking; PR comments and summaries bypass that pipeline entirely. matches against `isSensitiveEnvName` (the same *_KEY/*_TOKEN/*_SECRET/*_PASSWORD/*_CREDENTIAL surface that `normalizeEnv` registers with the runner) and only redacts values >= 8 chars to avoid false-positive substring hits. - add `agentHangReport.test.ts` covering the branchy bits: idle-seconds parsing, eventCount-zero copy with and without provider error, fence-escape against embedded triple backticks, 3 KB tail truncation, null-on-no-diagnostic, and secret redaction. `startedAtMs` is dropped from `AgentDiagnostic` — total runtime was the only consumer and idle seconds replaces it. * strip slop: drop tests, drop redactSecrets, simplify ternary - delete `agentHangReport.test.ts`. half the cases just pinned literal copy ("**Pullfrog stalled**", "check whether the model provider is reachable") which is exactly the "performative tests to every string utility" pattern AGENTS.md flags. the other half tested 2-5 line pure helpers (parseIdleSec / pickFence / truncation) that code review catches. the formatter is a best-effort string output; pinning it in tests creates churn without catching real regressions. - remove `redactSecrets` and revert the formatter's import. theatrical defense: opencode doesn't dump env on startup, bearer tokens aren't in request bodies, bash is denied. the action has many other PR-comment write paths that don't redact (comment.ts, errorReport.ts, the progress writer) — if PR-comment secret hygiene matters, it's a cross-cutting concern at the comment-write layer, not bolted onto one formatter. - factor the explanation triple-ternary into `formatExplanation` with early returns. same logic, easier to read. `isHang` gate, fence-length escaping, and idle-seconds parsing stay — those are real correctness fixes.	2026-05-14 04:13:26 +00:00
Colin McDonnell	b9383bbcfd	action: center provider-error log excerpt on the matched line (closes #703) ... the `» provider error detected (...)` excerpt was `chunk.substring(0, 500)` — the head of whatever stderr buffer node delivered. on big writes that's the front of an mcp tool-schema dump, not the matched error text. label was correct (regex.test on the whole chunk), excerpt was misleading. introduce findProviderErrorMatch(text) that returns { label, excerpt } where excerpt is a windowed slice centered on the regex match index: the matched line plus 1 line before and 2 lines after, hard-capped at 600 bytes. detectProviderError stays as a thin wrapper for label-only callers. both opencode and claude harnesses log match.excerpt instead of chunk.substring(0, 500). regression tests cover the multi-line buffer case, surrounding-line context, byte-cap fallback to matched-line-only, and head truncation of a single oversize line.	2026-05-14 03:59:45 +00:00
Colin McDonnell	8d6460da1c	fix: surface real tool error string in opencode log handler (#736) ... opencode's `ToolStateError` carries the failure reason on `state.error`, not `state.output`. our log handler was reading `state.output` and falling back to `(no error message)`, so every tool failure logged a useless line. type the state as a discriminated union (mirrors @opencode-ai/sdk) so the field misread becomes a compile error. operator-facing only: the model already received the real error via opencode's tool-result envelope (verified by running webfetch against a known-404 URL — model reported "Error: Request failed with status code: 404" verbatim). closes #662	2026-05-14 03:56:24 +00:00
Colin McDonnell	1f4c3031be	ci: filter test matrices by per-test coverage globs (#730) ... * ci: filter test matrices by per-test coverage globs to cut LLM spend every test in `crossagent/`, `agnostic/`, and every provider entry now declares a `coverage: string[]` of repo-relative globs. the new `changes` job runs `paths-filter` for a docs-only short-circuit, then pipes the changed-file list into `action/test/matrix.ts`, which intersects each entry's coverage against the diff and emits filtered `agents`, `agnostic`, `flagships`, and `aliases` matrices. main pushes and `workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety net. retires `changed-agents.sh` and the `MODE=flagships` branch in `list-aliases.ts` in favor of one consistent model. * ci(matrix): switch test discovery to dep-free static parsing the GHA `changes` job has no `node_modules` installed. the previous dynamic-import path pulled the test files transitively through `utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with ERR_MODULE_NOT_FOUND. parse the test files via regex instead so matrix.ts stays zero-dep — the chain (matrix -> coverage / providers / list-aliases / models) imports only node builtins and relative TS files. * ci(matrix): address PR #730 review feedback - drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`, `restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`, both already covered). - drop unused `coverageForProvider` export and its `byName` map from `providers.ts` (matrix.ts builds its own lookup inline). - derive the active agent list from `agents/index.ts` via the same dep-free regex tactic as `parseTestFile` instead of hardcoding `["claude", "opencode"]` — adding a new harness file now wires it into the dynamic matrix automatically. - treat `coverage: []` as `coverage: undefined` in `shouldRun` so an accidentally-empty array doesn't silently skip CI on every PR. - add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the `timeout` test's coverage — the activity-timeout enforcement path was the original reason the test exists. - ungate the `root` job (lint/format/typecheck/vitest). it's a required status check on `main`, so gating it on `code == 'true'` would make docs-only PRs unmergeable (skipped jobs don't satisfy required-check rules). the real LLM savings come from skipping the four matrices, not from skipping `root`. - harden the four matrix-job `if:` guards from `outputs.matrix && ...` to `outputs.matrix != '' && ...` — explicit > implicit short-circuit. - document `expandBraces`'s flat-only support so a future author isn't surprised by `{a,{b,c}}` not expanding. - fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".	2026-05-14 03:55:33 +00:00
Colin McDonnell	4ad649ebb9	action: extend shallow-unreachable deepen-retry to checkout_pr fetches (#734) ... extracts the deepen-retry helper from `GitFetchTool` into shared `$gitFetchWithDeepen` and applies it to every fetch in `checkoutPrBranch` (baseRef, pull/N/head, before_sha temp branch). on shallow clones with deep PR ancestry — the failure mode behind ~10 of 51 `heuristic:very-slow` runs in 24h on `remotion-dev/remotion` — the baseRef fetch was throwing `Could not read <sha>` to the agent before the compare-api deepen block could run. agents then burned 10+ minutes retrying `checkout_pr` and falling back to ad-hoc shell `git fetch --deepen` workarounds. also splits the analyzer's `heuristic:git-error-recovered` into `heuristic:git-shallow-unreachable` and `heuristic:git-shallow-lock` buckets so future audits surface this without manual log-grep. closes #656.	2026-05-14 03:44:08 +00:00
Colin McDonnell	2960d51493	shell tool: cap output at 5K chars and spill overflow to tempfile (#732) ... unbounded shell tool output blows the agent's context window on commands that dump big logs (test runners, build tools, grep on large trees). cap the inline body at 5000 chars; on overflow, persist the full output to ${PULLFROG_TEMP_DIR}/shell-<id>.log and return the tail prefixed with a sentinel pointing at the saved path. agents re-read the tempfile with cat/tail/grep when they need more.	2026-05-14 03:18:54 +00:00
Colin McDonnell	b6df2860c3	action: bump to 0.1.7 v0.1.7	2026-05-14 02:48:31 +00:00
Colin McDonnell	d495f0b984	surface BYOK failures + chronic-failures card + WorkflowRunStatus mirrors GitHub conclusions (#722) ... - Migrates `WorkflowRunStatus` from `running | completed | cancelled` to a 9-state mirror of `workflow_run.conclusion`. Backfill: old `completed → success`, `cancelled → failure`. New rows write `hook.workflow_run.conclusion` verbatim via `statusFromConclusion`. - Adds Discord links to `formatApiKeyErrorSummary` (both missing-key and 401 invalid-key shapes). - Repo console: `<ChronicFailuresCard>` fires when the last 3 terminal-state runs are all `failure`. Pure DB read; latest-run button hidden for pre-dispatch failures (`runId: null`). - `StatusIcon` distinguishes `cancelled` (gray X, intentional stop) from `failure` (red X) so the visual matches the chronic-card threshold. - Pre-dispatch failures (workflow lookup miss, dispatch API error) write `failure` instead of `cancelled` so they feed the card. - Cascade: every `status: "completed"` filter in billing routes / cron / cohort queries / analyzer becomes `status: "success"`. Verified end-to-end on `pullfrog/preview-722-failure-surfaces` — Better Stack logs confirm webhooks reached the preview deploy and all three e2e runs got `marked as failure (conclusion=failure)` via the new mapper. Closes #679, #702.	2026-05-14 02:39:41 +00:00
Colin McDonnell	206c11fe7c	review: drop misleading 'with the same arguments' from diff-coverage nudge ... agent is free to refine review body/comments on retry — there's no enforcement that the second call matches the first, and if reading the nudged region surfaces a new finding the agent should add it.	2026-05-14 02:36:06 +00:00
Colin McDonnell	7414c1e9ca	review: clarify diff-coverage nudge gives explicit license to skip generated artifacts ... the one-time pre-flight nudge said "optionally read" but never told the agent it's free to retry without reading when every unread region is generated (lockfiles, codegen, snapshots, migration metadata). audit #677 surfaced ~21 runs/24h burning an extra model turn re-reading drizzle snapshots, pnpm-lock, and *.gen.ts files purely to satisfy the gate. mode prompts only mention generated content in the "skip self-review entirely" path, not the "in-progress substantive review" path, so the in-the-moment error message was the gap. behavior unchanged for legitimately-unread source regions.	2026-05-14 02:30:37 +00:00