37d15a338d67df6b29e7810962c9cd80401f263d
1009 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wolfy
|
37d15a338d | chore: more improvement and reproducibility | ||
|
wolfy
|
fc11b91851 | chore: continue improving review feedback | ||
|
wolfy
|
4a1743126e | fix: diagnostic issues | ||
|
wolfy
|
57e6529f97 | fix: comment anchor, remove emojis | ||
|
wolfy
|
1f4f84ec40 | chore: some retry logic | ||
|
wolfy
|
5ea8a23d80 | fix: review should leave comments on actual files | ||
|
wolfy
|
1e839d36a9 | fix: review process and cleanup | ||
|
wolfy
|
41dbd09cc0 | fix: disable think and keep alive until manual unload | ||
|
wolfy
|
f88377cd1d | chore: bump context window to match what zed uses | ||
|
wolfy
|
fa2516e53e | chore: use thinking mode | ||
|
wolfy
|
0dc0f7eb53 | feat: add read_file tool | ||
|
wolfy
|
3f0d9a80c7 | fix: issue with shell calls | ||
|
wolfy
|
fe85adfa53 | fix: reviews failing to call next tool | ||
|
wolfy
|
0cf9df2bb6 | chore: revert back to initial instructions for the most part | ||
|
wolfy
|
f7d59cad03 | fix: issue properly basing diffs when tagged on pr | ||
|
wolfy
|
93471c9408 | feat: handle "suggestions" better | ||
|
wolfy
|
19671c6299 | feat: branch protection + deps caching | ||
|
wolfy
|
d5d2e5b58e | fix: get diffs properly | ||
|
wolfy
|
0438688e32 | fix: issues with pagination not resolving correct url templates | ||
|
wolfy
|
2aca1a3aa3 | feat: adapt pullfrog for gitea + ollama | ||
 Colin McDonnell
|
36ac64a5b6 |
fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy (#844)
* fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy OSS-allowlisted repos with `CODEX_AUTH_JSON` uploaded via `pullfrog auth codex` were still being routed through the OSS OpenRouter subsidy because two paths ignored managed credentials: - `hasProviderKey()` only checked `provider.envVars`, so an `openai/*` model with only `CODEX_AUTH_JSON` present silently fell back to `opencode/big-pickle` via `selectFallbackModelIfNeeded` — the maintainer saw "opencode/big-pickle (resolved from openai/gpt)" on CI even though Codex was configured. - `run-context` set `proxyModel` for every OSS run unconditionally, which the action runtime threads through `payload.proxyModel` and uses to overwrite `OPENROUTER_API_KEY`. Even if `big-pickle` fallback hadn't fired, the runner would consume the $10 OSS subsidy key instead of the user's ChatGPT subscription. Fix: - Add `getModelAuthEnvVars()` covering both `envVars` and `managedCredentials` in `action/utils/apiKeys.ts`; route `hasProviderKey` + `validateAgentApiKey` through it. - `run-context` now skips `proxyModel` for OSS repos when the configured model's provider has matching auth in Pullfrog-stored account/repo secrets, so the runner authenticates directly with the user's Codex subscription (or any other user-provided provider auth). Triggered by mrlubos (`hey-api/openapi-ts`). Companion follow-up tracked for the opaque "(no error message)" classifier swallow that masked the OSS $10 cap exhaustion on PR #3872 runs 25815370844 + 25815443234. * fix(oss): force Kimi K2 for OSS proxy + hide picker in console UI OSS-funded runs were resolving `repo.model` through OpenRouter, so a single Opus / GPT-5.5 run could burn an entire `oss_subsidy` key against the per-key cap and crash mid-stream (e.g. `hey-api/openapi-ts` PR #3872 runs `25815370844` + `25815443234`, ~$9.20 each on a single key). Force `DEFAULT_PROXY_MODEL` (Kimi K2.6 — ~10-50× cheaper) for every OSS proxy mint, regardless of `repo.model`. Per-run spend stays bounded within the cap by structure, not by hope. `repo.model` stays in the DB unchanged — overriding at runtime means leaving the program restores the user's prior pick without a migration. UI: hide the model picker entirely on OSS repos in `AgentSettings`. The field is effectively inert until the repo leaves the program, so exposing it as if it were live was misleading. Replaced with a banner naming Kimi K2 and pointing to `pullfrog auth …` as the opt-out path — that lands the user on the existing #844 bug-2 branch (Pullfrog-stored auth suppresses the OSS proxy entirely; runner uses user credentials + their preferred model). ModelCostsInfo already has its own `isOss` branch for the cost copy, so that section is unchanged. * fix(oss): lowercase comment casing per AGENTS.md * fix(oss): revert banner copy to 'It's on us.' framing per review Maintainer felt 'Kimi K2' as the banner headline lost the warm 'we've got you covered' framing that the existing OSS cost banner uses. Restore 'It's on us.' as the headline, move the model name into the body where it explains the hardcoded choice and points to the opt-out (pullfrog auth codex / account secret). * docs(agents): screenshots must be of the live route, never synthetic Caught myself building a temp `/dev/oss-ui-preview` route with hardcoded JSX copy-pasted from the real component just to grab a screenshot — the result told us nothing about whether the actual integrated UI worked, and the user (rightly) called it out as a waste. Strengthen the rule: screenshots must come from the live route in the running app, driven by the actual component tree and real props. Note the GH OAuth interstitial gotcha so the next agent gets through Clerk → GitHub sign-in on the first try instead of bailing to a fake render. Also bans side-by-side comparison screenshots unless explicitly requested. * fix(oss): one 'It's on us.' banner, not two OSS Agent settings was showing the message twice — once in the Model section, once in the Model costs section right below it. Fold the cost coverage into the model banner ('at no cost to you' + the spend stat) and hide the Model costs subsection entirely for OSS. ModelCostsInfo no longer needs `isOss` / `ossSpendThisMonthUsd` props — call site is gated, so the OSS branch is dead. Removed it and the now-unused props. Non-OSS rendering is unchanged: full Model picker + Model costs subsection with Router / BYOK branches. * feat(action): corepack-aware package manager provisioning before setup customer setup scripts that did `npm i -g pnpm && pnpm install` were installing whatever pnpm "latest" happens to be on the day the run fires, not what the repo declares — and pnpm 11.3 silently writes a new `packageManagerDependencies` block into lockfiles, which the agent's "always push changes" rule then packages into a noisy PR (see #844). resolve the project's pnpm/yarn pin from `package.json` (honoring pnpm 11+ precedence: `devEngines.packageManager` over `packageManager`) and activate it via `corepack prepare ... --activate` BEFORE the setup hook runs. corepack is bundled with node, so this is a no-op on managed infra; failure (no corepack, no network, range-only version) degrades to a warning and the existing PATH binary still runs. also replaces the legacy `npm install -g <pm>@<v>` path in prep with the same helper so behavior is consistent end-to-end. bun/deno still use the legacy installer because corepack doesn't ship shims for them. * chore(console): drop 'npm i -g pnpm' anti-pattern from setup-script placeholder the suggested example trained customers to install pnpm unpinned, which silently picks up whatever's latest at run time. that's exactly the behavior #844 traced lockfile drift back to. now that prep handles package-manager provisioning via corepack from the repo's declared pin, the placeholder is just a frozen-lockfile install — load-bearing only when the repo wants `pnpm install` to actually run (prep already does that), but a much safer default for customers who do paste it in. * refactor(action): introspect opencode models for BYOK detection Replace the static `provider.envVars + provider.managedCredentials` catalog gate in `selectFallbackModelIfNeeded` + `validateAgentApiKey` with two `opencode models` captures around the auth merge: - `captureBaselineModels` BEFORE dbSecrets + Codex auth.json - `captureAuthorizedModels` AFTER both The authorized set is the authoritative source for "can OpenCode route this model" — strictly more accurate than the catalog, which can miss new auth shapes (Codex was one, there will be more). The diff between baseline and authorized is logged as `BYOK auth enabled N model(s)` for operator visibility. Sequencing changes in main.ts: - `createTempDirectory` hoisted out of the try block so `PULLFROG_TEMP_DIR` is set before the early opencode install - `agents.opencode.install()` + baseline capture before dbSecrets - `installCodexAuth()` hoisted up (idempotent — agent re-calls it inside run() and writes the same file) - authorized capture after Codex auth.json materializes - fallback + validateAgentApiKey receive the authorized set as a parameter; tests inject directly with no mocks Deleted: `hasProviderKey`, `getModelAuthEnvVars`, `knownApiKeys` in `action/utils/apiKeys.ts` (only `selectFallbackModelIfNeeded` consumed them, and PR #844's catalog-extension fix is superseded by introspection). `getModelEnvVars` / `getModelManagedCredentials` stay exported for UI and the server-side OSS proxy heuristic in run-context/route.ts. For the claude agent path, validateAgentApiKey keeps the static single-provider check on `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` — `opencode models` is opencode-specific. validateBedrockSetup / validateVertexSetup also stay; they cover region/location/model-id which `opencode models` doesn't catch. When fallback engages, the post-fallback model is the guaranteed-free `opencode/big-pickle`, so validateAgentApiKey is skipped — the fallback gate already authoritatively decided "this model is OK to run". * test(oss): temp add preview-844 to ossRepos for O4 e2e * Revert "test(oss): temp add preview-844 to ossRepos for O4 e2e" This reverts commit 8167e560126b2ac516c32ba1c63c36aa32ae4019. * test(oss): temp add preview-844 to ossRepos for O5 e2e * fix(action): skip validateAgentApiKey when proxyModel is set The new opencode-models BYOK introspection in PR #844 captures the authorized set BEFORE runProxyResolution mints OPENROUTER_API_KEY, so the proxy slug (e.g. `openrouter/moonshotai/kimi-k2.6`) is never in the set. validateAgentApiKey then spuriously threw "no API key found" on every OSS run, even though the proxy key was minted correctly and the inference would have worked. Mirrors the analogous skip in `selectFallbackModelIfNeeded`: when proxyModel is set, the server-side gate (`run-context/route.ts`) is the authority and the proxy mint itself is the validation. Caught by O5 e2e on `pullfrog/preview-844-heyapi-oss-bug`. * Revert "test(oss): temp add preview-844 to ossRepos for O5 e2e" This reverts commit 3bae075ceeb188ee272c45c13b5080e15bcd00a5. * fix(action): discard hook-generated tracked-file drift before agent sees it addresses bug 3 in #844: customer setup/post-checkout hooks like `pnpm install` or `corepack prepare` left the working tree dirty (e.g. `M pnpm-lock.yaml`), the agent took the prompt's "must push" rule literally, opened a spurious bot PR for the lockfile drift, and we ate runs+spend on noise. after each setup / post-checkout hook (opt-in via `normalizeWorkingTreeAfter`), discard tracked-file mods with `git restore --staged --worktree .`. untracked files are preserved — a hook that materializes a `.env` from a template, or emits codegen output, stays visible to the agent. guarded by a pre-hook `git status --porcelain` snapshot: if the tree was already dirty before the hook ran (shouldn't happen — setup runs before any working-tree writes; checkout_pr refuses to run dirty), we warn and skip the discard rather than clobber whatever was there. prepush hook (action/mcp/git.ts) intentionally does NOT opt in — its job is to read the about-to-be-pushed state, not normalize it. * test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge) * fix(action): skip eager pnpm/npm/etc install when no lockfile exists second half of bug 3 in #844. the eager prep step assumed `pnpm install --frozen-lockfile` (and equivalents) would fail cleanly without a lockfile, leaving the tree untouched. that assumption is false for pnpm 11.1.1 against a no-deps `package.json`: the command reports "Already up to date" with exit 0 AND silently materializes an empty `pnpm-lock.yaml` despite the `--frozen-lockfile` flag. the resulting untracked file trips the post-run dirty-tree gate, the agent reads it as "must push uncommitted work", and a spurious "Add pnpm lockfile" PR lands. smoking gun: pullfrog/preview-844-heyapi-oss-bug PRs #1/#2/#3, all auto-opened by the bot against a repo that contains nothing but a one-line README + a no-deps package.json. guard explicitly with an `existsSync` per manager. if the lockfile is absent, skip eager prep entirely with an info log; the agent can install on demand via the `setup` lifecycle hook (which non-frozen `pnpm install` would handle correctly), or just leave deps uninstalled when the prompt doesn't need them (e.g. the O5 "tell me a joke" path). orthogonal to the lifecycle-hook normalization in 0051bd2a — together they cover the full bug 3 surface: - eager prep can't materialize a lockfile (this commit) - setup/postCheckout hooks that rewrite tracked files have the drift discarded before the agent sees it (prior commit) * fix(action): address Pullfrog review on hook normalization two fixes in `executeLifecycleHook` from review on f6f3b32: 1. pre-hook snapshot was `git status --porcelain` which counts untracked files; in practice any repo with pre-existing untracked content (e.g. `.plans/`, an ignored-but-not-yet-gitignored scratch dir, codegen artifacts) would trip the guard and silently skip normalization, defeating the fix. switch to `git diff --name-only HEAD` so the gate measures the same thing the discard targets — tracked-file mods only. pre-existing untracked files are safe regardless because `git restore --staged --worktree .` never touches them. 2. normalization fired only on the happy path; a hook that updated a lockfile then exploded on a peer-dep conflict left tracked drift for the agent. move the call into a `finally` so it runs on success, non-zero exit, timeout, AND spawn failure. the pre-hook guard still protects pre-existing work in every case. * Revert "test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge)" This reverts commit f6f3b325d6bf9a1720754ed1d39d248dab76cfa8. * fix(action): use detect lockfile strategy for eager-prep gate addresses Pullfrog review on be3c207b. two findings, one root cause: - the hardcoded LOCKFILE_BY_MANAGER map missed `bun.lockb` and `npm-shrinkwrap.json`, two managers' accepted lockfile variants. - `existsSync(join(cwd, lockfile))` only checked the immediate directory, breaking monorepo subpackages where the lockfile lives at the workspace root. both fall out by replacing the custom check with `detect({ strategies: ["lockfile"] })`. the detector already walks up the tree (subpackage → workspace root) and recognizes every accepted lockfile name across all managers it supports. restricting to the `lockfile` strategy is load- bearing: the default strategy set also matches on `packageManager` / `devEngines.packageManager` package.json fields, which would return non-null and re-mask the very case we're trying to detect (declared manager, no lockfile committed — the O5 / hey-api preview repro). drops the LOCKFILE_BY_MANAGER map entirely; no need for a second detect() call since the existing one was only used for `agent` resolution and that consumer is now after the lockfile gate, where `detected` is guaranteed non-null. |
||
|
Colin McDonnell
|
b49c1d9a57 |
postRun: forbid set_output in reflection prompt (gemini pro regression)
reflection turn is a meta-turn for editing the learnings file; the task's `result` output was already finalized on the previous turn. gemini pro re-triggers on the standing "call set_output when done" system instruction during reflection and clobbers the value with the literal word "done" (see ci run 26529624199, smoke test on providers-live google/gemini-pro). add an explicit prohibition to the reflection prompt; the snapshot/restore in runPostRunRetryLoop remains as defense in depth. |
||
|
Colin McDonnell
|
b0868d48e6 |
git tool: reject {command, args[0]} duplicates with a directed error
models occasionally call `pullfrog_git({command:"status", args:["status"]})`,
which shells out to `git status status`. git silently treats args[0] as a
pathspec — when no file/dir matches, status prints "nothing to commit,
working tree clean" even on a dirty tree. observed in production
(Skn0tt/beckerbuch run 26519563044): the agent looped trying to reconcile
that against a real diff, burned ~$3 / ~6min of opus, and only escaped when
it switched to `args: ["--porcelain"]`. generalises to every subcommand
(`diff diff`, `log log`, ...).
guard `args[0]?.toLowerCase() === command.toLowerCase()` with a directed
throw pointing the model at the disambiguated `args: ["--", "<name>"]`
escape hatch for the rare legitimate pathspec case (`--` works under every
subcommand, unlike a bare positional which can be parsed as a ref by
log/diff/checkout/restore/reset).
description also leads with the no-args case and explicitly forbids
repeating the subcommand in args. schema already had args.optional().
|
||
|
Colin McDonnell
|
c89b0c7b4a |
action/README: drop waitlist banner, point to GA console
also includes in-flight working-tree work: - postRun: snapshot/restore toolState.output across reflection turn so reflection prompt can't clobber task-turn output (gemini pro regression) - toolState: widen `output` to `string | undefined` for assignability - uninstallFeedback: suspend-mode emails now CTA the GitHub unsuspend page when accountType is known; delete events keep console pointer |
||
|
pullfrog[bot]
|
05d9343660 |
chore(models): bump resolved versions (#814)
* chore(models): bump resolved versions * chore(models): bump google + opencode gemini-flash to 3.5 --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> |
||
|
Colin McDonnell
|
3440292abb | release: bump action to 0.1.14 v0 v0.1.14 | ||
|
Colin McDonnell
|
585a5d21cc |
fix(askpass): scope code + script lifetime to one $git() call (#841)
* fix(askpass): scope code + script lifetime to one $git() call, not first password prompt LFS pre-push (and any auth-bound sibling subprocess) consumed the single-use code AND triggered the script self-delete, so git's own push call then hit `fatal: cannot exec '/tmp/pullfrog-…/askpass-…js'` and our server treated the legitimate retry as tamper, revoking the installation token. Observed on nteract/nteract#2987 (LFS repo). `gitAuthServer` codes are now `active` until `$git()`'s finally calls `revoke()`; the script no longer self-deletes (finally already unlinks). Replay after revoke still trips 409 + token revocation, which is the realistic exfiltration vector we care about. * fix(askpass): drop wall-clock TTL on active codes Copilot review on #841 noticed the 5-minute CODE_TTL_MS still applied to active codes, which would re-introduce the original LFS failure mode at a different boundary: a large LFS push lasting >5min would hit a 404 mid- call. $git() uses `activityTimeout: 0` precisely because git fetch/push can take arbitrarily long, so any wall-clock TTL on active codes is wrong. Active codes now live until revoke() is called (in $git()'s finally) or the auth server is closed. Revoked codes keep their 60s replay trap. * docs(askpass): purge stale single-use vocabulary; align error message Pullfrog review on #841 surfaced four doc-drift sites that still described the pre-PR single-use model: - wiki/security.md — 3 references (overview prose, bullets, threat-mitigation table) - action/utils/gitAuth.ts — file-level JSDoc + the 409 error message - wiki/askpass.md — error message quoted in the tamper-evident section - action/utils/gitAuthServer.ts — per-prompt invocation comment was ambiguous All updated to match the active|revoked vocabulary; error message is now "askpass code was replayed after revoke, token revoked". |
||
|
David Blass
|
fe2746198c |
fix: 6 unaddressed log-audit / run-audit findings (#840)
* fix: 6 unaddressed log-audit / run-audit findings - #836 + #818 (clerk middleware SyntaxError on action-runtime endpoints): narrow proxy.ts matcher to exclude /api/repo/<owner>/<repo>/run-context, /api/runtime/*, /api/proxy-token. these are server-to-server with their own auth and have no Clerk session to evaluate, so clerkMiddleware's decodeJwt throws turn into 500s on every request. - #837 (npx EBADDEVENGINES on customer's package.json): change runCli's bootstrap cwd from $GITHUB_WORKSPACE to os.tmpdir() so npm v11+ doesn't enforce devEngines.packageManager from the customer's tree before our bootstrap can install pullfrog. CLI process.chdir's to payload.cwd internally, so the runtime work still happens in $GITHUB_WORKSPACE. - #838 (createWorkflowDispatch silently dropped 39 user runs on a 5xx spike): add bounded retry (3 attempts, ~750ms total) on Octokit 5xx and network errors inside dispatchReservedRun. preserves the existing 422 "Unexpected inputs" path. retry budget stays well under GitHub's 10s webhook redelivery window. - #833 (bail() redirect("/signout") propagated NEXT_REDIRECT into webhook handlers, 40+ 500s/24h): drop the redirect side-effect; bail now just classifies bad-credentials as non-retryable and propagates. UI flows that wanted auto-signout on revoked tokens can detect it themselves; the side-effect was wrong for any non-page caller. - #835 (BYOK provider billing-exhausted fell through to raw error renderer, 37 review-mode runs/24h with no PR-side signal): - extend providerErrors patterns with Anthropic "credit balance is too low" + extract isProviderBillingExhausted / extractProviderId helpers - add a renderer branch in runErrorRenderer.ts that names the provider (parsed from providerID=) and links to its billing dashboard - route handleAgentResult's !result.success path through the renderer + reportErrorToComment with createIfMissing, so review-mode and silent triggers get an actionable PR comment regardless of mode - #834 (post-hook ERR_MODULE_NOT_FOUND already fixed on main, hardening): - add a vitest invariant that walks the entryPost.ts import graph and refuses any non-relative / non-node: specifier — catches the next `@actions/core` slip-up before publish - add an analyze-logs classifier so future entryPost crashes surface as failure:post-hook-module-not-found instead of hiding inside failure:unknown / failure:git-lock-file Co-authored-by: Cursor <cursoragent@cursor.com> * anneal: fix dead-code matcher, 404 url, silent-trigger gate, and grammar regression Round-1 anneal pass on the audit-fixes PR surfaced two critical issues + several majors that the original fixes shipped with: - proxy.ts matcher 1 (`(?!_next|[^?]*\.(...))`) still caught every /api/... route because matcher arrays are OR'd. The narrowing in matcher 2 was dead code → middleware still 500'd on /api/runtime/*, /api/proxy-token, /api/repo/.../run-context. Carve-out now lives in BOTH matchers. - opencode.ai/billing returns 404; canonical top-up surface is /zen. deepseek /usage is the consumption page, not the top-up flow (/top_up is correct). google /apikey is the keys list, not billing (/usage is the spend dashboard). All three URL strings updated. - handleAgentResult gated reportErrorToComment behind `if (!ctx.silent)`, contradicting the createIfMissing intent — silent IncrementalReview / pull_request_synchronize / auto-label still got zero PR signal on BYOK billing exhaustion, the exact failure mode #835 was meant to fix. Moved createIfMissing into finalizeSuccessRun's existing render-and- post block (single source of truth), reverted handleAgentResult to its prior shape. Side benefit: drops the double-PATCH that fired on every non-silent !success path with an existing progress comment. - Anthropic-direct error rendered "**Your your provider account is out of credit.**" because Anthropic SDK has no providerID= tag, so extractProviderId returned null and the headline composed "Your " + "your provider". Added detectProviderId Anthropic fallback (matches "Anthropic API" / "credit balance is too low") so the link is reachable AND made the headline conditional on whether a provider id was detected. - Reordered renderRunError classifier: BYOK billing-exhausted now runs BEFORE api-key auth detection. Providers commonly return 401 for billing exhaustion (DeepSeek, Gemini), and the OpenCode harness logs often include "API Error: 401" in the raw error body, which isApiKeyAuthError would otherwise match — surfacing "rotate your key" when the actual fix is "top up credits". - isTransientUpstreamError missed ENOTFOUND / ENETUNREACH / EHOSTUNREACH (undici DNS-class failures Octokit doesn't wrap with a status). Added to the prefix alternation. - Tightened "10s webhook redelivery budget" / "GitHub redelivers" wording in triggerWorkflow.ts and bail.ts JSDoc — GitHub's 10s is the response timeout (it doesn't auto-redeliver); upstream webhook proxy retries are what multiplied the failure. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal r2: unbreak proxy.ts matcher 2, extend carve-out, mkdtemp bootstrap Round-2 anneal (security + cross-cutting lenses) on top of the round-1 audit-fixes commit caught a critical regression + several majors: - proxy.ts matcher 2 from r1 (`/(api|trpc)(?!...)(.*)`) does NOT compile. path-to-regexp rejects a top-level `(?!` after `)` as "Pattern cannot start with '?'", and Next.js's SourceSchema runs the same validator at build time and aborts via `process.exit(1)`. PR #840's Vercel + preview deployments have been failing since 978eca26 for exactly this reason. Fixed by nesting the lookahead inside an outer parameter group: `/(api|trpc)((?!...).*)`. Same shape matcher 1 already uses, which is why m1 always compiled. Verified `pnpm next build` succeeds end-to-end. - proxy.ts carve-out was incomplete relative to its own justification. The "no Clerk session, decodeJwt 500" failure mode applies to ALL server-to-server action-runtime endpoints — five more share the exact shape: /api/repo/.../learnings, /api/repo/.../pr/.../summary-comment, /api/repo/.../issue/.../plan-comment, /api/workflow-run/, and /api/github/installation-token. Extended both matchers. /api/upload/ signed-url stays in (dual auth: Clerk session OR bearer JWT — needs middleware for the user path). - proxy.ts carve-outs were unanchored: a future /api/proxy-token-info, /api/proxy-tokens, or /api/repo/X/Y/run-context-foo would silently bypass Clerk. Added `(?:$|/)` for path-prefix carve-outs (allow exact match or sub-path), `$` for routes with browser-callable siblings (e.g. `learnings/history` is browser-Clerk, `learnings$` is action- bearer-JWT). Verified 30/31 routes via path-to-regexp test harness. - runCli.ts cwd flipped from $GITHUB_WORKSPACE to os.tmpdir() in r1 (#837 fix for npm v11 devEngines.packageManager EBADDEVENGINES). But $TMPDIR is overridable from a prior $GITHUB_ENV step — a customer- authored or compromised prior step can plant /atk/node_modules/ pullfrog/ and `echo "TMPDIR=/atk" >> $GITHUB_ENV`, and our npx --yes pullfrog@<v> bootstrap resolves the local install first, executing attacker code with full action env (provider keys, OIDC, installation token, CODEX_AUTH_JSON). Switched to mkdtempSync(join (tmpdir(), "pullfrog-bootstrap-")) — fresh per-invocation 0700 dir, not pre-writable by anything earlier in the job. - runLifecycle.ts: writeRunErrorOutputs (catch-path) didn't pass createIfMissing: true, contradicting the symmetric intent of the r1 finalizeSuccessRun fix. Silent triggers (IncrementalReview / pull_request_synchronize / auto-label) that throw past the success path still got zero PR signal — exact failure mode #835 was meant to close. Now both paths pass createIfMissing: true. - runErrorRenderer.ts detectProviderId regex `/Anthropic API|credit balance is too low/i` could mis-tag a non-Anthropic billing-exhausted error that mentioned "Anthropic API" in passing (fallback-chain agent prompt text, OpenCode harness logs). Tightened to /credit balance is too low/i — Anthropic-specific phrasing, sufficient for the direct- Anthropic SDK case the fallback exists to handle. - runErrorRenderer.ts JSDoc: r1's classifier reorder put hang at #6 in code but the JSDoc still listed it at #2. Reordered the doc to match dispatch order, with explicit note that hang is a sub-source for the api-key check (which is why hangBody is precomputed early). - analyze-logs.ts: ERR_MODULE_NOT_FOUND.*entryPost regex needs `s` flag so it survives Node v23+ stack-trace reformatting onto multiple lines. One-char fix to defend the #834 classification bucket. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Colin McDonnell
|
dc4dff98da |
allowlist middleapi/orpc for OSS, ship skill scaffold + no-mock audit
- utils/ossRepos.ts: add middleapi/orpc (orpc.dev, 5.2k stars) to the oss allowlist so any future install mints via mintOssKey ($10 cap, pullfrog absorbs). - scripts/skill.ts + pnpm skill: scaffold .agents/skills/<name>/SKILL.md + .claude/skills/<name> symlink. patch + skill skills written using it. - AGENTS.md: hard-ban vi.* mocking apis; document pnpm skill workflow. - audit follow-through: drop pure-mock test files (action/utils/lifecycle, action/utils/timer, test/handleNoInstall) and trim action/mcp/review to the non-mock cases. - wiki/scripts.md: row for scripts/skill.ts. |
||
|
Colin McDonnell
|
a0746dcc27 | release: bump action to 0.1.13 v0.1.13 | ||
|
Colin McDonnell
|
ed8ee363c0 | release: bump action to 0.1.12 v0.1.12 | ||
|
Colin McDonnell
|
f327f65413 |
revert(test): keep opencode default on sonnet, not kimi
c45a07e5 swapped the opencode harness default to moonshotai/kimi-k2, which broke push-enabled (Kimi reported delete_branch as "auth failed" where sonnet handles it). The agnostic test matrix expects Sonnet-grade tool-calling; switching to Kimi was scope creep on top of the original "don't default to opus" ask — opencode's default was already sonnet, not opus, so nothing needed to change there. |
||
|
Colin McDonnell
|
d93ddcbf4a |
expose push: restricted to users (#828)
* expose push: restricted to users the action runtime already understood `disabled | restricted | enabled` for `push` (see action/external.ts, action/mcp/git.ts); the DB enum, the zod schema, the React settings UI, and the action.yml input docs lagged behind. catches all four up. UI surfaces a 2-state toggle (restricted ↔ enabled) in the Security card alongside shell isolation — `disabled` stays in the enum so non-UI callers (action.yml input) still accept it but is intentionally hidden from the console. * address reviews: doc full restricted scope + disabled UI edge case reviewers flagged that "restricted" blocks more than default-branch pushes (also delete_branch + push_tags per action/mcp/git.ts:621-700), and that the popover's "off = full push access" claim is wrong for repos whose workflow sets `push: disabled` (also renders as OFF since the toggle is `props.push === "restricted"`). action.yml description now enumerates the full restricted scope, and the popover frames off behavior in terms of the workflow input so disabled repos aren't misrepresented. |
||
|
Colin McDonnell
|
e52206b8ca |
test: switch opus defaults to sonnet/kimi for cheaper coverage
- opencode test default: claude-sonnet-4-6 → moonshotai/kimi-k2. claude-code default stays on sonnet (the agent-under-test for that path is Claude Code); the opencode path doesn't need an Anthropic model. - byok-no-keys-fallback fixture: anthropic/claude-opus → moonshotai/kimi-k2. test verifies fallback to opencode/big-pickle, so the configured model is never invoked; picking a non-opus alias avoids burning credits if the fallback ever regresses. - BEDROCK_MODEL_ID in both test workflows: us.anthropic.claude-opus-4-6-v1 → us.anthropic.claude-sonnet-4-6. env var is required by the ci.test invariant that every provider's env vars are wired into the workflow. |
||
|
Colin McDonnell
|
fd2c67ab50 |
adhoc: push:restricted adversarial pentest (#827)
* adhoc: push:restricted adversarial pentest
enumerates the 16 attack vectors the deep audit identified as load-bearing
for `push: restricted`. used to drive e2e verification against the preview
repo's pullfrog.yml; also runnable via pnpm runtest locally.
validator only asserts that the repo's default branch SHA didn't move —
the per-attack outputs are the deliverable for human review (the test
exists to feed adversarial runs, not to be a CI guard).
* wipe runner leak surface before agent spawn
the GHA runner persists credentials inside $RUNNER_TEMP that an MCP-shell
agent can grep — _runner_file_commands/set_output_* (from any composite
step that called core.setOutput, e.g. pullfrog/pullfrog/get-installation-token
which leaks a ghs_… installation token), <uuid>.sh rendered step scripts
(whose run: | body embeds ${{ ... }} expressions literally before write),
and git-credentials-*.config from actions/checkout@v6.
snapshot-and-delete that surface at action startup, after our own token is
in memory and before setupGit. preserves $GITHUB_OUTPUT, $GITHUB_ENV, and
$GITHUB_STATE so pullfrog's result output and post: hook still work.
setupGit's existing removeIncludeIfEntries call strips the matching
dangling includeIf.gitdir:....path entries from the user's .git/config.
does not tighten isGitCommand — that's a UX guard, not a security
boundary, and trivially bypassable via bash -c, absolute paths, symlinks,
python subprocess. the security boundary is the absence of credentials on
disk for those bypassed shells to authenticate with.
verified end-to-end by re-firing action/test/adhoc/pushRestrictedAdversarial
against pullfrog/preview-827-push-restricted-pentest.
* preserve all runner file-command paths from wipe
addresses pullfrog review on f7f5143b: GITHUB_STEP_SUMMARY also lives at
$RUNNER_TEMP/_runner_file_commands/step_summary_<uuid> and is read by the
runner AFTER our step exits to render the job summary in the GH UI. wiping
it silently broke pullfrog's job summary output. preserve GITHUB_PATH too
for symmetry — it's the same allocation pattern, and a step or post hook
that appends a directory expects the file to exist.
set of file-command env vars enumerated in @actions/core:
GITHUB_ENV, GITHUB_OUTPUT, GITHUB_PATH, GITHUB_STATE, GITHUB_STEP_SUMMARY
|
||
|
Colin McDonnell
|
b6c57547ca |
fix(claude): use claude-code as skills CLI agent name
the skills CLI rejects "claude" — its valid list is claude-code, opencode, cursor, etc. caused agent-browser skill install to fail on every claude run. |
||
|
Colin McDonnell
|
e2eb26573f |
surface agent failures in job summary (#632) (#802)
* surface agent failures in job summary (#632) when the agent harness returns `{success: false, error}`, main.ts went through the success path so the catch block — which renders the `### ❌ Pullfrog failed` banner via renderRunError — never fired. result: the GitHub Actions job summary showed only the partial body + usage table, no error block. the progress comment had a narrow workaround that re-implemented the api-key classifier inline. unify: in finalizeSuccessRun, call renderRunError once when `!success`, use `.summary` for the job summary (prepended to the existing body/usage parts) and `.comment` for the progress comment. removes the duplicated isApiKeyAuthError / formatApiKeyErrorSummary branch and picks up BillingError reclassification + hang body for free. * docs: note dual-surface failure rendering in finalizeSuccessRun * fix copilot review nit: clarify which renderRunError classifications carry the H3 banner |
||
|
Colin McDonnell
|
01e4daa0b5 |
checkout_pr: refuse unconditionally on dirty working tree (#808)
* checkout_pr: refuse unconditionally on dirty working tree drop the live-HEAD comparison from the guard introduced in #796. any checkout_pr call with staged or unstaged changes now throws, even when HEAD is already on pr-N. no stashing, no idempotent escape hatch. motivation is the zed-industries/cloud (2026-05-18) incident: shared-cwd subagents make "carry edits along" semantics dangerous, and the HEAD-equality predicate let a re-checkout silently inherit working-tree state from a sibling agent. forcing commit/discard before any PR-context operation eliminates the entire carry-forward failure class. error names the PR number, lists dirty paths, and tells the agent to commit/push/restore/clean before retrying. * improve dirty-tree error: precise discard commands copilot caught two sloppy bits in the error string: - "push" alone does not clean a dirty tree (needs commit first) - bare `git clean` is a no-op without `-fd` reword to "commit (then push if needed), or discard with `git restore --staged --worktree .` / `git clean -fd`" so the guidance is actually actionable. * checkout_pr: initial-branch invariant setupGit captures `toolState.initialBranch` at run start via live `git rev-parse --abbrev-ref HEAD`. checkout_pr refuses unless current HEAD matches the run-entry branch or the target `pr-N` (idempotent same-PR re-checkout). uses live rev-parse, not toolState.issueNumber (poisonable per the PR #796 review). refusal error names the current branch, target PR, recovery path (`git checkout <initialBranch>` with the literal branch name), and explicitly states routing around via the `git` tool is not sanctioned. closes the zed-industries/cloud (2026-05-18) shape where a subagent parked HEAD on someone else's `pr-X` and the orchestrator's next checkout_pr inherited that position. * reviewfrog: enforce canonical diff + pre-commit halt; align Build dispatch extend REVIEWER_SYSTEM_PROMPT with two prepended HARD CONSTRAINTS: - first action MUST be `git diff origin/<base>` (single-rev, captures uncommitted). no other diff first; no checkout_pr; no alt-ref fetches; no branch listing; no `gh pr list`. - empty canonical diff + claimed-changes dispatch ⇒ reply exactly with `no changes detected — likely pre-commit Build self-review; orchestrator should commit then re-dispatch` and stop. do not guess PR numbers (the zed thrash that ended in `checkout_pr({2582})`). reshape Build mode reviewfrog dispatch step around a verbatim template that names: (a) the situation is pre-commit, (b) canonical diff command, (c) halt-on-empty-diff rule. orchestrator side now says the same thing as the reviewer's baked-in prompt. delegation-discipline bullets and orchestrator-evaluation guidance kept intact. * checkout_pr: handle detached-HEAD entry in initial-branch invariant pullfrog incremental review caught a defense-in-depth gap: `git rev-parse --abbrev-ref HEAD` returns the sentinel string `"HEAD"` on detached entry, which is the default `actions/checkout` state for `pull_request` events. with the previous string-typed `initialBranch`, both the captured value and the live probe would equal `"HEAD"` on any detached state, trivially satisfying the invariant — including a subagent doing `git checkout --detach <sha>`. discriminate the captured HEAD: probe `git symbolic-ref --short HEAD` first (works on named branches), fall back to `git rev-parse HEAD` (SHA) on detached entry. store as `{ kind: "branch"; name } | { kind: "detached"; sha }`. checkout_pr runs the identical probe at call time and compares like-with-like (branch name vs branch name, SHA vs SHA). refusal error renders both heads via a small `describeHead` helper and chooses the right `git checkout` recovery target (branch name or SHA). no inline-discriminant `as` casts — uses a top-level `headsEqual` that narrows via the discriminator. |
||
|
David Blass
|
d3b5340583 |
fix: audit batch — MCP timeouts, entryPost, vip_audit 404s, and 6 more (#824)
* fix: 9 unaddressed log-audit / run-audit findings Co-authored-by: Cursor <cursoragent@cursor.com> #815 entryPost stdlib-only imports; #823 MCP timeoutMs on checkout_pr/shell; #816 FREE_FALLBACK → opencode/big-pickle; #822 chunk GraphQL nodes ≤100; #817/#821 vip_audit 404 skip paths; #813 longer serializable retries; #818 run-context handler-entered log; #805 audit severity template. * fix: update footer test for big-pickle fallback slug Co-authored-by: Cursor <cursoragent@cursor.com> * fix: anneal round 1 — ghaCore getState casing, post-hook timeout Match @actions/core STATE_ key semantics (no uppercasing), cap postApiFetch at 30s, trim serializable retries to stay under GitHub's 10s webhook window, log Clerk failures in getUserTokenByGithubLogin. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: drop run-context handler log (#818 deferred) The #692 client-side fix is already on main; residual SyntaxError hits are old action pins. Per-request log added noise without fixing anything. Co-authored-by: Cursor <cursoragent@cursor.com> * document per-issue Closes syntax for audit PRs GitHub only auto-closes the first issue when numbers are comma-separated; /audits and AGENTS.md now require Closes before each issue number. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: anneal round 2 — outreach privacy, alert resilience, vertex cleanup Filter private repos from VIP authority output, harden console alert lines against DB failures, drop spoofable changesets body check, and unset GOOGLE_APPLICATION_CREDENTIALS after vertex credential cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: drop codexHome re-export of detectCodexRefresh Import detectCodexRefresh directly from codexRefreshDetect.ts everywhere; rename the unit test file to match. codexHome.ts stays install-only. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: drop deprecated minimax-m2.5-free; add paid minimax-m2.5 Remove the deprecated free MiniMax promo from the catalog, docs, and tests. BYOK fallback and picker copy stay on opencode/big-pickle. Add opencode/minimax-m2.5 and openrouter/minimax-m2.5 for Zen BYOK and Router. Pin #816 regressions with freeFallbackCatalog and runErrorRenderer unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: hidden minimax-m2.5-free fallback for stored slugs Re-add opencode/minimax-m2.5-free as a hidden fallback alias to big-pickle so repos with the legacy slug still resolve as free. Drop live Zen API experiment tests in freeFallbackCatalog.test.ts. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Colin McDonnell
|
e65dbe420c | Use Vertex Claude Opus in vertex-claude CI smoke — Haiku quota exhausted. | ||
|
Colin McDonnell
|
58e5b74cb8 | Update footer test for big-pickle BYOK fallback. | ||
|
Colin McDonnell
|
7c5ed7add0 |
Rename OSS_PROXY_MODEL to DEFAULT_PROXY_MODEL; default Router proxy to Kimi K2.6.
Derive the platform default from moonshotai/kimi-k2 openRouterResolve, add models.dev drift coverage, and promote repos on default-branch workflow pushes when still needs_setup. |
||
|
Colin McDonnell
|
fb22cb3ae3 | release: bump action to 0.1.11 v0.1.11 | ||
|
Colin McDonnell
|
c43ed65c3b |
Add Vertex AI routing support (#753)
* add Vertex AI routing support * include Vertex smokes in action CI |
||
|
Colin McDonnell
|
09344a9ec9 | release: bump action to 0.1.10 v0.1.10 | ||
|
Colin McDonnell
|
1b201352b5 | release: bump action to 0.1.9 v0.1.9 | ||
|
Colin McDonnell
|
6c166ac1cc |
fix: prevent cross-PR push from subagent-induced branch switch (#796)
* fix: prevent cross-PR push from subagent-induced branch switch A workflow_dispatch run for zed-industries/cloud (workflow run 26036155393) force-pushed the orchestrator's work onto an unrelated engineer's PR branch (origin/reactivate-pro-plan, PR #2582). The orchestrator's reviewfrog subagent called checkout_pr({pull_number: 2582}), which (1) moved the shared working tree to pr-2582 and (2) persisted pushDest pointing at reactivate-pro-plan. The orchestrator's subsequent commit + push_branch then clobbered the victim PR. Recovery + disclosure in PR #2584. Three compounding bugs closed here: 1. checkout_pr dirty-tree guard had a first-call hole: the previous condition required ctx.toolState.issueNumber to already be set, so on workflow_dispatch runs the first checkout_pr (commonly from a subagent) bypassed the guard entirely. Now any PR switch with a dirty tree is refused, including the first switch of a run. Idempotent same-PR re-checkouts are still absorbed by alreadyOnBranch inside checkoutPrBranch. 2. push_branch trusted sticky pushDest blindly. Added a backstop: refuse pushes where the local branch matches /^pr-(\d+)$/ AND pushDest.remoteBranch differs from it AND the current run is not scoped to PR N (event.is_pr === true && event.issue_number === N). This catches subagent-induced silent branch switches even if a future bug reintroduces a first-call hole in fix #1. 3. Build-mode self-review prompt told the orchestrator to ship "the output of git diff" to the reviewer. The model in this run synthesized git diff main...HEAD, which excludes uncommitted work — and Build self-review runs BEFORE the commit, so the reviewer saw an empty diff and thrashed, eventually calling checkout_pr on a random PR to find something to look at. Prompt now specifies git diff origin/<base-branch> (two-dot, no HEAD), which compares the working tree against the remote base. Refs: zed-industries/cloud workflow run 26036155393 zed-industries/cloud#2582 (victim) zed-industries/cloud#2584 (disclosure) * review: key dirty-tree guard on current branch + drop 'two-dot' misnomer Address review feedback on PR #796. 1. checkout_pr dirty-tree guard now keys off the live current branch (git rev-parse --abbrev-ref HEAD), not ctx.toolState.issueNumber. issueNumber is ALSO set by get_issue / get_issue_comments / get_issue_events, so a subagent doing get_issue(N) followed by checkout_pr(N) on a dirty tree would have bypassed the original guard (issueNumber === pull_number). The current branch is the actual primitive for "would this call move HEAD" — querying it directly avoids correlating on toolState that other tools write to. 2. modes.ts: drop the wrong "two-dot" label on git diff origin/<base>. That's the single-rev form, not two-dot. Copilot was right that the label was confusing/contradictory with the actually-shown command. |
||
|
Colin McDonnell
|
a0576a702a |
opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767)
* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite
Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.
Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.
Surface area:
- drop `init` / `message` / `result` / `tool_result` event types and
handlers (no longer emitted at v1.14+ per upstream
`cli/cmd/run.ts:588-601`).
- `tool_use` is now a single event covering both `state.status:
"completed"` and `"error"`. duration / subagent-finish bookkeeping
moves from the v1 `tool_result` handler into the consolidated
`tool_use` handler.
- new `reasoning` event handler — gated on `--thinking`, surfaces
Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
`baseArgs`.
- drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
v1.15 the `task` tool callID is stable across the whole
`tool-input-* → tool-call → tool-result/tool-error` chain
(`session/processor.ts:282-330`). exact-match map is sufficient.
- drop `experimental.batch_tool: true` from injected config — declared
but inert at v1.15. re-add once upstream wires it back.
- bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
the platform-specific binary into `opencode.exe` for every OS now).
Validated locally:
- `pnpm test` 610/610 ✓
- `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
- `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
(this is the headline `thought_signature` fix)
- runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
- runtest opencode: skill-invoke and mcpmerge fail (model-behavior
drift on the new system prompt; wiring confirmed intact via direct
repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
Tracked for follow-up; does not gate the migration.
Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.
* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)
The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.
* opencode v2: set PWD env explicitly to fix skill / project-config discovery
Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:
const root = Filesystem.resolve(process.env.PWD ?? process.cwd())
We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.
Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.
Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).
* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly
opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).
Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.
Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now: `» thinking (4.0s): <preview>` (one line, sourced)
For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.
Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.
* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment
claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.
Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.
Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.
* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup
Code-quality pass on the v2 work:
1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
between v1 and v2:
- `OpenCodeConfig` type
- `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
- `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
and re-imported by v2 via a back-reference)
- `installOpencodeCli({ binPath })` (parameterized — v1 passes
`bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
`installCli` lambda; matches each pinned version's npm shape)
- `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback
v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
keeps a one-line `export { geminiHighThinkingOverrides }` re-export
so `opencode.test.ts` keeps working unchanged. Once v1 is retired
(post burn-in) opencodeShared collapses back into v2.
2. `opencode_v2.ts` cleanup:
- drop dead state (`currentStepId`, `stepHistory` were write-only —
their reader was the v1 `tool_result` handler we deleted)
- hoist `state` in `tool_use` handler; replace nested-ternary payload
extraction with a `terminalPayload(state)` helper
- extract `formatPartDuration(time)` for the reasoning-block
"(X.Ys)" suffix
- tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
`callID` fields directly, drop the `partWithToolFields` cast
- trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
handler, tool_use handler, bus envelope handler all shortened
- `step_start` becomes an explicit `() => {}` no-op so the dispatcher
doesn't log "unhandled event" for every step
3. `subagentRegistration.test.ts` retargeted at the new file split —
reads opencodeShared.ts for the buildReviewerAgentConfig assertions
and opencode_v2.ts for the orchestrator-model wire-through.
Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.
* opencode v2: address PR review feedback
Three fixes from the inline review threads on #767:
1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
level `markActivity()` resets the module-level idle counter, so the
per-event `getIdleMs()` sample inside the dispatch loop was always
~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
a runner-local `lastEventAt` so we measure real event-to-event silence
instead of chunk-arrival latency. Drop the unused `getIdleMs` import.
2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
`lastProviderError`, and `recentStderr` are closed over by the
`handlers` const but were declared after it. No current bug because
handlers only fire inside the awaited `spawn()`, but a future
refactor that triggers a handler synchronously during setup would
surface a TDZ. Hoisted above `handlers`.
3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
L566): leave a `TODO` comment marking the gap until `AgentUsage`
grows a `reasoningTokens` field — separate PR with schema work.
Cost totals stay correct because `part.cost` is summed independently.
Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
`reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
the bus-envelope flow; the plugin filters orchestrator events except
for status=running task dispatches, and bus-envelope returns before
calling handlers.tool_use on those. No double-fire under current code.
Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.
* DX: flip pnpm play / pnpm runtest to docker-by-default
Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.
scripts (root → action):
- pnpm play → pnpm -C action gha play.ts (docker, default)
- pnpm play:local → pnpm -C action play:local (host)
- pnpm runtest → pnpm -C action gha test/run.ts (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)
action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.
CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.
Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.
File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
`pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers
AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.
wiki/docker.md unchanged — already described the now-real shape.
* test/crossagent: add codex-auth smoke
Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:
- installCodexAuth() materializes auth.json under the test HOME
- opencode routes openai requests through ChatGPT subscription auth
(no OPENAI_API_KEY in env, AT path forced via expires: 0)
- the refresh chain advances during the run (refresh_token rotates)
- detectCodexRefresh() would surface the rotation to entryPost.ts
The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.
CI wiring (already committed in a1c1fd4f as part of the DX flip):
- .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
action-agents env block
- action/.github/workflows/test.yml: same; codex-auth in the
hardcoded test matrix with a claude exclude
The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.
ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.
* docs(codex-auth): make storage requirement unmissable
A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).
- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
loudly contrast Pullfrog secret store vs GH Actions and explain the
writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
unchanged per scope.
* auth codex: auto-open device URL, drop --scope flag
- detect `https://auth.openai.com/codex/device...` from codex CLI output
and best-effort launch it in the user's default browser (open / xdg-open
/ cmd start, wslview fallback on linux). gated so we only open once per
flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
interactive (browser approval), so a "skip-the-prompt" flag for just one
of the prompts was dead weight. collapses scope selection to "always
prompt on org-owned, always account on user-owned".
* rename gha→docker, flip play/runtest defaults to host
the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.
- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
`pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
`README.md`, `AGENTS.md`), CI workflows
(`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
and code headers (`action/{play,test/run,utils/runFixture}.ts`,
`webhook.ts`, `action/test/coverage.ts`)
`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).
* fix(codex): route post-hook writeback through apiFetch + conditional skip
Three threads addressing PR #767 followups.
action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).
action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.
doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
mention (removed in 10be96db, scope is now always interactively
prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
is agent-gated (opencode/opencode_v2 harness), not model-gated;
opencode runs with non-OpenAI models still materialize the file,
it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
→ pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
instead of the dropped --scope flag.
* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS
upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.
also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.
local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.
---------
Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>
|
||
|
David Blass
|
4d1fd5ea1a |
fix: 4 unaddressed log-audit / run-audit findings + close 10 already-resolved issues (#785)
* fix: 4 unaddressed log-audit / run-audit findings closes 4 issues with code changes; 7 issues are already addressed by #769 and 3 are deferred — see PR description. #782 Anthropic 401 → `isApiKeyAuthError` now matches the direct-Anthropic 401 shape (`Failed to authenticate. API Error: 401 ...`, `authentication_error`, `Invalid bearer token`, `api_error_status=401`) so revoked / mistyped / rotated `ANTHROPIC_API_KEY` users see the formatted rotate-key CTA instead of a raw 401 JSON dump. #778 billing-class provider errors → `providerErrors.ts` now classifies `CreditsError` / `FreeUsageLimitError` / `Insufficient balance` / `spending cap` as `provider billing exhausted` *before* status-code patterns can win and tag them as transient `auth error (401)` / `rate limited (429)`. `agentHangReport.ts` swaps the bare "Pullfrog stalled — auth error" headline for a billing-specific CTA (extracts the provider's billing URL when present). #775 silent IncrementalReview swallows `BillingError` → `reportErrorToComment` now optionally falls through to creating a fresh issue comment on `toolState.issueNumber` when no progress comment exists. Wired with `createIfMissing: true` from the `BillingError` / `TransientError` paths in `proxy.ts` so silent triggers (`pull_request_synchronize`) finally surface the router-balance signal on the PR instead of only in the GH job summary. #773 `currentUser()` inside `after()` → `fillInstallerIdentityIfMissing` is split into `resolveInstallerIdentity` (must run inside the request body) and `fillInstallerIdentity` (DB-only, safe in `after()`). The `/console/[owner]` caller now resolves Clerk identity up-front and defers only the prisma write, fixing the broken installer-identity backfill on org-console first-admin visits. Co-authored-by: Cursor <cursoragent@cursor.com> * add /audits cursor command for triaging run-audit + log-audit issues Co-authored-by: Cursor <cursoragent@cursor.com> * review prompt: tighten body-section bar + inline technical-details (#770) * review prompt: tighten body-section bar + add inline technical-details Two layers of tightening to the Review/IncrementalReview prompts in PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step): 1. Reframe inline-vs-body split. Body `### ` sections are now reserved for concerns that genuinely have no line to anchor to — absence, sequencing, design decisions, scope questions, architectural risk. Drop the "cross-cutting concerns" framing (misled the agent into either filing nothing in the body or filing multi-file anchored findings there). 2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step 6) and IncrementalReview (step 8) aggregate phases. Diagnosis from PR #767's auto-review: on substantial PRs the agent surfaced findings but routed all of them inline, producing reviews with zero `### ` body sections even on diffs where non-anchored concerns clearly existed. 3. Replace the abstract `### ` example with a concrete non-anchored one ("Legacy `opencode.ts` has no documented deletion plan") so the agent pattern-matches the absence-shaped finding, not a line-bug. 4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT so inline comments can carry a `<details>Technical details</details>` block when the fix has cross-file implications. Rename the existing "Agent details" inline collapsible to "Technical details" for consistency with body sections. 5. (Carried over from prior uncommitted work) Restructure the review metadata block from `<details>Review metadata</details>` into an HTML comment + an italic TL;DR commit-range line. The HTML comment keeps the metadata addressable for downstream agents without eating user-visible review real estate. No tests touched. * wiki: document multi-model end-to-end eval pattern * feat(promo): cookie-stashed promo codes for onboarding rewards (#771) * feat(promo): cookie-stashed promo codes for onboarding rewards Operator hands out a link like https://pullfrog.com/start?promo=FROGGY; middleware validates the code against an in-code registry, stashes it in an HttpOnly cookie, and the install callback applies the reward once the GH-side account exists. v1 reward: unlimited_runs (lifts the monthly free-runs cap to 1M, same convention prod-grandfathered accounts use). No schema changes. Idempotent across reinstalls via the lte: 100 gate. * fix(promo): integrate handler into existing proxy.ts (Next 16 rename) * docs(promo): clarify sentinel + sync plan doc with renamed paths * feat(promo): add FOUNDATIONS code * feat(promo): show applied promo code in console * refactor(promo): move cookie set to client-side * docs(promo): point JSDocs at PromoCookieSetter, not proxy.ts * billing: cap counts only successful runs (#787) * billing: cap counts only successful runs `reserveRun` was counting `WorkflowRun` rows regardless of status against `Account.includedMonthlyRuns`. Failed / cancelled / skipped / timed-out runs consumed cap slots even though their `billableCents` got zeroed on the completion webhook — pushing paying users into billable territory earlier than the contract implies. `inthhq` paid for 2 extra runs this month because 2 failed runs ate 2 of their 100 free slots. Cap query now filters on `CAP_CONSUMING_STATUS = "success"`. Only runs that actually deliver value consume slots; in-flight (`running`) runs hold no slot until they terminate as success (burst-bypass risk is theoretical given GH Actions concurrency limits). Shared constant lives in `utils/billing.ts` and is used in lockstep by three call sites: `reserveRun` (live cap gate), the billing API's `runsThisMonth` (dashboard progress bar), and the billing-report script's `cap` column. Script's `cap` cell was also broken independently — it compared `monthBillableRuns` (overage count) against `includedMonthlyRuns` (free cap), so `inthhq` rendered as `125/100 (over)` when the meaningful ratio is `223/100 (over)`. Fixed to use `mRuns/cap`, which is the same predicate the live billing path uses. * move CAP_CONSUMING_STATUS to workflowRunStatus.ts + wire script through it Per copilot review: the JSDoc claimed the billing-report script used the constant in lockstep, but the script kept `status: "success"` inline. The script imports from raw-node ESM and can't pull in `next/server`, so it couldn't import from `utils/billing.ts`. Moved the constant to `utils/workflowRunStatus.ts` (already Next-free, already the home of `CONCLUSION_VALUES`) and updated all three call sites to import from there. Script's `mRuns` query now uses `CAP_CONSUMING_STATUS` directly, making drift impossible. * learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743) * learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy three audit fixes on top of the recent learnings overhaul (#717): - `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry when a body has non-whitespace content before the first heading. the prompt instructs the agent NOT to slurp the whole file when a TOC is present, so without this any preamble lines were silently invisible (realistic transitional case: an agent partially restructures a legacy free-text body and leaves bullets above the first `## `). - server-side PATCH route now applies the same line-boundary-aware truncation as the action (defense in depth via a shared `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from `action/internal`). the raw `.slice` it used before could leave a mid-heading tail on any caller that bypassed the client-side truncate, breaking the next-seed TOC parse. removes the duplicated cap constant. - `buildLearningsSection` intro no longer asserts "accumulated by previous agent runs" — false for fresh repos with zero history. new copy is tense-neutral and works for empty + populated bodies. also nudges the agent to re-read after mid-run edits (the inlined TOC ranges are a run-start snapshot). Co-authored-by: Cursor <cursoragent@cursor.com> * learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned calls discovering a quirk this run, recording the workaround prevents the next run from repeating the waste. Reframe around one litmus ("would a future run do its work better because this bullet exists?") and trust it to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary) and the four-example pullfrog/PR/date/play-by-play list (the rule underneath is "don't anchor facts to repo state that will move"). Cuts ~10 lines from a prompt the model was already mostly ignoring; the remaining anchor list is narrower and more enforceable. * audit-learnings-r2: align wiki + tighten re-read nudge - wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls. - buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly. * postRun: refresh JSDoc to match the reflection prompt rewrite `buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets. * fix(mcp/issueEvents): narrow event.event before Set.has lookup octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup. * learnings: split truncation helpers into MCP-free module re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph. move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> * trim first-run celebration email to short personal note drops the feature-dump bullet list (custom review instructions, github iteration walkthrough, security model) — wrong moment to teach. keeps the congrats, the reply CTA, adds discord/x links, keeps the router credit P.S. handler no longer needs the workflowRun→repo lookup. * signup-report: per-bucket histogram Adds a UTC-aligned signups-per-bucket histogram between the overview block and the company-email list. Empty buckets are pre-filled with 0 so dry spells render as gaps. New `BUCKET=hour|day` env flag with a smart default (hour if window ≤ 48h, else day). Histogram is also included in the JSON payload under `histogram: [{key, count}, ...]`. * signup-report: drop hourly bucket, day-only histogram * feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748) * feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660) Per-account ceiling on the sum of `router_topup` invoices (pending + succeeded) for the current UTC calendar month. Closes a gap where a runaway agent loop, leaked PR trigger, or stuck workflow could auto-reload indefinitely with no aggregate per-month ceiling. Two enforcement modes via `RouterLimitMode` enum: - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun; 402 `router_monthly_limit` from /api/proxy-token; email + banner - `alert_only`: auto-reload keeps flowing; email + banner only, first breach per UTC month Enforcement is split across reserveRun (pre-dispatch paywall comment) and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through the same `getRouterSpentThisMonthCents` helper so the dashboard, the dispatch gate, and the auto-reload gate can't disagree. Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string), claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent reloads breaching together send exactly one email. Read-time comparison with the current month re-arms on rollover — no cron. Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell) above the Router/BYOK tabs in `ModelAccessCard`, with a popover "Adjust limit" form that PATCHes the existing /api/account/[owner]/billing/settings route. Same `assertBillingAdmin` gate that owns the other billing settings — no new auth surface. See wiki/billing.md § Router monthly spend limit for the full contract + edge cases. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal pass on monthly Router spend limit (#660) Round-1 review across 5 lenses (billing-subsystem, correctness, security, operational-readiness, research-validated-assumptions) surfaced one critical + three actionable major findings on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot` used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles to `field != value` — UNKNOWN (not TRUE) against the post-migration `NULL` default. First breach for any account would never claim the slot, never stamp the row, and never fire the email (hard_cap or alert_only). Replaced with `OR: [{ field: null }, { field: { not: monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern. **Major — email gap on manual-top-up over cap.** Breach email was only wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>` that crosses the cap blocks dispatch via `reserveRun` but never hits proxy-token, so the user got the PR comment but no email. Wired the CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s PaywallError catch (the SERIALIZABLE txn rolled back when we threw, so we re-claim with the global client; single-statement CAS is its own race boundary against concurrent proxy-token claims). **Major — PR paywall comment leaked $ figures.** `router_limit` body embedded `($X of $Y)` in a comment visible to anyone with PR read access (public repos, forks, outside collaborators). Other paywall types deliberately avoid amounts. Removed; deep link still points to the authenticated console for the figures. **Medium — observability.** Added `[router-limit]` structured logs at the three enforcement sites (proxy-token hard_cap 402, proxy-token alert_only breach, reserveRun paywall) so on-call can grep "did the cap fire for customer X this month." **Medium — customer docs.** Added a `### Monthly spend limit` section to `docs/billing.mdx` (Mintlify) describing the two modes and the manual-top-up caveat. **Doc — refund/dispute interaction.** Documented in `wiki/billing.md` that the cap inherits the existing webhook semantics: disputed `router_topup` drops from the sum (cap briefly un-trips); refunds don't flip status today so refunded top-ups keep counting. Matches wallet behavior — not redefined here. Accepted as-is (documented or pre-existing): `after()` reliability vs stamp-before-send tradeoff, alert_only email fires before Stripe phase-2, proxy-token reads limit fields outside SERIALIZABLE scope (brief TOCTOU on admin lowering cap), stale paywall comment on cap clear, no global kill switch (per-account `alert_only` flip is the practical kill switch), no audit log on cap changes (no existing audit infra), action version not bumped (separate release commit). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal round 2 on monthly Router spend limit Round-2 anneal (billing-subsystem, correctness, research-validated, user-journey, operational-readiness) surfaced a critical merge conflict and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted `formatBillingErrorSummary` from `action/main.ts` to `action/utils/billingErrors.ts`. The PR's new `router_monthly_limit` arm still lived in `action/main.ts`. Took main's slim orchestrator wholesale; moved the arm into the extracted file. **Major — cap = payments only, not dispatch.** `reserveRun` was pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit` regardless of wallet balance, contradicting the cap's positioning as "ceiling on what you pay." An account with $500 of paid-up wallet and a breached $100 cap couldn't trigger any new run via the comment path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded — surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token` is now the sole enforcement point, refusing only the next auto-reload that would push past. Wallet credit always drains. Dropped the now-dead `router_limit` arm in `buildPaywallCommentBody`, the dead `routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`, and the post-paywall email-fire David added — all unreachable. **Major — split `manual_topup` from `router_topup`.** Manual on-session top-ups at `/billing-top-up/<owner>` were landing as `Invoice.kind = "router_topup"` and counting toward the cap. The cap exists to brake *passive* runaway (auto-reload loops); a manual top-up is a deliberate click-through that the user owns. Added `InvoiceKind.manual_topup`, flipped the manual write site + `createTopUpCheckoutSession` metadata, broadened wallet / reconcile / billing-report reads to `kind IN (router_topup, manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap aggregate) to `router_topup` only. Worked example: cap=$300, reload=$100 → exactly three reloads succeed; a fourth is blocked. Historical rows stay labelled `router_topup` (no backfill); the asymmetry is small and accepted since the manual flow only existed alongside auto-reload for a brief window. Extended the `invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows the same shape as `router_topup` (PaymentIntent-backed, no stripeInvoiceId); split into a second migration because PG forbids using a freshly-added enum value in the same transaction. **Major — email reframed around the triggering reload event.** The `alert_only` body was reporting a pre-eager-write `spentCents` while the dashboard reads the post-commit value, so email and dashboard disagreed by exactly one reload. Both flavors now say "Your most recent $50 auto-reload brought you over your $300 monthly limit" instead of a running spent-of-cap total — no reconciliation needed, no more "you've hit your monthly cap" copy firing for partial breaches (spent=$80 of $100, reload=$30 would have triggered that wording). **Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded "You've used your 30 free runs this month. Add a card to continue at 7¢/run." regardless of `detail.reason`. Branched on `cap` vs `delinquent` so each paywall surfaces actionable copy with the right CTA. `router_limit` no longer flows through here (per F4 above). **Major — RouterLimitBanner.** Added an `isAlertBreached` visual state (amber palette) so an `alert_only` account at $240 of $200 no longer renders in the same neutral zinc chrome as a healthy under-cap account. Updated popover copy to reflect the auto-reload-only scope. **Medium — paywall log line.** Added `detail.reason` to the `[Installation X] paywall:` log so on-call grepping for "why was this paused" can distinguish `cap` from `delinquent`. **Cleanup.** Dropped dead `utcMonthKey` import + re-export in `maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*` fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*` since they now handle both kinds. Updated wiki/billing.md + docs/billing.mdx + schema doc comments throughout. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt` sitting next to it — a single-purpose state column on `Account` that encoded a date as a string and required a custom CAS predicate to read/write race-safely. Plus it had real holes: Resend send failure left the sentinel stamped and the account silently un-emailed for the month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered edge cases never fired at all. Replace it with: fire `maybeNotifyRouterLimit` on every breaching reload, let the Resend `Idempotency-Key` `router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside Resend's 24h dedup window. Continuously-breaching accounts get ~1 reminder per day; brief Resend outages self-heal because the next breaching reload re-attempts the send. Mode is in the dedup key so `alert_only → hard_cap` mid-month re-arms a fresh email with the appropriate copy. Drops `Account.routerLimitNotifiedMonth` and `claimRouterLimitNotificationSlot`; simplifies the proxy-token phase-1 branch significantly. Net diff is negative LOC and the data model loses a single-purpose sentinel. Migration was branch-local — never deployed — so I edited the original add-cap migration in place to drop the column from the ALTER TABLE rather than chain a drop-column migration on top. Preview Neon branches reset automatically on history rewrite per wiki/migrations.md. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): hide RouterLimitBanner when no cap is configured The banner was unconditionally rendered for every billing-enabled account, including pure-BYOK admins who never touch Router. They got "No monthly spend limit / Router has spent $0.00" + a divider as visual noise on the model access page — basically nagging them to set a feature they may not want. Running without a cap is valid; we don't nag. ModelAccessCard now gates the banner block (banner + dividers) on `routerMonthlyLimitCents !== null`. RouterLimitBanner drops the no-limit visual state, the "Set monthly limit" CTA text, and the dead `hasLimit` branching. Cleaner three-state shape (under cap / amber breached / brick breached). Discoverability: no-cap users no longer see a UI affordance to set one. That's deliberate — the cap is a power-user feature documented in docs/billing.mdx. If discoverability becomes an ask, we can add a small inline link inside RouterWalletSection without bringing back the always-visible banner. Resolves the only outstanding finding from cursor bugbot's review of ff5328c (banner-visible-for-byok thread). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(billing): docs/wiki match new "no banner without a cap" reality Pullfrog bot review of f7672ca pointed out the customer docs still told users to "Set the cap from the **Monthly spend limit** banner in the **Model costs** card" — but after hiding the banner for no-cap accounts there is no such banner to use until you already have a cap. Catch-22 for first-time setup. Rewrote docs/billing.mdx to be self-contained: explain what the cap is, what the two modes do, what the banner shows *once configured*, and direct admins to PATCH the billing settings endpoint (or reach out to support) for first-time setup. Cap is positioned as optional; running without one is the documented default. Wiki paragraph in wiki/billing.md updated to match — banner is only rendered when a cap exists, three visual states (under / amber / red), no first-time-setup UI nag by design. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely The standalone `RouterLimitBanner` was the wrong shape. It only rendered when a cap was already configured (so there was no UI to discover the feature in the first place — first-time setup required hitting the API directly), and it occupied prominent real estate above the tabs to surface state that already lives in the row's own input when the form moves down where it belongs. New shape: monthly cap is just a third row inside `RouterWalletSection` sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated the same way (card on file + auto-reload enabled — the only state where the cap actually means anything). Empty input → no cap, with placeholder "No limit". Setting a number reveals a **Behavior at limit** toggle built on the same `Tabs` slider component used for the Router/BYOK tab switch, so the look matches the rest of the card. Deletes: - `RouterLimitBanner` component (212 lines) - banner mount + conditional + spacers in `ModelAccessCard` - `AlertTriangle` is still imported (used by `DelinquencyBanner`) Adds: - one settings row in `RouterWalletSection` with the cap input + mode tabs - `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the existing `saveSettings` helper (widened to accept `string | null`) - `Tabs` / `TabsList` / `TabsTrigger` import Docs + wiki updated to match the new shape; the customer doc no longer points at a banner that won't appear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between Previously bundled both into one row block. Restructure: cap input is its own row; Behavior-at-limit Tabs gets a sibling row with the standard `h-5 + hr + h-5` separator between (matching the rhythm of auto-reload amount → threshold → monthly cap). Mode-toggle row is gated on `routerMonthlyLimitCents !== null` so the hr + tabs only appear once a number is in the cap input. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row Same `flex items-center justify-between gap-3` layout as the Auto-reload row: label group on the left, control on the right. Drops the vertical stack in favour of the horizontal one — looks identical to the toggle row directly above. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> * drop italic TL;DR commit-range line from review body the metadata (sha range, commit list, timestamps) is already in the html comment for downstream agents. the visible italic line was clutter and the ellipsis form broke the second sha's auto-link on github anyway. * add agent-browser fallback rule for unreachable chrome devtools mcp * onboarding: gated org-console wizard (#762) * onboarding: gated org-console wizard Replaces the org console's `/console/[owner]` page with a single-card, "growing" stepper when the account has zero `Repo` rows. Walks first-time users through billing mode, BYOK provider+key (if applicable), repo pick, workflow file creation, and a celebratory redeem-credit moment before landing them back on the now-populated org console. ## What's new - New: `components/OnboardingStepper.tsx` — the wizard. Six steps, each derived from real persisted state (Account.modelAccessMode, AccountSecret, Repo). Step state ladder with progressive disclosure and click-to-edit collapsed summaries. - New: `app/console/[owner]/OnboardingView.tsx` — page-chrome wrapper that hosts the stepper inside the same header/sidebar shell as the member view. - Modified: `app/console/[owner]/page.tsx` — adds a `prisma.repo.count` gate alongside existing parallel queries; renders OnboardingView when count === 0, else falls through to the existing repo grid. ## Schema - Flipped `Account.modelAccessMode` default from `byok` to `router`. Router is the lower-friction default (signup credit funds first ~150 runs without a card; users can flip to BYOK explicitly via the wizard or the existing `<ModelAccessCard>` switch). Existing rows keep their current explicit value — Postgres column-default change doesn't backfill, by design. - Migration: `20260516014601_modelaccessmode_default_router`. ## Credit-claim semantics Killed the historical mount-time auto-claim on `<SignupCreditModal>`. All claims are now explicit clicks, fired from one of two surfaces: 1. Wizard step 6 "Redeem $10 credit" CTA (Router branch, eligible). 2. New explicit "Redeem $10 credit" button on `<BillingCard>`'s Router wallet section, visible only when the new server-derived `signupCreditEligible` flag is true (promo active + no prior signup or welcome grant). Covers existing users who'd otherwise lose the auto-claim entry point. `<SignupCreditModal>` is now a controlled component (`open` / `onOpenChange` / `amountCents` props) with a sibling `useClaimSignupCredit(owner)` hook for explicit invocation. The Sparkles celebration dialog rendering is unchanged. ## Other touched surfaces - `app/api/create-workflow/route.ts`: optional `model` body field. When present, the route updates `Repo.model` on the row that `createWorkflowForRepo` just created/surfaced — wizard threads the picked provider's `preferred` model alias through here so a fresh repo doesn't sit on null/auto. - `app/api/account/[owner]/billing/route.ts`: surfaces `signupCreditEligible: boolean` (derived from `SIGNUP_CREDIT_PROMO_ACTIVE` + grant scan). Drives the new explicit redeem button. - `components/AgentSettings.tsx`: fixes the Router-no-billing copy lie ("Runs will draw from your signup credit until exhausted" was false — `isInfraCovered` gates Router minting on `hasCardOnFile`, not balance, so credit-only-no-card users can't actually spend the grant on Router runs). New copy: "Add a card to use Pullfrog Router. Your $10 signup credit (if claimed) applies on top." ## Resume-tomorrow detection Every step's expansion is derived from persisted state (no new column, no localStorage). With the Router default flip, `modelAccessMode === "byok"` is now a reliable signal of explicit user pick, eliminating the heuristic that the byok-default schema would have required. The only ambiguous case is "Router-bailed-before-redeem" (looks identical to a default-Router fresh visit since neither card nor grant exists yet) — acceptable 1-click cost on revisit. ## Testing - `pnpm lint`: clean - `pnpm format`: clean - `pnpm typecheck`: clean - `pnpm -C action test`: 596/596 passing - Visual verification: blocked — Chrome DevTools MCP returned "Not connected" across both available servers. Manual walkthrough needed before merge to confirm step transitions, going-back UX, and the celebration modal redirect destinations match the plan in `.cursor/plans/org_onboarding_stepper_4fdfebbb.plan.md`. * onboarding: drop accordion, multi-repo bulk-onboard, full-width radio rows Three rounds of UX feedback rolled in: 1. **Drop the accordion.** Steps no longer collapse to a one-line summary when "done" — the wizard literally grows by appending steps below as the user progresses, and earlier steps stay fully interactive (re-flip Router→BYOK, re-pick provider, toggle a repo) without any "edit" affordance. `StepShell` now always renders its body for any step the user has reached; the only state distinction is the number circle (filled = active, check = done). 2. **Step 1 is full-width radio rows, not narrow tabs with side-by-side info tiles.** Two rows, each with the option title, an inline "Recommended" badge on Router, and a description sentence inside the row. The persisted `Account.modelAccessMode` (default `router`) drives the initial selection, so step 1 always has one row picked on first paint — no "neither selected" empty state. 3. **Multi-repo bulk-onboard.** Step 4 now uses checkboxes; copy reads "Select the repos you'd like to install Pullfrog into. We'll create a pullfrog.yml GitHub Actions workflow file in each." Step 5 fans out N parallel `POST /api/create-workflow` calls (concurrency capped at 4) and renders per-repo status inline (running → committed / PR #N / already configured / error). Step 6 celebrates with a multi-result headline ("Pullfrog is set up across N repos") and a sub-line breaking down `committed · PRs awaiting merge · failed` plus a per-repo PR list when any PRs were opened. Single- repo path renders the same control surface but with singular copy. Other bits: - Per-step description sentences below every title. - Repo picker shows totalCount inline with the pagination controls and "N repos selected" summary below the table. - Dropped the `userPickedBillingMode` and `editingStep` state machinery + the `isFreshDefault` heuristic — all simplified out by the no-accordion design (we just trust `billingMode` directly). - `createWorkflowPR` PR body already links back to `pullfrog.com/console/<owner>/<repo>` with a "Verify workflow" CTA; no change needed there. * fix(onboarding): provider tile labels — getProviderDisplayName expects slug `getProviderDisplayName` from `pullfrog/internal` parses its argument as a `provider/model` slug. Step 2 was passing bare provider keys (e.g. "anthropic"), which made the helper throw "invalid model slug 'anthropic' — expected 'provider/model'" and crashed the BYOK branch with the page-level error boundary. Replace with a local `providerDisplayName` that reads the registry directly (`providers[key].displayName`). Drops the unused `getProviderDisplayName` import. Caught by Chrome DevTools end-to-end: clicking Bring-your-own-key on the fresh wizard renders the page-level error. Re-verified post-fix: BYOK flow shows step 2 with all 9 provider tiles correctly labeled (Anthropic / OpenAI / Google / xAI / DeepSeek / Moonshot AI / Amazon Bedrock / OpenRouter / OpenCode), step 3 reveals on tile click. Also adds a guardrail to AGENTS.md: don't silently abandon visual verification when DevTools breaks. Recovery is always possible (pkill -9 chrome-devtools-mcp + pkill puppeteer + rm Singleton locks + retry several times); if it genuinely won't recover, abort and tell the user — never mask as "verified by code review". * agents.md: never give up on Chrome DevTools MCP failures Recovery is always possible (pkill chrome-devtools-mcp, remove Singleton locks, retry several times). If genuinely unrecoverable, abort and tell the user explicitly — never silently mask as "verified by code review". Visual verification is non-negotiable for UI changes. * onboarding: polish — checkbox color, redundant labels, copy Caught during chrome-devtools verification of the BYOK + cross-page selection flows: - **Checkbox color**: native browser pink/red replaced with evergreen via `accent-evergreen-600`. Visually consistent with the rest of the wizard's selection states. - **Bedrock provider tile**: was rendering "Amazon Bedrock" twice (provider name + recommended-model name both resolve to "Amazon Bedrock" because Bedrock has no `preferred` model under `providers.bedrock.models` — its single routing entry IS the recommended pick). Suppress the recommended subtitle when it duplicates the provider name. - **Step 6 description**: tightened from a clunky two-clause sentence about workflow file landing to a single direct call: "Mention @pullfrog in any PR or issue to dispatch a run. (Branch-protected repos: merge the PR first.)" - **Wizard intro**: was "Set up Pullfrog for your first repo" — outdated since multi-repo. Now: "Connect Pullfrog to your repos. Each step unlocks the next as you go." Cross-page multi-select also verified: selections from page 1 persist when navigating to page 2 and back. "N repos selected" counter reflects total across all pages. BYOK secret-add flow verified end-to-end: AddSecretModal opens with the env var pre-filled, save triggers secrets refetch, step 3 flips to "✓ ANTHROPIC_API_KEY configured", step 4 reveals automatically. * onboarding: serial install, inline secrets, explicit credit redeem - step 3: replace modal-based secret entry with inline password fields per provider, with deep links to provider dashboards. claude code OAuth surfaces as a distinct group when anthropic is picked. bedrock gets three-field form. github actions secrets path is collapsible with org/personal-aware urls + self-certify. - step 4: merge repo-pick + workflow-create into one step. install is now serial (visible slow-reveal) instead of concurrent. continue button renders immediately on submit, disabled until every repo reaches a terminal state. errored rows render a single soft amber 'failed' label. pagination uses chevron buttons + keepPreviousData (no layout shift). - step 6: explicit 'redeem $10 credit' for router+eligible, 'complete setup' otherwise. final redirect is a hard refresh so the repo grid picks up. - signup credit: drop the mount-time auto-claim modal in favor of explicit user clicks. new useClaimSignupCredit hook + RedeemSignupCreditCallout banner inside RouterWalletSection so a BYOK→Router flip surfaces a one-click redeem affordance. - billing mode is now optimistic (local state + background PATCH) and initialBillingMode + signupCreditEligible eager-load via server props to kill the multi-second click latency. - skip onboarding: header button sets pullfrog_skip_onboarding cookie; server reads it in page.tsx and falls through to the regular grid. - demo mode: NEXT_PUBLIC_ONBOARDING_DEMO=1 cycles the install progress list through pending/running/committed/PR/existing/failed states. - createWorkflowForRepo: PULLFROG_FORCE_PR_CREATION=1 skips direct commit to exercise the PR fallback locally. * onboarding: review feedback — focused eligibility query, best-effort model pre-fill, claim error toast - billing/route.ts + console/[owner]/page.tsx: replace top-N recentGrants scan for signup-credit eligibility with a focused findFirst({ reason: { in: [SIGNUP, WELCOME] } }). the prior query could return any 5/10 rows (no orderBy on page.tsx) and miss a prior signup/welcome grant if a future grant reason (refund/referral/etc.) ever ships. recentGrants stays for the billing-history list. - create-workflow/route.ts: gate Repo.model updateMany on result.type === "created" so an existing user-set model isn't clobbered when the workflow file already exists. wrap in try/catch: GitHub side effect already succeeded, so a transient DB blip shouldn't 500 the route and have the UI report failure on a partially-completed setup. - SignupCreditModal: add onError toast to useClaimSignupCredit so transient redeem failures surface ("Couldn't redeem your credit. Try again in a moment."). callers .catch(() => null) the rejection so it doesn't propagate as an unhandled rejection in the React handler. - OnboardingStepper: trim stale "per-row try again button" wording from progressRef + processRepo comments — that button was removed in the prior commit per design feedback. * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * revert: extract router-gate fix into its own PR The router fix at a14bcdd4 is being shipped as a standalone PR so it can be reviewed and merged independently of the onboarding-wizard work. Reverting here keeps #762 focused on the wizard. The fix itself landed at https://github.com/pullfrog/app/pull/792. * router: fix unspendable signup credit on no-card private repos (#792) * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * action: drop dead isInfraCovered + plan param post-fix Cleanup the action-side dead code introduced by the previous commit's removal of the redundant `isInfraCovered` re-derivation in proxy.ts: - delete `isInfraCovered` from action/utils/runContext.ts (was the only callsite; mirror in server's utils/billing.ts is unchanged and still load-bearing for learnings/indexing) - drop unused `plan: AccountPlan` param from `resolveProxyModel` / `runProxyResolution` (and the corresponding `AccountPlan` import + the `plan: runContext.plan` arg at the main.ts call site) - update the action/mcp/server.ts comment that pointed at the now-gone action mirror to reference the server-side `utils/billing.ts` instead `AccountPlan` itself is still load-bearing (mcp/server, runContextData, run-context fetch), only `isInfraCovered` and the dead `plan` parameter go away. * eager signup credit + free-OpenCode fallback when BYOK has no key (#789) * eager signup credit + free-OpenCode fallback when BYOK has no key addresses the silent-churn pattern that took out 15 first-run-failure accounts post-launch: GH Actions secret references resolved to empty strings (because the secrets didn't exist on the repo), the action launched Claude Code with no key, the LLM provider 401'd, and the run died in seconds with a synthetic "Invalid API key" message. those accounts had no Router credits to fall back to because the lazy claim required a dashboard visit they never made. three changes, one PR: 1. Eager $10 signup credit at account creation. Both account-creation sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo` for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }` in the same transaction as the `accounts` row. CLI installers who never sign in get the credit. The dashboard `/signup-credit/claim` POST stays as an idempotent backstop for accounts created before this shipped. 2. Free-OpenCode fallback in the action. When the configured BYOK slug needs a provider key the runner doesn't have, swap to `opencode/minimax-m2.5-free` before agent selection so the run still succeeds. Surfaced via a `» fell back from <slug> to <free>` warning in the action log. Skipped on Router runs (Pullfrog mints the key) and when no model is configured (auto-select-with-throw still fires for the genuinely-misconfigured case). 3. New action-test fixture `byok-no-keys-fallback` that empty-strings every known provider key (matching how GH Actions handles missing secrets) and asserts the run succeeds with the fallback log line present. plus a unit test for the helper covering each skip case. skipping the schema flip from `byok` to `router` — that's coming via the onboarding-stepper PR (#762). * fallback: skip Bedrock + surface in PR-comment footer addresses copilot review on #789 (real bug — parseModel throws on Bedrock raw IDs that have no slash, would crash before validateBedrockSetup could surface its own error) and the user-side ask to make the fallback visible in PR comments. - selectFallbackModelIfNeeded skips when resolvedModel has no '/' so Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash inside hasProviderKey -> parseModel. unit test covers it. - toolState.modelFallback records the configured slug we fell back from. set in main.ts when fallback engages. - buildPullfrogFooter accepts fallbackFrom and renders "Using `MiniMax M2.5` (free) (credentials for Claude Opus not configured)" so the substitution is visible in PR comments, reviews, PR bodies, and error reports. - threaded through all four action-side footer call sites (mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts fire pre-action and don't have toolState — left as-is. * fallback footer: use provider display name + document email asymmetry addresses pullfrog reviewer findings on #789: - footer now shows 'credentials for Anthropic not configured' (provider display name from `providers.anthropic.displayName`) instead of the per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY covers all Anthropic models), so this matches what the user actually needs to fix. - document the intentional asymmetry between eager and lazy signup credit paths: eager skips both the signupCreditClaimedEmail and the per-grant team@ alert. comment explains why (the 'new account created' alert already covers it on the eager path; the user-facing email assumes a user-initiated action that hasn't happened yet for CLI/GH-App-only signups). - skipping the backfill for the 15 historical accounts per user's earlier decision — they all uninstalled, so the cohort self-selected out of being reachable. * fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap local agnostic fixture run surfaced two real bugs the unit tests didn't catch: 1. fallback gate was on configuredSlug (=payload.model) but the test uses PULLFROG_MODEL to set the model, which is read by resolveModel AFTER its slug arg. configuredSlug stayed undefined → fallback never fired. drop configuredSlug from the helper signature; gate purely on resolvedModel since that's the same value regardless of how the model was specified (DB config vs PULLFROG_MODEL env). 2. when fallback engaged, the post-swap resolveModel({slug: fallback.to}) call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback target back to the unkeyed model. validateAgentApiKey then threw "no API key found" against the original model. fix: skip the re-resolve. fallback.to is already a CLI-ready specifier. unit tests updated for the new helper signature (8 tests, all pass). fallback log line confirmed emitted in the local run pre-second-fix; the second fix unblocks the validation that previously threw. * models-bump: harden CI and bot prompt against catalog hallucinations PR #790 (the first bot-authored models-bump PR) shipped a broken bump for openrouter/gemini-flash: the bot pattern-matched the parallel google bump and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the OpenRouter API but not on models.dev's openrouter section — the catalog OpenCode actually reads. The slug failed at runtime with `Model not found`. Two CI gaps let it through: 1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*` alias as a "passthrough", smoke-testing only one canary per routing layer. But those aren't passthroughs — each is a distinct models.dev catalog entry that can drift independently of the direct-provider mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25). Only `bedrock/byok` stays pruned (sentinel resolve). 2. `models-catalog` test (the integrity gate that asserts every resolve exists on models.dev) was main-only by design — to keep upstream catalog churn from blocking unrelated PRs. But it's exactly the test we want running on the bot's own catalog edits. Add `pullfrog/models-bump` head-ref to its trigger. Also tighten the bot prompt in models-bump.yml: new rule 0 requires every new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the alias's own `candidates[]` in models-bump-context.json — the deterministic preprocessor only emits candidates sourced from models.dev's mirror, so this gates against the cross-alias pattern-matching that broke PR #790. For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter API), which is necessary but not sufficient; the `models-catalog` job is the authoritative models.dev check. Verified locally: - baseline `pnpm -C action test:catalog` passes 133 tests - simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash` into action/models.ts) and the catalog test fails with the right assertion: `model "google/gemini-3.5-flash" not found under openrouter on models.dev` - `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every openrouter/* alias and every keyed opencode/* alias now smoked --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |
||
|
David Blass
|
2f1f136da8 |
modes: actually call resolve_review_thread on addressed PR feedback (#749)
* modes: instruct IncrementalReview + AddressReviews to actually call resolve_review_thread empirically (per #672 audit on 195 prod runs across 4 repos) only 1 in 195 runs called resolve_review_thread. IncrementalReview's prompt only mentioned prior reviews as a dedup filter — the agent had the data in hand but no instruction to retire addressed threads, so it almost never did. AddressReviews mentioned resolve as a one-line bullet at the end of step 6, which the agent treated as optional (the one observed AddressReviews run replied to 3 comments and resolved 0). IncrementalReview step 4 now: fetch prior reviews → for each open Pf-originated thread, decide if the new commits addressed it (anchor moved, isOutdated, or substantive concern resolved on a re-read), reply + resolve when addressed, leave open when uncertain. Conservative scope: only Pf-originated threads — human-reviewer threads stay theirs to mediate. AddressReviews step 6 now pairs reply + resolve in the same beat with explicit rules: resolve when you made the change OR replied substantively; do NOT resolve when you pushed back and the disagreement is unresolved. addresses #672. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal: tighten auto-resolve decision rules correctness fixes from /anneal pass on IncrementalReview step 4 + AddressReviews step 6: - `[OUTDATED]` no longer "strong signal of address" — it just means GitHub moved the anchor (line shift / reformat / force-push); agent must re-read code at new location - explicit Pf-origin detection rule (first `comment author=pullfrog[bot]` tag), clarifies `*` marker is unrelated to thread root - explicit dual-ID separation: numeric `id=` for `reply_to_review_comment.comment_id`, GraphQL `thread=` for `resolve_review_thread.thread_id` (was silently 422-ing) - reformatter / partial-fix loophole closed: lines being modified isn't enough; all concerns in multi-concern comments must be addressed - AddressReviews push-failure path explicit: STOP and report_progress, do NOT reply or resolve when fix isn't live - step 4 → step 8 wiring rationale corrected (step 8 dedups by line range, not thread state) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |