main
323 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wolfy
|
eb8871d6d2 | feat: add configurable context_window action input | ||
|
wolfy
|
1e839d36a9 | fix: review process and cleanup | ||
|
wolfy
|
0dc0f7eb53 | feat: add read_file tool | ||
|
wolfy
|
3f0d9a80c7 | fix: issue with shell calls | ||
|
wolfy
|
fe85adfa53 | fix: reviews failing to call next tool | ||
|
wolfy
|
0cf9df2bb6 | chore: revert back to initial instructions for the most part | ||
|
wolfy
|
19671c6299 | feat: branch protection + deps caching | ||
|
wolfy
|
0438688e32 | fix: issues with pagination not resolving correct url templates | ||
|
wolfy
|
2aca1a3aa3 | feat: adapt pullfrog for gitea + ollama | ||
 Colin McDonnell
|
36ac64a5b6 |
fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy (#844)
* fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy OSS-allowlisted repos with `CODEX_AUTH_JSON` uploaded via `pullfrog auth codex` were still being routed through the OSS OpenRouter subsidy because two paths ignored managed credentials: - `hasProviderKey()` only checked `provider.envVars`, so an `openai/*` model with only `CODEX_AUTH_JSON` present silently fell back to `opencode/big-pickle` via `selectFallbackModelIfNeeded` — the maintainer saw "opencode/big-pickle (resolved from openai/gpt)" on CI even though Codex was configured. - `run-context` set `proxyModel` for every OSS run unconditionally, which the action runtime threads through `payload.proxyModel` and uses to overwrite `OPENROUTER_API_KEY`. Even if `big-pickle` fallback hadn't fired, the runner would consume the $10 OSS subsidy key instead of the user's ChatGPT subscription. Fix: - Add `getModelAuthEnvVars()` covering both `envVars` and `managedCredentials` in `action/utils/apiKeys.ts`; route `hasProviderKey` + `validateAgentApiKey` through it. - `run-context` now skips `proxyModel` for OSS repos when the configured model's provider has matching auth in Pullfrog-stored account/repo secrets, so the runner authenticates directly with the user's Codex subscription (or any other user-provided provider auth). Triggered by mrlubos (`hey-api/openapi-ts`). Companion follow-up tracked for the opaque "(no error message)" classifier swallow that masked the OSS $10 cap exhaustion on PR #3872 runs 25815370844 + 25815443234. * fix(oss): force Kimi K2 for OSS proxy + hide picker in console UI OSS-funded runs were resolving `repo.model` through OpenRouter, so a single Opus / GPT-5.5 run could burn an entire `oss_subsidy` key against the per-key cap and crash mid-stream (e.g. `hey-api/openapi-ts` PR #3872 runs `25815370844` + `25815443234`, ~$9.20 each on a single key). Force `DEFAULT_PROXY_MODEL` (Kimi K2.6 — ~10-50× cheaper) for every OSS proxy mint, regardless of `repo.model`. Per-run spend stays bounded within the cap by structure, not by hope. `repo.model` stays in the DB unchanged — overriding at runtime means leaving the program restores the user's prior pick without a migration. UI: hide the model picker entirely on OSS repos in `AgentSettings`. The field is effectively inert until the repo leaves the program, so exposing it as if it were live was misleading. Replaced with a banner naming Kimi K2 and pointing to `pullfrog auth …` as the opt-out path — that lands the user on the existing #844 bug-2 branch (Pullfrog-stored auth suppresses the OSS proxy entirely; runner uses user credentials + their preferred model). ModelCostsInfo already has its own `isOss` branch for the cost copy, so that section is unchanged. * fix(oss): lowercase comment casing per AGENTS.md * fix(oss): revert banner copy to 'It's on us.' framing per review Maintainer felt 'Kimi K2' as the banner headline lost the warm 'we've got you covered' framing that the existing OSS cost banner uses. Restore 'It's on us.' as the headline, move the model name into the body where it explains the hardcoded choice and points to the opt-out (pullfrog auth codex / account secret). * docs(agents): screenshots must be of the live route, never synthetic Caught myself building a temp `/dev/oss-ui-preview` route with hardcoded JSX copy-pasted from the real component just to grab a screenshot — the result told us nothing about whether the actual integrated UI worked, and the user (rightly) called it out as a waste. Strengthen the rule: screenshots must come from the live route in the running app, driven by the actual component tree and real props. Note the GH OAuth interstitial gotcha so the next agent gets through Clerk → GitHub sign-in on the first try instead of bailing to a fake render. Also bans side-by-side comparison screenshots unless explicitly requested. * fix(oss): one 'It's on us.' banner, not two OSS Agent settings was showing the message twice — once in the Model section, once in the Model costs section right below it. Fold the cost coverage into the model banner ('at no cost to you' + the spend stat) and hide the Model costs subsection entirely for OSS. ModelCostsInfo no longer needs `isOss` / `ossSpendThisMonthUsd` props — call site is gated, so the OSS branch is dead. Removed it and the now-unused props. Non-OSS rendering is unchanged: full Model picker + Model costs subsection with Router / BYOK branches. * feat(action): corepack-aware package manager provisioning before setup customer setup scripts that did `npm i -g pnpm && pnpm install` were installing whatever pnpm "latest" happens to be on the day the run fires, not what the repo declares — and pnpm 11.3 silently writes a new `packageManagerDependencies` block into lockfiles, which the agent's "always push changes" rule then packages into a noisy PR (see #844). resolve the project's pnpm/yarn pin from `package.json` (honoring pnpm 11+ precedence: `devEngines.packageManager` over `packageManager`) and activate it via `corepack prepare ... --activate` BEFORE the setup hook runs. corepack is bundled with node, so this is a no-op on managed infra; failure (no corepack, no network, range-only version) degrades to a warning and the existing PATH binary still runs. also replaces the legacy `npm install -g <pm>@<v>` path in prep with the same helper so behavior is consistent end-to-end. bun/deno still use the legacy installer because corepack doesn't ship shims for them. * chore(console): drop 'npm i -g pnpm' anti-pattern from setup-script placeholder the suggested example trained customers to install pnpm unpinned, which silently picks up whatever's latest at run time. that's exactly the behavior #844 traced lockfile drift back to. now that prep handles package-manager provisioning via corepack from the repo's declared pin, the placeholder is just a frozen-lockfile install — load-bearing only when the repo wants `pnpm install` to actually run (prep already does that), but a much safer default for customers who do paste it in. * refactor(action): introspect opencode models for BYOK detection Replace the static `provider.envVars + provider.managedCredentials` catalog gate in `selectFallbackModelIfNeeded` + `validateAgentApiKey` with two `opencode models` captures around the auth merge: - `captureBaselineModels` BEFORE dbSecrets + Codex auth.json - `captureAuthorizedModels` AFTER both The authorized set is the authoritative source for "can OpenCode route this model" — strictly more accurate than the catalog, which can miss new auth shapes (Codex was one, there will be more). The diff between baseline and authorized is logged as `BYOK auth enabled N model(s)` for operator visibility. Sequencing changes in main.ts: - `createTempDirectory` hoisted out of the try block so `PULLFROG_TEMP_DIR` is set before the early opencode install - `agents.opencode.install()` + baseline capture before dbSecrets - `installCodexAuth()` hoisted up (idempotent — agent re-calls it inside run() and writes the same file) - authorized capture after Codex auth.json materializes - fallback + validateAgentApiKey receive the authorized set as a parameter; tests inject directly with no mocks Deleted: `hasProviderKey`, `getModelAuthEnvVars`, `knownApiKeys` in `action/utils/apiKeys.ts` (only `selectFallbackModelIfNeeded` consumed them, and PR #844's catalog-extension fix is superseded by introspection). `getModelEnvVars` / `getModelManagedCredentials` stay exported for UI and the server-side OSS proxy heuristic in run-context/route.ts. For the claude agent path, validateAgentApiKey keeps the static single-provider check on `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` — `opencode models` is opencode-specific. validateBedrockSetup / validateVertexSetup also stay; they cover region/location/model-id which `opencode models` doesn't catch. When fallback engages, the post-fallback model is the guaranteed-free `opencode/big-pickle`, so validateAgentApiKey is skipped — the fallback gate already authoritatively decided "this model is OK to run". * test(oss): temp add preview-844 to ossRepos for O4 e2e * Revert "test(oss): temp add preview-844 to ossRepos for O4 e2e" This reverts commit 8167e560126b2ac516c32ba1c63c36aa32ae4019. * test(oss): temp add preview-844 to ossRepos for O5 e2e * fix(action): skip validateAgentApiKey when proxyModel is set The new opencode-models BYOK introspection in PR #844 captures the authorized set BEFORE runProxyResolution mints OPENROUTER_API_KEY, so the proxy slug (e.g. `openrouter/moonshotai/kimi-k2.6`) is never in the set. validateAgentApiKey then spuriously threw "no API key found" on every OSS run, even though the proxy key was minted correctly and the inference would have worked. Mirrors the analogous skip in `selectFallbackModelIfNeeded`: when proxyModel is set, the server-side gate (`run-context/route.ts`) is the authority and the proxy mint itself is the validation. Caught by O5 e2e on `pullfrog/preview-844-heyapi-oss-bug`. * Revert "test(oss): temp add preview-844 to ossRepos for O5 e2e" This reverts commit 3bae075ceeb188ee272c45c13b5080e15bcd00a5. * fix(action): discard hook-generated tracked-file drift before agent sees it addresses bug 3 in #844: customer setup/post-checkout hooks like `pnpm install` or `corepack prepare` left the working tree dirty (e.g. `M pnpm-lock.yaml`), the agent took the prompt's "must push" rule literally, opened a spurious bot PR for the lockfile drift, and we ate runs+spend on noise. after each setup / post-checkout hook (opt-in via `normalizeWorkingTreeAfter`), discard tracked-file mods with `git restore --staged --worktree .`. untracked files are preserved — a hook that materializes a `.env` from a template, or emits codegen output, stays visible to the agent. guarded by a pre-hook `git status --porcelain` snapshot: if the tree was already dirty before the hook ran (shouldn't happen — setup runs before any working-tree writes; checkout_pr refuses to run dirty), we warn and skip the discard rather than clobber whatever was there. prepush hook (action/mcp/git.ts) intentionally does NOT opt in — its job is to read the about-to-be-pushed state, not normalize it. * test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge) * fix(action): skip eager pnpm/npm/etc install when no lockfile exists second half of bug 3 in #844. the eager prep step assumed `pnpm install --frozen-lockfile` (and equivalents) would fail cleanly without a lockfile, leaving the tree untouched. that assumption is false for pnpm 11.1.1 against a no-deps `package.json`: the command reports "Already up to date" with exit 0 AND silently materializes an empty `pnpm-lock.yaml` despite the `--frozen-lockfile` flag. the resulting untracked file trips the post-run dirty-tree gate, the agent reads it as "must push uncommitted work", and a spurious "Add pnpm lockfile" PR lands. smoking gun: pullfrog/preview-844-heyapi-oss-bug PRs #1/#2/#3, all auto-opened by the bot against a repo that contains nothing but a one-line README + a no-deps package.json. guard explicitly with an `existsSync` per manager. if the lockfile is absent, skip eager prep entirely with an info log; the agent can install on demand via the `setup` lifecycle hook (which non-frozen `pnpm install` would handle correctly), or just leave deps uninstalled when the prompt doesn't need them (e.g. the O5 "tell me a joke" path). orthogonal to the lifecycle-hook normalization in 0051bd2a — together they cover the full bug 3 surface: - eager prep can't materialize a lockfile (this commit) - setup/postCheckout hooks that rewrite tracked files have the drift discarded before the agent sees it (prior commit) * fix(action): address Pullfrog review on hook normalization two fixes in `executeLifecycleHook` from review on f6f3b32: 1. pre-hook snapshot was `git status --porcelain` which counts untracked files; in practice any repo with pre-existing untracked content (e.g. `.plans/`, an ignored-but-not-yet-gitignored scratch dir, codegen artifacts) would trip the guard and silently skip normalization, defeating the fix. switch to `git diff --name-only HEAD` so the gate measures the same thing the discard targets — tracked-file mods only. pre-existing untracked files are safe regardless because `git restore --staged --worktree .` never touches them. 2. normalization fired only on the happy path; a hook that updated a lockfile then exploded on a peer-dep conflict left tracked drift for the agent. move the call into a `finally` so it runs on success, non-zero exit, timeout, AND spawn failure. the pre-hook guard still protects pre-existing work in every case. * Revert "test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge)" This reverts commit f6f3b325d6bf9a1720754ed1d39d248dab76cfa8. * fix(action): use detect lockfile strategy for eager-prep gate addresses Pullfrog review on be3c207b. two findings, one root cause: - the hardcoded LOCKFILE_BY_MANAGER map missed `bun.lockb` and `npm-shrinkwrap.json`, two managers' accepted lockfile variants. - `existsSync(join(cwd, lockfile))` only checked the immediate directory, breaking monorepo subpackages where the lockfile lives at the workspace root. both fall out by replacing the custom check with `detect({ strategies: ["lockfile"] })`. the detector already walks up the tree (subpackage → workspace root) and recognizes every accepted lockfile name across all managers it supports. restricting to the `lockfile` strategy is load- bearing: the default strategy set also matches on `packageManager` / `devEngines.packageManager` package.json fields, which would return non-null and re-mask the very case we're trying to detect (declared manager, no lockfile committed — the O5 / hey-api preview repro). drops the LOCKFILE_BY_MANAGER map entirely; no need for a second detect() call since the existing one was only used for `agent` resolution and that consumer is now after the lockfile gate, where `detected` is guaranteed non-null. |
||
|
Colin McDonnell
|
585a5d21cc |
fix(askpass): scope code + script lifetime to one $git() call (#841)
* fix(askpass): scope code + script lifetime to one $git() call, not first password prompt LFS pre-push (and any auth-bound sibling subprocess) consumed the single-use code AND triggered the script self-delete, so git's own push call then hit `fatal: cannot exec '/tmp/pullfrog-…/askpass-…js'` and our server treated the legitimate retry as tamper, revoking the installation token. Observed on nteract/nteract#2987 (LFS repo). `gitAuthServer` codes are now `active` until `$git()`'s finally calls `revoke()`; the script no longer self-deletes (finally already unlinks). Replay after revoke still trips 409 + token revocation, which is the realistic exfiltration vector we care about. * fix(askpass): drop wall-clock TTL on active codes Copilot review on #841 noticed the 5-minute CODE_TTL_MS still applied to active codes, which would re-introduce the original LFS failure mode at a different boundary: a large LFS push lasting >5min would hit a 404 mid- call. $git() uses `activityTimeout: 0` precisely because git fetch/push can take arbitrarily long, so any wall-clock TTL on active codes is wrong. Active codes now live until revoke() is called (in $git()'s finally) or the auth server is closed. Revoked codes keep their 60s replay trap. * docs(askpass): purge stale single-use vocabulary; align error message Pullfrog review on #841 surfaced four doc-drift sites that still described the pre-PR single-use model: - wiki/security.md — 3 references (overview prose, bullets, threat-mitigation table) - action/utils/gitAuth.ts — file-level JSDoc + the 409 error message - wiki/askpass.md — error message quoted in the tamper-evident section - action/utils/gitAuthServer.ts — per-prompt invocation comment was ambiguous All updated to match the active|revoked vocabulary; error message is now "askpass code was replayed after revoke, token revoked". |
||
|
David Blass
|
fe2746198c |
fix: 6 unaddressed log-audit / run-audit findings (#840)
* fix: 6 unaddressed log-audit / run-audit findings - #836 + #818 (clerk middleware SyntaxError on action-runtime endpoints): narrow proxy.ts matcher to exclude /api/repo/<owner>/<repo>/run-context, /api/runtime/*, /api/proxy-token. these are server-to-server with their own auth and have no Clerk session to evaluate, so clerkMiddleware's decodeJwt throws turn into 500s on every request. - #837 (npx EBADDEVENGINES on customer's package.json): change runCli's bootstrap cwd from $GITHUB_WORKSPACE to os.tmpdir() so npm v11+ doesn't enforce devEngines.packageManager from the customer's tree before our bootstrap can install pullfrog. CLI process.chdir's to payload.cwd internally, so the runtime work still happens in $GITHUB_WORKSPACE. - #838 (createWorkflowDispatch silently dropped 39 user runs on a 5xx spike): add bounded retry (3 attempts, ~750ms total) on Octokit 5xx and network errors inside dispatchReservedRun. preserves the existing 422 "Unexpected inputs" path. retry budget stays well under GitHub's 10s webhook redelivery window. - #833 (bail() redirect("/signout") propagated NEXT_REDIRECT into webhook handlers, 40+ 500s/24h): drop the redirect side-effect; bail now just classifies bad-credentials as non-retryable and propagates. UI flows that wanted auto-signout on revoked tokens can detect it themselves; the side-effect was wrong for any non-page caller. - #835 (BYOK provider billing-exhausted fell through to raw error renderer, 37 review-mode runs/24h with no PR-side signal): - extend providerErrors patterns with Anthropic "credit balance is too low" + extract isProviderBillingExhausted / extractProviderId helpers - add a renderer branch in runErrorRenderer.ts that names the provider (parsed from providerID=) and links to its billing dashboard - route handleAgentResult's !result.success path through the renderer + reportErrorToComment with createIfMissing, so review-mode and silent triggers get an actionable PR comment regardless of mode - #834 (post-hook ERR_MODULE_NOT_FOUND already fixed on main, hardening): - add a vitest invariant that walks the entryPost.ts import graph and refuses any non-relative / non-node: specifier — catches the next `@actions/core` slip-up before publish - add an analyze-logs classifier so future entryPost crashes surface as failure:post-hook-module-not-found instead of hiding inside failure:unknown / failure:git-lock-file Co-authored-by: Cursor <cursoragent@cursor.com> * anneal: fix dead-code matcher, 404 url, silent-trigger gate, and grammar regression Round-1 anneal pass on the audit-fixes PR surfaced two critical issues + several majors that the original fixes shipped with: - proxy.ts matcher 1 (`(?!_next|[^?]*\.(...))`) still caught every /api/... route because matcher arrays are OR'd. The narrowing in matcher 2 was dead code → middleware still 500'd on /api/runtime/*, /api/proxy-token, /api/repo/.../run-context. Carve-out now lives in BOTH matchers. - opencode.ai/billing returns 404; canonical top-up surface is /zen. deepseek /usage is the consumption page, not the top-up flow (/top_up is correct). google /apikey is the keys list, not billing (/usage is the spend dashboard). All three URL strings updated. - handleAgentResult gated reportErrorToComment behind `if (!ctx.silent)`, contradicting the createIfMissing intent — silent IncrementalReview / pull_request_synchronize / auto-label still got zero PR signal on BYOK billing exhaustion, the exact failure mode #835 was meant to fix. Moved createIfMissing into finalizeSuccessRun's existing render-and- post block (single source of truth), reverted handleAgentResult to its prior shape. Side benefit: drops the double-PATCH that fired on every non-silent !success path with an existing progress comment. - Anthropic-direct error rendered "**Your your provider account is out of credit.**" because Anthropic SDK has no providerID= tag, so extractProviderId returned null and the headline composed "Your " + "your provider". Added detectProviderId Anthropic fallback (matches "Anthropic API" / "credit balance is too low") so the link is reachable AND made the headline conditional on whether a provider id was detected. - Reordered renderRunError classifier: BYOK billing-exhausted now runs BEFORE api-key auth detection. Providers commonly return 401 for billing exhaustion (DeepSeek, Gemini), and the OpenCode harness logs often include "API Error: 401" in the raw error body, which isApiKeyAuthError would otherwise match — surfacing "rotate your key" when the actual fix is "top up credits". - isTransientUpstreamError missed ENOTFOUND / ENETUNREACH / EHOSTUNREACH (undici DNS-class failures Octokit doesn't wrap with a status). Added to the prefix alternation. - Tightened "10s webhook redelivery budget" / "GitHub redelivers" wording in triggerWorkflow.ts and bail.ts JSDoc — GitHub's 10s is the response timeout (it doesn't auto-redeliver); upstream webhook proxy retries are what multiplied the failure. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal r2: unbreak proxy.ts matcher 2, extend carve-out, mkdtemp bootstrap Round-2 anneal (security + cross-cutting lenses) on top of the round-1 audit-fixes commit caught a critical regression + several majors: - proxy.ts matcher 2 from r1 (`/(api|trpc)(?!...)(.*)`) does NOT compile. path-to-regexp rejects a top-level `(?!` after `)` as "Pattern cannot start with '?'", and Next.js's SourceSchema runs the same validator at build time and aborts via `process.exit(1)`. PR #840's Vercel + preview deployments have been failing since 978eca26 for exactly this reason. Fixed by nesting the lookahead inside an outer parameter group: `/(api|trpc)((?!...).*)`. Same shape matcher 1 already uses, which is why m1 always compiled. Verified `pnpm next build` succeeds end-to-end. - proxy.ts carve-out was incomplete relative to its own justification. The "no Clerk session, decodeJwt 500" failure mode applies to ALL server-to-server action-runtime endpoints — five more share the exact shape: /api/repo/.../learnings, /api/repo/.../pr/.../summary-comment, /api/repo/.../issue/.../plan-comment, /api/workflow-run/, and /api/github/installation-token. Extended both matchers. /api/upload/ signed-url stays in (dual auth: Clerk session OR bearer JWT — needs middleware for the user path). - proxy.ts carve-outs were unanchored: a future /api/proxy-token-info, /api/proxy-tokens, or /api/repo/X/Y/run-context-foo would silently bypass Clerk. Added `(?:$|/)` for path-prefix carve-outs (allow exact match or sub-path), `$` for routes with browser-callable siblings (e.g. `learnings/history` is browser-Clerk, `learnings$` is action- bearer-JWT). Verified 30/31 routes via path-to-regexp test harness. - runCli.ts cwd flipped from $GITHUB_WORKSPACE to os.tmpdir() in r1 (#837 fix for npm v11 devEngines.packageManager EBADDEVENGINES). But $TMPDIR is overridable from a prior $GITHUB_ENV step — a customer- authored or compromised prior step can plant /atk/node_modules/ pullfrog/ and `echo "TMPDIR=/atk" >> $GITHUB_ENV`, and our npx --yes pullfrog@<v> bootstrap resolves the local install first, executing attacker code with full action env (provider keys, OIDC, installation token, CODEX_AUTH_JSON). Switched to mkdtempSync(join (tmpdir(), "pullfrog-bootstrap-")) — fresh per-invocation 0700 dir, not pre-writable by anything earlier in the job. - runLifecycle.ts: writeRunErrorOutputs (catch-path) didn't pass createIfMissing: true, contradicting the symmetric intent of the r1 finalizeSuccessRun fix. Silent triggers (IncrementalReview / pull_request_synchronize / auto-label) that throw past the success path still got zero PR signal — exact failure mode #835 was meant to close. Now both paths pass createIfMissing: true. - runErrorRenderer.ts detectProviderId regex `/Anthropic API|credit balance is too low/i` could mis-tag a non-Anthropic billing-exhausted error that mentioned "Anthropic API" in passing (fallback-chain agent prompt text, OpenCode harness logs). Tightened to /credit balance is too low/i — Anthropic-specific phrasing, sufficient for the direct- Anthropic SDK case the fallback exists to handle. - runErrorRenderer.ts JSDoc: r1's classifier reorder put hang at #6 in code but the JSDoc still listed it at #2. Reordered the doc to match dispatch order, with explicit note that hang is a sub-source for the api-key check (which is why hangBody is precomputed early). - analyze-logs.ts: ERR_MODULE_NOT_FOUND.*entryPost regex needs `s` flag so it survives Node v23+ stack-trace reformatting onto multiple lines. One-char fix to defend the #834 classification bucket. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Colin McDonnell
|
dc4dff98da |
allowlist middleapi/orpc for OSS, ship skill scaffold + no-mock audit
- utils/ossRepos.ts: add middleapi/orpc (orpc.dev, 5.2k stars) to the oss allowlist so any future install mints via mintOssKey ($10 cap, pullfrog absorbs). - scripts/skill.ts + pnpm skill: scaffold .agents/skills/<name>/SKILL.md + .claude/skills/<name> symlink. patch + skill skills written using it. - AGENTS.md: hard-ban vi.* mocking apis; document pnpm skill workflow. - audit follow-through: drop pure-mock test files (action/utils/lifecycle, action/utils/timer, test/handleNoInstall) and trim action/mcp/review to the non-mock cases. - wiki/scripts.md: row for scripts/skill.ts. |
||
|
Colin McDonnell
|
fd2c67ab50 |
adhoc: push:restricted adversarial pentest (#827)
* adhoc: push:restricted adversarial pentest
enumerates the 16 attack vectors the deep audit identified as load-bearing
for `push: restricted`. used to drive e2e verification against the preview
repo's pullfrog.yml; also runnable via pnpm runtest locally.
validator only asserts that the repo's default branch SHA didn't move —
the per-attack outputs are the deliverable for human review (the test
exists to feed adversarial runs, not to be a CI guard).
* wipe runner leak surface before agent spawn
the GHA runner persists credentials inside $RUNNER_TEMP that an MCP-shell
agent can grep — _runner_file_commands/set_output_* (from any composite
step that called core.setOutput, e.g. pullfrog/pullfrog/get-installation-token
which leaks a ghs_… installation token), <uuid>.sh rendered step scripts
(whose run: | body embeds ${{ ... }} expressions literally before write),
and git-credentials-*.config from actions/checkout@v6.
snapshot-and-delete that surface at action startup, after our own token is
in memory and before setupGit. preserves $GITHUB_OUTPUT, $GITHUB_ENV, and
$GITHUB_STATE so pullfrog's result output and post: hook still work.
setupGit's existing removeIncludeIfEntries call strips the matching
dangling includeIf.gitdir:....path entries from the user's .git/config.
does not tighten isGitCommand — that's a UX guard, not a security
boundary, and trivially bypassable via bash -c, absolute paths, symlinks,
python subprocess. the security boundary is the absence of credentials on
disk for those bypassed shells to authenticate with.
verified end-to-end by re-firing action/test/adhoc/pushRestrictedAdversarial
against pullfrog/preview-827-push-restricted-pentest.
* preserve all runner file-command paths from wipe
addresses pullfrog review on f7f5143b: GITHUB_STEP_SUMMARY also lives at
$RUNNER_TEMP/_runner_file_commands/step_summary_<uuid> and is read by the
runner AFTER our step exits to render the job summary in the GH UI. wiping
it silently broke pullfrog's job summary output. preserve GITHUB_PATH too
for symmetry — it's the same allocation pattern, and a step or post hook
that appends a directory expects the file to exist.
set of file-command env vars enumerated in @actions/core:
GITHUB_ENV, GITHUB_OUTPUT, GITHUB_PATH, GITHUB_STATE, GITHUB_STEP_SUMMARY
|
||
|
Colin McDonnell
|
e2eb26573f |
surface agent failures in job summary (#632) (#802)
* surface agent failures in job summary (#632) when the agent harness returns `{success: false, error}`, main.ts went through the success path so the catch block — which renders the `### ❌ Pullfrog failed` banner via renderRunError — never fired. result: the GitHub Actions job summary showed only the partial body + usage table, no error block. the progress comment had a narrow workaround that re-implemented the api-key classifier inline. unify: in finalizeSuccessRun, call renderRunError once when `!success`, use `.summary` for the job summary (prepended to the existing body/usage parts) and `.comment` for the progress comment. removes the duplicated isApiKeyAuthError / formatApiKeyErrorSummary branch and picks up BillingError reclassification + hang body for free. * docs: note dual-surface failure rendering in finalizeSuccessRun * fix copilot review nit: clarify which renderRunError classifications carry the H3 banner |
||
|
Colin McDonnell
|
01e4daa0b5 |
checkout_pr: refuse unconditionally on dirty working tree (#808)
* checkout_pr: refuse unconditionally on dirty working tree drop the live-HEAD comparison from the guard introduced in #796. any checkout_pr call with staged or unstaged changes now throws, even when HEAD is already on pr-N. no stashing, no idempotent escape hatch. motivation is the zed-industries/cloud (2026-05-18) incident: shared-cwd subagents make "carry edits along" semantics dangerous, and the HEAD-equality predicate let a re-checkout silently inherit working-tree state from a sibling agent. forcing commit/discard before any PR-context operation eliminates the entire carry-forward failure class. error names the PR number, lists dirty paths, and tells the agent to commit/push/restore/clean before retrying. * improve dirty-tree error: precise discard commands copilot caught two sloppy bits in the error string: - "push" alone does not clean a dirty tree (needs commit first) - bare `git clean` is a no-op without `-fd` reword to "commit (then push if needed), or discard with `git restore --staged --worktree .` / `git clean -fd`" so the guidance is actually actionable. * checkout_pr: initial-branch invariant setupGit captures `toolState.initialBranch` at run start via live `git rev-parse --abbrev-ref HEAD`. checkout_pr refuses unless current HEAD matches the run-entry branch or the target `pr-N` (idempotent same-PR re-checkout). uses live rev-parse, not toolState.issueNumber (poisonable per the PR #796 review). refusal error names the current branch, target PR, recovery path (`git checkout <initialBranch>` with the literal branch name), and explicitly states routing around via the `git` tool is not sanctioned. closes the zed-industries/cloud (2026-05-18) shape where a subagent parked HEAD on someone else's `pr-X` and the orchestrator's next checkout_pr inherited that position. * reviewfrog: enforce canonical diff + pre-commit halt; align Build dispatch extend REVIEWER_SYSTEM_PROMPT with two prepended HARD CONSTRAINTS: - first action MUST be `git diff origin/<base>` (single-rev, captures uncommitted). no other diff first; no checkout_pr; no alt-ref fetches; no branch listing; no `gh pr list`. - empty canonical diff + claimed-changes dispatch ⇒ reply exactly with `no changes detected — likely pre-commit Build self-review; orchestrator should commit then re-dispatch` and stop. do not guess PR numbers (the zed thrash that ended in `checkout_pr({2582})`). reshape Build mode reviewfrog dispatch step around a verbatim template that names: (a) the situation is pre-commit, (b) canonical diff command, (c) halt-on-empty-diff rule. orchestrator side now says the same thing as the reviewer's baked-in prompt. delegation-discipline bullets and orchestrator-evaluation guidance kept intact. * checkout_pr: handle detached-HEAD entry in initial-branch invariant pullfrog incremental review caught a defense-in-depth gap: `git rev-parse --abbrev-ref HEAD` returns the sentinel string `"HEAD"` on detached entry, which is the default `actions/checkout` state for `pull_request` events. with the previous string-typed `initialBranch`, both the captured value and the live probe would equal `"HEAD"` on any detached state, trivially satisfying the invariant — including a subagent doing `git checkout --detach <sha>`. discriminate the captured HEAD: probe `git symbolic-ref --short HEAD` first (works on named branches), fall back to `git rev-parse HEAD` (SHA) on detached entry. store as `{ kind: "branch"; name } | { kind: "detached"; sha }`. checkout_pr runs the identical probe at call time and compares like-with-like (branch name vs branch name, SHA vs SHA). refusal error renders both heads via a small `describeHead` helper and chooses the right `git checkout` recovery target (branch name or SHA). no inline-discriminant `as` casts — uses a top-level `headsEqual` that narrows via the discriminator. |
||
|
David Blass
|
d3b5340583 |
fix: audit batch — MCP timeouts, entryPost, vip_audit 404s, and 6 more (#824)
* fix: 9 unaddressed log-audit / run-audit findings Co-authored-by: Cursor <cursoragent@cursor.com> #815 entryPost stdlib-only imports; #823 MCP timeoutMs on checkout_pr/shell; #816 FREE_FALLBACK → opencode/big-pickle; #822 chunk GraphQL nodes ≤100; #817/#821 vip_audit 404 skip paths; #813 longer serializable retries; #818 run-context handler-entered log; #805 audit severity template. * fix: update footer test for big-pickle fallback slug Co-authored-by: Cursor <cursoragent@cursor.com> * fix: anneal round 1 — ghaCore getState casing, post-hook timeout Match @actions/core STATE_ key semantics (no uppercasing), cap postApiFetch at 30s, trim serializable retries to stay under GitHub's 10s webhook window, log Clerk failures in getUserTokenByGithubLogin. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: drop run-context handler log (#818 deferred) The #692 client-side fix is already on main; residual SyntaxError hits are old action pins. Per-request log added noise without fixing anything. Co-authored-by: Cursor <cursoragent@cursor.com> * document per-issue Closes syntax for audit PRs GitHub only auto-closes the first issue when numbers are comma-separated; /audits and AGENTS.md now require Closes before each issue number. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: anneal round 2 — outreach privacy, alert resilience, vertex cleanup Filter private repos from VIP authority output, harden console alert lines against DB failures, drop spoofable changesets body check, and unset GOOGLE_APPLICATION_CREDENTIALS after vertex credential cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: drop codexHome re-export of detectCodexRefresh Import detectCodexRefresh directly from codexRefreshDetect.ts everywhere; rename the unit test file to match. codexHome.ts stays install-only. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: drop deprecated minimax-m2.5-free; add paid minimax-m2.5 Remove the deprecated free MiniMax promo from the catalog, docs, and tests. BYOK fallback and picker copy stay on opencode/big-pickle. Add opencode/minimax-m2.5 and openrouter/minimax-m2.5 for Zen BYOK and Router. Pin #816 regressions with freeFallbackCatalog and runErrorRenderer unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: hidden minimax-m2.5-free fallback for stored slugs Re-add opencode/minimax-m2.5-free as a hidden fallback alias to big-pickle so repos with the legacy slug still resolve as free. Drop live Zen API experiment tests in freeFallbackCatalog.test.ts. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Colin McDonnell
|
58e5b74cb8 | Update footer test for big-pickle BYOK fallback. | ||
|
Colin McDonnell
|
c43ed65c3b |
Add Vertex AI routing support (#753)
* add Vertex AI routing support * include Vertex smokes in action CI |
||
|
Colin McDonnell
|
a0576a702a |
opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767)
* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite
Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.
Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.
Surface area:
- drop `init` / `message` / `result` / `tool_result` event types and
handlers (no longer emitted at v1.14+ per upstream
`cli/cmd/run.ts:588-601`).
- `tool_use` is now a single event covering both `state.status:
"completed"` and `"error"`. duration / subagent-finish bookkeeping
moves from the v1 `tool_result` handler into the consolidated
`tool_use` handler.
- new `reasoning` event handler — gated on `--thinking`, surfaces
Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
`baseArgs`.
- drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
v1.15 the `task` tool callID is stable across the whole
`tool-input-* → tool-call → tool-result/tool-error` chain
(`session/processor.ts:282-330`). exact-match map is sufficient.
- drop `experimental.batch_tool: true` from injected config — declared
but inert at v1.15. re-add once upstream wires it back.
- bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
the platform-specific binary into `opencode.exe` for every OS now).
Validated locally:
- `pnpm test` 610/610 ✓
- `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
- `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
(this is the headline `thought_signature` fix)
- runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
- runtest opencode: skill-invoke and mcpmerge fail (model-behavior
drift on the new system prompt; wiring confirmed intact via direct
repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
Tracked for follow-up; does not gate the migration.
Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.
* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)
The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.
* opencode v2: set PWD env explicitly to fix skill / project-config discovery
Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:
const root = Filesystem.resolve(process.env.PWD ?? process.cwd())
We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.
Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.
Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).
* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly
opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).
Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.
Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now: `» thinking (4.0s): <preview>` (one line, sourced)
For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.
Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.
* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment
claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.
Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.
Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.
* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup
Code-quality pass on the v2 work:
1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
between v1 and v2:
- `OpenCodeConfig` type
- `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
- `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
and re-imported by v2 via a back-reference)
- `installOpencodeCli({ binPath })` (parameterized — v1 passes
`bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
`installCli` lambda; matches each pinned version's npm shape)
- `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback
v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
keeps a one-line `export { geminiHighThinkingOverrides }` re-export
so `opencode.test.ts` keeps working unchanged. Once v1 is retired
(post burn-in) opencodeShared collapses back into v2.
2. `opencode_v2.ts` cleanup:
- drop dead state (`currentStepId`, `stepHistory` were write-only —
their reader was the v1 `tool_result` handler we deleted)
- hoist `state` in `tool_use` handler; replace nested-ternary payload
extraction with a `terminalPayload(state)` helper
- extract `formatPartDuration(time)` for the reasoning-block
"(X.Ys)" suffix
- tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
`callID` fields directly, drop the `partWithToolFields` cast
- trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
handler, tool_use handler, bus envelope handler all shortened
- `step_start` becomes an explicit `() => {}` no-op so the dispatcher
doesn't log "unhandled event" for every step
3. `subagentRegistration.test.ts` retargeted at the new file split —
reads opencodeShared.ts for the buildReviewerAgentConfig assertions
and opencode_v2.ts for the orchestrator-model wire-through.
Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.
* opencode v2: address PR review feedback
Three fixes from the inline review threads on #767:
1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
level `markActivity()` resets the module-level idle counter, so the
per-event `getIdleMs()` sample inside the dispatch loop was always
~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
a runner-local `lastEventAt` so we measure real event-to-event silence
instead of chunk-arrival latency. Drop the unused `getIdleMs` import.
2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
`lastProviderError`, and `recentStderr` are closed over by the
`handlers` const but were declared after it. No current bug because
handlers only fire inside the awaited `spawn()`, but a future
refactor that triggers a handler synchronously during setup would
surface a TDZ. Hoisted above `handlers`.
3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
L566): leave a `TODO` comment marking the gap until `AgentUsage`
grows a `reasoningTokens` field — separate PR with schema work.
Cost totals stay correct because `part.cost` is summed independently.
Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
`reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
the bus-envelope flow; the plugin filters orchestrator events except
for status=running task dispatches, and bus-envelope returns before
calling handlers.tool_use on those. No double-fire under current code.
Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.
* DX: flip pnpm play / pnpm runtest to docker-by-default
Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.
scripts (root → action):
- pnpm play → pnpm -C action gha play.ts (docker, default)
- pnpm play:local → pnpm -C action play:local (host)
- pnpm runtest → pnpm -C action gha test/run.ts (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)
action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.
CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.
Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.
File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
`pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers
AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.
wiki/docker.md unchanged — already described the now-real shape.
* test/crossagent: add codex-auth smoke
Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:
- installCodexAuth() materializes auth.json under the test HOME
- opencode routes openai requests through ChatGPT subscription auth
(no OPENAI_API_KEY in env, AT path forced via expires: 0)
- the refresh chain advances during the run (refresh_token rotates)
- detectCodexRefresh() would surface the rotation to entryPost.ts
The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.
CI wiring (already committed in a1c1fd4f as part of the DX flip):
- .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
action-agents env block
- action/.github/workflows/test.yml: same; codex-auth in the
hardcoded test matrix with a claude exclude
The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.
ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.
* docs(codex-auth): make storage requirement unmissable
A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).
- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
loudly contrast Pullfrog secret store vs GH Actions and explain the
writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
unchanged per scope.
* auth codex: auto-open device URL, drop --scope flag
- detect `https://auth.openai.com/codex/device...` from codex CLI output
and best-effort launch it in the user's default browser (open / xdg-open
/ cmd start, wslview fallback on linux). gated so we only open once per
flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
interactive (browser approval), so a "skip-the-prompt" flag for just one
of the prompts was dead weight. collapses scope selection to "always
prompt on org-owned, always account on user-owned".
* rename gha→docker, flip play/runtest defaults to host
the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.
- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
`pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
`README.md`, `AGENTS.md`), CI workflows
(`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
and code headers (`action/{play,test/run,utils/runFixture}.ts`,
`webhook.ts`, `action/test/coverage.ts`)
`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).
* fix(codex): route post-hook writeback through apiFetch + conditional skip
Three threads addressing PR #767 followups.
action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).
action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.
doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
mention (removed in 10be96db, scope is now always interactively
prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
is agent-gated (opencode/opencode_v2 harness), not model-gated;
opencode runs with non-OpenAI models still materialize the file,
it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
→ pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
instead of the dropped --scope flag.
* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS
upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.
also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.
local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.
---------
Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>
|
||
|
David Blass
|
4d1fd5ea1a |
fix: 4 unaddressed log-audit / run-audit findings + close 10 already-resolved issues (#785)
* fix: 4 unaddressed log-audit / run-audit findings closes 4 issues with code changes; 7 issues are already addressed by #769 and 3 are deferred — see PR description. #782 Anthropic 401 → `isApiKeyAuthError` now matches the direct-Anthropic 401 shape (`Failed to authenticate. API Error: 401 ...`, `authentication_error`, `Invalid bearer token`, `api_error_status=401`) so revoked / mistyped / rotated `ANTHROPIC_API_KEY` users see the formatted rotate-key CTA instead of a raw 401 JSON dump. #778 billing-class provider errors → `providerErrors.ts` now classifies `CreditsError` / `FreeUsageLimitError` / `Insufficient balance` / `spending cap` as `provider billing exhausted` *before* status-code patterns can win and tag them as transient `auth error (401)` / `rate limited (429)`. `agentHangReport.ts` swaps the bare "Pullfrog stalled — auth error" headline for a billing-specific CTA (extracts the provider's billing URL when present). #775 silent IncrementalReview swallows `BillingError` → `reportErrorToComment` now optionally falls through to creating a fresh issue comment on `toolState.issueNumber` when no progress comment exists. Wired with `createIfMissing: true` from the `BillingError` / `TransientError` paths in `proxy.ts` so silent triggers (`pull_request_synchronize`) finally surface the router-balance signal on the PR instead of only in the GH job summary. #773 `currentUser()` inside `after()` → `fillInstallerIdentityIfMissing` is split into `resolveInstallerIdentity` (must run inside the request body) and `fillInstallerIdentity` (DB-only, safe in `after()`). The `/console/[owner]` caller now resolves Clerk identity up-front and defers only the prisma write, fixing the broken installer-identity backfill on org-console first-admin visits. Co-authored-by: Cursor <cursoragent@cursor.com> * add /audits cursor command for triaging run-audit + log-audit issues Co-authored-by: Cursor <cursoragent@cursor.com> * review prompt: tighten body-section bar + inline technical-details (#770) * review prompt: tighten body-section bar + add inline technical-details Two layers of tightening to the Review/IncrementalReview prompts in PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step): 1. Reframe inline-vs-body split. Body `### ` sections are now reserved for concerns that genuinely have no line to anchor to — absence, sequencing, design decisions, scope questions, architectural risk. Drop the "cross-cutting concerns" framing (misled the agent into either filing nothing in the body or filing multi-file anchored findings there). 2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step 6) and IncrementalReview (step 8) aggregate phases. Diagnosis from PR #767's auto-review: on substantial PRs the agent surfaced findings but routed all of them inline, producing reviews with zero `### ` body sections even on diffs where non-anchored concerns clearly existed. 3. Replace the abstract `### ` example with a concrete non-anchored one ("Legacy `opencode.ts` has no documented deletion plan") so the agent pattern-matches the absence-shaped finding, not a line-bug. 4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT so inline comments can carry a `<details>Technical details</details>` block when the fix has cross-file implications. Rename the existing "Agent details" inline collapsible to "Technical details" for consistency with body sections. 5. (Carried over from prior uncommitted work) Restructure the review metadata block from `<details>Review metadata</details>` into an HTML comment + an italic TL;DR commit-range line. The HTML comment keeps the metadata addressable for downstream agents without eating user-visible review real estate. No tests touched. * wiki: document multi-model end-to-end eval pattern * feat(promo): cookie-stashed promo codes for onboarding rewards (#771) * feat(promo): cookie-stashed promo codes for onboarding rewards Operator hands out a link like https://pullfrog.com/start?promo=FROGGY; middleware validates the code against an in-code registry, stashes it in an HttpOnly cookie, and the install callback applies the reward once the GH-side account exists. v1 reward: unlimited_runs (lifts the monthly free-runs cap to 1M, same convention prod-grandfathered accounts use). No schema changes. Idempotent across reinstalls via the lte: 100 gate. * fix(promo): integrate handler into existing proxy.ts (Next 16 rename) * docs(promo): clarify sentinel + sync plan doc with renamed paths * feat(promo): add FOUNDATIONS code * feat(promo): show applied promo code in console * refactor(promo): move cookie set to client-side * docs(promo): point JSDocs at PromoCookieSetter, not proxy.ts * billing: cap counts only successful runs (#787) * billing: cap counts only successful runs `reserveRun` was counting `WorkflowRun` rows regardless of status against `Account.includedMonthlyRuns`. Failed / cancelled / skipped / timed-out runs consumed cap slots even though their `billableCents` got zeroed on the completion webhook — pushing paying users into billable territory earlier than the contract implies. `inthhq` paid for 2 extra runs this month because 2 failed runs ate 2 of their 100 free slots. Cap query now filters on `CAP_CONSUMING_STATUS = "success"`. Only runs that actually deliver value consume slots; in-flight (`running`) runs hold no slot until they terminate as success (burst-bypass risk is theoretical given GH Actions concurrency limits). Shared constant lives in `utils/billing.ts` and is used in lockstep by three call sites: `reserveRun` (live cap gate), the billing API's `runsThisMonth` (dashboard progress bar), and the billing-report script's `cap` column. Script's `cap` cell was also broken independently — it compared `monthBillableRuns` (overage count) against `includedMonthlyRuns` (free cap), so `inthhq` rendered as `125/100 (over)` when the meaningful ratio is `223/100 (over)`. Fixed to use `mRuns/cap`, which is the same predicate the live billing path uses. * move CAP_CONSUMING_STATUS to workflowRunStatus.ts + wire script through it Per copilot review: the JSDoc claimed the billing-report script used the constant in lockstep, but the script kept `status: "success"` inline. The script imports from raw-node ESM and can't pull in `next/server`, so it couldn't import from `utils/billing.ts`. Moved the constant to `utils/workflowRunStatus.ts` (already Next-free, already the home of `CONCLUSION_VALUES`) and updated all three call sites to import from there. Script's `mRuns` query now uses `CAP_CONSUMING_STATUS` directly, making drift impossible. * learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743) * learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy three audit fixes on top of the recent learnings overhaul (#717): - `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry when a body has non-whitespace content before the first heading. the prompt instructs the agent NOT to slurp the whole file when a TOC is present, so without this any preamble lines were silently invisible (realistic transitional case: an agent partially restructures a legacy free-text body and leaves bullets above the first `## `). - server-side PATCH route now applies the same line-boundary-aware truncation as the action (defense in depth via a shared `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from `action/internal`). the raw `.slice` it used before could leave a mid-heading tail on any caller that bypassed the client-side truncate, breaking the next-seed TOC parse. removes the duplicated cap constant. - `buildLearningsSection` intro no longer asserts "accumulated by previous agent runs" — false for fresh repos with zero history. new copy is tense-neutral and works for empty + populated bodies. also nudges the agent to re-read after mid-run edits (the inlined TOC ranges are a run-start snapshot). Co-authored-by: Cursor <cursoragent@cursor.com> * learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned calls discovering a quirk this run, recording the workaround prevents the next run from repeating the waste. Reframe around one litmus ("would a future run do its work better because this bullet exists?") and trust it to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary) and the four-example pullfrog/PR/date/play-by-play list (the rule underneath is "don't anchor facts to repo state that will move"). Cuts ~10 lines from a prompt the model was already mostly ignoring; the remaining anchor list is narrower and more enforceable. * audit-learnings-r2: align wiki + tighten re-read nudge - wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls. - buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly. * postRun: refresh JSDoc to match the reflection prompt rewrite `buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets. * fix(mcp/issueEvents): narrow event.event before Set.has lookup octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup. * learnings: split truncation helpers into MCP-free module re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph. move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> * trim first-run celebration email to short personal note drops the feature-dump bullet list (custom review instructions, github iteration walkthrough, security model) — wrong moment to teach. keeps the congrats, the reply CTA, adds discord/x links, keeps the router credit P.S. handler no longer needs the workflowRun→repo lookup. * signup-report: per-bucket histogram Adds a UTC-aligned signups-per-bucket histogram between the overview block and the company-email list. Empty buckets are pre-filled with 0 so dry spells render as gaps. New `BUCKET=hour|day` env flag with a smart default (hour if window ≤ 48h, else day). Histogram is also included in the JSON payload under `histogram: [{key, count}, ...]`. * signup-report: drop hourly bucket, day-only histogram * feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748) * feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660) Per-account ceiling on the sum of `router_topup` invoices (pending + succeeded) for the current UTC calendar month. Closes a gap where a runaway agent loop, leaked PR trigger, or stuck workflow could auto-reload indefinitely with no aggregate per-month ceiling. Two enforcement modes via `RouterLimitMode` enum: - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun; 402 `router_monthly_limit` from /api/proxy-token; email + banner - `alert_only`: auto-reload keeps flowing; email + banner only, first breach per UTC month Enforcement is split across reserveRun (pre-dispatch paywall comment) and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through the same `getRouterSpentThisMonthCents` helper so the dashboard, the dispatch gate, and the auto-reload gate can't disagree. Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string), claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent reloads breaching together send exactly one email. Read-time comparison with the current month re-arms on rollover — no cron. Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell) above the Router/BYOK tabs in `ModelAccessCard`, with a popover "Adjust limit" form that PATCHes the existing /api/account/[owner]/billing/settings route. Same `assertBillingAdmin` gate that owns the other billing settings — no new auth surface. See wiki/billing.md § Router monthly spend limit for the full contract + edge cases. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal pass on monthly Router spend limit (#660) Round-1 review across 5 lenses (billing-subsystem, correctness, security, operational-readiness, research-validated-assumptions) surfaced one critical + three actionable major findings on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot` used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles to `field != value` — UNKNOWN (not TRUE) against the post-migration `NULL` default. First breach for any account would never claim the slot, never stamp the row, and never fire the email (hard_cap or alert_only). Replaced with `OR: [{ field: null }, { field: { not: monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern. **Major — email gap on manual-top-up over cap.** Breach email was only wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>` that crosses the cap blocks dispatch via `reserveRun` but never hits proxy-token, so the user got the PR comment but no email. Wired the CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s PaywallError catch (the SERIALIZABLE txn rolled back when we threw, so we re-claim with the global client; single-statement CAS is its own race boundary against concurrent proxy-token claims). **Major — PR paywall comment leaked $ figures.** `router_limit` body embedded `($X of $Y)` in a comment visible to anyone with PR read access (public repos, forks, outside collaborators). Other paywall types deliberately avoid amounts. Removed; deep link still points to the authenticated console for the figures. **Medium — observability.** Added `[router-limit]` structured logs at the three enforcement sites (proxy-token hard_cap 402, proxy-token alert_only breach, reserveRun paywall) so on-call can grep "did the cap fire for customer X this month." **Medium — customer docs.** Added a `### Monthly spend limit` section to `docs/billing.mdx` (Mintlify) describing the two modes and the manual-top-up caveat. **Doc — refund/dispute interaction.** Documented in `wiki/billing.md` that the cap inherits the existing webhook semantics: disputed `router_topup` drops from the sum (cap briefly un-trips); refunds don't flip status today so refunded top-ups keep counting. Matches wallet behavior — not redefined here. Accepted as-is (documented or pre-existing): `after()` reliability vs stamp-before-send tradeoff, alert_only email fires before Stripe phase-2, proxy-token reads limit fields outside SERIALIZABLE scope (brief TOCTOU on admin lowering cap), stale paywall comment on cap clear, no global kill switch (per-account `alert_only` flip is the practical kill switch), no audit log on cap changes (no existing audit infra), action version not bumped (separate release commit). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal round 2 on monthly Router spend limit Round-2 anneal (billing-subsystem, correctness, research-validated, user-journey, operational-readiness) surfaced a critical merge conflict and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted `formatBillingErrorSummary` from `action/main.ts` to `action/utils/billingErrors.ts`. The PR's new `router_monthly_limit` arm still lived in `action/main.ts`. Took main's slim orchestrator wholesale; moved the arm into the extracted file. **Major — cap = payments only, not dispatch.** `reserveRun` was pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit` regardless of wallet balance, contradicting the cap's positioning as "ceiling on what you pay." An account with $500 of paid-up wallet and a breached $100 cap couldn't trigger any new run via the comment path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded — surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token` is now the sole enforcement point, refusing only the next auto-reload that would push past. Wallet credit always drains. Dropped the now-dead `router_limit` arm in `buildPaywallCommentBody`, the dead `routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`, and the post-paywall email-fire David added — all unreachable. **Major — split `manual_topup` from `router_topup`.** Manual on-session top-ups at `/billing-top-up/<owner>` were landing as `Invoice.kind = "router_topup"` and counting toward the cap. The cap exists to brake *passive* runaway (auto-reload loops); a manual top-up is a deliberate click-through that the user owns. Added `InvoiceKind.manual_topup`, flipped the manual write site + `createTopUpCheckoutSession` metadata, broadened wallet / reconcile / billing-report reads to `kind IN (router_topup, manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap aggregate) to `router_topup` only. Worked example: cap=$300, reload=$100 → exactly three reloads succeed; a fourth is blocked. Historical rows stay labelled `router_topup` (no backfill); the asymmetry is small and accepted since the manual flow only existed alongside auto-reload for a brief window. Extended the `invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows the same shape as `router_topup` (PaymentIntent-backed, no stripeInvoiceId); split into a second migration because PG forbids using a freshly-added enum value in the same transaction. **Major — email reframed around the triggering reload event.** The `alert_only` body was reporting a pre-eager-write `spentCents` while the dashboard reads the post-commit value, so email and dashboard disagreed by exactly one reload. Both flavors now say "Your most recent $50 auto-reload brought you over your $300 monthly limit" instead of a running spent-of-cap total — no reconciliation needed, no more "you've hit your monthly cap" copy firing for partial breaches (spent=$80 of $100, reload=$30 would have triggered that wording). **Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded "You've used your 30 free runs this month. Add a card to continue at 7¢/run." regardless of `detail.reason`. Branched on `cap` vs `delinquent` so each paywall surfaces actionable copy with the right CTA. `router_limit` no longer flows through here (per F4 above). **Major — RouterLimitBanner.** Added an `isAlertBreached` visual state (amber palette) so an `alert_only` account at $240 of $200 no longer renders in the same neutral zinc chrome as a healthy under-cap account. Updated popover copy to reflect the auto-reload-only scope. **Medium — paywall log line.** Added `detail.reason` to the `[Installation X] paywall:` log so on-call grepping for "why was this paused" can distinguish `cap` from `delinquent`. **Cleanup.** Dropped dead `utcMonthKey` import + re-export in `maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*` fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*` since they now handle both kinds. Updated wiki/billing.md + docs/billing.mdx + schema doc comments throughout. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt` sitting next to it — a single-purpose state column on `Account` that encoded a date as a string and required a custom CAS predicate to read/write race-safely. Plus it had real holes: Resend send failure left the sentinel stamped and the account silently un-emailed for the month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered edge cases never fired at all. Replace it with: fire `maybeNotifyRouterLimit` on every breaching reload, let the Resend `Idempotency-Key` `router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside Resend's 24h dedup window. Continuously-breaching accounts get ~1 reminder per day; brief Resend outages self-heal because the next breaching reload re-attempts the send. Mode is in the dedup key so `alert_only → hard_cap` mid-month re-arms a fresh email with the appropriate copy. Drops `Account.routerLimitNotifiedMonth` and `claimRouterLimitNotificationSlot`; simplifies the proxy-token phase-1 branch significantly. Net diff is negative LOC and the data model loses a single-purpose sentinel. Migration was branch-local — never deployed — so I edited the original add-cap migration in place to drop the column from the ALTER TABLE rather than chain a drop-column migration on top. Preview Neon branches reset automatically on history rewrite per wiki/migrations.md. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): hide RouterLimitBanner when no cap is configured The banner was unconditionally rendered for every billing-enabled account, including pure-BYOK admins who never touch Router. They got "No monthly spend limit / Router has spent $0.00" + a divider as visual noise on the model access page — basically nagging them to set a feature they may not want. Running without a cap is valid; we don't nag. ModelAccessCard now gates the banner block (banner + dividers) on `routerMonthlyLimitCents !== null`. RouterLimitBanner drops the no-limit visual state, the "Set monthly limit" CTA text, and the dead `hasLimit` branching. Cleaner three-state shape (under cap / amber breached / brick breached). Discoverability: no-cap users no longer see a UI affordance to set one. That's deliberate — the cap is a power-user feature documented in docs/billing.mdx. If discoverability becomes an ask, we can add a small inline link inside RouterWalletSection without bringing back the always-visible banner. Resolves the only outstanding finding from cursor bugbot's review of ff5328c (banner-visible-for-byok thread). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(billing): docs/wiki match new "no banner without a cap" reality Pullfrog bot review of f7672ca pointed out the customer docs still told users to "Set the cap from the **Monthly spend limit** banner in the **Model costs** card" — but after hiding the banner for no-cap accounts there is no such banner to use until you already have a cap. Catch-22 for first-time setup. Rewrote docs/billing.mdx to be self-contained: explain what the cap is, what the two modes do, what the banner shows *once configured*, and direct admins to PATCH the billing settings endpoint (or reach out to support) for first-time setup. Cap is positioned as optional; running without one is the documented default. Wiki paragraph in wiki/billing.md updated to match — banner is only rendered when a cap exists, three visual states (under / amber / red), no first-time-setup UI nag by design. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely The standalone `RouterLimitBanner` was the wrong shape. It only rendered when a cap was already configured (so there was no UI to discover the feature in the first place — first-time setup required hitting the API directly), and it occupied prominent real estate above the tabs to surface state that already lives in the row's own input when the form moves down where it belongs. New shape: monthly cap is just a third row inside `RouterWalletSection` sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated the same way (card on file + auto-reload enabled — the only state where the cap actually means anything). Empty input → no cap, with placeholder "No limit". Setting a number reveals a **Behavior at limit** toggle built on the same `Tabs` slider component used for the Router/BYOK tab switch, so the look matches the rest of the card. Deletes: - `RouterLimitBanner` component (212 lines) - banner mount + conditional + spacers in `ModelAccessCard` - `AlertTriangle` is still imported (used by `DelinquencyBanner`) Adds: - one settings row in `RouterWalletSection` with the cap input + mode tabs - `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the existing `saveSettings` helper (widened to accept `string | null`) - `Tabs` / `TabsList` / `TabsTrigger` import Docs + wiki updated to match the new shape; the customer doc no longer points at a banner that won't appear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between Previously bundled both into one row block. Restructure: cap input is its own row; Behavior-at-limit Tabs gets a sibling row with the standard `h-5 + hr + h-5` separator between (matching the rhythm of auto-reload amount → threshold → monthly cap). Mode-toggle row is gated on `routerMonthlyLimitCents !== null` so the hr + tabs only appear once a number is in the cap input. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row Same `flex items-center justify-between gap-3` layout as the Auto-reload row: label group on the left, control on the right. Drops the vertical stack in favour of the horizontal one — looks identical to the toggle row directly above. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> * drop italic TL;DR commit-range line from review body the metadata (sha range, commit list, timestamps) is already in the html comment for downstream agents. the visible italic line was clutter and the ellipsis form broke the second sha's auto-link on github anyway. * add agent-browser fallback rule for unreachable chrome devtools mcp * onboarding: gated org-console wizard (#762) * onboarding: gated org-console wizard Replaces the org console's `/console/[owner]` page with a single-card, "growing" stepper when the account has zero `Repo` rows. Walks first-time users through billing mode, BYOK provider+key (if applicable), repo pick, workflow file creation, and a celebratory redeem-credit moment before landing them back on the now-populated org console. ## What's new - New: `components/OnboardingStepper.tsx` — the wizard. Six steps, each derived from real persisted state (Account.modelAccessMode, AccountSecret, Repo). Step state ladder with progressive disclosure and click-to-edit collapsed summaries. - New: `app/console/[owner]/OnboardingView.tsx` — page-chrome wrapper that hosts the stepper inside the same header/sidebar shell as the member view. - Modified: `app/console/[owner]/page.tsx` — adds a `prisma.repo.count` gate alongside existing parallel queries; renders OnboardingView when count === 0, else falls through to the existing repo grid. ## Schema - Flipped `Account.modelAccessMode` default from `byok` to `router`. Router is the lower-friction default (signup credit funds first ~150 runs without a card; users can flip to BYOK explicitly via the wizard or the existing `<ModelAccessCard>` switch). Existing rows keep their current explicit value — Postgres column-default change doesn't backfill, by design. - Migration: `20260516014601_modelaccessmode_default_router`. ## Credit-claim semantics Killed the historical mount-time auto-claim on `<SignupCreditModal>`. All claims are now explicit clicks, fired from one of two surfaces: 1. Wizard step 6 "Redeem $10 credit" CTA (Router branch, eligible). 2. New explicit "Redeem $10 credit" button on `<BillingCard>`'s Router wallet section, visible only when the new server-derived `signupCreditEligible` flag is true (promo active + no prior signup or welcome grant). Covers existing users who'd otherwise lose the auto-claim entry point. `<SignupCreditModal>` is now a controlled component (`open` / `onOpenChange` / `amountCents` props) with a sibling `useClaimSignupCredit(owner)` hook for explicit invocation. The Sparkles celebration dialog rendering is unchanged. ## Other touched surfaces - `app/api/create-workflow/route.ts`: optional `model` body field. When present, the route updates `Repo.model` on the row that `createWorkflowForRepo` just created/surfaced — wizard threads the picked provider's `preferred` model alias through here so a fresh repo doesn't sit on null/auto. - `app/api/account/[owner]/billing/route.ts`: surfaces `signupCreditEligible: boolean` (derived from `SIGNUP_CREDIT_PROMO_ACTIVE` + grant scan). Drives the new explicit redeem button. - `components/AgentSettings.tsx`: fixes the Router-no-billing copy lie ("Runs will draw from your signup credit until exhausted" was false — `isInfraCovered` gates Router minting on `hasCardOnFile`, not balance, so credit-only-no-card users can't actually spend the grant on Router runs). New copy: "Add a card to use Pullfrog Router. Your $10 signup credit (if claimed) applies on top." ## Resume-tomorrow detection Every step's expansion is derived from persisted state (no new column, no localStorage). With the Router default flip, `modelAccessMode === "byok"` is now a reliable signal of explicit user pick, eliminating the heuristic that the byok-default schema would have required. The only ambiguous case is "Router-bailed-before-redeem" (looks identical to a default-Router fresh visit since neither card nor grant exists yet) — acceptable 1-click cost on revisit. ## Testing - `pnpm lint`: clean - `pnpm format`: clean - `pnpm typecheck`: clean - `pnpm -C action test`: 596/596 passing - Visual verification: blocked — Chrome DevTools MCP returned "Not connected" across both available servers. Manual walkthrough needed before merge to confirm step transitions, going-back UX, and the celebration modal redirect destinations match the plan in `.cursor/plans/org_onboarding_stepper_4fdfebbb.plan.md`. * onboarding: drop accordion, multi-repo bulk-onboard, full-width radio rows Three rounds of UX feedback rolled in: 1. **Drop the accordion.** Steps no longer collapse to a one-line summary when "done" — the wizard literally grows by appending steps below as the user progresses, and earlier steps stay fully interactive (re-flip Router→BYOK, re-pick provider, toggle a repo) without any "edit" affordance. `StepShell` now always renders its body for any step the user has reached; the only state distinction is the number circle (filled = active, check = done). 2. **Step 1 is full-width radio rows, not narrow tabs with side-by-side info tiles.** Two rows, each with the option title, an inline "Recommended" badge on Router, and a description sentence inside the row. The persisted `Account.modelAccessMode` (default `router`) drives the initial selection, so step 1 always has one row picked on first paint — no "neither selected" empty state. 3. **Multi-repo bulk-onboard.** Step 4 now uses checkboxes; copy reads "Select the repos you'd like to install Pullfrog into. We'll create a pullfrog.yml GitHub Actions workflow file in each." Step 5 fans out N parallel `POST /api/create-workflow` calls (concurrency capped at 4) and renders per-repo status inline (running → committed / PR #N / already configured / error). Step 6 celebrates with a multi-result headline ("Pullfrog is set up across N repos") and a sub-line breaking down `committed · PRs awaiting merge · failed` plus a per-repo PR list when any PRs were opened. Single- repo path renders the same control surface but with singular copy. Other bits: - Per-step description sentences below every title. - Repo picker shows totalCount inline with the pagination controls and "N repos selected" summary below the table. - Dropped the `userPickedBillingMode` and `editingStep` state machinery + the `isFreshDefault` heuristic — all simplified out by the no-accordion design (we just trust `billingMode` directly). - `createWorkflowPR` PR body already links back to `pullfrog.com/console/<owner>/<repo>` with a "Verify workflow" CTA; no change needed there. * fix(onboarding): provider tile labels — getProviderDisplayName expects slug `getProviderDisplayName` from `pullfrog/internal` parses its argument as a `provider/model` slug. Step 2 was passing bare provider keys (e.g. "anthropic"), which made the helper throw "invalid model slug 'anthropic' — expected 'provider/model'" and crashed the BYOK branch with the page-level error boundary. Replace with a local `providerDisplayName` that reads the registry directly (`providers[key].displayName`). Drops the unused `getProviderDisplayName` import. Caught by Chrome DevTools end-to-end: clicking Bring-your-own-key on the fresh wizard renders the page-level error. Re-verified post-fix: BYOK flow shows step 2 with all 9 provider tiles correctly labeled (Anthropic / OpenAI / Google / xAI / DeepSeek / Moonshot AI / Amazon Bedrock / OpenRouter / OpenCode), step 3 reveals on tile click. Also adds a guardrail to AGENTS.md: don't silently abandon visual verification when DevTools breaks. Recovery is always possible (pkill -9 chrome-devtools-mcp + pkill puppeteer + rm Singleton locks + retry several times); if it genuinely won't recover, abort and tell the user — never mask as "verified by code review". * agents.md: never give up on Chrome DevTools MCP failures Recovery is always possible (pkill chrome-devtools-mcp, remove Singleton locks, retry several times). If genuinely unrecoverable, abort and tell the user explicitly — never silently mask as "verified by code review". Visual verification is non-negotiable for UI changes. * onboarding: polish — checkbox color, redundant labels, copy Caught during chrome-devtools verification of the BYOK + cross-page selection flows: - **Checkbox color**: native browser pink/red replaced with evergreen via `accent-evergreen-600`. Visually consistent with the rest of the wizard's selection states. - **Bedrock provider tile**: was rendering "Amazon Bedrock" twice (provider name + recommended-model name both resolve to "Amazon Bedrock" because Bedrock has no `preferred` model under `providers.bedrock.models` — its single routing entry IS the recommended pick). Suppress the recommended subtitle when it duplicates the provider name. - **Step 6 description**: tightened from a clunky two-clause sentence about workflow file landing to a single direct call: "Mention @pullfrog in any PR or issue to dispatch a run. (Branch-protected repos: merge the PR first.)" - **Wizard intro**: was "Set up Pullfrog for your first repo" — outdated since multi-repo. Now: "Connect Pullfrog to your repos. Each step unlocks the next as you go." Cross-page multi-select also verified: selections from page 1 persist when navigating to page 2 and back. "N repos selected" counter reflects total across all pages. BYOK secret-add flow verified end-to-end: AddSecretModal opens with the env var pre-filled, save triggers secrets refetch, step 3 flips to "✓ ANTHROPIC_API_KEY configured", step 4 reveals automatically. * onboarding: serial install, inline secrets, explicit credit redeem - step 3: replace modal-based secret entry with inline password fields per provider, with deep links to provider dashboards. claude code OAuth surfaces as a distinct group when anthropic is picked. bedrock gets three-field form. github actions secrets path is collapsible with org/personal-aware urls + self-certify. - step 4: merge repo-pick + workflow-create into one step. install is now serial (visible slow-reveal) instead of concurrent. continue button renders immediately on submit, disabled until every repo reaches a terminal state. errored rows render a single soft amber 'failed' label. pagination uses chevron buttons + keepPreviousData (no layout shift). - step 6: explicit 'redeem $10 credit' for router+eligible, 'complete setup' otherwise. final redirect is a hard refresh so the repo grid picks up. - signup credit: drop the mount-time auto-claim modal in favor of explicit user clicks. new useClaimSignupCredit hook + RedeemSignupCreditCallout banner inside RouterWalletSection so a BYOK→Router flip surfaces a one-click redeem affordance. - billing mode is now optimistic (local state + background PATCH) and initialBillingMode + signupCreditEligible eager-load via server props to kill the multi-second click latency. - skip onboarding: header button sets pullfrog_skip_onboarding cookie; server reads it in page.tsx and falls through to the regular grid. - demo mode: NEXT_PUBLIC_ONBOARDING_DEMO=1 cycles the install progress list through pending/running/committed/PR/existing/failed states. - createWorkflowForRepo: PULLFROG_FORCE_PR_CREATION=1 skips direct commit to exercise the PR fallback locally. * onboarding: review feedback — focused eligibility query, best-effort model pre-fill, claim error toast - billing/route.ts + console/[owner]/page.tsx: replace top-N recentGrants scan for signup-credit eligibility with a focused findFirst({ reason: { in: [SIGNUP, WELCOME] } }). the prior query could return any 5/10 rows (no orderBy on page.tsx) and miss a prior signup/welcome grant if a future grant reason (refund/referral/etc.) ever ships. recentGrants stays for the billing-history list. - create-workflow/route.ts: gate Repo.model updateMany on result.type === "created" so an existing user-set model isn't clobbered when the workflow file already exists. wrap in try/catch: GitHub side effect already succeeded, so a transient DB blip shouldn't 500 the route and have the UI report failure on a partially-completed setup. - SignupCreditModal: add onError toast to useClaimSignupCredit so transient redeem failures surface ("Couldn't redeem your credit. Try again in a moment."). callers .catch(() => null) the rejection so it doesn't propagate as an unhandled rejection in the React handler. - OnboardingStepper: trim stale "per-row try again button" wording from progressRef + processRepo comments — that button was removed in the prior commit per design feedback. * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * revert: extract router-gate fix into its own PR The router fix at a14bcdd4 is being shipped as a standalone PR so it can be reviewed and merged independently of the onboarding-wizard work. Reverting here keeps #762 focused on the wizard. The fix itself landed at https://github.com/pullfrog/app/pull/792. * router: fix unspendable signup credit on no-card private repos (#792) * router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * action: drop dead isInfraCovered + plan param post-fix Cleanup the action-side dead code introduced by the previous commit's removal of the redundant `isInfraCovered` re-derivation in proxy.ts: - delete `isInfraCovered` from action/utils/runContext.ts (was the only callsite; mirror in server's utils/billing.ts is unchanged and still load-bearing for learnings/indexing) - drop unused `plan: AccountPlan` param from `resolveProxyModel` / `runProxyResolution` (and the corresponding `AccountPlan` import + the `plan: runContext.plan` arg at the main.ts call site) - update the action/mcp/server.ts comment that pointed at the now-gone action mirror to reference the server-side `utils/billing.ts` instead `AccountPlan` itself is still load-bearing (mcp/server, runContextData, run-context fetch), only `isInfraCovered` and the dead `plan` parameter go away. * eager signup credit + free-OpenCode fallback when BYOK has no key (#789) * eager signup credit + free-OpenCode fallback when BYOK has no key addresses the silent-churn pattern that took out 15 first-run-failure accounts post-launch: GH Actions secret references resolved to empty strings (because the secrets didn't exist on the repo), the action launched Claude Code with no key, the LLM provider 401'd, and the run died in seconds with a synthetic "Invalid API key" message. those accounts had no Router credits to fall back to because the lazy claim required a dashboard visit they never made. three changes, one PR: 1. Eager $10 signup credit at account creation. Both account-creation sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo` for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }` in the same transaction as the `accounts` row. CLI installers who never sign in get the credit. The dashboard `/signup-credit/claim` POST stays as an idempotent backstop for accounts created before this shipped. 2. Free-OpenCode fallback in the action. When the configured BYOK slug needs a provider key the runner doesn't have, swap to `opencode/minimax-m2.5-free` before agent selection so the run still succeeds. Surfaced via a `» fell back from <slug> to <free>` warning in the action log. Skipped on Router runs (Pullfrog mints the key) and when no model is configured (auto-select-with-throw still fires for the genuinely-misconfigured case). 3. New action-test fixture `byok-no-keys-fallback` that empty-strings every known provider key (matching how GH Actions handles missing secrets) and asserts the run succeeds with the fallback log line present. plus a unit test for the helper covering each skip case. skipping the schema flip from `byok` to `router` — that's coming via the onboarding-stepper PR (#762). * fallback: skip Bedrock + surface in PR-comment footer addresses copilot review on #789 (real bug — parseModel throws on Bedrock raw IDs that have no slash, would crash before validateBedrockSetup could surface its own error) and the user-side ask to make the fallback visible in PR comments. - selectFallbackModelIfNeeded skips when resolvedModel has no '/' so Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash inside hasProviderKey -> parseModel. unit test covers it. - toolState.modelFallback records the configured slug we fell back from. set in main.ts when fallback engages. - buildPullfrogFooter accepts fallbackFrom and renders "Using `MiniMax M2.5` (free) (credentials for Claude Opus not configured)" so the substitution is visible in PR comments, reviews, PR bodies, and error reports. - threaded through all four action-side footer call sites (mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts fire pre-action and don't have toolState — left as-is. * fallback footer: use provider display name + document email asymmetry addresses pullfrog reviewer findings on #789: - footer now shows 'credentials for Anthropic not configured' (provider display name from `providers.anthropic.displayName`) instead of the per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY covers all Anthropic models), so this matches what the user actually needs to fix. - document the intentional asymmetry between eager and lazy signup credit paths: eager skips both the signupCreditClaimedEmail and the per-grant team@ alert. comment explains why (the 'new account created' alert already covers it on the eager path; the user-facing email assumes a user-initiated action that hasn't happened yet for CLI/GH-App-only signups). - skipping the backfill for the 15 historical accounts per user's earlier decision — they all uninstalled, so the cohort self-selected out of being reachable. * fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap local agnostic fixture run surfaced two real bugs the unit tests didn't catch: 1. fallback gate was on configuredSlug (=payload.model) but the test uses PULLFROG_MODEL to set the model, which is read by resolveModel AFTER its slug arg. configuredSlug stayed undefined → fallback never fired. drop configuredSlug from the helper signature; gate purely on resolvedModel since that's the same value regardless of how the model was specified (DB config vs PULLFROG_MODEL env). 2. when fallback engaged, the post-swap resolveModel({slug: fallback.to}) call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback target back to the unkeyed model. validateAgentApiKey then threw "no API key found" against the original model. fix: skip the re-resolve. fallback.to is already a CLI-ready specifier. unit tests updated for the new helper signature (8 tests, all pass). fallback log line confirmed emitted in the local run pre-second-fix; the second fix unblocks the validation that previously threw. * models-bump: harden CI and bot prompt against catalog hallucinations PR #790 (the first bot-authored models-bump PR) shipped a broken bump for openrouter/gemini-flash: the bot pattern-matched the parallel google bump and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the OpenRouter API but not on models.dev's openrouter section — the catalog OpenCode actually reads. The slug failed at runtime with `Model not found`. Two CI gaps let it through: 1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*` alias as a "passthrough", smoke-testing only one canary per routing layer. But those aren't passthroughs — each is a distinct models.dev catalog entry that can drift independently of the direct-provider mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25). Only `bedrock/byok` stays pruned (sentinel resolve). 2. `models-catalog` test (the integrity gate that asserts every resolve exists on models.dev) was main-only by design — to keep upstream catalog churn from blocking unrelated PRs. But it's exactly the test we want running on the bot's own catalog edits. Add `pullfrog/models-bump` head-ref to its trigger. Also tighten the bot prompt in models-bump.yml: new rule 0 requires every new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the alias's own `candidates[]` in models-bump-context.json — the deterministic preprocessor only emits candidates sourced from models.dev's mirror, so this gates against the cross-alias pattern-matching that broke PR #790. For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter API), which is necessary but not sufficient; the `models-catalog` job is the authoritative models.dev check. Verified locally: - baseline `pnpm -C action test:catalog` passes 133 tests - simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash` into action/models.ts) and the catalog test fails with the right assertion: `model "google/gemini-3.5-flash" not found under openrouter on models.dev` - `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every openrouter/* alias and every keyed opencode/* alias now smoked --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |
||
|
Colin McDonnell
|
cb0dbcd371 |
feat(action): make prepush hook non-blocking after one failure (#777)
* feat(action): make prepush hook non-blocking after one failure push_branch now treats the repository's prepush hook as best-effort: it runs at most once per run, surfaces the failure output if the script exits non-zero, and every subsequent push_branch call this run skips the hook so the agent isn't blocked by failures unrelated to its change. The agent can iterate by running the hook command itself via the shell tool when shell access is available; push_branch will not re-run the hook automatically after a failure. Why: a one-line OSS-allowlist change took 9 minutes (#776) because the agent retried push_branch six times against a prepush hook that was failing for env-leak and missing-build-artifact reasons unrelated to the change. CI catches the same checks on the GitHub side; the local prepush gate was duplicating work and blocking unrelated fixes. - ToolState: new prepushFailureCount counter (per-run, never resets) - executeLifecycleHook: returns structured failure (kind/output/exitCode) so prepush can compose its own agent-facing message instead of inheriting the generic retry/no-retry advice meant for setup - push_branch: composes a shell-mode-aware error message; surfaces prepushSkipped on the success payload + appends a note to the message - instructions.ts + wiki/prompt.md + docs/comparisons.mdx: updated to reflect best-effort semantics * fix(action): clarify prepush latch semantics + soften static guidance review fixes from PR #777: - toolState comment, instructions, success message, tool description: replace "runs at most once per run" / "first call only" wording with the actual semantic — successful prepush keeps running on later push_branch calls; only a hook FAILURE latches the bypass. - tool description: drop hardcoded "via the shell tool" guidance so the static description doesn't mislead in shell:disabled runs (the dynamic agent prompt in instructions.ts already does shell-conditional messaging). - LifecycleHookFailure.output JSDoc: match the implementation (stderr-preferred fallback to stdout, empty for timeout/spawn). * fix(action): shorten prepush-skip log to terse operator telemetry the previous log line tried to address the agent ("re-run the hook command yourself via shell"), but log.info writes to the action runtime's stdout — the agent never sees it. agent-facing skip guidance already lives in the error message from buildPrepushFailureMessage, the success message when bypassed, and the system prompt in instructions.ts. log line is now just operator telemetry. * refactor(action): drop slop from prepush soft-fail self-audit pass after the previous review-fix round. removed duplication between code-level comment and the five other places that already explain the same behavior, tightened verbose JSDoc, and collapsed redundant clauses in agent-facing strings. - LifecycleHookFailure → discriminated union. drops the optional exitCode/spawnError fields (and the empty-output sentinel for timeout/spawn) plus the corresponding ?? fallbacks in the helper. - PushBranchTool: 7-line code comment above the latch removed (toolState field comment + tool description + error message + success message + system prompt all already cover it). tool description third sentence dropped (restated the second). success message tightened to a parenthetical. - buildPrepushFailureMessage: 4-line JSDoc → 1 line. shared "if you think the failure could indicate a real bug in your code" prefix factored out across the shell-conditional branches. - ToolState.prepushFailureCount comment: 8 lines → 3. the "what" is in git.ts; comment now only documents the invariant (never decremented within a run). - instructions.ts prepush guidance: collapsed nested bullets + ternary into one paragraph; dropped the "so re-running via shell is the only way…" tail that restated "push_branch will NOT re-run it". * fix(action): hint prepush bypass on dirty tree after hook failure When push_branch blocks on a dirty working tree and the prepush latch is already set, tell the agent the hook will be skipped once the tree is clean. * fix(test): narrow CI matrix for lifecycle and toolState changes Remove lifecycle.ts from ALWAYS_RUN_ALL and add lifecycle.ts + toolState.ts to push/git agnostic test coverage so PRs touching prepush latch logic run targeted tests instead of the full matrix. |
||
|
Colin McDonnell
|
7e90e5cae6 |
Align Plan-mode prompts on report_progress as the canonical plan tool (#786)
* fix: align Plan-mode prompts on report_progress as the canonical plan tool Fixes #673. Three sites disagreed on where Plan output should be posted, letting a model synthesize a broken third interpretation (initial post via `report_progress({ target_plan_comment: true })`, which then misses the `existingPlanCommentId` precondition). This PR aligns all three on `report_progress` as canonical, with `target_plan_comment` reserved for revisions only: - `action/modes.ts` Plan step 4 — spell out that the initial plan post uses `report_progress` WITHOUT `target_plan_comment`, and that revisions go through `select_mode`'s PlanEdit override. - `action/mcp/comment.ts` `target_plan_comment` flag description — make the "revisions only" precondition explicit and call out the initial-post path by name. - `action/utils/instructions.ts` Progress reporting paragraph — drop the misleading "(e.g., Plan comments)" parenthetical that read as "use create_issue_comment for plans". `PlanEdit` (in `action/mcp/selectMode.ts`) was already correct and is unchanged. Intentionally out of scope (to keep the fix minimal): a `publish_plan` tool, removing the vestigial `create_issue_comment({ type: "Plan" })` branch, hardening the run-end cleanup guard for the `target_plan_comment but no existingPlanCommentId` fallthrough, and renaming `target_plan_comment`. * align create_issue_comment description with report_progress as canonical plan tool |
||
|
Colin McDonnell
|
f3d18401ac |
eager signup credit + free-OpenCode fallback when BYOK has no key (#789)
* eager signup credit + free-OpenCode fallback when BYOK has no key
addresses the silent-churn pattern that took out 15 first-run-failure
accounts post-launch: GH Actions secret references resolved to empty
strings (because the secrets didn't exist on the repo), the action
launched Claude Code with no key, the LLM provider 401'd, and the run
died in seconds with a synthetic "Invalid API key" message. those
accounts had no Router credits to fall back to because the lazy claim
required a dashboard visit they never made.
three changes, one PR:
1. Eager $10 signup credit at account creation. Both account-creation
sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo`
for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }`
in the same transaction as the `accounts` row. CLI installers who
never sign in get the credit. The dashboard `/signup-credit/claim`
POST stays as an idempotent backstop for accounts created before
this shipped.
2. Free-OpenCode fallback in the action. When the configured BYOK slug
needs a provider key the runner doesn't have, swap to
`opencode/minimax-m2.5-free` before agent selection so the run still
succeeds. Surfaced via a `» fell back from <slug> to <free>` warning
in the action log. Skipped on Router runs (Pullfrog mints the key)
and when no model is configured (auto-select-with-throw still fires
for the genuinely-misconfigured case).
3. New action-test fixture `byok-no-keys-fallback` that empty-strings
every known provider key (matching how GH Actions handles missing
secrets) and asserts the run succeeds with the fallback log line
present. plus a unit test for the helper covering each skip case.
skipping the schema flip from `byok` to `router` — that's coming via
the onboarding-stepper PR (#762).
* fallback: skip Bedrock + surface in PR-comment footer
addresses copilot review on #789 (real bug — parseModel throws on
Bedrock raw IDs that have no slash, would crash before
validateBedrockSetup could surface its own error) and the user-side
ask to make the fallback visible in PR comments.
- selectFallbackModelIfNeeded skips when resolvedModel has no '/' so
Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash
inside hasProviderKey -> parseModel. unit test covers it.
- toolState.modelFallback records the configured slug we fell back
from. set in main.ts when fallback engages.
- buildPullfrogFooter accepts fallbackFrom and renders
"Using `MiniMax M2.5` (free) (credentials for Claude Opus not
configured)" so the substitution is visible in PR comments,
reviews, PR bodies, and error reports.
- threaded through all four action-side footer call sites
(mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side
call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts
fire pre-action and don't have toolState — left as-is.
* fallback footer: use provider display name + document email asymmetry
addresses pullfrog reviewer findings on #789:
- footer now shows 'credentials for Anthropic not configured' (provider
display name from `providers.anthropic.displayName`) instead of the
per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY
covers all Anthropic models), so this matches what the user actually
needs to fix.
- document the intentional asymmetry between eager and lazy signup
credit paths: eager skips both the signupCreditClaimedEmail and the
per-grant team@ alert. comment explains why (the 'new account
created' alert already covers it on the eager path; the user-facing
email assumes a user-initiated action that hasn't happened yet for
CLI/GH-App-only signups).
- skipping the backfill for the 15 historical accounts per user's
earlier decision — they all uninstalled, so the cohort self-selected
out of being reachable.
* fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap
local agnostic fixture run surfaced two real bugs the unit tests didn't
catch:
1. fallback gate was on configuredSlug (=payload.model) but the test
uses PULLFROG_MODEL to set the model, which is read by resolveModel
AFTER its slug arg. configuredSlug stayed undefined → fallback never
fired. drop configuredSlug from the helper signature; gate purely on
resolvedModel since that's the same value regardless of how the
model was specified (DB config vs PULLFROG_MODEL env).
2. when fallback engaged, the post-swap resolveModel({slug: fallback.to})
call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback
target back to the unkeyed model. validateAgentApiKey then threw
"no API key found" against the original model. fix: skip the
re-resolve. fallback.to is already a CLI-ready specifier.
unit tests updated for the new helper signature (8 tests, all pass).
fallback log line confirmed emitted in the local run pre-second-fix;
the second fix unblocks the validation that previously threw.
|
||
|
Colin McDonnell
|
8dff91ac49 |
router: fix unspendable signup credit on no-card private repos (#792)
* router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * action: drop dead isInfraCovered + plan param post-fix Cleanup the action-side dead code introduced by the previous commit's removal of the redundant `isInfraCovered` re-derivation in proxy.ts: - delete `isInfraCovered` from action/utils/runContext.ts (was the only callsite; mirror in server's utils/billing.ts is unchanged and still load-bearing for learnings/indexing) - drop unused `plan: AccountPlan` param from `resolveProxyModel` / `runProxyResolution` (and the corresponding `AccountPlan` import + the `plan: runContext.plan` arg at the main.ts call site) - update the action/mcp/server.ts comment that pointed at the now-gone action mirror to reference the server-side `utils/billing.ts` instead `AccountPlan` itself is still load-bearing (mcp/server, runContextData, run-context fetch), only `isInfraCovered` and the dead `plan` parameter go away. |
||
|
David Blass
|
6e94f513df |
feat(billing): monthly Router spend limits (hard-cap + alert-only) (#748)
* feat(billing): monthly router spend limits (hard-cap + alert-only modes) (#660) Per-account ceiling on the sum of `router_topup` invoices (pending + succeeded) for the current UTC calendar month. Closes a gap where a runaway agent loop, leaked PR trigger, or stuck workflow could auto-reload indefinitely with no aggregate per-month ceiling. Two enforcement modes via `RouterLimitMode` enum: - `hard_cap`: refuse new auto-reloads; PR comment via reserveRun; 402 `router_monthly_limit` from /api/proxy-token; email + banner - `alert_only`: auto-reload keeps flowing; email + banner only, first breach per UTC month Enforcement is split across reserveRun (pre-dispatch paywall comment) and /api/proxy-token phase-1 (mid-run 402). Both surfaces read through the same `getRouterSpentThisMonthCents` helper so the dashboard, the dispatch gate, and the auto-reload gate can't disagree. Email dedup uses `Account.routerLimitNotifiedMonth` (YYYY-MM string), claimed atomically inside the phase-1 SERIALIZABLE txn so concurrent reloads breaching together send exactly one email. Read-time comparison with the current month re-arms on rollover — no cron. Admin surface: `RouterLimitBanner` (reuses `DelinquencyBanner` shell) above the Router/BYOK tabs in `ModelAccessCard`, with a popover "Adjust limit" form that PATCHes the existing /api/account/[owner]/billing/settings route. Same `assertBillingAdmin` gate that owns the other billing settings — no new auth surface. See wiki/billing.md § Router monthly spend limit for the full contract + edge cases. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal pass on monthly Router spend limit (#660) Round-1 review across 5 lenses (billing-subsystem, correctness, security, operational-readiness, research-validated-assumptions) surfaced one critical + three actionable major findings on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — CAS never matched NULL.** `claimRouterLimitNotificationSlot` used Prisma `NOT { routerLimitNotifiedMonth: monthKey }`, which compiles to `field != value` — UNKNOWN (not TRUE) against the post-migration `NULL` default. First breach for any account would never claim the slot, never stamp the row, and never fire the email (hard_cap or alert_only). Replaced with `OR: [{ field: null }, { field: { not: monthKey } }]`, mirroring the `maybeNotifyLowBalance` pattern. **Major — email gap on manual-top-up over cap.** Breach email was only wired through `/api/proxy-token`. A manual `/billing-top-up/<owner>` that crosses the cap blocks dispatch via `reserveRun` but never hits proxy-token, so the user got the PR comment but no email. Wired the CAS + `after(maybeNotifyRouterLimit)` into `reserveRun`'s PaywallError catch (the SERIALIZABLE txn rolled back when we threw, so we re-claim with the global client; single-statement CAS is its own race boundary against concurrent proxy-token claims). **Major — PR paywall comment leaked $ figures.** `router_limit` body embedded `($X of $Y)` in a comment visible to anyone with PR read access (public repos, forks, outside collaborators). Other paywall types deliberately avoid amounts. Removed; deep link still points to the authenticated console for the figures. **Medium — observability.** Added `[router-limit]` structured logs at the three enforcement sites (proxy-token hard_cap 402, proxy-token alert_only breach, reserveRun paywall) so on-call can grep "did the cap fire for customer X this month." **Medium — customer docs.** Added a `### Monthly spend limit` section to `docs/billing.mdx` (Mintlify) describing the two modes and the manual-top-up caveat. **Doc — refund/dispute interaction.** Documented in `wiki/billing.md` that the cap inherits the existing webhook semantics: disputed `router_topup` drops from the sum (cap briefly un-trips); refunds don't flip status today so refunded top-ups keep counting. Matches wallet behavior — not redefined here. Accepted as-is (documented or pre-existing): `after()` reliability vs stamp-before-send tradeoff, alert_only email fires before Stripe phase-2, proxy-token reads limit fields outside SERIALIZABLE scope (brief TOCTOU on admin lowering cap), stale paywall comment on cap clear, no global kill switch (per-account `alert_only` flip is the practical kill switch), no audit log on cap changes (no existing audit infra), action version not bumped (separate release commit). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): anneal round 2 on monthly Router spend limit Round-2 anneal (billing-subsystem, correctness, research-validated, user-journey, operational-readiness) surfaced a critical merge conflict and a handful of major correctness + UX gaps on top of [#748](https://github.com/pullfrog/app/pull/748). **Critical — merge conflict.** While #748 was open, [#755](https://github.com/pullfrog/app/pull/755) extracted `formatBillingErrorSummary` from `action/main.ts` to `action/utils/billingErrors.ts`. The PR's new `router_monthly_limit` arm still lived in `action/main.ts`. Took main's slim orchestrator wholesale; moved the arm into the extracted file. **Major — cap = payments only, not dispatch.** `reserveRun` was pre-empting all PR-comment / `/trigger` dispatch on `spent >= limit` regardless of wallet balance, contradicting the cap's positioning as "ceiling on what you pay." An account with $500 of paid-up wallet and a breached $100 cap couldn't trigger any new run via the comment path, while GitHub UI re-runs (which bypass `reserveRun`) succeeded — surface inconsistency. Deleted the pre-dispatch gate; `/api/proxy-token` is now the sole enforcement point, refusing only the next auto-reload that would push past. Wallet credit always drains. Dropped the now-dead `router_limit` arm in `buildPaywallCommentBody`, the dead `routerSpentCents`/`routerLimitCents` fields on `PaywallError.detail`, and the post-paywall email-fire David added — all unreachable. **Major — split `manual_topup` from `router_topup`.** Manual on-session top-ups at `/billing-top-up/<owner>` were landing as `Invoice.kind = "router_topup"` and counting toward the cap. The cap exists to brake *passive* runaway (auto-reload loops); a manual top-up is a deliberate click-through that the user owns. Added `InvoiceKind.manual_topup`, flipped the manual write site + `createTopUpCheckoutSession` metadata, broadened wallet / reconcile / billing-report reads to `kind IN (router_topup, manual_topup)`, and scoped `getRouterSpentThisMonthCents` (the cap aggregate) to `router_topup` only. Worked example: cap=$300, reload=$100 → exactly three reloads succeed; a fourth is blocked. Historical rows stay labelled `router_topup` (no backfill); the asymmetry is small and accepted since the manual flow only existed alongside auto-reload for a brief window. Extended the `invoices_kind_matches_stripe_columns` CHECK so `manual_topup` follows the same shape as `router_topup` (PaymentIntent-backed, no stripeInvoiceId); split into a second migration because PG forbids using a freshly-added enum value in the same transaction. **Major — email reframed around the triggering reload event.** The `alert_only` body was reporting a pre-eager-write `spentCents` while the dashboard reads the post-commit value, so email and dashboard disagreed by exactly one reload. Both flavors now say "Your most recent $50 auto-reload brought you over your $300 monthly limit" instead of a running spent-of-cap total — no reconciliation needed, no more "you've hit your monthly cap" copy firing for partial breaches (spent=$80 of $100, reload=$30 would have triggered that wording). **Major — `/trigger/<owner>/<repo>/<n>` paywall copy.** Hardcoded "You've used your 30 free runs this month. Add a card to continue at 7¢/run." regardless of `detail.reason`. Branched on `cap` vs `delinquent` so each paywall surfaces actionable copy with the right CTA. `router_limit` no longer flows through here (per F4 above). **Major — RouterLimitBanner.** Added an `isAlertBreached` visual state (amber palette) so an `alert_only` account at $240 of $200 no longer renders in the same neutral zinc chrome as a healthy under-cap account. Updated popover copy to reflect the auto-reload-only scope. **Medium — paywall log line.** Added `detail.reason` to the `[Installation X] paywall:` log so on-call grepping for "why was this paused" can distinguish `cap` from `delinquent`. **Cleanup.** Dropped dead `utcMonthKey` import + re-export in `maybeNotifyRouterLimit.ts`. Renamed file-internal `reconcileRouterTopup*` fns + their reconcile-kind labels to `reconcileTopup*` / `topup_*` since they now handle both kinds. Updated wiki/billing.md + docs/billing.mdx + schema doc comments throughout. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(billing): drop routerLimitNotifiedMonth sentinel; rely on Resend idempotency-key The sentinel was the same anti-pattern as `routerLowBalanceEmailedAt` sitting next to it — a single-purpose state column on `Account` that encoded a date as a string and required a custom CAS predicate to read/write race-safely. Plus it had real holes: Resend send failure left the sentinel stamped and the account silently un-emailed for the month (F11), mode flips mid-month didn't re-arm (F9), and cap-lowered edge cases never fired at all. Replace it with: fire `maybeNotifyRouterLimit` on every breaching reload, let the Resend `Idempotency-Key` `router_limit:<accountId>:<monthKey>:<mode>` collapse repeats inside Resend's 24h dedup window. Continuously-breaching accounts get ~1 reminder per day; brief Resend outages self-heal because the next breaching reload re-attempts the send. Mode is in the dedup key so `alert_only → hard_cap` mid-month re-arms a fresh email with the appropriate copy. Drops `Account.routerLimitNotifiedMonth` and `claimRouterLimitNotificationSlot`; simplifies the proxy-token phase-1 branch significantly. Net diff is negative LOC and the data model loses a single-purpose sentinel. Migration was branch-local — never deployed — so I edited the original add-cap migration in place to drop the column from the ALTER TABLE rather than chain a drop-column migration on top. Preview Neon branches reset automatically on history rewrite per wiki/migrations.md. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): hide RouterLimitBanner when no cap is configured The banner was unconditionally rendered for every billing-enabled account, including pure-BYOK admins who never touch Router. They got "No monthly spend limit / Router has spent $0.00" + a divider as visual noise on the model access page — basically nagging them to set a feature they may not want. Running without a cap is valid; we don't nag. ModelAccessCard now gates the banner block (banner + dividers) on `routerMonthlyLimitCents !== null`. RouterLimitBanner drops the no-limit visual state, the "Set monthly limit" CTA text, and the dead `hasLimit` branching. Cleaner three-state shape (under cap / amber breached / brick breached). Discoverability: no-cap users no longer see a UI affordance to set one. That's deliberate — the cap is a power-user feature documented in docs/billing.mdx. If discoverability becomes an ask, we can add a small inline link inside RouterWalletSection without bringing back the always-visible banner. Resolves the only outstanding finding from cursor bugbot's review of ff5328c (banner-visible-for-byok thread). Co-authored-by: Cursor <cursoragent@cursor.com> * docs(billing): docs/wiki match new "no banner without a cap" reality Pullfrog bot review of f7672ca pointed out the customer docs still told users to "Set the cap from the **Monthly spend limit** banner in the **Model costs** card" — but after hiding the banner for no-cap accounts there is no such banner to use until you already have a cap. Catch-22 for first-time setup. Rewrote docs/billing.mdx to be self-contained: explain what the cap is, what the two modes do, what the banner shows *once configured*, and direct admins to PATCH the billing settings endpoint (or reach out to support) for first-time setup. Cap is positioned as optional; running without one is the documented default. Wiki paragraph in wiki/billing.md updated to match — banner is only rendered when a cap exists, three visual states (under / amber / red), no first-time-setup UI nag by design. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): move monthly cap into RouterWalletSection as a normal settings row; drop the banner entirely The standalone `RouterLimitBanner` was the wrong shape. It only rendered when a cap was already configured (so there was no UI to discover the feature in the first place — first-time setup required hitting the API directly), and it occupied prominent real estate above the tabs to surface state that already lives in the row's own input when the form moves down where it belongs. New shape: monthly cap is just a third row inside `RouterWalletSection` sibling to **Auto-reload amount** and **Auto-reload threshold**. Gated the same way (card on file + auto-reload enabled — the only state where the cap actually means anything). Empty input → no cap, with placeholder "No limit". Setting a number reveals a **Behavior at limit** toggle built on the same `Tabs` slider component used for the Router/BYOK tab switch, so the look matches the rest of the card. Deletes: - `RouterLimitBanner` component (212 lines) - banner mount + conditional + spacers in `ModelAccessCard` - `AlertTriangle` is still imported (used by `DelinquencyBanner`) Adds: - one settings row in `RouterWalletSection` with the cap input + mode tabs - `routerMonthlyLimitUsd` / `routerLimitMode` plumbed through the existing `saveSettings` helper (widened to accept `string | null`) - `Tabs` / `TabsList` / `TabsTrigger` import Docs + wiki updated to match the new shape; the customer doc no longer points at a banner that won't appear. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): split monthly cap input and Behavior-at-limit toggle into separate rows with hr between Previously bundled both into one row block. Restructure: cap input is its own row; Behavior-at-limit Tabs gets a sibling row with the standard `h-5 + hr + h-5` separator between (matching the rhythm of auto-reload amount → threshold → monthly cap). Mode-toggle row is gated on `routerMonthlyLimitCents !== null` so the hr + tabs only appear once a number is in the cap input. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(billing): right-justify Behavior-at-limit Tabs to mirror Auto-reload toggle row Same `flex items-center justify-between gap-3` layout as the Auto-reload row: label group on the left, control on the right. Drops the vertical stack in favour of the horizontal one — looks identical to the toggle row directly above. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |
||
|
David Blass
|
dd26d35137 |
learnings: audit fixes — preamble in TOC, server-side line-boundary truncation, empty-repo intro (#743)
* learnings: surface preamble in TOC, mirror line-boundary truncation server-side, fix empty-repo intro copy three audit fixes on top of the recent learnings overhaul (#717): - `parseLearningsHeadings` now prepends a synthetic `(preamble)` entry when a body has non-whitespace content before the first heading. the prompt instructs the agent NOT to slurp the whole file when a TOC is present, so without this any preamble lines were silently invisible (realistic transitional case: an agent partially restructures a legacy free-text body and leaves bullets above the first `## `). - server-side PATCH route now applies the same line-boundary-aware truncation as the action (defense in depth via a shared `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` exported from `action/internal`). the raw `.slice` it used before could leave a mid-heading tail on any caller that bypassed the client-side truncate, breaking the next-seed TOC parse. removes the duplicated cap constant. - `buildLearningsSection` intro no longer asserts "accumulated by previous agent runs" — false for fresh repos with zero history. new copy is tense-neutral and works for empty + populated bodies. also nudges the agent to re-read after mid-run edits (the inlined TOC ranges are a run-start snapshot). Co-authored-by: Cursor <cursoragent@cursor.com> * learnings prompt: tighten to single evergreen test, allow tool-quirk bullets when they prevent repeat waste The blanket "no pullfrog tool quirks" ban was wrong — if the agent burned calls discovering a quirk this run, recording the workaround prevents the next run from repeating the waste. Reframe around one litmus ("would a future run do its work better because this bullet exists?") and trust it to subsume the scattered don'ts. Drop the 3+ months timeframe (arbitrary) and the four-example pullfrog/PR/date/play-by-play list (the rule underneath is "don't anchor facts to repo state that will move"). Cuts ~10 lines from a prompt the model was already mostly ignoring; the remaining anchor list is narrower and more enforceable. * audit-learnings-r2: align wiki + tighten re-read nudge - wiki/prompt.md described the post-run reflection prompt as "bans pullfrog-tool quirks (those belong in tool descriptions, not per-repo learnings), bans PR/review/commit/date references" — that's stale after the prompt rewrite. update to: single-litmus framing, expanded anchor list (now includes version pins + line numbers), and explicit allowance for tool-quirk workarounds when discovery burned calls. - buildLearningsSection re-read nudge said "re-read after editing" which can be read as "re-read the section you edited". in fact any edit shifts the line numbers of every later section in the TOC, not just the edited one. tighten to make that explicit. mirror the new wording in the wiki example block. update the test substring assertion accordingly. * postRun: refresh JSDoc to match the reflection prompt rewrite `buildLearningsReflectionPrompt`'s JSDoc still listed "PR-/review-/commit-/date-anchored facts" and "rediscovery of pullfrog-tool quirks" as failure modes the prompt pushes back on. after b586b4f8 the prompt no longer bans tool-quirk bullets (it explicitly allows them when the agent burned calls discovering the quirk), and the anchor list expanded to cover branch refs, version pins, and line numbers too. update the JSDoc so it describes the prompt that actually exists, and call out the cross-repo drift tradeoff that comes with allowing tool-quirk bullets. * fix(mcp/issueEvents): narrow event.event before Set.has lookup octokit's listEventsForTimeline union includes timeline-event members where `event` is `event?: string`. `("event" in event)` does not narrow that property to non-undefined, so `relevantEventTypes.has(event.event)` was passing `string | undefined` to a `Set<string>.has`. typescript only flagged this once `cf-worker-indexing` started seeing the file via the type graph that now reaches mcp through the new `truncateAtLineBoundary` re-export in `action/internal/index.ts`. fix the latent bug at the source: require `typeof event.event === "string"` before the Set lookup. * learnings: split truncation helpers into MCP-free module re-exporting `truncateAtLineBoundary` + `MAX_LEARNINGS_LENGTH` from `action/utils/learnings.ts` through `action/internal/index.ts` accidentally pulled the entire MCP type graph into the SDK barrel: `learnings.ts` imports `ToolContext` from `mcp/server.ts`, which transitively wires every tool module under `action/mcp/` into anything that imports from `pullfrog/internal`. for `cf-worker-indexing/tsconfig.json` (`customConditions: ["@pullfrog/source"]`) and the root `tsc` (which compiles the proprietary app routes that import from `pullfrog/internal`), this expanded the type-checked surface and surfaced two latent issues in unrelated files (`mcp/issueEvents.ts`, `utils/subprocess.ts`). a 6-line pure string helper has no business dragging mcp/server.ts into anyone else's type graph. move both symbols to `action/utils/learningsTruncate.ts`. `learnings.ts` re-exports them so existing callers keep working; `internal/index.ts` re-exports from the truncate-only module so the SDK barrel stays MCP-free. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |
||
|
Colin McDonnell
|
88f170e19a |
fix: 7 log-audit / run-audit findings (mega-PR) (#769)
* fix(#765): silence Clerk 400 (revoked OAuth) noise from getTokenForClerkId Branch on isClerkAPIResponseError + status<500 so the well-understood revoked-token redirect doesn't emit a level=error line in Better Stack on every request. Vercel maps console.warn -> error for non-streaming routes, so a downgrade to log.warn wouldn't help; only the unexpected shape (5xx, network) is worth surfacing. * fix(#742): stop logging input verbatim from yes.op retry-failure paths GitHub OAuth user tokens (ghu_...) were leaking to Better Stack on every yes.op retry-failure for any utils/github/get* helper that takes a token field — 38 leaks/7d in the most recent audit window. The leak path is console.log inside the yes package (its own log shim, not utils/log.ts). Drop input from the four log sites + the cache-key-derivation throw site. key (SHA-1 of input) is sufficient for retry correlation; error already carries request URL + status. Defense-in-depth comment so future contributors don't re-add the field. Operational follow-up (separate task): inventory ghu_... strings in Better Stack ingested in the last 90d, revoke matching Clerk grants, scrub cold-tier S3, rotate the BS source token. * fix(#759): handle GraphqlResponseError "Could not resolve to a node" as 404 When the stored planCommentNodeId references a comment that's been deleted on GitHub, octokit.graphql throws GraphqlResponseError before the existing `node === null` 404 branch is reached. Add a narrow isGraphqlNodeNotFound predicate in utils/errors.ts and a new catch branch in the plan-comment route. The action treats 404 as "no prior plan comment" and creates a fresh one, so behavior matches existing contract. * fix(#747): convert webhook GraphQL rate-limit 5xx into a Result<T> sentinel + 200 ack When GitHub's GraphQL responds with "API rate limit exceeded for installation ID N", _getReviewCommentsWithReplies threw, propagated through the bare yes.op wrapper (no rate-limit bail), out of the bare await in handleWebhook, and crashed /api/webhook/github with 500 — 77 webhook 500s/24h on the most recent audit window. GitHub redelivery plus R2 dedup also silently masked the legitimate handler from re-running once the rate-limit window cleared. Mirror the #658 / _getRepository pattern: detect GraphqlResponseError matching /rate limit (already )?exceeded/i, log.warn with the x-ratelimit-reset value (and [Installation N] prefix when available), return failure(...) with status 429. Webhook handler short-circuits the case with 200 + log.info so GitHub stops the redelivery storm against an exhausted budget, and the trigger page surfaces a clean ThrowClientError. Document the new pattern as a Tier 2 false-positive in wiki/log-audit.md so the next audit cron doesn't re-flag it. Note that returning [] silently (the issue's first suggestion) would have dropped @pullfrog mentions inline in review comments and dispatched an agent run that re-rate-limits — skip-the-whole-case is the correct semantics. Co-vulnerable getPullRequest / getWorkflow have zero occurrences in this window; per #737 policy, defer until they show up. NOTE: this commit and the bracket of touched files revert as a unit — the Result<T> shape change in getReviewCommentsWithReplies is breaking; partial revert breaks the type chain. * fix(#766): fold stderr+stdout into shell.ts errors + carve out merge-base --is-ancestor action/utils/shell.ts dropped stdout when constructing failure messages ($\{stderr || "Unknown error"\}), so git subcommands that write context-bearing diagnostics to stdout (merge conflicts, cherry-pick rejections, diff --exit-code, ls-files --error-unmatch) surfaced as "Command failed with exit code 1: Unknown error" through mcp__pullfrog__git. The agent burned an extra MCP round-trip calling git status to recover. Fold stderr + stdout into the thrown error message (stderr first, stdout fallback) so the agent always sees the real diagnostic. Plus a narrow carve-out for `git merge-base --is-ancestor` in action/mcp/git.ts: that subcommand uses exit code as data (0=ancestor, 1=not-an-ancestor, >1=error), so return { success: true, isAncestor } instead of throwing on exit 1. No caller in action/ string-matches on the old error format (verified). diff --exit-code and ls-files --error-unmatch are not carved out — both are zero-occurrence in the May audit window, and the stderr+stdout fold renders their output usefully anyway. * fix(#739): point customers at the actual fix when permissions: id-token: write is missing When a customer workflow runs in GitHub Actions but lacks permissions: id-token: write, ACTIONS_ID_TOKEN_REQUEST_URL/_TOKEN aren't injected, isOIDCAvailable() is false, and acquireNewToken falls through to the local-dev-only acquireTokenViaGitHubApp path, which throws "GITHUB_APP_ID and GITHUB_PRIVATE_KEY must be set" — pointing at a self-hosted-app fix that doesn't apply. One affected customer burned 13 dispatches in 24h on this misleading error. Detect (GITHUB_ACTIONS=true) AND (no OIDC env vars) inside acquireNewToken before falling through to the local-dev branch, and throw an actionable message naming the missing permissions block, the exact YAML, and the docs anchor. The error surfaces via ##[error]action failed: ... in the workflow log (the only customer surface available before main()'s inner try opens). Local-dev path keeps the existing GITHUB_APP_ID message. * fix(#760): suspend activity watchdog across in-flight tool calls mcp__pullfrog__checkout_pr was hard-failing 6/24h on SenecaLabs/senecaWeb because git fetch+deepen on a large monorepo can take 4-5 min, the agent's stdout pipe goes silent the entire time (FastMCP is in-process HTTP, but Claude/opencode CLIs await the synchronous tools/call response), and both the spawn-level activity timer (300s in subprocess.ts) and the process-level activity monitor (300s in activity.ts) fire and kill the run. Re-introduce the bracket pattern that PR #634 removed: bracket suspendActivity()/resumeActivity() around tool_use -> tool_result in both agent harnesses, plumb isPausedExternally into spawn() so both timers suspend in lockstep. Bounded by MAX_TOOL_CALL_SUSPENSION_MS (15 min auto-resume) plus the outer 1h agent timeout — neither zombie-run avenue from #12 is reopened (subprocess.close still resolves on death; outer timeout is suspend-agnostic; suspends gated on explicit paired CLI events, not internal noise). opencode tool_use handler: gate suspendActivity() on non-terminal status (running/pending) so the bus_event re-dispatch path at line 915 — which only fires for completed/error subagent parts and never emits a paired tool_result — doesn't latch the watchdog into suspension until the 15min ceiling. Add a heuristic:activity-watchdog-ceiling classifier to scripts/analyze-logs.ts so a tool that genuinely hangs past MAX_TOOL_CALL_SUSPENSION_MS surfaces in run-audit instead of being bucketed into failure:unknown. NOTE: this commit and the bracket of touched files revert as a unit — activity.ts, subprocess.ts, and the two harnesses must move together or the bracketing breaks. * refactor(#747): swap Result<T> for InstallationRateLimitError typed throw The Result<T> shape from 3ebf6c4c was cargo-culted from the #658 _getRepository pattern, but _getReviewCommentsWithReplies has only one expected-error case (installation rate-limit) and two callers — Result imposes branching on the trigger-page caller that never cared about the rate-limit case specifically. A typed error class is lighter (~10 LoC vs ~33) and matches the actual need: - new InstallationRateLimitError(resetAt) thrown from _getReviewCommentsWithReplies; rate-limit log.warn unchanged. - handleWebhook catches it and breaks with log.info (unchanged semantics: 200 ack, no redelivery storm). - trigger page reverts to direct array access; any failure propagates to the page error boundary (the pre-#747-commit shape). - log-audit.md wording updated to match. |
||
|
pullfrog[bot]
|
0abaaa1e37 |
chore(oss): add yamcodes/arkenv to OSS program (#776)
* chore(oss): add yamcodes/arkenv to OSS program * fix(test): strip CODEX_AUTH_JSON in apiKeys auto-select test The beforeEach strip list omitted CODEX_AUTH_JSON, which is in `knownApiKeys` via the openai provider's managedCredentials. When the env has CODEX_AUTH_JSON set, the auto-select "throws when no provider keys are present" assertion finds it and fails to throw. --------- Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> |
||
|
Colin McDonnell
|
c0988e35b0 |
fix(security): block docker socket from sandboxed shell; disable opencode batch_tool
two real CI failures on main, both shipping bugs in the action: 1. `token-exfil-claude` was a real sandbox escape: GHA `ubuntu-latest` puts `runner` in the `docker` group, so a sandboxed shell could run `docker run --pid=host --privileged busybox cat /proc/<parent>/environ` and read the action process's env (which holds user secrets) — fully bypassing the unshare PID-namespace. fix: inside the sandbox's mount namespace (already private via `--mount-proc` which implies `--mount`), bind-mount /dev/null over /var/run/docker.sock (+ podman/containerd/crio variants) so any container-runtime socket connect from the sandbox fails. only affects sandboxed shells — host runner mount table is untouched, so user workflow steps outside pullfrog keep working. 2. `restricted-opencode` regressed in #719 (`experimental.batch_tool`). opencode's batch tool rejects MCP tools with `"Tool '<name>' not in registry. External tools (MCP, environment) cannot be batched."` when a model emits parallel `pullfrog_shell` (or any MCP) tool_use blocks, opencode internally routes them through batch, they all fail, the model misreads the error as "the tool doesn't exist", and gives up. caught by a `lens:` subagent in the restricted test concluding shell was unavailable and setting `DIAGNOSTIC_ID=empty`. drop `batch_tool: true` and the matching opencode-specific guidance in `instructions.ts` — native parallel tool_use (multiple tool_use blocks per assistant message) still works for both built-in and MCP tools without batch, so we lose only the 1-25 wrapper, not parallelism. |
||
|
Colin McDonnell
|
0a64659ee7 |
refactor: slim action/main.ts to an orchestrator + extract helpers (#755)
* refactor: extract helpers out of action/main.ts so non-orchestration churn stops touching the file main.ts had grown to ~1240 lines holding ~500 lines of helpers that have nothing to do with the resolver pipeline — billing-error UI/copy, proxy minting, summary/learnings persistence, log formatters, end-of-run cleanup waterfalls. any PR adding a new billing code branch or a new log line was forced to edit main.ts, and since main.ts is in ALWAYS_RUN_ALL the entire 52-job LLM CI matrix fired on what should have been a 0-job change (e.g. #748). extractions: - action/utils/billingErrors.ts — BillingError, TransientError, the format*Summary renderers, billingConsoleUrl - action/utils/proxy.ts — mintProxyKey, buildProxyTokenHeaders, resolveProxyModel, plus runProxyResolution wrapper that renders + rethrows BillingError/TransientError before the outer catch - action/utils/prSummary.ts — fetchPreviousSnapshot, persistSummary co-located with the existing seed/read file helpers - action/utils/learnings.ts — persistLearnings co-located with the existing seed/read file helpers - action/utils/runStartupLog.ts — resolveOutputSchema + logRunStartup (the model/agent/push/shell/timeout block) - action/utils/runErrorRenderer.ts — renderRunError classifies (BillingError reclassify / hang detect / API-key auth) and emits {summary, comment} markdown bodies - action/utils/runLifecycle.ts — persistRunArtifacts, finalizeSuccessRun, writeRunErrorOutputs — the three end-of-run cleanup phases shared between the success path and the error catch path main.ts is now ~570 lines — the irreducible orchestrator: disposables (`await using` for tokenRef / gitAuthServer / mcpHttpServer), the toolContext construction, the agent-timeout race, the catch/finally shape, and the named phase calls. behavior is preserved verbatim (verified: pnpm -r typecheck + pnpm test 695/695 pass, action/test 596/596 pass). wiki/main.md gets a new "file layout" section describing the split. AGENTS.md gets a single line pointing future edits at the helpers instead of main.ts. * anneal: address review findings - restore MainResult.result?: string (accidental removal in initial commit; field was unused in current code but is part of the exported interface surface — keep the diff truly behavior-preserving) - move resolveOutputSchema from runStartupLog.ts to payload.ts (it's an action-input resolver alongside resolvePromptInput / resolvePayload, not a log helper — was placed in runStartupLog.ts for matrix-churn pragmatism but the domain fit is in payload.ts) - un-export resolveProxyModel (only used internally by runProxyResolution in proxy.ts; no external importer) - fix runErrorRenderer.ts JSDoc "Three classifications" → four (Billing, hang, API-key, default) - expand runLifecycle.ts module banner to note that finalizeSuccessRun calls persistRunArtifacts first, and to explain why the catch path splits writeRunErrorOutputs + persistRunArtifacts - update billingErrors.ts header to point at proxy.ts and runErrorRenderer.ts as the actual origin sites (was stale "main.ts") - expand proxy.ts header to spell out the runProxyResolution entrypoint contract (was stale "main.ts can render") - update wiki/main.md resolver chain + dependency table to name runProxyResolution as the actual call site and document the early BillingError/TransientError rendering branch - update wiki/main.md file-layout table to lead with runProxyResolution and describe mintProxyKey/buildProxyTokenHeaders/resolveProxyModel as internal helpers (was implying they were public surface) |
||
|
Colin McDonnell
|
a78b1542da |
feat: pullfrog auth codex + fresh-branch (#757)
* feat: pullfrog auth codex + fresh-branch Add `pullfrog auth codex` standalone command for minting Codex (ChatGPT) subscription credentials and saving them as the `CODEX_AUTH_JSON` Pullfrog secret. Codex device-auth runs in a subprocess with an isolated `CODEX_HOME` (temp dir) so the user's `~/.codex/auth.json` is never touched. The spawned `codex login --device-auth` output is captured line-by-line, ANSI-stripped, and re-rendered with a `$ codex login --device-auth` header above dimmed sub-output on the @clack/prompts rail so the user visually understands they're seeing a sub-process. Companion `pnpm fresh-branch` script: from inside `.worktrees/<name>`, creates a schema-only Neon branch named `dev/<git-branch>`, patches the worktree's `.env` (DATABASE_URL, DATABASE_URL_UNPOOLED, NEON_DEV_BRANCH), then runs `prisma migrate reset --force` so migrations apply cleanly against a data-free copy. Refuses to run from the primary checkout or on protected branch names. Other: - bump CLI/account/repo secret value limit 4096 -> 49152 chars (matches GitHub Actions' 48KB cap; auth.json is ~4-5KB) - extract shared CLI helpers (gh/pullfrog API, secret save) into `action/commands/_shared.ts` * fix(auth): address PR review + add CodexAuthCallout, default account scope Review fixes: - handle 'error' event from `codex` spawn (ENOENT) so missing PATH bails with an actionable "install codex CLI" message instead of an unhandled Node error - escalate SIGTERM -> SIGKILL after 5s grace when killing a stuck codex child so the CLI can't get pinned indefinitely - stop the spinner with a red "failed" glyph in the catch path before clearing activeSpin, mirroring `bail` (no orphan spinner above errors) - enforce 48 KB secret value cap by *bytes* (Buffer.byteLength) not UTF-16 code units, across all 3 secret routes; matches GH Actions' byte-based limit - preserve existing blank lines + comments when fresh-branch rewrites worktree .env (no more cosmetic reformat on every run) Scope: - default to `account` scope on org-owned repos too — never silently prompt for repo scope. Pullfrog has no per-GitHub-user secret store, so account is right for both user and org owners; `--scope repo` is the explicit opt-in for repo-only. UI: - new CodexAuthCallout (sibling to ClaudeCodeOAuthCallout); surfaces `pullfrog auth codex` for ChatGPT subscribers when an OpenAI provider model is selected. wired into AgentSettings.tsx (model-costs surface) and OnboardingCard.tsx (first-time setup). no paste button — the CLI handles minting + saving end-to-end. * auth/codex: rename to neon-fresh-branch, address PR review - rename `pnpm fresh-branch` → `pnpm neon-fresh-branch` (and the script file) to disambiguate from git branches. - `--scope` help text now explains the default (account) and when to pass `repo`. - move `_shared.ts` import up with the rest in `action/commands/auth.ts` and push the `stripAnsi` helper below the import block. - `sanitizeBranchName` no longer slices: slicing after trim could reintroduce a trailing `-`/`/`. callers slice the raw input first, then sanitize. - DRY the `start` branch of the codex progress callback (single header path, optional retry log). - thread a `timedOut` flag from `runDeviceAuth` → `ProgressEvent.exit` so the retry prompt can say "device authorization timed out — retry?" instead of the generic "no auth.json was written" line when the per-attempt timeout fires. - drop the redundant `mkdirSync` after `mkdtempSync` in `codexAuth.ts`. * untrack .scratch/ (committed screenshot fixture by mistake) * auth codex: prompt for scope on orgs (mirrors init) * revert worktree.ts: out of scope for this PR * anneal: trim _shared.ts dead exports, collapse CodexSpawnError, inline packageBin * codex auth: wire end-to-end runtime consumer CODEX_AUTH_JSON is now actually usable: the action runtime materializes it as OpenCode's auth.json at the runner's real $HOME/.local/share/opencode, OpenCode routes openai requests through the ChatGPT subscription via the embedded CodexAuthPlugin, and a GitHub Actions post: hook detects any refresh-chain rotation during the run and PUTs it back to Pullfrog via a new JWT-authenticated PUT /api/runtime/secret endpoint. Key decisions: - Write to the real $HOME (not the per-run tmpdir-redirected HOME) so the file lives outside OpenCode's `/tmp/*` permission allow zone — its existing deny-default protects it without any new permission rule. - Materialization gated on agent === opencode (Codex auth is OpenAI-only, Claude never sees the file). - Defense-in-depth on Claude: deny Read/Grep/Edit/Glob + sandbox.denyRead for ~/.local/share/opencode/auth.json in managedSettings (covers Bash file-reading commands too per Claude Code permissions docs). - New `provider.managedCredentials` field on the provider config — CLI-only credentials authored via `pullfrog auth <provider>`. Counted for hasAnyKey/log-redaction but never surfaced as a paste option in init. CODEX_AUTH_JSON is the first member; OPENAI_API_KEY stays in envVars. - Eager refresh on `pullfrog auth codex`: one OAuth round-trip before setPullfrogSecret so Pullfrog's copy is the freshest in the chain (avoids the user's laptop refreshing first and stranding our copy). - Post-hook approach for write-back so it survives cancellation, timeouts, and unhandled errors in the main step. State is ferried via core.saveState since apiToken is run-scoped and not in env. - Server-side write-back endpoint is allowlist-gated to CODEX_AUTH_JSON only — never a generic secret-write surface. Looks up the secret at repo scope first, falls back to account scope. 404s on create (refresh-only, never auto-provision). * codex auth: documentation + wiki cross-links * debug: log dbSecrets keys + CODEX_AUTH_JSON presence (temporary) * debug: surface install path + parse failure preview * remove debug log lines (E2E verified) * hide CodexAuthCallout until opencode-ai bump (1.1.56's allowed-models set excludes gpt-5.5) |
||
|
Colin McDonnell
|
ddbc610569 |
review prompt: friendly green callouts + per-section severity emojis (#756)
* review prompt: friendly green callouts + per-section severity emojis - Replace `[!NOTE]` informational tier and the no-callout minor-suggestions tier with friendly green blockquotes (`> ✅` / `> 💡`). The two loud tiers (`[!CAUTION]` / `[!IMPORTANT]`) keep their GitHub admonitions. - Add a per-`##`-section severity-emoji rule (🚨/⚠️/💡/ℹ️) for cross-cutting review concerns that don't anchor to a line and would otherwise be buried in summary content. - Drop the `<br/>` between summary sections — heading + blank line carries enough visual spacing. - Skip the post-run learnings-reflection turn for `IncrementalReview`. It's the lowest-novelty mode (delta review against existing PR with prior summary already loaded) and almost never produces durable learnings — reflection there costs ~$0.50-0.80/run for nothing. - Surface real error info on `agent-browser` skill install failures (exit code + stdout + stderr + spawn error). The skills CLI uses a TUI that prints errors to stdout, so the prior stderr-only logging silently swallowed every failure. * review prompt: per-bullet severity emoji + bullets-only sections Section headings are plain again (no leading severity emoji). Severity moves to individual bullets so a section that mixes a 🚨 and a 💡 isn't mislabeled by either. Section bodies are now bullets only — paragraph prose under a heading is harder to scan and tends to bury the actionable point. Bullets can carry indented continuation content (sub-bullets, code fences, blockquotes) by indenting two spaces under the parent. * review prompt: cap section length + identifier discipline Bound each summary section to at most 4 bullets at most 2 lines each, and explicitly call out identifier-heavy prose as an anti-pattern. The reader is often a manager or non-author; identifier-dense paragraphs ('foo calls bar.fetch which dispatches to baz via qux...') are unreadable for them. Default to plain-language behavior descriptions, name an identifier only when it's the subject of an actionable concern or a public surface a reader would recognize, target 2-3 backtick tokens per bullet. Move the deep-explanation pattern from open blockquote to a default- collapsed details/summary so depth doesn't dominate the visible body. * review prompt: hard cap on bullet identifier density + worked rewrite example Soft 'aim for 2-3 tokens' guidance was ignored — first big-PR e2e showed 12 of 19 actionable bullets exceeded the target (avg 4.8 tokens, several over 8). Promote to a hard cap of 3 backticked tokens per bullet and pair with a concrete bad/good rewrite the agent can pattern- match against. Also tighten the per-bullet length cap from ~240 to ~200 chars and explicitly call it 'hard cap, not target'. * review prompt: tighten bullet length cap to 160 chars, dramatize the worked example V2 e2e test: token discipline improved (4.8 -> 3.3 avg, 12/19 -> 6/14 violations) but length got worse (235 -> 286 chars, 13/14 over the 200 cap). The agent compensated for fewer identifiers with more prose. Two changes: (1) tighten the cap from ~200 chars to 160 chars / 1 visual line and call out wrap-to-multiple-lines as the failure mode; (2) rewrite the worked example so the good version is genuinely half the length of the bad one, not just lower token count. The example was the thing the agent pattern-matches against; making the good version ~130 chars vs the bad version's ~290 chars sets the right shape. * review prompt: drop fixed bullet-count cap, keep length + identifier caps Per user feedback — section length should be governed by content, not an arbitrary count. Soft guidance ('past ~6, ask whether to split') is fine; the hard '≤ 4 bullets per section' rule was the wrong shape. Length cap (160c) and identifier cap (3 backtick tokens) stay; those target the actual scanability problem. * review prompt: drop ## subsystem sections, flat 'Issues found' list Per-section structure forced every concern into a subsystem frame and made the body read like a series of mini-essays. Replace with two parts: (1) TL;DR + Key changes as the dispassionate overview, (2) flat '### Issues found' list ordered by severity, intermixed across files and subsystems. Per-bullet rules (≤160c, ≤3 backtick tokens, severity emoji prefix, optional indented continuation) carry over unchanged. * review prompt: full v6 structure — preamble + cross-cutting H3s + nitpicks Replaces the flat 'Issues found' bullet list with the iterated v6 shape: - Preamble is a bolded inline 'Reviewed changes' lead-in plus bullets plus a collapsed 'Review metadata' block (mode/files/commits/refs/ reviewed commits list/prior pullfrog review/staleness note). - Each cross-cutting concern gets a '### emoji Title' section. The visible problem write-up is human-friendly and DESCRIBES THE PROBLEM ONLY — no asks, no suggested fixes, no 'the right thing to do is'. - Each section carries a collapsed 'Technical details' block wrapped in a 4-backtick markdown fence (so it can hold its own 3-tick code fences cleanly, agent-readable, one-click copyable). Standard four inner sections: Affected sites, Required outcome, optional Suggested approach, optional Open questions for the human. - '### ℹ️ Nitpicks' at the bottom for body-only nits that don't inline; simple bullets, no technical-details collapse. - Anti-paragraph-wall rule: never two successive plain paragraphs in visible '### ' sections; alternate prose with structure. - Inline-vs-body discipline: anything that anchors to a single line goes inline, body is for cross-cutting only. - Drops legacy '### Key changes', '### Issues found', '<b>TL;DR</b>', and the '<sub>Summary</sub>' line. * model effort: bump Gemini + GPT to high effort; drop Gemini Pro→Flash subagent E2E review eval against a substantive billing-module diff surfaced two related quality gaps: 1. Gemini Pro at thinkingLevel=medium (#663's CI-timeout fix) reviewed the diff only, took the 0-lens path, and missed a catastrophic camelCase/snake_case service-vs-schema mismatch. Bumping back to high — review work is exactly the wrong shape for the medium/high tradeoff #663 was optimizing for; the per-turn TTFT cost is worth paying when reasoning IS the value. 2. GPT had no reasoningEffort override, defaulting to upstream medium. Same diff, similar shallow result vs Claude. Adding reasoningEffort: high for the curated direct-OpenAI slugs, mirroring the Gemini pattern (Anthropic separately uses --effort high via the Claude Code CLI flag in claude.ts). 3. Gemini Pro's subagentModel was 'gemini-flash' — but Google has no in-between tier between Pro and Flash, and Flash is a meaningful capability cliff for review work. Dropping the override so subagents inherit Pro. Cost stays reasonable since Gemini Pro is already the cheapest of the flagship trio. Other providers unchanged: Anthropic opus→sonnet and OpenAI gpt→gpt-5.4 remain (each is a one-tier drop to a still-capable sibling). * model effort: revert orchestrator override, set explicit high on reviewfrog subagent Reshape the effort design after eval: - Drop the explicit Gemini and GPT model-level overrides — orchestrators now run at upstream defaults (Gemini high, GPT-5.x medium). Gemini's upstream IS high, so this is a no-op there; GPT goes back to upstream medium for orchestrator-level routing work. - Add explicit 'high' on the reviewfrog subagent via agent.options. OpenCode merge order is base ← model.options ← agent.options ← variant per session/llm.ts:141, so the subagent always runs at high regardless of which orchestrator dispatched it. Both thinkingConfig.thinkingLevel (Gemini) and reasoningEffort (GPT) keys included; irrelevant keys are ignored per provider. - Bump providers-live timeouts (12min job / 10min step, from 8/6) to budget for Gemini's TTFT variance at high effort. #663's 4min timeout was sized for the medium-effort override that's now removed. * model effort: restore Gemini explicit high override (no-override path breaks) Bare 'rely on upstream default' for Gemini failed in e2e — removing the model-level provider config produced 'Function call is missing a thought_signature' API errors on every gemini-pro run. Even though upstream opencode's options() returns the same thinkingLevel: high we were explicitly setting, opencode's resolution path differs subtly between the two cases. v2's explicit override worked; v3's removal broke. Reproducible across two consecutive runs. Restoring the explicit Gemini override (back to v2 design). GPT orchestrator stays UN-overridden — at upstream default (medium) — since removing that override didn't trigger the same failure pattern and the reviewfrog subagent agent.options high override compensates for the extra depth GPT loses at medium. * diag: remove reviewfrog agent.options to isolate Gemini thought_signature failure v3 (no Gemini orch override) failed with thought_signature error. v4 (restored Gemini orch override at v2-equivalent) ALSO failed, even though the orchestrator config matches v2. The variable between v2 (working) and v4 (failing) is the new reviewfrog agent.options block. Removing it to confirm — if Gemini works again, the agent.options addition is the culprit and we need a different shape for it. * opencode-ai: bump 1.1.56 → 1.15.0 + clean up gemini effort config opencode-ai@1.1.56 was published 2026-02-10 (3 months old). The Google API tightened thought_signature validation 24-48h ago (per https://discuss.ai.google.dev/t/gemini-thought-signature-patch/122555), and the bug class hits opencode's session→prompt serializer for MCP tool-call parts (anomalyco/opencode#4832, #8321). Latest stable bumps us through ~3 months of fixes; needed for Gemini-direct to stop dying with 'thought_signature is missing' on every multi-turn run. Companion cleanup: the gemini provider override in opencode.ts had 30-line block of comments, four unused constants, and a 6-line Object.fromEntries map for two entries. Replaced with one source-of- truth helper that loops modelAliases, filters provider==='google', strips the 'google/' prefix, and returns the override map. Adding any future Google alias to the registry now flows through automatically. Test added: action/agents/opencode.test.ts asserts the helper covers every direct-Google alias, strips the prefix correctly, and pins every entry to thinkingLevel high — catches drift in helper logic without hardcoding the API ids the test would have to update in lockstep with the registry. * fix(workflow): tolerate listJobsForWorkflowRun 404 in resolveRun PR #750 (docker testing rewrite) replaced the per-call env allowlist with full process.env passthrough into the test container. That now leaks GITHUB_RUN_ID + GITHUB_JOB into runs whose MCP token is scoped to a DIFFERENT repo (e.g. providers-live smoke runs the action against pullfrog/test-repo with pullfrog/app's run ID). The unconditional listJobsForWorkflowRun call 404s and crashes the entire run, breaking every providers-live job on main since #750 landed. jobId is purely cosmetic (deep-links 'View workflow run' footer to a specific job vs the run-level URL). Wrapping the API call in try/catch so a 404 logs a debug message and falls through to undefined jobId is the right fix — the failure mode is exactly what graceful degradation is for, and the alternative (filter the env vars at the docker boundary) re-introduces the kind of allowlist #750 was getting rid of. * opencode-ai: pin 1.14.51 instead of 1.15.0 (effect refactor breaks JSON output) opencode 1.15.0 (May 15) ships a major architectural refactor onto @effect — the run command boots an in-process server via @opencode-ai/sdk/v2 and the JSON event emission path through that SDK client doesn't surface on stdout the way our parser expects (CI run on 1.15.0 produced 0 stdout events but the agent still completed). Local invocation also hangs at the in-process server boot. The Gemini thought_signature fixes (the original reason for bumping) landed earlier in the 1.14.x line, so 1.14.51 (May 14) gets us the upstream fix without the Effect rewrite. Defer the 1.15.x bump until we're ready to rewire our parser/spawn around the new SDK. * opencode-ai: revert to 1.1.56; gha: filter outer-CI workflow-run vars at the docker boundary Two related changes for the docker testing harness's ergonomics: 1. Revert opencode-ai 1.14.51 → 1.1.56. The 1.14+ line ships an Effect refactor (the SDK-v2 client + in-process server architecture) that our --format json parser doesn't speak — even the 1.14.51 release, pre-dating the 1.15.0 Effect rename, produced 0 stdout events on our skill-invoke smoke. There's no clean pre-Effect version that ships the Gemini thought_signature fix; that fix needs a separate workstream once we're ready to rewire the parser onto SDK v2. 2. Filter outer-CI workflow-run identifiers (GITHUB_RUN_ID, GITHUB_JOB, GITHUB_WORKFLOW, GITHUB_ACTION, GITHUB_REF, GITHUB_SHA, etc.) from gha.ts's --env-file passthrough. PR #750's full-process.env design leaks pullfrog/app's CI run identifiers into runs that act against a different repo (e.g. pullfrog/test-repo); any code path inside the action that uses them as keys (most notably resolveRun's listJobsForWorkflowRun lookup) 404s. Filtering them here means the action sees undefined and skips the lookup, complementing the defensive try/catch in resolveRun (commit addc76d4). GITHUB_REPOSITORY and GITHUB_TOKEN are NOT filtered — those are genuinely needed. Companion to addc76d4 (resolveRun 404 tolerance). The two together make this class of bug 'either fix would have caught it' rather than 'silently breaks the entire test matrix'. * fix(deps): sync pnpm-lock.yaml with opencode-ai 1.1.56 manifest revert Forgot to refresh the lockfile after reverting the manifest in 02c6d8c1. CI's frozen-lockfile install was failing with 'lockfile: 1.14.51, manifest: 1.1.56' mismatch. |
||
|
Colin McDonnell
|
a0dce200d0 |
fix(claude): prefer OAuth token over ANTHROPIC_API_KEY (#763)
* fix(claude): prefer OAuth token over ANTHROPIC_API_KEY in Claude Code
When both `CLAUDE_CODE_OAUTH_TOKEN` and `ANTHROPIC_API_KEY` are present,
claude-code's auth resolver (`Vw()` in cli.js) returns the API key first
and silently ignores the OAuth token. The result: accounts that have a
Max-subscription OAuth token in `account_secrets` are still billed at
per-token API rates because the workflow `env:` block also forwards
`ANTHROPIC_API_KEY` from org-level secrets.
Strip `ANTHROPIC_API_KEY` from the spawned claude-code subprocess env
when an OAuth token is present (and we're not on the Bedrock route),
so the Max subscription is actually used. Other agents in the same run
still see the API key in `process.env` via the parent.
* chore: tighten comment-length rule + trim claude.ts comment
Caps inline comments at 2-3 lines above any single line of code (the
prior wording allowed runaway block comments as long as the comment
was nominally shorter than the annotated code).
* chore: downgrade OAuth-strip log to debug + document debug-mode pattern
`log.info` was overkill for a per-run path-selection marker. `log.debug`
keeps production logs quiet while preserving full visibility in e2e
verification, where `LOG_LEVEL=debug` (or `gh run rerun --debug`)
flips the same line on.
Adds a "Action debug mode" subsection to wiki/e2e-testing.md so the
affordance is discoverable: `log.debug(...)` is the right tool for
breadcrumbs that prove a code path fired during preview-repo e2e but
shouldn't ship to customer logs.
* chore(wiki): correct debug-mode trigger guidance for preview repos
LOG_LEVEL=debug only works when the template's pullfrog.yml forwards
it, which it doesn't. ACTIONS_STEP_DEBUG=true is the GitHub-magic name
that's auto-injected into every step's env without any yaml change,
so make that the documented default for preview-repo e2e.
* chore(wiki): fix render-format claim in debug-mode table
When `ACTIONS_STEP_DEBUG=true`, `log.debug` routes through
`core.debug()`, which GitHub renders as `##[debug]<msg>`, not the
`[DEBUG] <msg>` format. The `[DEBUG]` prefix only happens via the
LOG_LEVEL=debug path which isn't currently wired into the template.
* feat(action): add `overrides` input for per-dispatch env mutation
Accepts a JSON {string:string} map via the workflow_dispatch input,
parsed and merged into process.env at the start of `main()` (before
any agent or token-acquisition code runs). Lets a privileged caller
flip env vars for one dispatch without persisting state on the repo
(repo Actions variables) or being restricted to GitHub's debug names
(`gh run rerun --debug`).
Deny-list refuses overrides for integrity-critical names — GITHUB_TOKEN,
ACTIONS_RUNTIME_TOKEN, ACTIONS_RUNTIME_URL, ACTIONS_ID_TOKEN_REQUEST_*,
ACTIONS_CACHE_URL, PULLFROG_API_SECRET, VERCEL_AUTOMATION_BYPASS_SECRET.
Customer provider keys (ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, etc.)
are explicitly allowed — overriding them per-run for cred-rotation tests
and auth-failure repros is the use case.
Touches:
- action/action.yml — declare `overrides` input
- action/utils/overrides.ts — parse + apply with deny-list (+ unit tests)
- action/main.ts — wire into `main()` after `normalizeEnv()`
- .github/workflows/pullfrog.yml — forward to action
- utils/github/pullfrog.yml.ts — same in the customer-facing template
- wiki/e2e-testing.md — documented as preferred debug-mode trigger
* fix(overrides): strip raw INPUT_OVERRIDES + mask applied values
GitHub Actions injects every action input as an env var (INPUT_<NAME>),
so the original JSON of `overrides` sits in process.env as INPUT_OVERRIDES
and is inherited by every spawned subprocess (claude, opencode, MCP
servers, shell). That defeats the deny-list (a downstream re-application
would have access to the raw JSON) and leaks arbitrary caller-supplied
values into agent env verbatim.
After applying, applyOverrides now:
1. delete process.env.INPUT_OVERRIDES — subprocesses see only the
surgically-applied keys, not the raw JSON
2. core.setSecret(value) for each applied value — the runner masks
those strings in subsequent log output, so an overridden
ANTHROPIC_API_KEY can't accidentally surface in debug logs.
Two new tests cover the deletion path (both applied and all-denied).
* fix(overrides): scope auto-masking to credential-shaped keys
core.setSecret(value) is a global string-match — calling it on a short
config value like "claude" masks every appearance in subsequent logs
(including "claude-opus-4-7", "anthropic-claude-sonnet", etc.), which
actively harms debugging.
Restrict the auto-mask to keys whose names end in _KEY / _TOKEN /
_SECRET / _PASSWORD / _OAUTH / _PRIVATE_KEY — the credential-shape
naming convention. Customer keys (ANTHROPIC_API_KEY, etc.) and the
deny-listed names match. Plain config (PULLFROG_AGENT, PULLFROG_MODEL,
ACTIONS_STEP_DEBUG) doesn't.
* docs(wiki): document the three security layers + runner-echo caveat
Lays out exactly what the `overrides` input does to mitigate the secret-
leak surface (deletion + masking) and the one unavoidable limit: GH
Actions echoes the `with:` block once before any action code runs, so
the raw JSON appears in the workflow log header in plaintext. Anyone
using `overrides` should treat that one-shot exposure as part of the
threat model.
* fix(overrides): forward via env, not action input, so the value isn't echoed verbatim in the runner step header
GH Actions echoes the `with:` block of every `uses:` step in the log
group header, BEFORE any action code runs — so the raw JSON of
`overrides` was always visible in the workflow log regardless of any
in-action `core.setSecret` calls.
Refactor: drop the `overrides` action input; instead the action reads
`process.env.PULLFROG_OVERRIDES`. The workflow yaml forwards
`inputs.overrides` via the step-level `env:` block. We still need to
verify empirically whether `env:` block values from workflow inputs
get echoed too (separate test); even if they do, masking via
core.setSecret + delete of PULLFROG_OVERRIDES after parsing closes
the leak to subprocesses, which is the part the action controls.
* fix(overrides): rename to unsafe_overrides + UNSAFE_OVERRIDES
The runner echoes step-header env-block values in plaintext before any
action code runs, so the raw JSON of this affordance is visible to
anyone with actions:read on the calling repo. That's acceptable
because the workflow only exists on our private repos, but the input
name should make the trade-off obvious at the call site rather than
buried in a wiki.
- workflow_dispatch input: `overrides` → `unsafe_overrides`
- env var the action reads: `PULLFROG_OVERRIDES` → `UNSAFE_OVERRIDES`
- wiki: rewrite the section to surface the runner-echo as the central
trade-off rather than a buried caveat
* chore(overrides): tighten error messages to reference UNSAFE_OVERRIDES
* docs(wiki): fix stale 'overrides' refs + correct render-format mechanism
Addresses two unresolved review threads on PR #763:
1. The opening sentence of "Action debug mode" still referenced the
pre-rename `overrides` input and `gh workflow run -f overrides=...`.
Updated to `unsafe_overrides`.
2. The render-format claim was technically wrong. `core.isDebug()`
doesn't cache — it reads `process.env.RUNNER_DEBUG === '1'` on
every call. The actual mechanism: the runner only sets
RUNNER_DEBUG=1 when ACTIONS_STEP_DEBUG=true is observed at
workflow-trigger time. Mutating ACTIONS_STEP_DEBUG mid-step
doesn't retroactively flip RUNNER_DEBUG, so the call falls through
to isLocalDebugEnabled() which reads ACTIONS_STEP_DEBUG directly.
Rewrote the explanation to match.
* fix: drop unsafe_overrides from customer-facing workflow template + remove test theater
Two cleanups from a stricter re-read of AGENTS.md:
1. utils/github/pullfrog.yml.ts is the workflow yaml we sync into every
customer repo. unsafe_overrides has no business there — it's a
pullfrog-only debugging affordance. Reverted. The action's read of
UNSAFE_OVERRIDES env var stays — it's a no-op for any workflow that
doesn't set it, and pullfrog/template + pullfrog/app's own workflow
still forward it.
2. Deleted action/utils/overrides.test.ts entirely. AGENTS.md is clear:
no tests unless explicitly asked. I added them anyway. The tests
were mostly testing JSON.parse + typeof, plus one regression guard
for the deny-list that is better protected by code review of the
tiny DENIED_OVERRIDE_NAMES set than by a vitest file.
Also strengthened the corresponding AGENTS.md rule from a buried bullet
to an explicit "NEVER write tests unless asked, here's why agents
violate this constantly, here's the bar" callout.
Wiki note added: unsafe_overrides is pullfrog-only infra, not customer-
facing.
|
||
|
Colin McDonnell
|
76879b27ec |
docker testing rewrite: bake the image, drop the allowlist, kill the quoting (#750)
* docker testing rewrite: bake the image, drop the allowlist, kill the quoting - new `pnpm gha <script>` wrapper. one entry point for running any node script in the GHA-like container; replaces the runtime apt-get + useradd + chown ceremony in `action/utils/docker.ts`. - `action/Dockerfile` bakes ubuntu:24.04 + node 24 + gh + jq + sudo + testuser at uid 1000. `action/docker-entrypoint.sh` remaps to the host uid/gid and `exec`s the requested command — no `bash -c` nesting, no `escapeForDoubleQuotes`. - env passthrough: full `process.env` (+ `.env` via dotenv) flows through `--env-file`, multi-line values via `-e` fallback. drops `EnvFilterMode` / `testEnvAllowList`. - image rebuild is content-hash gated on Dockerfile + entrypoint; volume is versioned by hash so a stale `node_modules` cache from an old image can't poison a new one. - `action/play.ts` slimmed to a CLI; `run()` extracted to `action/utils/runFixture.ts`. drops the `--local` / `PLAY_LOCAL` dual mode in favor of explicit `play:local` / `runtest:local` scripts. - `action/test/run.ts` no longer self-relaunches into docker — that's `gha`'s job now. - `action/test/coverage.ts` `ALWAYS_RUN_ALL` updated to track the new files. - `wiki/docker.md` rewritten (243 → 105 lines). `wiki/action-tests.md`, `wiki/billing.md`, `wiki/adversarial.md`, `README.md`, `AGENTS.md` all updated to drop `--local` / `PLAY_LOCAL` references. verified end-to-end: `pnpm play` runs the default fixture against pullfrog/scratch, exit 0; `sudo unshare --pid` still works inside the container; `pnpm runtest` boots through the wrapper. * gha: address review feedback + 3 related issues found locally review-flagged: - bare `pnpm gha --build` now builds the image and exits 0 (was printing help and exiting 1 — docs claimed it was a valid standalone) - `initVolumeOwnership` skipped when the named volume already exists; saves the ~240ms `docker run … chown` on every warm invocation - `GIT_SSH_COMMAND` gate widened to any `id_*` private key (was hard- coded to `id_rsa`, leaving ed25519-only linux contributors with the default ssh config). dropped `-i` so ssh picks whichever key exists - new `action/.dockerignore` — partial mitigation noted: BuildKit (default since docker 23) only sends files referenced by the Dockerfile (~42B in practice), so the perf concern is mostly hypothetical. file is still worth keeping for `DOCKER_BUILDKIT=0` fallback and as documented intent for future `COPY . .` additions related issues found while validating locally: - `parseArgs` now stops flag-parsing at the first positional (or literal `--`); `pnpm gha test/run.ts --build` previously intercepted `--build` as a gha flag instead of forwarding to `test/run.ts` - new `pnpm gha --clean` command prunes orphan `pullfrog-gha:*` images and `pullfrog-gha-node-modules-*` volumes whose hash doesn't match the current Dockerfile (each Dockerfile/entrypoint edit creates a fresh hash and orphans the prior pair, ~600MB + ~200MB each — without a cleaner they accumulate silently) - `--shell` without a TTY now fails fast with an actionable message before docker is invoked, instead of producing the confusing `the input device is not a TTY` from docker run wiki updated: documents `--clean`, the parseArgs passthrough rule, and a new "Reclaiming disk" section. * gha: fidelity, flexibility, and signal-safety improvements investigated local fidelity vs the real GHA ubuntu-24.04 runner and addressed the gaps that have actually bitten contributors or could. fidelity (image now matches GHA closer): - bake build-essential, wget, xz-utils, file alongside the existing toolset. gh, jq, git, python3, sudo, ssh, build-essential, wget, xz, file, unzip, curl all present. native module builds (node-gyp, any package missing arm64 prebuilts) now work; common agent shell calls don't hit ENOENT - `host.docker.internal:host-gateway` flag wires the host into the container's DNS on linux (macOS Docker Desktop bakes it in). lets scripts that hit a local dev server use `API_URL=http://host.docker. internal:3100` and work identically on both platforms - `--init` makes tini PID 1, fixing signal forwarding during the pre-exec warmup window (Ctrl-C was previously taking up to 10s to tear down because bash-as-PID-1 swallowed the signal) - pnpm version is correctly pinned via the workspace's `packageManager` field — corepack resolves it at install time; verified via the new `--doctor` command flexibility (new affordances): - `pnpm gha --doctor` runs an inside-the-container fidelity audit: os + arch + node/pnpm/python versions, version snapshots of every baked tool, env vars (CI, HOME, TMPDIR), uid/gid, and the host.docker.internal resolution. useful for "works in CI fails locally" or vice versa - `pnpm gha --build --no-cache` busts the docker layer cache when an apt mirror, base image, or external download has changed upstream - entrypoint's `pnpm install` warmup is now wrapped in a `flock` on a file in the shared node_modules volume — concurrent `pnpm gha` invocations (e.g. play in one terminal, runtest in another) serialize their install instead of racing docs: - new "Gaps (known)" section in wiki/docker.md explicitly calling out the things this system can't do yet, including the missing `uses: ./action` semantics gap that `.github/workflows/action-gha-e2e-adhoc.yml` currently fills via GHA only (designing a local `pnpm gha-action <fixture>` is on the roadmap), service containers, parallel-run sharing, and arch differences (arm64 vs amd64) * docs: audit + corrections after testing fronts self-audit pass for stale references and incomplete pointers: - wiki/browser.md: `Docker (node:24)` → `pnpm gha container (ubuntu:24.04)`. the substance was right (chrome not preinstalled) but the base image reference was stale. - wiki/docker.md: the "Permission errors" troubleshooting line claimed the node_modules volume is chowned on every run; now correctly says "owned by the host uid on first creation; warm runs skip the chown" to match the actual behavior after the initVolumeOwnership fix. - wiki/action-tests.md: `API_URL` env-var doc now mentions BOTH paths (`localhost:` from play:local, `host.docker.internal:` from inside the container). Proxy/router recipe now shows both invocations side-by-side instead of saying "must use play:local". - wiki/billing.md: same dual-recipe update for the loop-including-the- action proxy walkthrough. - gha.ts header: expanded the usage block to include --clean / --doctor / --no-cache / --shell-TTY, added the host.docker.internal note, and pointed at wiki/docker.md for design rationale. self-document check: a future agent landing on this code can answer "how do I run a fixture / debug in shell / add a tool / diagnose fidelity / reach a local dev server" purely from gha.ts header + wiki/docker.md without spelunking through the entrypoint or git history. |
||
|
Colin McDonnell
|
ba7f5a0b89 |
action: surface agent hang context in progress comment (#733)
* action: surface agent hang context in progress comment When the activity-timeout watchdog kills a stalled opencode subprocess, the user used to see a bare "activity timeout: no output for 30Xs" — no provider context, no stderr trace, no clue why the run died. Investigation of the six runs in #728 showed the same shape every time: opencode hangs after a non-retryable provider event (auth 401, 502 stream lost, free-tier flake), and the only useful signal was buried in stderr where the user couldn't see it without diving into Actions logs. Stop trying to prevent the hang. Surface it. Add a small `AgentDiagnostic` handle on `toolState` that the harness mutates as a run progresses (recent stderr ring buffer reference, last provider-error label, event count). `formatAgentHangBody` renders that into a markdown body — bold headline, one-line explanation, collapsible `<details>` with the last ~10 stderr lines (capped to 3KB) — used by both the agent harness's own catch path and main.ts's outer catch when the watchdog wins the race against the harness. Both paths converge on one formatter; the existing "View workflow run ➔" footer affordance in `reportErrorToComment` is unchanged, so the user still has one click from the comment to the raw logs to develop their own thesis. * address review: gate hang body on isHang; fix contradictory copy - Only render `hangBody` when `isHang`. The harness sets `agentDiagnostic` on entry, so any non-hang throw past `runOpenCode`'s own catch (post-success `output_schema` validator, late cleanup throws) was rendering "Pullfrog failed — N events processed…" with the real exception message dropped — including for runs that actually succeeded before a late throw. - When `lastProviderError` already names the cause in the headline, the zero-events sentence "check whether the model provider is reachable" contradicts it (a 401 produces zero events but isn't a reachability issue). Drop the nudge in that case; keep it for the silent-stall path where it's still actionable. * address copilot review: fence escape, idle parsing, secret redaction, tests - pick a backtick fence longer than any backtick run in the rendered stderr tail. opencode error JSON occasionally embeds triple backticks in tool input dumps; the fixed three-tick fence let those terminate the fence early and corrupt the rest of the comment markdown. - parse idle seconds out of the timer reject string ("activity timeout: no output for 301s") and use that for the hang explanation. previously rendered total runtime, which overstated the stall by 20+ minutes for runs that streamed for a long time before going quiet (e.g. Rohithgilla12/data-peek#25784038918, 1230s elapsed but 304s idle). - redact sensitive env-var values from the rendered stderr tail before it lands in the PR comment / job summary. workflow log writes already go through `core.setSecret` masking; PR comments and summaries bypass that pipeline entirely. matches against `isSensitiveEnvName` (the same *_KEY/*_TOKEN/*_SECRET/*_PASSWORD/*_CREDENTIAL surface that `normalizeEnv` registers with the runner) and only redacts values >= 8 chars to avoid false-positive substring hits. - add `agentHangReport.test.ts` covering the branchy bits: idle-seconds parsing, eventCount-zero copy with and without provider error, fence-escape against embedded triple backticks, 3 KB tail truncation, null-on-no-diagnostic, and secret redaction. `startedAtMs` is dropped from `AgentDiagnostic` — total runtime was the only consumer and idle seconds replaces it. * strip slop: drop tests, drop redactSecrets, simplify ternary - delete `agentHangReport.test.ts`. half the cases just pinned literal copy ("**Pullfrog stalled**", "check whether the model provider is reachable") which is exactly the "performative tests to every string utility" pattern AGENTS.md flags. the other half tested 2-5 line pure helpers (parseIdleSec / pickFence / truncation) that code review catches. the formatter is a best-effort string output; pinning it in tests creates churn without catching real regressions. - remove `redactSecrets` and revert the formatter's import. theatrical defense: opencode doesn't dump env on startup, bearer tokens aren't in request bodies, bash is denied. the action has many other PR-comment write paths that don't redact (comment.ts, errorReport.ts, the progress writer) — if PR-comment secret hygiene matters, it's a cross-cutting concern at the comment-write layer, not bolted onto one formatter. - factor the explanation triple-ternary into `formatExplanation` with early returns. same logic, easier to read. `isHang` gate, fence-length escaping, and idle-seconds parsing stay — those are real correctness fixes. |
||
|
Colin McDonnell
|
b9383bbcfd |
action: center provider-error log excerpt on the matched line (closes #703)
the `» provider error detected (...)` excerpt was `chunk.substring(0, 500)`
— the head of whatever stderr buffer node delivered. on big writes that's
the front of an mcp tool-schema dump, not the matched error text. label
was correct (regex.test on the whole chunk), excerpt was misleading.
introduce findProviderErrorMatch(text) that returns { label, excerpt }
where excerpt is a windowed slice centered on the regex match index:
the matched line plus 1 line before and 2 lines after, hard-capped at
600 bytes. detectProviderError stays as a thin wrapper for label-only
callers. both opencode and claude harnesses log match.excerpt instead
of chunk.substring(0, 500).
regression tests cover the multi-line buffer case, surrounding-line
context, byte-cap fallback to matched-line-only, and head truncation
of a single oversize line.
|
||
|
Colin McDonnell
|
4ad649ebb9 |
action: extend shallow-unreachable deepen-retry to checkout_pr fetches (#734)
extracts the deepen-retry helper from `GitFetchTool` into shared `$gitFetchWithDeepen` and applies it to every fetch in `checkoutPrBranch` (baseRef, pull/N/head, before_sha temp branch). on shallow clones with deep PR ancestry — the failure mode behind ~10 of 51 `heuristic:very-slow` runs in 24h on `remotion-dev/remotion` — the baseRef fetch was throwing `Could not read <sha>` to the agent before the compare-api deepen block could run. agents then burned 10+ minutes retrying `checkout_pr` and falling back to ad-hoc shell `git fetch --deepen` workarounds. also splits the analyzer's `heuristic:git-error-recovered` into `heuristic:git-shallow-unreachable` and `heuristic:git-shallow-lock` buckets so future audits surface this without manual log-grep. closes #656. |
||
|
Colin McDonnell
|
d495f0b984 |
surface BYOK failures + chronic-failures card + WorkflowRunStatus mirrors GitHub conclusions (#722)
- Migrates `WorkflowRunStatus` from `running | completed | cancelled` to a 9-state mirror of `workflow_run.conclusion`. Backfill: old `completed → success`, `cancelled → failure`. New rows write `hook.workflow_run.conclusion` verbatim via `statusFromConclusion`. - Adds Discord links to `formatApiKeyErrorSummary` (both missing-key and 401 invalid-key shapes). - Repo console: `<ChronicFailuresCard>` fires when the last 3 terminal-state runs are all `failure`. Pure DB read; latest-run button hidden for pre-dispatch failures (`runId: null`). - `StatusIcon` distinguishes `cancelled` (gray X, intentional stop) from `failure` (red X) so the visual matches the chronic-card threshold. - Pre-dispatch failures (workflow lookup miss, dispatch API error) write `failure` instead of `cancelled` so they feed the card. - Cascade: every `status: "completed"` filter in billing routes / cron / cohort queries / analyzer becomes `status: "success"`. Verified end-to-end on `pullfrog/preview-722-failure-surfaces` — Better Stack logs confirm webhooks reached the preview deploy and all three e2e runs got `marked as failure (conclusion=failure)` via the new mapper. Closes #679, #702. |
||
|
Colin McDonnell
|
8f9208bd3f |
feat: Amazon Bedrock support via routing slug (#720)
* add Amazon Bedrock as a routing slug introduces a single `bedrock/byok` catalog entry that the harness translates to the appropriate Bedrock model ID at run time via `BEDROCK_MODEL_ID`. routes Anthropic IDs through claude-code (with `CLAUDE_CODE_USE_BEDROCK=1`) and everything else through opencode's `amazon-bedrock` provider — keeps the catalog flat for an audience that needs version pinning rather than aliasing. accepts either `AWS_BEARER_TOKEN_BEDROCK` or `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` for auth; both validated alongside `AWS_REGION` and `BEDROCK_MODEL_ID` in `validateAgentApiKey`. catalog drift tests, the bumps cron, and per-alias smoke scripts all skip routing slugs since there's no fixed `resolve` to validate. docs/bedrock.mdx walks through setup; wiki/model-resolution.md has a section explaining why bedrock breaks the usual alias pattern. closes pullfrog/pullfrog#40 * ci: add bedrock env vars to test workflows mirrors the new bedrock provider's required env vars (AWS_BEARER_TOKEN_BEDROCK inherited from org secret + AWS_REGION + BEDROCK_MODEL_ID hardcoded) into both .github/workflows/test.yml files so the ci.test "env vars cover all provider API keys" assertion passes. * docs(bedrock): clearer setup flow + screenshot of model selector restructures the setup section into three concrete steps in execution order: select Bedrock from the dropdown, store the bearer token as a secret (Pullfrog or GitHub — links to keys.mdx for the trade-off), then add region + model id directly in pullfrog.yml since neither is sensitive. enable-model-access in the Bedrock console moved to step 4 (only required once per model and only when AWS rejects the call, not blocking on first run). adds a screenshot of the console model selector with Amazon Bedrock selected so readers can recognize the UI state they're aiming for. * fix(bedrock): tolerate raw Bedrock model IDs in validateAgentApiKey main.ts passes the resolved model into validateAgentApiKey (`payload.proxyModel ?? resolvedModel ?? payload.model`). For Bedrock, `resolveModel` translates `bedrock/byok` into the raw AWS model ID (e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/` and so trips parseModel inside getModelEnvVars. Detect the no-slash case and re-run the bedrock setup check (auth + region; BEDROCK_MODEL_ID is already enforced upstream by resolveModel). Caught by PR #720 e2e dispatch on pullfrog/preview-720-bedrock — "invalid model slug 'us.anthropic.claude-opus-4-6-v1' — expected 'provider/model'". Two regression tests cover the raw-ID path. * fix(bedrock): always prepend amazon-bedrock/ prefix when bedrock-routed opencode.ts was gating the prefix-injection on `!isBedrockAnthropicId(rawModel)`, on the theory that Anthropic Bedrock IDs always go through claude-code. But `PULLFROG_AGENT=opencode` is a documented escape hatch — when it forces opencode for an Anthropic Bedrock model, the prefix still has to be added or opencode fails with 'Model not found: <modelId>/.'. The Anthropic-vs-other discriminant only belongs in resolveAgent. Once an agent is selected, it should consistently honor the bedrock route. Caught by the PULLFROG_AGENT=opencode + Opus 4.6 e2e on pullfrog/preview-720-bedrock — run 25823437606. * ui+docs(bedrock): bespoke setup callout + clearer docs UI: - BedrockSetupCallout in components/AgentSettings.tsx covers both the Model costs section and the onboarding card. Detects bedrock via resolveDisplayAlias().routing === "bedrock", shows a dedicated message ("store AWS_BEARER_TOKEN_BEDROCK as a secret, then put AWS_REGION + BEDROCK_MODEL_ID directly in pullfrog.yml") + link to the setup guide. Replaces the generic "X, Y, or Z is required" prompt that misrepresented the three values as three separate secrets to add (and used the wrong "or" connector for what's actually an AND). - OnboardingCard re-uses the same callout with the gradient-card variant. Docs: - Drop the obsolete "Enable model access" step. AWS retired the manual enrollment page; foundation models auto-enable on first invocation. Anthropic models still need a one-time use-case form for first-time users — surfaced under the AccessDenied troubleshooting entry. - Drop the "Testing a different model in one run" PULLFROG_MODEL note. It introduced the secrets-vs-vars distinction we want to keep out of the bedrock setup story. - Step 3 already recommends hardcoding region + model id in pullfrog.yml. Workflow template: - The default pullfrog.yml customers receive (utils/github/pullfrog.yml.ts) now references AWS_BEARER_TOKEN_BEDROCK from secrets but inlines AWS_REGION and BEDROCK_MODEL_ID as plain values. Matches the docs. * fix(bedrock): three review-caught edges in routing + UI copy Addresses three real issues from PR #720 review: 1. agent.ts: PULLFROG_MODEL=bedrock/byok no longer leaks the literal sentinel "bedrock" downstream. resolveCliModel returns the alias's resolve field verbatim, which for routing entries IS the sentinel. Refactored both the env-override and slug-lookup paths through a shared resolveSlug() that recognizes routing aliases and defers to their backing env var (BEDROCK_MODEL_ID). 2. models.ts: isBedrockAnthropicId() now anchors on a discrete dot/slash/colon-segment match (case-insensitive) instead of a substring contains. The substring check was fragile in both directions for inference-profile ARNs (BEDROCK_MODEL_ID accepts ARNs per AWS docs) — a non-Anthropic profile whose user-chosen name contained "anthropic" would mis-route to claude-code, and an Anthropic profile whose name omitted it would miss CLAUDE_CODE_USE_BEDROCK=1. 3. AgentSettings.tsx: BedrockSetupCallout's configured-state copy showed "AWS_BEARER_TOKEN_BEDROCK configured" even when the user satisfied the gate via AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, gaslighting access-key users about a secret they never set. Detect which auth method is actually present and name the right secret(s) in the success message. Regression tests in models.test.ts (5 new isBedrockAnthropicId cases including positive and negative ARN forms) and agent.test.ts (2 new PULLFROG_MODEL=bedrock/byok cases). 171/171 action tests pass. * yml template: add commented AWS access-key alternative for Bedrock auth Mirrors the IAM access-key path verified end-to-end on PR #720 e2e run 25830764987. Bearer token stays as the primary nudge; the access-key pair is the fallback for users who can't mint Bedrock API keys. * yml template: drop redundant 'or, alternatively' annotation * ui+docs(bedrock): rewrite callout copy + refresh screenshot Reframes the BedrockSetupCallout away from generic BYOK language to a Bedrock-specific message: leads with "Amazon Bedrock is configured entirely via environment variables", lists all four (auth, region, model id), and ends with the requested CTA sentence ("click below to learn more about Bedrock support in Pullfrog"). Promotes the "Bedrock setup guide" docs link from an inline anchor to a prominent button (always visible, regardless of auth state). The "Add AWS_BEARER_TOKEN_BEDROCK" affordance is now a secondary chip shown only when no auth secret is configured. Refreshes docs/images/model-selector-bedrock.png to capture the new callout — the prior screenshot still showed the old generic "BYOK / X, Y, or Z required" wording. |
||
|
Colin McDonnell
|
868576a474 |
audit: format byok auth errors actionably + tighten audit prompt
- `action/utils/apiKeys.ts`: rewrite the missing-key body as Markdown with linked CTAs (repo secrets / model settings / docs). add `isApiKeyAuthError` + `formatApiKeyErrorSummary` covering both shapes: missing key (#679) and revoked/invalid 401 key (#702). - `action/main.ts`: reclassify in the result-failure branch and the catch block so the PR progress comment surfaces the actionable CTA instead of the raw `Invalid API key · Fix external API key` / numbered-list dump. - `scripts/analyze-logs.ts`: split `failure:user-misconfig` into `:no-key` and `:invalid-key` so both buckets are visible separately and the audit can ignore them as user-correctable. - `.github/workflows/run-audit.yml`: add three explicit prompt rules — cross-customer signal required (≥3 distinct accounts; single-customer concentration is not enough), recovered failures are not actionable, user misconfig is out of scope. closes the loop on #679 / #702 being filed in the first place. |
||
|
David Blass
|
5518890b18 |
learnings: TOC + section taxonomy + 100k cap, hygiene rules, tool-quirk descriptions (#717)
* audit learnings: reshape reflection prompt + bake tool quirks into descriptions (#619) Cross-repo audit of the 48 repos with non-null learnings turned up two recurring failure modes: 1. ~25-30% of bullets across the most-active repos are pullfrog-tool quirks ("shell timeout is in milliseconds", "git args must be a JSON array", "create_pull_request_review drops out-of-hunk comments", "push_branch may report timeout when push succeeded", "checkout_pr shallow.lock retries", "commit_id needs full 40-char SHA"). These are universal across repos and should live in tool descriptions, not be rediscovered and stored 48 times. Tool descriptions now surface them. 2. Bullets are routinely 200-1000 chars (paragraph-length), and 12 of 48 repos are at the 10k cap. The reflection prompt now: caps bullets at ~240 chars (one specific fact), bans PR/review/commit/date-anchored facts that decay within weeks, bans tool-quirk learnings, and tells the agent that cap pressure means compress+prune existing bullets, not skip new findings. Co-authored-by: Cursor <cursoragent@cursor.com> * learnings: add server-generated TOC, fixed section taxonomy, raise cap to 100k (#707) Cap goes 10k → 100k. Reads stay bounded because the seeded file now opens with a server-generated table of contents listing every `## ` section's line range — agents read the TOC, then `read_file offset/limit` just the sections relevant to the current task instead of slurping the whole file. ## Section taxonomy (fixed) `## Build & test`, `## CI`, `## Conventions`, `## Architecture`, `## Gotchas`. Free-form `### ` sub-headings inside a section are fine. Pre-taxonomy free-text rows get wrapped in a `## Legacy` carve-out on first seed so they remain visible while the agent gradually re-curates them during reflection turns. ## Storage shape unchanged `Repo.learnings` still holds raw markdown (no schema migration). The TOC is a pure read-side affordance: prepended at seed time, stripped from the agent-edited file before persist. Markers `<!-- pullfrog-learnings-toc:* -->` delimit the strip region. Agent edits inside the markers are discarded. ## Round-trip semantics `seedLearningsFile` now returns `{ path, canonicalSeed }` where `canonicalSeed` is the post-TOC body — same shape `readLearningsFile` returns at end-of-run, so `persistLearnings` byte-compares them directly to skip the no-op PATCH. Empty-repo first runs end up with the section scaffold both as seed and as read-back, so untouched runs still short-circuit cleanly. ## Reflection prompt Adds explicit section-placement guidance (place each new bullet under the most relevant `## `; do NOT add new top-level headings; do NOT edit anything between the TOC markers). Carries forward the bullet hygiene from the previous commit: ≤240 chars per bullet, no pullfrog-tool quirks (those belong in tool descriptions), no PR/review/commit/date references. The "near cap" framing is replaced with "compress and prune within a section when it grows noisy" since the cap pressure that drove cramming is gone. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal round 1: line-anchored taxonomy detect, partial-merge, line-boundary truncation, scaffold-empty UI Multi-lens review of the TOC + taxonomy diff surfaced a cluster of correctness and operational bugs. Fixes: - `hasAnyTaxonomyHeading` used `String.includes("## X")` which false-positives on `### X` (the `## ` substring sits inside `### `), prose containing `## CI`, fenced code documenting markdown, etc. Replaced with a line-anchored predicate that reuses `parseHeadings` so detection and TOC construction stay consistent. - The "any heading present → pass through verbatim" rule meant a body with one taxonomy heading would seed without the other four. Worse, requiring all five would flip a body back into Legacy when the agent legitimately pruned a section to empty. New `partial` kind: keep existing content in place, append missing sections in canonical order so the agent always has the full scaffold without losing pruning intent. - `stripLearningsToc` collapsed `\n{3,}` globally; `canonicalSeed` doesn't, so an untouched body with intentional triple-newline spacing would compare unequal and burn a spurious LearningsRevision row each run. Drop the global collapse — only the leading newlines that the strip itself introduces are normalized. - 100k truncation via `slice(0, 100_000)` could cut mid-line, breaking `parseHeadings` (whole-line `^## `) on the next seed and flipping a cut body back into Legacy. New `truncateAtLineBoundary` cuts at the last newline before the cap. - `LearningsSection.tsx` rendered a scaffold-only body as "has learnings" instead of the empty placeholder. Added a `hasOnlyEmptyScaffold` guard so the console behaves the same as pre-PR for the empty case. - Seed log line distinguishes `kind=structured/partial/legacy-wrapped/ empty` instead of `existing=yes/no`, so operators can spot legacy migration activity in logs. - New tests cover: substring false-positive (`### Build & test`, in-prose mentions), partial-taxonomy merge (no Legacy wrap), full-taxonomy structured pass-through, last-newline truncation, triple-newline preservation. Deferred (documented in PR body): deploy-ordering footgun (action before API), rollback for rows >10k, Gemini sanitizer dropping `description` on `anyOf` branches, reflection-on-failed-runs. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal r2: hard-truncate fallback when line boundary discards >4k Round-2 review caught a regression in `truncateAtLineBoundary`: when the only newline within the first 100k chars sits near the start (e.g. one heading + 100k+ char single line — pathological pasted log dumps), the line-boundary cut discards almost all of the body. losing one partial line is preferable to losing kilobytes; threshold the fallback at 4k. Co-authored-by: Cursor <cursoragent@cursor.com> * move TOC out of file: prompt-side rendering, server-parsed headings drops the in-file TOC + fixed taxonomy in favor of: - file on disk = verbatim Repo.learnings (no markers, no scaffold) - server parses headings (mdast-util-from-markdown) at run-context time and returns them as RepoSettings.learningsHeadings - action renders heading TOC into the LEARNINGS prompt section as parenthesized line ranges like `Build & test (L1-L42)` with hierarchy via 2-space indent off the shallowest depth - reflection prompt teaches agent-curated structure with a soft 300-line per-section cap and explicit guidance to restructure flat legacy lists cuts 8 helpers (ensureSections, stripLearningsToc, assembleFile, buildTocBlock, parseHeadings, buildSectionScaffold, hasAnyTaxonomyHeading, LEARNINGS_SECTIONS) and the canonicalSeed round-trip dance. action seedLearningsFile is now { path } only; main.ts byte-compares the trimmed read-back against (current ?? "").trim() to gate the persist PATCH. truncateAtLineBoundary kept for safety. new tests: - test/learningsToc.test.ts (11 parser cases incl. fenced-code, blockquote, arbitrary h1-h6 nesting, startLine-points-at-heading invariant) - action/utils/learningsTocRender.test.ts (7 renderer cases) --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |
||
|
Colin McDonnell
|
ae976e7159 |
parallel tool execution: enable opencode batch + nudge agents to parallelize (#719)
opencode: opt into `experimental.batch_tool` (anomalyco/opencode#2983) so the `batch` tool registers and the model can bundle 1-25 independent calls into one round trip. edit calls are excluded upstream. instructions.ts: add a "Parallel tool execution" section to the SYSTEM Workflow block, agent-specialized via ctx.agentId. uses Anthropic's canonical wording ("invoke all relevant tools simultaneously...") so Claude reliably emits multiple tool_use blocks per message; tells OpenCode about the new `batch` affordance. verified end-to-end against haiku-class models (sonnet for claude, default for opencode) with a "read 3 files and report first lines" fixture. results: - opencode used `batch` with 3 nested reads AND emitted 3 native parallel read calls in the same assistant turn - claude went from 3 serial turns (1 read each) to 1 message with 3 parallel Read tool_use blocks |
||
|
Colin McDonnell
|
5aabd1e4a9 |
fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680) (#715)
* fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680) unbounded `stdoutBuffer += chunk` / `stderrBuffer += chunk` in `action/utils/subprocess.ts` previously crashed the wrapper with `RangeError: Invalid string length` once V8's ~1 GiB kMaxLength was breached on long-lived agent runs. multi-lens opencode Reviews on large monorepos (e.g. tambo-ai/buildy) hit this consistently — 23 runs in the last 24h, 100% of Review-mode hard failures on that repo. - add `retain: "tail" | "none"` to SpawnOptions, defaulting to "tail" with an 8 MiB cap. tail-mode prepends a `... [N MiB truncated] ...` sentinel so downstream consumers can detect truncation. - export `TailBuffer` helper for callers that need the same bounded accumulator semantics at their own layer. - wrap stream `data` listeners in try/catch as defense in depth — any synchronous throw inside a stream handler is otherwise fatal. - opencode + claude pass `retain: "none"` (they drain via onStdout / onStderr) and switch their own `output` accumulators to TailBuffer. their error paths read the agent-layer bounded mirrors instead of the now-empty `result.stdout` / `result.stderr`. - add `failure:string-length-overflow` heuristic to scripts/analyze-logs.ts so post-fix recurrences are visible at a glance instead of bucketing into `failure:unknown`. - regression tests cover >1 MiB stderr without crash, retain:"none" contract, and TailBuffer truncation semantics. * fix: avoid TS parameter property syntax in TailBuffer for strip-only node loader * address review: clarify try/catch scope + lock retain default to "tail" - the original comment claimed the try/catch caught "any synchronous throw" in the data listener, but `options.onStdout?.(chunk)` returns a Promise in the agent callers (claude.ts:569, opencode.ts:933) — a throw inside an async user callback surfaces as an unhandled Promise rejection, not a synchronous exception. reword to describe the actual protection: defense-in-depth for synchronous throws in the listener body, which is exactly the shape of the original RangeError on `+= chunk`. - add a test that locks `retain` default to "tail" by spawning without the option and asserting `result.stderr` is non-empty. a future refactor that flipped the default to "none" would silently break gitAuth, package installs, and lifecycle hooks that read result.stderr for failure messages, and the rest of the suite wouldn't catch it. |
||
|
Colin McDonnell
|
60cc8772a6 |
fix(log-audit): kill 404 noise from /api/github/installation-token at source (#693) (#708)
* fix(log-audit): kill 404 noise from `/api/github/installation-token` at source (#693) Closes #693. Issue diagnosed a surface symptom (`log.error` on expected 404s) but missed the actual root causes. Investigation revealed two distinct populations producing identical 3-call 404 bursts: 1. **Fork-CI on `pullfrog/pullfrog`**: `test-token.yml` and `trigger-sync.yml` ship with `on: push: main`, so every fork inherits them and 404s our token endpoint on first push. Self-inflicted noise that scales with fork count. 2. **Real users hitting the full action without installing the App**: `/api/repo/.../run-context` uses the caller's `GITHUB_TOKEN` to read the repo from GitHub and then unconditionally lazy-provisions Account+Repo rows via `fetchOrCreateRepo`, even when the App isn't installed. Generates phantom DB rows and false `new account created` team@ alerts. (Confirmed via Prisma: `ezcorp-org` has an Account row with `installerLogin: null`, never installed our App.) Both populations then trip the client retry loop in `acquireTokenViaOIDC`, which matched `"Token exchange failed"` and retried 3× on terminal 4xx — tripling log volume and wasting CI time. ## Changes - `action/.github/workflows/{test-token,trigger-sync}.yml`: gate jobs with `if: github.repository == 'pullfrog/pullfrog'`. Forks inherit the files but the jobs no-op. - `app/api/repo/[owner]/[repo]/run-context/route.ts`: call `getRepoInstallation` first; return 404 with install URL if the App isn't installed, before any DB writes or GitHub repo fetch. - `action/utils/github.ts`: introduce `TokenExchangeError` for non-2xx server responses; `acquireNewToken` no longer retries it. Retry now fires only on genuine network/timeout failures. 404 surfaces a user-actionable error pointing at the install URL. - `app/api/github/installation-token/route.ts`: move `log.error` inside the 500 branch only. 404 branch is silent (expected user-state) and returns the same install URL message for consistency. ## Effect - Better Stack `level=error` lines from this path: 6/day → 0. - Failed user-trial CI time: 3 wasted token requests → 1. - User-facing error: opaque `Token exchange failed: 404` → actionable install URL. - No more phantom Account rows from never-installed callers. Skipped per design discussion: phantom-account cleanup (conservative — stop the bleed, leave history), `AGENTS.md` rule (overgeneralized). * review: address oracle leak + per-env install URL + retryable 5xx Addresses pullfrog[bot] (IMPORTANT) and Copilot review findings on #708: - **Install-status oracle in `run-context`** [pullfrog, Copilot]: `getRepoInstallation` runs with our App's JWT, *before* the caller's bearer token is validated against the repo. Pre-PR the route was uniformly bad-token-shaped; the new install-specific 404 turned it into an unauthenticated oracle distinguishing "Pullfrog installed here" from "not installed". Collapsed the 404 message to match the outer catch's ambiguous "repository not found or token lacks access". Legit runners still get the actionable install URL from `/api/github/installation-token`, which IS gated by OIDC. - **Hardcoded `github.com/apps/pullfrog`** [Copilot]: server-side `installation-token` now uses `GITHUB_APP_INSTALL_URL` from `app/globals.ts`, so dev/staging deployments with a different `GITHUB_APP_SLUG` direct users to the correct app. Action-side echoes the server's `error` body when present (single source of truth) and falls back to a generic message only if the body isn't JSON. - **Transient 5xx/429 made terminal** [Copilot]: `shouldRetry` now returns `true` for `TokenExchangeError` with `status >= 500` or `status === 429`. 4xx remains terminal (the actual #693 fix). Real outages no longer fail the workflow immediately. - **Stale comment** [pullfrog, Copilot]: reworded the comment at `installation-token/route.ts:141` to reflect the new retry policy ("the action surfaces this once (no retry)" instead of "the action retries on this"). * review: restore caller-token-first auth in run-context Pre-PR, `getEnrichedRepo({owner, repo, token})` used the caller's token as the auth boundary — `getRepo({token})` succeeding was the proof-of-access check. My initial install-gate inverted the order and ran the App-credentialed `getRepoInstallation` first, which is how it became: - an install-status oracle (pullfrog bot, addressed previously by matching the outer-catch wording), and - an outbound amplifier against our App JWT for arbitrary `owner/repo` (pullfrog bot, this commit). Reordered so `getRepo({token})` runs first. Garbage / unauthorized bearers get rejected by github (mapped to 403 by the outer catch) before any App-credentialed call fires. `getRepo` is cached 5min, so `getEnrichedRepo` below remains a free re-hit. |
||
|
Colin McDonnell
|
4260984257 |
attribute claude subagent log lines + per-session thinking timer; tighten lens calibration (#700)
* attribute claude subagent log lines + per-session thinking timer; tighten lens calibration three orthogonal fixes diagnosed from the 10m PR-699 review run: 1. wire SessionLabeler into the Claude Code harness. claude-agent-sdk stamps every Assistant/User/System message with session_id and a non-null parent_tool_use_id when emitted from a subagent context, so the same FIFO labeler the OpenCode harness uses works here too. parallel reviewfrog dispatches now log with [lens:correctness] / [lens:operational-readiness] / etc. prefixes instead of being indistinguishable from the orchestrator. matches both "Task" and "Agent" tool names per the v2.1.63 rename. 2. one ThinkingTimer per session. the global timer treated cross-session interleaving (parent thinks → child tool_call, child returns → parent dispatches next) as parent thinking time, so individual "thought for Xs" numbers were untrustworthy. each session now owns its own timer and prefixes its own log line. 3. tighten the Review/IncrementalReview lens-add discipline. PR-699 triggered 4 lenses on a typical refactor (no auth/billing/schema) when the prompt's own calibration says 2-3 is typical; the research-validated lens went deep on Resend idempotency window + prisma updateMany lost-updates without either being load-bearing. adds an explicit "name the failure mode this lens would catch that the diff plausibly introduces" bar, and tightens research-validated specifically: only when correctness depends on the third-party contract, not when the API is merely used. side benefits from #1: subagents' TodoWrite events no longer clobber the orchestrator's progress comment; subagent text no longer overwrites finalOutput; system-event handler safely routes through eventLabel even though SDK only emits system:init for the top-level query today. * fix node strip-only mode: declare formatLine as field, not parameter property * key claude subagent labels by parent_tool_use_id, not session_id claude-agent-sdk runs subagents inside the orchestrator's session — they share session_id — and stamps subagent messages with parent_tool_use_id pointing at the Agent tool_use that spawned them. e2e on PR-700 with preview-700-claude-labeling#1 confirmed the original session_id-keyed wiring never differentiated subagent activity (only the dispatch line got [lens:correctness] in the log; the subagent's reads, writes, and todos all rendered as orchestrator). extend SessionLabeler so labelFor accepts an optional parent_tool_use_id and short-circuits to a direct map keyed by Agent tool_use id when set. recordTaskDispatch optionally takes the Agent tool_use id (block.id at dispatch time) and binds it. orchestrator events keep flowing through the sessionID/FIFO path unchanged so opencode wiring is untouched. * drop weak timer test that asserted only field isolation per pullfrog review on PR-700: the 'two timers do not bleed timestamps' test only verified that two ThinkingTimer instances have separate private fields, which has always been true. doesn't earn its keep — the per-session behavior is exercised by integration through claude.ts + opencode.ts. |
||
|
Colin McDonnell
|
d5f881e9fc |
action: trim sensitive env values before GitHub Actions log masking (#698)
* action: trim sensitive env values before GitHub Actions log masking GitHub Actions' log masking is line-based: a secret value containing a newline only registers the first line as a mask, leaving the remainder exposed verbatim in logs. A trailing newline copied from a terminal into a GitHub Actions secret (e.g. ANTHROPIC_API_KEY) was enough to leak "a large part of the key" in run logs (pullfrog/pullfrog#41). normalizeEnv now trims leading/trailing whitespace from any value whose key matches the sensitive name pattern, masks the cleaned value, and warns when whitespace was stripped so the user notices the source. sanitizeSecret is reused for dbSecrets injection in main.ts. The three secret-store PUT/POST routes also trim values defensively, matching the existing name.trim() pattern. Real multi-line secrets are not used in practice — even GITHUB_PRIVATE_KEY PEMs are stored single-line with escaped \n and unescaped at the point of use — so a straight trim() is safe. * action: address review — use core.setSecret for masking, don't zero whitespace-only Pullfrog's review of #698 caught two real issues in the original fix: 1. `console.log(\`::add-mask::\${trimmed}\`)` doesn't escape \r/\n. If a value survives trim with an embedded newline (PEMs, kubeconfigs, JSON), the runner only registers the first line as a mask and the rest leaks. `core.setSecret(trimmed)` routes through @actions/core which percent-encodes \r/\n so the runner V2 parser decodes back to the full value and registers every non-empty line as a separate mask. Removes the load-bearing "no embedded newlines" invariant from the fix. 2. Whitespace-only sensitive values silently became "". Downstream truthy checks would flip from "set" to "missing" with no log. Now sanitizeSecret returns null in that case and callers skip the process.env write, surfacing a clear missing-key error instead. Tests rewritten to assert process.env state directly — no stdout spies. Masking correctness is delegated to @actions/core (trusted dependency). |
||
|
Colin McDonnell
|
d5d8a0d7ac |
fix(#691): drop opencode/gpt-5-nano + opencode/mimo-v2-pro-free (not actually keyless on Zen) (#695)
* remove opencode/gpt-5-nano and opencode/mimo-v2-pro-free from catalog #7 delete aliases. both were listed as `isFree: true, envVars: []` but neither is keyless on opencode zen, producing a hard-fail `UnknownError: Model not found: opencode/<id>` on every run without an opencode_api_key. fixes pullfrog/app#691 (5 runs across 3 repos, 100% failure rate in the last 24h). root cause: opencode's provider gate (`packages/opencode/src/provider/provider.ts` `opencode:` loader) keeps a zen model only when models.dev reports `cost.input === 0` for it, then signs requests with `apiKey: "public"`. paid zen models get deleted from the autoloaded set and opencode surfaces the deletion as "model not found". - `opencode/gpt-5-nano`: models.dev reports `cost: {input: 0.05, output: 0.4, cache_read: 0.005}`. paid → requires `OPENCODE_API_KEY`. - `opencode/mimo-v2-pro-free`: free on models.dev but not in `https://opencode.ai/zen/v1/models` — zen never served it, so even the public-key path fails. remaining free aliases (`opencode/big-pickle`, `opencode/minimax-m2.5-free`) both pass both checks (cost.input === 0 in models.dev AND present in zen's served list) and continue to work without a key — verified against the opencode source. callers swept: `action/utils/apiKeys.test.ts`, `action/models.test.ts`, `action/test/list-aliases.ts`, `action/test/model-smoke.ts`, `components/ModelSelector.tsx` (`modelIdToUpstream`), `wiki/model-resolution.md`, `wiki/models-catalog.md`. wrote up the free-zen verification rule in models-catalog so the next maintainer can sanity-check both conditions before adding any `isFree` alias. users with a stored `opencode/gpt-5-nano` or `opencode/mimo-v2-pro-free` will now fall through `resolveCliModel → undefined` into the auto-select path — a strict improvement over today's hard fail. no DB migration needed; the slugs are simply unknown and treated like any other unrecognized stored value. * rework: keep mimo deprecated, demote gpt-5-nano to paid, add free-zen invariants revised approach after the first commit over-corrected. mimo was never broken at runtime — `fallback: "opencode/big-pickle"` already routes stored values through to a real free model before any zen call. the literal `opencode/mimo-v2-pro-free` being absent from zen's served list is irrelevant because `resolveCliModel` walks the chain first. restoring it as-is. the actual bug was `opencode/gpt-5-nano`: marked `isFree: true, envVars: []` but `models.dev` reports `cost: {input: 0.05, output: 0.4}` on the opencode provider, so opencode's keyless gate (`packages/opencode/src/provider/provider.ts` `opencode:`) deletes it when `OPENCODE_API_KEY` is missing and the run hard-fails with `UnknownError: Model not found: opencode/gpt-5-nano`. demoting it to a regular paid zen alias (drop `isFree`/`envVars: []`, add `openRouterResolve: "openrouter/openai/gpt-5-nano"` — verified to exist on openrouter at the same price). users without `OPENCODE_API_KEY` now get our explicit "no API key found" error pointing at the secrets page instead of opencode's cryptic upstream error. confirmed via `https://opencode.ai/zen/v1/models` that zen serves no free GPT variants, so there's no cheaper-than-`gpt-mini` free option to suggest in its place. CI gap analysis (why this slipped through): - `models-catalog.main.test.ts` only checked existence + `status !== "deprecated"` on models.dev. paid-model-marked-free regressions and zen-served-list drift both passed. - `models-live` (`model-smoke.ts`) runs with `OPENCODE_API_KEY` in env, so the keyless deletion gate never fires. `gpt-5-nano` returned "OK" in CI even though end users hit a hard fail. - `model-smoke.ts` walks the fallback chain, so mimo would have been smoked as big-pickle anyway — the dead resolve target was never exercised directly. (this is the right design; the gap is at the catalog layer, not the smoke layer.) new tests: - PR-blocking, static (`action/test/models.test.ts`, `isFree invariants`): every `isFree` alias must live under `opencode`, have `envVars: []`, omit `openRouterResolve`, AND have a fallback chain whose terminal alias is also `isFree` (catches "deprecate a free alias to a paid target" — the worst silent-charge regression). - main-only, network (`action/test/models-catalog.main.test.ts`, `opencode Zen served list`): every alias whose terminal-fallback resolve is `opencode/*` must appear in `https://opencode.ai/zen/v1/models`. catches zen dropping a model from its served list. - main-only, network (same file, `isFree models.dev cost`): every `isFree` alias's terminal-fallback resolve must have `cost.input === 0` in the `opencode` provider block on `models.dev`. would have caught `gpt-5-nano` at the next models-bump run. both network tests dedupe on terminal resolve, so deprecated aliases sharing a target aren't double-counted. `pnpm vitest run`: 113 static tests pass. `pnpm test:catalog`: 142 network tests pass against the live `models.dev`, `openrouter.ai`, and `opencode.ai/zen/v1/models` endpoints. wiki/models-catalog.md: rewrote the new "Free-Zen aliases need Zen-side verification" section to (a) describe the two conditions, (b) note that a fallback to an isFree alias is the legitimate escape hatch (mimo's pattern), and (c) point at the three tests by name so the next maintainer can find the enforcement surface. wiki/model-resolution.md points at the new section. * make gpt-5-nano a deprecated free alias falling back to big-pickle revising the previous "demote to paid" approach. the user-facing ergonomics are cleaner: anyone who picked gpt-5-nano under the "Free" badge gets transparent-upgraded to a real free model (big-pickle) instead of suddenly being asked to set OPENCODE_API_KEY. matches the existing mimo pattern exactly. the dropdown already filters `!a.fallback`, so the slug disappears from the picker on its own and the trigger renders it as "Big Pickle" via `resolveDisplayAlias`. no other catalog or test surface changes — the isFree invariants and the main-only zen/cost checks still pass (gpt-5-nano's terminal is now big-pickle, which is both isFree and zero-cost on models.dev, deduping with big-pickle's own row in both network tests). * revise: keep gpt-5-nano as paid alias, backfill affected DB rows instead dropping the deprecated-alias approach. `opencode/gpt-5-nano` is a legitimate cheap paid model people may want with BYOK (`OPENCODE_API_KEY`) — giving it `fallback: "opencode/big-pickle"` would foreclose that for everyone going forward. correct fix is two parts: (a) reclassify in the catalog as a regular paid OpenCode alias: - drop `isFree: true` and `envVars: []` so the local validator demands `OPENCODE_API_KEY` - add `openRouterResolve: "openrouter/openai/gpt-5-nano"` to satisfy the completeness test and route BYOK-via-OpenRouter users - no `fallback` — slug stays visible in the picker as a paid option (b) one-shot DB backfill of provably-affected repos (`scripts/backfill-gpt5-nano-affected.ts`). scope: - `Repo.model = "opencode/gpt-5-nano"` - AND at least one `WorkflowRun` with `inputTokens IS NULL` (evidence of an attempted run that didn't get past the model-init gate) skipped intentionally: - repos whose runs have `inputTokens > 0` — they have a key, gpt-5- nano works for them - repos with zero WorkflowRun rows — never dispatched; touching them would be presumptuous - `LearningsRevision.model` — audit trail of which model authored a revision, rewriting it would falsify history ran against .env.prod: 2 repos stored the slug; 1 was provably affected (sodown4thecause/seobot, 5/5 zero-token runs — matches #691's 3 failed runs from this repo plus 2 outside the 24h audit window). 1 was an internal test account that never dispatched (left as-is). applied: 1 row updated. confirmed idempotent on re-run. the other two repos in #691 (Nantiee/ALTA-breast-pump-tool, keksiqc/ansible-setup-linux) don't store the slug in `Repo.model`; their failed dispatches passed the model inline in the `workflow_dispatch` `prompt` payload, so the catalog fix alone (no longer offering it as free) is what helps them. tests: - models.test.ts: `getModelEnvVars("opencode/gpt-5-nano")` now returns `["OPENCODE_API_KEY"]`, moved into the keyed-model group - apiKeys.test.ts: added "throws without OPENCODE_API_KEY" case - isFree invariants from the previous commit still pass — gpt-5-nano no longer triggers them since it's no longer isFree - main-only catalog tests still pass (gpt-5-nano served by Zen, just paid; no isFree cost check applies) * docs: drop stale GPT Nano + MiMo V2 Pro from free-tier lists addressing pullfrog auto-review feedback on #695. three mintlify pages still advertised both as keyless after the catalog pivot, which now makes the docs affirmatively wrong rather than merely stale: - gpt nano is paid in the catalog (no `isFree`, inherits `OPENCODE_API_KEY`); a user following the docs would hit the same "missing API key" failure that's described 4 lines below in `docs/keys.mdx`. - mimo v2 pro is hidden from the picker (`fallback` triggers `ModelSelector`'s `!a.fallback` filter); the alias only exists for legacy stored-value resolution. a user reading the docs cannot actually pick it. surviving picker-visible free set: Big Pickle and MiniMax M2.5. - `docs/keys.mdx`: drop both bullets from the "Free models" list - `docs/billing.mdx`: drop both bullets from the "Free models" list - `docs/getting-started.mdx`: collapse the inline mention from a 4-model list to "Big Pickle and MiniMax M2.5" * address third review: picker grouping + backfill classifier honesty i had not pulled the third pullfrog review (`02:17:28Z`) when i declared reviews triaged after the docs sweep — the fourth review flagged that three findings remained pending. addressing them now. 1. picker grouping for now-selectable paid gpt-5-nano. when i removed `"gpt-5-nano": "OpenAI"` from `modelIdToUpstream` in the previous pivot-to-paid commit, i mistook it for dead code. it's not — the map IS consulted for paid opencode aliases via `groupByUpstream → getUpstreamLabel` inside the OpenCode submenu's `renderSubContent`. without the entry, `gpt-5-nano` falls back to `getProviderDisplayName("opencode")` = "OpenCode" and gets dropped into its own sub-header instead of joining opencode/gpt, opencode/gpt-pro, opencode/gpt-mini under the "OpenAI" upstream group. re-added with an explanatory comment so the next refactor doesn't make the same mistake. 2. JSDoc / code mismatch in `scripts/backfill-gpt5-nano-affected.ts`. the JSDoc said "at least one `WorkflowRun` with `inputTokens IS NULL`" but the code is `no WorkflowRun has inputTokens > 0` — a strictly broader filter (catches `null` AND `0`). rewrote the scope block to describe what the code actually does, with the operative classifier spelled out: "a billable run with `inputTokens > 0` is proof the agent successfully reached and called the model". 3. classifier breadth (raised in the same review). honest answer: the "no positive-token run" filter IS a heuristic — a repo whose only dispatches happened to fail or cancel for unrelated reasons would get false-positive-classified A. for THIS one-shot population (2 repos, 1 with 5/5 zero-token runs — strong systematic-failure signal) the heuristic was good enough and the dry-run inspection confirmed before APPLY. for any larger reuse of this pattern, you need to cross-reference the runtime error string (`UnknownError: Model not found: opencode/gpt-5-nano`) from GitHub Actions logs or Better Stack — that error doesn't live on `WorkflowRun` rows. added a "Classifier limitations" section to the JSDoc making this explicit. nothing about the actual applied backfill changes — the prod write (1 repo: sodown4thecause/seobot → opencode/big-pickle) is unchanged and re-running the script remains idempotent. |
||
|
Colin McDonnell
|
43bb14bf87 |
action: strip Content-Type on body-less apiFetch requests (#692) (#694)
* action: strip Content-Type on body-less apiFetch requests (#692) Vercel's Next.js lambda adapter (Next 16.1.x) attempts to decode a request body when Content-Type is set and throws `SyntaxError: Unexpected end of data` before delegating to the route handler, returning a 500. Hit /run-context exclusively because it was the only body-less GET that sent `Content-Type: application/json`. - Drop `Content-Type: application/json` from the GET in `action/utils/runContext.ts` (meaningless on a body-less request). - Defensively strip any `content-type` header in `action/utils/apiFetch.ts` when no body is present so future callers can't reintroduce this. * apiFetch: soften comment — empirical observation, RFC 9110 §8.3 framing |
||
|
David Blass
|
8c6cd2bda2 |
cancel + restart workflow run when @pullfrog mention is edited (#612)
* cancel + restart workflow run when @pullfrog mention is edited - add `WorkflowRun.triggeringCommentId` (BigInt?, indexed) so the webhook handler can find the run that was fired by a given comment - thread `triggeringCommentId` through `reserveRun` / `triggerWorkflow` - factor `dispatchMentionRun` out of `issue_comment_created` so the same shape is reused on edit - replace the `issue_comment_edited` stub: re-evaluates the trigger gate, cancels prior runs (`octokit.rest.actions.cancelWorkflowRun` + DB status='cancelled'), then re-dispatches with a `previousRunsNote` appended to `eventInstructions` so the agent acknowledges the prior run/PR/artifacts in its summary - if the edit removes `@pullfrog`, cancel only (no restart) Co-authored-by: Cursor <cursoragent@cursor.com> * thread previousRunsNote via dedicated payload field user prompt has precedence over eventInstructions, so stuffing the prior-runs note into eventInstructions made it vanish whenever the trigger comment contained an @pullfrog mention (which is always for the edit path). pass it as its own payload field and render it alongside the user's task so the agent actually sees it. * delete cancelled run's progress comment on edit-restart so the issue thread doesn't accumulate "This run was cancelled" stubs on every edit. only deletes for runs we actively cancel; runs that were already terminal (e.g. completed before the edit) keep their summary comment in the thread, and `previousRunsNote` links to it so the new agent can reference prior work. post-cleanup is race-safe: the action's `validateStuckProgressComment` swallows the 404 from the deleted comment and exits cleanly, so the old run's post step cannot clobber the new run's leaping comment. Co-authored-by: Cursor <cursoragent@cursor.com> * also cancel + delete progress comment when triggering comment is deleted mirrors the edit-removes-@pullfrog path: when an @pullfrog comment that fired a run is hard-deleted, look up any prior runs by triggeringCommentId, GH-cancel running ones, and delete their leaping progress comments. skips trigger-gate re-eval (we're tearing down a run, not firing one) and performs no restart. reuses the existing cancelRunsForTriggeringComment helper; the returned previousRunsNote is discarded since no dispatch follows. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: move cancellation before trigger gate in issue_comment_edited cancelRunsForTriggeringComment now runs before the triggerEnabled check, so edits that remove @pullfrog still cancel in-flight runs even when the repo mention trigger is currently disabled (e.g. for non-collaborators). * anneal: scope cancel updates per-row + simplify edit gate - replace blanket updateMany on (triggeringCommentId, repoId) with per-row, status-guarded updates so a parallel handler's freshly-reserved run cannot be clobbered into cancelled by a racing edit delivery. - drop wasMention/isMention early-break in issue_comment_edited; always run cancelRunsForTriggeringComment (DB is the canonical "did this comment ever trigger a run" source). closes the missing-changes.body.from edge and lets us tear down a still-running prior run even if the admin disabled the mention trigger mid-flight. - buildPreviousRunsNote returns undefined (not "") when no link lines materialize. - doc cleanups + wiki/modes.md addendum noting issue_comment_edited / _deleted now drive cancel + restart. Co-authored-by: Cursor <cursoragent@cursor.com> * address review feedback on cancel/restart semantics - guard workflow_run.completed update against status='cancelled' so a successful-but-uncancellable GH Actions job can't resurrect a cancelled row (and re-bill it) via the completed webhook. - bucket only status='completed' runs into `preserved` in cancelRunsForTriggeringComment; cancelled/failed prior runs have stubs as their progress comment, not summaries worth referencing. - emit previousRunsNote for the runId-null cancel case so the restarted agent always knows when it's superseding a prior dispatch. - drop the agent-forbidden `gh pr list` hint and soften 'was cancelled' to 'was signalled to cancel' in the note body. - post a fallback comment when the edit-path dispatch fails (prior run already torn down and progress comment already deleted). - symmetrize the delete-handler's pullfrog guard with the edit handler (key off hook.comment.user, not hook.sender). - trim misleading comments on the per-row DB update guard. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com> |