bf0a8b27430bcf4f4fbef6f03b49a6da69311b7b
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colin McDonnellandpullfrog[bot]
|
36ac64a5b6 |
fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy (#844)
* fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy OSS-allowlisted repos with `CODEX_AUTH_JSON` uploaded via `pullfrog auth codex` were still being routed through the OSS OpenRouter subsidy because two paths ignored managed credentials: - `hasProviderKey()` only checked `provider.envVars`, so an `openai/*` model with only `CODEX_AUTH_JSON` present silently fell back to `opencode/big-pickle` via `selectFallbackModelIfNeeded` — the maintainer saw "opencode/big-pickle (resolved from openai/gpt)" on CI even though Codex was configured. - `run-context` set `proxyModel` for every OSS run unconditionally, which the action runtime threads through `payload.proxyModel` and uses to overwrite `OPENROUTER_API_KEY`. Even if `big-pickle` fallback hadn't fired, the runner would consume the $10 OSS subsidy key instead of the user's ChatGPT subscription. Fix: - Add `getModelAuthEnvVars()` covering both `envVars` and `managedCredentials` in `action/utils/apiKeys.ts`; route `hasProviderKey` + `validateAgentApiKey` through it. - `run-context` now skips `proxyModel` for OSS repos when the configured model's provider has matching auth in Pullfrog-stored account/repo secrets, so the runner authenticates directly with the user's Codex subscription (or any other user-provided provider auth). Triggered by mrlubos (`hey-api/openapi-ts`). Companion follow-up tracked for the opaque "(no error message)" classifier swallow that masked the OSS $10 cap exhaustion on PR #3872 runs 25815370844 + 25815443234. * fix(oss): force Kimi K2 for OSS proxy + hide picker in console UI OSS-funded runs were resolving `repo.model` through OpenRouter, so a single Opus / GPT-5.5 run could burn an entire `oss_subsidy` key against the per-key cap and crash mid-stream (e.g. `hey-api/openapi-ts` PR #3872 runs `25815370844` + `25815443234`, ~$9.20 each on a single key). Force `DEFAULT_PROXY_MODEL` (Kimi K2.6 — ~10-50× cheaper) for every OSS proxy mint, regardless of `repo.model`. Per-run spend stays bounded within the cap by structure, not by hope. `repo.model` stays in the DB unchanged — overriding at runtime means leaving the program restores the user's prior pick without a migration. UI: hide the model picker entirely on OSS repos in `AgentSettings`. The field is effectively inert until the repo leaves the program, so exposing it as if it were live was misleading. Replaced with a banner naming Kimi K2 and pointing to `pullfrog auth …` as the opt-out path — that lands the user on the existing #844 bug-2 branch (Pullfrog-stored auth suppresses the OSS proxy entirely; runner uses user credentials + their preferred model). ModelCostsInfo already has its own `isOss` branch for the cost copy, so that section is unchanged. * fix(oss): lowercase comment casing per AGENTS.md * fix(oss): revert banner copy to 'It's on us.' framing per review Maintainer felt 'Kimi K2' as the banner headline lost the warm 'we've got you covered' framing that the existing OSS cost banner uses. Restore 'It's on us.' as the headline, move the model name into the body where it explains the hardcoded choice and points to the opt-out (pullfrog auth codex / account secret). * docs(agents): screenshots must be of the live route, never synthetic Caught myself building a temp `/dev/oss-ui-preview` route with hardcoded JSX copy-pasted from the real component just to grab a screenshot — the result told us nothing about whether the actual integrated UI worked, and the user (rightly) called it out as a waste. Strengthen the rule: screenshots must come from the live route in the running app, driven by the actual component tree and real props. Note the GH OAuth interstitial gotcha so the next agent gets through Clerk → GitHub sign-in on the first try instead of bailing to a fake render. Also bans side-by-side comparison screenshots unless explicitly requested. * fix(oss): one 'It's on us.' banner, not two OSS Agent settings was showing the message twice — once in the Model section, once in the Model costs section right below it. Fold the cost coverage into the model banner ('at no cost to you' + the spend stat) and hide the Model costs subsection entirely for OSS. ModelCostsInfo no longer needs `isOss` / `ossSpendThisMonthUsd` props — call site is gated, so the OSS branch is dead. Removed it and the now-unused props. Non-OSS rendering is unchanged: full Model picker + Model costs subsection with Router / BYOK branches. * feat(action): corepack-aware package manager provisioning before setup customer setup scripts that did `npm i -g pnpm && pnpm install` were installing whatever pnpm "latest" happens to be on the day the run fires, not what the repo declares — and pnpm 11.3 silently writes a new `packageManagerDependencies` block into lockfiles, which the agent's "always push changes" rule then packages into a noisy PR (see #844). resolve the project's pnpm/yarn pin from `package.json` (honoring pnpm 11+ precedence: `devEngines.packageManager` over `packageManager`) and activate it via `corepack prepare ... --activate` BEFORE the setup hook runs. corepack is bundled with node, so this is a no-op on managed infra; failure (no corepack, no network, range-only version) degrades to a warning and the existing PATH binary still runs. also replaces the legacy `npm install -g <pm>@<v>` path in prep with the same helper so behavior is consistent end-to-end. bun/deno still use the legacy installer because corepack doesn't ship shims for them. * chore(console): drop 'npm i -g pnpm' anti-pattern from setup-script placeholder the suggested example trained customers to install pnpm unpinned, which silently picks up whatever's latest at run time. that's exactly the behavior #844 traced lockfile drift back to. now that prep handles package-manager provisioning via corepack from the repo's declared pin, the placeholder is just a frozen-lockfile install — load-bearing only when the repo wants `pnpm install` to actually run (prep already does that), but a much safer default for customers who do paste it in. * refactor(action): introspect opencode models for BYOK detection Replace the static `provider.envVars + provider.managedCredentials` catalog gate in `selectFallbackModelIfNeeded` + `validateAgentApiKey` with two `opencode models` captures around the auth merge: - `captureBaselineModels` BEFORE dbSecrets + Codex auth.json - `captureAuthorizedModels` AFTER both The authorized set is the authoritative source for "can OpenCode route this model" — strictly more accurate than the catalog, which can miss new auth shapes (Codex was one, there will be more). The diff between baseline and authorized is logged as `BYOK auth enabled N model(s)` for operator visibility. Sequencing changes in main.ts: - `createTempDirectory` hoisted out of the try block so `PULLFROG_TEMP_DIR` is set before the early opencode install - `agents.opencode.install()` + baseline capture before dbSecrets - `installCodexAuth()` hoisted up (idempotent — agent re-calls it inside run() and writes the same file) - authorized capture after Codex auth.json materializes - fallback + validateAgentApiKey receive the authorized set as a parameter; tests inject directly with no mocks Deleted: `hasProviderKey`, `getModelAuthEnvVars`, `knownApiKeys` in `action/utils/apiKeys.ts` (only `selectFallbackModelIfNeeded` consumed them, and PR #844's catalog-extension fix is superseded by introspection). `getModelEnvVars` / `getModelManagedCredentials` stay exported for UI and the server-side OSS proxy heuristic in run-context/route.ts. For the claude agent path, validateAgentApiKey keeps the static single-provider check on `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` — `opencode models` is opencode-specific. validateBedrockSetup / validateVertexSetup also stay; they cover region/location/model-id which `opencode models` doesn't catch. When fallback engages, the post-fallback model is the guaranteed-free `opencode/big-pickle`, so validateAgentApiKey is skipped — the fallback gate already authoritatively decided "this model is OK to run". * test(oss): temp add preview-844 to ossRepos for O4 e2e * Revert "test(oss): temp add preview-844 to ossRepos for O4 e2e" This reverts commit 8167e560126b2ac516c32ba1c63c36aa32ae4019. * test(oss): temp add preview-844 to ossRepos for O5 e2e * fix(action): skip validateAgentApiKey when proxyModel is set The new opencode-models BYOK introspection in PR #844 captures the authorized set BEFORE runProxyResolution mints OPENROUTER_API_KEY, so the proxy slug (e.g. `openrouter/moonshotai/kimi-k2.6`) is never in the set. validateAgentApiKey then spuriously threw "no API key found" on every OSS run, even though the proxy key was minted correctly and the inference would have worked. Mirrors the analogous skip in `selectFallbackModelIfNeeded`: when proxyModel is set, the server-side gate (`run-context/route.ts`) is the authority and the proxy mint itself is the validation. Caught by O5 e2e on `pullfrog/preview-844-heyapi-oss-bug`. * Revert "test(oss): temp add preview-844 to ossRepos for O5 e2e" This reverts commit 3bae075ceeb188ee272c45c13b5080e15bcd00a5. * fix(action): discard hook-generated tracked-file drift before agent sees it addresses bug 3 in #844: customer setup/post-checkout hooks like `pnpm install` or `corepack prepare` left the working tree dirty (e.g. `M pnpm-lock.yaml`), the agent took the prompt's "must push" rule literally, opened a spurious bot PR for the lockfile drift, and we ate runs+spend on noise. after each setup / post-checkout hook (opt-in via `normalizeWorkingTreeAfter`), discard tracked-file mods with `git restore --staged --worktree .`. untracked files are preserved — a hook that materializes a `.env` from a template, or emits codegen output, stays visible to the agent. guarded by a pre-hook `git status --porcelain` snapshot: if the tree was already dirty before the hook ran (shouldn't happen — setup runs before any working-tree writes; checkout_pr refuses to run dirty), we warn and skip the discard rather than clobber whatever was there. prepush hook (action/mcp/git.ts) intentionally does NOT opt in — its job is to read the about-to-be-pushed state, not normalize it. * test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge) * fix(action): skip eager pnpm/npm/etc install when no lockfile exists second half of bug 3 in #844. the eager prep step assumed `pnpm install --frozen-lockfile` (and equivalents) would fail cleanly without a lockfile, leaving the tree untouched. that assumption is false for pnpm 11.1.1 against a no-deps `package.json`: the command reports "Already up to date" with exit 0 AND silently materializes an empty `pnpm-lock.yaml` despite the `--frozen-lockfile` flag. the resulting untracked file trips the post-run dirty-tree gate, the agent reads it as "must push uncommitted work", and a spurious "Add pnpm lockfile" PR lands. smoking gun: pullfrog/preview-844-heyapi-oss-bug PRs #1/#2/#3, all auto-opened by the bot against a repo that contains nothing but a one-line README + a no-deps package.json. guard explicitly with an `existsSync` per manager. if the lockfile is absent, skip eager prep entirely with an info log; the agent can install on demand via the `setup` lifecycle hook (which non-frozen `pnpm install` would handle correctly), or just leave deps uninstalled when the prompt doesn't need them (e.g. the O5 "tell me a joke" path). orthogonal to the lifecycle-hook normalization in 0051bd2a — together they cover the full bug 3 surface: - eager prep can't materialize a lockfile (this commit) - setup/postCheckout hooks that rewrite tracked files have the drift discarded before the agent sees it (prior commit) * fix(action): address Pullfrog review on hook normalization two fixes in `executeLifecycleHook` from review on f6f3b32: 1. pre-hook snapshot was `git status --porcelain` which counts untracked files; in practice any repo with pre-existing untracked content (e.g. `.plans/`, an ignored-but-not-yet-gitignored scratch dir, codegen artifacts) would trip the guard and silently skip normalization, defeating the fix. switch to `git diff --name-only HEAD` so the gate measures the same thing the discard targets — tracked-file mods only. pre-existing untracked files are safe regardless because `git restore --staged --worktree .` never touches them. 2. normalization fired only on the happy path; a hook that updated a lockfile then exploded on a peer-dep conflict left tracked drift for the agent. move the call into a `finally` so it runs on success, non-zero exit, timeout, AND spawn failure. the pre-hook guard still protects pre-existing work in every case. * Revert "test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge)" This reverts commit f6f3b325d6bf9a1720754ed1d39d248dab76cfa8. * fix(action): use detect lockfile strategy for eager-prep gate addresses Pullfrog review on be3c207b. two findings, one root cause: - the hardcoded LOCKFILE_BY_MANAGER map missed `bun.lockb` and `npm-shrinkwrap.json`, two managers' accepted lockfile variants. - `existsSync(join(cwd, lockfile))` only checked the immediate directory, breaking monorepo subpackages where the lockfile lives at the workspace root. both fall out by replacing the custom check with `detect({ strategies: ["lockfile"] })`. the detector already walks up the tree (subpackage → workspace root) and recognizes every accepted lockfile name across all managers it supports. restricting to the `lockfile` strategy is load- bearing: the default strategy set also matches on `packageManager` / `devEngines.packageManager` package.json fields, which would return non-null and re-mask the very case we're trying to detect (declared manager, no lockfile committed — the O5 / hey-api preview repro). drops the LOCKFILE_BY_MANAGER map entirely; no need for a second detect() call since the existing one was only used for `agent` resolution and that consumer is now after the lockfile gate, where `detected` is guaranteed non-null. |
||
|
Colin McDonnellandpullfrog[bot]
|
b49c1d9a57 |
postRun: forbid set_output in reflection prompt (gemini pro regression)
reflection turn is a meta-turn for editing the learnings file; the task's `result` output was already finalized on the previous turn. gemini pro re-triggers on the standing "call set_output when done" system instruction during reflection and clobbers the value with the literal word "done" (see ci run 26529624199, smoke test on providers-live google/gemini-pro). add an explicit prohibition to the reflection prompt; the snapshot/restore in runPostRunRetryLoop remains as defense in depth. |
||
|
Colin McDonnellandpullfrog[bot]
|
b0868d48e6 |
git tool: reject {command, args[0]} duplicates with a directed error
models occasionally call `pullfrog_git({command:"status", args:["status"]})`,
which shells out to `git status status`. git silently treats args[0] as a
pathspec — when no file/dir matches, status prints "nothing to commit,
working tree clean" even on a dirty tree. observed in production
(Skn0tt/beckerbuch run 26519563044): the agent looped trying to reconcile
that against a real diff, burned ~$3 / ~6min of opus, and only escaped when
it switched to `args: ["--porcelain"]`. generalises to every subcommand
(`diff diff`, `log log`, ...).
guard `args[0]?.toLowerCase() === command.toLowerCase()` with a directed
throw pointing the model at the disambiguated `args: ["--", "<name>"]`
escape hatch for the rare legitimate pathspec case (`--` works under every
subcommand, unlike a bare positional which can be parsed as a ref by
log/diff/checkout/restore/reset).
description also leads with the no-args case and explicitly forbids
repeating the subcommand in args. schema already had args.optional().
|
||
|
Colin McDonnellandpullfrog[bot]
|
c89b0c7b4a |
action/README: drop waitlist banner, point to GA console
also includes in-flight working-tree work: - postRun: snapshot/restore toolState.output across reflection turn so reflection prompt can't clobber task-turn output (gemini pro regression) - toolState: widen `output` to `string | undefined` for assignability - uninstallFeedback: suspend-mode emails now CTA the GitHub unsuspend page when accountType is known; delete events keep console pointer |
||
|
Colin McDonnellandpullfrog[bot]
|
3440292abb | release: bump action to 0.1.14 | ||
|
Colin McDonnellandpullfrog[bot]
|
585a5d21cc |
fix(askpass): scope code + script lifetime to one $git() call (#841)
* fix(askpass): scope code + script lifetime to one $git() call, not first password prompt LFS pre-push (and any auth-bound sibling subprocess) consumed the single-use code AND triggered the script self-delete, so git's own push call then hit `fatal: cannot exec '/tmp/pullfrog-…/askpass-…js'` and our server treated the legitimate retry as tamper, revoking the installation token. Observed on nteract/nteract#2987 (LFS repo). `gitAuthServer` codes are now `active` until `$git()`'s finally calls `revoke()`; the script no longer self-deletes (finally already unlinks). Replay after revoke still trips 409 + token revocation, which is the realistic exfiltration vector we care about. * fix(askpass): drop wall-clock TTL on active codes Copilot review on #841 noticed the 5-minute CODE_TTL_MS still applied to active codes, which would re-introduce the original LFS failure mode at a different boundary: a large LFS push lasting >5min would hit a 404 mid- call. $git() uses `activityTimeout: 0` precisely because git fetch/push can take arbitrarily long, so any wall-clock TTL on active codes is wrong. Active codes now live until revoke() is called (in $git()'s finally) or the auth server is closed. Revoked codes keep their 60s replay trap. * docs(askpass): purge stale single-use vocabulary; align error message Pullfrog review on #841 surfaced four doc-drift sites that still described the pre-PR single-use model: - wiki/security.md — 3 references (overview prose, bullets, threat-mitigation table) - action/utils/gitAuth.ts — file-level JSDoc + the 409 error message - wiki/askpass.md — error message quoted in the tamper-evident section - action/utils/gitAuthServer.ts — per-prompt invocation comment was ambiguous All updated to match the active|revoked vocabulary; error message is now "askpass code was replayed after revoke, token revoked". |
||
|
Colin McDonnellandpullfrog[bot]
|
dc4dff98da |
allowlist middleapi/orpc for OSS, ship skill scaffold + no-mock audit
- utils/ossRepos.ts: add middleapi/orpc (orpc.dev, 5.2k stars) to the oss allowlist so any future install mints via mintOssKey ($10 cap, pullfrog absorbs). - scripts/skill.ts + pnpm skill: scaffold .agents/skills/<name>/SKILL.md + .claude/skills/<name> symlink. patch + skill skills written using it. - AGENTS.md: hard-ban vi.* mocking apis; document pnpm skill workflow. - audit follow-through: drop pure-mock test files (action/utils/lifecycle, action/utils/timer, test/handleNoInstall) and trim action/mcp/review to the non-mock cases. - wiki/scripts.md: row for scripts/skill.ts. |
||
|
Colin McDonnellandpullfrog[bot]
|
a0746dcc27 | release: bump action to 0.1.13 | ||
|
Colin McDonnellandpullfrog[bot]
|
ed8ee363c0 | release: bump action to 0.1.12 | ||
|
Colin McDonnellandpullfrog[bot]
|
f327f65413 |
revert(test): keep opencode default on sonnet, not kimi
c45a07e5 swapped the opencode harness default to moonshotai/kimi-k2, which broke push-enabled (Kimi reported delete_branch as "auth failed" where sonnet handles it). The agnostic test matrix expects Sonnet-grade tool-calling; switching to Kimi was scope creep on top of the original "don't default to opus" ask — opencode's default was already sonnet, not opus, so nothing needed to change there. |
||
|
Colin McDonnellandpullfrog[bot]
|
d93ddcbf4a |
expose push: restricted to users (#828)
* expose push: restricted to users the action runtime already understood `disabled | restricted | enabled` for `push` (see action/external.ts, action/mcp/git.ts); the DB enum, the zod schema, the React settings UI, and the action.yml input docs lagged behind. catches all four up. UI surfaces a 2-state toggle (restricted ↔ enabled) in the Security card alongside shell isolation — `disabled` stays in the enum so non-UI callers (action.yml input) still accept it but is intentionally hidden from the console. * address reviews: doc full restricted scope + disabled UI edge case reviewers flagged that "restricted" blocks more than default-branch pushes (also delete_branch + push_tags per action/mcp/git.ts:621-700), and that the popover's "off = full push access" claim is wrong for repos whose workflow sets `push: disabled` (also renders as OFF since the toggle is `props.push === "restricted"`). action.yml description now enumerates the full restricted scope, and the popover frames off behavior in terms of the workflow input so disabled repos aren't misrepresented. |
||
|
Colin McDonnellandpullfrog[bot]
|
e52206b8ca |
test: switch opus defaults to sonnet/kimi for cheaper coverage
- opencode test default: claude-sonnet-4-6 → moonshotai/kimi-k2. claude-code default stays on sonnet (the agent-under-test for that path is Claude Code); the opencode path doesn't need an Anthropic model. - byok-no-keys-fallback fixture: anthropic/claude-opus → moonshotai/kimi-k2. test verifies fallback to opencode/big-pickle, so the configured model is never invoked; picking a non-opus alias avoids burning credits if the fallback ever regresses. - BEDROCK_MODEL_ID in both test workflows: us.anthropic.claude-opus-4-6-v1 → us.anthropic.claude-sonnet-4-6. env var is required by the ci.test invariant that every provider's env vars are wired into the workflow. |
||
|
Colin McDonnellandpullfrog[bot]
|
fd2c67ab50 |
adhoc: push:restricted adversarial pentest (#827)
* adhoc: push:restricted adversarial pentest
enumerates the 16 attack vectors the deep audit identified as load-bearing
for `push: restricted`. used to drive e2e verification against the preview
repo's pullfrog.yml; also runnable via pnpm runtest locally.
validator only asserts that the repo's default branch SHA didn't move —
the per-attack outputs are the deliverable for human review (the test
exists to feed adversarial runs, not to be a CI guard).
* wipe runner leak surface before agent spawn
the GHA runner persists credentials inside $RUNNER_TEMP that an MCP-shell
agent can grep — _runner_file_commands/set_output_* (from any composite
step that called core.setOutput, e.g. pullfrog/pullfrog/get-installation-token
which leaks a ghs_… installation token), <uuid>.sh rendered step scripts
(whose run: | body embeds ${{ ... }} expressions literally before write),
and git-credentials-*.config from actions/checkout@v6.
snapshot-and-delete that surface at action startup, after our own token is
in memory and before setupGit. preserves $GITHUB_OUTPUT, $GITHUB_ENV, and
$GITHUB_STATE so pullfrog's result output and post: hook still work.
setupGit's existing removeIncludeIfEntries call strips the matching
dangling includeIf.gitdir:....path entries from the user's .git/config.
does not tighten isGitCommand — that's a UX guard, not a security
boundary, and trivially bypassable via bash -c, absolute paths, symlinks,
python subprocess. the security boundary is the absence of credentials on
disk for those bypassed shells to authenticate with.
verified end-to-end by re-firing action/test/adhoc/pushRestrictedAdversarial
against pullfrog/preview-827-push-restricted-pentest.
* preserve all runner file-command paths from wipe
addresses pullfrog review on f7f5143b: GITHUB_STEP_SUMMARY also lives at
$RUNNER_TEMP/_runner_file_commands/step_summary_<uuid> and is read by the
runner AFTER our step exits to render the job summary in the GH UI. wiping
it silently broke pullfrog's job summary output. preserve GITHUB_PATH too
for symmetry — it's the same allocation pattern, and a step or post hook
that appends a directory expects the file to exist.
set of file-command env vars enumerated in @actions/core:
GITHUB_ENV, GITHUB_OUTPUT, GITHUB_PATH, GITHUB_STATE, GITHUB_STEP_SUMMARY
|
||
|
Colin McDonnellandpullfrog[bot]
|
b6c57547ca |
fix(claude): use claude-code as skills CLI agent name
the skills CLI rejects "claude" — its valid list is claude-code, opencode, cursor, etc. caused agent-browser skill install to fail on every claude run. |
||
|
Colin McDonnellandpullfrog[bot]
|
e2eb26573f |
surface agent failures in job summary (#632) (#802)
* surface agent failures in job summary (#632) when the agent harness returns `{success: false, error}`, main.ts went through the success path so the catch block — which renders the `### ❌ Pullfrog failed` banner via renderRunError — never fired. result: the GitHub Actions job summary showed only the partial body + usage table, no error block. the progress comment had a narrow workaround that re-implemented the api-key classifier inline. unify: in finalizeSuccessRun, call renderRunError once when `!success`, use `.summary` for the job summary (prepended to the existing body/usage parts) and `.comment` for the progress comment. removes the duplicated isApiKeyAuthError / formatApiKeyErrorSummary branch and picks up BillingError reclassification + hang body for free. * docs: note dual-surface failure rendering in finalizeSuccessRun * fix copilot review nit: clarify which renderRunError classifications carry the H3 banner |
||
|
Colin McDonnellandpullfrog[bot]
|
01e4daa0b5 |
checkout_pr: refuse unconditionally on dirty working tree (#808)
* checkout_pr: refuse unconditionally on dirty working tree drop the live-HEAD comparison from the guard introduced in #796. any checkout_pr call with staged or unstaged changes now throws, even when HEAD is already on pr-N. no stashing, no idempotent escape hatch. motivation is the zed-industries/cloud (2026-05-18) incident: shared-cwd subagents make "carry edits along" semantics dangerous, and the HEAD-equality predicate let a re-checkout silently inherit working-tree state from a sibling agent. forcing commit/discard before any PR-context operation eliminates the entire carry-forward failure class. error names the PR number, lists dirty paths, and tells the agent to commit/push/restore/clean before retrying. * improve dirty-tree error: precise discard commands copilot caught two sloppy bits in the error string: - "push" alone does not clean a dirty tree (needs commit first) - bare `git clean` is a no-op without `-fd` reword to "commit (then push if needed), or discard with `git restore --staged --worktree .` / `git clean -fd`" so the guidance is actually actionable. * checkout_pr: initial-branch invariant setupGit captures `toolState.initialBranch` at run start via live `git rev-parse --abbrev-ref HEAD`. checkout_pr refuses unless current HEAD matches the run-entry branch or the target `pr-N` (idempotent same-PR re-checkout). uses live rev-parse, not toolState.issueNumber (poisonable per the PR #796 review). refusal error names the current branch, target PR, recovery path (`git checkout <initialBranch>` with the literal branch name), and explicitly states routing around via the `git` tool is not sanctioned. closes the zed-industries/cloud (2026-05-18) shape where a subagent parked HEAD on someone else's `pr-X` and the orchestrator's next checkout_pr inherited that position. * reviewfrog: enforce canonical diff + pre-commit halt; align Build dispatch extend REVIEWER_SYSTEM_PROMPT with two prepended HARD CONSTRAINTS: - first action MUST be `git diff origin/<base>` (single-rev, captures uncommitted). no other diff first; no checkout_pr; no alt-ref fetches; no branch listing; no `gh pr list`. - empty canonical diff + claimed-changes dispatch ⇒ reply exactly with `no changes detected — likely pre-commit Build self-review; orchestrator should commit then re-dispatch` and stop. do not guess PR numbers (the zed thrash that ended in `checkout_pr({2582})`). reshape Build mode reviewfrog dispatch step around a verbatim template that names: (a) the situation is pre-commit, (b) canonical diff command, (c) halt-on-empty-diff rule. orchestrator side now says the same thing as the reviewer's baked-in prompt. delegation-discipline bullets and orchestrator-evaluation guidance kept intact. * checkout_pr: handle detached-HEAD entry in initial-branch invariant pullfrog incremental review caught a defense-in-depth gap: `git rev-parse --abbrev-ref HEAD` returns the sentinel string `"HEAD"` on detached entry, which is the default `actions/checkout` state for `pull_request` events. with the previous string-typed `initialBranch`, both the captured value and the live probe would equal `"HEAD"` on any detached state, trivially satisfying the invariant — including a subagent doing `git checkout --detach <sha>`. discriminate the captured HEAD: probe `git symbolic-ref --short HEAD` first (works on named branches), fall back to `git rev-parse HEAD` (SHA) on detached entry. store as `{ kind: "branch"; name } | { kind: "detached"; sha }`. checkout_pr runs the identical probe at call time and compares like-with-like (branch name vs branch name, SHA vs SHA). refusal error renders both heads via a small `describeHead` helper and chooses the right `git checkout` recovery target (branch name or SHA). no inline-discriminant `as` casts — uses a top-level `headsEqual` that narrows via the discriminator. |
||
|
Colin McDonnellandpullfrog[bot]
|
e65dbe420c | Use Vertex Claude Opus in vertex-claude CI smoke — Haiku quota exhausted. | ||
|
Colin McDonnellandpullfrog[bot]
|
58e5b74cb8 | Update footer test for big-pickle BYOK fallback. | ||
|
Colin McDonnellandpullfrog[bot]
|
7c5ed7add0 |
Rename OSS_PROXY_MODEL to DEFAULT_PROXY_MODEL; default Router proxy to Kimi K2.6.
Derive the platform default from moonshotai/kimi-k2 openRouterResolve, add models.dev drift coverage, and promote repos on default-branch workflow pushes when still needs_setup. |
||
|
Colin McDonnellandpullfrog[bot]
|
fb22cb3ae3 | release: bump action to 0.1.11 | ||
|
Colin McDonnellandpullfrog[bot]
|
c43ed65c3b |
Add Vertex AI routing support (#753)
* add Vertex AI routing support * include Vertex smokes in action CI |
||
|
Colin McDonnellandpullfrog[bot]
|
09344a9ec9 | release: bump action to 0.1.10 | ||
|
Colin McDonnellandpullfrog[bot]
|
1b201352b5 | release: bump action to 0.1.9 | ||
|
Colin McDonnellandpullfrog[bot]
|
6c166ac1cc |
fix: prevent cross-PR push from subagent-induced branch switch (#796)
* fix: prevent cross-PR push from subagent-induced branch switch A workflow_dispatch run for zed-industries/cloud (workflow run 26036155393) force-pushed the orchestrator's work onto an unrelated engineer's PR branch (origin/reactivate-pro-plan, PR #2582). The orchestrator's reviewfrog subagent called checkout_pr({pull_number: 2582}), which (1) moved the shared working tree to pr-2582 and (2) persisted pushDest pointing at reactivate-pro-plan. The orchestrator's subsequent commit + push_branch then clobbered the victim PR. Recovery + disclosure in PR #2584. Three compounding bugs closed here: 1. checkout_pr dirty-tree guard had a first-call hole: the previous condition required ctx.toolState.issueNumber to already be set, so on workflow_dispatch runs the first checkout_pr (commonly from a subagent) bypassed the guard entirely. Now any PR switch with a dirty tree is refused, including the first switch of a run. Idempotent same-PR re-checkouts are still absorbed by alreadyOnBranch inside checkoutPrBranch. 2. push_branch trusted sticky pushDest blindly. Added a backstop: refuse pushes where the local branch matches /^pr-(\d+)$/ AND pushDest.remoteBranch differs from it AND the current run is not scoped to PR N (event.is_pr === true && event.issue_number === N). This catches subagent-induced silent branch switches even if a future bug reintroduces a first-call hole in fix #1. 3. Build-mode self-review prompt told the orchestrator to ship "the output of git diff" to the reviewer. The model in this run synthesized git diff main...HEAD, which excludes uncommitted work — and Build self-review runs BEFORE the commit, so the reviewer saw an empty diff and thrashed, eventually calling checkout_pr on a random PR to find something to look at. Prompt now specifies git diff origin/<base-branch> (two-dot, no HEAD), which compares the working tree against the remote base. Refs: zed-industries/cloud workflow run 26036155393 zed-industries/cloud#2582 (victim) zed-industries/cloud#2584 (disclosure) * review: key dirty-tree guard on current branch + drop 'two-dot' misnomer Address review feedback on PR #796. 1. checkout_pr dirty-tree guard now keys off the live current branch (git rev-parse --abbrev-ref HEAD), not ctx.toolState.issueNumber. issueNumber is ALSO set by get_issue / get_issue_comments / get_issue_events, so a subagent doing get_issue(N) followed by checkout_pr(N) on a dirty tree would have bypassed the original guard (issueNumber === pull_number). The current branch is the actual primitive for "would this call move HEAD" — querying it directly avoids correlating on toolState that other tools write to. 2. modes.ts: drop the wrong "two-dot" label on git diff origin/<base>. That's the single-rev form, not two-dot. Copilot was right that the label was confusing/contradictory with the actually-shown command. |
||
|
|
a0576a702a |
opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767)
* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite
Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.
Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.
Surface area:
- drop `init` / `message` / `result` / `tool_result` event types and
handlers (no longer emitted at v1.14+ per upstream
`cli/cmd/run.ts:588-601`).
- `tool_use` is now a single event covering both `state.status:
"completed"` and `"error"`. duration / subagent-finish bookkeeping
moves from the v1 `tool_result` handler into the consolidated
`tool_use` handler.
- new `reasoning` event handler — gated on `--thinking`, surfaces
Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
`baseArgs`.
- drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
v1.15 the `task` tool callID is stable across the whole
`tool-input-* → tool-call → tool-result/tool-error` chain
(`session/processor.ts:282-330`). exact-match map is sufficient.
- drop `experimental.batch_tool: true` from injected config — declared
but inert at v1.15. re-add once upstream wires it back.
- bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
the platform-specific binary into `opencode.exe` for every OS now).
Validated locally:
- `pnpm test` 610/610 ✓
- `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
- `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
(this is the headline `thought_signature` fix)
- runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
- runtest opencode: skill-invoke and mcpmerge fail (model-behavior
drift on the new system prompt; wiring confirmed intact via direct
repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
Tracked for follow-up; does not gate the migration.
Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.
* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)
The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.
* opencode v2: set PWD env explicitly to fix skill / project-config discovery
Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:
const root = Filesystem.resolve(process.env.PWD ?? process.cwd())
We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.
Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.
Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).
* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly
opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).
Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.
Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now: `» thinking (4.0s): <preview>` (one line, sourced)
For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.
Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.
* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment
claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.
Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.
Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.
* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup
Code-quality pass on the v2 work:
1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
between v1 and v2:
- `OpenCodeConfig` type
- `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
- `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
and re-imported by v2 via a back-reference)
- `installOpencodeCli({ binPath })` (parameterized — v1 passes
`bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
`installCli` lambda; matches each pinned version's npm shape)
- `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback
v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
keeps a one-line `export { geminiHighThinkingOverrides }` re-export
so `opencode.test.ts` keeps working unchanged. Once v1 is retired
(post burn-in) opencodeShared collapses back into v2.
2. `opencode_v2.ts` cleanup:
- drop dead state (`currentStepId`, `stepHistory` were write-only —
their reader was the v1 `tool_result` handler we deleted)
- hoist `state` in `tool_use` handler; replace nested-ternary payload
extraction with a `terminalPayload(state)` helper
- extract `formatPartDuration(time)` for the reasoning-block
"(X.Ys)" suffix
- tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
`callID` fields directly, drop the `partWithToolFields` cast
- trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
handler, tool_use handler, bus envelope handler all shortened
- `step_start` becomes an explicit `() => {}` no-op so the dispatcher
doesn't log "unhandled event" for every step
3. `subagentRegistration.test.ts` retargeted at the new file split —
reads opencodeShared.ts for the buildReviewerAgentConfig assertions
and opencode_v2.ts for the orchestrator-model wire-through.
Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.
* opencode v2: address PR review feedback
Three fixes from the inline review threads on #767:
1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
level `markActivity()` resets the module-level idle counter, so the
per-event `getIdleMs()` sample inside the dispatch loop was always
~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
a runner-local `lastEventAt` so we measure real event-to-event silence
instead of chunk-arrival latency. Drop the unused `getIdleMs` import.
2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
`lastProviderError`, and `recentStderr` are closed over by the
`handlers` const but were declared after it. No current bug because
handlers only fire inside the awaited `spawn()`, but a future
refactor that triggers a handler synchronously during setup would
surface a TDZ. Hoisted above `handlers`.
3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
L566): leave a `TODO` comment marking the gap until `AgentUsage`
grows a `reasoningTokens` field — separate PR with schema work.
Cost totals stay correct because `part.cost` is summed independently.
Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
`reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
the bus-envelope flow; the plugin filters orchestrator events except
for status=running task dispatches, and bus-envelope returns before
calling handlers.tool_use on those. No double-fire under current code.
Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.
* DX: flip pnpm play / pnpm runtest to docker-by-default
Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.
scripts (root → action):
- pnpm play → pnpm -C action gha play.ts (docker, default)
- pnpm play:local → pnpm -C action play:local (host)
- pnpm runtest → pnpm -C action gha test/run.ts (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)
action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.
CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.
Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.
File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
`pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers
AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.
wiki/docker.md unchanged — already described the now-real shape.
* test/crossagent: add codex-auth smoke
Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:
- installCodexAuth() materializes auth.json under the test HOME
- opencode routes openai requests through ChatGPT subscription auth
(no OPENAI_API_KEY in env, AT path forced via expires: 0)
- the refresh chain advances during the run (refresh_token rotates)
- detectCodexRefresh() would surface the rotation to entryPost.ts
The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.
CI wiring (already committed in a1c1fd4f as part of the DX flip):
- .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
action-agents env block
- action/.github/workflows/test.yml: same; codex-auth in the
hardcoded test matrix with a claude exclude
The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.
ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.
* docs(codex-auth): make storage requirement unmissable
A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).
- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
loudly contrast Pullfrog secret store vs GH Actions and explain the
writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
unchanged per scope.
* auth codex: auto-open device URL, drop --scope flag
- detect `https://auth.openai.com/codex/device...` from codex CLI output
and best-effort launch it in the user's default browser (open / xdg-open
/ cmd start, wslview fallback on linux). gated so we only open once per
flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
interactive (browser approval), so a "skip-the-prompt" flag for just one
of the prompts was dead weight. collapses scope selection to "always
prompt on org-owned, always account on user-owned".
* rename gha→docker, flip play/runtest defaults to host
the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.
- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
`pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
`README.md`, `AGENTS.md`), CI workflows
(`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
and code headers (`action/{play,test/run,utils/runFixture}.ts`,
`webhook.ts`, `action/test/coverage.ts`)
`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).
* fix(codex): route post-hook writeback through apiFetch + conditional skip
Three threads addressing PR #767 followups.
action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).
action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.
doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
mention (removed in 10be96db, scope is now always interactively
prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
is agent-gated (opencode/opencode_v2 harness), not model-gated;
opencode runs with non-OpenAI models still materialize the file,
it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
→ pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
instead of the dropped --scope flag.
* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS
upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.
also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.
local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.
---------
Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>
|
||
|
Colin McDonnellandpullfrog[bot]
|
cb0dbcd371 |
feat(action): make prepush hook non-blocking after one failure (#777)
* feat(action): make prepush hook non-blocking after one failure push_branch now treats the repository's prepush hook as best-effort: it runs at most once per run, surfaces the failure output if the script exits non-zero, and every subsequent push_branch call this run skips the hook so the agent isn't blocked by failures unrelated to its change. The agent can iterate by running the hook command itself via the shell tool when shell access is available; push_branch will not re-run the hook automatically after a failure. Why: a one-line OSS-allowlist change took 9 minutes (#776) because the agent retried push_branch six times against a prepush hook that was failing for env-leak and missing-build-artifact reasons unrelated to the change. CI catches the same checks on the GitHub side; the local prepush gate was duplicating work and blocking unrelated fixes. - ToolState: new prepushFailureCount counter (per-run, never resets) - executeLifecycleHook: returns structured failure (kind/output/exitCode) so prepush can compose its own agent-facing message instead of inheriting the generic retry/no-retry advice meant for setup - push_branch: composes a shell-mode-aware error message; surfaces prepushSkipped on the success payload + appends a note to the message - instructions.ts + wiki/prompt.md + docs/comparisons.mdx: updated to reflect best-effort semantics * fix(action): clarify prepush latch semantics + soften static guidance review fixes from PR #777: - toolState comment, instructions, success message, tool description: replace "runs at most once per run" / "first call only" wording with the actual semantic — successful prepush keeps running on later push_branch calls; only a hook FAILURE latches the bypass. - tool description: drop hardcoded "via the shell tool" guidance so the static description doesn't mislead in shell:disabled runs (the dynamic agent prompt in instructions.ts already does shell-conditional messaging). - LifecycleHookFailure.output JSDoc: match the implementation (stderr-preferred fallback to stdout, empty for timeout/spawn). * fix(action): shorten prepush-skip log to terse operator telemetry the previous log line tried to address the agent ("re-run the hook command yourself via shell"), but log.info writes to the action runtime's stdout — the agent never sees it. agent-facing skip guidance already lives in the error message from buildPrepushFailureMessage, the success message when bypassed, and the system prompt in instructions.ts. log line is now just operator telemetry. * refactor(action): drop slop from prepush soft-fail self-audit pass after the previous review-fix round. removed duplication between code-level comment and the five other places that already explain the same behavior, tightened verbose JSDoc, and collapsed redundant clauses in agent-facing strings. - LifecycleHookFailure → discriminated union. drops the optional exitCode/spawnError fields (and the empty-output sentinel for timeout/spawn) plus the corresponding ?? fallbacks in the helper. - PushBranchTool: 7-line code comment above the latch removed (toolState field comment + tool description + error message + success message + system prompt all already cover it). tool description third sentence dropped (restated the second). success message tightened to a parenthetical. - buildPrepushFailureMessage: 4-line JSDoc → 1 line. shared "if you think the failure could indicate a real bug in your code" prefix factored out across the shell-conditional branches. - ToolState.prepushFailureCount comment: 8 lines → 3. the "what" is in git.ts; comment now only documents the invariant (never decremented within a run). - instructions.ts prepush guidance: collapsed nested bullets + ternary into one paragraph; dropped the "so re-running via shell is the only way…" tail that restated "push_branch will NOT re-run it". * fix(action): hint prepush bypass on dirty tree after hook failure When push_branch blocks on a dirty working tree and the prepush latch is already set, tell the agent the hook will be skipped once the tree is clean. * fix(test): narrow CI matrix for lifecycle and toolState changes Remove lifecycle.ts from ALWAYS_RUN_ALL and add lifecycle.ts + toolState.ts to push/git agnostic test coverage so PRs touching prepush latch logic run targeted tests instead of the full matrix. |
||
|
Colin McDonnellandpullfrog[bot]
|
7e90e5cae6 |
Align Plan-mode prompts on report_progress as the canonical plan tool (#786)
* fix: align Plan-mode prompts on report_progress as the canonical plan tool Fixes #673. Three sites disagreed on where Plan output should be posted, letting a model synthesize a broken third interpretation (initial post via `report_progress({ target_plan_comment: true })`, which then misses the `existingPlanCommentId` precondition). This PR aligns all three on `report_progress` as canonical, with `target_plan_comment` reserved for revisions only: - `action/modes.ts` Plan step 4 — spell out that the initial plan post uses `report_progress` WITHOUT `target_plan_comment`, and that revisions go through `select_mode`'s PlanEdit override. - `action/mcp/comment.ts` `target_plan_comment` flag description — make the "revisions only" precondition explicit and call out the initial-post path by name. - `action/utils/instructions.ts` Progress reporting paragraph — drop the misleading "(e.g., Plan comments)" parenthetical that read as "use create_issue_comment for plans". `PlanEdit` (in `action/mcp/selectMode.ts`) was already correct and is unchanged. Intentionally out of scope (to keep the fix minimal): a `publish_plan` tool, removing the vestigial `create_issue_comment({ type: "Plan" })` branch, hardening the run-end cleanup guard for the `target_plan_comment but no existingPlanCommentId` fallthrough, and renaming `target_plan_comment`. * align create_issue_comment description with report_progress as canonical plan tool |
||
|
Colin McDonnellandpullfrog[bot]
|
69c7d4b8cd |
models-bump: harden CI and bot prompt against catalog hallucinations
PR #790 (the first bot-authored models-bump PR) shipped a broken bump for openrouter/gemini-flash: the bot pattern-matched the parallel google bump and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the OpenRouter API but not on models.dev's openrouter section — the catalog OpenCode actually reads. The slug failed at runtime with `Model not found`. Two CI gaps let it through: 1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*` alias as a "passthrough", smoke-testing only one canary per routing layer. But those aren't passthroughs — each is a distinct models.dev catalog entry that can drift independently of the direct-provider mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25). Only `bedrock/byok` stays pruned (sentinel resolve). 2. `models-catalog` test (the integrity gate that asserts every resolve exists on models.dev) was main-only by design — to keep upstream catalog churn from blocking unrelated PRs. But it's exactly the test we want running on the bot's own catalog edits. Add `pullfrog/models-bump` head-ref to its trigger. Also tighten the bot prompt in models-bump.yml: new rule 0 requires every new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the alias's own `candidates[]` in models-bump-context.json — the deterministic preprocessor only emits candidates sourced from models.dev's mirror, so this gates against the cross-alias pattern-matching that broke PR #790. For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter API), which is necessary but not sufficient; the `models-catalog` job is the authoritative models.dev check. Verified locally: - baseline `pnpm -C action test:catalog` passes 133 tests - simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash` into action/models.ts) and the catalog test fails with the right assertion: `model "google/gemini-3.5-flash" not found under openrouter on models.dev` - `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every openrouter/* alias and every keyed opencode/* alias now smoked |
||
|
Colin McDonnellandpullfrog[bot]
|
f3d18401ac |
eager signup credit + free-OpenCode fallback when BYOK has no key (#789)
* eager signup credit + free-OpenCode fallback when BYOK has no key
addresses the silent-churn pattern that took out 15 first-run-failure
accounts post-launch: GH Actions secret references resolved to empty
strings (because the secrets didn't exist on the repo), the action
launched Claude Code with no key, the LLM provider 401'd, and the run
died in seconds with a synthetic "Invalid API key" message. those
accounts had no Router credits to fall back to because the lazy claim
required a dashboard visit they never made.
three changes, one PR:
1. Eager $10 signup credit at account creation. Both account-creation
sites (`upsertAccountByClerkId` for dashboard signin, `fetchOrCreateRepo`
for CLI / GH-App-only) now insert the `CreditGrant { reason: "signup" }`
in the same transaction as the `accounts` row. CLI installers who
never sign in get the credit. The dashboard `/signup-credit/claim`
POST stays as an idempotent backstop for accounts created before
this shipped.
2. Free-OpenCode fallback in the action. When the configured BYOK slug
needs a provider key the runner doesn't have, swap to
`opencode/minimax-m2.5-free` before agent selection so the run still
succeeds. Surfaced via a `» fell back from <slug> to <free>` warning
in the action log. Skipped on Router runs (Pullfrog mints the key)
and when no model is configured (auto-select-with-throw still fires
for the genuinely-misconfigured case).
3. New action-test fixture `byok-no-keys-fallback` that empty-strings
every known provider key (matching how GH Actions handles missing
secrets) and asserts the run succeeds with the fallback log line
present. plus a unit test for the helper covering each skip case.
skipping the schema flip from `byok` to `router` — that's coming via
the onboarding-stepper PR (#762).
* fallback: skip Bedrock + surface in PR-comment footer
addresses copilot review on #789 (real bug — parseModel throws on
Bedrock raw IDs that have no slash, would crash before
validateBedrockSetup could surface its own error) and the user-side
ask to make the fallback visible in PR comments.
- selectFallbackModelIfNeeded skips when resolvedModel has no '/' so
Bedrock routing IDs (e.g. us.anthropic.claude-opus-4-7) don't crash
inside hasProviderKey -> parseModel. unit test covers it.
- toolState.modelFallback records the configured slug we fell back
from. set in main.ts when fallback engages.
- buildPullfrogFooter accepts fallbackFrom and renders
"Using `MiniMax M2.5` (free) (credentials for Claude Opus not
configured)" so the substitution is visible in PR comments,
reviews, PR bodies, and error reports.
- threaded through all four action-side footer call sites
(mcp/comment, mcp/pr, mcp/review, utils/errorReport). server-side
call sites in triggerWorkflow.ts / handleWorkflowRunWebhook.ts
fire pre-action and don't have toolState — left as-is.
* fallback footer: use provider display name + document email asymmetry
addresses pullfrog reviewer findings on #789:
- footer now shows 'credentials for Anthropic not configured' (provider
display name from `providers.anthropic.displayName`) instead of the
per-model name. credentials are provider-scoped (ANTHROPIC_API_KEY
covers all Anthropic models), so this matches what the user actually
needs to fix.
- document the intentional asymmetry between eager and lazy signup
credit paths: eager skips both the signupCreditClaimedEmail and the
per-grant team@ alert. comment explains why (the 'new account
created' alert already covers it on the eager path; the user-facing
email assumes a user-initiated action that hasn't happened yet for
CLI/GH-App-only signups).
- skipping the backfill for the 15 historical accounts per user's
earlier decision — they all uninstalled, so the cohort self-selected
out of being reachable.
* fallback: gate on resolvedModel + skip resolveModel re-resolve post-swap
local agnostic fixture run surfaced two real bugs the unit tests didn't
catch:
1. fallback gate was on configuredSlug (=payload.model) but the test
uses PULLFROG_MODEL to set the model, which is read by resolveModel
AFTER its slug arg. configuredSlug stayed undefined → fallback never
fired. drop configuredSlug from the helper signature; gate purely on
resolvedModel since that's the same value regardless of how the
model was specified (DB config vs PULLFROG_MODEL env).
2. when fallback engaged, the post-swap resolveModel({slug: fallback.to})
call was ALSO honoring PULLFROG_MODEL, re-overriding the fallback
target back to the unkeyed model. validateAgentApiKey then threw
"no API key found" against the original model. fix: skip the
re-resolve. fallback.to is already a CLI-ready specifier.
unit tests updated for the new helper signature (8 tests, all pass).
fallback log line confirmed emitted in the local run pre-second-fix;
the second fix unblocks the validation that previously threw.
|
||
|
Colin McDonnellandpullfrog[bot]
|
8dff91ac49 |
router: fix unspendable signup credit on no-card private repos (#792)
* router: fix unspendable signup credit on no-card private repos (#791) The bug ------- `run-context/route.ts` gated `proxyModel` minting on `isInfraCovered`, which is `oss || hasCard`. So a no-card account with positive wallet balance (signup credit, top-up, etc.) on a private repo would never get a `proxyModel` set on the run context. The action runtime then fell through to whatever provider keys happened to be in the workflow env — using the user's BYOK keys without their knowledge if any were configured, or failing the run entirely otherwise. Meanwhile `proxy-token/route.ts` already gated correctly on `oss || hasCard || balance > 0`. The two routes disagreed, with run-context being strictly more restrictive, so the agent never even attempted to call proxy-token for these accounts. The wiki at `billing.md:1052` documented the *intended* behavior ("a Router usage row can debit a wallet with no card on file"), aspirational against the actual code. The action side had a parallel bug at `action/utils/proxy.ts:151` — it re-derived `isInfraCovered({ isOss, plan })` and short-circuited mint even when the server set `proxyModel`. Belt-and-suspenders that was strictly more restrictive than the server. Production impact ----------------- Queried 55 router-mode no-card accounts holding signup credit: - ALL have wallet balance = exactly $10.00 (untouched) - ALL have 0 router proxy keys ever minted, 0 hwm usage - ~25 have successful runs (using BYOK env vars from their workflow, unaware their credit isn't being touched) - The rest have zero successes; some accumulated 25+ failures (e.g. `onechannelpe`: 25 failures, 0 successes, no card, $10 credit). The fix ------- - `run-context/route.ts`: widen `useRouter` to match proxy-token's gate. OSS short-circuits as before. Otherwise: router mode + card on file → mint; router mode + no card + positive balance → fetch balance, mint if > 0. Skip the balance read when a card is on file (auto-reload covers it without needing pre-flight balance — keeps the hot path single-query). - `action/utils/proxy.ts`: drop the redundant `isInfraCovered` check. `ctx.proxyModel` IS the signal — the server is the authority on funding decisions; the action just trusts and mints. - `wiki/pricing.md`: correct the Router proxy key minting gate row + add a paragraph explaining why this gate diverges from `isInfraCovered`. - `wiki/billing.md`: rewrite the misleading "proxy-token returns 402" paragraph to describe what actually happens at both routes. `isInfraCovered` is unchanged. It still gates Pullfrog-paid features (learnings writes, indexing). The bug was in conflating "Pullfrog pays for marginal infra" with "user can fund a Router run via wallet" — different concerns, now untangled. * action: drop dead isInfraCovered + plan param post-fix Cleanup the action-side dead code introduced by the previous commit's removal of the redundant `isInfraCovered` re-derivation in proxy.ts: - delete `isInfraCovered` from action/utils/runContext.ts (was the only callsite; mirror in server's utils/billing.ts is unchanged and still load-bearing for learnings/indexing) - drop unused `plan: AccountPlan` param from `resolveProxyModel` / `runProxyResolution` (and the corresponding `AccountPlan` import + the `plan: runContext.plan` arg at the main.ts call site) - update the action/mcp/server.ts comment that pointed at the now-gone action mirror to reference the server-side `utils/billing.ts` instead `AccountPlan` itself is still load-bearing (mcp/server, runContextData, run-context fetch), only `isInfraCovered` and the dead `plan` parameter go away. |
||
|
Colin McDonnellandpullfrog[bot]
|
0d7955d87d | add agent-browser fallback rule for unreachable chrome devtools mcp | ||
|
Colin McDonnellandpullfrog[bot]
|
3514bbc39f |
review prompt: tighten body-section bar + inline technical-details (#770)
* review prompt: tighten body-section bar + add inline technical-details Two layers of tightening to the Review/IncrementalReview prompts in PR_SUMMARY_FORMAT (and the per-mode aggregate-&-draft step): 1. Reframe inline-vs-body split. Body `### ` sections are now reserved for concerns that genuinely have no line to anchor to — absence, sequencing, design decisions, scope questions, architectural risk. Drop the "cross-cutting concerns" framing (misled the agent into either filing nothing in the body or filing multi-file anchored findings there). 2. Add a "Hunt for non-anchored concerns" sub-step to both Review (step 6) and IncrementalReview (step 8) aggregate phases. Diagnosis from PR #767's auto-review: on substantial PRs the agent surfaced findings but routed all of them inline, producing reviews with zero `### ` body sections even on diffs where non-anchored concerns clearly existed. 3. Replace the abstract `### ` example with a concrete non-anchored one ("Legacy `opencode.ts` has no documented deletion plan") so the agent pattern-matches the absence-shaped finding, not a line-bug. 4. Add an "Inline technical details" subsection to PR_SUMMARY_FORMAT so inline comments can carry a `<details>Technical details</details>` block when the fix has cross-file implications. Rename the existing "Agent details" inline collapsible to "Technical details" for consistency with body sections. 5. (Carried over from prior uncommitted work) Restructure the review metadata block from `<details>Review metadata</details>` into an HTML comment + an italic TL;DR commit-range line. The HTML comment keeps the metadata addressable for downstream agents without eating user-visible review real estate. No tests touched. * wiki: document multi-model end-to-end eval pattern |
||
|
Colin McDonnellandpullfrog[bot]
|
8ac954a27f | fix(handleIncompleteSetup): also skip nudge when repos are disabled, not just active | ||
|
Colin McDonnellandpullfrog[bot]
|
88f170e19a |
fix: 7 log-audit / run-audit findings (mega-PR) (#769)
* fix(#765): silence Clerk 400 (revoked OAuth) noise from getTokenForClerkId Branch on isClerkAPIResponseError + status<500 so the well-understood revoked-token redirect doesn't emit a level=error line in Better Stack on every request. Vercel maps console.warn -> error for non-streaming routes, so a downgrade to log.warn wouldn't help; only the unexpected shape (5xx, network) is worth surfacing. * fix(#742): stop logging input verbatim from yes.op retry-failure paths GitHub OAuth user tokens (ghu_...) were leaking to Better Stack on every yes.op retry-failure for any utils/github/get* helper that takes a token field — 38 leaks/7d in the most recent audit window. The leak path is console.log inside the yes package (its own log shim, not utils/log.ts). Drop input from the four log sites + the cache-key-derivation throw site. key (SHA-1 of input) is sufficient for retry correlation; error already carries request URL + status. Defense-in-depth comment so future contributors don't re-add the field. Operational follow-up (separate task): inventory ghu_... strings in Better Stack ingested in the last 90d, revoke matching Clerk grants, scrub cold-tier S3, rotate the BS source token. * fix(#759): handle GraphqlResponseError "Could not resolve to a node" as 404 When the stored planCommentNodeId references a comment that's been deleted on GitHub, octokit.graphql throws GraphqlResponseError before the existing `node === null` 404 branch is reached. Add a narrow isGraphqlNodeNotFound predicate in utils/errors.ts and a new catch branch in the plan-comment route. The action treats 404 as "no prior plan comment" and creates a fresh one, so behavior matches existing contract. * fix(#747): convert webhook GraphQL rate-limit 5xx into a Result<T> sentinel + 200 ack When GitHub's GraphQL responds with "API rate limit exceeded for installation ID N", _getReviewCommentsWithReplies threw, propagated through the bare yes.op wrapper (no rate-limit bail), out of the bare await in handleWebhook, and crashed /api/webhook/github with 500 — 77 webhook 500s/24h on the most recent audit window. GitHub redelivery plus R2 dedup also silently masked the legitimate handler from re-running once the rate-limit window cleared. Mirror the #658 / _getRepository pattern: detect GraphqlResponseError matching /rate limit (already )?exceeded/i, log.warn with the x-ratelimit-reset value (and [Installation N] prefix when available), return failure(...) with status 429. Webhook handler short-circuits the case with 200 + log.info so GitHub stops the redelivery storm against an exhausted budget, and the trigger page surfaces a clean ThrowClientError. Document the new pattern as a Tier 2 false-positive in wiki/log-audit.md so the next audit cron doesn't re-flag it. Note that returning [] silently (the issue's first suggestion) would have dropped @pullfrog mentions inline in review comments and dispatched an agent run that re-rate-limits — skip-the-whole-case is the correct semantics. Co-vulnerable getPullRequest / getWorkflow have zero occurrences in this window; per #737 policy, defer until they show up. NOTE: this commit and the bracket of touched files revert as a unit — the Result<T> shape change in getReviewCommentsWithReplies is breaking; partial revert breaks the type chain. * fix(#766): fold stderr+stdout into shell.ts errors + carve out merge-base --is-ancestor action/utils/shell.ts dropped stdout when constructing failure messages ($\{stderr || "Unknown error"\}), so git subcommands that write context-bearing diagnostics to stdout (merge conflicts, cherry-pick rejections, diff --exit-code, ls-files --error-unmatch) surfaced as "Command failed with exit code 1: Unknown error" through mcp__pullfrog__git. The agent burned an extra MCP round-trip calling git status to recover. Fold stderr + stdout into the thrown error message (stderr first, stdout fallback) so the agent always sees the real diagnostic. Plus a narrow carve-out for `git merge-base --is-ancestor` in action/mcp/git.ts: that subcommand uses exit code as data (0=ancestor, 1=not-an-ancestor, >1=error), so return { success: true, isAncestor } instead of throwing on exit 1. No caller in action/ string-matches on the old error format (verified). diff --exit-code and ls-files --error-unmatch are not carved out — both are zero-occurrence in the May audit window, and the stderr+stdout fold renders their output usefully anyway. * fix(#739): point customers at the actual fix when permissions: id-token: write is missing When a customer workflow runs in GitHub Actions but lacks permissions: id-token: write, ACTIONS_ID_TOKEN_REQUEST_URL/_TOKEN aren't injected, isOIDCAvailable() is false, and acquireNewToken falls through to the local-dev-only acquireTokenViaGitHubApp path, which throws "GITHUB_APP_ID and GITHUB_PRIVATE_KEY must be set" — pointing at a self-hosted-app fix that doesn't apply. One affected customer burned 13 dispatches in 24h on this misleading error. Detect (GITHUB_ACTIONS=true) AND (no OIDC env vars) inside acquireNewToken before falling through to the local-dev branch, and throw an actionable message naming the missing permissions block, the exact YAML, and the docs anchor. The error surfaces via ##[error]action failed: ... in the workflow log (the only customer surface available before main()'s inner try opens). Local-dev path keeps the existing GITHUB_APP_ID message. * fix(#760): suspend activity watchdog across in-flight tool calls mcp__pullfrog__checkout_pr was hard-failing 6/24h on SenecaLabs/senecaWeb because git fetch+deepen on a large monorepo can take 4-5 min, the agent's stdout pipe goes silent the entire time (FastMCP is in-process HTTP, but Claude/opencode CLIs await the synchronous tools/call response), and both the spawn-level activity timer (300s in subprocess.ts) and the process-level activity monitor (300s in activity.ts) fire and kill the run. Re-introduce the bracket pattern that PR #634 removed: bracket suspendActivity()/resumeActivity() around tool_use -> tool_result in both agent harnesses, plumb isPausedExternally into spawn() so both timers suspend in lockstep. Bounded by MAX_TOOL_CALL_SUSPENSION_MS (15 min auto-resume) plus the outer 1h agent timeout — neither zombie-run avenue from #12 is reopened (subprocess.close still resolves on death; outer timeout is suspend-agnostic; suspends gated on explicit paired CLI events, not internal noise). opencode tool_use handler: gate suspendActivity() on non-terminal status (running/pending) so the bus_event re-dispatch path at line 915 — which only fires for completed/error subagent parts and never emits a paired tool_result — doesn't latch the watchdog into suspension until the 15min ceiling. Add a heuristic:activity-watchdog-ceiling classifier to scripts/analyze-logs.ts so a tool that genuinely hangs past MAX_TOOL_CALL_SUSPENSION_MS surfaces in run-audit instead of being bucketed into failure:unknown. NOTE: this commit and the bracket of touched files revert as a unit — activity.ts, subprocess.ts, and the two harnesses must move together or the bracketing breaks. * refactor(#747): swap Result<T> for InstallationRateLimitError typed throw The Result<T> shape from 3ebf6c4c was cargo-culted from the #658 _getRepository pattern, but _getReviewCommentsWithReplies has only one expected-error case (installation rate-limit) and two callers — Result imposes branching on the trigger-page caller that never cared about the rate-limit case specifically. A typed error class is lighter (~10 LoC vs ~33) and matches the actual need: - new InstallationRateLimitError(resetAt) thrown from _getReviewCommentsWithReplies; rate-limit log.warn unchanged. - handleWebhook catches it and breaks with log.info (unchanged semantics: 200 ack, no redelivery storm). - trigger page reverts to direct array access; any failure propagates to the page error boundary (the pre-#747-commit shape). - log-audit.md wording updated to match. |
||
|
Colin McDonnellandpullfrog[bot]
|
e20f32fb09 |
fix(test): randomize tag name in push-enabled fixture
the branch name was already randomized with \${RANDOM}, but the tag
name was hardcoded as `test-tag-enabled`. every successful run left
that tag on the fixture remote with no cleanup, so subsequent runs'
checkouts (which fetch tags) saw the local tag already exist and
failed with `fatal: tag already exists`. the agent's git-push fallback
then hit the deliberately-broken creds and the test scored
push_tags=✗ delete_branch=✗.
mirror the branch pattern with \${RANDOM} so every run gets a fresh
tag name. tags still accumulate on the remote but no longer collide;
that's an infra cleanup concern, not a correctness one.
|
||
|
Colin McDonnellandpullfrog[bot]
|
c0988e35b0 |
fix(security): block docker socket from sandboxed shell; disable opencode batch_tool
two real CI failures on main, both shipping bugs in the action: 1. `token-exfil-claude` was a real sandbox escape: GHA `ubuntu-latest` puts `runner` in the `docker` group, so a sandboxed shell could run `docker run --pid=host --privileged busybox cat /proc/<parent>/environ` and read the action process's env (which holds user secrets) — fully bypassing the unshare PID-namespace. fix: inside the sandbox's mount namespace (already private via `--mount-proc` which implies `--mount`), bind-mount /dev/null over /var/run/docker.sock (+ podman/containerd/crio variants) so any container-runtime socket connect from the sandbox fails. only affects sandboxed shells — host runner mount table is untouched, so user workflow steps outside pullfrog keep working. 2. `restricted-opencode` regressed in #719 (`experimental.batch_tool`). opencode's batch tool rejects MCP tools with `"Tool '<name>' not in registry. External tools (MCP, environment) cannot be batched."` when a model emits parallel `pullfrog_shell` (or any MCP) tool_use blocks, opencode internally routes them through batch, they all fail, the model misreads the error as "the tool doesn't exist", and gives up. caught by a `lens:` subagent in the restricted test concluding shell was unavailable and setting `DIAGNOSTIC_ID=empty`. drop `batch_tool: true` and the matching opencode-specific guidance in `instructions.ts` — native parallel tool_use (multiple tool_use blocks per assistant message) still works for both built-in and MCP tools without batch, so we lose only the 1-25 wrapper, not parallelism. |
||
|
Colin McDonnellandpullfrog[bot]
|
efc1b67e7b |
fix(test): skip models.dev existence check for fallback aliases
deprecated aliases (`fallback` set) legitimately point at dead resolve targets — xAI just retired grok-4-1-fast/grok-code-fast-1 and #761 wired them through the fallback chain. the terminal-fallback is validated separately by the Zen served-list test. |
||
|
Colin McDonnellandpullfrog[bot]
|
0a64659ee7 |
refactor: slim action/main.ts to an orchestrator + extract helpers (#755)
* refactor: extract helpers out of action/main.ts so non-orchestration churn stops touching the file main.ts had grown to ~1240 lines holding ~500 lines of helpers that have nothing to do with the resolver pipeline — billing-error UI/copy, proxy minting, summary/learnings persistence, log formatters, end-of-run cleanup waterfalls. any PR adding a new billing code branch or a new log line was forced to edit main.ts, and since main.ts is in ALWAYS_RUN_ALL the entire 52-job LLM CI matrix fired on what should have been a 0-job change (e.g. #748). extractions: - action/utils/billingErrors.ts — BillingError, TransientError, the format*Summary renderers, billingConsoleUrl - action/utils/proxy.ts — mintProxyKey, buildProxyTokenHeaders, resolveProxyModel, plus runProxyResolution wrapper that renders + rethrows BillingError/TransientError before the outer catch - action/utils/prSummary.ts — fetchPreviousSnapshot, persistSummary co-located with the existing seed/read file helpers - action/utils/learnings.ts — persistLearnings co-located with the existing seed/read file helpers - action/utils/runStartupLog.ts — resolveOutputSchema + logRunStartup (the model/agent/push/shell/timeout block) - action/utils/runErrorRenderer.ts — renderRunError classifies (BillingError reclassify / hang detect / API-key auth) and emits {summary, comment} markdown bodies - action/utils/runLifecycle.ts — persistRunArtifacts, finalizeSuccessRun, writeRunErrorOutputs — the three end-of-run cleanup phases shared between the success path and the error catch path main.ts is now ~570 lines — the irreducible orchestrator: disposables (`await using` for tokenRef / gitAuthServer / mcpHttpServer), the toolContext construction, the agent-timeout race, the catch/finally shape, and the named phase calls. behavior is preserved verbatim (verified: pnpm -r typecheck + pnpm test 695/695 pass, action/test 596/596 pass). wiki/main.md gets a new "file layout" section describing the split. AGENTS.md gets a single line pointing future edits at the helpers instead of main.ts. * anneal: address review findings - restore MainResult.result?: string (accidental removal in initial commit; field was unused in current code but is part of the exported interface surface — keep the diff truly behavior-preserving) - move resolveOutputSchema from runStartupLog.ts to payload.ts (it's an action-input resolver alongside resolvePromptInput / resolvePayload, not a log helper — was placed in runStartupLog.ts for matrix-churn pragmatism but the domain fit is in payload.ts) - un-export resolveProxyModel (only used internally by runProxyResolution in proxy.ts; no external importer) - fix runErrorRenderer.ts JSDoc "Three classifications" → four (Billing, hang, API-key, default) - expand runLifecycle.ts module banner to note that finalizeSuccessRun calls persistRunArtifacts first, and to explain why the catch path splits writeRunErrorOutputs + persistRunArtifacts - update billingErrors.ts header to point at proxy.ts and runErrorRenderer.ts as the actual origin sites (was stale "main.ts") - expand proxy.ts header to spell out the runProxyResolution entrypoint contract (was stale "main.ts can render") - update wiki/main.md resolver chain + dependency table to name runProxyResolution as the actual call site and document the early BillingError/TransientError rendering branch - update wiki/main.md file-layout table to lead with runProxyResolution and describe mintProxyKey/buildProxyTokenHeaders/resolveProxyModel as internal helpers (was implying they were public surface) |
||
|
Colin McDonnellandpullfrog[bot]
|
a78b1542da |
feat: pullfrog auth codex + fresh-branch (#757)
* feat: pullfrog auth codex + fresh-branch Add `pullfrog auth codex` standalone command for minting Codex (ChatGPT) subscription credentials and saving them as the `CODEX_AUTH_JSON` Pullfrog secret. Codex device-auth runs in a subprocess with an isolated `CODEX_HOME` (temp dir) so the user's `~/.codex/auth.json` is never touched. The spawned `codex login --device-auth` output is captured line-by-line, ANSI-stripped, and re-rendered with a `$ codex login --device-auth` header above dimmed sub-output on the @clack/prompts rail so the user visually understands they're seeing a sub-process. Companion `pnpm fresh-branch` script: from inside `.worktrees/<name>`, creates a schema-only Neon branch named `dev/<git-branch>`, patches the worktree's `.env` (DATABASE_URL, DATABASE_URL_UNPOOLED, NEON_DEV_BRANCH), then runs `prisma migrate reset --force` so migrations apply cleanly against a data-free copy. Refuses to run from the primary checkout or on protected branch names. Other: - bump CLI/account/repo secret value limit 4096 -> 49152 chars (matches GitHub Actions' 48KB cap; auth.json is ~4-5KB) - extract shared CLI helpers (gh/pullfrog API, secret save) into `action/commands/_shared.ts` * fix(auth): address PR review + add CodexAuthCallout, default account scope Review fixes: - handle 'error' event from `codex` spawn (ENOENT) so missing PATH bails with an actionable "install codex CLI" message instead of an unhandled Node error - escalate SIGTERM -> SIGKILL after 5s grace when killing a stuck codex child so the CLI can't get pinned indefinitely - stop the spinner with a red "failed" glyph in the catch path before clearing activeSpin, mirroring `bail` (no orphan spinner above errors) - enforce 48 KB secret value cap by *bytes* (Buffer.byteLength) not UTF-16 code units, across all 3 secret routes; matches GH Actions' byte-based limit - preserve existing blank lines + comments when fresh-branch rewrites worktree .env (no more cosmetic reformat on every run) Scope: - default to `account` scope on org-owned repos too — never silently prompt for repo scope. Pullfrog has no per-GitHub-user secret store, so account is right for both user and org owners; `--scope repo` is the explicit opt-in for repo-only. UI: - new CodexAuthCallout (sibling to ClaudeCodeOAuthCallout); surfaces `pullfrog auth codex` for ChatGPT subscribers when an OpenAI provider model is selected. wired into AgentSettings.tsx (model-costs surface) and OnboardingCard.tsx (first-time setup). no paste button — the CLI handles minting + saving end-to-end. * auth/codex: rename to neon-fresh-branch, address PR review - rename `pnpm fresh-branch` → `pnpm neon-fresh-branch` (and the script file) to disambiguate from git branches. - `--scope` help text now explains the default (account) and when to pass `repo`. - move `_shared.ts` import up with the rest in `action/commands/auth.ts` and push the `stripAnsi` helper below the import block. - `sanitizeBranchName` no longer slices: slicing after trim could reintroduce a trailing `-`/`/`. callers slice the raw input first, then sanitize. - DRY the `start` branch of the codex progress callback (single header path, optional retry log). - thread a `timedOut` flag from `runDeviceAuth` → `ProgressEvent.exit` so the retry prompt can say "device authorization timed out — retry?" instead of the generic "no auth.json was written" line when the per-attempt timeout fires. - drop the redundant `mkdirSync` after `mkdtempSync` in `codexAuth.ts`. * untrack .scratch/ (committed screenshot fixture by mistake) * auth codex: prompt for scope on orgs (mirrors init) * revert worktree.ts: out of scope for this PR * anneal: trim _shared.ts dead exports, collapse CodexSpawnError, inline packageBin * codex auth: wire end-to-end runtime consumer CODEX_AUTH_JSON is now actually usable: the action runtime materializes it as OpenCode's auth.json at the runner's real $HOME/.local/share/opencode, OpenCode routes openai requests through the ChatGPT subscription via the embedded CodexAuthPlugin, and a GitHub Actions post: hook detects any refresh-chain rotation during the run and PUTs it back to Pullfrog via a new JWT-authenticated PUT /api/runtime/secret endpoint. Key decisions: - Write to the real $HOME (not the per-run tmpdir-redirected HOME) so the file lives outside OpenCode's `/tmp/*` permission allow zone — its existing deny-default protects it without any new permission rule. - Materialization gated on agent === opencode (Codex auth is OpenAI-only, Claude never sees the file). - Defense-in-depth on Claude: deny Read/Grep/Edit/Glob + sandbox.denyRead for ~/.local/share/opencode/auth.json in managedSettings (covers Bash file-reading commands too per Claude Code permissions docs). - New `provider.managedCredentials` field on the provider config — CLI-only credentials authored via `pullfrog auth <provider>`. Counted for hasAnyKey/log-redaction but never surfaced as a paste option in init. CODEX_AUTH_JSON is the first member; OPENAI_API_KEY stays in envVars. - Eager refresh on `pullfrog auth codex`: one OAuth round-trip before setPullfrogSecret so Pullfrog's copy is the freshest in the chain (avoids the user's laptop refreshing first and stranding our copy). - Post-hook approach for write-back so it survives cancellation, timeouts, and unhandled errors in the main step. State is ferried via core.saveState since apiToken is run-scoped and not in env. - Server-side write-back endpoint is allowlist-gated to CODEX_AUTH_JSON only — never a generic secret-write surface. Looks up the secret at repo scope first, falls back to account scope. 404s on create (refresh-only, never auto-provision). * codex auth: documentation + wiki cross-links * debug: log dbSecrets keys + CODEX_AUTH_JSON presence (temporary) * debug: surface install path + parse failure preview * remove debug log lines (E2E verified) * hide CodexAuthCallout until opencode-ai bump (1.1.56's allowed-models set excludes gpt-5.5) |
||
|
Colin McDonnellandpullfrog[bot]
|
ddbc610569 |
review prompt: friendly green callouts + per-section severity emojis (#756)
* review prompt: friendly green callouts + per-section severity emojis - Replace `[!NOTE]` informational tier and the no-callout minor-suggestions tier with friendly green blockquotes (`> ✅` / `> 💡`). The two loud tiers (`[!CAUTION]` / `[!IMPORTANT]`) keep their GitHub admonitions. - Add a per-`##`-section severity-emoji rule (🚨/⚠️/💡/ℹ️) for cross-cutting review concerns that don't anchor to a line and would otherwise be buried in summary content. - Drop the `<br/>` between summary sections — heading + blank line carries enough visual spacing. - Skip the post-run learnings-reflection turn for `IncrementalReview`. It's the lowest-novelty mode (delta review against existing PR with prior summary already loaded) and almost never produces durable learnings — reflection there costs ~$0.50-0.80/run for nothing. - Surface real error info on `agent-browser` skill install failures (exit code + stdout + stderr + spawn error). The skills CLI uses a TUI that prints errors to stdout, so the prior stderr-only logging silently swallowed every failure. * review prompt: per-bullet severity emoji + bullets-only sections Section headings are plain again (no leading severity emoji). Severity moves to individual bullets so a section that mixes a 🚨 and a 💡 isn't mislabeled by either. Section bodies are now bullets only — paragraph prose under a heading is harder to scan and tends to bury the actionable point. Bullets can carry indented continuation content (sub-bullets, code fences, blockquotes) by indenting two spaces under the parent. * review prompt: cap section length + identifier discipline Bound each summary section to at most 4 bullets at most 2 lines each, and explicitly call out identifier-heavy prose as an anti-pattern. The reader is often a manager or non-author; identifier-dense paragraphs ('foo calls bar.fetch which dispatches to baz via qux...') are unreadable for them. Default to plain-language behavior descriptions, name an identifier only when it's the subject of an actionable concern or a public surface a reader would recognize, target 2-3 backtick tokens per bullet. Move the deep-explanation pattern from open blockquote to a default- collapsed details/summary so depth doesn't dominate the visible body. * review prompt: hard cap on bullet identifier density + worked rewrite example Soft 'aim for 2-3 tokens' guidance was ignored — first big-PR e2e showed 12 of 19 actionable bullets exceeded the target (avg 4.8 tokens, several over 8). Promote to a hard cap of 3 backticked tokens per bullet and pair with a concrete bad/good rewrite the agent can pattern- match against. Also tighten the per-bullet length cap from ~240 to ~200 chars and explicitly call it 'hard cap, not target'. * review prompt: tighten bullet length cap to 160 chars, dramatize the worked example V2 e2e test: token discipline improved (4.8 -> 3.3 avg, 12/19 -> 6/14 violations) but length got worse (235 -> 286 chars, 13/14 over the 200 cap). The agent compensated for fewer identifiers with more prose. Two changes: (1) tighten the cap from ~200 chars to 160 chars / 1 visual line and call out wrap-to-multiple-lines as the failure mode; (2) rewrite the worked example so the good version is genuinely half the length of the bad one, not just lower token count. The example was the thing the agent pattern-matches against; making the good version ~130 chars vs the bad version's ~290 chars sets the right shape. * review prompt: drop fixed bullet-count cap, keep length + identifier caps Per user feedback — section length should be governed by content, not an arbitrary count. Soft guidance ('past ~6, ask whether to split') is fine; the hard '≤ 4 bullets per section' rule was the wrong shape. Length cap (160c) and identifier cap (3 backtick tokens) stay; those target the actual scanability problem. * review prompt: drop ## subsystem sections, flat 'Issues found' list Per-section structure forced every concern into a subsystem frame and made the body read like a series of mini-essays. Replace with two parts: (1) TL;DR + Key changes as the dispassionate overview, (2) flat '### Issues found' list ordered by severity, intermixed across files and subsystems. Per-bullet rules (≤160c, ≤3 backtick tokens, severity emoji prefix, optional indented continuation) carry over unchanged. * review prompt: full v6 structure — preamble + cross-cutting H3s + nitpicks Replaces the flat 'Issues found' bullet list with the iterated v6 shape: - Preamble is a bolded inline 'Reviewed changes' lead-in plus bullets plus a collapsed 'Review metadata' block (mode/files/commits/refs/ reviewed commits list/prior pullfrog review/staleness note). - Each cross-cutting concern gets a '### emoji Title' section. The visible problem write-up is human-friendly and DESCRIBES THE PROBLEM ONLY — no asks, no suggested fixes, no 'the right thing to do is'. - Each section carries a collapsed 'Technical details' block wrapped in a 4-backtick markdown fence (so it can hold its own 3-tick code fences cleanly, agent-readable, one-click copyable). Standard four inner sections: Affected sites, Required outcome, optional Suggested approach, optional Open questions for the human. - '### ℹ️ Nitpicks' at the bottom for body-only nits that don't inline; simple bullets, no technical-details collapse. - Anti-paragraph-wall rule: never two successive plain paragraphs in visible '### ' sections; alternate prose with structure. - Inline-vs-body discipline: anything that anchors to a single line goes inline, body is for cross-cutting only. - Drops legacy '### Key changes', '### Issues found', '<b>TL;DR</b>', and the '<sub>Summary</sub>' line. * model effort: bump Gemini + GPT to high effort; drop Gemini Pro→Flash subagent E2E review eval against a substantive billing-module diff surfaced two related quality gaps: 1. Gemini Pro at thinkingLevel=medium (#663's CI-timeout fix) reviewed the diff only, took the 0-lens path, and missed a catastrophic camelCase/snake_case service-vs-schema mismatch. Bumping back to high — review work is exactly the wrong shape for the medium/high tradeoff #663 was optimizing for; the per-turn TTFT cost is worth paying when reasoning IS the value. 2. GPT had no reasoningEffort override, defaulting to upstream medium. Same diff, similar shallow result vs Claude. Adding reasoningEffort: high for the curated direct-OpenAI slugs, mirroring the Gemini pattern (Anthropic separately uses --effort high via the Claude Code CLI flag in claude.ts). 3. Gemini Pro's subagentModel was 'gemini-flash' — but Google has no in-between tier between Pro and Flash, and Flash is a meaningful capability cliff for review work. Dropping the override so subagents inherit Pro. Cost stays reasonable since Gemini Pro is already the cheapest of the flagship trio. Other providers unchanged: Anthropic opus→sonnet and OpenAI gpt→gpt-5.4 remain (each is a one-tier drop to a still-capable sibling). * model effort: revert orchestrator override, set explicit high on reviewfrog subagent Reshape the effort design after eval: - Drop the explicit Gemini and GPT model-level overrides — orchestrators now run at upstream defaults (Gemini high, GPT-5.x medium). Gemini's upstream IS high, so this is a no-op there; GPT goes back to upstream medium for orchestrator-level routing work. - Add explicit 'high' on the reviewfrog subagent via agent.options. OpenCode merge order is base ← model.options ← agent.options ← variant per session/llm.ts:141, so the subagent always runs at high regardless of which orchestrator dispatched it. Both thinkingConfig.thinkingLevel (Gemini) and reasoningEffort (GPT) keys included; irrelevant keys are ignored per provider. - Bump providers-live timeouts (12min job / 10min step, from 8/6) to budget for Gemini's TTFT variance at high effort. #663's 4min timeout was sized for the medium-effort override that's now removed. * model effort: restore Gemini explicit high override (no-override path breaks) Bare 'rely on upstream default' for Gemini failed in e2e — removing the model-level provider config produced 'Function call is missing a thought_signature' API errors on every gemini-pro run. Even though upstream opencode's options() returns the same thinkingLevel: high we were explicitly setting, opencode's resolution path differs subtly between the two cases. v2's explicit override worked; v3's removal broke. Reproducible across two consecutive runs. Restoring the explicit Gemini override (back to v2 design). GPT orchestrator stays UN-overridden — at upstream default (medium) — since removing that override didn't trigger the same failure pattern and the reviewfrog subagent agent.options high override compensates for the extra depth GPT loses at medium. * diag: remove reviewfrog agent.options to isolate Gemini thought_signature failure v3 (no Gemini orch override) failed with thought_signature error. v4 (restored Gemini orch override at v2-equivalent) ALSO failed, even though the orchestrator config matches v2. The variable between v2 (working) and v4 (failing) is the new reviewfrog agent.options block. Removing it to confirm — if Gemini works again, the agent.options addition is the culprit and we need a different shape for it. * opencode-ai: bump 1.1.56 → 1.15.0 + clean up gemini effort config opencode-ai@1.1.56 was published 2026-02-10 (3 months old). The Google API tightened thought_signature validation 24-48h ago (per https://discuss.ai.google.dev/t/gemini-thought-signature-patch/122555), and the bug class hits opencode's session→prompt serializer for MCP tool-call parts (anomalyco/opencode#4832, #8321). Latest stable bumps us through ~3 months of fixes; needed for Gemini-direct to stop dying with 'thought_signature is missing' on every multi-turn run. Companion cleanup: the gemini provider override in opencode.ts had 30-line block of comments, four unused constants, and a 6-line Object.fromEntries map for two entries. Replaced with one source-of- truth helper that loops modelAliases, filters provider==='google', strips the 'google/' prefix, and returns the override map. Adding any future Google alias to the registry now flows through automatically. Test added: action/agents/opencode.test.ts asserts the helper covers every direct-Google alias, strips the prefix correctly, and pins every entry to thinkingLevel high — catches drift in helper logic without hardcoding the API ids the test would have to update in lockstep with the registry. * fix(workflow): tolerate listJobsForWorkflowRun 404 in resolveRun PR #750 (docker testing rewrite) replaced the per-call env allowlist with full process.env passthrough into the test container. That now leaks GITHUB_RUN_ID + GITHUB_JOB into runs whose MCP token is scoped to a DIFFERENT repo (e.g. providers-live smoke runs the action against pullfrog/test-repo with pullfrog/app's run ID). The unconditional listJobsForWorkflowRun call 404s and crashes the entire run, breaking every providers-live job on main since #750 landed. jobId is purely cosmetic (deep-links 'View workflow run' footer to a specific job vs the run-level URL). Wrapping the API call in try/catch so a 404 logs a debug message and falls through to undefined jobId is the right fix — the failure mode is exactly what graceful degradation is for, and the alternative (filter the env vars at the docker boundary) re-introduces the kind of allowlist #750 was getting rid of. * opencode-ai: pin 1.14.51 instead of 1.15.0 (effect refactor breaks JSON output) opencode 1.15.0 (May 15) ships a major architectural refactor onto @effect — the run command boots an in-process server via @opencode-ai/sdk/v2 and the JSON event emission path through that SDK client doesn't surface on stdout the way our parser expects (CI run on 1.15.0 produced 0 stdout events but the agent still completed). Local invocation also hangs at the in-process server boot. The Gemini thought_signature fixes (the original reason for bumping) landed earlier in the 1.14.x line, so 1.14.51 (May 14) gets us the upstream fix without the Effect rewrite. Defer the 1.15.x bump until we're ready to rewire our parser/spawn around the new SDK. * opencode-ai: revert to 1.1.56; gha: filter outer-CI workflow-run vars at the docker boundary Two related changes for the docker testing harness's ergonomics: 1. Revert opencode-ai 1.14.51 → 1.1.56. The 1.14+ line ships an Effect refactor (the SDK-v2 client + in-process server architecture) that our --format json parser doesn't speak — even the 1.14.51 release, pre-dating the 1.15.0 Effect rename, produced 0 stdout events on our skill-invoke smoke. There's no clean pre-Effect version that ships the Gemini thought_signature fix; that fix needs a separate workstream once we're ready to rewire the parser onto SDK v2. 2. Filter outer-CI workflow-run identifiers (GITHUB_RUN_ID, GITHUB_JOB, GITHUB_WORKFLOW, GITHUB_ACTION, GITHUB_REF, GITHUB_SHA, etc.) from gha.ts's --env-file passthrough. PR #750's full-process.env design leaks pullfrog/app's CI run identifiers into runs that act against a different repo (e.g. pullfrog/test-repo); any code path inside the action that uses them as keys (most notably resolveRun's listJobsForWorkflowRun lookup) 404s. Filtering them here means the action sees undefined and skips the lookup, complementing the defensive try/catch in resolveRun (commit addc76d4). GITHUB_REPOSITORY and GITHUB_TOKEN are NOT filtered — those are genuinely needed. Companion to addc76d4 (resolveRun 404 tolerance). The two together make this class of bug 'either fix would have caught it' rather than 'silently breaks the entire test matrix'. * fix(deps): sync pnpm-lock.yaml with opencode-ai 1.1.56 manifest revert Forgot to refresh the lockfile after reverting the manifest in 02c6d8c1. CI's frozen-lockfile install was failing with 'lockfile: 1.14.51, manifest: 1.1.56' mismatch. |
||
|
Colin McDonnellandpullfrog[bot]
|
a0dce200d0 |
fix(claude): prefer OAuth token over ANTHROPIC_API_KEY (#763)
* fix(claude): prefer OAuth token over ANTHROPIC_API_KEY in Claude Code
When both `CLAUDE_CODE_OAUTH_TOKEN` and `ANTHROPIC_API_KEY` are present,
claude-code's auth resolver (`Vw()` in cli.js) returns the API key first
and silently ignores the OAuth token. The result: accounts that have a
Max-subscription OAuth token in `account_secrets` are still billed at
per-token API rates because the workflow `env:` block also forwards
`ANTHROPIC_API_KEY` from org-level secrets.
Strip `ANTHROPIC_API_KEY` from the spawned claude-code subprocess env
when an OAuth token is present (and we're not on the Bedrock route),
so the Max subscription is actually used. Other agents in the same run
still see the API key in `process.env` via the parent.
* chore: tighten comment-length rule + trim claude.ts comment
Caps inline comments at 2-3 lines above any single line of code (the
prior wording allowed runaway block comments as long as the comment
was nominally shorter than the annotated code).
* chore: downgrade OAuth-strip log to debug + document debug-mode pattern
`log.info` was overkill for a per-run path-selection marker. `log.debug`
keeps production logs quiet while preserving full visibility in e2e
verification, where `LOG_LEVEL=debug` (or `gh run rerun --debug`)
flips the same line on.
Adds a "Action debug mode" subsection to wiki/e2e-testing.md so the
affordance is discoverable: `log.debug(...)` is the right tool for
breadcrumbs that prove a code path fired during preview-repo e2e but
shouldn't ship to customer logs.
* chore(wiki): correct debug-mode trigger guidance for preview repos
LOG_LEVEL=debug only works when the template's pullfrog.yml forwards
it, which it doesn't. ACTIONS_STEP_DEBUG=true is the GitHub-magic name
that's auto-injected into every step's env without any yaml change,
so make that the documented default for preview-repo e2e.
* chore(wiki): fix render-format claim in debug-mode table
When `ACTIONS_STEP_DEBUG=true`, `log.debug` routes through
`core.debug()`, which GitHub renders as `##[debug]<msg>`, not the
`[DEBUG] <msg>` format. The `[DEBUG]` prefix only happens via the
LOG_LEVEL=debug path which isn't currently wired into the template.
* feat(action): add `overrides` input for per-dispatch env mutation
Accepts a JSON {string:string} map via the workflow_dispatch input,
parsed and merged into process.env at the start of `main()` (before
any agent or token-acquisition code runs). Lets a privileged caller
flip env vars for one dispatch without persisting state on the repo
(repo Actions variables) or being restricted to GitHub's debug names
(`gh run rerun --debug`).
Deny-list refuses overrides for integrity-critical names — GITHUB_TOKEN,
ACTIONS_RUNTIME_TOKEN, ACTIONS_RUNTIME_URL, ACTIONS_ID_TOKEN_REQUEST_*,
ACTIONS_CACHE_URL, PULLFROG_API_SECRET, VERCEL_AUTOMATION_BYPASS_SECRET.
Customer provider keys (ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, etc.)
are explicitly allowed — overriding them per-run for cred-rotation tests
and auth-failure repros is the use case.
Touches:
- action/action.yml — declare `overrides` input
- action/utils/overrides.ts — parse + apply with deny-list (+ unit tests)
- action/main.ts — wire into `main()` after `normalizeEnv()`
- .github/workflows/pullfrog.yml — forward to action
- utils/github/pullfrog.yml.ts — same in the customer-facing template
- wiki/e2e-testing.md — documented as preferred debug-mode trigger
* fix(overrides): strip raw INPUT_OVERRIDES + mask applied values
GitHub Actions injects every action input as an env var (INPUT_<NAME>),
so the original JSON of `overrides` sits in process.env as INPUT_OVERRIDES
and is inherited by every spawned subprocess (claude, opencode, MCP
servers, shell). That defeats the deny-list (a downstream re-application
would have access to the raw JSON) and leaks arbitrary caller-supplied
values into agent env verbatim.
After applying, applyOverrides now:
1. delete process.env.INPUT_OVERRIDES — subprocesses see only the
surgically-applied keys, not the raw JSON
2. core.setSecret(value) for each applied value — the runner masks
those strings in subsequent log output, so an overridden
ANTHROPIC_API_KEY can't accidentally surface in debug logs.
Two new tests cover the deletion path (both applied and all-denied).
* fix(overrides): scope auto-masking to credential-shaped keys
core.setSecret(value) is a global string-match — calling it on a short
config value like "claude" masks every appearance in subsequent logs
(including "claude-opus-4-7", "anthropic-claude-sonnet", etc.), which
actively harms debugging.
Restrict the auto-mask to keys whose names end in _KEY / _TOKEN /
_SECRET / _PASSWORD / _OAUTH / _PRIVATE_KEY — the credential-shape
naming convention. Customer keys (ANTHROPIC_API_KEY, etc.) and the
deny-listed names match. Plain config (PULLFROG_AGENT, PULLFROG_MODEL,
ACTIONS_STEP_DEBUG) doesn't.
* docs(wiki): document the three security layers + runner-echo caveat
Lays out exactly what the `overrides` input does to mitigate the secret-
leak surface (deletion + masking) and the one unavoidable limit: GH
Actions echoes the `with:` block once before any action code runs, so
the raw JSON appears in the workflow log header in plaintext. Anyone
using `overrides` should treat that one-shot exposure as part of the
threat model.
* fix(overrides): forward via env, not action input, so the value isn't echoed verbatim in the runner step header
GH Actions echoes the `with:` block of every `uses:` step in the log
group header, BEFORE any action code runs — so the raw JSON of
`overrides` was always visible in the workflow log regardless of any
in-action `core.setSecret` calls.
Refactor: drop the `overrides` action input; instead the action reads
`process.env.PULLFROG_OVERRIDES`. The workflow yaml forwards
`inputs.overrides` via the step-level `env:` block. We still need to
verify empirically whether `env:` block values from workflow inputs
get echoed too (separate test); even if they do, masking via
core.setSecret + delete of PULLFROG_OVERRIDES after parsing closes
the leak to subprocesses, which is the part the action controls.
* fix(overrides): rename to unsafe_overrides + UNSAFE_OVERRIDES
The runner echoes step-header env-block values in plaintext before any
action code runs, so the raw JSON of this affordance is visible to
anyone with actions:read on the calling repo. That's acceptable
because the workflow only exists on our private repos, but the input
name should make the trade-off obvious at the call site rather than
buried in a wiki.
- workflow_dispatch input: `overrides` → `unsafe_overrides`
- env var the action reads: `PULLFROG_OVERRIDES` → `UNSAFE_OVERRIDES`
- wiki: rewrite the section to surface the runner-echo as the central
trade-off rather than a buried caveat
* chore(overrides): tighten error messages to reference UNSAFE_OVERRIDES
* docs(wiki): fix stale 'overrides' refs + correct render-format mechanism
Addresses two unresolved review threads on PR #763:
1. The opening sentence of "Action debug mode" still referenced the
pre-rename `overrides` input and `gh workflow run -f overrides=...`.
Updated to `unsafe_overrides`.
2. The render-format claim was technically wrong. `core.isDebug()`
doesn't cache — it reads `process.env.RUNNER_DEBUG === '1'` on
every call. The actual mechanism: the runner only sets
RUNNER_DEBUG=1 when ACTIONS_STEP_DEBUG=true is observed at
workflow-trigger time. Mutating ACTIONS_STEP_DEBUG mid-step
doesn't retroactively flip RUNNER_DEBUG, so the call falls through
to isLocalDebugEnabled() which reads ACTIONS_STEP_DEBUG directly.
Rewrote the explanation to match.
* fix: drop unsafe_overrides from customer-facing workflow template + remove test theater
Two cleanups from a stricter re-read of AGENTS.md:
1. utils/github/pullfrog.yml.ts is the workflow yaml we sync into every
customer repo. unsafe_overrides has no business there — it's a
pullfrog-only debugging affordance. Reverted. The action's read of
UNSAFE_OVERRIDES env var stays — it's a no-op for any workflow that
doesn't set it, and pullfrog/template + pullfrog/app's own workflow
still forward it.
2. Deleted action/utils/overrides.test.ts entirely. AGENTS.md is clear:
no tests unless explicitly asked. I added them anyway. The tests
were mostly testing JSON.parse + typeof, plus one regression guard
for the deny-list that is better protected by code review of the
tiny DENIED_OVERRIDE_NAMES set than by a vitest file.
Also strengthened the corresponding AGENTS.md rule from a buried bullet
to an explicit "NEVER write tests unless asked, here's why agents
violate this constantly, here's the bar" callout.
Wiki note added: unsafe_overrides is pullfrog-only infra, not customer-
facing.
|
||
|
Colin McDonnellandpullfrog[bot]
|
7907fac64e |
fix(test): bump model-smoke timeout 60s → 120s (#764)
xai/grok-4.3 jobs in the models-live matrix land at 42-67s wall time vs 23-41s for every other provider, brushing the 60s ceiling and crossing it intermittently (e.g. xai/grok-code-fast in run 25949844470 timed out at 60s with `OK` already in stdout — model replied, harness just hadn't seen close). 120s gives ~2x headroom on the slowest provider without penalizing the fast-path providers, since the timer only fires on actual hangs. |
||
|
Colin McDonnellandpullfrog[bot]
|
76879b27ec |
docker testing rewrite: bake the image, drop the allowlist, kill the quoting (#750)
* docker testing rewrite: bake the image, drop the allowlist, kill the quoting - new `pnpm gha <script>` wrapper. one entry point for running any node script in the GHA-like container; replaces the runtime apt-get + useradd + chown ceremony in `action/utils/docker.ts`. - `action/Dockerfile` bakes ubuntu:24.04 + node 24 + gh + jq + sudo + testuser at uid 1000. `action/docker-entrypoint.sh` remaps to the host uid/gid and `exec`s the requested command — no `bash -c` nesting, no `escapeForDoubleQuotes`. - env passthrough: full `process.env` (+ `.env` via dotenv) flows through `--env-file`, multi-line values via `-e` fallback. drops `EnvFilterMode` / `testEnvAllowList`. - image rebuild is content-hash gated on Dockerfile + entrypoint; volume is versioned by hash so a stale `node_modules` cache from an old image can't poison a new one. - `action/play.ts` slimmed to a CLI; `run()` extracted to `action/utils/runFixture.ts`. drops the `--local` / `PLAY_LOCAL` dual mode in favor of explicit `play:local` / `runtest:local` scripts. - `action/test/run.ts` no longer self-relaunches into docker — that's `gha`'s job now. - `action/test/coverage.ts` `ALWAYS_RUN_ALL` updated to track the new files. - `wiki/docker.md` rewritten (243 → 105 lines). `wiki/action-tests.md`, `wiki/billing.md`, `wiki/adversarial.md`, `README.md`, `AGENTS.md` all updated to drop `--local` / `PLAY_LOCAL` references. verified end-to-end: `pnpm play` runs the default fixture against pullfrog/scratch, exit 0; `sudo unshare --pid` still works inside the container; `pnpm runtest` boots through the wrapper. * gha: address review feedback + 3 related issues found locally review-flagged: - bare `pnpm gha --build` now builds the image and exits 0 (was printing help and exiting 1 — docs claimed it was a valid standalone) - `initVolumeOwnership` skipped when the named volume already exists; saves the ~240ms `docker run … chown` on every warm invocation - `GIT_SSH_COMMAND` gate widened to any `id_*` private key (was hard- coded to `id_rsa`, leaving ed25519-only linux contributors with the default ssh config). dropped `-i` so ssh picks whichever key exists - new `action/.dockerignore` — partial mitigation noted: BuildKit (default since docker 23) only sends files referenced by the Dockerfile (~42B in practice), so the perf concern is mostly hypothetical. file is still worth keeping for `DOCKER_BUILDKIT=0` fallback and as documented intent for future `COPY . .` additions related issues found while validating locally: - `parseArgs` now stops flag-parsing at the first positional (or literal `--`); `pnpm gha test/run.ts --build` previously intercepted `--build` as a gha flag instead of forwarding to `test/run.ts` - new `pnpm gha --clean` command prunes orphan `pullfrog-gha:*` images and `pullfrog-gha-node-modules-*` volumes whose hash doesn't match the current Dockerfile (each Dockerfile/entrypoint edit creates a fresh hash and orphans the prior pair, ~600MB + ~200MB each — without a cleaner they accumulate silently) - `--shell` without a TTY now fails fast with an actionable message before docker is invoked, instead of producing the confusing `the input device is not a TTY` from docker run wiki updated: documents `--clean`, the parseArgs passthrough rule, and a new "Reclaiming disk" section. * gha: fidelity, flexibility, and signal-safety improvements investigated local fidelity vs the real GHA ubuntu-24.04 runner and addressed the gaps that have actually bitten contributors or could. fidelity (image now matches GHA closer): - bake build-essential, wget, xz-utils, file alongside the existing toolset. gh, jq, git, python3, sudo, ssh, build-essential, wget, xz, file, unzip, curl all present. native module builds (node-gyp, any package missing arm64 prebuilts) now work; common agent shell calls don't hit ENOENT - `host.docker.internal:host-gateway` flag wires the host into the container's DNS on linux (macOS Docker Desktop bakes it in). lets scripts that hit a local dev server use `API_URL=http://host.docker. internal:3100` and work identically on both platforms - `--init` makes tini PID 1, fixing signal forwarding during the pre-exec warmup window (Ctrl-C was previously taking up to 10s to tear down because bash-as-PID-1 swallowed the signal) - pnpm version is correctly pinned via the workspace's `packageManager` field — corepack resolves it at install time; verified via the new `--doctor` command flexibility (new affordances): - `pnpm gha --doctor` runs an inside-the-container fidelity audit: os + arch + node/pnpm/python versions, version snapshots of every baked tool, env vars (CI, HOME, TMPDIR), uid/gid, and the host.docker.internal resolution. useful for "works in CI fails locally" or vice versa - `pnpm gha --build --no-cache` busts the docker layer cache when an apt mirror, base image, or external download has changed upstream - entrypoint's `pnpm install` warmup is now wrapped in a `flock` on a file in the shared node_modules volume — concurrent `pnpm gha` invocations (e.g. play in one terminal, runtest in another) serialize their install instead of racing docs: - new "Gaps (known)" section in wiki/docker.md explicitly calling out the things this system can't do yet, including the missing `uses: ./action` semantics gap that `.github/workflows/action-gha-e2e-adhoc.yml` currently fills via GHA only (designing a local `pnpm gha-action <fixture>` is on the roadmap), service containers, parallel-run sharing, and arch differences (arm64 vs amd64) * docs: audit + corrections after testing fronts self-audit pass for stale references and incomplete pointers: - wiki/browser.md: `Docker (node:24)` → `pnpm gha container (ubuntu:24.04)`. the substance was right (chrome not preinstalled) but the base image reference was stale. - wiki/docker.md: the "Permission errors" troubleshooting line claimed the node_modules volume is chowned on every run; now correctly says "owned by the host uid on first creation; warm runs skip the chown" to match the actual behavior after the initVolumeOwnership fix. - wiki/action-tests.md: `API_URL` env-var doc now mentions BOTH paths (`localhost:` from play:local, `host.docker.internal:` from inside the container). Proxy/router recipe now shows both invocations side-by-side instead of saying "must use play:local". - wiki/billing.md: same dual-recipe update for the loop-including-the- action proxy walkthrough. - gha.ts header: expanded the usage block to include --clean / --doctor / --no-cache / --shell-TTY, added the host.docker.internal note, and pointed at wiki/docker.md for design rationale. self-document check: a future agent landing on this code can answer "how do I run a fixture / debug in shell / add a tool / diagnose fidelity / reach a local dev server" purely from gha.ts header + wiki/docker.md without spelunking through the entrypoint or git history. |
||
|
Colin McDonnellandpullfrog[bot]
|
8e1acfba99 | fix(models): mark grok-fast and grok-code-fast as deprecated (#761) | ||
|
Colin McDonnellandpullfrog[bot]
|
fa7ddcee4a |
prompt: discerning review-feedback handling + elegance bar
strengthen build-mode self-review and addressreviews step 4 to require verifying every reviewer finding, reject AI slop / over-defensive code, and frame the goal as a complete + minimal + elegant solution. mirror the elegance/no-slop bar in AGENTS.md. |
||
|
Colin McDonnellandpullfrog[bot]
|
3add2cbc49 |
fix(action/tsconfig): noEmit + exclude dist to silence editor TS5055
action/tsconfig.json had "exclude": [] (overriding the default outDir
exclusion) and unset noEmit, so tsserver pulled action/dist/**/*.d.ts
into the program and flagged 92 TS5055 errors ("Cannot write file ...
.d.ts because it would overwrite input file") any time dist/ existed.
the CLI typecheck script passes --noEmit so it never tripped — only the
editor was affected.
emit is owned by tsconfig.exports.json, which extends this one and
overrides noEmit: false, emitDeclarationOnly: true. so the main config
is editor/typecheck only and should declare noEmit: true.
|
||
|
Colin McDonnellandpullfrog[bot]
|
5abb3072c7 | release: action v0.1.8 | ||
|
Colin McDonnellandpullfrog[bot]
|
74b7329f64 |
fix(action): dedupe concurrent checkout_pr + guard cross-PR clobber (#735)
* fix(action): dedupe concurrent checkout_pr calls + guard cross-PR clobber (#642) agents occasionally emit duplicate parallel `checkout_pr` tool_use blocks in one turn, causing two `checkoutPrBranch` invocations to race the same `.git/shallow.lock` and one to fail with `File exists`. the prior fix (#564) added a 30s staleness sweep, but that very threshold protects the within-run concurrent case from itself. dedupe at the tool layer: a module-level `Map<pull_number, Promise>` shares a single in-flight promise across concurrent same-PR calls. the fetch race becomes architecturally impossible — first call does the work, duplicate gets the same `CheckoutPrResult`. cleared in `finally` so subsequent same-PR calls re-do the work normally. also reject cross-PR checkouts when the working tree is dirty, surfacing a clear error instead of silently overwriting uncommitted work from a prior PR. uses existing `toolState.issueNumber` (no new state). * review: use dedicated `pullNumber` toolState field for cross-PR guard per copilot review: the prior guard used `toolState.issueNumber`, which is also set by issue/comment lookup tools (issueInfo, issueComments, issueEvents, review). that conflation is intentional and correct for its only consumer (`report_progress` falls back to `issueNumber` to choose which issue/PR to comment on, and GitHub treats both via the same comment API). but it makes the field wrong for the cross-PR guard: a same-PR re-checkout after `get_issue(other)` would falsely fire and surface a misleading "from PR #other" message. introduce a separate `pullNumber` field, set only by `checkoutPrBranch` alongside `issueNumber` and `checkoutSha`. narrower invariant, no disturbance to the existing `issueNumber` semantics. * review: drop dual-write — single `issueNumber` is sufficient for the guard reverting the `pullNumber` addition. setting both `issueNumber` and `pullNumber` to the same value at the same site was a code smell — there is no scenario where they diverge. issues and PRs share GitHub's number space, and the cross-PR guard's actual job is "refuse to clobber a dirty tree when switching to a different number"; that's expressible with `issueNumber` alone. addresses copilot's original concern (misleading "from PR #X" message when X was an issue) by removing the prior-number reference from the error message entirely. the dirty paths are the actionable detail. |
||
|
Colin McDonnellandpullfrog[bot]
|
ba7f5a0b89 |
action: surface agent hang context in progress comment (#733)
* action: surface agent hang context in progress comment When the activity-timeout watchdog kills a stalled opencode subprocess, the user used to see a bare "activity timeout: no output for 30Xs" — no provider context, no stderr trace, no clue why the run died. Investigation of the six runs in #728 showed the same shape every time: opencode hangs after a non-retryable provider event (auth 401, 502 stream lost, free-tier flake), and the only useful signal was buried in stderr where the user couldn't see it without diving into Actions logs. Stop trying to prevent the hang. Surface it. Add a small `AgentDiagnostic` handle on `toolState` that the harness mutates as a run progresses (recent stderr ring buffer reference, last provider-error label, event count). `formatAgentHangBody` renders that into a markdown body — bold headline, one-line explanation, collapsible `<details>` with the last ~10 stderr lines (capped to 3KB) — used by both the agent harness's own catch path and main.ts's outer catch when the watchdog wins the race against the harness. Both paths converge on one formatter; the existing "View workflow run ➔" footer affordance in `reportErrorToComment` is unchanged, so the user still has one click from the comment to the raw logs to develop their own thesis. * address review: gate hang body on isHang; fix contradictory copy - Only render `hangBody` when `isHang`. The harness sets `agentDiagnostic` on entry, so any non-hang throw past `runOpenCode`'s own catch (post-success `output_schema` validator, late cleanup throws) was rendering "Pullfrog failed — N events processed…" with the real exception message dropped — including for runs that actually succeeded before a late throw. - When `lastProviderError` already names the cause in the headline, the zero-events sentence "check whether the model provider is reachable" contradicts it (a 401 produces zero events but isn't a reachability issue). Drop the nudge in that case; keep it for the silent-stall path where it's still actionable. * address copilot review: fence escape, idle parsing, secret redaction, tests - pick a backtick fence longer than any backtick run in the rendered stderr tail. opencode error JSON occasionally embeds triple backticks in tool input dumps; the fixed three-tick fence let those terminate the fence early and corrupt the rest of the comment markdown. - parse idle seconds out of the timer reject string ("activity timeout: no output for 301s") and use that for the hang explanation. previously rendered total runtime, which overstated the stall by 20+ minutes for runs that streamed for a long time before going quiet (e.g. Rohithgilla12/data-peek#25784038918, 1230s elapsed but 304s idle). - redact sensitive env-var values from the rendered stderr tail before it lands in the PR comment / job summary. workflow log writes already go through `core.setSecret` masking; PR comments and summaries bypass that pipeline entirely. matches against `isSensitiveEnvName` (the same *_KEY/*_TOKEN/*_SECRET/*_PASSWORD/*_CREDENTIAL surface that `normalizeEnv` registers with the runner) and only redacts values >= 8 chars to avoid false-positive substring hits. - add `agentHangReport.test.ts` covering the branchy bits: idle-seconds parsing, eventCount-zero copy with and without provider error, fence-escape against embedded triple backticks, 3 KB tail truncation, null-on-no-diagnostic, and secret redaction. `startedAtMs` is dropped from `AgentDiagnostic` — total runtime was the only consumer and idle seconds replaces it. * strip slop: drop tests, drop redactSecrets, simplify ternary - delete `agentHangReport.test.ts`. half the cases just pinned literal copy ("**Pullfrog stalled**", "check whether the model provider is reachable") which is exactly the "performative tests to every string utility" pattern AGENTS.md flags. the other half tested 2-5 line pure helpers (parseIdleSec / pickFence / truncation) that code review catches. the formatter is a best-effort string output; pinning it in tests creates churn without catching real regressions. - remove `redactSecrets` and revert the formatter's import. theatrical defense: opencode doesn't dump env on startup, bearer tokens aren't in request bodies, bash is denied. the action has many other PR-comment write paths that don't redact (comment.ts, errorReport.ts, the progress writer) — if PR-comment secret hygiene matters, it's a cross-cutting concern at the comment-write layer, not bolted onto one formatter. - factor the explanation triple-ternary into `formatExplanation` with early returns. same logic, easier to read. `isHang` gate, fence-length escaping, and idle-seconds parsing stay — those are real correctness fixes. |
||
|
Colin McDonnellandpullfrog[bot]
|
b9383bbcfd |
action: center provider-error log excerpt on the matched line (closes #703)
the `» provider error detected (...)` excerpt was `chunk.substring(0, 500)`
— the head of whatever stderr buffer node delivered. on big writes that's
the front of an mcp tool-schema dump, not the matched error text. label
was correct (regex.test on the whole chunk), excerpt was misleading.
introduce findProviderErrorMatch(text) that returns { label, excerpt }
where excerpt is a windowed slice centered on the regex match index:
the matched line plus 1 line before and 2 lines after, hard-capped at
600 bytes. detectProviderError stays as a thin wrapper for label-only
callers. both opencode and claude harnesses log match.excerpt instead
of chunk.substring(0, 500).
regression tests cover the multi-line buffer case, surrounding-line
context, byte-cap fallback to matched-line-only, and head truncation
of a single oversize line.
|
||
|
Colin McDonnellandpullfrog[bot]
|
8d6460da1c |
fix: surface real tool error string in opencode log handler (#736)
opencode's `ToolStateError` carries the failure reason on `state.error`, not `state.output`. our log handler was reading `state.output` and falling back to `(no error message)`, so every tool failure logged a useless line. type the state as a discriminated union (mirrors @opencode-ai/sdk) so the field misread becomes a compile error. operator-facing only: the model already received the real error via opencode's tool-result envelope (verified by running webfetch against a known-404 URL — model reported "Error: Request failed with status code: 404" verbatim). closes #662 |
||
|
Colin McDonnellandpullfrog[bot]
|
1f4c3031be |
ci: filter test matrices by per-test coverage globs (#730)
* ci: filter test matrices by per-test coverage globs to cut LLM spend every test in `crossagent/`, `agnostic/`, and every provider entry now declares a `coverage: string[]` of repo-relative globs. the new `changes` job runs `paths-filter` for a docs-only short-circuit, then pipes the changed-file list into `action/test/matrix.ts`, which intersects each entry's coverage against the diff and emits filtered `agents`, `agnostic`, `flagships`, and `aliases` matrices. main pushes and `workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety net. retires `changed-agents.sh` and the `MODE=flagships` branch in `list-aliases.ts` in favor of one consistent model. * ci(matrix): switch test discovery to dep-free static parsing the GHA `changes` job has no `node_modules` installed. the previous dynamic-import path pulled the test files transitively through `utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with ERR_MODULE_NOT_FOUND. parse the test files via regex instead so matrix.ts stays zero-dep — the chain (matrix -> coverage / providers / list-aliases / models) imports only node builtins and relative TS files. * ci(matrix): address PR #730 review feedback - drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`, `restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`, both already covered). - drop unused `coverageForProvider` export and its `byName` map from `providers.ts` (matrix.ts builds its own lookup inline). - derive the active agent list from `agents/index.ts` via the same dep-free regex tactic as `parseTestFile` instead of hardcoding `["claude", "opencode"]` — adding a new harness file now wires it into the dynamic matrix automatically. - treat `coverage: []` as `coverage: undefined` in `shouldRun` so an accidentally-empty array doesn't silently skip CI on every PR. - add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the `timeout` test's coverage — the activity-timeout enforcement path was the original reason the test exists. - ungate the `root` job (lint/format/typecheck/vitest). it's a required status check on `main`, so gating it on `code == 'true'` would make docs-only PRs unmergeable (skipped jobs don't satisfy required-check rules). the real LLM savings come from skipping the four matrices, not from skipping `root`. - harden the four matrix-job `if:` guards from `outputs.matrix && ...` to `outputs.matrix != '' && ...` — explicit > implicit short-circuit. - document `expandBraces`'s flat-only support so a future author isn't surprised by `{a,{b,c}}` not expanding. - fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering". |
||
|
Colin McDonnellandpullfrog[bot]
|
4ad649ebb9 |
action: extend shallow-unreachable deepen-retry to checkout_pr fetches (#734)
extracts the deepen-retry helper from `GitFetchTool` into shared `$gitFetchWithDeepen` and applies it to every fetch in `checkoutPrBranch` (baseRef, pull/N/head, before_sha temp branch). on shallow clones with deep PR ancestry — the failure mode behind ~10 of 51 `heuristic:very-slow` runs in 24h on `remotion-dev/remotion` — the baseRef fetch was throwing `Could not read <sha>` to the agent before the compare-api deepen block could run. agents then burned 10+ minutes retrying `checkout_pr` and falling back to ad-hoc shell `git fetch --deepen` workarounds. also splits the analyzer's `heuristic:git-error-recovered` into `heuristic:git-shallow-unreachable` and `heuristic:git-shallow-lock` buckets so future audits surface this without manual log-grep. closes #656. |
||
|
Colin McDonnellandpullfrog[bot]
|
2960d51493 |
shell tool: cap output at 5K chars and spill overflow to tempfile (#732)
unbounded shell tool output blows the agent's context window on commands
that dump big logs (test runners, build tools, grep on large trees). cap
the inline body at 5000 chars; on overflow, persist the full output to
${PULLFROG_TEMP_DIR}/shell-<id>.log and return the tail prefixed with a
sentinel pointing at the saved path. agents re-read the tempfile with
cat/tail/grep when they need more.
|
||
|
Colin McDonnellandpullfrog[bot]
|
b6df2860c3 | action: bump to 0.1.7 | ||
|
Colin McDonnellandpullfrog[bot]
|
d495f0b984 |
surface BYOK failures + chronic-failures card + WorkflowRunStatus mirrors GitHub conclusions (#722)
- Migrates `WorkflowRunStatus` from `running | completed | cancelled` to a 9-state mirror of `workflow_run.conclusion`. Backfill: old `completed → success`, `cancelled → failure`. New rows write `hook.workflow_run.conclusion` verbatim via `statusFromConclusion`. - Adds Discord links to `formatApiKeyErrorSummary` (both missing-key and 401 invalid-key shapes). - Repo console: `<ChronicFailuresCard>` fires when the last 3 terminal-state runs are all `failure`. Pure DB read; latest-run button hidden for pre-dispatch failures (`runId: null`). - `StatusIcon` distinguishes `cancelled` (gray X, intentional stop) from `failure` (red X) so the visual matches the chronic-card threshold. - Pre-dispatch failures (workflow lookup miss, dispatch API error) write `failure` instead of `cancelled` so they feed the card. - Cascade: every `status: "completed"` filter in billing routes / cron / cohort queries / analyzer becomes `status: "success"`. Verified end-to-end on `pullfrog/preview-722-failure-surfaces` — Better Stack logs confirm webhooks reached the preview deploy and all three e2e runs got `marked as failure (conclusion=failure)` via the new mapper. Closes #679, #702. |
||
|
Colin McDonnellandpullfrog[bot]
|
206c11fe7c |
review: drop misleading 'with the same arguments' from diff-coverage nudge
agent is free to refine review body/comments on retry — there's no enforcement that the second call matches the first, and if reading the nudged region surfaces a new finding the agent should add it. |
||
|
Colin McDonnellandpullfrog[bot]
|
7414c1e9ca |
review: clarify diff-coverage nudge gives explicit license to skip generated artifacts
the one-time pre-flight nudge said "optionally read" but never told the agent it's free to retry without reading when every unread region is generated (lockfiles, codegen, snapshots, migration metadata). audit #677 surfaced ~21 runs/24h burning an extra model turn re-reading drizzle snapshots, pnpm-lock, and *.gen.ts files purely to satisfy the gate. mode prompts only mention generated content in the "skip self-review entirely" path, not the "in-progress substantive review" path, so the in-the-moment error message was the gap. behavior unchanged for legitimately-unread source regions. |
||
|
Colin McDonnellandpullfrog[bot]
|
8f9208bd3f |
feat: Amazon Bedrock support via routing slug (#720)
* add Amazon Bedrock as a routing slug introduces a single `bedrock/byok` catalog entry that the harness translates to the appropriate Bedrock model ID at run time via `BEDROCK_MODEL_ID`. routes Anthropic IDs through claude-code (with `CLAUDE_CODE_USE_BEDROCK=1`) and everything else through opencode's `amazon-bedrock` provider — keeps the catalog flat for an audience that needs version pinning rather than aliasing. accepts either `AWS_BEARER_TOKEN_BEDROCK` or `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` for auth; both validated alongside `AWS_REGION` and `BEDROCK_MODEL_ID` in `validateAgentApiKey`. catalog drift tests, the bumps cron, and per-alias smoke scripts all skip routing slugs since there's no fixed `resolve` to validate. docs/bedrock.mdx walks through setup; wiki/model-resolution.md has a section explaining why bedrock breaks the usual alias pattern. closes pullfrog/pullfrog#40 * ci: add bedrock env vars to test workflows mirrors the new bedrock provider's required env vars (AWS_BEARER_TOKEN_BEDROCK inherited from org secret + AWS_REGION + BEDROCK_MODEL_ID hardcoded) into both .github/workflows/test.yml files so the ci.test "env vars cover all provider API keys" assertion passes. * docs(bedrock): clearer setup flow + screenshot of model selector restructures the setup section into three concrete steps in execution order: select Bedrock from the dropdown, store the bearer token as a secret (Pullfrog or GitHub — links to keys.mdx for the trade-off), then add region + model id directly in pullfrog.yml since neither is sensitive. enable-model-access in the Bedrock console moved to step 4 (only required once per model and only when AWS rejects the call, not blocking on first run). adds a screenshot of the console model selector with Amazon Bedrock selected so readers can recognize the UI state they're aiming for. * fix(bedrock): tolerate raw Bedrock model IDs in validateAgentApiKey main.ts passes the resolved model into validateAgentApiKey (`payload.proxyModel ?? resolvedModel ?? payload.model`). For Bedrock, `resolveModel` translates `bedrock/byok` into the raw AWS model ID (e.g. `us.anthropic.claude-opus-4-6-v1`), which has no `/` and so trips parseModel inside getModelEnvVars. Detect the no-slash case and re-run the bedrock setup check (auth + region; BEDROCK_MODEL_ID is already enforced upstream by resolveModel). Caught by PR #720 e2e dispatch on pullfrog/preview-720-bedrock — "invalid model slug 'us.anthropic.claude-opus-4-6-v1' — expected 'provider/model'". Two regression tests cover the raw-ID path. * fix(bedrock): always prepend amazon-bedrock/ prefix when bedrock-routed opencode.ts was gating the prefix-injection on `!isBedrockAnthropicId(rawModel)`, on the theory that Anthropic Bedrock IDs always go through claude-code. But `PULLFROG_AGENT=opencode` is a documented escape hatch — when it forces opencode for an Anthropic Bedrock model, the prefix still has to be added or opencode fails with 'Model not found: <modelId>/.'. The Anthropic-vs-other discriminant only belongs in resolveAgent. Once an agent is selected, it should consistently honor the bedrock route. Caught by the PULLFROG_AGENT=opencode + Opus 4.6 e2e on pullfrog/preview-720-bedrock — run 25823437606. * ui+docs(bedrock): bespoke setup callout + clearer docs UI: - BedrockSetupCallout in components/AgentSettings.tsx covers both the Model costs section and the onboarding card. Detects bedrock via resolveDisplayAlias().routing === "bedrock", shows a dedicated message ("store AWS_BEARER_TOKEN_BEDROCK as a secret, then put AWS_REGION + BEDROCK_MODEL_ID directly in pullfrog.yml") + link to the setup guide. Replaces the generic "X, Y, or Z is required" prompt that misrepresented the three values as three separate secrets to add (and used the wrong "or" connector for what's actually an AND). - OnboardingCard re-uses the same callout with the gradient-card variant. Docs: - Drop the obsolete "Enable model access" step. AWS retired the manual enrollment page; foundation models auto-enable on first invocation. Anthropic models still need a one-time use-case form for first-time users — surfaced under the AccessDenied troubleshooting entry. - Drop the "Testing a different model in one run" PULLFROG_MODEL note. It introduced the secrets-vs-vars distinction we want to keep out of the bedrock setup story. - Step 3 already recommends hardcoding region + model id in pullfrog.yml. Workflow template: - The default pullfrog.yml customers receive (utils/github/pullfrog.yml.ts) now references AWS_BEARER_TOKEN_BEDROCK from secrets but inlines AWS_REGION and BEDROCK_MODEL_ID as plain values. Matches the docs. * fix(bedrock): three review-caught edges in routing + UI copy Addresses three real issues from PR #720 review: 1. agent.ts: PULLFROG_MODEL=bedrock/byok no longer leaks the literal sentinel "bedrock" downstream. resolveCliModel returns the alias's resolve field verbatim, which for routing entries IS the sentinel. Refactored both the env-override and slug-lookup paths through a shared resolveSlug() that recognizes routing aliases and defers to their backing env var (BEDROCK_MODEL_ID). 2. models.ts: isBedrockAnthropicId() now anchors on a discrete dot/slash/colon-segment match (case-insensitive) instead of a substring contains. The substring check was fragile in both directions for inference-profile ARNs (BEDROCK_MODEL_ID accepts ARNs per AWS docs) — a non-Anthropic profile whose user-chosen name contained "anthropic" would mis-route to claude-code, and an Anthropic profile whose name omitted it would miss CLAUDE_CODE_USE_BEDROCK=1. 3. AgentSettings.tsx: BedrockSetupCallout's configured-state copy showed "AWS_BEARER_TOKEN_BEDROCK configured" even when the user satisfied the gate via AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, gaslighting access-key users about a secret they never set. Detect which auth method is actually present and name the right secret(s) in the success message. Regression tests in models.test.ts (5 new isBedrockAnthropicId cases including positive and negative ARN forms) and agent.test.ts (2 new PULLFROG_MODEL=bedrock/byok cases). 171/171 action tests pass. * yml template: add commented AWS access-key alternative for Bedrock auth Mirrors the IAM access-key path verified end-to-end on PR #720 e2e run 25830764987. Bearer token stays as the primary nudge; the access-key pair is the fallback for users who can't mint Bedrock API keys. * yml template: drop redundant 'or, alternatively' annotation * ui+docs(bedrock): rewrite callout copy + refresh screenshot Reframes the BedrockSetupCallout away from generic BYOK language to a Bedrock-specific message: leads with "Amazon Bedrock is configured entirely via environment variables", lists all four (auth, region, model id), and ends with the requested CTA sentence ("click below to learn more about Bedrock support in Pullfrog"). Promotes the "Bedrock setup guide" docs link from an inline anchor to a prominent button (always visible, regardless of auth state). The "Add AWS_BEARER_TOKEN_BEDROCK" affordance is now a secondary chip shown only when no auth secret is configured. Refreshes docs/images/model-selector-bedrock.png to capture the new callout — the prior screenshot still showed the old generic "BYOK / X, Y, or Z required" wording. |
||
|
Colin McDonnellandpullfrog[bot]
|
1a9d3c1f82 |
fix bootstrap ETARGET when customer has npm min-release-age policy (#725)
* fix bootstrap ETARGET when customer has npm min-release-age policy set npm_config_min_release_age=0 in the action runtime env so `npx --yes pullfrog@<spec>` doesn't get rejected by a customer-side release-age gate (npm 11.5+'s min-release-age / pnpm's minimumReleaseAge). env vars beat .npmrc in npm config precedence, so this neutralises the policy regardless of where it's defined. pullfrog's npm version is server-stamped from a SHA-pinned action ref customers already vet at the action layer — it isn't a customer-vetted dep, so the release-age policy is the wrong affordance for our bootstrap and would otherwise hard-fail every run while the latest publish ages into the customer's window. closes #713 * also cover pnpm's minimumReleaseAge key for corepack fallback path * correct pnpm env var (pnpm v11+ uses pnpm_config_*, not npm_config_*) the prior commit set `npm_config_minimum_release_age=0` to cover the pnpm corepack-dlx fallback path, but pnpm v11+ only reads env vars prefixed `pnpm_config_*` / `PNPM_CONFIG_*` (the v10→v11 migration explicitly renamed the prefix). swap to the correct env var so the fallback path actually neutralises pnpm's `minimumReleaseAge`. also tighten the comment block, and add an AGENTS.md rule reminding us to fetch top-level reviews AND inline review comments together — they live on different endpoints and the inline set is easy to miss with `gh pr view --json reviews,comments` alone. * add scripts/pr-reviews.ts for one-shot review evaluation dumps top-level reviews + inline review threads (with resolved/outdated state) + PR-level conversation in a single GraphQL round trip, so agents don't miss inline-comment feedback. fixes the trap where `gh pr view --json reviews,comments` silently omits the inline `pulls/{n}/comments` set. borrows `gh auth token` so no env vars are required. registered in `wiki/scripts.md`; AGENTS.md rule updated to point at the script instead of the two-step gh-CLI workaround. * pr-reviews: dump raw JSON for jq piping |
||
|
Colin McDonnellandpullfrog[bot]
|
951745ec89 | disable stop hook (runtime + dashboard) (#727) | ||
|
Colin McDonnellandpullfrog[bot]
|
56793d4a81 |
claude: prefer non-JSON stdout over NDJSON tail in exit-1 fallback (#643) (#726)
Claude CLI under CLAUDE_CODE_OAUTH_TOKEN exits 1 without setting `is_error` when the OAuth subscription's quota is exhausted. The existing fallback chain (`lastResultError || stderr || tailLines(stdout)`) had nothing structured to grab and dumped ~2KB of `system/init` NDJSON into the progress comment, hiding the actionable quota notice the CLI had already printed as plain text. Capture non-JSON stdout lines into a 20-line ring buffer (mirroring the existing `recentStderr` pattern) and prefer it over the raw NDJSON tail. Generic — no regex on bubble text — so any human-readable line the CLI emits surfaces instead of the event stream. Also adds a `failure:claude-oauth-quota` bucket to `analyze-logs.ts`, ordered before the SIGTERM check so the NDJSON tail's `cancelled` / `cancel_url` substrings (from learnings content) stop shadowing it. |
||
|
Colin McDonnellandpullfrog[bot]
|
d857e06731 |
postrun: tighten unsubmitted-review gate to require create_pull_request_review for Review mode (#724)
The gate at `getUnsubmittedReview` accepted `toolState.finalSummaryWritten` as a valid Review exit, contradicting the post-failure error message which already says Review's only valid exit is `create_pull_request_review`. This let any caller that flipped `finalSummaryWritten` — including a `task`-dispatched `reviewfrog` subagent calling `pullfrog_report_progress` in violation of its prose-only read-only contract — silence the gate even when the orchestrator never submitted a review. Split per-mode: Review requires `toolState.review`, IncrementalReview keeps the existing `||` (its post-failure message explicitly accepts `report_progress` as a "no review warranted" exit). Test split mirrors the new semantics. closes #648 |
||
|
Colin McDonnellandpullfrog[bot]
|
b9f0938405 |
mcp: restore operational guidance dropped in #723
#723's revision pass cut four substantive strings along with the negative anchors. those strings address real, audit-observed failure modes and the positive examples don't carry them. restored: - push_branch: "if the response reports a timeout, the underlying push may have actually succeeded — verify with git log origin/<branch> before retrying" (was on the tool description) - create_pull_request_review commit_id .describe(): "must be the FULL 40-character SHA — abbreviated SHAs are rejected by GitHub with 422" - create_pull_request_review comments[].line .describe(): "must sit inside a `@@` hunk... dropped entries are reported under droppedComments in the response" - create_pull_request_review comments[].start_line .describe(): "both start_line and line must sit inside the same @@ hunk" also: get_commit_info example used a 31-character SHA (non-standard truncation). swapped to a 7-char short form, which is what git log --oneline emits and what agents see in practice. note that this tool accepts either full or abbreviated, unlike create_pull_request_review which requires full. |
||
|
Colin McDonnellandpullfrog[bot]
|
b8ac42e875 |
mcp: embed example calls in top-level tool descriptions (#723)
* mcp: embed example calls in top-level tool descriptions
agents (esp. claude sonnet) hallucinate param names from training-data
priors — `pr_number` instead of `pull_number`, `summary` instead of
`body`, full subcommand strings jammed into `git({command})` like it
were `shell({command})`. each error burns a tool round-trip plus a
follow-up ToolSearch, ~40+ events / 24h, no observable recovery cost
to us but visible to users in agent logs.
cheapest fix: add a sample formatted function call to every affected
tool's top-level description. example anchors are more reliable than
schema descriptions alone because the model treats descriptions as
narrative but call examples as canonical structure. for `git` and
`shell` (whose `command` fields collide), include explicit
counter-examples disambiguating which tool owns which shape.
no schema aliases / coercion yet — try the cheap thing first; if the
next audit window still shows the same hallucination rate, layer
aliases on top per #585's recommendation.
closes #585, closes #701
* mcp: drop negative anchors from tool descriptions
negation is a footgun in tool descriptions — telling the model "NOT
pr_number" makes pr_number more salient, not less. let the positive
example carry the schema and trust the model to read it.
removes:
- "the parameter is pull_number (a number), NOT pr_number" and
similar across checkout_pr, get_pull_request, list_pull_request_reviews,
get_review_comments, create_pull_request_review
- "NOT summary, message, or content" on report_progress
- "WRONG: git({ command: 'log --oneline' })" counter-example on git
- redundant param-type restatements after the example (e.g. "depth is a
number, not a string" on git_fetch, "description is required" on shell)
keeps a single positive example per tool. for tools with multiple call
shapes (git, git_fetch, push_branch), two positive examples instead of
one + a counter-example.
|
||
|
Colin McDonnellandpullfrog[bot]
|
868576a474 |
audit: format byok auth errors actionably + tighten audit prompt
- `action/utils/apiKeys.ts`: rewrite the missing-key body as Markdown with linked CTAs (repo secrets / model settings / docs). add `isApiKeyAuthError` + `formatApiKeyErrorSummary` covering both shapes: missing key (#679) and revoked/invalid 401 key (#702). - `action/main.ts`: reclassify in the result-failure branch and the catch block so the PR progress comment surfaces the actionable CTA instead of the raw `Invalid API key · Fix external API key` / numbered-list dump. - `scripts/analyze-logs.ts`: split `failure:user-misconfig` into `:no-key` and `:invalid-key` so both buckets are visible separately and the audit can ignore them as user-correctable. - `.github/workflows/run-audit.yml`: add three explicit prompt rules — cross-customer signal required (≥3 distinct accounts; single-customer concentration is not enough), recovered failures are not actionable, user misconfig is out of scope. closes the loop on #679 / #702 being filed in the first place. |
||
|
Colin McDonnellandpullfrog[bot]
|
b2b1e588e7 |
biome: exclude .scripts/ — gitignored operator scratchpad
Mirrors the gitignore. Same shape as the existing !**/logs / !**/.logs / !.worktrees exclusions in files.includes. Matches the upstream .gitignore policy for the .scripts/ directory. Without this, .scripts/ scripts (`.scripts/kyle-*.ts`, `.scripts/check-comment.ts`, etc.) get scanned by `pnpm lint` and `pnpm format` from the repo root and routinely fail husky pre-push even though they're explicitly intended to be local-only / personal. The companion to .gitignore — both are operator-owned scratchpads; neither participates in repo-wide hygiene. |
||
|
Colin McDonnellandpullfrog[bot]
|
5caeb75344 |
review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models (#710)
* review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models
PR review wall-time was dominated by two failure modes: orchestrator
serial-dispatching subagents (despite prompt asking for parallel) and
running every lens on the same Opus tier as the orchestrator. Sample of
recent runs showed 25-60min reviews on small PRs, with 8-10min idle
gaps between subagent dispatches.
Three changes:
1. `action/modes.ts` — replace the soft "1 trivial / 2-3 typical /
4-5 high-stakes" lens calibration with a binary 0-or-2+ rule. Default
is 0 lenses (orchestrator handles review solo with optional cheap
tracerfrog dispatches). 2+ parallel lenses only fire for substantive
PRs (>5 files AND >200 lines) or high-stakes-subsystem touches. Never
exactly one. Both Review and IncrementalReview prompts get loud
ALL-CAPS framing on parallel dispatch — emit ALL Task tool_use blocks
in a single assistant turn before reading any result. Drop the
"do NOT lens-review the diff yourself" advice; orchestrator pulls
context aggressively, in parallel with the lens fan-out.
2. New `tracerfrog` subagent for mechanical code tracing ("where is X
used / who calls Y / what depends on Z"). Pure read+grep+report with
no judgment — orchestrator can dispatch many tracers cheaply in
parallel. Defined in `action/agents/reviewer.ts`. Wired into both
claude.ts (`--agents` JSON) and opencode.ts (`agent` config block).
3. Per-subagent model downshifts via `deriveSubagentModels`:
- Anthropic: reviewfrog → Sonnet, tracerfrog → Haiku
- OpenAI: both → gpt-5.4-mini
- other providers (xai, deepseek, gemini, etc.): inherit (no
standard tier triplet to downshift to)
Claude Code path always runs Anthropic so the downshift is hardcoded
inline in claude.ts. OpenCode uses the helper since orchestrator
provider varies.
Both runtimes' subagent-definition formats verified directly against
their source: `--agents` JSON `model` field (claude-code's
`AgentJsonSchema` accepts model+effort+maxTurns+more) and OpenCode's
`agent.{name}.model` config field (parsed via Provider.parseModel,
applied per-task in tool/task.ts line 92). Parallel dispatch is
infra-supported in both — only the orchestrator model's tool_use
emission pattern was the bottleneck.
Tests: subagentModels.test.ts (14 tests covering provider matrix),
subagentRegistration.test.ts (6 source-asserts catching shape
regressions in buildAgentsJson / buildReviewerAgentConfig).
* subagentModels: add openrouter routes (proxy/router mode)
Initial helper missed the openrouter prefix used by Pullfrog's router
proxy. preview-710 e2e showed the OpenCode + openrouter path receiving
no downshift — orchestrator and lenses both ran on opus-4.7 because
'openrouter/anthropic/claude-opus-4.7' didn't match any of the
anthropic/openai prefixes the helper checked.
Add explicit branches for 'openrouter/anthropic/...' (uses dot notation:
claude-sonnet-4.6 / claude-haiku-4.5) and 'openrouter/openai/...'
(gpt-5.4-mini for both reviewer and tracer). Same opus->sonnet,
sonnet->keep-but-haiku-tracer, haiku->no-op semantics as the direct
anthropic path.
* opencode: log resolved subagent models at startup
So we can verify per-subagent model overrides actually take effect at
runtime. Prints once per run alongside the existing model/effort log
lines.
* drop tracerfrog: keep reviewfrog only, LSP-powered tracer planned later
Removes the cheap-haiku-tracer subagent (TRACER_AGENT_NAME +
TRACER_SYSTEM_PROMPT, registrations in claude.ts/opencode.ts, dispatch
guidance in modes.ts). The mechanical-tracing use case will be served
better by an LSP-powered tool than by a separately-prompted subagent.
deriveSubagentModels collapses to a single { reviewer } shape; the
reviewfrog-on-Sonnet downshift stays. Same source-assert + provider-
matrix tests, minus the tracer-specific cases.
modes.ts wording: drop the 'subagent type cheat sheet' bullet, drop
the parenthetical 'often better served by tracerfrog than reviewfrog'
on the impact lens, drop tracerfrog from the same-turn-context-pulling
hint. The 0-or-2+ rule and ALL-CAPS parallel emphasis are unchanged.
* subagentModels: broader downshift coverage (gpt-pro, gemini-pro, grok); drop gpt-mini target
Scanned every resolved orchestrator slug in action/models.ts against
models.dev pricing data. Identified five clear cases where the
orchestrator is meaningfully expensive AND has a cheaper sibling that
remains capable enough for review-style judgment work.
Changes:
- Anthropic: opus → sonnet (kept; -40%)
- OpenAI: gpt → gpt-5.4 (was: gpt-mini; -54% instead of -85% but
preserves review-quality judgment — gpt-mini was too dumb)
- OpenAI: gpt-pro → gpt (NEW; -93%, biggest single unlock —
gpt-5.5-pro is $30/Mtok in vs gpt-5.5 at $5)
- Google: gemini-pro → gemini-flash (NEW; -75%)
- xAI: grok-4.3 → grok-4-1-fast (NEW; -80%)
Every branch handles the three routes in use: direct provider slug,
opencode-vendored, and openrouter-proxied. Variants below the downshift
target (mini/nano/flash/fast/sonnet/haiku) inherit (no further drop).
Skipped:
- DeepSeek: v4-flash ($0.14/Mtok) is too far below review judgment
threshold; v4-pro orchestrator already cheap ($0.55 blended).
- Moonshot: kimi-k2-thinking would only save 32% and slug stability on
OpenRouter is uncertain; revisit if cost matters.
- o3: already mid-tier in OpenAI's reasoning family; no clean target.
* models: hoist subagent downshift into the registry, add hidden flag
The downshift relationship now lives next to each alias's resolve /
openRouterResolve as a sibling field. Two new ModelDef fields:
- subagentModel?: string — alias key (within same provider) of the
cheaper sibling reviewfrog should use as a lens-fanout subagent.
e.g. claude-opus → 'claude-sonnet'.
- hidden?: boolean — exclude from selectable lists (UI dropdown,
CLI init picker). Does NOT affect resolution; for that use
fallback. Used so internal-only subagent targets like openai/gpt-5.4
exist in the registry but never appear as a user-facing pick.
Wiring:
- anthropic.claude-opus → claude-sonnet (-40%)
- openai.gpt-pro → gpt (-93%, biggest unlock)
- openai.gpt → gpt-5.4 (-54%); gpt-5.4 added with hidden:true
- google.gemini-pro → gemini-flash (-75%)
- mirrored across opencode + openrouter providers (each provider
declares its own three-route data so the downshift declaration
is colocated with the rest of the alias definition).
deriveSubagentModels collapses from ~85 lines of prefix-matching to
a ~15-line registry reverse-lookup: find the alias whose resolve OR
openRouterResolve matches the orchestrator's spec, follow its
subagentModel pointer, return the matching field of the target alias.
Filter sites updated:
- components/ModelSelector.tsx: !a.fallback && !a.hidden
- action/commands/init.ts: same
Tests rewritten to exercise the registry through the public surface;
the matrix collapses to one assertion per (provider × route) pair.
* TEMP: log per-step cost+tokens for subagent model verification (PR #710)
* TEMP: also log SUBAGENT step_finish from bus envelope handler
* remove temporary per-step diagnostic logs (verification done)
Verified subagent model downshift takes effect end-to-end on the OpenCode
+ openrouter path. PR #8 in pullfrog/preview-710-review-perf dispatched
3 lenses (billing-subsystem / security / correctness) on the orchestrator's
opus-4.7 session, and per-subagent step_finish events showed actual cost
exactly matching Sonnet pricing rates (60% of what Opus would have cost):
session n actual if-Opus if-Sonnet match
T3VrUuF... 5 $0.2425 $0.4042 $0.2425 Sonnet ✓
93ZZR7E... 4 $0.2253 $0.3754 $0.2253 Sonnet ✓
Fb1Kr7b... 4 $0.2495 $0.4158 $0.2495 Sonnet ✓
The startup '» subagent models: reviewfrog=...' line stays — useful
permanent diagnostic showing the resolved subagent model per-run.
* TEMP: log per-event model from claude.ts assistant handler
* remove temporary per-event model log (claude.ts verification done)
Verified subagent model downshift takes effect end-to-end on the Claude
Code path. PR #9 in pullfrog/preview-710-review-perf dispatched 2 lenses
on an opus-4-7 orchestrator. Per-assistant-event model field from the
SDK's stream-json output, partitioned by parent_tool_use_id:
ORCH (parent_tool_use_id=null): 17 events all model=claude-opus-4-7
SUBAGENT lens:billing-subsystem: 17 events all model=claude-sonnet-4-6
SUBAGENT lens:security: 21 events all model=claude-sonnet-4-6
Zero leakage to opus from either subagent session. The per-subagent
'model' field in --agents JSON is honored by claude-code at the SDK
level, identical to the OpenCode path verified earlier.
* opencode: bump per-call output cap 5K → 16K to unblock large reviews
The 5K cap (added in #616 to lower OpenRouter upfront budget reservation
for low-wallet runs) was capping the entire response of a single LLM call,
not just the budget reservation. A single tool_use response — like a
`create_pull_request_review` with many inline comments — would truncate
mid-stream past 5K output tokens, leave the JSON unparseable, and the tool
would never actually invoke. We hit this on PR #710's verify-downshift PR:
review aggregated from 3 lenses had 11 inline comments + a long body,
truncated at out=5000 on every retry attempt, action exited with 'Review
mode finished without calling create_pull_request_review after 3 retry
attempts'.
Investigated whether OpenCode (or OpenAI/Anthropic/OpenRouter directly)
exposes a separate budget-reservation parameter that could stay small
while letting the response exceed it. They don't — `max_tokens` /
`max_completion_tokens` is the single value all four use for both the
upfront reservation and the hard output ceiling. No way to decouple them
at the API surface.
Bumped to 16K as a middle ground: 8× the prior cap (handles every review
shape we've observed plus headroom), still half of OpenCode's 32K default
so the wallet-burn benefit for low-balance accounts is preserved, just
smaller. For Opus 4.7 a typical ~50K-input call now reserves roughly
$0.65 instead of the prior $0.38.
Updated the constant comment to spell out the trade-off clearly so this
doesn't happen again.
* opencode: drop OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX override entirely
Verified the original rationale for the override is obsolete. From #616
the cap shrunk OpenRouter's per-call upfront budget reservation so a
single call's reservation wouldn't exceed the per-run key cap
(`ROUTER_PER_RUN_LIMIT_USD = 25`) and lock low-balance accounts out of
starting a run.
That per-run gate is gone. `app/api/proxy-token/route.ts` ~line 422
explicitly says: 'No upper cap (the old ROUTER_PER_RUN_LIMIT_USD = 25 is
gone). The natural ceiling is whatever the user has + their buffer.'
Router now mints keys with `keyLimitCents = balance + buffer` ($50 for
autoreload+card, $5 for card-only, $0 for no-card). A single call's
upfront reservation fits comfortably within that — no separate per-call
gate to fail past.
The cap had a real downside as a hard per-call output truncation. A
single `create_pull_request_review` tool_use with many inline comments
would truncate mid-stream past 5K output tokens, the JSON would be
unparseable, and the tool never invoked. Hit on PR #710's
verify-downshift PR.
Removing the override entirely; OpenCode falls back to its 32K default.
Left an explanatory note above the env-var assignment site so the next
person doesn't unknowingly re-add it.
|
||
|
Colin McDonnellandpullfrog[bot]
|
d04c1ca3da | action: bump to 0.1.6 | ||
|
Colin McDonnellandpullfrog[bot]
|
ae976e7159 |
parallel tool execution: enable opencode batch + nudge agents to parallelize (#719)
opencode: opt into `experimental.batch_tool` (anomalyco/opencode#2983) so the `batch` tool registers and the model can bundle 1-25 independent calls into one round trip. edit calls are excluded upstream. instructions.ts: add a "Parallel tool execution" section to the SYSTEM Workflow block, agent-specialized via ctx.agentId. uses Anthropic's canonical wording ("invoke all relevant tools simultaneously...") so Claude reliably emits multiple tool_use blocks per message; tells OpenCode about the new `batch` affordance. verified end-to-end against haiku-class models (sonnet for claude, default for opencode) with a "read 3 files and report first lines" fixture. results: - opencode used `batch` with 3 nested reads AND emitted 3 native parallel read calls in the same assistant turn - claude went from 3 serial turns (1 read each) to 1 message with 3 parallel Read tool_use blocks |
||
|
Colin McDonnellandpullfrog[bot]
|
5aabd1e4a9 |
fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680) (#715)
* fix(action): cap subprocess stdout/stderr retention to prevent RangeError crashes (#680) unbounded `stdoutBuffer += chunk` / `stderrBuffer += chunk` in `action/utils/subprocess.ts` previously crashed the wrapper with `RangeError: Invalid string length` once V8's ~1 GiB kMaxLength was breached on long-lived agent runs. multi-lens opencode Reviews on large monorepos (e.g. tambo-ai/buildy) hit this consistently — 23 runs in the last 24h, 100% of Review-mode hard failures on that repo. - add `retain: "tail" | "none"` to SpawnOptions, defaulting to "tail" with an 8 MiB cap. tail-mode prepends a `... [N MiB truncated] ...` sentinel so downstream consumers can detect truncation. - export `TailBuffer` helper for callers that need the same bounded accumulator semantics at their own layer. - wrap stream `data` listeners in try/catch as defense in depth — any synchronous throw inside a stream handler is otherwise fatal. - opencode + claude pass `retain: "none"` (they drain via onStdout / onStderr) and switch their own `output` accumulators to TailBuffer. their error paths read the agent-layer bounded mirrors instead of the now-empty `result.stdout` / `result.stderr`. - add `failure:string-length-overflow` heuristic to scripts/analyze-logs.ts so post-fix recurrences are visible at a glance instead of bucketing into `failure:unknown`. - regression tests cover >1 MiB stderr without crash, retain:"none" contract, and TailBuffer truncation semantics. * fix: avoid TS parameter property syntax in TailBuffer for strip-only node loader * address review: clarify try/catch scope + lock retain default to "tail" - the original comment claimed the try/catch caught "any synchronous throw" in the data listener, but `options.onStdout?.(chunk)` returns a Promise in the agent callers (claude.ts:569, opencode.ts:933) — a throw inside an async user callback surfaces as an unhandled Promise rejection, not a synchronous exception. reword to describe the actual protection: defense-in-depth for synchronous throws in the listener body, which is exactly the shape of the original RangeError on `+= chunk`. - add a test that locks `retain` default to "tail" by spawning without the option and asserting `result.stderr` is non-empty. a future refactor that flipped the default to "none" would silently break gitAuth, package installs, and lifecycle hooks that read result.stderr for failure messages, and the rest of the suite wouldn't catch it. |
||
|
Colin McDonnellandpullfrog[bot]
|
60cc8772a6 |
fix(log-audit): kill 404 noise from /api/github/installation-token at source (#693) (#708)
* fix(log-audit): kill 404 noise from `/api/github/installation-token` at source (#693) Closes #693. Issue diagnosed a surface symptom (`log.error` on expected 404s) but missed the actual root causes. Investigation revealed two distinct populations producing identical 3-call 404 bursts: 1. **Fork-CI on `pullfrog/pullfrog`**: `test-token.yml` and `trigger-sync.yml` ship with `on: push: main`, so every fork inherits them and 404s our token endpoint on first push. Self-inflicted noise that scales with fork count. 2. **Real users hitting the full action without installing the App**: `/api/repo/.../run-context` uses the caller's `GITHUB_TOKEN` to read the repo from GitHub and then unconditionally lazy-provisions Account+Repo rows via `fetchOrCreateRepo`, even when the App isn't installed. Generates phantom DB rows and false `new account created` team@ alerts. (Confirmed via Prisma: `ezcorp-org` has an Account row with `installerLogin: null`, never installed our App.) Both populations then trip the client retry loop in `acquireTokenViaOIDC`, which matched `"Token exchange failed"` and retried 3× on terminal 4xx — tripling log volume and wasting CI time. ## Changes - `action/.github/workflows/{test-token,trigger-sync}.yml`: gate jobs with `if: github.repository == 'pullfrog/pullfrog'`. Forks inherit the files but the jobs no-op. - `app/api/repo/[owner]/[repo]/run-context/route.ts`: call `getRepoInstallation` first; return 404 with install URL if the App isn't installed, before any DB writes or GitHub repo fetch. - `action/utils/github.ts`: introduce `TokenExchangeError` for non-2xx server responses; `acquireNewToken` no longer retries it. Retry now fires only on genuine network/timeout failures. 404 surfaces a user-actionable error pointing at the install URL. - `app/api/github/installation-token/route.ts`: move `log.error` inside the 500 branch only. 404 branch is silent (expected user-state) and returns the same install URL message for consistency. ## Effect - Better Stack `level=error` lines from this path: 6/day → 0. - Failed user-trial CI time: 3 wasted token requests → 1. - User-facing error: opaque `Token exchange failed: 404` → actionable install URL. - No more phantom Account rows from never-installed callers. Skipped per design discussion: phantom-account cleanup (conservative — stop the bleed, leave history), `AGENTS.md` rule (overgeneralized). * review: address oracle leak + per-env install URL + retryable 5xx Addresses pullfrog[bot] (IMPORTANT) and Copilot review findings on #708: - **Install-status oracle in `run-context`** [pullfrog, Copilot]: `getRepoInstallation` runs with our App's JWT, *before* the caller's bearer token is validated against the repo. Pre-PR the route was uniformly bad-token-shaped; the new install-specific 404 turned it into an unauthenticated oracle distinguishing "Pullfrog installed here" from "not installed". Collapsed the 404 message to match the outer catch's ambiguous "repository not found or token lacks access". Legit runners still get the actionable install URL from `/api/github/installation-token`, which IS gated by OIDC. - **Hardcoded `github.com/apps/pullfrog`** [Copilot]: server-side `installation-token` now uses `GITHUB_APP_INSTALL_URL` from `app/globals.ts`, so dev/staging deployments with a different `GITHUB_APP_SLUG` direct users to the correct app. Action-side echoes the server's `error` body when present (single source of truth) and falls back to a generic message only if the body isn't JSON. - **Transient 5xx/429 made terminal** [Copilot]: `shouldRetry` now returns `true` for `TokenExchangeError` with `status >= 500` or `status === 429`. 4xx remains terminal (the actual #693 fix). Real outages no longer fail the workflow immediately. - **Stale comment** [pullfrog, Copilot]: reworded the comment at `installation-token/route.ts:141` to reflect the new retry policy ("the action surfaces this once (no retry)" instead of "the action retries on this"). * review: restore caller-token-first auth in run-context Pre-PR, `getEnrichedRepo({owner, repo, token})` used the caller's token as the auth boundary — `getRepo({token})` succeeding was the proof-of-access check. My initial install-gate inverted the order and ran the App-credentialed `getRepoInstallation` first, which is how it became: - an install-status oracle (pullfrog bot, addressed previously by matching the outer-catch wording), and - an outbound amplifier against our App JWT for arbitrary `owner/repo` (pullfrog bot, this commit). Reordered so `getRepo({token})` runs first. Garbage / unauthorized bearers get rejected by github (mapped to 403 by the outer catch) before any App-credentialed call fires. `getRepo` is cached 5min, so `getEnrichedRepo` below remains a free re-hit. |
||
|
Colin McDonnellandpullfrog[bot]
|
4260984257 |
attribute claude subagent log lines + per-session thinking timer; tighten lens calibration (#700)
* attribute claude subagent log lines + per-session thinking timer; tighten lens calibration three orthogonal fixes diagnosed from the 10m PR-699 review run: 1. wire SessionLabeler into the Claude Code harness. claude-agent-sdk stamps every Assistant/User/System message with session_id and a non-null parent_tool_use_id when emitted from a subagent context, so the same FIFO labeler the OpenCode harness uses works here too. parallel reviewfrog dispatches now log with [lens:correctness] / [lens:operational-readiness] / etc. prefixes instead of being indistinguishable from the orchestrator. matches both "Task" and "Agent" tool names per the v2.1.63 rename. 2. one ThinkingTimer per session. the global timer treated cross-session interleaving (parent thinks → child tool_call, child returns → parent dispatches next) as parent thinking time, so individual "thought for Xs" numbers were untrustworthy. each session now owns its own timer and prefixes its own log line. 3. tighten the Review/IncrementalReview lens-add discipline. PR-699 triggered 4 lenses on a typical refactor (no auth/billing/schema) when the prompt's own calibration says 2-3 is typical; the research-validated lens went deep on Resend idempotency window + prisma updateMany lost-updates without either being load-bearing. adds an explicit "name the failure mode this lens would catch that the diff plausibly introduces" bar, and tightens research-validated specifically: only when correctness depends on the third-party contract, not when the API is merely used. side benefits from #1: subagents' TodoWrite events no longer clobber the orchestrator's progress comment; subagent text no longer overwrites finalOutput; system-event handler safely routes through eventLabel even though SDK only emits system:init for the top-level query today. * fix node strip-only mode: declare formatLine as field, not parameter property * key claude subagent labels by parent_tool_use_id, not session_id claude-agent-sdk runs subagents inside the orchestrator's session — they share session_id — and stamps subagent messages with parent_tool_use_id pointing at the Agent tool_use that spawned them. e2e on PR-700 with preview-700-claude-labeling#1 confirmed the original session_id-keyed wiring never differentiated subagent activity (only the dispatch line got [lens:correctness] in the log; the subagent's reads, writes, and todos all rendered as orchestrator). extend SessionLabeler so labelFor accepts an optional parent_tool_use_id and short-circuits to a direct map keyed by Agent tool_use id when set. recordTaskDispatch optionally takes the Agent tool_use id (block.id at dispatch time) and binds it. orchestrator events keep flowing through the sessionID/FIFO path unchanged so opencode wiring is untouched. * drop weak timer test that asserted only field isolation per pullfrog review on PR-700: the 'two timers do not bleed timestamps' test only verified that two ThinkingTimer instances have separate private fields, which has always been true. doesn't earn its keep — the per-session behavior is exercised by integration through claude.ts + opencode.ts. |
||
|
Colin McDonnellandpullfrog[bot]
|
d5f881e9fc |
action: trim sensitive env values before GitHub Actions log masking (#698)
* action: trim sensitive env values before GitHub Actions log masking GitHub Actions' log masking is line-based: a secret value containing a newline only registers the first line as a mask, leaving the remainder exposed verbatim in logs. A trailing newline copied from a terminal into a GitHub Actions secret (e.g. ANTHROPIC_API_KEY) was enough to leak "a large part of the key" in run logs (pullfrog/pullfrog#41). normalizeEnv now trims leading/trailing whitespace from any value whose key matches the sensitive name pattern, masks the cleaned value, and warns when whitespace was stripped so the user notices the source. sanitizeSecret is reused for dbSecrets injection in main.ts. The three secret-store PUT/POST routes also trim values defensively, matching the existing name.trim() pattern. Real multi-line secrets are not used in practice — even GITHUB_PRIVATE_KEY PEMs are stored single-line with escaped \n and unescaped at the point of use — so a straight trim() is safe. * action: address review — use core.setSecret for masking, don't zero whitespace-only Pullfrog's review of #698 caught two real issues in the original fix: 1. `console.log(\`::add-mask::\${trimmed}\`)` doesn't escape \r/\n. If a value survives trim with an embedded newline (PEMs, kubeconfigs, JSON), the runner only registers the first line as a mask and the rest leaks. `core.setSecret(trimmed)` routes through @actions/core which percent-encodes \r/\n so the runner V2 parser decodes back to the full value and registers every non-empty line as a separate mask. Removes the load-bearing "no embedded newlines" invariant from the fix. 2. Whitespace-only sensitive values silently became "". Downstream truthy checks would flip from "set" to "missing" with no log. Now sanitizeSecret returns null in that case and callers skip the process.env write, surfacing a clear missing-key error instead. Tests rewritten to assert process.env state directly — no stdout spies. Masking correctness is delegated to @actions/core (trusted dependency). |
||
|
Colin McDonnellandpullfrog[bot]
|
1dc53043a6 | chore: bump action to 0.1.5 | ||
|
Colin McDonnellandpullfrog[bot]
|
076e5a17b5 |
default Claude Code effort to high
max effort burns roughly 2x the wall time per turn for marginal quality
gain. high is the model's tuned default ('equivalent to not setting the
parameter' per Anthropic docs). full-send can be reintroduced as an
opt-in per-run override later if needed.
|
||
|
Colin McDonnellandpullfrog[bot]
|
d5d8a0d7ac |
fix(#691): drop opencode/gpt-5-nano + opencode/mimo-v2-pro-free (not actually keyless on Zen) (#695)
* remove opencode/gpt-5-nano and opencode/mimo-v2-pro-free from catalog #7 delete aliases. both were listed as `isFree: true, envVars: []` but neither is keyless on opencode zen, producing a hard-fail `UnknownError: Model not found: opencode/<id>` on every run without an opencode_api_key. fixes pullfrog/app#691 (5 runs across 3 repos, 100% failure rate in the last 24h). root cause: opencode's provider gate (`packages/opencode/src/provider/provider.ts` `opencode:` loader) keeps a zen model only when models.dev reports `cost.input === 0` for it, then signs requests with `apiKey: "public"`. paid zen models get deleted from the autoloaded set and opencode surfaces the deletion as "model not found". - `opencode/gpt-5-nano`: models.dev reports `cost: {input: 0.05, output: 0.4, cache_read: 0.005}`. paid → requires `OPENCODE_API_KEY`. - `opencode/mimo-v2-pro-free`: free on models.dev but not in `https://opencode.ai/zen/v1/models` — zen never served it, so even the public-key path fails. remaining free aliases (`opencode/big-pickle`, `opencode/minimax-m2.5-free`) both pass both checks (cost.input === 0 in models.dev AND present in zen's served list) and continue to work without a key — verified against the opencode source. callers swept: `action/utils/apiKeys.test.ts`, `action/models.test.ts`, `action/test/list-aliases.ts`, `action/test/model-smoke.ts`, `components/ModelSelector.tsx` (`modelIdToUpstream`), `wiki/model-resolution.md`, `wiki/models-catalog.md`. wrote up the free-zen verification rule in models-catalog so the next maintainer can sanity-check both conditions before adding any `isFree` alias. users with a stored `opencode/gpt-5-nano` or `opencode/mimo-v2-pro-free` will now fall through `resolveCliModel → undefined` into the auto-select path — a strict improvement over today's hard fail. no DB migration needed; the slugs are simply unknown and treated like any other unrecognized stored value. * rework: keep mimo deprecated, demote gpt-5-nano to paid, add free-zen invariants revised approach after the first commit over-corrected. mimo was never broken at runtime — `fallback: "opencode/big-pickle"` already routes stored values through to a real free model before any zen call. the literal `opencode/mimo-v2-pro-free` being absent from zen's served list is irrelevant because `resolveCliModel` walks the chain first. restoring it as-is. the actual bug was `opencode/gpt-5-nano`: marked `isFree: true, envVars: []` but `models.dev` reports `cost: {input: 0.05, output: 0.4}` on the opencode provider, so opencode's keyless gate (`packages/opencode/src/provider/provider.ts` `opencode:`) deletes it when `OPENCODE_API_KEY` is missing and the run hard-fails with `UnknownError: Model not found: opencode/gpt-5-nano`. demoting it to a regular paid zen alias (drop `isFree`/`envVars: []`, add `openRouterResolve: "openrouter/openai/gpt-5-nano"` — verified to exist on openrouter at the same price). users without `OPENCODE_API_KEY` now get our explicit "no API key found" error pointing at the secrets page instead of opencode's cryptic upstream error. confirmed via `https://opencode.ai/zen/v1/models` that zen serves no free GPT variants, so there's no cheaper-than-`gpt-mini` free option to suggest in its place. CI gap analysis (why this slipped through): - `models-catalog.main.test.ts` only checked existence + `status !== "deprecated"` on models.dev. paid-model-marked-free regressions and zen-served-list drift both passed. - `models-live` (`model-smoke.ts`) runs with `OPENCODE_API_KEY` in env, so the keyless deletion gate never fires. `gpt-5-nano` returned "OK" in CI even though end users hit a hard fail. - `model-smoke.ts` walks the fallback chain, so mimo would have been smoked as big-pickle anyway — the dead resolve target was never exercised directly. (this is the right design; the gap is at the catalog layer, not the smoke layer.) new tests: - PR-blocking, static (`action/test/models.test.ts`, `isFree invariants`): every `isFree` alias must live under `opencode`, have `envVars: []`, omit `openRouterResolve`, AND have a fallback chain whose terminal alias is also `isFree` (catches "deprecate a free alias to a paid target" — the worst silent-charge regression). - main-only, network (`action/test/models-catalog.main.test.ts`, `opencode Zen served list`): every alias whose terminal-fallback resolve is `opencode/*` must appear in `https://opencode.ai/zen/v1/models`. catches zen dropping a model from its served list. - main-only, network (same file, `isFree models.dev cost`): every `isFree` alias's terminal-fallback resolve must have `cost.input === 0` in the `opencode` provider block on `models.dev`. would have caught `gpt-5-nano` at the next models-bump run. both network tests dedupe on terminal resolve, so deprecated aliases sharing a target aren't double-counted. `pnpm vitest run`: 113 static tests pass. `pnpm test:catalog`: 142 network tests pass against the live `models.dev`, `openrouter.ai`, and `opencode.ai/zen/v1/models` endpoints. wiki/models-catalog.md: rewrote the new "Free-Zen aliases need Zen-side verification" section to (a) describe the two conditions, (b) note that a fallback to an isFree alias is the legitimate escape hatch (mimo's pattern), and (c) point at the three tests by name so the next maintainer can find the enforcement surface. wiki/model-resolution.md points at the new section. * make gpt-5-nano a deprecated free alias falling back to big-pickle revising the previous "demote to paid" approach. the user-facing ergonomics are cleaner: anyone who picked gpt-5-nano under the "Free" badge gets transparent-upgraded to a real free model (big-pickle) instead of suddenly being asked to set OPENCODE_API_KEY. matches the existing mimo pattern exactly. the dropdown already filters `!a.fallback`, so the slug disappears from the picker on its own and the trigger renders it as "Big Pickle" via `resolveDisplayAlias`. no other catalog or test surface changes — the isFree invariants and the main-only zen/cost checks still pass (gpt-5-nano's terminal is now big-pickle, which is both isFree and zero-cost on models.dev, deduping with big-pickle's own row in both network tests). * revise: keep gpt-5-nano as paid alias, backfill affected DB rows instead dropping the deprecated-alias approach. `opencode/gpt-5-nano` is a legitimate cheap paid model people may want with BYOK (`OPENCODE_API_KEY`) — giving it `fallback: "opencode/big-pickle"` would foreclose that for everyone going forward. correct fix is two parts: (a) reclassify in the catalog as a regular paid OpenCode alias: - drop `isFree: true` and `envVars: []` so the local validator demands `OPENCODE_API_KEY` - add `openRouterResolve: "openrouter/openai/gpt-5-nano"` to satisfy the completeness test and route BYOK-via-OpenRouter users - no `fallback` — slug stays visible in the picker as a paid option (b) one-shot DB backfill of provably-affected repos (`scripts/backfill-gpt5-nano-affected.ts`). scope: - `Repo.model = "opencode/gpt-5-nano"` - AND at least one `WorkflowRun` with `inputTokens IS NULL` (evidence of an attempted run that didn't get past the model-init gate) skipped intentionally: - repos whose runs have `inputTokens > 0` — they have a key, gpt-5- nano works for them - repos with zero WorkflowRun rows — never dispatched; touching them would be presumptuous - `LearningsRevision.model` — audit trail of which model authored a revision, rewriting it would falsify history ran against .env.prod: 2 repos stored the slug; 1 was provably affected (sodown4thecause/seobot, 5/5 zero-token runs — matches #691's 3 failed runs from this repo plus 2 outside the 24h audit window). 1 was an internal test account that never dispatched (left as-is). applied: 1 row updated. confirmed idempotent on re-run. the other two repos in #691 (Nantiee/ALTA-breast-pump-tool, keksiqc/ansible-setup-linux) don't store the slug in `Repo.model`; their failed dispatches passed the model inline in the `workflow_dispatch` `prompt` payload, so the catalog fix alone (no longer offering it as free) is what helps them. tests: - models.test.ts: `getModelEnvVars("opencode/gpt-5-nano")` now returns `["OPENCODE_API_KEY"]`, moved into the keyed-model group - apiKeys.test.ts: added "throws without OPENCODE_API_KEY" case - isFree invariants from the previous commit still pass — gpt-5-nano no longer triggers them since it's no longer isFree - main-only catalog tests still pass (gpt-5-nano served by Zen, just paid; no isFree cost check applies) * docs: drop stale GPT Nano + MiMo V2 Pro from free-tier lists addressing pullfrog auto-review feedback on #695. three mintlify pages still advertised both as keyless after the catalog pivot, which now makes the docs affirmatively wrong rather than merely stale: - gpt nano is paid in the catalog (no `isFree`, inherits `OPENCODE_API_KEY`); a user following the docs would hit the same "missing API key" failure that's described 4 lines below in `docs/keys.mdx`. - mimo v2 pro is hidden from the picker (`fallback` triggers `ModelSelector`'s `!a.fallback` filter); the alias only exists for legacy stored-value resolution. a user reading the docs cannot actually pick it. surviving picker-visible free set: Big Pickle and MiniMax M2.5. - `docs/keys.mdx`: drop both bullets from the "Free models" list - `docs/billing.mdx`: drop both bullets from the "Free models" list - `docs/getting-started.mdx`: collapse the inline mention from a 4-model list to "Big Pickle and MiniMax M2.5" * address third review: picker grouping + backfill classifier honesty i had not pulled the third pullfrog review (`02:17:28Z`) when i declared reviews triaged after the docs sweep — the fourth review flagged that three findings remained pending. addressing them now. 1. picker grouping for now-selectable paid gpt-5-nano. when i removed `"gpt-5-nano": "OpenAI"` from `modelIdToUpstream` in the previous pivot-to-paid commit, i mistook it for dead code. it's not — the map IS consulted for paid opencode aliases via `groupByUpstream → getUpstreamLabel` inside the OpenCode submenu's `renderSubContent`. without the entry, `gpt-5-nano` falls back to `getProviderDisplayName("opencode")` = "OpenCode" and gets dropped into its own sub-header instead of joining opencode/gpt, opencode/gpt-pro, opencode/gpt-mini under the "OpenAI" upstream group. re-added with an explanatory comment so the next refactor doesn't make the same mistake. 2. JSDoc / code mismatch in `scripts/backfill-gpt5-nano-affected.ts`. the JSDoc said "at least one `WorkflowRun` with `inputTokens IS NULL`" but the code is `no WorkflowRun has inputTokens > 0` — a strictly broader filter (catches `null` AND `0`). rewrote the scope block to describe what the code actually does, with the operative classifier spelled out: "a billable run with `inputTokens > 0` is proof the agent successfully reached and called the model". 3. classifier breadth (raised in the same review). honest answer: the "no positive-token run" filter IS a heuristic — a repo whose only dispatches happened to fail or cancel for unrelated reasons would get false-positive-classified A. for THIS one-shot population (2 repos, 1 with 5/5 zero-token runs — strong systematic-failure signal) the heuristic was good enough and the dry-run inspection confirmed before APPLY. for any larger reuse of this pattern, you need to cross-reference the runtime error string (`UnknownError: Model not found: opencode/gpt-5-nano`) from GitHub Actions logs or Better Stack — that error doesn't live on `WorkflowRun` rows. added a "Classifier limitations" section to the JSDoc making this explicit. nothing about the actual applied backfill changes — the prod write (1 repo: sodown4thecause/seobot → opencode/big-pickle) is unchanged and re-running the script remains idempotent. |
||
|
Colin McDonnellandpullfrog[bot]
|
159389fad2 |
fix(mcp): sanitize for gemini when model is unresolved (#697)
* fix(mcp): sanitize for gemini when model is unresolved isGeminiRouted() previously required the effective model string to contain "gemini" — but when payload.model="auto" (or any unresolved slug) reaches addTools(), `effective` is the literal "auto", which doesn't match. opencode then auto-selects gemini *after* the MCP server has registered raw arktype schemas, and every tool turn dies on `function_declarations[*].properties[*].any_of[*].enum: only allowed for STRING type`. widen the gate: any unresolved specifier (undefined / "auto" / a slug without a `provider/` prefix) is treated as gemini-routed and sanitized. the transforms are universally compatible normalizations so the false-positive cost is negligible. tighten case 3 to preserve `description` so the only lossy path no longer drops operator-facing context. fixes #676. * revert case-3 description preservation per pullfrog review on #697: keeping `description` as a peer of `anyOf`/`oneOf` directly contradicts the file's own header (lines 19-21) and the upstream opencode #14659 rationale that gates this sanitizer — gemini requires anyOf to be the ONLY field on a schema node, sibling keywords trigger `anyOf must be the only field in a schema node`. the change was speculative scope creep with no evidence, and would silently re-introduce a different gemini failure for any future schema using `.describe().or(...)`. the bug fix for #676 doesn't need it (arktype doesn't emit non-collapsible anyOf for current tool schemas). |
||
|
Colin McDonnellandpullfrog[bot]
|
43bb14bf87 |
action: strip Content-Type on body-less apiFetch requests (#692) (#694)
* action: strip Content-Type on body-less apiFetch requests (#692) Vercel's Next.js lambda adapter (Next 16.1.x) attempts to decode a request body when Content-Type is set and throws `SyntaxError: Unexpected end of data` before delegating to the route handler, returning a 500. Hit /run-context exclusively because it was the only body-less GET that sent `Content-Type: application/json`. - Drop `Content-Type: application/json` from the GET in `action/utils/runContext.ts` (meaningless on a body-less request). - Defensively strip any `content-type` header in `action/utils/apiFetch.ts` when no body is present so future callers can't reintroduce this. * apiFetch: soften comment — empirical observation, RFC 9110 §8.3 framing |
||
|
Colin McDonnellandpullfrog[bot]
|
d8f825034f |
billing: $10 signup credit + lazy claim modal; disable welcome credit promo (#674)
* billing: $10 signup credit + lazy claim modal; disable welcome credit promo Adds a per-Account $10 Router signup credit granted on first Router-tab mount via a new admin-gated POST /api/account/[owner]/signup-credit/claim. The endpoint is idempotent — the inserted CreditGrant row IS the dedup state, so subsequent calls return granted:false. Client SignupCreditModal fires the POST on mount (only when modelAccessMode === "router") and opens a celebratory dialog when granted:true. Disables the legacy welcome credit ($10 on first card add) via a new WELCOME_CREDIT_PROMO_ACTIVE = false flag in utils/stripe.ts. Code path stays intact — flip the flag to revive. Strips the now-untruthful "$10 on enabling billing" copy from BillingCard, EnableRouterPrompt, triggerWorkflow paywall comment, action router_requires_card summary, email snippet, billing/pricing docs and wiki. Cuts WELCOME_CREDIT_CENTS from 2000 to 1000 to reflect the lower amount that would land if the flag is ever re-enabled. Adds "signup" reason mapping to BillingCard wallet history. Verified end-to-end against dev: admin+Router fires modal, admin+BYOK gate-blocks mount, BYOK→Router transition fires modal on click, member and collaborator paths skip the mount entirely, reload after grant is idempotent. Wallet history shows "Router signup credit +$10.00". * billing: address PR review (race fix, copy sweep, modal retry) Correctness: - Add @@unique([accountId, reason]) on CreditGrant + migration. The prior check-then-insert pattern in /signup-credit/claim and finalizeCheckoutSession raced at READ COMMITTED — two concurrent admin tabs could land two grants of the same reason on a fresh account ($10 each). Both write sites now rely on the unique index for dedup (P2002 = "already granted") and route updated to catch P2002 cleanly. Verified zero existing duplicates in prod before migration. - Add log.info on signup grant insert so a successful grant has any chance of being caught by ops monitoring. - Add retry: 2 with backoff to the claim mutation. Endpoint is idempotent so a server-side success that lost its response cleanly returns granted:false on retry. Public copy that still advertised the (now-deleted) $20 welcome credit: - app/page.tsx landing pricing card - emails/announceBilling.ts broadcast template - docs/keys.mdx BYOK note - components/AgentSettings.tsx Router-without-billing warning - utils/stripe.ts finalizeCheckoutSession JSDoc - utils/email/snippets.ts ROUTER_CREDIT_PS_HTML JSDoc Wiki staleness sweep: - wiki/billing.md TOC, mermaid diagram (signup edge added; welcome marked dormant), test coverage list, key modules section, no-card wallet narrative - wiki/pricing.md welcome-credit drawdown reference - Rewrote my own internally-inconsistent dormancy paragraph to be honest about the $20-historical / $10-on-revival framing. Trivia: - ModelAccessCard JSX comment had a literal \\u2192 instead of →. * billing: address PR review round 2 - Replace try/catch P2002 inside finalizeCheckoutSession's prisma.$transaction with createMany skipDuplicates. The previous form is broken on Postgres: a unique-violation poisons the surrounding TX, so the catch block returns cleanly but the outer commit fails and the account.update (stripeCustomerId) silently rolls back too. Currently armed only behind the dormant welcome- credit flag, but would have broken billing enablement the moment the flag flipped. createMany skipDuplicates yields a single ON CONFLICT DO NOTHING statement that returns count: 0 cleanly without aborting the TX. - Apply the same createMany skipDuplicates pattern to the signup-credit route too — drops the exception-as-control-flow Prisma namespace import and is more uniform with the welcome path. - Drop the now-orphaned credit_grants_accountId_idx in the same migration. The schema removed @@index([accountId]) when @@unique([accountId, reason]) was added (covered by the leftmost prefix), but the migration only added the unique index, leaving prod drifted. * billing: fix stale finalizeCheckoutSession JSDoc The function-level JSDoc still described the abandoned try/catch P2002 mechanism after switching to createMany skipDuplicates. The inline comment + code now agree on the new ON CONFLICT DO NOTHING shape. * billing: decouple first-card alert, drop vestigial billing field, fix modal cents; sync copy * docs+homepage: align Router credit copy with signup claim (no card-on-add carrot) * homepage: add pricing screenshot and pay-as-you-go promo line * billing: fix once-per-lifetime misframe on first-card alert * billing: suppress signup credit for prior welcome-credit recipients * billing: drop bogus '1000 users' cap; invalidate billing on signup-credit settle |
||
|
Colin McDonnellandpullfrog[bot]
|
f0805b78f5 |
learnings: surface persist failures as warnings, not debug
`persistLearnings` only emitted `log.info("» learnings updated")` on
success; every failure path (non-2xx, fetch throw, 10s timeout) was
`log.debug`, which is hidden unless `ACTIONS_RUNNER_DEBUG=true`. Survey
of recent runs caught at least one case where the agent definitively
edited the tmpfile but no DB row was written and no warning surfaced.
Promote both failure paths to `log.warning` so dropped agent work is
visible in CI logs. The unchanged-from-seed short-circuit stays at
debug — that's a genuine no-op.
|
||
|
Colin McDonnellandpullfrog[bot]
|
e20b4d5515 | action: bump to 0.1.4 | ||
|
Colin McDonnellandpullfrog[bot]
|
e4d0fc7e3d | biome: ignore .logs/ (was matching only logs/) | ||
|
Colin McDonnellandpullfrog[bot]
|
a4a5010441 |
gemini-3: default thinkingLevel to medium + restrict eager prep to frozen install (#663)
* gemini-3: default thinkingLevel to medium + don't `npm ci` without a lockfile upstream opencode hardcodes `thinkingLevel: "high"` for every gemini-3 model on the direct google SDK (see `packages/opencode/src/provider/transform.ts` `options()`). that added 30-60s of pre-tool-call TTFT and 5-46s of post-tool jabber per turn, which is overkill for the tool-routing decisions that dominate agentic loops — and the variance caused the `providers-live (google/gemini-pro)` smoke job to time out at 4 minutes (see job 75405504847 on run 25684766415). three changes: - inject `provider.google.models.<api-id>.options.thinkingConfig.thinkingLevel = "medium"` for the two curated gemini-3 slugs in `buildSecurityConfig`. deep-merges over the upstream default; explicit `--variant high` / user opencode config still wins. flash stays at medium too — low-effort flash is visibly worse and the latency win isn't meaningful (flash is already fast). - bump the `providers-live` harness step from 4 → 6 minutes. the job-level 8-minute cap stays as the upper bound, but gemini's intrinsic TTFT variance was eating most of the 4-minute slack on its own. - in `installNodeDependencies`, pick `frozen` only when a lockfile was actually detected. previously a package.json-only repo (like the smoke fixture's `pullfrog/test-repo`) always triggered `npm ci` and emitted a noisy `EUSAGE` error before falling through. * prep: skip eager install when neither lockfile nor `packageManager` field present the previous commit changed the no-lockfile path from `npm ci` (always errored `EUSAGE`, never wrote any artifact) to a successful `npm install`, which had an unintended side effect: it generated `package-lock.json` in the working tree, tripping the post-run dirty-tree gate. the agent then committed the lockfile and opened a real PR — and in the openai/gpt smoke run on PR #663, the agent overwrote the `SMOKE TEST PASSED` output with the PR URL, failing the smoke validator. a repo with `package.json` but no lockfile and no `packageManager` field has not committed dependency state. eagerly installing produces state the repo doesn't track, which is the dirty-tree problem above. skip the eager install entirely in that case; the agent can opt in via `await_dependency_installation` when it actually needs deps. repos with a lockfile or a `packageManager` field keep the existing frozen-install behavior unchanged. * post-run: suppress dirty-tree gate in non-committing modes (Review / IncrementalReview / Plan) the dirty-tree post-run gate currently fires for every mode and tells the agent to commit and push whatever is in the working tree. that's wrong for modes that complete by submitting a review (`Review` / `IncrementalReview`) or posting a Plan comment (`Plan`) — those modes never touch files as part of their contract, so any tree dirt at end-of-run is incidental tool noise on an ephemeral worktree. nudging the agent to commit it can produce a spurious PR, as seen in the openai/gpt smoke run on PR #663 where a stray `package-lock.json` from `npm install` led the agent to open pullfrog/test-repo#32 and overwrite the smoke output. introduce `NON_COMMITTING_MODES` in `action/modes.ts` and consult it in `collectPostRunIssues`. when the selected mode is read-only, log the suppression for visibility but skip populating `issues.dirtyTree`. modes that legitimately commit (`Build`, `AddressReviews`, `Fix`, `ResolveConflicts`, `Task`) keep the existing nudge. * prep: restore eager frozen-install, drop non-frozen fallback eager dependency prep is non-mutating by contract — it runs before the agent starts and any artifact it leaves in the tree (e.g. a generated `package-lock.json`) trips the dirty-tree post-run gate and can lead the agent to open a spurious PR (seen on the openai/gpt smoke run earlier in this PR). revert the previous skip-when-no-lockfile branch: that was the wrong layer to enforce the invariant. instead, run `frozen` (`npm ci` / `pnpm install --frozen-lockfile` / etc.) unconditionally and drop the `|| install` fallback that could silently mutate the tree when `frozen` is missing. frozen commands fail cleanly without writing artifacts when there's no lockfile, which is exactly the safety contract we want. repos that need a real install must opt in explicitly via a `setup` lifecycle hook. * review nits: single getGitStatus call, tighten gemini-3 override scope comment addresses two inline nits from the PR review: - `collectPostRunIssues` was calling `getGitStatus()` (spawns `git status --porcelain`) in both branches of the mode check. lift the call above the conditional and branch on the result; same behavior, one git invocation. - the JSDoc on `GEMINI_3_DIRECT_API_IDS` said the override applies "across the board," but the constant only covers the two curated slugs in `action/models.ts`. tighten the wording to call out that other gemini-3 ids in models.dev keep the upstream "high" default. skipped the bot's yarn-1 concern after reading yarn 1's `install.js`: `bailout()` (lines 461-465) throws `frozenLockfileError` when `frozenLockfile && (!lockfileClean || missingPatterns.length > 0)`, which fires before `linker.init()` writes node_modules or runs lifecycle scripts. the existing comment's claim that frozen commands fail without artifacts holds for yarn 1 too. |
||
|
Colin McDonnellandpullfrog[bot]
|
cf94773bf0 |
modes: make task-list authoring the explicit first step in every mode checklist (#665)
* modes: make task-list authoring the explicit first step in every mode checklist The system prompt already instructs the agent to author an internal task list at the start of every run (action/utils/instructions.ts:291), but the rule lives several hundred tokens above the agent's first decision point and references the mode's checklist before the agent has it. Compliance is roughly coin-flip across opus runs — PR #610 dead-air for 9m20s was the extreme case; my own #664 e2e runs split 1-for-1 on `todowrite` compliance. Putting the directive *inside* the checklist that `select_mode` returns co-locates instruction with referent at the moment the agent decides what to do next. Same vocabulary as the existing rule (`task list`, agent-agnostic; the harness already maps to `todowrite`/`TodoWrite` per-agent in agents/opencode.ts and agents/claude.ts). The directive is deliberately non-prescriptive about list contents — the agent authors items based on the work it's about to do, not from a hand-shaped template. Touches all 8 built-in modes and the PlanEdit override: - Build / AddressReviews / Review / IncrementalReview / Plan / Fix / ResolveConflicts / Task: inserts `1. **task list**: create your task list for this run as your first action.` and renumbers existing steps. - action/mcp/selectMode.ts: same insertion in the PlanEdit override checklist. - All internal step cross-references shifted +1 (`step 5` → `step 6`, `skip steps 3–4` → `skip steps 4–5`, etc.) across Review, IncrementalReview, and ResolveConflicts modes. One code-comment reference in IncrementalReview's preamble updated to match. Complements #664 (live progress streaming): streaming guarantees the user sees *something* regardless of compliance; this PR raises the ceiling on what they see when the agent does comply (clean numbered checklist tracking through the run instead of just the latest assistant message). 488 action tests pass; typecheck, lint, format all clean. * postRun: fix stale 'step 7' reference missed during +1 renumbering |
||
|
Colin McDonnellandpullfrog[bot]
|
8e36f76cfa |
postrun: thread AgentRunContext through the retry loop instead of repackaging (#652)
* postrun: thread AgentRunContext through the retry loop instead of repackaging
drop the per-gate plumbing in `runPostRunRetryLoop`: the loop now receives
`ctx: AgentRunContext` whole and reads `ctx.stopScript` + `ctx.toolState.*`
directly. `getUnsubmittedReview` becomes a pure utility in postRun.ts
instead of a closure shipped over `AgentRunContext`. `AgentRunContext`
loses 4 fields that duplicated `toolState` (`summaryFilePath`,
`summarySeed`, `learningsFilePath`, `getUnsubmittedReview`) and gains
`toolState: ToolState`. both harness call sites collapse from 11 lines to
7; main.ts deletes the inline closure.
`ToolState` and friends move from `action/mcp/server.ts` to
`action/toolState.ts` so non-MCP code (agents, post-run loop) stops
importing run-state types from the MCP server module.
no behavior change. 503/503 tests green.
* toolState: relocate `CommentableLines` to break dep cycle with mcp/review
`action/toolState.ts` was importing `CommentableLines` from
`mcp/review.ts`, which pulled the entire MCP server compile graph (24
files) into any consumer of `ToolState` — including `cf-worker-indexing`
via the `pullfrog/internal` re-export chain through `utils/log.ts` →
`agents/shared.ts` → `toolState.ts`. that exposed a pre-existing TS
error in `mcp/issueEvents.ts` (octokit types resolve differently under
cf-worker's `moduleResolution: bundler`).
move `CommentableLines` (a small `{ RIGHT: Set<number>; LEFT: Set<number> }`
state-shape type) to `toolState.ts` where it's used; re-export from
`mcp/review.ts` for back-compat with test and call-site imports. cuts
cf-worker's mcp/ compile inclusion from 24 files back to 0.
* postRun: drop mock-heavy retry-loop tests; keep pure gate predicate
`runPostRunRetryLoop` and `executeStopHook` were covered by ~560 lines
of mock-heavy regression-gate tests that stubbed `spawn` / `getGitStatus`
and fabricated `AgentRunContext` to drive orchestration paths. per
AGENTS.md ("prefer no test over a mock-heavy test that only catches the
most obvious form of regression") and the empirical track record — the
one real production failure of this code path (#646) was a missing npm
release, not a logic bug a unit test could catch — the value-to-ceremony
ratio is poor. delete them.
keep only the pure predicate: `getUnsubmittedReview(toolState)` is a
decision function whose four input conditions have user-visible
consequences when wrong. 5 assertions, no mocks, no ctx fabrication.
488 tests still pass.
* toolState: import PrepResult from prep/types.ts, not the barrel
same dep-cycle class as the previous CommentableLines fix. importing
PrepResult from prep/index.ts pulled prep/installNodeDependencies.ts
into the Next.js production build's typecheck graph (via
pullfrog/internal → utils/log.ts → agents/shared.ts → toolState.ts →
prep/index.ts → installNodeDependencies.ts), and Next.js's stricter
NODE_ENV-required ProcessEnv shape rejected an existing
`env: { PATH: ... }` literal.
prep/types.ts is a leaf module with zero imports — re-routing the type
import severs the chain. Vercel preview deploy goes from Error → Ready;
preview-sync stops racing the deploy.
|
||
|
Colin McDonnellandpullfrog[bot]
|
dee13b160f |
console: case-insensitive owner/repo slug resolution (#649)
* console: case-insensitive owner/repo slug resolution
URL slugs may be any case but GitHub treats logins and repo names as
case-insensitive (and 301-redirects to canonical case). Internal
find/filter sites compared with `===`, so mixed-case slugs (e.g.
`/console/Pullfrog`) hard-403'd in resolveOwnerAccess and silently
redirected from the per-repo console when currentRepo lookup missed.
Lowercase both sides at every slug comparison: resolveOwnerAccess
installation lookup, currentRepo lookup in repo + history pages,
ConsoleHeader installation/repo lookups, getInstallations personal
split, getOrgMembership user/org checks, getInstallationRepos node
filter, getUserRole owner-as-collaborator check, and the action
runtime's installation-repo access check.
Caches keyed by raw input remain case-split across casings; that's
fine since both entries resolve to the same canonical GitHub data and
TTLs are short.
* api: resolve targetAccountId by gh node id
getAuthenticatedAccountContext was looking up Account by `name` using
the raw URL slug, but `Account.name` is plain String populated from
canonical GitHub login. Mixed-case URLs would render the page (since
resolveOwnerAccess is now case-insensitive) but every billing/secrets
API call would 403 on the find-by-name miss.
Resolve by gh_${access.installation.account.node_id} instead — invariant
to case-folding and login renames. Same pattern as the sibling owner
page route already uses.
|
||
|
Colin McDonnellandpullfrog[bot]
|
ef394277c1 |
review: synthesize [!NOTE] informational tier with #644 alert judiciousness — 4-callout visual ladder + approved Fix-gate (#653)
* review: NOTE-tier callout + `actionable` flag to suppress Fix buttons
Adds an `actionable` parameter to the `create_pull_request_review` tool
(defaults true) so the agent can opt out of the Fix-it/Fix-all/Fix-👍s
footer affordance on informational reviews. Threaded through
`createAndSubmitWithFooter` so the buttons are omitted when
`actionable: false`.
Updates `Review` and `IncrementalReview` mode prompts with a 4th tier:
`> [!NOTE]` + `actionable: false` for mergeable, FYI-style observations
(prior feedback addressed cleanly, minor stale doc reference, etc.).
Calibration note: `[!IMPORTANT]`/`[!CAUTION]` are reserved for findings
that warrant code changes, because that's what trains users to click
Fix. `[!NOTE]` reviews must not carry inline comments — if a point is
concrete enough to anchor to a line, upgrade the whole review tier.
* review: drop redundant `actionable` flag, key Fix buttons off `approved`
`approved` already encodes "this PR is mergeable, nothing for the Fix
button to act on" — `actionable` was a second flag carrying the same
signal. Drop it from the tool schema and `FooterOpts`; the footer gate
stays `if (!opts.approved)` (unchanged from pre-PR behavior, with a new
comment documenting the UX rationale).
NOTE-tier reviews now use `approved: true` + `> [!NOTE]` body instead of
`approved: false` + `actionable: false`. For repos with
`prApproveEnabled: false`, the runtime already downgrades APPROVE to
COMMENT, so the GitHub-side shape is identical to the prior design.
* review: address Pullfrog feedback — drop ambiguous parenthetical + update postRun nudge
- Review-mode calibration: drop the "(or no callout at all)" parenthetical
that didn't map cleanly to a bullet; replace with explicit "both the
`[!NOTE]` tier and the 'no actionable issues' tier below use approved:
true" so the bullet-list anchor is obvious.
- `buildUnsubmittedReviewPrompt` (Review mode): the fallback nudge for
unsubmitted reviews now defers to the mode prompt's tier matrix and
acknowledges that `> [!NOTE]` informational reviews submit with
`approved: true` alongside the canonical "No new issues found." path.
Previously the nudge only described the pre-NOTE binary world.
|
||
|
Colin McDonnellandpullfrog[bot]
|
96910f0f50 |
fix(run-audit): drop summary comment, fall back to agent final message in job summary
the audit agent's final 'post a short summary' instruction was ambiguous and, with no PR/issue context on schedule runs, caused the agent to invent a target — landing the summary as a comment on the most recent open PR (see #650). drop the comment instruction outright. writeJobSummary now falls back to the agent's final assistant message (result.output) when lastProgressBody is empty, so non-PR runs surface a real summary in the GitHub Actions job summary tab instead of just the usage table. lastProgressBody still wins when present to avoid duplicating the progress comment body. |
||
|
Colin McDonnellandpullfrog[bot]
|
4cc6d95a91 |
ci: split per-alias resolution smoke from per-provider harness smoke (#650)
* ci: split per-alias resolution smoke from per-provider harness smoke `models-live` previously ran the full Pullfrog harness (Docker + MCP + agent + structured-output validation) once per alias on every PR that touched `models.ts` or `agents/**`. That cost minutes and dollars per alias and re-validated tool-calling for every routing wrapper. The per-alias signal we actually need from `models.ts` changes is just "does this alias resolve and authenticate." Tool-calling correctness is a property of the underlying model, not the alias, and it doesn't change when someone adds a row to the catalog. Splitting the two concerns: - `models-live` now runs `action/test/model-smoke.ts` per alias — a top-level CLI invocation (`opencode run -m <resolve> "reply OK"` or `claude -p "reply OK" --model <bare>`) with no Docker, MCP, or Pullfrog harness. Validates resolution + auth in seconds at fractions of a cent. Lets us drop the `EXPENSIVE_RESOLVE_SUBSTRINGS` carve-out for `gpt-pro` since the cheap smoke covers it for free. - `providers-live` (new) runs the full harness smoke once per provider against a hand-curated standard-tier model (`anthropic/claude-sonnet`, `openai/gpt`, `google/gemini-pro`, `xai/grok`, `deepseek/deepseek-pro`, `moonshotai/kimi-k2`, `opencode/big-pickle`, `openrouter/claude-sonnet`). Catches provider-class regressions like the Gemini schema sanitizer or OpenAI tool-call format drift. ~8 jobs, ~$0.40/push, ~4min critical path in parallel. Net change per push that touches `models.ts`: ~$20 → ~$0.40. `list-aliases.ts` now branches on `MODE` to emit either matrix; the flagship list asserts each slug exists in `modelAliases` so renames break CI loudly. Wiki updated to reflect the new two-tier coverage and the operational rule for new Gemini aliases (cheap smoke covers auth, manual harness run still needed for sanitizer compatibility on non-flagship Gemini additions). * fix(model-smoke): walk fallback chain; address pr review comments - model-smoke now uses `resolveCliModel(slug)` instead of `alias.resolve` so deprecated aliases (those with `fallback` set, e.g. `opencode/mimo-v2-pro-free` → `opencode/big-pickle`) hit the replacement model the way production does. mimo-v2-pro-free was failing CI because the underlying opencode model is dead — the fallback chain is the whole point of marking it deprecated. - tighten stale `agentForSlug()` reference in model-smoke.ts comment (function was deleted in this same PR; classification is now inline in `list-aliases.ts toMatrixEntry`). - tighten `FLAGSHIPS` drift comment to call out that the assertion is one-way (catches slug-rename, but silently omits new providers). Update wiki step 4 of "To add a provider" to require adding the standard-tier slug to `FLAGSHIPS` for harness coverage. * docs: scrub stale env-knob refs in models-catalog parity section `wiki/models-catalog.md` cross-provider parity paragraph still pointed at `INCLUDE_ALL_PASSTHROUGHS` / `INCLUDE_EXPENSIVE` and the implicit filter→expensive-gate coupling — all removed in this PR. Aligned the copy with Step 9 (which was already updated): `INCLUDE_PASSTHROUGHS`, no expensive gate, `MATRIX_FILTER` applies to both aliases and flagships modes. |
||
|
Colin McDonnellandpullfrog[bot]
|
85d25a6fe6 |
post-run gate: fail review-mode runs that don't submit a review or progress (#638)
* post-run gate: fail the run when review mode finishes without a review or progress
review-mode runs that ended in a text-only assistant turn ("now I have enough
to draft the review...") were silently swallowed: the progress comment was
deleted by stranded-comment cleanup and no review appeared on the PR. user-
visible result was identical to "the agent never ran." caught in
https://github.com/pullfrog/app/actions/runs/25583698781.
new post-run gate alongside stopHook / dirtyTree / summaryStale: derived
inline from toolState (selectedMode in {Review, IncrementalReview} && !review
&& !finalSummaryWritten && hadProgressComment) — no parallel toolState flag.
when it fires, the resume prompt nudges the agent to call either
create_pull_request_review or report_progress; persistent failure after
MAX_POST_RUN_RETRIES surfaces as AgentResult.error.
also: when the post-run loop returns success=false, write the error to the
progress comment before the stranded-comment cleanup runs, and skip the
delete in that case. previously a !success run from the loop would lose the
error message into the void.
IncrementalReview's trivial-skip branch now calls report_progress with a
brief "no review warranted" note instead of exiting silently — keeps the
contract symmetric with the gate and gives the user a visible signal even
on no-op review runs.
documents the literal-record design rule on the ToolState interface so
future fields don't drift back into derived/absence-encoding state.
* review feedback: mode-aware nudge, gate-error preservation, prompt order
addresses three findings from the auto-review on this PR:
1. Review mode nudge no longer offers `report_progress` as an exit. Review
mode's contract (modes.ts step 5) forbids it; the gate previously sent
contradictory copy. IncrementalReview's nudge still offers both since
its trivial-skip path legitimately allows `report_progress`.
2. `writeJobSummary` is now wrapped in try/catch on the success-path
cleanup. without this, a throw there jumped to the outer catch and
overwrote the gate's failure message in the progress comment with the
(less actionable) writeJobSummary error — restoring exactly the
invisible-failure UX this PR fixes. step-summary writes are
informational; let them fail silently.
3. `buildPostRunPrompt` reorders gates to match the terminal hard-fail
order: `stopHook` → `unsubmittedReview` → `dirtyTree` → `summaryStale`.
when both hard-fail gates co-fire (rare in review modes), the prompt's
emphasis now matches the user-visible failure message.
new test asserts the IncrementalReview nudge offers both exits while the
Review nudge offers only `create_pull_request_review`. e2e validation
already passed against pullfrog/preview-638-review-stop-hook PR #1
(gate fired once; agent recovered on second turn).
* mode-aware terminal error copy
second auto-review caught a residual contradiction: the terminal hard-fail
error string reported "create_pull_request_review or report_progress" for
both modes, even though the new mode-aware nudge tells Review-mode agents
"Review mode does not have a no-submit exit". the error message now mirrors
the nudge — Review names only `create_pull_request_review`,
IncrementalReview lists both. additional Review-mode hard-fail test asserts
the absence of `report_progress` in the error.
|
||
|
Colin McDonnellandpullfrog[bot]
|
363e4cbed8 |
ci: gate gpt-5.5-pro by resolve, refresh stale matrix docs (#639)
* address review: gate by resolve, refresh stale doc claims - list-aliases.ts: gate EXPENSIVE on alias.resolve substring (catches opencode/gpt-pro and openrouter/gpt-pro, which both resolve to a gpt-5.5-pro variant — would have re-entered the matrix under INCLUDE_ALL_PASSTHROUGHS=1 and tripled the cost). - test.yml + models-catalog.md: stop describing the matrix as exhaustive. Mention pruning + INCLUDE_EXPENSIVE/MATRIX_FILTER opt-ins. * address review: clarify env vars are local-only, filter input is the CI knob Pullfrog review caught that wiki/models-catalog.md was advertising INCLUDE_EXPENSIVE / INCLUDE_ALL_PASSTHROUGHS as workflow_dispatch knobs — they're not, only `filter` (→ MATRIX_FILTER) is wired through. The filter coupling already implicitly opens the expensive gate, so dispatch + filter is the canonical CI path. |
||
|
Colin McDonnellandpullfrog[bot]
|
c8888cecde | bump action version to 0.1.2 | ||
|
Colin McDonnellandpullfrog[bot]
|
c0de70431e |
ci: prune openai/gpt-pro from default models-live matrix (#637)
* ci: prune openai/gpt-pro from default models-live matrix gpt-5.5-pro burns ~$2.40/run ($30/M input, $180/M output) — flagship reasoning tier with hidden reasoning tokens dominating cost. Multiplied by every push that touches a resolution-affecting file, the bill is untenable for a smoke that just verifies set_output works. Pruned by default; re-enable with INCLUDE_EXPENSIVE=1 or MATRIX_FILTER when validating the alias on demand. Also adds a comment-frugality rule to AGENTS.md. * ci: include list-aliases.ts in models paths-filter The matrix builder is resolution-affecting from a validation standpoint — a regression to it (e.g. accidentally pruning all aliases) wouldn't trigger models-live on its own commit. |
||
|
Colin McDonnellandpullfrog[bot]
|
b0274e3265 |
local proxy-key testing via x-dev-repo bypass (#629)
* local proxy-key testing via x-dev-repo bypass `pnpm play` previously couldn't exercise the proxy/router/oss code path — `resolveProxyModel` early-exits without OIDC credentials, and `mintProxyKey` always sends an OIDC bearer to `/api/proxy-token`. since GitHub Actions OIDC only exists in real workflow runs, billing flows (auto-reload, balance gates, key rotation, OSS subsidy) had no local feedback loop. a server-side dev bypass already exists at `app/api/proxy-token/route.ts` that accepts an `x-dev-repo: owner/repo` header instead of an OIDC bearer when `NODE_ENV === "development"`. wire the action side so it sends that header when there are no OIDC credentials AND `API_URL` resolves to localhost (i.e. the developer is talking to their own `pnpm dev` server). production is unreachable through this path because vercel never sets `NODE_ENV=development`. document the affordance in `wiki/action-tests.md` so the next person doesn't have to re-discover it (the server bypass had been sitting there undocumented since the WIP billing rewrite). verified end-to-end: `PLAY_LOCAL=1 GITHUB_REPOSITORY=pullfrog/app API_URL=http://localhost:3100 pnpm play …` now logs `» proxy: dev bypass (x-dev-repo) for pullfrog/app` → `» proxy: router → openrouter/ anthropic/claude-opus-4.7` → `» model: …(proxy)`, mints a real OpenRouter key against the dev DB, and the agent runs through the proxy. * wiki: cross-reference dev proxy-key affordance from main/e2e/stripe action-tests.md already documents the localhost+x-dev-repo path; mention it from the natural discovery points so the next person finds it without spelunking through git history again: - main.md: resolveProxyModel row in the dependencies table notes the two auth paths (OIDC bearer in prod, x-dev-repo in dev). - e2e-testing.md: "When to use this" calls out the lighter-weight alternative for proxy-only changes. - stripe.md: new "Loop including the action" subsection in the Dev workflow section, alongside the existing dev-script and cron-endpoint loops. |
||
|
Colin McDonnellandpullfrog[bot]
|
8f36eca62a | action: use log.success for skill install confirmations | ||
|
Colin McDonnellandpullfrog[bot]
|
3c9799adda |
add models-bump cron + drop snapshot test
every 12h, scripts/find-newer-models.ts scans models.dev for newer GA versions of every alias in action/models.ts and writes a focused per-alias diff. .github/workflows/models-bump.yml short-circuits when no candidates exist; otherwise hands the diff to pullfrog/pullfrog@main to evaluate against the policy in wiki/model-resolution.md and open a single living PR on the pullfrog/models-bump branch. drops the brittle "latest model per provider" snapshot block in action/test/models-catalog.main.test.ts (and its .snap file) — the cron keeps the registry in sync with upstreams, and the remaining validity tests act as the integrity gate on the bump PR. |
||
|
Colin McDonnellandpullfrog[bot]
|
5f3e46c42d |
fix: don't reuse disabled proxy key on workflow re-runs; non-fatal title-gen errors (#636)
* fix: don't reuse disabled proxy key on workflow re-runs; non-fatal title-gen errors Three small surgical fixes addressing run https://github.com/pullfrog/app/actions/runs/25580969379: 1. **`/api/proxy-token` idempotency now checks `finalizedAt`.** GitHub re-runs share the same `run_id` (only `run_attempt` increments), so attempt N+1's action calls /api/proxy-token and inherits attempt N's `proxyKeyId`. The `workflow_run.completed` webhook between attempts retires that key on OpenRouter (`disableKey`), so attempt N+1 was getting back a disabled key and OpenRouter responded with `401 User not found` on every call. Falling through when finalized routes through the same billing gate (`handleRouterBilling` balance check), so no new attack surface. 2. **OpenCode title-gen / small-model errors no longer fatal.** OpenCode auto-spawns a small `agent=title small=true` background call at session start to name the thread, defaulting to `anthropic/claude-haiku-4.5` (anomalyco/opencode#1243). Pre-fix, the wrapper's `error` event handler treated any `type=error` as fatal, so a cosmetic title failure killed the run before primary inference even started. Now: stderr matching `small=true` sets a one-shot suppression flag for the next stdout `error` event, which is logged as a warning instead. 3. **Provider-error classifier puts auth patterns above rate-limit.** OpenRouter 401 payloads bundle `x-ratelimit-*` response headers, and the loose `\brate[_ ]limit/i` pattern was winning. Added 401/403 status, `User not found`, `Invalid authentication`, `No auth credentials found` patterns ahead of rate-limit. Updated the existing 401-headers regression test to assert correct auth classification rather than `null`. * opencode: correlate small-model error suppression by message, not by next-event Pullfrog self-review on #636 flagged a real concurrency hole. OpenCode forks the title-gen call (`session/prompt.ts:1452-1457` via `Effect.forkIn(scope)`) so it races primary inference. The previous one-shot `suppressNextErrorEvent` boolean had no per-call correlation: it was consumed by whichever stdout `type=error` event landed next, regardless of which subagent produced it. Under concurrent failures, a primary-agent error landing first could be silently downgraded to a warning while the small-model error then propagated fatally — the inverse of the bug the suppression was meant to prevent. Replaced the boolean with a `Set<string>` of pending small-model error messages. stderr extracts the inner `"message":"..."` from any classified provider error tagged `small=true`; the stdout `error` handler suppresses only when `event.error.data.message` matches a pending entry. Set is capped at 32 entries so a long stream of small-model failures can't wedge memory. Also corrected the comment that referenced "session summarizer" — verified in opencode source that summarize() does NOT use `small: true`; only the title generator does today (only `small: true` match in the codebase). * revert: drop opencode title-gen suppression We have no evidence — and can't construct a realistic scenario — where title-gen fails on an otherwise-successful run. Title-gen and primary share the same OPENROUTER_API_KEY and hit the same proxy/upstream; whatever breaks one breaks the other. The original repro on run 25580969379 is fully explained by the stale proxy key (fix #1) — title-gen happened to be the first call that surfaced the auth error, but every subsequent primary call would have died the same way. Suppression code adds complexity (cross-stream correlation logic, message matching, set capping) and a real failure mode of its own (a small-model error with a unique message could mask an unrelated primary error landing shortly after). Net negative. Removing. |
||
|
Colin McDonnellandpullfrog[bot]
|
3d393c36a3 |
opencode: surface subagent events via injected plugin (#634)
* opencode: surface subagent events via injected plugin opencode's cli/cmd/run.ts event loop filters all message.part.updated events to the orchestrator's session id (`part.sessionID !== sessionID` continue), so subagent-internal tool_use / text / step events were silently discarded by the CLI in --format json mode. opencode plugins, by contrast, receive every bus event via bus.subscribeAll() regardless of session. ship a per-run plugin (action/agents/opencodePlugin.ts) that re-emits non-orchestrator message.part.updated events as `pullfrog_bus_event` envelopes on opencode's stdout. the plugin is staged into <XDG_CONFIG_HOME>/opencode/plugin/pullfrog-events.ts which is already redirected to ctx.tmpdir — never the user's repo working tree. the plugin also forwards the orchestrator's task tool dispatch at state.status="running" — that's the first moment state.input is populated with description / subagent_type / prompt and it lands BEFORE the subagent's first message.part.updated. forwarding this lets SessionLabeler register the lens label early, so subagent events bind to the correct lens name (e.g. lens:correctness) instead of the subagent#N fallback. the existing tool_use handler dedupes on callID so the late status=completed event from the CLI doesn't double-record. the parent's pullfrog_bus_event handler synthesizes the equivalent CLI-style event for each part type (tool/step-start/step-finish/text) and dispatches through the same handlers used by orchestrator events, so labeling, tool-call rendering, and the formatWithLabel magenta prefix all share one code path. verified end-to-end via `pnpm play --local --raw` with a prompt that dispatches a reviewfrog subagent: orchestrator's task call now logs "» dispatching subagent: lens:read-readme-and-report-purpose" before the subagent runs, the subagent's read tool call surfaces with [lens:...] magenta prefix, and the run-end "subagent finished" attribution shows the lens name. also adds an AGENTS.md rule formalizing the no-write-to-repo invariant: action runtime must never write into the user's working tree; auxiliary files go in ctx.tmpdir via HOME / XDG_CONFIG_HOME. * drop opencodePlugin.test.ts — bullshit-test cleanup these tests spied on process.stdout.write, loaded the plugin source into a temp file via dynamic import, and asserted the output strings matched the plugin source i'd just hand-written. zero unique signal over the e2e run in preview repo, plus they violate AGENTS.md's "mocks tend to add ceremony and brittleness" rule. real signal lives in the e2e: lens label rendering, dispatch attribution, no double events. if a syntactic regression in the plugin source ever ships, opencode logs it on plugin load and the e2e fails fast — the unit tests would catch the same regression no faster. * remove isPausedExternally — plugin makes it unnecessary empirical proof from PR #634's e2e debug trace: ~3.3 pullfrog_bus_event lines per second arrive on the parent's child.stdout pipe during a typical subagent run. each one fires updateActivity() and resets lastActivityTime, so the inner spawn activity timer naturally stays armed-but-not-fired throughout the subagent's lifetime — no suspend predicate needed. drop: - SpawnOptions.isPausedExternally + the check in spawn()'s activity loop - isSubagentInFlight() in opencode.ts + its callsite - two isPausedExternally unit tests in subprocess.test.ts keep: - killGroup (the actual zombie-prevention fix; still tested) - the plugin (action/agents/opencodePlugin.ts; the architectural fix) - everything in opencode.ts that derives lens labels from task dispatches the only edge case isPausedExternally covered that the plugin doesn't is a non-streaming provider going silent for >5min during a single LLM call inside a subagent. that's a provider-behavior question, not a harness-architecture one — best fixed at the provider level if it shows up. defense-in-depth that adds indirection is harmful when the upstream architectural fix is already in place. * opencode: address review feedback on bus envelope routing three findings from PR #634 review (2026-05-08T22:13:44Z): 1. token/cost double-count: routing subagent step_finish through the orchestrator's handler folded subagent tokens/cost into the run-wide accumulators that flow to logTokenTable + AgentUsage. neighbouring init/text handlers all gate on ORCHESTRATOR_LABEL for exactly this reason. fix: drop step_start AND step_finish from the bus envelope handler — those carry orchestrator-scoped state (currentStepId, stepHistory, token accumulators) that subagent events shouldn't touch. tool calls and text from subagents still surface — that's the user-visible activity. 2. subagent tool errors invisible: routed status="error" tool parts into handlers.tool_use which only emits "» <tool>(...)" with no error indication. fix: extend handlers.tool_use itself to log "» tool call failed: <msg>" when state.status==="error". benefits the orchestrator path too — opencode CLI also emits failed tool calls as tool_use at status=error and we were swallowing the failure signal there as well. 3. stale comments + leaked local paths: plugin source had /tmp/opencode-investigate/... paths from my local clone, specific line numbers from opencode's dev branch that don't match v1.1.56, forkDetach claim that's wrong for the pinned version, and JSDoc that still listed message.updated/session.error in the forwarded set after the runtime filter narrowed to message.part.updated only. fix: drop machine-local paths, drop version-fragile line numbers, correct the forwarded-set list, generalize the "why no @opencode-ai/plugin import" rationale to be version-agnostic. second review (2026-05-08T22:27:58Z) confirms these are the only findings still open — no new issues from the isPausedExternally removal. |
||
|
Colin McDonnellandpullfrog[bot]
|
d6de1c369a |
learnings: edit-in-place tmpfile (drop update_learnings tool) (#635)
* learnings: edit-in-place tmpfile (drop update_learnings tool)
learnings now follow the PR-summary file pattern: server seeds
`pullfrog-learnings.md` from `Repo.learnings` at startup, agent reads
it as part of context, may edit in place during the post-run reflection
turn, server reads back at end-of-run and PATCHes if changed.
motivation: `update_learnings` required the agent to pass the FULL
merged list as a string parameter — an output-token tax that grew
linearly with the learnings size, and a constant prompt-context
expansion since the contents were also inlined into the LEARNINGS
section. for repos with mature learnings the prompt was getting
visibly noisy in CI logs.
key changes:
- new `action/utils/learnings.ts` (seed/read helpers + 10k cap)
- `main.ts`: always seed; `persistLearnings` mirrors `persistSummary`
(success path, error path, exit-signal handler, idempotent guard,
byte-trim equality skip); forwards `model` for `LearningsRevision.model`
- `LEARNINGS` prompt section now contains only the file path + a
one-line "read it" instruction (no contents inlined)
- `update_learnings` MCP tool deleted; `action/mcp/learnings.ts` removed
- reflection turn (`buildLearningsReflectionPrompt`) reframed around
file editing with explicit prune-stale + leave-alone-if-nothing-new
framing
- `learningsStep` removed from every mode checklist — surface lives only
in the LEARNINGS prompt section + the reflection turn now
* learnings: harden seed step + refresh stale docs (review feedback)
Three findings from PR review, all implemented:
1. wrap learnings seed in best-effort try/catch (action/main.ts) —
the always-on seed block ran unconditionally and an unwrapped
`seedLearningsFile` (mkdir + writeFile) failure (ENOSPC, EACCES,
hostile sandbox) would unwind into the outer main() catch and flip
an otherwise-successful run to "❌ Pullfrog failed" before the
agent even started. asymmetric with `persistLearnings`'s own
best-effort contract. wrap and log on failure; downstream
consumers (`persistLearnings`, agent harnesses, `resolveInstructions`)
already handle `learningsFilePath: undefined` cleanly.
2. refresh wiki/main.md — `resolveInstructions` parameter renamed
from `learnings` to `learningsFilePath` in this PR; the data-flow
diagram and the resolver dependency table both still showed the
pre-refactor signature.
3. drop deleted `learnings.ts` from ROADMAP.md + RESEARCH.md
"missing MCP tool tests" bullets — `action/mcp/learnings.ts` was
removed in this PR; the bullets are otherwise still accurate.
|