shockbot

ShockVPN/shockbot

Fork 0

Commit Graph

Author	SHA1	Message	Date
Colin McDonnell	36ac64a5b6	fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy (#844 ) * fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy OSS-allowlisted repos with `CODEX_AUTH_JSON` uploaded via `pullfrog auth codex` were still being routed through the OSS OpenRouter subsidy because two paths ignored managed credentials: - `hasProviderKey()` only checked `provider.envVars`, so an `openai/` model with only `CODEX_AUTH_JSON` present silently fell back to `opencode/big-pickle` via `selectFallbackModelIfNeeded` — the maintainer saw "opencode/big-pickle (resolved from openai/gpt)" on CI even though Codex was configured. - `run-context` set `proxyModel` for every OSS run unconditionally, which the action runtime threads through `payload.proxyModel` and uses to overwrite `OPENROUTER_API_KEY`. Even if `big-pickle` fallback hadn't fired, the runner would consume the $10 OSS subsidy key instead of the user's ChatGPT subscription. Fix: - Add `getModelAuthEnvVars()` covering both `envVars` and `managedCredentials` in `action/utils/apiKeys.ts`; route `hasProviderKey` + `validateAgentApiKey` through it. - `run-context` now skips `proxyModel` for OSS repos when the configured model's provider has matching auth in Pullfrog-stored account/repo secrets, so the runner authenticates directly with the user's Codex subscription (or any other user-provided provider auth). Triggered by mrlubos (`hey-api/openapi-ts`). Companion follow-up tracked for the opaque "(no error message)" classifier swallow that masked the OSS $10 cap exhaustion on PR #3872 runs 25815370844 + 25815443234. fix(oss): force Kimi K2 for OSS proxy + hide picker in console UI OSS-funded runs were resolving `repo.model` through OpenRouter, so a single Opus / GPT-5.5 run could burn an entire `oss_subsidy` key against the per-key cap and crash mid-stream (e.g. `hey-api/openapi-ts` PR #3872 runs `25815370844` + `25815443234`, ~$9.20 each on a single key). Force `DEFAULT_PROXY_MODEL` (Kimi K2.6 — ~10-50× cheaper) for every OSS proxy mint, regardless of `repo.model`. Per-run spend stays bounded within the cap by structure, not by hope. `repo.model` stays in the DB unchanged — overriding at runtime means leaving the program restores the user's prior pick without a migration. UI: hide the model picker entirely on OSS repos in `AgentSettings`. The field is effectively inert until the repo leaves the program, so exposing it as if it were live was misleading. Replaced with a banner naming Kimi K2 and pointing to `pullfrog auth …` as the opt-out path — that lands the user on the existing #844 bug-2 branch (Pullfrog-stored auth suppresses the OSS proxy entirely; runner uses user credentials + their preferred model). ModelCostsInfo already has its own `isOss` branch for the cost copy, so that section is unchanged. * fix(oss): lowercase comment casing per AGENTS.md * fix(oss): revert banner copy to 'It's on us.' framing per review Maintainer felt 'Kimi K2' as the banner headline lost the warm 'we've got you covered' framing that the existing OSS cost banner uses. Restore 'It's on us.' as the headline, move the model name into the body where it explains the hardcoded choice and points to the opt-out (pullfrog auth codex / account secret). * docs(agents): screenshots must be of the live route, never synthetic Caught myself building a temp `/dev/oss-ui-preview` route with hardcoded JSX copy-pasted from the real component just to grab a screenshot — the result told us nothing about whether the actual integrated UI worked, and the user (rightly) called it out as a waste. Strengthen the rule: screenshots must come from the live route in the running app, driven by the actual component tree and real props. Note the GH OAuth interstitial gotcha so the next agent gets through Clerk → GitHub sign-in on the first try instead of bailing to a fake render. Also bans side-by-side comparison screenshots unless explicitly requested. * fix(oss): one 'It's on us.' banner, not two OSS Agent settings was showing the message twice — once in the Model section, once in the Model costs section right below it. Fold the cost coverage into the model banner ('at no cost to you' + the spend stat) and hide the Model costs subsection entirely for OSS. ModelCostsInfo no longer needs `isOss` / `ossSpendThisMonthUsd` props — call site is gated, so the OSS branch is dead. Removed it and the now-unused props. Non-OSS rendering is unchanged: full Model picker + Model costs subsection with Router / BYOK branches. * feat(action): corepack-aware package manager provisioning before setup customer setup scripts that did `npm i -g pnpm && pnpm install` were installing whatever pnpm "latest" happens to be on the day the run fires, not what the repo declares — and pnpm 11.3 silently writes a new `packageManagerDependencies` block into lockfiles, which the agent's "always push changes" rule then packages into a noisy PR (see #844). resolve the project's pnpm/yarn pin from `package.json` (honoring pnpm 11+ precedence: `devEngines.packageManager` over `packageManager`) and activate it via `corepack prepare ... --activate` BEFORE the setup hook runs. corepack is bundled with node, so this is a no-op on managed infra; failure (no corepack, no network, range-only version) degrades to a warning and the existing PATH binary still runs. also replaces the legacy `npm install -g <pm>@<v>` path in prep with the same helper so behavior is consistent end-to-end. bun/deno still use the legacy installer because corepack doesn't ship shims for them. * chore(console): drop 'npm i -g pnpm' anti-pattern from setup-script placeholder the suggested example trained customers to install pnpm unpinned, which silently picks up whatever's latest at run time. that's exactly the behavior #844 traced lockfile drift back to. now that prep handles package-manager provisioning via corepack from the repo's declared pin, the placeholder is just a frozen-lockfile install — load-bearing only when the repo wants `pnpm install` to actually run (prep already does that), but a much safer default for customers who do paste it in. * refactor(action): introspect opencode models for BYOK detection Replace the static `provider.envVars + provider.managedCredentials` catalog gate in `selectFallbackModelIfNeeded` + `validateAgentApiKey` with two `opencode models` captures around the auth merge: - `captureBaselineModels` BEFORE dbSecrets + Codex auth.json - `captureAuthorizedModels` AFTER both The authorized set is the authoritative source for "can OpenCode route this model" — strictly more accurate than the catalog, which can miss new auth shapes (Codex was one, there will be more). The diff between baseline and authorized is logged as `BYOK auth enabled N model(s)` for operator visibility. Sequencing changes in main.ts: - `createTempDirectory` hoisted out of the try block so `PULLFROG_TEMP_DIR` is set before the early opencode install - `agents.opencode.install()` + baseline capture before dbSecrets - `installCodexAuth()` hoisted up (idempotent — agent re-calls it inside run() and writes the same file) - authorized capture after Codex auth.json materializes - fallback + validateAgentApiKey receive the authorized set as a parameter; tests inject directly with no mocks Deleted: `hasProviderKey`, `getModelAuthEnvVars`, `knownApiKeys` in `action/utils/apiKeys.ts` (only `selectFallbackModelIfNeeded` consumed them, and PR #844's catalog-extension fix is superseded by introspection). `getModelEnvVars` / `getModelManagedCredentials` stay exported for UI and the server-side OSS proxy heuristic in run-context/route.ts. For the claude agent path, validateAgentApiKey keeps the static single-provider check on `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` — `opencode models` is opencode-specific. validateBedrockSetup / validateVertexSetup also stay; they cover region/location/model-id which `opencode models` doesn't catch. When fallback engages, the post-fallback model is the guaranteed-free `opencode/big-pickle`, so validateAgentApiKey is skipped — the fallback gate already authoritatively decided "this model is OK to run". * test(oss): temp add preview-844 to ossRepos for O4 e2e * Revert "test(oss): temp add preview-844 to ossRepos for O4 e2e" This reverts commit 8167e560126b2ac516c32ba1c63c36aa32ae4019. * test(oss): temp add preview-844 to ossRepos for O5 e2e * fix(action): skip validateAgentApiKey when proxyModel is set The new opencode-models BYOK introspection in PR #844 captures the authorized set BEFORE runProxyResolution mints OPENROUTER_API_KEY, so the proxy slug (e.g. `openrouter/moonshotai/kimi-k2.6`) is never in the set. validateAgentApiKey then spuriously threw "no API key found" on every OSS run, even though the proxy key was minted correctly and the inference would have worked. Mirrors the analogous skip in `selectFallbackModelIfNeeded`: when proxyModel is set, the server-side gate (`run-context/route.ts`) is the authority and the proxy mint itself is the validation. Caught by O5 e2e on `pullfrog/preview-844-heyapi-oss-bug`. * Revert "test(oss): temp add preview-844 to ossRepos for O5 e2e" This reverts commit 3bae075ceeb188ee272c45c13b5080e15bcd00a5. * fix(action): discard hook-generated tracked-file drift before agent sees it addresses bug 3 in #844: customer setup/post-checkout hooks like `pnpm install` or `corepack prepare` left the working tree dirty (e.g. `M pnpm-lock.yaml`), the agent took the prompt's "must push" rule literally, opened a spurious bot PR for the lockfile drift, and we ate runs+spend on noise. after each setup / post-checkout hook (opt-in via `normalizeWorkingTreeAfter`), discard tracked-file mods with `git restore --staged --worktree .`. untracked files are preserved — a hook that materializes a `.env` from a template, or emits codegen output, stays visible to the agent. guarded by a pre-hook `git status --porcelain` snapshot: if the tree was already dirty before the hook ran (shouldn't happen — setup runs before any working-tree writes; checkout_pr refuses to run dirty), we warn and skip the discard rather than clobber whatever was there. prepush hook (action/mcp/git.ts) intentionally does NOT opt in — its job is to read the about-to-be-pushed state, not normalize it. * test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge) * fix(action): skip eager pnpm/npm/etc install when no lockfile exists second half of bug 3 in #844. the eager prep step assumed `pnpm install --frozen-lockfile` (and equivalents) would fail cleanly without a lockfile, leaving the tree untouched. that assumption is false for pnpm 11.1.1 against a no-deps `package.json`: the command reports "Already up to date" with exit 0 AND silently materializes an empty `pnpm-lock.yaml` despite the `--frozen-lockfile` flag. the resulting untracked file trips the post-run dirty-tree gate, the agent reads it as "must push uncommitted work", and a spurious "Add pnpm lockfile" PR lands. smoking gun: pullfrog/preview-844-heyapi-oss-bug PRs #1/#2/#3, all auto-opened by the bot against a repo that contains nothing but a one-line README + a no-deps package.json. guard explicitly with an `existsSync` per manager. if the lockfile is absent, skip eager prep entirely with an info log; the agent can install on demand via the `setup` lifecycle hook (which non-frozen `pnpm install` would handle correctly), or just leave deps uninstalled when the prompt doesn't need them (e.g. the O5 "tell me a joke" path). orthogonal to the lifecycle-hook normalization in 0051bd2a — together they cover the full bug 3 surface: - eager prep can't materialize a lockfile (this commit) - setup/postCheckout hooks that rewrite tracked files have the drift discarded before the agent sees it (prior commit) * fix(action): address Pullfrog review on hook normalization two fixes in `executeLifecycleHook` from review on f6f3b32: 1. pre-hook snapshot was `git status --porcelain` which counts untracked files; in practice any repo with pre-existing untracked content (e.g. `.plans/`, an ignored-but-not-yet-gitignored scratch dir, codegen artifacts) would trip the guard and silently skip normalization, defeating the fix. switch to `git diff --name-only HEAD` so the gate measures the same thing the discard targets — tracked-file mods only. pre-existing untracked files are safe regardless because `git restore --staged --worktree .` never touches them. 2. normalization fired only on the happy path; a hook that updated a lockfile then exploded on a peer-dep conflict left tracked drift for the agent. move the call into a `finally` so it runs on success, non-zero exit, timeout, AND spawn failure. the pre-hook guard still protects pre-existing work in every case. * Revert "test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge)" This reverts commit f6f3b325d6bf9a1720754ed1d39d248dab76cfa8. * fix(action): use detect lockfile strategy for eager-prep gate addresses Pullfrog review on be3c207b. two findings, one root cause: - the hardcoded LOCKFILE_BY_MANAGER map missed `bun.lockb` and `npm-shrinkwrap.json`, two managers' accepted lockfile variants. - `existsSync(join(cwd, lockfile))` only checked the immediate directory, breaking monorepo subpackages where the lockfile lives at the workspace root. both fall out by replacing the custom check with `detect({ strategies: ["lockfile"] })`. the detector already walks up the tree (subpackage → workspace root) and recognizes every accepted lockfile name across all managers it supports. restricting to the `lockfile` strategy is load- bearing: the default strategy set also matches on `packageManager` / `devEngines.packageManager` package.json fields, which would return non-null and re-mask the very case we're trying to detect (declared manager, no lockfile committed — the O5 / hey-api preview repro). drops the LOCKFILE_BY_MANAGER map entirely; no need for a second detect() call since the existing one was only used for `agent` resolution and that consumer is now after the lockfile gate, where `detected` is guaranteed non-null.	2026-05-28 00:26:35 +00:00
Colin McDonnell	c43ed65c3b	Add Vertex AI routing support (#753 ) * add Vertex AI routing support * include Vertex smokes in action CI	2026-05-20 16:49:08 +00:00
Colin McDonnell	a0576a702a	opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767 ) * opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference; `opencode_v2.ts` is the active runner via `agents/index.ts`. Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish tool call. The fix only exists in the `1.14.x`+ line, which also ships the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory. Also unblocks the Codex ChatGPT-subscription auth path. Surface area: - drop `init` / `message` / `result` / `tool_result` event types and handlers (no longer emitted at v1.14+ per upstream `cli/cmd/run.ts:588-601`). - `tool_use` is now a single event covering both `state.status: "completed"` and `"error"`. duration / subagent-finish bookkeeping moves from the v1 `tool_result` handler into the consolidated `tool_use` handler. - new `reasoning` event handler — gated on `--thinking`, surfaces Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to `baseArgs`. - drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at v1.15 the `task` tool callID is stable across the whole `tool-input-* → tool-call → tool-result/tool-error` chain (`session/processor.ts:282-330`). exact-match map is sufficient. - drop `experimental.batch_tool: true` from injected config — declared but inert at v1.15. re-add once upstream wires it back. - bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames the platform-specific binary into `opencode.exe` for every OS now). Validated locally: - `pnpm test` 610/610 ✓ - `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓ - `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls, multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓ (this is the headline `thought_signature` fix) - runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓ - runtest opencode: skill-invoke and mcpmerge fail (model-behavior drift on the new system prompt; wiring confirmed intact via direct repro showing both `robinMCP` and `pullfrog` MCP tools exposed). Tracked for follow-up; does not gate the migration. Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at v1.15 — verified upstream and reused as-is. Bus subscription via `bus.subscribeAll()` and the `event` hook still fan out every payload. * model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename) The v1.14+ postinstall.mjs renames the platform-specific binary to `bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows. Mirrors the fix in action/agents/opencode_v2.ts. * opencode v2: set PWD env explicitly to fix skill / project-config discovery Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15 reads `process.env.PWD` first (with `process.cwd()` as fallback) when resolving the SDK client's `directory` parameter — see upstream `cli/cmd/run.ts:282`: const root = Filesystem.resolve(process.env.PWD ?? process.cwd()) We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the `action/` directory, not the cloned test repo. Result: opencode creates two instances per session — one at `process.cwd()` (correct) and one at `PWD` (wrong) — and the agent's session runs in the PWD-derived one, which can't see the project's `.opencode/skills/` or `.claude/skills/`. Empirically traced via the full opencode stderr trace under the runtest harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a second `service=default directory=<harness-pwd> creating instance` line per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})` exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value` becomes accessible too. Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓, restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration match, passes on retry; unrelated to PWD). * opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end` on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block "thought for X s" duration straight from the runtime. The v1 ThinkingTimer heuristic — measuring wall-clock between markToolResult and the next markToolCall — was an approximation when no native source existed; with v2 it's redundant and noisy (it would log alongside the real reasoning event, and conflated network latency with model thinking). Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()` helper, both `markToolCall` / `markToolResult` call sites in `tool_use`. The `reasoning` handler now reads `part.time.start/end` directly and prefixes the visible preview with `(X.Ys)`. Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate) Output now: `» thinking (4.0s): <preview>` (one line, sourced) For models that don't emit reasoning (Sonnet without extended thinking, GPT-4o, etc.), there's just no thinking line — which matches reality better than the gap-heuristic, which would fire on any pause >3s including provider-side latency that wasn't actual model reasoning. Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations. * claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment claude-code 2.1.x reads `process.env.PWD` and registers it as a "session" additional-working-directory when it differs from `process.cwd()` (per the bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...) j.set(H, { path: H, source: "session" })`). Without overriding PWD on the spawn env, claude inherits the harness's PWD via `...process.env` — under `pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo — and adds the wrong dir to the agent's allowed working set. Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of "agent's session sees the wrong cwd" failures on the claude side. Also refresh the stale `action/agents/opencode.ts` reference in entryPost.ts to point at opencode_v2.ts (the active runner), with the v1 file noted as kept-for-reference. * opencode: extract shared helpers into opencodeShared.ts; v2 cleanup Code-quality pass on the v2 work: 1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers between v1 and v2: - `OpenCodeConfig` type - `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin) - `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1 and re-imported by v2 via a back-reference) - `installOpencodeCli({ binPath })` (parameterized — v1 passes `bin/opencode`, v2 passes `bin/opencode.exe` via a per-version `installCli` lambda; matches each pinned version's npm shape) - `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1 keeps a one-line `export { geminiHighThinkingOverrides }` re-export so `opencode.test.ts` keeps working unchanged. Once v1 is retired (post burn-in) opencodeShared collapses back into v2. 2. `opencode_v2.ts` cleanup: - drop dead state (`currentStepId`, `stepHistory` were write-only — their reader was the v1 `tool_result` handler we deleted) - hoist `state` in `tool_use` handler; replace nested-ternary payload extraction with a `terminalPayload(state)` helper - extract `formatPartDuration(time)` for the reasoning-block "(X.Ys)" suffix - tighten `OpenCodeBusEnvelopeEvent` type to include `tool` / `callID` fields directly, drop the `partWithToolFields` cast - trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning handler, tool_use handler, bus envelope handler all shortened - `step_start` becomes an explicit `() => {}` no-op so the dispatcher doesn't log "unhandled event" for every step 3. `subagentRegistration.test.ts` retargeted at the new file split — reads opencodeShared.ts for the buildReviewerAgentConfig assertions and opencode_v2.ts for the orchestrator-model wire-through. Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format + typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified against the refactored v2 runtime. * opencode v2: address PR review feedback Three fixes from the inline review threads on #767: 1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk- level `markActivity()` resets the module-level idle counter, so the per-event `getIdleMs()` sample inside the dispatch loop was always ~0ms — the "no activity for Xs" diagnostic never fired. Replaced with a runner-local `lastEventAt` so we measure real event-to-event silence instead of chunk-arrival latency. Drop the unused `getIdleMs` import. 2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`, `lastProviderError`, and `recentStderr` are closed over by the `handlers` const but were declared after it. No current bug because handlers only fire inside the awaited `spawn()`, but a future refactor that triggers a handler synchronously during setup would surface a TDZ. Hoisted above `handlers`. 3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at L566): leave a `TODO` comment marking the gap until `AgentUsage` grows a `reasoningTokens` field — separate PR with schema work. Cost totals stay correct because `part.cost` is summed independently. Other thread states for the record: - Copilot L63 (geminiHighThinkingOverrides import from legacy): already fixed by the opencodeShared.ts extraction in 83a7cab. - Copilot L672 (ThinkingTimer over-reports on terminal events): already fixed by dropping ThinkingTimer in a1e536b — we use opencode's own `reasoning.part.time.{start,end}` for thinking durations now. - Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked the bus-envelope flow; the plugin filters orchestrator events except for status=running task dispatches, and bus-envelope returns before calling handlers.tool_use on those. No double-fire under current code. Validated: 610/610 unit tests, lint + format + typecheck clean, skill-invoke-opencode ✓. * DX: flip pnpm play / pnpm runtest to docker-by-default Restores the script shape wiki/docker.md has documented since the docker rewrite (#750). PR #756 inadvertently reverted action/package.json's gha/play/runtest scripts to host-only and dropped the :local variants; the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2 migration agent ran `pnpm play --raw …` host-side throughout because the host entry was the only thing that existed. scripts (root → action): - pnpm play → pnpm -C action gha play.ts (docker, default) - pnpm play:local → pnpm -C action play:local (host) - pnpm runtest → pnpm -C action gha test/run.ts (docker, default) - pnpm runtest:local → pnpm -C action runtest:local (host) - pnpm gha is restored in action/package.json (re-adds `node gha.ts`) action/package.json deliberately ships only the :local variants — bare `pnpm -C action play` now errors instead of silently bypassing docker. This is a tradeoff per the user prompt's "consider whether NAMES should change" hint: the explicit error is worth the small CI churn. CI workflows: `.github/workflows/test.yml` and `action/.github/workflows/test.yml` flipped from `pnpm runtest …` to `pnpm runtest:local …`. Semantics unchanged — they still execute `node test/run.ts` directly on the GHA Linux runner; nesting docker on GHA is unnecessary overhead. Only the script name changed to match the new package.json. Webhook tester: the existing root `pnpm play` was actually a webhook handler smoke harness (root play.ts), unrelated to the action runtime. Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so `pnpm play` can carry the docker-by-default action shortcut without collision. README updated. File headers updated: - action/play.ts: invocation block now points at `pnpm play` / `pnpm play:local` - action/test/run.ts: same - action/gha.ts: usage block calls out the new shortcut wrappers AGENTS.md: extended the existing "local sanity checks of action tool logic" rule with the play / play:local / runtest / runtest:local selection guidance and the `cd action; pnpm play` footgun note. wiki/docker.md unchanged — already described the now-real shape. * test/crossagent: add codex-auth smoke Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies: - installCodexAuth() materializes auth.json under the test HOME - opencode routes openai requests through ChatGPT subscription auth (no OPENAI_API_KEY in env, AT path forced via expires: 0) - the refresh chain advances during the run (refresh_token rotates) - detectCodexRefresh() would surface the rotation to entryPost.ts The post-hook write-back fetch isn't reachable from `pnpm runtest` (it's a separate GHA `post:` step). The integration boundary that matters end-to-end is "did the on-disk auth.json change in a way detectCodexRefresh recognizes" — that's exactly what this test asserts. CI wiring (already committed in a1c1fd4f as part of the DX flip): - .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in action-agents env block - action/.github/workflows/test.yml: same; codex-auth in the hardcoded test matrix with a claude exclude The provisioning step on the user's side is `gh secret set CODEX_AUTH_JSON --repo pullfrog/app < auth.json`. ci.test.ts: expectedAgentEnvVars now includes provider `managedCredentials` so the "env vars cover all provider API keys" invariant stays self-correcting as more managed credentials land. * docs(codex-auth): make storage requirement unmissable A previous reviewing agent on this branch came away thinking `CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't — `entryPost.ts` rewrites the rotated refresh token after every run, and GH Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres storage breaks the refresh chain on the first rotation (~1h silent expiry). - wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold, with the words "GitHub Actions secrets DO NOT WORK" verbatim and an enumeration of broken alternatives. - action/utils/codexHome.ts + action/entryPost.ts: header comments now loudly contrast Pullfrog secret store vs GH Actions and explain the writeback constraint. - AGENTS.md: terse one-bullet rule next to the model-resolution rule so future agents don't repeat the mistake. - .github/workflows/test.yml + action/.github/workflows/test.yml: added a comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a CI smoke-testing shortcut, not the canonical pattern. CI wiring itself unchanged per scope. * auth codex: auto-open device URL, drop --scope flag - detect `https://auth.openai.com/codex/device...` from codex CLI output and best-effort launch it in the user's default browser (open / xdg-open / cmd start, wslview fallback on linux). gated so we only open once per flow; failures are swallowed so manual copy-paste still works. - drop the `--scope` flag entirely. the device-code flow is fundamentally interactive (browser approval), so a "skip-the-prompt" flag for just one of the prompts was dead weight. collapses scope selection to "always prompt on org-owned, always account on user-owned". * rename gha→docker, flip play/runtest defaults to host the previous shape conflated "real GitHub Actions" with the local docker container that mocks it, and made the slow docker path the default for fast-iteration scripts. - `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image tag `pullfrog-docker:`, volume `pullfrog-docker-node-modules-`, tmpdir, error messages) - `pnpm play` / `pnpm runtest` now default to host (fast iteration); `pnpm play:docker` / `pnpm runtest:docker` run inside the container - `pnpm gha` → `pnpm docker` (the container runner shortcut) - `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the bare name implied a webhook server, which hookdeck-cli already is) - update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`, `README.md`, `AGENTS.md`), CI workflows (`.github/workflows/test.yml`, `action/.github/workflows/test.yml`), and code headers (`action/{play,test/run,utils/runFixture}.ts`, `webhook.ts`, `action/test/coverage.ts`) `action/commands/gha.ts` keeps its name — it's the real GitHub Actions entry point for the `pullfrog gha` CLI command (not the docker mock). * fix(codex): route post-hook writeback through apiFetch + conditional skip Three threads addressing PR #767 followups. action/entryPost.ts: replace raw fetch() with apiFetch() so the PUT /api/runtime/secret call carries the x-vercel-protection-bypass header/query when targeting a preview deployment. raw fetch silently 401s against the Vercel SSO gate, so every preview-env Codex run was losing its rotated refresh token. production is unaffected (no SSO). action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON via new TestRunnerOptions.skipIf hook. when the secret is absent (forks, contributors without it), runTestForAgent short-circuits to a passing-with-skipped ValidationResult before any agent spawn — so the matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI on pullfrog/app and dev-local with .env both still run the test for real. printSingleValidation/printResults now render skipped entries distinctly. doc/comment drift: - docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag mention (removed in 10be96db, scope is now always interactively prompted or implicit). - wiki/codex-auth.md: tighten Claude-defense wording — materialization is agent-gated (opencode/opencode_v2 harness), not model-gated; opencode runs with non-OpenAI models still materialize the file, it's just not read. - action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts → pnpm docker / docker.ts (renamed in a2a63929). - app/api/runtime/secret/route.ts: refer to the save-time scope prompt instead of the dropped --scope flag. * smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS upgrade crossagent/smoke prompt to call pullfrog_git status before set_output. this exercises the 2nd model→agent round-trip across every providers-live flagship, catching bugs like the Gemini thought_signature echo that single-tool-call tests can't see. also adds the "bar for adding new LLM-driven tests" section to wiki/action-tests.md and an extension to the existing AGENTS.md no-tests rule pointing at it — prefer upgrading existing matrix entries over adding new ones. local: pnpm runtest smoke opencode passes against both anthropic/claude-sonnet-4-6 and google/gemini-pro. --------- Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>	2026-05-20 04:05:16 +00:00

Author

SHA1

Message

Date

Colin McDonnell

36ac64a5b6

fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy (#844 )

* fix(oss-codex): prefer user's uploaded Codex auth over OSS subsidy

OSS-allowlisted repos with `CODEX_AUTH_JSON` uploaded via `pullfrog auth
codex` were still being routed through the OSS OpenRouter subsidy because
two paths ignored managed credentials:

- `hasProviderKey()` only checked `provider.envVars`, so an `openai/*`
model with only `CODEX_AUTH_JSON` present silently fell back to
`opencode/big-pickle` via `selectFallbackModelIfNeeded` — the maintainer
saw "opencode/big-pickle (resolved from openai/gpt)" on CI even though
Codex was configured.
- `run-context` set `proxyModel` for every OSS run unconditionally, which
the action runtime threads through `payload.proxyModel` and uses to
overwrite `OPENROUTER_API_KEY`. Even if `big-pickle` fallback hadn't
fired, the runner would consume the $10 OSS subsidy key instead of the
user's ChatGPT subscription.

Fix:

- Add `getModelAuthEnvVars()` covering both `envVars` and
`managedCredentials` in `action/utils/apiKeys.ts`; route
`hasProviderKey` + `validateAgentApiKey` through it.
- `run-context` now skips `proxyModel` for OSS repos when the configured
model's provider has matching auth in Pullfrog-stored account/repo
secrets, so the runner authenticates directly with the user's Codex
subscription (or any other user-provided provider auth).

Triggered by mrlubos (`hey-api/openapi-ts`). Companion follow-up tracked
for the opaque "(no error message)" classifier swallow that masked the
OSS $10 cap exhaustion on PR #3872 runs 25815370844 + 25815443234.

* fix(oss): force Kimi K2 for OSS proxy + hide picker in console UI

OSS-funded runs were resolving `repo.model` through OpenRouter, so a
single Opus / GPT-5.5 run could burn an entire `oss_subsidy` key against
the per-key cap and crash mid-stream (e.g. `hey-api/openapi-ts` PR #3872
runs `25815370844` + `25815443234`, ~$9.20 each on a single key).

Force `DEFAULT_PROXY_MODEL` (Kimi K2.6 — ~10-50× cheaper) for every OSS
proxy mint, regardless of `repo.model`. Per-run spend stays bounded
within the cap by structure, not by hope. `repo.model` stays in the DB
unchanged — overriding at runtime means leaving the program restores the
user's prior pick without a migration.

UI: hide the model picker entirely on OSS repos in `AgentSettings`. The
field is effectively inert until the repo leaves the program, so
exposing it as if it were live was misleading. Replaced with a banner
naming Kimi K2 and pointing to `pullfrog auth …` as the opt-out path —
that lands the user on the existing #844 bug-2 branch (Pullfrog-stored
auth suppresses the OSS proxy entirely; runner uses user credentials
+ their preferred model).

ModelCostsInfo already has its own `isOss` branch for the cost copy, so
that section is unchanged.

* fix(oss): lowercase comment casing per AGENTS.md

* fix(oss): revert banner copy to 'It's on us.' framing per review

Maintainer felt 'Kimi K2' as the banner headline lost the warm 'we've
got you covered' framing that the existing OSS cost banner uses. Restore
'It's on us.' as the headline, move the model name into the body where
it explains the hardcoded choice and points to the opt-out (pullfrog
auth codex / account secret).

* docs(agents): screenshots must be of the live route, never synthetic

Caught myself building a temp `/dev/oss-ui-preview` route with hardcoded
JSX copy-pasted from the real component just to grab a screenshot — the
result told us nothing about whether the actual integrated UI worked,
and the user (rightly) called it out as a waste. Strengthen the rule:
screenshots must come from the live route in the running app, driven by
the actual component tree and real props. Note the GH OAuth interstitial
gotcha so the next agent gets through Clerk → GitHub sign-in on the
first try instead of bailing to a fake render. Also bans side-by-side
comparison screenshots unless explicitly requested.

* fix(oss): one 'It's on us.' banner, not two

OSS Agent settings was showing the message twice — once in the Model
section, once in the Model costs section right below it. Fold the cost
coverage into the model banner ('at no cost to you' + the spend stat)
and hide the Model costs subsection entirely for OSS. ModelCostsInfo
no longer needs `isOss` / `ossSpendThisMonthUsd` props — call site is
gated, so the OSS branch is dead. Removed it and the now-unused props.

Non-OSS rendering is unchanged: full Model picker + Model costs
subsection with Router / BYOK branches.

* feat(action): corepack-aware package manager provisioning before setup

customer setup scripts that did `npm i -g pnpm && pnpm install` were
installing whatever pnpm "latest" happens to be on the day the run fires,
not what the repo declares — and pnpm 11.3 silently writes a new
`packageManagerDependencies` block into lockfiles, which the agent's
"always push changes" rule then packages into a noisy PR (see #844).

resolve the project's pnpm/yarn pin from `package.json` (honoring pnpm
11+ precedence: `devEngines.packageManager` over `packageManager`) and
activate it via `corepack prepare ... --activate` BEFORE the setup hook
runs. corepack is bundled with node, so this is a no-op on managed
infra; failure (no corepack, no network, range-only version) degrades
to a warning and the existing PATH binary still runs.

also replaces the legacy `npm install -g <pm>@<v>` path in prep with
the same helper so behavior is consistent end-to-end. bun/deno still
use the legacy installer because corepack doesn't ship shims for them.

* chore(console): drop 'npm i -g pnpm' anti-pattern from setup-script placeholder

the suggested example trained customers to install pnpm unpinned, which
silently picks up whatever's latest at run time. that's exactly the
behavior #844 traced lockfile drift back to. now that prep handles
package-manager provisioning via corepack from the repo's declared pin,
the placeholder is just a frozen-lockfile install — load-bearing only
when the repo wants `pnpm install` to actually run (prep already does
that), but a much safer default for customers who do paste it in.

* refactor(action): introspect opencode models for BYOK detection

Replace the static `provider.envVars + provider.managedCredentials`
catalog gate in `selectFallbackModelIfNeeded` + `validateAgentApiKey`
with two `opencode models` captures around the auth merge:

- `captureBaselineModels` BEFORE dbSecrets + Codex auth.json
- `captureAuthorizedModels` AFTER both

The authorized set is the authoritative source for "can OpenCode route
this model" — strictly more accurate than the catalog, which can miss
new auth shapes (Codex was one, there will be more). The diff between
baseline and authorized is logged as `BYOK auth enabled N model(s)`
for operator visibility.

Sequencing changes in main.ts:

- `createTempDirectory` hoisted out of the try block so
`PULLFROG_TEMP_DIR` is set before the early opencode install
- `agents.opencode.install()` + baseline capture before dbSecrets
- `installCodexAuth()` hoisted up (idempotent — agent re-calls it
inside run() and writes the same file)
- authorized capture after Codex auth.json materializes
- fallback + validateAgentApiKey receive the authorized set as a
parameter; tests inject directly with no mocks

Deleted: `hasProviderKey`, `getModelAuthEnvVars`, `knownApiKeys` in
`action/utils/apiKeys.ts` (only `selectFallbackModelIfNeeded` consumed
them, and PR #844's catalog-extension fix is superseded by introspection).
`getModelEnvVars` / `getModelManagedCredentials` stay exported for UI
and the server-side OSS proxy heuristic in run-context/route.ts.

For the claude agent path, validateAgentApiKey keeps the static
single-provider check on `ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN`
— `opencode models` is opencode-specific. validateBedrockSetup /
validateVertexSetup also stay; they cover region/location/model-id
which `opencode models` doesn't catch.

When fallback engages, the post-fallback model is the guaranteed-free
`opencode/big-pickle`, so validateAgentApiKey is skipped — the fallback
gate already authoritatively decided "this model is OK to run".

* test(oss): temp add preview-844 to ossRepos for O4 e2e

* Revert "test(oss): temp add preview-844 to ossRepos for O4 e2e"

This reverts commit 8167e560126b2ac516c32ba1c63c36aa32ae4019.

* test(oss): temp add preview-844 to ossRepos for O5 e2e

* fix(action): skip validateAgentApiKey when proxyModel is set

The new opencode-models BYOK introspection in PR #844 captures the
authorized set BEFORE runProxyResolution mints OPENROUTER_API_KEY, so
the proxy slug (e.g. `openrouter/moonshotai/kimi-k2.6`) is never in
the set. validateAgentApiKey then spuriously threw "no API key found"
on every OSS run, even though the proxy key was minted correctly and
the inference would have worked.

Mirrors the analogous skip in `selectFallbackModelIfNeeded`: when
proxyModel is set, the server-side gate (`run-context/route.ts`) is
the authority and the proxy mint itself is the validation.

Caught by O5 e2e on `pullfrog/preview-844-heyapi-oss-bug`.

* Revert "test(oss): temp add preview-844 to ossRepos for O5 e2e"

This reverts commit 3bae075ceeb188ee272c45c13b5080e15bcd00a5.

* fix(action): discard hook-generated tracked-file drift before agent sees it

addresses bug 3 in #844: customer setup/post-checkout hooks like `pnpm install`
or `corepack prepare` left the working tree dirty (e.g. `M pnpm-lock.yaml`),
the agent took the prompt's "must push" rule literally, opened a spurious bot
PR for the lockfile drift, and we ate runs+spend on noise.

after each setup / post-checkout hook (opt-in via `normalizeWorkingTreeAfter`),
discard tracked-file mods with `git restore --staged --worktree .`. untracked
files are preserved — a hook that materializes a `.env` from a template, or
emits codegen output, stays visible to the agent.

guarded by a pre-hook `git status --porcelain` snapshot: if the tree was
already dirty before the hook ran (shouldn't happen — setup runs before any
working-tree writes; checkout_pr refuses to run dirty), we warn and skip the
discard rather than clobber whatever was there.

prepush hook (action/mcp/git.ts) intentionally does NOT opt in — its job is
to read the about-to-be-pushed state, not normalize it.

* test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge)

* fix(action): skip eager pnpm/npm/etc install when no lockfile exists

second half of bug 3 in #844. the eager prep step assumed `pnpm install
--frozen-lockfile` (and equivalents) would fail cleanly without a lockfile,
leaving the tree untouched. that assumption is false for pnpm 11.1.1 against
a no-deps `package.json`: the command reports "Already up to date" with exit 0
AND silently materializes an empty `pnpm-lock.yaml` despite the `--frozen-lockfile`
flag. the resulting untracked file trips the post-run dirty-tree gate, the
agent reads it as "must push uncommitted work", and a spurious
"Add pnpm lockfile" PR lands. smoking gun: pullfrog/preview-844-heyapi-oss-bug
PRs #1/#2/#3, all auto-opened by the bot against a repo that contains nothing
but a one-line README + a no-deps package.json.

guard explicitly with an `existsSync` per manager. if the lockfile is absent,
skip eager prep entirely with an info log; the agent can install on demand
via the `setup` lifecycle hook (which non-frozen `pnpm install` would handle
correctly), or just leave deps uninstalled when the prompt doesn't need them
(e.g. the O5 "tell me a joke" path).

orthogonal to the lifecycle-hook normalization in 0051bd2a — together they
cover the full bug 3 surface:
- eager prep can't materialize a lockfile (this commit)
- setup/postCheckout hooks that rewrite tracked files have the drift
discarded before the agent sees it (prior commit)

* fix(action): address Pullfrog review on hook normalization

two fixes in `executeLifecycleHook` from review on f6f3b32:

1. pre-hook snapshot was `git status --porcelain` which counts untracked
files; in practice any repo with pre-existing untracked content (e.g.
`.plans/`, an ignored-but-not-yet-gitignored scratch dir, codegen
artifacts) would trip the guard and silently skip normalization,
defeating the fix. switch to `git diff --name-only HEAD` so the gate
measures the same thing the discard targets — tracked-file mods only.
pre-existing untracked files are safe regardless because `git restore
--staged --worktree .` never touches them.

2. normalization fired only on the happy path; a hook that updated a
lockfile then exploded on a peer-dep conflict left tracked drift for
the agent. move the call into a `finally` so it runs on success,
non-zero exit, timeout, AND spawn failure. the pre-hook guard still
protects pre-existing work in every case.

* Revert "test(oss): temp add preview-844 to ossRepos for bug 3 e2e (revert before merge)"

This reverts commit f6f3b325d6bf9a1720754ed1d39d248dab76cfa8.

* fix(action): use detect lockfile strategy for eager-prep gate

addresses Pullfrog review on be3c207b. two findings, one root cause:

- the hardcoded LOCKFILE_BY_MANAGER map missed `bun.lockb` and
`npm-shrinkwrap.json`, two managers' accepted lockfile variants.
- `existsSync(join(cwd, lockfile))` only checked the immediate directory,
breaking monorepo subpackages where the lockfile lives at the workspace
root.

both fall out by replacing the custom check with `detect({ strategies:
["lockfile"] })`. the detector already walks up the tree (subpackage →
workspace root) and recognizes every accepted lockfile name across all
managers it supports. restricting to the `lockfile` strategy is load-
bearing: the default strategy set also matches on `packageManager` /
`devEngines.packageManager` package.json fields, which would return
non-null and re-mask the very case we're trying to detect (declared
manager, no lockfile committed — the O5 / hey-api preview repro).

drops the LOCKFILE_BY_MANAGER map entirely; no need for a second
detect() call since the existing one was only used for `agent`
resolution and that consumer is now after the lockfile gate, where
`detected` is guaranteed non-null.

2026-05-28 00:26:35 +00:00

Colin McDonnell

c43ed65c3b

Add Vertex AI routing support (#753 )

* add Vertex AI routing support

* include Vertex smokes in action CI

2026-05-20 16:49:08 +00:00

Colin McDonnell

a0576a702a

opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767 )

* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite

Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.

Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.

Surface area:

  - drop `init` / `message` / `result` / `tool_result` event types and
    handlers (no longer emitted at v1.14+ per upstream
    `cli/cmd/run.ts:588-601`).
  - `tool_use` is now a single event covering both `state.status:
    "completed"` and `"error"`. duration / subagent-finish bookkeeping
    moves from the v1 `tool_result` handler into the consolidated
    `tool_use` handler.
  - new `reasoning` event handler — gated on `--thinking`, surfaces
    Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
    `baseArgs`.
  - drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
    v1.15 the `task` tool callID is stable across the whole
    `tool-input-* → tool-call → tool-result/tool-error` chain
    (`session/processor.ts:282-330`). exact-match map is sufficient.
  - drop `experimental.batch_tool: true` from injected config — declared
    but inert at v1.15. re-add once upstream wires it back.
  - bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
    the platform-specific binary into `opencode.exe` for every OS now).

Validated locally:

  - `pnpm test` 610/610 ✓
  - `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
  - `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
    multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
    (this is the headline `thought_signature` fix)
  - runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
  - runtest opencode: skill-invoke and mcpmerge fail (model-behavior
    drift on the new system prompt; wiring confirmed intact via direct
    repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
    Tracked for follow-up; does not gate the migration.

Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.

* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)

The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.

* opencode v2: set PWD env explicitly to fix skill / project-config discovery

Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:

  const root = Filesystem.resolve(process.env.PWD ?? process.cwd())

We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.

Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.

Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).

* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly

opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).

Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.

Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now:    `» thinking (4.0s): <preview>` (one line, sourced)

For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.

Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.

* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment

claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.

Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.

Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.

* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup

Code-quality pass on the v2 work:

1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
   between v1 and v2:
   - `OpenCodeConfig` type
   - `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
   - `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
     and re-imported by v2 via a back-reference)
   - `installOpencodeCli({ binPath })` (parameterized — v1 passes
     `bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
     `installCli` lambda; matches each pinned version's npm shape)
   - `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback

   v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
   keeps a one-line `export { geminiHighThinkingOverrides }` re-export
   so `opencode.test.ts` keeps working unchanged. Once v1 is retired
   (post burn-in) opencodeShared collapses back into v2.

2. `opencode_v2.ts` cleanup:
   - drop dead state (`currentStepId`, `stepHistory` were write-only —
     their reader was the v1 `tool_result` handler we deleted)
   - hoist `state` in `tool_use` handler; replace nested-ternary payload
     extraction with a `terminalPayload(state)` helper
   - extract `formatPartDuration(time)` for the reasoning-block
     "(X.Ys)" suffix
   - tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
     `callID` fields directly, drop the `partWithToolFields` cast
   - trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
     handler, tool_use handler, bus envelope handler all shortened
   - `step_start` becomes an explicit `() => {}` no-op so the dispatcher
     doesn't log "unhandled event" for every step

3. `subagentRegistration.test.ts` retargeted at the new file split —
   reads opencodeShared.ts for the buildReviewerAgentConfig assertions
   and opencode_v2.ts for the orchestrator-model wire-through.

Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.

* opencode v2: address PR review feedback

Three fixes from the inline review threads on #767:

1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
   level `markActivity()` resets the module-level idle counter, so the
   per-event `getIdleMs()` sample inside the dispatch loop was always
   ~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
   a runner-local `lastEventAt` so we measure real event-to-event silence
   instead of chunk-arrival latency. Drop the unused `getIdleMs` import.

2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
   `lastProviderError`, and `recentStderr` are closed over by the
   `handlers` const but were declared after it. No current bug because
   handlers only fire inside the awaited `spawn()`, but a future
   refactor that triggers a handler synchronously during setup would
   surface a TDZ. Hoisted above `handlers`.

3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
   L566): leave a `TODO` comment marking the gap until `AgentUsage`
   grows a `reasoningTokens` field — separate PR with schema work.
   Cost totals stay correct because `part.cost` is summed independently.

Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
  fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
  fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
  `reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
  the bus-envelope flow; the plugin filters orchestrator events except
  for status=running task dispatches, and bus-envelope returns before
  calling handlers.tool_use on those. No double-fire under current code.

Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.

* DX: flip pnpm play / pnpm runtest to docker-by-default

Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.

scripts (root → action):
- pnpm play       → pnpm -C action gha play.ts          (docker, default)
- pnpm play:local → pnpm -C action play:local           (host)
- pnpm runtest    → pnpm -C action gha test/run.ts      (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local     (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)

action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.

CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.

Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.

File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
  `pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers

AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.

wiki/docker.md unchanged — already described the now-real shape.

* test/crossagent: add codex-auth smoke

Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:

  - installCodexAuth() materializes auth.json under the test HOME
  - opencode routes openai requests through ChatGPT subscription auth
    (no OPENAI_API_KEY in env, AT path forced via expires: 0)
  - the refresh chain advances during the run (refresh_token rotates)
  - detectCodexRefresh() would surface the rotation to entryPost.ts

The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.

CI wiring (already committed in a1c1fd4f as part of the DX flip):
  - .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
    action-agents env block
  - action/.github/workflows/test.yml: same; codex-auth in the
    hardcoded test matrix with a claude exclude

The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.

ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.

* docs(codex-auth): make storage requirement unmissable

A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).

- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
  with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
  enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
  loudly contrast Pullfrog secret store vs GH Actions and explain the
  writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
  future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
  comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
  CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
  unchanged per scope.

* auth codex: auto-open device URL, drop --scope flag

- detect `https://auth.openai.com/codex/device...` from codex CLI output
  and best-effort launch it in the user's default browser (open / xdg-open
  / cmd start, wslview fallback on linux). gated so we only open once per
  flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
  interactive (browser approval), so a "skip-the-prompt" flag for just one
  of the prompts was dead weight. collapses scope selection to "always
  prompt on org-owned, always account on user-owned".

* rename gha→docker, flip play/runtest defaults to host

the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.

- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
  tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
  tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
  `pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
  bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
  `README.md`, `AGENTS.md`), CI workflows
  (`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
  and code headers (`action/{play,test/run,utils/runFixture}.ts`,
  `webhook.ts`, `action/test/coverage.ts`)

`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).

* fix(codex): route post-hook writeback through apiFetch + conditional skip

Three threads addressing PR #767 followups.

action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).

action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.

doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
  mention (removed in 10be96db, scope is now always interactively
  prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
  is agent-gated (opencode/opencode_v2 harness), not model-gated;
  opencode runs with non-OpenAI models still materialize the file,
  it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
  → pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
  instead of the dropped --scope flag.

* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS

upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.

also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.

local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.

---------

Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>

2026-05-20 04:05:16 +00:00

3 Commits