36ac64a5b63a2b749e3dcc3b22e288966f94a298
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colin McDonnell
|
a0576a702a |
opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767)
* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite
Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.
Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.
Surface area:
- drop `init` / `message` / `result` / `tool_result` event types and
handlers (no longer emitted at v1.14+ per upstream
`cli/cmd/run.ts:588-601`).
- `tool_use` is now a single event covering both `state.status:
"completed"` and `"error"`. duration / subagent-finish bookkeeping
moves from the v1 `tool_result` handler into the consolidated
`tool_use` handler.
- new `reasoning` event handler — gated on `--thinking`, surfaces
Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
`baseArgs`.
- drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
v1.15 the `task` tool callID is stable across the whole
`tool-input-* → tool-call → tool-result/tool-error` chain
(`session/processor.ts:282-330`). exact-match map is sufficient.
- drop `experimental.batch_tool: true` from injected config — declared
but inert at v1.15. re-add once upstream wires it back.
- bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
the platform-specific binary into `opencode.exe` for every OS now).
Validated locally:
- `pnpm test` 610/610 ✓
- `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
- `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
(this is the headline `thought_signature` fix)
- runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
- runtest opencode: skill-invoke and mcpmerge fail (model-behavior
drift on the new system prompt; wiring confirmed intact via direct
repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
Tracked for follow-up; does not gate the migration.
Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.
* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)
The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.
* opencode v2: set PWD env explicitly to fix skill / project-config discovery
Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:
const root = Filesystem.resolve(process.env.PWD ?? process.cwd())
We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.
Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.
Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).
* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly
opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).
Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.
Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now: `» thinking (4.0s): <preview>` (one line, sourced)
For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.
Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.
* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment
claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.
Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.
Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.
* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup
Code-quality pass on the v2 work:
1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
between v1 and v2:
- `OpenCodeConfig` type
- `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
- `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
and re-imported by v2 via a back-reference)
- `installOpencodeCli({ binPath })` (parameterized — v1 passes
`bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
`installCli` lambda; matches each pinned version's npm shape)
- `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback
v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
keeps a one-line `export { geminiHighThinkingOverrides }` re-export
so `opencode.test.ts` keeps working unchanged. Once v1 is retired
(post burn-in) opencodeShared collapses back into v2.
2. `opencode_v2.ts` cleanup:
- drop dead state (`currentStepId`, `stepHistory` were write-only —
their reader was the v1 `tool_result` handler we deleted)
- hoist `state` in `tool_use` handler; replace nested-ternary payload
extraction with a `terminalPayload(state)` helper
- extract `formatPartDuration(time)` for the reasoning-block
"(X.Ys)" suffix
- tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
`callID` fields directly, drop the `partWithToolFields` cast
- trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
handler, tool_use handler, bus envelope handler all shortened
- `step_start` becomes an explicit `() => {}` no-op so the dispatcher
doesn't log "unhandled event" for every step
3. `subagentRegistration.test.ts` retargeted at the new file split —
reads opencodeShared.ts for the buildReviewerAgentConfig assertions
and opencode_v2.ts for the orchestrator-model wire-through.
Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.
* opencode v2: address PR review feedback
Three fixes from the inline review threads on #767:
1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
level `markActivity()` resets the module-level idle counter, so the
per-event `getIdleMs()` sample inside the dispatch loop was always
~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
a runner-local `lastEventAt` so we measure real event-to-event silence
instead of chunk-arrival latency. Drop the unused `getIdleMs` import.
2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
`lastProviderError`, and `recentStderr` are closed over by the
`handlers` const but were declared after it. No current bug because
handlers only fire inside the awaited `spawn()`, but a future
refactor that triggers a handler synchronously during setup would
surface a TDZ. Hoisted above `handlers`.
3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
L566): leave a `TODO` comment marking the gap until `AgentUsage`
grows a `reasoningTokens` field — separate PR with schema work.
Cost totals stay correct because `part.cost` is summed independently.
Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
`reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
the bus-envelope flow; the plugin filters orchestrator events except
for status=running task dispatches, and bus-envelope returns before
calling handlers.tool_use on those. No double-fire under current code.
Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.
* DX: flip pnpm play / pnpm runtest to docker-by-default
Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.
scripts (root → action):
- pnpm play → pnpm -C action gha play.ts (docker, default)
- pnpm play:local → pnpm -C action play:local (host)
- pnpm runtest → pnpm -C action gha test/run.ts (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)
action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.
CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.
Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.
File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
`pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers
AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.
wiki/docker.md unchanged — already described the now-real shape.
* test/crossagent: add codex-auth smoke
Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:
- installCodexAuth() materializes auth.json under the test HOME
- opencode routes openai requests through ChatGPT subscription auth
(no OPENAI_API_KEY in env, AT path forced via expires: 0)
- the refresh chain advances during the run (refresh_token rotates)
- detectCodexRefresh() would surface the rotation to entryPost.ts
The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.
CI wiring (already committed in a1c1fd4f as part of the DX flip):
- .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
action-agents env block
- action/.github/workflows/test.yml: same; codex-auth in the
hardcoded test matrix with a claude exclude
The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.
ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.
* docs(codex-auth): make storage requirement unmissable
A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).
- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
loudly contrast Pullfrog secret store vs GH Actions and explain the
writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
unchanged per scope.
* auth codex: auto-open device URL, drop --scope flag
- detect `https://auth.openai.com/codex/device...` from codex CLI output
and best-effort launch it in the user's default browser (open / xdg-open
/ cmd start, wslview fallback on linux). gated so we only open once per
flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
interactive (browser approval), so a "skip-the-prompt" flag for just one
of the prompts was dead weight. collapses scope selection to "always
prompt on org-owned, always account on user-owned".
* rename gha→docker, flip play/runtest defaults to host
the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.
- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
`pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
`README.md`, `AGENTS.md`), CI workflows
(`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
and code headers (`action/{play,test/run,utils/runFixture}.ts`,
`webhook.ts`, `action/test/coverage.ts`)
`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).
* fix(codex): route post-hook writeback through apiFetch + conditional skip
Three threads addressing PR #767 followups.
action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).
action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.
doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
mention (removed in 10be96db, scope is now always interactively
prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
is agent-gated (opencode/opencode_v2 harness), not model-gated;
opencode runs with non-OpenAI models still materialize the file,
it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
→ pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
instead of the dropped --scope flag.
* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS
upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.
also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.
local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.
---------
Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>
|
||
|
Colin McDonnell
|
5caeb75344 |
review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models (#710)
* review: 0-or-2+ lens rule, parallel-or-bust, downshifted subagent models
PR review wall-time was dominated by two failure modes: orchestrator
serial-dispatching subagents (despite prompt asking for parallel) and
running every lens on the same Opus tier as the orchestrator. Sample of
recent runs showed 25-60min reviews on small PRs, with 8-10min idle
gaps between subagent dispatches.
Three changes:
1. `action/modes.ts` — replace the soft "1 trivial / 2-3 typical /
4-5 high-stakes" lens calibration with a binary 0-or-2+ rule. Default
is 0 lenses (orchestrator handles review solo with optional cheap
tracerfrog dispatches). 2+ parallel lenses only fire for substantive
PRs (>5 files AND >200 lines) or high-stakes-subsystem touches. Never
exactly one. Both Review and IncrementalReview prompts get loud
ALL-CAPS framing on parallel dispatch — emit ALL Task tool_use blocks
in a single assistant turn before reading any result. Drop the
"do NOT lens-review the diff yourself" advice; orchestrator pulls
context aggressively, in parallel with the lens fan-out.
2. New `tracerfrog` subagent for mechanical code tracing ("where is X
used / who calls Y / what depends on Z"). Pure read+grep+report with
no judgment — orchestrator can dispatch many tracers cheaply in
parallel. Defined in `action/agents/reviewer.ts`. Wired into both
claude.ts (`--agents` JSON) and opencode.ts (`agent` config block).
3. Per-subagent model downshifts via `deriveSubagentModels`:
- Anthropic: reviewfrog → Sonnet, tracerfrog → Haiku
- OpenAI: both → gpt-5.4-mini
- other providers (xai, deepseek, gemini, etc.): inherit (no
standard tier triplet to downshift to)
Claude Code path always runs Anthropic so the downshift is hardcoded
inline in claude.ts. OpenCode uses the helper since orchestrator
provider varies.
Both runtimes' subagent-definition formats verified directly against
their source: `--agents` JSON `model` field (claude-code's
`AgentJsonSchema` accepts model+effort+maxTurns+more) and OpenCode's
`agent.{name}.model` config field (parsed via Provider.parseModel,
applied per-task in tool/task.ts line 92). Parallel dispatch is
infra-supported in both — only the orchestrator model's tool_use
emission pattern was the bottleneck.
Tests: subagentModels.test.ts (14 tests covering provider matrix),
subagentRegistration.test.ts (6 source-asserts catching shape
regressions in buildAgentsJson / buildReviewerAgentConfig).
* subagentModels: add openrouter routes (proxy/router mode)
Initial helper missed the openrouter prefix used by Pullfrog's router
proxy. preview-710 e2e showed the OpenCode + openrouter path receiving
no downshift — orchestrator and lenses both ran on opus-4.7 because
'openrouter/anthropic/claude-opus-4.7' didn't match any of the
anthropic/openai prefixes the helper checked.
Add explicit branches for 'openrouter/anthropic/...' (uses dot notation:
claude-sonnet-4.6 / claude-haiku-4.5) and 'openrouter/openai/...'
(gpt-5.4-mini for both reviewer and tracer). Same opus->sonnet,
sonnet->keep-but-haiku-tracer, haiku->no-op semantics as the direct
anthropic path.
* opencode: log resolved subagent models at startup
So we can verify per-subagent model overrides actually take effect at
runtime. Prints once per run alongside the existing model/effort log
lines.
* drop tracerfrog: keep reviewfrog only, LSP-powered tracer planned later
Removes the cheap-haiku-tracer subagent (TRACER_AGENT_NAME +
TRACER_SYSTEM_PROMPT, registrations in claude.ts/opencode.ts, dispatch
guidance in modes.ts). The mechanical-tracing use case will be served
better by an LSP-powered tool than by a separately-prompted subagent.
deriveSubagentModels collapses to a single { reviewer } shape; the
reviewfrog-on-Sonnet downshift stays. Same source-assert + provider-
matrix tests, minus the tracer-specific cases.
modes.ts wording: drop the 'subagent type cheat sheet' bullet, drop
the parenthetical 'often better served by tracerfrog than reviewfrog'
on the impact lens, drop tracerfrog from the same-turn-context-pulling
hint. The 0-or-2+ rule and ALL-CAPS parallel emphasis are unchanged.
* subagentModels: broader downshift coverage (gpt-pro, gemini-pro, grok); drop gpt-mini target
Scanned every resolved orchestrator slug in action/models.ts against
models.dev pricing data. Identified five clear cases where the
orchestrator is meaningfully expensive AND has a cheaper sibling that
remains capable enough for review-style judgment work.
Changes:
- Anthropic: opus → sonnet (kept; -40%)
- OpenAI: gpt → gpt-5.4 (was: gpt-mini; -54% instead of -85% but
preserves review-quality judgment — gpt-mini was too dumb)
- OpenAI: gpt-pro → gpt (NEW; -93%, biggest single unlock —
gpt-5.5-pro is $30/Mtok in vs gpt-5.5 at $5)
- Google: gemini-pro → gemini-flash (NEW; -75%)
- xAI: grok-4.3 → grok-4-1-fast (NEW; -80%)
Every branch handles the three routes in use: direct provider slug,
opencode-vendored, and openrouter-proxied. Variants below the downshift
target (mini/nano/flash/fast/sonnet/haiku) inherit (no further drop).
Skipped:
- DeepSeek: v4-flash ($0.14/Mtok) is too far below review judgment
threshold; v4-pro orchestrator already cheap ($0.55 blended).
- Moonshot: kimi-k2-thinking would only save 32% and slug stability on
OpenRouter is uncertain; revisit if cost matters.
- o3: already mid-tier in OpenAI's reasoning family; no clean target.
* models: hoist subagent downshift into the registry, add hidden flag
The downshift relationship now lives next to each alias's resolve /
openRouterResolve as a sibling field. Two new ModelDef fields:
- subagentModel?: string — alias key (within same provider) of the
cheaper sibling reviewfrog should use as a lens-fanout subagent.
e.g. claude-opus → 'claude-sonnet'.
- hidden?: boolean — exclude from selectable lists (UI dropdown,
CLI init picker). Does NOT affect resolution; for that use
fallback. Used so internal-only subagent targets like openai/gpt-5.4
exist in the registry but never appear as a user-facing pick.
Wiring:
- anthropic.claude-opus → claude-sonnet (-40%)
- openai.gpt-pro → gpt (-93%, biggest unlock)
- openai.gpt → gpt-5.4 (-54%); gpt-5.4 added with hidden:true
- google.gemini-pro → gemini-flash (-75%)
- mirrored across opencode + openrouter providers (each provider
declares its own three-route data so the downshift declaration
is colocated with the rest of the alias definition).
deriveSubagentModels collapses from ~85 lines of prefix-matching to
a ~15-line registry reverse-lookup: find the alias whose resolve OR
openRouterResolve matches the orchestrator's spec, follow its
subagentModel pointer, return the matching field of the target alias.
Filter sites updated:
- components/ModelSelector.tsx: !a.fallback && !a.hidden
- action/commands/init.ts: same
Tests rewritten to exercise the registry through the public surface;
the matrix collapses to one assertion per (provider × route) pair.
* TEMP: log per-step cost+tokens for subagent model verification (PR #710)
* TEMP: also log SUBAGENT step_finish from bus envelope handler
* remove temporary per-step diagnostic logs (verification done)
Verified subagent model downshift takes effect end-to-end on the OpenCode
+ openrouter path. PR #8 in pullfrog/preview-710-review-perf dispatched
3 lenses (billing-subsystem / security / correctness) on the orchestrator's
opus-4.7 session, and per-subagent step_finish events showed actual cost
exactly matching Sonnet pricing rates (60% of what Opus would have cost):
session n actual if-Opus if-Sonnet match
T3VrUuF... 5 $0.2425 $0.4042 $0.2425 Sonnet ✓
93ZZR7E... 4 $0.2253 $0.3754 $0.2253 Sonnet ✓
Fb1Kr7b... 4 $0.2495 $0.4158 $0.2495 Sonnet ✓
The startup '» subagent models: reviewfrog=...' line stays — useful
permanent diagnostic showing the resolved subagent model per-run.
* TEMP: log per-event model from claude.ts assistant handler
* remove temporary per-event model log (claude.ts verification done)
Verified subagent model downshift takes effect end-to-end on the Claude
Code path. PR #9 in pullfrog/preview-710-review-perf dispatched 2 lenses
on an opus-4-7 orchestrator. Per-assistant-event model field from the
SDK's stream-json output, partitioned by parent_tool_use_id:
ORCH (parent_tool_use_id=null): 17 events all model=claude-opus-4-7
SUBAGENT lens:billing-subsystem: 17 events all model=claude-sonnet-4-6
SUBAGENT lens:security: 21 events all model=claude-sonnet-4-6
Zero leakage to opus from either subagent session. The per-subagent
'model' field in --agents JSON is honored by claude-code at the SDK
level, identical to the OpenCode path verified earlier.
* opencode: bump per-call output cap 5K → 16K to unblock large reviews
The 5K cap (added in #616 to lower OpenRouter upfront budget reservation
for low-wallet runs) was capping the entire response of a single LLM call,
not just the budget reservation. A single tool_use response — like a
`create_pull_request_review` with many inline comments — would truncate
mid-stream past 5K output tokens, leave the JSON unparseable, and the tool
would never actually invoke. We hit this on PR #710's verify-downshift PR:
review aggregated from 3 lenses had 11 inline comments + a long body,
truncated at out=5000 on every retry attempt, action exited with 'Review
mode finished without calling create_pull_request_review after 3 retry
attempts'.
Investigated whether OpenCode (or OpenAI/Anthropic/OpenRouter directly)
exposes a separate budget-reservation parameter that could stay small
while letting the response exceed it. They don't — `max_tokens` /
`max_completion_tokens` is the single value all four use for both the
upfront reservation and the hard output ceiling. No way to decouple them
at the API surface.
Bumped to 16K as a middle ground: 8× the prior cap (handles every review
shape we've observed plus headroom), still half of OpenCode's 32K default
so the wallet-burn benefit for low-balance accounts is preserved, just
smaller. For Opus 4.7 a typical ~50K-input call now reserves roughly
$0.65 instead of the prior $0.38.
Updated the constant comment to spell out the trade-off clearly so this
doesn't happen again.
* opencode: drop OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX override entirely
Verified the original rationale for the override is obsolete. From #616
the cap shrunk OpenRouter's per-call upfront budget reservation so a
single call's reservation wouldn't exceed the per-run key cap
(`ROUTER_PER_RUN_LIMIT_USD = 25`) and lock low-balance accounts out of
starting a run.
That per-run gate is gone. `app/api/proxy-token/route.ts` ~line 422
explicitly says: 'No upper cap (the old ROUTER_PER_RUN_LIMIT_USD = 25 is
gone). The natural ceiling is whatever the user has + their buffer.'
Router now mints keys with `keyLimitCents = balance + buffer` ($50 for
autoreload+card, $5 for card-only, $0 for no-card). A single call's
upfront reservation fits comfortably within that — no separate per-call
gate to fail past.
The cap had a real downside as a hard per-call output truncation. A
single `create_pull_request_review` tool_use with many inline comments
would truncate mid-stream past 5K output tokens, the JSON would be
unparseable, and the tool never invoked. Hit on PR #710's
verify-downshift PR.
Removing the override entirely; OpenCode falls back to its 32K default.
Left an explanatory note above the env-var assignment site so the next
person doesn't unknowingly re-add it.
|