ShockVPN/shockbot
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
962 Commits 3 Branches 194 Tags
a0576a702a0709e2c164819876249ae202cb6e97
Commit Graph

5 Commits

Author SHA1 Message Date
Colin McDonnell a0576a702a opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite (#767) 
* opencode v2: harness adapted to opencode-ai 1.15+ SDK-v2 / Effect-ts CLI rewrite

Bumps `opencode-ai` from `1.1.56` → `1.15.1` and ports the harness to the
v2 NDJSON event contract. The legacy `opencode.ts` is kept as reference;
`opencode_v2.ts` is the active runner via `agents/index.ts`.

Why: `1.1.56` doesn't echo Gemini `thought_signature` back through the
MCP tool-call serializer, so direct-Google reviews 400 on the 3rd-ish
tool call. The fix only exists in the `1.14.x`+ line, which also ships
the SDK-v2 / Effect-ts CLI rewrite — taking the rewrite is mandatory.
Also unblocks the Codex ChatGPT-subscription auth path.

Surface area:

  - drop `init` / `message` / `result` / `tool_result` event types and
    handlers (no longer emitted at v1.14+ per upstream
    `cli/cmd/run.ts:588-601`).
  - `tool_use` is now a single event covering both `state.status:
    "completed"` and `"error"`. duration / subagent-finish bookkeeping
    moves from the v1 `tool_result` handler into the consolidated
    `tool_use` handler.
  - new `reasoning` event handler — gated on `--thinking`, surfaces
    Gemini-3 / OpenAI / Anthropic thinking blocks. `--thinking` added to
    `baseArgs`.
  - drop `pendingTaskDispatches` FIFO + `knownNonTaskCallIDs` set: at
    v1.15 the `task` tool callID is stable across the whole
    `tool-input-* → tool-call → tool-result/tool-error` chain
    (`session/processor.ts:282-330`). exact-match map is sufficient.
  - drop `experimental.batch_tool: true` from injected config — declared
    but inert at v1.15. re-add once upstream wires it back.
  - bin path: `bin/opencode` → `bin/opencode.exe` (postinstall renames
    the platform-specific binary into `opencode.exe` for every OS now).

Validated locally:

  - `pnpm test` 610/610 ✓
  - `pnpm play --raw` end-to-end with Anthropic via OpenRouter ✓
  - `pnpm play --raw` with `google/gemini-3.1-pro-preview`: 6 tool calls,
    multiple reasoning blocks visible, `set_output` propagates, exit 0 ✓
    (this is the headline `thought_signature` fix)
  - runtest opencode: smoke ✓, restricted ✓, nobash ✓, token-exfil ✓
  - runtest opencode: skill-invoke and mcpmerge fail (model-behavior
    drift on the new system prompt; wiring confirmed intact via direct
    repro showing both `robinMCP` and `pullfrog` MCP tools exposed).
    Tracked for follow-up; does not gate the migration.

Plugin (`opencodePlugin.ts`) and skill discovery paths are unchanged at
v1.15 — verified upstream and reused as-is. Bus subscription via
`bus.subscribeAll()` and the `event` hook still fan out every payload.

* model-smoke: bump opencode bin path to opencode.exe (v1.14+ rename)

The v1.14+ postinstall.mjs renames the platform-specific binary to
`bin/opencode.exe` for every OS (incl. linux/darwin), not just Windows.
Mirrors the fix in action/agents/opencode_v2.ts.

* opencode v2: set PWD env explicitly to fix skill / project-config discovery

Root cause for skill-invoke + mcpmerge harness regressions: opencode-ai 1.15
reads `process.env.PWD` first (with `process.cwd()` as fallback) when
resolving the SDK client's `directory` parameter — see upstream
`cli/cmd/run.ts:282`:

  const root = Filesystem.resolve(process.env.PWD ?? process.cwd())

We pass `cwd: repoDir` to spawn, but the child inherits the harness's PWD
via `...process.env`. Under `pnpm runtest` (and `pnpm play`) PWD is the
`action/` directory, not the cloned test repo. Result: opencode creates
two instances per session — one at `process.cwd()` (correct) and one at
`PWD` (wrong) — and the agent's session runs in the PWD-derived one,
which can't see the project's `.opencode/skills/` or `.claude/skills/`.

Empirically traced via the full opencode stderr trace under the runtest
harness: `service=skill count=3 init` (no `pullfrog-skill-check`) plus a
second `service=default directory=<harness-pwd> creating instance` line
per run. With `PWD=repoDir` set explicitly, `count=4 init` includes the
test skill, the agent reaches for `skill({"name":"pullfrog-skill-check"})`
exactly as the validator expects, and mcpmerge's `robinMCP_get_test_value`
becomes accessible too.

Validated locally: skill-invoke-opencode ✓, mcpmerge-opencode ✓, smoke ✓,
restricted ✓, nobash ✓, token-exfil ✓ (flaked once on a model-narration
match, passes on retry; unrelated to PWD).

* opencode v2: drop ThinkingTimer; use opencode's reasoning.part.time directly

opencode-ai 1.15 emits `reasoning` parts with `time.start` / `time.end`
on terminal state (`cli/cmd/run.ts:671`), giving us a precise per-block
"thought for X s" duration straight from the runtime. The v1
ThinkingTimer heuristic — measuring wall-clock between markToolResult
and the next markToolCall — was an approximation when no native source
existed; with v2 it's redundant and noisy (it would log alongside the
real reasoning event, and conflated network latency with model thinking).

Removed: `ThinkingTimer` import, `thinkingTimers` Map, `timerFor()`
helper, both `markToolCall` / `markToolResult` call sites in `tool_use`.
The `reasoning` handler now reads `part.time.start/end` directly and
prefixes the visible preview with `(X.Ys)`.

Output before: `» thinking: <preview>` + `» thought for 4.0s` (separate)
Output now:    `» thinking (4.0s): <preview>` (one line, sourced)

For models that don't emit reasoning (Sonnet without extended thinking,
GPT-4o, etc.), there's just no thinking line — which matches reality
better than the gap-heuristic, which would fire on any pause >3s
including provider-side latency that wasn't actual model reasoning.

Validated locally: skill-invoke ✓, mcpmerge ✓, smoke ✓, Gemini play
shows `» thinking (4.0s)` and `» thinking (0.8s)` from real durations.

* claude.ts: same PWD fix as opencode v2; entryPost: refresh stale comment

claude-code 2.1.x reads `process.env.PWD` and registers it as a "session"
additional-working-directory when it differs from `process.cwd()` (per the
bundled cli.js: `let H = process.env.PWD; if (H && H !== Y7() && ...)
j.set(H, { path: H, source: "session" })`). Without overriding PWD on the
spawn env, claude inherits the harness's PWD via `...process.env` — under
`pnpm runtest` / `pnpm play` that's `action/`, not the cloned test repo —
and adds the wrong dir to the agent's allowed working set.

Symmetric to the opencode v2 fix in 52337f9. Pre-empts the same class of
"agent's session sees the wrong cwd" failures on the claude side.

Also refresh the stale `action/agents/opencode.ts` reference in
entryPost.ts to point at opencode_v2.ts (the active runner), with the v1
file noted as kept-for-reference.

* opencode: extract shared helpers into opencodeShared.ts; v2 cleanup

Code-quality pass on the v2 work:

1. New `agents/opencodeShared.ts` (144 lines) for genuinely-shared helpers
   between v1 and v2:
   - `OpenCodeConfig` type
   - `geminiHighThinkingOverrides()` (registry-driven Gemini thinking pin)
   - `buildReviewerAgentConfig()` (reviewfrog config builder, was in v1
     and re-imported by v2 via a back-reference)
   - `installOpencodeCli({ binPath })` (parameterized — v1 passes
     `bin/opencode`, v2 passes `bin/opencode.exe` via a per-version
     `installCli` lambda; matches each pinned version's npm shape)
   - `autoSelectModel()` + `getOpenCodeModels()` model-registry fallback

   v2 drops the `import { ... } from "./opencode.ts"` back-reference; v1
   keeps a one-line `export { geminiHighThinkingOverrides }` re-export
   so `opencode.test.ts` keeps working unchanged. Once v1 is retired
   (post burn-in) opencodeShared collapses back into v2.

2. `opencode_v2.ts` cleanup:
   - drop dead state (`currentStepId`, `stepHistory` were write-only —
     their reader was the v1 `tool_result` handler we deleted)
   - hoist `state` in `tool_use` handler; replace nested-ternary payload
     extraction with a `terminalPayload(state)` helper
   - extract `formatPartDuration(time)` for the reasoning-block
     "(X.Ys)" suffix
   - tighten `OpenCodeBusEnvelopeEvent` type to include `tool` /
     `callID` fields directly, drop the `partWithToolFields` cast
   - trim docblocks per AGENTS.md "≤ 2-3 lines per code line": reasoning
     handler, tool_use handler, bus envelope handler all shortened
   - `step_start` becomes an explicit `() => {}` no-op so the dispatcher
     doesn't log "unhandled event" for every step

3. `subagentRegistration.test.ts` retargeted at the new file split —
   reads opencodeShared.ts for the buildReviewerAgentConfig assertions
   and opencode_v2.ts for the orchestrator-model wire-through.

Net: -306 source lines (1339+1130 → 1228+1031+144). Tests + lint + format
+ typecheck all green; skill-invoke-opencode ✓ and smoke ✓ verified
against the refactored v2 runtime.

* opencode v2: address PR review feedback

Three fixes from the inline review threads on #767:

1. Activity-diagnostic ordering bug (Copilot review at L705): the chunk-
   level `markActivity()` resets the module-level idle counter, so the
   per-event `getIdleMs()` sample inside the dispatch loop was always
   ~0ms — the "no activity for Xs" diagnostic never fired. Replaced with
   a runner-local `lastEventAt` so we measure real event-to-event silence
   instead of chunk-arrival latency. Drop the unused `getIdleMs` import.

2. TDZ-defensive hoist (Pullfrog review nit): `agentErrorEvent`,
   `lastProviderError`, and `recentStderr` are closed over by the
   `handlers` const but were declared after it. No current bug because
   handlers only fire inside the awaited `spawn()`, but a future
   refactor that triggers a handler synchronously during setup would
   surface a TDZ. Hoisted above `handlers`.

3. `step_finish.part.tokens.reasoning` follow-up (Pullfrog review at
   L566): leave a `TODO` comment marking the gap until `AgentUsage`
   grows a `reasoningTokens` field — separate PR with schema work.
   Cost totals stay correct because `part.cost` is summed independently.

Other thread states for the record:
- Copilot L63 (geminiHighThinkingOverrides import from legacy): already
  fixed by the opencodeShared.ts extraction in 83a7cab.
- Copilot L672 (ThinkingTimer over-reports on terminal events): already
  fixed by dropping ThinkingTimer in a1e536b — we use opencode's own
  `reasoning.part.time.{start,end}` for thinking durations now.
- Pullfrog L642 (onToolUse double-fire on subagent dispatch): re-checked
  the bus-envelope flow; the plugin filters orchestrator events except
  for status=running task dispatches, and bus-envelope returns before
  calling handlers.tool_use on those. No double-fire under current code.

Validated: 610/610 unit tests, lint + format + typecheck clean,
skill-invoke-opencode ✓.

* DX: flip pnpm play / pnpm runtest to docker-by-default

Restores the script shape wiki/docker.md has documented since the docker
rewrite (#750). PR #756 inadvertently reverted action/package.json's
gha/play/runtest scripts to host-only and dropped the :local variants;
the wiki kept the new shape, so docs and reality drifted. The OpenCode-v2
migration agent ran `pnpm play --raw …` host-side throughout because the
host entry was the only thing that existed.

scripts (root → action):
- pnpm play       → pnpm -C action gha play.ts          (docker, default)
- pnpm play:local → pnpm -C action play:local           (host)
- pnpm runtest    → pnpm -C action gha test/run.ts      (docker, default)
- pnpm runtest:local → pnpm -C action runtest:local     (host)
- pnpm gha is restored in action/package.json (re-adds `node gha.ts`)

action/package.json deliberately ships only the :local variants — bare
`pnpm -C action play` now errors instead of silently bypassing docker.
This is a tradeoff per the user prompt's "consider whether NAMES should
change" hint: the explicit error is worth the small CI churn.

CI workflows: `.github/workflows/test.yml` and
`action/.github/workflows/test.yml` flipped from `pnpm runtest …` to
`pnpm runtest:local …`. Semantics unchanged — they still execute
`node test/run.ts` directly on the GHA Linux runner; nesting docker on
GHA is unnecessary overhead. Only the script name changed to match the
new package.json.

Webhook tester: the existing root `pnpm play` was actually a webhook
handler smoke harness (root play.ts), unrelated to the action runtime.
Renamed root play.ts → webhook.ts and exposed it as `pnpm webhook` so
`pnpm play` can carry the docker-by-default action shortcut without
collision. README updated.

File headers updated:
- action/play.ts: invocation block now points at `pnpm play` /
  `pnpm play:local`
- action/test/run.ts: same
- action/gha.ts: usage block calls out the new shortcut wrappers

AGENTS.md: extended the existing "local sanity checks of action tool
logic" rule with the play / play:local / runtest / runtest:local
selection guidance and the `cd action; pnpm play` footgun note.

wiki/docker.md unchanged — already described the now-real shape.

* test/crossagent: add codex-auth smoke

Pins openai/gpt-5.5 (in opencode's Codex ALLOWED_MODELS) and runs the
full opencode harness against the env-provided CODEX_AUTH_JSON. Verifies:

  - installCodexAuth() materializes auth.json under the test HOME
  - opencode routes openai requests through ChatGPT subscription auth
    (no OPENAI_API_KEY in env, AT path forced via expires: 0)
  - the refresh chain advances during the run (refresh_token rotates)
  - detectCodexRefresh() would surface the rotation to entryPost.ts

The post-hook write-back fetch isn't reachable from `pnpm runtest`
(it's a separate GHA `post:` step). The integration boundary that
matters end-to-end is "did the on-disk auth.json change in a way
detectCodexRefresh recognizes" — that's exactly what this test asserts.

CI wiring (already committed in a1c1fd4f as part of the DX flip):
  - .github/workflows/test.yml: CODEX_AUTH_JSON via secrets in
    action-agents env block
  - action/.github/workflows/test.yml: same; codex-auth in the
    hardcoded test matrix with a claude exclude

The provisioning step on the user's side is `gh secret set
CODEX_AUTH_JSON --repo pullfrog/app < auth.json`.

ci.test.ts: expectedAgentEnvVars now includes provider
`managedCredentials` so the "env vars cover all provider API keys"
invariant stays self-correcting as more managed credentials land.

* docs(codex-auth): make storage requirement unmissable

A previous reviewing agent on this branch came away thinking
`CODEX_AUTH_JSON` could live in GitHub Actions secrets. It can't —
`entryPost.ts` rewrites the rotated refresh token after every run, and GH
Actions secrets are immutable at runtime, so any non-Pullfrog-Postgres
storage breaks the refresh chain on the first rotation (~1h silent
expiry).

- wiki/codex-auth.md: prominent `[!IMPORTANT]` callout above the fold,
  with the words "GitHub Actions secrets DO NOT WORK" verbatim and an
  enumeration of broken alternatives.
- action/utils/codexHome.ts + action/entryPost.ts: header comments now
  loudly contrast Pullfrog secret store vs GH Actions and explain the
  writeback constraint.
- AGENTS.md: terse one-bullet rule next to the model-resolution rule so
  future agents don't repeat the mistake.
- .github/workflows/test.yml + action/.github/workflows/test.yml: added a
  comment marking the existing `secrets.CODEX_AUTH_JSON` injection as a
  CI smoke-testing shortcut, not the canonical pattern. CI wiring itself
  unchanged per scope.

* auth codex: auto-open device URL, drop --scope flag

- detect `https://auth.openai.com/codex/device...` from codex CLI output
  and best-effort launch it in the user's default browser (open / xdg-open
  / cmd start, wslview fallback on linux). gated so we only open once per
  flow; failures are swallowed so manual copy-paste still works.
- drop the `--scope` flag entirely. the device-code flow is fundamentally
  interactive (browser approval), so a "skip-the-prompt" flag for just one
  of the prompts was dead weight. collapses scope selection to "always
  prompt on org-owned, always account on user-owned".

* rename gha→docker, flip play/runtest defaults to host

the previous shape conflated "real GitHub Actions" with the local docker
container that mocks it, and made the slow docker path the default for
fast-iteration scripts.

- `action/gha.ts` → `action/docker.ts` (banner, --doctor, --help, image
  tag `pullfrog-docker:*`, volume `pullfrog-docker-node-modules-*`,
  tmpdir, error messages)
- `pnpm play` / `pnpm runtest` now default to host (fast iteration);
  `pnpm play:docker` / `pnpm runtest:docker` run inside the container
- `pnpm gha` → `pnpm docker` (the container runner shortcut)
- `pnpm webhook` → `pnpm play:webhook` (fits the play: namespace; the
  bare name implied a webhook server, which hookdeck-cli already is)
- update docs (`wiki/{docker,action-tests,billing,adversarial,browser}.md`,
  `README.md`, `AGENTS.md`), CI workflows
  (`.github/workflows/test.yml`, `action/.github/workflows/test.yml`),
  and code headers (`action/{play,test/run,utils/runFixture}.ts`,
  `webhook.ts`, `action/test/coverage.ts`)

`action/commands/gha.ts` keeps its name — it's the real GitHub Actions
entry point for the `pullfrog gha` CLI command (not the docker mock).

* fix(codex): route post-hook writeback through apiFetch + conditional skip

Three threads addressing PR #767 followups.

action/entryPost.ts: replace raw fetch() with apiFetch() so the
PUT /api/runtime/secret call carries the x-vercel-protection-bypass
header/query when targeting a preview deployment. raw fetch silently
401s against the Vercel SSO gate, so every preview-env Codex run was
losing its rotated refresh token. production is unaffected (no SSO).

action/test/crossagent/codexAuth.ts: gate the test on CODEX_AUTH_JSON
via new TestRunnerOptions.skipIf hook. when the secret is absent
(forks, contributors without it), runTestForAgent short-circuits to a
passing-with-skipped ValidationResult before any agent spawn — so the
matrix's fail-fast: true setting doesn't cascade-cancel siblings. CI
on pullfrog/app and dev-local with .env both still run the test for
real. printSingleValidation/printResults now render skipped entries
distinctly.

doc/comment drift:
- docs/codex-auth.mdx, wiki/codex-auth.md: drop stale --scope flag
  mention (removed in 10be96db, scope is now always interactively
  prompted or implicit).
- wiki/codex-auth.md: tighten Claude-defense wording — materialization
  is agent-gated (opencode/opencode_v2 harness), not model-gated;
  opencode runs with non-OpenAI models still materialize the file,
  it's just not read.
- action/Dockerfile, action/docker-entrypoint.sh: pnpm gha / gha.ts
  → pnpm docker / docker.ts (renamed in a2a63929).
- app/api/runtime/secret/route.ts: refer to the save-time scope prompt
  instead of the dropped --scope flag.

* smoke: force ≥2 tool calls; document test-bar in wiki + AGENTS

upgrade crossagent/smoke prompt to call pullfrog_git status before
set_output. this exercises the 2nd model→agent round-trip across every
providers-live flagship, catching bugs like the Gemini thought_signature
echo that single-tool-call tests can't see.

also adds the "bar for adding new LLM-driven tests" section to
wiki/action-tests.md and an extension to the existing AGENTS.md
no-tests rule pointing at it — prefer upgrading existing matrix entries
over adding new ones.

local: pnpm runtest smoke opencode passes against both
anthropic/claude-sonnet-4-6 and google/gemini-pro.

---------

Co-authored-by: Colin McDonnell <colinmcd94@M1chelle.local>
 2026-05-20 04:05:16 +00:00
Colin McDonnell cb0dbcd371 feat(action): make prepush hook non-blocking after one failure (#777) 
* feat(action): make prepush hook non-blocking after one failure

push_branch now treats the repository's prepush hook as best-effort: it
runs at most once per run, surfaces the failure output if the script
exits non-zero, and every subsequent push_branch call this run skips
the hook so the agent isn't blocked by failures unrelated to its
change. The agent can iterate by running the hook command itself via
the shell tool when shell access is available; push_branch will not
re-run the hook automatically after a failure.

Why: a one-line OSS-allowlist change took 9 minutes (#776) because the
agent retried push_branch six times against a prepush hook that was
failing for env-leak and missing-build-artifact reasons unrelated to
the change. CI catches the same checks on the GitHub side; the local
prepush gate was duplicating work and blocking unrelated fixes.

- ToolState: new prepushFailureCount counter (per-run, never resets)
- executeLifecycleHook: returns structured failure (kind/output/exitCode)
  so prepush can compose its own agent-facing message instead of
  inheriting the generic retry/no-retry advice meant for setup
- push_branch: composes a shell-mode-aware error message; surfaces
  prepushSkipped on the success payload + appends a note to the message
- instructions.ts + wiki/prompt.md + docs/comparisons.mdx: updated to
  reflect best-effort semantics

* fix(action): clarify prepush latch semantics + soften static guidance

review fixes from PR #777:

- toolState comment, instructions, success message, tool description:
  replace "runs at most once per run" / "first call only" wording with
  the actual semantic — successful prepush keeps running on later
  push_branch calls; only a hook FAILURE latches the bypass.
- tool description: drop hardcoded "via the shell tool" guidance so
  the static description doesn't mislead in shell:disabled runs (the
  dynamic agent prompt in instructions.ts already does shell-conditional
  messaging).
- LifecycleHookFailure.output JSDoc: match the implementation
  (stderr-preferred fallback to stdout, empty for timeout/spawn).

* fix(action): shorten prepush-skip log to terse operator telemetry

the previous log line tried to address the agent ("re-run the hook
command yourself via shell"), but log.info writes to the action
runtime's stdout — the agent never sees it. agent-facing skip
guidance already lives in the error message from
buildPrepushFailureMessage, the success message when bypassed, and
the system prompt in instructions.ts. log line is now just operator
telemetry.

* refactor(action): drop slop from prepush soft-fail

self-audit pass after the previous review-fix round. removed
duplication between code-level comment and the five other places that
already explain the same behavior, tightened verbose JSDoc, and
collapsed redundant clauses in agent-facing strings.

- LifecycleHookFailure → discriminated union. drops the optional
  exitCode/spawnError fields (and the empty-output sentinel for
  timeout/spawn) plus the corresponding ?? fallbacks in the helper.
- PushBranchTool: 7-line code comment above the latch removed
  (toolState field comment + tool description + error message +
  success message + system prompt all already cover it). tool
  description third sentence dropped (restated the second). success
  message tightened to a parenthetical.
- buildPrepushFailureMessage: 4-line JSDoc → 1 line. shared "if you
  think the failure could indicate a real bug in your code" prefix
  factored out across the shell-conditional branches.
- ToolState.prepushFailureCount comment: 8 lines → 3. the "what" is
  in git.ts; comment now only documents the invariant (never
  decremented within a run).
- instructions.ts prepush guidance: collapsed nested bullets + ternary
  into one paragraph; dropped the "so re-running via shell is the only
  way…" tail that restated "push_branch will NOT re-run it".

* fix(action): hint prepush bypass on dirty tree after hook failure

When push_branch blocks on a dirty working tree and the prepush latch
is already set, tell the agent the hook will be skipped once the tree is clean.

* fix(test): narrow CI matrix for lifecycle and toolState changes

Remove lifecycle.ts from ALWAYS_RUN_ALL and add lifecycle.ts + toolState.ts
to push/git agnostic test coverage so PRs touching prepush latch logic run
targeted tests instead of the full matrix.
 2026-05-20 02:43:23 +00:00
Colin McDonnell 69c7d4b8cd models-bump: harden CI and bot prompt against catalog hallucinations 
PR #790 (the first bot-authored models-bump PR) shipped a broken bump for
openrouter/gemini-flash: the bot pattern-matched the parallel google bump
and fabricated `openrouter/google/gemini-3.5-flash`, which exists on the
OpenRouter API but not on models.dev's openrouter section — the catalog
OpenCode actually reads. The slug failed at runtime with `Model not found`.
Two CI gaps let it through:

1. `models-live` matrix pruned every `openrouter/*` and keyed `opencode/*`
   alias as a "passthrough", smoke-testing only one canary per routing
   layer. But those aren't passthroughs — each is a distinct models.dev
   catalog entry that can drift independently of the direct-provider
   mirror. Drop the pruning; smoke every keyed alias (53 jobs, up from 25).
   Only `bedrock/byok` stays pruned (sentinel resolve).

2. `models-catalog` test (the integrity gate that asserts every resolve
   exists on models.dev) was main-only by design — to keep upstream catalog
   churn from blocking unrelated PRs. But it's exactly the test we want
   running on the bot's own catalog edits. Add `pullfrog/models-bump`
   head-ref to its trigger.

Also tighten the bot prompt in models-bump.yml: new rule 0 requires every
new `resolve` to equal `<alias-provider>/<c.modelId>` for some `c` in the
alias's own `candidates[]` in models-bump-context.json — the deterministic
preprocessor only emits candidates sourced from models.dev's mirror, so
this gates against the cross-alias pattern-matching that broke PR #790.
For `openRouterResolve` the gate is `openRouterCandidates[]` (OpenRouter
API), which is necessary but not sufficient; the `models-catalog` job is
the authoritative models.dev check.

Verified locally:
- baseline `pnpm -C action test:catalog` passes 133 tests
- simulated the PR #790 hunk (sed'd `openrouter/google/gemini-3.5-flash`
  into action/models.ts) and the catalog test fails with the right
  assertion: `model "google/gemini-3.5-flash" not found under openrouter
  on models.dev`
- `FULL=1 node action/test/matrix.ts` emits 53 aliases (was 25); every
  openrouter/* alias and every keyed opencode/* alias now smoked
 2026-05-20 02:18:38 +00:00
Colin McDonnell 76879b27ec docker testing rewrite: bake the image, drop the allowlist, kill the quoting (#750) 
* docker testing rewrite: bake the image, drop the allowlist, kill the quoting

- new `pnpm gha <script>` wrapper. one entry point for running any node
  script in the GHA-like container; replaces the runtime apt-get +
  useradd + chown ceremony in `action/utils/docker.ts`.
- `action/Dockerfile` bakes ubuntu:24.04 + node 24 + gh + jq + sudo +
  testuser at uid 1000. `action/docker-entrypoint.sh` remaps to the host
  uid/gid and `exec`s the requested command — no `bash -c` nesting, no
  `escapeForDoubleQuotes`.
- env passthrough: full `process.env` (+ `.env` via dotenv) flows through
  `--env-file`, multi-line values via `-e` fallback. drops
  `EnvFilterMode` / `testEnvAllowList`.
- image rebuild is content-hash gated on Dockerfile + entrypoint; volume
  is versioned by hash so a stale `node_modules` cache from an old image
  can't poison a new one.
- `action/play.ts` slimmed to a CLI; `run()` extracted to
  `action/utils/runFixture.ts`. drops the `--local` / `PLAY_LOCAL` dual
  mode in favor of explicit `play:local` / `runtest:local` scripts.
- `action/test/run.ts` no longer self-relaunches into docker — that's
  `gha`'s job now.
- `action/test/coverage.ts` `ALWAYS_RUN_ALL` updated to track the new
  files.
- `wiki/docker.md` rewritten (243 → 105 lines). `wiki/action-tests.md`,
  `wiki/billing.md`, `wiki/adversarial.md`, `README.md`, `AGENTS.md` all
  updated to drop `--local` / `PLAY_LOCAL` references.

verified end-to-end: `pnpm play` runs the default fixture against
pullfrog/scratch, exit 0; `sudo unshare --pid` still works inside the
container; `pnpm runtest` boots through the wrapper.

* gha: address review feedback + 3 related issues found locally

review-flagged:
- bare `pnpm gha --build` now builds the image and exits 0 (was
  printing help and exiting 1 — docs claimed it was a valid standalone)
- `initVolumeOwnership` skipped when the named volume already exists;
  saves the ~240ms `docker run … chown` on every warm invocation
- `GIT_SSH_COMMAND` gate widened to any `id_*` private key (was hard-
  coded to `id_rsa`, leaving ed25519-only linux contributors with the
  default ssh config). dropped `-i` so ssh picks whichever key exists
- new `action/.dockerignore` — partial mitigation noted: BuildKit
  (default since docker 23) only sends files referenced by the
  Dockerfile (~42B in practice), so the perf concern is mostly
  hypothetical. file is still worth keeping for `DOCKER_BUILDKIT=0`
  fallback and as documented intent for future `COPY . .` additions

related issues found while validating locally:
- `parseArgs` now stops flag-parsing at the first positional (or
  literal `--`); `pnpm gha test/run.ts --build` previously
  intercepted `--build` as a gha flag instead of forwarding to
  `test/run.ts`
- new `pnpm gha --clean` command prunes orphan `pullfrog-gha:*`
  images and `pullfrog-gha-node-modules-*` volumes whose hash
  doesn't match the current Dockerfile (each Dockerfile/entrypoint
  edit creates a fresh hash and orphans the prior pair, ~600MB +
  ~200MB each — without a cleaner they accumulate silently)
- `--shell` without a TTY now fails fast with an actionable message
  before docker is invoked, instead of producing the confusing
  `the input device is not a TTY` from docker run

wiki updated: documents `--clean`, the parseArgs passthrough rule,
and a new "Reclaiming disk" section.

* gha: fidelity, flexibility, and signal-safety improvements

investigated local fidelity vs the real GHA ubuntu-24.04 runner and
addressed the gaps that have actually bitten contributors or could.

fidelity (image now matches GHA closer):
- bake build-essential, wget, xz-utils, file alongside the existing
  toolset. gh, jq, git, python3, sudo, ssh, build-essential, wget,
  xz, file, unzip, curl all present. native module builds (node-gyp,
  any package missing arm64 prebuilts) now work; common agent shell
  calls don't hit ENOENT
- `host.docker.internal:host-gateway` flag wires the host into the
  container's DNS on linux (macOS Docker Desktop bakes it in). lets
  scripts that hit a local dev server use `API_URL=http://host.docker.
  internal:3100` and work identically on both platforms
- `--init` makes tini PID 1, fixing signal forwarding during the
  pre-exec warmup window (Ctrl-C was previously taking up to 10s to
  tear down because bash-as-PID-1 swallowed the signal)
- pnpm version is correctly pinned via the workspace's
  `packageManager` field — corepack resolves it at install time;
  verified via the new `--doctor` command

flexibility (new affordances):
- `pnpm gha --doctor` runs an inside-the-container fidelity audit:
  os + arch + node/pnpm/python versions, version snapshots of every
  baked tool, env vars (CI, HOME, TMPDIR), uid/gid, and the
  host.docker.internal resolution. useful for "works in CI fails
  locally" or vice versa
- `pnpm gha --build --no-cache` busts the docker layer cache when
  an apt mirror, base image, or external download has changed
  upstream
- entrypoint's `pnpm install` warmup is now wrapped in a `flock` on
  a file in the shared node_modules volume — concurrent `pnpm gha`
  invocations (e.g. play in one terminal, runtest in another)
  serialize their install instead of racing

docs:
- new "Gaps (known)" section in wiki/docker.md explicitly calling
  out the things this system can't do yet, including the missing
  `uses: ./action` semantics gap that
  `.github/workflows/action-gha-e2e-adhoc.yml` currently fills via
  GHA only (designing a local `pnpm gha-action <fixture>` is on the
  roadmap), service containers, parallel-run sharing, and arch
  differences (arm64 vs amd64)

* docs: audit + corrections after testing fronts

self-audit pass for stale references and incomplete pointers:

- wiki/browser.md: `Docker (node:24)` → `pnpm gha container (ubuntu:24.04)`.
  the substance was right (chrome not preinstalled) but the base image
  reference was stale.
- wiki/docker.md: the "Permission errors" troubleshooting line claimed
  the node_modules volume is chowned on every run; now correctly says
  "owned by the host uid on first creation; warm runs skip the chown"
  to match the actual behavior after the initVolumeOwnership fix.
- wiki/action-tests.md: `API_URL` env-var doc now mentions BOTH paths
  (`localhost:` from play:local, `host.docker.internal:` from inside
  the container). Proxy/router recipe now shows both invocations
  side-by-side instead of saying "must use play:local".
- wiki/billing.md: same dual-recipe update for the loop-including-the-
  action proxy walkthrough.
- gha.ts header: expanded the usage block to include --clean / --doctor /
  --no-cache / --shell-TTY, added the host.docker.internal note, and
  pointed at wiki/docker.md for design rationale.

self-document check: a future agent landing on this code can answer
"how do I run a fixture / debug in shell / add a tool / diagnose
fidelity / reach a local dev server" purely from gha.ts header +
wiki/docker.md without spelunking through the entrypoint or git
history.
 2026-05-16 03:12:25 +00:00
Colin McDonnell 1f4c3031be ci: filter test matrices by per-test coverage globs (#730) 
* ci: filter test matrices by per-test coverage globs to cut LLM spend

every test in `crossagent/`, `agnostic/`, and every provider entry now
declares a `coverage: string[]` of repo-relative globs. the new `changes`
job runs `paths-filter` for a docs-only short-circuit, then pipes the
changed-file list into `action/test/matrix.ts`, which intersects each
entry's coverage against the diff and emits filtered `agents`,
`agnostic`, `flagships`, and `aliases` matrices. main pushes and
`workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety
net.

retires `changed-agents.sh` and the `MODE=flagships` branch in
`list-aliases.ts` in favor of one consistent model.

* ci(matrix): switch test discovery to dep-free static parsing

the GHA `changes` job has no `node_modules` installed. the previous
dynamic-import path pulled the test files transitively through
`utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with
ERR_MODULE_NOT_FOUND. parse the test files via regex instead so
matrix.ts stays zero-dep — the chain (matrix -> coverage / providers /
list-aliases / models) imports only node builtins and relative TS files.

* ci(matrix): address PR #730 review feedback

- drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`,
  `restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but
  the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`,
  both already covered).
- drop unused `coverageForProvider` export and its `byName` map from
  `providers.ts` (matrix.ts builds its own lookup inline).
- derive the active agent list from `agents/index.ts` via the same
  dep-free regex tactic as `parseTestFile` instead of hardcoding
  `["claude", "opencode"]` — adding a new harness file now wires it
  into the dynamic matrix automatically.
- treat `coverage: []` as `coverage: undefined` in `shouldRun` so an
  accidentally-empty array doesn't silently skip CI on every PR.
- add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the
  `timeout` test's coverage — the activity-timeout enforcement path
  was the original reason the test exists.
- ungate the `root` job (lint/format/typecheck/vitest). it's a required
  status check on `main`, so gating it on `code == 'true'` would make
  docs-only PRs unmergeable (skipped jobs don't satisfy required-check
  rules). the real LLM savings come from skipping the four matrices,
  not from skipping `root`.
- harden the four matrix-job `if:` guards from `outputs.matrix && ...`
  to `outputs.matrix != '' && ...` — explicit > implicit short-circuit.
- document `expandBraces`'s flat-only support so a future author isn't
  surprised by `{a,{b,c}}` not expanding.
- fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".
 2026-05-14 03:55:33 +00:00