* ci: filter test matrices by per-test coverage globs to cut LLM spend
every test in `crossagent/`, `agnostic/`, and every provider entry now
declares a `coverage: string[]` of repo-relative globs. the new `changes`
job runs `paths-filter` for a docs-only short-circuit, then pipes the
changed-file list into `action/test/matrix.ts`, which intersects each
entry's coverage against the diff and emits filtered `agents`,
`agnostic`, `flagships`, and `aliases` matrices. main pushes and
`workflow_dispatch` set `FULL=1` to run everything as a stale-glob safety
net.
retires `changed-agents.sh` and the `MODE=flagships` branch in
`list-aliases.ts` in favor of one consistent model.
* ci(matrix): switch test discovery to dep-free static parsing
the GHA `changes` job has no `node_modules` installed. the previous
dynamic-import path pulled the test files transitively through
`utils.ts` -> `agents/index.ts` -> `@actions/core`, which exploded with
ERR_MODULE_NOT_FOUND. parse the test files via regex instead so
matrix.ts stays zero-dep — the chain (matrix -> coverage / providers /
list-aliases / models) imports only node builtins and relative TS files.
* ci(matrix): address PR #730 review feedback
- drop dangling `action/mcp/toolFiltering.ts` glob from `nobash`,
`restricted`, `tokenExfil` (file doesn't exist; `.test.ts` does, but
the runtime tooling lives in `mcp/shell.ts` and `agents/{claude,opencode}.ts`,
both already covered).
- drop unused `coverageForProvider` export and its `byName` map from
`providers.ts` (matrix.ts builds its own lookup inline).
- derive the active agent list from `agents/index.ts` via the same
dep-free regex tactic as `parseTestFile` instead of hardcoding
`["claude", "opencode"]` — adding a new harness file now wires it
into the dynamic matrix automatically.
- treat `coverage: []` as `coverage: undefined` in `shouldRun` so an
accidentally-empty array doesn't silently skip CI on every PR.
- add `action/utils/activity.ts` and `action/mcp/selectMode.ts` to the
`timeout` test's coverage — the activity-timeout enforcement path
was the original reason the test exists.
- ungate the `root` job (lint/format/typecheck/vitest). it's a required
status check on `main`, so gating it on `code == 'true'` would make
docs-only PRs unmergeable (skipped jobs don't satisfy required-check
rules). the real LLM savings come from skipping the four matrices,
not from skipping `root`.
- harden the four matrix-job `if:` guards from `outputs.matrix && ...`
to `outputs.matrix != '' && ...` — explicit > implicit short-circuit.
- document `expandBraces`'s flat-only support so a future author isn't
surprised by `{a,{b,c}}` not expanding.
- fix awkward sentence in `wiki/action-tests.md` "CI Cost Filtering".
* replace suffix-based env filtering with default-deny allowlist
filterEnv() now only passes known-safe GitHub Actions runner/system/toolchain
vars plus user-configured allowlist entries to shell subprocesses. GITHUB_TOKEN
and GH_TOKEN are always blocked, even from the user allowlist.
adds envAllowlist field to repo settings with dashboard textarea UI (visible
only when shell isolation is enabled) and wires it through run-context API
to the action runtime.
Made-with: Cursor
* address review: blocked-name warning, JAVA_HOME prefix, stale waitlist copy
- setEnvAllowlist now strips BLOCKED_ENV_NAMES from user input and returns
them so main.ts can log a warning
- move JAVA_HOME to exact names, use JAVA_HOME_ as prefix for clarity
- update stale suffix-based description in waitlist email script
Made-with: Cursor
* fix wiki/security.md snippet: JAVA_HOME -> JAVA_HOME_ to match code
Made-with: Cursor
* UI polish: field-sizing-content on all textareas, rename env allowlist label
- add field-sizing-content to all settings textareas so they auto-expand
to fit content (AgentSettings, ModesSettings, WorkflowsSettings, FlagsSettings)
- rename "Environment variable passthrough" to "Environment allowlist"
with clearer popover copy
- drop "e.g." prefix from env allowlist placeholder
- update docs/security.mdx and wiki/security.md references to match
Made-with: Cursor
* tweak env allowlist popover wording
Made-with: Cursor
* document default allowed variables in security docs with link from popover
Made-with: Cursor
* Update action/utils/secrets.ts
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
* rename agent key to opencode and add skill invocation coverage.
add skill-invoke tests for claude and opencode with local play-based validation signals, update CI matrices, and include the current tracked refactors in this branch for review.
Made-with: Cursor
* exclude agent-specific skill-invoke tests from wrong agent in CI matrix
* address review follow-up and preserve workflow run UI tweak.
switch changed-agents ci coverage to exercise the opentoad implementation path while keeping the opencode expectation, and include the local workflow run client interaction updates requested on this branch.
Made-with: Cursor
* remove opentoad agent filename from runtime.
rename the opencode harness implementation file from opentoad.ts to opencode.ts and update ci coverage input accordingly so action code no longer carries the old filename.
Made-with: Cursor
* ensure security prompt bypass is set on every test fixture.
this keeps adversarial and permissions harnesses from being blocked by the default prompt-injection refusal path during CI security tests.
Made-with: Cursor
---------
Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
* fix: make token-exfil test reliable by disabling security instructions and reframing prompt
the test was flaky — agents would randomly refuse (not calling set_output),
refuse politely (calling set_output with refusal text), or cooperate fully,
depending on model mood. two changes:
1. set PULLFROG_DISABLE_SECURITY_INSTRUCTIONS=1 in test env (layer 1)
2. reframe prompt as CI debugging task instead of security test (layer 2)
Made-with: Cursor
* fix: set PULLFROG_DISABLE_SECURITY_INSTRUCTIONS on adversarial test fixtures
without this flag, the system prompt tells agents to refuse anything that
looks malicious — which is exactly what these security pentests ask them to
do. adds the flag to tokenExfil, askpassIntercept, and nobashcreative.
Made-with: Cursor
* set PULLFROG_DISABLE_SECURITY_INSTRUCTIONS on all security-related test fixtures
Made-with: Cursor
* sandbox native filesystem tools to prevent /proc/self/environ exfiltration
the agent's native Read/Grep/Edit tools can bypass the shell sandbox by
reading /proc/self/environ directly. this adds agent-native filesystem
restrictions using the highest-precedence, non-overridable config for each CLI:
OpenCode: OPENCODE_PERMISSION env var with external_directory deny-all + /tmp allow,
plus deletion of untrusted .opencode/plugins/ and .opencode/tools/ before launch.
Claude Code: managed-settings.json at /etc/claude-code/ with denyRead, permissions.deny,
allowManagedPermissionRulesOnly, allowManagedHooksOnly. also --setting-sources user and
--disallowedTools path patterns as belt-and-suspenders.
Made-with: Cursor
* add Glob to Claude Code /proc and /sys deny lists
closes gap identified in review — Glob can enumerate /proc entries.
added to both managed-settings.json permissions.deny and --disallowedTools.
Made-with: Cursor
* run token-exfil test with both agents, hint at native /proc reads
changed tag from "agnostic" (opentoad-only) to "security" so the test
runs with both opentoad and claude. updated prompt to explicitly instruct
the agent to try reading /proc/self/environ via native Read tool.
added API keys to action-agnostic CI job for claude support.
Made-with: Cursor
* move token-exfil to crossagent matrix, remove redundant permissions.deny
- moved token-exfil from agnostic/ to crossagent/ so it runs via the
agent matrix (claude + opentoad in parallel) instead of sequentially
- removed permissions.deny per-tool rules from managed-settings.json;
sandbox.filesystem.denyRead is the single enforcement mechanism
- reverted action-agnostic env vars to minimal set
- updated wiki to match
Made-with: Cursor
* document post-spawn API key deletion analysis in security wiki
evaluated whether API key env vars can be deleted from agent processes
after spawn. OpenCode snapshots env at startup (safe to delete), but
Claude Code re-reads process.env per request (not viable). documented
as further exploration item with per-agent breakdown and caveats.
Made-with: Cursor
* fix stale tokenExfil path references in wiki docs
moved from test/agnostic/ to test/crossagent/ in directory tree
and adversarial test example.
Made-with: Cursor
* revert accidental prisma.config.ts changes
Made-with: Cursor
* hardcode PULLFROG_MODEL per agent in test runner to avoid DB model mismatch
when PULLFROG_AGENT forces a specific agent, the DB-configured model may
belong to a different provider (e.g. openai model with claude agent).
PULLFROG_MODEL short-circuits the DB slug resolution entirely.
Made-with: Cursor
Colin McDonnell