Files
shockbot/agents/shared.ts
T
David Blass c6a757424c Stop hook + learnings reflection via post-run loop (#515) (#548)
* add stop hook + learnings reflection to post-run loop (#515)

stop hook (#515): repo-configured script that runs after the agent
finishes. non-zero exit resumes the agent with the hook output as
guidance; persistent failure (3 attempts) marks the run failed. the
dirty-tree and stop-hook gates share a single retry loop so a fix +
push happen in one turn.

learnings reflection: per Colin, the learnings step baked into mode
checklists rarely fires — the agent stays focused on the task and the
meta-ask falls through. the post-run loop now delivers a dedicated
one-shot --continue turn asking the agent to call update_learnings if
relevant, nothing else competing for attention. reflection doesn't
consume the gate-retry budget; if it dirties the tree, the next loop
iteration catches it via the dirty-tree gate.

plumbing: Repo.stopScript column + migration, zod schema, run-context
api, AgentSettings UI. RepoSettings.stopScript threads through to
AgentRunContext and into each agent harness.

subprocess-dependent logic lives in action/agents/postRun.ts to keep
action/agents/shared.ts lean — shared.ts is reachable from
pullfrog/internal, and pulling node:child_process through it leaks
into root tsc (which uses bundler resolution, not NodeNext).

* fix: preserve successful run when reflection turn fails

The post-run reflection turn (update_learnings nudge) is a best-effort
one-shot; its failure must not flip a successful run to failed. Prior
code overwrote `result` with the reflection's return value, so a model
API error during reflection caused the whole run to be reported as
failed even though the gated work had already completed cleanly.

Now: save the pre-reflection result, and if reflection returns
`success: false`, log a warning, restore the prior success, and exit
without re-invoking the gates (re-running a freshly-green stop hook
risks a flaky false-positive failure).

Adds action/agents/postRun.test.ts covering the reflection path —
previously uncovered.

* fix: surface both stop-hook stdout and stderr to the agent

The `(stderr || stdout)` heuristic in executeStopHook dropped stdout
entirely whenever stderr had any content. Scripts that emit a benign
warning to stderr and the actionable error to stdout (common for
wrapper scripts) starved the agent of the information it needed to
fix the issue.

Now concatenate both streams (stderr first, stdout second, skipping
empty ones) before truncation. This keeps stdout's tail — usually
where summaries and totals live — intact under the 4096-char cap.

* test: lock in the core post-run retry + reflection invariants

PR #548's test plan ships four manual verification scenarios.
Convert three to vitest coverage, catching regressions on the hottest
code paths:

- persistent stop hook failure exhausts MAX_POST_RUN_RETRIES and
  surfaces as AgentResult.error with both the retry count and the
  verbatim hook output (so the GitHub-comment rendering stays
  actionable).
- every gate retry is fed the hook output as the resume prompt.
- usage aggregates across the initial run plus every retry (billing
  relies on this).
- reflection turn still fires when no stop hook is configured and the
  tree is clean.

Manual item remaining is the full UI round-trip of the settings form,
which is out of scope for unit tests.

* test: cover executeStopHook soft-fail and truncation invariants

Three paths the PR documents but previously had no regression gates:

- timeout (SPAWN_TIMEOUT_CODE) and activity-timeout
  (SPAWN_ACTIVITY_TIMEOUT_CODE) must return null, not a failure. a
  hook that times out is an infra problem; retrying with an agent
  turn risks an infinite loop.
- spawn errors (ENOENT from a typoed binary, etc.) take the same
  soft-fail path for the same reason.
- oversize hook output is truncated to the last 4096 chars with a
  "truncated" marker, keeping the tail (where summaries live) and
  protecting the 65535-char GitHub-comment budget downstream.

Regression targets — a refactor that accidentally surfaces an infra
failure as a gate failure, or blows the comment budget, will now
fail loudly in CI.

* test: cover soft-fail, no-resume, and short-circuit invariants

Three more documented behaviors that previously had no regression
gates:

- dirty-tree-only is a soft-fail: persistent uncommitted changes log
  and warn but DO NOT flip the run to failed. a regression that
  started surfacing this as AgentResult.error would break every run
  that leaves a test fixture untracked.
- canResume=false + stop hook failure still surfaces the hook failure
  as AgentResult.error. the retry budget is zero so "N retry
  attempts" is correctly omitted from the message, but the run still
  reports WHY it failed rather than silently reporting success.
- initial result with success=false short-circuits the loop: no gate
  checks, no reflection, no resume calls. the original agent error
  flows through verbatim for clean triage.

Also reset mockedSpawn in beforeEach so test state doesn't leak
between cases.

* test: lock in the reflection-dirties-tree → dirty-tree-gate path

The PR description claims: "if the reflection turn dirties the tree,
the loop picks that up on the next iteration via the normal
dirty-tree gate." There was no regression gate on this invariant.

Without it, a refactor that moved the reflection out of the retry
loop (e.g., into a one-shot post-loop call) would silently bypass
the commit-before-you-finish contract whenever the agent misbehaves
during reflection — uncommitted changes would ship as part of the
run's "success" state.

The test sequences three getGitStatus returns (clean → dirty → clean)
and asserts two resume calls: REFLECTION first, then UNCOMMITTED
CHANGES with the dirtying file in the prompt.

* fix: preserve pre-reflection task output when reflection succeeds

the reflection turn's reply ("done" or "updated learnings with N bullets")
is a meta-ask, not a task summary. before this fix, result = reflectionResult
clobbered the original task's output on the returned AgentResult, so
downstream consumers (handleAgentResult's fallback path when toolState is
empty, programmatic callers of main()) saw the reflection's trivial reply
instead of the real summary.

spread reflectionResult to inherit fields subsequent gate retries need
(e.g. the new sessionId claude emits per --resume invocation), but keep
the pre-reflection output verbatim.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix: fall back to reflection's output when pre-reflection output is empty

the prior fix used `??` which only fell through on null/undefined. runs
that communicate exclusively through MCP tools (e.g. report_progress) and
emit no plain text leave result.output = "", which `??` preserved as-is —
dropping the reflection's reply and leaving handleAgentResult's fallback
path with nothing to show. switch to `||` so empty-string pre-reflection
output yields the reflection's output instead of ""; non-empty task output
still wins as intended.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test: drop reflection-failure-skips-hook test (over-specified control flow)

the test pinned the literal `break` in the post-reflection failure
branch with stopScript=null, asserting only that getGitStatus was
called once. that's not a behavior contract — a reasonable refactor
(e.g. `continue` to re-check gates with explicit flake guards) would
fail this test even though the new behavior would be fine. the
"does not flip a successful run to failed" test already covers the
only thing callers depend on.

* test: drop low-value mock-driven tests from postRun

- "fires the reflection turn when no stop hook is configured" — fully
  subsumed by the output-preservation test (asserts task output
  survives, which is only possible if reflection fired).
- "uses stdout alone" / "uses stderr alone" — pin format trivia
  (`filter(Boolean).join`) that LLMs ignore.
- "returns empty output (not undefined) when both streams are empty"
  — guards a TS-impossible case; every consumer uses `output || "(no output)"`.
- "returns null on activity-timeout" — duplicate of the timeout test;
  same `return null` branch with a different constant.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
2026-05-04 19:09:42 +00:00

226 lines
7.6 KiB
TypeScript

import { execFileSync } from "node:child_process";
import type { AgentId } from "../external.ts";
import { log } from "../utils/cli.ts";
import type { ResolvedInstructions } from "../utils/instructions.ts";
import type { ResolvedPayload } from "../utils/payload.ts";
import type { TodoTracker } from "../utils/todoTracking.ts";
// maximum number of stderr lines to keep in the rolling buffer during agent execution
export const MAX_STDERR_LINES = 20;
// ── post-run retry loop ────────────────────────────────────────────────────────
/**
* how many times the post-run loop may resume the agent to fix a dirty tree
* or a failing stop hook before giving up.
*/
export const MAX_POST_RUN_RETRIES = 3;
export function getGitStatus(): string {
try {
return execFileSync("git", ["status", "--porcelain"], {
encoding: "utf-8",
timeout: 10_000,
}).trim();
} catch {
return "";
}
}
export function buildCommitPrompt(status: string): string {
return [
`UNCOMMITTED CHANGES — the working tree is dirty. push all changes to a pull request (new or existing). \`git status\` must be clean before you finish.`,
"",
"```",
status,
"```",
].join("\n");
}
export interface StopHookFailure {
exitCode: number;
output: string;
}
export interface PostRunIssues {
stopHook?: StopHookFailure;
dirtyTree?: string;
}
export function hasPostRunIssues(issues: PostRunIssues): boolean {
return issues.stopHook !== undefined || issues.dirtyTree !== undefined;
}
/**
* token/cost usage data from a single agent run.
*
* NOTE on semantics: `inputTokens` here is the *total* billable input for the
* run — non-cached input + cache read + cache write — matching the per-agent
* SDK conventions. This is what gets persisted to `WorkflowRun.inputTokens`.
*
* The stdout token table and markdown step summary display a different "Input"
* column that shows only the non-cached portion (derivable as
* `inputTokens - cacheReadTokens - cacheWriteTokens`) so humans can see the
* cache hit ratio at a glance. Dashboards that query `WorkflowRun.inputTokens`
* directly are seeing the full total, not the log column.
*/
export interface AgentUsage {
agent: string;
/** full billable input: non-cached + cache read + cache write */
inputTokens: number;
outputTokens: number;
cacheReadTokens?: number | undefined;
cacheWriteTokens?: number | undefined;
costUsd?: number | undefined;
}
export interface AgentToolUseEvent {
toolName: string;
input: unknown;
}
/**
* Result returned by agent execution
*/
export interface AgentResult {
success: boolean;
output?: string | undefined;
error?: string | undefined;
metadata?: Record<string, unknown>;
usage?: AgentUsage | undefined;
}
/**
* Minimal context passed to agent.run()
*/
export interface AgentRunContext {
payload: ResolvedPayload;
resolvedModel?: string | undefined;
mcpServerUrl: string;
tmpdir: string;
instructions: ResolvedInstructions;
todoTracker?: TodoTracker | undefined;
/**
* user-configured stop hook script. runs after the agent finishes each
* attempt; non-zero exit resumes the agent with the hook output as
* guidance. null when the repo has no stop hook configured.
*/
stopScript?: string | null | undefined;
/**
* called synchronously when the agent subprocess is killed for inner
* activity timeout. lets main.ts tear down shared resources (MCP HTTP
* server) so lingering SSE reconnects don't keep the outer timer alive.
*/
onActivityTimeout?: (() => void) | undefined;
onToolUse?: ((event: AgentToolUseEvent) => void) | undefined;
}
export interface Agent {
name: AgentId;
install: (token?: string) => Promise<string>;
run: (ctx: AgentRunContext) => Promise<AgentResult>;
}
export const agent = (input: Agent): Agent => {
return {
...input,
run: async (ctx: AgentRunContext): Promise<AgentResult> => {
log.debug(`» payload: ${JSON.stringify(ctx.payload, null, 2)}`);
return input.run(ctx);
},
};
};
/** format a USD cost to 4 decimal places, always showing the leading zero */
export function formatCostUsd(costUsd: number): string {
return costUsd.toFixed(4);
}
/**
* merge two AgentUsage snapshots into one running total.
*
* both agent harnesses invoke their runner multiple times per `run()` when the
* post-run retry loop kicks in (MAX_POST_RUN_RETRIES). each invocation
* produces its own AgentUsage; we sum them so downstream callers (usage
* summary, WorkflowRun persistence) see the whole session — not just the
* final retry's slice.
*
* returns `undefined` when both sides are empty so callers can short-circuit
* without a special case. zero-valued cache / cost fields are dropped to
* `undefined` for symmetry with each harness's `buildUsage`.
*/
export function mergeAgentUsage(
a: AgentUsage | undefined,
b: AgentUsage | undefined
): AgentUsage | undefined {
// always return a fresh object — callers treat AgentUsage as immutable, and
// returning `a` / `b` directly would leak that invariant to future callers
if (!a && !b) return undefined;
if (!a) return { ...(b as AgentUsage) };
if (!b) return { ...a };
const cacheRead = (a.cacheReadTokens ?? 0) + (b.cacheReadTokens ?? 0);
const cacheWrite = (a.cacheWriteTokens ?? 0) + (b.cacheWriteTokens ?? 0);
const cost = (a.costUsd ?? 0) + (b.costUsd ?? 0);
return {
agent: a.agent,
inputTokens: a.inputTokens + b.inputTokens,
outputTokens: a.outputTokens + b.outputTokens,
cacheReadTokens: cacheRead > 0 ? cacheRead : undefined,
cacheWriteTokens: cacheWrite > 0 ? cacheWrite : undefined,
costUsd: cost > 0 ? cost : undefined,
};
}
/**
* unified per-run token table used by every agent harness.
*
* columns are kept stable across agents and models so downstream log parsers
* (scripts/token-usage.ts, cost dashboards) only have to understand one format:
*
* Input non-cached input tokens sent this run
* Cache Read input tokens served from prompt cache (Anthropic, etc.)
* Cache Write input tokens written to prompt cache this run
* Output assistant output tokens
* Total sum of the four columns — the real billable quantity
* Cost ($) USD cost reported by the provider (only rendered when known)
*
* models that don't report prompt caching leave Cache Read / Write at 0.
* OpenCode emits per-step `part.cost` sourced from models.dev (works across
* Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, OpenRouter, etc.);
* Claude CLI emits `total_cost_usd` on its final `result` event. pass the
* accumulated value via `costUsd` to render the Cost column.
*/
export function logTokenTable(t: {
input: number;
cacheRead: number;
cacheWrite: number;
output: number;
costUsd?: number | undefined;
}): void {
const total = t.input + t.cacheRead + t.cacheWrite + t.output;
// narrow costUsd to a concrete number so the render path doesn't need a cast
const costUsd = typeof t.costUsd === "number" && t.costUsd > 0 ? t.costUsd : undefined;
const headerRow: Array<{ data: string; header: true }> = [
{ data: "Input", header: true },
{ data: "Cache Read", header: true },
{ data: "Cache Write", header: true },
{ data: "Output", header: true },
{ data: "Total", header: true },
];
const dataRow: string[] = [
String(t.input),
String(t.cacheRead),
String(t.cacheWrite),
String(t.output),
String(total),
];
if (costUsd !== undefined) {
headerRow.push({ data: "Cost ($)", header: true });
dataRow.push(formatCostUsd(costUsd));
}
log.table([headerRow, dataRow]);
}