learnings: edit-in-place tmpfile (drop update_learnings tool) (#635)

* learnings: edit-in-place tmpfile (drop update_learnings tool)

learnings now follow the PR-summary file pattern: server seeds
`pullfrog-learnings.md` from `Repo.learnings` at startup, agent reads
it as part of context, may edit in place during the post-run reflection
turn, server reads back at end-of-run and PATCHes if changed.

motivation: `update_learnings` required the agent to pass the FULL
merged list as a string parameter — an output-token tax that grew
linearly with the learnings size, and a constant prompt-context
expansion since the contents were also inlined into the LEARNINGS
section. for repos with mature learnings the prompt was getting
visibly noisy in CI logs.

key changes:
- new `action/utils/learnings.ts` (seed/read helpers + 10k cap)
- `main.ts`: always seed; `persistLearnings` mirrors `persistSummary`
  (success path, error path, exit-signal handler, idempotent guard,
  byte-trim equality skip); forwards `model` for `LearningsRevision.model`
- `LEARNINGS` prompt section now contains only the file path + a
  one-line "read it" instruction (no contents inlined)
- `update_learnings` MCP tool deleted; `action/mcp/learnings.ts` removed
- reflection turn (`buildLearningsReflectionPrompt`) reframed around
  file editing with explicit prune-stale + leave-alone-if-nothing-new
  framing
- `learningsStep` removed from every mode checklist — surface lives only
  in the LEARNINGS prompt section + the reflection turn now

* learnings: harden seed step + refresh stale docs (review feedback)

Three findings from PR review, all implemented:

1. wrap learnings seed in best-effort try/catch (action/main.ts) —
   the always-on seed block ran unconditionally and an unwrapped
   `seedLearningsFile` (mkdir + writeFile) failure (ENOSPC, EACCES,
   hostile sandbox) would unwind into the outer main() catch and flip
   an otherwise-successful run to " Pullfrog failed" before the
   agent even started. asymmetric with `persistLearnings`'s own
   best-effort contract. wrap and log on failure; downstream
   consumers (`persistLearnings`, agent harnesses, `resolveInstructions`)
   already handle `learningsFilePath: undefined` cleanly.

2. refresh wiki/main.md — `resolveInstructions` parameter renamed
   from `learnings` to `learningsFilePath` in this PR; the data-flow
   diagram and the resolver dependency table both still showed the
   pre-refactor signature.

3. drop deleted `learnings.ts` from ROADMAP.md + RESEARCH.md
   "missing MCP tool tests" bullets — `action/mcp/learnings.ts` was
   removed in this PR; the bullets are otherwise still accurate.
This commit is contained in:
Colin McDonnell
2026-05-08 22:45:26 +00:00
committed by pullfrog[bot]
parent 2e6c01670e
commit d6de1c369a
12 changed files with 320 additions and 97 deletions
+3 -1
View File
@@ -727,7 +727,9 @@ export const claude = agent({
stopScript: ctx.stopScript,
summaryFilePath: ctx.summaryFilePath,
summarySeed: ctx.summarySeed,
reflectionPrompt: buildLearningsReflectionPrompt("claude"),
reflectionPrompt: ctx.learningsFilePath
? buildLearningsReflectionPrompt(ctx.learningsFilePath)
: undefined,
canResume: (r) => Boolean(r.sessionId),
resume: async (c) => {
const sessionId = c.previousResult.sessionId;
+3 -1
View File
@@ -1005,7 +1005,9 @@ export const opencode = agent({
stopScript: ctx.stopScript,
summaryFilePath: ctx.summaryFilePath,
summarySeed: ctx.summarySeed,
reflectionPrompt: buildLearningsReflectionPrompt("opencode"),
reflectionPrompt: ctx.learningsFilePath
? buildLearningsReflectionPrompt(ctx.learningsFilePath)
: undefined,
resume: async (c) =>
runOpenCode({
...runParams,
+8 -8
View File
@@ -42,10 +42,10 @@ describe("runPostRunRetryLoop — reflection turn", () => {
});
it("does not flip a successful run to failed when reflection returns success:false", async () => {
// the reflection turn is a best-effort nudge (update_learnings). if it
// fails — e.g. the model API errors mid-turn — the underlying task has
// already completed and been gated cleanly, so the run as a whole must
// still be reported as successful.
// the reflection turn is a best-effort nudge (edit the learnings
// tmpfile). if it fails — e.g. the model API errors mid-turn — the
// underlying task has already completed and been gated cleanly, so the
// run as a whole must still be reported as successful.
const initial = successResult({ output: "task done" });
const resume = vi
.fn<(ctx: { prompt: string; previousResult: AgentResult }) => Promise<AgentResult>>()
@@ -56,7 +56,7 @@ describe("runPostRunRetryLoop — reflection turn", () => {
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: call update_learnings if anything is worth saving",
reflectionPrompt: "REFLECTION: edit learnings file if anything is worth saving",
});
expect(result.success).toBe(true);
@@ -110,7 +110,7 @@ describe("runPostRunRetryLoop — reflection turn", () => {
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: consider update_learnings",
reflectionPrompt: "REFLECTION: consider editing learnings",
});
expect(result.success).toBe(true);
@@ -135,7 +135,7 @@ describe("runPostRunRetryLoop — reflection turn", () => {
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: consider update_learnings",
reflectionPrompt: "REFLECTION: consider editing learnings",
});
expect(result.success).toBe(true);
@@ -186,7 +186,7 @@ describe("runPostRunRetryLoop — reflection turn", () => {
initialUsage: undefined,
stopScript: null,
resume,
reflectionPrompt: "REFLECTION: consider update_learnings",
reflectionPrompt: "REFLECTION: consider editing learnings",
});
expect(result.success).toBe(true);
+19 -16
View File
@@ -1,5 +1,4 @@
import { readFile } from "node:fs/promises";
import { type AgentId, formatMcpToolRef } from "../external.ts";
import { LIFECYCLE_HOOK_TIMEOUT_MS } from "../lifecycle.ts";
import { log } from "../utils/cli.ts";
import {
@@ -157,26 +156,30 @@ export function buildPostRunPrompt(issues: PostRunIssues): string {
}
/**
* prompt for a dedicated post-run reflection turn nudging the agent to call
* `update_learnings` if it discovered anything worth persisting.
* prompt for a dedicated post-run reflection turn nudging the agent to edit
* the rolling learnings file if it discovered anything worth persisting.
*
* this exists because the learnings step baked into mode checklists is
* frequently ignored — the agent stays focused on the task and the meta-ask
* falls through. delivering it as its own resume turn, with nothing competing
* for attention, raises the fire rate substantially.
* this exists because passive "if you learned something, write it down"
* instructions baked into mode checklists are frequently ignored — the agent
* stays focused on the task and the meta-ask falls through. delivering it
* as its own resume turn, with nothing competing for attention, raises the
* fire rate substantially.
*
* the file is the single source of truth — there is no separate MCP tool
* call. the server reads the file at end-of-run and persists any edits to
* `Repo.learnings`.
*/
export function buildLearningsReflectionPrompt(agentId: AgentId): string {
const t = (name: string) => formatMcpToolRef(agentId, name);
export function buildLearningsReflectionPrompt(filePath: string): string {
return [
`REFLECTION — before you finish, think back over this task: did you discover anything about this repo's setup, test commands, conventions, or patterns that you are confident is correct and would reliably help future runs?`,
`REFLECTION — before you finish, think back over this task: did you discover anything about this repo's setup, test commands, conventions, or patterns that is high-confidence and would reliably help future runs?`,
"",
`if so, call \`${t("update_learnings")}\` to persist it.`,
`the rolling learnings file is at \`${filePath}\`. read it first if you haven't already, then edit it in place using your native file tools. the server reads this file at end-of-run and persists any changes — there is no tool to call.`,
"",
`rules:`,
`- only call \`${t("update_learnings")}\` when the finding is high-confidence and broadly useful. skip if unsure, speculative, or one-off.`,
`- pass the FULL merged list: existing learnings from the original prompt + your new discoveries. one fact per bullet, lines starting with \`- \`.`,
`- deduplicate, and drop bullets that are clearly wrong or no longer relevant to the current codebase.`,
`- if you already called \`${t("update_learnings")}\` earlier in this run, or nothing new is worth capturing, just reply "done" and stop — do not edit the repo for this reflection.`,
`keep the file healthy:`,
`- only add bullets when the finding is high-confidence AND broadly useful. skip speculative, one-off, or "maybe" findings.`,
`- prune bullets that are clearly wrong, no longer relevant, or low-signal (rarely useful). a focused, accurate file beats a long stale one.`,
`- format: flat bullet list, one fact per line starting with \`- \`. deduplicate against existing entries — if a bullet covers the same fact, update it in place instead of adding a duplicate.`,
`- leave the file alone if you have nothing substantively new to add and the existing entries still look healthy. silence is a valid outcome — just reply "done" and stop.`,
].join("\n");
}
+7
View File
@@ -133,6 +133,13 @@ export interface AgentRunContext {
* track of multi-step instructions.
*/
summarySeed?: string | undefined;
/**
* absolute path to the rolling repo-level learnings tmpfile. seeded for
* every run from `Repo.learnings`. used by the post-run reflection turn
* so the prompt can point the agent at a concrete path to edit; the
* file's content is read back and persisted by main.ts after the run.
*/
learningsFilePath?: string | undefined;
/**
* called synchronously when the agent subprocess is killed for inner
* activity timeout. lets main.ts tear down shared resources (MCP HTTP