5518890b18
* audit learnings: reshape reflection prompt + bake tool quirks into descriptions (#619) Cross-repo audit of the 48 repos with non-null learnings turned up two recurring failure modes: 1. ~25-30% of bullets across the most-active repos are pullfrog-tool quirks ("shell timeout is in milliseconds", "git args must be a JSON array", "create_pull_request_review drops out-of-hunk comments", "push_branch may report timeout when push succeeded", "checkout_pr shallow.lock retries", "commit_id needs full 40-char SHA"). These are universal across repos and should live in tool descriptions, not be rediscovered and stored 48 times. Tool descriptions now surface them. 2. Bullets are routinely 200-1000 chars (paragraph-length), and 12 of 48 repos are at the 10k cap. The reflection prompt now: caps bullets at ~240 chars (one specific fact), bans PR/review/commit/date-anchored facts that decay within weeks, bans tool-quirk learnings, and tells the agent that cap pressure means compress+prune existing bullets, not skip new findings. Co-authored-by: Cursor <cursoragent@cursor.com> * learnings: add server-generated TOC, fixed section taxonomy, raise cap to 100k (#707) Cap goes 10k → 100k. Reads stay bounded because the seeded file now opens with a server-generated table of contents listing every `## ` section's line range — agents read the TOC, then `read_file offset/limit` just the sections relevant to the current task instead of slurping the whole file. ## Section taxonomy (fixed) `## Build & test`, `## CI`, `## Conventions`, `## Architecture`, `## Gotchas`. Free-form `### ` sub-headings inside a section are fine. Pre-taxonomy free-text rows get wrapped in a `## Legacy` carve-out on first seed so they remain visible while the agent gradually re-curates them during reflection turns. ## Storage shape unchanged `Repo.learnings` still holds raw markdown (no schema migration). The TOC is a pure read-side affordance: prepended at seed time, stripped from the agent-edited file before persist. Markers `<!-- pullfrog-learnings-toc:* -->` delimit the strip region. Agent edits inside the markers are discarded. ## Round-trip semantics `seedLearningsFile` now returns `{ path, canonicalSeed }` where `canonicalSeed` is the post-TOC body — same shape `readLearningsFile` returns at end-of-run, so `persistLearnings` byte-compares them directly to skip the no-op PATCH. Empty-repo first runs end up with the section scaffold both as seed and as read-back, so untouched runs still short-circuit cleanly. ## Reflection prompt Adds explicit section-placement guidance (place each new bullet under the most relevant `## `; do NOT add new top-level headings; do NOT edit anything between the TOC markers). Carries forward the bullet hygiene from the previous commit: ≤240 chars per bullet, no pullfrog-tool quirks (those belong in tool descriptions), no PR/review/commit/date references. The "near cap" framing is replaced with "compress and prune within a section when it grows noisy" since the cap pressure that drove cramming is gone. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal round 1: line-anchored taxonomy detect, partial-merge, line-boundary truncation, scaffold-empty UI Multi-lens review of the TOC + taxonomy diff surfaced a cluster of correctness and operational bugs. Fixes: - `hasAnyTaxonomyHeading` used `String.includes("## X")` which false-positives on `### X` (the `## ` substring sits inside `### `), prose containing `## CI`, fenced code documenting markdown, etc. Replaced with a line-anchored predicate that reuses `parseHeadings` so detection and TOC construction stay consistent. - The "any heading present → pass through verbatim" rule meant a body with one taxonomy heading would seed without the other four. Worse, requiring all five would flip a body back into Legacy when the agent legitimately pruned a section to empty. New `partial` kind: keep existing content in place, append missing sections in canonical order so the agent always has the full scaffold without losing pruning intent. - `stripLearningsToc` collapsed `\n{3,}` globally; `canonicalSeed` doesn't, so an untouched body with intentional triple-newline spacing would compare unequal and burn a spurious LearningsRevision row each run. Drop the global collapse — only the leading newlines that the strip itself introduces are normalized. - 100k truncation via `slice(0, 100_000)` could cut mid-line, breaking `parseHeadings` (whole-line `^## `) on the next seed and flipping a cut body back into Legacy. New `truncateAtLineBoundary` cuts at the last newline before the cap. - `LearningsSection.tsx` rendered a scaffold-only body as "has learnings" instead of the empty placeholder. Added a `hasOnlyEmptyScaffold` guard so the console behaves the same as pre-PR for the empty case. - Seed log line distinguishes `kind=structured/partial/legacy-wrapped/ empty` instead of `existing=yes/no`, so operators can spot legacy migration activity in logs. - New tests cover: substring false-positive (`### Build & test`, in-prose mentions), partial-taxonomy merge (no Legacy wrap), full-taxonomy structured pass-through, last-newline truncation, triple-newline preservation. Deferred (documented in PR body): deploy-ordering footgun (action before API), rollback for rows >10k, Gemini sanitizer dropping `description` on `anyOf` branches, reflection-on-failed-runs. Co-authored-by: Cursor <cursoragent@cursor.com> * anneal r2: hard-truncate fallback when line boundary discards >4k Round-2 review caught a regression in `truncateAtLineBoundary`: when the only newline within the first 100k chars sits near the start (e.g. one heading + 100k+ char single line — pathological pasted log dumps), the line-boundary cut discards almost all of the body. losing one partial line is preferable to losing kilobytes; threshold the fallback at 4k. Co-authored-by: Cursor <cursoragent@cursor.com> * move TOC out of file: prompt-side rendering, server-parsed headings drops the in-file TOC + fixed taxonomy in favor of: - file on disk = verbatim Repo.learnings (no markers, no scaffold) - server parses headings (mdast-util-from-markdown) at run-context time and returns them as RepoSettings.learningsHeadings - action renders heading TOC into the LEARNINGS prompt section as parenthesized line ranges like `Build & test (L1-L42)` with hierarchy via 2-space indent off the shallowest depth - reflection prompt teaches agent-curated structure with a soft 300-line per-section cap and explicit guidance to restructure flat legacy lists cuts 8 helpers (ensureSections, stripLearningsToc, assembleFile, buildTocBlock, parseHeadings, buildSectionScaffold, hasAnyTaxonomyHeading, LEARNINGS_SECTIONS) and the canonicalSeed round-trip dance. action seedLearningsFile is now { path } only; main.ts byte-compares the trimmed read-back against (current ?? "").trim() to gate the persist PATCH. truncateAtLineBoundary kept for safety. new tests: - test/learningsToc.test.ts (11 parser cases incl. fenced-code, blockquote, arbitrary h1-h6 nesting, startLine-points-at-heading invariant) - action/utils/learningsTocRender.test.ts (7 renderer cases) --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
100 lines
3.3 KiB
TypeScript
100 lines
3.3 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { buildLearningsSection, renderLearningsToc } from "./instructions.ts";
|
|
import type { LearningsHeading } from "./runContext.ts";
|
|
|
|
const h = (depth: 1 | 2 | 3 | 4 | 5 | 6, title: string, startLine: number, endLine: number) => ({
|
|
depth,
|
|
title,
|
|
startLine,
|
|
endLine,
|
|
});
|
|
|
|
describe("renderLearningsToc", () => {
|
|
it("renders flat h2 list with parenthesized ranges, no hashes or backticks", () => {
|
|
const headings: LearningsHeading[] = [
|
|
h(2, "Build & test", 1, 18),
|
|
h(2, "Architecture", 19, 60),
|
|
];
|
|
expect(renderLearningsToc(headings)).toBe(
|
|
`- Build & test (L1-L18)
|
|
- Architecture (L19-L60)`
|
|
);
|
|
});
|
|
|
|
it("indents deeper headings 2 spaces per depth level past the shallowest", () => {
|
|
const headings: LearningsHeading[] = [
|
|
h(2, "Build & test", 1, 42),
|
|
h(3, "Local", 3, 18),
|
|
h(3, "CI", 19, 42),
|
|
h(2, "Architecture", 43, 210),
|
|
h(3, "Background workers", 80, 210),
|
|
];
|
|
expect(renderLearningsToc(headings)).toBe(
|
|
`- Build & test (L1-L42)
|
|
- Local (L3-L18)
|
|
- CI (L19-L42)
|
|
- Architecture (L43-L210)
|
|
- Background workers (L80-L210)`
|
|
);
|
|
});
|
|
|
|
it("treats the shallowest depth as the root column when no h2 is present", () => {
|
|
const headings: LearningsHeading[] = [h(3, "Only h3", 1, 5), h(4, "Sub h4", 2, 5)];
|
|
expect(renderLearningsToc(headings)).toBe(
|
|
`- Only h3 (L1-L5)
|
|
- Sub h4 (L2-L5)`
|
|
);
|
|
});
|
|
|
|
it("supports depths up through h6 with stable 2-space indent steps", () => {
|
|
const headings: LearningsHeading[] = [
|
|
h(2, "Two", 1, 10),
|
|
h(3, "Three", 2, 10),
|
|
h(4, "Four", 3, 10),
|
|
h(5, "Five", 4, 10),
|
|
h(6, "Six", 5, 10),
|
|
];
|
|
expect(renderLearningsToc(headings)).toBe(
|
|
`- Two (L1-L10)
|
|
- Three (L2-L10)
|
|
- Four (L3-L10)
|
|
- Five (L4-L10)
|
|
- Six (L5-L10)`
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("buildLearningsSection", () => {
|
|
it("returns empty string when no file path (seed step failed)", () => {
|
|
expect(buildLearningsSection({ filePath: null, headings: [] })).toBe("");
|
|
});
|
|
|
|
it("renders the no-headings affordance when the body has no structure", () => {
|
|
const out = buildLearningsSection({
|
|
filePath: "/tmp/run-1/pullfrog-learnings.md",
|
|
headings: [],
|
|
});
|
|
expect(out).toContain("************* LEARNINGS *************");
|
|
expect(out).toContain("/tmp/run-1/pullfrog-learnings.md");
|
|
expect(out).toContain("no headings yet");
|
|
expect(out).toContain("structure it with");
|
|
// does not include a TOC list when there are no headings
|
|
expect(out).not.toMatch(/\(L\d+-L\d+\)/);
|
|
});
|
|
|
|
it("renders the TOC inline with the file path and heading guidance", () => {
|
|
const out = buildLearningsSection({
|
|
filePath: "/tmp/run-1/pullfrog-learnings.md",
|
|
headings: [h(2, "Build & test", 1, 18), h(2, "Architecture", 19, 60)],
|
|
});
|
|
expect(out).toContain("************* LEARNINGS *************");
|
|
expect(out).toContain("/tmp/run-1/pullfrog-learnings.md");
|
|
expect(out).toContain("- Build & test (L1-L18)");
|
|
expect(out).toContain("- Architecture (L19-L60)");
|
|
expect(out).toContain("Each range starts at the section heading line");
|
|
// explicit "no hashes, no backticks" in the rendered list
|
|
expect(out).not.toContain("- `## Build");
|
|
expect(out).not.toContain("`## Build");
|
|
});
|
|
});
|