add diff coverage preflight for PR review submissions (#544)

* add one-time diff coverage preflight for PR reviews

track diff read coverage from agent tool-use events and run a one-time pre-flight before review submission, with explicit coverage skip reasons for low-value files like lockfiles.

Made-with: Cursor

* add manual dispatch fallback for preview deploy workflow

allow preview repo and preview sync jobs to be run via workflow_dispatch with explicit PR number and branch inputs, so preview provisioning can be retriggered when pull_request events fail to fire.

Made-with: Cursor

* fix manual preview dispatch PR input wiring

use normalized PR number and branch env values for comment creation and script env wiring so workflow_dispatch preview runs can create and update PR-specific preview resources.

Made-with: Cursor

* remove obsolete snapshots invalidated by checkout instructions change

* fix diff coverage read offset handling and add local sanity-check guidance

normalize read offset semantics for diff coverage tracking, reuse shared range counting in review preflight, add focused diff coverage unit tests, and document the local play.ts testing workflow in AGENTS.md.

Made-with: Cursor

* add regenerated mcp test snapshots

capture snapshot files generated by the review comment and checkout formatting tests during pre-push validation so the branch remains clean and reproducible.

Made-with: Cursor

* add diff coverage preflight instrumentation logs

log diff coverage initialization in checkout_pr and emit preflight state/breakdown diagnostics in create_pull_request_review to debug missing coverage enforcement in preview e2e runs.

Made-with: Cursor

* add env override to force local cli execution in action runtime

support explicit local-cli execution via PULLFROG_FORCE_LOCAL_CLI so preview workflows can run branch action code instead of the npm fallback package during e2e debugging.

Made-with: Cursor

* add preview e2e debugging learnings for action runtime validation

capture the preview execution-path gotchas and one-time preflight verification pattern in AGENTS.md so future investigations validate the real runtime and avoid npm fallback confusion.

Made-with: Cursor

* reduce diff coverage log noise while preserving failure visibility

downgrade verbose diff coverage lifecycle diagnostics to debug, keep a concise info-level pre-flight failure signal, and document preview runtime debugging learnings in AGENTS.md.

Made-with: Cursor

* WIP

* tune sync.md: ff override + softer overlap verification

Made-with: Cursor

* chore: bump models snapshot for claude-opus-4-7

Made-with: Cursor

* rip out coverage_skips waiver from diff coverage pre-flight

Made-with: Cursor

---------

Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>
This commit is contained in:
Colin McDonnell
2026-04-16 21:51:44 +00:00
committed by pullfrog[bot]
parent 5e6ff67623
commit 56a5d29598
11 changed files with 653 additions and 5 deletions
+4 -2
View File
@@ -130,7 +130,7 @@ ${learningsStep(t, 6)}`,
"Review code, PRs, or implementations; provide feedback or suggestions; identify issues; or check code quality, style, and correctness",
prompt: `### Checklist
1. Checkout the PR via \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. Read the diff to identify the major areas of change.
1. Checkout the PR via \`${t("checkout_pr")}\` — this returns PR metadata and a \`diffPath\`. read the diff TOC first and treat its file line ranges as your coverage checklist.
2. For each area of change:
- read the diff and trace data flow, check boundaries, and verify assumptions
@@ -147,6 +147,7 @@ ${learningsStep(t, 6)}`,
4. Submit — ALWAYS submit exactly one review via \`${t("create_pull_request_review")}\`.
Do NOT call \`report_progress\` — the review is the final record and the progress
comment will be cleaned up automatically.
note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session.
- **critical issues** (blocks merge — bugs, security, data loss):
\`approved: false\`. Body begins with a GitHub alert blockquote, e.g.:
@@ -165,7 +166,7 @@ ${learningsStep(t, 6)}`,
"Re-review a PR after new commits are pushed; focus on new changes since the last review",
prompt: `### Checklist
1. Checkout the PR via \`${t("checkout_pr")}\` — this returns PR metadata, \`diffPath\` (full diff), and \`incrementalDiffPath\` (changes since last reviewed version, if available).
1. Checkout the PR via \`${t("checkout_pr")}\` — this returns PR metadata, \`diffPath\` (full diff), and \`incrementalDiffPath\` (changes since last reviewed version, if available). read the diff TOC first and use its line ranges as your coverage checklist.
2. If \`incrementalDiffPath\` is present, read it to see what changed since the last review. This is a range-diff that isolates the net changes, filtering out base branch noise. If not present, fall back to reviewing the full PR diff.
@@ -189,6 +190,7 @@ ${learningsStep(t, 6)}`,
- in some cases you may receive a complete diff for the whole pull request instead of an incremental one. when this happens, you will need to determine what changes have happened since Pullfrog's most recent review.
7. Submit — Do NOT call \`report_progress\` or \`create_issue_comment\` — the review is the final record and the progress comment will be cleaned up automatically. the review body always includes the reviewed changes from step 6a. append \`Prior review feedback:\\n\` with the checklist from step 6b only if any prior comments were addressed. Follow these rules:
- note: the first create_pull_request_review submission may error with a one-time diff-coverage nudge listing unread TOC regions. retry the same call to proceed — optionally after reading the listed ranges. the pre-flight will not block again this session.
- IF NO NEW ISSUES, NON-SUBSTANTIVE CHANGES ONLY (trivial formatting, import reordering, comment tweaks): do NOT submit a review. Do NOT call \`report_progress\`. Exit — the progress comment will be cleaned up automatically.
- ELSE IF NEW CRITICAL ISSUES (blocks merge): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with a GitHub alert blockquote (e.g. \`> [!CAUTION]\\n> This PR introduces ...\`), then the reviewed changes summary and prior feedback (if any).
- ELSE IF NEW RECOMMENDED CHANGES (non-critical): call \`${t("create_pull_request_review")}\` with \`approved: false\`, all comments, and the review body. body opens with \`> [!IMPORTANT]\\n> ...\` alert, then the reviewed changes summary and prior feedback (if any).