mcp: embed example calls in top-level tool descriptions (#723)

* mcp: embed example calls in top-level tool descriptions

agents (esp. claude sonnet) hallucinate param names from training-data
priors — `pr_number` instead of `pull_number`, `summary` instead of
`body`, full subcommand strings jammed into `git({command})` like it
were `shell({command})`. each error burns a tool round-trip plus a
follow-up ToolSearch, ~40+ events / 24h, no observable recovery cost
to us but visible to users in agent logs.

cheapest fix: add a sample formatted function call to every affected
tool's top-level description. example anchors are more reliable than
schema descriptions alone because the model treats descriptions as
narrative but call examples as canonical structure. for `git` and
`shell` (whose `command` fields collide), include explicit
counter-examples disambiguating which tool owns which shape.

no schema aliases / coercion yet — try the cheap thing first; if the
next audit window still shows the same hallucination rate, layer
aliases on top per #585's recommendation.

closes #585, closes #701

* mcp: drop negative anchors from tool descriptions

negation is a footgun in tool descriptions — telling the model "NOT
pr_number" makes pr_number more salient, not less. let the positive
example carry the schema and trust the model to read it.

removes:
- "the parameter is pull_number (a number), NOT pr_number" and
  similar across checkout_pr, get_pull_request, list_pull_request_reviews,
  get_review_comments, create_pull_request_review
- "NOT summary, message, or content" on report_progress
- "WRONG: git({ command: 'log --oneline' })" counter-example on git
- redundant param-type restatements after the example (e.g. "depth is a
  number, not a string" on git_fetch, "description is required" on shell)

keeps a single positive example per tool. for tools with multiple call
shapes (git, git_fetch, push_branch), two positive examples instead of
one + a counter-example.
This commit is contained in:
Colin McDonnell
2026-05-13 22:45:08 +00:00
committed by pullfrog[bot]
parent 868576a474
commit b8ac42e875
11 changed files with 41 additions and 24 deletions
+4 -5
View File
@@ -320,16 +320,14 @@ export const CreatePullRequestReview = type({
)
.optional(),
commit_id: type.string
.describe(
"Optional SHA of the commit being reviewed. Defaults to latest. Must be the FULL 40-character SHA — abbreviated SHAs are rejected by GitHub with `422 Unprocessable Entity`. The PR-synchronize event payload's `head_sha` is already full-length."
)
.describe("Optional SHA of the commit being reviewed. Defaults to latest.")
.optional(),
comments: type({
path: type.string.describe(
"The file path to comment on (relative to repo root). Must be a file that appears in the PR diff."
),
line: type.number.describe(
"Line number to comment on. For multi-line ranges, this is the end line. Use NEW column from diff format. MUST sit inside a `@@` hunk in the PR diff — anchors on context-only or untouched lines are dropped silently (the rest of the review still posts; dropped entries are reported under `droppedComments` in the response)."
"Line number to comment on. For multi-line ranges, this is the end line. Use NEW column from diff format."
),
side: type
.enumerated("LEFT", "RIGHT")
@@ -347,7 +345,7 @@ export const CreatePullRequestReview = type({
.optional(),
start_line: type.number
.describe(
"Start line for multi-line comment ranges. Omit for single-line comments. The range [start_line, line] defines which lines a suggestion replaces. BOTH `start_line` AND `line` must sit inside the SAME `@@` hunk — a `start_line` outside the hunk causes the whole comment to be dropped even when `line` is valid. If you need to comment on context just above/below a hunk, shrink the range to a single line that is provably modified."
"Start line for multi-line comment ranges. Omit for single-line comments. The range [start_line, line] defines which lines a suggestion replaces."
)
.optional(),
})
@@ -363,6 +361,7 @@ export function CreatePullRequestReviewTool(ctx: ToolContext) {
name: "create_pull_request_review",
description:
"Submit a review for an existing pull request. " +
'Example: `create_pull_request_review({ pull_number: 1234, body: "LGTM", approved: true, comments: [{ path: "src/api.ts", line: 42, body: "nit: rename" }] })`. ' +
"Each call creates a permanent, visible review on the PR — NEVER submit test or diagnostic reviews. " +
"Reviews with no body AND no comments are silently skipped (nothing to post). " +
"IMPORTANT: 95%+ of feedback should be in 'comments' array with file paths and line numbers. " +