5f3e46c42d
* fix: don't reuse disabled proxy key on workflow re-runs; non-fatal title-gen errors Three small surgical fixes addressing run https://github.com/pullfrog/app/actions/runs/25580969379: 1. **`/api/proxy-token` idempotency now checks `finalizedAt`.** GitHub re-runs share the same `run_id` (only `run_attempt` increments), so attempt N+1's action calls /api/proxy-token and inherits attempt N's `proxyKeyId`. The `workflow_run.completed` webhook between attempts retires that key on OpenRouter (`disableKey`), so attempt N+1 was getting back a disabled key and OpenRouter responded with `401 User not found` on every call. Falling through when finalized routes through the same billing gate (`handleRouterBilling` balance check), so no new attack surface. 2. **OpenCode title-gen / small-model errors no longer fatal.** OpenCode auto-spawns a small `agent=title small=true` background call at session start to name the thread, defaulting to `anthropic/claude-haiku-4.5` (anomalyco/opencode#1243). Pre-fix, the wrapper's `error` event handler treated any `type=error` as fatal, so a cosmetic title failure killed the run before primary inference even started. Now: stderr matching `small=true` sets a one-shot suppression flag for the next stdout `error` event, which is logged as a warning instead. 3. **Provider-error classifier puts auth patterns above rate-limit.** OpenRouter 401 payloads bundle `x-ratelimit-*` response headers, and the loose `\brate[_ ]limit/i` pattern was winning. Added 401/403 status, `User not found`, `Invalid authentication`, `No auth credentials found` patterns ahead of rate-limit. Updated the existing 401-headers regression test to assert correct auth classification rather than `null`. * opencode: correlate small-model error suppression by message, not by next-event Pullfrog self-review on #636 flagged a real concurrency hole. OpenCode forks the title-gen call (`session/prompt.ts:1452-1457` via `Effect.forkIn(scope)`) so it races primary inference. The previous one-shot `suppressNextErrorEvent` boolean had no per-call correlation: it was consumed by whichever stdout `type=error` event landed next, regardless of which subagent produced it. Under concurrent failures, a primary-agent error landing first could be silently downgraded to a warning while the small-model error then propagated fatally — the inverse of the bug the suppression was meant to prevent. Replaced the boolean with a `Set<string>` of pending small-model error messages. stderr extracts the inner `"message":"..."` from any classified provider error tagged `small=true`; the stdout `error` handler suppresses only when `event.error.data.message` matches a pending entry. Set is capped at 32 entries so a long stream of small-model failures can't wedge memory. Also corrected the comment that referenced "session summarizer" — verified in opencode source that summarize() does NOT use `small: true`; only the title generator does today (only `small: true` match in the codebase). * revert: drop opencode title-gen suppression We have no evidence — and can't construct a realistic scenario — where title-gen fails on an otherwise-successful run. Title-gen and primary share the same OPENROUTER_API_KEY and hit the same proxy/upstream; whatever breaks one breaks the other. The original repro on run 25580969379 is fully explained by the stale proxy key (fix #1) — title-gen happened to be the first call that surfaced the auth error, but every subsequent primary call would have died the same way. Suppression code adds complexity (cross-stream correlation logic, message matching, set capping) and a real failure mode of its own (a small-model error with a unique message could mask an unrelated primary error landing shortly after). Net negative. Removing.
76 lines
4.2 KiB
TypeScript
76 lines
4.2 KiB
TypeScript
type ProviderErrorPattern = { regex: RegExp; label: string };
|
|
|
|
// status codes are only treated as provider errors when they are adjacent to
|
|
// a recognised status key. this rejects commit SHAs that happen to contain
|
|
// "429", version strings, file hashes, etc.
|
|
const statusKey = `\\b(?:status[_ ]?code|http[_ ]?status|status)["']?\\s*[:=]\\s*["']?`;
|
|
|
|
const PROVIDER_ERROR_PATTERNS: ProviderErrorPattern[] = [
|
|
// auth patterns must come BEFORE rate-limit patterns. OpenRouter 401 error
|
|
// payloads carry `x-ratelimit-*` response headers in the dump, and the
|
|
// free-form rate-limit regex below would otherwise win on word-boundary
|
|
// matches inside header names. canonical 401 messages: OpenRouter returns
|
|
// `{"error":{"message":"User not found","code":401}}` for disabled or
|
|
// invalid keys (https://openai.luzhipeng.com/docs/api/reference/errors-and-debugging).
|
|
{ regex: new RegExp(`${statusKey}401\\b`, "i"), label: "auth error (401)" },
|
|
{ regex: new RegExp(`${statusKey}403\\b`, "i"), label: "auth error (403)" },
|
|
{ regex: /\bUser not found\b/i, label: "auth error (invalid/disabled key)" },
|
|
{ regex: /\bInvalid authentication\b/i, label: "auth error (invalid credentials)" },
|
|
{ regex: /\bNo auth credentials found\b/i, label: "auth error (missing credentials)" },
|
|
{ regex: new RegExp(`${statusKey}429\\b`, "i"), label: "rate limited (429)" },
|
|
{ regex: new RegExp(`${statusKey}500\\b`, "i"), label: "provider 500 error" },
|
|
{ regex: new RegExp(`${statusKey}503\\b`, "i"), label: "provider unavailable (503)" },
|
|
// matches `rate limit`, `rate limited`, `rate limits exceeded`,
|
|
// `rate_limit_error`, `rate_limit_exceeded`. the leading `\b` + `[_ ]`
|
|
// separator rejects `x-ratelimit-*` / `anthropic-ratelimit-*` response
|
|
// headers (no separator between "rate" and "limit") which routinely
|
|
// appear in dumped 401 / 4xx error JSON.
|
|
{ regex: /\brate[_ ]limit/i, label: "rate limited" },
|
|
{ regex: /\bRESOURCE_EXHAUSTED\b/, label: "quota exhausted" },
|
|
// Google gRPC `INTERNAL` status. word-boundary anchors reject
|
|
// `INTERNAL_SERVER_ERROR` (HTTP 500 message that may appear in unrelated
|
|
// log lines) and identifiers like `INTERNALS`.
|
|
{ regex: /\bINTERNAL\b/, label: "provider internal error" },
|
|
{ regex: /\bUNAVAILABLE\b/, label: "provider unavailable" },
|
|
// matches `quota`, `insufficient_quota`, `quota_exceeded`, `quotaExceeded`.
|
|
// word-character lookarounds would reject `_quota` / `quotaX`; `quota` is
|
|
// specific enough that a plain substring match is safe.
|
|
{ regex: /quota/i, label: "quota error" },
|
|
// explicit zero-quota response, e.g. `{"limit": 0}`. the `\b` anchor
|
|
// around `limit` rejects keys like `time_limit` or `field_limit`.
|
|
{ regex: /["']?\blimit\b["']?\s*:\s*0\b/, label: "zero quota" },
|
|
];
|
|
|
|
export function detectProviderError(text: string): string | null {
|
|
for (const entry of PROVIDER_ERROR_PATTERNS) {
|
|
if (entry.regex.test(text)) return entry.label;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/**
|
|
* OpenRouter's response when the per-run key's remaining budget can't cover
|
|
* the agent's `max_tokens` reservation. Distinct from a generic provider error
|
|
* because it's a Pullfrog billing concern, not an upstream outage — the user's
|
|
* Router wallet ran out (or the key budget was undersized at mint time and the
|
|
* agent ran out of headroom partway through).
|
|
*
|
|
* Match must be specific to this exact OpenRouter error class. Generic "credits"
|
|
* or "limit" text shows up in unrelated errors and would mis-classify them.
|
|
*
|
|
* Sample:
|
|
* `APIError: This request requires more credits, or fewer max_tokens.
|
|
* You requested up to 32000 tokens, but can only afford 22800.`
|
|
*/
|
|
// `/s` (dotAll) lets `.*?` cross newlines so we still detect the error if any
|
|
// upstream layer reformats the message onto multiple lines. Without it, a
|
|
// single inserted `\n` would silently bypass the BillingError reclassification
|
|
// and the user would see the generic `❌ Pullfrog failed` dump instead of the
|
|
// actionable top-up CTA.
|
|
const ROUTER_KEYLIMIT_EXHAUSTED_PATTERN =
|
|
/requires more credits.*?fewer max_tokens|requested up to \d+ tokens.*?can only afford/is;
|
|
|
|
export function isRouterKeylimitExhaustedError(text: string): boolean {
|
|
return ROUTER_KEYLIMIT_EXHAUSTED_PATTERN.test(text);
|
|
}
|