Implement support for output schemas (#411)

* tweak examples

* tweak prompt

* Implement support for output schemas

* fix: add example for structured output with zod schema

* tweak

* remove redundant cast

* fix input name

* strip $schema

* hack around vendor requirement

* clarify required result output

---------

Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com>

README.md

@@ -1,5 +1,5 @@
-<!-- test preview system --> <!-- test bypass 2 --> <!-- trigger preview repo creation -->
+<!-- test preview system -->
-<p align="center">  
+<p align="center">
   <h1 align="center">
     <picture>
       <source media="(prefers-color-scheme: dark)" srcset="https://pullfrog.com/frog-white-200px.png">
@@ -24,27 +24,26 @@ Pullfrog is a GitHub bot that brings the full power of your favorite coding agen
 
 - **Tag `@pullfrog`** — Tag `@pullfrog` in a comment anywhere in your repo. It will pull in any relevant context using the action's internal MCP server and perform the appropriate task.
 - **Prompt from the web** — Trigger arbitrary tasks from the Pullfrog dashboard
-- **Automated triggers** — Configure Pullfrog to trigger agent runs in response to specific events. Each of these triggers can be associated with custom prompt instructions. 
+- **Automated triggers** — Configure Pullfrog to trigger agent runs in response to specific events. Each of these triggers can be associated with custom prompt instructions.
   - issue created
   - issue labeled
   - PR created
-  - PR review created 
+  - PR review created
   - PR review requested
   - and more...
 
 Pullfrog is the bridge between your preferred coding agents and GitHub. Use it for:
- 
+
 - **🤖 Coding tasks** — Tell `@pullfrog` to implement something and it'll spin up a PR. If CI fails, it'll read the logs and attempt a fix automatically. It'll automatically address any PR reviews too.
 - **🔍 PR review** — Coding agents are great at reviewing PRs. Using the "PR created" trigger, you can configure Pullfrog to auto-review new PRs.
 - **🤙 Issue management** — Via the "issue created" trigger, Pullfrog can automatically respond to common questions, create implementation plans, and link to related issues/PRs. Or (if you're feeling lucky) you can prompt it to immediately attempt a PR addressing new issues.
 - **Literally whatever** — Want to have the agent automatically add docs to all new PRs? Cut a new release with agent-written notes on every commit to `main`? Pullfrog lets you do it.
 
-
 <!-- Features
 - **Agent-agnostic** — Switch between agents with the click of a radio button.
 - ** -->
 
-<!-- 
+<!--
 ## Get started
 
 Install the Pullfrog GitHub App on your personal or organization account. During installation you can choose to limit access to a specific repo or repos. After installation, you'll be redirected to the Pullfrog dashboard where you'll see an onboarding flow. This flow will create your `pullfrog.yml` workflow and prompt you to set up API keys. Once you finish those steps (2 minutes) you're ready to rock.
@@ -124,14 +123,14 @@ on:
   pull_request_review:
     types: [submitted]
   # add other triggers as needed
-  
+
 
 jobs:
   pullfrog:
-    
+
     # trigger conditions (e.g. only run if @pullfrog is mentioned)
     if: contains(github.event.comment.body, '@pullfrog') || contains(github.event.issue.body, '@pullfrog')
-    
+
     permissions:
       id-token: write
       contents: write
@@ -148,3 +147,104 @@ jobs:
 
 </details>
 -->
+
+## Standalone Usage
+
+You can also use `pullfrog/pullfrog` as a step in your own workflows. The action exposes a `result` output that can be consumed by subsequent steps.
+
+### Example: Auto-generate release notes on new tags
+
+```yaml
+name: Release
+on:
+  push:
+    tags: ['v*']
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Generate release notes
+        id: notes
+        uses: pullfrog/pullfrog@v0
+        with:
+          prompt: |
+            Generate release notes for ${{ github.ref_name }}.
+            Compare commits between this tag and the previous tag.
+            Format as markdown: summary paragraph, then ### Features, ### Fixes, ### Breaking Changes sections.
+            Omit empty sections. Be concise.
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+
+      # write to file to avoid shell escaping issues with special characters
+      - name: Create GitHub release
+        run: |
+          notesfile="$RUNNER_TEMP/release-notes-$GITHUB_RUN_ID.md"
+          printf '%s' "$NOTES" > "$notesfile"
+          gh release create ${{ github.ref_name }} --title "${{ github.ref_name }}" --notes-file "$notesfile"
+        env:
+          GH_TOKEN: ${{ github.token }}
+          NOTES: ${{ steps.notes.outputs.result }}
+```
+
+### Example: Structured Output with Zod Schema
+
+You can force the agent to return structured JSON output by providing a JSON schema. This allows you to reliably parse and use the agent's response in subsequent workflow steps.
+
+You can define your JSON schema directly or uou can use any validation library that converts to JSON Schema. Here's an example using [Zod](https://zod.dev):
+
+```yaml
+name: Release Check
+on:
+  pull_request:
+    types: [closed]
+
+jobs:
+  check-release:
+    if: github.event.pull_request.merged == true
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install dependencies
+        run: npm install --no-save --no-package-lock zod @actions/core
+
+      - name: Generate Schema
+        id: schema
+        run: |
+          node -e '
+            import { z } from "zod";
+            import { setOutput } from "@actions/core";
+            const schema = z.object({
+              version: z.string().describe("Semantic version number (e.g. 1.0.0)"),
+              isBreaking: z.boolean().describe("Whether this release contains breaking changes"),
+              changelog: z.array(z.string()).describe("List of changes in this release"),
+            });
+            setOutput("schema", JSON.stringify(z.toJSONSchema(schema)));
+          '
+
+      - name: Analyze PR
+        id: analysis
+        uses: pullfrog/pullfrog@v0
+        with:
+          prompt: |
+            Analyze this PR and determine semantic versioning impact.
+            Return a JSON object matching the provided schema.
+          output_schema: ${{ steps.schema.outputs.schema }}
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+
+      - name: Process Result
+        run: |
+          # Parse the JSON result using fromJSON()
+          echo "Version: ${{ fromJSON(steps.analysis.outputs.result).version }}"
+          echo "Breaking: ${{ fromJSON(steps.analysis.outputs.result).isBreaking }}"
+```

action.yml

@@ -30,6 +30,9 @@ inputs:
   shell:
     description: "Shell permission: disabled, restricted (filters secrets from env vars), or enabled. Public repos default to restricted for security; private repos default to enabled."
     required: false
+  output_schema:
+    description: "JSON Schema (draft-07) for structured output validation. When provided, the action output becomes required and must conform to this schema."
+    required: false
   token:
     description: "GitHub-provided token with job-scoped permissions. Do not set this unless you know what you are doing."
     required: false
@@ -37,7 +40,7 @@ inputs:
 
 outputs:
   result:
-    description: "It's set when the prompt explicitly requests it. It can be used to capture an actionable output for the next step in the workflow."
+    description: "It's set when the prompt explicitly requests it and is required when output_schema is provided; use it to capture actionable output for the next workflow step."
 
 runs:
   using: "node24"

main.ts

@@ -1,4 +1,5 @@
 // changes to tool permissions should be reflected in wiki/granular-tools.md
+import * as core from "@actions/core";
 import { initToolState, startMcpHttpServer, type ToolState } from "./mcp/server.ts";
 import { computeModes } from "./modes.ts";
 import {
@@ -37,6 +38,23 @@ export interface MainResult {
   result?: string | undefined;
 }
 
+function resolveOutputSchema(): Record<string, unknown> | undefined {
+  const raw = core.getInput("output_schema");
+  if (!raw) return undefined;
+
+  let parsed: unknown;
+  try {
+    parsed = JSON.parse(raw);
+  } catch {
+    throw new Error(`invalid output_schema: not valid JSON`);
+  }
+  if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
+    throw new Error(`invalid output_schema: must be a JSON object`);
+  }
+  log.info("» structured output schema provided — output will be required");
+  return parsed as Record<string, unknown>;
+}
+
 async function writeJobSummary(toolState: ToolState): Promise<void> {
   const usageSummary = formatUsageSummary(toolState.usageEntries);
   const summaryParts = [toolState.lastProgressBody, usageSummary].filter(Boolean);
@@ -151,6 +169,8 @@ export async function main(): Promise<MainResult> {
 
     const modes = [...computeModes(), ...runContext.repoSettings.modes];
 
+    const outputSchema = resolveOutputSchema();
+
     // mcpServerUrl and tmpdir are set after server starts — delegate tool reads them at call time
     const toolContext = {
       repo: runContext.repo,
@@ -169,7 +189,7 @@ export async function main(): Promise<MainResult> {
       mcpServerUrl: "",
       tmpdir,
     };
-    await using mcpHttpServer = await startMcpHttpServer(toolContext);
+    await using mcpHttpServer = await startMcpHttpServer(toolContext, { outputSchema });
     toolContext.mcpServerUrl = mcpHttpServer.url;
     log.info(`» MCP server started at ${mcpHttpServer.url}`);
     timer.checkpoint("mcpServer");
@@ -178,6 +198,7 @@ export async function main(): Promise<MainResult> {
       payload,
       repo: runContext.repo,
       modes,
+      outputSchema,
     });
     // log instructions as soon as they are fully resolved
     const logParts = [
@@ -236,6 +257,13 @@ export async function main(): Promise<MainResult> {
       toolState.usageEntries.push(result.usage);
     }
 
+    // validate this before writing job summary to avoid masking the error
+    if (outputSchema && !toolState.output) {
+      throw new Error(
+        "output_schema was provided but agent did not call set_output — structured output is required"
+      );
+    }
+
     await writeJobSummary(toolState);
 
     // emit structured output marker for test validation

mcp/output.ts

@@ -1,3 +1,5 @@
+import type { StandardSchemaV1 } from "@standard-schema/spec";
+import { Ajv } from "ajv";
 import { type } from "arktype";
 import { log } from "../utils/cli.ts";
 import type { ToolContext } from "./server.ts";
@@ -7,29 +9,76 @@ export const SetOutputParams = type({
   value: type.string.describe("the output value to expose as a GitHub Action output"),
 });
 
-export function SetOutputTool(ctx: ToolContext) {
+type JsonSchema = Record<string, unknown>;
+
+function jsonSchemaToStandardSchema({
+  $schema: _,
+  ...jsonSchema
+}: JsonSchema): StandardSchemaV1<any> & { toJsonSchema(): JsonSchema } {
+  const ajv = new Ajv();
+  const validate = ajv.compile(jsonSchema);
+
+  return {
+    "~standard": {
+      version: 1,
+      // xsschema (used by fastmcp) hardcodes supported vendors instead of duck-typing toJsonSchema().
+      // "arktype" works because its converter is just `(schema) => schema.toJsonSchema()`.
+      vendor: "arktype",
+      validate(input: unknown) {
+        if (validate(input)) {
+          return { value: input };
+        }
+        return {
+          issues: (validate.errors ?? []).map((err) => ({
+            message: `${err.instancePath || "/"}: ${err.message ?? "validation error"}`,
+            path: err.instancePath ? err.instancePath.split("/").filter(Boolean) : [],
+          })),
+        };
+      },
+    },
+    toJsonSchema() {
+      return jsonSchema;
+    },
+  };
+}
+
+function storeOutput(ctx: ToolContext, value: string) {
+  const selfId = ctx.toolState.selfSubagentId;
+  if (selfId) {
+    const subagent = ctx.toolState.subagents.get(selfId);
+    if (subagent) {
+      subagent.output = value;
+      log.debug(`set_output: routed to subagent ${selfId} (value=${value.slice(0, 80)})`);
+      return { success: true, routed: "subagent" as const };
+    }
+    log.warning(
+      `set_output: selfSubagentId=${selfId} but subagent not found in map — routing to action output`
+    );
+  }
+  ctx.toolState.output = value;
+  return { success: true, routed: "action_output" as const };
+}
+
+export function SetOutputTool(ctx: ToolContext, outputSchema?: JsonSchema) {
+  if (outputSchema) {
+    return tool({
+      name: "set_output",
+      description:
+        "Set the structured action output. You MUST call this tool before finishing — the output is required. Pass the output object directly as the tool arguments (no wrapping needed).",
+      parameters: jsonSchemaToStandardSchema(outputSchema),
+      execute: execute(async (params) => {
+        return storeOutput(ctx, JSON.stringify(params));
+      }),
+    });
+  }
+
   return tool({
     name: "set_output",
     description:
       "Set the action output. When called by a subagent, returns a summary result to the orchestrator — this is the ONLY way to pass results back. When called by the orchestrator in standalone mode (trigger: unknown), exposes the value as the 'result' GitHub Action output for downstream workflow steps. Do NOT use this for progress reporting — use report_progress instead.",
     parameters: SetOutputParams,
     execute: execute(async (params) => {
-      const selfId = ctx.toolState.selfSubagentId;
+      return storeOutput(ctx, params.value);
-      if (selfId) {
-        const subagent = ctx.toolState.subagents.get(selfId);
-        if (subagent) {
-          subagent.output = params.value;
-          log.debug(
-            `set_output: routed to subagent ${selfId} (value=${params.value.slice(0, 80)})`
-          );
-          return { success: true, routed: "subagent" };
-        }
-        log.warning(
-          `set_output: selfSubagentId=${selfId} but subagent not found in map — routing to action output`
-        );
-      }
-      ctx.toolState.output = params.value;
-      return { success: true, routed: "action_output" };
     }),
   });
 }

mcp/server.ts

@@ -189,8 +189,10 @@ function isAddressInUse(error: unknown): boolean {
   return message.includes("eaddrinuse") || message.includes("address already in use");
 }
 
+type JsonSchema = Record<string, unknown>;
+
 // tools shared by both orchestrator and subagent servers
-function buildCommonTools(ctx: ToolContext): Tool<any, any>[] {
+function buildCommonTools(ctx: ToolContext, outputSchema?: JsonSchema): Tool<any, any>[] {
   const tools: Tool<any, any>[] = [
     StartDependencyInstallationTool(ctx),
     AwaitDependencyInstallationTool(ctx),
@@ -213,7 +215,7 @@ function buildCommonTools(ctx: ToolContext): Tool<any, any>[] {
     GitTool(ctx),
     GitFetchTool(ctx),
     UploadFileTool(ctx),
-    SetOutputTool(ctx),
+    SetOutputTool(ctx, outputSchema),
     FileReadTool(ctx),
     FileWriteTool(ctx),
     FileEditTool(ctx),
@@ -234,9 +236,9 @@ function buildCommonTools(ctx: ToolContext): Tool<any, any>[] {
 }
 
 // orchestrator gets common tools + delegation + remote-mutating tools
-function buildOrchestratorTools(ctx: ToolContext): Tool<any, any>[] {
+function buildOrchestratorTools(ctx: ToolContext, outputSchema?: JsonSchema): Tool<any, any>[] {
   return [
-    ...buildCommonTools(ctx),
+    ...buildCommonTools(ctx, outputSchema),
     ReportProgressTool(ctx),
     SelectModeTool(ctx),
     DelegateTool(ctx),
@@ -353,13 +355,18 @@ async function killBackgroundProcesses(toolState: ToolState): Promise<void> {
   backgroundProcesses.clear();
 }
 
+type McpHttpServerOptions = {
+  outputSchema?: JsonSchema | undefined;
+};
+
 /**
  * Start the orchestrator MCP HTTP server (has all tools including push/PR/delegation).
  */
 export async function startMcpHttpServer(
-  ctx: ToolContext
+  ctx: ToolContext,
+  options?: McpHttpServerOptions
 ): Promise<{ url: string; [Symbol.asyncDispose]: () => Promise<void> }> {
-  const tools = buildOrchestratorTools(ctx);
+  const tools = buildOrchestratorTools(ctx, options?.outputSchema);
   const startResult = await selectMcpPort(ctx, tools);
 
   return {

package.json

@@ -36,6 +36,7 @@
     "@opencode-ai/sdk": "^1.0.143",
     "@standard-schema/spec": "1.0.0",
     "@toon-format/toon": "^1.0.0",
+    "ajv": "^8.18.0",
     "arkregex": "0.0.5",
     "arktype": "2.1.29",
     "dotenv": "^17.2.3",

pnpm-lock.yaml

@@ -43,6 +43,9 @@ importers:
       '@toon-format/toon':
         specifier: ^1.0.0
         version: 1.4.0
+      ajv:
+        specifier: ^8.18.0
+        version: 8.18.0
       arkregex:
         specifier: 0.0.5
         version: 0.0.5
@@ -868,6 +871,9 @@ packages:
   ajv@8.17.1:
     resolution: {integrity: sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==}
 
+  ajv@8.18.0:
+    resolution: {integrity: sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A==}
+
   ansi-regex@5.0.1:
     resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==}
     engines: {node: '>=8'}
@@ -2280,6 +2286,13 @@ snapshots:
       json-schema-traverse: 1.0.0
       require-from-string: 2.0.2
 
+  ajv@8.18.0:
+    dependencies:
+      fast-deep-equal: 3.1.3
+      fast-uri: 3.1.0
+      json-schema-traverse: 1.0.0
+      require-from-string: 2.0.2
+
   ansi-regex@5.0.1: {}
 
   ansi-regex@6.2.2: {}

post

@@ -41348,6 +41348,7 @@ var package_default = {
     "@opencode-ai/sdk": "^1.0.143",
     "@standard-schema/spec": "1.0.0",
     "@toon-format/toon": "^1.0.0",
+    ajv: "^8.18.0",
     arkregex: "0.0.5",
     arktype: "2.1.29",
     dotenv: "^17.2.3",
@@ -41445,7 +41446,8 @@ var Inputs = type({
   "search?": ToolPermissionInput.or("undefined"),
   "push?": PushPermissionInput.or("undefined"),
   "shell?": ShellPermissionInput.or("undefined"),
-  "cwd?": type.string.or("undefined")
+  "cwd?": type.string.or("undefined"),
+  "output_schema?": type.string.or("undefined")
 });
 function resolvePromptInput() {
   const prompt = core3.getInput("prompt", { required: true });

utils/instructions.ts

@@ -10,6 +10,7 @@ interface InstructionsContext {
   payload: ResolvedPayload;
   repo: RunContextData["repo"];
   modes: Mode[];
+  outputSchema?: Record<string, unknown> | undefined;
 }
 
 function buildRuntimeContext(ctx: InstructionsContext): string {
@@ -118,13 +119,21 @@ Use the \`${ghPullfrogMcpName}\` MCP file tools for all file operations. Do NOT
 All file tools enforce repository-scoped access and prevent modifications to .git/.`;
 }
 
-function getStandaloneModeInstructions(trigger: string): string {
+function getStandaloneModeInstructions(
+  trigger: string,
+  outputSchema?: Record<string, unknown> | undefined
+): string {
   if (trigger !== "unknown") {
     return "";
   }
+
+  const outputRequirement = outputSchema
+    ? `**REQUIRED structured output:** You MUST call \`${ghPullfrogMcpName}/set_output\` before finishing. The tool expects a structured object matching a JSON Schema — inspect its parameter schema to see the exact shape. Omitting this call or providing non-conforming output will fail the action.`
+    : `When you complete your task, call \`${ghPullfrogMcpName}/set_output\` with the main result of your work (generated content, summary of changes, analysis results, etc.). This makes it available as a GitHub Action output named \`result\` for subsequent workflow steps to consume. When in doubt, prefer calling \`set_output\`—unused outputs are harmless, but missing outputs may break downstream steps.`;
+
   return `### Standalone mode
 
-You are running as a step in a user-defined CI workflow. When you complete your task, call \`${ghPullfrogMcpName}/set_output\` with the main result of your work (generated content, summary of changes, analysis results, etc.). This makes it available as a GitHub Action output named \`result\` for subsequent workflow steps to consume.`;
+You are running as a step in a user-defined CI workflow. ${outputRequirement}`;
 }
 
 // shared system prompt body used by both orchestrator and subagent instructions.
@@ -134,6 +143,7 @@ interface SystemPromptContext {
   trigger: string;
   priorityOrder: string;
   taskSection: string;
+  outputSchema?: Record<string, unknown> | undefined;
 }
 
 function buildSystemPrompt(ctx: SystemPromptContext): string {
@@ -192,7 +202,7 @@ ${getShellInstructions(ctx.shell)}
 
 ${getFileInstructions()}
 
-${getStandaloneModeInstructions(ctx.trigger)}
+${getStandaloneModeInstructions(ctx.trigger, ctx.outputSchema)}
 
 ## Workflow
 
@@ -406,6 +416,7 @@ If the task clearly requires no work, skip delegation. Call \`${ghPullfrogMcpNam
     trigger: ctx.payload.event.trigger,
     priorityOrder: orchestratorPriorityOrder,
     taskSection: orchestratorTaskSection,
+    outputSchema: ctx.outputSchema,
   });
 
   const contextSections = buildContextSections({

utils/payload.ts

@@ -54,6 +54,7 @@ export const Inputs = type({
   "push?": PushPermissionInput.or("undefined"),
   "shell?": ShellPermissionInput.or("undefined"),
   "cwd?": type.string.or("undefined"),
+  "output_schema?": type.string.or("undefined"),
 });
 
 export type Inputs = typeof Inputs.infer;