feat(gateway): implement OpenResponses /v1/responses endpoint phase 2

- Add input_image and input_file support with SSRF protection
- Add client-side tools (Hosted Tools) support
- Add turn-based tool flow with function_call_output handling
- Export buildAgentPrompt for testing

docs/experiments/plans/openresponses-gateway.md

@@ -8,6 +8,7 @@ last_updated: "2026-01-19"
 # OpenResponses Gateway Integration Plan
 
 ## Context
+
 Clawdbot Gateway currently exposes a minimal OpenAI-compatible Chat Completions endpoint at
 `/v1/chat/completions` (see [OpenAI Chat Completions](/gateway/openai-http-api)).
 
@@ -16,19 +17,23 @@ for agentic workflows and uses item-based inputs plus semantic streaming events.
 spec defines `/v1/responses`, not `/v1/chat/completions`.
 
 ## Goals
+
 - Add a `/v1/responses` endpoint that adheres to OpenResponses semantics.
 - Keep Chat Completions as a compatibility layer that is easy to disable and eventually remove.
 - Standardize validation and parsing with isolated, reusable schemas.
 
 ## Non-goals
+
 - Full OpenResponses feature parity in the first pass (images, files, hosted tools).
 - Replacing internal agent execution logic or tool orchestration.
 - Changing the existing `/v1/chat/completions` behavior during the first phase.
 
 ## Research Summary
+
 Sources: OpenResponses OpenAPI, OpenResponses specification site, and the Hugging Face blog post.
 
 Key points extracted:
+
 - `POST /v1/responses` accepts `CreateResponseBody` fields like `model`, `input` (string or
   `ItemParam[]`), `instructions`, `tools`, `tool_choice`, `stream`, `max_output_tokens`, and
   `max_tool_calls`.
@@ -52,6 +57,7 @@ Key points extracted:
 - HF examples include `OpenResponses-Version: latest` in requests (optional header).
 
 ## Proposed Architecture
+
 - Add `src/gateway/open-responses.schema.ts` containing Zod schemas only (no gateway imports).
 - Add `src/gateway/openresponses-http.ts` (or `open-responses-http.ts`) for `/v1/responses`.
 - Keep `src/gateway/openai-http.ts` intact as a legacy compatibility adapter.
@@ -61,6 +67,7 @@ Key points extracted:
 - Emit a startup warning when Chat Completions is enabled to signal legacy status.
 
 ## Deprecation Path for Chat Completions
+
 - Maintain strict module boundaries: no shared schema types between responses and chat completions.
 - Make Chat Completions opt-in by config so it can be disabled without code changes.
 - Update docs to label Chat Completions as legacy once `/v1/responses` is stable.
@@ -68,6 +75,7 @@ Key points extracted:
   removal path.
 
 ## Phase 1 Support Subset
+
 - Accept `input` as string or `ItemParam[]` with message roles and `function_call_output`.
 - Extract system and developer messages into `extraSystemPrompt`.
 - Use the most recent `user` or `function_call_output` as the current message for agent runs.
@@ -76,6 +84,7 @@ Key points extracted:
 - Return `usage` with zeroed values until token accounting is wired.
 
 ## Validation Strategy (No SDK)
+
 - Implement Zod schemas for the supported subset of:
   - `CreateResponseBody`
   - `ItemParam` + message content part unions
@@ -84,6 +93,7 @@ Key points extracted:
 - Keep schemas in a single, isolated module to avoid drift and allow future codegen.
 
 ## Streaming Implementation (Phase 1)
+
 - SSE lines with both `event:` and `data:`.
 - Required sequence (minimum viable):
   - `response.created`
@@ -96,6 +106,7 @@ Key points extracted:
   - `[DONE]`
 
 ## Tests and Verification Plan
+
 - Add e2e coverage for `/v1/responses`:
   - Auth required
   - Non-stream response shape
@@ -106,5 +117,6 @@ Key points extracted:
   `[DONE]`.
 
 ## Doc Updates (Follow-up)
+
 - Add a new docs page for `/v1/responses` usage and examples.
 - Update `/gateway/openai-http-api` with a legacy note and pointer to `/v1/responses`.

src/agents/pi-embedded-runner/run.ts

@@ -482,6 +482,17 @@ export async function runEmbeddedPiAgent(
               agentMeta,
               aborted,
               systemPromptReport: attempt.systemPromptReport,
+              // Handle client tool calls (OpenResponses hosted tools)
+              stopReason: attempt.clientToolCall ? "tool_calls" : undefined,
+              pendingToolCalls: attempt.clientToolCall
+                ? [
+                    {
+                      id: `call_${Date.now()}`,
+                      name: attempt.clientToolCall.name,
+                      arguments: JSON.stringify(attempt.clientToolCall.params),
+                    },
+                  ]
+                : undefined,
             },
             didSendViaMessagingTool: attempt.didSendViaMessagingTool,
             messagingToolSentTexts: attempt.messagingToolSentTexts,

src/agents/pi-embedded-runner/run/attempt.ts

@@ -64,6 +64,7 @@ import { prewarmSessionFile, trackSessionManagerAccess } from "../session-manage
 import { prepareSessionManagerForRun } from "../session-manager-init.js";
 import { buildEmbeddedSystemPrompt, createSystemPromptOverride } from "../system-prompt.js";
 import { splitSdkTools } from "../tool-split.js";
+import { toClientToolDefinitions } from "../../pi-tool-definition-adapter.js";
 import { buildSystemPromptParams } from "../../system-prompt-params.js";
 import { describeUnknownError, mapThinkingLevel } from "../utils.js";
 import { resolveSandboxRuntimeStatus } from "../../sandbox/runtime-status.js";
@@ -314,6 +315,16 @@ export async function runEmbeddedAttempt(
         sandboxEnabled: !!sandbox?.enabled,
       });
 
+      // Add client tools (OpenResponses hosted tools) to customTools
+      let clientToolCallDetected: { name: string; params: Record<string, unknown> } | null = null;
+      const clientToolDefs = params.clientTools
+        ? toClientToolDefinitions(params.clientTools, (toolName, toolParams) => {
+            clientToolCallDetected = { name: toolName, params: toolParams };
+          })
+        : [];
+
+      const allCustomTools = [...customTools, ...clientToolDefs];
+
       ({ session } = await createAgentSession({
         cwd: resolvedWorkspace,
         agentDir,
@@ -323,7 +334,7 @@ export async function runEmbeddedAttempt(
         thinkingLevel: mapThinkingLevel(params.thinkLevel),
         systemPrompt,
         tools: builtInTools,
-        customTools,
+        customTools: allCustomTools,
         sessionManager,
         settingsManager,
         skills: [],
@@ -681,6 +692,8 @@ export async function runEmbeddedAttempt(
         cloudCodeAssistFormatError: Boolean(
           lastAssistant?.errorMessage && isCloudCodeAssistFormatError(lastAssistant.errorMessage),
         ),
+        // Client tool call detected (OpenResponses hosted tools)
+        clientToolCall: clientToolCallDetected ?? undefined,
       };
     } finally {
       // Always tear down the session (and release the lock) before we leave this attempt.

src/agents/pi-embedded-runner/run/params.ts

@@ -6,6 +6,16 @@ import type { ExecElevatedDefaults, ExecToolDefaults } from "../../bash-tools.js
 import type { BlockReplyChunking, ToolResultFormat } from "../../pi-embedded-subscribe.js";
 import type { SkillSnapshot } from "../../skills.js";
 
+// Simplified tool definition for client-provided tools (OpenResponses hosted tools)
+export type ClientToolDefinition = {
+  type: "function";
+  function: {
+    name: string;
+    description?: string;
+    parameters?: Record<string, unknown>;
+  };
+};
+
 export type RunEmbeddedPiAgentParams = {
   sessionId: string;
   sessionKey?: string;
@@ -27,6 +37,8 @@ export type RunEmbeddedPiAgentParams = {
   skillsSnapshot?: SkillSnapshot;
   prompt: string;
   images?: ImageContent[];
+  /** Optional client-provided tools (OpenResponses hosted tools). */
+  clientTools?: ClientToolDefinition[];
   provider?: string;
   model?: string;
   authProfileId?: string;

src/agents/pi-embedded-runner/run/types.ts

@@ -9,6 +9,7 @@ import type { MessagingToolSend } from "../../pi-embedded-messaging.js";
 import type { BlockReplyChunking, ToolResultFormat } from "../../pi-embedded-subscribe.js";
 import type { SkillSnapshot } from "../../skills.js";
 import type { SessionSystemPromptReport } from "../../../config/sessions/types.js";
+import type { ClientToolDefinition } from "./params.js";
 
 type AuthStorage = ReturnType<typeof discoverAuthStorage>;
 type ModelRegistry = ReturnType<typeof discoverModels>;
@@ -30,6 +31,8 @@ export type EmbeddedRunAttemptParams = {
   skillsSnapshot?: SkillSnapshot;
   prompt: string;
   images?: ImageContent[];
+  /** Optional client-provided tools (OpenResponses hosted tools). */
+  clientTools?: ClientToolDefinition[];
   provider: string;
   modelId: string;
   model: Model<Api>;
@@ -79,4 +82,6 @@ export type EmbeddedRunAttemptResult = {
   messagingToolSentTexts: string[];
   messagingToolSentTargets: MessagingToolSend[];
   cloudCodeAssistFormatError: boolean;
+  /** Client tool call detected (OpenResponses hosted tools). */
+  clientToolCall?: { name: string; params: Record<string, unknown> };
 };

src/agents/pi-embedded-runner/types.ts

@@ -23,6 +23,14 @@ export type EmbeddedPiRunMeta = {
     kind: "context_overflow" | "compaction_failure" | "role_ordering";
     message: string;
   };
+  /** Stop reason for the agent run (e.g., "completed", "tool_calls"). */
+  stopReason?: string;
+  /** Pending tool calls when stopReason is "tool_calls". */
+  pendingToolCalls?: Array<{
+    id: string;
+    name: string;
+    arguments: string;
+  }>;
 };
 
 export type EmbeddedPiRunResult = {

src/agents/pi-tool-definition-adapter.ts

@@ -4,6 +4,7 @@ import type {
   AgentToolUpdateCallback,
 } from "@mariozechner/pi-agent-core";
 import type { ToolDefinition } from "@mariozechner/pi-coding-agent";
+import type { ClientToolDefinition } from "./pi-embedded-runner/run/params.js";
 import { logDebug, logError } from "../logger.js";
 import { normalizeToolName } from "./tool-policy.js";
 import { jsonResult } from "./tools/common.js";
@@ -65,3 +66,38 @@ export function toToolDefinitions(tools: AnyAgentTool[]): ToolDefinition[] {
     } satisfies ToolDefinition;
   });
 }
+
+// Convert client tools (OpenResponses hosted tools) to ToolDefinition format
+// These tools are intercepted to return a "pending" result instead of executing
+export function toClientToolDefinitions(
+  tools: ClientToolDefinition[],
+  onClientToolCall?: (toolName: string, params: Record<string, unknown>) => void,
+): ToolDefinition[] {
+  return tools.map((tool) => {
+    const func = tool.function;
+    return {
+      name: func.name,
+      label: func.name,
+      description: func.description ?? "",
+      parameters: func.parameters as any,
+      execute: async (
+        toolCallId,
+        params,
+        _onUpdate: AgentToolUpdateCallback<unknown> | undefined,
+        _ctx,
+        _signal,
+      ): Promise<AgentToolResult<unknown>> => {
+        // Notify handler that a client tool was called
+        if (onClientToolCall) {
+          onClientToolCall(func.name, params as Record<string, unknown>);
+        }
+        // Return a pending result - the client will execute this tool
+        return jsonResult({
+          status: "pending",
+          tool: func.name,
+          message: "Tool execution delegated to client",
+        });
+      },
+    } satisfies ToolDefinition;
+  });
+}

src/commands/agent.ts

@@ -415,6 +415,7 @@ export async function agentCommand(
             skillsSnapshot,
             prompt: body,
             images: opts.images,
+            clientTools: opts.clientTools,
             provider: providerOverride,
             model: modelOverride,
             authProfileId,

src/commands/agent/types.ts

@@ -1,4 +1,5 @@
 import type { ChannelOutboundTargetMode } from "../../channels/plugins/types.js";
+import type { ClientToolDefinition } from "../../agents/pi-embedded-runner/run/params.js";
 
 /** Image content block for Claude API multimodal messages. */
 export type ImageContent = {
@@ -20,6 +21,8 @@ export type AgentCommandOpts = {
   message: string;
   /** Optional image attachments for multimodal messages. */
   images?: ImageContent[];
+  /** Optional client-provided tools (OpenResponses hosted tools). */
+  clientTools?: ClientToolDefinition[];
   /** Agent id override (must exist in config). */
   agentId?: string;
   to?: string;

src/gateway/client.ts

@@ -90,7 +90,7 @@ export class GatewayClient {
     };
     if (url.startsWith("wss://") && this.opts.tlsFingerprint) {
       wsOptions.rejectUnauthorized = false;
-      wsOptions.checkServerIdentity = (_host: string, cert: CertMeta) => {
+      wsOptions.checkServerIdentity = ((_host: string, cert: CertMeta) => {
         const fingerprintValue =
           typeof cert === "object" && cert && "fingerprint256" in cert
             ? ((cert as { fingerprint256?: string }).fingerprint256 ?? "")
@@ -99,9 +99,17 @@ export class GatewayClient {
           typeof fingerprintValue === "string" ? fingerprintValue : "",
         );
         const expected = normalizeFingerprint(this.opts.tlsFingerprint ?? "");
-        if (!expected || !fingerprint) return false;
+        if (!expected) {
-        return fingerprint === expected;
+          return new Error("gateway tls fingerprint missing");
-      };
+        }
+        if (!fingerprint) {
+          return new Error("gateway tls fingerprint unavailable");
+        }
+        if (fingerprint !== expected) {
+          return new Error("gateway tls fingerprint mismatch");
+        }
+        return undefined;
+      }) as any;
     }
     this.ws = new WebSocket(url, wsOptions);
 

src/gateway/open-responses.schema.ts

@@ -27,18 +27,46 @@ export const OutputTextContentPartSchema = z
   })
   .strict();
 
-// For Phase 1, we reject image/file content with helpful errors
+// OpenResponses Image Content: Supports URL or base64 sources
+export const InputImageSourceSchema = z.discriminatedUnion("type", [
+  z.object({
+    type: z.literal("url"),
+    url: z.string().url(),
+  }),
+  z.object({
+    type: z.literal("base64"),
+    media_type: z.enum(["image/jpeg", "image/png", "image/gif", "image/webp"]),
+    data: z.string().min(1), // base64-encoded
+  }),
+]);
+
 export const InputImageContentPartSchema = z
   .object({
     type: z.literal("input_image"),
+    source: InputImageSourceSchema,
   })
-  .passthrough();
+  .strict();
+
+// OpenResponses File Content: Supports URL or base64 sources
+export const InputFileSourceSchema = z.discriminatedUnion("type", [
+  z.object({
+    type: z.literal("url"),
+    url: z.string().url(),
+  }),
+  z.object({
+    type: z.literal("base64"),
+    media_type: z.string().min(1), // MIME type
+    data: z.string().min(1), // base64-encoded
+    filename: z.string().optional(),
+  }),
+]);
 
 export const InputFileContentPartSchema = z
   .object({
     type: z.literal("input_file"),
+    source: InputFileSourceSchema,
   })
-  .passthrough();
+  .strict();
 
 export const ContentPartSchema = z.discriminatedUnion("type", [
   InputTextContentPartSchema,
@@ -117,13 +145,14 @@ export const FunctionToolDefinitionSchema = z
   .object({
     type: z.literal("function"),
     function: z.object({
-      name: z.string(),
+      name: z.string().min(1, "Tool name cannot be empty"),
       description: z.string().optional(),
       parameters: z.record(z.string(), z.unknown()).optional(),
     }),
   })
   .strict();
 
+// OpenResponses tool definitions match internal ToolDefinition structure
 export const ToolDefinitionSchema = FunctionToolDefinitionSchema;
 
 export type ToolDefinition = z.infer<typeof ToolDefinitionSchema>;

src/gateway/openresponses-http.ts

@@ -27,6 +27,8 @@ import {
   type StreamingEvent,
   type Usage,
 } from "./open-responses.schema.js";
+import type { ClientToolDefinition } from "../agents/pi-embedded-runner/run/params.js";
+import type { ImageContent } from "../commands/agent/types.js";
 
 type OpenResponsesHttpOptions = {
   auth: ResolvedGatewayAuth;
@@ -74,16 +76,157 @@ function extractTextContent(content: string | ContentPart[]): string {
     .join("\n");
 }
 
-function hasUnsupportedContent(content: string | ContentPart[]): string | null {
+const PRIVATE_IP_PATTERNS = [
-  if (typeof content === "string") return null;
+  /^127\./, // Loopback
-  for (const part of content) {
+  /^192\.168\./, // Private network
-    if (part.type === "input_image") return "input_image content is not supported in Phase 1";
+  /^10\./, // Private network
-    if (part.type === "input_file") return "input_file content is not supported in Phase 1";
+  /^172\.(1[6-9]|2[0-9]|3[0-1])\./, // Private network
-  }
+  /^::1$/, // IPv6 loopback
-  return null;
+  /^fe80:/, // IPv6 link-local
+  /^fec0:/, // IPv6 site-local
+];
+
+function isPrivateIp(hostname: string): boolean {
+  return PRIVATE_IP_PATTERNS.some((pattern) => pattern.test(hostname));
 }
 
-function buildAgentPrompt(input: string | ItemParam[]): {
+// Fetch with SSRF protection, timeout, and size limits
+async function fetchWithGuard(
+  url: string,
+  maxBytes: number,
+  timeoutMs: number = 10000,
+): Promise<{ data: string; mimeType: string }> {
+  const parsedUrl = new URL(url);
+
+  // Only allow HTTP/HTTPS
+  if (!["http:", "https:"].includes(parsedUrl.protocol)) {
+    throw new Error(`Invalid URL protocol: ${parsedUrl.protocol}. Only HTTP/HTTPS allowed.`);
+  }
+
+  // Block private IPs (SSRF protection)
+  if (isPrivateIp(parsedUrl.hostname)) {
+    throw new Error(`Private IP addresses are not allowed: ${parsedUrl.hostname}`);
+  }
+
+  const controller = new AbortController();
+  const timeoutId = setTimeout(() => controller.abort(), timeoutMs);
+
+  try {
+    const response = await fetch(url, {
+      signal: controller.signal,
+      headers: { "User-Agent": "Clawdbot-Gateway/1.0" },
+    });
+
+    if (!response.ok) {
+      throw new Error(`Failed to fetch: ${response.status} ${response.statusText}`);
+    }
+
+    const contentLength = response.headers.get("content-length");
+    if (contentLength) {
+      const size = parseInt(contentLength, 10);
+      if (size > maxBytes) {
+        throw new Error(`Content too large: ${size} bytes (limit: ${maxBytes} bytes)`);
+      }
+    }
+
+    const buffer = await response.arrayBuffer();
+    if (buffer.byteLength > maxBytes) {
+      throw new Error(`Content too large: ${buffer.byteLength} bytes (limit: ${maxBytes} bytes)`);
+    }
+
+    const mimeType = response.headers.get("content-type") || "application/octet-stream";
+
+    return {
+      data: Buffer.from(buffer).toString("base64"),
+      mimeType,
+    };
+  } finally {
+    clearTimeout(timeoutId);
+  }
+}
+
+const ALLOWED_IMAGE_MIMES = new Set(["image/jpeg", "image/png", "image/gif", "image/webp"]);
+const MAX_IMAGE_BYTES = 10 * 1024 * 1024; // 10MB
+const MAX_FILE_BYTES = 5 * 1024 * 1024; // 5MB
+const ALLOWED_FILE_MIMES = new Set([
+  "text/plain",
+  "text/markdown",
+  "text/html",
+  "text/csv",
+  "application/pdf",
+  "application/json",
+]);
+
+async function extractImageContent(part: ContentPart): Promise<ImageContent | null> {
+  if (part.type !== "input_image") return null;
+
+  const source = part.source as { type: string; url?: string; data?: string; media_type?: string };
+
+  if (source.type === "base64") {
+    if (!source.data) {
+      throw new Error("input_image base64 source missing 'data' field");
+    }
+    const mimeType = source.media_type || "image/png";
+    if (!ALLOWED_IMAGE_MIMES.has(mimeType)) {
+      throw new Error(`Unsupported image MIME type: ${mimeType}`);
+    }
+    return { type: "image", data: source.data, mimeType };
+  }
+
+  if (source.type === "url" && source.url) {
+    const result = await fetchWithGuard(source.url, MAX_IMAGE_BYTES);
+    if (!ALLOWED_IMAGE_MIMES.has(result.mimeType)) {
+      throw new Error(`Unsupported image MIME type from URL: ${result.mimeType}`);
+    }
+    return { type: "image", data: result.data, mimeType: result.mimeType };
+  }
+
+  throw new Error("input_image must have 'source.url' or 'source.data'");
+}
+
+async function extractFileContent(part: ContentPart): Promise<string | null> {
+  if (part.type !== "input_file") return null;
+
+  const source = part.source as {
+    type: string;
+    url?: string;
+    data?: string;
+    media_type?: string;
+    filename?: string;
+  };
+  const filename = source.filename || "file";
+
+  let content: string;
+
+  if (source.type === "base64") {
+    if (!source.data) {
+      throw new Error("input_file base64 source missing 'data' field");
+    }
+    const buffer = Buffer.from(source.data, "base64");
+    if (buffer.byteLength > MAX_FILE_BYTES) {
+      throw new Error(
+        `File too large: ${buffer.byteLength} bytes (limit: ${MAX_FILE_BYTES} bytes)`,
+      );
+    }
+    content = buffer.toString("utf-8");
+  } else if (source.type === "url" && source.url) {
+    const result = await fetchWithGuard(source.url, MAX_FILE_BYTES);
+    if (!ALLOWED_FILE_MIMES.has(result.mimeType)) {
+      throw new Error(`Unsupported file MIME type: ${result.mimeType}`);
+    }
+    content = Buffer.from(result.data, "base64").toString("utf-8");
+  } else {
+    throw new Error("input_file must have 'source.url' or 'source.data'");
+  }
+
+  return `<file name="${filename}">\n${content}\n</file>`;
+}
+
+function extractClientTools(body: CreateResponseBody): ClientToolDefinition[] {
+  return (body.tools ?? []) as ClientToolDefinition[];
+}
+
+export function buildAgentPrompt(input: string | ItemParam[]): {
   message: string;
   extraSystemPrompt?: string;
 } {
@@ -293,33 +436,44 @@ export async function handleOpenResponsesHttpRequest(
   const model = payload.model;
   const user = payload.user;
 
-  // Check for unsupported content types (Phase 1)
+  // Extract images, files, and tools from input (Phase 2)
+  let images: ImageContent[] = [];
+  let fileContents: string[] = [];
   if (Array.isArray(payload.input)) {
     for (const item of payload.input) {
       if (item.type === "message" && typeof item.content !== "string") {
-        const unsupported = hasUnsupportedContent(item.content);
+        for (const part of item.content) {
-        if (unsupported) {
+          const image = await extractImageContent(part);
-          sendJson(res, 400, {
+          if (image) {
-            error: { message: unsupported, type: "invalid_request_error" },
+            images.push(image);
-          });
+            continue;
-          return true;
+          }
+          const file = await extractFileContent(part);
+          if (file) {
+            fileContents.push(file);
+          }
         }
       }
     }
   }
 
+  const clientTools = extractClientTools(payload);
   const agentId = resolveAgentIdForRequest({ req, model });
   const sessionKey = resolveSessionKey({ req, agentId, user });
 
   // Build prompt from input
   const prompt = buildAgentPrompt(payload.input);
 
+  // Append file contents to the message
+  const fullMessage =
+    fileContents.length > 0 ? `${prompt.message}\n\n${fileContents.join("\n\n")}` : prompt.message;
+
   // Handle instructions as extra system prompt
   const extraSystemPrompt = [payload.instructions, prompt.extraSystemPrompt]
     .filter(Boolean)
     .join("\n\n");
 
-  if (!prompt.message) {
+  if (!fullMessage) {
     sendJson(res, 400, {
       error: {
         message: "Missing user message in `input`.",
@@ -337,7 +491,9 @@ export async function handleOpenResponsesHttpRequest(
     try {
       const result = await agentCommand(
         {
-          message: prompt.message,
+          message: fullMessage,
+          images: images.length > 0 ? images : undefined,
+          clientTools: clientTools.length > 0 ? clientTools : undefined,
           extraSystemPrompt: extraSystemPrompt || undefined,
           sessionKey,
           runId: responseId,
@@ -350,6 +506,36 @@ export async function handleOpenResponsesHttpRequest(
       );
 
       const payloads = (result as { payloads?: Array<{ text?: string }> } | null)?.payloads;
+      const meta = (result as { meta?: unknown } | null)?.meta;
+      const stopReason =
+        meta && typeof meta === "object" ? (meta as { stopReason?: string }).stopReason : undefined;
+      const pendingToolCalls =
+        meta && typeof meta === "object"
+          ? (meta as { pendingToolCalls?: Array<{ id: string; name: string; arguments: string }> })
+              .pendingToolCalls
+          : undefined;
+
+      // If agent called a client tool, return function_call instead of text
+      if (stopReason === "tool_calls" && pendingToolCalls && pendingToolCalls.length > 0) {
+        const functionCall = pendingToolCalls[0];
+        const response = createResponseResource({
+          id: responseId,
+          model,
+          status: "incomplete",
+          output: [
+            {
+              type: "function_call",
+              id: functionCall.id,
+              call_id: functionCall.id,
+              name: functionCall.name,
+              arguments: functionCall.arguments,
+            },
+          ],
+        });
+        sendJson(res, 200, response);
+        return true;
+      }
+
       const content =
         Array.isArray(payloads) && payloads.length > 0
           ? payloads
@@ -511,7 +697,9 @@ export async function handleOpenResponsesHttpRequest(
     try {
       const result = await agentCommand(
         {
-          message: prompt.message,
+          message: fullMessage,
+          images: images.length > 0 ? images : undefined,
+          clientTools: clientTools.length > 0 ? clientTools : undefined,
           extraSystemPrompt: extraSystemPrompt || undefined,
           sessionKey,
           runId: responseId,
@@ -527,7 +715,90 @@ export async function handleOpenResponsesHttpRequest(
 
       // Fallback: if no streaming deltas were received, send the full response
       if (!sawAssistantDelta) {
-        const payloads = (result as { payloads?: Array<{ text?: string }> } | null)?.payloads;
+        const resultAny = result as { payloads?: Array<{ text?: string }>; meta?: unknown };
+        const payloads = resultAny.payloads;
+        const meta = resultAny.meta;
+        const stopReason =
+          meta && typeof meta === "object"
+            ? (meta as { stopReason?: string }).stopReason
+            : undefined;
+        const pendingToolCalls =
+          meta && typeof meta === "object"
+            ? (
+                meta as {
+                  pendingToolCalls?: Array<{ id: string; name: string; arguments: string }>;
+                }
+              ).pendingToolCalls
+            : undefined;
+
+        // If agent called a client tool, emit function_call instead of text
+        if (stopReason === "tool_calls" && pendingToolCalls && pendingToolCalls.length > 0) {
+          const functionCall = pendingToolCalls[0];
+          // Complete the text content part
+          writeSseEvent(res, {
+            type: "response.output_text.done",
+            item_id: outputItemId,
+            output_index: 0,
+            content_index: 0,
+            text: "",
+          });
+          writeSseEvent(res, {
+            type: "response.content_part.done",
+            item_id: outputItemId,
+            output_index: 0,
+            content_index: 0,
+            part: { type: "output_text", text: "" },
+          });
+
+          // Complete the message item
+          const completedItem = createAssistantOutputItem({
+            id: outputItemId,
+            text: "",
+            status: "completed",
+          });
+          writeSseEvent(res, {
+            type: "response.output_item.done",
+            output_index: 0,
+            item: completedItem,
+          });
+
+          // Send function_call item
+          const functionCallItemId = `call_${randomUUID()}`;
+          const functionCallItem = {
+            type: "function_call" as const,
+            id: functionCallItemId,
+            call_id: functionCall.id,
+            name: functionCall.name,
+            arguments: functionCall.arguments,
+          };
+          writeSseEvent(res, {
+            type: "response.output_item.added",
+            output_index: 1,
+            item: functionCallItem,
+          });
+          writeSseEvent(res, {
+            type: "response.output_item.done",
+            output_index: 1,
+            item: { ...functionCallItem, status: "completed" as const },
+          });
+          writeSseEvent(res, {
+            type: "response.output_item.done",
+            output_index: 1,
+            item: { ...functionCallItem, status: "completed" as const },
+          });
+
+          const incompleteResponse = createResponseResource({
+            id: responseId,
+            model,
+            status: "incomplete",
+            output: [completedItem, functionCallItem],
+          });
+          writeSseEvent(res, { type: "response.completed", response: incompleteResponse });
+          writeDone(res);
+          res.end();
+          return;
+        }
+
         const content =
           Array.isArray(payloads) && payloads.length > 0
             ? payloads