clawdbot/docs/ios.md

summary, read_when

summary	read_when
iOS app (node): architecture + connection runbook	Pairing or reconnecting the iOS node Debugging iOS bridge discovery or auth Sending screen/canvas commands to iOS Designing iOS node + gateway integration Extending the Gateway protocol for node/canvas commands Implementing Bonjour pairing or transport security

iOS App (Node)

Status: prototype implemented (internal) · Date: 2025-12-13

Connection Runbook

This is the practical “how do I connect the iOS node” guide:

iOS app ⇄ (Bonjour + TCP bridge) ⇄ Gateway bridge ⇄ (loopback WS) ⇄ Gateway

The Gateway WebSocket stays loopback-only (ws://127.0.0.1:18789). The iOS node talks to the LAN-facing bridge (default tcp://0.0.0.0:18790) and uses Gateway-owned pairing.

Prerequisites

• You can run the Gateway on the “master” machine.
• iOS node app can reach the gateway bridge:
  • Same LAN with Bonjour/mDNS, or
  • Same Tailscale tailnet using Wide-Area Bonjour / unicast DNS-SD (see below), or
  • Manual bridge host/port (fallback)
• You can run the CLI (clawdbot) on the gateway machine (or via SSH).

1) Start the Gateway (with bridge enabled)

Bridge is enabled by default (disable via CLAWDBOT_BRIDGE_ENABLED=0).

pnpm clawdbot gateway --port 18789 --verbose

Confirm in logs you see something like:

• bridge listening on tcp://0.0.0.0:18790 (node)

For tailnet-only setups (recommended for Vienna ⇄ London), bind the bridge to the gateway machine’s Tailscale IP instead:

• Set bridge.bind: "tailnet" in ~/.clawdbot/clawdbot.json on the gateway host.
• Restart the Gateway / macOS menubar app.

2) Verify Bonjour discovery (optional but recommended)

From the gateway machine:

dns-sd -B _clawdbot-bridge._tcp local.

You should see your gateway advertising _clawdbot-bridge._tcp.

If browse works, but the iOS node can’t connect, try resolving one instance:

dns-sd -L "<instance name>" _clawdbot-bridge._tcp local.

More debugging notes: docs/bonjour.md.

Tailnet (Vienna ⇄ London) discovery via unicast DNS-SD

If the iOS node and the gateway are on different networks but connected via Tailscale, multicast mDNS won’t cross the boundary. Use Wide-Area Bonjour / unicast DNS-SD instead:

1. Set up a DNS-SD zone (example clawdbot.internal.) on the gateway host and publish _clawdbot-bridge._tcp records.
2. Configure Tailscale split DNS for clawdbot.internal pointing at that DNS server.

Details and example CoreDNS config: docs/bonjour.md.

3) Connect from the iOS node app

In the iOS node app:

• Pick the discovered bridge (or hit refresh).
• If not paired yet, it will initiate pairing automatically.
• After the first successful pairing, it will auto-reconnect strictly to the last discovered gateway on launch (including after reinstall), as long as the iOS Keychain entry is still present.

Connection indicator (always visible)

The Settings tab icon shows a small status dot:

• Green: connected to the bridge
• Yellow: connecting (subtle pulse)
• Red: not connected / error

4) Approve pairing (CLI)

On the gateway machine:

clawdbot nodes pending

Approve the request:

clawdbot nodes approve <requestId>

After approval, the iOS node receives/stores the token and reconnects authenticated.

Pairing details: docs/gateway/pairing.md.

5) Verify the node is connected

• In the macOS app: Instances tab should show something like iOS Node (...) with a green “Active” presence dot shortly after connect.
• Via nodes status (paired + connected):

  clawdbot nodes status
  

• Via Gateway (paired + connected):

  clawdbot gateway call node.list --params "{}"
  

• Via Gateway presence (legacy-ish, still useful):

  clawdbot gateway call system-presence --params "{}"
  

  Look for the node instanceId (often a UUID).

6) Drive the iOS Canvas (draw / snapshot)

The iOS node runs a WKWebView “Canvas” scaffold which exposes:

• window.__clawdbot.canvas
• window.__clawdbot.ctx (2D context)
• window.__clawdbot.setStatus(title, subtitle)

Gateway Canvas Host (recommended for web content)

If you want the node to show real HTML/CSS/JS that the agent can edit on disk, point it at the Gateway canvas host.

Note: nodes always use the standalone canvas host on canvasHost.port (default 18793), bound to the bridge interface.

1. Create ~/clawd/canvas/index.html on the gateway host.

2. Navigate the node to it (LAN):

clawdbot nodes invoke --node "iOS Node" --command canvas.navigate --params '{"url":"http://<gateway-hostname>.local:18793/__clawdbot__/canvas/"}'

Notes:

• The server injects a live-reload client into HTML and reloads on file changes.
• A2UI is hosted on the same canvas host at http://<gateway-host>:18793/__clawdbot__/a2ui/.
• Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of .local, e.g. http://<gateway-magicdns>:18793/__clawdbot__/canvas/.
• iOS may require App Transport Security allowances to load plain http:// URLs; if it fails to load, prefer HTTPS or adjust the iOS app’s ATS config.

Draw with canvas.eval

clawdbot nodes invoke --node "iOS Node" --command canvas.eval --params "$(cat <<'JSON'
{"javaScript":"(() => { const {ctx,setStatus} = window.__clawdbot; setStatus('Drawing','…'); ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle='#ff2d55'; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); setStatus(null,null); return 'ok'; })()"}
JSON
)"

Snapshot with canvas.snapshot

clawdbot nodes invoke --node 192.168.0.88 --command canvas.snapshot --params '{"maxWidth":900}'

The response includes { format, base64 } image data (default format="jpeg"; pass {"format":"png"} when you specifically need lossless PNG).

Common gotchas

• iOS in background: all canvas.* commands fail fast with NODE_BACKGROUND_UNAVAILABLE (bring the iOS node app to foreground).
• Return to default scaffold: canvas.navigate with {"url":""} or {"url":"/"} returns to the built-in scaffold page.
• mDNS blocked: some networks block multicast; use a different LAN or plan a tailnet-capable bridge (see docs/discovery.md).
• Wrong node selector: --node can be the node id (UUID), display name (e.g. iOS Node), IP, or an unambiguous prefix. If it’s ambiguous, the CLI will tell you.
• Stale pairing / Keychain cleared: if the pairing token is missing (or iOS Keychain was wiped), the node must pair again; approve a new pending request.
• App reinstall but no reconnect: the node restores instanceId + last bridge preference from Keychain; if it still comes up “unpaired”, verify Keychain persistence on your device/simulator and re-pair once.

Design + Architecture

Goals

• Build an iOS app that acts as a remote node for Clawdbot:
  • Voice trigger (wake-word / always-listening intent) that forwards transcripts to the Gateway agent method.
  • Canvas surface that the agent can control: navigate, draw/render, evaluate JS, snapshot.
• Dead-simple setup:
  • Auto-discover the host on the local network via Bonjour.
  • One-tap pairing with an approval prompt on the Mac.
  • iOS is never a local gateway; it is always a remote node.
• Operational clarity:
  • When iOS is backgrounded, voice may still run; canvas commands must fail fast with a structured error.
  • Provide settings: node display name, enable/disable voice wake, pairing status.

Non-goals (v1):

• Exposing the Node Gateway directly on the LAN.
• Supporting arbitrary third-party “plugins” on iOS.
• Perfect App Store compliance; this is internal-only initially.

Current repo reality (constraints we respect)

• The Gateway WebSocket server binds to 127.0.0.1:18789 (src/gateway/server.ts) with an optional CLAWDBOT_GATEWAY_TOKEN.
• The Gateway exposes a Canvas file server (canvasHost) on canvasHost.port (default 18793), so nodes can canvas.navigate to http://<lanHost>:18793/__clawdbot__/canvas/ and auto-reload on file changes (docs/configuration.md).
• macOS “Canvas” is controlled via the Gateway node protocol (canvas.*), matching iOS/Android (docs/mac/canvas.md).
• Voice wake forwards via GatewayChannel to Gateway agent (mac app: VoiceWakeForwarder → GatewayConnection.sendAgent).

Recommended topology (B): Gateway-owned Bridge + loopback Gateway

Keep the Node gateway loopback-only; expose a dedicated gateway-owned bridge to the LAN/tailnet.

iOS App ⇄ (TLS + pairing) ⇄ Bridge (in gateway) ⇄ (loopback) ⇄ Gateway WS (ws://127.0.0.1:18789)

Why:

• Preserves current threat model: Gateway remains local-only.
• Centralizes auth, rate limiting, and allowlisting in the bridge.
• Lets us unify “canvas node” semantics across mac + iOS without exposing raw gateway methods.

Security plan (internal, but still robust)

Transport

• Current (v0): bridge is a LAN-facing TCP listener with token-based auth after pairing.
• Next: wrap the bridge in TLS and prefer key-pinned or mTLS-like auth after pairing.

Pairing

• Bonjour discovery shows a candidate “Clawdbot Bridge” on the LAN.
• First connection:
  1. iOS generates a keypair (Secure Enclave if available).
  2. iOS connects to the bridge and requests pairing.
  3. The bridge forwards the pairing request to the Gateway as a pending request.
  4. Approval can happen via:
     • macOS UI (Clawdbot shows an alert with Approve/Reject/Later, including the node IP), or
     • Terminal/CLI (headless flows).
  5. Once approved, the bridge returns a token to iOS; iOS stores it in Keychain.
• Subsequent connections:
  • The bridge requires the paired identity. Unpaired clients get a structured “not paired” error and no access.

Gateway-owned pairing (Option B details)

Pairing decisions must be owned by the Gateway (clawd / Node) so nodes can be approved without the macOS app running.

Key idea:

• The Swift app may still show an alert, but it is only a frontend for pending requests stored in the Gateway.

Desired behavior:

• If the Swift UI is present: show alert with Approve/Reject/Later.
• If the Swift UI is not present: clawdbot CLI can list pending requests and approve/reject.

See docs/gateway/pairing.md for the API/events and storage.

CLI (headless approvals):

• clawdbot nodes pending
• clawdbot nodes approve <requestId>
• clawdbot nodes reject <requestId>

Authorization / scope control (bridge-side ACL)

The bridge must not be a raw proxy to every gateway method.

• Allow by default:
  • agent (with guardrails; idempotency required)
  • minimal system-event beacons (presence updates for the node)
  • node/canvas methods defined below (new protocol surface)
• Deny by default:
  • anything that widens control without explicit intent (future “shell”, “files”, etc.)
• Rate limit:
  • handshake attempts
  • voice forwards per minute
  • snapshot frequency / payload size

Protocol unification: add “node/canvas” to Gateway protocol

Principle

Unify mac Canvas + iOS Canvas under a single conceptual surface:

• The agent talks to the Gateway using a stable method set (typed protocol).
• The Gateway routes node-targeted requests to:
  • local mac Canvas implementation, or
  • remote iOS node via the bridge

Minimal protocol additions (v1)

Add to src/gateway/protocol/schema.ts (and regenerate Swift models):

Identity

• Node identity comes from connect.params.client.instanceId (stable), and connect.params.client.mode = "node" (or "ios-node").

Methods

• node.list → list paired/connected nodes + capabilities
• node.describe → describe a node (capabilities + supported node.invoke commands)
• node.invoke → send a command to a specific node
  • Params: { nodeId, command, params?, timeoutMs? }

Events

• node.event → async node status/errors
  • e.g. background/foreground transitions, voice availability, canvas availability

Node command set (canvas)

These are values for node.invoke.command:

• canvas.present / canvas.hide
• canvas.navigate with { url } (loads a URL; use "" or "/" to return to the default scaffold)
• canvas.eval with { javaScript }
• canvas.snapshot with { maxWidth?, quality?, format? }
• A2UI (mobile + macOS canvas):
  • canvas.a2ui.push with { messages: [...] } (A2UI v0.8 server→client messages)
  • canvas.a2ui.pushJSONL with { jsonl: "..." } (legacy alias)
  • canvas.a2ui.reset
  • A2UI is hosted by the Gateway canvas host (/__clawdbot__/a2ui/) on canvasHost.port. Commands fail if the host is unreachable.

Result pattern:

• Request is a standard req/res with ok / error.
• Long operations (loads, streaming drawing, etc.) may also emit node.event progress.

Current (implemented)

As of 2025-12-13, the Gateway supports node.invoke for bridge-connected nodes.

Example: draw a diagonal line on the iOS Canvas:

clawdbot nodes invoke --node ios-node --command canvas.eval --params '{"javaScript":"(() => { const {ctx} = window.__clawdbot; ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle=\"#ff2d55\"; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); return \"ok\"; })()"}'

Background behavior requirement

When iOS is backgrounded:

• Voice may still be active (subject to iOS suspension).
• All canvas.* commands must fail with a stable error code, e.g.:
  • NODE_BACKGROUND_UNAVAILABLE
  • Include retryable: true and retryAfterMs if we want the agent to wait.

iOS app architecture (SwiftUI)

App structure

• Single fullscreen Canvas surface (WKWebView).
• One settings entry point: a gear button that opens a settings sheet.
• All navigation is agent-driven (no local URL bar).

Components

• BridgeDiscovery: Bonjour browse + resolve (Network.framework NWBrowser)
• BridgeConnection: TCP session + pairing handshake + reconnect (TLS planned)
• NodeRuntime:
  • Voice pipeline (wake-word + capture + forward)
  • Canvas pipeline (WKWebView controller + snapshot + eval)
  • Background state tracking; enforces “canvas unavailable in background”

Voice in background (internal)

• Enable background audio mode (and required session configuration) so the mic pipeline can keep running when the user switches apps.
• If iOS suspends the app anyway, surface a clear node status (node.event) so operators can see voice is unavailable.

Code sharing (macOS + iOS)

Create/expand SwiftPM targets so both apps share:

• ClawdbotProtocol (generated models; platform-neutral)
• ClawdbotGatewayClient (shared WS framing + connect/req/res + seq-gap handling)
• ClawdbotKit (node/canvas command types + deep links + shared utilities)

macOS continues to own:

• local Canvas implementation details (custom scheme handler serving on-disk HTML, window/panel presentation)

iOS owns:

• iOS-specific audio/speech + WKWebView presentation and lifecycle

Repo layout

• iOS app: apps/ios/ (XcodeGen project.yml)
• Shared Swift packages: apps/shared/
• Lint/format: iOS target runs swiftformat --lint + swiftlint lint using repo configs (.swiftformat, .swiftlint.yml).

Generate the Xcode project:

cd apps/ios
xcodegen generate
open Clawdbot.xcodeproj

Storage plan (private by default)

iOS

• Canvas/workspace files (persistent, private):
  • Application Support/Clawdbot/canvas/<sessionKey>/...
• Snapshots / temp exports (evictable):
  • Library/Caches/Clawdbot/canvas-snapshots/<sessionKey>/...
• Credentials:
  • Keychain (paired identity + bridge trust anchor)

Related docs

• docs/gateway.md (gateway runbook)
• docs/gateway/pairing.md (approval + storage)
• docs/bonjour.md (discovery debugging)
• docs/discovery.md (LAN vs tailnet vs SSH)