From 5ef2666127e373b5168708ba222cdad5259d01ea Mon Sep 17 00:00:00 2001 From: Peter Steinberger Date: Sat, 20 Dec 2025 12:17:39 +0000 Subject: [PATCH] docs(canvas): update A2UI hosting --- docs/android/connect.md | 3 +- docs/ios/connect.md | 3 +- docs/ios/spec.md | 3 +- docs/mac/canvas.md | 26 ++++---- docs/refactor/canvas-a2ui.md | 124 ++++++++++++++++++++++++----------- 5 files changed, 104 insertions(+), 55 deletions(-) diff --git a/docs/android/connect.md b/docs/android/connect.md index 49c467b70..65fe284da 100644 --- a/docs/android/connect.md +++ b/docs/android/connect.md @@ -116,9 +116,10 @@ clawdis nodes invoke --node "" --command canvas.navigate --params Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of `.local`, e.g. `http://:18793/`. This server injects a live-reload client into HTML and reloads on file changes. +The A2UI host lives at `http://:18793/__clawdis__/a2ui/`. Canvas commands (foreground only): -- `canvas.eval`, `canvas.snapshot`, `canvas.navigate` (use `{"url":""}` or `{"url":"/"}` to return to the default canvas/A2UI scaffold). `canvas.snapshot` returns `{ format, base64 }` (default `format="jpeg"`). +- `canvas.eval`, `canvas.snapshot`, `canvas.navigate` (use `{"url":""}` or `{"url":"/"}` to return to the default scaffold). `canvas.snapshot` returns `{ format, base64 }` (default `format="jpeg"`). - A2UI: `canvas.a2ui.push`, `canvas.a2ui.reset` (`canvas.a2ui.pushJSONL` legacy alias) Camera commands (foreground only; permission-gated): diff --git a/docs/ios/connect.md b/docs/ios/connect.md index 8a73f274e..becc6b8f0 100644 --- a/docs/ios/connect.md +++ b/docs/ios/connect.md @@ -136,6 +136,7 @@ clawdis nodes invoke --node "iOS Node" --command canvas.navigate --params '{"url Notes: - The server injects a live-reload client into HTML and reloads on file changes. +- A2UI is hosted on the same canvas host at `http://:18793/__clawdis__/a2ui/`. - Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of `.local`, e.g. `http://:18793/`. - iOS may require App Transport Security allowances to load plain `http://` URLs; if it fails to load, prefer HTTPS or adjust the iOS app’s ATS config. @@ -159,7 +160,7 @@ The response includes `{ format, base64 }` image data (default `format="jpeg"`; ## Common gotchas - **iOS in background:** all `canvas.*` commands fail fast with `NODE_BACKGROUND_UNAVAILABLE` (bring the iOS node app to foreground). -- **Return to default scaffold:** `canvas.navigate` with `{"url":""}` or `{"url":"/"}` returns to the built-in canvas/A2UI scaffold. +- **Return to default scaffold:** `canvas.navigate` with `{"url":""}` or `{"url":"/"}` returns to the built-in scaffold page. - **mDNS blocked:** some networks block multicast; use a different LAN or plan a tailnet-capable bridge (see `docs/discovery.md`). - **Wrong node selector:** `--node` can be the node id (UUID), display name (e.g. `iOS Node`), IP, or an unambiguous prefix. If it’s ambiguous, the CLI will tell you. - **Stale pairing / Keychain cleared:** if the pairing token is missing (or iOS Keychain was wiped), the node must pair again; approve a new pending request. diff --git a/docs/ios/spec.md b/docs/ios/spec.md index bdce1e1a9..d1331c4c2 100644 --- a/docs/ios/spec.md +++ b/docs/ios/spec.md @@ -120,13 +120,14 @@ Add to `src/gateway/protocol/schema.ts` (and regenerate Swift models): ### Node command set (canvas) These are values for `node.invoke.command`: - `canvas.present` / `canvas.hide` -- `canvas.navigate` with `{ url }` (loads a URL; use `""` or `"/"` to return to the default canvas/A2UI scaffold) +- `canvas.navigate` with `{ url }` (loads a URL; use `""` or `"/"` to return to the default scaffold) - `canvas.eval` with `{ javaScript }` - `canvas.snapshot` with `{ maxWidth?, quality?, format? }` - A2UI (mobile + macOS canvas): - `canvas.a2ui.push` with `{ messages: [...] }` (A2UI v0.8 server→client messages) - `canvas.a2ui.pushJSONL` with `{ jsonl: "..." }` (legacy alias) - `canvas.a2ui.reset` + - A2UI is hosted by the Gateway canvas host (`/__clawdis__/a2ui/`); commands fail if the host is unreachable. Result pattern: - Request is a standard `req/res` with `ok` / `error`. diff --git a/docs/mac/canvas.md b/docs/mac/canvas.md index 367d894de..94d347eb0 100644 --- a/docs/mac/canvas.md +++ b/docs/mac/canvas.md @@ -10,7 +10,7 @@ read_when: Status: draft spec · Date: 2025-12-12 -Note: for iOS/Android nodes that should render agent-edited HTML/CSS/JS over the network, prefer the Gateway `canvasHost` (serves `~/clawd/canvas` over LAN/tailnet with live reload). This doc focuses on the macOS in-app canvas panel. See `docs/configuration.md`. +Note: for iOS/Android nodes that should render agent-edited HTML/CSS/JS over the network, prefer the Gateway `canvasHost` (serves `~/clawd/canvas` over LAN/tailnet with live reload). A2UI is also **hosted by the Gateway** over HTTP. This doc focuses on the macOS in-app canvas panel. See `docs/configuration.md`. Clawdis can embed an agent-controlled “visual workspace” panel (“Canvas”) inside the macOS app using `WKWebView`, served via a **custom URL scheme** (no loopback HTTP port required). @@ -41,17 +41,8 @@ Routing model: Directory listings are not served. -When `/` has no `index.html` yet, the handler serves a **built-in A2UI shell** (bundled with the macOS app). -This gives the agent a ready-to-render UI surface without requiring any on-disk HTML. - -If the A2UI shell resources are missing (dev misconfiguration), Canvas falls back to a simple built-in welcome page. - -### Reserved built-in paths - -The scheme handler serves bundled assets under: -- `clawdis-canvas:///__clawdis__/a2ui/...` - -This is reserved for app-owned assets (not session content) and is backed by `Bundle.module` resources. +When `/` has no `index.html` yet, the handler serves a **built-in scaffold page** (bundled with the macOS app). +This is a visual placeholder only (no A2UI renderer). ### Suggested on-disk location @@ -98,14 +89,20 @@ Use the main `clawdis` CLI; it invokes canvas commands via `node.invoke`. - `clawdis canvas present [--node ] [--target <...>] [--x/--y/--width/--height]` - Local targets map into the session directory via the custom scheme (directory targets resolve `index.html|index.htm`). - - If `/` has no index file, Canvas shows the built-in A2UI shell and returns `status: "a2uiShell"`. + - If `/` has no index file, Canvas shows the built-in scaffold page and returns `status: "welcome"`. - `clawdis canvas hide [--node ]` - `clawdis canvas eval --js [--node ]` - `clawdis canvas snapshot [--node ]` ### Canvas A2UI -Canvas includes a built-in **A2UI v0.8** renderer (Lit-based). The agent can drive it with JSONL **server→client protocol messages** (one JSON object per line): +Canvas A2UI is hosted by the **Gateway canvas host** at: + +``` +http(s)://:/__clawdis__/a2ui/ +``` + +The macOS app simply renders that page in the Canvas panel. The agent can drive it with JSONL **server→client protocol messages** (one JSON object per line): - `clawdis canvas a2ui push --jsonl [--node ]` - `clawdis canvas a2ui reset [--node ]` @@ -125,6 +122,7 @@ clawdis canvas a2ui push --jsonl /tmp/a2ui-v0.8.jsonl --node Notes: - This does **not** support the A2UI v0.9 examples using `createSurface`. +- A2UI **fails** if the Gateway canvas host is unreachable (no local fallback). ## Triggering agent runs from Canvas (deep links) diff --git a/docs/refactor/canvas-a2ui.md b/docs/refactor/canvas-a2ui.md index 1311fd803..f9e0d87f8 100644 --- a/docs/refactor/canvas-a2ui.md +++ b/docs/refactor/canvas-a2ui.md @@ -1,48 +1,96 @@ -# Canvas / A2UI +--- +summary: "Refactor: host A2UI from the Gateway (HTTP), remove app-bundled shells" +read_when: + - Refactoring Canvas/A2UI ownership or assets + - Moving UI rendering from native bundles into the Gateway + - Updating node canvas navigation or A2UI command flows +--- + +# Canvas / A2UI — HTTP-hosted from Gateway + +Status: Implemented · Date: 2025-12-20 ## Goal -- A2UI rendering works out-of-the-box (no per-user toggles). -- A2UI button clicks always reach the agent automatically. -- Canvas chrome (close button) stays readable on any content. +- Make the **Gateway (TypeScript)** the single owner of A2UI. +- Remove **app-bundled A2UI shells** (macOS, iOS, Android). +- A2UI renders only when the **Gateway is reachable** (acceptable failure mode). -## Current behavior -- Canvas can show a bundled A2UI shell at `/__clawdis__/a2ui/` when no session `index.html` exists. -- The A2UI shell forwards `a2ui.action` button clicks to native via `WKScriptMessageHandler` (`clawdisCanvasA2UIAction`). -- Native forwards the click to the gateway as an agent invocation. +## Decision +All A2UI HTML/JS assets are **served by the Gateway’s Canvas host** over HTTP. +Nodes (mac/iOS/Android) **navigate to the Gateway URL** before applying A2UI +messages. No local custom-scheme or bundled fallback remains. -## Fixes (2025-12-17) -- Close button: render a small vibrancy/material pill behind the “x” and reduce the button size for less visual weight. -- Click reliability: - - Allow A2UI clicks from any local Canvas path (not just `/` or the built-in A2UI shell). - - Inject an A2UI → native bridge at document start that listens for `a2uiaction` and forwards it: - - Prefer `WKScriptMessageHandler` when available. - - Otherwise fall back to an unattended `clawdis://agent?...&key=...` deep link (no prompt). - - Avoid double-sending actions when the bundled A2UI shell is present (let the shell forward clicks so it can resolve richer context). - - Intercept `clawdis://…` navigations inside the Canvas WKWebView and route them through `DeepLinkHandler` (no NSWorkspace bounce). - - `GatewayConnection` auto-starts the local gateway (and retries briefly) when a request fails in `.local` mode, so Canvas actions don’t silently fail if the gateway isn’t running yet. - - Fix a crash that made `clawdis canvas present`/`eval` look “hung”: - - `VoicePushToTalkHotkey`’s NSEvent monitor could call `@MainActor` code off-main, triggering executor checks / EXC_BAD_ACCESS on macOS 26.2. - - Now it hops back to the main actor before mutating state. - - Preserve in-page state when closing Canvas (hide the window instead of closing the `WKWebView`). - - Fix another “Canvas looks hung” source: node pairing approval used `NSAlert.runModal()` on the main actor, which stalls Canvas/IPC while the alert is open. - - Add UX feedback + better agent prompting: - - Show a small “Sending/Working” spinner when a button is clicked. - - Show “Updated/Failed” toasts (failures include the gateway error string). - - Send a compact, unambiguous agent message that includes machine identity + Canvas context (instead of a big JSON markdown block). - - Native acks the click back into the page via `clawdis:a2ui-action-status` so the UI can switch from “Sending…” to “Working…” immediately. +## Why +- One source of truth (TS) for A2UI rendering. +- Faster iteration (no app release required for A2UI updates). +- iOS/Android/macOS all behave identically. -## Suggested message format (token-efficient) -We want the model to immediately understand: -- This is a **Canvas UI event** (not user chat). -- It happened on **this specific Mac**. -- Default behavior is to **update the Canvas UI** (unless the button context says otherwise). +## New behavior (summary) +1) `canvas.a2ui.*` on any node: + - Ensure Canvas is visible. + - Navigate the node WebView to the Gateway A2UI URL. + - Apply/reset A2UI messages once the page is ready. +2) If Gateway is unreachable: + - A2UI fails with an explicit error (no fallback). -Proposed message line (single-line, parseable): +## Gateway changes + +### Serve A2UI assets +Add A2UI HTML/JS to the Gateway Canvas host, e.g.: ``` -CANVAS_A2UI action= session= surface= component= host= instance= ctx= default=update_canvas +/__clawdis__/a2ui/ -> index.html +/__clawdis__/a2ui/a2ui.bundle.js -> bundled A2UI runtime ``` -## Follow-ups -- Add a small “action sent / failed” debug overlay in the A2UI shell (dev-only) to make failures obvious. -- Decide whether non-local Canvas content should ever be allowed to emit A2UI actions (current stance: no, for safety). +Use the existing Canvas host server (`src/canvas-host/server.ts`) to serve these +assets and inject the action bridge + live reload if desired. + +### Derive HTTP host from WebSocket +Nodes derive the Canvas host URL from the Gateway WS URL: + +``` +ws://host:port -> http://host:port +wss://host:port -> https://host:port +``` + +The Gateway exposes a **canonical** `canvasHostUrl` in hello/bridge payloads +so nodes don’t need to guess. + +## Node changes (mac/iOS/Android) + +### Navigation path +Before applying A2UI: +- Navigate to `${canvasHostUrl}/__clawdis__/a2ui/`. + +### Remove bundled shells +Remove all fallback logic that serves A2UI from local bundles: +- macOS: remove custom-scheme fallback for `/__clawdis__/a2ui/` +- iOS/Android: remove packaged A2UI assets and "default scaffold" assumptions + +### Error behavior +If `canvasHostUrl` is missing or unreachable: +- `canvas.a2ui.push/reset` returns a clear error: + - `A2UI_HOST_UNAVAILABLE` or `A2UI_HOST_NOT_CONFIGURED` + +## Security / transport +- For non-TLS Gateway URLs (http), iOS/Android will need ATS exceptions. +- For TLS (https), prefer WSS + HTTPS with a valid cert. + +## Implementation plan +1) Gateway + - Add A2UI assets under `src/canvas-host/`. + - Serve them at `/__clawdis__/a2ui/` (align with existing naming). + - Expose `canvasHostUrl` in handshake + bridge hello payloads. +2) Node runtimes + - Update `canvas.a2ui.*` to navigate to `canvasHostUrl`. + - Remove custom-scheme A2UI fallback and bundled assets. +3) Tests + - TS: verify `/__clawdis__/a2ui/` responds with HTML + JS. + - Node: verify A2UI fails when host is unreachable and succeeds when reachable. +4) Docs + - Update `docs/mac/canvas.md`, `docs/ios/spec.md`, `docs/android/connect.md` + to remove local fallback assumptions and point to gateway-hosted A2UI. + +## Notes +- iOS/Android may still require ATS exceptions for `http://` canvas hosts.