let5see/clawdbot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(canvas): remove setMode; host A2UI in scaffold

Browse Source
This commit is contained in:
Peter Steinberger
2025-12-18 13:18:24 +01:00
parent dda6d7f9e1
commit 790079c3b6
12 changed files with 174 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/android/connect.md
View File

@@ -30,7 +30,7 @@ pnpm clawdis gateway --port 18789 --verbose
```
Confirm in logs you see something like:
- `bridge listening on tcp://0.0.0.0:18790 (Iris)`
- `bridge listening on tcp://0.0.0.0:18790 (node)`
For tailnet-only setups (recommended for Vienna ⇄ London), bind the bridge to the gateway machine’s Tailscale IP instead:
@@ -103,28 +103,22 @@ The Android node’s Chat sheet uses the gateway’s **primary session key** (`m
### Gateway Canvas Host (recommended for web content)
If you want the node to show real HTML/CSS/JS that the agent can edit on disk, enable the Gateway canvas host and point the node at it.
If you want the node to show real HTML/CSS/JS that the agent can edit on disk, point the node at the Gateway canvas host.
1) On the gateway host, enable `canvasHost` in `~/.clawdis/clawdis.json`:
1) Create `~/clawd/canvas/index.html` on the gateway host.
```json5
{
canvasHost: { enabled: true, root: "~/clawd/canvas", port: 18793, bind: "lan" }
}
```
2) Create `~/clawd/canvas/index.html`.
3) Navigate the node to it (LAN):
2) Navigate the node to it (LAN):
```bash
clawdis nodes invoke --node "<Android Node>" --command canvas.navigate --params '{"url":"http://<gateway-hostname>.local:18793/"}'
```
Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of `.local`, e.g. `http://<gateway-magicdns>:18793/`.
This server injects a live-reload client into HTML and reloads on file changes.
Canvas commands (foreground only):
- `canvas.eval`, `canvas.snapshot`, `canvas.navigate` (switches to web mode), `canvas.setMode` (use `"canvas"` to return)
- `canvas.eval`, `canvas.snapshot`, `canvas.navigate` (use `{"url":""}` or `{"url":"/"}` to return to the default canvas/A2UI scaffold)
- A2UI: `canvas.a2ui.push`, `canvas.a2ui.reset` (`canvas.a2ui.pushJSONL` legacy alias)
Camera commands (foreground only; permission-gated):

50
docs/ios/connect.md
View File

@@ -1,23 +1,23 @@
---
summary: "Runbook: connect/pair the iOS node (Iris) to a Clawdis Gateway and drive its Canvas"
summary: "Runbook: connect/pair the iOS node to a Clawdis Gateway and drive its Canvas"
read_when:
- Pairing or reconnecting the iOS node
- Debugging iOS bridge discovery or auth
- Sending screen/canvas commands to iOS
---
# iOS Node Connection Runbook (Iris)
# iOS Node Connection Runbook
This is the practical “how do I connect Iris” guide:
This is the practical “how do I connect the iOS node” guide:
**iOS app** ⇄ (Bonjour + TCP bridge) ⇄ **Gateway bridge** ⇄ (loopback WS) ⇄ **Gateway**
The Gateway WebSocket stays loopback-only (`ws://127.0.0.1:18789`). Iris talks to the LAN-facing **bridge** (default `tcp://0.0.0.0:18790`) and uses Gateway-owned pairing.
The Gateway WebSocket stays loopback-only (`ws://127.0.0.1:18789`). The iOS node talks to the LAN-facing **bridge** (default `tcp://0.0.0.0:18790`) and uses Gateway-owned pairing.
## Prerequisites
- You can run the Gateway on the “master” machine.
- Iris (iOS app) can reach the gateway bridge:
- iOS node app can reach the gateway bridge:
- Same LAN with Bonjour/mDNS, **or**
- Same Tailscale tailnet using Wide-Area Bonjour / unicast DNS-SD (see below), **or**
- Manual bridge host/port (fallback)
@@ -32,7 +32,7 @@ pnpm clawdis gateway --port 18789 --verbose
```
Confirm in logs you see something like:
- `bridge listening on tcp://0.0.0.0:18790 (Iris)`
- `bridge listening on tcp://0.0.0.0:18790 (node)`
For tailnet-only setups (recommended for Vienna ⇄ London), bind the bridge to the gateway machine’s Tailscale IP instead:
@@ -49,7 +49,7 @@ dns-sd -B _clawdis-bridge._tcp local.
You should see your gateway advertising `_clawdis-bridge._tcp`.
If browse works, but Iris can’t connect, try resolving one instance:
If browse works, but the iOS node can’t connect, try resolving one instance:
```bash
dns-sd -L "<instance name>" _clawdis-bridge._tcp local.
@@ -59,19 +59,19 @@ More debugging notes: `docs/bonjour.md`.
### Tailnet (Vienna ⇄ London) discovery via unicast DNS-SD
If Iris and the gateway are on different networks but connected via Tailscale, multicast mDNS won’t cross the boundary. Use Wide-Area Bonjour / unicast DNS-SD instead:
If the iOS node and the gateway are on different networks but connected via Tailscale, multicast mDNS won’t cross the boundary. Use Wide-Area Bonjour / unicast DNS-SD instead:
1) Set up a DNS-SD zone (example `clawdis.internal.`) on the gateway host and publish `_clawdis-bridge._tcp` records.
2) Configure Tailscale split DNS for `clawdis.internal` pointing at that DNS server.
Details and example CoreDNS config: `docs/bonjour.md`.
## 3) Connect from Iris (iOS)
## 3) Connect from the iOS node app
In Iris:
In the iOS node app:
- Pick the discovered bridge (or hit refresh).
- If not paired yet, Iris will initiate pairing automatically.
- After the first successful pairing, Iris will auto-reconnect **strictly to the last discovered gateway** on launch (including after reinstall), as long as the iOS Keychain entry is still present.
- If not paired yet, it will initiate pairing automatically.
- After the first successful pairing, it will auto-reconnect **strictly to the last discovered gateway** on launch (including after reinstall), as long as the iOS Keychain entry is still present.
### Connection indicator (always visible)
@@ -94,7 +94,7 @@ Approve the request:
clawdis nodes approve <requestId>
```
After approval, Iris receives/stores the token and reconnects authenticated.
After approval, the iOS node receives/stores the token and reconnects authenticated.
Pairing details: `docs/gateway/pairing.md`.
@@ -117,26 +117,18 @@ Pairing details: `docs/gateway/pairing.md`.
## 6) Drive the iOS Canvas (draw / snapshot)
Iris runs a WKWebView “Canvas” scaffold which exposes:
The iOS node runs a WKWebView “Canvas” scaffold which exposes:
- `window.__clawdis.canvas`
- `window.__clawdis.ctx` (2D context)
- `window.__clawdis.setStatus(title, subtitle)`
### Gateway Canvas Host (recommended for web content)
If you want Iris to show real HTML/CSS/JS that the agent can edit on disk, enable the Gateway canvas host and point Iris at it.
If you want the node to show real HTML/CSS/JS that the agent can edit on disk, point it at the Gateway canvas host.
1) On the gateway host, enable `canvasHost` in `~/.clawdis/clawdis.json`:
1) Create `~/clawd/canvas/index.html` on the gateway host.
```json5
{
canvasHost: { enabled: true, root: "~/clawd/canvas", port: 18793, bind: "lan" }
}
```
2) Create `~/clawd/canvas/index.html`.
3) Navigate Iris to it (LAN):
2) Navigate the node to it (LAN):
```bash
clawdis nodes invoke --node "iOS Node" --command canvas.navigate --params '{"url":"http://<gateway-hostname>.local:18793/"}'
@@ -144,6 +136,7 @@ clawdis nodes invoke --node "iOS Node" --command canvas.navigate --params '{"url
Notes:
- The server injects a live-reload client into HTML and reloads on file changes.
- Tailnet (optional): if both devices are on Tailscale, use a MagicDNS name or tailnet IP instead of `.local`, e.g. `http://<gateway-magicdns>:18793/`.
- iOS may require App Transport Security allowances to load plain `http://` URLs; if it fails to load, prefer HTTPS or adjust the iOS app’s ATS config.
### Draw with `canvas.eval`
@@ -165,11 +158,12 @@ The response includes `base64` PNG data (for debugging/verification).
## Common gotchas
- **iOS in background:** all `canvas.*` commands fail fast with `NODE_BACKGROUND_UNAVAILABLE` (bring Iris to foreground).
- **iOS in background:** all `canvas.*` commands fail fast with `NODE_BACKGROUND_UNAVAILABLE` (bring the iOS node app to foreground).
- **Return to default scaffold:** `canvas.navigate` with `{"url":""}` or `{"url":"/"}` returns to the built-in canvas/A2UI scaffold.
- **mDNS blocked:** some networks block multicast; use a different LAN or plan a tailnet-capable bridge (see `docs/discovery.md`).
- **Wrong node selector:** `--node` can be the node id (UUID), display name (e.g. `iOS Node`), IP, or an unambiguous prefix. If it’s ambiguous, the CLI will tell you.
- **Stale pairing / Keychain cleared:** if the pairing token is missing (or iOS Keychain was wiped), Iris must pair again; approve a new pending request.
- **App reinstall but no reconnect:** Iris restores `instanceId` + last bridge preference from Keychain; if it still comes up “unpaired”, verify Keychain persistence on your device/simulator and re-pair once.
- **Stale pairing / Keychain cleared:** if the pairing token is missing (or iOS Keychain was wiped), the node must pair again; approve a new pending request.
- **App reinstall but no reconnect:** the node restores `instanceId` + last bridge preference from Keychain; if it still comes up “unpaired”, verify Keychain persistence on your device/simulator and re-pair once.
## Related docs

3
docs/ios/spec.md
View File

@@ -120,10 +120,9 @@ Add to `src/gateway/protocol/schema.ts` (and regenerate Swift models):
### Node command set (canvas)
These are values for `node.invoke.command`:
- `canvas.show` / `canvas.hide`
- `canvas.navigate` with `{ url }` (Canvas URL or https URL; switches mode to `"web"`)
- `canvas.navigate` with `{ url }` (loads a URL; use `""` or `"/"` to return to the default canvas/A2UI scaffold)
- `canvas.eval` with `{ javaScript }`
- `canvas.snapshot` with `{ maxWidth?, quality?, format? }`
- `canvas.setMode` with `{ mode: "canvas" | "web" }` (use `"canvas"` to return to the scaffold)
- A2UI (mobile + macOS canvas):
- `canvas.a2ui.push` with `{ messages: [...] }` (A2UI v0.8 server→client messages)
- `canvas.a2ui.pushJSONL` with `{ jsonl: "..." }` (legacy alias)