Docs: document canvas.* node.invoke commands

This commit is contained in:
Peter Steinberger
2025-12-18 01:17:27 +00:00
parent 21a27e3b65
commit 272015c701
5 changed files with 16 additions and 19 deletions

View File

@@ -31,7 +31,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
### iOS node (Iris) ### iOS node (Iris)
- New iOS companion app that pairs to the Gateway bridge, reports presence as a node, and exposes a WKWebView “Canvas” for agent-driven UI. - New iOS companion app that pairs to the Gateway bridge, reports presence as a node, and exposes a WKWebView “Canvas” for agent-driven UI.
- `clawdis nodes invoke` supports `screen.eval` and `screen.snapshot` to drive and verify the iOS Canvas (fails fast when Iris is backgrounded). - `clawdis nodes invoke` supports `canvas.eval` and `canvas.snapshot` to drive and verify the iOS Canvas (fails fast when Iris is backgrounded).
- Voice wake words are configurable in-app; Iris reconnects to the last bridge when credentials are still present in Keychain. - Voice wake words are configurable in-app; Iris reconnects to the last bridge when credentials are still present in Keychain.
### WhatsApp & agent experience ### WhatsApp & agent experience

View File

@@ -123,7 +123,7 @@ Build/run the mac app with `./scripts/restart-mac.sh` (packages, installs, and l
Iris is an internal/prototype iOS app that connects as a **remote node**: Iris is an internal/prototype iOS app that connects as a **remote node**:
- **Voice trigger:** forwards transcripts into the Gateway (agent runs + wakeups). - **Voice trigger:** forwards transcripts into the Gateway (agent runs + wakeups).
- **Canvas screen:** a WKWebView + `<canvas>` surface the agent can control (via `screen.eval` / `screen.snapshot` over `node.invoke`). - **Canvas screen:** a WKWebView + `<canvas>` surface the agent can control (via `canvas.eval` / `canvas.snapshot` over `node.invoke`).
- **Discovery + pairing:** finds the bridge via Bonjour (`_clawdis-bridge._tcp`) and uses Gateway-owned pairing (`clawdis nodes pending|approve`); `clawdis nodes status` shows paired nodes + capabilities. - **Discovery + pairing:** finds the bridge via Bonjour (`_clawdis-bridge._tcp`) and uses Gateway-owned pairing (`clawdis nodes pending|approve`); `clawdis nodes status` shows paired nodes + capabilities.
Runbook: `docs/ios/connect.md` Runbook: `docs/ios/connect.md`

View File

@@ -102,7 +102,7 @@ The Android nodes Chat sheet uses the gateways **primary session key** (`m
## 7) Canvas + camera ## 7) Canvas + camera
Canvas commands (foreground only): Canvas commands (foreground only):
- `screen.eval`, `screen.snapshot`, `screen.navigate`, `screen.setMode` - `canvas.eval`, `canvas.snapshot`, `canvas.navigate`, `canvas.setMode`
Camera commands (foreground only; permission-gated): Camera commands (foreground only; permission-gated):
- `camera.snap` (jpg) - `camera.snap` (jpg)

View File

@@ -122,19 +122,19 @@ Iris runs a WKWebView “Canvas” scaffold which exposes:
- `window.__clawdis.ctx` (2D context) - `window.__clawdis.ctx` (2D context)
- `window.__clawdis.setStatus(title, subtitle)` - `window.__clawdis.setStatus(title, subtitle)`
### Draw with `screen.eval` ### Draw with `canvas.eval`
```bash ```bash
clawdis nodes invoke --node "iOS Node" --command screen.eval --params "$(cat <<'JSON' clawdis nodes invoke --node "iOS Node" --command canvas.eval --params "$(cat <<'JSON'
{"javaScript":"(() => { const {ctx,setStatus} = window.__clawdis; setStatus('Drawing','…'); ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle='#ff2d55'; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); setStatus(null,null); return 'ok'; })()"} {"javaScript":"(() => { const {ctx,setStatus} = window.__clawdis; setStatus('Drawing','…'); ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle='#ff2d55'; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); setStatus(null,null); return 'ok'; })()"}
JSON JSON
)" )"
``` ```
### Snapshot with `screen.snapshot` ### Snapshot with `canvas.snapshot`
```bash ```bash
clawdis nodes invoke --node 192.168.0.88 --command screen.snapshot --params '{"maxWidth":900}' clawdis nodes invoke --node 192.168.0.88 --command canvas.snapshot --params '{"maxWidth":900}'
``` ```
The response includes `base64` PNG data (for debugging/verification). The response includes `base64` PNG data (for debugging/verification).

View File

@@ -117,14 +117,11 @@ Add to `src/gateway/protocol/schema.ts` (and regenerate Swift models):
### Node command set (screen-focused) ### Node command set (screen-focused)
These are values for `node.invoke.command`: These are values for `node.invoke.command`:
- `screen.show` / `screen.hide` - `canvas.show` / `canvas.hide`
- `screen.navigate` with `{ url }` (Canvas URL or https URL) - `canvas.navigate` with `{ url }` (Canvas URL or https URL)
- `screen.eval` with `{ javaScript }` - `canvas.eval` with `{ javaScript }`
- `screen.snapshot` with `{ maxWidth?, quality?, format? }` - `canvas.snapshot` with `{ maxWidth?, quality?, format? }`
- `screen.setMode` with `{ mode: "canvas" | "web" }` - `canvas.setMode` with `{ mode: "canvas" | "web" }`
Alias:
- `canvas.*` is accepted as a synonym for `screen.*` (e.g. `canvas.eval``screen.eval`).
Result pattern: Result pattern:
- Request is a standard `req/res` with `ok` / `error`. - Request is a standard `req/res` with `ok` / `error`.
@@ -135,13 +132,13 @@ As of 2025-12-13, the Gateway supports `node.invoke` for bridge-connected nodes.
Example: draw a diagonal line on the iOS Canvas: Example: draw a diagonal line on the iOS Canvas:
```bash ```bash
clawdis nodes invoke --node ios-node --command screen.eval --params '{"javaScript":"(() => { const {ctx} = window.__clawdis; ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle=\"#ff2d55\"; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); return \"ok\"; })()"}' clawdis nodes invoke --node ios-node --command canvas.eval --params '{"javaScript":"(() => { const {ctx} = window.__clawdis; ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle=\"#ff2d55\"; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); return \"ok\"; })()"}'
``` ```
### Background behavior requirement ### Background behavior requirement
When iOS is backgrounded: When iOS is backgrounded:
- Voice may still be active (subject to iOS suspension). - Voice may still be active (subject to iOS suspension).
- **All `screen.*` commands must fail** with a stable error code, e.g.: - **All `canvas.*` commands must fail** with a stable error code, e.g.:
- `NODE_BACKGROUND_UNAVAILABLE` - `NODE_BACKGROUND_UNAVAILABLE`
- Include `retryable: true` and `retryAfterMs` if we want the agent to wait. - Include `retryable: true` and `retryAfterMs` if we want the agent to wait.
@@ -222,8 +219,8 @@ open Clawdis.xcodeproj
- Implement bridge routing + ACLs - Implement bridge routing + ACLs
4) **iOS screen/canvas** 4) **iOS screen/canvas**
- WKWebView screen surface - WKWebView screen surface
- `screen.navigate/eval/snapshot` - `canvas.navigate/eval/snapshot`
- Background fast-fail for `screen.*` - Background fast-fail for `canvas.*`
5) **Unify mac Canvas under the same node.invoke** 5) **Unify mac Canvas under the same node.invoke**
- Keep existing implementation, but expose it through the unified protocol path so the agent uses one API. - Keep existing implementation, but expose it through the unified protocol path so the agent uses one API.