Docs: document canvas.* node.invoke commands
This commit is contained in:
@@ -31,7 +31,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
|
|||||||
|
|
||||||
### iOS node (Iris)
|
### iOS node (Iris)
|
||||||
- New iOS companion app that pairs to the Gateway bridge, reports presence as a node, and exposes a WKWebView “Canvas” for agent-driven UI.
|
- New iOS companion app that pairs to the Gateway bridge, reports presence as a node, and exposes a WKWebView “Canvas” for agent-driven UI.
|
||||||
- `clawdis nodes invoke` supports `screen.eval` and `screen.snapshot` to drive and verify the iOS Canvas (fails fast when Iris is backgrounded).
|
- `clawdis nodes invoke` supports `canvas.eval` and `canvas.snapshot` to drive and verify the iOS Canvas (fails fast when Iris is backgrounded).
|
||||||
- Voice wake words are configurable in-app; Iris reconnects to the last bridge when credentials are still present in Keychain.
|
- Voice wake words are configurable in-app; Iris reconnects to the last bridge when credentials are still present in Keychain.
|
||||||
|
|
||||||
### WhatsApp & agent experience
|
### WhatsApp & agent experience
|
||||||
|
|||||||
@@ -123,7 +123,7 @@ Build/run the mac app with `./scripts/restart-mac.sh` (packages, installs, and l
|
|||||||
Iris is an internal/prototype iOS app that connects as a **remote node**:
|
Iris is an internal/prototype iOS app that connects as a **remote node**:
|
||||||
|
|
||||||
- **Voice trigger:** forwards transcripts into the Gateway (agent runs + wakeups).
|
- **Voice trigger:** forwards transcripts into the Gateway (agent runs + wakeups).
|
||||||
- **Canvas screen:** a WKWebView + `<canvas>` surface the agent can control (via `screen.eval` / `screen.snapshot` over `node.invoke`).
|
- **Canvas screen:** a WKWebView + `<canvas>` surface the agent can control (via `canvas.eval` / `canvas.snapshot` over `node.invoke`).
|
||||||
- **Discovery + pairing:** finds the bridge via Bonjour (`_clawdis-bridge._tcp`) and uses Gateway-owned pairing (`clawdis nodes pending|approve`); `clawdis nodes status` shows paired nodes + capabilities.
|
- **Discovery + pairing:** finds the bridge via Bonjour (`_clawdis-bridge._tcp`) and uses Gateway-owned pairing (`clawdis nodes pending|approve`); `clawdis nodes status` shows paired nodes + capabilities.
|
||||||
|
|
||||||
Runbook: `docs/ios/connect.md`
|
Runbook: `docs/ios/connect.md`
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ The Android node’s Chat sheet uses the gateway’s **primary session key** (`m
|
|||||||
## 7) Canvas + camera
|
## 7) Canvas + camera
|
||||||
|
|
||||||
Canvas commands (foreground only):
|
Canvas commands (foreground only):
|
||||||
- `screen.eval`, `screen.snapshot`, `screen.navigate`, `screen.setMode`
|
- `canvas.eval`, `canvas.snapshot`, `canvas.navigate`, `canvas.setMode`
|
||||||
|
|
||||||
Camera commands (foreground only; permission-gated):
|
Camera commands (foreground only; permission-gated):
|
||||||
- `camera.snap` (jpg)
|
- `camera.snap` (jpg)
|
||||||
|
|||||||
@@ -122,19 +122,19 @@ Iris runs a WKWebView “Canvas” scaffold which exposes:
|
|||||||
- `window.__clawdis.ctx` (2D context)
|
- `window.__clawdis.ctx` (2D context)
|
||||||
- `window.__clawdis.setStatus(title, subtitle)`
|
- `window.__clawdis.setStatus(title, subtitle)`
|
||||||
|
|
||||||
### Draw with `screen.eval`
|
### Draw with `canvas.eval`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
clawdis nodes invoke --node "iOS Node" --command screen.eval --params "$(cat <<'JSON'
|
clawdis nodes invoke --node "iOS Node" --command canvas.eval --params "$(cat <<'JSON'
|
||||||
{"javaScript":"(() => { const {ctx,setStatus} = window.__clawdis; setStatus('Drawing','…'); ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle='#ff2d55'; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); setStatus(null,null); return 'ok'; })()"}
|
{"javaScript":"(() => { const {ctx,setStatus} = window.__clawdis; setStatus('Drawing','…'); ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle='#ff2d55'; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); setStatus(null,null); return 'ok'; })()"}
|
||||||
JSON
|
JSON
|
||||||
)"
|
)"
|
||||||
```
|
```
|
||||||
|
|
||||||
### Snapshot with `screen.snapshot`
|
### Snapshot with `canvas.snapshot`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
clawdis nodes invoke --node 192.168.0.88 --command screen.snapshot --params '{"maxWidth":900}'
|
clawdis nodes invoke --node 192.168.0.88 --command canvas.snapshot --params '{"maxWidth":900}'
|
||||||
```
|
```
|
||||||
|
|
||||||
The response includes `base64` PNG data (for debugging/verification).
|
The response includes `base64` PNG data (for debugging/verification).
|
||||||
|
|||||||
@@ -117,14 +117,11 @@ Add to `src/gateway/protocol/schema.ts` (and regenerate Swift models):
|
|||||||
|
|
||||||
### Node command set (screen-focused)
|
### Node command set (screen-focused)
|
||||||
These are values for `node.invoke.command`:
|
These are values for `node.invoke.command`:
|
||||||
- `screen.show` / `screen.hide`
|
- `canvas.show` / `canvas.hide`
|
||||||
- `screen.navigate` with `{ url }` (Canvas URL or https URL)
|
- `canvas.navigate` with `{ url }` (Canvas URL or https URL)
|
||||||
- `screen.eval` with `{ javaScript }`
|
- `canvas.eval` with `{ javaScript }`
|
||||||
- `screen.snapshot` with `{ maxWidth?, quality?, format? }`
|
- `canvas.snapshot` with `{ maxWidth?, quality?, format? }`
|
||||||
- `screen.setMode` with `{ mode: "canvas" | "web" }`
|
- `canvas.setMode` with `{ mode: "canvas" | "web" }`
|
||||||
|
|
||||||
Alias:
|
|
||||||
- `canvas.*` is accepted as a synonym for `screen.*` (e.g. `canvas.eval` → `screen.eval`).
|
|
||||||
|
|
||||||
Result pattern:
|
Result pattern:
|
||||||
- Request is a standard `req/res` with `ok` / `error`.
|
- Request is a standard `req/res` with `ok` / `error`.
|
||||||
@@ -135,13 +132,13 @@ As of 2025-12-13, the Gateway supports `node.invoke` for bridge-connected nodes.
|
|||||||
|
|
||||||
Example: draw a diagonal line on the iOS Canvas:
|
Example: draw a diagonal line on the iOS Canvas:
|
||||||
```bash
|
```bash
|
||||||
clawdis nodes invoke --node ios-node --command screen.eval --params '{"javaScript":"(() => { const {ctx} = window.__clawdis; ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle=\"#ff2d55\"; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); return \"ok\"; })()"}'
|
clawdis nodes invoke --node ios-node --command canvas.eval --params '{"javaScript":"(() => { const {ctx} = window.__clawdis; ctx.clearRect(0,0,innerWidth,innerHeight); ctx.lineWidth=6; ctx.strokeStyle=\"#ff2d55\"; ctx.beginPath(); ctx.moveTo(40,40); ctx.lineTo(innerWidth-40, innerHeight-40); ctx.stroke(); return \"ok\"; })()"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
### Background behavior requirement
|
### Background behavior requirement
|
||||||
When iOS is backgrounded:
|
When iOS is backgrounded:
|
||||||
- Voice may still be active (subject to iOS suspension).
|
- Voice may still be active (subject to iOS suspension).
|
||||||
- **All `screen.*` commands must fail** with a stable error code, e.g.:
|
- **All `canvas.*` commands must fail** with a stable error code, e.g.:
|
||||||
- `NODE_BACKGROUND_UNAVAILABLE`
|
- `NODE_BACKGROUND_UNAVAILABLE`
|
||||||
- Include `retryable: true` and `retryAfterMs` if we want the agent to wait.
|
- Include `retryable: true` and `retryAfterMs` if we want the agent to wait.
|
||||||
|
|
||||||
@@ -222,8 +219,8 @@ open Clawdis.xcodeproj
|
|||||||
- Implement bridge routing + ACLs
|
- Implement bridge routing + ACLs
|
||||||
4) **iOS screen/canvas**
|
4) **iOS screen/canvas**
|
||||||
- WKWebView screen surface
|
- WKWebView screen surface
|
||||||
- `screen.navigate/eval/snapshot`
|
- `canvas.navigate/eval/snapshot`
|
||||||
- Background fast-fail for `screen.*`
|
- Background fast-fail for `canvas.*`
|
||||||
5) **Unify mac Canvas under the same node.invoke**
|
5) **Unify mac Canvas under the same node.invoke**
|
||||||
- Keep existing implementation, but expose it through the unified protocol path so the agent uses one API.
|
- Keep existing implementation, but expose it through the unified protocol path so the agent uses one API.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user