let5see/clawdbot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(webchat): SwiftUI-only WebChat UI

Browse Source 
# Conflicts:
#	apps/macos/Package.swift
This commit is contained in:
Peter Steinberger
2025-12-17 23:05:28 +01:00
parent ca85d217ec
commit 875cf9a054
7452 changed files with 218086 additions and 776630 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/clawd.md
View File

@@ -70,13 +70,7 @@ clawdis login
clawdis gateway --port 18789
```
3) Start the local WebChat UI (optional, but great for debugging):
```bash
clawdis webchat
```
4) Put a minimal config in `~/.clawdis/clawdis.json`:
3) Put a minimal config in `~/.clawdis/clawdis.json`:
```json5
{

7
docs/gateway.md
View File

@@ -61,10 +61,9 @@ See also: `docs/presence.md` for how presence is produced/deduped and why `insta
- `shutdown` — Gateway is exiting; payload includes `reason` and optional `restartExpectedMs`. Clients should reconnect.
## WebChat integration
- WebChat serves static assets locally (default port 18788, configurable).
- The WebChat backend keeps a single WS connection to the Gateway for control/data; all sends and agent runs flow through that connection.
- Remote use goes through the same SSH/Tailscale tunnel; if a gateway token is configured, WebChat must include it during connect.
- macOS app also connects via this WS (one socket); it hydrates presence from the initial snapshot and listens for `presence` events to update the UI.
- WebChat is a native SwiftUI UI that talks directly to the Gateway WebSocket for history, sends, abort, and events.
- Remote use goes through the same SSH/Tailscale tunnel; if a gateway token is configured, the client includes it during `connect`.
- macOS app connects via a single WS (shared connection); it hydrates presence from the initial snapshot and listens for `presence` events to update the UI.
## Typing and validation
- Server validates every inbound frame with AJV against JSON Schema emitted from the protocol definitions.

5
docs/index.md
View File

@@ -39,7 +39,7 @@ WhatsApp / Telegram
│
├─ Pi agent (RPC)
├─ CLI (clawdis …)
├─ WebChat (loopback UI)
├─ Chat UI (SwiftUI)
├─ macOS app (Clawdis.app)
└─ iOS node (Iris) via Bridge + pairing
```
@@ -82,9 +82,6 @@ clawdis login
# Run the Gateway (leave running)
clawdis gateway --port 18789
# Open the local WebChat UI
clawdis webchat
```
Send a test message (requires a running Gateway):

1
docs/mac/browser.md
View File

@@ -37,7 +37,6 @@ Optional (advanced, can be hidden behind Debug initially):
Clawdis already uses:
- Gateway WebSocket: `18789`
- WebChat HTTP: `18788`
- Bridge (voice/iris): `18790`
For the clawd browser-control server, use “family” ports:

2
docs/mac/canvas.md
View File

@@ -85,7 +85,7 @@ Expose Canvas via the existing `clawdis-mac` → control socket → app routing
- Capture a snapshot image of the current canvas view.
- Optionally set panel placement (screen `x/y` + `width/height`) when showing/navigating.
This should be modeled after `WebChatManager`/`WebChatWindowController` but targeting `clawdis-canvas://…` URLs.
This should be modeled after `WebChatManager`/`WebChatSwiftUIWindowController` but targeting `clawdis-canvas://…` URLs.
Related:
- For “invoke the agent again from UI” flows, prefer the macOS deep link scheme (`clawdis://agent?...`) so *any* UI surface (Canvas, WebChat, native views) can trigger a new agent run. See `docs/clawdis-mac.md`.

7
docs/mac/remote.md
View File

@@ -29,9 +29,8 @@ This flow lets the macOS app act as a full remote control for a Clawdis gateway
4) Health checks and Web Chat will now run through this SSH tunnel automatically.
## Web Chat over SSH
- The mac app serves the WebChat assets locally (from the app bundle) and connects to the gateway over the forwarded WebSocket control port (default 18789).
- The gateway’s own loopback WebChat HTTP server (default 18788, see `webchat.port`) is not required in remote mode.
- Keep the feature enabled in *Settings → Config → Web chat*. Disable it to hide the menu entry entirely.
- Web Chat connects to the gateway over the forwarded WebSocket control port (default 18789).
- There is no separate WebChat HTTP server anymore.
## Permissions
- The remote host needs the same TCC approvals as local (Automation, Accessibility, Screen Recording, Microphone, Speech Recognition, Notifications). Run onboarding on that machine to grant them once.
@@ -44,7 +43,7 @@ This flow lets the macOS app act as a full remote control for a Clawdis gateway
## Troubleshooting
- **exit 127 / not found**: `clawdis` isn’t on PATH for non-login shells. Add it to `/etc/paths`, your shell rc, or symlink into `/usr/local/bin`/`/opt/homebrew/bin`.
- **Health probe failed**: check SSH reachability, PATH, and that Baileys is logged in (`clawdis status --json`).
- **Web Chat stuck**: confirm the gateway is running on the remote host and `webchat.enabled` is true; ensure the forwarded port matches *Settings → Config*. Since RPC is in-process, PATH is no longer a factor.
- **Web Chat stuck**: confirm the gateway is running on the remote host and the forwarded port matches the gateway WS port; the UI requires a healthy WS connection.
- **Voice Wake**: trigger phrases are forwarded automatically in remote mode; no separate forwarder is needed.
## Notification sounds

27
docs/mac/webchat.md
View File

@@ -5,32 +5,23 @@ read_when:
---
# Web Chat (macOS app)
The macOS menu bar app embeds the WebChat UI in a WKWebView and reuses the **primary Clawd session** (`main` by default, configurable via `inbound.session.mainKey`).
The macOS menu bar app shows the WebChat UI as a native SwiftUI view and reuses the **primary Clawd session** (`main` by default, configurable via `inbound.session.mainKey`).
- **Local mode**: loads the gateway’s loopback WebChat HTTP server (default port 18788, see `webchat.port`).
- **Remote mode**: serves the WebChat assets locally from the mac app bundle (via `WebChatServer`) and only forwards the gateway WebSocket control port over SSH.
- **Local mode**: connects directly to the local Gateway WebSocket.
- **Remote mode**: forwards the Gateway WebSocket control port over SSH and uses that as the data plane.
## Launch & debugging
- Manual: Lobster menu → “Open Chat”.
- Auto-open for testing: run `dist/Clawdis.app/Contents/MacOS/Clawdis --webchat` (or pass `--webchat` to the binary launched by launchd). The window opens on startup.
- Inspect: right-click the web view → “Inspect Element” (developerExtras enabled). Console logs go to the Swift logger (subsystem `com.steipete.clawdis`, category `WebChat`). The HTML boot script also writes status text into the `#app` div until the panel mounts.
- WK logs: navigation lifecycle, readyState, js location, and JS errors/unhandled rejections are mirrored to OSLog for easier diagnosis.
- Logs: see `./scripts/clawlog.sh` (subsystem `com.steipete.clawdis`, category `WebChatSwiftUI`).
## How it’s wired
- Assets: `apps/macos/Sources/Clawdis/Resources/WebChat/` contains the `pi-web-ui` dist plus a local import map pointing at bundled vendor modules and a tiny `pi-ai` stub. Everything is served from the static host at `/` (legacy `/webchat/*` still works).
- Bridge: none. The web UI connects directly to the Gateway WebSocket (default 18789) and uses `chat.history`/`chat.send` plus `chat/presence/tick/health` events. No `/rpc` or file-watcher socket path remains.
- Session: always primary; multiple transports (WhatsApp/Telegram/Desktop) share the same session key so context is unified.
- Debug-only: a native SwiftUI “glass” chat UI (same WS transport, attachments + thinking selector) can replace the WKWebView. Enable it via Debug → “Use SwiftUI web chat (glass, gateway WS)” (default off).
- Implementation: `apps/macos/Sources/Clawdis/WebChatSwiftUI.swift` hosts `ClawdisChatUI` and speaks to the Gateway over `GatewayConnection`.
- Data plane: Gateway WebSocket methods `chat.history`, `chat.send`, `chat.abort`; events `chat`, `agent`, `presence`, `tick`, `health`.
- Session: currently always primary (`main`). Session switching UI is intentionally hidden for now.
## Security / surface area
- Loopback server only; remote mode forwards only the gateway WebSocket control port over SSH. CSP is set to `default-src 'self' 'unsafe-inline' data: blob:`.
- Web Inspector is opt-in via right-click; otherwise WKWebView stays in the app sandbox.
- Remote mode forwards only the Gateway WebSocket control port over SSH.
## Known limitations
- Text-only, single-turn (no streaming); tools/attachments not yet plumbed.
- Uses a stubbed pi-ai for model metadata; model selection is fixed to the primary Clawd backend.
## Updating the bundle
1) Ensure `../pi-mono` is present and built (`pnpm install` + `pnpm build` inside `packages/web-ui`).
2) Copy vendor deps into `Resources/WebChat/vendor` (currently synced from `../pi-mono/node_modules`).
3) Rebuild/restart the mac app with `./scripts/restart-mac.sh` so the new assets land in the bundle.
- The UI is optimized for the primary session and typical “chat” usage (not a full browser-based sandbox surface).

14
docs/remote.md
View File

@@ -27,18 +27,12 @@ With the tunnel up:
- `clawdis health` and `clawdis status --deep` now reach the remote gateway via `ws://127.0.0.1:18789`.
- `clawdis gateway {status,health,send,agent,call}` can also target the forwarded URL via `--url` when needed.
## WebChat over SSH
## Chat UI over SSH
Forward both the WebChat HTTP port and the Gateway WS port:
WebChat no longer uses a separate HTTP port. The SwiftUI chat UI connects directly to the Gateway WebSocket.
```bash
ssh -N \
-L 18788:127.0.0.1:18788 \
-L 18789:127.0.0.1:18789 \
user@host
```
Then open `http://127.0.0.1:18788/webchat/` locally. (Details: `docs/webchat.md`.)
- Forward `18789` over SSH (see above), then connect clients to `ws://127.0.0.1:18789`.
- On macOS, prefer the app’s “Remote over SSH” mode, which manages the tunnel automatically.
## macOS app “Remote over SSH”

39
docs/webchat.md
View File

@@ -3,41 +3,32 @@ summary: "Loopback WebChat static host and Gateway WS usage for chat UI"
read_when:
- Debugging or configuring WebChat access
---
# WebChat (loopback + SSH tunnel)
# WebChat (SwiftUI + Gateway WS)
Updated: 2025-12-09
Updated: 2025-12-17
## What it is
- A local web UI for chatting with the Gateway, now WS-only for data.
- Static assets served by the WebChat HTTP server (default port **18788**, configurable).
- The browser/WebView connects directly to the Gateway WebSocket (`ws://127.0.0.1:18789` by default) for history, sends, and events. No file watching or HTTP RPC.
- Trust model: access is granted by being on localhost or inside your SSH/Tailscale tunnel. No additional auth prompts once you can reach the box.
- `webchat.gatewayPort` config can point at a non-default Gateway port if needed.
## Endpoints
- UI is served at the root: `http://127.0.0.1:<port>/` (legacy `/webchat/` still works).
- `GET /` (or `/webchat/*`) → static assets only. No RPC endpoints.
- Data plane is entirely on the Gateway WS (`ws://127.0.0.1:<gatewayPort>`): methods `chat.history`, `chat.send`; events `chat`, `presence`, `tick`, `health`.
- A native SwiftUI chat UI (macOS app / iOS) that talks directly to the Gateway WebSocket.
- No embedded browser/WKWebView and no bundled static WebChat HTTP server.
- Data plane is entirely Gateway WS: methods `chat.history`, `chat.send`, `chat.abort`; events `chat`, `agent`, `presence`, `tick`, `health`.
## How it connects
- Browser/WebView performs Gateway WS `connect`, then calls `chat.history` for bootstrap and `chat.send` for sends; listens to `chat/presence/tick/health` events.
- No session file watching. History comes from the Gateway via `chat.history`.
- The UI performs Gateway WS `connect`, then calls `chat.history` for bootstrap and `chat.send` for sends; it listens to `chat/agent/presence/tick/health` events.
- History comes from the Gateway via `chat.history` (no local session file watching).
- If Gateway WS is unavailable, the UI surfaces the error and blocks send.
## Remote use
- SSH tunnel example: `ssh -N -L 18788:127.0.0.1:18788 -L 18789:127.0.0.1:18789 user@host`.
- Browse to `http://127.0.0.1:18788/webchat/?session=<key>` through the tunnel; the backend WS also rides the tunnel.
- In remote mode, the macOS app forwards the Gateway WebSocket control port via SSH and uses that for the SwiftUI chat UI.
- You generally should not need to manage tunnels manually; see `RemoteTunnelManager` in the app.
## Config
- `webchat.enabled` (default true)
- `webchat.port` (default 18788)
- Gateway WS port is set by `clawdis gateway --port`; WebChat expects it at 18789 unless overridden.
- WebChat does not have a separate HTTP port/config anymore.
- Gateway WS is configured via the app’s gateway endpoint settings (`GatewayEndpointStore`) or `clawdis gateway --port` when running locally.
## Failure handling
- UI errors when the Gateway handshake fails or the WS drops; no HTTP fallback.
- WebChat does not attempt fallback transports; the Gateway WS is required.
- UI errors when the Gateway handshake fails or the WS drops.
- No fallback transport; the Gateway WS is required.
## Dev notes
- Assets live in `apps/macos/Sources/Clawdis/Resources/WebChat`.
- Static host: `src/webchat/server.ts` (loopback-only HTTP).
- macOS glue: `WebChatWindow.swift` + `WebChatTunnel` for SSH -L helpers; WKWebView talks directly to Gateway WS.
- macOS glue: `apps/macos/Sources/Clawdis/WebChatSwiftUI.swift` + `apps/macos/Sources/Clawdis/WebChatManager.swift`.
- Remote tunnel helper: `apps/macos/Sources/Clawdis/RemotePortTunnel.swift`.