diff --git a/Peekaboo b/Peekaboo index 3c5576b27..4807e6fdf 160000 --- a/Peekaboo +++ b/Peekaboo @@ -1 +1 @@ -Subproject commit 3c5576b271be0ccbd1e42947fa4c655a394db8ff +Subproject commit 4807e6fdf016f013dba8a32fa4b2528182539fad diff --git a/docs/clawdis-mac.md b/docs/clawdis-mac.md index 07bcbcc9c..5cf6af67a 100644 --- a/docs/clawdis-mac.md +++ b/docs/clawdis-mac.md @@ -14,7 +14,7 @@ Author: steipete · Status: draft spec · Date: 2025-12-05 - Owns TCC prompts (Notifications, Accessibility, Screen Recording, Automation/AppleScript, Microphone, Speech Recognition). - Brokers privileged actions via local IPC: - Clawdis control socket (app-specific actions like notify/run) - - PeekabooBridge socket (`bridge.sock`) for UI automation (see `docs/mac/peekaboo.md`) + - PeekabooBridge socket (`bridge.sock`) for UI automation brokering (consumed by `peekaboo`; see `docs/mac/peekaboo.md`) - Provides a tiny CLI (`clawdis-mac`) that talks to the app; Node/TS shells out to it. - Replace the separate notifier helper pattern (Oracle) with a built-in notifier. - Offer a first-run experience similar to VibeTunnel’s onboarding (permissions + CLI install). @@ -47,7 +47,7 @@ struct Response { ok: Bool; message?: String; payload?: Data } - The control-socket server rejects oversize/unknown cases and validates the caller by code signature TeamID (with a `DEBUG`-only same-UID escape hatch controlled by `CLAWDIS_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`). UI automation is not part of `ClawdisIPC.Request`: -- `clawdis-mac ui …` speaks **PeekabooBridge** (see `docs/mac/peekaboo.md`). +- UI automation is handled via the separate PeekabooBridge socket and is surfaced by the `peekaboo` CLI (see `docs/mac/peekaboo.md`). ## App UX (Clawdis) - MenuBarExtra icon only (LSUIElement; no Dock). @@ -71,15 +71,7 @@ UI automation is not part of `ClawdisIPC.Request`: - Subcommands (text by default; `--json` for machine output; non-zero exit on failure): - `notify --title --body [--sound] [--priority passive|active|timeSensitive] [--delivery system|overlay|auto]` - `ensure-permissions --cap accessibility --cap screenRecording [--interactive]` - - `ui permissions status` - - `ui frontmost` - - `ui apps` - - `ui windows [--bundle-id ]` - - `ui screenshot [--screen-index ] [--bundle-id ] [--window-index ] [--watch] [--scale native|1x]` - - `ui see [--bundle-id ] [--window-index ] [--snapshot-id ]` - - `ui click --on [--bundle-id ] [--snapshot-id ] [--double|--right]` - - `ui type --text [--into ] [--bundle-id ] [--snapshot-id ] [--clear] [--delay-ms ]` - - `ui wait --on [--bundle-id ] [--snapshot-id ] [--timeout ]` + - UI automation + capture: use `peekaboo …` (Clawdis hosts PeekabooBridge; see `docs/mac/peekaboo.md`) - `run -- cmd args... [--cwd] [--env KEY=VAL] [--timeout 30] [--needs-screen-recording]` - `status` - Sounds: supply any macOS alert name with `--sound` per notification; omit the flag to use the system default. There is no longer a persisted “default sound” in the app UI. @@ -87,14 +79,14 @@ UI automation is not part of `ClawdisIPC.Request`: - Delivery: `overlay` and `auto` show an in-app toast panel (bypasses Notification Center/Focus). - Internals: - For app-specific commands (`notify`, `ensure-permissions`, `run`, `status`): build `ClawdisIPC.Request`, send over the control socket. - - For UI automation (`ui …`): connect to PeekabooBridge hosts (Peekaboo.app → Clawdis.app) and send one JSON request per command (see `docs/mac/peekaboo.md`). + - UI automation is intentionally not exposed via `clawdis-mac`; it lives behind PeekabooBridge and is surfaced by the `peekaboo` CLI. ## Integration with clawdis/Clawdis (Node/TS) - Add helper module that shells to `clawdis-mac`: - Prefer `ensure-permissions` before actions that need TCC. - Use `notify` for desktop toasts; fall back to JS notifier only if CLI missing or platform ≠ macOS. - Use `run` for tasks requiring privileged UI context (screen-recorded terminal runs, etc.). - - For UI automation, `clawdis ui …` is a convenience passthrough to `clawdis-mac ui …` (text by default; add `--json` to the outer `clawdis` command for structured output). + - For UI automation, shell out to `peekaboo …` (text by default; add `--json` for structured output) and rely on PeekabooBridge host selection (Peekaboo.app → Clawdis.app → local). ## Deep links (URL scheme) diff --git a/docs/mac/peekaboo.md b/docs/mac/peekaboo.md index 4fe7e8931..fe865e81b 100644 --- a/docs/mac/peekaboo.md +++ b/docs/mac/peekaboo.md @@ -1,15 +1,15 @@ --- summary: "Plan for integrating Peekaboo automation into Clawdis via PeekabooBridge (socket-based TCC broker)" read_when: - - Adding UI automation commands + - Hosting PeekabooBridge in Clawdis.app - Integrating Peekaboo as a submodule - - Changing clawdis-mac IPC/output formats + - Changing PeekabooBridge protocol/paths --- # Peekaboo Bridge in Clawdis (macOS UI automation broker) ## TL;DR - **Peekaboo removed its XPC helper** and now exposes privileged automation via a **UNIX domain socket bridge** (`PeekabooBridge` / `PeekabooBridgeHost`, socket name `bridge.sock`). -- Clawdis integrates by **hosting the same bridge** inside **Clawdis.app** (optional, user-toggleable), and by making `clawdis-mac ui …` act as a **bridge client**. +- Clawdis integrates by **optionally hosting the same bridge** inside **Clawdis.app** (user-toggleable). The primary client is the **`peekaboo` CLI** (installed via npm); Clawdis does not need its own `ui …` CLI surface. - For **visualizations**, we keep them in **Peekaboo.app** (best UX); Clawdis stays a thin broker host. No visualizer toggle in Clawdis. Non-goals: @@ -31,9 +31,8 @@ Reference (Peekaboo submodule): `docs/bridge-host.md`. - **Peekaboo.app** (preferred; also provides visualizations + controls) - **Clawdis.app** (secondary; “thin host” only) - **Bridge clients** (trigger single actions): - - `clawdis-mac ui …` - - `clawdis ui …` (Node/TS convenience wrapper; shells out to `clawdis-mac ui …`) - - Node/Gateway shells out to `clawdis-mac` + - `peekaboo …` (preferred; humans + agents) + - Optional: Clawdis/Node shells out to `peekaboo` when it needs UI automation/capture ### Host discovery (client-side) Order is deliberate: @@ -68,26 +67,22 @@ What Clawdis should *not* embed: - **XPC**: don’t reintroduce helper targets; use the bridge. ## IPC / CLI surface -### Namespacing -Add new automation commands behind a `ui` prefix: -- `clawdis-mac ui …` for UI automation + visualization-related actions. -- Keep existing top-level commands (`notify`, `run`, `canvas …`, etc.) for compatibility. +### No `clawdis-mac ui …` +We avoid a parallel “Clawdis UI automation CLI”. Instead: +- `peekaboo` is the user/agent-facing CLI surface for automation and capture. +- Clawdis.app can host PeekabooBridge as a **thin TCC broker** so Peekaboo can piggyback on Clawdis permissions when Peekaboo.app isn’t running. -Screenshot cutover: -- Remove legacy screenshot endpoints/commands. -- Ship only `clawdis-mac ui screenshot` (no aliases). +### Diagnostics +Use Peekaboo’s built-in diagnostics to see which host would be used: +- `peekaboo bridge status` +- `peekaboo bridge status --verbose` +- `peekaboo bridge status --json` ### Output format -Change `clawdis-mac` to default to human text output: -- **Default**: plain text; errors are string messages to stderr; exit codes indicate success/failure. -- **`--json`**: structured output (for agents/scripts) with stable schemas. - -This applies globally, not only `ui` commands. - -Note (current state as of 2025-12-13): `clawdis-mac` prints text by default; use `--json` for structured output. +Peekaboo commands default to human text output. Add `--json` for a structured envelope. ### Timeouts -Default timeout for UI actions: **10 seconds** end-to-end. +Default timeout for UI actions: **10 seconds** end-to-end (client enforced; host should also enforce per-operation). ## Coordinate model (multi-display) Requirement: coordinates are **per screen**, not global. @@ -111,18 +106,18 @@ Expose window/app targeting in the UI surface (align with Peekaboo targeting): - by window title substring - by (app, index) -Current `clawdis-mac ui …` support: +Peekaboo CLI targeting (agent-friendly): - `--bundle-id ` for app targeting -- `--window-index ` (0-based) for disambiguating within an app when capturing (see/screenshot) +- `--window-index ` (0-based) for disambiguating within an app when capturing All “see/click/type/scroll/wait” requests should accept a target (default: frontmost). ## “See” + click packs (Playwright-style) Behavior stays aligned with Peekaboo: -- `ui see` returns element IDs (e.g. `B1`, `T3`) with bounds/labels. +- `peekaboo see` returns element IDs (e.g. `B1`, `T3`) with bounds/labels. - Follow-up actions reference those IDs without re-scanning. -`clawdis-mac ui see` should: +`peekaboo see` should: - capture (optionally targeted) window/screen - return a screenshot **file path** (default: temp directory) - return a list of elements (text or JSON) @@ -132,9 +127,9 @@ Snapshot lifecycle requirement: - Snapshot scoping: “implicit snapshot” is **per target bundle id** (reuse last snapshot for that app when snapshot id is omitted). Practical flow (agent-friendly): -- `clawdis-mac ui frontmost` returns the focused app (bundle id) + focused window (title/id) so follow-up calls can pass `--bundle-id …`. -- `clawdis-mac ui see --bundle-id X` updates the implicit snapshot for `X`. -- `clawdis-mac ui click --bundle-id X --on B1` reuses the most recent snapshot for `X` when `--snapshot-id` is omitted. +- `peekaboo list apps` / `peekaboo list windows` provide bundle-id context for targeting. +- `peekaboo see --bundle-id X` updates the implicit snapshot for `X`. +- `peekaboo click --bundle-id X --on B1` reuses the most recent snapshot for `X` when `--snapshot-id` is omitted. ## Visualizer integration Keep visualizations in **Peekaboo.app** for now. @@ -142,11 +137,11 @@ Keep visualizations in **Peekaboo.app** for now. - Any “visualizer enabled/disabled” setting is controlled in Peekaboo.app. ## Screenshots (legacy → Peekaboo takeover) -Clawdis uses `clawdis-mac ui screenshot` and returns a file path (default location: temp directory) instead of raw image bytes. +Clawdis should not grow a separate screenshot CLI surface. Migration plan: -- Bridge host performs capture and returns a temp file path. -- No legacy aliases; make the old screenshot surface disappear cleanly. +- Use `peekaboo capture …` / `peekaboo see …` (returns a file path, default temp directory). +- Once Clawdis’ legacy screenshot plumbing is replaced, remove it cleanly (no aliases). ## Permissions behavior If required permissions are missing: @@ -164,22 +159,9 @@ Debug-only escape hatch (development convenience): - “allow same-UID callers” means: *skip codesign checks for clients running under the same Unix user*. - This must be **opt-in**, **DEBUG-only**, and guarded by an env var (Peekaboo uses `PEEKABOO_ALLOW_UNSIGNED_SOCKET_CLIENTS=1`). -## Current `clawdis-mac ui` commands (Dec 2025) -All commands default to text output. Add `--json` right after `clawdis-mac` for a structured envelope. - -- `clawdis-mac ui permissions status` -- `clawdis-mac ui frontmost` -- `clawdis-mac ui apps` -- `clawdis-mac ui windows [--bundle-id ]` -- `clawdis-mac ui screenshot [--screen-index ] [--bundle-id ] [--window-index ] [--watch] [--scale native|1x]` -- `clawdis-mac ui see [--bundle-id ] [--window-index ] [--snapshot-id ]` -- `clawdis-mac ui click --on [--bundle-id ] [--snapshot-id ] [--double|--right]` -- `clawdis-mac ui type --text [--into ] [--bundle-id ] [--snapshot-id ] [--clear] [--delay-ms ]` -- `clawdis-mac ui wait --on [--bundle-id ] [--snapshot-id ] [--timeout ]` - ## Next integration steps (after this doc) 1. Add Peekaboo as a git submodule (nested submodules OK). -2. Add a small `clawdis-mac ui …` surface that speaks PeekabooBridge (text by default, `--json` for structured). -3. Host `PeekabooBridgeHost` inside Clawdis.app behind a single setting (“Enable Peekaboo Bridge”, default on). -4. Implement the minimum operation set needed for agents (see/click/type/scroll/wait/screenshot, plus list apps/windows/screens). +2. Host `PeekabooBridgeHost` inside Clawdis.app behind a single setting (“Enable Peekaboo Bridge”, default on). +3. Ensure Clawdis hosts the bridge at `~/Library/Application Support/clawdis/bridge.sock` and speaks the PeekabooBridge JSON protocol. +4. Validate with `peekaboo bridge status --verbose` that Peekaboo can select Clawdis as the fallback host (no auto-launch). 5. Keep all protocol decisions aligned with Peekaboo (coordinate system, element IDs, snapshot scoping, error envelopes).