diff --git a/CHANGELOG.md b/CHANGELOG.md index 53a570538..821f72215 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ - CLI: remove `update`, `gateway-daemon`, `gateway {install|uninstall|start|stop|restart|daemon status|wake|send|agent}`, and `telegram` commands; move `login/logout` to `providers login/logout` (top-level aliases hidden); use `daemon` for service control, `send`/`agent`/`wake` for RPC, and `nodes canvas` for canvas ops. ### Fixes +- CLI/Daemon: add `clawdbot logs` tailing and improve restart/service hints across platforms. - Auto-reply: keep typing indicators alive during tool execution without changing typing-mode semantics. Thanks @thesash for PR #452. - macOS: harden Voice Wake tester/runtime (pause trigger, mic persistence, local-only tester) and keep transcript logs private. Thanks @xadenryan for PR #438. - macOS: preserve node bridge tunnel port override so remote nodes connect on the bridge port. Thanks @sircrumpet for PR #364. diff --git a/docs/cli/index.md b/docs/cli/index.md index 5b95bf1b1..3196b6ab2 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -423,6 +423,15 @@ Notes: - `daemon install` defaults to Node runtime; use `--runtime bun` only when WhatsApp is disabled. - `daemon install` options: `--port`, `--runtime`, `--token`. +### `logs` +Tail Gateway file logs via RPC. + +Examples: +```bash +clawdbot logs --follow +clawdbot logs --limit 200 +``` + ### `gateway ` Gateway RPC helpers (use `--url`, `--token`, `--password`, `--timeout`, `--expect-final` for each). diff --git a/docs/gateway/index.md b/docs/gateway/index.md index 2a0a73121..5a969bc70 100644 --- a/docs/gateway/index.md +++ b/docs/gateway/index.md @@ -166,12 +166,14 @@ clawdbot daemon status clawdbot daemon install clawdbot daemon stop clawdbot daemon restart +clawdbot logs --follow ``` Notes: - `daemon status` probes the Gateway RPC by default (same URL/token defaults as `gateway status`). - `daemon status --deep` adds system-level scans (LaunchDaemons/system units). - `daemon status` now reports runtime state (PID/exit status) and port collisions when the gateway isn’t reachable. +- `logs` tails the Gateway file log via RPC (no manual `tail`/`grep` needed). - If other gateway-like services are detected, the CLI warns. We recommend **one gateway per machine**; one gateway can host multiple agents. - Cleanup: `clawdbot daemon uninstall` (current service) and `clawdbot doctor` (legacy migrations). @@ -179,6 +181,7 @@ Bundled mac app: - Clawdbot.app can bundle a bun-compiled gateway binary and install a per-user LaunchAgent labeled `com.clawdbot.gateway`. - To stop it cleanly, use `clawdbot daemon stop` (or `launchctl bootout gui/$UID/com.clawdbot.gateway`). - To restart, use `clawdbot daemon restart` (or `launchctl kickstart -k gui/$UID/com.clawdbot.gateway`). + - `launchctl` only works if the LaunchAgent is installed; otherwise use `clawdbot daemon install` first. ## Supervision (systemd user unit) Create `~/.config/systemd/user/clawdbot-gateway.service`: diff --git a/docs/gateway/logging.md b/docs/gateway/logging.md index a761afb82..f8b7555a3 100644 --- a/docs/gateway/logging.md +++ b/docs/gateway/logging.md @@ -24,6 +24,11 @@ Clawdbot uses a file logger backed by `tslog` ([`src/logging.ts`](https://github The file format is one JSON object per line. The Control UI Logs tab tails this file via the gateway (`logs.tail`). +CLI can do the same: + +```bash +clawdbot logs --follow +``` **Verbose vs. log levels** diff --git a/docs/gateway/troubleshooting.md b/docs/gateway/troubleshooting.md index 89b8f8956..5f8960238 100644 --- a/docs/gateway/troubleshooting.md +++ b/docs/gateway/troubleshooting.md @@ -23,8 +23,10 @@ clawdbot doctor Doctor/daemon will show runtime state (PID/last exit) and log hints. **Logs:** -- macOS: `~/.clawdbot/logs/gateway.log` and `gateway.err.log` -- Linux: `journalctl --user -u clawdbot-gateway.service -n 200 --no-pager` +- Preferred: `clawdbot logs --follow` +- File logs (always): `/tmp/clawdbot/clawdbot-YYYY-MM-DD.log` (or your configured `logging.file`) +- macOS LaunchAgent (if installed): `~/.clawdbot/logs/gateway.log` and `gateway.err.log` +- Linux systemd (if installed): `journalctl --user -u clawdbot-gateway.service -n 200 --no-pager` - Windows: `schtasks /Query /TN "Clawdbot Gateway" /V /FO LIST` ### Address Already in Use (Port 18789) @@ -74,6 +76,8 @@ cat ~/.clawdbot/clawdbot.json | jq '.routing.groupChat, .whatsapp.groups, .teleg **Check 3:** Check the logs ```bash +clawdbot logs --follow +# or if you want quick filters: tail -f "$(ls -t /tmp/clawdbot/clawdbot-*.log | head -1)" | grep "blocked\\|skip\\|unauthorized" ``` @@ -126,7 +130,7 @@ clawdbot status clawdbot status --deep # View recent connection events -tail -100 /tmp/clawdbot/clawdbot-*.log | grep "connection\\|disconnect\\|logout" +clawdbot logs --limit 200 | grep "connection\\|disconnect\\|logout" ``` **Fix:** Usually reconnects automatically once the Gateway is running. If you’re stuck, restart the Gateway process (however you supervise it), or run it manually with verbose output: diff --git a/docs/install/updating.md b/docs/install/updating.md index b20ac04cd..8320021f0 100644 --- a/docs/install/updating.md +++ b/docs/install/updating.md @@ -90,12 +90,14 @@ CLI (works regardless of OS): clawdbot daemon stop clawdbot daemon restart clawdbot gateway --port 18789 +clawdbot logs --follow ``` If you’re supervised: - macOS launchd (app-bundled LaunchAgent): `launchctl kickstart -k gui/$UID/com.clawdbot.gateway` - Linux systemd user service: `systemctl --user restart clawdbot-gateway.service` - Windows (WSL2): `systemctl --user restart clawdbot-gateway.service` + - `launchctl`/`systemctl` only work if the service is installed; otherwise run `clawdbot daemon install`. Runbook + exact service labels: [Gateway runbook](/gateway) diff --git a/docs/platforms/macos.md b/docs/platforms/macos.md index 7e3671e37..33b775b9c 100644 --- a/docs/platforms/macos.md +++ b/docs/platforms/macos.md @@ -23,6 +23,8 @@ launchctl kickstart -k gui/$UID/com.clawdbot.gateway launchctl bootout gui/$UID/com.clawdbot.gateway ``` +`launchctl` only works if the LaunchAgent is installed; otherwise run `clawdbot daemon install` first. + Details: [Gateway runbook](/gateway) and [Bundled bun Gateway](/platforms/mac/bun). ## Purpose diff --git a/docs/providers/telegram.md b/docs/providers/telegram.md index bcb59733b..cf4e8c307 100644 --- a/docs/providers/telegram.md +++ b/docs/providers/telegram.md @@ -208,7 +208,7 @@ Outbound Telegram API calls retry on transient network/429 errors with exponenti - If `telegram.groups` is set, the group must be listed or use `"*"` - Check Privacy Settings in @BotFather → "Group Privacy" should be **OFF** - Verify bot is actually a member (not just an admin with no read access) -- Check gateway logs: `journalctl --user -u clawdbot -f` (look for "skipping group message") +- Check gateway logs: `clawdbot logs --follow` (look for "skipping group message") **Bot responds to mentions but not `/activation always`:** - The `/activation` command updates session state but doesn't persist to config diff --git a/docs/start/faq.md b/docs/start/faq.md index 59ffb409f..e5c14e6d5 100644 --- a/docs/start/faq.md +++ b/docs/start/faq.md @@ -363,6 +363,12 @@ Default log file: You can set a stable path via `logging.file`. File log level is controlled by `logging.level`. Console verbosity is controlled by `--verbose` and `logging.consoleLevel`. +Fastest log tail: + +```bash +clawdbot logs --follow +``` + ### What’s the fastest way to get more details when something fails? Start the Gateway with `--verbose` to get more console detail. Then inspect the log file for provider auth, model routing, and RPC errors. diff --git a/src/auto-reply/reply/commands.ts b/src/auto-reply/reply/commands.ts index 28d78230e..78bc7010f 100644 --- a/src/auto-reply/reply/commands.ts +++ b/src/auto-reply/reply/commands.ts @@ -24,7 +24,10 @@ import { formatUsageSummaryLine, loadProviderUsageSummary, } from "../../infra/provider-usage.js"; -import { triggerClawdbotRestart } from "../../infra/restart.js"; +import { + scheduleGatewaySigusr1Restart, + triggerClawdbotRestart, +} from "../../infra/restart.js"; import { enqueueSystemEvent } from "../../infra/system-events.js"; import { parseAgentSessionKey } from "../../routing/session-key.js"; import { resolveSendPolicy } from "../../sessions/send-policy.js"; @@ -360,11 +363,32 @@ export async function handleCommands(params: { ); return { shouldContinue: false }; } + const hasSigusr1Listener = process.listenerCount("SIGUSR1") > 0; + if (hasSigusr1Listener) { + scheduleGatewaySigusr1Restart({ reason: "/restart" }); + return { + shouldContinue: false, + reply: { + text: "⚙️ Restarting clawdbot in-process (SIGUSR1); back in a few seconds.", + }, + }; + } const restartMethod = triggerClawdbotRestart(); + if (!restartMethod.ok) { + const detail = restartMethod.detail + ? ` Details: ${restartMethod.detail}` + : ""; + return { + shouldContinue: false, + reply: { + text: `⚠️ Restart failed (${restartMethod.method}).${detail}`, + }, + }; + } return { shouldContinue: false, reply: { - text: `⚙️ Restarting clawdbot via ${restartMethod}; give me a few seconds to come back online.`, + text: `⚙️ Restarting clawdbot via ${restartMethod.method}; give me a few seconds to come back online.`, }, }; } diff --git a/src/cli/daemon-cli.ts b/src/cli/daemon-cli.ts index 48f612e35..0e818246b 100644 --- a/src/cli/daemon-cli.ts +++ b/src/cli/daemon-cli.ts @@ -28,6 +28,7 @@ import { type PortListener, type PortUsageStatus, } from "../infra/ports.js"; +import { getResolvedLoggerSettings } from "../logging.js"; import { defaultRuntime } from "../runtime.js"; import { createDefaultDeps } from "./deps.js"; import { withProgress } from "./progress.js"; @@ -171,11 +172,24 @@ function renderRuntimeHints( ): string[] { if (!runtime) return []; const hints: string[] = []; + const fileLog = (() => { + try { + return getResolvedLoggerSettings().file; + } catch { + return null; + } + })(); + if (runtime.missingUnit) { + hints.push("Service not installed. Run: clawdbot daemon install"); + if (fileLog) hints.push(`File logs: ${fileLog}`); + return hints; + } if (runtime.status === "stopped") { + if (fileLog) hints.push(`File logs: ${fileLog}`); if (process.platform === "darwin") { const logs = resolveGatewayLogPaths(process.env); - hints.push(`Logs: ${logs.stdoutPath}`); - hints.push(`Errors: ${logs.stderrPath}`); + hints.push(`Launchd stdout (if installed): ${logs.stdoutPath}`); + hints.push(`Launchd stderr (if installed): ${logs.stderrPath}`); } else if (process.platform === "linux") { hints.push( "Logs: journalctl --user -u clawdbot-gateway.service -n 200 --no-pager", @@ -188,17 +202,22 @@ function renderRuntimeHints( } function renderGatewayServiceStartHints(): string[] { + const base = ["clawdbot daemon install", "clawdbot gateway"]; switch (process.platform) { case "darwin": return [ + ...base, `launchctl bootstrap gui/$UID ~/Library/LaunchAgents/${GATEWAY_LAUNCH_AGENT_LABEL}.plist`, ]; case "linux": - return [`systemctl --user start ${GATEWAY_SYSTEMD_SERVICE_NAME}.service`]; + return [ + ...base, + `systemctl --user start ${GATEWAY_SYSTEMD_SERVICE_NAME}.service`, + ]; case "win32": - return [`schtasks /Run /TN "${GATEWAY_WINDOWS_TASK_NAME}"`]; + return [...base, `schtasks /Run /TN "${GATEWAY_WINDOWS_TASK_NAME}"`]; default: - return []; + return base; } } @@ -261,6 +280,12 @@ function printDaemonStatus(status: DaemonStatus, opts: { json: boolean }) { defaultRuntime.log( `Service: ${service.label} (${service.loaded ? service.loadedText : service.notLoadedText})`, ); + try { + const logFile = getResolvedLoggerSettings().file; + defaultRuntime.log(`File logs: ${logFile}`); + } catch { + // ignore missing config/log resolution + } if (service.command?.programArguments?.length) { defaultRuntime.log( `Command: ${service.command.programArguments.join(" ")}`, @@ -280,7 +305,12 @@ function printDaemonStatus(status: DaemonStatus, opts: { json: boolean }) { defaultRuntime.error(`RPC probe: failed (${rpc.error})`); } } - if (service.loaded && service.runtime?.status === "stopped") { + if (service.runtime?.missingUnit) { + defaultRuntime.error("Service unit not found."); + for (const hint of renderRuntimeHints(service.runtime)) { + defaultRuntime.error(hint); + } + } else if (service.loaded && service.runtime?.status === "stopped") { defaultRuntime.error( "Service is loaded but not running (likely exited immediately).", ); @@ -292,6 +322,7 @@ function printDaemonStatus(status: DaemonStatus, opts: { json: boolean }) { defaultRuntime.error( `LaunchAgent label cached but plist missing. Clear with: launchctl bootout gui/$UID/${GATEWAY_LAUNCH_AGENT_LABEL}`, ); + defaultRuntime.error("Then reinstall: clawdbot daemon install"); } if (status.port && shouldReportPortUsage(status.port.status, rpc?.ok)) { for (const line of formatPortDiagnostics({ diff --git a/src/cli/logs-cli.ts b/src/cli/logs-cli.ts new file mode 100644 index 000000000..6cb4dc660 --- /dev/null +++ b/src/cli/logs-cli.ts @@ -0,0 +1,96 @@ +import { setTimeout as delay } from "node:timers/promises"; +import type { Command } from "commander"; +import { defaultRuntime } from "../runtime.js"; +import { addGatewayClientOptions, callGatewayFromCli } from "./gateway-rpc.js"; + +type LogsTailPayload = { + file?: string; + cursor?: number; + size?: number; + lines?: string[]; + truncated?: boolean; + reset?: boolean; +}; + +type LogsCliOptions = { + limit?: string; + maxBytes?: string; + follow?: boolean; + interval?: string; + json?: boolean; + url?: string; + token?: string; + timeout?: string; + expectFinal?: boolean; +}; + +function parsePositiveInt(value: string | undefined, fallback: number): number { + if (!value) return fallback; + const parsed = Number.parseInt(value, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; +} + +async function fetchLogs( + opts: LogsCliOptions, + cursor: number | undefined, +): Promise { + const limit = parsePositiveInt(opts.limit, 200); + const maxBytes = parsePositiveInt(opts.maxBytes, 250_000); + const payload = await callGatewayFromCli("logs.tail", opts, { + cursor, + limit, + maxBytes, + }); + if (!payload || typeof payload !== "object") { + throw new Error("Unexpected logs.tail response"); + } + return payload as LogsTailPayload; +} + +export function registerLogsCli(program: Command) { + const logs = program + .command("logs") + .description("Tail gateway file logs via RPC") + .option("--limit ", "Max lines to return", "200") + .option("--max-bytes ", "Max bytes to read", "250000") + .option("--follow", "Follow log output", false) + .option("--interval ", "Polling interval in ms", "1000") + .option("--json", "Emit JSON payloads", false); + + addGatewayClientOptions(logs); + + logs.action(async (opts: LogsCliOptions) => { + const interval = parsePositiveInt(opts.interval, 1000); + let cursor: number | undefined; + let first = true; + + while (true) { + const payload = await fetchLogs(opts, cursor); + const lines = Array.isArray(payload.lines) ? payload.lines : []; + if (opts.json) { + defaultRuntime.log(JSON.stringify(payload, null, 2)); + } else { + if (first && payload.file) { + defaultRuntime.log(`Log file: ${payload.file}`); + } + for (const line of lines) { + defaultRuntime.log(line); + } + if (payload.truncated) { + defaultRuntime.error("Log tail truncated (increase --max-bytes)."); + } + if (payload.reset) { + defaultRuntime.error("Log cursor reset (file rotated)."); + } + } + cursor = + typeof payload.cursor === "number" && Number.isFinite(payload.cursor) + ? payload.cursor + : cursor; + first = false; + + if (!opts.follow) return; + await delay(interval); + } + }); +} diff --git a/src/cli/program.ts b/src/cli/program.ts index ca5621491..7012bcd5e 100644 --- a/src/cli/program.ts +++ b/src/cli/program.ts @@ -37,6 +37,7 @@ import { registerDnsCli } from "./dns-cli.js"; import { registerDocsCli } from "./docs-cli.js"; import { registerGatewayCli } from "./gateway-cli.js"; import { registerHooksCli } from "./hooks-cli.js"; +import { registerLogsCli } from "./logs-cli.js"; import { registerModelsCli } from "./models-cli.js"; import { registerNodesCli } from "./nodes-cli.js"; import { registerPairingCli } from "./pairing-cli.js"; @@ -616,6 +617,7 @@ Examples: registerDaemonCli(program); registerGatewayCli(program); + registerLogsCli(program); registerModelsCli(program); registerNodesCli(program); registerTuiCli(program); diff --git a/src/commands/doctor-format.ts b/src/commands/doctor-format.ts index 1f71350fc..ecadd104d 100644 --- a/src/commands/doctor-format.ts +++ b/src/commands/doctor-format.ts @@ -1,6 +1,7 @@ import { GATEWAY_LAUNCH_AGENT_LABEL } from "../daemon/constants.js"; import { resolveGatewayLogPaths } from "../daemon/launchd.js"; import type { GatewayServiceRuntime } from "../daemon/service-runtime.js"; +import { getResolvedLoggerSettings } from "../logging.js"; type RuntimeHintOptions = { platform?: NodeJS.Platform; @@ -42,19 +43,33 @@ export function buildGatewayRuntimeHints( if (!runtime) return hints; const platform = options.platform ?? process.platform; const env = options.env ?? process.env; + const fileLog = (() => { + try { + return getResolvedLoggerSettings().file; + } catch { + return null; + } + })(); if (runtime.cachedLabel && platform === "darwin") { hints.push( `LaunchAgent label cached but plist missing. Clear with: launchctl bootout gui/$UID/${GATEWAY_LAUNCH_AGENT_LABEL}`, ); + hints.push("Then reinstall: clawdbot daemon install"); + } + if (runtime.missingUnit) { + hints.push("Service not installed. Run: clawdbot daemon install"); + if (fileLog) hints.push(`File logs: ${fileLog}`); + return hints; } if (runtime.status === "stopped") { hints.push( "Service is loaded but not running (likely exited immediately).", ); + if (fileLog) hints.push(`File logs: ${fileLog}`); if (platform === "darwin") { const logs = resolveGatewayLogPaths(env); - hints.push(`Logs: ${logs.stdoutPath}`); - hints.push(`Errors: ${logs.stderrPath}`); + hints.push(`Launchd stdout (if installed): ${logs.stdoutPath}`); + hints.push(`Launchd stderr (if installed): ${logs.stderrPath}`); } else if (platform === "linux") { hints.push( "Logs: journalctl --user -u clawdbot-gateway.service -n 200 --no-pager", diff --git a/src/infra/restart.ts b/src/infra/restart.ts index 2a6fb6107..1663d1f9c 100644 --- a/src/infra/restart.ts +++ b/src/infra/restart.ts @@ -1,38 +1,97 @@ import { spawnSync } from "node:child_process"; +import { + GATEWAY_LAUNCH_AGENT_LABEL, + GATEWAY_SYSTEMD_SERVICE_NAME, +} from "../daemon/constants.js"; -const DEFAULT_LAUNCHD_LABEL = "com.clawdbot.mac"; -const DEFAULT_SYSTEMD_UNIT = "clawdbot-gateway.service"; +export type RestartAttempt = { + ok: boolean; + method: "launchctl" | "systemd" | "supervisor"; + detail?: string; + tried?: string[]; +}; -export function triggerClawdbotRestart(): - | "launchctl" - | "systemd" - | "supervisor" { +function formatSpawnDetail(result: { + error?: unknown; + status?: number | null; + stdout?: string | Buffer | null; + stderr?: string | Buffer | null; +}): string { + const clean = (value: string | Buffer | null | undefined) => { + const text = + typeof value === "string" + ? value + : value + ? value.toString() + : ""; + return text.replace(/\s+/g, " ").trim(); + }; + if (result.error) return String(result.error); + const stderr = clean(result.stderr); + if (stderr) return stderr; + const stdout = clean(result.stdout); + if (stdout) return stdout; + if (typeof result.status === "number") return `exit ${result.status}`; + return "unknown error"; +} + +function normalizeSystemdUnit(raw?: string): string { + const unit = raw?.trim(); + if (!unit) return `${GATEWAY_SYSTEMD_SERVICE_NAME}.service`; + return unit.endsWith(".service") ? unit : `${unit}.service`; +} + +export function triggerClawdbotRestart(): RestartAttempt { + const tried: string[] = []; if (process.platform !== "darwin") { if (process.platform === "linux") { - const unit = process.env.CLAWDBOT_SYSTEMD_UNIT || DEFAULT_SYSTEMD_UNIT; - const userRestart = spawnSync("systemctl", ["--user", "restart", unit], { - stdio: "ignore", + const unit = normalizeSystemdUnit(process.env.CLAWDBOT_SYSTEMD_UNIT); + const userArgs = ["--user", "restart", unit]; + tried.push(`systemctl ${userArgs.join(" ")}`); + const userRestart = spawnSync("systemctl", userArgs, { + encoding: "utf8", }); if (!userRestart.error && userRestart.status === 0) { - return "systemd"; + return { ok: true, method: "systemd", tried }; } - const systemRestart = spawnSync("systemctl", ["restart", unit], { - stdio: "ignore", + const systemArgs = ["restart", unit]; + tried.push(`systemctl ${systemArgs.join(" ")}`); + const systemRestart = spawnSync("systemctl", systemArgs, { + encoding: "utf8", }); if (!systemRestart.error && systemRestart.status === 0) { - return "systemd"; + return { ok: true, method: "systemd", tried }; } - return "systemd"; + const detail = [ + `user: ${formatSpawnDetail(userRestart)}`, + `system: ${formatSpawnDetail(systemRestart)}`, + ].join("; "); + return { ok: false, method: "systemd", detail, tried }; } - return "supervisor"; + return { + ok: false, + method: "supervisor", + detail: "unsupported platform restart", + }; } - const label = process.env.CLAWDBOT_LAUNCHD_LABEL || DEFAULT_LAUNCHD_LABEL; + const label = + process.env.CLAWDBOT_LAUNCHD_LABEL || GATEWAY_LAUNCH_AGENT_LABEL; const uid = typeof process.getuid === "function" ? process.getuid() : undefined; const target = uid !== undefined ? `gui/${uid}/${label}` : label; - spawnSync("launchctl", ["kickstart", "-k", target], { stdio: "ignore" }); - return "launchctl"; + const args = ["kickstart", "-k", target]; + tried.push(`launchctl ${args.join(" ")}`); + const res = spawnSync("launchctl", args, { encoding: "utf8" }); + if (!res.error && res.status === 0) { + return { ok: true, method: "launchctl", tried }; + } + return { + ok: false, + method: "launchctl", + detail: formatSpawnDetail(res), + tried, + }; } export type ScheduledRestart = {