clawdbot/extensions/tlon/README.md

# Clawdbot Tlon/Urbit Integration

Complete documentation for integrating Clawdbot with Tlon Messenger (built on Urbit).

## Overview

This extension enables Clawdbot to:
- Monitor and respond to direct messages on Tlon Messenger 
- Monitor and respond to group channel messages when mentioned
- Auto-discover available group channels
- Use per-conversation subscriptions for reliable message delivery
- **Automatic AI model fallback** - Seamlessly switches from Anthropic to OpenAI when rate limited (see [FALLBACK.md](./FALLBACK.md))

**Ship:** ~sitrul-nacwyl
**Test User:** ~malmur-halmex

## Architecture

### Files

- **`index.js`** - Plugin entry point, registers the Tlon channel adapter
- **`monitor.js`** - Core monitoring logic, handles incoming messages and AI dispatch
- **`urbit-sse-client.js`** - Custom SSE client for Urbit HTTP API
- **`core-bridge.js`** - Dynamic loader for clawdbot core modules
- **`package.json`** - Plugin package definition
- **`FALLBACK.md`** - AI model fallback system documentation

### How It Works

1. **Authentication**: Uses ship name + code to authenticate via `/~/login` endpoint
2. **Channel Creation**: Creates Tlon Messenger channel via PUT to `/~/channel/{uid}`
3. **Activation**: Sends "helm-hi" poke to activate channel (required!)
4. **Subscriptions**:
   - **DMs**: Individual subscriptions to `/dm/{ship}` for each conversation
   - **Groups**: Individual subscriptions to `/{channelNest}` for each channel
5. **SSE Stream**: Opens server-sent events stream for real-time updates
6. **Auto-Reconnection**: Automatically reconnects if SSE stream dies
   - Exponential backoff (1s to 30s delays)
   - Up to 10 reconnection attempts
   - Generates new channel ID on each attempt
7. **Auto-Discovery**: Queries `/groups-ui/v6/init.json` to find all available channels
8. **Dynamic Refresh**: Polls every 2 minutes for new conversations/channels
9. **Message Processing**: When bot is mentioned, routes to AI via clawdbot core
10. **AI Fallback**: Automatically switches providers when rate limited
   - Primary: Anthropic Claude Sonnet 4.5
   - Fallbacks: OpenAI GPT-4o, GPT-4 Turbo
   - Automatic cooldown management
   - See [FALLBACK.md](./FALLBACK.md) for details

## Configuration

### 1. Install Dependencies

```bash
cd ~/.clawdbot/extensions/tlon
npm install
```

### 2. Configure Credentials

Edit `~/.clawdbot/clawdbot.json`:

```json
{
  "channels": {
    "tlon": {
      "enabled": true,
      "ship": "your-ship-name",
      "code": "your-ship-code",
      "url": "https://your-ship-name.tlon.network",
      "showModelSignature": false,
      "dmAllowlist": ["~friend-ship-1", "~friend-ship-2"],
      "defaultAuthorizedShips": ["~malmur-halmex"],
      "authorization": {
        "channelRules": {
          "chat/~host-ship/channel-name": {
            "mode": "open",
            "allowedShips": []
          },
          "chat/~another-host/private-channel": {
            "mode": "restricted",
            "allowedShips": ["~malmur-halmex", "~sitrul-nacwyl"]
          }
        }
      }
    }
  }
}
```

**Configuration Options:**
- `enabled` - Enable/disable the Tlon channel (default: `false`)
- `ship` - Your Urbit ship name (required)
- `code` - Your ship's login code (required)
- `url` - Your ship's URL (required)
- `showModelSignature` - Append model name to responses (default: `false`)
  - When enabled, adds `[Generated by Claude Sonnet 4.5]` to the end of each response
  - Useful for transparency about which AI model generated the response
- `dmAllowlist` - Ships allowed to send DMs (optional)
  - If omitted or empty, all DMs are accepted (default behavior)
  - Ship names can include or omit the `~` prefix
  - Example: `["~trusted-friend", "~another-ship"]`
  - Blocked DMs are logged for visibility
- `defaultAuthorizedShips` - Ships authorized in new/unconfigured channels (default: `["~malmur-halmex"]`)
  - New channels default to `restricted` mode using these ships
- `authorization` - Per-channel access control (optional)
  - `channelRules` - Map of channel nest to authorization rules
    - `mode`: `"open"` (all ships) or `"restricted"` (allowedShips only)
    - `allowedShips`: Array of authorized ships (only for `restricted` mode)

**For localhost development:**
```json
"url": "http://localhost:8080"
```

**For Tlon-hosted ships:**
```json
"url": "https://{ship-name}.tlon.network"
```

### 3. Set Environment Variable

The monitor needs to find clawdbot's core modules. Set the environment variable:

```bash
export CLAWDBOT_ROOT=/opt/homebrew/lib/node_modules/clawdbot
```

Or if clawdbot is installed elsewhere:
```bash
export CLAWDBOT_ROOT=$(dirname $(dirname $(readlink -f $(which clawdbot))))
```

**Make it permanent** (add to `~/.zshrc` or `~/.bashrc`):
```bash
echo 'export CLAWDBOT_ROOT=/opt/homebrew/lib/node_modules/clawdbot' >> ~/.zshrc
```

### 4. Configure AI Authentication

The bot needs API credentials to generate responses.

**Option A: Use Claude Code CLI credentials**
```bash
clawdbot agents add main
# Select "Use Claude Code CLI credentials"
```

**Option B: Use Anthropic API key**
```bash
clawdbot agents add main
# Enter your API key from console.anthropic.com
```

### 5. Start the Gateway

```bash
CLAWDBOT_ROOT=/opt/homebrew/lib/node_modules/clawdbot clawdbot gateway
```

Or create a launch script:

```bash
cat > ~/start-clawdbot.sh << 'EOF'
#!/bin/bash
export CLAWDBOT_ROOT=/opt/homebrew/lib/node_modules/clawdbot
clawdbot gateway
EOF
chmod +x ~/start-clawdbot.sh
```

## Usage

### Testing

1. Send a DM from another ship to ~sitrul-nacwyl
2. Mention the bot: `~sitrul-nacwyl hello there!`
3. Bot should respond with AI-generated reply

### Monitoring Logs

Check gateway logs:
```bash
tail -f /tmp/clawdbot/clawdbot-$(date +%Y-%m-%d).log
```

Look for these indicators:
- `[tlon] Successfully authenticated to https://...`
- `[tlon] Auto-discovered N chat channel(s)`
- `[tlon] Connected! All subscriptions active`
- `[tlon] Received DM from ~ship: "..." (mentioned: true)`
- `[tlon] Dispatching to AI for ~ship (DM)`
- `[tlon] Delivered AI reply to ~ship`

### Group Channels

The bot automatically discovers and subscribes to all group channels using **delta-based discovery** for efficiency.

**How Auto-Discovery Works:**
1. **On startup:** Fetches changes from the last 5 days via `/groups-ui/v5/changes/~YYYY.M.D..20.19.51..9b9d.json`
2. **Periodic refresh:** Checks for new channels every 2 minutes
3. **Smart caching:** Only fetches deltas, not full state each time

**Benefits:**
- Reduced bandwidth usage
- Faster startup (especially for ships with many groups)
- Automatically picks up new channels you join
- Context of recent group activity

**Manual Configuration:**

To disable auto-discovery and use specific channels:

```json
{
  "channels": {
    "tlon": {
      "enabled": true,
      "ship": "your-ship-name",
      "code": "your-ship-code",
      "url": "https://your-ship-name.tlon.network",
      "autoDiscoverChannels": false,
      "groupChannels": [
        "chat/~host-ship/channel-name",
        "chat/~another-host/another-channel"
      ]
    }
  }
}
```

### Model Signatures

The bot can append the AI model name to each response for transparency. Enable this feature in your config:

```json
{
  "channels": {
    "tlon": {
      "enabled": true,
      "ship": "your-ship-name",
      "code": "your-ship-code",
      "url": "https://your-ship-name.tlon.network",
      "showModelSignature": true
    }
  }
}
```

**Example output with signature enabled:**
```
User: ~sitrul-nacwyl explain quantum computing
Bot: Quantum computing uses quantum mechanics principles like superposition
     and entanglement to perform calculations...

     [Generated by Claude Sonnet 4.5]
```

**Supported model formats:**
- `Claude Opus 4.5`
- `Claude Sonnet 4.5`
- `GPT-4o`
- `GPT-4 Turbo`
- `Gemini 2.0 Flash`

When using the [AI fallback system](./FALLBACK.md), signatures automatically reflect which model generated the response (e.g., if Anthropic is rate limited and OpenAI is used, the signature will show `GPT-4o`).

### Channel History Summarization

The bot can summarize recent channel activity when asked. This is useful for catching up on conversations you missed.

**Trigger phrases:**
- `~bot-ship summarize this channel`
- `~bot-ship what did I miss?`
- `~bot-ship catch me up`
- `~bot-ship tldr`
- `~bot-ship channel summary`

**Example:**
```
User: ~sitrul-nacwyl what did I miss?
Bot: Here's a summary of the last 50 messages:

Main topics discussed:
1. Discussion about Urbit networking (Ames protocol)
2. Planning for next week's developer meetup
3. Bug reports for the new UI update

Key decisions:
- Meetup scheduled for Thursday at 3pm EST
- Priority on fixing the scrolling issue

Notable participants: ~malmur-halmex, ~bolbex-fogdys
```

**How it works:**
- Fetches the last 50 messages from the channel
- Sends them to the AI for summarization
- Returns a concise summary with main topics, decisions, and action items

### Thread Support

The bot automatically maintains context in threaded conversations. When you mention the bot in a reply thread, it will respond within that thread instead of posting to the main channel.

**Example:**
```
Main channel post:
  User A: ~sitrul-nacwyl what's the capital of France?
  Bot: Paris is the capital of France.
    └─ User B (in thread): ~sitrul-nacwyl and what's its population?
       └─ Bot (in thread): Paris has a population of approximately 2.2 million...
```

**Benefits:**
- Keeps conversations organized
- Reduces noise in main channel
- Maintains conversation context within threads

**Technical Details:**
The bot handles both top-level posts and thread replies with different data structures:
- Top-level posts: `response.post.r-post.set.essay`
- Thread replies: `response.post.r-post.reply.r-reply.set.memo`

When replying in a thread, the bot uses the `parent-id` from the incoming message to ensure the reply stays within the same thread.

**Note:** Thread support is automatic - no configuration needed.

### Link Summarization

The bot can fetch and summarize web content when you share links.

**Example:**
```
User: ~sitrul-nacwyl can you summarize this https://example.com/article
Bot: This article discusses... [summary of the content]
```

**How it works:**
- Bot extracts URLs from rich text messages (including inline links)
- Fetches the web page content
- Summarizes using the WebFetch tool

### Channel Authorization

Control which ships can invoke the bot in specific group channels. **New channels default to `restricted` mode** for security.

#### Default Behavior

**DMs:** Always open (no restrictions)
**Group Channels:** Restricted by default, only ships in `defaultAuthorizedShips` can invoke the bot

#### Configuration

```json
{
  "channels": {
    "tlon": {
      "enabled": true,
      "ship": "sitrul-nacwyl",
      "code": "your-code",
      "url": "https://sitrul-nacwyl.tlon.network",
      "defaultAuthorizedShips": ["~malmur-halmex"],
      "authorization": {
        "channelRules": {
          "chat/~bitpyx-dildus/core": {
            "mode": "open"
          },
          "chat/~nocsyx-lassul/bongtable": {
            "mode": "restricted",
            "allowedShips": ["~malmur-halmex", "~sitrul-nacwyl"]
          }
        }
      }
    }
  }
}
```

#### Authorization Modes

**`open`** - Any ship can invoke the bot when mentioned
- Good for public channels
- No `allowedShips` needed

**`restricted`** (default) - Only specific ships can invoke the bot
- Good for private/work channels
- Requires `allowedShips` list
- New channels use `defaultAuthorizedShips` if no rule exists

#### Examples

**Make a channel public:**
```json
"chat/~bitpyx-dildus/core": {
  "mode": "open"
}
```

**Restrict to specific users:**
```json
"chat/~nocsyx-lassul/bongtable": {
  "mode": "restricted",
  "allowedShips": ["~malmur-halmex"]
}
```

**New channel (no config):**
- Mode: `restricted` (safe default)
- Allowed ships: `defaultAuthorizedShips` (e.g., `["~malmur-halmex"]`)

#### Behavior

**Authorized mention:**
```
~malmur-halmex: ~sitrul-nacwyl tell me about quantum computing
Bot: [Responds with answer]
```

**Unauthorized mention (silently ignored):**
```
~other-ship: ~sitrul-nacwyl tell me about quantum computing
Bot: [No response, logs show access denied]
```

**Check logs:**
```bash
tail -f /tmp/tlon-fallback.log | grep "Access"
```

You'll see:
```
[tlon] ✅ Access granted: ~malmur-halmex in chat/~host/channel (authorized user)
[tlon] ⛔ Access denied: ~other-ship in chat/~host/channel (restricted, allowed: ~malmur-halmex)
```

## Technical Deep Dive

### Urbit HTTP API Flow

1. **Login** (POST `/~/login`)
   - Sends `password={code}`
   - Returns authentication cookie in `set-cookie` header

2. **Channel Creation** (PUT `/~/channel/{channelId}`)
   - Channel ID format: `{timestamp}-{random}`
   - Body: array of subscription objects
   - Response: 204 No Content

3. **Channel Activation** (PUT `/~/channel/{channelId}`)
   - **Critical:** Must send helm-hi poke BEFORE opening SSE stream
   - Poke structure:
     ```json
     {
       "id": timestamp,
       "action": "poke",
       "ship": "sitrul-nacwyl",
       "app": "hood",
       "mark": "helm-hi",
       "json": "Opening API channel"
     }
     ```

4. **SSE Stream** (GET `/~/channel/{channelId}`)
   - Headers: `Accept: text/event-stream`
   - Returns Server-Sent Events
   - Format:
     ```
     id: {event-id}
     data: {json-payload}

     ```

### Subscription Paths

#### DMs (Chat App)
- **Path:** `/dm/{ship}`
- **App:** `chat`
- **Event Format:**
  ```json
  {
    "id": "~ship/timestamp",
    "whom": "~other-ship",
    "response": {
      "add": {
        "memo": {
          "author": "~sender-ship",
          "sent": 1768742460781,
          "content": [
            {
              "inline": [
                "text",
                {"ship": "~mentioned-ship"},
                "more text",
                {"break": null}
              ]
            }
          ]
        }
      }
    }
  }
  ```

#### Group Channels (Channels App)
- **Path:** `/{channelNest}`
- **Channel Nest Format:** `chat/~host-ship/channel-name`
- **App:** `channels`
- **Event Format:**
  ```json
  {
    "response": {
      "post": {
        "id": "message-id",
        "r-post": {
          "set": {
            "essay": {
              "author": "~sender-ship",
              "sent": 1768742460781,
              "kind": "/chat",
              "content": [...]
            }
          }
        }
      }
    }
  }
  ```

### Text Extraction

Message content uses inline format with mixed types:
- Strings: plain text
- Objects with `ship`: mentions (e.g., `{"ship": "~sitrul-nacwyl"}`)
- Objects with `break`: line breaks (e.g., `{"break": null}`)

Example:
```json
{
  "inline": [
    "Hey ",
    {"ship": "~sitrul-nacwyl"},
    " how are you?",
    {"break": null},
    "This is a new line"
  ]
}
```

Extracts to: `"Hey ~sitrul-nacwyl how are you?\nThis is a new line"`

### Mention Detection

Simple includes check (case-insensitive):
```javascript
const normalizedBotShip = botShipName.startsWith("~")
  ? botShipName
  : `~${botShipName}`;
return messageText.toLowerCase().includes(normalizedBotShip.toLowerCase());
```

Note: Word boundaries (`\b`) don't work with `~` character.

## Troubleshooting

### Issue: "Cannot read properties of undefined (reading 'href')"

**Cause:** Some clawdbot dependencies (axios, Slack SDK) expect browser globals

**Fix:** Window.location polyfill is already added to monitor.js (lines 1-18)

### Issue: "Unable to resolve Clawdbot root"

**Cause:** core-bridge.js can't find clawdbot installation

**Fix:** Set `CLAWDBOT_ROOT` environment variable:
```bash
export CLAWDBOT_ROOT=/opt/homebrew/lib/node_modules/clawdbot
```

### Issue: SSE Stream Returns 403 Forbidden

**Cause:** Trying to open SSE stream without activating channel first

**Fix:** Send helm-hi poke before opening stream (urbit-sse-client.js handles this)

### Issue: No Events Received After Subscribing

**Cause:** Wrong subscription path or app name

**Fix:**
- DMs: Use `/dm/{ship}` with `app: "chat"`
- Groups: Use `/{channelNest}` with `app: "channels"`

### Issue: Messages Show "[object Object]"

**Cause:** Not handling inline content objects properly

**Fix:** Text extraction handles mentions and breaks (monitor.js `extractMessageText()`)

### Issue: Bot Not Detecting Mentions

**Cause:** Message doesn't contain bot's ship name

**Debug:**
```bash
tail -f /tmp/clawdbot/clawdbot-*.log | grep "mentioned:"
```

Should show:
```
[tlon] Received DM from ~malmur-halmex: "~sitrul-nacwyl hello..." (mentioned: true)
```

### Issue: "No API key found for provider 'anthropic'"

**Cause:** AI authentication not configured

**Fix:** Run `clawdbot agents add main` and configure credentials

### Issue: Gateway Port Already in Use

**Fix:**
```bash
# Stop existing instance
clawdbot daemon stop

# Or force kill
lsof -ti:18789 | xargs kill -9
```

### Issue: Bot Stops Responding (SSE Disconnection)

**Cause:** Urbit SSE stream disconnected (sent "quit" event or stream ended)

**Symptoms:**
- Logs show: `[SSE] Received event: {"id":X,"response":"quit"}`
- No more incoming SSE events
- Bot appears online but doesn't respond to mentions

**Fix:** The bot now **automatically reconnects**! Look for these log messages:
```
[SSE] Stream ended, attempting reconnection...
[SSE] Reconnection attempt 1/10 in 1000ms...
[SSE] Reconnecting with new channel ID: xxx-yyy
[SSE] Reconnection successful!
```

**Manual restart if needed:**
```bash
kill $(pgrep -f "clawdbot gateway")
CLAWDBOT_ROOT=/opt/homebrew/lib/node_modules/clawdbot clawdbot gateway
```

**Configuration options** (in urbit-sse-client.js constructor):
```javascript
new UrbitSSEClient(url, cookie, {
  autoReconnect: true,          // Default: true
  maxReconnectAttempts: 10,     // Default: 10
  reconnectDelay: 1000,         // Initial delay: 1s
  maxReconnectDelay: 30000,     // Max delay: 30s
  onReconnect: async (client) => {
    // Optional callback for resubscription logic
  }
})
```

## Development Notes

### Testing Without Clawdbot

You can test the Urbit API directly:

```javascript
import { UrbitSSEClient } from "./urbit-sse-client.js";

const api = new UrbitSSEClient(
  "https://sitrul-nacwyl.tlon.network",
  "your-cookie-here"
);

// Subscribe to DMs
await api.subscribe({
  app: "chat",
  path: "/dm/malmur-halmex",
  event: (data) => console.log("DM:", data),
  err: (e) => console.error("Error:", e),
  quit: () => console.log("Quit")
});

// Connect
await api.connect();

// Send a DM
await api.poke({
  app: "chat",
  mark: "chat-dm-action",
  json: {
    ship: "~malmur-halmex",
    diff: {
      id: `~sitrul-nacwyl/${Date.now()}`,
      delta: {
        add: {
          memo: {
            content: [{ inline: ["Hello!"] }],
            author: "~sitrul-nacwyl",
            sent: Date.now()
          },
          kind: null,
          time: null
        }
      }
    }
  }
});
```

### Debugging SSE Events

Enable verbose logging in urbit-sse-client.js:

```javascript
// Line 169-171
if (parsed.response !== "subscribe" && parsed.response !== "poke") {
  console.log("[SSE] Received event:", JSON.stringify(parsed).substring(0, 500));
}
```

Remove the condition to see all events:
```javascript
console.log("[SSE] Received event:", JSON.stringify(parsed).substring(0, 500));
```

### Channel Nest Format

Format: `{type}/{host-ship}/{channel-name}`

Examples:
- `chat/~bitpyx-dildus/core`
- `chat/~malmur-halmex/v3aedb3s`
- `chat/~sitrul-nacwyl/tm-wayfinding-group-chat`

Parse with:
```javascript
const match = channelNest.match(/^([^/]+)\/([^/]+)\/(.+)$/);
const [, type, hostShip, channelName] = match;
```

### Auto-Discovery Endpoint

Query: `GET /~/scry/groups-ui/v6/init.json`

Response structure:
```json
{
  "groups": {
    "group-id": {
      "channels": {
        "chat/~host/name": { ... },
        "diary/~host/name": { ... }
      }
    }
  }
}
```

Filter for chat channels only:
```javascript
if (channelNest.startsWith("chat/")) {
  channels.push(channelNest);
}
```

## Implementation Timeline

### Major Milestones

1. ✅ Plugin structure and registration
2. ✅ Authentication and cookie management
3. ✅ Channel creation and activation (helm-hi poke)
4. ✅ SSE stream connection
5. ✅ DM subscription and event parsing
6. ✅ Group channel support
7. ✅ Auto-discovery of channels
8. ✅ Per-conversation subscriptions
9. ✅ Text extraction (mentions and breaks)
10. ✅ Mention detection
11. ✅ Node.js polyfills (window.location)
12. ✅ Core module integration
13. ⏳ API authentication (user needs to configure)

### Key Discoveries

- **Helm-hi requirement:** Must send helm-hi poke before opening SSE stream
- **Subscription paths:** Frontend uses `/v3` globally, but individual `/dm/{ship}` and `/{channelNest}` paths work better
- **Event formats:** V3 API uses `essay` and `memo` structures (not older `writs` format)
- **Inline content:** Mixed array of strings and objects (mentions, breaks)
- **Tilde handling:** Ship mentions already include `~` prefix
- **Word boundaries:** `\b` regex doesn't work with `~` character
- **Browser globals:** axios and Slack SDK need window.location polyfill
- **Module resolution:** Need CLAWDBOT_ROOT for dynamic imports

## Resources

- **Tlon Apps GitHub:** https://github.com/tloncorp/tlon-apps
- **Urbit HTTP API:** @urbit/http-api package
- **Tlon Frontend Code:** `/tmp/tlon-apps/packages/shared/src/api/chatApi.ts`
- **Clawdbot Docs:** https://docs.clawd.bot/
- **Anthropic Provider:** https://docs.clawd.bot/providers/anthropic

## Future Enhancements

- [ ] Support for message reactions
- [ ] Support for message editing/deletion
- [ ] Support for attachments/images
- [ ] Typing indicators
- [ ] Read receipts
- [ ] Message threading
- [ ] Channel-specific bot personas
- [ ] Rate limiting
- [ ] Message queuing for offline ships
- [ ] Metrics and monitoring

## Credits

Built for integrating Clawdbot with Tlon messenger.

**Developer:** Claude (Sonnet 4.5)
**Platform:** Tlon Messenger built on Urbit