feat: add OpenProse plugin skills
This commit is contained in:
Peter Steinberger
478
extensions/open-prose/skills/prose/state/filesystem.md
Normal file
478
extensions/open-prose/skills/prose/state/filesystem.md
Normal file
@@ -0,0 +1,478 @@
|
||||
---
|
||||
role: file-system-state-management
|
||||
summary: |
|
||||
File-system state management for OpenProse programs. This approach persists
|
||||
execution state to the `.prose/` directory, enabling inspection, resumption,
|
||||
and long-running workflows.
|
||||
see-also:
|
||||
- ../prose.md: VM execution semantics
|
||||
- in-context.md: In-context state management (alternative approach)
|
||||
- sqlite.md: SQLite state management (experimental)
|
||||
- postgres.md: PostgreSQL state management (experimental)
|
||||
- ../primitives/session.md: Session context and compaction guidelines
|
||||
---
|
||||
|
||||
# File-System State Management
|
||||
|
||||
This document describes how the OpenProse VM tracks execution state using **files in the `.prose/` directory**. This is one of two state management approaches (the other being in-context state in `in-context.md`).
|
||||
|
||||
## Overview
|
||||
|
||||
File-based state persists all execution artifacts to disk. This enables:
|
||||
|
||||
- **Inspection**: See exactly what happened at each step
|
||||
- **Resumption**: Pick up interrupted programs
|
||||
- **Long-running workflows**: Handle programs that exceed context limits
|
||||
- **Debugging**: Trace through execution history
|
||||
|
||||
**Key principle:** Files are inspectable artifacts. The directory structure IS the execution state.
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
# Project-level state (in working directory)
|
||||
.prose/
|
||||
├── .env # Config/telemetry (simple key=value format)
|
||||
├── runs/
|
||||
│ └── {YYYYMMDD}-{HHMMSS}-{random}/
|
||||
│ ├── program.prose # Copy of running program
|
||||
│ ├── state.md # Execution state with code snippets
|
||||
│ ├── bindings/
|
||||
│ │ ├── {name}.md # Root scope bindings
|
||||
│ │ └── {name}__{execution_id}.md # Scoped bindings (block invocations)
|
||||
│ ├── imports/
|
||||
│ │ └── {handle}--{slug}/ # Nested program executions (same structure recursively)
|
||||
│ └── agents/
|
||||
│ └── {name}/
|
||||
│ ├── memory.md # Agent's current state
|
||||
│ ├── {name}-001.md # Historical segments (flattened)
|
||||
│ ├── {name}-002.md
|
||||
│ └── ...
|
||||
└── agents/ # Project-scoped agent memory
|
||||
└── {name}/
|
||||
├── memory.md
|
||||
├── {name}-001.md
|
||||
└── ...
|
||||
|
||||
# User-level state (in home directory)
|
||||
~/.prose/
|
||||
└── agents/ # User-scoped agent memory (cross-project)
|
||||
└── {name}/
|
||||
├── memory.md
|
||||
├── {name}-001.md
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Run ID Format
|
||||
|
||||
Format: `{YYYYMMDD}-{HHMMSS}-{random6}`
|
||||
|
||||
Example: `20260115-143052-a7b3c9`
|
||||
|
||||
No "run-" prefix needed—the directory name makes context obvious.
|
||||
|
||||
### Segment Numbering
|
||||
|
||||
Segments use 3-digit zero-padded numbers: `captain-001.md`, `captain-002.md`, etc.
|
||||
|
||||
If a program exceeds 999 segments, extend to 4 digits: `captain-1000.md`.
|
||||
|
||||
---
|
||||
|
||||
## File Formats
|
||||
|
||||
### `.prose/.env`
|
||||
|
||||
Simple key=value configuration file:
|
||||
|
||||
```env
|
||||
OPENPROSE_TELEMETRY=enabled
|
||||
USER_ID=user-a7b3c9d4e5f6
|
||||
SESSION_ID=sess-1704326400000-x9y8z7
|
||||
```
|
||||
|
||||
**Why this format:** Self-evident, no JSON parsing needed, familiar to developers.
|
||||
|
||||
---
|
||||
|
||||
### `state.md`
|
||||
|
||||
The execution state file shows the program's current position using **annotated code snippets**. This makes it self-evident where execution is and what has happened.
|
||||
|
||||
**Only the VM writes this file.** Subagents never modify `state.md`.
|
||||
|
||||
The format shows:
|
||||
- **Full history** of executed code with inline annotations
|
||||
- **Current position** clearly marked with status
|
||||
- **~5-10 lines ahead** of current position (what's coming next)
|
||||
- **Index** of all bindings and agents with file paths
|
||||
|
||||
```markdown
|
||||
# Execution State
|
||||
|
||||
run: 20260115-143052-a7b3c9
|
||||
program: feature-implementation.prose
|
||||
started: 2026-01-15T14:30:52Z
|
||||
updated: 2026-01-15T14:35:22Z
|
||||
|
||||
## Execution Trace
|
||||
|
||||
```prose
|
||||
agent researcher:
|
||||
model: sonnet
|
||||
prompt: "You research topics thoroughly"
|
||||
|
||||
agent captain:
|
||||
model: opus
|
||||
persist: true
|
||||
prompt: "You coordinate and review"
|
||||
|
||||
let research = session: researcher # --> bindings/research.md
|
||||
prompt: "Research AI safety"
|
||||
|
||||
parallel:
|
||||
a = session "Analyze risk A" # --> bindings/a.md (complete)
|
||||
b = session "Analyze risk B" # <-- EXECUTING
|
||||
|
||||
loop until **analysis complete** (max: 3): # [not yet entered]
|
||||
session "Synthesize"
|
||||
context: { a, b, research }
|
||||
|
||||
resume: captain # [...next...]
|
||||
prompt: "Review the synthesis"
|
||||
context: synthesis
|
||||
```
|
||||
|
||||
## Active Constructs
|
||||
|
||||
### Parallel (lines 14-16)
|
||||
|
||||
- a: complete
|
||||
- b: executing
|
||||
|
||||
### Loop (lines 18-21)
|
||||
|
||||
- status: not yet entered
|
||||
- iteration: 0/3
|
||||
- condition: **analysis complete**
|
||||
|
||||
## Index
|
||||
|
||||
### Bindings
|
||||
|
||||
| Name | Kind | Path | Execution ID |
|
||||
|------|------|------|--------------|
|
||||
| research | let | bindings/research.md | (root) |
|
||||
| a | let | bindings/a.md | (root) |
|
||||
| result | let | bindings/result__43.md | 43 |
|
||||
|
||||
### Agents
|
||||
|
||||
| Name | Scope | Path |
|
||||
|------|-------|------|
|
||||
| captain | execution | agents/captain/ |
|
||||
|
||||
## Call Stack
|
||||
|
||||
| execution_id | block | depth | status |
|
||||
|--------------|-------|-------|--------|
|
||||
| 43 | process | 3 | executing |
|
||||
| 42 | process | 2 | waiting |
|
||||
| 41 | process | 1 | waiting |
|
||||
```
|
||||
|
||||
**Status annotations:**
|
||||
|
||||
| Annotation | Meaning |
|
||||
|------------|---------|
|
||||
| `# --> bindings/name.md` | Output written to this file |
|
||||
| `# <-- EXECUTING` | Currently executing this statement |
|
||||
| `# (complete)` | Statement finished successfully |
|
||||
| `# [not yet entered]` | Block not yet reached |
|
||||
| `# [...next...]` | Coming up next |
|
||||
| `# <-- RETRYING (attempt 2/3)` | Retry in progress |
|
||||
|
||||
---
|
||||
|
||||
### `bindings/{name}.md`
|
||||
|
||||
All named values (input, output, let, const) are stored as binding files.
|
||||
|
||||
```markdown
|
||||
# research
|
||||
|
||||
kind: let
|
||||
|
||||
source:
|
||||
```prose
|
||||
let research = session: researcher
|
||||
prompt: "Research AI safety"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
AI safety research covers several key areas including alignment,
|
||||
robustness, and interpretability. The field has grown significantly
|
||||
since 2020 with major contributions from...
|
||||
```
|
||||
|
||||
**Structure:**
|
||||
- Header with binding name
|
||||
- `kind:` field indicating type (input, output, let, const)
|
||||
- `source:` code snippet showing origin
|
||||
- `---` separator
|
||||
- Actual value below
|
||||
|
||||
**The `kind` field distinguishes:**
|
||||
|
||||
| Kind | Meaning |
|
||||
|------|---------|
|
||||
| `input` | Value received from caller |
|
||||
| `output` | Value to return to caller |
|
||||
| `let` | Mutable variable |
|
||||
| `const` | Immutable variable |
|
||||
|
||||
### Anonymous Session Bindings
|
||||
|
||||
Sessions without explicit output capture still produce results:
|
||||
|
||||
```prose
|
||||
session "Analyze the codebase" # No `let x = ...` capture
|
||||
```
|
||||
|
||||
These get auto-generated names with an `anon_` prefix:
|
||||
|
||||
- `bindings/anon_001.md`
|
||||
- `bindings/anon_002.md`
|
||||
- etc.
|
||||
|
||||
This ensures all session outputs are persisted and inspectable.
|
||||
|
||||
---
|
||||
|
||||
### Scoped Bindings (Block Invocations)
|
||||
|
||||
When a binding is created inside a block invocation, it's scoped to that execution frame to prevent collisions across recursive calls.
|
||||
|
||||
**Naming convention:** `{name}__{execution_id}.md`
|
||||
|
||||
Examples:
|
||||
- `bindings/result__43.md` — binding `result` in execution_id 43
|
||||
- `bindings/parts__44.md` — binding `parts` in execution_id 44
|
||||
|
||||
**File format with execution scope:**
|
||||
|
||||
```markdown
|
||||
# result
|
||||
|
||||
kind: let
|
||||
execution_id: 43
|
||||
|
||||
source:
|
||||
```prose
|
||||
let result = session "Process chunk"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Processed chunk into 3 sub-parts...
|
||||
```
|
||||
|
||||
**Scope resolution:** The VM resolves variable references by checking:
|
||||
1. `{name}__{current_execution_id}.md`
|
||||
2. `{name}__{parent_execution_id}.md`
|
||||
3. Continue up the call stack
|
||||
4. `{name}.md` (root scope)
|
||||
|
||||
The first match wins.
|
||||
|
||||
**Example directory for recursive calls:**
|
||||
|
||||
```
|
||||
bindings/
|
||||
├── data.md # Root scope input
|
||||
├── result__1.md # First process() invocation
|
||||
├── parts__1.md # Parts from first invocation
|
||||
├── result__2.md # Recursive call (depth 2)
|
||||
├── parts__2.md # Parts from depth 2
|
||||
├── result__3.md # Recursive call (depth 3)
|
||||
└── ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Agent Memory Files
|
||||
|
||||
#### `agents/{name}/memory.md`
|
||||
|
||||
The agent's current accumulated state:
|
||||
|
||||
```markdown
|
||||
# Agent Memory: captain
|
||||
|
||||
## Current Understanding
|
||||
|
||||
The project is implementing a REST API for user management.
|
||||
Architecture uses Express + PostgreSQL. Test coverage target is 80%.
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- 2026-01-15: Approved JWT over session tokens (simpler stateless auth)
|
||||
- 2026-01-15: Set 80% coverage threshold (balances quality vs velocity)
|
||||
|
||||
## Open Concerns
|
||||
|
||||
- Rate limiting not yet implemented on login endpoint
|
||||
- Need to verify OAuth flow works with new token format
|
||||
```
|
||||
|
||||
#### `agents/{name}/{name}-NNN.md` (Segments)
|
||||
|
||||
Historical records of each invocation, flattened in the same directory:
|
||||
|
||||
```markdown
|
||||
# Segment 001
|
||||
|
||||
timestamp: 2026-01-15T14:32:15Z
|
||||
prompt: "Review the research findings"
|
||||
|
||||
## Summary
|
||||
|
||||
- Reviewed: docs from parallel research session
|
||||
- Found: good coverage of core concepts, missing edge cases
|
||||
- Decided: proceed with implementation, note gaps for later
|
||||
- Next: review implementation against identified gaps
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Who Writes What
|
||||
|
||||
| File | Written By |
|
||||
|------|------------|
|
||||
| `state.md` | VM only |
|
||||
| `bindings/{name}.md` | Subagent |
|
||||
| `agents/{name}/memory.md` | Persistent agent |
|
||||
| `agents/{name}/{name}-NNN.md` | Persistent agent |
|
||||
|
||||
The VM orchestrates; subagents write their own outputs directly to the filesystem. **The VM never holds full binding values—it tracks file paths.**
|
||||
|
||||
---
|
||||
|
||||
## Subagent Output Writing
|
||||
|
||||
When the VM spawns a session, it tells the subagent where to write output.
|
||||
|
||||
### For Regular Sessions
|
||||
|
||||
```
|
||||
When you complete this task, write your output to:
|
||||
.prose/runs/20260115-143052-a7b3c9/bindings/research.md
|
||||
|
||||
Format:
|
||||
# research
|
||||
|
||||
kind: let
|
||||
|
||||
source:
|
||||
```prose
|
||||
let research = session: researcher
|
||||
prompt: "Research AI safety"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
[Your output here]
|
||||
```
|
||||
|
||||
### For Persistent Agents (resume:)
|
||||
|
||||
```
|
||||
Your memory is at:
|
||||
.prose/runs/20260115-143052-a7b3c9/agents/captain/memory.md
|
||||
|
||||
Read it first to understand your prior context. When done, update it
|
||||
with your compacted state following the guidelines in primitives/session.md.
|
||||
|
||||
Also write your segment record to:
|
||||
.prose/runs/20260115-143052-a7b3c9/agents/captain/captain-003.md
|
||||
```
|
||||
|
||||
### What Subagents Return to the VM
|
||||
|
||||
After writing output, the subagent returns a **confirmation message**—not the full content:
|
||||
|
||||
**Root scope (outside block invocations):**
|
||||
```
|
||||
Binding written: research
|
||||
Location: .prose/runs/20260115-143052-a7b3c9/bindings/research.md
|
||||
Summary: AI safety research covering alignment, robustness, and interpretability with 15 citations.
|
||||
```
|
||||
|
||||
**Inside block invocation (include execution_id):**
|
||||
```
|
||||
Binding written: result
|
||||
Location: .prose/runs/20260115-143052-a7b3c9/bindings/result__43.md
|
||||
Execution ID: 43
|
||||
Summary: Processed chunk into 3 sub-parts for recursive processing.
|
||||
```
|
||||
|
||||
The VM records the location and continues. It does NOT read the file—it passes the reference to subsequent sessions that need the context.
|
||||
|
||||
---
|
||||
|
||||
## Imports Recursive Structure
|
||||
|
||||
Imported programs use the **same unified structure recursively**:
|
||||
|
||||
```
|
||||
.prose/runs/{id}/imports/{handle}--{slug}/
|
||||
├── program.prose
|
||||
├── state.md
|
||||
├── bindings/
|
||||
│ └── {name}.md
|
||||
├── imports/ # Nested imports go here
|
||||
│ └── {handle2}--{slug2}/
|
||||
│ └── ...
|
||||
└── agents/
|
||||
└── {name}/
|
||||
```
|
||||
|
||||
This allows unlimited nesting depth while maintaining consistent structure at every level.
|
||||
|
||||
---
|
||||
|
||||
## Memory Scoping for Persistent Agents
|
||||
|
||||
| Scope | Declaration | Path | Lifetime |
|
||||
|-------|-------------|------|----------|
|
||||
| Execution (default) | `persist: true` | `.prose/runs/{id}/agents/{name}/` | Dies with run |
|
||||
| Project | `persist: project` | `.prose/agents/{name}/` | Survives runs in project |
|
||||
| User | `persist: user` | `~/.prose/agents/{name}/` | Survives across projects |
|
||||
| Custom | `persist: "path"` | Specified path | User-controlled |
|
||||
|
||||
---
|
||||
|
||||
## VM Update Protocol
|
||||
|
||||
After each statement completes, the VM:
|
||||
|
||||
1. **Confirms** subagent wrote its output file(s)
|
||||
2. **Updates** `state.md` with new position and annotations
|
||||
3. **Continues** to next statement
|
||||
|
||||
The VM never does compaction—that's the subagent's responsibility.
|
||||
|
||||
---
|
||||
|
||||
## Resuming Execution
|
||||
|
||||
If execution is interrupted, resume by:
|
||||
|
||||
1. Reading `.prose/runs/{id}/state.md` to find current position
|
||||
2. Loading all bindings from `bindings/`
|
||||
3. Continuing from the marked position
|
||||
|
||||
The `state.md` file contains everything needed to understand where execution stopped and what has been accomplished.
|
||||
380
extensions/open-prose/skills/prose/state/in-context.md
Normal file
380
extensions/open-prose/skills/prose/state/in-context.md
Normal file
@@ -0,0 +1,380 @@
|
||||
---
|
||||
role: in-context-state-management
|
||||
summary: |
|
||||
In-context state management using the narration protocol with text markers.
|
||||
This approach tracks execution state within the conversation history itself.
|
||||
The OpenProse VM "thinks aloud" to persist state—what you say becomes what you remember.
|
||||
see-also:
|
||||
- ../prose.md: VM execution semantics
|
||||
- filesystem.md: File-system state management (alternative approach)
|
||||
- sqlite.md: SQLite state management (experimental)
|
||||
- postgres.md: PostgreSQL state management (experimental)
|
||||
- ../primitives/session.md: Session context and compaction guidelines
|
||||
---
|
||||
|
||||
# In-Context State Management
|
||||
|
||||
This document describes how the OpenProse VM tracks execution state using **structured narration** in the conversation history. This is one of two state management approaches (the other being file-based state in `filesystem.md`).
|
||||
|
||||
## Overview
|
||||
|
||||
In-context state uses text-prefixed markers to persist state within the conversation. The VM "thinks aloud" about execution—what you say becomes what you remember.
|
||||
|
||||
**Key principle:** Your conversation history IS the VM's working memory.
|
||||
|
||||
---
|
||||
|
||||
## When to Use In-Context State
|
||||
|
||||
In-context state is appropriate for:
|
||||
|
||||
| Factor | In-Context | Use File-Based Instead |
|
||||
|--------|------------|------------------------|
|
||||
| Statement count | < 30 statements | >= 30 statements |
|
||||
| Parallel branches | < 5 concurrent | >= 5 concurrent |
|
||||
| Imported programs | 0-2 imports | >= 3 imports |
|
||||
| Nested depth | <= 2 levels | > 2 levels |
|
||||
| Expected duration | < 5 minutes | >= 5 minutes |
|
||||
|
||||
Announce your state mode at program start:
|
||||
|
||||
```
|
||||
OpenProse Program Start
|
||||
State mode: in-context (program is small, fits in context)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The Narration Protocol
|
||||
|
||||
Use text-prefixed markers for each state change:
|
||||
|
||||
| Marker | Category | Usage |
|
||||
|--------|----------|-------|
|
||||
| [Program] | Program | Start, end, definition collection |
|
||||
| [Position] | Position | Current statement being executed |
|
||||
| [Binding] | Binding | Variable assignment or update |
|
||||
| [Input] | Input | Receiving inputs from caller |
|
||||
| [Output] | Output | Producing outputs for caller |
|
||||
| [Import] | Import | Fetching and invoking imported programs |
|
||||
| [Success] | Success | Session or block completion |
|
||||
| [Warning] | Error | Failures and exceptions |
|
||||
| [Parallel] | Parallel | Entering, branch status, joining |
|
||||
| [Loop] | Loop | Iteration, condition evaluation |
|
||||
| [Pipeline] | Pipeline | Stage progress |
|
||||
| [Try] | Error handling | Try/catch/finally |
|
||||
| [Flow] | Flow | Condition evaluation results |
|
||||
| [Frame+] | Call Stack | Push new frame (block invocation) |
|
||||
| [Frame-] | Call Stack | Pop frame (block completion) |
|
||||
|
||||
---
|
||||
|
||||
## Narration Patterns by Construct
|
||||
|
||||
### Session Statements
|
||||
|
||||
```
|
||||
[Position] Executing: session "Research the topic"
|
||||
[Task tool call]
|
||||
[Success] Session complete: "Research found that..."
|
||||
[Binding] let research = <result>
|
||||
```
|
||||
|
||||
### Parallel Blocks
|
||||
|
||||
```
|
||||
[Parallel] Entering parallel block (3 branches, strategy: all)
|
||||
- security: pending
|
||||
- perf: pending
|
||||
- style: pending
|
||||
[Multiple Task calls]
|
||||
[Parallel] Parallel complete:
|
||||
- security = "No vulnerabilities found..."
|
||||
- perf = "Performance is acceptable..."
|
||||
- style = "Code follows conventions..."
|
||||
[Binding] security, perf, style bound
|
||||
```
|
||||
|
||||
### Loop Blocks
|
||||
|
||||
```
|
||||
[Loop] Starting loop until **task complete** (max: 5)
|
||||
|
||||
[Loop] Iteration 1 of max 5
|
||||
[Position] session "Work on task"
|
||||
[Success] Session complete
|
||||
[Loop] Evaluating: **task complete**
|
||||
[Flow] Not satisfied, continuing
|
||||
|
||||
[Loop] Iteration 2 of max 5
|
||||
[Position] session "Work on task"
|
||||
[Success] Session complete
|
||||
[Loop] Evaluating: **task complete**
|
||||
[Flow] Satisfied!
|
||||
|
||||
[Loop] Loop exited: condition satisfied at iteration 2
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```
|
||||
[Try] Entering try block
|
||||
[Position] session "Risky operation"
|
||||
[Warning] Session failed: connection timeout
|
||||
[Binding] err = {message: "connection timeout"}
|
||||
[Try] Executing catch block
|
||||
[Position] session "Handle error" with context: err
|
||||
[Success] Recovery complete
|
||||
[Try] Executing finally block
|
||||
[Position] session "Cleanup"
|
||||
[Success] Cleanup complete
|
||||
```
|
||||
|
||||
### Variable Bindings
|
||||
|
||||
```
|
||||
[Binding] let research = "AI safety research covers..." (mutable)
|
||||
[Binding] const config = {model: "opus"} (immutable)
|
||||
[Binding] research = "Updated research..." (reassignment, was: "AI safety...")
|
||||
```
|
||||
|
||||
### Input/Output Bindings
|
||||
|
||||
```
|
||||
[Input] Inputs received:
|
||||
topic = "quantum computing" (from caller)
|
||||
depth = "deep" (from caller)
|
||||
|
||||
[Output] output findings = "Research shows..." (will return to caller)
|
||||
[Output] output sources = ["arxiv:2401.1234", ...] (will return to caller)
|
||||
```
|
||||
|
||||
### Block Invocation and Call Stack
|
||||
|
||||
Track block invocations with frame markers:
|
||||
|
||||
```
|
||||
[Position] do process(data, 5)
|
||||
[Frame+] Entering block: process (execution_id: 1, depth: 1)
|
||||
Arguments: chunk=data, depth=5
|
||||
|
||||
[Position] session "Split into parts"
|
||||
[Task tool call]
|
||||
[Success] Session complete
|
||||
[Binding] let parts = <result> (execution_id: 1)
|
||||
|
||||
[Position] do process(parts[0], 4)
|
||||
[Frame+] Entering block: process (execution_id: 2, depth: 2)
|
||||
Arguments: chunk=parts[0], depth=4
|
||||
Parent: execution_id 1
|
||||
|
||||
[Position] session "Split into parts"
|
||||
[Task tool call]
|
||||
[Success] Session complete
|
||||
[Binding] let parts = <result> (execution_id: 2) # Shadows parent's 'parts'
|
||||
|
||||
... (continues recursively)
|
||||
|
||||
[Frame-] Exiting block: process (execution_id: 2)
|
||||
|
||||
[Position] session "Combine results"
|
||||
[Task tool call]
|
||||
[Success] Session complete
|
||||
|
||||
[Frame-] Exiting block: process (execution_id: 1)
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Each `[Frame+]` must have a matching `[Frame-]`
|
||||
- `execution_id` uniquely identifies each invocation
|
||||
- `depth` shows call stack depth (1 = first level)
|
||||
- Bindings include `(execution_id: N)` to indicate scope
|
||||
- Nested frames show `Parent: execution_id N` for the scope chain
|
||||
|
||||
### Scoped Binding Narration
|
||||
|
||||
When inside a block invocation, always include the execution_id:
|
||||
|
||||
```
|
||||
[Binding] let result = "computed value" (execution_id: 43)
|
||||
```
|
||||
|
||||
For variable resolution across scopes:
|
||||
|
||||
```
|
||||
[Binding] Resolving 'config': found in execution_id 41 (parent scope)
|
||||
```
|
||||
|
||||
### Program Imports
|
||||
|
||||
```
|
||||
[Import] Importing: @alice/research
|
||||
Fetching from: https://p.prose.md/@alice/research
|
||||
Inputs expected: [topic, depth]
|
||||
Outputs provided: [findings, sources]
|
||||
Registered as: research
|
||||
|
||||
[Import] Invoking: research(topic: "quantum computing")
|
||||
[Input] Passing inputs:
|
||||
topic = "quantum computing"
|
||||
|
||||
[... imported program execution ...]
|
||||
|
||||
[Output] Received outputs:
|
||||
findings = "Quantum computing uses..."
|
||||
sources = ["arxiv:2401.1234"]
|
||||
|
||||
[Import] Import complete: research
|
||||
[Binding] result = { findings: "...", sources: [...] }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Context Serialization
|
||||
|
||||
**In-context state passes values, not references.** This is the key difference from file-based and PostgreSQL state. The VM holds binding values directly in conversation history.
|
||||
|
||||
When passing context to sessions, format appropriately:
|
||||
|
||||
| Context Size | Strategy |
|
||||
|--------------|----------|
|
||||
| < 2000 chars | Pass verbatim |
|
||||
| 2000-8000 chars | Summarize to key points |
|
||||
| > 8000 chars | Extract essentials only |
|
||||
|
||||
**Format:**
|
||||
```
|
||||
Context provided:
|
||||
---
|
||||
research: "Key findings about AI safety..."
|
||||
analysis: "Risk assessment shows..."
|
||||
---
|
||||
```
|
||||
|
||||
**Limitation:** In-context state cannot support RLM-style "environment as variable" patterns where agents query arbitrarily large bindings. For programs with large intermediate values, use file-based or PostgreSQL state instead.
|
||||
|
||||
---
|
||||
|
||||
## Complete Execution Trace Example
|
||||
|
||||
```prose
|
||||
agent researcher:
|
||||
model: sonnet
|
||||
|
||||
let research = session: researcher
|
||||
prompt: "Research AI safety"
|
||||
|
||||
parallel:
|
||||
a = session "Analyze risk A"
|
||||
b = session "Analyze risk B"
|
||||
|
||||
loop until **analysis complete** (max: 3):
|
||||
session "Synthesize"
|
||||
context: { a, b, research }
|
||||
```
|
||||
|
||||
**Narration:**
|
||||
```
|
||||
[Program] Program Start
|
||||
Collecting definitions...
|
||||
- Agent: researcher (model: sonnet)
|
||||
|
||||
[Position] Statement 1: let research = session: researcher
|
||||
Spawning with prompt: "Research AI safety"
|
||||
Model: sonnet
|
||||
[Task tool call]
|
||||
[Success] Session complete: "AI safety research covers alignment..."
|
||||
[Binding] let research = <result>
|
||||
|
||||
[Position] Statement 2: parallel block
|
||||
[Parallel] Entering parallel (2 branches, strategy: all)
|
||||
[Task: "Analyze risk A"] [Task: "Analyze risk B"]
|
||||
[Parallel] Parallel complete:
|
||||
- a = "Risk A: potential misalignment..."
|
||||
- b = "Risk B: robustness concerns..."
|
||||
[Binding] a, b bound
|
||||
|
||||
[Position] Statement 3: loop until **analysis complete** (max: 3)
|
||||
[Loop] Starting loop
|
||||
|
||||
[Loop] Iteration 1 of max 3
|
||||
[Position] session "Synthesize" with context: {a, b, research}
|
||||
[Task with serialized context]
|
||||
[Success] Result: "Initial synthesis shows..."
|
||||
[Loop] Evaluating: **analysis complete**
|
||||
[Flow] Not satisfied (synthesis is preliminary)
|
||||
|
||||
[Loop] Iteration 2 of max 3
|
||||
[Position] session "Synthesize" with context: {a, b, research}
|
||||
[Task with serialized context]
|
||||
[Success] Result: "Comprehensive analysis complete..."
|
||||
[Loop] Evaluating: **analysis complete**
|
||||
[Flow] Satisfied!
|
||||
|
||||
[Loop] Loop exited: condition satisfied at iteration 2
|
||||
|
||||
[Program] Program Complete
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## State Categories
|
||||
|
||||
The VM must track these state categories in narration:
|
||||
|
||||
| Category | What to Track | Example |
|
||||
|----------|---------------|---------|
|
||||
| **Import Registry** | Imported programs and aliases | `research: @alice/research` |
|
||||
| **Agent Registry** | All agent definitions | `researcher: {model: sonnet, prompt: "..."}` |
|
||||
| **Block Registry** | All block definitions (hoisted) | `review: {params: [topic], body: [...]}` |
|
||||
| **Input Bindings** | Inputs received from caller | `topic = "quantum computing"` |
|
||||
| **Output Bindings** | Outputs to return to caller | `findings = "Research shows..."` |
|
||||
| **Variable Bindings** | Name -> value mapping (with execution_id) | `result = "..." (execution_id: 3)` |
|
||||
| **Variable Mutability** | Which are `let` vs `const` vs `output` | `research: let, findings: output` |
|
||||
| **Execution Position** | Current statement index | Statement 3 of 7 |
|
||||
| **Loop State** | Counter, max, condition | Iteration 2 of max 5 |
|
||||
| **Parallel State** | Branches, results, strategy | `{a: complete, b: pending}` |
|
||||
| **Error State** | Exception, retry count | Retry 2 of 3, error: "timeout" |
|
||||
| **Call Stack** | Stack of execution frames | See below |
|
||||
|
||||
### Call Stack State
|
||||
|
||||
For block invocations, track the full call stack:
|
||||
|
||||
```
|
||||
[CallStack] Current stack (depth: 3):
|
||||
execution_id: 5 | block: process | depth: 3 | status: executing
|
||||
execution_id: 3 | block: process | depth: 2 | status: waiting
|
||||
execution_id: 1 | block: process | depth: 1 | status: waiting
|
||||
```
|
||||
|
||||
Each frame tracks:
|
||||
- `execution_id`: Unique ID for this invocation
|
||||
- `block`: Name of the block
|
||||
- `depth`: Position in call stack
|
||||
- `status`: executing, waiting, or completed
|
||||
|
||||
---
|
||||
|
||||
## Independence from File-Based State
|
||||
|
||||
In-context state and file-based state (`filesystem.md`) are **independent approaches**. You choose one or the other based on program complexity.
|
||||
|
||||
- **In-context**: State lives in conversation history
|
||||
- **File-based**: State lives in `.prose/runs/{id}/`
|
||||
|
||||
They are not designed to be complementary—pick the appropriate mode at program start.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
In-context state management:
|
||||
|
||||
1. Uses **text-prefixed markers** to track state changes
|
||||
2. Persists state in **conversation history**
|
||||
3. Is appropriate for **smaller, simpler programs**
|
||||
4. Requires **consistent narration** throughout execution
|
||||
5. Makes state **visible** in the conversation itself
|
||||
|
||||
The narration protocol ensures that the VM can recover its execution state by reading its own prior messages. What you say becomes what you remember.
|
||||
875
extensions/open-prose/skills/prose/state/postgres.md
Normal file
875
extensions/open-prose/skills/prose/state/postgres.md
Normal file
@@ -0,0 +1,875 @@
|
||||
---
|
||||
role: postgres-state-management
|
||||
status: experimental
|
||||
summary: |
|
||||
PostgreSQL-based state management for OpenProse programs. This approach persists
|
||||
execution state to a PostgreSQL database, enabling true concurrent writes,
|
||||
network access, team collaboration, and high-throughput workloads.
|
||||
requires: psql CLI tool in PATH, running PostgreSQL server
|
||||
see-also:
|
||||
- ../prose.md: VM execution semantics
|
||||
- filesystem.md: File-based state (default, simpler)
|
||||
- sqlite.md: SQLite state (queryable, single-file)
|
||||
- in-context.md: In-context state (for simple programs)
|
||||
- ../primitives/session.md: Session context and compaction guidelines
|
||||
---
|
||||
|
||||
# PostgreSQL State Management (Experimental)
|
||||
|
||||
This document describes how the OpenProse VM tracks execution state using a **PostgreSQL database**. This is an experimental alternative to file-based state (`filesystem.md`), SQLite state (`sqlite.md`), and in-context state (`in-context.md`).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**Requires:**
|
||||
1. The `psql` command-line tool must be available in your PATH
|
||||
2. A running PostgreSQL server (local, Docker, or cloud)
|
||||
|
||||
### Installing psql
|
||||
|
||||
| Platform | Command | Notes |
|
||||
|----------|---------|-------|
|
||||
| macOS (Homebrew) | `brew install libpq && brew link --force libpq` | Client-only; no server |
|
||||
| macOS (Postgres.app) | Download from https://postgresapp.com | Full install with GUI |
|
||||
| Debian/Ubuntu | `apt install postgresql-client` | Client-only |
|
||||
| Fedora/RHEL | `dnf install postgresql` | Client-only |
|
||||
| Arch Linux | `pacman -S postgresql-libs` | Client-only |
|
||||
| Windows | `winget install PostgreSQL.PostgreSQL` | Full installer |
|
||||
|
||||
After installation, verify:
|
||||
|
||||
```bash
|
||||
psql --version # Should output: psql (PostgreSQL) 16.x
|
||||
```
|
||||
|
||||
If `psql` is not available, the VM will offer to fall back to SQLite state.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
PostgreSQL state provides:
|
||||
|
||||
- **True concurrent writes**: Row-level locking allows parallel branches to write simultaneously
|
||||
- **Network access**: Query state from any machine, external tools, or dashboards
|
||||
- **Team collaboration**: Multiple developers can share run state
|
||||
- **Rich SQL**: JSONB queries, window functions, CTEs for complex state analysis
|
||||
- **High throughput**: Handle 1000+ writes/minute, multi-GB outputs
|
||||
- **Durability**: WAL-based recovery, point-in-time restore
|
||||
|
||||
**Key principle:** The database is a flexible, shared workspace. The VM and subagents coordinate through it, and external tools can observe and query execution state in real-time.
|
||||
|
||||
---
|
||||
|
||||
## Security Warning
|
||||
|
||||
**⚠️ Credentials are visible to subagents.** The `OPENPROSE_POSTGRES_URL` connection string is passed to spawned sessions so they can write their outputs. This means:
|
||||
|
||||
- Database credentials appear in subagent context and may be logged
|
||||
- Treat these credentials as **non-sensitive**
|
||||
- Use a **dedicated database** for OpenProse, not your production systems
|
||||
- Create a **limited-privilege user** with access only to the `openprose` schema
|
||||
|
||||
**Recommended setup:**
|
||||
```sql
|
||||
-- Create dedicated user with minimal privileges
|
||||
CREATE USER openprose_agent WITH PASSWORD 'changeme';
|
||||
CREATE SCHEMA openprose AUTHORIZATION openprose_agent;
|
||||
GRANT ALL ON SCHEMA openprose TO openprose_agent;
|
||||
-- User can only access the openprose schema, nothing else
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## When to Use PostgreSQL State
|
||||
|
||||
PostgreSQL state is for **power users** with specific scale or collaboration needs:
|
||||
|
||||
| Need | PostgreSQL Helps |
|
||||
|------|------------------|
|
||||
| >5 parallel branches writing simultaneously | SQLite locks; PostgreSQL doesn't |
|
||||
| External dashboards querying state | PostgreSQL is designed for concurrent readers |
|
||||
| Team collaboration on long workflows | Shared network access; no file sync needed |
|
||||
| Outputs exceeding 1GB | Bulk ingestion; no single-file bottleneck |
|
||||
| Mission-critical workflows (hours/days) | Robust durability; point-in-time recovery |
|
||||
|
||||
**If none of these apply, use filesystem or SQLite state.** They're simpler and sufficient for 99% of programs.
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```
|
||||
Is your program <30 statements with no parallel blocks?
|
||||
YES -> Use in-context state (zero friction)
|
||||
NO -> Continue...
|
||||
|
||||
Do external tools (dashboards, monitoring, analytics) need to query state?
|
||||
YES -> Use PostgreSQL (network access required)
|
||||
NO -> Continue...
|
||||
|
||||
Do multiple machines or team members need shared access to the same run?
|
||||
YES -> Use PostgreSQL (collaboration)
|
||||
NO -> Continue...
|
||||
|
||||
Do you have >5 concurrent parallel branches writing simultaneously?
|
||||
YES -> Use PostgreSQL (concurrency)
|
||||
NO -> Continue...
|
||||
|
||||
Will outputs exceed 1GB or writes exceed 100/minute?
|
||||
YES -> Use PostgreSQL (scale)
|
||||
NO -> Use filesystem (default) or SQLite (if you want SQL queries)
|
||||
```
|
||||
|
||||
### The Concurrency Case
|
||||
|
||||
The primary motivation for PostgreSQL is **concurrent writes in parallel execution**:
|
||||
|
||||
- SQLite uses table-level locks: parallel branches serialize
|
||||
- PostgreSQL uses row-level locks: parallel branches write simultaneously
|
||||
|
||||
If your program has 10 parallel branches completing at once, PostgreSQL will be 5-10x faster than SQLite for the write phase.
|
||||
|
||||
---
|
||||
|
||||
## Database Setup
|
||||
|
||||
### Option 1: Docker (Recommended)
|
||||
|
||||
The fastest path to a running PostgreSQL instance:
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name prose-pg \
|
||||
-e POSTGRES_DB=prose \
|
||||
-e POSTGRES_HOST_AUTH_METHOD=trust \
|
||||
-p 5432:5432 \
|
||||
postgres:16
|
||||
```
|
||||
|
||||
Then configure the connection:
|
||||
|
||||
```bash
|
||||
mkdir -p .prose
|
||||
echo "OPENPROSE_POSTGRES_URL=postgresql://postgres@localhost:5432/prose" > .prose/.env
|
||||
```
|
||||
|
||||
Management commands:
|
||||
|
||||
```bash
|
||||
docker ps | grep prose-pg # Check if running
|
||||
docker logs prose-pg # View logs
|
||||
docker stop prose-pg # Stop
|
||||
docker start prose-pg # Start again
|
||||
docker rm -f prose-pg # Remove completely
|
||||
```
|
||||
|
||||
### Option 2: Local PostgreSQL
|
||||
|
||||
For users who prefer native PostgreSQL:
|
||||
|
||||
**macOS (Homebrew):**
|
||||
|
||||
```bash
|
||||
brew install postgresql@16
|
||||
brew services start postgresql@16
|
||||
createdb myproject
|
||||
echo "OPENPROSE_POSTGRES_URL=postgresql://localhost/myproject" >> .prose/.env
|
||||
```
|
||||
|
||||
**Linux (Debian/Ubuntu):**
|
||||
|
||||
```bash
|
||||
sudo apt install postgresql
|
||||
sudo systemctl start postgresql
|
||||
sudo -u postgres createdb myproject
|
||||
echo "OPENPROSE_POSTGRES_URL=postgresql:///myproject" >> .prose/.env
|
||||
```
|
||||
|
||||
### Option 3: Cloud PostgreSQL
|
||||
|
||||
For team collaboration or production:
|
||||
|
||||
| Provider | Free Tier | Cold Start | Best For |
|
||||
|----------|-----------|------------|----------|
|
||||
| **Neon** | 0.5GB, auto-suspend | 1-3s | Development, testing |
|
||||
| **Supabase** | 500MB, no auto-suspend | None | Projects needing auth/storage |
|
||||
| **Railway** | $5/mo credit | None | Simple production deploys |
|
||||
|
||||
```bash
|
||||
# Example: Neon
|
||||
echo "OPENPROSE_POSTGRES_URL=postgresql://user:pass@ep-name.us-east-2.aws.neon.tech/neondb?sslmode=require" >> .prose/.env
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Location
|
||||
|
||||
The connection string is stored in `.prose/.env`:
|
||||
|
||||
```
|
||||
your-project/
|
||||
├── .prose/
|
||||
│ ├── .env # OPENPROSE_POSTGRES_URL=...
|
||||
│ └── runs/ # Execution metadata and attachments
|
||||
│ └── {YYYYMMDD}-{HHMMSS}-{random}/
|
||||
│ ├── program.prose # Copy of running program
|
||||
│ └── attachments/ # Large outputs (optional)
|
||||
├── .gitignore # Should exclude .prose/.env
|
||||
└── your-program.prose
|
||||
```
|
||||
|
||||
**Run ID format:** `{YYYYMMDD}-{HHMMSS}-{random6}`
|
||||
|
||||
Example: `20260116-143052-a7b3c9`
|
||||
|
||||
### Environment Variable Precedence
|
||||
|
||||
The VM checks in this order:
|
||||
|
||||
1. `OPENPROSE_POSTGRES_URL` in `.prose/.env`
|
||||
2. `OPENPROSE_POSTGRES_URL` in shell environment
|
||||
3. `DATABASE_URL` in shell environment (common fallback)
|
||||
|
||||
### Security: Add to .gitignore
|
||||
|
||||
```gitignore
|
||||
# OpenProse sensitive files
|
||||
.prose/.env
|
||||
.prose/runs/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Responsibility Separation
|
||||
|
||||
This section defines **who does what**. This is the contract between the VM and subagents.
|
||||
|
||||
### VM Responsibilities
|
||||
|
||||
The VM (the orchestrating agent running the .prose program) is responsible for:
|
||||
|
||||
| Responsibility | Description |
|
||||
|----------------|-------------|
|
||||
| **Schema initialization** | Create `openprose` schema and tables at run start |
|
||||
| **Run registration** | Store the program source and metadata |
|
||||
| **Execution tracking** | Update position, status, and timing as statements execute |
|
||||
| **Subagent spawning** | Spawn sessions via Task tool with database instructions |
|
||||
| **Parallel coordination** | Track branch status, implement join strategies |
|
||||
| **Loop management** | Track iteration counts, evaluate conditions |
|
||||
| **Error aggregation** | Record failures, manage retry state |
|
||||
| **Context preservation** | Maintain sufficient narration in the main thread |
|
||||
| **Completion detection** | Mark the run as complete when finished |
|
||||
|
||||
**Critical:** The VM must preserve enough context in its own conversation to understand execution state without re-reading the entire database. The database is for coordination and persistence, not a replacement for working memory.
|
||||
|
||||
### Subagent Responsibilities
|
||||
|
||||
Subagents (sessions spawned by the VM) are responsible for:
|
||||
|
||||
| Responsibility | Description |
|
||||
|----------------|-------------|
|
||||
| **Writing own outputs** | Insert/update their binding in the `bindings` table |
|
||||
| **Memory management** | For persistent agents: read and update their memory record |
|
||||
| **Segment recording** | For persistent agents: append segment history |
|
||||
| **Attachment handling** | Write large outputs to `attachments/` directory, store path in DB |
|
||||
| **Atomic writes** | Use transactions when updating multiple related records |
|
||||
|
||||
**Critical:** Subagents write ONLY to `bindings`, `agents`, and `agent_segments` tables. The VM owns the `execution` table entirely. Completion signaling happens through the substrate (Task tool return), not database updates.
|
||||
|
||||
**Critical:** Subagents must write their outputs directly to the database. The VM does not write subagent outputs—it only reads them after the subagent completes.
|
||||
|
||||
**What subagents return to the VM:** A confirmation message with the binding location—not the full content:
|
||||
|
||||
**Root scope:**
|
||||
```
|
||||
Binding written: research
|
||||
Location: openprose.bindings WHERE name='research' AND run_id='20260116-143052-a7b3c9' AND execution_id IS NULL
|
||||
Summary: AI safety research covering alignment, robustness, and interpretability with 15 citations.
|
||||
```
|
||||
|
||||
**Inside block invocation:**
|
||||
```
|
||||
Binding written: result
|
||||
Location: openprose.bindings WHERE name='result' AND run_id='20260116-143052-a7b3c9' AND execution_id=43
|
||||
Execution ID: 43
|
||||
Summary: Processed chunk into 3 sub-parts for recursive processing.
|
||||
```
|
||||
|
||||
The VM tracks locations, not values. This keeps the VM's context lean and enables arbitrarily large intermediate values.
|
||||
|
||||
### Shared Concerns
|
||||
|
||||
| Concern | Who Handles |
|
||||
|---------|-------------|
|
||||
| Schema evolution | Either (use `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE` as needed) |
|
||||
| Custom tables | Either (prefix with `x_` for extensions) |
|
||||
| Indexing | Either (add indexes for frequently-queried columns) |
|
||||
| Cleanup | VM (at run end, optionally delete old data) |
|
||||
|
||||
---
|
||||
|
||||
## Core Schema
|
||||
|
||||
The VM initializes these tables using the `openprose` schema. This is a **minimum viable schema**—extend freely.
|
||||
|
||||
```sql
|
||||
-- Create dedicated schema for OpenProse state
|
||||
CREATE SCHEMA IF NOT EXISTS openprose;
|
||||
|
||||
-- Run metadata
|
||||
CREATE TABLE IF NOT EXISTS openprose.run (
|
||||
id TEXT PRIMARY KEY,
|
||||
program_path TEXT,
|
||||
program_source TEXT,
|
||||
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
status TEXT NOT NULL DEFAULT 'running'
|
||||
CHECK (status IN ('running', 'completed', 'failed', 'interrupted')),
|
||||
state_mode TEXT NOT NULL DEFAULT 'postgres',
|
||||
metadata JSONB DEFAULT '{}'::jsonb
|
||||
);
|
||||
|
||||
-- Execution position and history
|
||||
CREATE TABLE IF NOT EXISTS openprose.execution (
|
||||
id SERIAL PRIMARY KEY,
|
||||
run_id TEXT NOT NULL REFERENCES openprose.run(id) ON DELETE CASCADE,
|
||||
statement_index INTEGER NOT NULL,
|
||||
statement_text TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'executing', 'completed', 'failed', 'skipped')),
|
||||
started_at TIMESTAMPTZ,
|
||||
completed_at TIMESTAMPTZ,
|
||||
error_message TEXT,
|
||||
parent_id INTEGER REFERENCES openprose.execution(id) ON DELETE CASCADE,
|
||||
metadata JSONB DEFAULT '{}'::jsonb
|
||||
);
|
||||
|
||||
-- All named values (input, output, let, const)
|
||||
CREATE TABLE IF NOT EXISTS openprose.bindings (
|
||||
name TEXT NOT NULL,
|
||||
run_id TEXT NOT NULL REFERENCES openprose.run(id) ON DELETE CASCADE,
|
||||
execution_id INTEGER, -- NULL for root scope, non-null for block invocations
|
||||
kind TEXT NOT NULL CHECK (kind IN ('input', 'output', 'let', 'const')),
|
||||
value TEXT,
|
||||
source_statement TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
attachment_path TEXT,
|
||||
metadata JSONB DEFAULT '{}'::jsonb,
|
||||
PRIMARY KEY (name, run_id, COALESCE(execution_id, -1)) -- Composite key with scope
|
||||
);
|
||||
|
||||
-- Persistent agent memory
|
||||
CREATE TABLE IF NOT EXISTS openprose.agents (
|
||||
name TEXT NOT NULL,
|
||||
run_id TEXT, -- NULL for project-scoped and user-scoped agents
|
||||
scope TEXT NOT NULL CHECK (scope IN ('execution', 'project', 'user', 'custom')),
|
||||
memory TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
metadata JSONB DEFAULT '{}'::jsonb,
|
||||
PRIMARY KEY (name, COALESCE(run_id, '__project__'))
|
||||
);
|
||||
|
||||
-- Agent invocation history
|
||||
CREATE TABLE IF NOT EXISTS openprose.agent_segments (
|
||||
id SERIAL PRIMARY KEY,
|
||||
agent_name TEXT NOT NULL,
|
||||
run_id TEXT, -- NULL for project-scoped agents
|
||||
segment_number INTEGER NOT NULL,
|
||||
timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
prompt TEXT,
|
||||
summary TEXT,
|
||||
metadata JSONB DEFAULT '{}'::jsonb,
|
||||
UNIQUE (agent_name, COALESCE(run_id, '__project__'), segment_number)
|
||||
);
|
||||
|
||||
-- Import registry
|
||||
CREATE TABLE IF NOT EXISTS openprose.imports (
|
||||
alias TEXT NOT NULL,
|
||||
run_id TEXT NOT NULL REFERENCES openprose.run(id) ON DELETE CASCADE,
|
||||
source_url TEXT NOT NULL,
|
||||
fetched_at TIMESTAMPTZ,
|
||||
inputs_schema JSONB,
|
||||
outputs_schema JSONB,
|
||||
content_hash TEXT,
|
||||
metadata JSONB DEFAULT '{}'::jsonb,
|
||||
PRIMARY KEY (alias, run_id)
|
||||
);
|
||||
|
||||
-- Indexes for common queries
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_run_id ON openprose.execution(run_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_status ON openprose.execution(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_parent_id ON openprose.execution(parent_id) WHERE parent_id IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_metadata_gin ON openprose.execution USING GIN (metadata jsonb_path_ops);
|
||||
CREATE INDEX IF NOT EXISTS idx_bindings_run_id ON openprose.bindings(run_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_bindings_execution_id ON openprose.bindings(execution_id) WHERE execution_id IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_agents_run_id ON openprose.agents(run_id) WHERE run_id IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_agents_project_scoped ON openprose.agents(name) WHERE run_id IS NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_segments_lookup ON openprose.agent_segments(agent_name, run_id);
|
||||
```
|
||||
|
||||
### Schema Conventions
|
||||
|
||||
- **Timestamps**: Use `TIMESTAMPTZ` with `NOW()` (timezone-aware)
|
||||
- **JSON fields**: Use `JSONB` for structured data in `metadata` columns (queryable, indexable)
|
||||
- **Large values**: If a binding value exceeds ~100KB, write to `attachments/{name}.md` and store path
|
||||
- **Extension tables**: Prefix with `x_` (e.g., `x_metrics`, `x_audit_log`)
|
||||
- **Anonymous bindings**: Sessions without explicit capture use auto-generated names: `anon_001`, `anon_002`, etc.
|
||||
- **Import bindings**: Prefix with import alias for scoping: `research.findings`, `research.sources`
|
||||
- **Scoped bindings**: Use `execution_id` column—NULL for root scope, non-null for block invocations
|
||||
|
||||
### Scope Resolution Query
|
||||
|
||||
For recursive blocks, bindings are scoped to their execution frame. Resolve variables by walking up the call stack:
|
||||
|
||||
```sql
|
||||
-- Find binding 'result' starting from execution_id 43 in run '20260116-143052-a7b3c9'
|
||||
WITH RECURSIVE scope_chain AS (
|
||||
-- Start with current execution
|
||||
SELECT id, parent_id FROM openprose.execution WHERE id = 43
|
||||
UNION ALL
|
||||
-- Walk up to parent
|
||||
SELECT e.id, e.parent_id
|
||||
FROM openprose.execution e
|
||||
JOIN scope_chain s ON e.id = s.parent_id
|
||||
)
|
||||
SELECT b.* FROM openprose.bindings b
|
||||
WHERE b.name = 'result'
|
||||
AND b.run_id = '20260116-143052-a7b3c9'
|
||||
AND (b.execution_id IN (SELECT id FROM scope_chain) OR b.execution_id IS NULL)
|
||||
ORDER BY
|
||||
CASE WHEN b.execution_id IS NULL THEN 1 ELSE 0 END, -- Prefer scoped over root
|
||||
b.execution_id DESC NULLS LAST -- Prefer deeper (more local) scope
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
**Simpler version if you know the scope chain:**
|
||||
|
||||
```sql
|
||||
-- Direct lookup: check current scope (43), then parent (42), then root (NULL)
|
||||
SELECT * FROM openprose.bindings
|
||||
WHERE name = 'result'
|
||||
AND run_id = '20260116-143052-a7b3c9'
|
||||
AND (execution_id = 43 OR execution_id = 42 OR execution_id IS NULL)
|
||||
ORDER BY execution_id DESC NULLS LAST
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Interaction
|
||||
|
||||
Both VM and subagents interact via the `psql` CLI.
|
||||
|
||||
### From the VM
|
||||
|
||||
```bash
|
||||
# Initialize schema
|
||||
psql "$OPENPROSE_POSTGRES_URL" -f schema.sql
|
||||
|
||||
# Register a new run
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "
|
||||
INSERT INTO openprose.run (id, program_path, program_source, status)
|
||||
VALUES ('20260116-143052-a7b3c9', '/path/to/program.prose', 'program source...', 'running')
|
||||
"
|
||||
|
||||
# Update execution position
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "
|
||||
INSERT INTO openprose.execution (run_id, statement_index, statement_text, status, started_at)
|
||||
VALUES ('20260116-143052-a7b3c9', 3, 'session \"Research AI safety\"', 'executing', NOW())
|
||||
"
|
||||
|
||||
# Read a binding
|
||||
psql "$OPENPROSE_POSTGRES_URL" -t -A -c "
|
||||
SELECT value FROM openprose.bindings WHERE name = 'research' AND run_id = '20260116-143052-a7b3c9'
|
||||
"
|
||||
|
||||
# Check parallel branch status
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "
|
||||
SELECT metadata->>'branch' AS branch, status FROM openprose.execution
|
||||
WHERE run_id = '20260116-143052-a7b3c9' AND metadata->>'parallel_id' = 'p1'
|
||||
"
|
||||
```
|
||||
|
||||
### From Subagents
|
||||
|
||||
The VM provides the database path and instructions when spawning:
|
||||
|
||||
**Root scope (outside block invocations):**
|
||||
|
||||
```
|
||||
Your output goes to PostgreSQL state.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Connection | `postgresql://user:***@host:5432/db` |
|
||||
| Schema | `openprose` |
|
||||
| Run ID | `20260116-143052-a7b3c9` |
|
||||
| Binding | `research` |
|
||||
| Execution ID | (root scope) |
|
||||
|
||||
When complete, write your output:
|
||||
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "
|
||||
INSERT INTO openprose.bindings (name, run_id, execution_id, kind, value, source_statement)
|
||||
VALUES (
|
||||
'research',
|
||||
'20260116-143052-a7b3c9',
|
||||
NULL, -- root scope
|
||||
'let',
|
||||
E'AI safety research covers alignment, robustness...',
|
||||
'let research = session: researcher'
|
||||
)
|
||||
ON CONFLICT (name, run_id, COALESCE(execution_id, -1)) DO UPDATE
|
||||
SET value = EXCLUDED.value, updated_at = NOW()
|
||||
"
|
||||
```
|
||||
|
||||
**Inside block invocation (include execution_id):**
|
||||
|
||||
```
|
||||
Your output goes to PostgreSQL state.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Connection | `postgresql://user:***@host:5432/db` |
|
||||
| Schema | `openprose` |
|
||||
| Run ID | `20260116-143052-a7b3c9` |
|
||||
| Binding | `result` |
|
||||
| Execution ID | `43` |
|
||||
| Block | `process` |
|
||||
| Depth | `3` |
|
||||
|
||||
When complete, write your output:
|
||||
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "
|
||||
INSERT INTO openprose.bindings (name, run_id, execution_id, kind, value, source_statement)
|
||||
VALUES (
|
||||
'result',
|
||||
'20260116-143052-a7b3c9',
|
||||
43, -- scoped to this execution
|
||||
'let',
|
||||
E'Processed chunk into 3 sub-parts...',
|
||||
'let result = session \"Process chunk\"'
|
||||
)
|
||||
ON CONFLICT (name, run_id, COALESCE(execution_id, -1)) DO UPDATE
|
||||
SET value = EXCLUDED.value, updated_at = NOW()
|
||||
"
|
||||
```
|
||||
|
||||
For persistent agents (execution-scoped):
|
||||
|
||||
```
|
||||
Your memory is in the database:
|
||||
|
||||
Read your current state:
|
||||
psql "$OPENPROSE_POSTGRES_URL" -t -A -c "SELECT memory FROM openprose.agents WHERE name = 'captain' AND run_id = '20260116-143052-a7b3c9'"
|
||||
|
||||
Update when done:
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "UPDATE openprose.agents SET memory = '...', updated_at = NOW() WHERE name = 'captain' AND run_id = '20260116-143052-a7b3c9'"
|
||||
|
||||
Record this segment:
|
||||
psql "$OPENPROSE_POSTGRES_URL" -c "INSERT INTO openprose.agent_segments (agent_name, run_id, segment_number, prompt, summary) VALUES ('captain', '20260116-143052-a7b3c9', 3, '...', '...')"
|
||||
```
|
||||
|
||||
For project-scoped agents, use `run_id IS NULL` in queries:
|
||||
|
||||
```sql
|
||||
-- Read project-scoped agent memory
|
||||
SELECT memory FROM openprose.agents WHERE name = 'advisor' AND run_id IS NULL;
|
||||
|
||||
-- Update project-scoped agent memory
|
||||
UPDATE openprose.agents SET memory = '...' WHERE name = 'advisor' AND run_id IS NULL;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Context Preservation in Main Thread
|
||||
|
||||
**This is critical.** The database is for persistence and coordination, but the VM must still maintain conversational context.
|
||||
|
||||
### What the VM Must Narrate
|
||||
|
||||
Even with PostgreSQL state, the VM should narrate key events in its conversation:
|
||||
|
||||
```
|
||||
[Position] Statement 3: let research = session: researcher
|
||||
Spawning session, will write to state database
|
||||
[Task tool call]
|
||||
[Success] Session complete, binding written to DB
|
||||
[Binding] research = <stored in openprose.bindings>
|
||||
```
|
||||
|
||||
### Why Both?
|
||||
|
||||
| Purpose | Mechanism |
|
||||
|---------|-----------|
|
||||
| **Working memory** | Conversation narration (what the VM "remembers" without re-querying) |
|
||||
| **Durable state** | PostgreSQL database (survives context limits, enables resumption) |
|
||||
| **Subagent coordination** | PostgreSQL database (shared access point) |
|
||||
| **Debugging/inspection** | PostgreSQL database (queryable history) |
|
||||
|
||||
The narration is the VM's "mental model" of execution. The database is the "source of truth" for resumption and inspection.
|
||||
|
||||
---
|
||||
|
||||
## Parallel Execution
|
||||
|
||||
For parallel blocks, the VM uses the `metadata` JSONB field to track branches. **Only the VM writes to the `execution` table.**
|
||||
|
||||
```sql
|
||||
-- VM marks parallel start
|
||||
INSERT INTO openprose.execution (run_id, statement_index, statement_text, status, started_at, metadata)
|
||||
VALUES ('20260116-143052-a7b3c9', 5, 'parallel:', 'executing', NOW(),
|
||||
'{"parallel_id": "p1", "strategy": "all", "branches": ["a", "b", "c"]}'::jsonb)
|
||||
RETURNING id; -- Save as parent_id (e.g., 42)
|
||||
|
||||
-- VM creates execution record for each branch
|
||||
INSERT INTO openprose.execution (run_id, statement_index, statement_text, status, started_at, parent_id, metadata)
|
||||
VALUES
|
||||
('20260116-143052-a7b3c9', 6, 'a = session "Task A"', 'executing', NOW(), 42, '{"parallel_id": "p1", "branch": "a"}'::jsonb),
|
||||
('20260116-143052-a7b3c9', 7, 'b = session "Task B"', 'executing', NOW(), 42, '{"parallel_id": "p1", "branch": "b"}'::jsonb),
|
||||
('20260116-143052-a7b3c9', 8, 'c = session "Task C"', 'executing', NOW(), 42, '{"parallel_id": "p1", "branch": "c"}'::jsonb);
|
||||
|
||||
-- Subagents write their outputs to bindings table (see "From Subagents" section)
|
||||
-- Task tool signals completion to VM via substrate
|
||||
|
||||
-- VM marks branch complete after Task returns
|
||||
UPDATE openprose.execution SET status = 'completed', completed_at = NOW()
|
||||
WHERE run_id = '20260116-143052-a7b3c9' AND metadata->>'parallel_id' = 'p1' AND metadata->>'branch' = 'a';
|
||||
|
||||
-- VM checks if all branches complete
|
||||
SELECT COUNT(*) AS pending FROM openprose.execution
|
||||
WHERE run_id = '20260116-143052-a7b3c9'
|
||||
AND metadata->>'parallel_id' = 'p1'
|
||||
AND parent_id IS NOT NULL
|
||||
AND status NOT IN ('completed', 'failed', 'skipped');
|
||||
```
|
||||
|
||||
### The Concurrency Advantage
|
||||
|
||||
Each subagent writes to a different row in `openprose.bindings`. PostgreSQL's row-level locking means **no blocking**:
|
||||
|
||||
```
|
||||
SQLite (table locks):
|
||||
Branch 1 writes -------|
|
||||
Branch 2 waits ------|
|
||||
Branch 3 waits -----|
|
||||
Total time: 3 * write_time (serialized)
|
||||
|
||||
PostgreSQL (row locks):
|
||||
Branch 1 writes --|
|
||||
Branch 2 writes --| (concurrent)
|
||||
Branch 3 writes --|
|
||||
Total time: ~1 * write_time (parallel)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Loop Tracking
|
||||
|
||||
```sql
|
||||
-- Loop metadata tracks iteration state
|
||||
INSERT INTO openprose.execution (run_id, statement_index, statement_text, status, started_at, metadata)
|
||||
VALUES ('20260116-143052-a7b3c9', 10, 'loop until **analysis complete** (max: 5):', 'executing', NOW(),
|
||||
'{"loop_id": "l1", "max_iterations": 5, "current_iteration": 0, "condition": "**analysis complete**"}'::jsonb);
|
||||
|
||||
-- Update iteration
|
||||
UPDATE openprose.execution
|
||||
SET metadata = jsonb_set(metadata, '{current_iteration}', '2')
|
||||
WHERE run_id = '20260116-143052-a7b3c9' AND metadata->>'loop_id' = 'l1' AND parent_id IS NULL;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
```sql
|
||||
-- Record failure
|
||||
UPDATE openprose.execution
|
||||
SET status = 'failed',
|
||||
error_message = 'Connection timeout after 30s',
|
||||
completed_at = NOW()
|
||||
WHERE id = 15;
|
||||
|
||||
-- Track retry attempts in metadata
|
||||
UPDATE openprose.execution
|
||||
SET metadata = jsonb_set(jsonb_set(metadata, '{retry_attempt}', '2'), '{max_retries}', '3')
|
||||
WHERE id = 15;
|
||||
|
||||
-- Mark run as failed
|
||||
UPDATE openprose.run SET status = 'failed' WHERE id = '20260116-143052-a7b3c9';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Project-Scoped and User-Scoped Agents
|
||||
|
||||
Execution-scoped agents (the default) use `run_id = specific value`. **Project-scoped agents** (`persist: project`) and **user-scoped agents** (`persist: user`) use `run_id IS NULL` and survive across runs.
|
||||
|
||||
For user-scoped agents, the VM maintains a separate connection or uses a naming convention to distinguish them from project-scoped agents. One approach is to prefix user-scoped agent names with `__user__` in the same database, or use a separate user-level database configured via `OPENPROSE_POSTGRES_USER_URL`.
|
||||
|
||||
### The run_id Approach
|
||||
|
||||
The `COALESCE` trick in the primary key allows both scopes in one table:
|
||||
|
||||
```sql
|
||||
PRIMARY KEY (name, COALESCE(run_id, '__project__'))
|
||||
```
|
||||
|
||||
This means:
|
||||
- `name='advisor', run_id=NULL` has PK `('advisor', '__project__')`
|
||||
- `name='advisor', run_id='20260116-143052-a7b3c9'` has PK `('advisor', '20260116-143052-a7b3c9')`
|
||||
|
||||
The same agent name can exist as both project-scoped and execution-scoped without collision.
|
||||
|
||||
### Query Patterns
|
||||
|
||||
| Scope | Query |
|
||||
|-------|-------|
|
||||
| Execution-scoped | `WHERE name = 'captain' AND run_id = '{RUN_ID}'` |
|
||||
| Project-scoped | `WHERE name = 'advisor' AND run_id IS NULL` |
|
||||
|
||||
### Project-Scoped Memory Guidelines
|
||||
|
||||
Project-scoped agents should store generalizable knowledge that accumulates:
|
||||
|
||||
**DO store:** User preferences, project context, learned patterns, decision rationale
|
||||
**DO NOT store:** Run-specific details, time-sensitive information, large data
|
||||
|
||||
### Agent Cleanup
|
||||
|
||||
- **Execution-scoped:** Can be deleted when run completes or after retention period
|
||||
- **Project-scoped:** Only deleted on explicit user request
|
||||
|
||||
```sql
|
||||
-- Delete execution-scoped agents for a completed run
|
||||
DELETE FROM openprose.agents WHERE run_id = '20260116-143052-a7b3c9';
|
||||
|
||||
-- Delete a specific project-scoped agent (user-initiated)
|
||||
DELETE FROM openprose.agents WHERE name = 'old_advisor' AND run_id IS NULL;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Large Outputs
|
||||
|
||||
When a binding value is too large for comfortable database storage (>100KB):
|
||||
|
||||
1. Write content to `attachments/{binding_name}.md`
|
||||
2. Store the path in the `attachment_path` column
|
||||
3. Leave `value` as a summary
|
||||
|
||||
```sql
|
||||
INSERT INTO openprose.bindings (name, run_id, kind, value, attachment_path, source_statement)
|
||||
VALUES (
|
||||
'full_report',
|
||||
'20260116-143052-a7b3c9',
|
||||
'let',
|
||||
'Full analysis report (847KB) - see attachment',
|
||||
'attachments/full_report.md',
|
||||
'let full_report = session "Generate comprehensive report"'
|
||||
)
|
||||
ON CONFLICT (name, run_id) DO UPDATE
|
||||
SET value = EXCLUDED.value, attachment_path = EXCLUDED.attachment_path, updated_at = NOW();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resuming Execution
|
||||
|
||||
To resume an interrupted run:
|
||||
|
||||
```sql
|
||||
-- Find current position
|
||||
SELECT statement_index, statement_text, status
|
||||
FROM openprose.execution
|
||||
WHERE run_id = '20260116-143052-a7b3c9' AND status = 'executing'
|
||||
ORDER BY id DESC LIMIT 1;
|
||||
|
||||
-- Get all completed bindings
|
||||
SELECT name, kind, value, attachment_path FROM openprose.bindings
|
||||
WHERE run_id = '20260116-143052-a7b3c9';
|
||||
|
||||
-- Get agent memory states
|
||||
SELECT name, scope, memory FROM openprose.agents
|
||||
WHERE run_id = '20260116-143052-a7b3c9' OR run_id IS NULL;
|
||||
|
||||
-- Check parallel block status
|
||||
SELECT metadata->>'branch' AS branch, status
|
||||
FROM openprose.execution
|
||||
WHERE run_id = '20260116-143052-a7b3c9'
|
||||
AND metadata->>'parallel_id' IS NOT NULL
|
||||
AND parent_id IS NOT NULL;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Flexibility Encouragement
|
||||
|
||||
PostgreSQL state is intentionally **flexible**. The core schema is a starting point. You are encouraged to:
|
||||
|
||||
- **Add columns** to existing tables as needed
|
||||
- **Create extension tables** (prefix with `x_`)
|
||||
- **Store custom metrics** (timing, token counts, model info)
|
||||
- **Build indexes** for your query patterns
|
||||
- **Use JSONB operators** for semi-structured data queries
|
||||
|
||||
Example extensions:
|
||||
|
||||
```sql
|
||||
-- Custom metrics table
|
||||
CREATE TABLE IF NOT EXISTS openprose.x_metrics (
|
||||
id SERIAL PRIMARY KEY,
|
||||
run_id TEXT REFERENCES openprose.run(id) ON DELETE CASCADE,
|
||||
execution_id INTEGER REFERENCES openprose.execution(id) ON DELETE CASCADE,
|
||||
metric_name TEXT NOT NULL,
|
||||
metric_value NUMERIC,
|
||||
recorded_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
metadata JSONB DEFAULT '{}'::jsonb
|
||||
);
|
||||
|
||||
-- Add custom column
|
||||
ALTER TABLE openprose.bindings ADD COLUMN IF NOT EXISTS token_count INTEGER;
|
||||
|
||||
-- Create index for common query
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_bindings_created ON openprose.bindings(created_at);
|
||||
```
|
||||
|
||||
The database is your workspace. Use it.
|
||||
|
||||
---
|
||||
|
||||
## Comparison with Other Modes
|
||||
|
||||
| Aspect | filesystem.md | in-context.md | sqlite.md | postgres.md |
|
||||
|--------|---------------|---------------|-----------|-------------|
|
||||
| **State location** | `.prose/runs/{id}/` files | Conversation history | `.prose/runs/{id}/state.db` | PostgreSQL database |
|
||||
| **Queryable** | Via file reads | No | Yes (SQL) | Yes (SQL) |
|
||||
| **Atomic updates** | No | N/A | Yes (transactions) | Yes (ACID) |
|
||||
| **Concurrent writes** | Yes (different files) | N/A | **No (table locks)** | **Yes (row locks)** |
|
||||
| **Network access** | No | No | No | **Yes** |
|
||||
| **Team collaboration** | Via file sync | No | Via file sync | **Yes** |
|
||||
| **Schema flexibility** | Rigid file structure | N/A | Flexible | Very flexible (JSONB) |
|
||||
| **Resumption** | Read state.md | Re-read conversation | Query database | Query database |
|
||||
| **Complexity ceiling** | High | Low (<30 statements) | High | **Very high** |
|
||||
| **Dependency** | None | None | sqlite3 CLI | psql CLI + PostgreSQL |
|
||||
| **Setup friction** | Zero | Zero | Low | Medium-High |
|
||||
| **Status** | Stable | Stable | Experimental | **Experimental** |
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
PostgreSQL state management:
|
||||
|
||||
1. Uses a **shared PostgreSQL database** for all runs
|
||||
2. Provides **true concurrent writes** via row-level locking
|
||||
3. Enables **network access** for external tools and dashboards
|
||||
4. Supports **team collaboration** on shared run state
|
||||
5. Allows **flexible schema evolution** with JSONB and custom tables
|
||||
6. Requires the **psql CLI** and a running PostgreSQL server
|
||||
7. Is **experimental**—expect changes
|
||||
|
||||
The core contract: the VM manages execution flow and spawns subagents; subagents write their own outputs directly to the database. Completion is signaled through the Task tool return, not database updates. External tools can query execution state in real-time.
|
||||
|
||||
**PostgreSQL state is for power users.** If you don't need concurrent writes, network access, or team collaboration, filesystem or SQLite state will be simpler and sufficient.
|
||||
572
extensions/open-prose/skills/prose/state/sqlite.md
Normal file
572
extensions/open-prose/skills/prose/state/sqlite.md
Normal file
@@ -0,0 +1,572 @@
|
||||
---
|
||||
role: sqlite-state-management
|
||||
status: experimental
|
||||
summary: |
|
||||
SQLite-based state management for OpenProse programs. This approach persists
|
||||
execution state to a SQLite database, enabling structured queries, atomic
|
||||
transactions, and flexible schema evolution.
|
||||
requires: sqlite3 CLI tool in PATH
|
||||
see-also:
|
||||
- ../prose.md: VM execution semantics
|
||||
- filesystem.md: File-based state (default, more prescriptive)
|
||||
- in-context.md: In-context state (for simple programs)
|
||||
- ../primitives/session.md: Session context and compaction guidelines
|
||||
---
|
||||
|
||||
# SQLite State Management (Experimental)
|
||||
|
||||
This document describes how the OpenProse VM tracks execution state using a **SQLite database**. This is an experimental alternative to file-based state (`filesystem.md`) and in-context state (`in-context.md`).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**Requires:** The `sqlite3` command-line tool must be available in your PATH.
|
||||
|
||||
| Platform | Installation |
|
||||
|----------|--------------|
|
||||
| macOS | Pre-installed |
|
||||
| Linux | `apt install sqlite3` / `dnf install sqlite3` / etc. |
|
||||
| Windows | `winget install SQLite.SQLite` or download from sqlite.org |
|
||||
|
||||
If `sqlite3` is not available, the VM will fall back to filesystem state and warn the user.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
SQLite state provides:
|
||||
|
||||
- **Atomic transactions**: State changes are ACID-compliant
|
||||
- **Structured queries**: Find specific bindings, filter by status, aggregate results
|
||||
- **Flexible schema**: Add columns and tables as needed
|
||||
- **Single-file portability**: The entire run state is one `.db` file
|
||||
- **Concurrent access**: SQLite handles locking automatically
|
||||
|
||||
**Key principle:** The database is a flexible workspace. The VM and subagents share it as a coordination mechanism, not a rigid contract.
|
||||
|
||||
---
|
||||
|
||||
## Database Location
|
||||
|
||||
The database lives within the standard run directory:
|
||||
|
||||
```
|
||||
.prose/runs/{YYYYMMDD}-{HHMMSS}-{random}/
|
||||
├── state.db # SQLite database (this file)
|
||||
├── program.prose # Copy of running program
|
||||
└── attachments/ # Large outputs that don't fit in DB (optional)
|
||||
```
|
||||
|
||||
**Run ID format:** Same as filesystem state: `{YYYYMMDD}-{HHMMSS}-{random6}`
|
||||
|
||||
Example: `.prose/runs/20260116-143052-a7b3c9/state.db`
|
||||
|
||||
### Project-Scoped and User-Scoped Agents
|
||||
|
||||
Execution-scoped agents (the default) live in the per-run `state.db`. However, **project-scoped agents** (`persist: project`) and **user-scoped agents** (`persist: user`) must survive across runs.
|
||||
|
||||
For project-scoped agents, use a separate database:
|
||||
|
||||
```
|
||||
.prose/
|
||||
├── agents.db # Project-scoped agent memory (survives runs)
|
||||
└── runs/
|
||||
└── {id}/
|
||||
└── state.db # Execution-scoped state (dies with run)
|
||||
```
|
||||
|
||||
For user-scoped agents, use a database in the home directory:
|
||||
|
||||
```
|
||||
~/.prose/
|
||||
└── agents.db # User-scoped agent memory (survives across projects)
|
||||
```
|
||||
|
||||
The `agents` and `agent_segments` tables for project-scoped agents live in `.prose/agents.db`, and for user-scoped agents live in `~/.prose/agents.db`. The VM initializes these databases on first use and provides the correct path to subagents.
|
||||
|
||||
---
|
||||
|
||||
## Responsibility Separation
|
||||
|
||||
This section defines **who does what**. This is the contract between the VM and subagents.
|
||||
|
||||
### VM Responsibilities
|
||||
|
||||
The VM (the orchestrating agent running the .prose program) is responsible for:
|
||||
|
||||
| Responsibility | Description |
|
||||
|----------------|-------------|
|
||||
| **Database creation** | Create `state.db` and initialize core tables at run start |
|
||||
| **Program registration** | Store the program source and metadata |
|
||||
| **Execution tracking** | Update position, status, and timing as statements execute |
|
||||
| **Subagent spawning** | Spawn sessions via Task tool with database path and instructions |
|
||||
| **Parallel coordination** | Track branch status, implement join strategies |
|
||||
| **Loop management** | Track iteration counts, evaluate conditions |
|
||||
| **Error aggregation** | Record failures, manage retry state |
|
||||
| **Context preservation** | Maintain sufficient narration in the main conversation thread so execution can be understood and resumed |
|
||||
| **Completion detection** | Mark the run as complete when finished |
|
||||
|
||||
**Critical:** The VM must preserve enough context in its own conversation to understand execution state without re-reading the entire database. The database is for coordination and persistence, not a replacement for working memory.
|
||||
|
||||
### Subagent Responsibilities
|
||||
|
||||
Subagents (sessions spawned by the VM) are responsible for:
|
||||
|
||||
| Responsibility | Description |
|
||||
|----------------|-------------|
|
||||
| **Writing own outputs** | Insert/update their binding in the `bindings` table |
|
||||
| **Memory management** | For persistent agents: read and update their memory record |
|
||||
| **Segment recording** | For persistent agents: append segment history |
|
||||
| **Attachment handling** | Write large outputs to `attachments/` directory, store path in DB |
|
||||
| **Atomic writes** | Use transactions when updating multiple related records |
|
||||
|
||||
**Critical:** Subagents write ONLY to `bindings`, `agents`, and `agent_segments` tables. The VM owns the `execution` table entirely. Completion signaling happens through the substrate (Task tool return), not database updates.
|
||||
|
||||
**Critical:** Subagents must write their outputs directly to the database. The VM does not write subagent outputs—it only reads them after the subagent completes.
|
||||
|
||||
**What subagents return to the VM:** A confirmation message with the binding location—not the full content:
|
||||
|
||||
**Root scope:**
|
||||
```
|
||||
Binding written: research
|
||||
Location: .prose/runs/20260116-143052-a7b3c9/state.db (bindings table, name='research', execution_id=NULL)
|
||||
Summary: AI safety research covering alignment, robustness, and interpretability with 15 citations.
|
||||
```
|
||||
|
||||
**Inside block invocation:**
|
||||
```
|
||||
Binding written: result
|
||||
Location: .prose/runs/20260116-143052-a7b3c9/state.db (bindings table, name='result', execution_id=43)
|
||||
Execution ID: 43
|
||||
Summary: Processed chunk into 3 sub-parts for recursive processing.
|
||||
```
|
||||
|
||||
The VM tracks locations, not values. This keeps the VM's context lean and enables arbitrarily large intermediate values.
|
||||
|
||||
### Shared Concerns
|
||||
|
||||
| Concern | Who Handles |
|
||||
|---------|-------------|
|
||||
| Schema evolution | Either (use `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE` as needed) |
|
||||
| Custom tables | Either (prefix with `x_` for extensions) |
|
||||
| Indexing | Either (add indexes for frequently-queried columns) |
|
||||
| Cleanup | VM (at run end, optionally vacuum) |
|
||||
|
||||
---
|
||||
|
||||
## Core Schema
|
||||
|
||||
The VM initializes these tables. This is a **minimum viable schema**—extend freely.
|
||||
|
||||
```sql
|
||||
-- Run metadata
|
||||
CREATE TABLE IF NOT EXISTS run (
|
||||
id TEXT PRIMARY KEY,
|
||||
program_path TEXT,
|
||||
program_source TEXT,
|
||||
started_at TEXT DEFAULT (datetime('now')),
|
||||
updated_at TEXT DEFAULT (datetime('now')),
|
||||
status TEXT DEFAULT 'running', -- running, completed, failed, interrupted
|
||||
state_mode TEXT DEFAULT 'sqlite'
|
||||
);
|
||||
|
||||
-- Execution position and history
|
||||
CREATE TABLE IF NOT EXISTS execution (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
statement_index INTEGER,
|
||||
statement_text TEXT,
|
||||
status TEXT, -- pending, executing, completed, failed, skipped
|
||||
started_at TEXT,
|
||||
completed_at TEXT,
|
||||
error_message TEXT,
|
||||
parent_id INTEGER REFERENCES execution(id), -- for nested blocks
|
||||
metadata TEXT -- JSON for construct-specific data (loop iteration, parallel branch, etc.)
|
||||
);
|
||||
|
||||
-- All named values (input, output, let, const)
|
||||
CREATE TABLE IF NOT EXISTS bindings (
|
||||
name TEXT,
|
||||
execution_id INTEGER, -- NULL for root scope, non-null for block invocations
|
||||
kind TEXT, -- input, output, let, const
|
||||
value TEXT,
|
||||
source_statement TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
updated_at TEXT DEFAULT (datetime('now')),
|
||||
attachment_path TEXT, -- if value is too large, store path to file
|
||||
PRIMARY KEY (name, IFNULL(execution_id, -1)) -- IFNULL handles NULL for root scope
|
||||
);
|
||||
|
||||
-- Persistent agent memory
|
||||
CREATE TABLE IF NOT EXISTS agents (
|
||||
name TEXT PRIMARY KEY,
|
||||
scope TEXT, -- execution, project, user, custom
|
||||
memory TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
updated_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Agent invocation history
|
||||
CREATE TABLE IF NOT EXISTS agent_segments (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
agent_name TEXT REFERENCES agents(name),
|
||||
segment_number INTEGER,
|
||||
timestamp TEXT DEFAULT (datetime('now')),
|
||||
prompt TEXT,
|
||||
summary TEXT,
|
||||
UNIQUE(agent_name, segment_number)
|
||||
);
|
||||
|
||||
-- Import registry
|
||||
CREATE TABLE IF NOT EXISTS imports (
|
||||
alias TEXT PRIMARY KEY,
|
||||
source_url TEXT,
|
||||
fetched_at TEXT,
|
||||
inputs_schema TEXT, -- JSON
|
||||
outputs_schema TEXT -- JSON
|
||||
);
|
||||
```
|
||||
|
||||
### Schema Conventions
|
||||
|
||||
- **Timestamps**: Use ISO 8601 format (`datetime('now')`)
|
||||
- **JSON fields**: Store structured data as JSON text in `metadata`, `*_schema` columns
|
||||
- **Large values**: If a binding value exceeds ~100KB, write to `attachments/{name}.md` and store path
|
||||
- **Extension tables**: Prefix with `x_` (e.g., `x_metrics`, `x_audit_log`)
|
||||
- **Anonymous bindings**: Sessions without explicit capture (`session "..."` without `let x =`) use auto-generated names: `anon_001`, `anon_002`, etc.
|
||||
- **Import bindings**: Prefix with import alias for scoping: `research.findings`, `research.sources`
|
||||
- **Scoped bindings**: Use `execution_id` column—NULL for root scope, non-null for block invocations
|
||||
|
||||
### Scope Resolution Query
|
||||
|
||||
For recursive blocks, bindings are scoped to their execution frame. Resolve variables by walking up the call stack:
|
||||
|
||||
```sql
|
||||
-- Find binding 'result' starting from execution_id 43
|
||||
WITH RECURSIVE scope_chain AS (
|
||||
-- Start with current execution
|
||||
SELECT id, parent_id FROM execution WHERE id = 43
|
||||
UNION ALL
|
||||
-- Walk up to parent
|
||||
SELECT e.id, e.parent_id
|
||||
FROM execution e
|
||||
JOIN scope_chain s ON e.id = s.parent_id
|
||||
)
|
||||
SELECT b.* FROM bindings b
|
||||
LEFT JOIN scope_chain s ON b.execution_id = s.id
|
||||
WHERE b.name = 'result'
|
||||
AND (b.execution_id IN (SELECT id FROM scope_chain) OR b.execution_id IS NULL)
|
||||
ORDER BY
|
||||
CASE WHEN b.execution_id IS NULL THEN 1 ELSE 0 END, -- Prefer scoped over root
|
||||
s.id DESC NULLS LAST -- Prefer deeper (more local) scope
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
**Simpler version if you know the scope chain:**
|
||||
|
||||
```sql
|
||||
-- Direct lookup: check current scope, then parent, then root
|
||||
SELECT * FROM bindings
|
||||
WHERE name = 'result'
|
||||
AND (execution_id = 43 OR execution_id = 42 OR execution_id IS NULL)
|
||||
ORDER BY execution_id DESC NULLS LAST
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Interaction
|
||||
|
||||
Both VM and subagents interact via the `sqlite3` CLI.
|
||||
|
||||
### From the VM
|
||||
|
||||
```bash
|
||||
# Initialize database
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "CREATE TABLE IF NOT EXISTS..."
|
||||
|
||||
# Update execution position
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "
|
||||
INSERT INTO execution (statement_index, statement_text, status, started_at)
|
||||
VALUES (3, 'session \"Research AI safety\"', 'executing', datetime('now'))
|
||||
"
|
||||
|
||||
# Read a binding
|
||||
sqlite3 -json .prose/runs/20260116-143052-a7b3c9/state.db "
|
||||
SELECT value FROM bindings WHERE name = 'research'
|
||||
"
|
||||
|
||||
# Check parallel branch status
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "
|
||||
SELECT statement_text, status FROM execution
|
||||
WHERE json_extract(metadata, '$.parallel_id') = 'p1'
|
||||
"
|
||||
```
|
||||
|
||||
### From Subagents
|
||||
|
||||
The VM provides the database path and instructions when spawning:
|
||||
|
||||
**Root scope (outside block invocations):**
|
||||
|
||||
```
|
||||
Your output database is:
|
||||
.prose/runs/20260116-143052-a7b3c9/state.db
|
||||
|
||||
When complete, write your output:
|
||||
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "
|
||||
INSERT OR REPLACE INTO bindings (name, execution_id, kind, value, source_statement, updated_at)
|
||||
VALUES (
|
||||
'research',
|
||||
NULL, -- root scope
|
||||
'let',
|
||||
'AI safety research covers alignment, robustness...',
|
||||
'let research = session: researcher',
|
||||
datetime('now')
|
||||
)
|
||||
"
|
||||
```
|
||||
|
||||
**Inside block invocation (include execution_id):**
|
||||
|
||||
```
|
||||
Execution scope:
|
||||
execution_id: 43
|
||||
block: process
|
||||
depth: 3
|
||||
|
||||
Your output database is:
|
||||
.prose/runs/20260116-143052-a7b3c9/state.db
|
||||
|
||||
When complete, write your output:
|
||||
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "
|
||||
INSERT OR REPLACE INTO bindings (name, execution_id, kind, value, source_statement, updated_at)
|
||||
VALUES (
|
||||
'result',
|
||||
43, -- scoped to this execution
|
||||
'let',
|
||||
'Processed chunk into 3 sub-parts...',
|
||||
'let result = session \"Process chunk\"',
|
||||
datetime('now')
|
||||
)
|
||||
"
|
||||
```
|
||||
|
||||
For persistent agents (execution-scoped):
|
||||
|
||||
```
|
||||
Your memory is in the database:
|
||||
.prose/runs/20260116-143052-a7b3c9/state.db
|
||||
|
||||
Read your current state:
|
||||
sqlite3 -json .prose/runs/20260116-143052-a7b3c9/state.db "SELECT memory FROM agents WHERE name = 'captain'"
|
||||
|
||||
Update when done:
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "UPDATE agents SET memory = '...', updated_at = datetime('now') WHERE name = 'captain'"
|
||||
|
||||
Record this segment:
|
||||
sqlite3 .prose/runs/20260116-143052-a7b3c9/state.db "INSERT INTO agent_segments (agent_name, segment_number, prompt, summary) VALUES ('captain', 3, '...', '...')"
|
||||
```
|
||||
|
||||
For project-scoped agents, use `.prose/agents.db`. For user-scoped agents, use `~/.prose/agents.db`.
|
||||
|
||||
---
|
||||
|
||||
## Context Preservation in Main Thread
|
||||
|
||||
**This is critical.** The database is for persistence and coordination, but the VM must still maintain conversational context.
|
||||
|
||||
### What the VM Must Narrate
|
||||
|
||||
Even with SQLite state, the VM should narrate key events in its conversation:
|
||||
|
||||
```
|
||||
[Position] Statement 3: let research = session: researcher
|
||||
Spawning session, will write to state.db
|
||||
[Task tool call]
|
||||
[Success] Session complete, binding written to DB
|
||||
[Binding] research = <stored in state.db>
|
||||
```
|
||||
|
||||
### Why Both?
|
||||
|
||||
| Purpose | Mechanism |
|
||||
|---------|-----------|
|
||||
| **Working memory** | Conversation narration (what the VM "remembers" without re-querying) |
|
||||
| **Durable state** | SQLite database (survives context limits, enables resumption) |
|
||||
| **Subagent coordination** | SQLite database (shared access point) |
|
||||
| **Debugging/inspection** | SQLite database (queryable history) |
|
||||
|
||||
The narration is the VM's "mental model" of execution. The database is the "source of truth" for resumption and inspection.
|
||||
|
||||
---
|
||||
|
||||
## Parallel Execution
|
||||
|
||||
For parallel blocks, the VM uses the `metadata` JSON field to track branches. **Only the VM writes to the `execution` table.**
|
||||
|
||||
```sql
|
||||
-- VM marks parallel start
|
||||
INSERT INTO execution (statement_index, statement_text, status, metadata)
|
||||
VALUES (5, 'parallel:', 'executing', '{"parallel_id": "p1", "strategy": "all", "branches": ["a", "b", "c"]}');
|
||||
|
||||
-- VM creates execution record for each branch
|
||||
INSERT INTO execution (statement_index, statement_text, status, parent_id, metadata)
|
||||
VALUES (6, 'a = session "Task A"', 'executing', 5, '{"parallel_id": "p1", "branch": "a"}');
|
||||
|
||||
-- Subagent writes its output to bindings table (see "From Subagents" section)
|
||||
-- Task tool signals completion to VM via substrate
|
||||
|
||||
-- VM marks branch complete after Task returns
|
||||
UPDATE execution SET status = 'completed', completed_at = datetime('now')
|
||||
WHERE json_extract(metadata, '$.parallel_id') = 'p1' AND json_extract(metadata, '$.branch') = 'a';
|
||||
|
||||
-- VM checks if all branches complete
|
||||
SELECT COUNT(*) as pending FROM execution
|
||||
WHERE json_extract(metadata, '$.parallel_id') = 'p1' AND status != 'completed';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Loop Tracking
|
||||
|
||||
```sql
|
||||
-- Loop metadata tracks iteration state
|
||||
INSERT INTO execution (statement_index, statement_text, status, metadata)
|
||||
VALUES (10, 'loop until **analysis complete** (max: 5):', 'executing',
|
||||
'{"loop_id": "l1", "max_iterations": 5, "current_iteration": 0, "condition": "**analysis complete**"}');
|
||||
|
||||
-- Update iteration
|
||||
UPDATE execution
|
||||
SET metadata = json_set(metadata, '$.current_iteration', 2),
|
||||
updated_at = datetime('now')
|
||||
WHERE json_extract(metadata, '$.loop_id') = 'l1';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
```sql
|
||||
-- Record failure
|
||||
UPDATE execution
|
||||
SET status = 'failed',
|
||||
error_message = 'Connection timeout after 30s',
|
||||
completed_at = datetime('now')
|
||||
WHERE id = 15;
|
||||
|
||||
-- Track retry attempts in metadata
|
||||
UPDATE execution
|
||||
SET metadata = json_set(metadata, '$.retry_attempt', 2, '$.max_retries', 3)
|
||||
WHERE id = 15;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Large Outputs
|
||||
|
||||
When a binding value is too large for comfortable database storage (>100KB):
|
||||
|
||||
1. Write content to `attachments/{binding_name}.md`
|
||||
2. Store the path in the `attachment_path` column
|
||||
3. Leave `value` as a summary or null
|
||||
|
||||
```sql
|
||||
INSERT INTO bindings (name, kind, value, attachment_path, source_statement)
|
||||
VALUES (
|
||||
'full_report',
|
||||
'let',
|
||||
'Full analysis report (847KB) - see attachment',
|
||||
'attachments/full_report.md',
|
||||
'let full_report = session "Generate comprehensive report"'
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resuming Execution
|
||||
|
||||
To resume an interrupted run:
|
||||
|
||||
```sql
|
||||
-- Find current position
|
||||
SELECT statement_index, statement_text, status
|
||||
FROM execution
|
||||
WHERE status = 'executing'
|
||||
ORDER BY id DESC LIMIT 1;
|
||||
|
||||
-- Get all completed bindings
|
||||
SELECT name, kind, value, attachment_path FROM bindings;
|
||||
|
||||
-- Get agent memory states
|
||||
SELECT name, memory FROM agents;
|
||||
|
||||
-- Check parallel block status
|
||||
SELECT json_extract(metadata, '$.branch') as branch, status
|
||||
FROM execution
|
||||
WHERE json_extract(metadata, '$.parallel_id') IS NOT NULL
|
||||
AND parent_id = (SELECT id FROM execution WHERE status = 'executing' AND statement_text LIKE 'parallel:%');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Flexibility Encouragement
|
||||
|
||||
Unlike filesystem state, SQLite state is intentionally **less prescriptive**. The core schema is a starting point. You are encouraged to:
|
||||
|
||||
- **Add columns** to existing tables as needed
|
||||
- **Create extension tables** (prefix with `x_`)
|
||||
- **Store custom metrics** (timing, token counts, model info)
|
||||
- **Build indexes** for your query patterns
|
||||
- **Use JSON functions** for semi-structured data
|
||||
|
||||
Example extensions:
|
||||
|
||||
```sql
|
||||
-- Custom metrics table
|
||||
CREATE TABLE x_metrics (
|
||||
execution_id INTEGER REFERENCES execution(id),
|
||||
metric_name TEXT,
|
||||
metric_value REAL,
|
||||
recorded_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Add custom column
|
||||
ALTER TABLE bindings ADD COLUMN token_count INTEGER;
|
||||
|
||||
-- Create index for common query
|
||||
CREATE INDEX idx_execution_status ON execution(status);
|
||||
```
|
||||
|
||||
The database is your workspace. Use it.
|
||||
|
||||
---
|
||||
|
||||
## Comparison with Other Modes
|
||||
|
||||
| Aspect | filesystem.md | in-context.md | sqlite.md |
|
||||
|--------|---------------|---------------|-----------|
|
||||
| **State location** | `.prose/runs/{id}/` files | Conversation history | `.prose/runs/{id}/state.db` |
|
||||
| **Queryable** | Via file reads | No | Yes (SQL) |
|
||||
| **Atomic updates** | No | N/A | Yes (transactions) |
|
||||
| **Schema flexibility** | Rigid file structure | N/A | Flexible (add tables/columns) |
|
||||
| **Resumption** | Read state.md | Re-read conversation | Query database |
|
||||
| **Complexity ceiling** | High | Low (<30 statements) | High |
|
||||
| **Dependency** | None | None | sqlite3 CLI |
|
||||
| **Status** | Stable | Stable | **Experimental** |
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
SQLite state management:
|
||||
|
||||
1. Uses a **single database file** per run
|
||||
2. Provides **clear responsibility separation** between VM and subagents
|
||||
3. Enables **structured queries** for state inspection
|
||||
4. Supports **atomic transactions** for reliable updates
|
||||
5. Allows **flexible schema evolution** as needed
|
||||
6. Requires the **sqlite3 CLI** tool
|
||||
7. Is **experimental**—expect changes
|
||||
|
||||
The core contract: the VM manages execution flow and spawns subagents; subagents write their own outputs directly to the database. Both maintain the principle that what happens is recorded, and what is recorded can be queried.
|
||||
Reference in New Issue
Block a user