diff --git a/docs/testing.md b/docs/testing.md index f45f50519..6a2c7f528 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -8,31 +8,59 @@ read_when: # Testing -Clawdbot has three Vitest suites (unit, e2e, live) plus a couple Docker helpers for “run with my real keys” smoke checks. +Clawdbot has three Vitest suites (unit/integration, e2e, live) and a small set of Docker runners. + +This doc is a “how we test” guide: +- What each suite covers (and what it deliberately does *not* cover) +- Which commands to run for common workflows (local, pre-push, debugging) +- How live tests discover credentials and select models/providers +- How to add regressions for real-world model/provider issues ## Quick start -- Full gate (what we expect before push): `pnpm lint && pnpm build && pnpm test` +Most days: +- Full gate (expected before push): `pnpm lint && pnpm build && pnpm test` + +When you touch tests or want extra confidence: - Coverage gate: `pnpm test:coverage` - E2E suite: `pnpm test:e2e` -- Live suite (opt-in, Clawdbot only): `CLAWDBOT_LIVE_TEST=1 pnpm test:live` -- Live suite (opt-in, includes provider live tests too): `LIVE=1 pnpm test:live` + +When debugging real providers/models (requires real creds; skipped by default): +- Live suite (models only): `CLAWDBOT_LIVE_TEST=1 pnpm test:live` +- Live suite (models + providers): `LIVE=1 pnpm test:live` + +Tip: when you only need one failing case, prefer narrowing live tests via the allowlist env vars described below. ## Test suites (what runs where) +Think of the suites as “increasing realism” (and increasing flakiness/cost): + ### Unit / integration (default) - Command: `pnpm test` - Config: `vitest.config.ts` - Files: `src/**/*.test.ts` -- Scope: pure unit tests + in-process integration tests (gateway server auth, routing, tooling, parsing, config). +- Scope: + - Pure unit tests + - In-process integration tests (gateway auth, routing, tooling, parsing, config) + - Deterministic regressions for known bugs +- Expectations: + - Runs in CI + - No real keys required + - Should be fast and stable ### E2E (gateway smoke) - Command: `pnpm test:e2e` - Config: `vitest.e2e.config.ts` - Files: `src/**/*.e2e.test.ts` -- Scope: multi-instance gateway end-to-end behavior (WebSocket/HTTP/node pairing), heavier networking surface. +- Scope: + - Multi-instance gateway end-to-end behavior + - WebSocket/HTTP surfaces, node pairing, and heavier networking +- Expectations: + - Runs in CI (when enabled in the pipeline) + - No real keys required + - More moving parts than unit tests (can be slower) ### Live (real providers + real models) @@ -40,34 +68,86 @@ Clawdbot has three Vitest suites (unit, e2e, live) plus a couple Docker helpers - Config: `vitest.live.config.ts` - Files: `src/**/*.live.test.ts` - Default: **skipped** unless `CLAWDBOT_LIVE_TEST=1` or `LIVE=1` -- Scope: “does this provider/model actually work today with real creds”. +- Scope: + - “Does this provider/model actually work *today* with real creds?” + - Catch provider format changes, tool-calling quirks, auth issues, and rate limit behavior +- Expectations: + - Not CI-stable by design (real networks, real provider policies, quotas, outages) + - Costs money / uses rate limits + - Prefer running narrowed subsets instead of “everything” + +## Which suite should I run? + +Use this decision table: +- Editing logic/tests: run `pnpm test` (and `pnpm test:coverage` if you changed a lot) +- Touching gateway networking / WS protocol / pairing: add `pnpm test:e2e` +- Debugging “my bot is down” / provider-specific failures / tool calling: run a narrowed `pnpm test:live` ## Live: model smoke (profile keys) -Two layers: +Live tests are split into two layers so we can isolate failures: +- “Direct model” tells us the provider/model can answer at all with the given key. +- “Gateway smoke” tells us the full gateway+agent pipeline works for that model (sessions, history, tools, sandbox policy, etc.). -1. Direct model completion (no gateway): - - Test: `src/agents/models.profiles.live.test.ts` - - Goal: enumerate discovered models, use `getApiKeyForModel` to pick ones you have creds for, then run a small completion. - - Selection: - - `CLAWDBOT_LIVE_ALL_MODELS=1` (required to run the suite) - - `CLAWDBOT_LIVE_MODELS=all` or comma allowlist (`openai/gpt-5.2,anthropic/claude-opus-4-5,...`) - - `CLAWDBOT_LIVE_REQUIRE_PROFILE_KEYS=1` to ensure creds come from the profile store (not ad-hoc env). - - Regression hook: OpenAI Responses tool-only → follow-up path (the `reasoning` replay class) is covered here. +### Layer 1: Direct model completion (no gateway) -2. Gateway + dev agent smoke (what “@clawdbot” actually does): - - Test: `src/gateway/gateway-models.profiles.live.test.ts` - - Goal: spin up an in-process gateway, create/patch a `agent:dev:*` session, iterate models-with-keys, and assert “meaningful” responses. - - Covers providers present in your `models.json`/config (e.g. OpenAI, Anthropic, Google Gemini, `google-antigravity`, etc.) as long as a key/profile is available. - - Selection: - - `CLAWDBOT_LIVE_GATEWAY=1` - - `CLAWDBOT_LIVE_GATEWAY_ALL_MODELS=1` (scan all discovered models with available keys) - - `CLAWDBOT_LIVE_GATEWAY_MODELS=all` or comma allowlist - - Extra regression: for OpenAI Responses/Codex Responses models, force a tool-call-only turn followed by a user question (the exact failure mode that produced `400 … reasoning … required following item`). +- Test: `src/agents/models.profiles.live.test.ts` +- Goal: + - Enumerate discovered models + - Use `getApiKeyForModel` to select models you have creds for + - Run a small completion per model (and targeted regressions where needed) +- How to enable: + - `CLAWDBOT_LIVE_TEST=1` or `LIVE=1` + - `CLAWDBOT_LIVE_ALL_MODELS=1` (required for this test to run) +- How to select models: + - `CLAWDBOT_LIVE_MODELS=all` to run everything with keys + - or `CLAWDBOT_LIVE_MODELS="openai/gpt-5.2,anthropic/claude-opus-4-5,..."` (comma allowlist) +- Where keys come from: + - By default: profile store and env fallbacks + - Set `CLAWDBOT_LIVE_REQUIRE_PROFILE_KEYS=1` to enforce **profile store** only +- Why this exists: + - Separates “provider API is broken / key is invalid” from “gateway agent pipeline is broken” + - Contains small, isolated regressions (example: OpenAI Responses/Codex Responses reasoning replay + tool-call flows) + +### Layer 2: Gateway + dev agent smoke (what “@clawdbot” actually does) + +- Test: `src/gateway/gateway-models.profiles.live.test.ts` +- Goal: + - Spin up an in-process gateway + - Create/patch a `agent:dev:*` session (model override per run) + - Iterate models-with-keys and assert: + - “meaningful” response (no tools) + - a real tool invocation works (read probe) + - optional extra tool probes (bash+read probe) + - OpenAI regression paths (tool-call-only → follow-up) keep working +- How to enable: + - `CLAWDBOT_LIVE_TEST=1` or `LIVE=1` + - `CLAWDBOT_LIVE_GATEWAY=1` (required for this test to run) +- How to select models: + - `CLAWDBOT_LIVE_GATEWAY_ALL_MODELS=1` to scan all discovered models with keys + - or set `CLAWDBOT_LIVE_GATEWAY_MODELS="provider/model,provider/model,..."` to narrow quickly +- Optional tool-calling stress: + - `CLAWDBOT_LIVE_GATEWAY_TOOL_PROBE=1` enables an extra “bash writes file → read reads it back → echo nonce” check. + - This is specifically meant to catch tool-calling compatibility issues across providers (formatting, history replay, tool_result pairing, etc.). + +### Recommended live recipes + +Narrow, explicit allowlists are fastest and least flaky: + +- Single model, direct (no gateway): + - `CLAWDBOT_LIVE_TEST=1 CLAWDBOT_LIVE_ALL_MODELS=1 CLAWDBOT_LIVE_MODELS="openai/gpt-5.2" pnpm test:live src/agents/models.profiles.live.test.ts` + +- Single model, gateway smoke: + - `LIVE=1 CLAWDBOT_LIVE_GATEWAY=1 CLAWDBOT_LIVE_GATEWAY_ALL_MODELS=1 CLAWDBOT_LIVE_GATEWAY_MODELS="openai/gpt-5.2" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts` + +- Tool calling across several providers (bash + read probe): + - `LIVE=1 CLAWDBOT_LIVE_GATEWAY=1 CLAWDBOT_LIVE_GATEWAY_ALL_MODELS=1 CLAWDBOT_LIVE_GATEWAY_TOOL_PROBE=1 CLAWDBOT_LIVE_GATEWAY_MODELS="openai/gpt-5.2,anthropic/claude-opus-4-5,google/gemini-flash-latest,zai/glm-4.7,minimax/minimax-m2.1" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts` ## Credentials (never commit) -Live tests discover credentials the same way the CLI does: +Live tests discover credentials the same way the CLI does. Practical implications: +- If the CLI works, live tests should find the same keys. +- If a live test says “no creds”, debug the same way you’d debug `clawdbot models list` / model selection. - Profile store: `~/.clawdbot/credentials/` (preferred; what “profile keys” means in the tests) - Config: `~/.clawdbot/clawdbot.json` (or `CLAWDBOT_CONFIG_PATH`) @@ -97,5 +177,15 @@ Run docs checks after doc edits: `pnpm docs:list`. ## Offline regression (CI-safe) +These are “real pipeline” regressions without real providers: - Gateway tool calling (mock OpenAI, real gateway + agent loop): `src/gateway/gateway.tool-calling.mock-openai.test.ts` - Gateway wizard (WS `wizard.start`/`wizard.next`, writes config + auth enforced): `src/gateway/gateway.wizard.e2e.test.ts` + +## Adding regressions (guidance) + +When you fix a provider/model issue discovered in live: +- Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation) +- If it’s inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars +- Prefer targeting the smallest layer that catches the bug: + - provider request conversion/replay bug → direct models test + - gateway session/history/tool pipeline bug → gateway live smoke or CI-safe gateway mock test