Multi-Instance Watchtower — Research Meeting Minutes

Date: 2026-02-17
Meeting start: 00:25 EST | Meeting end: —
Duration: —
Attendees: Phil (product owner), CoS (facilitator), SDK Researcher, Discord Researcher, Infra/Resource Analyst, Brain DB Analyst

1. SDK Researcher — Multi-Instance Feasibility

• No singletons or global state in the SDK. Each ClaudeSDKClient is independent — own transport, own subprocess, own message stream.
• Process model: Each client spawns its own claude Node.js subprocess via anyio.open_process(). Communication is JSON over stdin/stdout pipes.
• Working directory isolation: Each subprocess gets its own cwd — fully isolated at the OS level.
• Settings per instance: The --settings CLI flag is per-subprocess. Each worker can use a different settings file.
• Concurrent asyncio: Multiple clients can run query() concurrently in the same event loop. Each has its own TaskGroup and message stream.
• One minor wart: A cosmetic env var (CLAUDE_CODE_ENTRYPOINT) gets mutated globally, but it's harmless — same value written by all clients.
• No nesting issue: Workers are spawned by the Python coordinator, not by CC itself, so the CLAUDECODE env var detection doesn't fire.

Resource per instance

2. Discord Researcher — Channel Management

• Bot can create channels: guild.create_text_channel(name, category, overwrites). Needs manage_channels permission.
• Categories supported: Create a "Watchtower Workers" category, nest worker channels inside it. Clean sidebar UX.
• Bot can delete channels: channel.delete() — same permission. Cleanup on session end.
• Per-channel permissions: Overwrites at creation — lock to Phil-only and the bot. Everyone else denied.
• Multi-channel routing: Single on_message handler + a dict[channel_id → worker] registry. No separate listeners needed.
• Channel naming: 1-100 chars, lowercase + hyphens. Discord does NOT enforce uniqueness (use IDs as keys).
• Rate limits: ~2 channel creates per 10 seconds. Irrelevant for spawning a handful of workers.
• Hard limit: 500 channels per guild. Ephemeral pattern (create on start, delete on end) keeps this clean.

Recommended pattern

• Channels are ephemeral — created when worker starts, deleted when worker ends.
• Named worker-<short-id> or worker-czechwriter for readability.
• If Phil wants history, keep a persistent log channel for session summaries.

3. Infra/Resource Analyst — Deadpool Capacity & Clone Model

Memory budget (32GB)

Clone-based workspaces

• Clone location: Linux filesystem at /home/plangeberg/watchtower-workers/<worker-id>/ — substantially faster than /mnt/d/ (5-15x for git ops).
• Clone scope: Just the needed subrepo (e.g., czechsuma-labs/czechwriter). Use SDK's --add-dir flag for read access to brain/ context if needed.
• Shallow clones: git clone --depth 1 keeps disk usage minimal (~10-50MB per worker).
• Best option: Bare clone mirror on Linux FS + git worktree add for workers. Faster than network clones, shared object store.

Cleanup

• Normal: commit → push → rm -rf worker dir.
• Crash: On startup, scan worker dirs for orphans (check PIDs). worker.json metadata per worker.

Conflict handling — recommended approach

• Worker branches: Each worker commits to worker/<id>. No push collisions possible.
• CoS (or Phil) merges worker branches into main after review.
• Alternative: Lock one worker per repo (simpler, less parallel).

WSL2 prep needed

• Raise fs.inotify.max_user_watches to 524288 (default 8192 will exhaust with multiple Node.js processes).
• Set .wslconfig: memory=20GB, swap=4GB — leaves 12GB for Windows.

4. Brain DB Analyst — Shared State & Concurrent Writes

The real conflict risk

• 95% of the risk is one file: THREADS.md — specifically the todo list and thread status sections.
• Everything else (PHIL.md, contexts, sessions, handoffs, runbooks) is either read-only, append-only, or session-scoped. No concurrency risk.

Recommended phased approach

1. Phase 1 (ship with multi-instance): "CoS owns THREADS.md" rule. Worker instances run in a restricted mode — no writes to brain/. This is basically extending !secret mode to all workers. Zero new infrastructure needed.
2. Phase 2 (when pain is real): SQLite DB in WAL mode at brain/brain.db.
   • Tables: threads, todos, parking_lot, review_items, delegated
   • WAL mode allows concurrent reads + serialized writes with auto-retry
   • Python CLI wrapper (brain-db) — CC calls it via Bash instead of editing THREADS.md
   • Auto-renders THREADS.md as a read-only artifact after every write (Phil's view doesn't change)
   • Migration script parses existing THREADS.md → DB rows (~150 lines)
   • CLI tool ~200 lines. Minimal command set: brain-db todos, brain-db todo-done "text", etc.

What stays as files forever

• PHIL.md, CHIEF-OF-STAFF.md, contexts/* — read-only for CC, rare human edits
• Sessions/* — append-only, one file per session
• Handoffs — create-once, consume-once lifecycle
• Runbooks, parking-lot detail files — reference docs, not concurrent data

5. Architecture Summary

Watchtower starts → CoS session in main channel (primary working copy)
    │
    Phil: "spin up a worker for CzechWriter"
    │
    ▼
CoS checks: resources OK? repo locked? → YES
    │
    ├── Creates Discord channel: #worker-czechwriter
    ├── Clones repo to /home/plangeberg/watchtower-workers/wt-001/
    ├── Spawns new ClaudeSDKClient(cwd=clone_path)
    ├── Registers channel_id → worker in routing dict
    └── Tells Phil: "Channel ready, go talk to it"

Phil switches to #worker-czechwriter → works directly with that CC instance
    │
    Done → Phil says "end session" or tells CoS to kill it
    │
    ▼
Cleanup:
    ├── Worker commits to branch worker/wt-001
    ├── Worker pushes branch
    ├── Clone directory deleted
    ├── Discord channel deleted (or archived)
    └── CoS merges branch to main (or Phil reviews first)

6. Proposed Tickets

7. Open Questions for Phil

1. Max concurrent workers: Default to 4? Or let it float based on memory?
2. Worker branch merging: CoS auto-merges to main? Or Phil reviews the branch first?
3. Channel persistence: Delete channels when worker ends? Or keep for history (collapse into archive category)?
4. Repo mapping: Should CoS know which repos map to which project names? (e.g., "CzechWriter" → czechsuma-labs/czechwriter) Or does Phil specify the path?
5. Naming: The app is still called Watchtower (pending rename). Do these tickets go into the existing Watchtower backlog, or does multi-instance warrant its own project name?

8. Risks & Mitigations