Clarence:
Designing an Autonomous AI Collaborator

Token Budget as a Design Material

Token cost shapes every architectural decision. The March 25-26 optimization pass made this explicit: self-audit prompt trimmed from 7,582 chars to 1,276 chars. Bootstrap files trimmed from 11 to 7 (~18KB total). Memory files reduced from 106KB to 40KB. The lightContext flag enabled on every cron job. The total effect: every session starts with less noise and more signal, and the overnight loop runs at zero model cost.

Model switching is now immediate. openclaw models set <model> changes the active model with no restart. This means routing decisions can be made at the task level rather than the configuration level. The system can adapt to what each job actually needs.

Single Source of Truth Architecture

Five sub-agent workspaces are symlinked to the parent workspace. Every agent reads from the same files. IDENTITY.md was merged into SOUL.md. Legacy databases were archived and retired into a single authoritative store: clarence.db. The pattern is consistent: eliminate duplication, reduce the surface area where state can diverge.

This matters more than it sounds. When multiple agents can write to the same knowledge base but read from different workspace copies, you get silent divergence. Symlinks solved this at the filesystem level without requiring any coordination protocol.

Bridging the Orchestrator and Model Providers

OpenClaw speaks the OpenAI-compatible API. The underlying model providers speak their own protocols. A custom Rust bridge translates between them, making Claude and Gemini available to the orchestrator without separate API key configurations per agent.

This adds a dependency layer that can fail independently of either system. The tradeoff was worth it: deeper integration with the model toolchain, including tool access and session context that direct API calls would not provide in the same form.

SQLite Knowledge Database + RAG

Long-term memory is stored in a single consolidated SQLite database (clarence.db) with 2,318 memories, 1,877 entities, and 9,376 facts, shared by all agents through a custom MCP server. The schema separates concerns: a profiles table holds identity facts (agent names, user preferences, project constants) with deterministic key lookup. No fuzzy search for things that must be exact. A memories table stores durable knowledge with soft invalidation: when a fact changes, the old record is marked invalid and a new one is written, preserving the audit trail.

The RAG layer is live: 2,372 memory vectors and 9,376 fact vectors with sentence-transformer embeddings (all-MiniLM-L6-v2), running fully locally via sqlite-vec. No separate vector database, no network hop. Agents query the knowledge base by meaning: “what does James think about AI agent UX?” returns the five most relevant records across all tables, regardless of how they were originally tagged.

The conversation distillation pipeline (conversation-distill.py) processes Telegram conversations nightly, extracting decisions, corrections, and preferences into the memory DB. This is what makes the memory system feel alive rather than static. James corrects something once in conversation, and it persists. The knowledge base grew from ~170 to 2,318 memories in part because this pipeline captures context that would otherwise evaporate.

An Obsidian vault syncs bidirectionally with the database. New vault notes are picked up by the nightly embedding job automatically. Writing in Obsidian feeds the RAG layer without any additional wiring.

Tailscale for Device Continuity

Clarence runs on an Ubuntu Linux server. James moves between Linux, Mac, iPhone, and iPad. Tailscale creates a private mesh VPN connecting all four, so the Telegram interface and Brain Reader HTTP server are reachable from any device without exposing anything to the public internet. This was a day-one decision and it has been the most friction-free part of the stack.

@ClarenceTheOG: Extending into Public Space

The Twitter/X bot extends the output surface beyond private Telegram and brain files into a public space. This raises the design stakes: errors that stay in a log file are recoverable. Errors that get posted publicly are not. Clarence has standing permission to post but James retains approval for anything commercially sensitive or identity-critical.

The Write-Only Audit Loop

The self-audit produces detailed, specific proposals every night. The quick-wins queue is populated with concrete autonomous tasks. The autonomous employee is configured to execute them. But the execution-to-proposal ratio is lower than it should be.

The problems are at the execution layer: the autonomous employee sometimes fails to find the queue file, sometimes falls back to its own priority logic, and sometimes the queue was not written correctly. The audit system reports on this failure. The audit is self-aware about it. But self-awareness does not fix reliability. This is the most honest statement about the current system.

Budget Constraint as Design Constraint

The entire routing policy exists because running expensive models at scale has real cost. A fleet of cron jobs running nightly would be expensive if all of them used Opus. The March 25-26 pass eliminated expensive models from cron entirely, splitting them across Gemini Flash and MiniMax. But this creates a different kind of debt.

The tradeoff is quality degradation at the mechanical tier. When a task gets routed to a cheaper model, the output quality drops and it is not always obvious why. Budget-aware routing requires continuous calibration, and the calibration is never quite finished.

Memory Growth Without Garbage Collection

The knowledge base grew from ~170 to 2,318 memories. The conversation distillation pipeline accelerated that growth. But more memories does not automatically mean better recall. As the database scales, the vector search returns increasingly similar results, and the signal-to-noise ratio in retrieved context degrades. Memory needs pruning and consolidation, not just accumulation. This is the next hard problem.

Log Noise as a Signal Problem

Three tools were in the agent tool profile but unavailable at runtime, generating dozens of WARN log entries daily. Each warning was individually harmless. Together they buried real errors. Bruno's security audits were scanning logs where signal-to-noise had degraded enough that genuine failures could be missed. The fix was explicit: disable the three tools at the config level rather than leaving them as dead references. This is a documented case of how low-stakes configuration drift compounds into an observability gap.

Spontaneous Task Delegation Breaks Down

The overnight cron loop works because every job has a clear prompt, a fixed model, and a predictable execution path. Spontaneous tasks during live sessions are a different problem. When a cheaper model receives an unstructured real-time request, it sometimes hallucinates tool syntax instead of executing actual tool calls, dumping raw markup into Telegram messages where a human expects a coherent response.

This reveals a gap between scheduled autonomy and reactive autonomy. The system is reliable when it knows what to do in advance. It degrades when asked to improvise with models that lack the reasoning depth to handle ambiguity. The current workaround is routing complex spontaneous work to higher-capability models, but this defeats the cost architecture. The real fix is better prompt scaffolding for spontaneous tasks, not just better models.

Visibility of System Status (Nielsen #1)

This is the most persistent unsolved design problem in the system. Nielsen's first usability heuristic states that a system should always keep users informed about what is going on through appropriate feedback within reasonable time. Clarence violates this consistently.

Twenty-six cron jobs run overnight. Many complete successfully but report “not-delivered” on their Telegram notifications. The Sergeant-at-Arms posts a digest, but only if something needs attention, which means silence is ambiguous: does silence mean everything worked, or that the reporting layer itself failed? When James wakes up, the system status is reconstructed from scattered log files and database queries rather than surfaced through a coherent status interface.

The problem compounds during live sessions. When agents are delegated tasks in parallel, there is no progress indicator, no heartbeat, no way to distinguish “working on it” from “silently failed.” The user is left interpreting silence, which is the opposite of visibility. This is not a logging problem. It is a feedback design problem, and solving it requires treating system status as a first-class UX surface rather than a side effect of Telegram messages.