Tracing LLM Agents in Postgres: Why We Built a Ledger Instead of OpenTelemetry

Every AI agent in Connect writes down what it did. Each model turn, each tool call, the tokens it spent, what it cost and whether it worked ends up in one Postgres table that the executive portal, the evaluation loop and the schedule version history all read from. There's no OpenTelemetry collector, no hosted tracing vendor and no second database just for telemetry.

That was a deliberate choice, and it cost us some things. This article covers how the ledger works: the two recorders that feed it, the table shapes, what we redact and why, and the admin views built on top. It also covers the reasoning, because "just use OTel" is the default answer and we think it's often the wrong one for a product team running agents for its own customers.

The problem: agents in two stores, questions across all of them

Connect runs several agents. The chat assistant, the Project Info Sheet orchestrator and its section workers, the Schedule Builder, the App Builder, an Updates drafter and a few smaller helpers. They don't all share a runtime. Chat, the Info Sheet, the Schedule Builder and the App Builder run on the OpenAI Agents SDK. The shorter single-shot agents (Means & Methods, Updates, alert drafting, job walks) run on a lightweight agent loop we drive directly. Their working state lives in different places too: chat history in Postgres, Info Sheet turns in a per-workspace document store.

The questions executives ask cut across all of that:

• Which agents are people actually using, and what does each one cost per week?
• Why did this turn fail, and which tool call broke it?
• Did last Tuesday's prompt change make answers better or worse?
• How was revision 7 of this schedule produced?

You can't answer those by fanning out over one document database per workspace plus Postgres and merging the results in application code. So every agent appends a compact, queryable copy of each turn to one table, agent_events, and every dashboard query hits that table alone.

What one ledger row holds

One row is one agent turn. A turn is an LLM interaction that may involve several model calls and tool calls. The columns fall into five groups:

A few of these are worth calling out.

Reasoning and cache tokens get their own columns. Folding them into "input" and "output" hides the two numbers that move cost the most on modern models. Prompt-cache hit rates tell you whether a prompt restructure saved money. Reasoning tokens tell you whether a thinking-level change was worth it.

The prompt version id records which published configuration governed the turn. When a quality score drops, you can see whether it dropped for turns on the new version or across the board.

There are no foreign keys back to messages or conversations. The ledger is append-only and written best-effort, so it can't depend on the row it describes having committed first.

There's an older ai_usage table from before the ledger. It only held token counts and never got populated for chat. We left it in place rather than write a migration to drop a harmless table. New code reads agent_events.

Two recorders, one sink

Because the agents run on two different runtimes, there are two recorders. Both produce the same record shape and hand it to the same sink.

The runtime recorder for our own agent loop

RuntimeTraceRecorder subscribes to the agent loop's lifecycle events. One recorder lives per agent instance, and a fresh request context is resolved at each run start, so a long-lived agent object can't leak one user's identity into another user's trace.

The event flow:

1. On run start, resolve the trace context (agent kind, workspace, conversation, turn, user, prompt version, tool definitions) and reset state. A run with no context throws. Untraced agent runs are treated as a wiring bug.
2. On each model turn start, open a pending span with a generated span id, the system prompt and the projected model input.
3. On each tool execution start, push a tool call with sanitized arguments and a start time.
4. On each tool execution end, record duration, success or failure and the sanitized result.
5. On turn end, attach the final assistant message, its usage and stop reason, and close the span.
6. On run end, build one record, then persist it.

Spans come out as a simple tree: one model span per turn with its tool spans as children. That's enough for a waterfall view without adopting a span model built for distributed systems.

The persistence step is the part that matters most. If the sink throws, the recorder writes one line to stderr and returns. A telemetry failure never fails a user's turn. The tests cover exactly this: a sink that throws still has to let the user's request finish normally.

The SDK trace processor for the long-loop agents

The Agents SDK, which runs chat, the Info Sheet and the builders, has its own tracing model. Out of the box it exports traces to OpenAI's hosted dashboard. We didn't want customer project data leaving our boundary for a vendor's trace viewer, so LedgerTraceProcessor is installed with setTraceProcessors, which replaces the default exporter. Adding a processor alongside the default one would have quietly kept exporting.

The processor has to deal with an awkward ordering problem. Two things have to happen before a trace is complete: the SDK has to end the trace, and our runner has to report the outcome (did the run succeed, what was the final text, which message did it produce). Either can arrive first. So:

1. The runner registers the trace id with its context before the run starts. Traces with no registration aren't ours and are ignored.
2. Span-end events accumulate in memory, keyed by trace id. Nothing is written per span.
3. Whichever of "trace ended" and "outcome reported" arrives second triggers one flush.
4. If only the outcome arrives, a 60-second grace timer flushes with whatever spans exist.
5. A failed write retries with backoff up to three attempts, then gives up and logs a structured warning.
6. On shutdown, the processor drains in-flight writes and makes a final flush pass.

The runner's outcome is treated as authoritative. A missing trace-end only means the SDK was slow. A missing outcome, for example after a restart mid-run, gets stop reason trace_incomplete so it shows up honestly instead of looking like a success.

The row is keyed by both ids. Our turn_id and conversation_id are how the rest of Connect refers to the work. The SDK's trace id goes in a separate nullable column with its own index. We kept them apart on purpose: conflating the two would have made every non-SDK agent carry a fake SDK id, and made SDK debugging depend on our naming.

The processor also maps SDK span types onto the ledger's shape. Response spans become model turns. Function spans become tool calls attached to their parent model turn by span ancestry, not by timestamp. Orphaned function calls with no model parent are kept in a synthetic turn rather than dropped. It also pulls out per-response details that the hosted viewer would have shown: reasoning summaries, service tier, prompt-cache key and cache-write tokens, and request fingerprints. Those let us debug cache misses from our own data.

Redaction is in the recorder, not the viewer

Traces contain whatever users and tools put into them, which for us means drawings, contracts and schedules. We sanitize at write time so the ledger never holds something the viewer would have to hide.

• Any object key that looks like an API key, authorization header, password, secret, token or signed URL is replaced with a placeholder, recursively.
• String values are scrubbed for secret-key patterns, bearer tokens and signature query parameters in URLs.
• Every item is capped at 4,000 characters. Oversized values become a truncated preview with a flag, so the viewer can say "this was cut" instead of pretending it's complete.
• Images become a MIME type, never bytes.
• Redacted reasoning blocks are dropped. Visible reasoning summaries are kept.

The bound matters as much as the redaction. Without it, one agent that reads a 300-page specification would make its trace row enormous and slow down every list query that touches it.

MCP tool calls get a separate, stricter table

Connect also exposes an MCP server so tools like Claude can work with a user's projects. An MCP tool call isn't an agent turn. There's no model, no tokens and no cost. It's a data-access request. Mixing it into agent_events would distort every per-turn metric, so migration 051 added mcp_tool_events: one row per call with tool name, which MCP surface, the advertised tool profile, user, workspace and project (pulled from the call's arguments when present), status, error and duration.

What it deliberately doesn't store is the content:

• Arguments are hashed. A SHA-256 of the canonical arguments supports dedupe and correlation ("is this client retrying the same call?") without keeping the values.
• The preview is short and redacted. A JSON preview of the arguments is capped at 300 characters. Known free-text and bulk fields (content, body, markdown, message, data, XER payloads and similar) are replaced with a note like "[1,842 chars omitted]".
• Results are never stored. Tool results can carry tenant documents. Usage analytics don't need them.

The recording call is wrapped so a failed telemetry insert can never affect the tool call itself. It has five indexes, each matching an access path the dashboard actually uses: recent feed, per tool, per workspace, per user and per project.

The admin views built on top

Because everything is in one table, the executive views are ordinary SQL with date filters: no ETL, no warehouse. The overview, daily time series, agent graph, trace list and conversation view are all queries over agent_events, and the agent graph (agents, tools, models, users and handoffs) needs no extra instrumentation. We cover what those pages are for in the executive admin portal article. The part worth describing here is how the trace tree is assembled.

The client-side logs model has its own tests for the hard cases: a child agent whose parent row hasn't arrived yet, sibling agents whose parent is missing, a parent that persists late, and cyclic or invalid timings. The rule throughout is to show what was recorded and never invent a parent or a timing to make the tree look tidy.

The same ledger now serves members as well. Schedule versions carry the conversation and turn that produced them, so "how was this revision built?" resolves to that turn's tool calls. The member-facing read filters by workspace and project inside the query and returns a projection with no cost or model internals.

Why a Postgres ledger and not OpenTelemetry or hosted tracing

The decision is recorded in an ADR and the decision log. Connect has zero OTel. Structured application logs plus this ledger are the whole observability story for agents. The reasons:

1. The questions are product questions, not infrastructure questions. "Cost per workspace per agent", "turns on prompt version 12 versus 11" and "eval score by user" are joins against tenants, users, config versions and eval results, which all live in Postgres already. In a tracing backend you'd be exporting that context as span attributes and then rebuilding the joins somewhere else.
2. The data is customer data. Agent traces contain project documents. Keeping them in the same database, under the same access control and backups as the documents themselves, is simpler to defend than adding a processor for a third party.
3. Telemetry feeds features. The eval loop grades ledger rows. Golden examples are promoted from ledger rows. Schedule version history links to ledger rows. That only works if the ledger is a real table you can foreign-key evaluation results to.
4. One write per run is cheap. A turn produces one insert, not dozens of span exports. At our volume Postgres doesn't notice.
5. One less system to run. No collector, no sampling configuration, no second retention policy.

What it costs:

• No cross-service distributed tracing. If Connect were a dozen services, we'd want OTel for request flow. It's a modular monolith, so we don't.
• We own the viewer. The waterfall, the trace detail and the conversation grouping are code we wrote and test.
• Retention and table growth are our problem. The jsonb content columns are bounded, but the table still grows with usage.
• Best-effort means gaps are possible. A crashed process can lose a trace. We'd rather lose a trace than fail a user's turn, and incomplete traces are labeled rather than hidden.

Why this matters if you're building something similar

• Decide what your traces are for before picking a tool. If the main consumers are cost reports, quality reviews and audit trails keyed to your own tenants, a table in your primary database will probably serve you better than a span store.
• Make telemetry best-effort, and test that it is. Write the test where the sink throws and the user's request still succeeds.
• Redact and bound at write time. Viewers change. Whatever is in the table is what you're holding.
• Keep identities separate. Store your own turn id and the framework's trace id side by side. Don't make one pretend to be the other.
• Replace default exporters explicitly. Agent SDKs often ship tracing to the vendor by default. Check what "add a processor" actually does.
• Separate model turns from data-access calls. Tool-server usage and agent turns have different shapes. Keep them in different tables, and store far less for the one that touches raw tenant data.

Where to go next

• The series overview: how we built Connect
• What the ledger is graded against: LLM evals, golden sets and an autopilot gate
• The portal that reads it: building an executive admin portal for AI agents
• What happens inside one traced turn: anatomy of an AI chat turn
• The member MCP whose calls land in mcp_tool_events: building an enterprise MCP server with OAuth

Need observability your agents' customers can trust? Plan your build.