Prompt Caching for AI Agents: How We Structure Prompts and Model Clients in Connect

Most of what an AI agent costs, in money and in waiting time, is input tokens it has already seen. A chat turn resends the same primer and answering rules every time. A Schedule Builder turn resends a firm's whole scheduling SOP on every model call, and one build makes hundreds of them. Prompt caching turns that repetition into a discount and a latency win, but only if the prompts and the model clients are built for it. We designed Connect, the agentic scheduling platform we built on top of Syncify, around that from the start.

This article covers how the caching works across Connect's agents: how prompts are ordered, how cache keys are chosen per role, how long sessions stay inside the context window, when cheap child models and lower reasoning effort take over, and how we check whether any of it is working. The chat-specific prompt assembly is walked through step by step in anatomy of an AI chat turn. This article is the cross-agent view.

How implicit prompt caching actually behaves

Connect's long-running agents (chat, the Info Sheet, the App Builder and the Schedule Builder) run on the OpenAI Agents SDK against Azure OpenAI through Azure AI Foundry, using the Responses API. The narrower agents run on a lighter runtime, but the same layout rules apply to their prompts. The provider offers implicit prompt caching. Three properties of it drive every decision below:

1. It is a prefix cache. A request reuses cached computation up to the first byte that differs from an earlier request. One changed character near the top throws away everything after it.
2. A cache key routes requests. Requests that share a key are steered to where the matching prefix is likely to be warm. The key is a routing hint. It is not a data boundary, and the cache still matches on content.
3. Nothing is guaranteed. A hit is not promised, and how cached tokens are billed depends on the deployment. So a cache strategy that is never measured is a hope, not a strategy.

The practical consequence is that prompt layout becomes an architectural concern. Where a block sits in the prompt decides whether it is paid for once per session or once per call.

Static first, volatile last

Every Connect agent builds its prompt in two halves: an invariant prefix that is byte-identical across calls, then a volatile tail. The rule is simple: nothing that changes per call (timestamps, ids, retrieved passages, live state) may sit above anything that does not.

Here is an illustrative layout for the Schedule Builder, the agent with the longest sessions. Block sizes are orders of magnitude, not exact figures.

Two details in that table are worth calling out.

The identity line names the project. That means the warm prefix is effectively per project, not shared across all builds. We accepted that. Within one build the prefix is byte-identical across a few dozen user turns and hundreds of model calls, which is where nearly all the repetition is.

The volatile state sits at the end of the instructions. The runtime appends a compact summary of the Build Spec and the last build report after the stable prompt on every pass. When that summary changes between repair passes, everything after it, including session history, is re-read. Within a single pass, though, the instructions are fixed and history only grows by appending, so each tool call in the loop hits on everything before the newest item. That is the shape you want: one cold read per pass, warm reads for every step inside it.

Chat follows the same pattern at a smaller scale: domain primer, curated knowledge (budgeted at 20,000 characters), the Syncify tools primer and the answering rules lead; retrieved passages, history and attachments trail. Toggles like "tools available" vary by the shape of a turn, not its content, so turns of the same shape share a prefix.

Split stable and volatile versions of the same thing

Curated knowledge and knowledge retrieved for the current message look like the same thing, but curated knowledge is stable for weeks and retrieved knowledge changes on every message. Carried together, they would put a volatile block high in the prompt and break the cache for everything below it. So they are separate prompt parameters in separate positions: curated knowledge in the prefix, retrieved knowledge in the tail. If two things share a label but not a rate of change, split them.

Tool definitions are part of the prefix

Tool schemas are sent with every request and count toward the cached prefix. After the Schedule Builder's first draft pass, the research and setup tools are dropped from the offer, because every strict tool schema offered slows the model's writing. That costs one cold read on the next pass. We took that trade on purpose: a smaller tool list speeds up every pass after it. The broader point is that changing the tool list mid-session is a cache decision, and it should be made knowingly.

Cache keys: one per role, never per tenant

Every model client in Connect stamps a prompt-cache key onto each request. The keys follow three rules: they carry a version, they carry no tenant or project data, and each distinct prompt role gets its own.

The version suffix is the cheap insurance. When the shape of a prefix changes (a block moves, a new section is added), we bump it, and nothing can be routed toward a stale slot.

Keeping tenant data out of keys is not a security control, because the key is not a boundary. It keeps identifiers out of a field that ends up in provider logs and our own traces, and lets identical invariant prefixes share warmth across workspaces.

The model client: a thin subclass

All of this lives in small subclasses of the Agents SDK's Responses model, one per agent family, each overriding the method that builds the request. The pattern, written from scratch here:

class CachedModel extends ResponsesModel {
  buildRequest(req, stream) {
    const body = super.buildRequest(req, stream)
    body.prompt_cache_key ??= KEY            // a per-call override wins
    body.prompt_cache_options = { mode: "implicit", ttl: "30m", ...body.prompt_cache_options }
    return body
  }
}

Two choices matter. Defaults are applied with "only if unset", so an agent that passes its own key through provider data (the reviewers, the section workers) is not overwritten. And the 30-minute implicit TTL fits how people work: long enough to cover a planner reading an agent's question and answering it, short enough not to hold prefixes nobody is using.

The Schedule Builder's client also sets a two-minute timeout on streamed requests that have not returned response headers, so a dead stream is abandoned and retried rather than held open for the SDK's much longer default. Retries are where a warm cache quietly pays off: the retried request resends the same prefix and reads most of it from cache.

Long sessions: output reserves and server-side compaction

Short chat turns run plain, with a per-turn output ceiling of 4,000 tokens. The long-running agents need more structure, so their clients set three things on every request: a fixed output reserve, truncation disabled, and server-side compaction above a threshold.

Truncation is disabled on purpose. Silent truncation drops the oldest context without telling anyone, and the oldest context is often the instruction that matters. With truncation off and compaction on, the provider summarises older history explicitly once the threshold is crossed, and the run keeps going.

The reserve is part of the budget. A threshold must leave room for the reserve plus the next input. Setting the threshold close to the window and the reserve large is how a long tool loop ends in a context-length error on its last, most important call.

Compaction and caching interact. Compaction rewrites the history portion of the request, so the history after it reads cold on the next call. The instructions above it are untouched, which is another reason to keep the expensive, stable material in the instructions rather than in replayed messages.

Chat history is bounded twice

Chat does not replay a message array at all. Recent turns are folded into the volatile tail as a plain transcript, bounded to the last 10 turns and a 6,000-character budget filled newest-first. That keeps the volatile tail small, so the uncached part of each chat request stays cheap no matter how long the conversation runs. Attachments get their own 24,000-character budget for the same reason.

Cheap work goes to cheap models

Not every model call deserves the main model or deep reasoning. Connect pushes side tasks down:

A published agent version can override the thinking level, and when it does it wins. Otherwise effort is tiered by the shape of the work.

Stepping effort down for bulk repair

The Schedule Builder makes the most interesting effort decision. A build turn is a draft followed by repair passes, each fixing the failures the acceptance gates found. The runtime runs:

1. The first draft at full configured effort, because a lower-effort draft writes a thinner schedule.
2. Bulk repair passes, while more than five gate failures remain, one effort level lower.
3. The last few failures at full effort again, because at low effort each last fix tends to knock another one loose.

The step-down has a floor: it never drops below low. It sheds latency on the passes where the work is mechanical and keeps the reasoning for the passes where a fix has to hold.

Measuring it: cache tokens in the ledger

Every agent run writes one row to agent_events, the Postgres trace ledger described in LLM agent tracing on a Postgres ledger. Cache usage gets first-class columns rather than being folded into input:

• Cache-read tokens, taken from the cached-token count in the response's input usage details.
• Cache-write tokens, recorded only when the provider reports them, and stored as unknown rather than zero when it does not.
• Reasoning tokens, kept separate from output for the same reason.

The executive dashboard sums those columns over any time range next to input tokens, cost and duration, so a cache hit rate is a division away: cache-read tokens over input tokens, per agent, per day. When a prompt change lands and that ratio drops, it shows up the next morning, not on the invoice.

The Info Sheet's client goes further, because its sessions are the most expensive. Each request carries hashed fingerprints of the full request, the instructions, the tool list, the settings and each input segment, plus the cache key, in request metadata. The trace processor stores them with each model turn. When a hit rate falls, comparing fingerprints between two consecutive calls shows exactly which part changed: a tool schema that reordered, a setting that flipped, an input segment that should have been stable. On endpoints that support it, the client also asks the provider to compare each request with the previous response and records its cache diagnostics.

Why this matters if you're building something similar

• Treat prompt order as architecture. Write the rule down (invariant first, volatile last, nothing per-call above the retrieved content) and review against it.
• Split blocks by rate of change, not by topic. Two kinds of "knowledge" that change at different speeds belong in different positions.
• Give every prompt role its own versioned key. Small side prompts sharing a slot with a large one will evict it. Bump the version when a prefix changes shape.
• Keep tenant data out of cache keys. The key is a routing hint that ends up in logs, not a boundary.
• Pair an output reserve with explicit compaction, and turn silent truncation off. Budget the threshold so the reserve and the next input always fit.
• Push side tasks to a child model at minimal effort, and step effort down for mechanical bulk work. Keep full effort for drafts and for the last fixes that have to hold.
• Log cache-read tokens per run. The provider does not guarantee hits. Your ledger is the only place you will find out whether the strategy works.

Where to go next

• How we built Connect: architecture of an enterprise AI platform, the series overview.
• Anatomy of an AI chat turn, the chat prompt assembled step by step.
• The AI Schedule Builder: build spec and gates, the agent with the longest sessions.
• LLM agent tracing on a Postgres ledger, where cache and reasoning tokens are recorded.
• Lessons from vector search on Cosmos DB for MongoDB vCore, the retrieval tier that feeds the volatile tail.

Running agents whose token bills are growing faster than their usage? Talk to us about it.