The Prefix Cache Is a Latency Weapon: Keep Your Prompt Still

People imagine a chatbot's latency as a property of the model - how fast it can generate. In truth, a lot of what you feel as "speed" is work that was already done before your user typed anything. Large models now ship with a mechanism called the prefix cache (or KV cache): when the beginning of a prompt matches something the model recently processed, it skips recomputing those tokens and the answer starts almost instantly. It's like having the engine already warm.

This cache has one cruel property, though: it only works on a static prefix. Any single thing in it that changes turn to turn invalidates the cache from that point on. And a team that cares about low latency turns that into a whole design philosophy.

What invalidates the cache

Take the naive agent. Its system prompt starts: "You are an advisor. Today is {date}. The user is on page {page}. You have these tools: [a long list of twenty tools]." It looks reasonable. But that prompt changes every turn - date, page, maybe even tools. So the prefix that could be cached... isn't one. The model recomputes it all every time. You're paying to re-traverse thousands of tokens that are byte-for-byte identical to last time.

Now the right instinct: put what's static first; put what changes behind it. The system prompt - role, tone, rules, safety boundaries, tool descriptions - stays put, identical turn to turn. Only after it, at the point where the cache would end anyway, comes the narrow per-turn bundle: today's date, current page, the catalog of currently-unlocked skills. The front cache holds. The per-turn injection costs little, because it's small.

Few tools from the start

Here's the second, less obvious move. A long tool list in the system prompt isn't just cognitive noise for the model - it's cache payload. The more tool descriptions you carry in the prefix, the harder it is to keep the prefix stable, because every new tool you add changes the text and thus what would cache. And a long prompt costs more time on the first pass all by itself.

The move that frees you: don't publish all your tools at once. Start with a narrow set - say three tools that cover the overwhelming majority of everyday questions - and hide the rest behind a meta-tool that unlocks skills only when the model needs them. The ordinary answer sees only the small, stable prompt. The complex answer pays one extra step to summon the skills it needs - which is correct, because that work really is harder.

You get three wins at once. The prompt is short, so inference is faster even without cache. The prompt is stable, so the cache hits. And the model is less confused by a long menu of options, which lifts quality - and that, honestly, matters as much as speed.

Situational guidance without breaking the prefix

The trick teams forget: they want the agent to behave differently depending on the situation - the tone for someone still exploring differs from someone already choosing. The instinct is to swap the system prompt per situation. It "works," but it breaks the cache, because that dynamic bit sits in the prefix.

The better path leaves the system prompt alone and routes situational guidance through a different channel - a tool that classifies the turn's situation and returns a situational playbook as a tool result, slotted in behind the cacheable prefix. The static part stays static. The dynamic guidance lands where the cache ended anyway. Same effect, the cache stays whole.

What to take with you

The prefix cache isn't a detail the optimization team gets to later. It's a structural decision about what goes where in your prompt. The rule is simple and unforgiving: everything in the prefix must be immutable turn to turn. Role, tone, boundaries, descriptions of the always-on tools - that's the prefix. Date, page context, the skill catalog, situational guidance - that comes behind it. Do this from day one and your "first token" will feel faster than the model's own decision-time would justify. Skip it and you're paying to recompute things it already computed once.

---

Your "fast model" sagging under a dynamic system prompt? Most of the time-to-first-token gain we see comes from making the prefix cache actually hit. Let's look at what's changing in your prompt when it shouldn't.