Small Leaks: What Costs Latency When You're Not Looking

When a team sets a goal of "answer in three seconds," they often imagine one big problem - a faster model, better search. In reality it's a budget, broken into dozens of small expenditures, and it's often not the one big expenditure that puts you over. It's the small leaks, none significant on their own, that together decide. Here are four that get forgotten.

Stream, don't wait

The biggest perceived latency isn't the time until the answer ends - it's the time until anything first appears. A user watching text start to flow waits patiently. A user staring at a spinner grows impatient after a second. From that comes the rule: stream tokens as soon as you have them. Don't wait until the answer is complete to send it. Every chunk you send earlier pulls perceived speed down, even when total time is the same.

And it scales. If the answer carries entities - clickable cards, details - don't send them after the text. Send them in parallel, in their own frame, before the text ends. The frontend then has data for the popover ready when the user clicks the link - no spinner, no second request. Parallelizing frames is the same trick as parallelizing tool calls: independent work should run concurrently, not in sequence.

No new handshake

Here's a leak that looks innocent but compounds. When your agent calls an embedding API on every query and every call opens a fresh TLS connection, you pay a handshake - tens of milliseconds - on every request. On a single request no one notices. Across thousands of requests a day it's a consistent tax that buys nothing.

The fix is a shared keep-alive pool - the connection opens once and stays, subsequent calls reuse it. No fresh handshake, no TCP and TLS waiting. It's one of those tricks that won't fix your main bottleneck but frees up budget that something more important will need.

Fail soft, don't break

What happens when a tool fails inside an agent loop? The naive version raises an exception, the loop collapses, the user gets an error, the conversation has to restart. That's expensive not just because the user leaves - it's expensive in latency too, because a restart means the whole turn again, from zero.

The better invariant: every tool fails soft - it returns an error message instead of raising. The agent loop doesn't break. The model gets "this tool isn't working right now, try another way" and continues. Most turns where something breaks then complete with a reasonable answer ("I couldn't find that, but here are alternatives") instead of a hard error. And the ones that don't complete cost one turn, not a full restart. Soft-fail isn't just nicer UX - it's a latency optimization, because it eliminates work you'd otherwise redo.

Stop when you can

The last leak is the sneakiest, because it looks like correct work. The agent recognizes the query is ambiguous and should ask the user for clarification. The instinct is: ask, and then wait for what the model says next. But there's nothing to say - the answer is the question. And yet you let the agent make another model-call to "summarize" a situation it already summarized.

The fix is early termination: once the clarifying question is done, the turn ends. No further model-call. It's one of those moments where the agent loop has to know when to stop - and the team that doesn't make it pays a whole extra model-call for an answer that was already written. You can call it return_direct: the tool's result is the answer, the model isn't called again.

What to take with you

None of these leaks alone turns a slow bot fast. But together they're the difference between a team that makes the limit and a team that doesn't, and can't tell why - because no single step stands out. Stream what you have, send parallel frames, hold keep-alive, fail soft, stop when the answer's already written. The latency budget doesn't get spent on one big expenditure; it bleeds through a lot of small ones nobody was watching.

---

Your bot "makes three seconds" most of the time but sometimes doesn't, and you can't tell why? It's usually not the big things - it's leaks that compound. Let's find them.