Expensive Work Belongs Off the Request Path

There's a simple split that decides whether a chatbot over a large database will be fast or slow. On one side is work that's expensive and predictable - document embeddings, city geocoding, page re-indexing, node summarization. On the other is work that's cheap and question-dependent - a few SQL lookups over an index, composing the answer. A team that cares about low latency knows which is which and never lets them share a path.

The principle: expensive work that changes rarely doesn't run in the request. Push it into an offline pipeline, store the results, and let the request only read. What's left on the request path is the cheap, indexed part.

Embeddings are background work

Take the clearest case. Vector search needs embeddings - a vector representation of every document, entity, chunk. Computing an embedding is fairly expensive: a network call to a model, tokenization, model work. Do this in the request - embed the question, sure, but embed documents only then? - and every query pays that whole cost. Yet a document's embedding doesn't change with the question. It's a property of the document.

The right shape: embed documents offline, in an incremental pipeline, and store the vectors alongside them. The request then embeds only the question (one call) and does a cheap nearest-neighbor lookup in the index. The key property of that offline pipeline is incrementality: for each row, store a content hash and the model version. Skip unchanged rows. A model bump forces a re-embed. Editing one entity then costs one embed call, not a re-embed of the whole database. Without that, the offline pipeline stalls every update and the team stops running it - and then the embeddings drift out of sync and search gets worse.

Geocoding: one table, no API

Here's a nice example of how far the principle goes. The agent answers "where can I find X nearest" - it needs to turn a place into coordinates and compute distance to all entities. The naive version calls a geocoding API on every such query. That's expensive (quota, latency, money) and fragile (the API can be down).

The better version notices that catalog places change rarely. Geocode them once, with a backfill script, and store the coordinates in a table. Most queries then just read coordinates from the database - no external call. The geocoding API stays only for the case where the user types a village or address that isn't in the catalog. And even there it deserves a cache: positive results and ZERO_RESULTS are cached (a badly-typed place mustn't burn quota every turn), but transient errors mustn't be cached - a retry might succeed. What was meant to be an expensive API call becomes a narrow exception.

One source of truth, no sync

Here's a structural decision that looks like architecture but is really a latency trick. A lot of "latency" in systems with a search engine isn't inference - it's synchronization. You edit an entity in the database, the search engine has to reindex it, and until then it returns stale data. The team that spends time tuning this sync never reaches "edited → consistent immediately."

The solution, where it's possible: runtime search runs purely over the database that's the source of truth. No separate search engine to synchronize. Hybrid search - full-text, trigrams, vectors - runs as SQL queries over indexes on one database. An edited entity is consistent in the next query because it's the same database. No replica, no lag, no endless sync work.

Fusion without tuning

When you run several search strategies at once - lexical, trigram, vector - you have to merge their scores. The instinct is a weighted sum, which drags you into endless tuning of weights between incommensurable scores (cosine vs. ts_rank). The better choice is Reciprocal Rank Fusion: scores aren't summed, they're combined through rank - each strategy contributes 1 / (k + rank). There's nothing to tune. It works across strategies out of the box. The team tuning weights on a hybrid search engine is usually tuning the wrong thing - and burning time they could've spent on something that actually moves latency.

Push the computation into the database

Last move. When the answer needs a computation over results - sort entities by distance from point X - the instinct is to pull rows into the app and compute in the interpreter. That's expensive: you fetch data you don't need and do work the database can do itself. The better path: write that computation as a SQL expression and run it in the query, over the index. Distance, ranking, aggregation - all expressible in SQL, running alongside the search. One query, one round-trip, result done. And a statement_timeout guards against a query that went bad - better a fast error than a hanging request.

What to take with you

Low latency over a large database isn't about a faster model. It's about what you don't let onto the request path. Embeddings, geocoding, summarization, re-indexing - expensive and predictable - belong in an offline pipeline, incremental, storing hash and model version. The request should do only cheap indexed lookups over one database that's the source of truth. Score fusion should be tune-free. Computation over results should run in SQL, not the app. The team that holds this line finds that its "speed" stopped being about inference long ago - inference is now the cheapest part of the path.

---

Your chatbot paying latency for work that changes once a week? Most of the gain on big RAG databases comes from what you push off the request path. Let's look at what your bot computes at runtime even though it hasn't changed since yesterday.