The LLM Latency Playbook for Next.js
Written by Crexed
April 9, 2026
Latency kills AI UX faster than accuracy issues.
You can improve the experience dramatically without changing the model at all.
From skeletons and first-token streaming to retrieval caching and clear status messages, small frontend and infrastructure choices compound into experiences users describe as “instant.”
Stream the First Token
Streaming reduces perceived latency. Show partial output quickly, even if the full response takes longer to finish.
Example: Progressive UI for a Summary
A simple pattern: render a skeleton immediately, stream the first sentence as soon as it’s available, then progressively fill in bullet points and citations. Users perceive the app as responsive even when the final answer takes longer.
Cache the Right Things
Prompt templates
Cache stable system prompts and reusable context fragments.
Retrieval results
Cache top-k doc chunks for repeated queries within a short TTL.
UI shells
Render the layout immediately and hydrate AI content progressively.
What Not to Cache
Caching can backfire if you cache user-specific or fast-changing data. Avoid caching anything that can leak sensitive information across users. Prefer short TTLs and keyed caches (by user, org, and permissions) for retrieval and personalization.
Design for Uncertainty
Use loading states that explain what’s happening (retrieving, drafting, verifying) and allow users to interrupt or refine the query.
Build a Latency Budget (So You Know What to Fix)
Break end-to-end latency into components: network time, retrieval time, model time, and rendering time. When you can see which part dominates, optimizations become straightforward.
Perceived latency
Time to first visible progress (skeleton, first token, or status message).
Time to usable
When the user can act on the output (first bullet points, partial draft).
Time to complete
When the final, formatted answer is done.
Conclusion
Fast AI UX is mostly good product engineering: stream early, cache carefully, and design for uncertainty. When users see progress quickly, they trust the system regardless of the final completion time.