Voice Latency Under 1 Second: The Architecture Behind a Real-Time AI Travel Assistant

I built TourizmAI because voice is brutally honest: a chat UI can hide latency behind typing indicators, but a spoken interface gets judged in milliseconds, and once the system pauses too long the user stops feeling like they are in a conversation and starts feeling like they are waiting on infrastructure. Around 1 second is the point where a user can notice delay but still stay in flow — which is exactly why I treated sub-second perceived latency as a product requirement rather than a performance nice-to-have.

For a tourism assistant, that threshold matters even more because travel queries are short, contextual, and often interrupt-driven: "nearby cafes," "best time to visit Rajgir," "book a guide," "translate this," and "what's open right now" all feel conversational. If the assistant takes two or three seconds before speaking, the entire experience feels less like a guide and more like a laggy IVR.

Voice Pipeline

The runtime path is straightforward on paper: user speech comes in, Vapi handles the live voice pipeline, speech is transcribed, the text goes to an LLM through OpenRouter, the model response is post-processed, TTS generates audio, and the client starts playback as soon as it has enough buffered audio to speak naturally. Vapi's documented pipeline is essentially this exact sequence — user audio, VAD, transcription, start-speaking decision, LLM, TTS, optional wait delay, then assistant audio — and endpointing and interruption handling are first-order latency variables, not implementation details.

OpenRouter was the right model gateway because it exposes hundreds of models behind one API with streaming support and fallback handling. This let me switch between Claude for nuanced itinerary generation, GPT-4-class models for tool-heavy orchestration, and Llama-family models for cheaper background tasks — without rewriting the transport layer every time I wanted to tune quality versus cost.

The big implementation decision was to optimize for perceived latency instead of full-response latency: I do not wait for the perfect answer before speaking; I wait for enough grounded intent to start the response, then stream the rest through the system.

Retrieval Under 300ms

Travel assistants live or die on retrieval quality because generic model knowledge is not enough for local recommendations, seasonal context, curated itineraries, or business-specific travel inventory. I kept the semantic search path separate from the LLM path and pushed retrieval into a fast pre-answer stage. Upstash Vector was a good fit because it supports similarity search with metadata filtering — letting me scope results by city, category, language, seasonality, budget band, and tenant before the model ever saw a chunk of context.

To keep recommendation fetches under 300ms, I paired Upstash Vector with a Redis-first lookup strategy: the first Redis layer caches normalized query fingerprints and final hit lists, and the second caches hydrated place cards, summaries, and ranking features — so most repeated or near-repeated tourism queries avoid a cold vector round trip entirely.

Upstash Redis's globally replicated architecture serves reads from nearby replicas, which matters for a latency-sensitive data layer receiving geographically distributed traffic. The practical effect is that queries like "family-friendly places near me" or "2-day Nalanda itinerary" often resolve in one fast cache read plus a lightweight rerank step.

Concurrency at 50+ Users

Handling 50-plus concurrent voice users was less about raw CPU and more about preventing contention between three very different workloads: live audio streams, retrieval bursts, and model streaming responses. I kept the Hono.js backend thin and event-driven, moved conversation state and user data into Neon PostgreSQL, used Clerk Auth at the edge for session-aware access control, and treated each voice call as a state machine with hard time budgets for STT, retrieval, inference, and TTS.

The key scaling trick was isolation: speech ingestion, vector retrieval, model inference orchestration, and response persistence each ran as separate bounded phases. A slow recommendation lookup could not freeze audio handling, and a slower model could not block transcript commits or presence updates. I also avoided per-user in-memory dependency on a single node, so I could fan requests out horizontally and recover cleanly if one process died mid-conversation.

What surprised me is that concurrency failures usually looked like latency failures before they looked like outages: queue depth would creep up, first-audio times would drift, and users would start talking over the assistant. I monitored turn-start delay, partial transcript delay, vector search percentile latency, and time-to-first-audio as separate SLOs — not just errors per minute.

Profiling and Lessons

Latency profiling taught me that developers often blame the model first when the real tax is elsewhere — especially in endpointing, buffering, and avoidable waits. VAD-based interruption can react in roughly 50–100ms, while transcription-based interruption is slower (often 200–500ms), and waitSeconds is applied after LLM and TTS work is already done. A badly chosen delay can sabotage an otherwise fast stack.

My first real optimization win came from shaving silence, not tokens: tighter endpointing, faster cache hits, smaller retrieval payloads, and earlier TTS kickoff mattered more than squeezing a few percentage points out of prompt length. The second win was routing by task instead of loyalty to one model — OpenRouter made it easy to send real-time travel Q&A through a fast model, itinerary polishing through a stronger model, and fallback traffic to cheaper options using the same API surface.

The lesson I'd pass on to any developer building voice AI: instrument every boundary. Time disappears in tiny places — microphone buffering, STT finalization, vector misses, auth checks, prompt assembly, TTS startup, client audio buffering — and if you don't measure those separately, you'll spend weeks "optimizing AI" when the real problem is that your system is waiting politely in six places before it says a word.