Vector Databases & RAG: What I Learned Building a 10,000-Document Search System

When people say "RAG," they often make it sound like a product feature, but in a production AI agent platform it is really a control system for truth: instead of asking the model to hallucinate its way through documentation, contracts, tickets, and internal playbooks, you turn those assets into embeddings, retrieve the most relevant chunks at query time, and force the answer to anchor itself to retrieved context.

Why RAG Matters

For agents, retrieval is not optional. The moment your workflow moves from "answer a question" to "decide which tool to call, which policy applies, and which document version is authoritative," raw model memory stops being enough. The retrieval layer becomes the difference between an assistant that sounds smart and an agent that is actually operationally safe.

My biggest lesson was that a good RAG system does not just improve answer quality — it narrows the action space and gives the agent a defensible basis for every downstream decision. In practice, that meant my Hono.js backend treated retrieval as a first-class subsystem: every query produced a retrieval trace, every returned chunk carried source metadata and version stamps, and every answer was assembled from a ranked context window.

Qdrant, Upstash, and the Dual-Vector Split

I tested both Qdrant and Upstash Vector as if I needed one winner, and that turned out to be the wrong framing. The real question was not "which is better" but "which job is each one better at." Qdrant gives you rich payload storage in JSON, filtering on payload values, and payload indexing on specific fields. Upstash Vector is a serverless vector database built around DiskANN with metadata support and low-latency similarity queries.

So I ended up with a dual-vector architecture: Qdrant held the full corpus as the system of retrieval record — document chunks, tenant metadata, ACL hints, embedding version, source type, and freshness timestamps — while Upstash Vector held a hot working set of recent, high-frequency, high-value chunks for edge-facing paths.

The opinionated part: if your entire product is one small corpus with one retrieval pattern, pick one store and keep your life simple. But once you have heterogeneous access patterns, multi-tenant filters, and a mix of hot and cold knowledge, one vector store usually becomes a compromise, not a solution.

The Cache Architecture

The fastest retrieval is the retrieval you do not repeat. The biggest performance win did not come from swapping embedding models or tweaking top-k — it came from a two-layer Redis cache in front of the vector path. Layer one cached query plans and hit lists using a normalized fingerprint (tenant + query hash + filters + embeddingVersion + topK). Layer two cached hydrated chunk payloads and rerank-ready text blocks keyed by chunk ID.

I used Upstash Redis because its globally replicated architecture places reads from nearby replicas — the perfect fit for a cache in front of a latency-sensitive retrieval service. With that setup, warm searches routinely landed under 500ms end to end: 20–40ms for cache lookup, 80–150ms for hot-vector retrieval on a layer-one miss, 40–70ms for chunk hydration on a layer-two miss, and the rest on reranking and response assembly.

Chunking and Embeddings

Semantic chunking mattered more than I expected. Fixed 500-token windows are easy to implement but terrible at preserving meaning across heterogeneous sources like docs, tickets, PDFs, and internal wikis. I chunked by structure first and size second: headings, list boundaries, table sections, code fences, and paragraph groups defined the candidate segments, then a token budget step merged or split them to stay within a stable embedding envelope while preserving local coherence.

For embeddings, I used OpenAI's embeddings API in batches. The API accepts arrays of strings, but it imposes practical limits on per-request size and total token budget — which forces you to build a real ingestion queue instead of pretending you can dump arbitrary document blobs into one call.

My best results came from versioned embeddings plus aggressive metadata: every chunk stored docId, chunkIndex, sectionPath, tenantId, sourceUpdatedAt, embeddingVersion, and a lightweight content checksum.

Production Gotchas

The first real production bug was stale embeddings, not bad retrieval. Documents change quietly while vectors look valid forever. A system that reindexes only on explicit upload events will drift out of sync and still return high-similarity results for obsolete text. The fix was to treat embeddings as derived artifacts with versioning, checksums, and freshness metadata, then use partial payload updates in Qdrant to mark chunks dirty before asynchronous re-embedding jobs rewrote the vector state.

The second gotcha was cold starts across the whole path — not just one service. Cold edge function, cold Redis connection, cold vector client, cold embedding request, cold document hydration from Postgres — all stack. My search SLA improved only after I added scheduled warmers for hot tenants, kept a rolling cache of top queries, and separated ingestion workers from request-time retrieval.

The third gotcha was cost. Most teams waste money on embeddings before they waste money on inference. If you re-embed entire documents on every edit, keep giant chunks, skip deduplication, and store every low-value chunk in your hot path, your bill grows long before your quality does. The boring fixes were the best ones: checksum-based skip logic, document-level diffing, TTLs on hot-vector mirrors, and keeping Neon PostgreSQL as the source-of-truth catalog so vector stores stayed optimized for retrieval.