Vector search demos are easy. Vector search in production — under load, with stale data, and a real budget — is a different animal. Three lessons from the trenches.
1. Postgres + pgvector is enough longer than you'd think
Every "best vector DB" post starts with a benchmark of seven specialized engines. You probably don't need any of them.
For most B2B products, the corpus fits in tens of millions of rows, latency tolerance is 100–300ms, and recall is bottlenecked by your prompt and chunking, not the index. pgvector with HNSW gets you all of that, while keeping your operational surface area at one database.
The day you genuinely outgrow Postgres, you'll know. Until then, every other engine is one more thing that pages someone at 3am.
2. Chunking matters more than the model
I've spent more time tuning chunk size and overlap than picking embedding models. The differences between OpenAI's, Cohere's, and the open-source contenders are real but small. The differences between 200-token chunks vs 800-token chunks, with 0% vs 20% overlap, are huge.
Rules I follow:
- Start at 500 tokens, 15% overlap. Adjust based on document type.
- Respect natural boundaries. Splitting mid-sentence wrecks recall. Use heading and paragraph boundaries.
- Store the parent. Index the chunk, but return enough surrounding context to ground the LLM. Retrieval-then-rerank with the full parent works better than just returning the matching chunk.
3. Re-embed on schedule, not on every change
The naive approach: every time a document changes, regenerate its embeddings. Fine for low-write apps. Painful when a client's content team makes 5,000 small edits a day.
I queue re-embedding jobs and batch them — usually nightly. The user-facing latency stays low, the API costs stay predictable, and stale-by-a-day is fine for almost every use case I've shipped.
If you genuinely need real-time freshness, that's a different conversation — and probably means a different architecture.
What I'd skip
- Specialty vector DBs until Postgres groans. Pinecone and friends are fine, just not the first reach.
- Premature reranking layers. Get retrieval right first. A reranker on top of bad chunks is just makeup.
- "Hybrid search" before you've measured. It usually helps, but pick a baseline first so you know.
The hard part of "AI features" is rarely the model. It's the boring data plumbing around it.