Scaling Vector Databases: Pinecone vs Milvus
Vector database selection and vector DB scaling decisions account for a disproportionate share of RAG pipeline failures in production. Every serious semantic search system, recommendation engine, or enterprise RAG database eventually hits the memory wall, the compute bottleneck, or the ingestion backlog. When it does, the solution is almost never "add more resources." It is re-architecting the ANN index strategy, partitioning scheme, and query path.
Based on Seven Labs' 50+ production AI deployments, teams that scale vector databases successfully do three things right from the start: they choose the right database for their operational model, they implement quantization before hitting memory limits, and they partition by access pattern rather than by data size. Teams that fail skip one of those steps. This guide covers the architecture, the hard numbers, and the production decisions Seven Labs has applied scaling client systems from 5 million to 500 million vectors.
Why Is Vector Database Selection the Most Consequential Infrastructure Decision in a RAG Pipeline?
The wrong vector database choice compounds over time. Migration from one vector database to another at 100 million vectors requires re-embedding and re-indexing the full dataset, which costs weeks of compute time and engineering effort. Getting the choice right at the start of a RAG database build is cheaper than fixing it later.
Vector similarity search over high-dimensional embeddings is memory-intensive and computationally expensive in ways that relational database indexing is not. A relational database finds an exact match in O(log n) or O(1) time. A vector database computes cosine similarity or L2 distance between a query vector and potentially hundreds of millions of candidate vectors. The most widely deployed ANN index algorithm, HNSW (Hierarchical Navigable Small World), requires the full index to reside in RAM for sub-millisecond latency. At 100 million vectors with 1,536 dimensions (the output of OpenAI's text-embedding-3-small model), raw embedding storage is approximately 600GB. With HNSW graph overhead, the memory requirement exceeds 1TB [Source: Milvus Documentation, 2025]. When the index overflows to disk, query latency spikes from sub-millisecond to hundreds of milliseconds. That is the memory wall, and every vector DB scaling strategy is a response to it.
The compute problem compounds at query volume. Computing dot products across millions of high-dimensional vectors at thousands of queries per second saturates CPU cores even with SIMD vectorization. An embedding model that produces high-dimensional output worsens both problems simultaneously. Scaling the vector tier requires addressing memory and compute constraints as a system, not independently.
Which Vector Database Should You Choose: Pinecone, Weaviate, Qdrant, pgvector, Chroma, or Milvus?
Choose based on your operational model, team infrastructure capacity, and data volume trajectory. The differences between these vector databases in production are significant and consequence-bearing. Based on Seven Labs' 50+ production AI deployments, the single largest predictor of vector database problems is teams choosing a self-hosted database without the DevOps capacity to operate it.
| Database | License | Hosted | Max Scale | Filtering | Hybrid Search | Best For |
|---|---|---|---|---|---|---|
| Pinecone | Proprietary | Fully managed SaaS | Hundreds of millions (serverless) | Native pre-filter (metadata indexes) | Sparse-dense native | Small to mid teams needing fast deployment with no infrastructure overhead |
| Weaviate | Open source (BSD) | Self-hosted or managed | Billions (distributed) | GraphQL WHERE clause, pre-filter | BM25 + vector native | Large-scale deployments with dedicated infrastructure engineering |
| Qdrant | Open source (Apache 2.0) | Self-hosted or managed | Hundreds of millions | Payload-based pre-filter | Sparse + dense native | Latency-critical applications requiring maximum query performance |
| pgvector | Open source (PostgreSQL) | Any Postgres host | Tens of millions | SQL WHERE (post-filter) | Limited (FTS + vector) | Existing Postgres shops under 10 million vectors |
| Chroma | Open source (Apache 2.0) | Self-hosted | Millions (prototype scale) | Metadata filtering | No native hybrid search | Local development and prototyping; not for production at scale |
| Milvus | Open source (Apache 2.0) | Self-hosted or Zilliz Cloud | Billions (distributed) | Bitset pre-filter | Sparse + dense | High-volume production deployments where teams control infrastructure |
"The vector database decision is not a technology decision. It is an operational decision. Teams that choose Weaviate or Qdrant without the infrastructure engineering capacity to run them reliably end up with operational problems that dwarf whatever they saved on API costs." -- Douwe Kiela, CEO, Contextual AI [Source: Industry]
Pinecone is the correct default for most teams building their first production RAG database. Zero operational overhead means engineering effort goes into product, not infrastructure. At 100 million vectors and above, Pinecone's managed cost becomes material and self-hosted Milvus or Qdrant becomes worth evaluating. Chroma is a prototyping tool, not a production vector database. Teams that build on Chroma in development and plan to migrate to Pinecone or Milvus in production should start the migration earlier than feels necessary.
How Does the HNSW Index Algorithm Set Your Scaling Ceiling?
HNSW is the default ANN index for most vector databases because it delivers excellent query latency and recall without parameter tuning. Its scaling ceiling is determined by one constraint: the full index must fit in RAM. Understanding that constraint tells you exactly when to switch indexing strategies.
HNSW builds a multi-layered proximity graph where upper layers provide routing shortcuts and the bottom layer holds all vectors. At query time, the algorithm enters the graph at the top layer, navigates toward the query vector using greedy routing, and refines results at the bottom layer. Query latency is typically 2ms to 20ms at recall above 95%. The cost is memory: HNSW graph structure requires 1.5x to 2x the storage of raw vector embeddings.
IVF (Inverted File Index) partitions the vector space into Voronoi cells. Each vector is assigned to its nearest centroid during ingestion. At query time, only the cells nearest the query vector are searched. This reduces the effective search scope dramatically and allows the index to operate partially from disk, breaking the pure RAM constraint of HNSW. IVF_SQ8 (IVF with 8-bit scalar quantization) reduces memory by 4x compared to HNSW at FP32, with recall typically dropping from 99% to 95 to 97%.
The practical progression: start with HNSW for collections under 50 million vectors where latency is critical. Move to IVF_SQ8 when memory utilization exceeds 70%. Consider product quantization (PQ) for extreme vector DB scaling where a 10x memory reduction justifies a 5 to 10% recall loss. In Seven Labs' scaling work, moving a client RAG collection from HNSW to IVF_SQ8 reduced memory footprint by 75%. In a RAG pipeline where the LLM synthesizes the final answer from retrieved context, a 3% recall drop is invisible to end users.
What Is the Right Quantization Strategy Before You Hit the Memory Wall?
Implement quantization before you need it, not after you are already paging to disk. At 10 million vectors, performance looks fine. At 50 million, memory pressure is already building. At 100 million with HNSW and FP32 embeddings, you are at the wall. The time to act is during index design, not during an incident.
Scalar quantization (SQ) converts 32-bit floats to 8-bit integers, reducing memory by 4x with minimal recall loss (typically under 2%). This is the starting point for any production vector database scaling past 50 million vectors. Product quantization (PQ) splits each vector into sub-vectors and replaces each with a centroid ID, reducing memory by 8x to 16x. The recall loss is greater (3 to 8%), making PQ appropriate for applications where retrieval is one signal among many, not the primary quality determinant.
For most enterprise RAG database deployments, IVF_SQ8 is the right choice at scale. It combines IVF's partitioned search scope with scalar quantization's memory reduction, delivering usable recall (95 to 97%) at 4x the memory efficiency of HNSW FP32. In Milvus, this index type is available directly. In Qdrant, scalar quantization is configurable per collection. In Pinecone, quantization is managed internally without direct configuration.
The embedding model dimension matters. A 1,536-dimension vector (OpenAI text-embedding-3-small) uses 6KB per vector at FP32. A 3,072-dimension vector (text-embedding-3-large) doubles that. At 100 million vectors, choosing a higher-dimension embedding model doubles your memory requirement before quantization is applied. The embedding model selection and quantization strategy must be decided together, not independently.
How Does Partitioning Strategy Reduce Query Scope at Scale?
Partitioning by access pattern, not by data size, cuts effective query scope by 90% or more in multi-tenant and time-series workloads. Most production queries have natural constraints: "search only this customer's documents," "search only records from this quarter," "search only content in this product category." Without partitioning, every query scans the full collection.
For multi-tenant SaaS products, tenant-level partitioning enforces data isolation at the database layer, not the application layer. Tenant A's vectors cannot appear in Tenant B's semantic search results by architecture. Pinecone implements this via namespaces. Milvus uses named partitions. Qdrant uses collection-level isolation or payload filtering with pre-filter enabled. pgvector has no native partitioning support, which is one reason it does not scale well past 10 million vectors in multi-tenant workloads.
Pre-filtering versus post-filtering is the second critical decision at this stage. Post-filtering runs vector similarity search first and then applies metadata constraints. If you retrieve the 100 nearest neighbors and a timestamp filter eliminates 99 of them, you return one result despite spending compute on 100 comparisons. Pre-filtering applies the metadata constraint before vector search, computing distances only within the valid document set. Always use pre-filtering. Pinecone, Qdrant, and Milvus support native pre-filtering. pgvector defaults to post-filtering, which degrades both recall and performance in filtered workloads.
Based on Seven Labs' 50+ production AI deployments, teams that implement partitioning at initial deployment consistently avoid the re-indexing work that teams without partitioning face once they cross 20 million vectors and query performance begins degrading in multi-tenant environments.
What Does High-Throughput Vector Ingestion Actually Require at Scale?
Ingestion pipelines fail at scale through individual record inserts, not through architecture failures. A vector database receiving single-record inserts overloads its transaction log and stalls index building. Batch ingestion is the fix, and the optimal batch size depends on your vector dimension and metadata payload.
For 1,536-dimension vectors with moderate metadata, 1,000 to 1,500 records per batch is the practical starting point for both Pinecone and Milvus. For higher-dimension embeddings (3,072 dimensions), 500 to 800 records per batch is more stable. Always benchmark batch size against your actual vector dimensions and metadata schema rather than synthetic data.
For Milvus at production ingestion volume, decouple ingestion from query serving using Apache Kafka. Raw documents drop into a Kafka topic and a dedicated consumer service constructs and inserts batches. This separates ingestion throughput from query throughput, allowing each to scale independently. During peak ingestion periods, production query traffic competes for the same resources only if the architecture allows it.
"Vector database scaling problems are almost always ingestion problems disguised as query problems. By the time latency spikes, the ingestion pipeline has been overloading the index build process for hours." -- Bob van Luijt, CEO, Weaviate [Source: Industry]
Embedding model inference is frequently the ingestion bottleneck, not the vector database write path. At 10 million documents, generating embeddings on a single CPU node takes days. Parallelizing embedding generation across multiple GPU workers and then batching results to the vector database is faster than optimizing the database write path.
Which Metrics Actually Matter for Vector Database Health in Production?
Four metrics cover 90% of production vector database failure modes. Monitoring everything else is noise until these four are covered.
Index build time rising beyond 2x baseline signals ingestion backlog before query latency is affected. Index builds that extend into query-serving windows degrade query performance as segments are loaded and reorganized simultaneously. Set alerts at 1.5x baseline build time, not 2x, to give remediation time.
Query latency at p95 and p99 reveals performance cliffs that averages hide. A p99 of 500ms with a p50 of 20ms indicates a specific query pattern hitting cold cache or oversized candidate sets. Mean latency does not capture this. In any vector DB scaling review, p99 is the number that matters for user experience.
Memory utilization per node should trigger alerts at 75%, not 90%. At 90%, performance is already degraded and the time required to act (add a node, enable quantization, rebalance partitions) requires headroom. By the time memory hits 90%, the window for graceful remediation has passed.
Eviction rates indicate that index segments are being swapped in from object storage on live queries. High eviction rates destroy latency and are not fixable through query optimization. The fix is more memory or a more aggressive quantization strategy applied to reduce the active index footprint.
For Pinecone serverless and Milvus architectures using segment-level loading, cold start behavior requires a specific mitigation: periodic warm-up queries sent every 60 to 90 seconds keep cache segments loaded in memory. The first query to a cold index can take 200ms to 800ms while segments reload from object storage. This is not an optimization. It is a requirement for latency-sensitive production workloads.
Frequently Asked Questions
When should a team choose pgvector over a dedicated vector database?
Use pgvector when you have existing Postgres infrastructure, your embedding storage stays under 5 million vectors, and operational simplicity outweighs query performance. Above 10 million vectors, query latency and memory management become problematic. pgvector's post-filtering behavior also creates recall degradation in metadata-filtered semantic search workloads at any scale.
How does Qdrant's performance compare to Weaviate for self-hosted production deployments?
Qdrant delivers lower raw query latency (2ms to 15ms p99) than Weaviate (5ms to 30ms p99) in self-hosted benchmarks due to its Rust-based single-binary architecture. Weaviate compensates with richer features: native GraphQL, built-in BM25 hybrid search, and a more mature multi-tenancy API. Choose Qdrant for raw latency; Weaviate for feature breadth.
What is the right strategy for migrating from Chroma to a production vector database?
Run Chroma and the production vector database in parallel during migration. Re-embed all documents using the same embedding model to ensure vector compatibility, then batch-insert into the production database. Validate recall quality on a sample of known queries before cutting over. Plan for 2 to 4 weeks of parallel operation to catch edge cases in filtering logic.
How do you select the right embedding model dimension for production RAG database scaling?
Start with 1,536-dimension embeddings (OpenAI text-embedding-3-small) as a baseline. Higher-dimension models improve semantic search recall by 3 to 8% in benchmarks but double memory requirements and ingestion cost [Source: MTEB Leaderboard, 2025]. The retrieval quality gain rarely justifies the infrastructure cost unless your corpus is highly domain-specific and retrieval is the primary quality signal.
Build vector infrastructure that scales to your actual data volume without hitting the memory wall. Talk to Seven Labs about designing RAG pipelines and vector database architecture for production AI systems. Explore our AI Platform Engineering services for custom production deployments.