Vector Databases: A Systematic Introduction and Survey

Vector databases have rapidly emerged as a core infrastructure layer underpinning the latest generation of AI applications, particularly those that demand rapid, semantically meaningful retrieval at scale or depend on retrieval-augmented generation (RAG) pipelines. As the field matures, a sophisticated ecosystem of tools, algorithms, and operational paradigms has developed to address the unique challenges posed by managing and searching high-dimensional vector spaces. This paper provides a comprehensive academic overview of vector databases, delving into foundational theory, algorithmic advances, operational considerations, evolving use cases, and a survey of leading production systems as of late 2025.

Introduction

The landscape of information retrieval has been fundamentally transformed since 2020, driven by the explosive progress in deep learning-based embedding models and dense retrieval techniques. Embeddings derived from large neural networks now capture nuanced relationships within text, images, audio, and multimodal data, enabling systems to reason about semantic similarity far beyond the capabilities of traditional keyword or attribute-based search. This evolution has exposed the limitations of legacy database paradigms—relational, document-oriented, and key-value stores—when tasked with performing similarity search over massive collections of high-dimensional vectors. The need for systems capable of executing approximate nearest neighbor (ANN) search in hundreds or thousands of dimensions, with low latency and at scale, has catalyzed the rise of vector databases as a specialized solution.

Definitions and Core Concepts

At their core, vector databases are engineered to ingest, store, index, and efficiently retrieve dense, real-valued vectors—embeddings—typically residing in ℝ^d, where d often ranges from 384 to 4096 in contemporary models. Each embedding φ(x) ∈ ℝ^d encapsulates semantic properties of the underlying data point x, whether it be a sentence, image, user profile, or protein sequence. These embeddings are usually generated by deep neural networks trained on large, diverse datasets, and capture complex relationships that can be exploited for downstream tasks.  

Similarity between embeddings is quantified using mathematical distance functions such as cosine similarity, Euclidean distance, or inner product. The canonical operation is the k-approximate nearest neighbor search: for a given query vector q and a dataset X of n vectors, the objective is to rapidly identify the k vectors in X most similar to q, according to the chosen metric. Vector databases are optimized to perform these searches with high recall and low latency, even as n scales to billions or more.3. The Curse of Dimensionality and the Need for Approximate Methods

In high-dimensional spaces, exact nearest-neighbour search rapidly becomes intractable. Vector databases therefore employ approximate algorithms that trade bounded recall for dramatic improvements in latency and throughput. The dominant paradigms in 2025 are graph-based and quantisation-based methods.

3.1 Graph-based Indexing (HNSW)

Graph-based indexing has revolutionized ANN search with structures such as Hierarchical Navigable Small World (HNSW) graphs (Malkov & Yashunin, 2018). HNSW organizes embeddings into multi-layered, small-world graphs that allow efficient traversal from entry points to local neighborhoods. This enables logarithmic-time searches with a favorable balance between recall and latency, making HNSW the backbone of many enterprise and open-source vector databases. Its robustness, tunability, and scalability have led to widespread adoption in systems like Pinecone, Weaviate, Qdrant, and the pgvector extension for PostgreSQL.

3.2 Quantisation and Inverted File Methods

For extreme-scale collections—on the order of billions to hundreds of billions of vectors—quantization-based techniques, particularly Product Quantization (PQ) and its derivatives, become essential. PQ compresses vectors into compact codes that approximate original distances, drastically reducing storage and accelerating search. Often, these are combined with Inverted File (IVF) indices, which partition the search space into manageable clusters, further optimizing retrieval. Open-source projects like Milvus and Meta’s FAISS have operationalized these methods, enabling large organizations to build scalable, cost-effective vector search infrastructure.

3.3 Disk-based and Emerging Algorithms

As vector collection sizes grow toward the trillion-scale, disk-based algorithms have gained traction. Microsoft’s DiskANN (Subramanya et al., 2023) exemplifies this trend by leveraging SSD-optimized data structures to maintain fast query times with minimal memory footprints. Additionally, the field is witnessing early adoption of learned indexes—data-driven approaches that construct index structures tailored to the distribution of embeddings—heralding a new era of adaptive and workload-aware vector search.

4. Principal Application DomainsThe versatility of vector databases is reflected in their rapidly expanding application domains:

• Retrieval-Augmented Generation (RAG): RAG architectures combine LLMs with vector search to inject up-to-date, contextually relevant information into generated outputs, overcoming the limitations of static training data and enabling dynamic knowledge retrieval.
• Long-context LLMs: By storing and retrieving extended context windows as embeddings, vector databases empower language models to operate over much larger effective contexts.
• Semantic and Hybrid Search: Embedding-based search offers robust semantic matching, often supplemented with hybrid techniques that blend vector and keyword filtering for more precise results.
• Recommendation Engines: User, product, and interaction embeddings drive personalized recommendation systems, improving relevance and engagement.
• Multimodal Retrieval: Vector databases enable unified retrieval across text, image, video, and audio modalities, supporting use cases ranging from media search to cross-modal reasoning.
• Anomaly and Fraud Detection: Behavioral embeddings can capture subtle patterns in user or transaction data, facilitating the detection of outliers and fraudulent activity.
• Bioinformatics and Drug Discovery: Similarity search among biological sequences, molecular structures, or compounds accelerates research pipelines in the life sciences.
• Memory Layers for Autonomous Agents: Embedding-based memory stores allow AI agents and robots to recall and generalize from past experiences, supporting continual learning and complex reasoning.

5. Survey of Leading Systems (November 2025)

1. Pinecone – Provides a fully managed, serverless vector database platform leveraging HNSW. It stands out for its strong enterprise features, compliance support, and robust SLAs, making it a preferred choice for regulated industries and mission-critical workloads.
2. Weaviate – An open-source, cloud-native system with a flexible hybrid search engine, GraphQL API, and a thriving community. Weaviate has become a mainstay in research and developer circles, valued for its extensibility and ease of use.
3. Milvus (Zilliz) – Designed for massive scale, Milvus excels at handling collections from billions up to hundreds of billions of vectors, with deep support for quantization and distributed deployment. Its popularity in Asia-Pacific underscores its global reach.
4. Qdrant – Written in Rust, Qdrant delivers high performance for on-premises deployments, advanced payload filtering, and granular access controls, making it a compelling choice for organizations with strict data residency and compliance requirements.
5. pgvector – By extending PostgreSQL with HNSW and IVF capabilities, pgvector enables seamless integration of vector search into existing relational workflows, accelerating adoption among enterprises seeking simplicity and familiarity.
6. Chroma – Optimized for lightweight, embedded use cases, Chroma is the go-to solution for prototyping, education, and rapid experimentation.
7. LanceDB – Emphasizes a columnar, versioned storage model suitable for data science workflows and multimodal datasets, facilitating reproducibility and efficient analytics.
8. Vespa – A large-scale engine with integrated vector support, Vespa powers demanding applications at web-scale, as exemplified by its deployment at Verizon Media.
9. Elasticsearch & OpenSearch – These established search platforms have incorporated dense_vector fields and HNSW indexing, enabling organizations to extend their existing infrastructure to support vector search without major overhauls.
10. Redis (RedisVector) –Offers in-memory vector search capabilities, fitting naturally into environments already leveraging Redis for caching or real-time analytics.
    

6. Selection Criteria and Decision Framework

Academic and industrial practitioners may select systems according to the following dimensions:

• Operational model (managed vs self-hosted)
• Scale requirements (millions vs billions vs trillions)
• Compliance and data-sovereignty constraints
• Need for hybrid search and complex filtering
• Integration with existing infrastructure (PostgreSQL, Redis, etc.)
• Maturity of multimodal and graph traversal features

7. Conclusion

Vector databases have transitioned from a research curiosity to foundational infrastructure for artificial intelligence systems. The combination of mature approximate indexing algorithms, production-grade implementations, and rapidly improving embedding models has made semantic retrieval a commodity capability. As embedding dimensionality and dataset sizes continue to grow, further advances in learned indexes, quantisation, and disk-based methods are anticipated.

Researchers and practitioners entering the field in late 2025 are encouraged to begin experimentation with lightweight systems (Chroma, LanceDB, or pgvector) before progressing to managed or large-scale alternatives according to their specific requirements.