New Index Preview: RaBitQ-empowered DiskANN Index

From day one, VectorChord's core indexing solution has relied on a powerful combination: the cluster-based IVF algorithm and RaBitQ quantization—what we ship as vchordrq (IVF+RaBitQ). This approach consistently delivers excellent low-latency, high-recall performance across a wide range of use cases and remains our recommended default for most scenarios.

At the same time, we recognize that graph-based indexes can outperform IVF+RaBitQ on certain datasets and at specific (often higher) recall targets. To give you the best tool for every job—and because our goal is to be a one‑stop solution for vector search—VectorChord 0.5 introduces experimental support for the DiskANN algorithm as an additional option.

What is DiskANN and Why Does It Matter?

Traditional top-tier graph algorithms (like HNSW and NSG) achieve their speed by keeping the entire graph in memory, which drives up hardware costs and caps single-node scale. SSDs are cheaper, but historically their random I/O made on-disk graphs impractical without big latency penalties.

DiskANN's mission is to break this memory barrier. It’s a graph ANN designed from the ground up to store and search billion-scale datasets on inexpensive SSDs while maintaining competitive recall and low latency.

A visualization from the original DiskANN paper.

How our variant differs

We implement a variant of the original DiskANN design: instead of the paper’s default Product Quantization (PQ), our implementation uses RaBitQ as the quantization layer. RaBitQ is more accurate and comes with theoretical guarantees, which helps us deliver better recall–latency trade‑offs compared with vanilla PQ used in original implementations. In practice, this change enables stronger accuracy at the same footprint or faster queries at the same recall.

In VectorChord 0.5, we've integrated DiskANN into the existing ecosystem so you can evaluate it side-by-side with vchordrq.

When DiskANN Can Shine

• Workload + target dependent gains. For some model families (e.g., OpenAI/Cohere‑style embeddings) and high-recall settings, DiskANN can deliver lower latency and/or higher recall than IVF+RaBitQ on the same hardware.

• Stable and predictable build memory. It avoids the KMeans phase of IVF, reducing memory fluctuations during index build and simplifying capacity planning.

• Ecosystem compatibility. Keep using VBase for metadata filtering and RaBitQ quantization for smaller footprint and faster re-ranking.

Important Considerations (Read This First)

DiskANN is still experimental in VectorChord, and—like all graph ANN methods—it entails trade‑offs:

• Updates & build speed. Index builds and updates are much slower than with vchordrq (IVF+RaBitQ). If you have frequent inserts/deletes or need fast rebuilds, vchordrq is the better fit.

• Dynamic data performance. Modifying graph connectivity is computationally heavier than updating IVF lists; expect better operational ergonomics with vchordrq for mutable datasets.

• Operational simplicity. vchordrq tends to be simpler to tune and operate for most teams.

Our recommendation: Start with vchordrq (IVF+RaBitQ) for the majority of use cases. Consider DiskANN when you’re chasing top‑tier recall on relatively static corpora, want tighter tail latency at high recall, or when build‑time memory predictability is a hard requirement. Your feedback will help us harden this option for production.

How to Get Started

docker run \
  --name vectorchord-demo \
  -e POSTGRES_PASSWORD=mysecretpassword \
  -p 5432:5432 \
  -d tensorchord/vchord-postgres:pg17-v0.5.2

-- 1. Enable the VectorChord extension
postgres=# CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
NOTICE:  installing required extension "vector"
CREATE EXTENSION
postgres=# \dx
                                                 List of installed extensions
  Name   | Version |   Schema   |                                         Description
---------+---------+------------+---------------------------------------------------------------------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
 vchord  | 0.5.0   | public     | vchord: Vector database plugin for Postgres, written in Rust, specifically designed for LLM
 vector  | 0.8.0   | public     | vector data type and ivfflat and hnsw access methods
(3 rows)

-- 2. Create a table with a vector column
postgres=# CREATE TABLE items (id bigserial PRIMARY KEY, embedding vector(3));
CREATE TABLE
-- 3. Insert some sample data
postgres=# INSERT INTO items (embedding) SELECT ARRAY[random(), random(), random()]::real[] FROM generate_series(1, 1000);
INSERT 0 1000
-- 4. Build the DiskANN graph index using the 'vchordg' method
postgres=# CREATE INDEX ON items USING vchordg (embedding vector_l2_ops);
CREATE INDEX
-- 5. Run an approximate nearest neighbor search!
postgres=# SELECT * FROM items ORDER BY embedding <-> '[3,1,2]' LIMIT 5;
 id  |             embedding
-----+-----------------------------------
 370 | [0.9644321,0.7480163,0.95816356]
 186 | [0.9583021,0.6364455,0.9882274]
 303 | [0.9735422,0.9101731,0.92470014]
 793 | [0.93719786,0.8919436,0.9729206]
 432 | [0.97542423,0.62692064,0.9525037]
(5 rows)

Check out our official documentation for detailed instructions on configuring and using the new graph index: Guide to Using Graph Index in VectorChord

Proactive Monitoring: Recall Measurement Tool for IVF+RaBitQ

A potential challenge with vector search index is "data distribution drift". As you continuously add new data—especially when the new data's vector distribution differs significantly from the initial dataset—the index's recall can degrade over time, impacting query quality.

Previously, quantifying this degradation was difficult, often relying on intuition or complex offline evaluations to decide when an index rebuild was necessary.

To empower you to proactively and easily monitor your index's health, version 0.5 introduces a powerful new function: vchordrq_evaluate_query_recall.

This function allows you to use a small, representative set of query vectors and their ground truth nearest neighbors to quickly and accurately assess the actual recall of your current IVF+RaBitQ index.

What It Does For You:

• Quantify Performance: Turn the "feel" of your index's quality into a hard, measurable number.

• Inform Decisions: By running evaluations periodically, you can clearly visualize the recall trend and make data-driven decisions on when to rebuild your index for optimal performance.

• Enhance Service Quality: Ensure your live vector search service consistently meets its quality SLAs and prevent business impact from silent index degradation.

How to Use It

Here's how to put it into practice:

-- Select ctid from your vector table with the target query vector to evaluate
postgres=# SELECT vchordrq_evaluate_query_recall(query => $$
  SELECT ctid FROM items ORDER BY embedding <-> '[3,1,2]' LIMIT 10$$);
 vchordrq_evaluate_query_recall
--------------------------------
                              1
(1 row)

For more details and configuration, you can refer to Measuring Recall for IVF+RaBitQ Indexes

Automatically Record Queries for Recall Tracking

Keeping a high-quality vector search service isn't just about measuring recall once—it’s about measuring it continually on real traffic. Starting in v0.5.2, VectorChord can automatically record (sample) your production queries so you can evaluate recall on the exact vectors users searched for.

Why it matters

• Hands‑off data collection: No custom logging pipelines or app changes.

• Representative signals: Evaluate recall on the same distributions your system sees in production.

• Privacy‑aware & lightweight: You control when sampling is enabled and how much to collect.

Enable query sampling

-- Turn on sampling and set sensible limits
SET vchordrq.query_sampling_enable = on;
SET vchordrq.query_sampling_max_records = 1000;  -- cap the total stored samples
SET vchordrq.query_sampling_rate = 0.01;         -- sample every 100 query (adjust as needed)

After enabling, run your normal vector searches (e.g., SELECT * FROM items ORDER BY embedding <-> '[3,1,2]' LIMIT 10;). VectorChord will capture the essential components of each sampled query: schema, index, table, column, operator, and vector value.

Inspect what was recorded

SELECT * FROM vchordrq_sampled_queries('items_embedding_idx');
-- schema_name | index_name          | table_name | column_name | operator |    value
-- ------------+---------------------+------------+-------------+----------+--------------
-- public      | items_embedding_idx | items      | embedding   | <->      | [0.5,0.25,1]

Evaluate recall on recorded queries

SELECT AVG(recall_value) FROM (
    SELECT vchordrq_evaluate_query_recall(
            format(
                'SELECT ctid FROM %I.%I ORDER BY %I OPERATOR(%s) %L LIMIT 10',
                lq.schema_name,
                lq.table_name,
                lq.column_name,
                lq.operator,
                lq.value
            )
    ) AS recall_value
    FROM vchordrq_sampled_queries('items_embedding_idx') AS lq
) AS eval_results;

Note: If you run PostgreSQL replication with primary/standby servers, sampled queries are not synchronized via replication or backup. Each server records its own queries based on the traffic it serves.

Summary

VectorChord 0.5 brings unprecedented flexibility to our users. The experimental DiskANN support offers a new high-performance option for specific workloads, while the recall measurement tool gives you fine-grained monitor over the operational health of your existing IVF indexes.

We believe that empowering users with more choices and greater control is the driving force behind VectorChord's evolution. We sincerely invite you to try out the new version and share your feedback, especially on how DiskANN performs in your real-world scenarios.

• Download/Star on GitHub: https://github.com/tensorchord/VectorChord

• Join the Community: https://discord.gg/KqswhpVgdU

Ready to take it for a spin? Head over to our GitHub repository, download the latest release, and see what VectorChord 0.5 can do for you

What is PostgreSQL?

PostgreSQL is an open‑source, enterprise‑class relational database management system (RDBMS) renowned for reliability, data integrity, and standards compliance. It supports advanced features such as window functions, materialized views, JSONB storage, and a robust extension mechanism that allows developers to add capabilities. Learn more about PostgreSQL here.

Full‑Text Search with tsvector

PostgreSQL has its own tool for basic text search called tsvector. Think of it as a special column that stores the important words (lexemes) from each document in a way the database can search quickly. When you add a GIN or GiST index on that column, PostgreSQL can find rows that match a keyword almost instantly. You write searches with the @@ operator, and the full details are in the PostgreSQL manual here.

Where native ranking falls short

While the ts_rank function can sort matches, its scoring method is basic. Long documents that mention a term once may outrank short documents focused entirely on the query, and rare but important words carry little extra weight. In large datasets, the perceived relevance of results often suffers.

What’s New: VectorChord Adds BM25

VectorChord‑BM25 is a lightweight PostgreSQL extension that augments the existing text‑search stack with the industry‑standard BM25 ranking formula. Instead of exporting data to an external search engine, you can keep everything inside a single database, gaining:

• Ranking that rewards distinctive terms and penalises irrelevant verbosity

• In‑index scoring, reducing query latency

• Seamless SQL workflow—documents remain rows, searches remain queries

What is BM25?

BM25 (Best Matching 25) is a probabilistic retrieval model widely adopted by search engines and academic literature. It evaluates how often a query term appears in a document, how rare that term is across the entire corpus, and how long each document is. The algorithm is discussed in detail on its Wikipedia page, but the core idea is straightforward:

• Term Frequency (TF) – More occurrences of a word in a document increase its relevance but with diminishing returns.

• Inverse Document Frequency (IDF) – Rare words are more informative than common ones.

• Document Length Normalisation – Shorter documents aren’t unfairly penalised when they focus on the query topic.

VectorChord implements BM25 through a new index type and operator (<&>), letting PostgreSQL compute scores while scanning the index.

Two Practical Examples

Below are concise demonstrations that illustrate how to adopt VectorChord‑BM25 with a pre‑trained tokenizer and, for specialised domains, with a custom model.

Example 1: Quick Start with a Pre‑Trained Model

# Spin up a pre‑configured PostgreSQL instance
docker run --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg17-latest

-- Inside psql
CREATE EXTENSION pg_tokenizer CASCADE;
CREATE EXTENSION vchord_bm25  CASCADE;

-- Register the LLMLingua‑2 tokenizer
SELECT create_tokenizer('llm_tok', $$ model = "llmlingua2" $$);

CREATE TABLE articles (
    id   SERIAL PRIMARY KEY,
    body TEXT,
    emb  bm25vector
);

INSERT INTO articles(body) VALUES
('PostgreSQL is a powerful open‑source database system.'),
('BM25 is a ranking function used by search engines.');

-- Tokenise each row
UPDATE articles SET emb = tokenize(body, 'llm_tok');

-- Build the BM25 index
CREATE INDEX articles_emb_bm25 ON articles USING bm25 (emb bm25_ops);

-- Query with relevance ordering
SELECT id,
       body,
       emb <&> to_bm25query('articles_emb_bm25',
                            tokenize('open source database', 'llm_tok')) AS score
FROM   articles
ORDER  BY score         -- lower (more negative) = higher relevance
LIMIT  10;

The query returns rows already ranked by BM25, producing more intuitive results than the default ts_rank.

Example 2: Custom Model for Domain‑Specific Vocabulary

Domain‑specific text—medical notes, legal briefs, technical logs—often includes jargon absent from general models. VectorChord lets you train a custom tokenizer directly in SQL.

-- 1. Create a text analyzer with Unicode segmentation, lowercasing,
--    stop‑word removal, and stemming.
SELECT create_text_analyzer('tech_analyzer', $$
pre_tokenizer = "unicode_segmentation"
[[character_filters]]
to_lowercase = {}
[[token_filters]]
stopwords = "nltk_english"
[[token_filters]]
stemmer = "english_porter2"
$$);

-- 2. Train a model on your own corpus and set up automatic embedding
SELECT create_custom_model_tokenizer_and_trigger(
    tokenizer_name     => 'tech_tok',
    model_name         => 'tech_model',
    text_analyzer_name => 'tech_analyzer',
    table_name         => 'tickets',
    source_column      => 'issue_text',
    target_column      => 'embedding');

-- 3. Insert support tickets; embeddings are generated via trigger
INSERT INTO tickets(issue_text)
VALUES ('Kubernetes pod fails with ExitCode 137 after OOM kill.'),
       ('Network latency spikes to 250ms during peak hours.');

-- 4. Build an index and query as before
CREATE INDEX tickets_emb_bm25 ON tickets USING bm25 (embedding bm25_ops);

SELECT issue_text,
       embedding <&> to_bm25query('tickets_emb_bm25',
                                  tokenize('OOM kill ExitCode 137', 'tech_tok')) AS score
FROM   tickets
ORDER  BY score
LIMIT  5;

With a tailor‑made vocabulary, the database recognises abbreviations such as “OOM” and “Kubernetes,” yielding more precise rankings for technical support scenarios.

Final Thoughts

VectorChord‑BM25 brings search‑engine quality relevance to the PostgreSQL ecosystem without introducing an additional service layer. By combining flexible tokenization, an efficient BM25 index, and familiar SQL, it enables developers to deliver significantly better search experiences while preserving the operational simplicity of a single database system. If your application relies on PostgreSQL and demands accurate text ranking, VectorChord‑BM25 is well worth exploring.

If you're working with similarity searches, RAG pipelines, or other AI-driven applications on PostgreSQL, these updates are for you. We've focused on two major areas: significantly enhancing I/O for cold queries by leveraging PostgreSQL's evolving capabilities, and optimizing filtered vector searches.

🚀 Major Improvement 1: Advanced I/O for Disk-Bound Indexes (2x-3x Lower Cold Query Latency!)

VectorChord has supported disk-based indexing since its early versions. Previously, a challenge was I/O efficiency, as full-precision vectors were read one-by-one due to limitations in PostgreSQL's older internal APIs for buffer operations.

VectorChord 0.4 addresses this with a rewritten page layout and the adoption of modern streaming I/O techniques. We're proud to be one of the first, if not the first, PostgreSQL extensions to adopt these new streaming I/O APIs as they become available. This allows us to pipeline I/O with computation and issue disk I/O operations more effectively. As a part of this effort, we've also contributed initial PostgreSQL 18 support to the pgrx framework to enable testing and development with these upcoming AIO capabilities, benefiting the wider PostgrSQL community.

Our approach adapts to your PostgreSQL version:

• On PostgreSQL 17 (and newer with io_method=sync): Utilizing madvise for Prefetching When VectorChord calls PostgreSQL's new streaming I/O APIs, PostgreSQL's underlying implementation on PG17 (or PG18 with io_method=sync) may internally use madvise(MADV_WILLNEED). This hints the OS kernel about upcoming data needs, enabling reads from disk to be cached in memory with page cache first. When actual I/O occurs, PostgreSQL can read directly from the page cache instead of waiting for disk I/O.

• Preparing for Asynchronous I/O: PostgreSQL 18 & io_uring Looking ahead to PostgreSQL 18's true Asynchronous I/O (AIO) with io_uring, VectorChord 0.4 is engineered for integration. This will allow data to be fetched directly into PostgreSQL's shared buffers with minimal overhead and maximal efficiency.

• For Earlier PostgreSQL Versions (Pre-PG17): Streaming I/O Interface for Prefetching For users on older PostgreSQL versions, we've implemented a similar streaming I/O interface within VectorChord for backward compatibility. While it doesn't leverage kernel-level AIO or the newest PostgreSQL internal APIs, this interface allows us to achieve effective prefetching of buffers. This means users on earlier PostgreSQL versions can also benefit from the improved I/O patterns and reduced latency when fetching full-precision vectors.

The Impact of Advanced I/O for Disk Indexes: These I/O enhancements result in a 2x-3x reduction in latency for cold queries in our benchmarks. This directly improves tail latency, especially when queries require re-ranking with disk-resident vectors.

🎯 Major Improvement 2: Pre-filtering for Faster Filtered Searches (Up to 3x Faster!)

Vector search is often combined with metadata filtering. Addressing this efficiently has been a key focus for us. We were among the first to tackle the vector filtering challenge in PostgreSQL by introducing VBASE-style filtering in pgvecto.rs, which significantly improved performance over pgvector.

Now, with VectorChord 0.4, we're pushing the performance to the next level by introducing robust prefiltering support.

• Post-filtering (Previous Method): Find top-K vectors, then filter. Inefficient for selective filters.

• Pre-filtering (New in 0.4): Based on bit vector scan results, identify rows satisfying metadata filters first, then perform vector distance calculations only on this smaller set.

We now use efficient mechanisms with quantized bit-vector scans to identify matching rows before vector distance computations. Since filter checks are much lighter than distance calculations, this provides a significant speedup. User can enable this with SET vchordrq.prefilter=ON.

The Impact: Our benchmarks show up to 3x faster search performance when pre-filtering is applicable, compared to our already optimized VBASE post-filtering approach. This offers considerable advantages for complex queries.

✨ Other Notable Improvements:

• Optimized Residual Quantization (~20% QPS Boost): Thanks to Jianyang (author of RaBitQ), we've refined residual quantization. By reformulating <o, q-c> as <o, q> - <o, c>, the query vector q is quantized only once, yielding a ~20% QPS improvement. We now recommend user to enable residual quantization for L2 distance.

• Optimized Rotation Matrix & Simplified Configuration: Also thanks to Jianyang, a new Fast Hadamard Transform optimizes rotation matrices and removes the prewarm_dim GUC, simplifying configuration and slightly speeding up the process.

• Rewritten Documentation: We've comprehensively rewritten our documentation, now offering more user guidance and a detailed API reference to help you get the most out of VectorChord. Check https://docs.vectorchord.ai/vectorchord/.

Get Started with VectorChord 0.4!

We believe this release significantly enhances vector search capabilities in PostgreSQL.

• Download/Star on GitHub: https://github.com/tensorchord/VectorChord

• Join the Community: https://discord.gg/KqswhpVgdU We encourage you to try VectorChord 0.4, benchmark it with your workloads, and share your feedback.

Happy Vector Searching!

We're attempting to show you what kind of hardware setup you need to allow you to search through this vast dataset using VectorChord. We want to make sure that you have a good experience with a search that is both accurate and fast.

VectorChord is a PostgreSQL extension that may bring high-performance vector database capabilities directly into your PostgreSQL database with high compatibility with the popular pgvector extension's interface where possible.

The version of VectorChord to be used for this tutorial must be v0.3.0 or later. While VectorChord is compatible with most of the recent pgvector versions, pgvector 0.8.0 was extensively tested.

Dataset

The dataset used for this experiment is LAION-400M, which is a dataset with approximately 400 million vectors derived from images. Each vector is 512-dimensional and was generated from a CLIP model.

The total vector data is approximately 400 GB. The vectors have been normalized, so it is feasible to use multiple distance measures (L2, cosine, dot product). To be consistent with other benchmarks on this dataset and following the nature of CLIP embeddings, we employed the cosine distance measure in this experiment.

LAION-400M is divided into 409 chunks, each containing around 1 million vectors. We will process these chunks sequentially for uploading.

Algorithm

Searching through 400 million vectors is a huge challenge, especially as the data size means much of it must live on disk rather than in memory. We need algorithms built for this scale! A popular and effective method for handling large, disk-based vector data is the graph-based DiskANN, known for its Low memory and resource cost.

However, being derived from HNSW, DiskANN also has some inherited disadvantages:

• Slow index building: For large scale datasets, it's well known that building indexes with DiskANN can take a long time.

• Poor insert performance: For streaming inserted vectors, updating a graph-based index requires traversing the partial graph. This is even more costly because DiskANN's on-disk graph must be loaded into memory.

VectorChord addresses these problems with its VChordRQ index, which combines a cluster-based index (ivf) with a clever data compression scheme called RabitQ. In addition to better precision and latency in both memory and disk, RabitQ's strength lies in its strong theoretical guarantee of accuracy, which is different from other compression techniques. This helps us achieve an excellent trade-off between accuracy and efficiency!

Hardware

Based on our experiments, we determined the following cloud instance type to be sufficient to index and query the dataset with acceptable latency:

• Instance Type: AWS EC2 i8g.2xlarge ($501.0720 / month)

• CPU: 8 vCPUs (ARM64)

• RAM: 64GB

• Storage: 1875 GB NVMe SSD

This configuration provides the necessary compute and ample fast local NVMe storage. The 64GB of RAM serves as cache to accelerate search queries.

For the PostgreSQL setup within this instance, we adjusted the shared_buffers parameter:

• Set to 48GB during the index building process.

• Increased to 54GB for search evaluation to allow more data caching for search performance.

Uploading and Indexing

The VectorChord VChordRQ index uses an Index-based Vector File (IVF) structuring. While VChordRQ can compute the required centroids during the index building process, the building of an index of this size (400M vectors) pre-calculating the centroids out of the system significantly accelerates the process of building an index.

For LAION-400M, we performed an external k-means clustering step to determine these centroids. This was done on an instance with an A10 GPU for faster computation.

• The clustering was configured for nlist = 160000 clusters.

• The clustering took approximately 220 seconds per iteration for 25 iterations, totaling about 1.5 hours.

Once we have computed the centroids, we set up the VectorChord-enabled PostgreSQL database. We used a pre-built Docker image for convenience:

-- Mount PGDATA path to the NVMe SSD
-- sudo mount /dev/nvme1n1 /data
docker run --name vchord-pg17 \
-e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -v /data/pg:/var/lib/postgresql/data \
-d tensorchord/vchord-postgres:pg17-v0.3.0

This Docker image contains PostgreSQL 17 with pgvector 0.8.0 and VectorChord 0.3.0 installed.

The core of the upload process involves inserting the pre-computed centroids into a dedicated table and then inserting the vectors from the LAION dataset chunks into the main data table. This can be done efficiently using a client library like psycopg in Python with batched inserts.

First, insert the centroids into a table (e.g., laion_centroids):

-- The centroids table we created is:
-- CREATE TABLE laion_centroids (id SERIAL PRIMARY KEY, vector vector(512));
INSERT INTO laion_centroids (vector) VALUES ('[...centroid vector...]'); -- Insert all 160000 centroids

Then, insert the vectors from the dataset into the data table (e.g., laion):

-- The data table we created is:
-- CREATE TABLE laion (id BIGINT PRIMARY KEY, embedding vector(512));
-- Note: VectorChord is compatible with pgvector's vector insert syntax.
INSERT INTO laion (id, embedding) VALUES (0, '[...vector data...]'); -- Batch inserts for chunks

After inserting all data, we build the VectorChord index (vchordrq). vchordrq stands for RabitQ, which is VectorChord's highly optimized index type based on the IVF structure combined with quantization and efficient reranking.

-- Note: The metric operator vector_cosine_ops is specified here, similar to pgvector. 
-- The $$ syntax is used to pass build options as a text block.
CREATE INDEX laion_embedding_idx ON laion USING vchordrq (embedding vector_cosine_ops) WITH (options = $$
[build.external]
table = 'public.laion_centroids' -- Link to the pre-computed centroids
$$);

The image below shows the system resource utilization (CPU, memory, and disk usage) on the instance during the upload and indexing process. The disk usage graph shows the data being written (initial steep increase) and the subsequent index building (steady increase), resulting in a final size over 1TB. The CPU and memory graphs indicate the resources consumed throughout this process.

Ground Truth Data

To correctly measure search performance and accuracy, comparing to ground truth is vital. The LAION data set comes without inherent ground truth. We are in debt to the Qdrant team for having published the ground truth data they had prepared for their tutorial on this dataset.

Their methodology involved performing a full-scan nearest neighbor search for the first 100 vectors in the dataset to find the top 50 true nearest neighbors for each query. We used this same exact ground truth file to evaluate the VectorChord search accuracy.

Search Query

VectorChord search employs two main parameters to control the performance-accuracy trade-off: nprobe and epsilon.

• nprobe: Configured using the vchord.probes setting. This parameter determines how many clusters (partitions) of the IVF index are searched in the initial phase. A higher nprobe increases the search scope, potentially finding better results but also increasing latency.

• epsilon: Configured using the vchord.epsilon setting. This parameter controls the reranking precision. After fetching the initial candidates from the nprobed clusters, VectorChord performs a reranking step using more precise vector data. A larger epsilon means more candidates are considered and potentially reranked for higher recall, impacting latency.

These parameters are set as PostgreSQL session variables before executing the search query:

-- Example settings
SET vchord.probes = 100; -- Configure nprobe
SET vchord.epsilon = 0.8; -- Configure epsilon
​
-- The search query itself uses the standard pgvector distance operator
SELECT id FROM laion ORDER BY embedding <=> '[...query_vector...]' LIMIT 50;

The ORDER BY embedding <=> '[...]' clause leverages the index to find the approximate nearest neighbors based on the specified distance metric (cosine distance, like CLIP). The LIMIT 50 retrieves the top 50 results.

Running Search Requests

After the index built is finished, we ran queries over the 100 ground truth vectors with varying nprobe and epsilon settings to try out performance versus precision trade-offs.

We expressed search latency in milliseconds (ms). "Cold" latency refers to instances where vector data of interest may need to be read from disk, while "warm" latency implies that data will be in memory.

Here are the average latencies and corresponding Precision@50 scores across the 100 queries:

The table clearly illustrates the trade-off:

• Increasing nprobe and epsilon generally leads to higher Precision@50.

• However, this comes at the cost of increased search latency, especially noticeable in cold scenarios where more disk I/O is required to fetch data from more clusters and perform more extensive reranking.

• Warm cache performance is significantly faster, highlighting the importance of sufficient RAM for caching frequently accessed data.

• It is possible to have over 90% precision through warm latencies less than 50ms, and increasing precision above this yields hundreds of milliseconds or even more than a second latency for best recall, depending on the parameters selected.

The graph below shows the trade-off between Queries Per Second (QPS) and Precision@50 for VectorChord (cold and warm cache) alongside Qdrant, with Qdrant’s metrics taken from their published benchmarks on a comparable 8 vCPU, 64 GB machine rather than run on the same hardware.

Conclusion

In this tutorial, we illustrated how VectorChord as an extension to PostgreSQL can be utilized to upload, index, and search a gigantic 400 million vector dataset cost-efficiently on relatively small hardware with fast NVMe SSD.

By using the VectorChord VChordRQ index with external centroids, we were able to handle the scale. Having carefully set nprobe and epsilon at query time provides precise control over the critical trade-off between search speed and accuracy, allowing users to tune the system to their specific needs.

VectorChord's compatibility with the pgvector interface simplifies adoption for current pgvector users, and its vchordrq index provides powerful capabilities for large-scale vector search directly within the robust and familiar PostgreSQL environment.

An example could be from a collection of historical documents. If a user asks about a specific event, and a retrieved chunk says: "The treaty was signed, bringing an end to hostilities."

Without the surrounding text, this chunk lacks vital context such as which treaty was signed, who the involved parties were, and when or where this significant event took place. Relying solely on this decontextualized snippet would provide an incomplete and potentially inaccurate picture to the LLM, hindering its ability to generate a comprehensive and accurate answer about the historical event.

The example highlights that while chunking aids retrieval efficiency, it can inadvertently strip away the necessary context that gives meaning and relevance to the individual pieces of information. This loss of context can significantly impact the quality and accuracy of the responses generated by LLMs in a RAG system.

To tackle this problem, Anthropic explores a method called Contextual Retrieval.

Basically, it could be treated as the chunk augmentation with the document context.

This article will walk through the steps to improve the retrieval accuracy with the help of contextual information. We use the same prompt, dataset, and metrics described in the anthropic tutorial. Note that we use a different embedding model and LLM in this experiment, so the results will differ. However, this does not reflect the model's accuracy, as the dataset consists of code rather than natural language.

We mainly use Gemini models to generate the chunk context and text embedding, other models also work. The embedding quality affects the vector retrieval metrics heavily, we recommend checking the leaderboard to find the one that suits your use case. For the keyword tokenization, we use the default BERT tokenizer, you can also try other solutions.

The rerank part is not the main point of this article, if you want to know more about the rerank methods, check our article about the cross-encoder rerank and token-level late interaction rerank. BTW, since we support multivector MaxSim index, the token-level late interaction rerank can be done more efficiently.

Prepare the environment

We provide both vector search and keyword search features in our VectorChord-suite docker image.

docker run --rm -d --name vdb -e POSTGRES_PASSWORD=postgres -p 5432:5432 ghcr.io/tensorchord/vchord-suite:pg17-20250414

We will use the vechord Python library to reduce the boilerplate code and simplify the process.

pip install 'vechord[gemini]'

Define the RAG-related tables

Let’s get started with the basic RAG process.

from vechord.spec import (
    ForeignKey,
    Keyword,
    PrimaryKeyAutoIncrease,
    Table,
    UniqueIndex,
    Vector,
)

DenseVector = Vector[768]

class Document(Table, kw_only=True):
    uid: Optional[PrimaryKeyAutoIncrease] = None
    uuid: Annotated[str, UniqueIndex()]
    content: str

class Chunk(Table, kw_only=True):
    uid: Optional[PrimaryKeyAutoIncrease] = None
    doc_uuid: Annotated[str, ForeignKey[Document.uuid]]
    index: int
    content: str
    vector: DenseVector
    keyword: Keyword

class Query(Table, kw_only=True):
    uid: Optional[PrimaryKeyAutoIncrease] = None
    content: str
    answer: str
    doc_uuids: list[str]
    chunk_index: list[int]
    vector: DenseVector

This includes the vector embedding index and keyword index.

To load the datasets:

from vechord.embedding import GeminiDenseEmbedding
from vechord.registry import VechordRegistry

emb = GeminiDenseEmbedding()
vr = VechordRegistry("anthropic", "postgresql://postgres:postgres@172.17.0.1:5432/")

vr.register([Document, Chunk, Query])

def load_data(filepath: str):
    with open(filepath, "r", encoding="utf-8") as f:
        docs = json.load(f)
        for doc in docs:
            vr.insert(
                Document(
                    uuid=doc["original_uuid"],
                    content=doc["content"],
                )
            )
            for chunk in doc["chunks"]:
                vr.insert(
                    Chunk(
                        doc_uuid=doc["original_uuid"],
                        index=chunk["original_index"],
                        content=chunk["content"],
                        vector=emb.vectorize_chunk(chunk["content"]),
                        keyword=Keyword(chunk["content"]),
                    )
                )

def load_query(filepath: str):
    queries = []
    with open(filepath, "r", encoding="utf-8") as f:
        for line in f:
            query = json.loads(line)
            queries.append(
                Query(
                    content=query["query"],
                    answer=query["answer"],
                    doc_uuids=[x[0] for x in query["golden_chunk_uuids"]],
                    chunk_index=[x[1] for x in query["golden_chunk_uuids"]],
                    vector=emb.vectorize_query(query["query"]),
                )
            )
    vr.copy_bulk(queries)

Evaluation

Now we have everything for the basic RAG process. Let’s define the evaluation metric Pass@k , which means that retrieved top-k chunks contain how many of the groundtruth chunks.

def evaluate(topk=5, search_func=vector_search):
    print(f"TopK={topk}, search by: {search_func.__name__}")
    queries: list[Query] = vr.select_by(Query.partial_init())
    total_score = 0
    start = perf_counter()
    for query in queries:
        chunks: list[Chunk] = search_func(query, topk)
        count = 0
        for doc_uuid, chunk_index in zip(
            query.doc_uuids, query.chunk_index, strict=True
        ):
            for chunk in chunks:
                if chunk.doc_uuid == doc_uuid and chunk.index == chunk_index:
                    count += 1
                    break
        score = count / len(query.doc_uuids)
        total_score += score

    print(
        f"Pass@{topk}: {total_score / len(queries):.4f}, total queries: {len(queries)}, QPS: {len(queries) / (perf_counter() - start):.3f}"
    )

We can try different retrieval strategies like vector search, keyword search, hybrid search with fusion or rerank.

Those strategies can be defined as:

from vechord.rerank import CohereReranker, ReciprocalRankFusion

def vector_search(query: Query, topk: int) -> list[Chunk]:
    return vr.search_by_vector(Chunk, query.vector, topk=topk)

def keyword_search(query: Query, topk: int) -> list[Chunk]:
    return vr.search_by_keyword(Chunk, query.content, topk=topk)

def hybrid_search_fuse(query: Query, topk: int) -> list[Chunk]:
    rrf = ReciprocalRankFusion()
    return rrf.fuse([vector_search(query, topk), keyword_search(query, topk)])[:topk]

def hybrid_search_rerank(query: Query, topk: int, boost=3) -> list[Chunk]:
    ranker = CohereReranker()
    vecs = vector_search(query, topk * boost)
    keys = keyword_search(query, topk * boost)
    chunks = list({chunk.uid: chunk for chunk in vecs + keys}.values())
    indices = ranker.rerank(query.content, [chunk.content for chunk in chunks])
    return [chunks[i] for i in indices[:topk]]

Contextual retrieval

The LLM generates contextual information with the prompt like:

prompt = (
    "<document>\n{whole_document}\n</document>"
    "Here is the chunk we want to situate within the whole document \n"
    "<chunk>\n{chunk}\n</chunk>\n"
    "Please give a short succinct context to situate this chunk within "
    "the overall document for the purposes of improving search retrieval "
    "of the chunk. Answer only with the succinct context and nothing else."
)

The prompt above is a general one designed for retrieval tasks. You may find that a more specialized prompt offers better performance for your specific use case.

This feature is already included in the vechord library.

GeminiAugmenter also supports the prompt caching like Claude, but it requires the cached content to be at least 32768 tokens, which is not the case for this dataset.

from vechord.augment import GeminiAugmenter

class ContextualChunk(Table, kw_only=True):
    uid: Optional[PrimaryKeyAutoIncrease] = None
    doc_uuid: Annotated[str, ForeignKey[Document.uuid]]
    index: int
    content: str
    context: str
    vector: DenseVector
    keyword: Keyword

def load_contextual_chunks(filepath: str):
    augmenter = GeminiAugmenter()

    with open(filepath, "r", encoding="utf-8") as f:
        docs = json.load(f)
        for doc in docs:
            augmenter.reset(doc["content"])
            chunks = doc["chunks"]
            augments = augmenter.augment_context([chunk["content"] for chunk in chunks])
            if len(augments) != len(chunks):
                print(f"augments length not match for uuid: {doc['original_uuid']}, {len(augments)} != {len(chunks)}")
            for chunk, context in zip(chunks, augments, strict=False):
                contextual_content = f"{chunk['content']}\n\n{context}"
                vr.insert(
                    ContextualChunk(
                        doc_uuid=doc["original_uuid"],
                        index=chunk["original_index"],
                        content=chunk["content"],
                        context=context,
                        vector=emb.vectorize_chunk(contextual_content),
                        keyword=Keyword(contextual_content),
                    )
                )

Then, we can expand the strategies like:

def vector_contextual_search(query: Query, topk: int) -> list[ContextualChunk]:
    return vr.search_by_vector(ContextualChunk, query.vector, topk=topk)

def keyword_contextual_search(query: Query, topk: int) -> list[ContextualChunk]:
    return vr.search_by_keyword(ContextualChunk, query.content, topk=topk)

def hybrid_contextual_search_fuse(query: Query, topk: int) -> list[ContextualChunk]:
    rrf = ReciprocalRankFusion()
    return rrf.fuse(
        [vector_contextual_search(query, topk), keyword_contextual_search(query, topk)]
    )[:topk]

def hybrid_contextual_search_rerank(
    query: Query, topk: int, boost=3
) -> list[ContextualChunk]:
    ranker = CohereReranker()
    vecs = vector_contextual_search(query, topk * boost)
    keys = keyword_contextual_search(query, topk * boost)
    chunks = list({chunk.uid: chunk for chunk in vecs + keys}.values())
    indices = ranker.rerank(
        query.content, [f"{chunk.content}\n{chunk.context}" for chunk in chunks]
    )
    return [chunks[i] for i in indices[:topk]]

Benchmark

• topk=5

• topk=10

All the code can be found in our vechord repository.

Many solutions feature complex architectures, using distinct databases for vector search, keyword search, and primary data storage. This commonly results in data synchronization challenges, greater infrastructure complexity, and higher operational costs.

What if you could build a powerful, production-ready RAG system directly within your trusted PostgreSQL database?

Enter the VectorChord Suite, a collection of PostgreSQL extensions designed to bring high-performance vector search, native BM25 ranking, and flexible tokenization capabilities right into Postgres. Let's explore the components and how they enable next-generation RAG solutions.

What is the VectorChord Suite?

The suite comprises three key PostgreSQL extensions working in concert:

1. VectorChord: The core vector search engine. It's specifically designed for scalable, high-performance, and disk-efficient vector similarity search within PostgreSQL.

2. VectorChord-bm25: This extension implements the sophisticated BM25 ranking algorithm directly inside PostgreSQL, leveraging efficient Block-WeakAnd algorithms. BM25 is a standard for relevance ranking based on keyword frequency and document characteristics.

3. pg_tokenizer.rs: Provides essential text tokenization capabilities needed for effective full-text search, enabling fine-grained control over how text is processed for full-text search.

By combining these extensions, you unlock powerful capabilities for building advanced RAG systems entirely within PostgreSQL.

How to Use the VectorChord Suite

You can use the tensorchord/vchord-suite image to run multiple extensions which are provided by TensorChord. The image is based on the official Postgres image and includes the following extensions:

docker run   \           
  --name vchord-suite  \
  -e POSTGRES_PASSWORD=postgres  \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg17-latest
  # If you want to use ghcr image, you can change the image to `ghcr.io/tensorchord/vchord-suite:pg17-latest`.
  # if you want to use the specific version, you can use the tag `pg17-20250414`, supported version can be found in the support matrix.

CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;
\dx
pg_tokenizer | 0.1.0   | tokenizer_catalog | pg_tokenizer
vchord       | 0.3.0   | public            | vchord: Vector database plugin for Postgres, written in Rust, specifically designed for LLM
vchord_bm25  | 0.2.0   | bm25_catalog      | vchord_bm25: A postgresql extension for bm25 ranking algorithm
vector       | 0.8.0   | public            | vector data type and ivfflat and hnsw access methods

Use Cases for the VectorChord Suite

Use Case 1: Powerful Hybrid Search with Native BM25 and VectorChord

In the RAG era, effective retrieval is paramount. Neither keyword search nor vector search alone is perfect:

• Keyword Search (like BM25): Excels at precision, finding documents with exact keyword matches. It's great for structured queries and term-specific searches. However, it struggles with synonyms, paraphrasing, and understanding the underlying meaning or semantic intent. (Leverages VectorChord-bm25 and pg_tokenizer).

• Vector Search: Captures deep semantic meaning and relationships between concepts, allowing it to find relevant information even if the exact keywords aren't present. However, it can sometimes lack precision for queries demanding specific term matches. (Leverages VectorChord).

The Solution: Hybrid Search. By combining the strengths of both approaches within Postgres, the VectorChord Suite bridges this gap. You can run a query that leverages:

• VectorChord-bm25 (powered by pg_tokenizer) for keyword precision.

• VectorChord for semantic understanding.

The results from both can be intelligently combined (e.g., using reciprocal rank fusion - RRF or Model-based rerank) to produce a final ranking that is more accurate, contextually relevant, and semantically aware than either method could achieve alone. This delivers significantly better retrieval performance for your RAG applications.

Use Case 2: Beyond Text - OCR-Free RAG with ColQwen2 & VectorChord

Building RAG systems for documents like PDFs or scanned images often involves cumbersome pre-processing pipelines. Traditional methods rely heavily on Optical Character Recognition (OCR) and layout analysis to extract text. This process can be:

• Slow: OCR can be computationally expensive.

• Error-Prone: OCR accuracy varies significantly depending on document quality.

• Lossy: Crucial visual contexts like tables, figures, formatting, and relative positioning is often lost during text extraction.

The Solution: OCR-Free, Visually-Aware RAG. What if you could query documents based on how they look and their content, without explicit OCR?

This is now achievable by combining:

1. Multi-Modal Vision Language Models (VLMs): Models like ColQwen2 can process images (document pages) and generate embeddings that capture both textual content and visual layout information.

2. VectorChord's Multi-Vector Capabilities: VectorChord can efficiently store and search multiple vectors per document within Postgres – allowing you to store embeddings representing different aspects (e.g., text content, visual layout).

With this setup, you can query your document database using prompts that reference visual elements ("Find documents with a bar chart comparing sales figures") or combined textual and visual cues, directly within PostgreSQL. This simplifies your RAG stack, potentially boosts retrieval accuracy by preserving visual context, and eliminates the bottlenecks associated with traditional OCR pipelines.

Why Build RAG in PostgreSQL?

Leveraging the VectorChord Suite within Postgres offers significant advantages:

• Unified Data: Keep your source data, text, metadata, and vector embeddings all in one place.

• Reduced Complexity: Eliminate the need for separate vector databases and synchronization pipelines.

• Leverage Existing Infrastructure: Utilize your existing Postgres expertise, tooling, and operational practices.

• Transactional Integrity: Benefit from PostgreSQL's robust ACID compliance.

• Rich Ecosystem: Access the wide array of tools and features available within the Postgres ecosystem.

Conclusion

The VectorChord Suite transforms PostgreSQL into a powerful, production-ready platform for building advanced RAG solutions. By integrating high-performance vector search (VectorChord), sophisticated keyword ranking (VectorChord-bm25), and flexible text processing (pg_tokenizer), you can implement cutting-edge techniques like hybrid search and OCR-free multi-modal retrieval directly within your database. Simplify your architecture, enhance retrieval accuracy, and unlock the full potential of RAG with VectorChord in PostgreSQL.

Previous Blog Post

In our previous blog, we shared some user cases where all images can be replaced with VectorChord Suite image.

• Supercharge vector search with ColBERT rerank in PostgreSQL

• Hybrid search with Postgres Native BM25 and VectorChord

• Beyond Text: Unlock OCR-Free RAG in PostgreSQL with Modal & VectorChord

References

• https://blog.vectorchord.ai/vectorchord-store-400k-vectors-for-1-in-postgresql

• https://blog.vectorchord.ai/vectorchord-bm25-revolutionize-postgresql-search-with-bm25-ranking-3x-faster-than-elasticsearch

• https://blog.vectorchord.ai/vectorchord-bm25-introducing-pgtokenizera-standalone-multilingual-tokenizer-for-advanced-search

This post is your guide to building exactly that: an OCR-free RAG system directly within your familiar PostgreSQL database. We'll leverage the power of the ColQwen2 Vision Language Model, the efficiency of VectorChord for multi-vector search in Postgres, and the scalability of Modal for GPU-powered embedding generation. Get ready to simplify your RAG stack and potentially boost your retrieval accuracy, all without complex pre-processing.

We'll cover:

• What ColQwen2 is and why it's a game-changer.

• How VectorChord makes advanced vector search possible in Postgres.

• A step-by-step tutorial to build and evaluate the system.

What is ColQwen2? The Power of Visual Understanding

To grasp ColQwen2, let's first look at its foundation: ColPali. As introduced in the paper "ColPali: Efficient Document Retrieval with Vision Language Models", ColPali represents a novel approach using Vision Language Models (VLMs). Instead of relying on imperfect OCR, it directly indexes documents using their rich visual features – text, images, tables, layout, everything the eye can see.

Think about the limitations of traditional OCR: complex layouts get mangled, tables become gibberish, and images are often ignored entirely. It's like trying to understand a book by only reading a flawed transcript. ColPali avoids this by using a powerful VLM (originally PaliGemma) to create embeddings that capture the document's holistic visual nature. Two key concepts make it shine:

1. Contextualized Vision Embeddings: Generating rich embeddings directly from the document image using a VLM.

2. Late Interaction: This clever technique allows the query's textual meaning to directly interact with the document's detailed visual features at search time. It's not just matching text summaries; it's comparing the query concept against the visual evidence within the document page.

The ColPali architecture (Image from the ColPali paper)

ColQwen2 builds upon this powerful ColPali architecture but swaps the underlying VLM for the more recent Qwen2-VL-2B. It generates ColBERT-style multi-vector representations, capturing fine-grained details from both text and images. As seen on the vidro-leaderboard, ColQwen2 delivers impressive performance with practical model size.

How Does VectorChord Enable ColQwen2 in Postgres?

This is where VectorChord becomes the crucial piece of the puzzle, bringing this cutting-edge VLM capability into your PostgreSQL database. ColQwen2 (and ColPali) relies heavily on those multi-vector representations and the Late Interaction mechanism, specifically requiring an efficient MaxSim (Maximum Similarity) operation.

Calculating MaxSim – finding the highest similarity score between any vector in the query set and any vector in the document set – can be computationally brutal, especially across millions of document vectors. VectorChord tackles this head-on:

• Native Multi-Vector Support: It's designed from the ground up to handle multi-vector data efficiently within Postgres.

• Optimized MaxSim: Drawing inspiration from the WARP paper, VectorChord uses techniques like dynamic similarity imputation to dramatically speed up MaxSim calculations, making large-scale visual document retrieval feasible.

• Hybrid Search Ready: Beyond multi-vector, it also supports dense, sparse, and hybrid search (check out our previous post!).

• Scalable & Disk-Friendly: Designed for performance without demanding excessive resources.

In short, VectorChord transforms PostgreSQL into a powerhouse capable of handling the advanced vector search techniques required by models like ColQwen2 or ColPali.

Tutorial: Building Your OCR-Free RAG System

Alright, theory's great, but let's roll up our sleeves and build this thing! We'll walk through setting up the environment, processing data using Modal for scalable embedding generation, indexing into VectorChord within Postgres, and finally, evaluating our shiny new OCR-free RAG system.

Prerequisites

Before we start, ensure you have:

• A PostgreSQL instance (Docker recommended) with the VectorChord extension installed OR a VectorChord Cloud cluster.

• A Modal account (free tier available). Modal's fast GPU provisioning and scaling are perfect for the embedding generation step. To efficiently process a large volume of documents using the ColQwen2 model, it's crucial to leverage Modal's rapid startup and GPU expansion features. This approach will significantly reduce the time required for local processing.

If you want to reproduce the tutorial quickly, you can use the tensorchord/vchord-suite image to run multiple extensions that TensorChord provides.

You can run the following command to build and start Postgres with VectorChord-BM25 and VectorChord.

docker run   \           
  --name vchord-suite  \
  -e POSTGRES_PASSWORD=postgres  \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg17-latest

CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;
\dx
pg_tokenizer | 0.1.0   | tokenizer_catalog | pg_tokenizer
vchord       | 0.3.0   | public            | vchord: Vector database plugin for Postgres, written in Rust, specifically designed for LLM
vchord_bm25  | 0.2.0   | bm25_catalog      | vchord_bm25: A postgresql extension for bm25 ranking algorithm
vector       | 0.8.0   | public            | vector data type and ivfflat and hnsw access methods

Set up Modal:

$ pip install modal
$ python3 -m modal setup  # click the link to authorize

Step 1: Load the Data (Using Modal Volumes)

We'll use the ViDoRe Benchmark dataset. To handle this data efficiently across potentially distributed Modal functions, we'll download it to a Modal Volume. Volumes provide persistent, shared storage ideal for this 'download once, process many times' scenario.

image = modal.Image.debian_slim().pip_install("datasets","huggingface_hub","Pillow")
DATASET_DIR = "/data"
DATASET_VOLUME  = modal.Volume.from_name(
    "colpali-dataset", create_if_missing=True
)
app = modal.App(image=image)

@app.function(volumes={DATASET_DIR: DATASET_VOLUME}, timeout=3000)
def download_dataset(cache=False) -> None:
    from datasets import load_dataset
    from tqdm import tqdm

    collection_dataset_names = get_collection_dataset_names("vidore/vidore-benchmark-667173f98e70a1c0fa4db00d") 
    for dataset_name in tqdm(collection_dataset_names, desc="vidore benchmark dataset(s)"):
        dataset = load_dataset(dataset_name, split="test",num_proc=10)
        unique_indices = dataset.to_pandas().drop_duplicates(subset="image_filename", keep="first").index #to remove repeating PDF pages with different queries
        dataset = dataset.select(unique_indices)
        dataset.save_to_disk(f"{DATASET_DIR}/{dataset_name}")

Modal offers a straightforward Python API for interacting with its platform. With modal.Image, you can set the base image for the Modal App, and the function decorator helps define the function to be executed within the Modal application. To download the dataset to the Modal Volume, execute the following command. Modal will then automatically build the Docker image and promptly launch the container to run your function.

Run the download function:

$ modal run dataset.py::download_dataset

Modal handles building the environment and running the script in the cloud.

Step 2: Process Data & Generate Embeddings (With Modal & ColQwen2)

This is the heavy lifting: converting document images into ColQwen2 multi-vector embeddings. Doing this locally for many documents would be slow. Modal shines here:

• Easy GPU Access & Autoscaling: Launch a ColQwen2 model service on Modal and use it to generate the image embeddings and query embeddings for the dataset. The decision to implement an HTTP service instead of using the SDK directly was made to ensure seamless auto-scaling during large-scale embedding operations. If you want to deploy the embedding service as a persistent web endpoint, you can directly change modal.cls to modal.web_server for ColPaliServer and running modal deploy.

• Recovery: We checkpoint progress to a separate Modal Volume, allowing us to resume embedding generation if interrupted.

# embedding.py (Illustrative - keep original code)
# ... imports, volumes setup ...
modal_app = modal.App() # Define app

# Function to coordinate embedding generation
@modal_app.function(...)
def embed_dataset(down_scale: float = 1.0, batch_size: int = BATCH_SIZE):
    # ... (logic for loading dataset names, handling checkpoints) ...
    colpali_server = ColPaliServer() # Get handle to our GPU class
    # ... (loop through datasets, batch items, call server for embeddings) ...
    # ... (save embeddings and update checkpoint) ...
    print("Embedding generation complete.")

# server.py (Illustrative - keep original code)
# Class running on GPU to serve embedding requests
@modal_app.cls(gpu=GPU_CONFIG, ...)
class ColPaliServer:
    @modal.enter()
    def load_model_and_start_server(self):
        # ... (Load ColQwen2 model, start internal FastAPI server) ...
        self.client = httpx.AsyncClient(...) # Client to talk to internal server

    @modal.exit()
    def shutdown_server(self):
        # ... (Cleanup) ...

    # Method called by embed_dataset function
    @modal.method()
    async def embed_images(self, images: List[str]) -> np.ndarray:
        # ... (Prepare batch, send request to internal server via self.client) ...
        # ... (Decode response using msgspec for speed) ...
        return embeddings_numpy_array

    # ... (Potentially add embed_queries method too) ...

# colpali.py
class ColPaliModel:
    def __init__(self, model_name: str = "vidore/colqwen2-v1.0", cache_dir: str="/model"):
        # ...
        if self.model_name == "vidore/colqwen2-v1.0":
            from colpali_engine.models import ColQwen2, ColQwen2Processor
            from transformers.utils.import_utils import is_flash_attn_2_available

            # load model
            model = ColQwen2.from_pretrained(
                self.model_name,
                torch_dtype=torch.bfloat16,
                device_map="cuda:0", 
                attn_implementation="flash_attention_2" if is_flash_attn_2_available() else None,
                cache_dir=self.cache_dir,
            ).eval()

            colpali_processor = ColQwen2Processor.from_pretrained(
                self.model_name,
                cache_dir=self.cache_dir,
            )
    # ... functions for embedding images and queries
    # @modal.method()
    # async def batch_embed_images(self, images_base64: List[str]) -> np.ndarray:
    # @modal.method()
    # async def batch_embed_queries(self, queries: List[str]) -> np.ndarray:

Generate the embeddings:

$ modal run embedding.py::embed_dataset

Step 3: Create Index in VectorChord (Using Vechord SDK)

With embeddings generated and saved in a Modal Volume, we first need to bring them local.

$ modal volume get colpali-embedding-checkpoint /path/to/local/vidore_embeddings

Now, we'll use the vechord SDK to load these embeddings into our PostgreSQL database and create the necessary multi-vector index. vechord provides a Pythonic, ORM-like way to interact with VectorChord in Postgres.

MultiVector = List[Vector[128]] # Assuming 128 dimensions for ColQwen2

lists = 2500 # lists is the number of the cluster
# Define the database table schema using vechord
class Image(Table, kw_only=True):
    uid: Optional[PrimaryKeyAutoIncrease] = None
    image_embedding: Annotated[MultiVector, MultiVectorIndex(lists=lists)] # Stores the document's visual embeddings 
    query_embedding: Annotated[MultiVector, MultiVectorIndex(lists=lists)]  # Stores the query's embeddings 
    query: str = None
    dataset: Optional[str] = None
    dataset_id: Optional[int] = None

# Connect to your PostgreSQL database
# Ensure the URL points to your local Docker or VectorChord Cloud instance
DB_URL = "postgresql://postgres:postgres@127.0.0.1:5432/postgres" # Default DB
vr = VechordRegistry("colpali", DB_URL)
vr.register([Image]) # Creates the table and multi-vector index if they don't exist

# Function to load embeddings from disk and yield Image objects
@vr.inject(output=Image)
def load_image_embeddings(path: str) -> Iterator[Image]:
    # ... (logic to load numpy arrays from disk, convert to List[Vector]) ...
    # ... (yield Image(...) instances) ...
    print(f"Loaded embeddings from {path}")

if __name__ == "__main__":
    # Path where you downloaded the embeddings from Modal
    embedding_dir = "/path/to/local/vidore_embeddings"
    load_image_embeddings(embedding_dir) # This triggers vechord to insert data
    print("Data loaded and indexed into VectorChord.")

Run the indexer script, vechord handles table creation, index creation, and data insertion.

Step 4: Evaluation - Does it Work?

The final step is crucial: evaluating retrieval performance. Does this OCR-free system deliver accurate results, and can it be fast? We'll use vechord to run queries against our indexed data and look at NDCG@10 and Recall@10. We'll also test the impact of VectorChord's WARP optimization, which accelerates MaxSim calculations. In this tutorial, we will use vidore/arxivqa_test_subsampled dataset for evaluation queries, the data are shown below.

TOP_K = 10

# Define structure for results (optional but good practice)
class Evaluation(msgspec.Struct):
    map: float
    ndcg: float
    recall: float

TOP_K = 10
def evaluate(queries: list[Image], probes: int, max_maxsim_tuples: int) -> list[Evaluation]:
    result  = []
    for query in queries:
        vector = query.query_embedding
        docs: list[Image] = vr.search_by_multivec(
            Image, vector, topk=TOP_K, probe=probes, max_maxsim_tuples=max_maxsim_tuples
        )
        score = BaseEvaluator.evaluate_one(query.uid, [doc.uid for doc in docs])
        result.append(Evaluation(
            map=score.get("map"),
            ndcg=score.get("ndcg"),
            recall=score.get(f"recall_{TOP_K}"),
        ))
    return result

if __name__ == "__main__":
    # Select some queries from the benchmark dataset stored in the DB
    # Example: Get 100 queries from a specific subset
    test_queries: list[Image] = vr.select_by(
        Image.partial_init(dataset="vidore/arxivqa_test_subsampled"), limit=100
    )

    # ... optimize the param ...
    res: list[Evaluation] = evaluate(queries, probes=probes, maxsim_refine=maxsim_refine)
    print("ndcg@10", sum(r.ndcg for r in res) / len(res))
    print("recall@10", sum(r.recall for r in res) / len(res))
    print(f"Total execution time: {total_time:.4f} seconds")

Running the evaluation script yields impressive results:

# Disable WARP
ndcg@10 0.8615
recall@10 0.92
Total execution time: 810 seconds # Baseline accuracy and time

# Enable WARP
# There is no need to focus on specific times, 
# only relative times need to be taken into account, 
# as the tests were performed on a local model with poor performance.
ndcg@10 0.8353
recall@10 0.90
Total execution time: 41 seconds # Dramatic WARP Speed Boost

Analysis:

• High Baseline Accuracy: First, let's look at the baseline performance without WARP optimization. The system achieves an excellent NDCG@10 of 0.8615 and Recall@10 of 0.92. This confirms the fundamental effectiveness of ColQwen2 embeddings paired with VectorChord's precise MaxSim search, delivering state-of-the-art results for visual document retrieval even without speed optimizations.

• Dramatic WARP Speed Boost: Now, observe the impact of enabling VectorChord's WARP optimization. The total execution time plummets from a substantial 810 seconds down to just 41 seconds! This represents a massive ~18.7x speedup for the evaluation query set. This clearly demonstrates WARP's power in dramatically accelerating the computationally intensive MaxSim operations required for late-interaction models like ColQwen2.

• Minimal Accuracy Trade-off: Impressively, this significant speed gain comes at the cost of only a very minor dip in retrieval accuracy. The NDCG@10 slightly decreases to 0.8353 (a difference of less than 0.3), and Recall@10 reduces minimally to 0.9 (a difference of only 0.02).

Conclusion: Visual RAG Made Simpler in Postgres

In this tutorial, we successfully built a high-performance, OCR-free RAG system by bringing together the visual understanding capabilities of ColQwen2, the scalable multi-vector search of VectorChord within PostgreSQL, and the efficient GPU processing of Modal. We saw how this stack allows us to directly query documents based on their visual content, bypassing the pitfalls of traditional OCR pipelines.

The ability to seamlessly integrate this visual dimension into your RAG system directly within Postgres opens up exciting possibilities for applications dealing with visually rich documents like scientific papers, invoices, product manuals, historical archives, and much more.

Ready to try it yourself?

• Explore the Code: https://github.com/xieydd/vectorchord-colqwen2

• Dive Deeper into VectorChord: Check out the VectorChord documentation or try the hassle-free VectorChord Cloud.

• Experiment: Try different VLMs or datasets.

• Share Your Thoughts: Let us know your experiences or questions in the comments below!

This approach represents a significant step towards simpler, more robust, and visually-aware document retrieval systems.

References

• https://huggingface.co/vidore/colqwen2-v1.0

• https://huggingface.co/blog/manu/colpali

• https://blog.vespa.ai/scaling-colpali-to-billions/

Beyond Single Vectors: Why Multi-Vector Rocks

For years, vector search primarily represented entire documents or queries as single, dense vectors. While powerful, this approach involves averaging or compressing complex information into one representation, inevitably leading to a loss of nuance. Imagine trying to capture the entire meaning of a detailed technical document in a single sentence – you'd lose specifics!

Enter multi-vector representations and late interaction, pioneered by models like ColBERT. Instead of one vector per item, these models generate multiple vectors – often one for each token (word or sub-word). The real magic happens during the search process:

1. Query Encoding: Your search query is also broken down into multiple token vectors (let's call them q1, q2, ..., qN).

2. Document Encoding: Similarly, the document is represented by its token vectors (d1, d2, ... dM).

3. Late Interaction & MaxSim Aggregation: Instead of one comparison, we perform fine-grained matching. The core idea is the Maximum Similarity (MaxSim) operation. For each query token vector (like q_i), we find the document token vector (d_j) that has the highest similarity (e.g., cosine similarity or dot product) to it across all document tokens (d1 through dM). This process is repeated for every query token (q1, q2, ..., qN). The final relevance score for the document is then calculated by summing up these maximum similarity values obtained for each query token. This operation is inherently asymmetric, focusing on how well each part of the query is represented somewhere in the document.

Why is this better?

• Contextual Nuance: Late interaction with MaxSim allows the model to capture fine-grained semantic relationships. It can identify if specific important terms from the query strongly match specific parts of the document, rather than relying on a potentially diluted overall average similarity.

• Improved Relevance: By considering focused, token-level interactions and summing the best matches for each query term, models like ColBERT achieve state-of-the-art retrieval quality. This fine-grained approach is particularly powerful for content-rich queries with multiple terms, leading to a better understanding of user intent. On the BEIR benchmark, this translates to a significant performance boost: With the same ModernBert model, Colbert variant achieves 51.6 NDCG@10 compared to 41.6 for dense vector variant.

Our previous blog post explored using ColBERT for reranking, showcasing its power:
Supercharge Vector Search with ColBERT Rerank in PostgreSQL

VectorChord 0.3: Bringing Efficient MaxSim to Postgres, Inspired by WARP

VectorChord 0.3 directly confronts the high computational cost of late interaction. We've integrated a highly optimized multi-vector late-interaction MaxSim operator and index into our core Rust engine, drawing inspiration from the groundbreaking WARP engine (Paper: WARP: An Efficient Engine for Multi-Vector Retrieval, Code: jlscheerer/xtr-warp).

The genius of the WARP approach lies in a fundamental insight: the complex multi-vector MaxSim calculation (comparing N query vectors to M document vectors) can be cleverly decomposed into multiple, independent single-vector search processes. Instead of one massive N x M comparison, WARP effectively performs N separate searches, one for each query token vector, against the indexed document vectors to find its best match.

Our core improvement in VectorChord 0.3's MaxSim scanner is built on this concept. We've implemented MaxSim by leveraging and orchestrating VectorChord's existing, highly optimized single-vector search infrastructure. When a MaxSim query is executed with an index, VectorChord performs multiple single-vector searches—one for each query vector—using our underlying index structures (IVF combined with RaBitQ). It then efficiently aggregates the results based on the MaxSim summation rule. For estimating missing values, we adopt the approach from WARP, using the distance to the centroid and the cumulative cluster size to estimate MaxSim scores for potential candidates.

Reusing our optimized single-vector search components ensures MaxSim benefits from existing performance tuning and stability. Importantly for users, it provides a seamless experience: you can leverage the same familiar index types (IVF with RaBitQ) for both single-vector and multi-vector MaxSim search, without needing a separate system. VectorChord 0.3 proudly stands as the first Postgres extension to deliver this efficient, decomposed MaxSim implementation, making state-of-the-art multi-vector retrieval practical and performant directly within your database.

Powerful Use Cases Unlocked

This new efficiency opens the door to practical implementations of cutting-edge techniques:

1. High-Performance ColBERT Reranking: Apply ColBERT's superior relevance ranking to a candidate set retrieved by a faster first-stage search (like traditional vector search or keyword search) without incurring prohibitive latency penalties. Get the best of both speed and quality.

2. OCR-Free Document Search (ColPali/ColQwen): Imagine searching directly within scanned documents, PDFs, or images without needing a separate, often error-prone, OCR (Optical Character Recognition) step. Models like ColPali or ColQwen generate token embeddings directly from image patches. With VectorChord 0.3's efficient MaxSim, you can now perform late-interaction search over these visual token embeddings directly in Postgres, enabling powerful search over documents previously inaccessible to pure text-based methods.

Stay tuned! We have another article coming soon, diving deep into how VectorChord 0.3 enables revolutionary OCR-free RAG pipelines.

Performance Highlights

We put VectorChord 0.3's new multi-vector MaxSim capabilities to the test on the FiQA (Financial Opinion Mining and Question Answering) dataset. This standard benchmark includes 57,000 documents with approximately 15 million cumulative tokens.

Our initial benchmark results, using ColBERTv2 powered by VectorChord's optimized engine, show promising performance:

• Relevance: We achieved an NDCG@10 score of 34.1. This compares favorably to the 33.6 NDCG@10 reported for the same dataset in the original WARP paper.

• Speed: Queries were executed efficiently, averaging just 35 milliseconds per query.

While these are preliminary results based on a single dataset as we continue optimization, they demonstrate VectorChord 0.3's potential. Users can now leverage the advanced relevance capabilities of multi-vector search and late interaction with impressive speed, approaching the latency often associated with simpler single-vector methods, all directly within their Postgres database.

Get started with VectorChord 0.3

Prerequisites:

• PostgreSQL server with the VectorChord v0.3 installed

You can use our VectorChord-Suite image:

docker run --rm --name vchord_db -d -e POSTGRES_PASSWORD=postgres -p 5432:5432 \
    ghcr.io/tensorchord/vchord-postgres:pg17-v0.3.0

Step 1: Create a Table for Multi-Vector Data

First, define a table to store your data. The key difference is using the array type for your vector column (vector[]) to hold multiple vectors per row.

-- Define a table to store items (e.g., documents), each potentially having multiple vectors.
-- Replace '128' with your actual vector dimensionality.
CREATE TABLE doc (
    id SERIAL PRIMARY KEY, -- A unique identifier for each item
    vecs vector(128)[] -- The column storing an ARRAY of 128-dimensional vectors
);

Step 2: Insert Multi-Vector Data

Insert data into your table. The vecs column takes a PostgreSQL array containing vector types.

-- Insert sample data: one document with 2 vectors, another with 3.
-- Ensure vector dimensions match your table definition (128 in this example).
INSERT INTO doc (id, vecs) VALUES
    (1, array[array[0.1, 0.2, ..., 0.9]::vector, array[0.8, 0.7, ..., 0.1]::vector]),
    (2, array[array[0.5, 0.5, ..., 0.5]::vector, array[0.3, 0.4, ..., 0.7]::vector, array[0.9, 0.1, ..., 0.4]::vector]);
-- Add more data as needed...

Step 3: Create a vchordrq Index with MaxSim Support

To accelerate MaxSim searches, create an index using the vchordrq method and specify the vector_maxsim_ops operator class. A crucial parameter here is build.internal.lists.

• Calculate n: Estimate the total number of individual vectors across your entire dataset.

$$n=N_{doc}*\text{avg}(N_{vector\_per\_doc})$$

• Set lists: The recommended range for build.internal.lists is

$$4 * \sqrt n < \text{lists} < 8 * \sqrt n$$

Choose a value within this range (often powers of 2 work well).

-- Example Calculation:
-- If you have 1,000,000 documents (rows) with an average of 5 vectors each:
-- n = 1,000,000 * 5 = 5,000,000 sqrt(n) ≈ 2236 
-- Lower bound: 4 * 2236 ≈ 8944
-- Upper bound: 8 * 2236 ≈ 17888
-- A good value for 'lists' could be 16384 (power of 2).
-- Assume the K-means clusters are balanced, each will have about 305 vectors.

-- Create the index using vchordrq and vector_maxsim_ops
CREATE INDEX doc_vecs_idx ON doc USING vchordrq (vecs vector_maxsim_ops)
WITH (options = $$
build.internal.lists = [16384] -- Adjust this value based on your calculation!
$$);

Step 4: Configure Search Parameters (Optional but Recommended)

Before querying, you can tune runtime parameters for performance and accuracy:

• vchordrq.probes: Controls how many index lists (clusters) are checked during a search. Higher values increase accuracy (recall) but slow down the search. A common starting point for finding the top 10 results (LIMIT 10) is 32.

• vchordrq.maxsim_refine: Limits the number of vector pairs re-computed with the original precision (otherwise will use the bit distance) for each candidate query token vector. It’s related to the probes, a value of 20% of probed vectors will be sufficient.

-- Set runtime parameters for the current session/transaction
SET vchordrq.probes = 32; -- Adjust based on desired recall vs. speed trade-off
SET vchordrq.maxsim_refine = 2000; -- Adjust based on desired recall vs. speed trade-off

Step 5: Perform a MaxSim Similarity Search

Now you can query using the @# MaxSim operator. Provide your query vectors as a PostgreSQL array of vector types.

-- Find the top 10 documents most similar to the given set of query vectors
SELECT id FROM doc 
ORDER BY vecs @# ARRAY[array[0.4, 0.1, ..., 0.8]::vector, array[0.7, 0.2, ..., 0.3]::vector]
LIMIT 10;

You've now successfully set up a table, indexed it for multi-vector MaxSim search, and executed your first query using VectorChord 0.3's MaxSim operator! Experiment with the probes and max_maxsim_tuples parameters to find the best balance of speed and accuracy for your specific use case.

Earth Genome's Earth Index platform is tackling this challenge head-on. It now offers global coverage, making the entire land surface of our planet searchable using the power of AI. This isn't just a technical achievement; it's a new tool designed to empower environmental stewardship by providing accessible, actionable intelligence from satellite data.

We at VectorChord are incredibly proud that our PostgreSQL vector search extension is playing a crucial role in powering this ambitious and vital mission.

What is Earth Index Doing? Turning Pixels into Planetary Insights

Imagine finding a specific type of small-scale environmental impact, like an unreported quarry or a new patch of deforestation, somewhere within satellite imagery covering millions of square kilometers. Traditionally, analyzing this data required significant expertise and resources, making tracking specific threats across vast, remote areas difficult, especially for conservation groups, journalists, and regulators with limited budgets. The animation below illustrates how efficiently Earth Index pinpoints the illegal narcotrafficking airstrips in the Amazon, akin to finding a toothpick on a soccer field.

But how does Earth Index make finding environmental features of interest possible on a global scale? The magic lies in a cutting-edge approach using AI foundation models trained via contrastive learning and the resulting vector embeddings.

Here’s a glimpse into their innovative process:

1. Tiling the Globe: Earth Index divides the Earth's entire land surface into over 3.2 billion overlapping tiles, each roughly 10 hectares.

2. Learning the "Planetary DNA" with Contrastive Self-Supervision: Earth Index leverages breakthroughs in AI, specifically foundation models trained for Earth Observation (EO) using self-supervised, contrastive learning methods (like the DINO objective). The foundation model (e.g., an SSL4EO Vision Transformer) learns the fundamental visual language of satellite imagery by distinguishing between pairs of images from the same location (positive pairs) and pairs from different locations (negative pairs). This happens across diverse global data (like Harmonized Landsat Sentinel-2, Sentinel-1 radar) without needing explicit human labels for features like "forest" or "city."

3. Generating Vector Embeddings: Once pre-trained, this model acts as an encoder. Imagery for each tile is fed into it, producing a vector embedding – a rich numerical representation or fingerprint capturing the tile's essential visual and structural characteristics in a highly compressed format (>10,000x compression). This high-dimensional vector acts like a unique "Planetary DNA" for that piece of land.

4. Creating a Searchable Planet: These 3.2 billion vector embeddings form the core of Earth Index. They need to be stored in a specialized system optimized for rapidly finding vectors that are numerically similar to each other. Users can provide an example of what they're looking for (e.g., by clicking on a known illegal mine or a specific type of agricultural infrastructure on the map), and Earth Index searches this massive dataset to find other locations across the globe (or within a specific region) with similar vector fingerprints.

Real-World Impact: Earth Index in Action for Environmental Protection

This technological foundation translates directly into powerful tools for understanding and protecting our environment. It empowers journalists (like those at Mongabay and Radio Free Europe), indigenous communities, conservation organizations (like the World Bank), and researchers to uncover hidden environmental changes, monitor industrial footprints, and gather evidence for conservation efforts with unprecedented ease and speed. The figure shows the shrimp farms in the Gulf of Fonseca.

Here are some examples of how Earth Index is being used:

• Combating Deforestation and Illegal Activities: In collaboration with Mongabay, Earth Index was used to pierce through dense canopy cover and identify hidden, illegal narcotrafficking airstrips in the Amazon. Similarly, the platform can rapidly locate illegal gold mining scars, like those devastating parts of the Yanomami Indigenous Territory, helping to quantify the scale of the problem and guide enforcement efforts.

• Monitoring Resource Extraction: Radio Free Europe utilized Earth Index to identify over 1300 quarries across the Balkans, revealing that roughly half lacked proper permits, highlighting potential environmental damage and unregulated extraction. This approach can monitor fracking wellpads, rubber processing plants, or other industrial facilities.

• Protecting Water Quality and Ecosystems: Earth Index can locate Concentrated Animal Feeding Operations (CAFOs) near sensitive waterways to assess runoff risks or identify expansions of shrimp farming into vital mangrove ecosystems.

• Tracking Agricultural Expansion: The tool helps monitor the conversion of natural habitats into agricultural land by searching for patterns associated with new clearings or specific crop types.

The Infrastructure Challenge: Searching Billions of Vectors Affordably

Generating 3.2 billion high-dimensional vectors is one challenge; hosting and searching them efficiently presents another significant hurdle. Earth Index needed a solution that could:

• Handle Massive Scales: Searching billions of vectors requires specialized indexing and querying capabilities.

• Integrate Seamlessly with Geospatial Data: Earth Index heavily relies on geospatial operations (like filtering results within specific administrative boundaries or proximity to rivers). This made PostgreSQL with PostGIS a natural and essential choice for their core database. Standalone vector databases often lack the rich geospatial features required.

• Maintain Performance: Users need results relatively quickly for the platform to be effective for investigation and monitoring.

• Control Costs: Storing and searching billions of vectors, especially if requiring vast amounts of RAM for purely in-memory indexes, can become prohibitively expensive for many organizations, particularly non-profits.

Earth Index needed a vector search solution that lived within their chosen PostgreSQL environment, could scale massively, perform well, and wouldn't break the bank on hardware costs.

Why PostgreSQL Rocks for Planetary-Scale Vectors

While specialized vector databases exist, Earth Index faced a critical need: seamlessly integrating vector similarity search with rich geospatial data and operations. This is where PostgreSQL, combined with extensions, truly shines:

• Unified Data Platform: PostgreSQL, especially with the powerful PostGIS extension, allowed Earth Index to keep their vector embeddings alongside their crucial geospatial metadata (tile locations, administrative boundaries, proximity to features like rivers). This avoids the complexity and potential synchronization issues of managing separate databases for vector search and relational/geospatial data.

• Leveraging the Power of SQL & PostGIS: Earth Index could combine vector similarity searches with standard SQL filters and complex PostGIS spatial queries (e.g., "find vectors similar to this example within this specific national park boundary" or "near this river system") all within a single query interface. Standalone vector databases often lack this mature geospatial query capability.

• Maturity and Ecosystem: Building on PostgreSQL means inheriting its decades of development, renowned stability, robust tooling, and wide community support – essential for a mission-critical platform like Earth Index.

However, searching 3.2 billion vectors efficiently requires more than just standard PostgreSQL capabilities.

Why Earth Index Chose VectorChord for Performance and Affordability

While the PostgreSQL ecosystem provided the ideal foundation, achieving the required performance and affordability at the scale of 3.2 billion vectors presented a significant challenge. Earth Index needed a solution that could deliver speed without breaking the bank.

Finding the right balance of performance, cost, and integration was challenging. Hutch Ingold, CTO of Earth Genome, explains their considerations:

When we evaluated solutions for our 3.2 billion vectors – recognizing the rapidly evolving tech landscape – standard pgvector didn't meet our performance requirements at that scale. Dedicated options like Qdrant posed difficulties: hosted versions were too expensive, and self-hosting clustered deployments on our Kubernetes infrastructure seemed insufficiently mature at the time. We also examined cloud services, such as Google's Vertex AI Vector Search, but the list price of $237,000 per month for our dataset was simply prohibitive.

This is where VectorChord became the enabling technology:

• Performance Beyond Vanilla PGVector: VectorChord delivers the necessary speed enhancements – up to 5x faster queries, 16x higher insert throughput, and 16x quicker index building – addressing the performance limitations encountered with standard pgvector at this massive scale.

• Affordable Scale via Disk-Based Indexing: Critically, VectorChord's optimized disk-based indexing (leveraging cost-effective SSDs) directly counters the prohibitive costs highlighted by the CTO. It avoids the massive RAM requirements of purely in-memory approaches and the exorbitant monthly fees of managed cloud vector services, making the 3.2 billion vector index financially feasible.

• Native Integration & Scalability: As a PostgreSQL extension, VectorChord provided these benefits within their chosen database, requiring no infrastructure changes. It's built to scale efficiently, handling the billions of vectors needed for Earth Index's global monitoring platform while maintaining interactive performance.

VectorChord provided the crucial combination of high performance, massive scalability, and cost-effectiveness directly within the familiar and powerful PostgreSQL environment that Earth Index needed.

Earth Index: Technical Architecture and Setup

To manage its planetary dataset effectively and affordably, Earth Index built its backend on a cluster of three powerful AWS EC2 i8g.16xlarge instances, powered by the latest Graviton4 ARM processor and Nitro SSD. Each provides 512 GB RAM, 64 vCPUs, and 15 TB of NVMe storage, offering a solid foundation for both computation and disk I/O. This setup, costing around $12,000 per month, delivers massive cost savings compared to the estimated $237,000 monthly fee for a comparable managed cloud vector service, making the project economically viable. These instances run PostgreSQL enhanced with VectorChord and PostGIS, efficiently hosting both the vector data and other application components.

The core data resides in a partitioned table structure across the three nodes. The primary table, globe24, stores each tile's unique ID, its geographic point geometry (geometry(Point, 4326)), a partition key (partkey), and the crucial 384-dimensional AI-generated embedding stored efficiently as halfvec(384). This sharding distributes the 3.2 billion rows across the cluster, allowing for parallel processing.

A key advantage of this architecture is the ability to perform complex, integrated queries. Earth Index commonly executes SQL statements that seamlessly combine VectorChord's fast vector search (embedding <=> '{vector}'::vector) with PostGIS geospatial filtering (st_within(geom, '{geom}')) and partition pruning (partkey IN (...)). This allows users to find visually similar locations within specific geographic boundaries efficiently. Even with billions of vectors, this integrated approach yields practical performance, achieving a median query latency (p50) of approximately 761 ms for typical searches returning the top 500 results.

Enabling Environmental Action at Scale

By providing a cost-effective, scalable, and tightly integrated vector search solution within PostgreSQL, VectorChord helps enable Earth Index's groundbreaking work. It allows them to manage their enormous "Planetary DNA" database affordably, ensuring that this powerful tool for environmental monitoring can reach the organizations and communities who need it most.

Join the Earth Index waitlist today to explore the environmental challenges in your neighborhood, understand your ecological footprint, and be part of a global effort to protect the planet.

We are thrilled to see Earth Index leverage VectorChord to turn satellite data into actionable environmental intelligence. Their success demonstrates the power of combining cutting-edge AI, robust database technology, and efficient vector search to address critical global challenges.

Interested in bringing large-scale vector search to your PostgreSQL database? Learn more about VectorChord or get started today!

Specialized vector databases aren’t always needed. With PostgreSQL extensions like pgvector, pgvectorscale, VectorChord (which evolved from pgvecto.rs)​​, you get vector search plus relational power—no extra infrastructure. But adoption isn’t seamless. Here’s what real users say:

• The query takes 30 seconds for the first time and 100ms if it's a repeat query. The problem is very slow IO.

• Is it normal to take 3 hours to build the index for 12 million rows with 75 dimension vector?

• After 13 hours of building index for 200 million vectors of 75 dimensions each, I have noticed a heavy memory usage at 100GB/124GB at 30% of building index, and server will eventually crash.

• We're hitting some challenges, particularly around need for frequent index updates, throughput constraints, lack of parallel index scans and no built-in quantization.

In this article, we'll take a deep dive into these questions and help you make informed decisions by understanding not only the "what" but also the "why" behind these tools. Let's explore how to navigate these complexities and choose the right solution for your needs.

What do we value most in a vector database?

In PostgreSQL, shared_buffers is a crucial memory area acting as the primary cache for data pages, including table data and indexes. By storing frequently accessed vector index blocks in this memory buffer, PostgreSQL significantly reduces slow disk I/O. When a query needs data, PostgreSQL first checks shared_buffers. If the data is present, it's retrieved quickly without disk access. If not, the data is fetched from disk and loaded into shared_buffers for future queries. To configure this, use ALTER SYSTEM SET shared_buffers = 'xGB' in psql and restart the server.

Let's say you have some vectors in a table, after the index is built, it's time to start a query. Even though you have set enough shared_buffers for PostgreSQL, it still needs to read the index from disk for the first query. However, for the repeated query, since the vector can be retrieved directly from shared_buffers, it can be much faster.

While the first query after a cold start or index change matters, sustained performance often depends on how quickly queries execute once the 'hot' or frequently accessed parts of the index are cached in memory (shared_buffers), mimicking the repeated query scenario. Therefore, optimizing for this cached performance is crucial for many real-world applications.

In this blog, we will compare the performance of vector similarity search extensions in these situations:

• Query performance when memory is sufficient

• Query performance when memory is low and disk becomes a bottleneck

We will also discuss other aspects related to index building, such as:

• Index build speed: How long does it take to build the index? If I have multiple cores in the instance, can it use them all to speed things up?

• Memory usage: How much memory should be reserved for index creation? This is the most important consideration when choosing an instance.

• Disk usage: How many vectors can be hosted on a given amount of disk space? This is also a consideration when choosing an instance that uses non-expandable NVMe SSD (a type of high-performance solid state drive) storage.

• Ease of use: Can I use a simple CREATE INDEX command to create an index? How difficult is it to tune for query performance?

Finally, we will discuss the insertion performance after index building is complete. This is important if the user has streamed data that will trigger frequent index updates.

Experimental setup

Here we present information about the hardware and datasets used in all of the following experiments.

In this table, we show the version of all the extensions we use in these experiments:

Vectors in memory: for small scale data

For a vector search system, if you want the best performance, it's important to make sure that the entire index is cached in shared_buffers, which is more appropriate on a memory-optimized instance. In this situation, the size of shared_buffers should ideal be large enough to hold the entire index.

For our experiment, we choose a medium-sized dataset, LAION-5m, which consists of 5 million vectors with 768 dimensions. To host LAION-5m in shared_buffers, we can use the AWS instance I4I Extra Large, with a 937GB NVMe SSD and 32GB of memory.

On this machine, we set the shared_buffers to 24GB and measure the performance of the first query and then the repeated query. The following is the result of the repeated query performance over pgvector, pgvectorscale and VectorChord.

The following graphs illustrate the trade-off between query speed (QPS) and search quality (Recall@10/100). Recall@10 measures the percentage of true nearest neighbors found within the top 10 results. In general, higher QPS can be achieved at the cost of lower recall, and vice versa. A better system may achieve higher recall at the same QPS, or higher QPS at the same recall level. Our goal is typically to maximize QPS while maintaining a high recall target (e.g., 95%).

To measure first query performance, we cleared shared_buffers before running the 10,000 test queries sequentially. Although the buffer starts empty, index/data pages loaded into the cache by earlier queries within the same test run can sometimes be reused by subsequent queries. This can cause the instantaneous QPS to increase during the run. We report the average QPS across all 10,000 queries to provide a representative measure of performance with a cold cache.

In conclusion, VectorChord consistently achieves higher QPS at high recall levels (e.g., >95% Recall@10) compared to the other extensions. When it comes to the other extensions, pgvectorscale performs better for first query, but once data is cached, pgvector will outperform it.

Vectors in disk: for large scale data

There are times when things are different. For billions of vectors, it is not possible to use memory to store the entire index on a single machine. When the size of the vectors to be read exceeds the size of memory, the speedup effect of shared_buffers is greatly reduced, and most of the data is fetched from disk.

While shared_buffers becomes less effective, the operating system's page cache can still provide some caching benefits, although its performance is generally lower than shared_buffers.

To illustrate this situation, we can host LAION-5m on a much smaller machine, C6ID Large, with 4GB of memory and set shared_buffers to 2GB. This configuration ensures that memory will be insufficient and most queries will be forced to go to disk.

In these cases, the difference in performance between the first and repeated queries will be much smaller due to insufficient memory. Therefore, it's acceptable to evaluate only the repeated query performance of each extension.

As always, VectorChord consistently achieves higher QPS at high recall levels (e.g., >95% Recall@10) compared to the other extensions, followed by pgvector and pgvectorscale with similar QPS.

Experience in index building

As we discussed earlier, users also care about some index building metrics: Index build time, memory usage, disk usage and usability. We measured these metrics during the CREATE INDEX step of the above experiment on the Laion-5m dataset:

• Core utilization: Percentage of available CPU cores effectively used during index creation. Higher values indicate better parallelism.

• Index build time: The time taken by the SQL command to build the index. If the index build requires an additional step (VectorChord external build), this should also be included.

• Memory usage: Peak memory usage during the index build, including the 1G shared_buffers allocated for PostgreSQL.

• Disk usage: The size of PGDATA on disk after index building is finished, including vectors and the index.

All query performance is measured after an external build, on a standalone machine with A10 GPU. If you insist on using the same I4I Extra Large instance for external builds, the index build time will increase to 50 minutes, which is still competitive.

In this case, we set shared_buffers to 1GB and logged the peak memory usage of each extension. Both VectorChord's internal build and VectorChord's external build are set to 25 iters for K-means clustering. For ease of use, pgvector, pgvectorscale and VectorChord's internal build have similar syntax with a single SQL to build the index, while VectorChord's external build is more complicated.

For most aspects, VectorChord's external build gained a leading position, but this partly sacrificed the ease of use, as developers need to implement an additional K-means clustering step outside of PostgreSQL and manage the resulting centroids.

If you want to start with a simple CREATE INDEX command for a straightforward approach, then VectorChord's external build may not be the best choice , you can turn to other extensions / methods:

• Choose VectorChord's internal build for a faster index build

• Choose pgvector if you really don't have enough memory

• Choose pgvectorscale to reduce disk requirements or get better capacity

Insertion performance: for stream-generated data

With streaming vector data, insertion speed can become a new concern. In real-time data analysis systems, the number of new vectors written per second can be very high. Each time new vectors are added to an index, the index itself must be updated. This capability is usually referred to as insertion performance.

So, we set up an experiment to see how well the insertion worked. After building the index for 5 million vectors, we insert another 100,000 random generated vectors into the table one after the other and calculate the insertion time of all.

As you can see, VectorChord (1565 Insert/Sec) performs much better than pgvector (246 Insert/Sec) and pgvectorscale (107 Insert/Sec) when inserting data. For most workloads, 100 Insert/Sec is more than enough. However, if you find that a lot of vectors are continuously inserted in your production (for example, 500 Insert/Sec), this could be a problem.

Summary

Based on the results above, this table shows the differences between the three extensions in various aspects. We can see that the main advantage of VectorChord is concentrated in performance, while pgvector and pgvectorscale have their own advantages in terms of usability and capacity.

If you want to see more detailed differences, this table shows them between the three extensions in various aspects. We can see that the main advantage of VectorChord is concentrated in performance, while pgvector and pgvectorscale have their own advantages in terms of usability and capacity.

Want to know more about how VectorChord compares to other extensions? In future blogs, we will explore the differences of PostgreSQL vector extensions in more scenarios, such as bm25 hybrid search (combining traditional keyword search with vector similarity search) and filtered vector search. Ready to try VectorChord for your vector search needs? Download the extension and feel free to join our Discord community or contact us at support@tensorchord.ai.

In their benchmark, Neon compared query performance on their database platform with their pg_search extension (based on Rust's Tantivy library via pgrx) against the Postgres built-in fulltext search setting with tsvector and GIN index. They commendably stated they optimized this standard setup by adding GIN indexes where appropriate (benchmark code available here).

However, while adding GIN indexes is a necessary first step, their results showing significantly slower performance for the "standard" setup suggest crucial additional optimization steps for PostgreSQL FTS were likely missed. The conclusion that standard FTS is inherently much slower than pg_search might be based on an unintentionally handicapped baseline.

Let's dive into how to correctly set up and use standard PostgreSQL FTS for optimal performance, addressing the specific configuration flaws likely present in the baseline used in the Neon benchmark, and demonstrating the true speed of built-in FTS. We'll show concrete numbers demonstrating a ~50x performance increase just by applying these standard optimizations to the baseline configuration.

The Benchmark Setup (Recap from Neon's Analysis)

The analysis used a table structure similar to this with 10 million log entries:

CREATE TABLE benchmark_logs (
    id SERIAL PRIMARY KEY,
    message TEXT,
    country VARCHAR(255),
    severity INTEGER,
    timestamp TIMESTAMP,
    metadata JSONB
);

They tested various query types. Many involved searching the message text field, using queries structured like this in their "standard Postgres" examples:

-- Create index on to_tsvector('english', message)
CREATE INDEX message_gin ON benchmark_logs USING gin (to_tsvector('english', message));
-- Example problematic query structure (likely used in Neon's baseline)
SELECT country, COUNT(*)
FROM benchmark_logs
WHERE to_tsvector('english', message) @@ to_tsquery('english', 'research')
GROUP BY country
ORDER BY country;

Even with a GIN index present on message, this query structure and the likely default GIN index settings are where the performance issues for the standard FTS baseline begin.

Our Test Environment Details (Replicating the Baseline Scenario)

To demonstrate the impact of proper optimization on standard FTS, we used the following setup:

• Instance: AWS EC2 i7ie.xlarge with local NVMe SSD (minimizing I/O impact).

• CPU: 4 vCPUs.

• PostgreSQL Configuration: PostgreSQL 16 via Docker, configured for the hardware:

    docker run -d \
      --name my-postgres \
      --network=host \
      -v /mnt/localssd/pgdata:/var/lib/postgresql/data \
      -e POSTGRES_PASSWORD=mysecretpassword \
      postgres:latest \
      -c shared_buffers=8GB \
      -c maintenance_work_mem=8GB \
      -c max_parallel_workers=4 \
      -c max_worker_processes=4
  

• Parallelism Note: This setup provides max_parallel_workers_per_gather = 2. The Neon post mentioned their test environment used 8 parallel workers due to larger instances, implying potentially higher parallelism (max_parallel_workers_per_gather up to 8) than our setup. Our optimized results for standard FTS were achieved with less query parallelism.

Mistake #1: Calculating tsvector On-the-Fly (Major issue)

The sample queries shown in the Neon blog (and common in basic FTS examples) calculate the tsvector within the WHERE clause:

WHERE to_tsvector('english', message) @@ to_tsquery('english', 'research')

This forces PostgreSQL to:

1. Perform Expensive Computation: Run to_tsvector() (parsing, stemming, etc.) repeatedly for many rows during query execution.

2. Limit Index Efficiency: Prevent the most direct and efficient use of the GIN index, even if one exists on the base message column.

The Fix (For a Proper Standard FTS Baseline): Pre-calculate and store the tsvector.

1. Add a tsvector column:

    ALTER TABLE benchmark_logs ADD COLUMN message_tsvector tsvector;
   

2. Populate the column:

    UPDATE benchmark_logs SET message_tsvector = to_tsvector('english', message);
   

3. Index the tsvector column (with fastupdate=off): (As shown in Fix #1)

4. Rewrite queries:

    -- Optimized standard FTS query
    SELECT country, COUNT(*)
    FROM benchmark_logs
    WHERE message_tsvector @@ to_tsquery('english', 'research') -- Use the indexed column!
    GROUP BY country
    ORDER BY country;
   

Mistake #2 : Ignoring GIN Index fastupdate (Minor issue)

While Neon's benchmark correctly identified GIN as the index type for standard FTS, it likely used the default setting: fastupdate=on.

• fastupdate=on (Default): Prioritizes index update speed by using pending lists. This significantly slows down searches over time, especially on large, static datasets used for benchmarks, as both the main index and growing pending lists must be scanned. It also contributes to index bloat and less efficient page structure.

• fastupdate=off: Prioritizes search speed. Index updates are slower, but the resulting index is more compact and significantly faster to query as there are no pending lists to scan. Essential for read-heavy workloads or benchmarking search performance.

For a fair comparison focused on search speed, the standard FTS baseline should have used fastupdate=off.

The Fix (For a Proper Standard FTS Baseline):

-- Create the GIN index correctly for search speed
CREATE INDEX idx_gin_logs_message_tsvector
ON benchmark_logs USING GIN (message_tsvector) -- Index the dedicated tsvector column
WITH (fastupdate = off);

The Performance Impact: A ~50x Speedup for Standard FTS

What happens when we apply these standard optimizations to the standard PostgreSQL FTS setup? We tested a query in Neon's examples on the 10-million-row dataset:

• Unoptimized Standard FTS (Neon's Baseline): ~41301 ms (41.3 seconds)

• Optimized Standard FTS (Our Fixes): ~877 ms (0.88 seconds)

This demonstrates a ~50x speed improvement for standard PostgreSQL FTS achieved simply by applying well-established best practices. This transforms the baseline performance dramatically, suggesting the comparison point used in the Neon benchmark may not have represented the true potential of built-in FTS. This speedup was achieved even with potentially less query parallelism available than in Neon's tests.

The Grain of Truth: Ranking (ts_rank) Performance

Neon's analysis did likely highlight a valid point regarding ranking speed. Functions like ts_rank or ts_rank_cd can be relatively slow in standard PostgreSQL compared to just finding matches. This is because ranking requires fetching and processing data for all matching rows before limiting, which can be I/O and CPU intensive.

Beyond Standard FTS: Advanced Ranking with VectorChord-BM25

While standard FTS excels at finding matches quickly, if sophisticated, high-performance ranking is critical, specialized tools are often required. Instead of relying solely on built-in functions, consider VectorChord-BM25.

VectorChord-BM25 is a PostgreSQL extension specifically designed for fast, relevance-based ranking using the well-regarded BM25 algorithm. BM25 goes beyond simple term matching, estimating relevance based on term frequency within a document and inverse document frequency across the entire collection.

Key advantages reported for VectorChord-BM25 include:

• High Performance: Designed for speed, reportedly outperforming other solutions like Elasticsearch (up to 3x faster) and potentially pg_search for ranking tasks.

• BM25 Algorithm: Implements a standard, effective relevance ranking model.

• Specialized Indexing: Uses a dedicated bm25 index type (leveraging optimizations like Block WeakAnd) which is crucial for both performance and calculating the global statistics needed for BM25 scoring.

• Dedicated Data Type: Introduces a bm25vector type to store tokenized representations suitable for BM25.

If your application heavily relies on the quality and speed of relevance ranking, and standard ts_rank proves insufficient, VectorChord-BM25 presents a powerful, integrated alternative within PostgreSQL.

Conclusion: Standard FTS is Faster Than You Might Think

Standard PostgreSQL FTS is a highly capable and performant feature for finding documents when correctly optimized using tsvector columns and properly configured GIN indexes (with fastupdate=off for reads). Benchmarks suggesting otherwise may be comparing against unoptimized baselines.

For advanced relevance ranking where standard ts_rank falls short, specialized extensions like VectorChord-BM25 offer significant performance improvements by implementing algorithms like BM25 with dedicated data types and optimized indexing strategies.

Choose the right tool for the job: optimize standard FTS first, and if high-performance relevance ranking is paramount, explore dedicated extensions built for that purpose.

VectorChord-BM25 allows you to leverage the power of the BM25 algorithm, a standard in information retrieval, without needing external search engines. This release marks a significant step forward, focusing heavily on enhancing the core text processing component: tokenization, unlocking greater flexibility and significantly improved multilingual support.

The Big News: Introducing pg_tokenizer.rs

The cornerstone of VectorChord-BM25 0.2 is a completely refactored and decoupled tokenizer extension: pg_tokenizer.rs.

Why the change? We realized that tokenization – the process of breaking down text into meaningful units (tokens) – is incredibly complex and highly dependent on the specific language and use case. Supporting multiple languages with their unique rules, custom dictionaries, stemming, stop words, and different tokenization strategies within a single monolithic extension was becoming cumbersome.

By moving the tokenizer into its own dedicated project (pg_tokenizer.rs) under the permissive Apache License, we achieve several key benefits:

1. Modularity & Flexibility: Developers can now use or customize the tokenizer independently of the core BM25 ranking logic.

2. Easier Contribution: The focused nature of pg_tokenizer.rs makes it simpler for the community to contribute new language support, tokenization techniques, or custom filters.

3. Faster Iteration: We can now develop and release improvements to the tokenizer more rapidly without needing a full VectorChord-BM25 release cycle.

4. Enhanced Customization: Users gain significantly more control over how their text, regardless of language, is processed before ranking.

What's New in Tokenization (Thanks to pg_tokenizer.rs)

This new architecture enables several powerful features in v0.2:

• Expanded Language Support: Directly handle diverse linguistic needs with dedicated tokenizers like Jieba (Chinese) and Lindera (Japanese), alongside powerful multilingual LLM-based tokenizers (like Gemma2 and LLMLingua2) trained on vast datasets covering a wide array of languages.

• Richer Tokenization Features: You now have more granular control over the tokenization pipeline:

  • Custom Stop Words: Define your own lists of words to ignore during indexing and search.

  • Custom Stemmers: Apply stemming rules for various supported languages or even define custom ones.

  • Custom Synonyms: Define synonym lists to treat different words as equivalent (e.g., "postgres", "postgresql", "pgsql").

  • Language-Specific Options: Leverage fine-grained controls available within specific tokenizers (like Lindera or Jieba) when needed.

Show Me the Code!

Let's see how easy it is to use the new tokenizer features.

1. Using a Pre-trained Multilingual LLM Tokenizer (LLMLingua2)

LLM-based tokenizers are great for handling text from many different languages.

-- Enable the extensions (if not already done)
CREATE EXTENSION IF NOT EXISTS vchord_bm25;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer;

-- Update search_path for the first time
ALTER SYSTEM SET search_path TO "$user", public, tokenizer_catalog, bm25_catalog;
SELECT pg_reload_conf();

-- Create a tokenizer configuration using the LLMLingua2 model
SELECT create_tokenizer('llm_tokenizer', $$
model = "llmlingua2"
$$);

-- Tokenize some English text
SELECT tokenize('PostgreSQL is a powerful, open source database.', 'llm_tokenizer');
-- Output: {2795,7134,158897,83,10,113138,4,9803,31344,63399,5} -- Example token IDs

-- Tokenize some Spanish text (LLMLingua2 handles multiple languages)
SELECT tokenize('PostgreSQL es una potente base de datos de código abierto.', 'llm_tokenizer');
-- Output: {2795,7134,158897,198,220,105889,3647,8,13084,8,55845,118754,5} -- Example token IDs

-- Integrate with a table
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    passage TEXT,
    embedding bm25vector
);

INSERT INTO documents (passage) VALUES ('PostgreSQL is a powerful, open source database.');

UPDATE documents
SET embedding = tokenize(passage, 'llm_tokenizer')
WHERE id = 1; -- Or process the whole table

2. Creating a Custom Tokenizer with Filters (Example: English)

This example defines a custom pipeline, including lowercasing, Unicode normalization, skipping non-alphanumeric tokens, using NLTK English stop words, and the Porter2 stemmer. It then automatically trains a model and sets up a trigger to tokenize text on insert/update.

CREATE TABLE articles (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding bm25vector
);

-- Define a custom text analysis pipeline
SELECT create_text_analyzer('english_analyzer', $$
pre_tokenizer = "unicode_segmentation"  # Basic word splitting
[[character_filters]]
to_lowercase = {}                       # Lowercase everything
[[character_filters]]
unicode_normalization = "nfkd"          # Normalize Unicode
[[token_filters]]
skip_non_alphanumeric = {}              # Remove punctuation-only tokens
[[token_filters]]
stopwords = "nltk_english"              # Use built-in English stopwords
[[token_filters]]
stemmer = "english_porter2"             # Apply Porter2 stemming
$$);

-- Create tokenizer, custom model based on 'articles.content', and trigger
SELECT create_custom_model_tokenizer_and_trigger(
    tokenizer_name => 'custom_english_tokenizer',
    model_name => 'article_model',
    text_analyzer_name => 'english_analyzer',
    table_name => 'articles',
    source_column => 'content',
    target_column => 'embedding'
);

-- Now, inserts automatically generate tokens
INSERT INTO articles (content) VALUES
('VectorChord-BM25 provides advanced ranking features for PostgreSQL users.');

SELECT embedding FROM articles WHERE id = 1;
-- Output: {1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 7:1, 8:1}
-- Bm25vector based on the custom model and pipeline

3. Using Jieba for Chinese Text

CREATE TABLE chinese_docs (
    id SERIAL PRIMARY KEY,
    passage TEXT,
    embedding bm25vector
);

-- Define a text analyzer using the Jieba pre-tokenizer
SELECT create_text_analyzer('jieba_analyzer', $$
[pre_tokenizer.jieba]
# Optional Jieba configurations can go here
$$);

-- Create tokenizer, custom model, and trigger for Chinese text
SELECT create_custom_model_tokenizer_and_trigger(
    tokenizer_name => 'chinese_tokenizer',
    model_name => 'chinese_model',
    text_analyzer_name => 'jieba_analyzer',
    table_name => 'chinese_docs',
    source_column => 'passage',
    target_column => 'embedding'
);

-- Insert Chinese text
INSERT INTO chinese_docs (passage) VALUES
('红海早过了，船在印度洋面上开驶着。'); -- Example sentence

SELECT embedding FROM chinese_docs WHERE id = 1;
-- Output: {1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 7:1, 8:1, 9:1, 10:1, 11:1, 12:1}
-- Bm25vector based on Jieba segmentation

(For full examples, including custom stop words and synonyms, please refer to the pg_tokenizer.rs documentation.)

Understanding the Tokenizer Configuration

The new tokenizer system revolves around two core concepts:

1. Text Analyzer: Defines how raw text is processed into a sequence of tokens. It consists of:

   • character_filters: Modify text before splitting (e.g., lowercasing, Unicode normalization).

   • pre_tokenizer: Splits the text into initial tokens (e.g., based on Unicode rules, Jieba, Lindera).

   • token_filters: Modify or filter tokens after splitting (e.g., stop word removal, stemming, synonym replacement).

2. Model: Defines the mapping from the processed tokens to the final integer token IDs used by BM25. Models can be:

   • pre-trained: Use established vocabularies and rules (like bert-base-uncased, llmlingua2). Great for general purpose and multilingual use.

   • custom: Build a vocabulary dynamically from your own data, tailored specifically to your corpus and language(s).

You can define these components separately or inline them when creating a tokenizer using a simple TOML configuration format passed as a string in SQL.

Get Started with VectorChord-BM25 0.2!

This release significantly boosts the flexibility and power of VectorChord-BM25, especially for users dealing with multiple languages or needing fine-grained control over text processing.

• GitHub Repository: https://github.com/tensorchord/VectorChord-bm25

• Tokenizer Documentation: https://github.com/tensorchord/pg_tokenizer.rs

We encourage you to try out version 0.2 and explore the new tokenization capabilities. Your feedback is invaluable – please report any issues or suggest features on our GitHub repository.

Upgrade your PostgreSQL full-text search today with the enhanced multilingual flexibility of VectorChord-BM25 0.2 and pg_tokenizer.rs!

This article explores how to implement Hybrid Search in Postgres using VectorChord-bm25 for keyword-based retrieval and VectorChord for semantic search. By leveraging VectorChord-bm25's BM25 ranking with the Block-WeakAnd algorithm and VectorChord's advanced vector similarity capabilities, you can build a robust search system that seamlessly integrates keyword precision with semantic understanding. Whether you're building a recommendation engine, a document retrieval system, or an enterprise search solution, this guide will walk you through the steps to unlock the full potential of hybrid search.

All related benchmark codes can be found here.

Hybrid Search Explained

Hybrid Search merges the results from vector search and keyword search. Vector search is based on the semantic similarity between the query and the documents, while keyword search relies on the BM25 ranking algorithm. However, it is important to note that BM25 itself is only responsible for scoring documents, while the broader keyword search process also includes steps such as tokenization, indexing, and query parsing. Hybrid Search utilizes reranker or fuses the results from both methods. Fuses include RRF (Reciprocal Rank Fusion) or weighted scoring for multiple recall strategies. Model-based rerank includes cross encoder models like bge-reranker-v2-m3 and multi-vector representation model ColBERT.

What is the VectorChord?

You may be familiar with pgvector, an open-source vector similarity search extension for Postgres. However, scalable or performance-critical scenarios may require a more advanced vector search solution. VectorChord is an excellent choice.

Check out the blog post "VectorChord: Store 400k Vectors for $1 in PostgreSQL" to learn more about its motivation and design.

What is VectorChord-BM25

VectorChord-BM25 is a PostgreSQL extension for keyword search. It not only implements BM25 ranking but also includes a tokenizer and a Block-WeakAnd index to improve speed.

What is BM25?

BM25 (Best Matching 25) is a probabilistic ranking function used in information retrieval to assess how well a document matches a query. It calculates relevance scores based on term frequency (TF) and inverse document frequency (IDF), while also applying document length normalization. The formula ensures that terms appearing frequently within a document (TF) and those rare across the corpus (IDF) are appropriately weighted, improving search accuracy and relevance. IDF measures how often a word appears across the document collection. The fewer the occurrences, the higher its value. TF represents the frequency of a specific word from the query appearing in the given document. A higher TF indicates a stronger relevance between the query and the document.

Why a Postgres native BM25 Ranking implementation is needed?

In our previous blog post, VectorChord-BM25: Revolutionizing PostgreSQL Search with BM25 Ranking — 3x Faster Than Elasticsearch, my colleague Allen has already provided a detailed and sufficient explanation, which can be referred to.

Tutorial: Use VectorChord-BM25 and VectorChord implement Hybrid Search

In this tutorial, we will guide you through the steps to implement Hybrid Search using VectorChord-BM25 and VectorChord in PostgreSQL. We will cover the following topics:

• Semantic Search with VectorChord

• Keyword Search with VectorChord-BM25

• Rerank

Prerequisites

1. Postgres with VectorChord-BM25 and VectorChord

If you want to reproduce the tutorial quickly, you can use the tensorchord/vchord-suite image to run multiple extensions that TensorChord provides.

You can run the following command to build and start Postgres with VectorChord-BM25 and VectorChord.

docker run   \           
  --name vchord-suite  \
  -e POSTGRES_PASSWORD=postgres  \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg17-latest

CREATE EXTENSION IF NOT EXISTS vchord CASCADE;
CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;
CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;
\dx
pg_tokenizer | 0.1.0   | tokenizer_catalog | pg_tokenizer
vchord       | 0.3.0   | public            | vchord: Vector database plugin for Postgres, written in Rust, specifically designed for LLM
vchord_bm25  | 0.2.0   | bm25_catalog      | vchord_bm25: A postgresql extension for bm25 ranking algorithm
vector       | 0.8.0   | public            | vector data type and ivfflat and hnsw access methods

2. Prepare the Embedding Model and Data

For embedding, we can use a pre-trained embedding model like BGE-M3 to generate embeddings for the documents. BGE-M3 is a high-quality embedding model known for its versatility in multi-functionality, multi-linguality, and multi-granularity.

For validation, we use the BEIR dataset, a heterogeneous benchmark for information retrieval. It is easy to use and allows you to evaluate your models across 15+ diverse IR datasets.

First, you need to load your data into PostgreSQL.Then you can use the following SQL query to generate embeddings for the documents.

with self.conn.cursor() as cursor:
    cursor.execute(
        f"CREATE TABLE IF NOT EXISTS {self.dataset}_corpus (id TEXT, text TEXT, emb vector({self.vector_dim}), bm25 bm25vector);"
    )

    for did, doc in tqdm(zip(doc_ids, docs), desc="insert corpus"):
        emb = self.sentence_encoder.encode_doc(doc)
        cursor.execute(
            f"INSERT INTO {self.dataset}_corpus (id, text, emb) VALUES (%s, %s, %s)",
            (did, doc, emb),
        )

For embedding:

from FlagEmbedding import BGEM3FlagModel
class SentenceEmbedding:
    def __init__(self, model_name: str = "BAAI/bge-m3"):
        self.model = BGEM3FlagModel(
            model_name,
            use_fp16=True,
        )
    def encode_docs(self, documents: list[str]):
        return self.model.encode(
            documents,
            batch_size=32,
            max_length=8192,
        )['dense_vecs']

The data process and embedding generation code can be found here.

Semantic Search with VectorChord

For semantic search, we utilize the RabitQ algorithm, which is highly optimized in the VectorChord PostgreSQL extension. RabitQ is a quantization algorithm for high-dimensional spaces, designed to improve the storage and retrieval efficiency of high-dimensional data, such as embedded vectors.

It achieves this by mapping high-dimensional vectors to a low-dimensional discrete space while preserving the similarity information of the original vectors. This process reduces storage requirements and computational costs while maintaining high retrieval accuracy.

centroids = min(4 * int(self.num**0.5), self.num // 40)
ivf_config = f"""
residual_quantization = true
[build.internal]
lists = [{centroids}]
build_threads = {workers}
spherical_centroids = false
"""
with self.conn.cursor() as cursor:
    cursor.execute(f"SET max_parallel_maintenance_workers TO {workers}")
    cursor.execute(f"SET max_parallel_workers TO {workers}")
    cursor.execute(
        f"CREATE INDEX {self.dataset}_rabitq ON {self.dataset}_corpus USING vchordrq (emb vector_l2_ops) WITH (options = $${ivf_config}$$)"
    )

If you find that building the index on your own dataset is too slow, you can utilize external build to accelerate the process. For more details, please refer to our previous blog: Benefits and Steps of External Centroids Building in VectorChord.

Once the index is built, you can use the following SQL query to search for similar documents:

probe = int(0.1 * min(4 * int(self.num**0.5), self.num // 40))
with self.conn.cursor() as cursor:
    cursor.execute(f"SET vchordrq.probes = {probe}")
    cursor.execute(
        f"select q.id as qid, c.id, c.score from {self.dataset}_query q, lateral ("
        f"select id, {self.dataset}_corpus.emb <-> q.emb as score from "
        f"{self.dataset}_corpus order by score limit {topk}) c;"
    )

Key-Word Search with VectorChord-BM25

For keyword search, we need to use a tokenizer to convert the text into a BM25 vector, which is similar to a sparse vector that stores the vocabulary ID and frequency.

cursor.execute(
    f"SELECT create_tokenizer('{self.dataset}_token', $$",
    f"tokenizer = 'unicode'",
    f"stopwords = 'nltk'",
    f"table = '{self.dataset}_corpus'",
    f"column = 'text'",
    f"$$);"
)
cursor.execute(
    f"UPDATE {self.dataset}_corpus SET bm25 = tokenize(text, '{self.dataset}_token')"
)

Let me explain this operation in detail. The tokenize function is used to tokenize the text with the specified tokenizer. In this case, we used the Unicode tokenizer. The output of the tokenize function is a BM25 vector, which is a sparse vector that stores the vocabulary ID and frequency of each word in the text. For example, 1035:7 means the word with vocabulary ID 1035 appears 7 times in the text.

postgres=# select bm25 from fiqa_corpus limit 1;
-- Output: {1035:7, 1041:1, 1996:1, 1997:1, 1999:3, 2010:3, 2015:7, 2019:1, 2022:1, 2028:4, 2036:2, 2041:1, 2051:2, 2054...

After creating the index, you can calculate the BM25 score between the query and the vectors. Note that the BM25 score is negative, meaning that the higher the score, the more relevant the document is. We intentionally make it negative so that you can use the default ORDER BY to retrieve the most relevant documents first.

with self.conn.cursor() as cursor:
    cursor.execute(
        f"SELECT q.id AS qid, c.id, c.score FROM {self.dataset}_query q, LATERAL ("
        f"SELECT id, {self.dataset}_corpus.bm25 <&> to_bm25query('{self.dataset}_text_bm25', q.text , '{self.dataset}_token') AS score "
        f"FROM {self.dataset}_corpus "
        f"ORDER BY score "
        f"LIMIT {topk}) c;"
    )

Rerank/Fuse

Once you obtain the results from both semantic search and keyword search, you can use a fuse or rerank process to merge the results.

1. Fuse (Reciprocal Rank Fusion (RRF))

The advantage of RRF lies in its independence from specific score units; it is based on ranking for fusion, making it suitable for sorting systems with different scoring criteria.

$$\text{RRF}(d) = \sum_{i=1}^{n} \frac{1}{k + \text{rank}_i(d)}$$

Among them:

• d is the document.

• n is the number of ranking systems.

• rank is the ranking in the sorting system.

• k is the adjustment parameter, the value is usually 60 (empirical value), used to control the impact of ranking on scores.

for rank, (query_id, doc_id, _) in enumerate(result, start=1):
    if query_id not in rrf_scores:
        rrf_scores[query_id] = {}
    if doc_id not in rrf_scores[query_id]:
        rrf_scores[query_id][doc_id] = 0
    # Calculate and accumulate RRF scores
    rrf_scores[query_id][doc_id] += 1 / (k + rank)

2. Cross-Encoder model Rerank

In semantic search, we already use Bi-Encoder vectorized the documents and queries separately. But this independent encoding leads to a lack of interaction between the query and the document. Cross-Encoder model will input the query and document as a whole into the model, the model will see the content of both at the same time, capturing the fine-grained semantic relationship between them through the Transformer layer. Compared with RRF, Cross-Encoder model can capture the fine-grained semantic relationship between the query and the document, and it can be more accurate but slower.

reranker = FlagReranker(
    'BAAI/bge-reranker-v2-m3', 
    query_max_length=256,
    passage_max_length=512,
    use_fp16=True,
    devices=["cuda:0"] # change ["cpu"] if you do not have gpu, but it will be very slow
) # Setting use_fp16 to True speeds up computation with a slight performance degradation

for query_id, docs in tqdm(results.items()):
    scores = reranker.compute_score(pairs, normalize=True) 
    for i, doc_id in enumerate(docs):
        bge_scores[query_id][doc_id] = scores[i]

Evaluation

We have tested the NDCG@10 with those methods on several BEIR datasets. Here are the top10 results:

The table above demonstrates several conclusions:

• Compared to BM25 and semantic search, the cross-encoder model rerank can significantly improve search performance across different datasets.

• On some datasets, RRF may lead the performance decrease. Please conduct verification tests before deciding whether to use RRF. If effective, choosing RRF will be a very economical choice, as it is much, much faster than reranking with the cross-encoder model, and almost no resource consumption.

All related benchmark codes can be found here. If you have any questions, please feel free to contact us on Discord or email us at vectorchord-inquiry@tensorchord.ai.

Future Work

The results highlight the great potential of the hybrid search method. In the future, we will focus on the following aspects to advance RAG:

• Improve the reranking method: Explore the other model-based rerank, such as ColBERT or ColPali, to enhance reranking performance.

• Integrate graph search: Investigate the use of the Apache AGE extension for graph search and integrate it with the hybrid search method to further improve search performance.

References

• https://github.com/tensorchord/vectorChord/

• https://github.com/tensorchord/VectorChord-bm25

• https://techcommunity.microsoft.com/blog/adforpostgresql/introducing-the-graphrag-solution-for-azure-database-for-postgresql/4299871

• https://www.pinecone.io/learn/hybrid-search-intro/

• https://www.pinecone.io/learn/series/rag/rerankers/

• https://blog.vespa.ai/improving-zero-shot-ranking-with-vespa-part-two/

What’s New?

• BM25 Scoring & Ranking: Get more precise and relevant search results with BM25, helping you find what matters most.

• Optimized Indexing: Thanks to the Block WeakAnd algorithm, searches are quicker and more efficient, even with large datasets.

• Enhanced Tokenization: Improved stemming and stop word handling mean better accuracy and smoother performance.

We built VectorChord-BM25 to be simple, powerful, and fully integrated with PostgreSQL—because we believe great tools should make your life easier, not more complicated. We hope you’ll give it a try and let us know what you think.

BM25

Before we get to the exciting news, let’s take a quick look at BM25, the algorithm that powers modern search engines. BM25 is a probabilistic ranking function that determines how relevant a document is to a search query.

The BM25 formula might look a bit intimidating at first, but let’s break it down step by step to make it easy to understand. Here’s it:

$$\text{score}(Q, D) = \sum_{q \in Q} \text{IDF}(q) \cdot \frac{f(q, D) \, (k_1 + 1)}{f(q, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

Term Frequency (TF): The term f(q,D) represents how often the query term q appears in document D. The more a term appears in a document, the higher its relevance score. For example, if "AI" appears 5 times in Document A and 2 times in Document B, Document A gets a higher score for this term.

Inverse Document Frequency (IDF): The term IDF(q) measures how rare or common the term q is across all documents. Rare terms (e.g., "NVIDIA") are given more weight than common terms (e.g., "revenue"). This ensures that unique terms have a greater impact on the relevance score.

Document Length Normalization: The term ∣D∣ represents the length of document D, while avgdl is the average length of all documents in the collection. This part of the formula adjusts the score to account for document length, ensuring shorter documents aren’t unfairly penalized. For instance, a concise report won’t be overshadowed by a lengthy one that only briefly mentions the query term.

Tuning Parameters: The parameters k1 and b allow the formula to be fine-tuned. k1 controls the extent to which term frequency impacts the score, while b balances the effect of document length normalization. These parameters can be adjusted to optimize results for specific datasets.

Consider searching a database of financial reports for "Tesla stock performance." A report that mentions "Tesla" ten times will score higher than one that mentions it only twice. However, the term "stock" might appear in many reports, so it’s given less weight than "Tesla," which is more specific. Furthermore, a short, concise report about Tesla’s stock performance won’t be overshadowed by a lengthy report that only briefly mentions Tesla.

Existing Solution in Postgres

Now that we’ve covered the basics, let’s take a look at the existing solutions for full-text search in PostgreSQL.

PostgreSQL provides built-in support for full-text search through the combination of the tsvector data type and GIN (Generalized Inverted Index) indexes. Here’s how it works: text is first converted into a tsvector, which tokenizes the content into lexemes—standardized word forms that make searching more efficient. A GIN index is then created on the tsvector column, significantly speeding up query performance and enabling fast retrieval of relevant documents, even when dealing with large text fields. This integration of text processing and advanced indexing makes PostgreSQL a powerful and scalable solution for applications that require efficient full-text search.

-- Create a table with a text column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    content_vector tsvector
);

-- Insert sample data with tsvector conversion
INSERT INTO documents (content, content_vector) VALUES
('PostgreSQL is a powerful, open-source database system.', to_tsvector(...)),
('Full-text search in PostgreSQL is efficient and scalable.', to_tsvector(...)),
('BM25 is a ranking function used by search engines.', to_tsvector(...));

-- Create a GIN index on the tsvector column
CREATE INDEX idx_content_vector ON documents USING GIN (content_vector);

-- Query using tsvector, rank results with ts_rank, and leverage the GIN index
SELECT id, content, ts_rank(content_vector, to_tsquery('english', 'PostgreSQL & search')) AS rank
FROM documents
WHERE content_vector @@ to_tsquery('english', 'PostgreSQL & search')
ORDER BY rank DESC;

However, PostgreSQL has a limitation: it lacks modern relevance scoring mechanisms like BM25. Instead, it returns all matching documents and relies on ts_rank to re-rank them, which can be inefficient. This makes it challenging for users to quickly identify the most important results, especially when dealing with large datasets.

Another solution is ParadeDB, which pushes full-text search queries down to Tantivy for results. It supports BM25 scoring and complex query patterns like negative terms, aiming to be a complete replacement for ElasticSearch. However, it uses its own unique syntax for filtering and querying and delegates filtering operations to Tantivy instead of relying on Postgres directly. Its implementation requires several hooks into Postgres' query planning and storage, potentially leading to compatibility issues.

In contrast, VectorChord-BM25 takes a different approach. It focuses exclusively on bringing BM25 ranking to PostgreSQL in a lightweight and native way. We implemented the BM25 ranking algorithm and the Block WeakAnd technique from scratch, building it as a custom operator and index (similar to pgvector) to accelerate queries. Designed to be intuitive and efficient, VectorChord-BM25 provides a seamless API for enhanced full-text search and ranking, all while staying fully integrated with PostgreSQL’s ecosystem.

VectorChord-BM25

Our implementation introduces a novel approach by developing the BM25 index and search algorithm from the ground up, while seamlessly integrating with PostgreSQL’s existing development interfaces to ensure maximum compatibility.

Inspired by the PISA engine, Tantivy, and Lucene, we incorporated the BlockMax WeakAnd algorithm to facilitate efficient score-based filtering and ranking. Furthermore, we employ bitpacking for ID compression to enhance overall efficiency and re-implemented a user data-based tokenizer to more closely align with ElasticSearch’s performance.

The table below compares the Top1000 Queries Per Second (QPS)—a metric that measures how many queries a system can process per second while retrieving the top 1000 results for each query—between our implementation and ElasticSearch across various datasets from bm25-benchmarks:

On average, our implementation achieves 3 times higher QPS compared to ElasticSearch across the tested datasets, showcasing its efficiency and scalability. However, speed alone isn’t sufficient—we also prioritize accuracy. To ensure relevance, we evaluated NDCG@10 (Normalized Discounted Cumulative Gain at 10), a key metric for assessing ranking quality.

The table below compares the NDCG@10 scores between VectorChord-BM25 and ElasticSearch across various datasets:

We have dedicated substantial effort to align VectorChord-BM25 with ElasticSearch’s behavior, ensuring a fair and precise comparison. As demonstrated in the table, our implementation achieves NDCG@10 scores that are comparable across most datasets, with certain cases even surpassing ElasticSearch (e.g., trec-covid and scifact).

We’ll share the technical details of our alignment efforts in a later section, including how we addressed tokenization, scoring, and other critical components to achieve these results. Before that, let’s explore how to use VectorChord-BM25 in PostgreSQL.

Quick start

To get started with VectorChord-BM25, we’ve put together a detailed guide in our GitHub README that walks you through the installation and configuration process. Below, you’ll find a complete example showing how to use VectorChord-BM25 for BM25 full-text search in Postgres. Each SQL snippet comes with a clear explanation of what it does and why it’s useful.

The extension consists of three main components:

• Tokenizer: Converts text into a bm25vector, which is similar to a sparse vector that stores vocabulary IDs and their frequencies.

• bm25vector: Represents the tokenized text in a format suitable for BM25 scoring.

• bm25vector Index: Speeds up the search and ranking process, making it more efficient.

If you’d like to tokenize some text, you can use the tokenize function. It takes two arguments: the text you want to tokenize and the name of the tokenizer.

-- tokenize text with bert tokenizer
SELECT tokenize('A quick brown fox jumps over the lazy dog.', 'Bert');
-- Output: {2474:1, 2829:1, 3899:1, 4248:1, 4419:1, 5376:1, 5831:1}
-- The output is a bm25vector, 2474:1 means the word with id 2474 appears once in the text.

One unique aspect of the BM25 score is that it relies on a global document frequency. This means the score of a word in a document is influenced by how frequently that word appears across all documents in the set. To calculate the BM25 score between a bm25vector and a query, you’ll first need a document set. Once that’s in place, you can use the <&> operator to perform the calculation.

Here is an example step by step. First, create a table and insert some documents:

-- Setup the document table
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    passage TEXT,
    embedding bm25vector
);

INSERT INTO documents (passage) VALUES
('PostgreSQL is a powerful, open-source object-relational database system. It has over 15 years of active development.'),
('Full-text search is a technique for searching in plain-text documents or textual database fields. PostgreSQL supports this with tsvector.'),
...
('Effective search ranking algorithms, such as BM25, improve search results by understanding relevance.');

Then tokenize it

UPDATE documents SET embedding = tokenize(passage, 'Bert');

Create the index on the bm25vector column so that we can collect the global document frequency.

CREATE INDEX documents_embedding_bm25 ON documents USING bm25 (embedding bm25_ops);

Now we can compute the BM25 score between the query and the vectors. It’s worth noting that the BM25 score is negative—this is by design. A higher (less negative) score indicates a more relevant document. We intentionally made the score negative so that you can use the default ORDER BY clause to easily retrieve the most relevant documents first.

SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', 'PostgreSQL', 'Bert') AS rank
FROM documents
ORDER BY rank
LIMIT 10;

Other Tokenizers

In addition to BERT, VectorChord-BM25 also supports Tocken and Unicode Tokenizers. Tocken is a Unicode tokenizer pre-trained on the wiki-103-raw dataset with a minimum frequency (min_freq) of 10. Since it’s a pre-trained tokenizer, you can use it similarly to BERT.

SELECT tokenize('A quick brown fox jumps over the lazy dog.', 'Tocken');

Unicode is a tokenizer that builds a vocabulary list from your data, similar to the standard behavior in Elasticsearch and other full-text search engines. To enable this, you need to create a specific one for your data using create_unicode_tokenizer_and_trigger(vocab_list_name, table_name, source_text_column, tokenized_vec_column).

CREATE TABLE documents (id SERIAL, text TEXT, embedding bm25vector);
SELECT create_unicode_tokenizer_and_trigger('test_token', 'documents', 'text', 'embedding');

INSERT INTO documents (text) VALUES ('PostgreSQL is a powerful, open-source object-relational database system.');
CREATE INDEX documents_embedding_bm25 ON documents USING bm25 (embedding bm25_ops);
SELECT id, text, embedding <&> to_bm25query('documents_embedding_bm25', 'PostgreSQL', 'test_token') AS rank
    FROM documents
    ORDER BY rank
    LIMIT 10;

Please check out the tokenizer documentation if you want to know more about it.

Faster than ElasticSearch (Really?)

To ensure our new plugin provides meaningful and relevant results—not just faster performance—we rigorously evaluated VectorChord-BM25 using bm25-benchmarks. We leveraged the widely recognized BEIR benchmark for information retrieval, focusing on two key metrics: QPS (Queries Per Second) to measure speed and NDCG@10 (Normalized Discounted Cumulative Gain at 10) to assess relevance and ranking accuracy.

In our initial version, we were thrilled to see our system achieve 3-5 times higher QPS compared to ElasticSearch. However, we also noticed that on certain datasets, our NDCG@10 scores were significantly lower than ElasticSearch’s. After a thorough analysis, we realized these gaps weren’t caused by our indexing implementation but rather by differences in tokenization approaches.

To address this, we invested considerable effort to align our tokenization process with ElasticSearch’s, ensuring a more accurate and fair comparison.

Issue 1: Stopword List

ElasticSearch defaults to using Lucene’s Standard Tokenizer, which relies on a relatively short stopword list. In contrast, our initial implementation utilized NLTK’s much more comprehensive stopword list. As a result, ElasticSearch’s searches include certain stopwords, leading to longer inverted index lists that require additional scanning time—ultimately impacting performance.

Issue 2: Stemming

We initially used the snowball stemmer from the rust-stemmer library to handle word variations, but we observed discrepancies compared to ElasticSearch’s implementation. Upon investigation, we discovered that the rust-stemmer’s version of snowball was outdated. Following the guidelines from the official snowball repository, we regenerated the files using the latest version.

When we aligned both the stopword list and stemmer between our system and ElasticSearch, our performance advantage decreased from 300% to 40%. Even so, on three datasets—nq, fever, and climate-fever—a noticeable performance gap persisted. A deeper comparison revealed a subtle but critical detail: in the bm25-benchmark, ElasticSearch preprocesses data differently from other frameworks, which contributed to the remaining discrepancy.

Issue 3: Data Preprocessing

In the BEIR dataset, each document includes both a title and a text field. While most frameworks concatenate these fields into a single string for indexing, ElasticSearch accepts JSON input and indexes the title and text separately. During querying, ElasticSearch performs a multi_match operation, searching both fields independently and combining their scores (using the higher score plus 0.5 times the lower score). This approach yields significantly better NDCG@10 results but requires searching two separate indexes, which can substantially impact performance.

To ensure a fair comparison, we re-ran our ElasticSearch tests by concatenating the title and text fields. With this adjustment, VectorChord-BM25 was able to match ElasticSearch’s results. Interestingly, ElasticSearch’s QPS (geometric mean) increased from 135 to 341 when using concatenated fields, making it 25% faster than VectorChord-BM25’s 271.91 QPS in Top10 query tests. However, in Top1000 tests, VectorChord-BM25 achieved a QPS of 112 compared to ElasticSearch’s 49—making our implementation 2.26 times faster.

This experience highlights the challenges of conducting fair performance comparisons between different BM25 full-text search implementations. The tokenizer is an exceptionally complex and influential component, and ElasticSearch’s intricate default settings add another layer of complexity to the evaluation process.

Future

We’re still in the early stages of this project, and we’ve already pinpointed several areas for performance optimization. Tokenization is an inherently complex process—even for English, we face numerous decisions and trade-offs. Our next step is to fully decouple the tokenization process, transforming it into an independent and extensible extension. This will enable us to support multiple languages, allow users to customize tokenization for better results, and even incorporate advanced features like synonym handling.

Our ultimate goal is to empower users to perform high-quality, relevance-based full-text searches on PostgreSQL with ease. Combined with VectorChord, we aim to deliver a first-class data infrastructure for RAG (Retrieval-Augmented Generation). After all, choosing PostgreSQL is always a solid decision!

If you have any questions about vector search or full-text search in PostgreSQL, feel free to reach out to us! You can connect with us at:

• Discord: https://discord.gg/KqswhpVgdU

• Email: vectorchord-inquery@tensorchord.ai

More Benchmarks

In addition to the Top1000 benchmarks, we have also conducted extensive evaluations for Top10 results. These benchmarks provide further insights into the performance of our implementation across different datasets. If you’re interested in exploring these results in detail, feel free to refer to the data provided here.