Adding a reranker to your vector database

Practical · ~10 min read · Updated 25 Aug 2026

Vector databases return neighbours, not answers. The rerank stage sits between the database and your prompt, and the integration is the same three lines everywhere: raise the limit, score the candidates, truncate. What changes per database is only how you ask for more rows and how you carry the payload back.

The pattern

Whatever the store, reranking changes your retrieval call in exactly one way — you ask for more rows than you intend to use:

CANDIDATES = 50   # what you ask the database for
KEEP = 5          # what reaches the prompt

rows = store.search(query_vector, limit=CANDIDATES)
scores = reranker.predict([(query, r.text) for r in rows])
top = [r for _, r in sorted(zip(scores, rows), key=lambda p: -p[0])][:KEEP]

Two details matter more than the database choice. First, keep the original row objects through the rerank so you still have ids, metadata and permissions on the other side — reranking text alone and then trying to match it back by string is a reliable way to lose your primary keys. Second, the reranker sees only the text you hand it, so if your stored text is a heading-plus-body blob, that is what gets scored.

pgvector (Postgres)

pgvector returns rows ordered by distance. You raise the LIMIT, keep the ids, and rank in the application layer:

import psycopg
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def search(conn, query: str, query_vec, keep: int = 5):
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT id, content
            FROM documents
            ORDER BY embedding <=> %s
            LIMIT 50
            """,
            (query_vec,),
        )
        rows = cur.fetchall()

    pairs = [(query, content) for _id, content in rows]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, rows), key=lambda p: -p[0])
    return [row for _score, row in ranked[:keep]]

<=> is cosine distance; use <-> for L2 or <#> for inner product, matching whatever your index was built with. Note that raising the limit from 5 to 50 interacts with your index settings — with HNSW you may need SET LOCAL hnsw.ef_search above the default so the index actually considers enough neighbours to return 50 good ones. A limit larger than ef_search silently gives you padding rather than candidates, which looks exactly like a reranker that does not help.

Qdrant

Same shape, with the payload carried along so you do not lose metadata:

from qdrant_client import QdrantClient

client = QdrantClient(url="http://localhost:6333")

def search(query: str, query_vec, keep: int = 5):
    hits = client.query_points(
        collection_name="documents",
        query=query_vec,
        limit=50,
        with_payload=True,
    ).points

    scores = reranker.predict([(query, h.payload["text"]) for h in hits])
    ranked = sorted(zip(scores, hits), key=lambda p: -p[0])
    return [h for _s, h in ranked[:keep]]

If you filter by tenant, permission or date, apply that filter in the query_points call rather than after reranking. Filtering afterwards means you paid to score rows you then threw away, and your 50 candidates might collapse to 3.

Elasticsearch

Elasticsearch is where reranking pays off most visibly, because you can feed it a hybrid candidate set — BM25 and vectors fail on different queries, and the cross-encoder sorts out the union:

resp = es.search(
    index="documents",
    size=50,
    query={"bool": {"should": [
        {"match": {"content": query}},
        {"knn": {"field": "embedding", "query_vector": query_vec, "k": 50}},
    ]}},
)

hits = resp["hits"]["hits"]
scores = reranker.predict([(query, h["_source"]["content"]) for h in hits])
ranked = sorted(zip(scores, hits), key=lambda p: -p[0])
top = [h for _s, h in ranked[:5]]

The BM25 and kNN scores are on different scales and are not meaningfully comparable, which is the usual argument for reciprocal rank fusion. A cross-encoder sidesteps that problem entirely: it re-scores every candidate on one scale of its own, so how the candidate got into the pool stops mattering.

When the database reranks for you

Several stores now offer a rerank step inside the query — as a managed integration with a hosted model, or as a native second stage. Using it is a reasonable default, and it saves you a network hop and a chunk of glue code.

What you trade away is worth knowing before you commit:

• Model choice narrows to what the vendor integrates, which may not include the model that wins on your domain.
• Evaluation gets harder — it is more work to A/B two rerankers when one of them lives inside the query engine.
• Your API key and your passages now flow through the database vendor as well as the model vendor.

The application-layer pattern above stays portable across all of them, so it is a sound place to start even if you later move the stage into the database. Check your store's current documentation for what it supports — this area moves quickly, and we don't keep a feature matrix across stores. Pinecone, below, is the one store we track; its figures are re-checked against Pinecone’s own pages every day.

Pinecone, as a worked example

Pinecone reranks in two ways: inside a query, by adding a rerank block to its search-with-text call so results come back already reordered, or as a separate inference.rerank call on any list of documents. It hosts three rerankers:

On the paid plans each costs $2 per 1,000 requests (Cohere’s is counted in “rerank units”). Two details from Pinecone’s documentation are worth knowing before you build on it:

• cohere-rerank-3.5 is gone in all but name. Pinecone deprecated it on 1 July 2026 and has answered requests for it with cohere-rerank-4-fast since 31 August. Rankings carry over, but the scores are on a different scale, so any fixed score threshold needs re-tuning.
• Cohere’s model runs on Azure AI, and Pinecone says its requests may be processed outside the United States. That is the “your passages flow through another vendor” trade-off above, made concrete.

pinecone-rerank-v0’s 512-token limit counts the query and the document together, and by default anything longer is cut at the limit without an error, so chunk documents to fit. bge-reranker-v2-m3 defaults the other way and returns an error instead.

Pitfalls

Cross-encoder cost scales linearly with the candidate count, so going from 50 to 200 roughly quadruples the rerank latency and, on a hosted API, the bill. The cost calculator puts numbers on the second half of that.

Keep reading