Zen Reranker: Two-Stage Retrieval Done Right

Embedding-based retrieval is fast but imprecise. Cross-encoder reranking is precise but slow. The combination unlocks the best of both. Today we release the Zen Reranker, purpose-built for two-stage retrieval.

Two-Stage Retrieval

Modern retrieval pipelines typically operate in two stages:

    Query -> [Embedding Retrieval] -> Top-K Candidates -> [Reranker] -> Final Results
             (fast, approximate)                          (slow, precise)
    

Stage 1 : Bi-encoder embeddings enable fast approximate search over millions of documents. Retrieve top-100 to top-1000 candidates.

Stage 2 : Cross-encoder reranker scores each candidate against the query with full attention. Reorder to get final top-10 or top-20.

The reranker sees query-document pairs together, enabling much finer relevance distinctions than independent embeddings.

The Zen Reranker

Our reranker builds on several key design decisions:

Architecture

• Base : 330M parameter encoder-only transformer
• Input : Concatenated query and document with separator tokens
• Output : Single relevance score (0-1)
• Context : 512 tokens (query + document combined)

Training Data

We curated training data from multiple sources:

1. MS MARCO : Search query-passage pairs (positive and hard negatives)
2. Natural Questions : Question-answer pairs from Wikipedia
3. Synthetic pairs : LLM-generated query-document pairs with labels
4. Human judgments : 50K expert-annotated pairs across domains

Total: 12M training pairs with 4-way relevance labels.

Training Objective

We use a listwise loss that considers the full ranking:

    def listwise_loss(scores, labels):
        # Softmax over candidate scores
        probs = softmax(scores)
        # Cross-entropy with label distribution
        return -sum(labels * log(probs))
    

This outperforms pointwise binary classification by teaching the model to rank, not just classify.

Benchmark Results

BEIR Reranking

Reranking BM25 top-100 results:

Reranking Zen Embeddings

Combined with our 7680d embeddings:

The two-stage pipeline achieves +12% improvement with acceptable latency overhead.

Domain-Specific Performance

Specialized domains with complex language benefit most.

Usage

Basic Usage

    from zen.reranker import ZenReranker
    
    reranker = ZenReranker.from_pretrained("zoo-labs/zen-reranker")
    
    query = "What causes climate change?"
    documents = [
        "Greenhouse gases trap heat in the atmosphere...",
        "The weather today is sunny and warm...",
        "CO2 emissions from burning fossil fuels..."
    ]
    
    scores = reranker.score(query, documents)
    # [0.92, 0.12, 0.87]
    

Integration with Retrievers

    from zen.retrieval import ZenRetriever
    from zen.reranker import ZenReranker
    
    retriever = ZenRetriever("zoo-labs/zen-embed-xl")
    reranker = ZenReranker.from_pretrained("zoo-labs/zen-reranker")
    
    # Stage 1: Fast retrieval
    candidates = retriever.retrieve(query, k=100)
    
    # Stage 2: Precise reranking
    reranked = reranker.rerank(query, candidates, k=10)
    

Batched Inference

For production workloads:

    # Batch queries for efficiency
    queries = ["query 1", "query 2", ...]
    candidate_lists = [[docs...], [docs...], ...]
    
    results = reranker.batch_rerank(
        queries, 
        candidate_lists,
        batch_size=32,
        k=10
    )
    

Deployment Considerations

Hardware Requirements

• Minimum : 4GB GPU memory
• Recommended : 8GB+ for batched inference
• CPU : Viable for low-throughput (<10 QPS)

Latency Optimization

1. Batching : Process multiple query-doc pairs together
2. Quantization : INT8 reduces latency 40% with <1% quality loss
3. Early termination : Stop scoring when top-k is confident
4. Caching : Cache scores for repeated query-document pairs

Scaling

For high-throughput applications:

• Deploy multiple replicas behind load balancer
• Use async inference with request queuing
• Consider distilled smaller models for extreme latency requirements

Model Release

The Zen Reranker is available under Apache 2.0:

• Hugging Face : huggingface.co/zoo-labs/zen-reranker
• ONNX : Optimized for deployment
• TensorRT : NVIDIA-optimized variant

Conclusion

Two-stage retrieval with a quality reranker is the pragmatic choice for production search systems. The Zen Reranker provides state-of-the-art reranking in an efficient, easy-to-deploy package.

Fast first, then precise.

Zach Kelling is a co-founder of Zoo Labs Foundation.