arXiv:2607.11933v1 Announce Type: new Abstract: Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using a two-stage

中文全文翻译正在补齐，可先查看下方英文原文。

[Submitted on 11 Jul 2026]

Title:Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking

View a PDF of the paper titled Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking, by Shreeya Dasa Lakshminath and 1 other authors

View PDF HTML (experimental)

Abstract:Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using a two-stage pipeline: supervised fine-tuning on a custom query-document relevance dataset via the Unsloth framework with LoRA adapters, followed by 4-bit quantization for efficient inference. The resulting model replaces the cross-encoder in a dual-retriever RAG pipeline combining BM25 and dense vector search. Evaluated on a domain-specific question-answering benchmark using the RAGAS framework, our fine-tuned LLaMA 3 reranker achieves gains of 14% in answer relevancy, 16% in context precision, 19% in answer similarity, and 21% in answer correctness over the cross-encoder baseline, while reducing inference overhead through 4-bit quantization. These results demonstrate that instruction-tuned LLMs can be adapted into accurate, efficient rerankers without the quadratic complexity of traditional cross-encoders.

Submission history

From: Shreeya Dasa Lakshminath [

view email

]

[v1]

Sat, 11 Jul 2026 04:31:20 UTC (784 KB)