Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
LLM rerankers in conversational recommendation show performance gains that are highly dependent on retrieval protocol, candidate pool composition, and decoding settings Best proprietary reranker achieves NDCG@10 of 0.1497 vs 0.0939 for strongest non-LLM baseline under strict candidate-aware scoring on ReDial Unconstrained zero-shot generation yields artificially inflated scores (0.2925), revealing a significant evaluation gap compared to matched-pool evaluation Switching from semantic to collabo
Analysis
TL;DR
- LLM rerankers in conversational recommendation show performance gains that are highly dependent on retrieval protocol, candidate pool composition, and decoding settings
- Best proprietary reranker achieves NDCG@10 of 0.1497 vs 0.0939 for strongest non-LLM baseline under strict candidate-aware scoring on ReDial
- Unconstrained zero-shot generation yields artificially inflated scores (0.2925), revealing a significant evaluation gap compared to matched-pool evaluation
- Switching from semantic to collaborative-filtering candidates improves NDCG@10 by over 50%, demonstrating candidate generation is the dominant factor
- No open-weight LLM outperformed the tuned shallow autoencoder baseline under the strict evaluation protocol
Why It Matters
This work exposes a critical reproducibility and evaluation problem in the LLM recommendation space: many reported gains may reflect favorable evaluation protocols rather than genuine model superiority. For practitioners building conversational recommenders, it highlights that investing in candidate generation and pool management may yield far greater returns than simply upgrading the reranker model.
Technical Details
- Evaluated on the ReDial conversational movie recommendation benchmark using a shared retrieve-then-rerank pipeline, comparing proprietary LLMs, open-weight LLMs, fine-tuned models, collaborative-filtering, and sequential baselines
- Systematically varied three key hyperparameters: candidate-pool size, first-stage retriever type (semantic vs. collaborative-filtering), and decoding temperature (0 to 1.0)
- Under strict candidate-aware scoring with a top-250 semantic pool, the best proprietary reranker reached NDCG@10 of 0.1497; the same model scored 0.2925 in zero-shot generation mode, revealing a 2x inflation from unconstrained scoring
- Temperature increases from 0 to 1.0 raised top-10 Jaccard distance from 0.0900 to 0.1240 for the best proprietary reranker with negligible NDCG impact, while weaker LLMs showed substantially larger degradation
- Collaborative-filtering candidates outperformed semantic candidates by over 50% in NDCG@10 for both proprietary and open-weight rerankers
Industry Insight
- Researchers and practitioners should treat candidate generation, pool size, scoring policy, and decoding configuration as first-class reporting requirements rather than implementation footnotes; without these, benchmark comparisons are unreliable
- The finding that open-weight LLMs failed to beat a shallow autoencoder suggests that for recommendation reranking, architectural simplicity and task-specific tuning may still outperform general-purpose LLM capability
- Teams should prioritize investing in diverse, high-quality candidate generation (e.g., hybrid semantic + collaborative approaches) over selecting increasingly capable rerankers, as the former drives significantly larger performance gains
Disclaimer: The above content is generated by AI and is for reference only.