SemEval 2026 Task 8 (MTRAGEval)
Lightweight tri-fusion retrieval and faithful generation for multi-turn RAG
PFW Task 8 at SemEval-2026 Task 8: Lightweight Tri-Fusion Retrieval with Prompt-Engineered Faithful Generation for Multi-Turn RAG
Taleef Tamsal and Jonathan Rusert. SemEval 2026, pp. 1526–1532.
MTRAGEval tests retrieval-augmented generation over multi-turn conversations, where each new question depends on what came before. Our system fuses three retrievers (BM25, SPLADE-v3, and Jina Embeddings v4) with weighted reciprocal rank fusion and prompts GPT-4o or GPT-4o-mini zero-shot to answer from the retrieved passages.
Results
| Task | Metric | Score | Rank |
|---|---|---|---|
| A: retrieval | nDCG@5 | 0.433 | 20 of 38 |
| B: generation from reference passages | H-mean | 0.756 | 6 of 26 |
| C: retrieval plus generation | H-mean | 0.533 | 14 of 29 |
What we learned
- Telling the model exactly how to format citations raised citation use from 4% to 93% on a 100-example development sample. Chain-of-thought prompting alone did not.
- The system handled unanswerable questions almost perfectly (H = 0.990) but did poorly on underspecified ones, where it should have asked a clarifying question and instead answered or refused.