Challenge Overview
Retrieval Augmented Generation (RAG) has emerged as a key technology to mitigate the issues that Large Language Models (LLMs) face when they lack adequate knowledge. Given a user's request, a RAG system searches auxiliary sources to augment the prompt associated with the request with relevant content. RAG is attracting a great deal of attention from the AI community, yet it is still hard to assess the quality of RAG systems in a systematic manner.
The goal of the SIGIR'2025 LiveRAG Challenge (organized by the Technology Innovation Institute, with support from AI71, AWS and Pinecone and which took place over March-May 2025) was to allow research teams across academia and industry to advance their RAG research and compare the performance of their solutions with other teams, on a fixed corpus (derived from the publicly available FineWeb) and a fixed open-source LLM, Falcon3-10B-Instruct.
After an application process, 40 selected teams were awarded up to 1500 USD in AWS compute credits to train their RAG solution, and up to 750 USD in Pinecone compute credits to use/generate their RAG indices. They were also given early access to TII's DataMorgana tool to help them generate synthetic benchmarks for training and testing.
During the Live Challenge Day, on May 12, 2025, the teams were provided with a stream of unseen questions. twenty-five teams returned valid answers under the two-hour time limit. Results and finalists are listed below.
Challenge Results
We are delighted to list below (sorted by team's name alphabetical order) the finalists of the SIGIR'2025 LiveRAG Challenge.
The prize winners will be announced at the LiveRAG Workshop in Padua, Italy on July 17, 2025.
Note that prizes will be awarded only to teams that have at least one team member registered to and present at the workshop.
Team Number | Team Name | Team members | Institution |
---|---|---|---|
2636 | Magikarp | Tong Zhou | Institute of Automation Chinese Academy of Sciences, China |
2614 | RAGtifier | William Xion, Hailay Teklehaymanot, Oleh Astappiev, Tim Cofala | L3S Research Center, Leibniz University Hannover, Germany |
2615 | RMIT-ADMS | Oleg Zendel, Kun Ran, Shuoqi Sun, Dinh Anh Khoi Nguyen, Damiano Spina | RMIT, Australia |
2596 | UDInfo | Damian Martinez, Catalina Riano, Hui Fang | University of Delaware, USA |
The finalists were identified after a thorough validation and assessment of the teams' artifacts. This included the Correctness and Faithfulness scores, as computed by DataMorgana, following the official evaluation guidelines, manual examination of results by annotators, PC members reviews of the teams' reports, and code repositories.
The leaderboards (one for each session), with the Correctness and Faithfulness scores, are given below, with rows sorted by Correctness.
Session 1 - May 12, 07:00 - 09:00 UTC
Rank | Team Number | Team Name | Team Members | Institution | Correctness [-1:2] | Faithfulness [-1:1] |
---|---|---|---|---|---|---|
1 | 2615 | RMIT-ADMS | Oleg Zendel, Kun Ran, Shuoqi Sun, Dinh Anh Khoi Nguyen, Damiano Spina | RMIT, Australia | 1.199317 | 0.477382 |
2 | 2587 | RUC_DeepSearch | Guanting Dong, Xiaoxi Li, Yuyao Zhang, Mengjie Deng, Yutao Zhu | Renmin University of China, China | 0.969273 | 0.387808 |
3 | 2620 | Ped100X | Saksorn Ruangtanusak, Natthapath Rungseesiripak, Peerawat Rojratchadakorn, Monthol Charattrakool, Natapong Nitarach | SCBX, Thailand | 0.928893 | 0.043381 |
4 | 2677 | PRMAS-DRCA | Priyanshu Raj Mall, Aman Sinha, Dwaipayan Roy | Indian Institute of Science Education and Research, Kolkata, India | 0.922780 | 0.410600 |
5 | 2668 | Hybrid Search with Graph | Junjie Huang, Guo Chen, Maolin Zheng, Sha Hu, Tao Jia | College of Computer and Information Science, Southwest University, China | 0.875091 | 0.315802 |
6 | 2617 | BagBag | Shuailong Sang, Yourui Ye, Shimao Chu, Kun Zhang | Hefei University of Technology, China | 0.694073 | -0.911353 |
7 | 2669 | UniClustRAG | Juli Bakagianni, John Pavlopoulos, Aristidis Likas | Athens University of Economics and Business, Greece | 0.685146 | 0.460062 |
8 | 2624 | METURAG | Tizian Peer, Seymanur Ozen, Tugba Taskaya Temizel | 0.673451 | 0.325339 | |
9 | 2643 | DeepRAG | Djellel Difallah, Prince Larbi Ampofo, Ola El Khatib, Filip Mislov | New York University, United Arab Emirates | 0.566053 | 0.097828 |
10 | 2635 | UiS-IAI | Weronika Łajewska, Ivica Kostric, Gabriel Iturra-Bocaz, Mariam Arustashvili, Krisztian Balog | University of Stavanger, Norway | 0.552328 | 0.433697 |
11 | 2665 | SNU-LDILab | Soyoung Yoon, Minseong Hwang, Jongyoon Kim, Dohyeon Lee, Seung-won Hwang | Interdisciplinary Program in Artificial Intelligence (IPAI), Seoul National University, South Korea | 0.517367 | 0.103027 |
12 | 2586 | Gravitational Lens | Yifei Wang, Boyu Ren, Pengqian Han, Shuangyan Deng, Jiamou Liu | School of Computer Science, The University of Auckland, New Zealand | 0.376637 | -0.988097 |
Session 2 - May 12, 15:00 - 17:00 UTC
Rank | Team Number | Team Name | Team Members | Institution | Correctness [-1:2] | Faithfulness [-1:1] |
---|---|---|---|---|---|---|
1 | 2636 | Magikarp | Tong Zhou | Institute of Automation Chinese Academy of Sciences, China | 1.231578 | 0.656464 |
2 | 2596 | UDInfo | Damian Martinez, Catalina Riano, Hui Fang | University of Delaware, USA | 1.200586 | 0.623175 |
3 | 2614 | RAGtifier | William Xion, Hailay Teklehaymanot, Oleh Astappiev, Tim Cofala | L3S Research Center, Leibniz University Hannover, Germany | 1.134454 | 0.552365 |
4 | 2626 | HLTCOE | Eugene Yang, Andrew Yates, Orion Weller, Kevin Duh, Dawn Lawrie | Johns Hopkins University, USA | 1.070111 | 0.340711 |
5 | 2591 | Ragmatazz | Matthias Krüger, David Fisher, Scott Stults | OpenSource Connections, Germany | 1.011956 | 0.519394 |
6 | 2611 | ScaledRAG | Alireza Salemi, Mukta Maddipatla, Hamed Zamani | University of Massachusetts Amherst, USA | 0.996348 | 0.418273 |
7 | 2664 | Emorag | Chase Fensore, Kaustubh Dhole, Joyce Ho, Eugene Agichtein | Emory University, USA | 0.890718 | 0.556581 |
8 | 2671 | Graph-Enhanced RAG | Zhili Shen, Chenxin Diao, Pascual Merita, Pavlos Vougiouklis, Jeff Pan | Huawei Technologies, United Kingdom | 0.875714 | 0.529335 |
9 | 2650 | Multi-Agent Adaptive RAG | Ines Besrour, Jingbo He, Tobias Schreieder, Michael Färber | TU Dresden, Germany | 0.836110 | 0.200420 |
10 | 2660 | Starlight | To Eun Kim, Fernando Diaz | Carnegie Mellon University, USA | 0.818337 | 0.433003 |
11 | 2648 | NoobRAG | A F M Mohimenul Joaa, Ramy Boulos, Himanshu Manoj Kaloni, Michael Färber | TU Dresden, Germany | 0.655292 | 0.154648 |
12 | 2580 | UIUC-RAGents | Eric Modesitt, Ke Yang, Chengxiang Zhai | University of Illinois at Urbana Champaign, USA | 0.565043 | -0.302616 |
13 | 2652 | AugmentRAG-TUD | Alisamar Husain, Hardik Ghoshal, Mario Tawfelis | TU Dresden, Germany | 0.532533 | 0.655634 |
Challenge Calendar
Date (2025) | Details |
---|---|
Mar 3 |
Application submission deadline - SIGIR2025 easychair site (Select: SIGIR2025 LiveRAG Challenge track) |
Mar 12 |
|
Mar 20 |
Training and testing tool (DataMorgana) made available to teams |
May 5 |
"Dry" test for participants of live service on a small question set |
May 12 | Live Challenge Day hosted on Hugging Face competition platform – test questions shared and live service for answers submission opens |
May 23 |
Short paper submission deadline - SIGIR2025 easychair site (Select: SIGIR2025 LiveRAG Challenge track) |
June 12 |
Short paper notification and announcement of finalists |
July 17 |
Remark: Registration and attendance at the workshop by at least one author/team-member is required to be considered for prizes |
All details about the Challenge including the challenge overview and details, the application process, eligibility, submission instructions, challenge guidelines and more, are available on the Challenge Web site
Prizes
- First Prize: $5000
- Second Prize: $3000
- Third Prize: $2000
Organization
PC Members
- Charles L. A. Clarke, University of Waterloo
- Yi Chang, Jilin University
- Ido Guy, Meta
- Oren Kurland, Technion, Israel Institute of Technology
- Yiqun Liu, Tsinghua University
- Antonio Mallia, Pinecone
- Marc Najork, Google DeepMind
- Fabrizio Silvestri, Sapienza Università di Roma
- Ian Soboroff, NIST
- Emine Yilmaz, University College London and Amazon
- Elad Yom-Tov, Bar-Ilan University
Organizing Team
David Carmel1, Simone Filice1, Mehdi Ghissassi2, Hakim Hacid1, Guy Horowitz1, Zohar Karnin1, Liane Lewin-Eytan1, Yoelle Maarek1, Ran Tavory1, and Oren Somekh1
- 1Technology Innovation Institute
- 2AI71
By applying to the Challenge, each team agreed to the Challenge Terms and Conditions and committed to strictly adhering to the Challenge Guidelines.
Contact:
For any question about the challenge, please send mail to sigir2025-liverag-gen@tii.ae.