Challenge Overview

Retrieval Augmented Generation (RAG) has emerged as a key technology to mitigate the issues that Large Language Models (LLMs) face when they lack adequate knowledge. Given a user's request, a RAG system searches auxiliary sources to augment the prompt associated with the request with relevant content. RAG is attracting a great deal of attention from the AI community, yet it is still hard to assess the quality of RAG systems in a systematic manner.

The goal of the SIGIR'2025 LiveRAG Challenge (organized by the Technology Innovation Institute, with support from AI71, AWS and Pinecone and which took place over March-May 2025) was to allow research teams across academia and industry to advance their RAG research and compare the performance of their solutions with other teams, on a fixed corpus (derived from the publicly available FineWeb) and a fixed open-source LLM, Falcon3-10B-Instruct.

After an application process, 40 selected teams were awarded up to 1500 USD in AWS compute credits to train their RAG solution, and up to 750 USD in Pinecone compute credits to use/generate their RAG indices. They were also given early access to TII's DataMorgana tool to help them generate synthetic benchmarks for training and testing.

During the Live Challenge Day, on May 12, 2025, the teams were provided with a stream of unseen questions. twenty-five teams returned valid answers under the two-hour time limit. Results and finalists are listed below.



Challenge Results

We are delighted to list below (sorted by team's name alphabetical order) the finalists of the SIGIR'2025 LiveRAG Challenge.

The prize winners will be announced at the LiveRAG Workshop in Padua, Italy on July 17, 2025.
Note that prizes will be awarded only to teams that have at least one team member registered to and present at the workshop.

Team Number Team Name Team members Institution
2636 Magikarp Tong Zhou Institute of Automation Chinese Academy of Sciences, China
2614 RAGtifier William Xion, Hailay Teklehaymanot, Oleh Astappiev, Tim Cofala L3S Research Center, Leibniz University Hannover, Germany
2615 RMIT-ADMS Oleg Zendel, Kun Ran, Shuoqi Sun, Dinh Anh Khoi Nguyen, Damiano Spina RMIT, Australia
2596 UDInfo Damian Martinez, Catalina Riano, Hui Fang University of Delaware, USA

The finalists were identified after a thorough validation and assessment of the teams' artifacts. This included the Correctness and Faithfulness scores, as computed by DataMorgana, following the official evaluation guidelines, manual examination of results by annotators, PC members reviews of the teams' reports, and code repositories.

The leaderboards (one for each session), with the Correctness and Faithfulness scores, are given below, with rows sorted by Correctness.



Session 1 - May 12, 07:00 - 09:00 UTC

Rank Team Number Team Name Team Members Institution Correctness [-1:2] Faithfulness [-1:1]
1 2615 RMIT-ADMS Oleg Zendel, Kun Ran, Shuoqi Sun, Dinh Anh Khoi Nguyen, Damiano Spina RMIT, Australia 1.199317 0.477382
2 2587 RUC_DeepSearch Guanting Dong, Xiaoxi Li, Yuyao Zhang, Mengjie Deng, Yutao Zhu Renmin University of China, China 0.969273 0.387808
3 2620 Ped100X Saksorn Ruangtanusak, Natthapath Rungseesiripak, Peerawat Rojratchadakorn, Monthol Charattrakool, Natapong Nitarach SCBX, Thailand 0.928893 0.043381
4 2677 PRMAS-DRCA Priyanshu Raj Mall, Aman Sinha, Dwaipayan Roy Indian Institute of Science Education and Research, Kolkata, India 0.922780 0.410600
5 2668 Hybrid Search with Graph Junjie Huang, Guo Chen, Maolin Zheng, Sha Hu, Tao Jia College of Computer and Information Science, Southwest University, China 0.875091 0.315802
6 2617 BagBag Shuailong Sang, Yourui Ye, Shimao Chu, Kun Zhang Hefei University of Technology, China 0.694073 -0.911353
7 2669 UniClustRAG Juli Bakagianni, John Pavlopoulos, Aristidis Likas Athens University of Economics and Business, Greece 0.685146 0.460062
8 2624 METURAG Tizian Peer, Seymanur Ozen, Tugba Taskaya Temizel 0.673451 0.325339
9 2643 DeepRAG Djellel Difallah, Prince Larbi Ampofo, Ola El Khatib, Filip Mislov New York University, United Arab Emirates 0.566053 0.097828
10 2635 UiS-IAI Weronika Łajewska, Ivica Kostric, Gabriel Iturra-Bocaz, Mariam Arustashvili, Krisztian Balog University of Stavanger, Norway 0.552328 0.433697
11 2665 SNU-LDILab Soyoung Yoon, Minseong Hwang, Jongyoon Kim, Dohyeon Lee, Seung-won Hwang Interdisciplinary Program in Artificial Intelligence (IPAI), Seoul National University, South Korea 0.517367 0.103027
12 2586 Gravitational Lens Yifei Wang, Boyu Ren, Pengqian Han, Shuangyan Deng, Jiamou Liu School of Computer Science, The University of Auckland, New Zealand 0.376637 -0.988097


Session 2 - May 12, 15:00 - 17:00 UTC

Rank Team Number Team Name Team Members Institution Correctness [-1:2] Faithfulness [-1:1]
1 2636 Magikarp Tong Zhou Institute of Automation Chinese Academy of Sciences, China 1.231578 0.656464
2 2596 UDInfo Damian Martinez, Catalina Riano, Hui Fang University of Delaware, USA 1.200586 0.623175
3 2614 RAGtifier William Xion, Hailay Teklehaymanot, Oleh Astappiev, Tim Cofala L3S Research Center, Leibniz University Hannover, Germany 1.134454 0.552365
4 2626 HLTCOE Eugene Yang, Andrew Yates, Orion Weller, Kevin Duh, Dawn Lawrie Johns Hopkins University, USA 1.070111 0.340711
5 2591 Ragmatazz Matthias Krüger, David Fisher, Scott Stults OpenSource Connections, Germany 1.011956 0.519394
6 2611 ScaledRAG Alireza Salemi, Mukta Maddipatla, Hamed Zamani University of Massachusetts Amherst, USA 0.996348 0.418273
7 2664 Emorag Chase Fensore, Kaustubh Dhole, Joyce Ho, Eugene Agichtein Emory University, USA 0.890718 0.556581
8 2671 Graph-Enhanced RAG Zhili Shen, Chenxin Diao, Pascual Merita, Pavlos Vougiouklis, Jeff Pan Huawei Technologies, United Kingdom 0.875714 0.529335
9 2650 Multi-Agent Adaptive RAG Ines Besrour, Jingbo He, Tobias Schreieder, Michael Färber TU Dresden, Germany 0.836110 0.200420
10 2660 Starlight To Eun Kim, Fernando Diaz Carnegie Mellon University, USA 0.818337 0.433003
11 2648 NoobRAG A F M Mohimenul Joaa, Ramy Boulos, Himanshu Manoj Kaloni, Michael Färber TU Dresden, Germany 0.655292 0.154648
12 2580 UIUC-RAGents Eric Modesitt, Ke Yang, Chengxiang Zhai University of Illinois at Urbana Champaign, USA 0.565043 -0.302616
13 2652 AugmentRAG-TUD Alisamar Husain, Hardik Ghoshal, Mario Tawfelis TU Dresden, Germany 0.532533 0.655634




Challenge Calendar

Date (2025) Details
Mar 3 Feb 24 Application submission deadline - SIGIR2025 easychair site (Select: SIGIR2025 LiveRAG Challenge track)
Mar 12
  • Application submission notification to selected teams
  • Opening of easychair site for short paper submission
  • AWS and Pinecone resources and credits made available to selected teams together with detailed operational instructions
Mar 20 Mar 15 Training and testing tool (DataMorgana) made available to teams
May 5 May 8 "Dry" test for participants of live service on a small question set
May 12 Live Challenge Day hosted on Hugging Face competition platform – test questions shared and live service for answers submission opens
May 23 May 19 Short paper submission deadline - SIGIR2025 easychair site (Select: SIGIR2025 LiveRAG Challenge track)
June 12 May 29 Short paper notification and announcement of finalists
July 17
  • LiveRAG Workshop at SIGIR'2025 in Padua, Italy
  • Presentation of research by selected teams
  • Announcement of winner and runner(s)-up

Remark: Registration and attendance at the workshop by at least one author/team-member is required to be considered for prizes


All details about the Challenge including the challenge overview and details, the application process, eligibility, submission instructions, challenge guidelines and more, are available on the Challenge Web site



Prizes

  • First Prize: $5000
  • Second Prize: $3000
  • Third Prize: $2000


Organization

PC Members

  • Charles L. A. Clarke, University of Waterloo
  • Yi Chang, Jilin University
  • Ido Guy, Meta
  • Oren Kurland, Technion, Israel Institute of Technology
  • Yiqun Liu, Tsinghua University
  • Antonio Mallia, Pinecone
  • Marc Najork, Google DeepMind
  • Fabrizio Silvestri, Sapienza Università di Roma
  • Ian Soboroff, NIST
  • Emine Yilmaz, University College London and Amazon
  • Elad Yom-Tov, Bar-Ilan University

Organizing Team

David Carmel1, Simone Filice1, Mehdi Ghissassi2, Hakim Hacid1, Guy Horowitz1, Zohar Karnin1, Liane Lewin-Eytan1, Yoelle Maarek1, Ran Tavory1, and Oren Somekh1

  • 1Technology Innovation Institute
  • 2AI71



By applying to the Challenge, each team agreed to the Challenge Terms and Conditions and committed to strictly adhering to the Challenge Guidelines.



Contact:

For any question about the challenge, please send mail to sigir2025-liverag-gen@tii.ae.