Challenge Overview

Retrieval Augmented Generation (RAG) has emerged as a key technology to mitigate the issues that Large Language Models (LLMs) face when they lack adequate knowledge. Given a user's request, a RAG system searches auxiliary sources to augment the prompt associated with the request with relevant content. RAG is attracting a great deal of attention from the AI community, yet it is still hard to assess the quality of RAG systems in a systematic manner.

The goal of the LiveRAG Challenge is to allow research teams across academia and industry to advance their RAG research and compare the performance of their solutions with other teams, on a fixed corpus (derived from the publicly available FineWeb) and a fixed open-source LLM, Falcon3-10B-Instruct.

The SIGIR'2025 LiveRAG Challenge is organized by TII (Technology Innovation Institute) with support from AI71, AWS, Pinecone, and Hugging Face. It requires an application process, after which selected teams will be

  1. Awarded up to 1500 USD in AWS compute credits to train their RAG solution, and up to 750 USD in Pinecone compute credits to use/generate their RAG indices.
  2. Given early access to TII's DataMorgana tool to help them generate synthetic benchmarks for training and testing.
During the Live Challenge Day, the teams will be provided with a stream of unseen questions and will have to return their answers under strict response-time constraints. Finalists will be requested to present their results at the LiveRAG workshop day to be held at the SIGIR'2025 conference, during which winners will be announced and prizes will be awarded.

Challenge Calendar

Date (2025) Details
Mar 3 Feb 24 Application submission deadline - SIGIR2025 easychair site (Select: SIGIR2025 LiveRAG Challenge track)
Mar 12
  • Application submission notification to selected teams
  • Opening of easychair site for short paper submission
  • AWS and Pinecone resources and credits made available to selected teams together with detailed operational instructions
Mar 20 Mar 15 Training and testing tool (DataMorgana) made available to teams
May 5 May 8 "Dry" test for participants of live service on a small question set
May 12 Live Challenge Day hosted on Hugging Face competition platform – test questions shared and live service for answers submission opens
May 19 Short paper submission deadline - SIGIR2025 easychair site (Select: SIGIR2025 LiveRAG Challenge track)
May 29 Short paper notification and announcement of finalists
July 17
  • LiveRAG Workshop at SIGIR'2025 in Padua, Italy
  • Presentation of research by selected teams
  • Announcement of winner and runner(s)-up


All details about the Challenge including the challenge overview and details, the application process, eligibility, submission instructions, challenge guidelines and more, are available on the Challenge Web site



Prizes

  • First Prize: $5000
  • Second Prize: $3000
  • Third Prize: $2000


If no academic team ranks among the top three, the first-ranking academic team in the top ten teams will be considered for the third prize



Organization

PC Members

  • Charles L. A. Clarke, University of Waterloo
  • Yi Chang, Jilin University
  • Ido Guy, Meta
  • Oren Kurland, Technion, Israel Institute of Technology
  • Yiqun Liu, Tsinghua University
  • Antonio Mallia, Pinecone
  • Marc Najork, Google DeepMind
  • Fabrizio Silvestri, Sapienza Università di Roma
  • Ian Soboroff, NIST
  • Emine Yilmaz, University College London and Amazon
  • Elad Yom-Tov, Bar-Ilan University

Organizing Team

David Carmel1, Simone Filice1, Mehdi Ghissassi2, Hakim Hacid1, Guy Horowitz1, Zohar Karnin1, Liane Lewin-Eytan1, Yoelle Maarek1, Ran Tavory1, and Oren Somekh1

  • 1Technology Innovation Institute
  • 2AI71



By applying to the Challenge, each team agrees to the Challenge Terms and Conditions and commits to strictly adhering to the Challenge Guidelines.



Contact:

For any question about the challenge, please send mail to sigir2025-liverag-gen@tii.ae.