Spaces:
Running
Running
Create Live_Challenge_Day_and_Dry_Test_Instructions.md
Browse files
Operational_Instructions/Live_Challenge_Day_and_Dry_Test_Instructions.md
ADDED
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
**Live Challenge Day (May 12)**
|
2 |
+
|
3 |
+
1. We will conduct two live sessions on May 12 as part of the Live Challenge Day (calendar invites will be sent shortly):
|
4 |
+
* Session 1: 7:00 β 9:00 UTC
|
5 |
+
* Session 2: 15:00 β 17:00 UTC
|
6 |
+
|
7 |
+
2. Just before your session start time, your team leader will receive an email containing the Question file (500 questions) β see Question file [json schema](Question_File.json.schema) and [example](Question_File_Example.json)
|
8 |
+
|
9 |
+
3. You must generate and submit your Answer file β see Answer file [json schema](Answe_File.json.schema) and [example](Answer_File_Example.json) β within 2 hours from your session start time\
|
10 |
+
3.1 **Remark:** Details regarding the exact submission process will be provided soon
|
11 |
+
|
12 |
+
4. Please refer to the LiveRAG Challenge [Evaluation Guidelines](LiveRAG_Evaluation_Guidelines.md) for important information about the evaluation process
|
13 |
+
|
14 |
+
5. You must share your RAG system Git repository with us by May 13 AoE via email at sigir2025-liverag-gen@tii.ae to enable result reproduction
|
15 |
+
|
16 |
+
6. Additional information:\
|
17 |
+
6.1 The automatic evaluation results leaderboard will be published on the HuggingFace Challenge page once the evaluation process is completed
|
18 |
+
6.2 The top-performing teams will undergo manual evaluation to determine the final winners, who will be announced on July 17 during the SIGIR 2025 LiveRAG Workshop
|
19 |
+
|
20 |
+
**Dry Test (May 5)**
|
21 |
+
|
22 |
+
1. To ensure a smooth Live Challenge Day, we will conduct a Dry Test on May 5 (calendar invites will be sent shortly)
|
23 |
+
|
24 |
+
2. Each Dry Test Question file will contain 50 questions
|
25 |
+
|
26 |
+
3. Your answers will not be evaluated, and no leaderboard will be published for the Dry Test. Otherwise, instructions 1β4 from the Live Challenge Day also apply
|