Update index.html
Browse files- index.html +239 -16
index.html
CHANGED
|
@@ -1,19 +1,242 @@
|
|
| 1 |
<!doctype html>
|
| 2 |
<html>
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
</html>
|
|
|
|
|
|
| 1 |
<!doctype html>
|
| 2 |
<html>
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8" />
|
| 5 |
+
<meta name="viewport" content="width=device-width" />
|
| 6 |
+
<title>Iqra’Eval Shared Task</title>
|
| 7 |
+
<link rel="stylesheet" href="style.css" />
|
| 8 |
+
</head>
|
| 9 |
+
<body>
|
| 10 |
+
<div class="card">
|
| 11 |
+
<h1>Iqra’Eval Shared Task</h1>
|
| 12 |
+
|
| 13 |
+
<!-- Overview Section -->
|
| 14 |
+
<h2>Overview</h2>
|
| 15 |
+
<p>
|
| 16 |
+
<strong>Iqra’Eval</strong> is a shared task aimed at advancing <strong>automatic assessment of Qur’anic recitation pronunciation</strong> by leveraging computational methods to detect and diagnose pronunciation errors. The focus on Qur’anic recitation provides a standardized and well-defined context for evaluating Modern Standard Arabic (MSA) pronunciation, where precise articulation is not only valued but essential for correctness according to established Tajweed rules.
|
| 17 |
+
</p>
|
| 18 |
+
<p>
|
| 19 |
+
Participants will develop systems capable of:
|
| 20 |
+
</p>
|
| 21 |
+
<ul>
|
| 22 |
+
<li>Detecting whether a segment of Qur’anic recitation contains pronunciation errors.</li>
|
| 23 |
+
<li>Diagnosing the nature of the error (e.g., substitution, deletion, or insertion of phonemes).</li>
|
| 24 |
+
</ul>
|
| 25 |
+
|
| 26 |
+
<!-- Timeline Section -->
|
| 27 |
+
<h2>Timeline</h2>
|
| 28 |
+
<ul>
|
| 29 |
+
<li><strong>June 1, 2025</strong>: Official announcement of the shared task</li>
|
| 30 |
+
<li><strong>June 5, 2025</strong>: Release of training data, development set (QuranMB), phonetizer script, and baseline systems</li>
|
| 31 |
+
<li><strong>July 24, 2025</strong>: Registration deadline and release of test data</li>
|
| 32 |
+
<li><strong>July 27, 2025</strong>: End of evaluation cycle (test set submission closes)</li>
|
| 33 |
+
<li><strong>July 30, 2025</strong>: Final results released</li>
|
| 34 |
+
<li><strong>August 15, 2025</strong>: System description paper submissions due</li>
|
| 35 |
+
<li><strong>August 22, 2025</strong>: Notification of acceptance</li>
|
| 36 |
+
<li><strong>September 5, 2025</strong>: Camera-ready versions due</li>
|
| 37 |
+
</ul>
|
| 38 |
+
|
| 39 |
+
<!-- Task Description -->
|
| 40 |
+
<h2>Task Description</h2>
|
| 41 |
+
<p>
|
| 42 |
+
The Iqra’Eval shared task focuses on automatic mispronunciation detection and diagnosis in Qur’anic recitation. Given:
|
| 43 |
+
</p>
|
| 44 |
+
<ol>
|
| 45 |
+
<li>A speech segment (an audio clip of a Qur’anic verse recitation), and</li>
|
| 46 |
+
<li>A fully vowelized reference transcript (the corresponding Qur’anic text, fully diacritized),</li>
|
| 47 |
+
</ol>
|
| 48 |
+
<p>
|
| 49 |
+
the goal is to identify any pronunciation errors, localize them within the phoneme sequence, and classify the type of error based on Tajweed rules.
|
| 50 |
+
</p>
|
| 51 |
+
<p>
|
| 52 |
+
Each participant’s system must predict the sequence of phonemes that the reciter actually produced. A standardized phonemizer (Nawar Halabi’s phonetizer) will be used to generate the “gold” phoneme sequence from the reference transcript for comparison.
|
| 53 |
+
</p>
|
| 54 |
+
<p>
|
| 55 |
+
<strong>Key subtasks:</strong>
|
| 56 |
+
</p>
|
| 57 |
+
<ul>
|
| 58 |
+
<li>Compare predicted phoneme sequence vs. gold reference.</li>
|
| 59 |
+
<li>Detect substitutions (e.g., pronouncing /q/ as /k/), deletions (e.g., dropping a hamza), or insertions (e.g., adding an extra vowel) of phonemes.</li>
|
| 60 |
+
<li>Localize the error to a specific phoneme index in the utterance.</li>
|
| 61 |
+
<li>Classify what type of mistake occurred based on Tajweed (e.g., madd errors, ikhfa, idgham, etc.).</li>
|
| 62 |
+
</ul>
|
| 63 |
+
|
| 64 |
+
<!-- Example & Illustration -->
|
| 65 |
+
<h2>Example</h2>
|
| 66 |
+
<p>
|
| 67 |
+
Suppose the reference verse (fully vowelized) is:
|
| 68 |
+
</p>
|
| 69 |
+
<blockquote>
|
| 70 |
+
<p>
|
| 71 |
+
إِنَّ اللَّهَ عَلَىٰ كُلِّ شَيْءٍ قَدِيرٌ
|
| 72 |
+
<br />
|
| 73 |
+
(inna l-lāha ʿalā kulli shay’in qadīrun)
|
| 74 |
+
</p>
|
| 75 |
+
</blockquote>
|
| 76 |
+
<p>
|
| 77 |
+
The gold phoneme sequence (using the standard phonemizer) might be:
|
| 78 |
+
</p>
|
| 79 |
+
<pre>
|
| 80 |
+
inna l l aa h a ʕ a l a k u l l i ʃ a y ’ i n q a d i r u n
|
| 81 |
+
</pre>
|
| 82 |
+
<p>
|
| 83 |
+
If a reciter mispronounces “قَدِيرٌ” (qadīrun) as “كَدِيرٌ” (kadīrun), that corresponds to a substitution at the very start of that word: phoneme /q/ → /k/.
|
| 84 |
+
</p>
|
| 85 |
+
<p>
|
| 86 |
+
A well-trained system should:
|
| 87 |
+
</p>
|
| 88 |
+
<ol>
|
| 89 |
+
<li>Flag the pronunciation of “قَدِيرٌ” as erroneous,</li>
|
| 90 |
+
<li>Identify that the first phoneme in that word was substituted (“/q/” → “/k/”), and</li>
|
| 91 |
+
<li>Classify it under the Tajweed error category “Ghunnah/Qaf vs. Kaf error.”</li>
|
| 92 |
+
</ol>
|
| 93 |
+
<div style="text-align: center; margin: 1em 0;">
|
| 94 |
+
<img src="images/pronunciation_assessment_arabic.png" alt="Pronunciation Assessment in Arabic" style="max-width: 100%; height: auto;" />
|
| 95 |
+
<p style="font-size: 0.9em; color: #555;">
|
| 96 |
+
<em>Figure: Example of a phoneme-level comparison between reference vs. predicted for an Arabic Qur’anic recitation.</em>
|
| 97 |
+
</p>
|
| 98 |
+
</div>
|
| 99 |
+
|
| 100 |
+
<!-- Evaluation Criteria -->
|
| 101 |
+
<h2>Evaluation Criteria</h2>
|
| 102 |
+
<p>
|
| 103 |
+
Systems will be scored on their ability to detect and correctly classify phoneme-level errors:
|
| 104 |
+
</p>
|
| 105 |
+
<ul>
|
| 106 |
+
<li><strong>Detection accuracy:</strong> Did the system spot that a phoneme-level error occurred in the segment?</li>
|
| 107 |
+
<li><strong>Localization precision:</strong> Did the system mark the correct positions (indices) in the phoneme sequence where the error(s) occurred?</li>
|
| 108 |
+
<li><strong>Classification F1-score:</strong> Given that an error is detected at a particular position, did the system assign the correct error type (e.g., substitution vs. insertion vs. deletion, plus the specific Tajweed subcategory)?</li>
|
| 109 |
+
</ul>
|
| 110 |
+
<p>
|
| 111 |
+
A final <strong>Composite Error Score (CES)</strong> will be computed by combining:
|
| 112 |
+
</p>
|
| 113 |
+
<ol>
|
| 114 |
+
<li>Boundary-aware detection accuracy (punish off-by-one index errors lightly),</li>
|
| 115 |
+
<li>Per-error-type classification F1-score (substitution, deletion, insertion), and</li>
|
| 116 |
+
<li>Overall phoneme-sequence alignment score (Levenshtein-based alignment to reward correct sequences).
|
| 117 |
+
<!-- Note: Detailed weightings will be released along with the test data. -->
|
| 118 |
+
</li>
|
| 119 |
+
</ol>
|
| 120 |
+
<p>
|
| 121 |
+
<em>(Detailed evaluation weights and scripts will be made available on June 5, 2025.)</em>
|
| 122 |
+
</p>
|
| 123 |
+
|
| 124 |
+
<!-- Submission Details -->
|
| 125 |
+
<h2>Submission Details (Draft)</h2>
|
| 126 |
+
<p>
|
| 127 |
+
Participants are required to submit a CSV file named <code>submission.csv</code> containing the predicted phoneme sequences for each audio sample. The file must have exactly two columns:
|
| 128 |
+
</p>
|
| 129 |
+
<ul>
|
| 130 |
+
<li><strong>ID:</strong> Unique identifier of the audio sample.</li>
|
| 131 |
+
<li><strong>Labels:</strong> The predicted phoneme sequence, with each phoneme separated by a single space.</li>
|
| 132 |
+
</ul>
|
| 133 |
+
<p>
|
| 134 |
+
Below is a minimal example illustrating the required format:
|
| 135 |
+
</p>
|
| 136 |
+
<pre>
|
| 137 |
+
ID,Labels
|
| 138 |
+
0000_0001, i n n a m a a y a k h a l l a h a m i n ʕ i b a a d i h u l ʕ u l a m
|
| 139 |
+
0000_0002, m a a n a n s a k h u m i n i ʕ a a y a t i n
|
| 140 |
+
0000_0003, y u k h i k u m u n n u ʔ a u ʔ a m a n a t a n m m i n h u
|
| 141 |
+
…
|
| 142 |
+
</pre>
|
| 143 |
+
<p>
|
| 144 |
+
The first column (ID) should match exactly the audio filenames (without extension). The second column (Labels) is the predicted phoneme string.
|
| 145 |
+
</p>
|
| 146 |
+
<p>
|
| 147 |
+
<strong>Important:</strong>
|
| 148 |
+
<ul>
|
| 149 |
+
<li>Use UTF-8 encoding.</li>
|
| 150 |
+
<li>Do not include extra spaces at the start or end of each line.</li>
|
| 151 |
+
<li>Submit a single CSV file (no archives). Filename must be <code>submission.csv</code>.</li>
|
| 152 |
+
</ul>
|
| 153 |
+
</p>
|
| 154 |
+
|
| 155 |
+
<!-- Dataset Description -->
|
| 156 |
+
<h2>Dataset Description</h2>
|
| 157 |
+
<p>
|
| 158 |
+
All data are hosted on Hugging Face. Two main splits are provided:
|
| 159 |
+
</p>
|
| 160 |
+
<ul>
|
| 161 |
+
<li>
|
| 162 |
+
<strong>Training set:</strong> 79 hours of Modern Standard Arabic (MSA) speech, augmented with multiple Qur’anic recitations.
|
| 163 |
+
<br />
|
| 164 |
+
<code>df = load_dataset("mostafaashahin/IqraEval_Training_Data", split="train")</code>
|
| 165 |
+
</li>
|
| 166 |
+
<li>
|
| 167 |
+
<strong>Development set (QuranMB):</strong> 3.4 hours reserved for tuning and validation.
|
| 168 |
+
<br />
|
| 169 |
+
<code>df = load_dataset("mostafaashahin/IqraEval_Training_Data", split="dev")</code>
|
| 170 |
+
</li>
|
| 171 |
+
</ul>
|
| 172 |
+
<p>
|
| 173 |
+
A sample submission file (<code>sample_submission.csv</code>) is also provided in the repository.
|
| 174 |
+
</p>
|
| 175 |
+
<p>
|
| 176 |
+
<strong>Column Definitions:</strong>
|
| 177 |
+
</p>
|
| 178 |
+
<ul>
|
| 179 |
+
<li><code>sentence</code>: Original sentence text (may be partially diacritized or non-diacritized).</li>
|
| 180 |
+
<li><code>q_index</code>: If from the Quran, the verse index (0–6265, including Basmalah); otherwise <code>-1</code>.</li>
|
| 181 |
+
<li><code>start_word_index</code>, <code>end_word_index</code>: Word positions within the verse (or <code>-1</code> if non-Quranic).</li>
|
| 182 |
+
<li><code>tashkeel_sentence</code>: Fully diacritized sentence (auto-generated via a diacritization tool).</li>
|
| 183 |
+
<li><code>phoneme</code>: Phoneme sequence corresponding to the diacritized sentence (Nawar Halabi phonetizer).</li>
|
| 184 |
+
</ul>
|
| 185 |
+
<p>
|
| 186 |
+
<strong>Data Splits:</strong>
|
| 187 |
+
<br />
|
| 188 |
+
• Training (train): 79 hours total<br />
|
| 189 |
+
• Development (dev): 3.4 hours total
|
| 190 |
+
</p>
|
| 191 |
+
|
| 192 |
+
<!-- Additional TTS Data -->
|
| 193 |
+
<h2>TTS Data (Optional Use)</h2>
|
| 194 |
+
<p>
|
| 195 |
+
We also provide a high-quality TTS corpus for auxiliary experiments (e.g., data augmentation, synthetic pronunciation error simulation). This TTS set can be loaded via:
|
| 196 |
+
</p>
|
| 197 |
+
<ul>
|
| 198 |
+
<li><code>df_tts = load_dataset("IqraEval/Iqra_TTS", split="train")</code></li>
|
| 199 |
+
</ul>
|
| 200 |
+
<p>
|
| 201 |
+
Researchers who wish to experiment with “synthetic mispronunciations” can use the TTS waveform + forced-alignment pipeline to generate various kinds of pronunciation errors in a controlled manner.
|
| 202 |
+
</p>
|
| 203 |
+
|
| 204 |
+
<!-- Resources & Links -->
|
| 205 |
+
<h2>Resources</h2>
|
| 206 |
+
<ul>
|
| 207 |
+
<li>
|
| 208 |
+
<a href="https://huggingface.co/datasets/mostafaashahin/IqraEval_Training_Data" target="_blank">
|
| 209 |
+
Training & Development Data on Hugging Face
|
| 210 |
+
</a>
|
| 211 |
+
</li>
|
| 212 |
+
<li>
|
| 213 |
+
<a href="https://huggingface.co/datasets/IqraEval/Iqra_train" target="_blank">
|
| 214 |
+
IqraEval_Training_Data (alias)
|
| 215 |
+
</a>
|
| 216 |
+
</li>
|
| 217 |
+
<li>
|
| 218 |
+
<a href="https://huggingface.co/datasets/IqraEval/Iqra_TTS" target="_blank">
|
| 219 |
+
IqraEval TTS Data on Hugging Face
|
| 220 |
+
</a>
|
| 221 |
+
</li>
|
| 222 |
+
<li>
|
| 223 |
+
<a href="https://github.com/Iqra-Eval/interspeech_IqraEval" target="_blank">
|
| 224 |
+
Baseline systems & training scripts (GitHub)
|
| 225 |
+
</a>
|
| 226 |
+
</li>
|
| 227 |
+
</ul>
|
| 228 |
+
<p>
|
| 229 |
+
<em>
|
| 230 |
+
For detailed instructions on data access, phonetizer installation, and baseline usage, please refer to the GitHub README.
|
| 231 |
+
</em>
|
| 232 |
+
</p>
|
| 233 |
+
|
| 234 |
+
<!-- Placeholder for Future Details -->
|
| 235 |
+
<h2>Future Updates</h2>
|
| 236 |
+
<p>
|
| 237 |
+
Further details on <strong>evaluation criteria</strong> (exact scoring weights), <strong>submission templates</strong>, and any clarifications will be posted on the shared task website when test data are released (June 5, 2025). Stay tuned!
|
| 238 |
+
</p>
|
| 239 |
+
</div>
|
| 240 |
+
</body>
|
| 241 |
</html>
|
| 242 |
+
|