Spaces:

IqraEval
/

SharedTask_ArabicNLP2025

Running

App Files Files Community

01Yassine commited on 10 days ago

Commit

290e06d

verified ·

1 Parent(s): cb75d3d

Update index.html

Browse files

Files changed (1) hide show

index.html +211 -178

index.html CHANGED Viewed

@@ -1,173 +1,127 @@
 <!doctype html>
 <html lang="en">
 <head>
-<meta charset="utf-8" />
-<meta name="viewport" content="width=device-width" />
-<title>Iqra’Eval Shared Task</title>
-<style>
-  /* Color Palette */
-  :root {
-    --navy-blue: #001f4d;
-    --coral: #ff6f61;
-    --light-gray: #f5f7fa;
-    --text-dark: #222;
-  }
-  body {
-    font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
-    background-color: var(--light-gray);
-    color: var(--text-dark);
-    margin: 20px;
-    line-height: 1.6;
-  }
-  h1, h2, h3 {
-    color: var(--navy-blue);
-    font-weight: 700;
-    margin-top: 1.2em;
-  }
-  h1 {
-    text-align: center;
-    font-size: 2.8rem;
-    margin-bottom: 0.3em;
-  }
-  h2 {
-    border-bottom: 3px solid var(--coral);
-    padding-bottom: 0.3em;
-  }
-  h3 {
-    color: var(--coral);
-    margin-top: 1em;
-  }
-  p {
-    max-width: 900px;
-    margin: 0.8em auto;
-  }
-  strong {
-    color: var(--navy-blue);
-  }
-  ul {
-    max-width: 900px;
-    margin: 0.5em auto 1.5em auto;
-    padding-left: 1.2em;
-  }
-  ul li {
-    margin: 0.4em 0;
-  }
-  code {
-    background-color: #eef4f8;
-    color: var(--navy-blue);
-    padding: 2px 6px;
-    border-radius: 4px;
-    font-family: Consolas, monospace;
-    font-size: 0.9em;
-  }
-  pre {
-    max-width: 900px;
-    background-color: #eef4f8;
-    color: var(--navy-blue);
-    padding: 1em;
-    border-radius: 8px;
-    overflow-x: auto;
-    font-family: Consolas, monospace;
-    font-size: 0.95em;
-    margin: 0.8em auto;
-  }
-  a {
-    color: var(--coral);
-    text-decoration: none;
-  }
-  a:hover {
-    text-decoration: underline;
-  }
-  .card {
-    max-width: 1200px;
-    background: white;
-    margin: 0 auto 40px auto;
-    padding: 2em 2.5em;
-    box-shadow: 0 4px 14px rgba(0,0,0,0.1);
-    border-radius: 12px;
-  }
-  /* Centering images and captions */
-  div img {
-    display: block;
-    margin: 20px auto;
-    max-width: 100%;
-    height: auto;
-    border-radius: 8px;
-    box-shadow: 0 4px 8px rgba(0,31,77,0.15);
-  }
-  .centered p {
-    text-align: center;
-    font-style: italic;
-    color: var(--navy-blue);
-    margin-top: 0.4em;
-  }
-  .highlight {
-    color: var(--coral);
-    font-weight: 700;
-  }
-  /* Lists inside paragraphs */
-  p > ul {
-    margin-top: 0.3em;
-  }
-</style>
 </head>
 <body>
-<div class="card">
     <h1>Iqra’Eval Shared Task</h1>
-    <div>
-      <img src="IqraEval.png" alt="IqraEval Logo" />
-    </div>
-    <!-- Overview Section -->
     <h2>Overview</h2>
     <p>
-        <strong>Iqra'Eval</strong> is a shared task aimed at advancing <strong>automatic assessment of Qur’anic recitation pronunciation</strong> by leveraging computational methods to detect and diagnose pronunciation errors. The focus on Qur’anic recitation provides a standardized and well-defined context for evaluating Modern Standard Arabic (MSA) pronunciation.
     </p>
     <p>
-        Participants will develop systems capable of detecting mispronunciations (e.g., substitution, deletion, or insertion of phonemes).
     </p>
-    <!-- Timeline Section -->
     <h2>Timeline</h2>
     <ul>
-        <li><strong>June 1, 2025</strong>: Official announcement of the shared task</li>
-        <li><strong>June 10, 2025</strong>: Release of training data, development set (QuranMB), phonetizer script, and baseline systems</li>
-        <li><strong>July 24, 2025</strong>: Registration deadline and release of test data</li>
-        <li><strong>July 27, 2025</strong>: End of evaluation cycle (test set submission closes)</li>
-        <li><strong>July 30, 2025</strong>: Final results released</li>
-        <li><strong>August 15, 2025</strong>: System description paper submissions due</li>
-        <li><strong>August 22, 2025</strong>: Notification of acceptance</li>
-        <li><strong>September 5, 2025</strong>: Camera-ready versions due</li>
     </ul>
-    <!-- Task Description -->
     <h2>Task Description: Quranic Mispronunciation Detection System</h2>
     <p>
-      The aim is to design a model to detect and provide detailed feedback on mispronunciations in Quranic recitations.
-      Users read aloud vowelized Quranic verses; this model predicts the phoneme sequence uttered by the speaker, which may contain mispronunciations.
-      Models are evaluated on the <strong>QuranMB.v2</strong> dataset, which contains human‐annotated mispronunciations.
     </p>
     <div class="centered">
       <img src="task.png" alt="System Overview" />
       <p>Figure: Overview of the Mispronunciation Detection Workflow</p>
@@ -175,7 +129,7 @@
     <h3>1. Read the Verse</h3>
     <p>
-      The user is shown a <strong>Reference Verse</strong> (What should have been said) in Arabic script along with its corresponding <strong>Reference Phoneme Sequence</strong>.
     </p>
     <p><strong>Example:</strong></p>
     <ul>
@@ -188,64 +142,143 @@
     <h3>2. Save Recording</h3>
     <p>
-      The user recites the verse aloud; the system captures and stores the audio waveform for subsequent analysis.
     </p>
     <h3>3. Mispronunciation Detection</h3>
     <p>
-      The stored audio is fed into a <strong>Mispronunciation Detection Model</strong>.
-      This model predicts the phoneme sequence uttered by the speaker, which may contain mispronunciations.
     </p>
     <p><strong>Example of Mispronunciation:</strong></p>
     <ul>
-      <li><strong>Reference Phoneme Sequence (What should have been said):</strong> <code>&lt; i n n a SS A f aa w a l m a r w a t a m i n $ a E a a &lt; i r i l l a h i</code></li>
-      <li><strong>Model Phoneme Prediction (What is predicted):</strong> <code>&lt; i n n a SS A f aa w a l m a r w a t a m i n s a E a a &lt; i r u l l a h i</code></li>
       <li>
-        <strong>Annotated Phoneme Sequence (What is said):</strong>
         <code>&lt; i n n a SS A f aa w a l m a r w <span class="highlight">s</span> a E a a &lt; i <span class="highlight">r u</span> l l a h i</code>
       </li>
     </ul>
     <p>
-      In this case, the phoneme <code>$</code> was mispronounced as <code>s</code>, and <code>i</code> was mispronounced as <code>u</code>.
     </p>
     <p>
-      The annotated phoneme sequence indicates that the phoneme <code>ta</code> was omitted, but the model failed to detect it.
     </p>
-    <h2>Training Dataset: Description</h2>
     <p>
-        All data are hosted on Hugging Face. Two main splits are provided:
     </p>
     <ul>
-        <li>
-            <strong>Training set:</strong> 79 hours of Modern Standard Arabic (MSA) Quran recitations (5,167 audio files)
-        </li>
-        <li>
-            <strong>Evaluation set:</strong> QuranMB.v2 dataset with phoneme-level mispronunciation annotations, which includes:
-            <ul>
-                <li>QuranMB-Train: 9 hours (1,218 files) for development</li>
-                <li>QuranMB-Test: 8 hours (1,018 files) for evaluation</li>
-            </ul>
-        </li>
     </ul>
-    <h2>Submission Guidelines</h2>
     <p>
-      Participants should submit their predicted phoneme sequences on the test set by the deadline (July 27, 2025). Submissions will be automatically evaluated using the official scoring scripts.
     </p>
-    <h2>Evaluation Metrics</h2>
     <p>
-      Systems will be evaluated based on phoneme error rates (PER) computed over the test set, measuring accuracy in detecting and localizing mispronunciations.
     </p>
-    <h2>Contact and Support</h2>
     <p>
-      For inquiries and support, reach out to the task coordinators at
-      <a href="mailto:support@iqraeval.org">support@iqraeval.org</a>.
     </p>
-</div>
 </body>
 </html>

 <!doctype html>
 <html lang="en">
 <head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width" />
+  <title>Iqra’Eval Shared Task</title>
+  <style>
+    :root {
+      --navy-blue: #001f4d;
+      --coral: #ff6f61;
+      --light-gray: #f5f7fa;
+      --text-dark: #222;
+    }
+    body {
+      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
+      background-color: var(--light-gray);
+      color: var(--text-dark);
+      margin: 20px;
+      line-height: 1.6;
+    }
+    h1, h2, h3 {
+      color: var(--navy-blue);
+      font-weight: 700;
+      margin-top: 1.2em;
+    }
+    h1 {
+      text-align: center;
+      font-size: 2.8rem;
+      margin-bottom: 0.3em;
+    }
+    h2 {
+      border-bottom: 3px solid var(--coral);
+      padding-bottom: 0.3em;
+    }
+    h3 {
+      color: var(--coral);
+      margin-top: 1em;
+    }
+    p, ul, pre {
+      max-width: 900px;
+      margin: 0.8em auto;
+    }
+    ul { padding-left: 1.2em; }
+    ul li { margin: 0.4em 0; }
+    code {
+      background-color: #eef4f8;
+      color: var(--navy-blue);
+      padding: 2px 6px;
+      border-radius: 4px;
+      font-family: Consolas, monospace;
+      font-size: 0.9em;
+    }
+    pre {
+      background-color: #eef4f8;
+      padding: 1em;
+      border-radius: 8px;
+      overflow-x: auto;
+      font-size: 0.95em;
+    }
+    a {
+      color: var(--coral);
+      text-decoration: none;
+    }
+    a:hover { text-decoration: underline; }
+    .card {
+      max-width: 960px;
+      background: white;
+      margin: 0 auto 40px;
+      padding: 2em 2.5em;
+      box-shadow: 0 4px 14px rgba(0,0,0,0.1);
+      border-radius: 12px;
+    }
+    img {
+      display: block;
+      margin: 20px auto;
+      max-width: 100%;
+      height: auto;
+      border-radius: 8px;
+      box-shadow: 0 4px 8px rgba(0,31,77,0.15);
+    }
+    .centered p {
+      text-align: center;
+      font-style: italic;
+      color: var(--navy-blue);
+      margin-top: 0.4em;
+    }
+    .highlight {
+      color: var(--coral);
+      font-weight: 700;
+    }
+    /* nested lists in paragraphs */
+    p > ul { margin-top: 0.3em; }
+  </style>
 </head>
 <body>
+  <div class="card">
     <h1>Iqra’Eval Shared Task</h1>
+    <img src="IqraEval.png" alt="IqraEval Logo" />
     <h2>Overview</h2>
     <p>
+      <strong>Iqra'Eval</strong> is a shared task aimed at advancing <strong>automatic assessment of Qur’anic recitation pronunciation</strong> by leveraging computational methods to detect and diagnose pronunciation errors. The focus on Qur’anic recitation provides a standardized and well-defined context for evaluating Modern Standard Arabic (MSA) pronunciation.
     </p>
     <p>
+      Participants will develop systems capable of detecting mispronunciations (e.g., substitution, deletion, or insertion of phonemes).
     </p>
     <h2>Timeline</h2>
     <ul>
+      <li><strong>June 1, 2025</strong>: Official announcement</li>
+      <li><strong>June 10, 2025</strong>: Release of training data, dev set, phonetizer, baselines</li>
+      <li><strong>July 24, 2025</strong>: Registration deadline & test data release</li>
+      <li><strong>July 27, 2025</strong>: Test set submission closes</li>
+      <li><strong>July 30, 2025</strong>: Final results released</li>
+      <li><strong>August 15, 2025</strong>: System description papers due</li>
+      <li><strong>August 22, 2025</strong>: Notification of acceptance</li>
+      <li><strong>September 5, 2025</strong>: Camera-ready versions due</li>
     </ul>
     <h2>Task Description: Quranic Mispronunciation Detection System</h2>
     <p>
+      Design a model to detect and provide detailed feedback on mispronunciations in Quranic recitations. Users read vowelized verses; the model predicts the spoken phoneme sequence and flags deviations. Evaluation is on the <strong>QuranMB.v2</strong> dataset with human‐annotated errors.
     </p>
     <div class="centered">
       <img src="task.png" alt="System Overview" />
       <p>Figure: Overview of the Mispronunciation Detection Workflow</p>
     <h3>1. Read the Verse</h3>
     <p>
+      System shows a <strong>Reference Verse</strong> plus its <strong>Reference Phoneme Sequence</strong>.
     </p>
     <p><strong>Example:</strong></p>
     <ul>
     <h3>2. Save Recording</h3>
     <p>
+      User recites; system captures and stores the audio waveform.
     </p>
     <h3>3. Mispronunciation Detection</h3>
     <p>
+      Model predicts the phoneme sequence—deviations from reference indicate mispronunciations.
     </p>
     <p><strong>Example of Mispronunciation:</strong></p>
     <ul>
+      <li><strong>Reference:</strong> <code>&lt; i n n a SS A f aa w a l m a r w a t a m i n $ a E a a &lt; i r i l l a h i</code></li>
+      <li><strong>Predicted:</strong> <code>&lt; i n n a SS A f aa w a l m a r w a t a m i n s a E a a &lt; i r u l l a h i</code></li>
       <li>
+        <strong>Annotated:</strong>
         <code>&lt; i n n a SS A f aa w a l m a r w <span class="highlight">s</span> a E a a &lt; i <span class="highlight">r u</span> l l a h i</code>
       </li>
     </ul>
     <p>
+      Here, <code>$</code>→<code>s</code> and <code>i</code>→<code>u</code>; omission of <code>ta</code> went undetected.
+    </p>
+    <h2>Training Dataset: Description</h2>
+    <p>
+      Hosted on Hugging Face:
+    </p>
+    <ul>
+      <li>
+        <strong>Training:</strong> 79 h of MSA speech (Qur’anic recitations)
+        <code>load_dataset("IqraEval/Iqra_train", split="train")</code>
+      </li>
+      <li>
+        <strong>Development:</strong> 3.4 h for tuning
+        <code>load_dataset("IqraEval/Iqra_train", split="dev")</code>
+      </li>
+    </ul>
+    <p>
+      <strong>Columns:</strong>
+      <ul>
+        <li><code>audio</code>: waveform</li>
+        <li><code>sentence</code>: original text</li>
+        <li><code>index</code>: verse ID or –1</li>
+        <li><code>tashkeel_sentence</code>: fully diacritized</li>
+        <li><code>phoneme</code>: Nawar Halabi phonetizer output</li>
+      </ul>
     </p>
+    <h2>Training Dataset: TTS Data (Optional)</h2>
     <p>
+      Auxiliary high-quality TTS corpus for augmentation:
+      <code>load_dataset("IqraEval/Iqra_TTS")</code>
     </p>
+    <h2>Test Dataset: QuranMB_v2</h2>
     <p>
+      98 verses × 18 speakers ≈ 2 h, with deliberate errors and human annotations.
+      <code>load_dataset("IqraEval/Iqra_QuranMB_v2")</code>
     </p>
+    <h2>Resources & Links</h2>
     <ul>
+      <li><a href="https://github.com/Iqra-Eval/MSA_phonetiser" target="_blank">Phonetiser script (GitHub)</a></li>
+      <li><a href="https://huggingface.co/datasets/IqraEval/Iqra_train" target="_blank">Training & Dev Data (Hugging Face)</a></li>
+      <li><a href="https://huggingface.co/datasets/IqraEval/Iqra_TTS" target="_blank">TTS Data (Hugging Face)</a></li>
+      <li><a href="https://github.com/Iqra-Eval/interspeech_IqraEval" target="_blank">Baseline Systems & Scripts (GitHub)</a></li>
     </ul>
+    <p><em>See the main <a href="https://github.com/Iqra-Eval" target="_blank">GitHub</a> for full instructions.</em></p>
+    <h2>Submission Details (Draft)</h2>
+    <p>
+      Submit a UTF-8 CSV named <code>teamID_submission.csv</code> with two columns:
+    </p>
+    <ul>
+      <li><strong>ID:</strong> audio filename (no extension)</li>
+      <li><strong>Labels:</strong> predicted phoneme sequence (space-separated)</li>
+    </ul>
+    <pre>
+ID,Labels
+0000_0001, i n n a m a a y a …
+0000_0002, m a a n a n s a …
+…
+    </pre>
     <p>
+      <strong>Note:</strong> no extra spaces, single CSV, no archives.
     </p>
+    <h2>Evaluation Criteria</h2>
+    <p>
+      Leaderboard based on phoneme-level F1-score.
+      We use a hierarchical evaluation (detection + diagnostic) per <a href="https://arxiv.org/pdf/2310.13974" target="_blank">MDD Overview</a>.
+    </p>
+    <ul>
+      <li><em>What is said</em>: annotated phoneme sequence</li>
+      <li><em>What is predicted</em>: model output</li>
+      <li><em>What should have been said</em>: reference sequence</li>
+    </ul>
+    <p>From these we compute:</p>
+    <ul>
+      <li><strong>TA:</strong> correct phonemes accepted</li>
+      <li><strong>TR:</strong> mispronunciations correctly detected</li>
+      <li><strong>FR:</strong> correct phonemes flagged as errors</li>
+      <li><strong>FA:</strong> mispronunciations missed</li>
+    </ul>
+    <p>Rates:</p>
+    <ul>
+      <li><strong>FRR:</strong> FR/(TA+FR)</li>
+      <li><strong>FAR:</strong> FA/(FA+TR)</li>
+      <li><strong>DER:</strong> DE/(CD+DE)</li>
+    </ul>
     <p>
+      Plus standard Precision, Recall, F1 for detection:
+      <ul>
+        <li>Precision = TR/(TR+FR)</li>
+        <li>Recall = TR/(TR+FA)</li>
+        <li>F1 = 2·P·R/(P+R)</li>
+      </ul>
     </p>
+    <h2>Potential Research Directions</h2>
+    <ol>
+      <li><strong>Advanced Models:</strong> fine-tune Wav2Vec2.0, HuBERT on Arabic/Quranic speech.</li>
+      <li><strong>Data Augmentation:</strong> use SpeechBlender to synthesize mispronunciations.</li>
+      <li><strong>Pattern Analysis:</strong> statistical study of QuranMB errors to guide training.</li>
+    </ol>
+    <h2>Future Updates</h2>
     <p>
+      Detailed scoring weights, submission templates, and clarifications will be posted on the shared task site when test data is released (June 5, 2025).
     </p>
+    <h2>References</h2>
+    <ul>
+      <li>El Kheir Y. et al., “SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation,” arXiv:2211.00923, 2022.</li>
+      <li>Al Harere A. & Al Jallad K., “Mispronunciation Detection of Basic Quranic Recitation Rules using Deep Learning,” arXiv:2305.06429, 2023.</li>
+      <li>Aly S. A. et al., “ASMDD: Arabic Speech Mispronunciation Detection Dataset,” arXiv:2111.01136, 2021.</li>
+      <li>Moustafa A. & Aly S. A., “Efficient Voice Identification Using Wav2Vec2.0 and HuBERT…,” arXiv:2111.06331, 2021.</li>
+      <li>El Kheir Y. et al., “Automatic Pronunciation Assessment – A Review,” arXiv:2310.13974, 2021.</li>
+    </ul>
+  </div>
 </body>
 </html>