Iqra’Eval Shared Task

IqraEval Logo

Overview

Iqra'Eval is a shared task aimed at advancing automatic assessment of Qur’anic recitation pronunciation by leveraging computational methods to detect and diagnose pronunciation errors. The focus on Qur’anic recitation provides a standardized and well-defined context for evaluating Modern Standard Arabic (MSA) pronunciation.

Participants will develop systems capable of detecting mispronunciations (e.g., substitution, deletion, or insertion of phonemes).

Timeline

Task Description: Quranic Mispronunciation Detection System

Design a model to detect and provide detailed feedback on mispronunciations in Quranic recitations. Users read vowelized verses; the model predicts the spoken phoneme sequence and flags deviations. Evaluation is on the QuranMB.v2 dataset with human‐annotated errors.

System Overview

Figure: Overview of the Mispronunciation Detection Workflow

1. Read the Verse

System shows a Reference Verse plus its Reference Phoneme Sequence.

Example:

2. Save Recording

User recites; system captures and stores the audio waveform.

3. Mispronunciation Detection

Model predicts the phoneme sequence—deviations from reference indicate mispronunciations.

Example of Mispronunciation:

Here, $→s and i→u; omission of ta went undetected.

Training Dataset: Description

Hosted on Hugging Face:

Columns:

Training Dataset: TTS Data (Optional)

Auxiliary high-quality TTS corpus for augmentation: load_dataset("IqraEval/Iqra_TTS")

Test Dataset: QuranMB.v2

98 verses × 18 speakers ≈ 2 h, with deliberate errors and human annotations. load_dataset("IqraEval/Iqra_QuranMB_v2")

Resources & Links

Submission Details (Draft)

Submit a UTF-8 CSV named teamID_submission.csv with two columns: 

ID,Labels
0000_0001, i n n a m a a y a …
0000_0002, m a a n a n s a …
…

Note: no extra spaces, single CSV, no archives.

Evaluation Criteria

IqraEval Leaderboard is based on phoneme-level F1-score. We use a hierarchical evaluation (detection + diagnostic) per MDD Overview.

From these we compute:

Rates:

Plus standard Precision, Recall, F1 for detection:

Suggested Research Directions

  1. Advanced Mispronunciation Detection Models
    Apply state-of-the-art self-supervised models (e.g., Wav2Vec2.0, HuBERT), using variants that are pre-trained/fine-tuned on Arabic speech. These models can then be fine-tuned on Quranic recitations to improve phoneme-level accuracy.
  2. Data Augmentation Strategies
    Create synthetic mispronunciation examples using pipelines like SpeechBlender. Augmenting limited Arabic/Quranic speech data helps mitigate data scarcity and improves model robustness.
  3. Analysis of Common Mispronunciation Patterns
    Perform statistical analysis on the QuranMB dataset to identify prevalent errors (e.g., substituting similar phonemes, swapping vowels). These insights can drive targeted training and tailored feedback rules.

Registration

Teams and individual participants must register to gain access to the test set. Please complete the registration form using the link below:

Registration Form

Registration opens on June 10, 2025.

Future Updates

Further details on the open-set leaderboard submission will be posted on the shared task website (June 20, 2025). Stay tuned!

Contact and Support

For inquiries and support, reach out to the task coordinators at iqraeval@googlegroups.com.

References