|
|
<!DOCTYPE html> |
|
|
<html> |
|
|
<head> |
|
|
<meta charset="utf-8"> |
|
|
<meta name="description" content="JQL: Judging Quality across Languages - A pipeline for multilingual data filtering."> |
|
|
<meta name="viewport" content="width=device-width, initial-scale=1"> |
|
|
<title>JQL: Judging Quality across Languages</title> |
|
|
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> |
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css"> |
|
|
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css"> |
|
|
<link rel="stylesheet" |
|
|
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> |
|
|
<style> |
|
|
body { font-family: 'Noto Sans', sans-serif; } |
|
|
.hero.is-primary { background-color: #f9d5e5; } |
|
|
.subtitle img { max-width: 100%; height: auto; } |
|
|
.section-title { margin-top: 2em; } |
|
|
</style> |
|
|
</head> |
|
|
<body> |
|
|
<section class="hero is-primary"> |
|
|
<div class="hero-body"> |
|
|
<div class="container has-text-centered"> |
|
|
<h1 class="title is-1">🦊 JQL: Judging Quality across Languages</h1> |
|
|
<p class="subtitle is-5">Scalable and lightweight multilingual data filtering with LLM-based annotators</p> |
|
|
</div> |
|
|
</div> |
|
|
</section> |
|
|
<section class="hero"> |
|
|
<div class="hero-body"> |
|
|
<div class="container is-max-desktop"> |
|
|
<div class="columns is-centered"> |
|
|
<div class="column has-text-centered"> |
|
|
<h1 class="title is-1 publication-title">Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models</h1> |
|
|
<div class="is-size-5 publication-authors"> |
|
|
<span class="author-block">Mehdi Ali<sup>1,2</sup>†,</span> |
|
|
<span class="author-block">Manuel Brack<sup>3,5</sup>†,</span> |
|
|
<span class="author-block">Max Lübbering<sup>1,2</sup>†,</span> |
|
|
<span class="author-block">Elias Wendt<sup>5</sup>†,</span> |
|
|
<span class="author-block">Abbas Goher Khan<sup>1</sup>†,</span> |
|
|
<span class="author-block">Richard Rutmann<sup>1,2</sup>,</span> |
|
|
<span class="author-block">Alex Jude<sup>2</sup>,</span> |
|
|
<span class="author-block">Maurice Kraus<sup>5</sup>,</span> |
|
|
<span class="author-block">Alexander Arno Weber<sup>1,2</sup>,</span> |
|
|
<span class="author-block">Felix Stollenwerk<sup>6</sup>,</span> |
|
|
<span class="author-block">David Kaczér<sup>1</sup>,</span> |
|
|
<span class="author-block">Florian Mai<sup>1</sup>,</span> |
|
|
<span class="author-block">Lucie Flek<sup>1</sup>,</span> |
|
|
<span class="author-block">Rafet Sifa<sup>1,2</sup>,</span> |
|
|
<span class="author-block">Nicolas Flores-Herr<sup>2</sup>,</span> |
|
|
<span class="author-block">Joachim Köhler<sup>1,2</sup>,</span> |
|
|
<span class="author-block">Patrick Schramowski<sup>3,4,5</sup>,</span> |
|
|
<span class="author-block">Michael Fromm<sup>1,2</sup>,</span> |
|
|
<span class="author-block">Kristian Kersting<sup>3,4,5</sup></span> |
|
|
</div> |
|
|
|
|
|
<div class="is-size-5 publication-authors"> |
|
|
<span class="author-block"><sup>1</sup>Lamarr Institute,</span> |
|
|
<span class="author-block"><sup>2</sup>Fraunhofer IAIS,</span> |
|
|
<span class="author-block"><sup>3</sup>DFKI SAINT,</span> |
|
|
<span class="author-block"><sup>4</sup>Hessian AI,</span> |
|
|
<span class="author-block"><sup>5</sup>Computer Science Department, TU Darmstadt,</span> |
|
|
<span class="author-block"><sup>6</sup>AI Sweden</span> |
|
|
</div> |
|
|
|
|
|
<div class="column has-text-centered"> |
|
|
<span class="link-block"> |
|
|
<a href="https://arxiv.org/abs/2505.22232" target="_blank" |
|
|
class="external-link button is-normal is-rounded is-dark"> |
|
|
<span class="icon"> |
|
|
<i class="ai ai-arxiv"></i> |
|
|
</span> |
|
|
<span>arXiv</span> |
|
|
</a> |
|
|
</span> |
|
|
|
|
|
<span class="link-block"> |
|
|
<a href="https://github.com/JQL-AI/JQL-Annotation-Pipeline/" target="_blank" |
|
|
class="external-link button is-normal is-rounded is-dark"> |
|
|
<span class="icon"> |
|
|
<i class="fab fa-github"></i> |
|
|
</span> |
|
|
<span>Code</span> |
|
|
</a> |
|
|
</span> |
|
|
|
|
|
<span class="link-block"> |
|
|
<a href="https://huggingface.co/datasets/Jackal-AI/JQL-Human-Edu-Annotations" target="_blank" |
|
|
class="external-link button is-normal is-rounded is-dark"> |
|
|
<span class="icon"> |
|
|
<i class="far fa-images"></i> |
|
|
</span> |
|
|
<span>Human Annotations</span> |
|
|
|
|
|
<span class="link-block"> |
|
|
<a href="https://huggingface.co/datasets/Jackal-AI/JQL-LLM-Edu-Annotations" target="_blank" |
|
|
class="external-link button is-normal is-rounded is-dark"> |
|
|
<span class="icon"> |
|
|
<i class="far fa-images"></i> |
|
|
</span> |
|
|
<span>LLM Annotations</span> |
|
|
<span class="link-block"> |
|
|
<a href="https://huggingface.co/Jackal-AI/JQL-Edu-Heads" target="_blank" |
|
|
class="external-link button is-normal is-rounded is-dark"> |
|
|
<span class="icon"> |
|
|
<i class="far fa-images"></i> |
|
|
</span> |
|
|
<span>Lightweight Annotator</span> |
|
|
</a> |
|
|
</div> |
|
|
</div> |
|
|
</div> |
|
|
</div> |
|
|
</div> |
|
|
</div> |
|
|
</section> |
|
|
<section class="section"> |
|
|
<div class="container content"> |
|
|
<p> |
|
|
High-quality multilingual data is crucial for training effective large language models (LLMs). |
|
|
<strong>JQL (Judging Quality across Languages)</strong> is a scalable and lightweight multilingual data filtering approach that distills the judgment capabilities of strong |
|
|
multilingual LLMs into efficient cross-lingual annotators. |
|
|
</p> |
|
|
<p> |
|
|
Overall, JQL improves data quality, retains more tokens, and generalizes to unseen languages. It outperforms heuristic baselines and enables cost-efficient multilingual pretraining data curation at scale. |
|
|
</p> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<section class="section"> |
|
|
<div class="container content"> |
|
|
<h2 class="title is-3">🧩 Main Pipeline Steps</h2> |
|
|
<figure> |
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64bfc4d55ce3d382c05c0f9a/1zPQcwqt9Li_gCvd04_2_.png" alt="JQL Pipeline Overview"> |
|
|
<figcaption><em>Figure 1: Overview of the JQL pipeline</em></figcaption> |
|
|
</figure> |
|
|
|
|
|
<ol> |
|
|
<li><strong>📋 Ground Truth Creation:</strong> Human annotators label monolingual documents based on a structured instruction prompt. These documents are translated into all target languages to create a multilingual gold-standard dataset. (See Figure 1)</li> |
|
|
<li><strong>🤖 LLM-as-a-Judge Selection & Data Annotation:</strong> Strong multilingual LLMs (e.g., Gemma, Mistral, LLaMA) are evaluated against the ground truth, and top-performing models are used to produce synthetic annotations. (See Figure 1)</li> |
|
|
<li><strong>🪶 Lightweight Annotator Training:</strong> Train compact regression heads on frozen multilingual embeddings to create efficient, high-throughput annotators. (See Figure 1)</li> |
|
|
<li><strong>🚀 Scalable Data Filtering:</strong> Use trained annotators to filter large-scale pretraining corpora using quantile thresholds. (See Figure 1)</li> |
|
|
</ol> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<section class="section"> |
|
|
<div class="container content"> |
|
|
<h2 class="title is-3">📊 Results</h2> |
|
|
<ul> |
|
|
<li><strong>✔️ Accuracy:</strong> Good correlation with human ground truth</li> |
|
|
<li><strong>📈 Downstream LLM Training:</strong> |
|
|
<ul> |
|
|
<li>Benchmark performance improvement over FineWeb2</li> |
|
|
<li>Higher document retention vs. FineWeb2 heuristic filter</li> |
|
|
<li>Effective dynamic threshold strategies: Trade-off document quality for quantity</li> |
|
|
</ul> |
|
|
</li> |
|
|
<li><strong>⚡ Annotation Speed:</strong> ~11,000 docs/min (A100 GPU, avg. 690 tokens)</li> |
|
|
</ul> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<section class="section"> |
|
|
<div class="container content"> |
|
|
<h2 class="title is-3">📁 Available Artifacts</h2> |
|
|
<ul> |
|
|
<li><a href="https://huggingface.co/datasets/Jackal-AI/JQL-Human-Edu-Annotations" target="_blank">📄 Ground truth annotations in 35 languages</a></li> |
|
|
<li><a href="https://huggingface.co/datasets/Jackal-AI/JQL-LLM-Edu-Annotations" target="_blank">🧠 Synthetic LLM-annotated dataset (14M+ documents)</a></li> |
|
|
<li><a href="https://huggingface.co/Jackal-AI/JQL-Edu-Heads" target="_blank">🪶 Lightweight annotation models</a>: |
|
|
<ul> |
|
|
<li>JQL-Gemma</li> |
|
|
<li>JQL-Mistral</li> |
|
|
<li>JQL-Llama</li> |
|
|
</ul> |
|
|
</li> |
|
|
<li>🛠️ Training & inference scripts</li> |
|
|
<ul> |
|
|
<li><a href="https://huggingface.co/Jackal-AI/JQL-Edu-Heads" target="_blank">Web Corpus Annotation</a></li> |
|
|
<li>More coming soon</li> |
|
|
</ul> |
|
|
<li>🗄️ Large-scale dataset coming soon</li> |
|
|
</ul> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
<section class="section"> |
|
|
<div class="container content"> |
|
|
<h2 class="title is-3">📜 Citation</h2> |
|
|
<p>If you use JQL, the annotations, or the pretrained annotators, please cite the paper:</p> |
|
|
<pre><code>@article{ali2024jql, |
|
|
title={Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Modelss}, |
|
|
author={Ali, Mehdi and Brack, Manuel and Lübbering, Max and Wendt, Elias and Khan, Abbas Goher and Rutmann, Richard and Jude, Alex and Kraus, Maurice and Weber, Alexander Arno and Stollenwerk, Felix and Kaczér, David and Mai, Florian and Flek, Lucie and Sifa, Rafet and Flores-Herr, Nicolas and Köhler, Joachim and Schramowski, Patrick and Fromm, Michael and Kersting, Kristian}, |
|
|
journal={arXiv preprint arXiv:2505.22232}, |
|
|
year={2025} |
|
|
}</code></pre> |
|
|
</div> |
|
|
</section> |
|
|
|
|
|
</body> |
|
|
</html> |
