---
title: Semantic Deduplication
emoji: 🧹
colorFrom: green
colorTo: green
sdk: gradio
sdk_version: 5.0.2
app_file: app.py
pinned: false
license: mit
short_description: Deduplicate HuggingFace datasets in seconds
hf_oauth: true
hf_oauth_scopes:
  - write-repo
  - manage-repo
---

# Semantic Text Deduplication Using SemHash

This Gradio application performs **semantic deduplication** on HuggingFace datasets using [SemHash](https://github.com/MinishLab/semhash) with [Model2Vec](https://github.com/MinishLab/model2vec) embeddings.

## Features

- **Two deduplication modes**:
  - **Single dataset**: Find and remove duplicates within one dataset
  - **Cross-dataset**: Remove entries from Dataset 2 that are similar to entries in Dataset 1

- **Customizable similarity threshold**: Control how strict the deduplication should be (0.0 = very loose, 1.0 = exact matches only)

- **Detailed results**: View statistics and examples of found duplicates with word-level differences highlighted

- **Hub Integration**: 🆕 **Push deduplicated datasets directly to the Hugging Face Hub** after logging in

## How to Use

### 1. Choose Deduplication Type
- **Cross-dataset**: Useful for removing training data contamination from test sets
- **Single dataset**: Clean up duplicate entries within a single dataset

### 2. Configure Datasets
- Enter the HuggingFace dataset names (e.g., `SetFit/amazon_massive_scenario_en-US`)
- Specify the dataset splits (e.g., `train`, `test`, `validation`)
- Set the text column name (usually `text`, `sentence`, or `content`)

### 3. Set Similarity Threshold
- **0.9** (default): Good balance between precision and recall
- **Higher values** (0.95-0.99): More conservative, only removes very similar texts
- **Lower values** (0.7-0.85): More aggressive, may remove semantically similar but different texts

### 4. Run Deduplication
Click **"Deduplicate"** to start the process. You'll see:
- Loading progress for datasets
- Deduplication progress
- Results with statistics and example duplicates

### 5. Push to Hub (New!)
After deduplication completes:
1. **Log in** with your Hugging Face account using the login button
2. Enter a **dataset name** for your cleaned dataset
3. Click **"Push to Hub"** to upload the deduplicated dataset

The dataset will be saved as `your-username/dataset-name` and be publicly available.

## Technical Details

- **Embedding Model**: Uses `minishlab/potion-base-8M` (Model2Vec) for fast, efficient text embeddings
- **Deduplication Algorithm**: SemHash for scalable semantic similarity detection
- **Backend**: Runs on CPU (may be slow for large datasets on free tier)

## Local Usage

For faster processing of large datasets, run locally:

```bash
git clone <repository-url>
cd semantic-deduplication
pip install -r requirements.txt
python app.py
```

## Examples

### Cross-dataset Deduplication
Remove test set contamination:
- **Dataset 1**: `your-org/training-data` (split: `train`)
- **Dataset 2**: `your-org/test-data` (split: `test`)
- **Result**: Clean test set with training examples removed

### Single Dataset Cleaning
Remove duplicates from a dataset:
- **Dataset 1**: `common_voice` (split: `train`)
- **Result**: Training set with duplicate audio transcriptions removed

## Notes

- The app preserves all original columns from the datasets
- Only the text similarity is used for deduplication decisions
- Deduplicated datasets maintain the same structure as the original
- OAuth login is required only for pushing to the Hub, not for deduplication