diff --git a/.gitattributes b/.gitattributes
index c7d9f3332a950355d5a77d85000f05e6f45435ea..e0446aeb926578756e42600f812e1c4824dce3ad 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -32,3 +32,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
+doc/vsr_1.gif filter=lfs diff=lfs merge=lfs -text
+doc/vsr_2.gif filter=lfs diff=lfs merge=lfs -text
+pipelines/data/noise/babble_noise.wav filter=lfs diff=lfs merge=lfs -text
+pipelines/data/noise/pink_noise.wav filter=lfs diff=lfs merge=lfs -text
+pipelines/data/noise/white_noise.wav filter=lfs diff=lfs merge=lfs -text
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000000000000000000000000000000000000..8e31708daf8c81aa8aa1ed44d4325e049fa9cde3
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,33 @@
+License
+
+Copyright (c) 2020, Pingchuan Ma
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+4. Using our content for individual research and private study.
+
+5. Using our content to build a research prototype to compare with your
+ organisation’s own existing technology. This work must be done separately from
+ your organisation’s commercial product development. The BBC’s content must not
+ be used for training any existing or new technology, algorithms or models that
+ will be sold commercially. And it must not be used to train technology to
+ improve or enhance your operational systems. It can only be used for
+ comparative or benchmarking purposes.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED.
diff --git a/README.md b/README.md
index efdf680ca8443a8c623f0f2b4c8f4c7b355eeee6..9593460cc0d3d9854d644a471ae38110459fcd34 100644
--- a/README.md
+++ b/README.md
@@ -1,12 +1,315 @@
----
-title: Auto Avsr
-emoji: 📈
-colorFrom: purple
-colorTo: pink
-sdk: gradio
-sdk_version: 3.35.1
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+Visual Speech Recognition for Multiple Languages
+
+
+
+[📘Introduction](#Introduction) |
+[🛠️Preparation](#Preparation) |
+[📊Benchmark](#Benchmark-evaluation) |
+[🔮Inference](#Speech-prediction) |
+[🐯Model zoo](#Model-Zoo) |
+[📝License](#License)
+
+
+## Authors
+
+[Pingchuan Ma](https://mpc001.github.io/), [Alexandros Haliassos](https://dblp.org/pid/257/3052.html), [Adriana Fernandez-Lopez](https://scholar.google.com/citations?user=DiVeQHkAAAAJ), [Honglie Chen](https://scholar.google.com/citations?user=HPwdvwEAAAAJ), [Stavros Petridis](https://ibug.doc.ic.ac.uk/people/spetridis), [Maja Pantic](https://ibug.doc.ic.ac.uk/people/mpantic).
+
+## Update
+
+`2023-03-27`: We have released our AutoAVSR models for LRS3, see [here](#autoavsr-models).
+
+## Introduction
+
+This is the repository of [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://arxiv.org/abs/2303.14307) and [Visual Speech Recognition for Multiple Languages](https://arxiv.org/abs/2202.13084), which is the successor of [End-to-End Audio-Visual Speech Recognition with Conformers](https://arxiv.org/abs/2102.06657). By using this repository, you can achieve the performance of 19.1%, 1.0% and 0.9% WER for automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR) on LRS3.
+
+## Tutorial
+
+We provide a tutorial [](https://colab.research.google.com/drive/1jfb6e4xxhXHbmQf-nncdLno1u0b4j614) to show how to use our Auto-AVSR models to perform speech recognition (ASR, VSR, and AV-ASR), crop mouth ROIs or extract visual speech features.
+
+## Demo
+
+English -> Mandarin -> Spanish | French -> Portuguese -> Italian |
+:-------------------------------:|:------------------------------------:
+
| 
|
+
+
+
+[Youtube](https://youtu.be/FIau-6JA9Po) |
+[Bilibili](https://www.bilibili.com/video/BV1Wu411D7oP)
+
+
+## Preparation
+1. Clone the repository and enter it locally:
+
+```Shell
+git clone https://github.com/mpc001/Visual_Speech_Recognition_for_Multiple_Languages
+cd Visual_Speech_Recognition_for_Multiple_Languages
+```
+
+2. Setup the environment.
+```Shell
+conda create -y -n autoavsr python=3.8
+conda activate autoavsr
+```
+
+3. Install pytorch, torchvision, and torchaudio by following instructions [here](https://pytorch.org/get-started/), and install all packages:
+
+```Shell
+pip install -r requirements.txt
+conda install -c conda-forge ffmpeg
+```
+
+4. Download and extract a pre-trained model and/or language model from [model zoo](#Model-Zoo) to:
+
+- `./benchmarks/${dataset}/models`
+
+- `./benchmarks/${dataset}/language_models`
+
+5. [For VSR and AV-ASR] Install [RetinaFace](./tools) or [MediaPipe](https://pypi.org/project/mediapipe/) tracker.
+
+### Benchmark evaluation
+
+```Shell
+python eval.py config_filename=[config_filename] \
+ labels_filename=[labels_filename] \
+ data_dir=[data_dir] \
+ landmarks_dir=[landmarks_dir]
+```
+
+- `[config_filename]` is the model configuration path, located in `./configs`.
+
+- `[labels_filename]` is the labels path, located in `${lipreading_root}/benchmarks/${dataset}/labels`.
+
+- `[data_dir]` and `[landmarks_dir]` are the directories for original dataset and corresponding landmarks.
+
+- `gpu_idx=-1` can be added to switch from `cuda:0` to `cpu`.
+
+### Speech prediction
+
+```Shell
+python infer.py config_filename=[config_filename] data_filename=[data_filename]
+```
+
+- `data_filename` is the path to the audio/video file.
+
+- `detector=mediapipe` can be added to switch from RetinaFace to MediaPipe tracker.
+
+### Mouth ROIs cropping
+
+```Shell
+python crop_mouth.py data_filename=[data_filename] dst_filename=[dst_filename]
+```
+
+- `dst_filename` is the path where the cropped mouth will be saved.
+
+## Model zoo
+
+### Overview
+
+We support a number of datasets for speech recognition:
+- [x] [Lip Reading Sentences 2 (LRS2)](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs2.html)
+- [x] [Lip Reading Sentences 3 (LRS3)](https://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs3.html)
+- [x] [Chinese Mandarin Lip Reading (CMLR)](https://www.vipazoo.cn/CMLR.html)
+- [x] [CMU Multimodal Opinion Sentiment, Emotions and Attributes (CMU-MOSEAS)](http://immortal.multicomp.cs.cmu.edu/cache/multilingual)
+- [x] [GRID](http://spandh.dcs.shef.ac.uk/gridcorpus)
+- [x] [Lombard GRID](http://spandh.dcs.shef.ac.uk/avlombard)
+- [x] [TCD-TIMIT](https://sigmedia.tcd.ie)
+
+### AutoAVSR models
+
+
+
+Lip Reading Sentences 3 (LRS3)
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| - | 19.1 |[GoogleDrive](http://bit.ly/40EAtyX) or [BaiduDrive](https://bit.ly/3ZjbrV5)(key: dqsy) | 891 |
+| **Audio-only** |
+| - | 1.0 |[GoogleDrive](http://bit.ly/3ZSdh0l) or [BaiduDrive](http://bit.ly/3Z1TlGU)(key: dvf2) | 860 |
+| **Audio-visual** |
+| - | 0.9 |[GoogleDrive](http://bit.ly/3yRSXAn) or [BaiduDrive](http://bit.ly/3LAxcMY)(key: sai5) | 1540 |
+| **Language models** |
+| - | - |[GoogleDrive](http://bit.ly/3FE4XsV) or [BaiduDrive](http://bit.ly/3yRI5SY)(key: t9ep) | 191 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/33rEsax) or [BaiduDrive](https://bit.ly/3rwQSph)(key: mi3c) | 18577 |
+
+
+
+### VSR for multiple languages models
+
+
+
+Lip Reading Sentences 2 (LRS2)
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| - | 26.1 |[GoogleDrive](https://bit.ly/3I25zrH) or [BaiduDrive](https://bit.ly/3BAHBkH)(key: 48l1) | 186 |
+| **Language models** |
+| - | - |[GoogleDrive](https://bit.ly/3qzWKit) or [BaiduDrive](https://bit.ly/3KgAL7T)(key: 59u2) | 180 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/3jSMMoz) or [BaiduDrive](https://bit.ly/3BuIwBB)(key: 53rc) | 9358 |
+
+
+
+
+
+
+Lip Reading Sentences 3 (LRS3)
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| - | 32.3 |[GoogleDrive](https://bit.ly/3Bp4gjV) or [BaiduDrive](https://bit.ly/3rIzLCn)(key: 1b1s) | 186 |
+| **Language models** |
+| - | - |[GoogleDrive](https://bit.ly/3qzWKit) or [BaiduDrive](https://bit.ly/3KgAL7T)(key: 59u2) | 180 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/33rEsax) or [BaiduDrive](https://bit.ly/3rwQSph)(key: mi3c) | 18577 |
+
+
+
+
+
+
+
+Chinese Mandarin Lip Reading (CMLR)
+
+
+
+| Components | CER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| - | 8.0 |[GoogleDrive](https://bit.ly/3fR8RkU) or [BaiduDrive](https://bit.ly/3IyACLB)(key: 7eq1) | 195 |
+| **Language models** |
+| - | - |[GoogleDrive](https://bit.ly/3fPxXAJ) or [BaiduDrive](https://bit.ly/3rEcErr)(key: k8iv) | 187 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/3bvetPL) or [BaiduDrive](https://bit.ly/3o2u53d)(key: 1ret) | 3721 |
+
+
+
+
+
+
+CMU Multimodal Opinion Sentiment, Emotions and Attributes (CMU-MOSEAS)
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| Spanish | 44.5 |[GoogleDrive](https://bit.ly/34MjWBW) or [BaiduDrive](https://bit.ly/33rMq3a)(key: m35h) | 186 |
+| Portuguese | 51.4 |[GoogleDrive](https://bit.ly/3HjXCgo) or [BaiduDrive](https://bit.ly/3IqbbMg)(key: wk2h) | 186 |
+| French | 58.6 |[GoogleDrive](https://bit.ly/3Ik6owb) or [BaiduDrive](https://bit.ly/35msiQG)(key: t1hf) | 186 |
+| **Language models** |
+| Spanish | - |[GoogleDrive](https://bit.ly/3rppyJN) or [BaiduDrive](https://bit.ly/3nA3wCN)(key: 0mii) | 180 |
+| Portuguese | - |[GoogleDrive](https://bit.ly/3gPvneF) or [BaiduDrive](https://bit.ly/33vL8Es)(key: l6ag) | 179 |
+| French | - |[GoogleDrive](https://bit.ly/3LDChSn) or [BaiduDrive](https://bit.ly/3sNnNql)(key: 6tan) | 179 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/34Cf6ak) or [BaiduDrive](https://bit.ly/3BiFG4c)(key: vsic) | 3040 |
+
+
+
+
+
+
+
+GRID
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| Overlapped | 1.2 |[GoogleDrive](https://bit.ly/3Aa6PWn) or [BaiduDrive](https://bit.ly/3IdamGh)(key: d8d2) | 186 |
+| Unseen | 4.8 |[GoogleDrive](https://bit.ly/3patMVh) or [BaiduDrive](https://bit.ly/3t6459A)(key: ttsh) | 186 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/2Yzu1PF) or [BaiduDrive](https://bit.ly/30fucjG)(key: 16l9) | 1141 |
+
+You can include `data_ext=.mpg` in your command line to match the video file extension in the GRID dataset.
+
+
+
+
+
+
+Lombard GRID
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| Unseen (Front Plain) | 4.9 |[GoogleDrive](https://bit.ly/3H5zkGQ) or [BaiduDrive](https://bit.ly/3LE1xI6)(key: 38ds) | 186 |
+| Unseen (Side Plain) | 8.0 |[GoogleDrive](https://bit.ly/3BsGOSO) or [BaiduDrive](https://bit.ly/3sRZYNY)(key: k6m0) | 186 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/354YOH0) or [BaiduDrive](https://bit.ly/3oWUCA4)(key: cusv) | 309 |
+
+You can include `data_ext=.mov` in your command line to match the video file extension in the Lombard GRID dataset.
+
+
+
+
+
+
+TCD-TIMIT
+
+
+
+| Components | WER | url | size (MB) |
+|:----------------------|:----:|:---------------------------------------------------------------------------------------:|:-----------:|
+| **Visual-only** |
+| Overlapped | 16.9 |[GoogleDrive](https://bit.ly/3Fv7u61) or [BaiduDrive](https://bit.ly/33rPlZN)(key: jh65) | 186 |
+| Unseen | 21.8 |[GoogleDrive](https://bit.ly/3530d0N) or [BaiduDrive](https://bit.ly/3nxZjzC)(key: n2gr) | 186 |
+| **Language models** |
+| - | - |[GoogleDrive](https://bit.ly/3qzWKit) or [BaiduDrive](https://bit.ly/3KgAL7T)(key: 59u2) | 180 |
+| **Landmarks** |
+| - | - |[GoogleDrive](https://bit.ly/3HYmifr) or [BaiduDrive](https://bit.ly/3JFJ6RH)(key: bnm8) | 930 |
+
+