README.md

---
title: RelightVid
emoji: 🎥
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 5.23.3
app_file: app.py
license: mit
---

<!-- # <img src="assets/icon.png" style="vertical-align: -14px;" :height="50px" width="50px"> RelightVid -->
# RelightVid

**[RelightVid: Temporal-Consistent Diffusion Model for Video Relighting](https://arxiv.org/abs/2501.16330)**
</br>
[Ye Fang](https://github.com/Aleafy)\*,
[Zeyi Sun](https://github.com/SunzeY)\*,
[Shangzhan Zhang](https://zhanghe3z.github.io/),
[Tong Wu](https://wutong16.github.io/),
[Yinghao Xu](https://justimyhxu.github.io/),
[Pan Zhang](https://panzhang0212.github.io/),
[Jiaqi Wang](https://myownskyw7.github.io/),
[Gordon Wetzstein](https://web.stanford.edu/~gordonwz/),
[Dahua Lin](http://dahua.site/)

<p style="font-size: 0.6em; margin-top: -1em">*Equal Contribution</p>
<p align="center">
<a href="https://arxiv.org/abs/2501.16330"><img src="https://img.shields.io/badge/arXiv-Paper-<color>"></a>
<a href="https://sunzey.github.io/Make-it-Real"><img src="https://img.shields.io/badge/Project-Website-red"></a>
<a href="https://www.youtube.com/watch?v=_j-t8592GCM"><img src="https://img.shields.io/static/v1?label=Demo&message=Video&color=orange"></a>
<a href="" target='_blank'>
<img src="https://visitor-badge.laobi.icu/badge?page_id=Aleafy.RelightVid&left_color=gray&right_color=blue">
</a>
</p>


![Demo](./assets/demo.gif)


## 📜 News
<!-- 🚀 [2024/6/8] We release our [inference pipeline of Make-it-Real](#⚡-quick-start), including material matching and generation of albedo-only 3D objects.

🚀 [2024/6/8] [Material library annotations](#📦-data-preparation) generated by GPT-4V and [data engine](#⚡-quick-start) are released! -->

✨ [2025/3/12] The [inference code](xxx), [project page](xxx) and [huggingface demo](xxx) are released!

✨ [2025/1/27] We release the [paper](https://arxiv.org/abs/2501.16330) of RelightVid!

## 💡 Highlights
- 🔥 We first demonstrate that **GPT-4V** can effectively **recognize and describe materials**, allowing our model to precisely identifies and aligns materials with the corresponding components of 3D objects.
- 🔥 We construct a **Material Library** containing thousands of materials with highly
detailed descriptions readily for MLLMs to look up and assign.
- 🔥 **An effective pipeline** for texture segmentation, material identification and matching, enabling the high-quality application of materials to
3D assets.

## 👨‍💻 Todo
- [ ] Evaluation for Existed and Model-Generated Assets (both code & test assets)
- [ ] More Interactive Demos (huggingface, jupyter) 
- [x] Make-it-Real Pipeline Inference Code
- [x] Highly detailed Material Library annotations (generated by GPT-4V) 
- [x] Paper and Web Demos

## 💾 Installation

   ```bash
   git clone https://github.com/Aleafy/RelightVid.git
   cd RelightVid

   conda create -n relitv python=3.10 
   conda activate relitv
   
   pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu118
   pip install -r requirements.txt
   ```





## 📦 Data Preparation
 1. **Annotations**: in `data/material_lib/annotations` [folder](data/material_lib/annotations), include:
    - Highly-detailed descriptions by GPT-4V: offering thorough descriptions of the material’s visual characteristics and rich semantic information.
    - Category-tree: Divided into a hierarchical structure with coarse and fine granularity, it includes over 80 subcategories.
 2. **PBR Maps**: You can download the complete PBR data collection at [Huggingface](https://huggingface.co/datasets/gvecchio/MatSynth/tree/main), or download the data used in our project at [OpenXLab](https://openxlab.org.cn/datasets/YeFang/MatSynth/tree/main) (Recommended). (If you have any questions, please refer to [issue#5](https://github.com/Aleafy/Make_it_Real/issues/5))
 3. **Material Images(optinal)**: You can download the material images file [here](https://drive.google.com/file/d/1ob7CV6JiaqFyjuCzlmSnBuNRkzt2qMSG/view?usp=sharing), to check and visualize the material appearance.

<pre>
Make_it_Real
└── data
    └── material_lib
        ├── annotations
        ├── mat_images
        └── pbr_maps
            └── train
                ├── Ceremic
                ├── Concrete
                ├── ...
                └── Wood
</pre>



## ⚡ Quick Start
#### Inference
```bash
python main.py --obj_dir <object_dir> --exp_name <unique_exp_name> --api_key <your_own_gpt4_api_key>
```
- To ensure proper network connectivity for GPT-4V, add proxy environment settings in [main.py](https://github.com/Aleafy/Make_it_Real/blob/feb3563d57fbe18abbff8d4abfb48f71cc8f967b/main.py#L18) (optional). Also, please verify the reachability of your [API host](https://github.com/Aleafy/Make_it_Real/blob/feb3563d57fbe18abbff8d4abfb48f71cc8f967b/utils/gpt4_query.py#L68).
- Result visualization (blender engine) is located in the `output/refine_output` dir. You can compare the result with that in `output/ori_output`. 

#### Annotation Engine

```bash
cd scripts/gpt_anno
python gpt4_query_mat.py
```
`Note`: Besides functinoning as annotation engine, you can also use this code ([gpt4_query_mat.py](https://github.com/Aleafy/Make_it_Real/blob/main/scripts/gpt_anno/gpt4_query_mat.py)) to test the GPT-4V connection simply.

<!-- [annotation code](scripts/gpt_anno) -->
<!-- #### Evalutation -->



## ❤️ Acknowledgments
- [MatSynth](https://huggingface.co/datasets/gvecchio/MatSynth/tree/main): a Physically Based Rendering (PBR) materials dataset, which offers extensive high-resolusion tilable pbr maps to look up.
- [TEXTure](https://github.com/TEXTurePaper/TEXTurePaper): Wonderful text-guided texture generation model, and the codebase we built upon.
- [SoM](https://som-gpt4v.github.io/): Draw visual cues on images to facilate GPT-4V query better.
- [Material Palette](https://github.com/astra-vision/MaterialPalette): Excellent exploration of material extraction and generation, offers good insights and comparable setting.

## ✒️ Citation
If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝
```bibtex
@misc{fang2024makeitreal,
      title={Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials}, 
      author={Ye Fang and Zeyi Sun and Tong Wu and Jiaqi Wang and Ziwei Liu and Gordon Wetzstein and Dahua Lin},
      year={2024},
      eprint={2404.16829},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```