AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding
ahmed-masry
Abstract
A new vision-text alignment method, AlignVLM, effectively maps visual features to LLM embeddings, improving performance in document understanding and robustness to noise.
Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), often produce out-of-distribution or noisy inputs, leading to misalignment between the modalities. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where scanned document images must be accurately mapped to their textual content. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods. We provide further analysis demonstrating improved vision-text feature alignment and robustness to noise.
Community
Happy to announce AlignVLM📏: a novel approach to bridging vision and language latent spaces for multimodal understanding in VLMs! 🌍📄🖼️
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Optimizing Vision-Language Interactions Through Decoder-Only Models (2024)
- HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding (2024)
- LEO: Boosting Mixture of Vision Encoders for Multimodal Large Language Models (2025)
- Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks (2025)
- Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding (2025)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (2024)
- VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
I'm curious about using ALIGNVLM strategy for non-text use cases, e.g. general VQA instead of Text/Chart/OCR related VQA. Wondering why the paper evaluates majorly on the text/document/OCR based image understanding?
Hi Guys
I have created a PoC of using ALIGN module in smolVLM and trained it on roughly 300k images. IMO the eval results are impressive. Here is the link: https://github.com/mvish7/AlignVLM/tree/main
Hi Guys,
Thank you for your interest in our work! We’ve received several questions about the effectiveness of the Align Connector compared to the MLP Connector on general vision-language tasks. To address this, we conducted an additional experiment where we trained the models on the Mammoth-VL dataset, a general vision-language instruction dataset, instead of BigDocs. We then evaluated them on a range of benchmarks such as MMLU, MMVet, SeedBench, POPE, and GQA.
The results in the table below show that ALIGN outperforms the MLP connector across all the benchmarks.
We’re currently in the process of releasing the official codebase and plan to update the arXiv paper soon with the latest results and findings. But I am also glad and excited to see some open-source reimplementation of our Align connector from the community!
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper