Soundwave: Less is More for Speech-Text Alignment in LLMs
Yoohao
Abstract
Soundwave addresses the representation space gap and sequence length inconsistency in end-to-end speech large language models using an efficient training strategy and novel architecture, outperforming Qwen2-Audio with significantly less data.
Existing end-to-end speech large language models (LLMs) usually rely on large-scale annotated data for training, while data-efficient training has not been discussed in depth. We focus on two fundamental problems between speech and text: the representation space gap and sequence length inconsistency. We propose Soundwave, which utilizes an efficient training strategy and a novel architecture to address these issues. Results show that Soundwave outperforms the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks, using only one-fiftieth of the training data. Further analysis shows that Soundwave still retains its intelligence during conversation. The project is available at https://github.com/FreedomIntelligence/Soundwave.
Community
Soundwave is a Speech LLM that can process various speech tasks (e.g., speech recognition and speech translation). It is trained with just one-fiftieth of the data size compared to Qwen2-Audio, while achieving better performance ont AIR-Bench speech tasks. More details can be found at https://github.com/FreedomIntelligence/Soundwave.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia (2025)
- Zero-resource Speech Translation and Recognition with LLMs (2024)
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (2025)
- DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities (2025)
- OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis (2025)
- Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition (2025)
- Audio-Language Models for Audio-Centric Tasks: A survey (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Excellent work! I believe the alignment adaptor output, supervised by the CTC loss, captures more semantic information while reducing acoustic dependency. However, Soundwave achieves SOTA performance in speech emotion recognition (0.635 on MELD). I’d love to hear your thoughts on this.
Thank you for paying attention to our work. That’s a good question, and we have considered it as well. We use two branches to address this problem in the shrinking adapter (see Figure below). The content features are shrunk based on the CTC prediction, and then we use the shrunk features as queries, applying cross-attention to extract paralinguistic information from the original sequence. Finally, the two types of feautures are fused as the output.
In short, the content features represent 'what the person says,' while the auxiliary features represent 'how the person says it.' The complete speech features are decomposed into semantic and acoustic features, making it easier to detect emotions for LLMs.
"We first identify two fundamental challenges to achieve alignment between speech and text (Zhang et al., 2023): (1) the representation space gap and (2) sequence length inconsistency."
Maybe the authors did not mean it but this sentence gives the impression that the above two challenges are identified by the current work, which is quite misleading. Those two challenges are widely known in the speech translation literature (the first is typically called "modality gap"), so instead of "We first identify two fundamental challenges..." it should have been something along the line of "It's well-known that there are two fundamental challenges...". See for example Le et al.'s Pre-training for Speech Translation: CTC Meets Optimal Transport
(https://arxiv.org/abs/2301.11716 ) and references therein (the idea of using CTC for speech-text alignment is also well-known, per this paper).
Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper