Sound is a critical part of multimodal perception. For a system — be it a voice assistant, a next-generation security monitor, or an autonomous agent — to behave naturally, it must demonstrate a full spectrum of auditory capabilities. These capabilities include transcription, classification, retrieval, reasoning, segmentation, clustering, reranking, and reconstruction.
These diverse functions rely on transforming raw sound into an intermediate representation, or embedding. But research into improving the auditory capabilities of multimodal perception models has been fragmented, and there remain important unanswered questions: How do we compare performance across domains like human speech and bioacoustics? What is the true performance potential we are leaving on the table? And could a single, general-purpose sound embedding serve as the foundation for all these capabilities?
To investigate these queries and accelerate progress toward robust machine sound intelligence, we created the Massive Sound Embedding Benchmark (MSEB), presented at NeurIPS 2025.
MSEB provides the necessary structure to answer these questions by:
- Standardizing evaluation for a comprehensive suite of eight real-world capabilities that we believe every human-like intelligent system must possess.
- Providing an open and extensible framework that allows researchers to seamlessly integrate and evaluate any model type — from conventional downstream uni-modal models to cascade models to end-to-end multimodal embedding models.
- Establishing clear performance goals to objectively highlight research opportunities beyond current state-of-the-art approaches.
Our initial experiments confirm that current sound representations are far from universal, revealing substantial performance “headroom” (i.e., maximum improvement possible) across all eight tasks.