Wearable devices, from smartwatches to fitness trackers, have become ubiquitous, continuously capturing a rich stream of data about our lives. They record our heart rate, count our steps, track our fitness and sleep, and much more. This deluge of information holds immense potential for personalized health and wellness. However, while we can easily see what our body is doing (e.g., a heart rate of 150 bpm), the crucial context of why (say, “a brisk uphill run” vs. “a stressful public speaking event”) is often missing. This gap between raw sensor data and its real-world meaning has been a major barrier to unlocking the full potential of these devices.
The primary challenge lies in the scarcity of large-scale datasets that pair sensor recordings with rich, descriptive text. Manually annotating millions of hours of data is prohibitively expensive and time-consuming. To solve this, and to truly let wearable data “speak for itself”, we need models that can learn the intricate connections between sensor signals and human language directly from the data.
In “SensorLM: Learning the Language of Wearable Sensors”, we introduce SensorLM, a family of sensor–language foundation models that bridges this gap. Pre-trained on an unprecedented 59.7 million hours of multimodal sensor data from over 103,000 individuals, SensorLM learns to interpret and generate nuanced, human-readable descriptions from high-dimensional wearable data, setting a new state of the art in sensor data understanding.