Wearable AI systems that combine multiple types of sensors are moving beyond the laboratory to detect everyday health behaviors in real-world conditions, according to recent research from Northwestern Engineering. These systems integrate cameras, accelerometers, motion sensors, and thermal sensors to identify patterns like eating, smoking, and sun exposure, offering a new approach to health monitoring that prioritizes privacy and energy efficiency.

What Makes Multimodal Wearable AI Different from Traditional Health Trackers?

Traditional fitness trackers count steps and measure heart rate, but they miss the behavioral context that matters most to clinicians and health researchers. Glenn Fernandes, who recently completed his PhD in computer science at Northwestern Engineering, has spent the past four years building wearable systems that understand not just what your body is doing, but what you're actually doing with your hands and mouth.

The key difference lies in combining multiple data streams simultaneously. Instead of relying on a single sensor, these systems use audio-visual and motion data together, creating a richer picture of behavior. Fernandes' work on a platform called HabitSense demonstrates this approach: a neck-worn device that uses RGB cameras, thermal imaging, and motion sensors to detect hand-to-mouth behaviors in everyday life.

How Do Researchers Make AI Wearables Actually Work in Daily Life?

Building a wearable system that works in a laboratory is one challenge; making it work when someone is eating lunch at their desk, walking outside, or moving through a crowded space is entirely different. Fernandes emphasized that technical sophistication alone isn't enough. The system must address real-world constraints that academic researchers often overlook.

"A technically impressive model is only one part of the solution. The system also has to work in messy real-world conditions and be meaningful to users," Fernandes explained.

Glenn Fernandes, PhD in Computer Science, Northwestern Engineering

This philosophy shaped every aspect of his research. Rather than optimizing purely for accuracy, Fernandes and his team at the Health-Aware Bits (HABits) Lab considered whether clinicians would actually trust the system's predictions, whether patients would feel comfortable wearing it all day, and whether the battery would last long enough to be practical.

Steps to Building Interpretable AI Health Systems

• Involve Multiple Disciplines: Fernandes' most impactful projects required collaboration across hardware engineering, machine learning, behavioral health, and human-centered design, ensuring the system addressed clinical needs, user comfort, and technical feasibility simultaneously.
• Translate AI Outputs for End Users: A project called PRIMO created an explainable AI interface that helped clinicians understand why the system made specific predictions, recognizing that interpretability is a communication and trust problem, not just a technical feature.
• Test in Real-World Conditions: Rather than validating systems only in controlled laboratory settings, the research prioritized deployment in everyday environments where people actually eat, work, and move, revealing challenges that lab testing would never surface.
• Prioritize Privacy and Energy Efficiency: Multimodal wearables process sensitive data like video and audio, requiring careful design choices about what data is stored locally versus transmitted, and how to minimize battery drain from continuous sensing.

Why Are Clinicians and Researchers Excited About This Approach?

For behavioral health researchers and clinicians, the ability to detect and quantify behaviors like eating frequency, smoking episodes, or sun exposure in real-world settings opens new possibilities for understanding and treating chronic conditions. Traditional methods rely on patient self-reporting, which is often inaccurate or incomplete. Wearable multimodal AI systems provide objective, continuous data.

Fernandes' work demonstrates that this isn't just about collecting more data; it's about collecting the right data in a way that people will actually use. His internships at Dolby Laboratories and Meta Reality Labs Research exposed him to how academic insights translate into products that millions of people interact with daily. At Meta Reality Labs, he worked specifically on wearable AI and multimodal sensing, bridging the gap between research and real-world deployment.

The interdisciplinary nature of this work also reflects a broader shift in AI research. Rather than treating audio-visual AI as a pure computer science problem, Fernandes and his advisers recognized that the most impactful systems emerge when technical design is informed by the people, contexts, and real-world constraints that surround the problem. This philosophy extends beyond wearables to any AI system intended for human use.

As Fernandes moves into his role as a research engineer at Meta Reality Labs, the implications of multimodal wearable AI continue to expand. Systems that can understand human behavior through multiple sensory channels, while respecting privacy and operating efficiently on battery-powered devices, represent a significant step forward in making AI health tools practical for clinical and consumer use.