A Nigerian computer scientist who lost his hearing at age twelve has developed an AI system that restores emotional context to automated captions, solving a critical accessibility gap that mainstream speech-to-text tools have ignored for years. Dr. Sunday David Ubur, now a University of California Chancellor's Postdoctoral Fellow at UC Irvine, built Interpretive Caption by combining OpenAI Whisper's speech recognition technology with specialized acoustic models that detect and classify vocal emotion in real time.

Why Do Standard Captions Fail Deaf and Hard-of-Hearing Students?

For the world's 430 million Deaf and Hard-of-Hearing individuals, mainstream automated speech-to-text tools like Zoom captions, Microsoft Teams subtitles, and YouTube auto-transcripts strip away critical human information. While these systems transcribe words with acceptable accuracy, they eliminate vocal cadence, emotional tone, pedagogical emphasis, sarcasm, and urgency. In high-stakes environments like STEM classrooms, medical consultations, or engineering briefings, an instructor's vocal inflection often signals whether a remark is casual or critical. Flat, unannotated text leaves deaf students constantly switching their attention between lecture slides, blackboard diagrams, hand gestures, and a text box at the bottom of the screen, causing what researchers call "split-attention" fatigue.

How Does Interpretive Caption Restore Emotional Context to Text?

Dr. Ubur's solution merges OpenAI Whisper's speech recognition transformers with Wav2Vec2, a specialized deep acoustic model fine-tuned to detect and classify vocal emotion across ten distinct states, ranging from calm explanations to forceful, urgent commands. Rather than cluttering the display with animated avatars or distracting emojis, which Dr. Ubur's neurophysiological studies using 14-channel electroencephalography (EEG) brain-wave monitoring proved actually reduces user concentration, he engineered a minimalist interface called "Progressive Disclosure." Lightweight, single-letter markers appear alongside text segments and expand into plain-language descriptions only when users intentionally hover over them.

Taking the innovation further, Dr. Ubur built augmented reality frameworks in Unity that anchor floating, head-stabilized, emotionally annotated captions directly beside a speaker's face inside next-generation headsets like the Meta Quest and Apple Vision Pro. In controlled laboratory trials, this spatial breakthrough achieved remarkable results:

• Technical Comprehension: STEM concept comprehension improved from a 54 percent baseline under conventional flat captions to nearly 100 percent.
• Cognitive Load: Subjective mental workload and cognitive fatigue decreased by a full quarter, or 25 percent.
• Research Support: The work received funding from the University of California Chancellor's Office and the National Science Foundation (NSF).

What Personal Challenges Drove This Innovation?

Dr. Ubur's journey to this breakthrough began when he lost his hearing at age twelve in Kano, Nigeria, after a playground incident and exposure to loud traditional drums. Rather than accepting the societal script that told him to "sit down, stay quiet, and manage whatever crumbs come your way," he pursued a rigorous computer science degree at the University of Abuja without access to sign language interpreters, institutional accommodations, or digital note-takers. He performed what he now theoretically describes as the "second shift" of access labor, waking before dawn to claim front-row seats, lip-reading lecturers, borrowing classmates' notes, and spending midnight hours in library corners reconstructing entire semester curricula from textbooks and determination.

In his groundbreaking autoethnographic paper for the ACM ASSETS conference, Dr. Ubur encapsulated this reality with a phrase that has echoed across international academic auditoriums: "My body became the access infrastructure". This personal experience motivated him to ensure that others would not have to build accessibility through sheer physical endurance.

How Did Dr. Ubur Gain International Recognition?

Dr. Ubur's competitive drive caught international attention early. In 2017, he outpaced tens of thousands of applicants across the African continent to secure the prestigious U.S. State Department's Mandela Washington Fellowship, also known as the Young African Leaders Initiative (YALI), landing at the McCombs School of Business at the University of Texas at Austin and Gallaudet University in Washington, D.C. . During this pivotal fellowship, he encountered formal captioning, American Sign Language interpreting infrastructure, and institutional accommodations for the first time, illuminating the staggering divide between Western assistive access and the structural neglect across the Global South.

He subsequently earned the British Government's fiercely contested Chevening Scholarship, completing a Master of Science in Software Development at Coventry University in the United Kingdom, where he mastered distributed enterprise architectures and asynchronous event pipelines. He was then admitted directly to Virginia Tech to complete his Ph.D. under the mentorship of internationally renowned virtual environments pioneer, Prof. Denis Gračanin.

What Is the Broader Impact of This Research?

Dr. Ubur's scholarly portfolio spans the absolute pinnacle of computing venues, including ACM CHI, ACM ASSETS, and IEEE VR, where his papers routinely set benchmarks for human-in-the-loop computing. His research is conducted in collaboration with leading accessibility researchers such as Prof. Stacy M. Branham at UC Irvine, helping shape how next-generation artificial intelligence, spatial computing, and assistive reading platforms are designed. Back home in Nigeria, where the full implementation of the national Disability Act remains an ongoing campaign and deaf youth continue to face systemic exclusion, Dr. Ubur's achievements serve as a sharp, unapologetic reality check to government policymakers, university administrators, and society at large.