Google DeepMind has released a breakthrough artificial intelligence model that translates sign language directly into text on consumer smartphones, marking the first time sign language AI has moved from research labs into everyday products for Deaf and hard of hearing users. The new sign-language-to-text (SL2T) model powers sign-to-text dictation in Gboard and Live Transcribe on Pixel 11 phones, starting with American Sign Language (ASL) to English, with additional languages and devices coming soon.

While AI has transformed how hearing people interact with technology through voice dictation and automatic translation, the world's more than 200 sign languages and 70 million Deaf and hard of hearing people have been largely left behind. This gap reflects both the technical complexity of processing sign language and widespread misconceptions about how these languages work. SL2T represents a fundamental shift in closing that divide.

Why Is Sign Language AI So Much Harder Than Speech Recognition?

Sign languages present two core challenges that speech-to-text systems do not face. First, sign languages are independent, natural languages with their own distinct grammars and lexicons, not simply English expressed through hand movements. This means the AI must perform true machine translation, not just convert hand shapes into English words. Second, the model must learn to perceive and understand complex physical movement across the entire body. Sign languages convey meaning through simultaneous movements of the hands, arms, torso, head, and face, requiring the AI to track these movements at high frame rates.

Early attempts to solve this problem, such as sign language gloves, failed because they treated sign languages as if they were just "English on the hands." SL2T takes a fundamentally different approach by combining sophisticated computer vision with full-fledged language translation capabilities.

How Does SL2T Actually Work?

Google's team built SL2T by training the model on over 100,000 hours of data across more than 50 sign languages, with roughly a quarter of the data in ASL. Training jointly on diverse languages, dialects, and proficiency levels causes the model to learn shared underlying structures, outperforming single-language models in experiments.

The system works by tracking pose landmarks, or specific points on the signer's body, rather than processing raw video. An on-device model called MediaPipe Holistic identifies the location of these points, and only the geometric coordinates are sent to the server for translation. This approach protects user privacy by allowing the original video to be discarded immediately. SL2T then translates this coordinate sequence directly into text, bypassing intermediate annotations called "glosses" that were used in prior sign language translation work.

The performance gains are substantial. SL2T achieved a zero-shot score of 70 BLEURT on the FLEURS-ASL benchmark, which assesses ASL to English translation quality. This score is significantly higher than any previously reported result, demonstrating a major leap forward in translation accuracy.

What Practical Benefits Does This Bring to Users?

For Deaf users, the new feature enables signing to their phone anywhere they would normally type. This includes searching the web, drafting messages or documents, and asking Gemini, Google's AI assistant, to solve queries or execute tasks. In Live Transcribe, Deaf users can sign responses in conversations instead of having to type back and forth. According to Google's testers, signing in ASL is faster, more natural, and more delightful than typing in English.

Beyond convenience, this technology opens new possibilities for bridging the communication gap between Deaf and hearing communities. Sign languages are the primary languages of Deaf communities around the world and the cornerstone of Deaf cultural identity. Supporting sign language processing in the same way that hearing people benefit from spoken language processing represents a significant step toward genuine digital accessibility.

Steps to Ensure Responsible Deployment of Sign Language AI

• Community-Centered Design: Google worked with Deaf perspectives at every stage, from conceptualization by Sam Sepah, a Deaf Googler, to data collection with Deaf partners, evaluation in Deaf user studies, and impact assessment with Deaf experts.
• Participatory Governance: The company established the AI Sign Language Advisory Committee (AISLAC), bringing together global Deaf organizations and subject-matter experts to directly influence development priorities and ensure the communities most impacted by the technology shape its future.
• Transparent Impact Reporting: Google co-authored a joint impact report detailing the technology's capabilities and current limitations, with plans to continue this collaborative approach for all major sign language releases.

The team also worked hard on practical issues that matter in real-world use. These included minimizing streaming latency so responses feel immediate, preventing hallucination on non-signing inputs, ensuring fairness for the 10 percent of signers who are left-handed, and improving performance for one-handed signing, which is used while holding a smartphone in the other hand.

What Are the Current Limitations?

While SL2T represents a major breakthrough, occasional errors remain in rare signs, rapid fingerspelling, passive constructions, classifier depictions, and tense without context. For example, the word "prey" was occasionally mistranslated as "grey," and some complex grammatical structures still present challenges. These limitations are being addressed as the model is deployed and refined through real-world use.

The rollout is beginning with American Sign Language on Pixel 11 phones, but Google's team is working to expand this technology into additional sign languages, sign language generation, and frontier AI capabilities. The company stated that "bringing ASL input to users' phones is only the beginning," signaling a commitment to achieving full parity with spoken and written languages across the global Deaf community.

This release demonstrates how AI can address accessibility gaps that have persisted for decades, but only when built with the communities it serves rather than for them. As more sign languages are added and the technology improves, SL2T could fundamentally reshape digital accessibility for hundreds of millions of people worldwide.

" }