Artificial intelligence has become remarkably good at processing written and spoken language. Modern systems can translate between dozens of spoken languages, transcribe conversations, summarize speech, and generate natural-sounding text almost instantly.
Sign languages present a much harder challenge.
They are not simply spoken languages expressed through hand movements. Sign languages have their own vocabularies, grammar, syntax and linguistic conventions. Meaning can be communicated simultaneously through handshape, movement, facial expression, body posture, eye gaze and the use of physical space.
That makes automatic sign-language translation fundamentally different from ordinary text translation.
Now, researchers are increasingly trying to solve the problem using the same technological revolution that produced today’s large language models and vision-language systems. Instead of forcing sign language through an artificial intermediate representation called a “gloss,” newer systems are learning to translate sign-language video directly into natural-language sentences.
A 2026 survey by researchers from Sejong University, Khalifa University and Princess Nourah bint Abdulrahman University describes this transition toward large language models, vision-language models and gloss-free sign-language translation as an important direction for the field.
The implications extend well beyond another AI benchmark. If these systems become reliable enough, they could eventually make digital communication, education, public services and everyday interactions more accessible to millions of Deaf and hard-of-hearing people.
But getting there is considerably more difficult than simply pointing a camera at someone’s hands.
Sign Language Is a Real Language, Not a Gesture Code
One of the most important misconceptions about sign-language AI is the assumption that a system only needs to recognize hand gestures and replace them with words.
That approach misses much of what makes sign languages languages.
A signer can communicate information through several channels simultaneously. Hand movements may convey one part of a message while facial expressions, body position, mouth movements and spatial relationships contribute additional grammatical or semantic information.
The relationship between a sign and a spoken-language word is also not necessarily one-to-one.
A sign-language sentence may need to be reorganized substantially when translated into written English, Arabic, Chinese or another spoken language. The system therefore has to perform something closer to genuine language translation than simple gesture recognition.
Researchers describe sign-language translation as a transformation from visual language into spoken-language text, rather than merely recognizing individual signs.
This distinction is crucial.
An AI system that recognizes a sequence of signs correctly can still produce a poor translation if it fails to understand the grammatical relationship between them.
What Are Glosses?
For many years, researchers addressed this problem by introducing an intermediate representation called a gloss.
A gloss is essentially a textual label assigned to a sign. Researchers can annotate a video with a sequence of labels representing the signs that appear.
The system then performs two steps:
Sign video → gloss sequence → spoken-language sentence
This approach makes the problem easier for machine-learning systems because the gloss layer provides an explicit linguistic bridge between visual information and written language.
But there is a major problem.
Glosses are not the same thing as sign language.
They are simplified representations created for linguistic annotation. They cannot fully capture facial expression, spatial relationships, timing, body movement or other information that can be essential to meaning.
They are also expensive to produce.
One analysis cited by researchers estimates that gloss annotation can require roughly an hour of work for 90 seconds of video.
That creates a major scalability problem.
If researchers want to build translation systems for hundreds of sign languages, manually annotating enormous quantities of video becomes extremely difficult.
The Gloss-Free Approach
Gloss-free translation attempts to remove this intermediate stage.
Instead of training an AI system to produce a gloss sequence first, researchers train it to learn a direct relationship:
Sign-language video → natural-language sentence
This is considerably more difficult because the model receives much less explicit supervision.
The AI has to determine which visual patterns matter, understand the temporal structure of signing, infer linguistic meaning and then generate an appropriate sentence.
Modern architectures try to accomplish this by combining powerful visual encoders with language models.
The basic idea is similar to multimodal AI systems that can analyze photographs and answer questions about them.
But a sign-language video contains another dimension that an ordinary photograph does not: time.
The model must understand how a hand moves from one position to another, how long a movement lasts, when one sign transitions into another and how facial and body movements interact with the hands.
Why Large Language Models Matter
Large language models have transformed natural-language processing because they have learned enormous amounts about syntax, vocabulary, context and semantic relationships from large datasets.
Researchers are now exploring whether that linguistic knowledge can be connected to sign-language video.
One strategy involves adding specialized adapter modules that allow visual representations extracted from sign-language video to interact with a largely pretrained language model.
Instead of teaching an entire language model from scratch, researchers can effectively teach it how to interpret a new visual input modality.
Another approach uses vision-language pretraining.
Models based on technologies such as CLIP-style visual encoders and DINOv2 visual representations already learn relationships between visual patterns and semantic concepts. Researchers can adapt those capabilities to sign-language video.
The 2026 survey identifies systems such as SignLLM, DiffSLT and LLaVA-SLT as examples of research exploring these combinations of visual processing, language models and generative AI.
This is important because it changes the question.
Instead of asking:
How can we teach an AI to recognize every individual sign?
researchers can increasingly ask:
How can we give a language model the visual and linguistic capabilities needed to understand an entire signed utterance?
That is a much more ambitious problem.
The AI Has to Understand Motion, Not Just Images
A sign-language translator cannot simply examine isolated video frames.
Imagine a signer moving a hand from left to right.
The individual frames may look relatively meaningless. The important information is contained in the movement itself.
Consequently, sign-language models need to capture:
- Handshape
- Hand position
- Movement direction
- Movement speed
- Orientation
- Facial expression
- Body posture
- Eye gaze
- Spatial relationships
- Temporal relationships
- Transitions between signs
The system must also distinguish meaningful movements from incidental movements.
That makes sign-language translation a spatiotemporal problem: the AI needs to understand both where things are and how they change over time.
Recent research has therefore explored mechanisms that progressively expand the model’s attention across video frames so that it can capture both local movements and longer-range relationships.
The Problem of Similar-Looking Signs
Another difficulty is visual ambiguity.
Different signs can contain similar hand movements or body configurations. Context may determine which interpretation is correct.
A person might make a movement that looks ambiguous in isolation but becomes obvious when viewed within an entire sentence.
This means successful systems cannot depend exclusively on hand-pose recognition.
They need contextual understanding.
Recent research has specifically identified visual ambiguity, imperfect alignment between video segments and spoken words, and imbalances in word frequency as continuing problems for gloss-free systems.
A model may recognize the visual action correctly but still choose an inappropriate word because it does not understand the surrounding sentence.
Signers Do Not All Sign the Same Way
There is another challenge that is easy to overlook: people have individual signing styles.
Two people communicating the same idea may differ in:
- Signing speed
- Hand position
- Movement size
- Facial expression
- Body posture
- Regional vocabulary
- Personal style
- Camera angle
- Lighting
- Background
An AI system trained primarily on a small number of signers may perform well when tested on people who resemble its training data but perform poorly on unfamiliar signers.
This is a serious issue because benchmark performance can sometimes give a misleading impression of real-world generalization.
A 2025 study examining signer-independent evaluation found that performance for several prominent gloss-free systems dropped sharply when the models were tested under conditions designed to prevent them from relying on signer-specific patterns.
In other words, an AI that performs impressively in a controlled laboratory setting may still have difficulty with an unfamiliar person using a different signing style.
That distinction is essential for accessibility technology.
The Data Problem Is Bigger Than the Model Problem
AI researchers often talk about larger models, better architectures and more computing power.
But sign-language translation has another fundamental constraint:
There is not enough high-quality data.
The world’s sign languages are extremely diverse, and many have relatively little digitized training material.
The most frequently used research datasets tend to represent a small subset of sign languages, while many other communities remain poorly represented.
Research reviewing sign-language datasets has warned that the field relies heavily on a relatively small number of benchmarks, creating risks of overfitting and misleading evaluations.
This creates a feedback loop.
Languages with more data receive better AI systems.
Better AI systems attract more research.
More research produces more data and investment.
Languages with little data can consequently fall further behind.
For sign-language technology to become genuinely global, researchers will need approaches that work effectively with smaller datasets rather than assuming that every language has access to enormous training corpora.
American and British Sign Language Are Not Universal
Another common misconception is that there is one universal sign language.
There isn’t.
American Sign Language and British Sign Language, for example, are different languages. Their users do not simply share a single global vocabulary expressed through different accents.
There are hundreds of sign languages and varieties around the world.
This creates an enormous challenge for AI developers.
A model designed for American Sign Language cannot automatically be assumed to understand Arabic Sign Language, Japanese Sign Language, Malaysian Sign Language or other languages.
The problem becomes particularly important in multilingual environments.
Researchers are therefore increasingly interested in multilingual sign-language datasets and models capable of transferring knowledge between languages without assuming that all languages work identically.
Arabic Sign Language Illustrates the Opportunity
The research described in the 2026 survey places particular attention on Arabic Sign Language and the need for better accessibility technologies in large-scale settings.
This has practical implications in places where millions of people gather.
For example, Saudi Arabia receives huge numbers of visitors for Hajj and Umrah. Providing accessible communication services for Deaf visitors can be challenging because the number of professional human interpreters is limited compared with the scale of the population moving through major religious and public facilities.
AI-based translation could eventually provide an additional accessibility layer.
But such applications also demonstrate why accuracy matters.
A mistake in a casual conversation is one thing.
A mistake involving medical instructions, emergency information, transportation, immigration or religious procedures can have much more serious consequences.
Translation Is Not the Same as Recognition
It is useful to separate several related technologies.
Sign-language recognition
The system identifies individual signs or sequences of signs.
Sign-language-to-text translation
The system converts a continuous signed utterance into natural written language.
Sign-language generation
The system produces signs from written or spoken language, potentially through an avatar or generated video.
Sign-language interpretation
Human interpreters provide contextual, interactive communication between signing and non-signing people.
These technologies overlap but are not interchangeable.
A system that recognizes individual signs does not automatically possess the linguistic ability required for translation.
Similarly, generating a visually convincing signing avatar does not guarantee that the generated signing is linguistically or culturally appropriate.
From Text Back to Signing
The long-term goal is not necessarily one-way translation.
Researchers are also investigating systems that can go in the opposite direction:
Text or speech → sign language
This could eventually enable a hearing person to type a message that an AI converts into signing.
Some research systems are already exploring sign-language generation alongside translation. The broader vision is a two-way communication system in which visual and written languages can be translated in both directions.
However, generating natural sign language may be just as challenging as recognizing it.
A generated avatar would need to reproduce more than hand movements. It would need appropriate facial expressions, timing, body posture and spatial grammar.
A technically impressive avatar could still produce unnatural or linguistically incorrect signing.
Why Automatic Translation Metrics Are Not Enough
Another issue missing from many discussions of AI is how translation quality is measured.
Researchers often use metrics such as BLEU, ROUGE and CIDEr.
These metrics were largely developed for evaluating text-generation or machine-translation systems.
They can provide useful numerical comparisons, but they have limitations.
Consider two translations:
“The boy went to the store because he needed milk.”
and
“He went to the shop to buy milk.”
They communicate substantially similar information but contain different words.
A purely word-overlap-oriented metric may not fully recognize their semantic equivalence.
The problem becomes even more complicated when translating between fundamentally different linguistic structures.
Researchers have therefore called for greater use of human evaluation and more consistent experimental procedures.
Benchmark Scores Can Be Misleading
Recent research has raised another important warning.
Different papers can report impressive improvements while using different training configurations, video encoders, preprocessing techniques and evaluation procedures.
A 2025 comparative study that reimplemented several gloss-free models under standardized conditions found that some published performance advantages became considerably smaller when the systems were evaluated using the same experimental setup.
This does not mean that progress is fake.
It means that researchers need to distinguish between:
- Better model architecture
- Better training data
- Better preprocessing
- Better visual backbones
- Better optimization
- Benchmark familiarity
- Genuine generalization
That distinction is particularly important when technologies are eventually promoted as accessibility tools.
The Real-World Camera Problem
Laboratory demonstrations often assume favorable conditions.
Real life does not.
A sign-language AI deployed through a smartphone might encounter:
- Poor lighting
- Backlighting
- Crowded backgrounds
- Multiple people signing
- Partial obstruction
- Different camera angles
- Low-resolution video
- Fast signing
- Clothing that blends into the background
- Limited internet connectivity
- Regional signing variations
A system that works beautifully in a controlled studio can therefore perform very differently in a train station, hospital, classroom or outdoor environment.
Developing reliable real-world systems requires training and testing under these conditions.
Privacy Is Another Major Concern
Sign-language translation systems may rely on cameras pointed directly at people’s faces and bodies.
That creates privacy questions.
A future translation application could potentially record:
- Facial expressions
- Conversations
- Locations
- Personal interactions
- Medical discussions
- Workplace communications
- Family conversations
If video is uploaded to a remote server for processing, users need to understand what happens to the footage.
Is it stored?
Is it used for training?
Who can access it?
Can the user delete it?
Can the system process the video locally on a phone instead?
These questions are especially important because accessibility technologies should not require people to surrender unnecessary personal information simply to communicate.
The Deaf Community Needs to Be Part of the Development Process
Technical performance is only one measure of whether sign-language AI is useful.
Deaf communities have historically had complicated relationships with technologies developed in their name but without meaningful participation from Deaf users.
That makes co-design important.
Researchers and technology companies should involve Deaf people in decisions about:
- What problems should be solved
- Which languages should receive funding
- How datasets are collected
- How translations are evaluated
- What privacy protections are needed
- How errors should be communicated
- Which applications are appropriate
- How the technology should be deployed
A technically sophisticated system can still fail if it solves the wrong problem.
AI Should Complement Human Interpreters, Not Simply Replace Them
It is tempting to imagine a future in which AI eliminates the need for sign-language interpreters.
That conclusion would be premature.
Human interpreters do much more than convert one vocabulary into another. They understand context, manage conversations, clarify ambiguity and adapt dynamically to participants.
There are also situations where the consequences of mistranslation are particularly serious.
AI could instead become another tool available to interpreters and Deaf users.
For example, it could assist with:
- Everyday conversations
- Captioning
- Customer-service interactions
- Educational materials
- Public announcements
- Online meetings
- Basic information services
- Navigation
- Search and information retrieval
Human expertise could remain essential for complex or high-stakes communication.
The Technology Is Already Moving Beyond Research Papers
This field is no longer purely theoretical.
In August 2026, Google DeepMind announced a massively multilingual sign-language-to-text model intended to support new sign-language features for Deaf and hard-of-hearing users. The company described the technology as an attempt to extend recent AI advances beyond spoken languages and toward the world’s more than 200 sign languages.
That development illustrates the broader trend.
Sign-language AI is beginning to move from specialized academic experiments toward products that can potentially be used by ordinary people.
But commercialization also increases the importance of transparency.
Users need to know not only that a model works on a benchmark, but also where it works, where it fails and which signers and languages were represented during development.
What Happens When AI Makes a Mistake?
A translation system might turn a sentence into something that sounds perfectly grammatical but has the wrong meaning.
This is particularly dangerous because fluent language can create a false sense of confidence.
A human listener might assume:
“The AI produced a clear sentence, so it must have understood the signer.”
But fluency and accuracy are different properties.
Future systems may therefore need confidence indicators, uncertainty detection and mechanisms for asking users to repeat or clarify an utterance.
Instead of pretending that every translation is correct, an AI could say, in effect:
“I am not sufficiently confident in this interpretation.”
That may be far more useful than confidently producing an incorrect sentence.
A More Important Goal Than Perfect Translation
The ultimate objective should not necessarily be to create an AI that translates every sign perfectly.
The more meaningful goal is to reduce communication barriers while preserving users’ autonomy, privacy and linguistic identity.
That means evaluating systems according to real-world questions:
- Can unfamiliar signers use them?
- Do they work across different environments?
- Do they support multiple sign languages?
- Are errors understandable?
- Can users correct the system?
- Is processing private?
- Are Deaf people involved in development?
- Does the technology improve access to education and public services?
- Does it work for people outside major research languages?
Those questions are harder than producing another benchmark score.
They are also more important.
The Future of Sign-Language AI
The development of gloss-free translation represents a significant conceptual change.
Instead of constructing increasingly elaborate pipelines around manually created glosses, researchers are attempting to let AI learn the relationship between continuous signing and natural language directly.
New architectures are experimenting with visual-language pretraining, large language models, multimodal alignment, contrastive learning, temporal modeling and other techniques.
Research published in 2026 demonstrates that the field is continuing to improve through methods designed to align visual and textual representations without gloss supervision.
Yet the biggest breakthrough may ultimately come from somewhere other than model architecture.
It could come from better datasets.
It could come from better evaluation.
It could come from multilingual collaboration.
It could come from on-device AI that protects privacy.
Or it could come from closer cooperation between engineers, linguists, interpreters and Deaf communities.
The technology is therefore best understood as an emerging infrastructure rather than a finished product.
A New Generation of Accessibility Technology
For decades, computers primarily interacted with humans through keyboards, screens and eventually microphones.
The current generation of AI is making visual communication increasingly important.
Sign language is an especially powerful test of this transition because it forces machines to understand movement, space, facial expression, context and language simultaneously.
That makes the problem technically difficult.
It also makes the potential reward unusually significant.
If AI can learn to understand sign languages without reducing them to crude word lists, it could help create communication tools that respect the complexity of visual languages rather than forcing them into the structure of spoken-language systems.
The most important development, therefore, is not simply that an AI model can recognize a hand gesture.
It is that machines are beginning to approach sign language as language.
And that distinction could shape the next generation of accessible technology.
Frequently Asked Questions
1. What is gloss-free sign-language translation?
Gloss-free sign-language translation is an AI approach that converts continuous sign-language video directly into natural-language text without first requiring manually created gloss annotations. The goal is to allow the model to learn the relationship between visual signing and linguistic meaning more directly.
2. Why are glosses being removed?
Gloss annotation is expensive, time-consuming and imperfect. Glosses are also simplified representations and cannot capture every aspect of sign languages, including facial expression, spatial grammar and other simultaneous visual information. Eliminating gloss supervision could make it easier to develop systems for languages that lack large annotated datasets.
3. Can AI translate every sign language?
No. Current systems do not provide universal, equally reliable translation across the world’s sign languages. Research datasets are heavily concentrated in a relatively small number of languages, and many sign languages remain under-resourced. Dataset diversity and signer-independent testing remain major challenges.
4. Can sign-language AI replace human interpreters?
AI may become a useful communication and accessibility tool, but reliable replacement of professional interpreters is a much stronger claim. Human interpretation involves context, ambiguity management and interaction, while AI systems can still make serious errors, particularly with unfamiliar signers, languages and real-world conditions.
5. When will sign-language translation become reliable enough for everyday use?
There is no single reliable date. Research and commercial development are advancing rapidly, including multilingual sign-language-to-text systems announced in 2026. However, widespread reliable deployment will depend not only on model quality but also on diverse datasets, privacy protections, signer-independent evaluation, multilingual coverage, real-world testing and meaningful involvement of Deaf communities.
Conclusion
AI is entering one of the most difficult areas of language technology: understanding human communication that exists primarily in visual space.
Gloss-free sign-language translation offers a promising path because it removes an expensive intermediate layer and allows modern multimodal AI systems to learn directly from video and language.
But the challenge is much larger than recognizing hands.
A successful system must understand movement, timing, facial expression, body posture, context, grammar and cultural variation. It must generalize beyond the people and datasets used for training. It must also be transparent about uncertainty and designed with the people who will actually use it.
The next breakthrough in sign-language AI may therefore not simply be a larger model.
It may be an AI system trained on more diverse languages, tested on unfamiliar signers, evaluated by Deaf users and designed around the realities of everyday communication.
That would turn sign-language translation from an impressive research demonstration into something far more valuable: technology that genuinely expands access without diminishing the languages it is designed to understand.
Sources Bioengineer

