For millions of deaf and hard-of-hearing individuals around the world, communication barriers can make everyday interactions challenging. Traditional solutions, like sign language interpreters, are often scarce, expensive and dependent on human availability. In an increasingly digital world, the demand for smart, assistive technologies that offer real-time, accurate and accessible communication solutions is growing, aiming to bridge this critical gap.
American Sign Language (ASL) is one of the most widely used sign languages, consisting of distinct hand gestures that represent letters, words and phrases. Existing ASL recognition systems often struggle with real-time performance, accuracy and robustness across diverse environments.
A major challenge in ASL systems lies in distinguishing visually similar gestures such as "A" and "T" or "M" and "N," which often leads to misclassifications. Additionally, the dataset quality presents significant obstacles, including poor image resolution, motion blur, inconsistent lighting, and variations in hand sizes, skin tones and backgrounds. These factors introduce bias and reduce the model's ability to generalize across different users and environments.
To tackle these challenges, researchers from the College of Engineering and Computer Science at Florida Atlantic University have developed an innovative real-time ASL interpretation system. Combining the object detection power of YOLOv11 with MediaPipe's precise hand tracking, the system can accurately recognize ASL alphabet letters in real time. Using advanced deep learning and key hand point tracking, it translates ASL gestures into text, enabling users to interactively spell names, locations and more with remarkable accuracy.
At its core, a built-in webcam serves as a contact-free sensor, capturing live visual data that is converted into digital frames for gesture analysis. MediaPipe identifies 21 keypoints on each hand to create a skeletal map, while YOLOv11 uses these points to detect and classify ASL letters with high precision.
"What makes this system especially notable is that the entire recognition pipeline – from capturing the gesture to classifying it – operates seamlessly in real time, regardless of varying lighting conditions or backgrounds," said Bader Alsharif, the first author and a Ph.D. candidate in the FAU Department of Electrical Engineering and Computer Science . "And all of this is achieved using standard, off-the-shelf hardware. This underscores the system's practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications."
Results of the study, published in the journal Sensors
