He Robotics and Three-Dimensional Vision Group (RoViT) of the University of Alicante (UA) He has designed Sign4all, the first application capable of recognizing and interpreting the Spanish sign language alphabet in real time (known by the initials LSE). This is an advance that contributes to breaking the communication barriers between deaf and hearing people in situations as everyday as going to a doctor’s office or eating in a restaurant. According to the latest Survey on disability, personal autonomy and dependency situations of the National Institute of Statistics (INE), in Spain there are 1,230,000 people with a hearing disability of different types and degrees. Of these, 27,300 people use sign language to communicate.
Thanks to the use of different computer vision and deep learning techniques (deep learning), PhD in Computer Science and UA researcher Ester Martínez, together with doctoral student Francisco Morillas, have developed this low-cost tool in order to be able to offer assistance to deaf people when they cannot be accompanied by an interpreter.
Sign4all, after capturing the person and extracting the details of the skeleton of the arms and hands, encodes the left part of the body in blue and the right part in red, maintaining the user’s anonymity at all times. From this moment on, the application translates the sign used by the deaf person in real time and, in the opposite sense, is capable of signing Spanish words typed by the hearing person through a virtual avatar. “The idea is that this entire process can be done by downloading an application and using the camera of the mobile phone or a tablet, so that it can be used anywhere,” explains Martínez.
“After many tests, Sign4all manages to interpret and recognize the LSE alphabet with an accuracy of 80%”, explains the UA researcher. “Although this result corresponds to the data alphabet, we are working on a version with a specific vocabulary belonging to the field of daily tasks where we can interpret complete sentences”, she adds.
The UA work team has been “training” this new system for months and introducing more and more signs. In this sense, a collaboration has emerged with the Spanish Language and Signed Languages Research Group (GRILES) of the University of Vigo, a group with extensive experience in the study of this language and its use in different territories. “From the University of Vigo they are collecting images with interpreters and we are processing all that data. In this way, we can expand the vocabulary of our LSE recognition and interpretation system much faster and improve its operation”, points out Ester Martínez.
