For a computer, understanding sign language has always been like trying to read a dictionary that only includes words you already know. That’s because traditional systems for recognizing individual hand signs are often set up in a very specific way. To explain, these systems, which scientists call Isolated Sign Language Recognition, learn to identify a particular sign, say hello, only after being shown many examples of hello. Each of those examples comes with a specific tag, a kind of linguistic label called a gloss. The computer links the visual information of the sign directly to that one tag.

The key thing to understand here is what this means for new signs. If the system later encounters a sign for, say, "yesterday," but was never trained on a "yesterday gloss label, it simply cannot recognize it. This limitation is known as closed-set classification. It means these systems are excellent at recognizing signs they’ve been explicitly taught, but they cannot generalize. They can’t grasp the meaning of a sign if it falls outside their predefined vocabulary. So, every time someone wanted to teach a computer a new set of signs, they had to start from scratch, meticulously tagging each new sign with its own gloss. This takes immense effort.

Now, imagine a different kind of dictionary—one that doesn’t just tell you what a word is called, but describes what it means and how you would use it. This is the underlying idea behind a powerful new approach to understanding sign language. Rather than relying on rigid, pre-set labels, a team led by Santiago Poveda-GutiĂ©rrez, Hideki Nakayama, and Mayumi Bono has developed what they call a "reverse sign language dictionary." This work, appearing as a preprint on arXiv, promises to allow computers to understand a far wider range of signs, including those they've never seen before. It has two main steps. First, the system watches a video of someone signing. It’s designed to handle continuous signing, which means it can process natural, flowing sign language, not just individual, separated signs. As it watches, a special computer program, trained to understand both video and language, gets to work. Think of this program as an incredibly observant transcriber. Its job is to generate a detailed, free-form written description of the hand movements, the arm positions, and the facial expressions that make up each sign. It’s not just giving it a single word like "hello," but describing the action of the sign itself.

Once the system has this procedural description—a sentence or two about how the sign is articulated—it moves to the second step. Here, another smart program comes into play. This one specializes in understanding the meaning within written sentences, and it can even work across different languages. Its task is to take the freshly generated description of the sign and compare it to a vast collection of existing sign descriptions. These aren’t just gloss labels; they are rich, human-readable descriptions of what each sign looks like and what it conveys. The program then finds the description in its collection that is the most similar to the one it just created from the video.

Think of it this way: instead of seeing a sign and looking it up by its name, the system sees a sign, describes what it sees, and then searches a library of descriptions to find a match. This is why it’s called a reverse dictionary. It starts with the sign’s appearance and works backward to find its meaning, without needing a pre-assigned gloss label. This frees the system from the closed-set problem. It enables what is known as open-vocabulary sign recognition, meaning it can understand signs beyond a fixed, pre-programmed list.

The researchers tested their system using 1,300 distinct sign-level segments taken from a corpus of Japanese Sign Language conversations. Their target vocabulary, the collection of descriptions the system would search through, contained 503 entries. The results were quite encouraging, especially when considering unseen-class retrieval—the ability to identify signs it had never encountered during its initial training.

Initially, without much fine-tuning, the system could identify the correct sign description within its top ten guesses for unseen signs 11.5% of the time. While that might sound modest, it’s a significant leap beyond traditional systems, which simply cannot recognize these new signs at all. For context, a random guess from its 503-entry vocabulary would only give it a 2% chance. With further enhancements to the way the system processes descriptions and understands the connections between language and vision, that figure rose to 21.0% for unseen signs. When identifying signs it had encountered before, the system's accuracy improved from an initial 4.5% to a much more substantial 49% after these refinements. One of the language components of the system already accurately captures nearly 100% of the meaning in paraphrased descriptions, which is a strong starting point.

This capability to recognize signs it has never explicitly learned opens up new possibilities for sign language technology. It makes the system far more flexible and less dependent on painstaking manual labeling. There’s still work to do, of course. The researchers note that the quality of the video captions—the free-form descriptions generated in the first step—is an area for future improvement. But the core breakthrough is clear. We are moving closer to a future where computers can genuinely learn and understand the rich, dynamic language of signing, not just as a fixed list of gestures, but as a living, evolving conversation. ◆

Sourced from: "A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval" by Santiago Poveda-Gutiérrez, Hideki Nakayama, and Mayumi Bono