Fingerspelling with Machine Learning
Experiences,Products
Google Gemini - Generative AI Apps
Experiences
Type:
Products
Experiences
Branding
Platform
Client:
NVIDIA and The American Society for Deaf Children
Deliverables:
Animation, Concept, Design, Development, UI, UX
Signs - Expanding access to language
Signs, an AI-powered web tool, is designed to bridge communication gaps between Deaf and hearing communities! We, in partnership with NVIDIA and the American Society for Deaf Children (ASDC), have created an interactive platform where users can learn American Sign Language (ASL) with real-time feedback from an AI-driven 3D avatar. Signs also empowers everyone, whether you are just starting with ASL or a fluent signer, to contribute their skills, crowdsourcing ASL video content to expand accessibility and understanding for everyone. With hearing loss affecting millions worldwide, and that number projected to double by 2050, we believe tools like Signs are crucial. This initiative reflects our ongoing commitment to using technology for social good, and we're honored to partner with NVIDIA and the ASDC on this important work. We believe improving ASL accessibility is an ongoing effort that deserves attention and innovation, and we're excited to be a part of it. Join us in building a more connected world!
Concept
Building on our success with Fingerspelling, which uses machine vision to teach the ASL alphabet, we developed Signs to further expand ASL accessibility. Signs currently teaches select words, with plans for significant expansion. Critically, user contributions to the Signs dataset will not only enrich our own learning tools but also empower others to develop innovative products and experiences, fostering broader access to the language.
Learn
Signs offers two main sections, starting with "Learn." Here, users learn various signs step-by-step, receiving real-time feedback from machine vision to ensure accuracy and provide practice opportunities. Learners progress through four levels of increasing difficulty, earning points along the way.
Contribute
The second section, "Contribute," invites users to submit signs, helping us build a robust and ever-expanding dataset. These contributions will directly enrich both the dataset and the "Learn" section.
Design
Our goal was to design a simple yet highly functional UI that makes learning ASL feel intuitive and engaging—almost like playing a video game. The primary focus was to ensure users could navigate effortlessly while immersing themselves in the joy of learning. By prioritizing clarity and usability, we created an interface that encourages exploration, interaction, and a sense of accomplishment at every step.
