Talking Fingers is an ASL learning app built around real-time signing feedback using computer vision. Our team took the app from concept to a working iOS and macOS experience in one semester.
Beginner ASL learners need to build new physical vocabulary and grammar, often without feedback or guidance.
A working iOS and macOS ASL learning app that guides learners with camera-based feedback and AI-powered exercises.
As Design Manager, I led 4 designers to bring life to the technical vision our tech leads had of using computer vision, spaced repetition, and AI-generated practice to transform ASL learning. We shaped the learning experience while working closely with developers throughout implementation.
We analyzed 7 comparable apps with education or camera feedback features to explore how to bring language learning and physical movement feedback into one experience.
| Product category | Learning & practice | Feedback & progress | Gaps we can fill |
|---|---|---|---|
| ASL appsLingvano, ASL Bloom | Video lessons and sign imitation. | Limited feedback on signing accuracy. | Help learners understand how they're signing. |
| Language appsDuolingo, Babbel, Quizlet | Short exercises and repeated practice. | Streaks, scores, and visible progress. | Make ASL practice varied and progress easy to see. |
| Camera-based fitnessTempo, Clearspace | Guided physical movements. | Camera-based feedback on form. | Make movement feedback understandable during signing. |
We interviewed 3 people with different ASL learning experiences, from a prospective beginner to someone with over four years of experience.
I need reassurance!
Learners wanted confirmation that they were signing correctly.
I can't just do drills.
Repeating isolated signs could get boring without varied practice.
Show me my progress.
Visible progress could make learning feel more manageable.
We organized Talking Fingers into three learning modes with clear progression, gradually building skills and confidence.
Introduces new signs.
Reinforces them through repetition & recall.
Brings everything together in sentences.
Users generate sessions based on categories they want to practice and apply signs in context. Our computer vision recognition system evaluates each sign individually, so beginners can identify their weak spots.
To keep learners focused on signing, we showed the ASL gloss upfront instead of requiring them to reconstruct sentences before unlocking the camera. This simplified grammar practice while preserving ASL's distinct sentence structure.
Unlike spoken-language exercises, signing practice requires learners to see and adjust their own movements. Early wireframes gave more space to instructions than the camera, so I enlarged the camera and reduced the surrounding UI to keep the learner's movements front and center.
Users can customize how they track progress with widgets both in profile and outside of the app, encouraging user engagement through gamification and action-oriented design.
We developed a playful, approachable visual language with soft colors, friendly typography, and a hand-inspired tulip mascot that emphasizes growth and encouragement.
The resting pose that greets learners across the app.
I maintained a shared Figma component system to keep our five designers aligned while working alongside developers. We shared designs early and refined them through critiques, which led to smooth adaptation when transitioning from iOS to the macOS experience.
Our team delivered a working iOS and macOS app with live computer vision feedback and AI-generated practice, sweeping all 3 showcase awards (a first in the club's history!).
This was my first time leading a design team, and I quickly learned that being a good leader doesn't stop at delegating tasks. I wanted my four designers to feel supported, confident contributing ideas, and excited about what we were building. Seeing them grow into their roles and take ownership of their work by Demo Day was one of the most rewarding parts of the project.
We started the semester with an ambitious technical vision and somehow had to turn that into a working app in just a few months. Design and development happened at the same time, which meant sharing unfinished ideas, adapting to technical constraints, and sometimes making decisions before we had all the answers. It was exciting to watch everything come together, but it also taught me to communicate early and stay flexible when things didn't go as planned.
Best Overall · Best Design · Audience Choice