1 System Overview
Bridging the gap between complex AI models and accessible end-user applications is a significant engineering challenge. This project aimed to democratize American Sign Language (ASL) education by building a browser-based recognition system capable of running on consumer hardware without expensive server-side GPU costs. The goal was to combat Language Deprivation Syndrome by providing an accessible, automated tutor for families of deaf children.
I architected and deployed the full-stack web application, transforming a raw deep learning model into a production-ready interactive tool. While the underlying model was trained on the Google Isolated Sign Language Recognition dataset, my focus was on Model Serving and System Latency Optimization. I engineered the pipeline to load and run the TensorFlow Lite model directly in the client's browser, ensuring real-time performance regardless of the user's internet speed.
2 Technical Implementation
2.1 Client-Side Inference & Visualization
To achieve real-time responsiveness, I implemented a client-side inference engine. By decoupling the processing from the backend, I reduced server load and eliminated network latency during practice sessions.
- MediaPipe Integration: Utilized MediaPipe Holistic to extract 3D landmarks of hands, face, and pose in real-time within the browser.
- Visual Feedback Loop: Built a React-based interface that overlays skeleton tracking on the video feed. This provides immediate visual confirmation to users, closing the feedback loop between human gesture and machine interpretation.
2.2 Data Pipeline & Deployment
Beyond the UI, I built the infrastructure required to scale the application:
- AWS Deployment: Configured a serverless hosting architecture using AWS S3 for static assets and model weights. This ensured high availability and decoupled the frontend from the storage logic.
- Data Collection: Developed a recording module that captures user interactions. This feature not only allows users to review their progress but also serves as a data ingestion pipeline to collect real-world samples for future model fine-tuning.
This project demonstrated the viability of Edge AI in educational tools. The system's engineering design and user efficacy were documented in a technical paper accepted for publication, validating both the technological implementation and its practical application.
3 Selected visuals
