Engineered for Accessibility
from First Principles.
Sign Language Studio is an open, high-performance computer vision platform designed to translate American Sign Language directly on the client edge. Engineered with on-device deep learning, zero cloud ingestion, and real-time WebRTC peer-to-peer rooms.
Client-Side Neural Inference
The vision stream tracks 21 articulated 3D hand keypoints at 30 to 60 FPS using MediaPipe Hands. Landmarks are normalized relative to wrist origin and evaluated by a 4-layer Sequential Deep Neural Network in TensorFlow.js. Neural inference executes with ~15-25ms forward passes on WebGL/GPU, driving ~400ms debounced recognition lock-in for sustained gesture confirmation, ensuring video frames never leave the device.
Bidirectional P2P Channels
The platform establishes direct browser-to-browser WebRTC DataChannels using PeerJS. Hearing users can speak naturally via the Web Speech API, with words tokenized and translated into illustrated sign sequences in real time. Deaf users sign directly into the camera, streaming synthesized speech and textual feedback to their partner.
Practice Academy & Vision AI
The interactive Practice Laboratory provides pose hold-and-verify feedback, circular SVG progress rings, streak multiplier scoring, and a 60-second speed-run mode. An integrated vision assistant powered by Puter.js allows users to upload sign photographs and receive step-by-step diagnostic signing corrections.
Hardware Note : Smart Glove Integration
The custom Bluetooth Low Energy (BLE) sensor glove (ESP32 micro-controller and MPU-6050 6-axis IMU) was built as an experimental hardware extension to study rotational kinematics and trajectory disambiguation for dynamic gestures. The glove is completely optional and not required to use Sign Language Studio. All primary capabilities (alphabet finger-spelling, numerical counting, dynamic signs, practice quizzes, sign dictionary, and two-way rooms) function with 100% fidelity using any standard laptop or mobile camera through MediaPipe computer vision.
System Feature : Native Windows OS Gesture Controller
The Windows OS Controller module connects real-time computer vision tracking directly to native Windows operating system automation via an ultra-low-latency WebSocket bridge. Native desktop execution is fully functional and operational. Users can steer cursor coordinates across multi-monitor setups, perform left/right/double clicks, drag-and-drop windows and files, scroll documents, adjust master system volume and display brightness, trigger screenshot snipping tools, and virtually inject recognized text sentences into any focused Windows application.