Project Details
[Return to Previous Page]): Beyond Eye-Gaze: Natural Speech Interfaces for Assistive Communication
Company: Gabe Brown Family
Major(s):
Primary: BME
Secondary: EE
Optional: CMPSC, ME
Non-Disclosure Agreement: NO
Intellectual Property: YES
This project gives you the chance to help restore faster and more natural communication for Gabe, a Penn State College of Engineering alumnus who relies on an eye-gaze device after a traumatic brain injury. Instead of typing letter by letter with his eyes, your team will design a wearable silent speech interface that senses the residual movements of his face, jaw, and throat during attempted speech and converts them into spoken messages. The system is meant to complement his existing eye-gaze interface, which stays available for confirmation, correction, training labels, and reliable fallback communication. On the hardware side, you will build a piezoelectric-first wearable using flexible PVDF films coupled to the skin at candidate sites including the underside of the chin, the temporomandibular joint, and the lateral throat, with a repeatable mounting system built from medical-grade tape, a soft patch, or an adjustable collar. A miniature inertial measurement unit will be integrated into the housing to capture head and jaw motion and help separate attempted speech from unrelated movement. You will evaluate three configurations: piezoelectric sensing alone, IMU sensing alone, and the two combined. One or two dry EMG channels may be retained as a comparison modality and added to the final wearable only if they measurably improve cross-day reliability or accuracy. A previous student team built a facial EMG system using MyoWare sensors and an Arduino Uno, and this project moves that work toward lighter, less obtrusive mechanical sensing. On the software side, you will collect synchronized recordings during attempted speech, then train and compare lightweight personalized classifiers such as support vector machines, one-dimensional convolutional networks, and few shot or contrastive models suited to limited training data. Low confidence predictions should be rejected rather than spoken, and the existing eye-gaze system can supply confirmation and accurate training labels. Inference should run locally where practical to keep latency low and keep the user's data on the device. The first target is a reliable personalized vocabulary of roughly 10 to 20 words, commands, or phrases selected by Gabe and his family, not open vocabulary speech reconstruction. Expected deliverables include a working real-time wearable prototype, clear system diagrams and code documentation, and evaluation data covering macro F1-score, false activation rate, decision latency, calibration and setup time, accuracy after the device is removed and replaced, and performance across multiple days. Design goals are at least 90% same-session accuracy and 80% cross-session accuracy on the personalized closed set. The work spans wearable sensor design, signal processing, applied machine learning, and embedded real-time inference, and it gives a fellow engineer a faster way to talk with his family and caregivers.
