The post describes the first phase of my high school senior year project to convert signals obtained from an OpenBCI system to synthesized speech.
The objective of this project was to explore whether a low-cost BCI device could isolate sensorimotor rhythms (SMRs) evoked by motor imagery as direct EEG signals, and subsequently convert them through a dedicated software solution to a binary Morse code-like output and ultimately into speech. This type of device could be useful to individuals with conditions that render them non-verbal and more specifically, virtually immobile (patients with severe ALS causing partial or total locked-in syndrome).
Partial LIS patients would retain some voluntary motor function (e.g. blinking, ear wiggling, teeth grinding), but are unable to communicate verbally or through ASL. Therefore, a device that takes advantage of the remaining motor functions to establish a method of communication could be an effective tool for these individuals. The first phase of the project, described in this report, aimed to capture electromyographic signals from facial muscles and convert them into speech, with the objective to replace them with SMR signals in the second phase of the project.
The BCI device was 3D printed using free design and print files found on OpenBCI, an open source BCI solution. The device uses an 8-channel Cyton board for feature extraction and amplification, which then sends the signal to the computer via Bluetooth. Passive electrodes are placed in standardized locations around the headset and connected to the Cyton board up to 8 at a time. The electrodes, board, and associated accessories were all purchased or printed via instructions found on the OpenBCI website.
The dedicated software package was developed with Microsoft Visual Studio 2017 Community version and consisted in 4 parts: signal acquisition and processing module was based on the excellent conversion of the open source OpenBCI_GUI into C# from Marcin Jukiewicz http://openbci.com/community/c-and-vs2013-example-of-openbci-data-acquisition. Signal peaks were detected through a z-score algorithm example described here https://stackoverflow.com/questions/22583391/peak-signal-detection-in-realtime-timeseries-data/22640362#22640362 (user Jean_Paul). Morse-to-text conversion was based on this sample code https://code.sololearn.com/cVh3TmuQ6n8k/#cs, while for the final text-to-speech conversion, the Microsoft SpeechSynthesizer Class was used (described here https://docs.microsoft.com/en-us/dotnet/api/system.speech.synthesis.speechsynthesizer?view=netframework-4.7.2).
Instead of trying to start by capturing the SMR signals, the project began with an exploration of possible stronger signals that could be isolated in independent channels in order to first develop a binary system of communication (one movement corresponds to a “dot”, while the other corresponds to a “line”) which could then be converted to a Morse code-like structure. After several rounds of trial-and-error, it was found that eye blinking and ear movements were the best candidates for simple, independent signals. Other combinations tried were right and left eye winks, as well as blinks and jaw clenching.
The electrode placements used in the final project were FP1 and T6 electrode locations on the international standardized system of EEG electrode placement (one on the left lateral frontal bone to pick up blinks, and the other on the right temporal bone to pick up ear movement). As shown in the BCI recording video, the respective electrodes showed isolated activity in their own channels when a blink or ear movement was made.
The signal acquisition/processing part of the software picked up the signals from these movements and applied a bandpass filter of 1-50 Hz and a notch filter at 60 Hz to remove environmental electrical noise. A moving average for both signals was used in order to make the blank signal as smooth as possible. The peak detection portion of the software then identified a threshold for which amplitudes (in mV) qualify as signals in either channel.
The Morse-to-text part of the program then converted these binary signals into lines and dots corresponding to Morse code. However, traditional Morse code proved ineffective due to difficulty in distinguishing between words and letters of different lengths – in physical Morse code (tapping), there are precise lengths of pauses to denote separate letters and words. This would have been difficult to program and keep consistent on the user side, so a 5-symbol Morse code system was established instead. Each letter – as well as spaces and periods – would be a unique 5 symbol combination of dots and lines so that timing of the signal would not need to be factored. This was changed in the Morse-to-text part of the program, and then delivered to the standard text-to-speech section, which reads aloud the previous text at the end of a sentence (when a period is seen).
Over the course of the experiments, the interface received a reasonable level of reliability/reproducibility (approximately 60%) with the muscular signals. The user would also need to be familiar with the 5-symbol Morse code system, which requires a learning period to retain. Though some exploration with SMR signal acquisition was done, no pronounced features were recorded with either the OpenBCI GUI or the Visual C# software version. More practice/experience with this device in the future would likely yield higher efficiency in communication and more reliable and discrete signals in each channel.