This semester I’m in a course called Body Electric, taught by Conor Russomanno. Conor is a co-founder of OpenBCI, an open source EEG (brainwave-sensing) platform that is the focus of our class this semester. In the class we’ve been learning about human-computer interface more broadly, but our task over this seven week course has also been to develop a creative, interactive application for the OpenBCI. For the past few weeks, Mathura, Wangshu, Christina, and I have been hard at work on our project: DarkMaze.
It took quite a bit of deliberation to arrive at our concept for the project. We started off with the capabilities of the OpenBCI, which can sense EEG, EMC, ECG, and also comes with an onboard accelerometer. We also knew that we wanted to build something physical to interface with through the OpenBCI, since a tangible form is a much more satisfying interaction than a merely digital one. I was also interested in the brainwaves associated with various dream states as well as sensory deprivation states, but the clunkiness of the OpenBCI Ultracortex was bound to interfere with sleep and sensory deprivation. The Ultracortex, the 3D-printed headset that the platform rests in, and the electrodes are not very comfortable for long term use.
The problem with EEG of course is that brainwaves are not readily available for conscious manipulation. Event-related potentials are a fairly accurate way of predicting reactions (such as p300), and audio-visual entrainment can also be employed to match brainwave frequency to the frequency of audiovisual input, but neither of these seemed like robust interactions to us. We decided instead to use EEG, EMG, and accelerometer data all together, while keeping everything located around the headset as opposed to breaking leads out for ECG or reading EMG of the limbs. For EEG data we decided to have focus be our parameter, defined by high alpha and low beta waves.
The idea we settled on was a physical maze. The maze would be dark, obscured by a one-way mirrored acrylic that only lit up and revealed the details of the maze once the user demonstrated focus through EEG. The user would control the movement of the ball in the maze through a combination of eye blinks (left and right) and head tilts (up and down). We titled the project DarkMaze. I pictured something sleek and dark and futuristic, just as mind-controlled things ought to be. Our idea was that DarkMaze would be simultaneously training focus and memory – you’d have to be focused to reveal the maze, and then either maintain focus while you navigated it (which was likely very difficult) or remember the layout of the maze and navigate it blindly.
Our original concept was to have electromagnets pull a metal ball through the maze. We played around with a few magnets around the shop and did research into buying and creating our own electromagnet. We’d need something pretty strong, since it would need to pull a ball from a distance of about 10 inches (which we surmised would be the size of the maze). You can make some pretty strong electromagnets at home (check out this one made from a microwave transformer), but even very strong magnets don’t have a very far reach. Sure, it might be able to lift over a hundred pounds, but that weight has to be pretty close to the magnet. So we scrapped the magnet idea. We toyed with a few other ideas – a magnet mounted to a CNC-type arm, a magnetized toy car on a track, a system using a ping-pong ball and pneumatic pressure, etc. – but in the end we decided the simplest and most effective way would be to create a tilt maze. That we should be able to control with two servo motors.
Next we had to record some data using the various inputs (alpha, blinks, and tilts) so that we could work independently of the OpenBCI. Plus, I’m the only one whom the Neocortex fits, and my brain can only be in so many places at once.
Wangshu’s head is definitely not big enough for this Neocortex
So my head was the chosen one. We found that I can achieve alpha waves with much more facility than others we’ve tested on. I’m guessing this is because I do a lot of yoga, or maybe that I’m just generally a pretty relaxed person. We tried a few methods to pinpoint my focused state, including having me read and having me focus on an object. Which happened to be the banana I had on had.
There is no banana
It’s interesting trying to describe my mental state when trying to attain focus. When focusing on the banana, for instance, I sort of fixate on it and its contours, and let the rest of the world fall away. It’s just me and the banana.
Anyway, we got nice recordings of all three inputs so that we could work independently. Mathura and I took the eye blinks, Wangshu took alpha, and Christina took the accelerometer. We were to reconvene a week later to put everything together.
Mathura and I got together a few times, working with the recorded data and live data from my eye blinks. We had to repurpose the EMG processing and visualization already in OpenBCI to make it work for us. Instead of the EMG visualization, we drew a ball on the GUI whose X position we tied to eye blinks. We set thresholds for channels 1 and 2 – the channels reading the electrical signals causing my eyes to blink/wink – and had them switch the state of booleans (moveLeft and moveRight). We had to muck around with the thresholds and frame rates for quite a while to get the ball moving properly. We also had to distinguish blinks from winks. I’ve been using the terms somewhat interchangeably here, but it was important that only a significantly strong one-eyed wink change the X position, and not, for instance, a two-eyed blink in which the left and right eyes were slightly offset, for instance.
Another thing we had to account for is my left-right dominance. It’s much easier for me to wink my right eye and in fact, when I blink my left, my right eye closes slightly too. Some funny nerve crossing, I guess, or simply a matter of weakness. I never wink with my left eye, so I’ve been trying to train myself. But we made a workaround by lowering the threshold for my left eye. Anyway, in the end we got it to work pretty well!
When we got back together the next week, Wangshu also had some code that really faithfully read when my recorded brain went into alpha. It worked just as well when we tested it with live data, too! Christina wasn’t able to get to the accelerometer data, unfortunately. It took her and Mathura the greater part of the day to get to the data and make it change the Y position of the GUI ball. I’m not actually sure how they ended up doing it, because Wangshu and I spent that time making a virtual mockup of the maze in Unity.
Wangshu found a really great Processing “robot” that performs keystrokes. We attached those functions to our left-right and alpha booleans and aimed to have these control a Unity sketch. Wangshu knows her way around the program pretty well, so I watched over her shoulder and gleaned as much as I could. It doesn’t seem too complicated, really! Anyway, we set up a basic maze and turned off the lights in Unity, to have them activated by the alpha wave readings. It actually worked like a charm, and by the time we had Unity up and running we were finally able to get the proper accelerometer readings and join the three separate codes together. And it worked!
This is me winning the virtual DarkMaze:
We planned to bring this digital prototype to the next class, which would take place at the Neural Hacking meeting. This was our first change to get user testing from someone other than me, which was a really valuable experience.
Nick had a little bit of a difficult time with DarkMaze at the beginning, but he got the hang of it.
This guy was a pro from the get-go!
He had so much control with our interface that Mathura altered the code to let him just draw with the GUI ball we created a while back.
Anyway, it was really exciting to test our prototype on real people and have it work so well. I think our testers were really excited by our project. The meetup was really cool aside from that. Getting to meet and chat with brain enthusiasts from all over was really inspiring to me, and I definitely picked up some inspiration from the evening. I mean, who can resist shooting lasers into someone’s brain? (This was my first time hearing about low-level laser therapy.)
So the next step for DarkMaze is to make it physical. We’d been planning out how to go about this – buying materials, designing the maze, fabrication, circuitry, etc. – when Wangshu pointed out we could perhaps just buy a readymade tilt maze and hack it. This idea appealed to all of us, knowing that we were pressed for time with only a week left until we have to show our final projects. So today we ordered a readymade maze and laser cut a prototype labyrinth. We already have one-way mirrored acrylic and superbright LEDs to illuminate it. We have servos and breadboards and Arduinos, so we seem to be in pretty good shape. We’re just going to laser cut our own labyrinth (so that it’s not too difficult, which the readymade one is) and glue it into the frame of the maze. The servos and the programming may take some time, but with a week left and our combined skills, I’m confident we can pull it off. And I gotta say, I’m really proud of where we’ve gotten with DarkMaze as it is. The final product is going to be awesome.
You can find our code on Github, here.
Leave a Reply
You must be logged in to post a comment.