By adding a third controller via EMG signal by forearm flexing, user can signal his/her intent on different gestures.
https://docs.openbci.com/docs/07Examples/18-EMGProjects/EMG_Chrome_Dino_Game
This can overcome problems of unintentional gesture signals and allow wider range of gesture controls in games.
————————————————————————————-
knowledge needed
-python to create a python server -> https://pymotw.com/2/socket/tcp.html
-c++ to create an unreal engine tcp client -> https://docs.unrealengine.com/en-US/API/Runtime/Sockets/index.html
-familiarity with unreal engine blueprints
-familiarity with oculus quest 2 app development
hardware required
-oculus quest 2
-oculus quest link cable
-computer (run unreal on this)
hardware setup
-follow the dino example in open bci website and set up -> https://docs.openbci.com/docs/07Examples/18-EMGProjects/EMG_Chrome_Dino_Game
link open bci cython board to unreal engine
-add a python tcp socket server functionality in the python script found in the dino example
-create an actor component in unreal engine c++, add a c++ tcp socket client to listen to the signals from the python side
-from the dino example, you are able to get 2 signals (dino jump, dino dont jump), after creating the link, your unreal engine project should be able to listen to these signals
the steps below onwards are only to support the features as shown in the video -> https://youtu.be/9AAbZETRIpw
gesture setup
-allow gesture recognition oculus quest on unreal
-record and save your intended gesture (open hand/close hand)
-load your saved gestures to use them when running unreal
link emg signal from c++ unreal tcp to unreal blueprint
-create a (actor component) c++ unreal event and broadcast signal
-use this emg signal to form a controller together with the hand gestures recognized
the steps below onwards are set up in unreal blueprints and can be varied according to your use case
emg signal with gesture recognition
-on emg signal up + hand open -> activate object movement and rotate
-on emg signal down + hand open -> deactivate object movement and rotate
-on emg signal up + hand crush -> activate object crushing
-on emg signal down + hand crush -> deactivate object crushing
note* without the emg sigal, we can only have 2 out of 4 signals to control the game
on emg signal up + hand open
-have the targetted object(box) be parented to the wrist of any hand and offset to its original position
-this will allow the object to rotate or move according to the behavior of the wrist
on emg signal down + hand open
-unparent the targetted object
on emg signal up + hand crush
-have the targetted object(box) be parented to the wrist of any hand and offset to its original position
-save the position of both left hand and right hand
-on event tick, feed the distance between both hands the crush functionality of the object
on emg signal down + hand crush
-unparent the targetted object
-on event tick, stop feeding the distance values between both hands to the crush functionality of the object
the steps below onwards depends on the 3d software you use – blender/maya/3ds max/houdini
crush functionality
-create a crush version (using point and click 3d vertex translations or cloth simulations) from the targetted object(box)
-import that crush version object into unreal
-set up morph target in blueprint of the normal object with the crush version
-have a function to receive (distance values from both hands) to control the morph values
you can contact me if you have more problems
Hi Hansoo — this video looks really incredible.
I’ve published this post in “private” mode for now, which means only the author (you) and OpenBCI admins (me) can see it.
I would love to add more detail on your video, and how you created it. Tutorial content is extremely helpful for our users.
1. Are there extra details you can share?
2. Where did you place the electrodes?
3. What code did you use to get these commands into the 3D environment?
4. Are you using Unity? Unreal?
Any extra details that would help other community members replicate your work would be great to add to the post. Feel free to email me directly joseph (at) openbci.com