OpenBCI to MNE Python ?

felipev · October 2019

Hi everyone ?
Someone does know how to use this script with a raw or BDF file? --> https://github.com/OpenBCI/OpenBCI_MNE

I had more luck using MNElab but it just give me the option to get a time/frequency plot but I need topoplots ?

wjcroft · October 2019

Mentioning Daniel Rivas, @drivas, who authored some of the commits on the MNE repo you mention.

William

drivas · October 2019

Hi @felipev!

I have not tested OpenBCI's BDF output since it was broken when I started working with OpenBCI around last year, but if it now outputs correctly formatted BDF, then MNE's mne.io.read_raw_bdf called with the path to your data file should return to you a full mne.io.Raw object.

As for my patch for the .txt output files, I believe it still has some bugs, like assuming a sample counter looping until 128 whereas it could also be 255, and not dealing with the new (since about a year) human-readable timestamp column. Its original implementation also included an interpolation fix for dropped samples that was buggy since it assumed 1) only single samples are dropped (not multiple samples in a row) and 2) that sample order is preserved, which I have shown can be false. I might get around updating it on GitHub.

For now, I suggest to try simply saving in BDF and opening the data with read_raw_bdf

retiutut · October 2019

@drivas BDF should be fixed!!! Give it a go!

jamaker · November 2019

How can I set BDF export in the OpenBCI GUI? I couldn't find any export options.
thanks!

edit: Found it. I was looking with synthetic data selected...

retiutut · November 2019

Click the BDF+ button in the Control Panel

retiutut · November 2019

griffinmilsap · April 2020

Hey @retiutut, @drivas,
I've been doing a bunch of development recently on the OpenBCI platform, and I typically use mne as a file importer for my python analysis scripts. I've noticed that using the default configuration for mne's read_raw_bdf produces corrupted results. There may be a way to change the parameterization to fix this, but I wanted to post my findings so that anyone using the bdf export functionality with MNE knows about this issue:

I made two recordings on my Cyton V3-based headset (November 2019 OpenBCI GUI) where I relaxed with my eyes open, back-to-back with no system reset or cap refit. Occasionally I closed my eyes to verify the presence of alpha in the signal (which was clearly visible in the timeseries). One of these recordings was saved into the .bdf format, and the other was saved into the OpenEEG RAW txt format. During both recordings, the data looked quite stable like reasonable EEG during the whole recording (except for the recording that was saved into the RAW format, wherein the signal dropped for inexplicable reasons about 25 seconds into the recording. I first opened and visualized the .bdf file (which had repeatedly been giving me problems)

bdf_fname = 'OpenBCI-BDF-2020-04-24_15-43-17.bdf'
data_file = os.path.join( data_dir, bdf_fname )
cnt = mne.io.read_raw_bdf( data_file, preload = True )
cnt = cnt.pick_channels( cnt.ch_names[ :5 ] )

# Convert data to xarray for further analysis
data = xr.DataArray( cnt.to_data_frame() ) \
         .assign_coords( time = np.arange( len( cnt ) ) / cnt.info[ 'sfreq' ] ) \
         .assign_attrs( fs = cnt.info[ 'sfreq' ] )

fig, ax = plt.subplots( dpi = 100, figsize = ( 10.0, 4.0 ) )
traces = ax.plot( data.time, data )
ax.legend( traces, data.channel.values )
ax.set_xlabel( 'Time (sec)' )
ax.set_ylabel( 'Voltage (muV)' )
ax.set_title( bdf_fname );

This output looks very corrupted, but I can't verify that its a problem with the bdf file or the way that MNE reads the bdf files that OpenBCI GUI produces. I opened the file in EDF Browser and it looks similar, leading me to believe there's an issue with the way that OpenBCI produces the BDF outputs.

I then looked at the RAW .txt file:

obci_fname = 'OpenBCI-RAW-2020-04-24_15-42-29.txt'
data_file = os.path.join( data_dir, obci_fname )

with open( data_file, 'r' ) as in_f:
    header = True
    while header:
        line = next( in_f )
        if line[0] != '%':
            header = False
    data_arr = [ np.array( line.strip( ' \n' ).split( ', ' )[1:6] ) 
                 for line in in_f ]
    data_arr = np.array( data_arr ).astype( np.float )

# Convert OpenBCI data to an xarray because I like these data structures
data = xr.DataArray( data_arr, dims = [ 'time', 'channel' ] ) \
         .assign_coords( time = np.arange( data_arr.shape[0] ) / 250.0 ) \
         .assign_coords( channel = [ 'EEG %d' % ( i + 1 ) 
                                     for i in range( data_arr.shape[1] ) ] ) \
         .assign_attrs( fs = 250 )

fig, ax = plt.subplots( dpi = 100, figsize = ( 10.0, 4.0 ) )
traces = ax.plot( data.time, data )
ax.legend( traces, data.channel.values )
ax.set_xlabel( 'Time (sec)' )
ax.set_ylabel( 'Voltage (muV)' )
ax.set_title( obci_fname );

This looks much more like I would expect the data to look -- There are still significant DC offsets, as this data is unfiltered, but it appears to represent the EEG much more faithfully (until the signal dropped, at least)

I suspect there's still issues with the BDF output, or at least the way MNE treats the OpenBCI BDF files
-Griff

retiutut · April 2020

When I said it was fixed above, I was referring to the files being valid. Corrupt files were being produced.

griffinmilsap · April 2020

@retiutut,
I see - all good! I had seen https://github.com/OpenBCI/OpenBCI_GUI/issues/441 and noted that it had been closed, and all indications that BDF output was broken seemed to be reversed as of the November 2019 release -- I didn't know if this was still a known issue, as I couldn't find an open issue on the GUI github page. I can make an issue there, but it might be worth just deprecating the BDF output anyway - it's probably a nightmare to maintain and the OpenBCI CSV files are much easier to work with.

Huge props for this GUI by the way - it has made signal debugging and visualization of my signal so easy -- I wish other EEG devices had such comprehensive visual output

-Griff

retiutut · April 2020

@griffinmilsap I will surely investigate BDF mode again once the core of GUI v5 is done. We will still have OpenBCI Data Format (CSV) as an option, and this will be tested before seeing if BDF is working (per your post earlier) and worth keeping moving forward.

In your pictures, I think there should be more than 5 channels of data if you are using Cyton. Not sure why this is happening. Even when channels are turned off, it still saves a value of 0 to file.

I just tested making and reading a BDF file using GUI v4.2.0 standalone on Windows 10, and it looks ok in the EDF Browser:

__

Here is a recording I made just now using Ganglion+WiFi (30 secs nothing attached, ~30 secs messing with pins):
https://gofile.io/?c=hc9lMB

Can you try EDF Browser and MNE with the above file?

Take care,
RW

griffinmilsap · April 2020

It looks like that recording is faithfully imported using MNE, as well as EDF Browser.

bdf_fname = 'OpenBCI-BDF-2020-04-25_16-14-37.bdf'
data_file = os.path.join( data_dir, bdf_fname )
cnt = mne.io.read_raw_bdf( data_file, preload = True )

# Convert data to xarray for further analysis
data = xr.DataArray( cnt.to_data_frame() ) \
         .assign_coords( time = np.arange( len( cnt ) ) / cnt.info[ 'sfreq' ] ) \
         .assign_attrs( fs = cnt.info[ 'sfreq' ] )

gain = 200000
fig, ax = plt.subplots( dpi = 100, figsize = ( 10.0, 10.0 ) )
offsets = data.channel.copy( data = np.arange( len( data.channel ) ) ) * gain
traces = ax.plot( data.time, data + offsets )
ax.set_xlabel( 'Time (sec)' )
ax.set_ylabel( 'Voltage (muV)' )
ax.set_title( bdf_fname )

scale = gain / 4
tick_locs, tick_labels = zip( *[ ( [ off.item() - scale, off.item(), off.item() + scale ],
                                   [ str( -scale ), str( ch.item() ), str( scale ) ] )
                                 for off, ch in zip( offsets, data.channel ) ] )
tick_locs, tick_labels = tuple( [ functools.reduce( lambda a, b: a + b, l )
                                  for l in [ tick_locs, tick_labels ] ] )

ax.set_yticks( tick_locs )
ax.set_yticklabels( tick_labels )
ax.grid( True )

ax.set_xlim( 40, 50 )

This is the same stretch of data plotted in EDFBrowser and it seems to match up very well.

So a couple of thoughts. My Cyton BDF file does contain 8 full channels of data, I just decided to plot the only channels that were connected (I'm only using 5 right now). My broken BDF file does actually contain zeros all the way across channels 6, 7, and 8 (mostly -- there was an electrical pop that hit those channels briefly too).. ALSO You mentioned you recorded in GUI 4.2.0 but I'm only using GUI 4.1.7 (if you can think of a bug you fixed between the versions)

The corruption in my remaining bdf channels looks a LOT like some sort of signed/unsigned conversion issue.. The channels appear to sit around some high or low value, then when they deviate negatively or positively, they wrap around to the opposite really high/low value creating some large square waves.. The raw OpenBCI recording is fine, and the signal in the viewer reflects the OpenBCI file so I suspect its something specific to the BDF writer.. It might be specific to Cyton given its bit depth compared to the ganglion (which was used for the sample file you gave me)..

I'll poke around the github and let you know if there's anything that jumps out to me.

Billh · April 2020

The possible errors are not just signed/unsigned, but 24 bit integer versus 16 or 32 bit integer, and bigendian versus littleendian. I would think that the MNE code would never have worked at all if those errors were present though.

Billh · April 2020

See: https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataLogging.pde#L1082

I think that although the OpenBCI boards send byte order bigendian whereas the byte order in the BDF file is littleendian, the BIT order should NOT be switched--just the BYTE order. I suggest trying the BDF writer without the call to swapByte, just doing the reordering of the byte order, not the bit order.

EDIT: I put in a PR on the Processing code, but it needs testing.

griffinmilsap · April 2020

@Billh
Hahahaha oh my I think you're on to something here! I would have never thought about endianness... I suspect you're right and I'm going to give your pull request a test today on my Apple hardware. Funny thing is if you just do a test without signals connected and rake your finger across the pins, the output with/without this corruption is probably pretty similar (hence the previous ganglion recording looking alright), but any real recording would be all over the place provided a deflection that changes the second byte.

Huge props for this suggestion/fix!

griffinmilsap · April 2020

@Billh,
I just pulled your fork and made two recordings on my setup, back-to-back. The OpenBCI file closely resembles generated .BDF file, and I can safely say that with some filtering the two files are both usable now, at least for my purposes.

In my recordings, I had channel 6, 7, and 8 "turned off" in the GUI, and in both cases there were zeros ("0.0") written to the OpenBCI file for all those channels, but in the BDF+ file written by your fork, the channels had nonzero entries: -187500 for one session, and 0.01117587 for another session I recorded to double-check this behavior. I had to reseat the cap between these sessions and my cap may have some loose connections that caused the difference between these recordings, but I'm unsure if its expected behavior for the BDF to have nonzero entries for these "off" channels. In both recordings, the static value never changes, but it is still non-zero.

I then had the thought that the absolute values that these channels assume may not be the same between the two files, and that there may be some static DC shift between the BDF file and the OpenBCI file (possibly caused by an endian-conversion-issue on an analog gain factor or something) but I couldn't confirm because my signals were drifting all over the place in both files. I don't personally care about the absolute muV values of my recordings (or really, even the relative values -- I just look for spectral changes in my use case) but someone may care about this if it is indeed an issue.

It'd be handy to be able to record/save an OpenBCI file, then play it back WITH saving into the BDF format and verify the two files contain the same numbers - this would be a good test of the BDF+ format output.

Great insight on the endianness issue!

-Griff

Billh · April 2020

The point is that the bits should never have been flipped. There might be some gain modification going on with one save format and not another, but that would not have to do with the storage to BDF itself.

retiutut · May 2020

@billh was able to identify and prototype a fix for BDF files that has now been merged into the upcoming GUI v5.

Thank you for the code contribution! This issue has been resolved!

https://github.com/OpenBCI/OpenBCI_GUI/pull/737#event-3297192891

Billh · May 2020

No problem. The BDF format is a good interchange format, since it can be more compact than CSV and is one that most EEG packages can use, so I hope that OpenBCI's version of it works better from here on.

OpenBCI to MNE Python ?

Comments