script to convert SD card file

armz12

Is there a script available (C, python, etc) to convert the SD card file (hex format) to voltages, etc without having to use the OpenBCI GUI?

wjcroft

Armz, hi.

See the code from lines 1461 to 1651 in,

https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataLogging.pde

You could write a standalone Processing app with just this section. Let us know if you do that with a link to your Github.

William

Samvb

Hi wjcroft,
I am also looking for a script to convert SD card files (hex) to decimal.
Is there a new link to the above code?
Thanks,
Sam

wjcroft

Richard @retiutut, do you know what section of the GUI code now handles decoding of SD card hex data? The link above no longer works because that filename 'DataLogging.pde', no longer exists. I do see the file 'DataSourceSDCard.pde'...

Thanks, William

retiutut

Correct. Let me break down what happens in the new DataSourceSDCard.pde (https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataSourceSDCard.pde)!

parseRow(String[] row, int numChannels) parses the values for all channels which exist as a "row" in the CSV file: https://github.com/OpenBCI/OpenBCI_GUI/blob/4ab8cf7e024aed367e0cca07f02ad4e3b4366836/OpenBCI_GUI/DataSourceSDCard.pde#L76

parseInt24Hex(String hex) converts the values for each channel: https://github.com/OpenBCI/OpenBCI_GUI/blob/4ab8cf7e024aed367e0cca07f02ad4e3b4366836/OpenBCI_GUI/DataSourceSDCard.pde#L287

And then there is a Processing method called unhex(): https://processing.org/reference/unhex_.html

pluto

Hi,
@retiutut, Is there any python script to do so (SD Card hex to decimal). Since direct conversion is not the actual way to do so.

pluto

Hi All,
Hope you all are doing well.
1. I just saw a repo (depricated) named: PyOpenBCI, does this python script converts the SD Card hex code to decimal values? If yes, I would really like to know on how to do it?

2. Also, if I just take a raw value from the SD Card hex values, what is the way to convert those hex values into the actual decimal values (since direct conversion of those hex values did not give same results as the converted files from Processing SD Card converter).

@wjcroft, I will be grateful for your help.

retiutut

I have made an issue on the GUI Github repo to address this.
https://github.com/OpenBCI/OpenBCI_GUI/issues/1014

pluto

Thanks for the help @retiutut.
As you have mentioned above, the use of parseInt24Hex(String hex), under that I could see one unhex function, can you describe what does it do?

pluto

Hi,
I have two files with me, SD Card log, and converted of the same, let us take a value in SD Card log for example:
F260C9, this is converted as: -19954.006, HOW?
How do I know, how this is converted into that particular value?
Can you tell any simple method to interpret this, not specifically any code, any simple explanation is welcome enough.

@wjcroft, please help me out on this, kinda urgent.

Thanks & Regards,
Pluto

retiutut

https://processing.org/reference/unhex_.html

wjcroft

@pluto, hi. I merged your new thread into this existing one. It helps to put related questions into similar threads.

https://docs.openbci.com/Cyton/CytonDataFormat/#24-bit-signed-data-values

William

pluto

Thanks @wjcroft @retiutut, I could figure out the conversion while looking at the conversion file and then comparing with original hex, got this tip from firmware that those are negative hex, and else was easy to implement, thanks a lot for your kind support.

pluto

Hi,
I am seeing that while conversion from hex to decimal in SD Card log, there is an scale factor used.
I am aware it depends on Vref and Gain, and I have considered same as what is used by OpenBCI and having few questions about it which are as follow:

1. Till what decimal place is the scale factor used for conversion? (Since I could see while converting from python script of mine vs converted by OpenBCI gives different results from 4th or 5th decimal places, sometimes 2nd and 3rd too)

2. If possible, can you give me the exact value of the scale factor computed? (Since I actually can not print the scale factor used from OpenBCI, and this would be helpful to compare mine scale factor vs the one used)

@wjcroft, @retiutut I will be grateful for your help on this.

Thanks and Regards,
Pluto

wjcroft

@pluto, hi.

I merged your 'scale factor' thread, into this existing thread, because the general topic is still conversion of the hex file.

The calculation, including scale factor, is given here,

https://docs.openbci.com/Cyton/CytonDataFormat/#interpreting-the-eeg-data

William

wjcroft

As stated at that link, the default value is "0.02235 microVolts per count". In other words, each additional count of the low order bit of the raw sample value, adds two hundredths of a microvolt. Since EEG below a microvolt is mostly considered noise, the low order digits in the scale factor (0.00035) have low relevance.

pluto

Thanks a lot for you help @wjcroft,
I just tested it with this scenario:

    scale_factor = 0.02235
    hex_string_to_convert = FA8CFA
    Equivalent decimal value converted by OpenBCI GUI 4.2.0 = -7982.3887
    Equivalent decimal value converted by my script =         -7981.7661 (This value considers the above scale factor used)

Previously I was using scale factor without any round off, and I got results as below:

    scale_factor is:  0.022351744455307063
    hex_string_to_convert = FA8CFA
    Equivalent decimal value converted by OpenBCI GUI 4.2.0 = -7982.3887
    Equivalent decimal value converted by my script =         -7982.38909034599

In the second case where I did not round off the scale factor, the values are still different from 3rd decimal places itself, and even if I round off my value (decimal of hex) to 4 decimal place, I still won't get the same result as openbci.

What is that I am missing??

wjcroft

Pluto, hi.

The minor differences plus or minus an absolute microvolt, really don't make much difference, as previously mentioned. A single microvolt level signal in EEG is usually to close to noise level. Also, all your conversions are using the same scale value, so it is the relative microvolt amplitude levels within the same recording, that are important.

However I do note that your sign seems to be wrong in some of the results you show, unless that is a typo on your part.

Richard @retiutut may want to make a note that the docs page showing the sample 24 and 16 bit conversion code, has a subtle typo. That likely got introduced in one of the docusaurus hosting conversions. This is a link showing what the code samples used to look like:

https://www.titanwolf.org/Network/q/a46315a4-d0c9-44e0-a7b4-0f6749094598/y

Whereas the current example shows <! instead of <<:

https://docs.openbci.com/Cyton/CytonDataFormat/#24-bit-signed-data-values

Regards, William

pluto

Hi William,
Yes that was a typo from my side of missing -ve sign in openbci conversion. I have corrected that now.

Considering about the values, since we have once source file for the log (from SD Card), and the values are pre-recorded, the conversion should provide me the same results, since it is not realtime deal with EEG signal.
Let us consider the value as just a value. And particular conversion of some value should always give me same result, not different right? Like for example FF will always give me -1 everywhere.

What I mean to say is, if we have one particular value (let us take for example: FA8CFA), knowing it is negative hex it will always have one particular value, considering till what value we take the deicmal.

From my manual calculation, I am getting the exact same result as my script, which is not matching from 3rd decimal of OpenBCI.

So how can both values differ? (Considering at the end both are just hex values)

Regards,
Pluto

wjcroft

Do you understand what I am saying about the extremely low relevance of single microvolt or sub microvolt EEG changes? Use whatever conversion factor you desire. If you can find one that matches exactly the GUI conversion, cool. But don't sweat about single microvolt or sub-microvolt differences. EEG is just noise at that amplitude level.

I suggest consulting the OpenBCI source code if you want to see exactly what steps were used to derive the scale value.

macoffee

I would like a python script to convert hexadecimal data (recorded in Cyton) recorded on an SD card to decimal.
Thank you in advance.

wjcroft

Hi @macoffee, I merged your new thread into this existing thread on the same subject.

See some of the previous comments. I believe there is still an issue open on this subject,

https://github.com/OpenBCI/OpenBCI_GUI/issues/1014

Richard @retiutut may have further comments. If you look at previous comments above, you can see the source code link,

https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataSourceSDCard.pde

@retiutut said:
Correct. Let me break down what happens in the new DataSourceSDCard.pde (https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataSourceSDCard.pde)!

parseRow(String[] row, int numChannels) parses the values for all channels which exist as a "row" in the CSV file: https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataSourceSDCard.pde#L76

parseInt24Hex(String hex) converts the values for each channel: https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/DataSourceSDCard.pde#L287

And then there is a Processing method called unhex(): https://processing.org/reference/unhex_.html

max_eastwood

I've made a python script sd_convert.py, you can modify it for your needs:

https://github.com/roflecopter/openbci-session

wjcroft

Thanks so much @max_eastwood ! I'm hoping that your script can be more widely promoted. @retiutut.