How are the csv files encoded?
alias123
california
in OpenBCI_GUI
I made a bit of a mistake in analyzing my OpenBCI headset data in Python: I did not specify the encoding type when doing pd.csv_read. I read them in, processed them, and saved them as .npy files in a directory.
I run the .npy files through a machine learning algorithm to look for patterns and have found when I run them on OS X I get ~70% accuracy and when I run exactly the same code on the exact same data on Windows 10 I get ~35% accuracy. I'm troubleshooting now and one of the things it might be is that the encoding of the data is causing problems, which leads me to my question:
How are the CSV files that OpenBCI outputs encoded?
Comments
Alias, hi.
The CSV files produced by OpenBCI_GUI have a header comment showing the format. Can you explain how your .npy files are generated? Have you looked inside the original file produced by the GUI?
The GUI CSV file format changed with the new v5 GUI. Prior to that another, simpler format was used.
Regards, William
Hi William,
Sorry my original question wasn't totally clear. I see the way the data is setup (e.g. the headers/columns) when I look at the raw CSV file in say Notepad. In Python you typically import files with a "encoding=utf-8" or something similar when opening CSV files. I'm curious if utf-8 is indeed the format that the OpenBCI GUI saves the CSV data as. I am using V5 GUI for reference.
Normally what I'd do is simply re-import the data CSV files, but as luck would have it I purged all of them after I processed them.
As for how I generate my .npy files:
utf-8 is a common encoding, but is identical with plain ascii text when no special Unicode characters are involved. So generally the 'encoding' you used for importing should not matter much. Encoding is only critical with "special characters" outside normal ascii.
https://www.google.com/search?q=ascii+vs+utf8
So if you know the original CSV data format, I still don't get your question. Don't you have all the info you need in your .npy, provided you interpret it correctly??
None of the GUI CSV file formats (v5 or earlier) ever used Unicode characters. They are all just plain 7-bit ascii files.
Hmm ok back to the drawing board to figure out why identical code+data is performing 2x better on one operating system vs. another
Thanks for your quick responses William!
Update
The source of my huge reduction in training accuracy was that I had inadvertently updated from Tensorflow 1.15 on the first system to Tensorflow 2.x on the second system.
After some poking around, I found that TF2.x made "eager execution" the default and for whatever reason that was breaking my model. I used the following code to disable it and my accuracy went back to where it was beforehand:
tf.compat.v1.disable_eager_execution()