Ganglion OTA programming for Windows [new instructions]
Hi, I have a Ganglion (bought post-2017) and I'm looking for a way to program it OTA on Windows, trying to avoid the need to get myself an FTDI programmer. I went through the Ganglion programming guide under Documentation, but the OTA example given seems to be limited to MacOSx only. Has anyone here been able to program their ganglion using Windows and if so, what are the extra steps I'd have to follow, so that I dont land up bricking the board?
Comments
Tek, hi.
The docs page you refer to,
https://docs.openbci.com/docs/03Ganglion/GanglionProgram
DOES reference some macOS folder locations, to locate the 'variant.h' file and make some modifications. Actually this docs file contains a mixture of Mac and Windows instructions. It does need cleaning up to flesh out the Windows side. My impression is that the same directory tree (Windows uses backslash instead of forward slash):
.../Java/portable/packages/Simblee/hardware/Simblee/1.1.0/variants/Simblee/variant.h
[the 1.1.0 shown on the docs page, I believe is now 1.1.4]
Will be found inside your Windows Arduino folder, after you have loaded the Arduino app with the Simblee board files. Note that there are some dead links at the top of that docs page. Sorry about that. We uploaded all the Simblee related files and docs to our own tree some time ago. And I assumed the docs page would have been updated as well. We'll put that in the queue. Mentioning Richard @retiutut. Our local copy of these files are in these URLs / folders:
https://openbci.com/arduino/ [board files and such]
https://openbci.com/arduino/simblee/ [contains original Simblee documentation pdf documents]
The first folder has a json file: https://openbci.com/arduino/package_simblee166_index.json
This URL is what you point the Windows Arduino app to, in order to load the Simblee board files. That will have the effect of unzipping the http://openbci.com/arduino/simblee_253.tar.gz zip file, inside the Arduino app board-files subfolder. If you don't see this tree inside your Arduino app folder (typically in 'c:\Program Files\Arduino') it may instead be in 'C:\Users\username\Documents\Arduino\libraries'. You can also try using the Windows Start menu search box to locate 'variant.h'.
Let us know if this is sufficient to get going. And if you have any further tips or hints when you went through the upload.
Regards, William
Here is an open issue on our repo. With more background.
https://github.com/OpenBCI/OpenBCI_Ganglion_Library/issues/17
Hello William, thank you for taking the time for your detailed reply! I did come across the same issue page before and followed the helpful info in there. So I managed to get the Simblee board installed into my Arduino v1.8.5 platform via this link on the page: http://openbci.com/arduino/package_simblee166_index.json
The Simblee version that gets installed is actually v1.14 and on Windows, the default installation path is as follows (just including this for future reference, incase anyone else finds this helpful):
C:\Users\{username}\AppData\Local\Arduino15\packages\Simblee\hardware\Simblee\1.1.4\
Navigating further into \variants\Simblee\, I edited the variants.h header file in there and modified the Simblee pin defines as instructed on the Ganglion programming page: https://docs.openbci.com/docs/03Ganglion/GanglionProgram
I followed all the steps faithfully and everything is fine right until the point where I'm required to copy/replace the platform.txt file, which you'll find in the default Simblee installation path mentioned above. My concern is that the required files (mkdfuzip, nrfutil_macosx & platform.txt) that are available for download from the Ganglion page are clearly MacOSx-centric. And when I compared the compile variables in both copies of platform.txt, these lines stand out:
2 of these lines are commented out and the one in the middle is meant for MacOS.. So it's unclear how the eventual OTA zip will be created or generated specifically for Windows? Is it possible that the original platform.txt file is meant for compiling on Windows already or am I just overthinking this??
I should add that I was able to compile DefaultGanglion.ino successfully after selecting the Simblee board option in Arduino. I then followed the last step to create the final OTA file. Exported the compiled Binary (under Sketch in Arduino) and to my pleasant surprise, I did find the binary within the sketch folder, which I then renamed to DefaultGanglion.ino.Simblee
So before I attempt to upload/flash this final hex to the Ganglion OTA, are there any precautions I should take? And just INCASE I botch this up, will I be able to reprogram the Ganglion physically using a FTDI breakout later? I do not want to risk this bricking my Ganglion because I've got so many projects hanging on this only board I have :'D
Okay, I REALLY need help with this now. I finally took the FTDI route and attempted to program my Ganglion with the DefaultGanglion.ino that I compiled as mentioned earlier. The upload via FTDI breakout went through successfully and the Ganglion began blinking once it was done. Ran OpenBCI GUI, connected via BLE (not using the WIFI shield) and the GUI recognized and listed the Ganglion board correctly. However, when I started the system and ran the Impedance Check just as an initial test, it started displaying extremely high impedance values. Normally, even if the electrodes werent connected, I'd see a maximum of ~450-650 kOhms.. but now, the values are in the Mega Ohms range!! Also, when I tried to Start the Data Stream, there's absolutely no data flowing in through the GUI, the graphs are all blank.
At first, I thought this might've been due to a conflict with the compile definitions in platform.txt OR that I needed to strip the code down further to run only with BLE, since I dont have or use the Wifi shield. So, I decided to compile the GanglionNoWifi sketch that comes with OpenBCI library's examples. Please see the screenshot below:
As you can see, the compilation within Arduino and upload to the board went through successfully, the simple compile output was as follows:
No joy though, uploading it to my Ganglion yields the same problem. I have a feeling that my board was possibly flashed with an earlier version of the firmware, since this was bought in Feb 2017.. but it looks like version 2.0 of the firmware was released only by August 2017, is this right? If so, could someone please point me to the older Ganglion v1.x firmware or sketch that I can compile and upload to bring my board back to normal. I'd really appreciate a reply soon, thanks!
Tek, hi.
The following is just my hunch. I believe ALL the Ganglion firmware compiles at OpenBCI (by AJ Keller, Joel Murphy, etc.) were done with the Mac toolchain, not the Windows toolchain version. It's possible that there is something wonky with the Windows based compiler / loader, etc. -- that was not present on the Mac. Also note that original Simblee was at 1.1.0 level at the time the docs page was written. And this json points to the 1.1.4 version. The links / files in this json, is entirely from Simblee's archive, so should be correct. And one would assume that 1.1.4 would be even more refined and bug free than 1.1.0. But clearly Simblee support and manufacturing all disappeared at some point. Which is why we archived the json and files linked therein (except for the OS dependent toolchains.)
At this point it seems prudent to try the toolchain / files, saved by Shirley from the lab development machine. Unfortunately this is Mac based. Do you have access to a Mac, perhaps owned by a friend? This Google Drive link is in the issue conversation below, March 6 2019 by Shirley.
https://github.com/OpenBCI/OpenBCI_Ganglion_Library/issues/17
https://drive.google.com/file/d/1FyqoNxvTGLvanM5F23RMR9kC89zmaaKB/view
Apologies if this seems like a wild goose chase, but at this point (as you know), both AJ and Joel have left for other projects. And the engineering team at OpenBCI is focused on a next generation VR/AR product. So really that Google Drive snapshot is the only known resource for Ganglion firmware.
Regards, William
See this forum thread. The author successfully built on Linux, and his resultant upload worked.
https://openbci.com/forum/index.php?p=/discussion/1880/anyone-built-the-firmware-for-the-ganglion-on-linux-nrfutil
Thank you once again for your prompt responses & helpful instructions, William!
Yes, I suspect the same with the Windows compiler/toolchain but I don't think the problem is with the Simblee library version#1.1.4 because if it was, then I don't think I would've been able to have compiled within Arduino successfully on Windows at all.
Fortunately I do have access to Mac, infact I'd converted my PC into a dual boot Hackintosh + Windows, so thankfully I have the best of worlds here I have already been digging into the Mac toolchain and comparing the required defines as found in the respective platform.txt files, before I attempted compiling on my Mac. The one thing I must mention that I didn't notice earlier is that the actual nrfutil.exe file was missing from the Arduino15 toolchain/package that Shirley had uploaded to the Drive, but that doesn't surprise me since that toolchain is centered around Mac, like you said. However, this exe does show up in the original Simblee package (downloaded via the json) & in the OpenHAK repository as well.
Oh and another thing: the platform.txt inside the same package mentioned above is actually similar to the original platform.txt that gets downloaded from the Simblee/json.. it does NOT seem to have the extra compiler definitions or compile patterns as seen in the other platform.txt that's meant to compile for Ganglion OTA. So was Shirley's version meant to flash directly to the board via FTDI? Her preferences also point to Upload using Programmer, by the way.. if so, which programmer is this now?
Finally, I used this very same Mac package to try & compile the simple DefaultGanglion sketch on MacOSx. I didnt have to edit the variants.h file since the Simblee pins were already redefined within her Library15 for Ganglion. However, even though the compile 'looks' successful in Arduino, this error pops up repeatedly:
Invalid library found in /Users/teknomage/Documents/Arduino/libraries/Simblee: no headers files (.h) found in /Users/teknomage/Documents/Arduino/libraries/Simblee
Looks like I'm going to have to backtrack a bit and maybe refresh the Simblee library/package before I attempt compiling on my Mac again. Will keep you posted..
Oh, I dont mind playing around with code & all the compilation variations, I just wish I had a clearer view of the end in sight, lol. And I completely understand the current situation, William.. which makes it all the more admirable how you've been holding down the fort in the absence of others. I'm sure there are other users like me who are very grateful for your timely & helpful guidance and all the knowledge you share with us, especially here on the forums!
I'm gonna keep digging and will report here as soon as I make some progress. Cheers!
Quick Update: I managed to get it compiled on my Mac and then flashed OTA to the Ganglion successfully via nrfToolbox on Android. Simple differences I noticed when exporting the binary from Arduino: the target zip file is created correctly within the Arduino sketch folder, just as it should be on MacOSx. But on Windows, only the .hex file is generated, not the required zip with its constituent .bin, .dat and manifest files. Even if you were to zip that hex file and attempt to upload it OTA, nrfToolbox doesnt allow it as a valid upload, you'll get this => Upload Failed: DFU File Error (even though the documentation states that it does accept hex files for backward compatibility)
I do think this should be mentioned or corrected on the Ganglion programming page.. and the fact that programming the board on Windows is not viable as of now.
Would the Windows upload work using the FTDI friend / hardware method? Would the Linux toolchain version do any better? What steps are missing from the Windows build script?
I really wouldnt be able to say, Williams.. this is the first time I've ever worked with Simblee. What I do know for sure is that, with the current toolchain, OTA programming for Ganglion on Windows doesnt work as it should and FTDI does so only partially. What I mean by partially is this: Let's say I were to modify the DefaultGanglion sketch and insert my own code after the Ganglion routine within the main loop(), for e.g., a simple blinking light routine, involving any one of the digital pins from the Simblee breakout terminals. It will compile on Arduino without any errors but If I were to upload it directly to the Ganglion via FTDI, MY section of the code will work just fine but the streaming of data from Ganglion doesnt work as expected (if I were to run the OpenBCI GUI and test the board, the impedance values are totally out of whack and I'm unable to initiate the data stream at all). As for OTA, forget about it.. when you export the binary from Arduino, it will only create the .hex file but not the required zip archive that can be accepted by nrfToolbox & uploaded to the Ganglion. This is what happens on Windows.
On my Mac, from what I see so far, I am able to compile and upload the modified code successfully either via FTDI or OTA. However, when I test my Ganglion with OpenBCI GUI, the impedance values are within the correct range BUT the actual data streamed is being totally RAILED, and that's even when all the electrodes are connected properly to the board. What could be causing the data to rail like this?
I must mention this Railing happens ONLY when I modify the firmware within Arduino to include my own code (even though I leave the Ganglion routine untouched). If I revert to the original DefaultGanglion without any changes, then the Ganglion works as normal when I upload the firmware either via FTDI or OTA.
What about this previous thread, which succeeded with Linux?
re: even the Mac build fails, if any mods are included
This is with Shirley's zipped build folder?
re: [on Mac] "If I revert to the original DefaultGanglion without any changes, then the Ganglion works as normal when I upload the firmware either via FTDI or OTA."
Then is it possible your mods are suspect? Are you changing any of the pin mappings, for example? Or doing something that alters the timing of the main loop? What if some resource is really tight, and even your 'simple' mod is throwing something over the edge / off?
I'm glad you managed to get your Ganglion working with the default code. I don't believe more than a couple people have attempted modifying the Ganglion. Can you try a Private Message to the author of the Linux thread?
William
I figured out the problem, and it was such a noob mistake on my part, lol ** facepalm **
My modded code had the Serial() running for debugging purposes, which would print straight to the Serial Monitor.. but what was not obvious to me earlier was that my Serial calls were interfering with Simblee's serial transmission itself. Once I disabled the serial calls in my code and uploaded the mod, everything works fine now.. i.e., Ganglion is transmitting correctly AND my section of the code is functioning properly as well. These mods were compiled and uploaded from Mac, but still no joy with Windows.
And yes, I used Shirley's zipped folder although when I come to think of it, the Library15 package I have now on my Mac is actually a combination of the original Simblee package (initially downloaded within Arudino via the json), but I later MERGED Shirley's package with my local Library15 (instead of replacing it). Shirley's platform.txt needed modifying as well, and I mentioned that on the linux thread too. Anyways, I'm including my copy of platform.txt below (with some additional comments & edits), just incase this is useful for anyone else in the future. Whoever wants to use it, simply replace your local copy of platform.txt with this instead, then restart Arduino and compile away!
Great!!
If you happen to have a Linux partition there, would be instructive to see if that works any better than Windows. The other Linux related thread claims it worked for him.
William
Thanks William! I dont have a Linux partition but I have no doubt that it would compile just fine on Linux. I am more interested in getting it to work on Windows.. and to that end, I have more GOOD news: I finally got it to compile and work successfully on Windows as well! The trick on Windows is that, even though the Arudino preferences.txt may reside inside the root Arduino15 library (under %AppData%\Local) sub-folder , the actual packages are being stored within the Arduino's installation folder instead, specifically under: %ArduinoInstallPath%\data\packages
I got the initial hint to this from the Hardware path mentioned within preferences.txt as seen below. So, in my case, my Arduino installation path is => C:\Arduino\v1.8.5\
last.ide.1.8.5.hardwarepath=C:\Arduino\v1.8.5\hardware
The second hint I got that confirmed this was when I attempted to compile my mod within Arduino and a temp path showed up in the compile log. This might be different for those who follow the 'portable' approach, but I didnt want to take that route because I have many other libraries in my main Library for other projects I work on and I didnt want to have to move them around. So what I found out was that the temporary Build (preceding the actual compilation) seems to go into Temp folder, for e.g.:
%AppData%\Local\Temp\arduino_build_838649
, and this temp Build folder seems to change every time you Compile or Upload.When I dug deeper into this Temp build folder, I realized that the required build package was pointing to the main Arduino installation folder, NOT the Arduino15 folder that's found within %AppData%\Local. I do believe this ought to be corrected or atleast mentioned in the Ganglion programming section for Windows
Anyways, when I checked my main Arduino program folder, sure enough I found another untouched, barebones Simblee package sitting under:
%ArduinoInstallPath%\data\packages
Once I identified this different package source, I replaced it with the more complete Simblee package I had under %AppData%\Local\Arduino15. Replaced it entirely, including the platform.txt as well (with the modified one mentioned in my previous post).
After this, when I compiled and then Exported the binary from within Arduino, it worked like a charm! The Zip archive for OTA is being created perfectly with the required .bin, .dat and manifest files contained within, just as they should be. And uploading to Ganglion either via FTDI or OTA works perfectly fine now on Windows with either method.
By the way, I also realized from the compiler defines (within platform.txt) that we could manually generate this zip file from the hex file with nrfutil.exe, even if the Zip file generation were to fail for any reason. You'll have to first navigate to the folder containing nrfutil.exe and/or mention the requisite paths to your sketch/hex file, but the basic command is as follows :
nrfutil.exe dfu genpkg --application "DefaultGanglion.ino.Simblee.hex" "DefaultGanglion.ino.Simblee.zip"
Run this inside Command Prompt and once it's done, you'll find the target zip file within your sketch folder.
Finally.. if you feel that a copy of my Library15\Simblee package folder would be useful to anyone else in the future, I could upload an archive of it to my google drive and share it with y'all. I'm just glad I finally got this FTDI/OTA compilation method to work on Windows, phew!
Tek, that is great, thanks so much for sleuthing this out. Fantastic detective work.
One request. Your description above has all the details, but is a bit hard to follow as an actual change to the docs page. Assuming that a future user who is building on Windows, uses the json Board Manager file we have on our server (not the macOS folder that Shirley made) -- can you make a numbered list? Which would indicate additional build steps / adjustments needed to get the proper files moved to where they need to live and build on Windows?
Thanks again.
Sure William, I'd be glad to. Will do so once I find some time during this week..
@teknomage, hi. Resurrecting this old thread. Above I asked:
Would you mind posting such a list of steps?
Thanks, William
@wjcroft: Hey William, I know this comes much later but let's just say this year has been INSANE for me, for lack of a better word. Anyway, I finally got down to documenting the steps required to build from source and program Ganglion for Windows OTA specifically. I have included a doc file with all the instructions required. If you wish to upload this to the Ganglion documentation section for Windows, then there maybe some terms or sections that may need editing. Please feel free to do with this document as you wish, I just hope it proves to be useful to other users who might want to modify their own Ganglions on Windows. Cheers!
https://docs.google.com/document/d/1aT_bjucwLBBR9oSzh72mQ143XScDjKm-/edit?usp=sharing&ouid=112967961883656547419&rtpof=true&sd=true
AJ / teknomage,
Hey thanks so much for putting this wonderful document together and sleuthing out the steps to get Windows OTA working (!)
Richard @retiutut and @Shirley, can we upload this document to our Simblee programming pages? While the link above may work now, it may not be guaranteed to stay around indefinitely.
Best regards, William
My pleasure, William. By the way, in the doc, I have also included a Gdrive link to my copy of the Simblee package for Windows, comes with the platform and variant files already edited, plus a newer version of nrfutil (for anyone who feels adventurous enough to play around with it)
Mentioning @Shirley.