data stream freezes after some minutes of use
Hello,
After setting up the device as described in the tutorial and installing the processing OpenBCI GUI, I've run some sessions to collect EEG data. Every time after around 5-10 minutes the communication breaks und no data is collected anymore. Only the blue LED is on. I tried to remove/reinsert the dongle and power down/up the OpenBCI module, but it won’t work again immediately. Even if I restart the PC it is not working. I have to wait for a few minutes and the orange LED is blinking and I get data from the board. But again after some minutes the communication breaks. I first thought it could be the OpenBCI GUI code maybe, but the same thing happens when I’m using OpenVIBE.
I've tried it on Win 7, Win 10 and Mac OS X El Capitan. The drivers are up to date (re-installed them to be sure) and also the firmware has been reloaded, new batteries, but nothing changes. Therefor I assume the issue could be with the hardware. I'm using the OpenBCI board V3 32-bit.
These issues sound a bit similar, but I think mine is different:
http://openbci.com/forum/index.php?p=/discussion/554/usb-dongle-stopped-working-suddenly
http://openbci.com/forum/index.php?p=/discussion/429/broken-dongle
Any ideas what the issue could be or what tests I could perform?
Regards,
Roger
After setting up the device as described in the tutorial and installing the processing OpenBCI GUI, I've run some sessions to collect EEG data. Every time after around 5-10 minutes the communication breaks und no data is collected anymore. Only the blue LED is on. I tried to remove/reinsert the dongle and power down/up the OpenBCI module, but it won’t work again immediately. Even if I restart the PC it is not working. I have to wait for a few minutes and the orange LED is blinking and I get data from the board. But again after some minutes the communication breaks. I first thought it could be the OpenBCI GUI code maybe, but the same thing happens when I’m using OpenVIBE.
I've tried it on Win 7, Win 10 and Mac OS X El Capitan. The drivers are up to date (re-installed them to be sure) and also the firmware has been reloaded, new batteries, but nothing changes. Therefor I assume the issue could be with the hardware. I'm using the OpenBCI board V3 32-bit.
These issues sound a bit similar, but I think mine is different:
http://openbci.com/forum/index.php?p=/discussion/554/usb-dongle-stopped-working-suddenly
http://openbci.com/forum/index.php?p=/discussion/429/broken-dongle
Any ideas what the issue could be or what tests I could perform?
Regards,
Roger
Comments
It looks like your situation is more similar to,
http://openbci.com/forum/index.php?p=/discussion/324/data-stream-freezes-during-long-sessions
Except that your freeze is happening in a much shorter time frame. I have encountered the long period freezes myself during ~60 minute or more neurofeedback sessions. But I've also wondered if there is some kind of component heat-up issue involved. Since my powering down and restarting works for a period of time, but this period of time is frequently less than the initial freeze up time period.
After this has happened when I've felt around on the main board and dongle with my fingers, I cannot detect any 'hot spots' on the components. But I have wondered if it might be something going on with the RFduino's since they are likely pulling more power with their microwave transmissions.
One possible experiment would be adjusting the RFduino power output levels downward and seeing if the freeze times get longer. However in my case that would still be 45 or more minutes until freeze. Making that testing somewhat awkward. It might be that with your particular boards there, you have an ideal set for testing the freeze failure and getting to the bottom of what is going on. I'll send an email to Joel @biomurph and include you.
William
Thanks for your replies. Indeed, the issue seems more like the one you mentioned in your first post. As you said my issue is a 'hard' freeze where the components need to 'cool down' for some reason. The emerergency reset of OpenViBE in case of a 'soft' freeze is not working, as no commands can be sent to the board anymore. Just stopping and restarting the application code does not work. The third freeze type is nothing I have encountered so far, the freeze characteristic is always the same with red LED going off and no data is received (actually no communication is possible). As the red LED is not even flickering when the application code is
trying to send commands, I again assume the issue is on the dongle, let me know you have different thoughts.
The two things I will try:
Thanks again!
Roger
Yannick @yj is the SD-card-only expert, I'm going to let him comment on the command strings / file closeout.
My current guess is that the issue is not with the dongle, but rather something going on related to the RFduino on the main board. It is the RFduino that is using the most power and conceivably something getting warm. The RFduino on the dongle is transmitting much shorter acknowledgment packets. But you're right, it could be either of them or both of them.
William
To address your question on power level adjustment. Here's the RFduino pdf manual,
http://www.rfduino.com/wp-content/uploads/2014/03/rfduino.ble_.programming.reference.pdf
On page 5 is this section:
This variable allows you to set the GZLL transmit power in dBm. You can select any value between -20 to +4 dBm in 4dBm increments. (ex. -20, -16, -12, -8, -4, 0, +4)
Example:
RFduinoGZLL.txPowerLevel = +4; //Sets the transmit power to max +4dBm
Here's the conversion between dBm and milliwatts,
https://en.wikipedia.org/wiki/DBm
According to that table, +4 dBm is 2.5 milliwatts, and 10 meter range in the Bluetooth standard. But it looks like the API interface can only go down in 4 dBm steps, so next lower, 0 dBm would be 1 milliwatt and is listed as a 1 meter range. Although not sure how closely this corresponds to how RFDigital adjusts their transmitter. GZLL is a proprietary protocol, and not the standard Bluetooth specs quoted in the Wikipedia page. So range at 0 dBM may have to be determined experimentally.
----
Joel @biomurph, although whenever I've felt around on the main board components after a freeze -- I've not sensed any components as being 'warm', consider this: the RFduino 'chip' has a hollow sheet metal shield over it. Which is about 2 mm off the chip die surface. If some very small area of the die (the RF transmitter portion) happened to get significantly above design specs -- that would likely not be felt on the sheet metal shield. Because (a) the tiny tiny hot spot would be balanced off by the majority cold portions of the chip. (b) the shield itself would smear out any temperature gradients that one could feel on it's top surface.
I get the impression that most projects using the RFduino are using the BLE stack, and fairly low throughput. I would estimate that OpenBCI is one of the few projects that is driving it as fast and as continuously we are with the GZLL. You also mentioned that we have a custom GZLL stack from RFDigital that eliminates channel hopping or other delay mechanisms that would slow down GZLL. So we indeed are probably driving the highest sustained throughput compared to all other RFduino customers.
William
@wjcroft : the "SD card Only" present on github, has been writen by Joel. It has been for me a starting point from which I increased the sample rate and I suppressed overruns.
@Rogerio : Most operations on the OBCI Card , are controled by single letters, alphanumeric chars.
Look in the main code "OpenBCI_32bit.ino" it is rather simple.
Send these letters one by one.
for exemple : 'A' followed by 'N'
A opens a file which size is 11000 block of 512 chars (around 5 minutes)
N starts streaming (starts the A/D, etc..)
When the file is filled , it is closed and streaming stopped
You can stop prematurely sending 's' then you should close the file by hand , sending 'j'
To send these letters, use the Arduino IDE menu Tool/Serial Monitor
you can also use the OpenBCI GUI (proccessing). On your right there is a white '<' to open a window from which all chars typed are sent to the OBCI Board. And look at the bottom of the processing window to see the result. for exemple try '?'
Lets come back to our problem. The first day I experienced the freezing issue... Since then I use the SD only version.
A few days ago I recorded 4 hours at a 2kHz sample rate and 24h at 500Hz. No problem...
So as you write : "it shows that the issue is not in a component that collects/processes the signals from the sensors."
@biomurph, @wjcroft, @ Rogerio, @....
To test if the "Radio problem" is located on the Dongle or on the OBCI Board we need (much time), two Boards and two Dongles using the same channel.
As soon as the freezing occures between a couple of them, test the communication between :
the "Fresh" Dongle with the "Hot" Board and
the "Fresh" Board with the "Hot" Dongle...
Which one of these combinations will work? May be none...
Results might be instructive... Suspense...
Yannick.
Last weekend I tried the two things:
1. Test if the freeze also happens when only writing to the SD card
2. Lower the power output levels to check if I can expand the timeframe before the freeze
Here are my results:
(To mention: I did have some issues with the upload as the progress stopped at the step "Verify Flash" and threw out the following error: error at address 1D000204: file=..., mem=... which could be fixed using the trick mentioned in http://openbci.com/index.php/forum/#/discussion/208/chipkit-uploading-tips-restarting-bootloader)
After I managed to upload the OpenBCI_32bit_SD_Only.ino I ran some test using the write to SD card only option. I could run sessions for over 1 hour without issues. As mentioned before also by @yj my conclusion was that the issue is not in a component that collects/processes the signals from the sensors.
So with this result I wanted to try the second test as proposed by @wjcroft and lowering the output power level. First of all I was not able to upload the passthru code on the dongle radio at all. I followed the instructions on http://docs.openbci.com/tutorials/03-Upload_Code_to_OpenBCI_Dongle#upload-code-to-openbci-radios-setting-up-your-system-to-program-openbci-dongle (using Arduino IDE Version 1.5.8, dongle set to RESET). But I always got the error
get
fail.... fail... fail....
I wonder if maybe the switch is broken and the dongle is not really accepting code uploads to it.
It was interesting what then happened. To verify that the freezes still happen when streaming the data, I ran some sessions with normal streaming mode. And surprisingly also these sessions where able to run for over 1 hour without the freeze. There where some interruptions, but the cause was different and I could restart the stream immediately by stopping and starting the streaming mode in the processing GUI. I'm still using the OpenBCI_32bit_SD_Only.ino so I'm wondering if there is any other change in the code that could have fixed my freezing issue. I haven't compared the two code files yet, but will do this evening. Also I have forgotten to switch back to the GPIO6 position on the dongle, maybe this is also a factor (and again makes me wonder if the switch is working properly). Will test with GPIO6 position also this evening.
Roger
The indicator diode on the main board is lit. The battery voltage is 3.9x Volts.
Normally when I run my code, I'll see a brief flash of the RXD diode (green) on the dongle, then a corresponding brief flash of the TXD diode (red) on the dongle. Like I said, it works perfectly fine for a while. Then the next time I run the code, only the RXD diode flashes. No corresponding flash of the TXD diode. I then get an error message from the "print_incoming_text" function (which is part of the __init__ function) that it didn't receive anything from the main board.
And no matter how many times I push the RST button on the main board, or toggle the switch between OFF and PC, the comms won't come back to life.
Any ideas?? What do you want me to check or post (code, etc.)? Thanks!
http://openbci.com/forum/index.php?p=/discussion/595/data-stream-freezes-after-a-few-minutes-of-use
If you have this type of freeze scenario, the board runs fine for an hour or so, then stops. It usually can be resumed by doing a soft reset or restart.
Does the transfer restart if you power down everything (poweroff mainboard, remove dongle, reinsert, then powerup mainboard.)
If you ARE having this type of freeze, I may merge this thread into the other so we can better track this.
My apologies, the sentence "It has happened for two straight days, after working just fine for a couple hours." was poorly worded. Let me attempt to reword, as follows.
The previous two days, the same thing has happened: the system works perfectly fine for 1-2 hrs, then suddenly the main board radio stops talking. And let me make a further clarification: my EEG recording sessions last for approx 3min 20sec at a time (40 continuous cues to the subject, lasting for 5 sec each). So I'm able to run these sessions fairly constantly with varying amounts of break time between. Then suddenly, after being able to do this for 1-2 hrs the last couple days, the main board stops talking.
But I'm not sure if you're talking about literally running the board (streaming data @250 samples/sec) continuously for an hour. I don't do anything like that.
SO ... I don't think my problem belongs under this thread title. But since you've already moved it, that's fine. Maybe the thread title could be adjusted to something like "Data stream/main board radio issues"?
I have NOT attempted:
- turning off main board, unplugging dongle, replugging dongle, turning on main board (but I suspect this won't work)
- letting the main board rest for 5-10 mins and then trying again (or some other amount of time)
I use the Adafruit lithium battery pack + charger suggested in the Mark III headset page here: https://github.com/OpenBCI/Ultracortex/tree/master/Mark_3 --> https://www.adafruit.com/products/1578 https://www.adafruit.com/products/1304
The charger is charging the battery up to almost 4.2V (I leave it charged overnight, green diode is lit in the morning). ASIDE: seems a bit odd, because the Adafruit webpage shows it to be a 3.7V battery, while the charger is talking about a 3.7V/4.2V battery. Maybe that's a 70%/100% charge level indication?
HYPOTHESIS: the main board radio requires a voltage level of >= ~3.95 (? or something) in order to function. Once the battery level drops low enough, the power regulator output dips enough so that the blue diode stays lit (and probably the micro still works, 3.3V?) but the radio stops working.
TEST: I assume my system will work fine for a while today, then stop working again. Guessing the voltage level will have dropped to around 3.92-3.93 or so. At that point, I'll try plugging in the AA battery holder that came with the kit which is around 5.5V now.
Please try the complete power off / on sequence we described. That always resolves my freezes regardless of the type. It may not be that a 'continuous' stream for an hour or so is necessary, but rather just the cumulative time. Also if you can try reflashing your mainboard, that changed things for @Rogerio.
I use one of the Adafruit Powerboost boosters which provides a constant 5v. And still have the freeze after about an hour of use. So I don't think it is an issue with your lithium setup there.
I changed the title of the thread slightly to cover all these bases.
My failure mode would never recover when toggling the power switch between OFF/PC. But I also did not let it sit for 5-10mins before trying again.
No failure mode today after 12 runs. Battery got down to 3.97V. But in light of your comment about using a constant voltage supply, then that probably kills my hypothesis. I am not charging the battery tonight. So we'll see how long it lasts tomorrow.
I'll test everything, if I can get it to fail the same way again, and report back the results.
I'm going to keep running the board without recharge as low as the batt voltage can go until either the failure mode reappears and isn't solved by the unplug/replug sequence or the board just shuts down due to lack of batt power.
So on the average, about how much cumulative run time do you need before you see a freeze? As I said previously, it takes about an hour for me. Initially @Rogerio was seeing failures within just 5 to 10 minutes. But after he reflashed his chipKIT, the time required then went up to one hour as well. Mysterious why a reflash would effect this. Possibly a combination of marginal flash memory plus the radio problem.
If you are seeing short times until your freeze, try a reflash of the chipKIT.
Voltage is 100% not the issue, as I was using the AA battery pack and it's voltage is around 5V right now.
The power down, unplug, re-plug, power up sequence solved the issue again.
Quite a while of cumulative streaming time between my last post and today, maybe 1-2 hrs total, maybe more. I guess somehow the radio pair get discombobulated now and then.
Anyhow, the sequence has solved it each time, so I guess not much else to report. Thanks again for the help.