data stream freezes after some minutes of use

edited April 2016 in Cyton
Hello,

After setting up the device as described in the tutorial and installing the processing OpenBCI GUI, I've run some sessions to collect EEG data. Every time after around 5-10 minutes the communication breaks und no data is collected anymore. Only the blue LED is on. I tried to remove/reinsert the dongle and power down/up the OpenBCI module, but it won’t work again immediately. Even if I restart the PC it is not working. I have to wait for a few minutes and the orange LED is blinking and I get data from the board. But again after some minutes the communication breaks. I first thought it could be the OpenBCI GUI code maybe, but the same thing happens when I’m using OpenVIBE.

I've tried it on Win 7, Win 10 and Mac OS X El Capitan. The drivers are up to date (re-installed them to be sure) and also the firmware has been reloaded, new batteries, but nothing changes. Therefor I assume the issue could be with the hardware. I'm using the OpenBCI board V3 32-bit.

These issues sound a bit similar, but I think mine is different:

http://openbci.com/forum/index.php?p=/discussion/554/usb-dongle-stopped-working-suddenly
http://openbci.com/forum/index.php?p=/discussion/429/broken-dongle

Any ideas what the issue could be or what tests I could perform?

Regards,
Roger

Comments

  • wjcroftwjcroft Mount Shasta, CA
    Roger, hi.

    It looks like your situation is more similar to,

    http://openbci.com/forum/index.php?p=/discussion/324/data-stream-freezes-during-long-sessions

    Except that your freeze is happening in a much shorter time frame. I have encountered the long period freezes myself during ~60 minute or more neurofeedback sessions. But I've also wondered if there is some kind of component heat-up issue involved. Since my powering down and restarting works for a period of time, but this period of time is frequently less than the initial freeze up time period.

    After this has happened when I've felt around on the main board and dongle with my fingers, I cannot detect any 'hot spots' on the components. But I have wondered if it might be something going on with the RFduino's since they are likely pulling more power with their microwave transmissions.

    One possible experiment would be adjusting the RFduino power output levels downward and seeing if the freeze times get longer. However in my case that would still be 45 or more minutes until freeze. Making that testing somewhat awkward. It might be that with your particular boards there, you have an ideal set for testing the freeze failure and getting to the bottom of what is going on. I'll send an email to Joel @biomurph and include you.

    William
  • wjcroftwjcroft Mount Shasta, CA
    Rereading the long freeze thread, I now recall that Jeremy @jfrey has some code in the OpenViBE acquisition server that attempts to do a soft reset and restart when this condition is detected. But from your experience and my own experience I know that there are a couple of different failure modes for the freeze:

    (1) in the 'soft' freeze failure mode, it's possible to restart just by stopping and restarting the application code on the laptop. This does work for me most of the time. And is the basis of Jeremy's workaround.

    (2) in the 'hard' freeze failure mode, application stop/start is not sufficient. The main board or main board and dongle both must be powered down, then back up in order to allow application to restart. And in your case you appear to have a hard / cool off situation, where the components must cool off for 5-10 minutes before proceeding.

    ---

    Here's a couple more related threads, discussing storing to the SD card alone, and disabling the RFduino transmission. This workaround by @yj seems to confirm that the freezes are related to main board RFduino data stream transmissions.

  • wjcroftwjcroft Mount Shasta, CA
    A third freeze mode that I forgot to mention. This seems to happen the least frequently of the two mentioned above:

    (3) the samples continue to come in (dongle red light still flickers), but something has gone south.

    (a) in one type of situation the frame counter apparently stops incrementing, but the frames are still arriving. This is logged as 'sync' errors that are visible in BrainBay as a status area (bottom of window) counter that starts incrementing continuously until the design is halted.

    (b) Just yesterday using BioEra, samples continued to come in, but the raw EEG traces were just all over the place; not valid EEG data, unclear if these were just random values coming in. Had not seen this freeze mode before. Yesterday a stop/start of the BioEra design resumed normally.
  • Hi William,

    Thanks for your replies. Indeed, the issue seems more like the one you mentioned in your first post. As you said my issue is a 'hard' freeze where the components need to 'cool down' for some reason. The emerergency reset of OpenViBE in case of a 'soft' freeze is not working, as no commands can be sent to the board anymore. Just stopping and restarting the application code does not work. The third freeze type is nothing I have encountered so far, the freeze characteristic is always the same with red LED going off and no data is received (actually no communication is possible). As the red LED is not even flickering when the application code is
    trying to send commands, I again assume the issue is on the dongle, let me know you have different thoughts.

    The two things I will try:
    1. Save data only to the SD card. Uploading the OpenBCI_32-write-to-DS-only code should not be the problem, but I'm not too sure about the commands. Do I need to send something like 'N' 'G' and close after the test 'j' from a serial terminal (sorry this is new for me)?. If writing to SD only is possible for a longer period of time, it would show that the issue is not in a component that collects/processes the signals from the sensors. And we could assume that it most probably is the dongle?
    2. Lower the RFduino power output levels. Ok I will need some help with this one. Could you give me short instructions how to achieve this and to what level I should lower the power ouput? Is there something in the tutorials/guides (I couldn't find it :-))

    Thanks again!

    Roger

  • wjcroftwjcroft Mount Shasta, CA
    Roger, hi. Joel received our email and mentioned to me he would comment later today.

    Yannick @yj is the SD-card-only expert, I'm going to let him comment on the command strings / file closeout.

    My current guess is that the issue is not with the dongle, but rather something going on related to the RFduino on the main board. It is the RFduino that is using the most power and conceivably something getting warm. The RFduino on the dongle is transmitting much shorter acknowledgment packets. But you're right, it could be either of them or both of them.

    William
  • biomurphbiomurph Brooklyn, NY
    I don't think that there is an issue with anything getting too hot on the board, if all of the parts are 'good' then the board draws less than 60mA. The RFduino is super low-powered. Is it possible that there is some radio interference? Have you tried running the system in different locations?


  • wjcroftwjcroft Mount Shasta, CA
    Joel, I think you mentioned once that you had reproduced the freeze problem there in the lab? But that it took a randomly long amount of time. For my own use with BrainBay and BioEra, it seems to be about an hour. How long did it take on your tests? We can't all be seeing RF interference problems. (Me, you, Roger, Yannick, Jeremy, etc.) If Roger has a particular board that fails more often, it could be a blessing in disguise. :-)

  • wjcroftwjcroft Mount Shasta, CA
    @Rogerio,

    To address your question on power level adjustment. Here's the RFduino pdf manual,

    http://www.rfduino.com/wp-content/uploads/2014/03/rfduino.ble_.programming.reference.pdf

    On page 5 is this section:

    RFduinoGZLL.txPowerLevel

    This variable allows you to set the GZLL transmit power in dBm. You can select any value between -20 to +4 dBm in 4dBm increments. (ex. -20, -16, -12, -8, -4, 0, +4)

    Example:
    RFduinoGZLL.txPowerLevel = +4; //Sets the transmit power to max +4dBm

    Here's the conversion between dBm and milliwatts,

    https://en.wikipedia.org/wiki/DBm

    According to that table, +4 dBm is 2.5 milliwatts, and 10 meter range in the Bluetooth standard. But it looks like the API interface can only go down in 4 dBm steps, so next lower, 0 dBm would be 1 milliwatt and is listed as a 1 meter range. Although not sure how closely this corresponds to how RFDigital adjusts their transmitter. GZLL is a proprietary protocol, and not the standard Bluetooth specs quoted in the Wikipedia page. So range at 0 dBM may have to be determined experimentally.

    ----

    Joel @biomurph, although whenever I've felt around on the main board components after a freeze -- I've not sensed any components as being 'warm', consider this: the RFduino 'chip' has a hollow sheet metal shield over it. Which is about 2 mm off the chip die surface. If some very small area of the die (the RF transmitter portion) happened to get significantly above design specs -- that would likely not be felt on the sheet metal shield. Because (a) the tiny tiny hot spot would be balanced off by the majority cold portions of the chip. (b) the shield itself would smear out any temperature gradients that one could feel on it's top surface.

    I get the impression that most projects using the RFduino are using the BLE stack, and fairly low throughput. I would estimate that OpenBCI is one of the few projects that is driving it as fast and as continuously we are with the GZLL. You also mentioned that we have a custom GZLL stack from RFDigital that eliminates channel hopping or other delay mechanisms that would slow down GZLL. So we indeed are probably driving the highest sustained throughput compared to all other RFduino customers.

    William

  • yjyj France , Bordeaux
    edited January 2016
    Hello all,

    @wjcroft : the "SD card Only"  present on github,   has been writen by Joel.    It has been for me a starting point from which I increased the sample rate and I suppressed overruns.

    @Rogerio : Most operations on the OBCI Card , are controled by single letters, alphanumeric chars.
    Look in the main code "OpenBCI_32bit.ino" it is rather simple.

    Send these letters one by one.
    for exemple :   'A' followed by 'N'
    A     opens a file  which size is 11000 block of 512 chars (around 5 minutes)
    N     starts streaming (starts the A/D, etc..)
    When the file is filled , it is closed and streaming stopped
    You can stop prematurely sending   's'   then you should close the file by hand , sending   'j'

    To send these letters, use the Arduino IDE  menu Tool/Serial Monitor
    you can also use the OpenBCI GUI (proccessing). On your right there is a white '<' to open a window from which all chars typed are sent to the OBCI Board.   And look at the bottom of the processing window to see the result. for exemple try '?'

    Lets come back to our problem.   The first day I experienced the freezing issue... Since then I use the SD only version.
    A few days ago I recorded 4 hours at a 2kHz sample rate    and     24h at 500Hz.  No problem...
    So as you write : "it shows that the issue is not in a component that collects/processes the signals from the sensors."

    @biomurph, @wjcroft, @ Rogerio,   @....

    To test if the "Radio problem"  is located on  the Dongle or on the OBCI Board we need (much time),  two Boards and   two Dongles  using the same channel.
    As soon as the freezing occures between a couple of them,  test the communication between :
         the "Fresh" Dongle with the "Hot" Board   and
         the "Fresh" Board  with the  "Hot" Dongle...
     Which one of these combinations will work?   May be none...
     Results might be instructive...  Suspense...

        Yannick.

       
  • wjcroftwjcroft Mount Shasta, CA
    edited January 2016
    As far as RFduino range goes at the default power settings, here are some actual figures.

    A few hundred feet, line of sight. Statement by RFDigital admin.

    Even with metal box shielding, ~ 45 yards (135 feet) !

    So these are MUCH greater than the figure quoted in the Wikipedia article on dBm for +4 dBm (2.5 mW), which was 10 meters. Thus it seems likely that setting the power level to 0 dBm (1 mW), would offer acceptable range within the same room. The figures quoted in the links above are I'm sure BLE and not GZLL. But you would think the same transmitter powers would give similar ranges.

    William
  • Last weekend I tried the two things:

    1. Test if the freeze also happens when only writing to the SD card

    2. Lower the power output levels to check if I can expand the timeframe before the freeze

    Here are my results:

    (To mention: I did have some issues with the upload as the progress stopped at the step "Verify Flash" and threw out the following error: error at address 1D000204: file=..., mem=... which could be fixed using the trick mentioned in http://openbci.com/index.php/forum/#/discussion/208/chipkit-uploading-tips-restarting-bootloader)

    After I managed to upload the OpenBCI_32bit_SD_Only.ino I ran some test using the write to SD card only option. I could run sessions for over 1 hour without issues. As mentioned before also by @yj my conclusion was that the issue is not in a component that collects/processes the signals from the sensors.

    So with this result I wanted to try the second test as proposed by @wjcroft and lowering the output power level. First of all I was not able to upload the passthru code on the dongle radio at all. I followed the instructions on http://docs.openbci.com/tutorials/03-Upload_Code_to_OpenBCI_Dongle#upload-code-to-openbci-radios-setting-up-your-system-to-program-openbci-dongle (using Arduino IDE Version 1.5.8, dongle set to RESET). But I always got the error

    get

    fail.... fail... fail....

    I wonder if maybe the switch is broken and the dongle is not really accepting code uploads to it.

    It was interesting what then happened. To verify that the freezes still happen when streaming the data, I ran some sessions with normal streaming mode. And surprisingly also these sessions where able to run for over 1 hour without the freeze. There where some interruptions, but the cause was different and I could restart the stream immediately by stopping and starting the streaming mode in the processing GUI. I'm still using the OpenBCI_32bit_SD_Only.ino so I'm wondering if there is any other change in the code that could have fixed my freezing issue. I haven't compared the two code files yet, but will do this evening. Also I have forgotten to switch back to the GPIO6 position on the dongle, maybe this is also a factor (and again makes me wonder if the switch is working properly). Will test with GPIO6 position also this evening.

    Roger





  • biomurphbiomurph Brooklyn, NY
    @Rogerio
    for dongle upload, do you have the correct board and serial port selected?
  • edited January 2016
    Hi all,

    After some more tests it looks like I can run sessions for around 1 hour before it freezes. Although it's not perfect, for my project this should be sufficient. I will still try to test what happens if I lower the power output levels. 

    @biomurph Yes, I have followed the instructions in the tutorials section.

    @wjcroft As I don't have a DMM, testing the switch contacts is not possible at the moment. Maybe I'll find somebody who has one. But it still seems likely that there is something wrong with the switch as I can upload code to the board even if the switch is on the reset position... I'll look for a Adafruit FTDI Friend, with this I should be able to upload the radio code to the board.




  • wjcroftwjcroft Mount Shasta, CA
    Roger, thanks. I also get the freeze at about the 1 hour point, then more frequently as something seems to be "warmed up". I wonder if there is something about your original mainboard firmware that was causing the 5-10 minute freezes. Have you ever reloaded the default firmware and tested that?
  • edited April 2016
    [Original thread title: COMM between dongle and main board dying, for no reason??]

    Recording 16-channel EEG using the main board + daisy board. I import the open_bci_v3 code module in my python code. It works perfectly, but then for no apparent reason the main board will no longer communicate with the dongle!

    The indicator diode on the main board is lit. The battery voltage is 3.9x Volts.


    Normally when I run my code, I'll see a brief flash of the RXD diode (green) on the dongle, then a corresponding brief flash of the TXD diode (red) on the dongle. Like I said, it works perfectly fine for a while. Then the next time I run the code, only the RXD diode flashes. No corresponding flash of the TXD diode. I then get an error message from the "print_incoming_text" function (which is part of the __init__ function) that it didn't receive anything from the main board.

    And no matter how many times I push the RST button on the main board, or toggle the switch between OFF and PC, the comms won't come back to life.


    Any ideas?? What do you want me to check or post (code, etc.)? Thanks!
  • wjcroftwjcroft Mount Shasta, CA
    How often does it fail, and how long do you have to run before it freezes? See if this thread matches what you are seeing.

    http://openbci.com/forum/index.php?p=/discussion/595/data-stream-freezes-after-a-few-minutes-of-use

    If you have this type of freeze scenario, the board runs fine for an hour or so, then stops. It usually can be resumed by doing a soft reset or restart.

    Does the transfer restart if you power down everything (poweroff mainboard, remove dongle, reinsert, then powerup mainboard.)

    If you ARE having this type of freeze, I may merge this thread into the other so we can better track this.

  • edited April 2016
    @wjcroft,

    Thanks for the link, I should've known there was already a thread on this. Doh! 

    From your link, it's definitely the mode "2" version. In other words, once the failure occurs then it doesn't matter how many times I run my python code. It always comes back that the print_incoming_text function didn't get anything back from the main board.

    It has happened for two straight days, after working just fine for a couple hours.

    I have not tried unplugging the dongle, then reinserting it and then powering up the main board again. But in my opinion, I don't think it's the dongle. Reason being that I always see the brief flash on the RXD (green) diode, which I assume is the dongle attempting to "hail" the radio on the main board. But the main board just up and stops talking, even though the battery voltage seems to be just fine. And the blue diode on the main board is lit.


    In the other thread you mentioned that maybe the radio on the main board is overheating and so the radio stops working. But I would tend to agree with the "hardware guy" (biomurph) that shouldn't be the case given the power draw. Something is going wrong with the main board radio. But it's so weird that it would work fine for a couple hours and then suddenly decide to stop working.


    Please let me know what I can post or do to try to fix this.


    PS - is it possible to just bypass the radio and plug directly into the PC serial?? I'm guessing the answer is no, but thought I'd ask anyway.
  • wjcroftwjcroft Mount Shasta, CA
    How long of a session can you have before it fails? After you restart, how long does that run until it fails again?

    This failure scenario happens with my board after about an hour of continuous use. After a reset, it will go about another hour, then freeze again. I do believe this is happening on the mainboard RFduino. It's unclear at this point if reducing the power level would show an improvement. It's not a matter of the total power draw of the RFduino module, that indeed is low. But the section of the RFduino chip that does RF amplification may rise enough in temperature to go out of spec. And since the metal shield of the RFduino prevents locating any hotspots, the only way to confirm would be to reduce the power level.


  • wjcroftwjcroft Mount Shasta, CA
    I merged the new thread that @leka0024 started into this existing thread on the same topic. All previous commenters on this thread get notifications as new posts are made on the issue.

    After @Rogerio reflashed the mainboard chipKIT, he was able to go an hour before the freeze happens. Leka, that could be something you could try.

    William
  • edited April 2016
    @wjcroft

    My apologies, the sentence "It has happened for two straight days, after working just fine for a couple hours." was poorly worded. Let me attempt to reword, as follows.

    The previous two days, the same thing has happened: the system works perfectly fine for 1-2 hrs, then suddenly the main board radio stops talking. And let me make a further clarification: my EEG recording sessions last for approx 3min 20sec at a time (40 continuous cues to the subject, lasting for 5 sec each). So I'm able to run these sessions fairly constantly with varying amounts of break time between. Then suddenly, after being able to do this for 1-2 hrs the last couple days, the main board stops talking.

    But I'm not sure if you're talking about literally running the board (streaming data @250 samples/sec) continuously for an hour. I don't do anything like that.

    SO ... I don't think my problem belongs under this thread title. But since you've already moved it, that's fine. Maybe the thread title could be adjusted to something like "Data stream/main board radio issues"?


    I have NOT attempted:

    - turning off main board, unplugging dongle, replugging dongle, turning on main board (but I suspect this won't work)
    - letting the main board rest for 5-10 mins and then trying again (or some other amount of time)


    I use the Adafruit lithium battery pack + charger suggested in the Mark III headset page here: https://github.com/OpenBCI/Ultracortex/tree/master/Mark_3  --> https://www.adafruit.com/products/1578  https://www.adafruit.com/products/1304

    The charger is charging the battery up to almost 4.2V (I leave it charged overnight, green diode is lit in the morning). ASIDE: seems a bit odd, because the Adafruit webpage shows it to be a 3.7V battery, while the charger is talking about a 3.7V/4.2V battery. Maybe that's a 70%/100% charge level indication?


    HYPOTHESIS: the main board radio requires a voltage level of >= ~3.95 (? or something) in order to function. Once the battery level drops low enough, the power regulator output dips enough so that the blue diode stays lit (and probably the micro still works, 3.3V?) but the radio stops working.

    TEST: I assume my system will work fine for a while today, then stop working again. Guessing the voltage level will have dropped to around 3.92-3.93 or so. At that point, I'll try plugging in the AA battery holder that came with the kit which is around 5.5V now.
  • wjcroftwjcroft Mount Shasta, CA
    Leka, hi.

    Please try the complete power off / on sequence we described. That always resolves my freezes regardless of the type. It may not be that a 'continuous' stream for an hour or so is necessary, but rather just the cumulative time. Also if you can try reflashing your mainboard, that changed things for @Rogerio.

    I use one of the Adafruit Powerboost boosters which provides a constant 5v. And still have the freeze after about an hour of use. So I don't think it is an issue with your lithium setup there.

    I changed the title of the thread slightly to cover all these bases.

  • edited April 2016
    @wjcroft

    My failure mode would never recover when toggling the power switch between OFF/PC. But I also did not let it sit for 5-10mins before trying again.

    No failure mode today after 12 runs. Battery got down to 3.97V. But in light of your comment about using a constant voltage supply, then that probably kills my hypothesis. I am not charging the battery tonight. So we'll see how long it lasts tomorrow.

    I'll test everything, if I can get it to fail the same way again, and report back the results.
  • wjcroftwjcroft Mount Shasta, CA
    Unpluging and replugging the dongle was necessary for some of my freezes.
  • @wjcroft , ran the headset several times today and got the failure mode to appear. Batt voltage was down to 3.95V. However, the sequence of powering down main board, unplugging dongle, replugging dongle and powering up the main board allowed normal function to continue.

    I'm going to keep running the board without recharge as low as the batt voltage can go until either the failure mode reappears and isn't solved by the unplug/replug sequence or the board just shuts down due to lack of batt power.
  • wjcroftwjcroft Mount Shasta, CA
    Leka, that's good to hear. I'm pretty sure the 3.3v regulator on the mainboard is an LDO, low dropout type. Meaning that stable regulation should happen all the way down to 3.3v.

    So on the average, about how much cumulative run time do you need before you see a freeze? As I said previously, it takes about an hour for me. Initially @Rogerio was seeing failures within just 5 to 10 minutes. But after he reflashed his chipKIT, the time required then went up to one hour as well. Mysterious why a reflash would effect this. Possibly a combination of marginal flash memory plus the radio problem.

    If you are seeing short times until your freeze, try a reflash of the chipKIT.

  • edited April 2016
    Update: the headset continued to run without any problems all the way down to 3.3V on the batt. In fact, I even switched workstations and am now working on a Win 7 machine. Never ran into the problem again ... until today.

    Voltage is 100% not the issue, as I was using the AA battery pack and it's voltage is around 5V right now.

    The power down, unplug, re-plug, power up sequence solved the issue again.

    Quite a while of cumulative streaming time between my last post and today, maybe 1-2 hrs total, maybe more. I guess somehow the radio pair get discombobulated now and then.

    Anyhow, the sequence has solved it each time, so I guess not much else to report. Thanks again for the help.
Sign In or Register to comment.