Dropped packets: counter and timestamp questions [resolved]

edited May 2021 in Ganglion

Hi all, I just had a really quick question that I was unable to find on the forums already.

So in my recordings (using Ganglion and OpenBCI GUI v4) I will sometimes notice around 1 second of data has been dropped. For Instance, Ill find inconsistencies in the counter (see below example) where the counter will jump from say 192 to 189, indicating I have lost about 1 second of data. However, I noticed the timestamps do not reflect this theoretical 1 second jump in time. Is this because timestamps are computed when the packet is received by the CPU? Also, is there anything I can do after recording to compensate for these dropped packets?

189,-118.52,-233.51,-205.48,54.75,0.0,0.0,0.0, 14:04:43.094,1607195083094
190,-117.64,-225.27,-208.3,53.77,0.0,0.0,0.0, 14:04:43.094,1607195083094
191,-115.01,-224.55,-207.91,55.5,0.0,0.0,0.0, 14:04:43.105,1607195083105
192,-118.7,-230.71,-213.13,47.87,0.0,0.0,0.0, 14:04:43.105,1607195083105
189,-1098.92,746.69,-217.92,-143.61,0.0,0.0,0.0, 14:04:43.121,1607195083121
190,-1165.16,-230.99,-1195.28,-140.74,0.0,0.0,0.0, 14:04:43.121,1607195083121
195,-1168.83,-235.27,-1199.79,-143.1,0.0,0.0,0.0, 14:04:43.122,1607195083122
196,-1167.2,-231.56,-1200.68,-141.47,0.0,0.0,0.0, 14:04:43.122,1607195083122

Comments

  • wjcroftwjcroft Mount Shasta, CA
    edited May 2021

    BPoole, hi.

    Is this because timestamps are computed when the packet is received by the CPU?

    That is correct. Time stamps reflect the time the BLE packets arrive at the laptop.

    re: repeated packet number 189 samples

    Hmm, this is very odd. Richard @retiutut, do you have any impressions of what is going on here? To me this looks like some kind of glitch in the Brainflow library. Aggravated by Bluetooth radio packet loss. Note that there is now a 'Packet Loss' GUI widget that can help to analyze packet loss situations and fine tune placement of your dongle (keep reading).

    Although the Ganglion sample rate is 200 Hz, as far as radio packets, it is only sending 100 packets per second in a compressed data stream. See below link. The compression scheme works in intervals of 100. Only at the beginning of these intervals are 'uncompressed' samples sent. Then for the in between samples, 'delta' (change) packets contain the difference between 'now' and the previous packet.

    https://docs.openbci.com/docs/03Ganglion/GanglionDataFormat

    The 'packet numbers' shown before each line of the GUI CSV Ganglion data, must be assigned internally by the Brainflow library as a means to show the user if discrepancies occur. These number DO NOT reflect the actual radio packet numbers. At least I believe that is the case.


    As far as the best way to resolve this issue in your own case: generally you want to improve radio reception at the BLED112 dongle. And reduction of potential sources of EMF electromagnetic field interference (extension cords, wall warts, power supplies, wall floor ceiling conduits, LED lamps, etc.) A common trick is to use a usb extension cord, and get the BLED112 closer to your Ganglion. Even a simple draping of the BLED112 over the top of your computer monitor, will get it closer and away from metal table tops and other objects that can interfere with reception.

    Best regards, William

  • wjcroftwjcroft Mount Shasta, CA

    The Packet Loss widget, I don't think is documented yet in the Widget guide, but is accessed similarly to the other widgets.

    https://docs.openbci.com/docs/06Software/01-OpenBCISoftware/GUIWidgets

  • edited May 2021

    To me this looks like some kind of glitch in the Brainflow library. Aggravated by Bluetooth radio packet loss.

    One of the things I've notice, I even saw it today when I was doing some testing, is that this issue seems to happen after "actual" packets were reported lost by OpenBCI v4 GUI. Now, I don't have access to the packet loss widget as I am currently unable to get the OpenBCI GUI v5.0.4 to work with my Ganglion board. However, I have paid close attention to the terminal output which does report missing packets for the v4 GUI. In doing so, I have noticed that the while the v4 GUI is recording data it will report missing packets, although it never reports hundreds of missing packets, just a few, say 4-6. There seems to be a disconnect between the recorded packet number (indicating false amounts of dropped samples) and the actual number of dropped packets (reported while recording by the GUI) . Maybe these aforementioned observations are inline with your theory?

    As a note, I wrote my algorithm which uses these packet numbers to interpolate missing samples as most missing samples are very few in number. Since interpolating hundreds of samples isn't really practical, I was trying to figure out if these packet numbers were a false flag or not.

  • wjcroftwjcroft Mount Shasta, CA

    Now, I don't have access to the packet loss widget as I am currently unable to get the OpenBCI GUI v5.0.4 to work with my Ganglion board.

    What is the issue preventing your using the latest GUI? Please try the usb extension cord idea, it will definitely greatly improve any packet loss.

    Regardless of how the real radio packet sequence numbers relate to the numbers produced by Brainflow, the skips do indicate lossage. It's just unclear how many were actually lost. Best solution will be to simply improve your radio connection.

    Regards, William

  • edited May 2021

    There are a few things that make the v5 GUI unusable for me at the moment that I haven't figured out how to solve. It should be noted that I am using Ubuntu 18.04 and v5.0.4 of the GUI.

    Normally, when I first start the GUI it will load fine. When I go to search for the Ganglion board it shows up under "BLE Devices" and I can easily start the GUI. However, when Im done recording and close the current recording session I am unable to find the Ganglion board again. The Ganglion board light will be flashing indicating it has been released. Further, I get the following error.

    ControlPanel: Found BLED112 Dongle on COM port: /dev/ttyACM0
    openbci_gui_helpers.GanglionError: Error in scan for ganglions:PORT_OPEN_ERROR
        at openbci_gui_helpers.GUIHelper.scan_for_ganglions(GUIHelper.java:69)
        at OpenBCI_GUI$BLEBox$1.run(OpenBCI_GUI.java:5260)
    

    I can solve this error by running the GUI with sudo. However, the Ganglion board still doesn't appear in "BLE Devices" until I completely restart my computer. I should note that my user is apart of the "dialout" group as well.

    Sometimes, if I wait long enough or unplug the BLE dongle or turn off and on the board, the Ganglion board will reappear in "BLE Devices". However, when I go to start the GUI I get the following error.

    ERROR: brainflow.BrainFlowError: Error in prepare_session:GENERAL_ERROR when initializing Brainflow board. Data will not stream.
    brainflow.BrainFlowError: Error in prepare_session:GENERAL_ERROR
        at brainflow.BoardShim.prepare_session(BoardShim.java:598)
        at OpenBCI_GUI$BoardBrainFlow.initializeInternal(OpenBCI_GUI.java:2459)
        at OpenBCI_GUI$BoardGanglion.initializeInternal(OpenBCI_GUI.java:3737)
        at OpenBCI_GUI$Board.initialize(OpenBCI_GUI.java:1990)
        at OpenBCI_GUI.initSystem(OpenBCI_GUI.java:681)
        at OpenBCI_GUI.systemInitSession(OpenBCI_GUI.java:982)
        at OpenBCI_GUI.draw(OpenBCI_GUI.java:559)
        at processing.core.PApplet.handleDraw(PApplet.java:2475)
        at processing.opengl.PSurfaceJOGL$DrawListener.display(PSurfaceJOGL.java:866)
        at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:692)
        at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:674)
        at jogamp.opengl.GLAutoDrawableBase$2.run(GLAutoDrawableBase.java:443)
        at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1293)
        at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147)
        at com.jogamp.newt.opengl.GLWindow.display(GLWindow.java:759)
        at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:81)
        at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:452)
        at com.jogamp.opengl.util.FPSAnimator$MainTask.run(FPSAnimator.java:178)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
    Channel count set to 4
    openBCI_GUI: haltSystem: Halting system for reconfiguration of settings...
    [DEFAULT]: Data stream is already stopped.
    [ERROR]: Failed to initialize board. Please check that the board is on and has power. See Console Log for more details.
    OpenBCI_GUI::Control Panel: Playback history file not found.
    
  • wjcroftwjcroft Mount Shasta, CA

    OK, sorry, did not see that you are already in 'dialout'. Maybe Richard @retiutut would have a suggestion.

  • wjcroftwjcroft Mount Shasta, CA
    edited May 2021

    It's possible the usb ports on your laptop are wonky. Suggest trying other usb ports, or using on a usb hub extender. Or trying another laptop. Does it work on Windows or work using Linux on another machine? Is your Linux distro the latest available, etc.

  • It's possible the usb ports on your laptop are wonky. Suggest trying other usb ports, or using on a usb hub extender. Or trying another laptop. Does it work on Windows or work using Linux on another machine? Is your Linux distro the latest available, etc.

    I have tried v5.0.4 of the GUI on another computer which runs Ubuntu 18.04 and it too had the same issues even though the user had access to the "dialout" group. Ill have to try it on Windows and report back here.

  • wjcroftwjcroft Mount Shasta, CA
    edited May 2021

    Ah, again apologies for not looking more closely at your errors. Your output above seems to have some correlation to issues with the OpenGL, graphics library. This library depends upon having up to date graphics / card / chip drivers installed on your laptop. Please visit your manufacturers website, and check if you can get latest graphics drivers. Same would apply to the other machine as well.

  • wjcroftwjcroft Mount Shasta, CA

    Also check the Console Log as suggested in the error list above.

  • edited May 2021

    Your output above seems to have some correlation to issues with the OpenGL, graphics library. This library depends upon having up to date graphics / card / chip drivers installed on your laptop. Please visit your manufacturers website, and check if you can get latest graphics drivers.

    After checking, I believe I have the latest graphics drivers and OpenGL mesa drivers on both PCs.

    Also check the Console Log as suggested in the error list above.

    The errors I linked above were taken directly form the terminal output (I assume this is the same as the console log). Looking at the console output closer it seems the GUI will find the BLED112 device but either the GanglionError will be thrown and the Ganglion board won't be found or no error will be thrown and the device will still not be found.

    ControlPanel: Found BLED112 Dongle on COM port: /dev/ttyACM0
    [DEFAULT]: BLE Devices Refreshing in progress
    openbci_gui_helpers.GanglionError: Error in scan for ganglions:GENERAL_ERROR
        at openbci_gui_helpers.GUIHelper.scan_for_ganglions(GUIHelper.java:69)
        at OpenBCI_GUI$BLEBox$1.run(OpenBCI_GUI.java:5260)
    
  • wjcroftwjcroft Mount Shasta, CA

    Richard @retiutut, do you see any pattern, or have any suggestions?

    Bpoole, have you tried running under Windows 10 (up to date), to see if this could be an issue with your Linux distro?

    It's remotely possible that your BLED112 is flakey, but I would think in such cases it would fail solidly. Not work once after boot, then fail. Your failure pattern sounds to me like issues the OS is having with usb connect / disconnect / reset.

    William

  • edited May 2021

    Hmm, this is very odd. Richard @retiutut, do you have any impressions of what is going on here? To me this looks like some kind of glitch in the Brainflow library. Aggravated by Bluetooth radio packet loss. Note that there is now a 'Packet Loss' GUI widget that can help to analyze packet loss situations and fine tune placement of your dongle (keep reading).

    With further experimenting and investigating, Im fairly certain there is some bug occurring here. I have recorded around 240 files across 6 different subjects where this "bug" occurs. The defining quality of this bug is packet number always seems to "mess up" 2 packet numbers before 2-6 packets are actually dropped. To further confirm this, I ran a 5 more experiments over the weekend and recorded all the number of packets that were dropped per each recording, according to the GUI terminal output. The GUI only ever reported 4-6 packets dropped. However, upon closer inspection of the recording files, some of the files contained this bug of then packet number "messing up" and making it look like 180+ packets were lost right before (exactly 2 samples before) the actual 4-6 packets were lost.

    As I'm using this data for research, I have been trying to confirm if this packet number issues is a bug or I was really losing loads of packets. As I compute the timestamp of each sample based on each recording's sampling rate, having 180+ missing samples would mean my timestamps would be ~1 second off. Thus in turn throwing off my mapping of external labels/events to the EEG data and distorting or even nullifying the signal averages. However, my signal averages are still visible, even when only ploting the averages of the files affected by this "bug".

  • wjcroftwjcroft Mount Shasta, CA

    PLEASE experiment with the usb extension cord suggestion made earlier. Drape the dongle over some object in your vicinity, such as your plastic monitor frame. Regardless of how the packet loss is recording (whether the actual sequence numbers shown are accurate or not), your first priority should be to get your packet loss to be close to zero.

    It's also possible that other EMF interference (mentioned previously) in your area, could be confusing the BLED112 reception. As a 'best' case example, you could see what the loss is, when the dongle (on extension) is very close to the Ganglion itself. Thus minimizing the distance between the two. Just draping it over an object however will likely make a large improvement.

    Regards,William

  • edited May 2021

    Bpoole, have you tried running under Windows 10 (up to date), to see if this could be an issue with your Linux distro?

    Yes, I just tried it on Windows (as I dual boot Windows and Linux) and the GUI and connection works quite flawlessly. It is most definitely an OS issue.

  • edited May 2021

    PLEASE experiment with the usb extension cord suggestion made earlier. Drape the dongle over some object in your vicinity, such as your plastic monitor frame. Regardless of how the packet loss is recording (whether the actual sequence numbers shown are accurate or not), your first priority should be to get your packet loss to be close to zero.

    I will do so for future experiments. However, I needed a way of confirming whether I needed to rerecord all my past experiments or not. I rarely do get much to any packet loss (once again according to the GUI terminal output while recording). I'd say it is rare if it does occur and when it does it only seems to be 2-6 samples.

  • wjcroftwjcroft Mount Shasta, CA

    Yes, I just tried it on Windows (as I dual boot Windows and Linux) and the GUI and connection works quite flawlessly. It is most definitely an OS issue.

    OK, so this eliminates all the other possibilities we talked about: poor radio reception, Brainflow issues, EMF interference, etc.

    You may want to try a better Linux distribution. Because other Linux users of the BLED112 have not had these issues.

  • OK, so this eliminates all the other possibilities we talked about: poor radio reception, Brainflow issues, EMF interference, etc.

    I should clarify, when I said "flawlessly" I meant connecting to the GUI to board (which was not the case on Linux). I have not been able to test if packets are being dropped or not.

    Also, Im using Ubuntu 18.04 which I would assume to be a commonly used Linux distribution, no?

  • wjcroftwjcroft Mount Shasta, CA

    Also, Im using Ubuntu 18.04 which I would assume to be a commonly used Linux distribution, no?

    The '18' in the version indicates it was from 2018, possibly three years old at this point. Although this version was a LTS long term support, it obviously has some flaws that were not fixed.

  • The '18' in the version indicates it was from 2018, possibly three years old at this point. Although this version was a LTS long term support, it obviously has some flaws that were not fixed.

    Ah right, do you think upgrading to 20.04 or 21.04 could potentially alleviate the issues?

  • wjcroftwjcroft Mount Shasta, CA

    I have not been able to test if packets are being dropped or not.

    Please use the working Windows 10 that you have, and the latest GUI, Packet Loss widget. If you are still seeing lossage with that, then ASAP obtain a usb extension cord as mentioned.

  • wjcroftwjcroft Mount Shasta, CA

    Ah right, do you think upgrading to 20.04 or 21.04 could potentially alleviate the issues?

    Worth a try. Could still be some basic misunderstanding that Ubuntu has with usb devices such as the BLED112. Windows gets it. The BLED112 is just a serial COM port as far as the OS goes.

  • edited May 2021

    Worth a try. Could still be some basic misunderstanding that Ubuntu has with usb devices such as the BLED112. Windows gets it. The BLED112 is just a serial COM port as far as the OS goes.

    Good news! Upgrading to Ubuntu 20.04 fixes the issues and allows the v5.0.4 GUI to work properly (as in I can actually connect to the Ganglion board without running into errors)! The only caveat being the board doesn't disconnect from the GUI even after the GUI is exits (i.e., the Ganglion board light doesn't return to blinking).

    Just as note, when I upgrade it did break the OpenBCI Hub for v4.2.0 but this post alleviated the issue!

  • retiututretiutut Louisiana, USA
    edited May 2021

    I have run 100+ sessions with Ganglion and BLED112 over the years. I regularly get 0% packet loss. I can create packet loss by walking to the other side of the house or put a running microwave between me and the dongle. It's an incredibly stable connection.

    I recently upgraded my Linux from Ubuntu 18 to Ubuntu 20, and it should work the same if the OS is allowed to access the dongle and it has the correct permissions (aka dialout group).

  • retiututretiutut Louisiana, USA
    edited May 2021

    The only caveat being the board doesn't disconnect from the GUI even after the GUI is exits (i.e., the Ganglion board light doesn't return to blinking).

    This might be worth looking into, as it involves the BrainFlow backend. GUI sends the correct commands to release the session. Could also be that the command doesn't make it to the board in your setup.

    In 2021, GUI v4 is just a "sanity check". GUI v5 has many stability improvements and is recommended for all users.

Sign In or Register to comment.