Ganglion BLE dropped packet IDs (with Android library)

edited January 2018 in Ganglion
As we are currently still working on the BLE to Android library, I noticed some pretty severe packet loss, which comes out to about a third of all packets.

https://pastebin.com/McjdMhm2

What seems really worrying is that frequently 50+ packets seem to be dropped, which looks like this

01-06 22:12:06.725 3661-4145/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 146
01-06 22:12:06.760 3661-3686/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 0
01-06 22:12:06.762 3661-3686/com.example.david.mymind_v2 E/GanglionBluetoothLeService: Warning: dropped 54 packets.
01-06 22:12:06.764 3661-3687/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 101

or this

01-06 22:12:10.871 3661-3687/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 122
01-06 22:12:10.904 3661-3686/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 0
01-06 22:12:10.906 3661-3686/com.example.david.mymind_v2 E/GanglionBluetoothLeService: Warning: dropped 78 packets.
01-06 22:12:10.908 3661-3686/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 101

or this 

01-06 22:13:26.127 3661-3751/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 137
01-06 22:13:26.131 3661-3751/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 0
01-06 22:13:26.132 3661-3751/com.example.david.mymind_v2 E/GanglionBluetoothLeService: Warning: dropped 63 packets.
01-06 22:13:26.134 3661-3751/com.example.david.mymind_v2 E/GanglionBluetoothLeService:  PacketID 101

Whenever this happens, it always resets to a raw packet and I was wondering if there are even that many packets lost or if thats some kind of bug where it starts over with the packet IDs?


«1

Comments

  • wjcroftwjcroft Mount Shasta, CA
    Florian, hi.

    Is it possible that your "receive packet" routine in Java, needs to be using some kind of better buffering or callback scheme? Such that latencies in the processing time for packets (due to other demands by the Android OS and other apps) -- does not cause packets to drop.

    In other words, you may need to have sufficient buffering ability in your Android app, so that even if the app cannot get much CPU time for even some hundreds of milliseconds -- you still can buffer the packets so they are kept when CPU is available.

    Mentioning AJ @pushtheworld.

    William

  • Hi William,

    That's a good point, I will make sure to pay close attention to it. 

    I dug into it and noticed that we made a mistake which periodically sent a command to the Ganglion, which in turn caused it to lose a lot of packets at once.
    Now that this issue has been fixed, the packet loss is down to about 6.5%.
    This still seems highish to me, especially with the board and tablet lying next to each other on my desk about 20 centimeters apart.
    For reference the tablet is a Huawei Mediapad T3 10 running on Android v7.0 with Bluetooth v4.0.

    Curious I decided to try it on another tablet and 2 Android phones with mixed results. 
    To isolate the BLE connection during debugging, I removed every bit of code used for decompression etc and only tracked the packetID to count the number of lost packets.
    Here are the results:

    Redmi Note 3 (MTK) on Android 5.0.2 with Bluetooth v4.1:

    A/GanglionBluetoothLeService: Lost: 0 out of 64509 % of loss= 0.0


    Doogee X5max pro on Android 6.0 with Bluetooth v4.0

    E/GanglionBluetoothLeService: Lost: 3596 out of 63550 % of loss= 5.658536585365853


    Huawei Mediapad T3 10 on Android 7.0 with Bluetooth v4.0

    E/GanglionBluetoothLeService: Lost: 2689 out of 42640 % of loss= 6.306285178236398


    Teclast X98 Plus II on Android 5.1.1 with Bluetooth v4.0

    A/GanglionBluetoothLeService: Lost: 808 out of 65388 % of loss= 1.235700740196978

    Any ideas on this? Could this issue come from "weaker" Bluetooth chips or is 6% packet loss to be expected?

    Florian
  • wjcroftwjcroft Mount Shasta, CA
    Florian, thanks.

    I would say more likely these different packet lossage figures on different platforms, have to do with CPU availability latencies. Each hardware platform has a unique OS and app environment with different CPU demands. 

    Ideally I think you want your packet collection to happen at something like a 'high-level' interrupt basis, dumped into some kind of (for example) circular buffer. The packets are then removed from this buffer when CPU is available to process them. I have no idea how Android / Java sets that up internally with the BLE library api. 

    But I do know that games and such that demand 'real-time' response characteristics, sometimes use a "closer to the metal" strategy, some examples,




    William

  • I wonder if there is a way to change the interrupt prototcol. Sounds like William may be on to something, make sure you don’t get stuck Processing the data and run out of time to collect the data from the internal buffer. At that distance you should not have 6.5% packet loss.

    Could you share some code? I don’t have an Android but I’ve written enough ganglion drivers to see if something is up!
  • Thanks for the suggestions William. I am not sure if there is anything that can be done, at least easily, but I don't know enough yet about this. 
     

    As mentioned in the previous comment, to isolate this issue, I removed every bit of processing within the app aside from counting the lost packets and the percentage remained within margin of error.

    The code can be found on our github page: 

    which is just an extension of the official BLE code sample from google.

    I could be wrong, but since the code is taken from one of Googles official samples I expected it to meet the requirements.
    From what I have read so far the BLE throughput is device specific as connection interval and packets per interval are variable.

    Theoretical rundown here:

    Practically it looks like 8kb/s is the throughput we can expect
    The currently available average throughput is approximately 8kbps through common mobile operating systems, regardless of the module being used.

    With 101 packets per second at 20 Byte each we are looking at 2020 Bytes/s which right in bewteen the average and maximum throughput. 

  • @fwende wow this is great information! Thanks so much for your research!

    I totally agree though, the Ganglion as of today is running close enough to the max. On some MacOS computers, even very new models, the BLE is at the upper limit of and packets are dropped! I have been working hard to make a new driver based on BLED112 which is a dongle that will handle the bluetooth comms.

    I think we need to reduce the data rate though, especially to support these lower end devices. We could do a high pass filter on the ganglion microcontroller, such as 0.5Hz, and then downsample to 14/16 bits instead of 18/19 and reduce the number of times we need to send data out of the Ganglion.

    I agree, if you took from Google sample code and are not doing anything else in the processing, then the issue is that these packets must not be getting to the software layer stack.
  • That seems in line with how iOS performs worse than Android, due to a longer minimum connection interval and fewer packets per interval as mentioned here. It would not surprise me if MacOS at least in the older versions has the same issue.

    I am not very familiar with the Ganglion Firmware, but would it be possible to adjust the sampling rate to lets say 180 Hz and keeping the 19Bit compression to send 90 packets/s? With 10% less packets and the so far highest loss percentage of about 7% this should be "safe", while still beeing sufficient for most tasks.





  • Nope the sample rate can only be increased :/ anyother ideas?
  • wjcroftwjcroft Mount Shasta, CA
    Florian, hi.

    Did you look into the link I mentioned on the Native Development Kit?


    This is how game developers get real-time performance, by putting some code into C or C++ form. When your code runs in Java there is a lot of overhead and caching, garbage collection, JIT compilation, etc. etc. potential delays and stuttering.

  • I did not get to it yet and since I haven't done this at all it will probably be quite a bit of work.

    Thanks for the suggestions to both of you, I'll keep you updated.

  • Keep us in the loop! I am very interested to hear if you get better results!
  • wjcroftwjcroft Mount Shasta, CA
    I did a bit of looking into the NDK approach. The Android Bluetooth stack is quite complex, and that API is NOT provided by default with the NDK kit. So some of the articles I read suggested to access Bluetooth 'natively', open the connection with the normal Java API. Then when you transition to real-time streaming, it may be possible to use a "JNI Bridge" approach. JNI is the Java Native language Interface spec, that describes how to call from Java down into C code, or vice versa.

    If you do a Google search on NDK Bluetooth, you'll turn up quite a few articles.

  • edited January 2018
    A quick update on the current situation:
    The NDK approach didn't quite convince as it seems like a lot of work with possibly no real benefit.

    Developing your app with native code using the Android NDK isn't necessarily more efficient than programming with the Java language. ..... .Native code is primarily useful when you have an existing native codebase that you want to port to Android, not for "speeding up" parts of your Android app written with the Java language.

    I read up on how to best debug BLE connections and found out that you can enable the Bluetooth HCI log which can be used to monitor the BLE packets. This log is created on the device and contains information about all incoming and outgoing bluetooth traffic.
    With a program like Wireshark the Log can then be analyzed.

    So that is what I did and I noticed that the packets shown as dropped in our App did not register at all, which I verified by comparing the packet id. 

    If you are interested in seeing this for yourself here is a link to the Logcat and the hci log:

    Logcat

    Hci Log
    Packets containing the data have the protocol ATT, length 32 and are from the source f4:e5...

    Btw. the connection interval is 7.5ms taken also from the log. This seems to be the standard setting for Android device
  • This is the same problem I saw on some MacOS computers!!!!
  • This is why I’ve been working on a BLED112 driver
  • wjcroftwjcroft Mount Shasta, CA
    Nice detective work.

    You can see in the diagrams how complex the Android Bluetooth stack is,


    It looks to me like at least three layers of processes: app, Bluetooth service process, then 'stack' / HIDL, which I think may be inside the Linux kernel (thus not an actual user space process per se, but still requiring scheduling, resource contention and context switching.) One or more elements of this chain are simply not getting the CPU cycles they need to service the incoming packets.

    And... I've seen it written in multiple places that Bluetooth 4.x (Low Energy) was never intended for continuous packet streaming, but only periodic status updates. So because of this implicit limitation, the APIs and such were not designed to operate in some kind of circular or double buffered, interrupt serviced manner. The entire Bluetooth stack chain can really only deal with one BLE packet at a time.
  • @pushtheworld
     
    You were saying that the sample rate on the ganglion could only be increased, could you please explain why that is the case and would I need to do to get the sample rate down?

    Thanks!
  • wjcroftwjcroft Mount Shasta, CA
    edited September 2018
    @fwende, hi.

    Since AJ has the BLED112 support written, here is another approach: use an Android device that supports plugging in the BLED112 as a serial port device. The BLED112 apparently has enough buffering that it can deal with OS latencies. You may need what is called an OTG On The Go adapter dongle, that converts the Android micro usb jack, to a full size Type A USB jack operating in host mode.

    Regards,

    William

  • wjcroftwjcroft Mount Shasta, CA
    Corrected my typo in last post, OTA -> OTG,

  • @wjcroft Thanks for the suggestion, this sounds very interesting. 
     
    Would there still be a way to decrease the sample rate by modifying the hardware? We are trying to evaluate our options. 

    Thanks!
  • wjcroftwjcroft Mount Shasta, CA
    Florian, hi.

    If you look at the MCP3912 (ADC) data rate table on page 23, OSR, Output Sample Rate


    You can see that the sample rates are adjusted by factors of 2. The Ganglion MCLK is 3.276 MHz (not 4 MHz) So since the default OSR rate is 200 sps, (I believe) that means you could drop to 100 sps, but then this would be pretty dismal for EEG. 200 is already on the low side (250 or 256 being more typical.) 100 sps would be dicey, since you typically want better than just the Nyquist limit (sample rate/2), which causes lots of aliasing. So say 4 samples per waveform would limit you to 25 Hz maximum EEG. Not very appealing, but possibly could work if you are only interested in low EEG bands.

    I think your best strategy is to take the existing code base for the BLED112 (see Hub code), and try to adapt your Java code to that. The BLED112 has sufficient buffering that it will insulate you from Android OS latencies. And almost all Android phones support OTG these days, especially tablets, and flagship models.

    One last option for adjusting sample rate, would be to futz with the crystal clock that drives the ADC, MCLK. But not very practical. You are going to encounter a range of OS latencies on various Android platforms. So (without BLED112), finding a rate that works would not guarantee you working on all devices.

    Regards,

    William

  • WonkybadonkWonkybadonk Idaho
    edited September 2018
    @fwende Seconding @wjcroft as far as perusing some existing code that streams BLED112 to Ganglion. github.com/OpenBCI/OpenBCI_C might also be enlightening if not so wonderfully OOPed as a java implementation. I've been able to get this streaming data from my Ganglion to the BLED112 in an Ubuntu 18.04 environment with only minor "hey bro, try sudo" issues. As per the @wjcroft comment above using your java to call the OpenBCI_C driver using NDK might not end up being as bad as you think. But then again, the jvms all usually hate me every time I use them.

    The BLE stacks all look a little gnarly compared to the plug and play of the BLED112 so far for me. I was attempting to see if I could figure out how to implement the Bluez stack to get the OpenBCI_C talking to either the CSR dongle or the BLED112 but there's a lot to learn. I'll take a fork at the GanglionAndroidBluetooth repo and see if I can do anything useful on my end :)

    EDIT: Disregard. I misunderstood.
  • wjcroftwjcroft Mount Shasta, CA
    Wonky, hi, Please reply to this post on the other thread,

  • edited September 2018
    Thank you William, as always you are of great help. 

    For now we would be happy to work just with the alpha waves so 12Hz would be enough for us. 
    However if we are looking at 100sps we should get to roughly 45 Hz with Nyquist + 10%, which is the practical minimum iirc. What is your reasoning for 25Hz? 
    The issue with the using the BLED112 is that we are currently working on cross platform compatibility (android + ios) with unity. 
    Supporting this would be a lot of work, which would be better spent on other things. 

    So for using half the sample rate would I just need to change this value
    curSampleRate = SAMPLE_RATE_200;
    in here:

  • wjcroftwjcroft Mount Shasta, CA
    Nyquist limit gives terrible aliasing. Imagine say a 50 hz sine wave that we are sampling. If you are lucky to sample it at the + and - peaks (very lucky indeed), then your 100 hz sample rate gives you basically a triangle wave. OK, that's barely passible. 

    NOW, shift your samples of the 50 hz wave so you capture it at the points where it crosses the axes. Now your wave is grossly distorted. Your 10% will help a little. But 4 or 5 samples per waveform is more likely to give you decent data.

    re: curSampleRate, yes.

    re: android vs ios. My misunderstanding. I thought you were working with Java and Android from your previous posts. Not iOS and Swift or Objective-C. Since Android devices are so much more affordable, your project would have wider leverage with Android. And iOS does support serial port devices,


    Regards,

    William

  • wjcroftwjcroft Mount Shasta, CA
    Your existing code base for parsing the Ganglion packets should not need any modifications. All you need to do is add a layer to your implementation that talks with the BLED112 to get your data stream, instead of the previous Android BLE stack API. Should be straightforward. The BLED112 can buffer multiple BLE packets, unlike the Android OS, which was not intended to stream BLE.
  • wjcroftwjcroft Mount Shasta, CA
    edited September 2018
    re: curSampleRate

    This is an enum, and current code is designed with  SAMPLE_RATE_200 as the lowest possible value. You will need to tweek other sections of code, such as the function at line #754 (config_MCP3912(unsigned long gain)) so you get the proper sample rate code into the register.


    You may also need to tweek other sections of code so that the radio packets are transmitted less often, not sure what governs this.

    Also note that even with these tweeks, you may still get some packet loss, because of timing dependencies in the Android BLE stack. That stack was never intended to operate with streaming data, because it has limited ability to buffer multiple radio packets. As your test data (at 200hz sample rate) has already shown.
  • Do you have something more in depth to read about the point you are making? I have heard, read and learned about the theorem a couple of times but never encountered the 4 or 5 samples per waveform you were mentioning, however I haven't really used it in practice.

    We switched over from pure Android to a game engine, so it is not on you for misunderstanding. 

    Dealing with this issue without the need for extra hardware would be preferred, but I'll look into both options. I ll update this when I have gotten to it, if you (or someone else) would be interested in the packet loss rate with 100sps.

    Thanks,
    Florian
  • wjcroftwjcroft Mount Shasta, CA
    Note that according to the documentation, the Simblee BLE radio transmissions from the Ganglion are ALREADY limited to around 100 times per second, at the 200 hz ADC sample rate. That is why they had to use the compression. So... That means that your BLE packet loss, is ALREADY happening at around 100 hz radio packet generation frequency. 

    You may try to drop that frequency to only say 50 radio packets per second(?) But still no guarantee you will avoid two packets arriving so quickly that the Android OS buffering cannot accommodate. 


    re: Nyquist and aliasing. Do you understand the example I gave with the 50 hz samples of a 100 hz waveform? It will be hugely and badly distorted if sampled close to the zero crossings. The only way to avoid this is with more samples per waveform.



    Regards,

    William

Sign In or Register to comment.