Packet Loss Analysis: Bug or Real Loss? Need Help Understanding
DashBarkHuss
Chicago
Packet Loss Analysis: Packet ID labeling bug or real loss? Need Help Understanding
Recording Details
- Duration: ~5 hours (304 minutes)
- Board: Cyton Daisy
- File format: BrainFlow CSV
- Total packets received: 2,281,491
- Board: Cyton-Daisy
Cyton Daisy Packet ID System
Important context: The Cyton Daisy labels packets from 0 to 254, skipping odd numbers (0, 2, 4, 6, ..., 252, 254), then wrapping back to 0. This gives us 128 unique packet IDs per loop.
The Problem
I'm analyzing packet loss in my recording and found patterns that could be interpreted as either bugs in packet ID labeling OR real packet loss. I need help determining which.
Pattern #1: Duplicate Packet ID + Skipped ID
- Pattern: The same packet ID appears twice in a row, and the expected next ID is skipped
Example: Received IDs
136, 136, 140but expected136, 138, 140- Packet ID 136 appears twice
- Packet ID 138 is completely missing
If this is real packet loss: Packets with duplicate IDs represent actual lost data (127 packets lost per wraparound)
- If this is a bug: The board failed to increment packet IDs correctly, but the data itself is present with wrong labels (minimal actual data loss)
Pattern #2: Backwards Packet
- Pattern: Packet ID decreases instead of increasing
Example: Received IDs
120, 118but expected120, 122- After packet 120, the next should be 122
- Instead received 118 (going backwards by 2)
If this is real packet loss: 126 packets lost in a wraparound (120 → 122 → ... → 254 → 0 → ... → 118)
- If this is a bug: The board sent packets out of order or mislabeled them, but the data is present
Two Interpretations
Interpretation A: It's a Bug
- Packet IDs got scrambled/mislabeled
- Data exists but labels are wrong
- Evidence: Bimodal distribution, clustered with other bugs
Interpretation B: Real Packet Loss
- Hardware correctly reports wraparound (120→122→...→254→0→...→118)
- Actually lost 126 or 127 packets in transmission
Key Evidence
The reason it doesn't seem likely that these are real packet losses is because of the distribution of packet loss in my 5 hours of data.
Distribution of Packets Lost Per Packet Gap:
1-23 packets lost: 1454 gaps ██████████████████████████████████████████████████
24-99 packets lost: 6 gaps
100-125 packets lost: 148 gaps █████
126 packets lost (backwards pattern): 109 gaps ████
127 packets lost (duplicate/wraparound): 885 gaps ██████████████████████████████
This gap in distribution (24-99 range) suggests the 100-127 losses aren't natural wireless interference, but rather a bug in the packet ID labeling. If these were real packet losses, I'd expect to see the number of gaps taper off as the number of packets lost increases.
Timeline shows extreme clustering:
This could be a cluster of bugs, but it could also be a cluster of packet loss so it's not particularly informative but I thought I'd share it anyways.
- All three suspicious patterns (127, 126, and 100-125 packet losses) explode simultaneously at 3.3 hours
The 198-204 minute window contains:
- 652 gaps with 127 packets lost (duplicates)
- 108 gaps with 126 packets lost (backwards)
- 147 gaps with 100-125 packets lost
Outside this window, packet loss is minimal and normal (1-23 packets lost)
Timeline Visualization:
Time 127 (dup) 126 (back) 100-125 Visualization
--------------------------------------------------------------------------------
0-6m 3 0 0 D
24-30m 15 0 0 D
...
192-198m 25 1 0 D B <-- First backwards
198-204m 652 108 147 DDD BBB SSS <-- MASSIVE CLUSTER
204-210m 1 0 0 D
...
Legend: D = 127 packet loss (duplicate), B = 126 packet loss (backwards), S = 100-125 packet loss
Questions for the Community
Is this a known OpenBCI bug? Has anyone seen backwards packet IDs or duplicate IDs?
How can I definitively tell the difference between real packet loss and packet ID bugs?
Should I assume packets are always delivered in
sequential order, or could backwards jumps (120 →
118), duplicates (120 → 120), and large gaps
indicate out-of-order delivery or mislabeled packet IDs?Is there a standard method for detecting accurate packet loss with
OpenBCI/BrainFlow? Am I using the right approach (packet ID
sequence analysis), or is there additional metadata I should be checking?
Impact on My Data
I'm using this EEG data for AI sleep stage classification. This will be used on live streamed data to score sleep in real time, though I'm testing it on recorded data. If the backwards patterns and duplicates are losses, I need to understand what amount of samples were lost to create accurate 30 second epochs. If this is a bug, it changes the amount of samples lost.
Any insights appreciated!
Comments
Hi Dash,
Are you using the previously suggested (on other threads) trick, of putting the dongle on a usb extension cord, and draping the cord+dongle in a way that gets the dongle away from nearby surfaces such as metal cabinets, tables, other equipment? For example, draping the dongle+cord over the top of a nearby monitor so it hangs in the air a few inches away from the monitor?
Is there any correlation with your packet loss and other wireless gear or EMF radiation sources in your area? Do you have access to an EMF meter or app?
https://www.google.com/search?q=emf+meter+app
Low cost EMF meters at Amazon.
https://www.amazon.com/s?k=emf+meter
EMF fields are generated by a wide variety of equipment: AC extension cords, power supplies, conduits (walls, floors, ceilings), LED or CFL lamps, wireless gear, etc. Are there any other nodes in your area using Bluetooth, internet-of-things, etc.?
In general there are no bugs in the Cyton / dongle radio transmissions. Are you getting any info from the GUI Packet Loss widget that would help to pinpoint the cause of the packet loss? The solution is to just get better radio connection between Cyton and Dongle. Keep moving things around until the widget reports better stats.
William
I’m analyzing someone else’s long recording and notice ~10 to ~27-minute stretches with no packet loss, followed by short pockets of loss, sometimes just one packet lost. The worst pocket lasting about 3 minutes a few hours in with a lot of lost packets.
Thanks for the EMF tips. Still, it seems unrealistic to expect zero packet loss during a 4–8 hour session. Shouldn’t the goal be accurate detection and compensation rather than perfect transmission?
If so, what’s the best way to detect packet loss precisely so I can interpolate missing samples?
How accurate is the GUI’s packet-loss widget? If it says “40 packets lost,” is that really 40 samples missing, or is it more of a heuristic? If it’s reliable, maybe I can use the same logic in my Python script.
What I don’t understand is how detection can be fully accurate if it relies only on packet IDs. The counter wraps every 256 samples (or 128 in Cyton-Daisy, which uses even numbers 0–254). That means a normal-looking increment could either mean no loss or a very large one, depending on how many wrap-arounds occurred. How does GUI system tell the difference?
Excuse me, are you saying that your packet loss issue is ONLY with this recording? For the solution, then on future sessions, just use the previous tips I gave. That should reduce packet loss to the minimum.
Please try the previous tips and see what your loss rate is. It should be minimal.
First, use the previous suggestions, then your lost packets, if any, should be easily detected with the packet counter.
Use the widget to see what your general loss rate is, and if you can make any improvements to your environment.
I'm not looking at the code, but my guess is that the widget is just doing simple subtractions. Source code is here:
https://github.com/OpenBCI/OpenBCI_GUI/blob/master/OpenBCI_GUI/W_PacketLoss.pde
“are you saying that your packet loss issue is ONLY with this recording?”
No this is just the sleep scored data set I currently am working with to examine packet loss programmatically. But in the gui sometimes we don’t see packet loss when setting up then find it hours later. I don’t think I’ve ever seen a 4 hour + recording with no packet loss.
Are you saying it’s possible to get packet loss down to zero, or just minimize it?
If zero packet loss is expected then we can aim for that. Is this what most people do and they never have packet loss issues?
If zero loss is not expected, is there a standard way for dealing with packet loss once it’s minimized as far as possible?
My current plan is to use whatever logic the packet loss widget uses and to get the number of packets lost to interpolate. But I’m just unclear if this is the standard or correct route.
Also sincere apologies if I’m being frustrating! I really do not mean to be and appreciate your help. No pressure if you’re not sure of the answer.
Please answer if you are already using the usb extension cord tip I mentioned several comments earlier. This is highly recommended.
Small / tiny amounts of packet loss are normal. And if by monitoring the packet counter you can see a skipped sample, you can try to interpolate. Use the GUI packet widget to fine tune your environmental setup to minimize lossage. This may involve moving equipment, or rooms, etc. An EMF detection gizmo is inexpensive and can find interfering sources of noise in your environment.
Is the Cyton stationary during your sleep monitoring, or is it attached to your head? Now imagine that your bed has a metal frame (box spring or mattress springs) and during the night, at times the Cyton is close to a metal portion of the bed. This has the potential of reducing the packet transmission reception, because some portion of the transmitted signal (from the RFduino on the back of the mainboard) is being absorbed by the bed and does not reach the dongle. This is the same reason we try to clear the dongle to be away from other surfaces.
No, we are using the USB-C adapter that was recommended (on here or the GitHub?). I can add a USB extension
to this setup.
The EMF detection device looks useful and not very expensive, so I'll order one of those.
The device is not connected to our heads when we do these recordings. We usually leave it on a nightstand.
With the extension cord and the EMF detector, I'm sure we can reduce the packet loss more. But I'm still trying to understand interpolation for the remaining packet loss.
I posted on the GUI repo and the firmware repo to try to better understand how accurately we can detect the exact amount of samples lost.
OpenBCI_GUI - Packet Loss Widget: Potential Wraparound Ambiguity Issue
Cyton Firmware - Is Accurate Packet Loss Detection Possible with 1-Byte Packet IDs?
My biggest confusion is around how the 256-value wraparound (0-255) seems to limit our ability to accurately detect packet loss.
The RFduino's used on the mainboard and dongle are no longer stocked at distributors, no longer manufactured. But OpenBCI does have stock. They have a fixed packet size which is very small, about 33 bytes. It is not possible to add more bytes to the packet counter, as the packet is already very small:
https://docs.openbci.com/Cyton/CytonDataFormat/#binary-format
As I mentioned previously, the best strategy is to aim for only a tiny / small amount of packet loss. Therefore you are not going to be seeing situations with hundreds of contiguous packets lost.
With your extension cord, get a simple USB-A type extension cord, plug the dongle into the end of it, and drape it over the top of your display monitor screen. If your nightstand has any metal components, try to get the mainboard away from those, perhaps set it on top of a small box or stack of books. Be sure the mainboard is not near any AC cables, power supplies,
Since your entire setup is fixed, you could just position the dongle next to the mainboard on the nightstand. This then is only inches between them.
True, but I could potentially create custom firmware that removes some of the columns I don't need (accelerometers) to then create a 3-byte packet ID. Then no wraparounds would occur.
Yes we can shoot for this but without accurate packet loss detection can be sure we even achieved no hundreds of contiguous packets?
In the course of eight hours it's seems unlikely that something might not cause a disruption. Especially if our subject walks to the bathroom. Even a small amount of packet loss, under 1% of packet loss results in up to 4.8 minutes in total over eight hours. So I still think it makes sense to have some understanding of how to best handle packet loss outside of just preventing it.
Please try my previous suggestions. I believe you will find that your packet loss then becomes minimal. No need for three byte packet numbers. Modifying the firmware would be very difficult as multiple items would need rebuilding: mainboard, dongle, Brainflow, GUI, etc. You would also then be out of sync with the rest of the Cyton universe. So using any of the 3rd party apps would break:
https://docs.openbci.com/Software/CompatibleThirdPartySoftware/LSL/
Yes I was looking into altering all of those- though I forgot about the dongle.
Ok I will try that when we do our next recordings. However like I said I’m unclear how we can confirm that packet loss is minimal if the packet detection isn’t accurate.
While waiting to do our next recording, I've further analyzed 127-sized-packet losses in my Cyton Daisy recordings and found more evidence these are not real data losses but rather a packet numbering bug.
I think we already had minimal packet loss here.
The math doesn't add up.
Unusual packet loss distribution
In 1 hour of recording (3600 seconds wall-clock time), I found only 3 gap sizes:
Out of 111 gaps, 98% of "lost" packets come from exactly 127-packet gaps. There are zero gap sizes between 3-126. Why would there be no medium-sized gaps?
Sample count vs wall clock math
If we look at 1 hour of wall time, we'd expect about 450,000 samples. If we really lost 5,419 packets we'd expect to see less (450,000-lost samples). However we received 450,891 samples.
We can calculate the amount of samples sent by the board by adding amount received and amount lost. We can use this to see the sampling rate.
⚖️ If 127-sized-packet losses are real:
???? If 127-sized-packet losses are a bug:
The 0.2% sample rate deviation matches known Cyton behavior (no crystal oscillator, but consistent, slightly inaccurate sampling). The 1.4% deviation would be 5-7x higher than typical.
Why exactly 127?
127 packets = one less than a full packet number loop (128 even numbers: 0, 2, 4...254). This creates transitions like
136→136after wrapping around. Having this exact number appear 42 times, with zero other medium-sized gaps and no missing samples compared to wall time, suggests a systematic issue with how packet numbers are being tracked rather than real data loss.Didn't you say previously that sometimes your Cyton is out of radio range with the dongle, when you go to the bathroom?
Why not just use the timestamps (assigned by Brainflow when received) along with the packet numbers, to flag when there is a long period of no samples?
If you have further questions, I suggest you email to (contact at openbci.com) and send a link to this Forum thread. I do not believe the Cyton / dongle packet transmission has any bugs, it is a very simple protocol. If something odd is happening with your experiment setup (large numbers of lost packets), then I would first advise optimizing your setup to minimize or flag transmission out-of-range errors.
In practice when you do it overnight study this is a possibility. I don't believe this happened during this first hour of recording. But even if it did, I think this should still be true:
Packets lost+packets received==packets sent. But that's not what I'm currently seeing; we are seeing way more packets than expected.I'm looking into doing this, and I will share my solution when done, thought it seems a little inaccurate since arrival times are not sample times- but it could be a good enough heuristic. But in my code I'm ignoring the 127 packet losses because they just don't make sense with the wall time and other patterns I'm seeing.
We scheduled a new recording in the EMF-free labs an Northwestern for next week plus ordered an EMF meter to help with setup just incase. We'll see if we see a similar pattern in the new recording.
The upcoming data collection is with a different Cyton and computer, so it won't be a perfect replication. I will try to order an EMF device over to Hawaii for Dr. Karen Konkoly to test with since the original data is from her setup.
I will report back and contact openbci if we're still seeing this issue.