Best practices for handling uneven timestamps from Brainflow/Cyton
programmatix
UK
Hi there.
I'm capturing data from the Cyton using some simple Brainflow Python code that writes to file. The key part of this is 'board.add_streamer(f"file://{filename}:w")'.
I've recently found that the timestamps in this file are quite odd. Ideally they'd be 4ms part, but instead they are mostly something like 0.025ms apart, with an intermittent jump of about 60 milliseconds.
Data:
$ head -n 20 2024-09-13-21-24-32.brainflow.csv | cut -f 23
1726259073.024538
1726259073.024589
1726259073.024616 <-- 0.0027ms gap from previous here
1726259073.024641
1726259073.024665
1726259073.024690
1726259073.024713
1726259073.024737
1726259073.024762
1726259073.024786
1726259073.024810
1726259073.024834
1726259073.024859
1726259073.024883
1726259073.024907
1726259073.085258 <-- ~60 millisecond jump here
1726259073.085329
1726259073.085353
1726259073.085378
1726259073.085402
I'm recording on a Raspberry Pi 5, from the dongle.
I've read through all the forum answers I could find, such as
https://openbci.com/forum/index.php?p=/discussion/1328/csv-data-files-contain-only-bursts-of-data-not-continous-streaming
https://openbci.com/forum/index.php?p=/discussion/2329/how-steady-is-cytons-sampling-rate-resolved
I've learnt from this that the Cyton itself should be outputting at something very close to a steady 250Hz (maybe with a small amount of fixed drift). So the buffering I'm seeing is probably introduced in some part of the stack between the dongle and the Brainflow.
Does anyone have any best practices for how to handle this? Should I be trying to recalculate estimated real timestamps based on the expected sample rate (which will be challenging, as the whole reason I'm doing this timestamp analysis is because I want to handle dropout periods where the Cyton couldn't communicate with the dongle). Are there any driver tweaks people can recommend? Should I be using the FTDI driver?
Comments
Hi programmatix,
The time stamps are NOT generated inside the Cyton. Instead they only are assigned after the radio packets arrive at the laptop in the Brainflow library. Thus they are subject to various buffering delays imposed by the radio transmission, USB system, OS scheduling latencies, etc.
You have to only depend upon the 250 Hz sample clock in the Cyton. This clock is not crystal controlled, so the resulting final sample rate may have a slight offset of say a few hundreths Hz plus or minus. This stays constant and does not jitter.
Only use timestamps to determine rough time of day, not sample times.
William
Follow the recommendations for your appropriate OS,
https://docs.openbci.com/Troubleshooting/FTDI_Fix_Windows/
https://docs.openbci.com/Troubleshooting/FTDI_Fix_Mac/
To avoid dropouts, keep your Cyton within a few meters of the dongle. Use a USB extension cable to get the dongle AWAY from the table and computer cabinet. A suggestion is to drape the dongle / cable, over the top of a nearby monitor. This gets it a couple feet away from the table which might be metallic.
Hi @wjcroft, thank you for the tips.
The dropouts are unavoidable (at least when streaming to the dongle), as they happen when I physically leave the room where the dongle needs to be - this is part of a home PSG setup. When the Cyton is in the same room as the dongle, the signal is ultra-reliable.
I understand that the timestamps are not coming from the Cyton, and that the buffering is coming from some part of the stack and could be related to many factors. I'm coming to the conclusion that it's going to be very challenging and probably unfruitful to debug that stack (getting into the realm of real-time computing), and I'm probably never going to get reliable timestamps that way. So I'm currently focusing on an approach that writes to the SD card instead. This is raising its own separate problems, in that I'm not reliably getting data files written for the expected period - but I'll raise those in a separate thread if needed, to keep this one on-topic.
One quick question on FTDI, and just out of curiosity: I did previously find the FTDI driver fix link you posted, but I wasn't certain if the FTDI driver still helps? The documentation (and my own experiences) suggest that the dongle currently works fine without it, and it was plug-and-play on both the Pi and a Windows laptop. Does the FTDI driver provide any benefits these days?
Appreciate your help again.
So my impression is that you have the dongle in the bedroom along with the computer. And if you leave the sleep environment the signal is unreliable. Isn't this ok, since you already know that you are in a non-sleep period?
If you want wider coverage, consider using a longer usb extension cable, up to 16 feet, which could be positioned at the doorway of your room, possibly allowing coverage of the radio signal outside the bedroom.
Again, I don't understand why you need millisecond accuracy timestamps. They certainly are accurate at the level of seconds. And this would allow you to see long gaps indicating some kind of radio coverage issue.
Don't think you need to do the SD card.
In the past, the FTDI 'hack' was needed to avoid extra buffering in the usb serial port driver. It seems likely that this fix has been incorporated into the standard OS drivers. If you are not seeing repeating gaps in your sample stream (such as viewing with the GUI), then the usb buffering is not happening.
William
I believe the GUI still has a special widget named "Packet Loss". Watching those numbers change as you reposition your Cyton and dongle can help fine tune the locations.
That's correct, the dongle is on a Pi 5 in the bedroom.
And just to be clear I'm not complaining about the unreliable signal - it's totally expected when I'm the other end of the house, and when I'm near the dongle, the signal is great.
The dropouts are only an issue because when I convert from the Brainflow CSV format into EDF and MNE FIF formats, those do not allow storing per-sample timestamps, and the disappearing data causes artifacts such as my sleep appearing to end 15 minutes earlier than it really did, the EEG data not matching up with external sources (video), and so on.
I appreciate the tips on trying to avoid the dropouts, but it's not really possible in my situation. The Pi/dongle is already pretty close to the bedroom door, and there are just times when I need to leave the bedroom and be well out of range - mundane tasks like getting a drink etc. The only way to avoid them completely is to use the SD card method, which is part of why that's appealing.
Yes for sure, I only need accuracy of around a second, to allow me to sync up with sources like a video stream. The issue is more the complexity of working out what timestamps are missing, given the bursty nature of the Brainflow CSV data. But, that isn't impossible, it is an option.
But I'm currently leaning towards getting an SD card workflow working. It avoids the dropout issue completely, and it removes points of failure (the Pi, streaming).
It does has its own workflow issues, particularly that I have to store the initiation time somewhere external. If you'd consider a feature request, it would be great if I could send over an epoch start timestamp with my 'K' command, which would be written into the file instead of the current timestamp, and/or used as the file creation time. This would save me needing to write the start time into a database somewhere, plus avoids the current issue of having N files all named OBCI_XX.txt with the same creation time. (I will rename them of course when I have this workflow solved, using the stored initiation timestamp - but it would be great to just avoid both issues in the first place.)
Oh I should mention also, that while I only need second accuracy for the current PSG focus, I would like the option to do ERPey things eventually, and will need millisecond accuracy then.