Synchronising Python datetime's to raw data & alternatives [resolved]
I have recorded my events with the datetime in python, since I think that's what OpenBCI cyton does too in the raw data. Since it's accessing the same system time it should be the same. So theoretically:
(last_datetime - first_datetime) * 250 = number of samples in the raw data right?
where first and last datetime are from the last column in the raw data
That's not the case, for me:
(last_datetime - first_datetime) * 250 = 235,957 samples
raw data = 200,880 samples
What the heck?!
Because now I can't trust my recorded event datetimes.
What is going on?
How do people usually synchronise their data?
Comments
Mango, hi.
We've explained the sample rate and buffering issues on previous threads,
https://openbci.com/forum/index.php?p=/discussion/2831/sampling-rate-cyton-daisy-in-unity
https://openbci.com/forum/index.php?p=/discussion/2329/how-steady-is-cytons-sampling-rate-resolved
The setting for the buffering on your FTDI driver can also play a factor,
https://docs.openbci.com/docs/10Troubleshooting/FTDI_Fix_Windows
Regards, William
Thanks man, so from what I understand the timestamps on the EEG data are what are received at the end, hence not are not exactly when the EEG samples were actually recorded. This results in some latency issues and some samples are omitted.
If I simply care about the frequency distribution 1sec from the onset of the event. Would I still be able to line up the python datetime I recorded from the system to the timestamps on the raw data roughly?
In other words, would it line up enough for it to be useful?
Any timestamps are only approximate because of buffering, latencies, OS and driver timing, radio packet delays, etc. So Just treat the timestamps as giving you an approximate idea of the date, and time within a second or two.
No, that is not true. All the samples are there and at the chosen sample rate. For Cyton, 250.xxxxx. As explained in the previous links, there is not a crystal controlled clock on the Cyton ADS. So rate may differ very slightly from 250.0000. For example might be 250.0234. This is considered a slight 'drift', but will not effect your signal processing. Note that 'drift' is different from 'jitter', where the sample interval keeps changing around. That is not the case.
If you want a general solution for time synchronization in the lab environment, LSL does that.
https://www.google.com/search?q=labstreaminglayer
If you do NOT need this level of sophistication in event synchronization, I suggest you use the more feature-full and easier to use Brainflow interface. But maybe you are already doing that.
https://brainflow.org/
William
Thanks William!
After testing Brainflow for 10 seconds I am getting 2484 samples which means it's only off by 16 samples. This means I can use the python datetime objects to synchronise with the timestamps I get from data right?
Umm, I'm not sure how you are measuring "10 seconds", but 2484 / 10 = 248.4 samples in your measured interval of 10 seconds. 248.4 Hz, samples per second. The sample clock on the Cyton is never off by that amount. At most the 'drift' or slew might be a tenth of a Hz or so. Possibly as much as a few tenths of a Hz.
So I would suspect your calculation is not measuring the time as accurately as you assume.
If you are only synchronizing to a one or two second level, then real time clock timestamps should work.
As the previous links pointed out, the timestamps recorded by both the GUI and Brainflow, are assigned when the packets are received at the laptop CPU. Not when the packet is received by the Cyton mainboard. Thus such timestamps reflect OS, buffering, radio, serial port, etc. latencies and buffering.
William
Well, I set the the port to be open for 10 seconds when using BrainFlow.
I see, I guess what I should be asking is what is the best way to synchronise data and record an event?
Im looking for an event that happens in a 200ms window so I do need things to line up.
Still not understanding how even though I ran it for 10 seconds I'm not getting 2500 samples. I would understand if I got 2500 samples but the timestamps were off but just missing timestamps?
Do you understand that most laptop / desktop operating systems have many many levels of latencies and buffering involved? Including OS process scheduling, Python interpreter, etc.
I already explained and gave links, showing that the Cyton sample rate is NOT exactly 250.0000 Hz. But rather differs by around a few TENTHS of a Hz. This is because Cyton ADC does not include a crystal controlled sample clock. But the 250 rate is "close enough" for any signal processing you need to do.
"Timestamps" you see in GUI CSV or Brainflow, are assigned imprecisely after all the buffering and latencies already described. These are assigned by the laptop CPU, ONLY after all buffering and latencies contribute inaccuracies.
If your event has a digital or analog output signal, then it could received as an "external trigger", this is recorded simultaneously with the 8 data channels. Using the 'Aux' channels, 3 of them. These serve dual purpose and can act either as accelerometer 3 axis inputs, or as trigger channel inputs.
https://docs.openbci.com/docs/02Cyton/CytonExternal
Regards, William
Corrected typo: "...differs by around a few TENTHS of a Hz..."