Bursting behavior of data from Python?

edited April 2016 in Software
Not sure if this should go in software forum, please move if appropriate.

I use Python to interface with the hardware. I have a V3-32 main board and a Daisy board, collecting 16 channels EEG data. I don't see any markings on the USB dongle to indicate a version, so there must just be one?

Anyhow, here's the problem: the time stamps for the samples being recorded in the callback function (the arg for board.start_streaming) are showing that the samples are bursting at roughly 0.5sec apart. In other words, every 0.5sec or so there are a bunch of data points. It's the same for every channel. In theory, the callback should be getting called every 1/125 seconds and so the time stamp of each sample should be separated by roughly 0.008 sec.

Here is a very rough psudo-code for Python:

import time

def callback(sample):
store_chan_data(sample.channel_data)
store_timestamp(time.clock() - time_master)

main:
time_master = time.clock()
board.start_streaming(call=callback)


Any idea why it's bursting like this? Or is that just how the radio functions?? If that's just how the radio functions, then can I just assume the samples actually are being collected 1/125 sec apart on the actual OpenBCI hardware?

If that's not how it's supposed to work, is it more likely an issue with the serial port, the PySerial module or something wrong with the radio firmware?


Thanks!

Comments

  • Perhaps it would be easier to just ask this: is it a fair assumption to just set the timestamps based off the sample.id number (a continuous version, not the one that loops around every 255)??

    Like, if the sample.id numbers are [0,1,3,5,8,9], then the equivalent timestamps would be 1/125 * [0,1,3,5,8,9]  ??


    That would make things a lot easy and I could just get rid of the Python timing.
  • wjcroftwjcroft Mount Shasta, CA
    If you use the search box on the lower right for Python, you can see several other threads reporting anomalies with the data stream.

    http://openbci.com/forum/index.php?p=/discussion/643/packet-loss-with-openvibe-and-python

    http://openbci.com/forum/index.php?p=/discussion/378/openbci-python-unreliable-linux-mac-acquisition

    Etc.

    The data packets and their sequence numbers should reliably increment by 1 each time. Only very occasionally should you see other increments when a packet is lost. How does the data stream look with the OpenBCI_GUI? I assume you have applied the FTDI OS adjustments to avoid serial stream buffering.

    http://openbci.com/forum/index.php?p=/search&Search=LatencyTimer

    William

  • Hi @wjcroft

    I'm not actually losing the packets. I'm getting all the sample.id numbers (in sequence). The issue I'm facing is that when I use Python's time modules to time all the function calls, then the timing of when the samples are recording isn't every 1/125 sec. It's much more bursty behavior. And that could be a number of reasons.

    But if the OpenBCI hardware can reasonable ensure that sample.id = i is spaced apart from sample.id = i+1 by 1/125sec, then I'll just use (a continuous version of) the sample.id times the sampling period as a "timestamp".
  • wjcroftwjcroft Mount Shasta, CA
    Yes, the samples are spaced by either 8 ms (or 4 ms 8 channel). Isn't that apparent when you run OpenBCI GUI??

    It would be great if some resolution was found for the Python issues. Many users have reported this, as the previous links show. It also seems to be OS dependent. Can you try another OS?

  • Why would OpenBCI's GUI know the "true" timing of the samples?? Unless the hardware is also sending timing information in the packet (and nothing I've read so far shows that it is), then the GUI is making the same assumption.

    My guess is the radio uses some kind of burst mode to save power, rather than a truely continuous stream. Thus, when an independent timer (like in Python) times how long it takes between calls to the Callback, which depends on the serial port reading function in open_bci_v3.py, it shows these bursts, as there is nothing on the serial port for 0.5sec then all of a sudden there's a bunch of data for several sample.id numbers worth.

    Just a guess though.
  • wjcroftwjcroft Mount Shasta, CA
    No, my suggestion for you to try the same test with the GUI was to confirm that there is no bursting with the GUI. Please do this test. The graphs should update smoothly. The issue is with Python, not the OpenBCI hardware. Unless you have a very unusual situation.

    Yes the radio is bursting at 4 or 8 ms intervals. One sample packet per transmission.

    Have you configured the LatencyTimer as suggested?

  • edited April 2016
    I already know that the GUI does indeed plot smoothly, as I ran it when I first moved the hardware to our Win7 station. But again, that could simply be it assuming the timing and having enough data to update the graph on a timely/smooth manner. For example, if 0.5sec * 125Hz worth of samples arrive every 0.5 sec, then it could update the graph with one data point every 1/125 sec and not miss a beat.

    I don't really want to debate on the GUI. I don't use it, other than as a sanity check when I first bring the hardware to a new machine.

    The issue could well be with Python or more precisely the PySerial module. I don't know. Truly, all I care about is the timing between samples being 1/125 sec. I'll trust the OpenBCI hardware, within whatever PPM tolerance the crystal, A/D and micro are able to maintain.
  • wjcroftwjcroft Mount Shasta, CA
    Can you please confirm that you have set the LatencyTimer? Without that adjustment, the serial link is indeed bursty. And shows as such on the GUI.

    re: PPM crystal specs. The mainboard sample clock is NOT independently crystal controlled, and some slight drift is to be expected. Also see,

    http://openbci.com/forum/index.php?p=/discussion/313/ads1299-sample-rate-drift-or-jitter

    http://openbci.com/forum/index.php?p=/discussion/129/ads1299-sample-rate-threading



  • Thanks for the info about the clock drift.

    Lo and behold, you were right after all. I will swear to you that the GUI ran smoothly when I first came over to this station, but indeed when I just ran it now it was only updating roughly twice per second. The LatencyTimer was 16ms. Changing to 1ms smoothed out the GUI and I assume would have also helped with my timing issue using the Python timers. Of course, now I've rewritten all my code to assume the timestamp from the OpenBCI sample.id number. Oh well...

    Thanks for the patience!! And sorry for being a mule. Ha!
Sign In or Register to comment.