CSV data files contain only bursts of data, not continous streaming

njnj Santa Cruz, CA
in Cyton

I
have the Mark IV headset with the Cyton 8-channel board connecting via
the Cyton bluetooth. I use the stand-alone GUI application v3.0.1.
Windows 10 on a fairly high-end PC.

I
am trying to get started with recording data. The GUI is set to 250 Hz.

I see csv data files being created in the
"SavedData" folder. However, when inspect them it seems the data is transmitted only in "bursts". A burst
lasts about 4 milliseconds. During that time period I get 3-5 rows each millisecond (as read from the time stamp), so all in all there are about 12-20 rows.  Then there is NO data for about 0.47
seconds, followed by a second similar burst, followed by another 0.47 second gap. This pattern repeats. Why isn't it there a continuous streaming? Shouldn't a sample 250 Hz imply that I get precisely 1 row every 4 millisecond? I am not very savvy with electronics so perhaps I am missing something basic here.

Comments

  • wjcroftwjcroft Mount Shasta, CA
    You need to adjust the Latency setting,


  • njnj Santa Cruz, CA
    Thanks!
  • rivardarivarda
    How is the Latency adjusted on a Mac?
  • wjcroftwjcroft Mount Shasta, CA
    http://docs.openbci.com/Tutorials/10-Mac_FTDI_Driver_Fix
  • rivardarivarda
    Thanks for the help.  I removed the existing FTDI driver but then stopped.  The link instructions refer to a dongle that did not ship with my Ganglion board.  Do I need to purchase this dongle in order to change the latency setting, or is there an alternative method?  My issue is not just with live streaming data, but also with the data saved as CSV.  Does this issue exist with data coming over the WiFi shield?  I planed on purchasing the shield in the future if I could get some reasonably good data out of this system.
  • wjcroftwjcroft Mount Shasta, CA
    Rivarda, hi.

    Sorry, I thought you had the Cyton. The two links above only apply to Cyton. What hardware / software are you using with Ganglion? Have you tried on another laptop? Are you using the CSR dongle or Mac or what?

    If you are seeing stuttering with Ganglion, it has no connection to the FTDI stuttering seen on Cyton.

    I would first try a test with another system to rule out problems with your hardware or OS. Mentioning AJ @pushtheworld.

    William
  • rivardarivarda
    I am on a Mac powerbook.  The Ganglion is connected over bluetooth.  The CSV data time stamps show data coming in packets as described in this discussion.  What makes you think that it is a problem with my laptop?  
  • wjcroftwjcroft Mount Shasta, CA
    We don't know where the issue is. A common strategy to narrow down the causes is to try with a different OS or laptop. What Mac OS are you running? Have you tried other versions of the OpenBCI_GUI, etc.? This is the first I've heard of the stuttering with Ganglion; so I don't believe it has been posted here on the forum before. I mentioned AJ because he is the developer.

    Regards,

  • rivardarivarda
    I am running 10.12.6.  I have only used the most current version of the GUI (3.0.1).  In the data set below you can see that data points 15-18 all have identical time stamps.

    Data
    0, 16.22, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.225
    1, 10.71, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.227
    2, 7.22, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.227
    3, 17.84, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.251
    4, 14.46, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.251
    5, 2.57, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.252
    6, 14.49, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.252
    7, 15.00, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.281
    8, -3.12, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.281
    9, 5.37, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.282
    10, 16.06, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.282
    11, 7.01, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.310
    12, 4.07, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.310
    13, 12.02, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.311
    14, 13.81, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.311
    15, 9.65, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.341
    16, 17.63, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.341
    17, 18.75, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.341
    18, -0.57, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.341
    19, 8.43, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.370
    20, 21.87, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.370
    21, 6.24, 0.00, 0.00, 0.00, 0.000, 0.000, 0.000, 16:54:33.371

  • wjcroftwjcroft Mount Shasta, CA
    AJ @pushtheworld, could you comment on the Ganglion timestamps he is seeing? Shouldn't the samples be advancing by about 5 milliseconds per line?

    Both Ganglion and Cyton have crystal controlled clocks that set the actual sample rates very close to 200 or 250 (respectively) samples per second. 
  • pushtheworldpushtheworld
    These timestamps are stored at the time of writing to the file, not when the sample was actually taken. Further, the ganglion specifically, sends two samples per packet, which is why we see always two identical timestamps after each other. Because the processing app cannot do BLE connection to the Ganglion.


    rivarda i would also like to reiterate that this thread has nothing to do with your issue.
  • wjcroftwjcroft Mount Shasta, CA
    "i would also like to reiterate that this thread has nothing to do with your issue."

    What is being referred to is the initial portion of this thread, which was caused by the Cyton serial port latency. This Ganglion timestamp appearance IS indeed a separate topic.

    AJ, do the time stamps work differently (more accurately) on the Cyton? 
  • njnj Santa Cruz, CA
    (Had to take a break in this work. Back now...). I adjusted the latency setting to 1 as your tutorial described. It led to some improvement but the data is still coming in bursts. Prior to the change there were gaps of about 0.47 seconds and the gaps were very regular. Now, the gap size varies. On average a gap lasts 0.3 seconds but there are some sub-second stretches without a gap. I have tested this on two computers, with similar results. (Again, one of the PC's is pretty high-end). Please advice.


  • pushtheworldpushtheworld
    If you want better then that time stamps, I would recommend opening an
    issue on the OpenBCI GUI github page requesting the feature! We could
    implement the time sync feature baked into the Cyton in the GUI. We can get the time stamps evenly spaced using a micro implementation found on the OpenBCI_NodeJS_Cyton
  • njnj Santa Cruz, CA
    Hi Pushtheworld - are you saying that what I experience right now with large gaps is currently the regular expected behavior? (I hope not...)
  • wjcroftwjcroft Mount Shasta, CA
    @nj, do you have the latest FTDI drivers? With the proper driver and the latency setting adjustment mentioned previously, there should be no large gaps like you are seeing.


    William

  • njnj Santa Cruz, CA
    edited December 2017
    Yes, I had installed FTDI drivers from the page you are referring to. I re-did it again to be sure. So, when I go to the Windows Device Manager at look at the Port COM settings it says "Driver: FTDI", "Driver Date: 8/16/2017" and "Driver Version: 2.12.28.0".  If I see this, I assume that the drivers are OK. Is that correct?

    And yes, the USB serial port settings are: "Latency timer: 1 ms" (changed from default).

    I also tried changing the USB Transfer rate (receive and send): both were set to 4096 bytes. I changed it to 64. That did not help. The gaps are still there.

    The reason it took me so long to answer this thread is that I also suddenly started to get railed or distorted signals on all channels at which point I was close to giving up on this product. Today, most channels started to work again and I have no idea why. Perhaps due to adjustment of the physical fit. It seems extremely sensitive in that respect.

     
     


  • wjcroftwjcroft Mount Shasta, CA
    @nj, thanks for the update.

    When did you purchase your Cyton? Unless it was this year, you may have older firmware in the mainboard or dongle. I think you can check the firmware revision level from the GUI.

    Still... It's very unusual that the Latency fix does not solve your issue. The transmit / receive buffer sizes should be left at their 4096 size default.
  • njnj Santa Cruz, CA
    edited December 2017
    The Cyton kit was purchased around May this year. I could not figure out how to get the firmward revision level.

    However, I took a closer a look at the data. As I said before, the time stamp does not progress evenly. There are 10-20 rows with exactly the same timestamps, or the total progression for these 10-20 rows is at most 1-2 milliseconds. Then, a comes a leap of perhaps 50-100 ms. This pattern repeats.

    But, it appears like I am no longer missing any data points. Rather, the data points are not getting a correct time stamp. This is based on the following observations:

    If I take take a random sequence of say 1000 rows, then the difference between the first and last time stamp is in the vicinity of 4 seconds, which corresponds to a new row on average every 4 ms ( = 250 Hz), which to my understanding is what it should be. Second, when I plot the series of electrode signals (Ch1) with row number (not timestamp) on the x-axis I get nice looking oscillations.  

    Here are a few rows from the file. (I removed some columns and transformed "logged_time" into "elapsed_time" so time starts at row zero). I added a column "Diff" that shows the progression elapsed_time. 

    (To make sure we're on the same page: I take my data from the text file that is generated in the GUI subfolder "SavedData")

    ----

    elapsed_time Diff Ch1
    00:19.986 4974.74
    00:19.986 00:00.000 1829.13
    00:19.986 00:00.000 2469.17
    00:19.986 00:00.000 5653.31
    00:19.986 00:00.000 5486.01
    00:19.986 00:00.000 2151.18
    00:19.986 00:00.000 2066.26
    00:19.986 00:00.000 5307.15
    00:20.077 00:00.091 5890.33
    00:20.077 00:00.000 2640.68
    00:20.077 00:00.000 1710.71
    00:20.077 00:00.000 4748.05
    00:20.077 00:00.000 6169.22
    00:20.077 00:00.000 3234.43
    00:20.077 00:00.000 1490.66
    00:20.077 00:00.000 4161.16
    00:20.077 00:00.000 6297.67
    00:20.078 00:00.001 3830.55
    00:20.078 00:00.000 1483.6
    00:20.078 00:00.000 3555.07
    00:20.078 00:00.000 6163.02
    00:20.078 00:00.000 4372.14
    00:20.078 00:00.000 1519.34
    00:20.078 00:00.000 2946.81
    00:20.078 00:00.000 5983.87
    00:20.078 00:00.000 4996.44
    00:20.078 00:00.000 1786.19
    00:20.078 00:00.000 2366.6
    00:20.078 00:00.000 5543.03
    00:20.079 00:00.001 5509.26
    00:20.079 00:00.000 2175.81
    00:20.166 00:00.087 1923.79
    00:20.166 00:00.000 5136.41

  • njnj Santa Cruz, CA
    So, it seems this has been discussed before in multiple places. For example:

    https://github.com/OpenBCI/OpenBCI_GUI/issues/129

    My understanding is now that the timestamp (in the text file in "SavedData") is set by the PC. Is this timestamp inherently unreliable? If so, how do people do their analysis? Do you just assume 250 Hz and that each new data row means time has progressed 4 ms? (unless you need to synch against a stimuli).




Sign In or Register to comment.

© Vanilla Keystone Theme 2026