problem with accumulating delay when using LSL Widget

edited April 2019 in OpenBCI_GUI
Hi,

In my research group, we did some investigation of the delay and jitter existing due to problems with the sampling rate. For the testing purposes, we used OP803SL phototransistor connected to the first channel of Cyton+Daisy (16ch in total), that produce square impulses hardware - hw markers). For stimulus display (a white circle on the screen) we used OpenVIbe Designer with the following configuration:

1) OpenVIbe connected to the OpenBCI_GUI via LSL Widget (LSL Streaming option in OpenVIbe acquisition server) with and without drift correction of 2ms / 8blocks /sec
2) OpenVIbe connected to the OpenBCI_Python via lsl_streaming (LSL Streaming option in OpenVIbe acquisition server) with and without drift correction of 2ms / 8blocks /sec
3) OpenVIbe connected directly via OpenVIbe driver from acquisition server with and without drift correction of 2ms / 8blocks /sec
4) OpenVIbe connected to Micromed (Treviso, Italy) via TCP streaming (System Acquisition Plus<--->Acquisition Server Openvibe)

The white circle appeared every 1.5sec (approx. 40min in total, 1600 repetitions). The recordings have been done on a machine with Windows 10 with delay settings set to minimum (according to the instructions). The data have been saved as gdf files with the software markers inserted by OpenVibe (red - marker 0). The difference between 0 (red) and 5 (green) is the effective delay of the system.

image
All 16 channels were recorded, so that the full capacity of transmission can be tested. The electrodes 2-16 were kept in the water to avoid railing .(red 0- software marker (sw), green - 5 marker inserted using (hw- OP803SL), 7 (magenta)-end of stimuli presentation sw) 

1) using OpenBCI_GUI via LSL Widget we obtained this:

No drift correction 
image


With drift correction

image


  
What we can see from these two figures is is that the delay at the 1st 1 trial is about 100ms(t=1.5s) and it grows up to 2500ms (2.5sec) at 1600th appearance of the circle at approx. t=40min. We can see that the delay grows linearly telling us that there are probably additional samples added either by OpenVIbe Acquisition Server or by OpenBCI_GUI. Furthermore, the drift correction enabled on OpenVIbe acquisition server didn't show any effect. (see the difference between these two photos).

2) Using OpenBCI_python script for LSL streaming we obtained the same results. 

3) Using OpenVIbe drivers that directly interface with the Board http://openvibe.inria.fr/drivers-openbci/

No drift correction (keep in mind that here the Y-axis are different)

mean (stdev) delay: 45 (7,5) msec


image

Drift correction 

image


The results here are much better and acceptable. The delay of 50ms with a small standard deviation and no jitter seems to fit most of the experiments. However, what I notice is this pattern of delay that is specific for blocks and I don't understand what is the cause? 


Finally,4) we compared with the CE medical device 32 electrodes Micromed system that interface with the OpenVIbe via TCP streaming between OpenVibe Acquisition Server and System Evolution (Acquisition software by Micromed). We did this just to have a point of reference about a possible delay. We have obtained this:

With drift correction (very similar results without):
Mean: 110.89 ms
St.dev: 19.5 ms

image


It is not perfect, but relatively constant. The bursts seem to be characteristics of the TCP congestion control since the test in this case really involved two different PC-s directly connected with 100Mbps ethernet.

In conclusion, we have to see what is really going with LSL Widget/Acquisition Server, since this behavior is critical not only for ERP studies but also for many other applications. The delay of 2.5sec is unacceptable. On the other hand, i would recommend OpenVIBE driver, but it is outdated, no impedance measurement available, no channel configuration possibility, so it has to be combined with OpenBCI_GUI, that also can create some problems (such as "port in use", because OpenBCI-hub was not closed properly. etc.).



Comments

  • wjcroftwjcroft Mount Shasta, CA
    Alex, hi.

    Thanks for your thorough explanation and documentation of your equipment setup and measurements.

    I'm not familiar in depth with how the OpenViBE AS or drivers do drift and jitter correction, but do note that the OpenBCI_GUI and OpenBCI_Python have no such software layers. My impression is that the direct (older) Cyton driver in OpenViBE, DOES make use of the drift / jitter correction capability that is available in the OpenViBE environment. So this explains the difference in the graphs you are seeing.

    You may not be aware of this, but the sample rate clock on the Cyton is not at precisely 250.00 Hz. This is because (as part of cost and board space factors), it was decided to not use a separate on board crystal oscillator. Thus the actual sample rate coming out of the Cyton might be for example 250.4 Hz, or even 249.5 Hz, etc. The sample rate has very little if any jitter. But it does have a long term and stabile 'drift' away from the exact 250.00 Hz sample rate that parts of OpenViBE may expect. This is not that unusual in EEG equipment, and is the reason the drift adjustment control panel was provided in some of the OpenViBE EEG drivers.

    From your graph we can calculate: 2.5 seconds drift / 40 mins (2400 seconds) = .00104 = 1.04 ms drift per second. 

    At exactly 250.00 Hz, samples come in every 1 / 250 = .004000 seconds = 4 ms per sample

    For example if the clock was actually 249.5 Hz, then 1 / 249.5 = .004008 = 4 ms + 8 microseconds per sample

    1.00104 (drift factor) * 250.00 = 250.26 Hz, so this appears to be the approximate sample rate of your Cyton. Completely within norms.

    ----

    My suggestion as far as ERP studies go, is that you base your analysis on the optical trigger alignment. And forget about precisely counting 250.00 samples per second. If you do need some sort of drift adjustment, I believe you could perform that in some sort of preprocessing step, using similar methodology to how the OpenViBE driver control panel is adjusting it.

    Hope this makes some sense. I think the slight difference from 250.00 Hz is what is causing your concern. This does not have to impact ERP studies or BCI fields, including neurofeedback.

    Regards,

    William Croft

  • wjcroftwjcroft Mount Shasta, CA
    A post on the OpenViBE forum, with much the same comments,


    Note that the original Cyton driver by Jeremy Frey was written in the OpenViBE 1.x era. At that time drift correction was thought to be a reasonable approach. With OpenViBE 2.x that has changed. And the adjustments for drift in the Acquisition Server (for LSL), might not be doing what you expect. 

    Both my comment and the link above, emphasize the importance of using trigger information for your ERP protocols. Recording the trigger information directly on a Cyton EEG channel or Aux channel is optimal. With LSL you also have the option of recording markers / triggers in the LSL stream. But I'd always prefer the direct approach, since there are no hidden variables.

    William
  • wjcroftwjcroft Mount Shasta, CA
    http://openvibe.inria.fr/acquisition-server/

    "The drift correction feature is by default disabled in OpenViBE 2.0.0. When the module is disabled, it will still monitor the drift, but no corrections will be performed."
  • edited April 2019
    Thank you for your detailed response. Of course, optical triggers are the best way to align markers with the signal, and it's been used even with much more expensive hardware. 

    However, the problem is real-time processing and how to handle the drift in the case of BCI? Let's say that LSL is used with BCILAB or OpenVIbe, and the BCI/Neurofeedback session approx. lasts 20-30min, that means that at the end of the continuous session there is a delay of 2sec? As I said, I am not saying that there is a problem with OpenBCI, I just want to point out that there is a need for controlled drift correction in some way (during the experimental break for example). So, maybe backward communication from OpenVIbe/BCILAB to OpenBCI_GUI, etc. is needed? Or some other proposal how to handle it in real-time applications?
  • wjcroftwjcroft Mount Shasta, CA
    Did you read the posted link above from the OpenViBE forum? According to that, OpenViBE officially (as of V2) no longer recommends drift correction. Instead just base your ERP and DSP calculations on the trigger information. 

    There is no concept  "that means that at the end of the continuous session there is a delay of 2sec?". This is only true if you are using the V1 OpenViBE strategy. They have abandoned that approach.

  • wjcroftwjcroft Mount Shasta, CA
    edited April 2019
    You might consider posting some comments on the OpenViBE forum, asking what are their recommendations, since the V1 drift correction has been abandoned. Ask them how to avoid your seeming paradox. This is at root a conundrum based on V1 OpenViBE definitions and concepts. So hopefully V2 OpenViBE has an improved framework for resolving this.

    What the OpenViBE forum link above is trying to convey, is that your DSP analysis (if doing ERP studies), should only concern itself with the samples surrounding the event. Extending out from the event perhaps plus and minus seconds at most. There is no need to require 250.000 Hz accuracy over 10 or 20 minute intervals. 

    If you join the OpenViBE forum as a member, you could actually reply to that very thread link mentioned above. Since Jussi Lindgren (who posted the reply) is a primary OpenViBE developer, you can trust that he is familiar with both V1 and V2 OpenViBE methodologies. 

    Regards,

    William

Sign In or Register to comment.