driver "Latency Timer" fix for macOS 11+ & M1 M2

ScottThomasMillerScottThomasMiller Brooklyn!
edited October 19 in Cyton

Hi! I am trying to apply the FTDI fix on my MacBook M1 by following these instructions:

https://docs.openbci.com/Troubleshooting/FTDI_Fix_Mac/

When I run this instruction:

sudo kextunload /System/Library/Extensions/FTDIUSBSerialDriver.kext

I get the following error:

Executing: /usr/bin/kmutil unload -p /System/Library/Extensions/FTDIUSBSerialDriver.kext
Error Domain=KMErrorDomain Code=71 "Could not find: No bundle was found at /System/Library/Extensions/FTDIUSBSerialDriver.kext." UserInfo={NSLocalizedDescription=Could not find: No bundle was found at /System/Library/Extensions/FTDIUSBSerialDriver.kext.}

I've tried FTDI versions 1.4.7 and 2.4.4. Both installers run without errors, but neither seem to install the driver into /System/Library/Extensions/FTDIUSBSerialDriver.kext.

I do however find the driver installed under /System/Volumes/Data/Library/Extensions/FTDIUSBSerialDriver.kext, but when I try to unload it I get a different error:

Executing: /usr/bin/kmutil unload -p /System/Volumes/Data/Library/Extensions/FTDIUSBSerialDriver.kext
Error Domain=KMErrorDomain Code=3 "Error occurred unloading extensions: Missing extension with identifier com.FTDI.driver.FTDIUSBSerialDriver : Incompatible architecture: Binary is for x86_64, but needed arch arm64e" UserInfo={NSLocalizedDescription=Error occurred unloading extensions: Missing extension with identifier com.FTDI.driver.FTDIUSBSerialDriver : Incompatible architecture: Binary is for x86_64, but needed arch arm64e}

Please help.

«1

Comments

  • wjcroftwjcroft Mount Shasta, CA

    Scott, hi.

    The VCP page,

    https://ftdichip.com/drivers/vcp-drivers/

    Shows only Intel x64 drivers for macOS.

    The M1 is an ARM chip. Have you tried reaching out by phone or email to FTDI, to see what they recommend for the M1 Macs?

    https://ftdichip.com/technical-support/

    Seems like they should be aware of this situation and working on a solution. In the meanwhile, if you are using the OpenBCI_GUI to gather the data stream, it already performs (by default) some internal buffering operations to reduce the impact of the driver stuttering. But if you are instead using Brainflow or other real-time streaming software to access the Cyton data, expecting that there will be zero latency introduced -- then a fix from FTDI will be more urgent for you.

    I somehow thought that some OpenBCI engineering staff were already testing with M1, but that might be only with the GUI. Mentioning Richard @retiutut.

    Regards, William

  • Hi William. The stuttering happens with the OpenBCI GUI, and with my custom Swift app which uses BrainFlow. Is there no other workaround or solution for the M1? Maybe running in Rosetta mode, or using the WiFi shield?

    I sent an email to [email protected] I'll keep you posted.

    --Scott

  • wjcroftwjcroft Mount Shasta, CA

    Another comment I would have for FTDI, is that they should make adjustments in the driver parameters EASY to do on Mac. As you can see on Windows, this is trivial using their control panel. No such ability on Mac. Which is why the workaround involves editing an 'Info.plist' file.

    It's possible that FTDI would know how to do this for Apple's default driver.

  • I'm working with a support person named Cameron via email. He is responding quickly. I asked him which built-in driver I should unload. He replied that he needs to verify with their Mac team in the UK.

    While I wait, I am trying to hack this out myself. I think I found the built-in driver on my M1. I first did:

    ioreg -w0 -rc IOUSBHostInterface -l |grep FT

    And I noticed in the output a driver named com.apple.DriverKit-AppleUSBFTDI. So I unplugged my dongle, unloaded the driver via:

    kmutil unload -b com.apple.DriverKit-AppleUSBFTDI

    When I plugged the dongle back in and looked for its device in /dev/cu*, it's not there! So far so good. Next I unzipped Cameron's recommended 1.4.7 driver, moved its contents into my Applications folder, ran it, and granted it permission to run via Security & Privacy.

    I can now see /dev/cu.usbserial-DM0258EJ. But I cannot find the Info.plist to edit. There is no such directory /System/Library/Extensions/FTDIUSBSerialDriver.kext.

    --Scott

  • wjcroftwjcroft Mount Shasta, CA

    It might be best to continue the conversation with the actual real engineers at FTDI in the UK. The support person you are emailing with is unlikely to have any detailed information on how to alter the latency timer settings. FTDI engineers really need to test and document how this is done with the new beta test version of the driver and M1. It's frankly daunting the current OpenBCI instructions on how to modify the 'Info.plist' parameters, which are targeted at the previous kext driver version. This new driver is using the new 'Dext' (driver extension) technology.

    I'm going to send you my email address. Please forward your conversation with FTDI. And any future emails you get regarding the M1 support. I'll followup with them and you with more specific tech information on what we are trying to achieve: how that was done in the past and asking how Dext's handle this.

    William

  • wjcroftwjcroft Mount Shasta, CA

    Scott, yes, I discovered the Info.plist myself a couple days ago. No surprise that the whole app package is code signed.

    I hope that FTDI replies to the email we sent. They obviously have the ability to alter these driver settings. And should give Mac users the same capabilities that Windows users already have with the driver control panel. I will 'remind' them of this with an email followup.

    William

  • I saw your email. Thanks.

    I used my local ID to sign it, but apparently that's not good enough. The internet seems to think I need to pay $100 for a real Apple developer ID, and use that to sign it instead.

    I did the following to sign it:

    codesign --deep --force --verbose --sign "[email protected]" ~/Downloads/FTDIUSBSerialDextInstaller_1_4_7.app
    /Users/scottmiller/Downloads/FTDIUSBSerialDextInstaller_1_4_7.app: replacing existing signature
    /Users/scottmiller/Downloads/FTDIUSBSerialDextInstaller_1_4_7.app: signed app bundle with Mach-O universal (x86_64 arm64) [com.ftdi.vcp.dext]

    But when I click the install button, I get: "Invalid code signature or missing entitlements"

    I've been meaning to get a real Apple dev ID anyway, so I guess it's time I shelled out the $100.

    --Scott

  • wjcroftwjcroft Mount Shasta, CA
    edited November 2021

    Just posted this on the FTDI Community Forum:

    https://www.ftdicommunity.com/index.php?topic=547.msg2009#msg2009


    Quote from: FTDI Community on September 27, 2021, 05:47:33 AM
    Hello,

    That would be the last version of macOS to support .kext drivers, this would be 10.15, but there is an extra notarization step for .kext drivers required by Apple in 10.15, that is not noted in that document. As such it would be safe to assume these instructions work up for drivers up till macOS 10.14.

    Best Regards,
    FTDI Community

    Hi "FTDI Community" / moderator,

    I've been in contact with @ScottThomasMiller regarding his difficulty adjusting the macOS 11 FTDI 'latency' setting. (macOS 11 is required for Apple hardware using the new M1 CPU.) For our Windows customers, this is extremely easy: they just bring up the Windows driver control panel, and type in the new Latency value. In this case '1' ms is the Latency value we need for optimum packet latency with the OpenBCI Cyton product.

    What we are requesting: is a similarly easy way to do this on macOS 11. That should not require modifying Info.plist files, or signing a driver. It should be an external file that you look for, upon driver loading time. This would have fields similar to a .plist, for adjusting values like are possible on Windows. But it would not have to be a .plist, could even be something simple like one line per adjustment: "variablename value".

    Can you give us a timeline for when FTDI will release this macOS 11 compatible driver with adjustable settings? The lack of this is causing great difficulty with our Mac customers.

    Thanks,

    William Croft
    OpenBCI Forum admin

  • William is correct. It might be that FTDI is having trouble getting the necessary entitlement from Apple. Or they've essentially closed shop and have no one around who remembers how to code :)

  • neurosynthneurosynth Valencia, ES

    Hey everyone! Was wondering if there was any update/solution for this? Just ordered a new M1 Macbook (preloaded with Monterrey and un-downgradable ???? might be a whole different ball game than Big Sur) and wanted to head off any potential issues, perhaps there is workaround involving Rosetta, or even Wine?

    Those forum responses by FTDI did not exactly inspire confidence, but @ScottThomasMiller and @wjcroft , perhaps they sent you guys some more encouraging emails? (Thanks for all of the legwork so far)

    Spoke with @retiutut a couple weeks ago, he mentioned that the FTDI buffer is stuck at 16ms. While not ideal, would it be possible just to smooth the incoming data and live with extra 16ms latency? I use the cyton for audio and synthesizer applications; 16ms is right at the upper threshold of noticeable latency, and this will likely much more forgiving than latency felt from playing a physical instrument.

    Thanks!
    -- Calvin

  • ScottThomasMillerScottThomasMiller Brooklyn!
    edited December 2021

    Hi Calvin. Still no help from FTDI. I had to give up and move on. My team decided to live with the jitter we have, for now. We are only at the very earliest, proof-of-concept stage, so we don't yet require extreme precision and accuracy.

    For smoothing we use Mathematica to ingest the EEG and marker streams, and do lots of maths on it. Even still our primary researcher sits in a wooden shed behind his house to run the experiments, so that he doesn't have to deal with the noise from the A/C circuits in his house, which I personally think is pure genius.

    --Scott

  • neurosynthneurosynth Valencia, ES
    edited December 2021

    Hi Scott,

    Gotcha, good to know the performance isn't a total dealbreaker, my work is also pretty early stage right now but hoping this can be resolved by summer 2022.

    I came across a completely unrelated thread of people struggling with FTDI/M1 issues; someone mentioned that the application Serial 2.0 fixed their problem. But I'm not going to speculate too much until I get everything together in the same room, maybe everything is fixed in Monterrey. No experience with Mathematica, do most of my work in Max/PD and Python, hopefully I can find a way in Max to smooth everything over.

    Funny you mention the wood shed, I recently discovered an entire cottage industry of pricey RF-blocking hats on amazon for people afraid of 5G, perhaps one of those would work well on top of an ultracortex

  • mkeetermkeeter Cambridge, MA

    I ran into the same issue with the latency timer, and discovered this thread while searching for answers.

    In the end, I wrote a small C program that uses libftdi to adjust the latency timer. This fixed my issue, and may help you all as well!

    You can install libftdi through Homebrew, then compile and run the program with something like

    cc -I/opt/homebrew/include/libftdi1 -L/opt/homebrew/lib -lftdi1 main.c -o adjust_latency_timer
    ./adjust_latency_timer
    

    You'll have to run the program each time you connect the FTDI cable to the computer.

    (There are a bunch of other details in this blog post)

    Hope that helps!

  • wjcroftwjcroft Mount Shasta, CA

    @neurosynth, thanks for that report on your M1 experience. This thread is primarily dealing with the FTDI 'latency' issue on M1 machines, which causes 'stuttering' of received radio packets from the Cyton. The actual bottleneck is in the dongle FTDI usb-serial chip, which is mistakenly buffering and holding onto packet data. This was not so much an issue with earlier Mac, because a workaround was possible using driver Plists. On Windows the fix is trivial, because the setting can be made in a driver control panel UI.


    We just had a new member, Matt Keeter, @mkeeter, join. Who is going to post on this thread regarding his macOS workaround for setting fixing the latency issue. His blog has a section on FTDI:

    https://mattkeeter.com/blog/2022-05-31-xmodem/#ftdi

    And he offers this macOS stand alone C program to set the latency. Matt, where is this latency value stored then, in the macOS driver, or some type of flash ram on the FTDI chip? So this program just needs to run a single time?

    https://gist.github.com/mkeeter/c43c3990ecdb8dcb6547ac3dbac8e881

    I hope you have updated the FTDI community thread on this same deficiency:

    https://www.ftdicommunity.com/index.php?topic=547.msg2009#msg2009

    Mentioning @ScottThomasMiller @retiutut

    Regards, William

  • mkeetermkeeter Cambridge, MA
    edited June 3

    I think the latency timer value is stored in volatile memory on the FTDI chip, so it's reset to the default of 16 if you unplug the FTDI cable from the computer.

    I did a quick test to confirm this – since the program reads and prints the latency timer before setting it, I see

    $ ./adjust_latency_timer
    Got latency 16
    Set latency to 2
    
    $ # re-run with cable still plugged in
    $ ./adjust_latency_timer
    Got latency 2
    Set latency to 2
    
    $ # unplug + replugging cable, and the value resets itself
    $ ./adjust_latency_timer
    Got latency 16
    Set latency to 2
    
  • How exciting! Thanks mkeeter. I will check it out.
    After 8 months I finally heard back from Apple Feedback Assistant regarding this issue. Their reply is:

    • Latency timer can be configured with ioctl IOSSDATALAT.
    • buffer size is not configurable.

    --Scott

  • wjcroftwjcroft Mount Shasta, CA
    edited June 9

    Scott, thanks.

    Regarding the ioctl call, is that with Apple's driver, or FTDI's? Seems unlikely FTDI would not know about their own supported driver calls. On the other hand, if this call is NOT in their driver, should be easier for FTDI to add this to their code than the link I shared via email.

    Most likely this ioctl does not 'remember' the value across dongle insertions.

    Quite a few matches online:

    https://www.google.com/search?q=ioctl+IOSSDATALAT

    William

  • sthsth US
    edited August 31

    The Apple built in FTDI driver is on both Apple Silicon and the Intel systems. You can set the latency after opening the file reference with the

             if ( -1 == ioctl(serialfd, IOSSDATALAT, &microseconds) ) {
                // set latency
                fprintf(stderr, "\nError calling ioctl(..., IOSSDATALAT, ...)\n");
                return(-1);
              }
    

    Despite the Apple documentation claiming the parameter is set in microseconds, testing has shown that it is unfortunately set in milliseconds. I have successfully set the latency to 1 milliseconds. A value of zero does something weird and should be avoided. In addition a bit more info is in the ioss.h file in the IOKit Serial Framework.

  • wjcroftwjcroft Mount Shasta, CA

    @sth and @ScottThomasMiller, thanks so much for verifying the Apple driver ioctl IOSSDATALAT command works, and the proper units.

    If the FTDI version is mistakenly installed, can that be permanently undone, so that the Apple driver takes precedence?

    Richard @retiutut and @Shirley, I hope this latency mod can get into new OpenBCI_GUI versions. And also wonder if it is supported in previous macOS versions in the 10.x series. Big Sur is 11.x, Monterey is 12.x.

    William

  • retiututretiutut Louisiana, USA

    Thanks, I have added an official issue to the GUI repo so this can hopefully be added to the next version of the GUI. Thank you all for investigating this!

    https://github.com/OpenBCI/OpenBCI_GUI/issues/1079

  • if the FTDI driver is to be removed, remove it from /Library/Extensions (and maybe /Library/StagedExtensions/Library/Extensions) then reboot. You can check for any non-apple drivers being active with the following terminal command, kextstat | grep -v com.apple
    This latency is supported and tested by me as working in Mojave, and Monterey, I haven't tested Big Sur but it has been around for awhile. I would expect it also works in Catalina, other 10.X series. As to integrating it into the OpenBCI that is up to someone else. I have a completely different application.

    I have attached a test program file. Sorry it has my specific FTDI test device with a loop back connector hard wired into the code, but change it to your serial number. Basically if I set the latency to 2, it takes 2 seconds for a 1000 characters, if I set it to 1 it takes 1 second. Have fun. Dang, I can't attach files... Ok, here is the code.

    #include <unistd.h>
    #include <stdio.h>
    #include <fcntl.h>
    #include <sys/ioctl.h>
    #include <sys/time.h>
    #include <IOKit/serial/ioss.h>
    
    #define ITERATIONS     1000
    
    /* globals used for caching the time */
    u_int64_t startTime;
    
    void StartClock ( void ) {
      startTime = clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW);
    }
    
    void StopClock( double *call_time ) {
      u_int64_t deltaTime = clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW) - startTime;
      *call_time = (double) deltaTime / 1000.0;
      if (*call_time < 0.0) fprintf(stderr, "start nSec = %llu, Delta nSec = %llu\n", startTime, deltaTime);
    }
    
    int tty_disconnect(int fd) {
      if (fd == -1) return -1;
    
      tcflush(fd, TCIOFLUSH);
    
      if (close(fd) != 0) return -1;
      return 0;
    }
    
    int
    main() {
      unsigned long microseconds = 2ULL;
      int serialfd, index;
      unsigned char char1, rchar;
      double elapsedtime;
      speed_t speed = 57600;
      struct termios tty_setting;
    
      printf("About to  set IOSSDATALAT = <%ld> microseconds\n", microseconds);
      printf("IOSSDATALAT = 0x%lx\n", IOSSDATALAT);
      printf("sizeof unsigned long = %lu\n", sizeof(unsigned long));
      printf("Sizeof microseconds = %lu\n", sizeof(microseconds));
      printf("Sizeof int = %lu\n", sizeof(int));
      printf("Sizeof IOSSDATALAT = %lu\n", sizeof(IOSSDATALAT));
    
      if ( -1 == (serialfd = open("/dev/cu.usbserial-FTDITBSI", O_RDWR | O_NOCTTY | O_CLOEXEC )) ) {
        fprintf(stderr, "Error opening Device\n");
        return(-2);
      }
    
      if ( -1 == ioctl(serialfd, IOSSDATALAT, &microseconds) ) {
        // set latency
        fprintf(stderr, "\nError calling ioctl(..., IOSSDATALAT, ...)\n");
        return(-1);
      }
    
      if ( -1 == ioctl(serialfd, IOSSIOSPEED, &speed) ) {
        perror("Error setting device speed: ");
        return (-1);
      }
    
      tcgetattr(serialfd, &tty_setting);
      if ( cfsetspeed(&tty_setting, speed) < 0) {
        perror("Error setting device speed: ");
        return (-1);
      }
    
      tty_setting.c_cflag &= ~(CSIZE | CSTOPB | PARENB | PARODD | HUPCL | CRTSCTS);
      tty_setting.c_cflag |= (CLOCAL | CREAD);
      tty_setting.c_cflag |= CS8;
      tty_setting.c_cflag |= PARENB;
      tty_setting.c_cflag &= ~PARODD;
      tty_setting.c_iflag &= ~(PARMRK | ISTRIP | IGNCR | ICRNL | INLCR | IXOFF | IXON | IXANY);
      tty_setting.c_iflag |= INPCK | IGNPAR | IGNBRK;
    
      /* Raw output.*/
      tty_setting.c_oflag &= ~(OPOST | ONLCR);
      /* Raw output.*/
      tty_setting.c_oflag &= ~(OPOST | ONLCR);
    
      /* Local Modes
      Don't echo characters. Don't generate signals.
      Don't process any characters.*/
      tty_setting.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG | IEXTEN | NOFLSH | TOSTOP);
      tty_setting.c_lflag |= NOFLSH;
    
      /* blocking read until 1 char arrives */
      tty_setting.c_cc[VMIN]  = 1;
      tty_setting.c_cc[VTIME] = 0;
    
      /* now clear input and output buffers and activate the new terminal settings */
      tcflush(serialfd, TCIOFLUSH);
      cfmakeraw( &tty_setting );
      if (tcsetattr(serialfd, TCSANOW, &tty_setting)) {
        perror("tty_connect: failed setting attributes on serial port.");
        tty_disconnect(serialfd);
        return -1;
      }
    
      fprintf(stderr, "Success calling ioctl(..., IOSSDATALAT, ...)\n");
      fprintf(stdout, "Starting sending %d charcters\n", ITERATIONS);
      StartClock();
      for (index=0; index<ITERATIONS; index++) {
        if (-1 == write(serialfd, &char1, 1)) perror("Error writing to device: ");
        if (-1 == tcdrain(serialfd)) perror("Error calling tcdrain: ");
        if (-1 == read(serialfd, &rchar, 1)) perror("Error reading from device: ");
        if (char1 != rchar) {
          fprintf(stderr, "Bad character match\n");
          break;
        }
      }
      StopClock(&elapsedtime);
      printf("Total Time = %f uSeconds\n", elapsedtime);
      printf("Time per iteration = %f uSeconds\n", elapsedtime/ITERATIONS);
      close(serialfd);
      return (0);
    }
    
  • Hi everyone, I appreciate the information you've included in this discussion. I'm currently running a course with students who have Apple M1 and M2 chip laptops in addition to Intel Macs and Windows machines. We are encountering this latency issue across the board, with the exception of my M2 Mac. Is there a solution that is suitable for a classroom of students to try?

  • wjcroftwjcroft Mount Shasta, CA

    @brainmusic, thanks for your comment.

    Can you clarify why ONLY your own M2 Mac is not affected by the latency? Are you doing anything different on that machine? Can you confirm that all other M1 and M2 Macs at your school ARE seeing the latency?

    Mentioning Richard @retiutut, @Shirley and @Andrey1994.

    From Richard's previous comment we know an issue is open,

    https://github.com/OpenBCI/OpenBCI_GUI/issues/1079

    And that Andrey has a PR,

    https://github.com/brainflow-dev/brainflow/pull/551/files

    The question is, how can @brainmusic test this modified Brainflow-GUI 'easily' without needing to himself custom build Brainflow and the GUI?

    Regards, William

  • Do any of the Arm based systems have any drivers in /Library/Extensions other than the ones distributed by Apple?
    Are all the Arm based systems running the same version of Mac OS?

  • this pr is not merged yet because I still need some help with testing it, I have no hardware for it. Until its merged and added into the gui the only options is to build everything manually

  • wjcroftwjcroft Mount Shasta, CA

    Andrey and Richard @retiutut, thanks.

    Unfortunately I doubt that the recent novice users commenting on this thread have the tech skills to both build Brainflow test editions (from a Github Pull Request), and to use that to build a test GUI installer.

    So I'm kindly asking if Richard or you or some other knowledgable engineer can do that test build and make it available. As we can see there are several eager test users who DO have the new macOS and M1 M2 hardware on this thread.

    Mentioning @Shirley, @Conor, @joeartuso. Please see this previous comment,

    https://openbci.com/forum/index.php?p=/discussion/comment/18393/#Comment_18393

    Thanks, William

  • Andrey! William! What's up? I've been swamped lately but I can carve out some time to assist.

  • wjcroftwjcroft Mount Shasta, CA

    Scott, hi. Thanks. Can you look over the last few comments on the thread here? If you are able to build Brainflow from Andrey's Pull Request and then incorporate it in the GUI, and then make that available to @brainmusic, that would be great. Then we can get test results from both you and Brainmusic.

    OTHERWISE, it may be best for engineers inside OpenBCI to coordinate the testing. As subsequent mods may be needed.

  • Hi everyone @retiutut, @wjcroft, @ScottThomasMiller, @Andrey1994, @sth, I really appreciate the swift group response!
    @wjcroft, I will need to recruit a few exemplar students (Intel Mac, PC, M1, etc.) as testers to try to answer your questions in more detail. Unfortunately, as it is a classroom environment it is hard to keep track of who installed the FTDI drivers, etc. and what effect this had on the GUI behavior.
    Interestingly, while I did not get an error message in installing the FTDI driver on my M2 machine, I could not find the corresponding .kext file under /System/Library/Extensions or the other places mentioned in this thread. I'm presuming it did not install correctly, thus saving me from the latency issues(?)
    As an aside, I noticed that sometimes the group is mentioning using BrainFlow-- the last time I used OpenBCI extensively was 5+ years ago before BrainFlow existed, and I'm accustomed to using the UDP/LSL outputs to stream data to other software. Are your setups dependent on using BrainFlow as an alternative to this, and would you suggest anyone having these problems switch over?

  • wjcroftwjcroft Mount Shasta, CA

    Grace / Brainmusic, hi.

    Impressive that your electronic music classes at University of Colorado are using the OpenBCI Cytons.

    re: FTDI. NO, the new potential solution is using the built-in Apple serial port driver, which has a way of canceling the latency issue via a special system call. The FTDI customization information on our website was written for older macOS versions and Intel hardware; so should be updated ASAP once this testing is complete. None of the kext / editing Info.plist stuff applies on M1 M2.

    re: Brainflow. This is automatically built into the GUI as a library. So once the new Brainflow M1 M2 mods are tested with the GUI, all GUI modes should work without latency, including the Networking Widget streaming via UDP or LSL. However if you wish to write custom applications in a range of languages that process the data stream directly (no LSL or UDP), we recommend the Brainflow library. This is superior to LSL, UDP, OSC, etc. access.

    https://brainflow.readthedocs.io/en/stable/Examples.html

Sign In or Register to comment.