measured shield regulated 3.3V supply, has .6V dip at 1 second intervals, with resonant ringing

alwayswearshatsalwayswearshats
edited May 2019 in Wifi Shield
Hello,

I have been using my Cyton + WiFi Shield with success but I have noticed that there is an audible (5 kHz+) noise coming from the WiFi Shield. I am powering both boards with a 3.7V Li-ion battery. If I disconnect the Cyton, the audible noise is still present with the WiFi Shield alone.

To further investigate, I used my adjustable DC power supply (Agilent E3620A) with the WiFi Shield on its own, probing the Vbatt+ (directly across the power supply) with my scope.

With the WiFi Shield powered ON, there is a 10 Hz (1 ms duration) pulse that causes my supply voltage to pull down by 200 mV. I think this is some kind of beacon pulse put out by the ESP8266.

With the supply set at 3.50V, there was no audible noise, but as I increased this to 3.75V, I began to hear it and noticeable ringing appeared in the supply trace on my scope. Increasing further caused ringing to get larger in magnitude and appear before, during, and/or after the pulse. Once the supply voltage reached 5.25V, the noise was no longer audible, but there was still some ringing in the scope trace.

Here are scope screenshots with 0.25V increments in the supply voltage:

For the heck of it, I placed a 470uF capacitor across the supply and it did reduce the magnitude of the voltage droop slightly, but it didn't lessen the resonant ringing observed without it.

Wondering if anyone has come across this or has any suggestions on how to suppress the ringing.

Thank you
«1

Comments

  • wjcroftwjcroft Mount Shasta, CA
    Always, hi.

    Interesting. Apparently the ESP chip can make significant current draws in bursts, up to .4 A. Yikes. So this ringing you are seeing would appear to be the rest of the power lines (and inherent capacitance / inductance) responding in a resonant fashion. This page suggests a choke in series to block the resonance.


    In our situation the ESP is not self destructing. But I'm curious now, is this why TWO battery supplies are recommended? Could they have dropped that to one with proper power supply filtering / blocking the ringing getting back to the mainboard Cyton??

    And yet...  All the little boards such as those from Adafruit and others, employing the ESP8266, just use a simple linear 3.3v regulator, like the Wifi Shield board. So how come no one else is complaining about this ringing? Could it have to do with the shortness of their traces? None of these other uC breakout boards are using large caps or chokes.

    Curious of your thoughts.

    Regards,

    William

  • wjcroftwjcroft Mount Shasta, CA
    Mentioning @produceconsumerobot, @daniellasry, @conor.

    Dang wish I could @ mention AJ or Joel. But they've moved on. 
  • wjcroftwjcroft Mount Shasta, CA
    Always, hi. Can you clarify? I believe you are a Build-It-Yourself guy? Are both your Cyton and Wifi Shield your builds? Is it possible there are some circuit anomalies on your Shield, that are not present in our production boards? Such as unusual ground or power planes, etc.?
  • wjcroftwjcroft Mount Shasta, CA
    It's possible this circuit ringing is nothing to be concerned about. At least when the board is separately powered. As you measured, the ringing is only in range of .2 volts.

    And it doesn't get through to the other side of the regulator, correct?
  • alwayswearshatsalwayswearshats
    edited April 2019
    Hi William, thank you for your reply.

    Yes, this is a DIY, and it is possible that these PCBs aren't of the best quality (JLCPCB). The WiFi shield did not have too many components and the smallest were 0603 which weren't too bad to solder under a microscope. I used the 1.0.0 board revision to make the PCBs, without any alterations.

    The only differences with my WiFi Shield with the production model are that I could not source the same C1, C2, and C4 capacitors and had to use alternatives. Wondering if mine have lower ESR and cause stability issues with the AP2112 regulator...

    ComponentValueBOMBOM Modified
    C10.1uFGRM188R71H104KA93D06035C104KAT4A
    C21uFGRM188R61E105KA12DTMK107BJ105KA-T
    C310uFGRM188C80G106ME47DGRM188C80G106ME47D
    C4100uFJMK325ABJ107MM-P12106D107MAT2A‎

    Aside from intermittent bursts that show up in OpenBCI recordings while using the WiFi Shield, I am able to collect data without issue. The audible noise of the board is annoying, though.

    I just grabbed screencaptures for the 3.3V regulated line:

    Looks like it is affected as well (0.6V droop at-worst), and oddly the pulse flips polarity at around 4.25V
  • wjcroftwjcroft Mount Shasta, CA
    Jason, hi.

    We've made several additions to the Github issue, I believe your power supply findings are quite relevant. Feel free to add any comments there.


    and

  • wjcroftwjcroft Mount Shasta, CA
    Changed title of this thread to:

    "measured shield regulated 3.3V supply, has .6V dip at 1 second intervals, with resonant ringing"

    The fact that these dips are not being regulated by the 3.3V regulator, could be responsible for some of the noise and packet losses that have been seen.

    A possible workaround, mentioned on the Github issue post above, is to drop the RF output power of the ESP8266, so it is not causing .4 amp current demands.

    Jason @alwayswearshats, have you done any more experiments or measurements with your setup? You mentioned a test with a 470uF cap across the battery? Would that possibly work better with placing the cap AFTER the regulator? Was your .6V drop measurement made with or without this cap? 

    The general practice with decoupling capacitors, is to place them as close to the position of the momentary current drains as possible. Yet with .4A surges, I've not calculated what sized cap would absorb that kind of momentary load.

    The current idea, is to test new firmware that reduces the RF power output level of the ESP. This should drop the required current surges.

    Please continue to share your thoughts.

    Regards,

    William

  • alwayswearshatsalwayswearshats
    Hi William, thank you for the suggestions. I am going to be out of town for the weekend. I will give some of your suggestions a try when I return.

    Cheers
  • alwayswearshatsalwayswearshats
    edited May 2019
    Hi William, I tried a few things that you had suggested and tinkered around a bit. Apologies for the rambling essay in advance...

    ELECTROLYTIC CAPACITOR ACROSS ESP8266 SUPPLY

    I ended up soldering a 330uF across 3.3V and GND on the top side of the WiFi Shield and kept the leads as shorts as I could and this seemed to do a great job at keeping the 3.3V steady. Using my DC supply I varied the pre-regulator voltage between 3.7V and 5.5V and I no longer observed the oscillations or heard any ringing with this fix.

    Although this kept the WiFi Shield D2 LED solid during data transmission (with no packets dropped), the intermittent spike noise still remained in the traces.

    RF POWER REDUCTION

    I also reduced the RF power output to +5.0 dBm using the method that you had suggested. I did this by adding the following line to the OpenBCI_WIFI/examples/DefaultWifiShield/DefaultWifiShield.ino Arduino file.

    void setup() {
      initializeVariables();

      WiFi.mode(WIFI_AP_STA);
      // WiFi.mode(WIFI_STA);
      // WiFi.mode(WIFI_AP);

      WiFi.setOutputPower(5.0);
    ...
    }

    For some reason when I programmed my WiFi Shield directly from this modified '.ino' sketch in Arduino IDE on my Mac the WiFi Shield did not function properly (D2/D3/D4 did not light up upon power-up and ESP8266 was flickering). I ended up getting this to work by exporting the sketch to a compiled binary file and used the esptool.py approach from the OpenBCI tutorial.

    Unfortunately, using this lower RF power setting (+5.0 vs. default +20.5 dBm) gave quite poor results. In the same room that I normally get solid connection with the +20.5 dBm setting, the +5.0 dBm was dropping packets profusely and with the D2 LED not staying steady. Also, the spike noise that we were trying to lessen was still present.

    REMARKS

    In my opinion, I think there are two likely sources for these noise spikes:

    1) These are due to packets being dropped because of a weak power supply and/or weak RF output power setting. These spikes show up any time your WiFi Shield D2 LED blinks, with a corresponding println() statements in the Processing console. These also show up when you start or stop the stream due to discontinuities between current data and past data or missing data. With a 330uF electrolytic capacitor across the ESP8266 3.3V/GND and four charged NiMH AA, I don't see these. Also, to clear up a comment that I had made a while ago, the spikes appear in both the SD card and stream data files.

    2) These are somewhat tricky to pinpoint, but they are likely due to RF noise or something to do with the bias settings. These do not correspond to D2 LED blinks or packet loss println() statements in Processing. These are intermittent but often show up in a cluster of multiple evenly-spaced spikes. I went through many of datasets taken on different days and I noticed that the spacing between spikes of a given cluster group were consistently 0.65 (~1.5 Hz) seconds apart. Wondering if others with WiFi Shields could check to see if the 0.65s is consistent with their data as it might give a clue...

    I also noticed that if you change the hardware BIAS/SRB1/SRB2 settings or turn channels ON and OFF in the GUI during data streaming you can get the spikes to appear. This kinda makes sense because there would be abrupt DC level changes that would show up as spikes in the filtered data.
  • wjcroftwjcroft Mount Shasta, CA
    Jason, I want to congratulate you on your fantastic engineering detective and remediation work.

    Your work on this shield power supply issue will benefit all Wifi Shield users.

    Can you clarify a couple things. 

    (1) with the 330uF cap on the shield regulated 3.3V side, you can run indefinitely with no noise spikes and no packet loss (except on stream start or setting changes.) Correct?

    (2) have you tried various protocol settings in terms of direct, AP / TCP, UDP, UDPx3. Any difference in performance or loss / noise WITH the cap? WITHOUT the cap, which mode is worst for packet loss / best? The consensus is that UDPx3 is our recommendation without any hardware fixes. But your mod may allow all modes to function as expected.

    (3) if the noise spikes only occur on packet loss, and if no packet loss happens -- Then does this mod essentially fix the noise issue? This may need to be verified in high wifi utilization areas (potential channel interference / retransmissions).

    (4) for smallest form factor it seems like a low voltage electrolytic would give the smallest profile: I see sizes for 6.3V, 10V and 16V are available. There are also Tantalum small caps, but sometimes these have odd failure modes. Even with electrolytics, you want to go with a high quality brand, such as Panasonic. I've repaired monitors before where the failing components were cheap Chinese electrolytics. Could you go even smaller that the 330 uF? Probably not much size advantage though.

    Thanks again,

    William

    PS mentioning Richard @retiutut, Sean @produceconsumerobot, @Conor, Daniel @daniellasry, @Shirley, @openbci, AJ @ajk.
    Need to send Jason some kind of prize or reward(!)

  • wjcroftwjcroft Mount Shasta, CA
    Reading again your post now for clarification, I see that the total noise spike issue did NOT improve with the cap, although we still don't know if this might be dependent on which protocol mode is in operation. However the cap DOES eliminate the packet loss. And we know that noise spikes occur during packet loss situations. More clues...
  • alwayswearshatsalwayswearshats
    edited May 2019
    Hi William, hopefully I can address your questions.

    1)
    When I connect to my WiFi Shield using WiFi Direct (TCP), I do not drop packets (20 minutes is longest I've checked, though). I actually had to walk about 25 ft (+20.5 dBm setting) with my laptop until a handful of packets were dropped.

    When I connect my WiFi Shield through my local network (WiFi Station, TCP), the results are largely dependent on my router connection. There are a few notable bad spots in my house where my laptop has a tough time making a solid connection to my router, and this is also where the WiFi Shield drops significant packets.

    2)
    All my testing has been using TCP in either WiFi Direct or WiFi Station mode. UDP and UDPx3 have never worked for me. UDP will load the Time Series frame but when I click "Start Data Stream" there is no data. UDPx3 doesn't load the Time Series frame for me but gives a blank frame with an error message: "Wifi Shield firmware is out of date". My WiFi Shield is loaded with "DefaultWifiShield.2.0.5.bin" firmware. If someone could point me in the right direction to getting UDP/UDPx3 up and running I would give those a try.

    3)
    Unfortunately, spikes not related to packet drop are present.

    4)
    The 330uF (25V) was something I had lying around that wasn't overly bulky (0.8mm x 13mm can) with a decent capacitance. I am sure one could get away with something a bit lower in capacitance. I think the larger ESR and capacitance of the electrolytic might have tamed the regulator's oscillations. I would need to test this further, though.
  • wjcroftwjcroft Mount Shasta, CA
    Jason, thanks.

    DefaultWifiShield.2.0.5.bin is the latest ESP firmware according to the issues page. Are you running the standalone GUI or from Processing IDE? I'd suggest the IDE, instructions here,


    Then look over Sean's @produceconsumerobot 's suggestions on this issue,


    Thanks for continuing to sleuth this out with us.

    Regards,

    William

  • wjcroftwjcroft Mount Shasta, CA
    Richard @retiutut, in your issue post (on noise spikes with shield UDP mode), 


    You show a time series with noise spikes. Jason, with a small hardware mod (adding a capacitor to improve power supply stability), has shown that this eliminates packet loss. (And therefor some but not all spikes.) However effects of various transport mechanisms is not clear yet. He is testing exclusively with TCP. (And seeing cyclical noise at about .65 second intervals.) He would like to test with UDP, and UDPx3, but is hitting some GUI roadblocks. Can you give him some hints? I already pointed him at Sean's hints in Issue 82,


    Thanks,

  • wjcroftwjcroft Mount Shasta, CA
    Jason, this beta is in final testing and very close to final release status. Might address any UDP issues you're having.

  • retiututretiutut Louisiana, USA
    edited May 2019
    We haven't messed with UDP/UDPx3 with the recent updates. What I can tell you is that the Accelerometer Widget has changed for the better, but it hasn't been tested for WiFi+Cyton, and there is an issue with cranking up the sample rate to 1600Hz with Ganglion+Wifi in the accelerometer widget only.

    I would recommend using the TimeSeries widget only when testing WiFi-related issues and the GUI. Adjust the Layout to show only one widget, and select TimeSeries. The GUI will remember your settings for next time when you stop system or close the GUI.
  • wjcroftwjcroft Mount Shasta, CA
    edited May 2019
    Jason, a question on your previous comment:

    "Unfortunately, using this lower RF power setting (+5.0 vs. default +20.5 dBm) gave quite poor results. In the same room that I normally get solid connection with the +20.5 dBm setting, the +5.0 dBm was dropping packets profusely and with the D2 LED not staying steady. Also, the spike noise that we were trying to lessen was still present."

    So at power output 20.5 dBm and then 5 dBm, you saw no change in the amplitude of the spike noise? Was your time series on auto scale, vs. fixed?

    If these spikes are indeed some kind of bleeding of the wifi RF into the Cyton circuit board, one might expect they go down (even just a little), with less power.

    One other thought on these spikes. Could they be related to the "revision level" or manufacturer of the ESP8266 package? I don't think ALL Wifi Shield users are seeing the spikes. Otherwise there would not be such variations in reporting of working Shields. I would say MOST users are not seeing the spikes. Wonder if it might be related to the manufacture date of the shield itself, since that determines ESP manufacturer and rev level. Is there any way to read out this kind of info from the ROM built into the ESP?


    William

  • alwayswearshatsalwayswearshats
    Hi William, I was using a fixed scale in Time Series. To be honest, I was not really paying much attention to the amplitudes of the spikes when at +20.5 dB and +5.0 dB - just if they were present or not.
  • alwayswearshatsalwayswearshats
    edited May 2019
    Also, I spent some time today trying to better characterize the WiFi Shield 3.3V supply stability. I took a spare PCB and just soldered the power supply components (bypassing the switch). I used an electronic load to draw ~300 mA at 10 Hz via function generator and NMOS transistor.

    More info on the method I used:
    http://rohmfs.rohm.com/en/products/databook/applinote/ic/power/linear_regulator/linearreg_easy_stability_app-e.pdf

    Here are the related images for the test setup and results that I found:

    Takes-aways:
    - Latest WiFi Shield has stability issues dependent on input supply voltage (battery) between 4.00V and 4.50V
    - Worst stability at ~4.25V with loud audible noise ~15 kHz

    In these measurements, I did not test how adding electrolytic capacitors across 3.3V would help. I am going to perform the same tests with different electrolytic capacitors to see which does the best job at suppressing oscillations.
  • wjcroftwjcroft Mount Shasta, CA
    Jason, thanks.

    Wow, great test / measurement setup and graphs. Did you do that output plotting all by hand, or is that generated by a piece of test equipment you have? Very cool.

    Looking forward to your characterization of the optimal uF cap. The 330 uF is not that unwieldy, but it does seem on the large side. The 1 Hz regular 'beacon' pulses are one thing, but it might also be informative to just watch with your scope the 3.3V DC level while the ESP is under more typical load during a session. So the power spikes will be much more dense than even 10 Hz?

    Did you ever try the UDP or UDPx3 with the latest GUI 4.1.2 beta? TCP will probably have the worst noise spikes because doing a lot of work with packet transmit / receive / acknowledgement. Whereas UDPx3 will mostly just be blasting out packets, I don't even think they are acked. 

    Regards,

    William

  • alwayswearshatsalwayswearshats
    Hi William, I just tried the 4.1.2 beta GUI release for OS X and UDP/UPDx3 still don't work for my WiFi Shield with Cyton.
  • alwayswearshatsalwayswearshats
    edited May 2019
    Here is an update regarding the WiFi Shield AP2112 3.3V supply output capacitor debug.

    After thorough stability testing, I have identified the source of the 8 kHz to 20 kHz ringing is the 100uF ceramic capacitor 'C4'. Removing this capacitor alleviates the instability in the power supply, but the 10uF ceramic capacitor 'C3' cannot handle the large current draw, so the supply droops a few hundred mV.

    Using my 300mA electronic load test setup (see above), it seems that an electrolytic capacitor with a value of at least 100uF would do the trick to avoid oscillations and keep the droop less than 100 mV (with Vin > 4.00V). When Vin was below 4.00V, the droop was many 100s of mV irrespective of the electrolytic's value.

    Probing the 3.3V supply on my WiFi Shield with different capacitor combinations, I observed very similar results as my test setup, except it seems that the load is a bit larger than 300mA. From the frequency response traces, you can see that the 100uF ceramic (C4) causes oscillations while the electrolytics (or no electrolytic) caused NO sustained oscillations. Looking at the time series for the WiFi Shield beacon pulses, it seems that the higher the value of the electrolytic the better, but 1000uF was no better than 680uF under worst case Vin=4.00V.

    Testing Results:

    Remarks:
    • 3.7V LiPo for the WiFi Shield + Cyton IS NOT recommended.
    • 4xAA NiMH or Alkaline (4.8V to 6.0V) IS recommended.
    • WiFi Shield 100uF ceramic capacitor 'C4' causes instability in AP2112 power supply, for 3.75V < Vin < 4.50V (likely because of VERY low ESR).
    • Removing 100uF 'C4' and soldering electrolytic capacitor close to ESP8266 supply rails with value 330uF <= Cout <= 680uF IS recommended to keep voltage droop <= 200mV (Vin=4.00V).
    • If Vin >= 4.50V and Cout >= 100uF, droop is less than 100mV.
  • alwayswearshatsalwayswearshats
    edited May 2019
    [Jason, I just edited your above post to insert your requested text edit. Thanks.]

    Edit above:

    "From the frequency response traces, you can see that the 100uF ceramic (C4) causes oscillations while the electrolytics (or no electrolytic) caused oscillations"

    should read:

    "From the frequency response traces, you can see that the 100uF ceramic (C4) causes oscillations while the electrolytics (or no electrolytic) caused NO sustained oscillations"
  • wjcroftwjcroft Mount Shasta, CA
    edited May 2019
    Jason, thanks for these measurements and trials.

    Question: for our users in the field that already have the shield, and would like to make the simplest possible improvement (by adding ONE capacitor), what do you suggest? I understood from your initial posts on this thread, that soldering in a single 330uF electrolytic across the regulated 3.3V rails, corrected ALL the power supply issues. Did this not also correct any instability from C4? 

    So you instead are recommending removing C4 and replacing it with a 680uF? Seems in that case that soldering directly onto the now empty C4 SMT pads would be easiest for a novice soldering person. Is there a significant advantage to actually placing it as close as possible to the ESP?

    Thanks again for your engineering expertise.

    William

  • wjcroftwjcroft Mount Shasta, CA
    edited September 2019
    Wouldn't it be easiest for a novice soldering person, to just leave the 100uF SMT cap in place, then solder one lead at a time to a radial electrolytic, such as the 680uF?

    Also, especially in the case of the Ultracortex and Shield with Daisy (16 channels), the weight of the AA battery pack(s) is substantial. If we use the 680uF Panasonic ultra low ESR,

    https://www.digikey.com/catalog/en/partgroup/fr-series/3611
    https://www.element14.com/community/docs/DOC-43515/l/new-low-esr-radial-capacitor-from-panasonic

    Is a LiPo battery cell (around ~3.7v) going to be that much worse than the AA pack?

    Regards, William
  • alwayswearshatsalwayswearshats

    Hi William, it has been a while since I reviewed this, but my recommendation remains to remove C4 (100uF ceramic) and replace it with a >300uF electrolytic capacitor. I would not recommend ultra low ESR capacitors as that is very likely why C4 was a problem for this voltage regulator.

    Regarding the LiPo, I would not recommend using one with the Cyton+WiFi Shield as there is not enough voltage headroom. I still recommend using 4 AA batteries. It's a bit awkward but I superglued a 4xAA battery pack to a comfort unit screwed into Cz. Works fine.

  • wjcroftwjcroft Mount Shasta, CA
    edited September 2020

    Here's a good Application Note from Texas Instruments, discussing how the output capacitor needs careful consideration for LDO low drop out linear regulators. See page 5 and beyond for the linear regulator section. I might see if I can find a suitable not-low-ESR cap. Tantalum type might be the way to go.

  • ShirleyShirley

    @alwayswearshats Hope you had a nice weekend! I'm reviving this thread to update you on the wifi shield. After reviewing the above comments, it appears the way to go is to remove C4 (100uF ceramic) and replace it with a >300uF electrolytic capacitor (noting to not use an ultra low ESR capacitor), as you clarified. I'm getting ready to have OpenBCI's PCB manufacturer update the wifi shield assembly files accordingly. If there's anything you'd like to mention before I proceed, please let me know!! You can reach me directly at [email protected]

    -Shirley

  • PBarrowsPBarrows UK

    Does this fix work? I can't find anything attesting to whether the WiFi shield can now be used without issue. I would appreciate advice on whether this fix has been successful.
    PaulB

  • wjcroftwjcroft Mount Shasta, CA

    @PBarrows, no, the Wifi Shield is no longer supported. The larger capacitor was 'thought' to be a temporary workaround for this Cyton-Shield issue. But in fact the root cause is in the variant of the Wifi Shield firmware that talks to Cyton. It seems that that interaction between the Cyton firmware and the Shield firmware, is causing the Shield ESP8266 chip to draw excessive current, hence the voltage dips. And eventually dropped packets.

    This never happened when the Shield was used the Ganglion. So at least that configuration is still workable. After some thought the engineering team decided it would be better to move on from the Shield concept, and has plans for a V2 Cyton board. (Incorporating better faster Wifi link.) Unclear timeframe at this point, because the Galea product line took precedence.

    If you have a Ganglion-Shield setup, everything should still work fine.

    Regards, William

Sign In or Register to comment.

© Vanilla Keystone Theme 2026