16 ms·
Debugging bare-metal STM32 from the seventh level of hell
- the__alchemist 4y agoYikes! That sounds frustrating. As the article concluded with, the way forward appears at the end; ST adding an errata for this. The elephant in the room: How is the author getting STM32 G4s? I've been F5ing this every day with no luck for months: https://octopart.com/search?q=stm32g4+ceu¤cy=USD&specs=0 https://octopart.com/search?q=stm32g4+ceu¤cy=USD&specs...
- jpieper 4y agoOrder far in advance....? These particular chips arrived more than a year ago. But yes, the shortage is hitting hard here too. The last tray I received was one year ago, after which all orders have been unfulfilled.
- realcertify 4y agoThere's plenty of STM32 of any flavor in China. Alibaba is a good source. There may be fakes though, but reputable resellers (pay attention to reviews and tenure) normally sell genuine chips.
- the__alchemist 4y agoThanks for the tip! Will check that out. Worth taking the risk.
- pabs3 4y agoAre fakes of lesser quality? How do you tell fake from genuine?
- kragen 4y agoI've gotten some Blue Pills with (properly labeled) CKS32s on them, which might be of lesser quality, but I've also seen people reporting getting Blue Pills with (properly labeled) GD32s, which are higher quality than the ST part, at least in the sense that they run successfully at higher clock speeds. (I couldn't tell you if they have more problems turning on many peripherals at close-together times, or if they have more noise on their ADCs, or something.)
- civilitty 4y agoYou have to research the counterfeit market for each chip/product series. There's no hard and fast rule for their quality because many of them come from the same factories that make the real chips when unscrupulous fabs have shadow shifts manufacturing their clients' designs for themselves or just straight up copy and rebrand it.
- jhallenworld 4y agoIf you can use Cortex-M0+ (so not likely for this project, BLDC where you want floating point), maybe think about RP2040 (Raspberry Pi Pico chip)? More than 100K of them in stock at Digikey, only $1 each... But they come in only one package type, and use external flash (an advantage, IMHO). I've recently started looking at them. The SDK is nice except that they chose to use CMake. I designed it out: https://github.com/nklabs/libnklabs-pico https://github.com/nklabs/libnklabs-pico
- dmitrygr 4y ago> an advantage, IMHO it isn't An extra chip means higher cost to produce & assemble board, larger board size, more pins wasted on this nonsense, most fast-edge signals to route, more passives, extra risk to handle for one extra chip being out of stock, and it is much easier to extract firmware than even from a "protected" stm32 Also wasting RAM (and power for it) on code, or random (between high and very high) latency of XIP from SPI flash
- jhallenworld 4y agoBut balanced against basically unlimited flash size. Many of my projects end up having an external SPI-flash for one reason or another anyway. Another similar chip is the i.MX rt1020 from NXP (except Cortex-M7 and way more expensive). The one gotcha was that there was only one QSPI-flash controller even though there were two ports. It meant that the extra port (which we assumed would be available during architecture) was not fully usable without interfering with the firmware.
- dmitrygr 4y ago> Many of my projects end up having an external SPI-flash for one reason or another anyway. for what? What sort of possible thing would you need to do on a c-m0 that needs more than 128K of CODE!?
- jhallenworld 4y ago
- seidleroni 4y agoGetting samples of G4's was pretty easy. Since its a fairly new part, there was a large stock that distributors had for engineering samples. The story is very different for things like the F5/F7's and even getting small sample volumes is very hard.
- dognotdog 4y agoI just bought 10 off digikey last week, to have some prototyping stock. Looks like they still got some: https://www.digikey.com/en/products/detail/stmicroelectronics/STM32G491CEU6/13592603 https://www.digikey.com/en/products/detail/stmicroelectronic... Octopart, etc haven't been able to accurately track inventory through this disruption.
- girvo 4y agoWe can buy them in 10s at work without too much gnashing of teeth, but the thousands we need is impossible for another 18 months as far as we can tell. Edited to add: not that it matters now, we moved to a different MCU that we could get larger shipments of, and we’re lucky enough our application of them can be flexible enough to do so!
- the__alchemist 4y agoGood to know! I've been trying to get hold of certain H7 and G4 variants in a specific footprint. Family and footprint are a hard requirement, but not too picky otherwise. Checking it out. Of note regarding the one you linked - great find; looks like that Octopart link was bad! I actually have some 491s, and they're great, except that the OSS firmware I'm using them with for this project doesn't support them. Got my own firmware working on them though.
- jpieper 4y agoI'm curious what other OSS firmware is using the G4 series?
- the__alchemist 4y agoBetaflight I also have custom flight controller firmware in Rust that's running fine on G491, but Betaflight needs 47x or 48x. No luck on getting H7s.
- baybal2 4y agoSTM32s are actually very nice things to debug by the standards of the embedded scene. At least 1 set of stm32 cube tools will be always working. It's not uncommon for microcontroller vendors to not to provide any tooling at all, and just say "Use JTAG" without any further explanations. And then JTAG may, or may not work. Or only work on some decade old 3rd party IDE + 3rd party USB (or even RS232...) JTAG debugger + pre sp2 win xp version combo.
- matrix_overload 4y ago>It seems that the ADCs on the STM32G4 do not like to be turned on in rapid succession, and if they do, bad things can happen like having the prescaler flipped to a different value without it showing in the corresponding register. This sounds very, VERY much like an incorrectly configured clock where some of the peripherals would end up with a clock frequency slightly above what they were designed for. Will work 99% of the time and will give you hell for the remaining 1%. Much more likely than stumbling upon an undiscovered errata in a fairly popular device family with 10+ years of history. Could also be flakey power (check your decoupling capacitors) or an outright b0rked chip/board.
- deleted 4y ago[deleted]
- jhallenworld 4y agoMaybe, anyway it's worth opening a case with ST support. I found a bug in STM32L0 a few years ago, and they did help (they found a software workaround which was not yet documented- had to do with waking up from deep sleep).
- khoozer 4y agoOr it was just faulty firmware and he did hit the (almost) correct conclusion: ADEN bit on g4: "Note: The software is allowed to set ADEN only when all bits of ADC_CR registers are 0 (ADCAL = 0, JADSTART = 0, ADSTART = 0, ADSTP = 0, ADDIS = 0 and ADEN = 0) except for bit ADVREGEN which must be 1 (and the software must have wait for the startup time of the voltage regulator" No errata needed, it's clearly stated that you cannot just set ADEN without waiting for certain other conditions.
- jpieper 4y agoActually, if you look at the firmware at the time, the proper procedure was followed as far as I can tell. All of the necessary bits were set and checked with the appropriate delays where required. https://github.com/mjbots/moteus/blob/dcb900c92ffd5d5c8f54053d5a21fe11c4c6f4f3/fw/bldc_servo.cc#L611 https://github.com/mjbots/moteus/blob/dcb900c92ffd5d5c8f5405... Notably, the datasheet is largely silent on interactions between different ADCs during initialization.
- bunnie 4y agoHats off to you, that was a tough one to find! Learned a couple of tricks too, so thanks for sharing.
- vardump 4y agoThis is really just how low-level embedded land is like. Hardware is often buggy.
- HeyLaughingBoy 4y agoThankfully it's usually not this bad. I remember spending a lot of time debugging a problem because although the signs pointed to bad hardware, it was in such a commonly-used feature that I knew it had to be my code. Otherwise, everyone else who used the part would be screaming at the manufacturer. After I finally admitted defeat and called my TI (Texas Instruments) support engineer, he said, "oh, that's a brand new part. You guys are actually only the second company we've sold it to. Thanks for the bug report, but we just discovered it last week so it'll be fixed in the next rev."
- fxtentacle 4y agoMost likely, the author is using counterfeit hardware. I haven't seen authentic ones in stock anytime this year. And bugs in rare edge cases is precisely what I would expect from a price-conscious copy.
- jpieper 4y agoSurprisingly, it is almost certainly genuine. These particular chips likely came from a batch delivered in April of 2021 from Mouser, who isn't known for their shoddy sourcing practices.
- fxtentacle 4y agoAgree, I've never had bad experiences with Mouser.
- duskwuff 4y agoIt's also a relatively new part which hasn't developed the kind of demand that leads to counterfeiting yet. Most of the counterfeiters are targeting well-established old parts like the STM32F103.
- varispeed 4y agoHow did they manage to get STM32s? My friend's small business had to close, because key MCU is not available anywhere and Chinese sell it for the price of the end product. Makes business no longer viable. He spent over a month designing board for another MCU that was available at the time, but as soon as he was ready to produce more the alternative was gone too. It's like chasing own tail. Big corporations can post huge back-orders, but small business can't tie up that kind of sums of money for unknown period of time. He placed orders for that MCU over a year ago and distributors still tell the production is delayed. Meanwhile Chinese don't seem to be short of supply and are milking the desperate market. Seems like a state intervention would be welcome here.
- kragen 4y agoProbably the beginning of that month of designing would have been a better time to order the microcontrollers than the end of it. The answer to "how did jpieper get STM32s" is "he ordered them from Mouser in early 02021": https://news.ycombinator.com/item?id=32383312 https://news.ycombinator.com/item?id=32383312 What kind of state intervention are you thinking of? I spent some time trying to think of a state policy that would reduce the risk that small businesses would lose access to parts crucial to their products, rather than increasing it, but I couldn't think of one.
- varispeed 4y agoThe simplest intervention would be to offer state backed loans that could be used for back orders and to be repaid 3-6 months after the delivery. It's not ideal, probably it will create further delays, but it would put small business on the level playing field with the big corporations when it comes to orders. The criteria for eligibility may be difficult to define too, so that fraud is minimised. Maybe such surge of demand could incentivise MCU manufacturers to secure more fab capacity. State could also make scalping illegal. For instance buying MCUs with a sole purpose to hold on them and then resell at inflated price would be illegal. This could be done by setting maximum margin someone could ask, that should be no higher than Mouser or Digikey have. So anyone selling MCUs at 10x the price would have them confiscated and get fined. Foreign online stores that facilitate such sellers like Aliexpress would risk being sanctioned and banned.
- retSava 4y agoThe initial state of doubting the test fixtures (healthy skepticism!) highlights that it's very useful to keep a few known-good hw specimens over time, so that when you get a new batch that fails, you can test your test equipment against the older, known good hardware. Also swap firmware versions back and forth on old/new hw and see how behavior/metrics change.
- elisharobinson 4y agothis has "i just changed the comments and it works now but i dont know why " vibe to it.
- nottorp 4y ago> The initialization sequence for the ADC is documented as requiring a wait until the ADRDY flag is set, so the fix is just to wait for that for each ADC in turn before enabling the next one. Hmm? Ok, it may be the clock or something else subtle but if they tell you to wait...
- jpieper 4y agoIn this case, the firmware did wait for the ADRDY flag. It just waited for all 5 to be set, then moved on to enable all 5 ADCs simultaneously. The easy fix was to just do those serially instead.
- bradfa 4y agoTangent: reading this, all I could think was "you're allocating memory in a motor controller?!?" and then "you're using interrupts in a motor controller!?!". Surely the bug write up was good and it is interesting, but all of the actual hard real time systems I've ever seen or talked to engineers who actually worked on them always avoided interrupts and never allocate memory. Both of these activities in hard real time code can produce very unexpected results which may be similar to what the author found with enabling the ADCs in rapid succession.
- markrages 4y agoThe "standard" motor control setup for FOC is to do the ADC measurement synchronously with the PWM, to sample the current while the low-side switch is closed. Then the flux angles and voltages are computed immediately afterwards, and the PWM duty updated for the next cycle. This all happens in the interrupt, in a very hard real-time system -- start missing interrupts and all hell breaks loose. PWM is run as fast as you can get away with, 20 kHz minimum (human hearing). The controller spends maybe 80% of itw time in interrupt context, leaving main context for tasks that are not time-sensitive.
- frmdstryr 4y ago> always avoided interrupts I'm curious... how does one achieve precise timing without interrupts?
- jpieper 4y ago
- marssaxman 4y agoThis is a great writeup. Thank you for sharing. I sure do love and miss doing this kind of engineering work.
- lebuffon 4y agoForth is very handy for this kind of new iron startup work. You can interrogate the hardware interactively for those really thorny problems and confirm your hypotheses immediately. You can write short test scripts and send them via the terminal and/or write and send short Assembler routines and test those interactively. Each routine you define is interactively usable to assist with higher level testing and/or can be further compiled together to make higher level or looping tests. It's old and it's weird but it's still a good tool for this kind of work. It may mean you only have to enter the top domain of Hell. :-) https://mecrisp-stellaris-folkdoc.sourceforge.io/flashing-mecrisp_stellaris.html https://mecrisp-stellaris-folkdoc.sourceforge.io/flashing-me... *I have no connection to mecrisp Forth but know it to have a good reputation. It is a a native code compiler with an integrated text interpreter.
- jpieper 4y agoYeah, I have (maybe slightly fond?) memories of using Forth to develop a tape drive reader and writer for an undergraduate lab project. It is wonderful for some things, although in this case, where the problem was literally which addresses the instructions got assigned to, it is unclear if it would have made anything better.