4 ms·

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed a

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board. The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week. It took half a year of debugging and tens of millions of dollars in losses. We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.

Fire-Dragon-DoL 3y ago
Oh god, that's insane
mghardal 3y ago
Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.
- calamari4065 3y ago
  >The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.
  - MassPikeMike 3y ago
    Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.
  - eep_social 3y ago
    And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.
  - onetimeuse92304 3y ago
    We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them. Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).
  - deleted 3y ago
    [deleted]
- rcxdude 3y ago
  from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.
  - onetimeuse92304 3y ago
    Yes, it was separate chip (per description). Yes, it was SPI flash. And yes, the word was 16 bits.
- onetimeuse92304 3y ago
  1. What does choice of MCU have to do with it? 2. Noise can definitely affect other components. Traces are antennas that both broadcast as well as receive signals. As technology marches forward, the voltages and tolerances for communication become smaller and tighter. At the same time, you integrate more functionality on the board so maybe your sensitive MCU and flash and ram are all living close by to some radio equipment and PSU and whatever else. Power interruptions to certain components can create momentary flashes of noise that can act as EMP bomb going off. On other occasions a noisy, badly designed high frequency power component can emit noise that is picked up by some random traces due to their geometric configuration and length. Your comment suggests you never worked on a large project with thousands of components and dozens of major chips?
vardump 3y ago
Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.
- phkahler 3y ago
  >> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring. I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.
  - hypothesis 3y ago
    I wonder if anyone else instantly gets a visual after reading your story: “Business woman on plane: Are there a lot of these kinds of accidents? Narrator: You wouldn't believe. Business woman on plane: Which car company do you work for? Narrator: A major one.”
    - phkahler 3y ago
      I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.
      - actionfromafar 3y ago
        And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different. How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?
deleted 3y ago
[deleted]
onetimeuse92304 3y ago
I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).
kragen 3y ago
epic debugging story
dartos 3y ago
I think something similar happened to my sd card at a hackathon years ago. We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged. https://devpost.com/software/xcaliber https://devpost.com/software/xcaliber

Fire-Dragon-DoL 3y ago

Oh god, that's insane

mghardal 3y ago

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

calamari4065 3y ago
>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.
- MassPikeMike 3y ago
  Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.
- eep_social 3y ago
  And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.
- onetimeuse92304 3y ago
  We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them. Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).
- deleted 3y ago
  [deleted]
rcxdude 3y ago
from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.
- onetimeuse92304 3y ago
  Yes, it was separate chip (per description). Yes, it was SPI flash. And yes, the word was 16 bits.
onetimeuse92304 3y ago
1. What does choice of MCU have to do with it? 2. Noise can definitely affect other components. Traces are antennas that both broadcast as well as receive signals. As technology marches forward, the voltages and tolerances for communication become smaller and tighter. At the same time, you integrate more functionality on the board so maybe your sensitive MCU and flash and ram are all living close by to some radio equipment and PSU and whatever else. Power interruptions to certain components can create momentary flashes of noise that can act as EMP bomb going off. On other occasions a noisy, badly designed high frequency power component can emit noise that is picked up by some random traces due to their geometric configuration and length. Your comment suggests you never worked on a large project with thousands of components and dozens of major chips?

vardump 3y ago

Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

phkahler 3y ago
>> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring. I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.
- hypothesis 3y ago
  I wonder if anyone else instantly gets a visual after reading your story: “Business woman on plane: Are there a lot of these kinds of accidents? Narrator: You wouldn't believe. Business woman on plane: Which car company do you work for? Narrator: A major one.”
  - phkahler 3y ago
    I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.
    - actionfromafar 3y ago
      And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different. How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?

deleted 3y ago

[deleted]

onetimeuse92304 3y ago

I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).

kragen 3y ago

epic debugging story

dartos 3y ago

I think something similar happened to my sd card at a hackathon years ago. We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged. https://devpost.com/software/xcaliber https://devpost.com/software/xcaliber