5 ms·

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesy

by nabla9 3y ago

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once a month or so. Oracle had to send on-site team to monitor the database operations to catch the bug in action. They basically had their own datacenter to record all the logs from database, networks and operating system. It took weeks for them to catch the bug. It was related to certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.

onetimeuse92304 3y ago
The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board. The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week. It took half a year of debugging and tens of millions of dollars in losses. We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.
- Fire-Dragon-DoL 3y ago
  Oh god, that's insane
- mghardal 3y ago
  Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.
  - calamari4065 3y ago
    >The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.
    - MassPikeMike 3y ago
      Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.
    - eep_social 3y ago
      And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.
    - onetimeuse92304 3y ago
      We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them. Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).
    - deleted 3y ago
      [deleted]
  - rcxdude 3y ago
    from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.
- vardump 3y ago
  Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.
  - phkahler 3y ago
    >> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring. I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.
    - hypothesis 3y ago
      I wonder if anyone else instantly gets a visual after reading your story: “Business woman on plane: Are there a lot of these kinds of accidents? Narrator: You wouldn't believe. Business woman on plane: Which car company do you work for? Narrator: A major one.”
      - phkahler 3y ago
        I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.
        actionfromafar 3y ago
        And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different. How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?
- deleted 3y ago
  [deleted]
- onetimeuse92304 3y ago
  I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).
- kragen 3y ago
  epic debugging story
- dartos 3y ago
  I think something similar happened to my sd card at a hackathon years ago. We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged. https://devpost.com/software/xcaliber https://devpost.com/software/xcaliber
vardump 3y ago
Network drivers contribute to "bugs" surprisingly often. I've had my share of issues with them. I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.
- YetAnotherNick 3y ago
  How can network driver corrupt a database?
  - throwaway2037 3y ago
    By corrupting the data that was received.
  - toast0 3y ago
    Most network drivers run in kernel context and can access and write to everything. Additionally, unless your system has an iommu and configures it to strictly limit your NIC, most NICs can DMA to any address in memory. Unintentionally providing the wrong address to receive a packet with is a great way to corrupt a database.
rincebrain 3y ago
First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris. _Some_ of the Linux systems had a problem where eventually, the NIC would stop responding and dmesg would flood with "Tx Unit Hang" or "Rx Unit Hang", and not work again until a reboot. If we swapped the NIC from a system where this never happened to one with the same model NIC where this happened, the problem kept happening - that is, it was something about the system+NIC, not just the NIC. Intel eventually gave us a shipping label to borrow one of our systems when they said they hadn't been able to reproduce it but kept having people report it, and we said "we have a number of them". They made some really exciting noises after some debugging, and eventually concluded the issue was that our really cheapass Sempron board that we sent them screwed up badly if you tried negotiating 64-bit PCI in their normal PCI slots, and would sometimes miss the messages between the NIC and the host saying that Tx Ring buffer 0 is free/full, etc, and eventually, it would miss so many of those that it would think the NIC had no more buffers, and here we are. They couldn't reproduce it because all their testbeds were well-designed and tested things. Choice quotes mid-debugging include "I do not see how anyone could make a PCI device work reliably in this system". They did figure out a workaround, and ship it, and asked if they could keep the system permanently for future testing. But that was exciting. [1] - https://github.com/openzfs/zfs/pull/15588 https://github.com/openzfs/zfs/pull/15588
- rincebrain 3y ago
  A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them. Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired... This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing. Huh. So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it. So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up. The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that. I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.
akira2501 3y ago
> certain network driver in the operating system and the bug manifested only with some specific network traffic pattern. Found a similar issue in a switch stack relating to multicast stream delivery. Knowing about this class of bug and having isolated it down to the switch, as the only possible device in the path, being responsible, I filed a ticket with the appropriate team to get the stack rebooted. They were understandably skeptical of my supposed prescience and so I had to spend a considerably larger amount of time writing test cases and running deep packet captures in order to prove that a single bit, in the port field, was getting flipped under circumstances that were repeatable but not entirely explainable without even more work. Finally, they relented and agreed that, just turning it off, then on again, was indeed the best initial solution. Fortunately with all the test harnesses in place we were able to prove that it cleared the issue on that set of ports.
weebull 3y ago
I never really accept bug resolutions like that. Oracle corrupted data because a network driver fed it bad information? Did this network driver manage to preserve the legality of the corrupted info? Unlikely, so why did oracle believe it? If the corrupted message was still legal, then it sounds like the protocol isn't robust. I'm sure there's all kinds of nuance to your tale, but it still sounds like Oracle's bug to me. They just found what it was in your system that introduced the behaviour they hadn't anticipated, and then got you to remove it.
- kragen 3y ago
  network drivers can usually corrupt arbitrary kernel memory

by nabla9 3y ago

onetimeuse92304 3y ago

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board. The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week. It took half a year of debugging and tens of millions of dollars in losses. We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.

Fire-Dragon-DoL 3y ago
Oh god, that's insane
mghardal 3y ago
Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.
- calamari4065 3y ago
  >The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.
  - MassPikeMike 3y ago
    Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.
  - eep_social 3y ago
    And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.
  - onetimeuse92304 3y ago
    We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them. Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).
  - deleted 3y ago
    [deleted]
- rcxdude 3y ago
  from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.
vardump 3y ago
Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.
- phkahler 3y ago
  >> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring. I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.
  - hypothesis 3y ago
    I wonder if anyone else instantly gets a visual after reading your story: “Business woman on plane: Are there a lot of these kinds of accidents? Narrator: You wouldn't believe. Business woman on plane: Which car company do you work for? Narrator: A major one.”
    - phkahler 3y ago
      I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.
      - actionfromafar 3y ago
        And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different. How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?
deleted 3y ago
[deleted]
onetimeuse92304 3y ago
I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).
kragen 3y ago
epic debugging story
dartos 3y ago
I think something similar happened to my sd card at a hackathon years ago. We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged. https://devpost.com/software/xcaliber https://devpost.com/software/xcaliber

vardump 3y ago

Network drivers contribute to "bugs" surprisingly often. I've had my share of issues with them. I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.

YetAnotherNick 3y ago
How can network driver corrupt a database?
- throwaway2037 3y ago
  By corrupting the data that was received.
- toast0 3y ago
  Most network drivers run in kernel context and can access and write to everything. Additionally, unless your system has an iommu and configures it to strictly limit your NIC, most NICs can DMA to any address in memory. Unintentionally providing the wrong address to receive a packet with is a great way to corrupt a database.

rincebrain 3y ago

First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris. _Some_ of the Linux systems had a problem where eventually, the NIC would stop responding and dmesg would flood with "Tx Unit Hang" or "Rx Unit Hang", and not work again until a reboot. If we swapped the NIC from a system where this never happened to one with the same model NIC where this happened, the problem kept happening - that is, it was something about the system+NIC, not just the NIC. Intel eventually gave us a shipping label to borrow one of our systems when they said they hadn't been able to reproduce it but kept having people report it, and we said "we have a number of them". They made some really exciting noises after some debugging, and eventually concluded the issue was that our really cheapass Sempron board that we sent them screwed up badly if you tried negotiating 64-bit PCI in their normal PCI slots, and would sometimes miss the messages between the NIC and the host saying that Tx Ring buffer 0 is free/full, etc, and eventually, it would miss so many of those that it would think the NIC had no more buffers, and here we are. They couldn't reproduce it because all their testbeds were well-designed and tested things. Choice quotes mid-debugging include "I do not see how anyone could make a PCI device work reliably in this system". They did figure out a workaround, and ship it, and asked if they could keep the system permanently for future testing. But that was exciting. [1] - https://github.com/openzfs/zfs/pull/15588 https://github.com/openzfs/zfs/pull/15588

rincebrain 3y ago
A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them. Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired... This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing. Huh. So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it. So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up. The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that. I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.

akira2501 3y ago

> certain network driver in the operating system and the bug manifested only with some specific network traffic pattern. Found a similar issue in a switch stack relating to multicast stream delivery. Knowing about this class of bug and having isolated it down to the switch, as the only possible device in the path, being responsible, I filed a ticket with the appropriate team to get the stack rebooted. They were understandably skeptical of my supposed prescience and so I had to spend a considerably larger amount of time writing test cases and running deep packet captures in order to prove that a single bit, in the port field, was getting flipped under circumstances that were repeatable but not entirely explainable without even more work. Finally, they relented and agreed that, just turning it off, then on again, was indeed the best initial solution. Fortunately with all the test harnesses in place we were able to prove that it cleared the issue on that set of ports.

weebull 3y ago

I never really accept bug resolutions like that. Oracle corrupted data because a network driver fed it bad information? Did this network driver manage to preserve the legality of the corrupted info? Unlikely, so why did oracle believe it? If the corrupted message was still legal, then it sounds like the protocol isn't robust. I'm sure there's all kinds of nuance to your tale, but it still sounds like Oracle's bug to me. They just found what it was in your system that introduced the behaviour they hadn't anticipated, and then got you to remove it.

kragen 3y ago
network drivers can usually corrupt arbitrary kernel memory