7 ms·

Disks lie. And the controllers that run them are partners in crime (2012)

83 points by ecliptik 3y ago

jandrewrogers 3y ago
Disks are foreign nodes in a distributed system. Treat them as such.
- pyrolistical 3y ago
  Then what does this mean for consensus algorithm like raft? Wouldn’t you need to power cycle disks and read back what you just wrote to ensure it actually got written and not just buffered?
  - vlovich123 3y ago
    You’ll always get back what you wrote even if it wasn’t written durably. It’s kind of how that works. With consensus you make the following kind of argument: 1. The kind of drives you’re using if you’re running a distributed consensus in production is going to be enterprise where these kind of games are played a lot less 2. Establishing consensus is time consuming - by the time peers ack it, you will have flushed to disk. In a common 3 replica scenario, that means 1/3 nodes is taken care of. 3. The probability of 2 disks failing simultaneously in ways that cause a problem is pretty small. But yeah, if your concensus has to be durably written and your flash controller is lying to you, your concensus is worthless (even reading what you wrote isn’t good enough because again - controller can lie to you and satisfy reads from the buffer too)
  - mmis1000 3y ago
    I think most modern disks just never design with sudden power-lost safety in mind (some enterprise disks do have such guarantee, by adding a battery or a capacitor to buy time for properly flush the cache to disk). A disk AND an UPS is probably the minimal combination that you can guarantee that your data is actually persisted.
johnbender 3y ago
If you view a file system as running concurrently with another instance of itself where it could be preempted at any time indefinitely and where your algorithm for ensuring your crash protocol invariant must be lock free (two threads helps here) then you get a good sense for the complexity of the problem.
- dan-robertson 3y ago
  … and filesystems themselves are multithreaded, and your writes can get reordered quite dramatically.
Dalewyn 3y ago
What we singularly call a computer is actually a collection of computers. Each HDD and/or SSD is its own computer, with its own CPU (controller) and RAM (cache). Each video card is its own computer with its own CPU (GPU) and RAM (video RAM). Every USB controller is its own computer with its own CPU and RAM. Each NIC is its own computer with its own CPU and RAM. And so on and so forth. Do disks lie? Fuck yeah, and so does everything else. Much like the human world, everyone (everything) lies so the system as a whole can (hopefully) keep running smoothly.
- thfuran 3y ago
  >everyone (everything) lies so the system as a whole can (hopefully) keep running smoothly. That's not how you make a smoothly running system, it's how you make one that insists everything is fine while it's all going to shit.
  - jurgenaut23 3y ago
    Great analogy of the wider world, by the way.
  - rollcat 3y ago
    Not necessarily. Sometimes you are forced to lie, because another component couldn't bear the truth. Disks were an example of that basically from day 1. I can't recall the exact details, but it was something about the disk geometry (heads/cylinders/sectors) that didn't fit in an 8bit integer, which was what BIOS expected. So the disks had to make something up, and translate requests on the go, otherwise a chunk of the total capacity would be inaccessible, which would be inexcusable in a race to be the biggest. You could call it "adversarial compatibility", and we owe a lot of tech progress to it.
    - Sakos 3y ago
      I love the term "adversarial compatibility". If I had to describe technological progress in computing for the past 40 years, that's what I'd use.
    - 3y ago
Galanwe 3y ago
I guess I've been living under a rock, because I had the souvenir that disks with volatile write caches did have a small power bank, able to sustain the disk for the few seconds it needed to flush its caches (what it called a BBU?) Which leads me to another question, should it really be the responsibility of each component to gracefully handle a random, instantaneous power loss? Couldn't we have a small power bank on the motherboard keeping, say, 2 seconds of power and signaling emergency shutdown to components? In a sense, that is very similar to sending a SIGTERM to a process to allow it to gracefully stop.
- gavinsyancey 3y ago
  My understanding is that disks use the inertia of their spinning platters to power themselves long enough to flush their cache in the event of sudden power loss.
  - Galanwe 3y ago
    Yeah, I just had a look and it seems BBUs don't actually power the disk. They just keep the volatile cache alive until the disk is powered to flush them.
  - tinus_hn 3y ago
    They use this to finish writing the sector they were writing so they can guarantee they always completely write the sector or not at all. As far as I know this mechanism is not used for write caches.
- josephg 3y ago
  Consumer drives are generally junk. Personally I think it should be the responsibility of the operating system. All filesystem operations on modern computers should be atomic, and either entirely succeed or entirely fail. And the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them. Hilariously, the OS already uses transactions for its own filesystem metadata operations. But they don't share that capability with end user applications, which end up either being quietly broken in the event of power loss (most apps) or horribly complex to make up for the OS's shortfalls (most databases). There are dozens of solutions in userland, from git's content-addressing to on-disk B+trees. There should just be a simple, effective, correct transactional filesystem API provided by the OS that applications can simply use. The best part of this proposal: a solution implemented by the OS directly would probably be faster than what most people would implement in userland anyway. Its ridiculous this doesn't already exist. There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state.
nurple 3y ago
What I get from this is that a major method of rent seeking used by disk manufacturers, to differentiate expensive enterprise disk from cheap consumer disk, was colluding to keep the simple and effective write queue reporting from being shipped to the latter. It didn't require nvram, or complex microcode that knew how to process leftover wire requests, it only had to report back when each chunk it was already tracking in its buffers got persisted. To me it sounds like it's the manufacturers were the true partners in crime here.
- dan-robertson 3y ago
  The article is from 10 years ago. Most people use flash now which is different (see eg https://dl.acm.org/doi/pdf/10.1145/3064176.3064187 https://dl.acm.org/doi/pdf/10.1145/3064176.3064187). I think higher prices for enterprise-grade disks are more about reliability than NCQ behaviour.
  - jacquesm 3y ago
    Try doing long, sustained out-of-order writes and see how long you can keep that up before you're down to very low transfer rates.
    - walth 3y ago
      Can you point me to any further specifics on this?
      - jacquesm 3y ago
        Essentially it will work fantastically until you hit the capacity of the cache memory and then the write rate will drop, sometimes only a bit, more often quite a lot. There are drives that can maintain their write rate 'forever', but those are substantially more expensive, for instance the Samsung 970 pro, see this graph from a Tom's Hardware article, it's a good graph because it very clearly shows how much cache the drives have and then it's game over: https://cdn.mos.cms.futurecdn.net/V5reyvV4UAwHxEMQBP2XWS-970-80.png.webp https://cdn.mos.cms.futurecdn.net/V5reyvV4UAwHxEMQBP2XWS-970... A review of that drive: https://www.tomshardware.com/reviews/samsung-970-pro-ssd-review,5572-2.html https://www.tomshardware.com/reviews/samsung-970-pro-ssd-rev...
ChrisArchitect 3y ago
Some previous discussion from back then: https://news.ycombinator.com/item?id=4491554 https://news.ycombinator.com/item?id=4491554

83 points by ecliptik 3y ago

jandrewrogers 3y ago

Disks are foreign nodes in a distributed system. Treat them as such.

pyrolistical 3y ago
Then what does this mean for consensus algorithm like raft? Wouldn’t you need to power cycle disks and read back what you just wrote to ensure it actually got written and not just buffered?
- vlovich123 3y ago
  You’ll always get back what you wrote even if it wasn’t written durably. It’s kind of how that works. With consensus you make the following kind of argument: 1. The kind of drives you’re using if you’re running a distributed consensus in production is going to be enterprise where these kind of games are played a lot less 2. Establishing consensus is time consuming - by the time peers ack it, you will have flushed to disk. In a common 3 replica scenario, that means 1/3 nodes is taken care of. 3. The probability of 2 disks failing simultaneously in ways that cause a problem is pretty small. But yeah, if your concensus has to be durably written and your flash controller is lying to you, your concensus is worthless (even reading what you wrote isn’t good enough because again - controller can lie to you and satisfy reads from the buffer too)
- mmis1000 3y ago
  I think most modern disks just never design with sudden power-lost safety in mind (some enterprise disks do have such guarantee, by adding a battery or a capacitor to buy time for properly flush the cache to disk). A disk AND an UPS is probably the minimal combination that you can guarantee that your data is actually persisted.

johnbender 3y ago

If you view a file system as running concurrently with another instance of itself where it could be preempted at any time indefinitely and where your algorithm for ensuring your crash protocol invariant must be lock free (two threads helps here) then you get a good sense for the complexity of the problem.

dan-robertson 3y ago
… and filesystems themselves are multithreaded, and your writes can get reordered quite dramatically.

Dalewyn 3y ago

What we singularly call a computer is actually a collection of computers. Each HDD and/or SSD is its own computer, with its own CPU (controller) and RAM (cache). Each video card is its own computer with its own CPU (GPU) and RAM (video RAM). Every USB controller is its own computer with its own CPU and RAM. Each NIC is its own computer with its own CPU and RAM. And so on and so forth. Do disks lie? Fuck yeah, and so does everything else. Much like the human world, everyone (everything) lies so the system as a whole can (hopefully) keep running smoothly.

thfuran 3y ago
>everyone (everything) lies so the system as a whole can (hopefully) keep running smoothly. That's not how you make a smoothly running system, it's how you make one that insists everything is fine while it's all going to shit.
- jurgenaut23 3y ago
  Great analogy of the wider world, by the way.
- rollcat 3y ago
  Not necessarily. Sometimes you are forced to lie, because another component couldn't bear the truth. Disks were an example of that basically from day 1. I can't recall the exact details, but it was something about the disk geometry (heads/cylinders/sectors) that didn't fit in an 8bit integer, which was what BIOS expected. So the disks had to make something up, and translate requests on the go, otherwise a chunk of the total capacity would be inaccessible, which would be inexcusable in a race to be the biggest. You could call it "adversarial compatibility", and we owe a lot of tech progress to it.
  - Sakos 3y ago
    I love the term "adversarial compatibility". If I had to describe technological progress in computing for the past 40 years, that's what I'd use.
  - 3y ago

Galanwe 3y ago

I guess I've been living under a rock, because I had the souvenir that disks with volatile write caches did have a small power bank, able to sustain the disk for the few seconds it needed to flush its caches (what it called a BBU?) Which leads me to another question, should it really be the responsibility of each component to gracefully handle a random, instantaneous power loss? Couldn't we have a small power bank on the motherboard keeping, say, 2 seconds of power and signaling emergency shutdown to components? In a sense, that is very similar to sending a SIGTERM to a process to allow it to gracefully stop.

gavinsyancey 3y ago
My understanding is that disks use the inertia of their spinning platters to power themselves long enough to flush their cache in the event of sudden power loss.
- Galanwe 3y ago
  Yeah, I just had a look and it seems BBUs don't actually power the disk. They just keep the volatile cache alive until the disk is powered to flush them.
- tinus_hn 3y ago
  They use this to finish writing the sector they were writing so they can guarantee they always completely write the sector or not at all. As far as I know this mechanism is not used for write caches.
josephg 3y ago
Consumer drives are generally junk. Personally I think it should be the responsibility of the operating system. All filesystem operations on modern computers should be atomic, and either entirely succeed or entirely fail. And the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them. Hilariously, the OS already uses transactions for its own filesystem metadata operations. But they don't share that capability with end user applications, which end up either being quietly broken in the event of power loss (most apps) or horribly complex to make up for the OS's shortfalls (most databases). There are dozens of solutions in userland, from git's content-addressing to on-disk B+trees. There should just be a simple, effective, correct transactional filesystem API provided by the OS that applications can simply use. The best part of this proposal: a solution implemented by the OS directly would probably be faster than what most people would implement in userland anyway. Its ridiculous this doesn't already exist. There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state.

nurple 3y ago

What I get from this is that a major method of rent seeking used by disk manufacturers, to differentiate expensive enterprise disk from cheap consumer disk, was colluding to keep the simple and effective write queue reporting from being shipped to the latter. It didn't require nvram, or complex microcode that knew how to process leftover wire requests, it only had to report back when each chunk it was already tracking in its buffers got persisted. To me it sounds like it's the manufacturers were the true partners in crime here.

dan-robertson 3y ago
The article is from 10 years ago. Most people use flash now which is different (see eg https://dl.acm.org/doi/pdf/10.1145/3064176.3064187 https://dl.acm.org/doi/pdf/10.1145/3064176.3064187). I think higher prices for enterprise-grade disks are more about reliability than NCQ behaviour.
- jacquesm 3y ago
  Try doing long, sustained out-of-order writes and see how long you can keep that up before you're down to very low transfer rates.
  - walth 3y ago
    Can you point me to any further specifics on this?
    - jacquesm 3y ago
      Essentially it will work fantastically until you hit the capacity of the cache memory and then the write rate will drop, sometimes only a bit, more often quite a lot. There are drives that can maintain their write rate 'forever', but those are substantially more expensive, for instance the Samsung 970 pro, see this graph from a Tom's Hardware article, it's a good graph because it very clearly shows how much cache the drives have and then it's game over: https://cdn.mos.cms.futurecdn.net/V5reyvV4UAwHxEMQBP2XWS-970-80.png.webp https://cdn.mos.cms.futurecdn.net/V5reyvV4UAwHxEMQBP2XWS-970... A review of that drive: https://www.tomshardware.com/reviews/samsung-970-pro-ssd-review,5572-2.html https://www.tomshardware.com/reviews/samsung-970-pro-ssd-rev...

ChrisArchitect 3y ago

Some previous discussion from back then: https://news.ycombinator.com/item?id=4491554 https://news.ycombinator.com/item?id=4491554