3 ms·

> The failure mode of EBS on Ubuntu is extremely severe: because EBS volumes are network drives masquerading as block devices http://joyent.com/blog/magical-blo

by leoc 14y ago

> The failure mode of EBS on Ubuntu is extremely severe: because EBS volumes are network drives masquerading as block devices http://joyent.com/blog/magical-block-store-when-abstractions-fail-us http://joyent.com/blog/magical-block-store-when-abstractions... , they break abstractions in the Linux operating system. This has led to really terrible failure scenarios for us, where a failing EBS volume causes an entire box to lock up, leaving it inaccessible and affecting even operations that don’t have any direct requirement of disk activity. Assuming that I/O is reliable and fast-ish was a bad OS design decision in the 1970s, as the Unix guys themelves soon realised, though it was an understandable mistake at the time. Continuing to develop and use OSes with an I/O interface that assumes I/O is reliable and fast-ish, in 2012 - it's pretty bad, isn't it?

zurn 14y ago
Hardware and software have co-evolved, so that disks provide an illusion of error free operation until they throw in the towel and die. And they have consistent performance. This has worked OK so far. With network filesystems (eg. NFS) you can choose to return an I/O error to the application when you hit a timeout or a network error (the -o intr mount option). This is rarely used since applications aren't used to dealing with them. So can't really blame the OS here either.
- spartango 14y ago
  On one hand you're right: when block devices fail, they're pretty much gone--if you've ever tried to read from a bad sector you'll know this exactly, or worse still on SSDs, where the disk will fail to show up to the bus. That said, I'm not sure I agree with the idea that we got away with a lack of error handling because disks had consistent performance. Magnetic disks have always had incredibly inconsistent random IO performance, and even inconsistent performance between different parts of the platter(s). And in the spirit of co-evolution, we engineered around it: OS disk caches are critical to decent performance on HDDs. I think it's not that we found disks to be consistent, as much as our solution to their suboptimal behaviors was caching, rather than error reporting/timeouts. I believe this is because caching was the most transparent approach; a good cache makes a variable-speed disk look just like an ideal disk, so an application can be written assuming the disk is perfect. Interestingly, as distributed systems have evolved, we've ended up having to engineer the error-handling constructs that might have been used for block devices; we see them in network filesystems (as mentioned) as well as most other network services. Applications have been designed to deal with errors. We just haven't propagated those constructs down to the disk devices in the recent past.
  - zurn 14y ago
    > Magnetic disks have always had incredibly inconsistent random IO performance From a non-realtime app POV disk seek performance is pretty consistent: you get 5-25ms seek cies that center around 10ms. Especially In contrast to network backed where you get to contend with hiccups and contention with other users. OS disk caching came about for a different reason.

by leoc 14y ago

zurn 14y ago

Hardware and software have co-evolved, so that disks provide an illusion of error free operation until they throw in the towel and die. And they have consistent performance. This has worked OK so far. With network filesystems (eg. NFS) you can choose to return an I/O error to the application when you hit a timeout or a network error (the -o intr mount option). This is rarely used since applications aren't used to dealing with them. So can't really blame the OS here either.

spartango 14y ago
On one hand you're right: when block devices fail, they're pretty much gone--if you've ever tried to read from a bad sector you'll know this exactly, or worse still on SSDs, where the disk will fail to show up to the bus. That said, I'm not sure I agree with the idea that we got away with a lack of error handling because disks had consistent performance. Magnetic disks have always had incredibly inconsistent random IO performance, and even inconsistent performance between different parts of the platter(s). And in the spirit of co-evolution, we engineered around it: OS disk caches are critical to decent performance on HDDs. I think it's not that we found disks to be consistent, as much as our solution to their suboptimal behaviors was caching, rather than error reporting/timeouts. I believe this is because caching was the most transparent approach; a good cache makes a variable-speed disk look just like an ideal disk, so an application can be written assuming the disk is perfect. Interestingly, as distributed systems have evolved, we've ended up having to engineer the error-handling constructs that might have been used for block devices; we see them in network filesystems (as mentioned) as well as most other network services. Applications have been designed to deal with errors. We just haven't propagated those constructs down to the disk devices in the recent past.
- zurn 14y ago
  > Magnetic disks have always had incredibly inconsistent random IO performance From a non-realtime app POV disk seek performance is pretty consistent: you get 5-25ms seek cies that center around 10ms. Especially In contrast to network backed where you get to contend with hiccups and contention with other users. OS disk caching came about for a different reason.