3 ms·

> You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. > > ban SSD from

by wtbob 10y ago

> You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. > > ban SSD from critical systems. > That's an over-reaction if I've ever seen one. No, it's really not. SFJulie's point is that we don't know enough about the way SSDs fail to build a good failure model for them, and that we currently don't seem to have a good way to even detect that they are failing. If you can't build a failure model for something, you can't build a good risk model including it. That's kinda her point. FWIW, I don't know that I fully agree with that point: it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? > If it's something that has to guarantee high performance under massive load, SSDs are what you need, period. We seemed to do alright four years ago, no? SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them.

hga 10y ago
it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? To me the three scariest words in engineering are "common mode failure". I personally never buy more than one hard disk at a time, so that I'll hopefully avoid something like both haven been shaken too much in their shipping container (the best explanation anyone was able to come up with for a Seagate failure pattern some time ago). So, yes, the above, to detect and recover, but plenty of services need very up to date backups, so if I was designing such a system, I'd make sure the (near-)real time backup would be to SSDs that share nothing in common with my production ones, different manufacturer, chip sets, etc. Or if the load could handle it ... ah, I see why Seagate is still selling real SAS enterprise disks (last time I checked, if I remember correctly, <= 1TB and up to 15K RPM). We understand those, not that they won't also fail on you.
bluecmd 10y ago
I don't get where you got the notion that you need to understand the failure model to rely on it. It's certainly good to, and it will prevent over provisioning, but a lot of storage solutions today are so complex that understanding the failure models is practically impossible. S3? Super-black box. Basically all you need to define is the expected output. "99.9pct of my reads are within 1ms" + "The read response is correct 99.9999% of the time" and then verify. Eject the disk if that doesn't hold, restore from replica. The rate of replacements is a factor of how close you are to the true failure model.
kabdib 10y ago
> SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them. Lots more money. Many, many more hard drives, with failure rates to match. And you're still not going to get there. We started running a 240TB all SSD array a few months ago and we get sub-millisecond latencies on most reads and writes. When we see an excursion to 10ms that's cause for concern; 50ms and we do a root-cause. We can peg multiple 16GBit fiber channels and it's okay. And it fits in half a rack; try that with disk. I cannot imagine doing this with spinning rust.

by wtbob 10y ago

hga 10y ago

it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? To me the three scariest words in engineering are "common mode failure". I personally never buy more than one hard disk at a time, so that I'll hopefully avoid something like both haven been shaken too much in their shipping container (the best explanation anyone was able to come up with for a Seagate failure pattern some time ago). So, yes, the above, to detect and recover, but plenty of services need very up to date backups, so if I was designing such a system, I'd make sure the (near-)real time backup would be to SSDs that share nothing in common with my production ones, different manufacturer, chip sets, etc. Or if the load could handle it ... ah, I see why Seagate is still selling real SAS enterprise disks (last time I checked, if I remember correctly, <= 1TB and up to 15K RPM). We understand those, not that they won't also fail on you.

bluecmd 10y ago

I don't get where you got the notion that you need to understand the failure model to rely on it. It's certainly good to, and it will prevent over provisioning, but a lot of storage solutions today are so complex that understanding the failure models is practically impossible. S3? Super-black box. Basically all you need to define is the expected output. "99.9pct of my reads are within 1ms" + "The read response is correct 99.9999% of the time" and then verify. Eject the disk if that doesn't hold, restore from replica. The rate of replacements is a factor of how close you are to the true failure model.

kabdib 10y ago

> SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them. Lots more money. Many, many more hard drives, with failure rates to match. And you're still not going to get there. We started running a 240TB all SSD array a few months ago and we get sub-millisecond latencies on most reads and writes. When we see an excursion to 10ms that's cause for concern; 50ms and we do a root-cause. We can peg multiple 16GBit fiber channels and it's okay. And it fits in half a rack; try that with disk. I cannot imagine doing this with spinning rust.