4 ms·

Compared to HDD SMART yields quite a lot of false negative/positive hence it is proving harder than HDD means to detect failures. Failure rates seems to not fi

by SFJulie 10y ago

Compared to HDD SMART yields quite a lot of false negative/positive hence it is proving harder than HDD means to detect failures. Failure rates seems to not fit the models and be a tad higher than expected. Most SSD are reliable, the only problem is they are failing in a way that is tough to detect leading to consequences we may ignore. (falsely positive functioning SSD in operations and we may experience silent corruptions) Given the "too many knobs" of the SSD they can defect in non predictable snowballing effects (chaotic and dramatic). Basically this study confirms that IEEE standards for rating MTBF are to be reconsidered drastically and that our models of failure for SSD are far from totally being well understood and that SSD is in production while all costs are not totally yet known. The Cost of SSD specific failures is not yet known and they urge SSD makers to begin studying the effects of their "knobs" on reliability Even though it is written in a very scientific neutral tone trying not to scare people, it basically can be used as a strong evidence to ban SSD from critical systems. It basically is an heavy blow to SSD industry since it attacks it on the costs model that is based on its expected reliability and says that these drives are basically still an unknown territory when it comes to failures.

toyg 10y ago
You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. You offset the costs of worst-case scenario readiness with the productivity and financial gains of using the tech. If SSDs make your business 2x as productive or 2x as remunerative, the cost of an extra backup or two is simply a blip. > ban SSD from critical systems. That's an over-reaction if I've ever seen one. I guess it depends what a "critical system" is for you. If it's something that has to guarantee high performance under massive load, SSDs are what you need, period. If it's something that has to guarantee 100-year data storage under a mountain, then there might be better choices, but they might not involve spinning drives either.
- SFJulie 10y ago
  Well, what about a falsely "functioning" SSD in a system with an SLA of 99.999%. Backups may not be the answer. Ok, it may be RAID1+0 but, what about data corruptions? If primary fails, and secondary is "ok" what will be the state of your data? You don't roll back an image when everything is green. The point is that in critical systems according to the SLA or the critical aspect of the mission (flight recorder, medical instruments...) silent failures, silent data corruption may not be ok. However for 99% of the business that is making money out of SSD right now in the world (advertisement, marketing) this is not really a concern. They don't aim at an absolute reliability. Most industry just aim at good enough. Some small islands of IT aims at doing a legally accurate book-keeping. Like banking, accounting, legal electronic archiving for instance... This topic is covered in Signal Processing in the first chapter of the cost of False Positive and False Negative. Security and risk management norms force to enumerate and evaluate the costs and predicting the outcomes of failures in terms of costs. Here, the paper is saying SSD works pretty much fine all the time. But for failure, there is a lot we don't know. How do you evaluate the cost of a risk you cannot compute? And how does another solution that maybe slower rates in risk management with a better failure model that can be better predicted? The scarriest part in the study is the implication that their could be propagation of failures by coupling in certain environment and that some failure could be snowballing. SSD are affected by heat/memory usage, CPU load is affected by failures, heat rises, SSD are more likely to fail .... They are implying that failures can propagate or spread or happen like a contamination. 1 SSD failing alone is already a problem, even small chances of coupling in failures is making the potentiality for a massive failure. Would you take that risk with a SLA of 99.999%?
  - toyg 10y ago
    I honestly don't believe in five-nines and all that, but my experience is admittedly limited to sectors where it's more about point-in-time backup reliability (finance management) than anything else. As I said, it depends on what your definition of "critical system" is. For aviation, for instance, there are many other factors (shielding etc); no hardware is perfect for all uses. > How do you evaluate the cost of a risk you cannot compute? Easy: you come up with a worst-case scenario, make plans to address it, and the cost of implementing such plans is your cost. Say you assume you could get silent errors: you build test routines for actual data integrity and implement test systems for continuous checks. The cost of building and maintaining all that is the cost of (accepting) your risk.
    - hga 10y ago
      Telcos, at least in times past, really tried to achieve a true 5 9s. I worked for Lucent in 2001 on an ultimately doomed product that was essential to the company's independent survival, a Media Gateway (https://en.wikipedia.org/wiki/Media_gateway https://en.wikipedia.org/wiki/Media_gateway) to supply the other half of their successful Softswitch Media Gateway Controller product, and my NoVa group worked with the Cincinnati people who were responsible for the 5ESS. One of my tasks was vetting the low level design of the persistent database, and it was intense at all levels. No spinning rust, that's for sure. Point-in-time recovery is a snap compared to that domain.
    - marcosdumay 10y ago
      Good thing you pointed at aviation. Aviation standard goal is 9 nines of reliability, and many systems get there.
  - AlgorithmicTime 10y ago
    >Ok, it may be RAID1+0 but, what about data corruptions? ZFS.
- wtbob 10y ago
  > You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. > > ban SSD from critical systems. > That's an over-reaction if I've ever seen one. No, it's really not. SFJulie's point is that we don't know enough about the way SSDs fail to build a good failure model for them, and that we currently don't seem to have a good way to even detect that they are failing. If you can't build a failure model for something, you can't build a good risk model including it. That's kinda her point. FWIW, I don't know that I fully agree with that point: it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? > If it's something that has to guarantee high performance under massive load, SSDs are what you need, period. We seemed to do alright four years ago, no? SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them.
  - hga 10y ago
    it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? To me the three scariest words in engineering are "common mode failure". I personally never buy more than one hard disk at a time, so that I'll hopefully avoid something like both haven been shaken too much in their shipping container (the best explanation anyone was able to come up with for a Seagate failure pattern some time ago). So, yes, the above, to detect and recover, but plenty of services need very up to date backups, so if I was designing such a system, I'd make sure the (near-)real time backup would be to SSDs that share nothing in common with my production ones, different manufacturer, chip sets, etc. Or if the load could handle it ... ah, I see why Seagate is still selling real SAS enterprise disks (last time I checked, if I remember correctly, <= 1TB and up to 15K RPM). We understand those, not that they won't also fail on you.
  - bluecmd 10y ago
    I don't get where you got the notion that you need to understand the failure model to rely on it. It's certainly good to, and it will prevent over provisioning, but a lot of storage solutions today are so complex that understanding the failure models is practically impossible. S3? Super-black box. Basically all you need to define is the expected output. "99.9pct of my reads are within 1ms" + "The read response is correct 99.9999% of the time" and then verify. Eject the disk if that doesn't hold, restore from replica. The rate of replacements is a factor of how close you are to the true failure model.
rsync 10y ago
"Most SSD are reliable, the only problem is they are failing in a way that is tough to detect leading to consequences we may ignore. (falsely positive functioning SSD in operations and we may experience silent corruptions)" These SSDs are not failing for physical reasons, which is the model we have in our minds for HDD failure. They are failing for logical reasons. In the near term, I think there is a very simple way to protect against this and that is to mix models. At any given time, you can buy the very latest Intel SSD and also the one-model-ago Intel SSD on Amazon. They're both fantastic and wonderfully performant and their sizes are typically identical. So, for instance, when we make a boot mirror out of SSDs, we buy one of each and make our mirror with those two. Now we know that if there is some silent, insidious, bizarro firmware bug that makes the drive brick or causes it to burn itself out early, it can't affect them both identically (which is important, since they have identical workloads, since they are mirrors). This isn't as useful on a large array since there are not 12 or 15 reliable manufacturers to mix between - but in a large array, you aren't giving each drive an identical workload like you are with a mirror. I'll bet a lot of SSD usage is confined to boot mirrors, so I think this is a decent practice that a lot of people can adopt.

by SFJulie 10y ago

toyg 10y ago

You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. You offset the costs of worst-case scenario readiness with the productivity and financial gains of using the tech. If SSDs make your business 2x as productive or 2x as remunerative, the cost of an extra backup or two is simply a blip. > ban SSD from critical systems. That's an over-reaction if I've ever seen one. I guess it depends what a "critical system" is for you. If it's something that has to guarantee high performance under massive load, SSDs are what you need, period. If it's something that has to guarantee 100-year data storage under a mountain, then there might be better choices, but they might not involve spinning drives either.

SFJulie 10y ago
Well, what about a falsely "functioning" SSD in a system with an SLA of 99.999%. Backups may not be the answer. Ok, it may be RAID1+0 but, what about data corruptions? If primary fails, and secondary is "ok" what will be the state of your data? You don't roll back an image when everything is green. The point is that in critical systems according to the SLA or the critical aspect of the mission (flight recorder, medical instruments...) silent failures, silent data corruption may not be ok. However for 99% of the business that is making money out of SSD right now in the world (advertisement, marketing) this is not really a concern. They don't aim at an absolute reliability. Most industry just aim at good enough. Some small islands of IT aims at doing a legally accurate book-keeping. Like banking, accounting, legal electronic archiving for instance... This topic is covered in Signal Processing in the first chapter of the cost of False Positive and False Negative. Security and risk management norms force to enumerate and evaluate the costs and predicting the outcomes of failures in terms of costs. Here, the paper is saying SSD works pretty much fine all the time. But for failure, there is a lot we don't know. How do you evaluate the cost of a risk you cannot compute? And how does another solution that maybe slower rates in risk management with a better failure model that can be better predicted? The scarriest part in the study is the implication that their could be propagation of failures by coupling in certain environment and that some failure could be snowballing. SSD are affected by heat/memory usage, CPU load is affected by failures, heat rises, SSD are more likely to fail .... They are implying that failures can propagate or spread or happen like a contamination. 1 SSD failing alone is already a problem, even small chances of coupling in failures is making the potentiality for a massive failure. Would you take that risk with a SLA of 99.999%?
- toyg 10y ago
  I honestly don't believe in five-nines and all that, but my experience is admittedly limited to sectors where it's more about point-in-time backup reliability (finance management) than anything else. As I said, it depends on what your definition of "critical system" is. For aviation, for instance, there are many other factors (shielding etc); no hardware is perfect for all uses. > How do you evaluate the cost of a risk you cannot compute? Easy: you come up with a worst-case scenario, make plans to address it, and the cost of implementing such plans is your cost. Say you assume you could get silent errors: you build test routines for actual data integrity and implement test systems for continuous checks. The cost of building and maintaining all that is the cost of (accepting) your risk.
  - hga 10y ago
    Telcos, at least in times past, really tried to achieve a true 5 9s. I worked for Lucent in 2001 on an ultimately doomed product that was essential to the company's independent survival, a Media Gateway (https://en.wikipedia.org/wiki/Media_gateway https://en.wikipedia.org/wiki/Media_gateway) to supply the other half of their successful Softswitch Media Gateway Controller product, and my NoVa group worked with the Cincinnati people who were responsible for the 5ESS. One of my tasks was vetting the low level design of the persistent database, and it was intense at all levels. No spinning rust, that's for sure. Point-in-time recovery is a snap compared to that domain.
  - marcosdumay 10y ago
    Good thing you pointed at aviation. Aviation standard goal is 9 nines of reliability, and many systems get there.
- AlgorithmicTime 10y ago
  >Ok, it may be RAID1+0 but, what about data corruptions? ZFS.
wtbob 10y ago
> You start from the common but misguided assumption that "unknowns" are an absolute evil, whereas they are just another risk factor to weigh. > > ban SSD from critical systems. > That's an over-reaction if I've ever seen one. No, it's really not. SFJulie's point is that we don't know enough about the way SSDs fail to build a good failure model for them, and that we currently don't seem to have a good way to even detect that they are failing. If you can't build a failure model for something, you can't build a good risk model including it. That's kinda her point. FWIW, I don't know that I fully agree with that point: it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? > If it's something that has to guarantee high performance under massive load, SSDs are what you need, period. We seemed to do alright four years ago, no? SSDs are not a requirement: one can get equivalent system performance, with predictable failures, by spending more money on more hard drives and CPUs to talk to them.
- hga 10y ago
  it seems to me that cryptographic integrity checks and backups would suffice, but … maybe not? To me the three scariest words in engineering are "common mode failure". I personally never buy more than one hard disk at a time, so that I'll hopefully avoid something like both haven been shaken too much in their shipping container (the best explanation anyone was able to come up with for a Seagate failure pattern some time ago). So, yes, the above, to detect and recover, but plenty of services need very up to date backups, so if I was designing such a system, I'd make sure the (near-)real time backup would be to SSDs that share nothing in common with my production ones, different manufacturer, chip sets, etc. Or if the load could handle it ... ah, I see why Seagate is still selling real SAS enterprise disks (last time I checked, if I remember correctly, <= 1TB and up to 15K RPM). We understand those, not that they won't also fail on you.
- bluecmd 10y ago
  I don't get where you got the notion that you need to understand the failure model to rely on it. It's certainly good to, and it will prevent over provisioning, but a lot of storage solutions today are so complex that understanding the failure models is practically impossible. S3? Super-black box. Basically all you need to define is the expected output. "99.9pct of my reads are within 1ms" + "The read response is correct 99.9999% of the time" and then verify. Eject the disk if that doesn't hold, restore from replica. The rate of replacements is a factor of how close you are to the true failure model.

rsync 10y ago

"Most SSD are reliable, the only problem is they are failing in a way that is tough to detect leading to consequences we may ignore. (falsely positive functioning SSD in operations and we may experience silent corruptions)" These SSDs are not failing for physical reasons, which is the model we have in our minds for HDD failure. They are failing for logical reasons. In the near term, I think there is a very simple way to protect against this and that is to mix models. At any given time, you can buy the very latest Intel SSD and also the one-model-ago Intel SSD on Amazon. They're both fantastic and wonderfully performant and their sizes are typically identical. So, for instance, when we make a boot mirror out of SSDs, we buy one of each and make our mirror with those two. Now we know that if there is some silent, insidious, bizarro firmware bug that makes the drive brick or causes it to burn itself out early, it can't affect them both identically (which is important, since they have identical workloads, since they are mirrors). This isn't as useful on a large array since there are not 12 or 15 reliable manufacturers to mix between - but in a large array, you aren't giving each drive an identical workload like you are with a mirror. I'll bet a lot of SSD usage is confined to boot mirrors, so I think this is a decent practice that a lot of people can adopt.