8 ms·

Since most drives are started and used concurrently this bug would blow any RAID set up. There's a dark day coming for some sysadmins.

cesarb 7y ago
That's only if the sysadmin was trusting a single server with the data, instead of a pair of redundant servers. Which were probably installed and started up at nearly the same time. Oops. This bug has the potential of simultaneously damaging whole sets of servers, if they were bought and installed in bulk. Dark day indeed.
- abarringer 7y ago
  We have a cluster of four nodes that were all setup and brought online within hours of each other. The entire cluster could blow up within a couple hours if not patched.
  - Scoundreller 7y ago
    I guess cluster nodes should be scheduled to be taken down for random amounts of time so that they fail in sequence more gracefully. Have an “off on weekends” node. And 24/7 nodes.
vidarh 7y ago
This is why I never use drives from the same batch, ideally never the same model, and usually not the same manufacturer. It happens way too regularly that drives start failing around the same time.
- GrayShade 7y ago
  > I never use drives from the same batch Note that it wouldnțt help in this instance, as the bug is caused by the amount of time a drive was running. Different manufacturers would work, yes.
  - vidarh 7y ago
    That's true. But it's easier to sell people on avoiding mixing batches, and it catches the most common reliability issues. I'd never personally trust my own files to drives from a single manufacturer, though - I've seen too many problems with that.
  - CarlRJ 7y ago
    Need to age a few of the drives by a hundred hours before putting them in the set.
  - imtringued 7y ago
    The problem with SSDs is that they are too reliable and when they fail they fail reliably. The only reason why they fail is usually an intrinsic flaw in the hardware design or firmware which all SSDs of the same model share equally.
- Piskvorrr 7y ago
  Amen. Bought a pair of brand new disks some years ago, which failed days into the deployment...apparently from a submarine batch. Luckily the array had another, older disk, which kept it up until a replacement arrived.
  - vidarh 7y ago
    Brand new disks are particularly troublesome - worth doing a burn-in of hammering them for a few days (or longer if you can take the time) to weed out the worst ones.
- ghaff 7y ago
  Good luck doing that at scale though. You can mix things up to some degree (and probably should) but if you need thousands of drives you're going to end up with lots from the same batch.
  - vidarh 7y ago
    Instead of ordering 5k drives of a single model, you order from 2-3 manufacturer, and split the order between 2-3 different models from each, and build each array from one drive from each distinct type of drive.
- AdamJacobMuller 7y ago
  I don't see the value in that in most cases, honestly. If you have, say, a 10-drive wide RAID6 you would need to source drives from 5 manufacturers/batches/models in order to be resilient to that kind of failure. Even if that was feasible that seems horrible to maintain long-term. Doing a red/blue setup where your red systems use one type of drive and your blue systems use another type of drive seems like it could be reasonably accomplished.
  - vidarh 7y ago
    > If you have, say, a 10-drive wide RAID6 you would need to source drives from 5 manufacturers/batches/models in order to be resilient to that kind of failure. Even if that was feasible that seems horrible to maintain long-term. If anything, it's easier to maintain, as all you need to ensure on replacing a drive is to not unintentionally make the array have too many of one type of drive. In practice, it means you just regularly cycle what model you buy for your spares instead of the often totally counter-productive practice of making extra effort to find a supply of the exact same model. In effect, most places I've done this, it has simply translated into refilling our spares from the currently most cost-effective model or two, and cycling manufacturers, instead of continuing to buy the same model. The point is not to religiously prevent any kind of potentially unfortunate mixing, because these errors are fairly rare, but to reduce a very real chance using very simple means. Over the 20+ years I've been doing this, I've seen at least 4-5 cases where homogenous raid arrays have been a major liability (the first one, that taught me to avoid this was the infamous IBM Death Star, where the film on the platters was almost totally scraped off; we had an array that we thankfully didn't lose data from, thanks to backups and careful management once the drives started failing once a week - only for it to take the array 4-5 days to rebuild... we didn't lose data, but we lost a lot of time babysitting that system and working around our dependency on it as a precaution). I started mixing manufacturers after having had near-misses with several arrays with OCZ drives, where it appears to have been firmware problems across drive models. > Doing a red/blue setup where your red systems use one type of drive and your blue systems use another type of drive seems like it could be reasonably accomplished. You need multiple systems too, but the point is that every hour a system is down because of an easily avoidable problem is an hour where your system has reduced resilience and capacity. It's trivial to prevent these kinds of errors from taking down a raid array, so it's pretty pointless not to.
- HorstG 7y ago
  Different SSD vendors is impossible with HP servers and controllers, they only talk to their own expensive gear. So the disk diversity option is off the table for HP customers.
  - vidarh 7y ago
    That would be a deal-breaker for me in choosing HP servers then, as that just seems like begging for trouble.
manls 7y ago
One of the general best practices is to have diversity in the array of drives. It's not for bugs like this though although it helps for bugs like this. It's to ensure that not all disks fail at the same time. If you use disks from the same batch in a RAID, they would all begin to fail around the same time, because all of them have the same lifetime more or less.
angrygoat 7y ago
> By disregarding this notification and not performing the recommended resolution, the customer accepts the risk of incurring future related errors. This seems incredibly rich. If you have a bunch of this kit, and you don't immediately shut it down to apply firmware updates, then HPE wash their hands of the consequences.

cesarb 7y ago

That's only if the sysadmin was trusting a single server with the data, instead of a pair of redundant servers. Which were probably installed and started up at nearly the same time. Oops. This bug has the potential of simultaneously damaging whole sets of servers, if they were bought and installed in bulk. Dark day indeed.

abarringer 7y ago
We have a cluster of four nodes that were all setup and brought online within hours of each other. The entire cluster could blow up within a couple hours if not patched.
- Scoundreller 7y ago
  I guess cluster nodes should be scheduled to be taken down for random amounts of time so that they fail in sequence more gracefully. Have an “off on weekends” node. And 24/7 nodes.

vidarh 7y ago

This is why I never use drives from the same batch, ideally never the same model, and usually not the same manufacturer. It happens way too regularly that drives start failing around the same time.

GrayShade 7y ago
> I never use drives from the same batch Note that it wouldnțt help in this instance, as the bug is caused by the amount of time a drive was running. Different manufacturers would work, yes.
- vidarh 7y ago
  That's true. But it's easier to sell people on avoiding mixing batches, and it catches the most common reliability issues. I'd never personally trust my own files to drives from a single manufacturer, though - I've seen too many problems with that.
- CarlRJ 7y ago
  Need to age a few of the drives by a hundred hours before putting them in the set.
- imtringued 7y ago
  The problem with SSDs is that they are too reliable and when they fail they fail reliably. The only reason why they fail is usually an intrinsic flaw in the hardware design or firmware which all SSDs of the same model share equally.
Piskvorrr 7y ago
Amen. Bought a pair of brand new disks some years ago, which failed days into the deployment...apparently from a submarine batch. Luckily the array had another, older disk, which kept it up until a replacement arrived.
- vidarh 7y ago
  Brand new disks are particularly troublesome - worth doing a burn-in of hammering them for a few days (or longer if you can take the time) to weed out the worst ones.
ghaff 7y ago
Good luck doing that at scale though. You can mix things up to some degree (and probably should) but if you need thousands of drives you're going to end up with lots from the same batch.
- vidarh 7y ago
  Instead of ordering 5k drives of a single model, you order from 2-3 manufacturer, and split the order between 2-3 different models from each, and build each array from one drive from each distinct type of drive.
AdamJacobMuller 7y ago
I don't see the value in that in most cases, honestly. If you have, say, a 10-drive wide RAID6 you would need to source drives from 5 manufacturers/batches/models in order to be resilient to that kind of failure. Even if that was feasible that seems horrible to maintain long-term. Doing a red/blue setup where your red systems use one type of drive and your blue systems use another type of drive seems like it could be reasonably accomplished.
- vidarh 7y ago
  > If you have, say, a 10-drive wide RAID6 you would need to source drives from 5 manufacturers/batches/models in order to be resilient to that kind of failure. Even if that was feasible that seems horrible to maintain long-term. If anything, it's easier to maintain, as all you need to ensure on replacing a drive is to not unintentionally make the array have too many of one type of drive. In practice, it means you just regularly cycle what model you buy for your spares instead of the often totally counter-productive practice of making extra effort to find a supply of the exact same model. In effect, most places I've done this, it has simply translated into refilling our spares from the currently most cost-effective model or two, and cycling manufacturers, instead of continuing to buy the same model. The point is not to religiously prevent any kind of potentially unfortunate mixing, because these errors are fairly rare, but to reduce a very real chance using very simple means. Over the 20+ years I've been doing this, I've seen at least 4-5 cases where homogenous raid arrays have been a major liability (the first one, that taught me to avoid this was the infamous IBM Death Star, where the film on the platters was almost totally scraped off; we had an array that we thankfully didn't lose data from, thanks to backups and careful management once the drives started failing once a week - only for it to take the array 4-5 days to rebuild... we didn't lose data, but we lost a lot of time babysitting that system and working around our dependency on it as a precaution). I started mixing manufacturers after having had near-misses with several arrays with OCZ drives, where it appears to have been firmware problems across drive models. > Doing a red/blue setup where your red systems use one type of drive and your blue systems use another type of drive seems like it could be reasonably accomplished. You need multiple systems too, but the point is that every hour a system is down because of an easily avoidable problem is an hour where your system has reduced resilience and capacity. It's trivial to prevent these kinds of errors from taking down a raid array, so it's pretty pointless not to.
HorstG 7y ago
Different SSD vendors is impossible with HP servers and controllers, they only talk to their own expensive gear. So the disk diversity option is off the table for HP customers.
- vidarh 7y ago
  That would be a deal-breaker for me in choosing HP servers then, as that just seems like begging for trouble.

manls 7y ago

One of the general best practices is to have diversity in the array of drives. It's not for bugs like this though although it helps for bugs like this. It's to ensure that not all disks fail at the same time. If you use disks from the same batch in a RAID, they would all begin to fail around the same time, because all of them have the same lifetime more or less.

angrygoat 7y ago

> By disregarding this notification and not performing the recommended resolution, the customer accepts the risk of incurring future related errors. This seems incredibly rich. If you have a bunch of this kit, and you don't immediately shut it down to apply firmware updates, then HPE wash their hands of the consequences.