3 ms·

On Monday at 3:30am PDT, we experienced a hardware failure in our primary encryption hardware device. The failure cascaded to the backup slave device as w

by sync 14y ago

On Monday at 3:30am PDT, we experienced a hardware failure in our primary encryption hardware device. The failure cascaded to the backup slave device as well. How does a hardware failure cascade to a backup slave device?

phillc 14y ago
One of the racks fell over onto the other racks, like dominios.
- eckyptang 14y ago
  Ha - that made me laugh. Ironically something not far off from that happened to a contract outfit I was working for. They had a telecoms rack rather than a proper one and they couldn't bolt 9x maxed out DL380G2 servers on it as it started to bend. They decided the best approach was to just pile them up on the rack bed. This was until the one at the bottom needed a RAM upgrade. Cue 6 people carefully trying to lift up 8x DL380's in one go and sliding various OReilly books underneath. The inevitable happened - a hand gave way resulting in a pile of DL380s spread all over the machine room floor and a broken toe from someone in the hosting facility. Nothing went down though!
shimonamit 14y ago
"This failure corrupted encryption keys used to access stored credit cards to process recurring transactions" Looks like they were sharing encryption keys by replication or shared storage.
deleted 14y ago
[deleted]
smoyer 14y ago
In the same way a slave database can be corrupted by the master when it has a failure. If you're encrypting data on the master and replicating to the slave, improperly encrypted data may be propagated before a fail-over. This is a case where having the master fail completely and go off-line is preferable to a "working failure".
- mike-cardwell 14y ago
  This is why I use incremental backups, and check them regularly.
- awicklander 14y ago
  But doesn't a 'hardware failure' imply failure to a specific piece of equipment/hardware? A cascading issue with actual data isn't a hardware failure, it's a software failure.
alexkus 14y ago
"Flailover" The primary fails because of a bug and so it fails over to the backup which flails-over because of the exact same bug, c.f. https://en.wikipedia.org/wiki/Ariane_5_Flight_501 https://en.wikipedia.org/wiki/Ariane_5_Flight_501

by sync 14y ago

phillc 14y ago

One of the racks fell over onto the other racks, like dominios.

eckyptang 14y ago
Ha - that made me laugh. Ironically something not far off from that happened to a contract outfit I was working for. They had a telecoms rack rather than a proper one and they couldn't bolt 9x maxed out DL380G2 servers on it as it started to bend. They decided the best approach was to just pile them up on the rack bed. This was until the one at the bottom needed a RAM upgrade. Cue 6 people carefully trying to lift up 8x DL380's in one go and sliding various OReilly books underneath. The inevitable happened - a hand gave way resulting in a pile of DL380s spread all over the machine room floor and a broken toe from someone in the hosting facility. Nothing went down though!

shimonamit 14y ago

"This failure corrupted encryption keys used to access stored credit cards to process recurring transactions" Looks like they were sharing encryption keys by replication or shared storage.

deleted 14y ago

[deleted]

smoyer 14y ago

In the same way a slave database can be corrupted by the master when it has a failure. If you're encrypting data on the master and replicating to the slave, improperly encrypted data may be propagated before a fail-over. This is a case where having the master fail completely and go off-line is preferable to a "working failure".

mike-cardwell 14y ago
This is why I use incremental backups, and check them regularly.
awicklander 14y ago
But doesn't a 'hardware failure' imply failure to a specific piece of equipment/hardware? A cascading issue with actual data isn't a hardware failure, it's a software failure.

alexkus 14y ago

"Flailover" The primary fails because of a bug and so it fails over to the backup which flails-over because of the exact same bug, c.f. https://en.wikipedia.org/wiki/Ariane_5_Flight_501 https://en.wikipedia.org/wiki/Ariane_5_Flight_501