5 ms·

From the article: "When the error happened, the system should have carried out what’s called a failover -- an automatic switching to the No. 2 device. But for

by bunnie 6y ago

From the article: "When the error happened, the system should have carried out what’s called a failover -- an automatic switching to the No. 2 device. But for reasons the exchange’s executives couldn’t explain, that process also failed. That had a knock-on effect on servers called information distribution gateways that are meant to send market information to traders."

seppel 6y ago
couldn't explain = didn't want to :) But anyway, it also shows that the failover was not properly tested.
- rightbyte 6y ago
  Or that the fault killing the first on also killed the failover? Like a rampage JIS encoding control character running over the wires.
  - marcan_42 6y ago
    It was a memory error. Hardware failure. The failover process is what went majorly wrong here. It seems they had a press conference with more details in Japanese.
karmakaze 6y ago
Automatic failover is like data backups, if not regularly tested it's like it doesn't exist.
- vmception 6y ago
  Thats a bit extreme in my experience, I would say its just closer to luck
  - rwbhn 6y ago
    I would be very curious to hear about your favorable experience with untested failover systems. My last job we tested our active/standby failover like clockwork every 3 months. It served us very well.
    - vmception 6y ago
      My experience is that they worked when needed and it was a sigh of relief, and a little pat on the back that we perceived reality correctly when we had the foresight to set them up.
  - ncmncm 6y ago
    An untested failover system is indistinguishable from a box burning power and doing nothing. On a failure, it might work. But there might not be a failure. Same difference.
  - karmakaze 6y ago
    It's sort-of a saying in the industry that can be applied to many things. The biggest problem is the people who might know how to make it work are no longer there or have forgotten. If it's automated there's a chance it might work, but if it's slightly off then it's even harder to comprehend and adjust. I've seen backups and failovers not work so many times that it's an amazing surprise when one actually works--usually after laborious manual off-script intervention and invention. I was being somewhat kind to failovers, I have seen smallish backups work on several occasions. An untested disaster recovery is the worst. Edit: think of an unexercised procedure as not "It works for me" but rather "it worked for me once."
- renewiltord 6y ago
  Failover sounds like the sort of thing you just do all the time. i.e. at the extreme, you treat failover like loadbalancing.
  - ptr 6y ago
    These kinds of systems typically don’t use load balancing, they’re provisioned for the expected max capacity needed.
- solarkraft 6y ago
  It seems like for true redundancy one should have multiple failover devices with regular actual switching between them.
- fomine3 6y ago
  I don't think failover won't work in every failure scenario, but won't work in this particular scenario. We can't test all possible failure situation regularly.
numpad0 6y ago
Someone speculated that since this is a RAM failure, the storage in question might kept crawling with reduced RAM and full workload so failover didn’t kick in(either failed from high load, or didn’t trigger from the fact that faulty RAM is successfully isolated). Sounds plausible.
- toast0 6y ago
  RAM failures are fun. Maybe 15 years ago, I had one box reboot itself and come back up with only ~ 16 MB out of an expected 1 GB or so; it was pretty overprovisioned and was able to sort of keep up working from swap but was sending alarms on a system that was always quiet (near idle cpu normally, important but low throughput system). More recently, I've had a couple systems with so high a rate of correctable ECC errors that machine check processing ate nearly all the cpu; but it still had enough to pass healthchecks, but not handle requests in a timely fashion, but didn't manage to get an uncorrectable error that would have paniced the machine so failover could work.

by bunnie 6y ago

seppel 6y ago

couldn't explain = didn't want to :) But anyway, it also shows that the failover was not properly tested.

rightbyte 6y ago
Or that the fault killing the first on also killed the failover? Like a rampage JIS encoding control character running over the wires.
- marcan_42 6y ago
  It was a memory error. Hardware failure. The failover process is what went majorly wrong here. It seems they had a press conference with more details in Japanese.

karmakaze 6y ago

Automatic failover is like data backups, if not regularly tested it's like it doesn't exist.

vmception 6y ago
Thats a bit extreme in my experience, I would say its just closer to luck
- rwbhn 6y ago
  I would be very curious to hear about your favorable experience with untested failover systems. My last job we tested our active/standby failover like clockwork every 3 months. It served us very well.
  - vmception 6y ago
    My experience is that they worked when needed and it was a sigh of relief, and a little pat on the back that we perceived reality correctly when we had the foresight to set them up.
- ncmncm 6y ago
  An untested failover system is indistinguishable from a box burning power and doing nothing. On a failure, it might work. But there might not be a failure. Same difference.
- karmakaze 6y ago
  It's sort-of a saying in the industry that can be applied to many things. The biggest problem is the people who might know how to make it work are no longer there or have forgotten. If it's automated there's a chance it might work, but if it's slightly off then it's even harder to comprehend and adjust. I've seen backups and failovers not work so many times that it's an amazing surprise when one actually works--usually after laborious manual off-script intervention and invention. I was being somewhat kind to failovers, I have seen smallish backups work on several occasions. An untested disaster recovery is the worst. Edit: think of an unexercised procedure as not "It works for me" but rather "it worked for me once."
renewiltord 6y ago
Failover sounds like the sort of thing you just do all the time. i.e. at the extreme, you treat failover like loadbalancing.
- ptr 6y ago
  These kinds of systems typically don’t use load balancing, they’re provisioned for the expected max capacity needed.
solarkraft 6y ago
It seems like for true redundancy one should have multiple failover devices with regular actual switching between them.
fomine3 6y ago
I don't think failover won't work in every failure scenario, but won't work in this particular scenario. We can't test all possible failure situation regularly.

numpad0 6y ago

Someone speculated that since this is a RAM failure, the storage in question might kept crawling with reduced RAM and full workload so failover didn’t kick in(either failed from high load, or didn’t trigger from the fact that faulty RAM is successfully isolated). Sounds plausible.

toast0 6y ago
RAM failures are fun. Maybe 15 years ago, I had one box reboot itself and come back up with only ~ 16 MB out of an expected 1 GB or so; it was pretty overprovisioned and was able to sort of keep up working from swap but was sending alarms on a system that was always quiet (near idle cpu normally, important but low throughput system). More recently, I've had a couple systems with so high a rate of correctable ECC errors that machine check processing ate nearly all the cpu; but it still had enough to pass healthchecks, but not handle requests in a timely fashion, but didn't manage to get an uncorrectable error that would have paniced the machine so failover could work.