3 ms·

What ticked me is that when the primary system threw in the towel, an EXACT SAME system took over and ran the exact same code on the exact same data as the prim

What ticked me is that when the primary system threw in the towel, an EXACT SAME system took over and ran the exact same code on the exact same data as the primary. I know that with code and algorithms it's not always the case but even then you know what doing the same thing over and over expecting different results defines... Yes, it can be argued that the software should've had more graceful failure modes and this shouldn't have thrown a critical exception. It can be argued that the programmers should've seen this possibility. We can argue a lot of things about this. But the reality is that this is a mission-critical system. And for such systems, there're ways to mitigate all of these mistakes and allow the system to continue functioning. The easiest (but least safe) one would be to have the secondary system loaded with code that does the same thing but written by a different team/vendor. It reduces the chance from 100% to much-much less that if any input provokes an unforseen, system-breaking bug in the primary, the same input will provoke the same bug in the secondary. An even better solution is to have a triumvirate system, where all 3 have code written by different teams, and they always compare results. If 3 agree, great, if 2 agree, not so great but safe to assume that the bug is in the 1 not the 2 (but should throw an alert for the supervisors that the whole system is in a degraded mode where any further node failure is a showstopper), and if all disagree, grind everything to a halt because the world is ending, and let the humans handle it. It can be refined even further. And it's not something new. So why wasn't this system implemented in such a way? (Aside from cost. I don't care about anyones cost-cutting incentives in mission-critical systems. Sorry capitalism...)

throwaway894345 3y ago
> Aside from cost. I don't care about anyones cost-cutting incentives in mission-critical systems. Sorry capitalism... Capitalism is happy to have redundancy in mission critical systems all the time. Why would it care here?
- drachir91 3y ago
  I don't know but in recent years I'm increasingly seeing mission critical systems having only token or "apparent" rendundancies instead of real ones, and couldn't find any other rationale than cost savings and shareholder bottom lines. I'm not saying that capitalism = bad, it's mostly better than the alternatives, but just like its most direct competitor, it suffers from bad implementations across the world and unbounded human greed. A recent and very "in the face" example, also from the air travel industry would be the B737 Max and its AoA sensors. There were two, for two flight computers, but MCAS only used 1 flight computer and 1 AoA sensor, despite the already existing crosslinks between the flight computers and the sensors... Pofit maxing first with the "no need for a new type rating for the pilots", then cost-cutting first in aeronautical engineering (solving an airframe design problem with software, plus designing a flight envelope protection system that can overpower the human pilots). Then cost-cutting in software engineering and QC, rushing out software made by (probably) inexperienced in the field engineers and failing to properly test it and ensure that it had the needed redundancy.
crabbone 3y ago
> an EXACT SAME system took over and ran the exact same code Did you ever work with HA systems? Because this is how they work. It's two copies of the same system intended for the cases when eg. hardware fails, or network partitioning happens etc.
- drachir91 3y ago
  No, I do not. But HA systems work like that because hardware or network failure is what they are designed to guard against, not a latent bug in the software logic. If there's a software bug, both systems will exhibit the same behavior, so HA fails there.
  - crabbone 3y ago
    In practice, you have two kinds of HA systems (based on this criteria): * Live + standby. Typically, the state of the live system is passively replicasted to the standby, where standby is meant to take over if it doesn't hear from the live one / the live one sends nonsense. (For example, you can use Kubernetes API server in this capacity). * Consensus systems where each actor plays the same role, while there's an "elected" master which deals with synchronization of the system state. (For example, you can use Etcd). In either case, it's the same program, but with a somewhat different state. It doesn't make sense to make different programs to deal with this problem because you will have double the amount of bugs for no practical gains. It's a lot more likely that two different programs will fail to communicate to each other than one program communicating to its own replica. Also, if you believe you were right the first time: why would you make the other one different? You will definitely want to choose the better of the two and have copies of that than have a better and a worse work together...
  - barrucadu 3y ago
    How can you tell whether the problem is due to a software bug or due to a hardware fault though? The software could have thrown the "catastrophic failure, stop the world" exception due to memory corruption.
burntwater 3y ago
I'm wondering if the backup system could have a delayed queue; say, 30 seconds behind. If the primary fails, and exactly 30 seconds later the secondary system fails, you have reasonable assurance that it was queue input that caused the failure. Rollback to the last successful queue input, skip and flag the suspect input, and see if the next input is successful.
- drachir91 3y ago
  This looks to me like it could work, but would need a ready force of technicians always expecting something like that so they can troubleshoot it in a timely manner.

throwaway894345 3y ago

> Aside from cost. I don't care about anyones cost-cutting incentives in mission-critical systems. Sorry capitalism... Capitalism is happy to have redundancy in mission critical systems all the time. Why would it care here?

drachir91 3y ago
I don't know but in recent years I'm increasingly seeing mission critical systems having only token or "apparent" rendundancies instead of real ones, and couldn't find any other rationale than cost savings and shareholder bottom lines. I'm not saying that capitalism = bad, it's mostly better than the alternatives, but just like its most direct competitor, it suffers from bad implementations across the world and unbounded human greed. A recent and very "in the face" example, also from the air travel industry would be the B737 Max and its AoA sensors. There were two, for two flight computers, but MCAS only used 1 flight computer and 1 AoA sensor, despite the already existing crosslinks between the flight computers and the sensors... Pofit maxing first with the "no need for a new type rating for the pilots", then cost-cutting first in aeronautical engineering (solving an airframe design problem with software, plus designing a flight envelope protection system that can overpower the human pilots). Then cost-cutting in software engineering and QC, rushing out software made by (probably) inexperienced in the field engineers and failing to properly test it and ensure that it had the needed redundancy.

crabbone 3y ago

> an EXACT SAME system took over and ran the exact same code Did you ever work with HA systems? Because this is how they work. It's two copies of the same system intended for the cases when eg. hardware fails, or network partitioning happens etc.

drachir91 3y ago
No, I do not. But HA systems work like that because hardware or network failure is what they are designed to guard against, not a latent bug in the software logic. If there's a software bug, both systems will exhibit the same behavior, so HA fails there.
- crabbone 3y ago
  In practice, you have two kinds of HA systems (based on this criteria): * Live + standby. Typically, the state of the live system is passively replicasted to the standby, where standby is meant to take over if it doesn't hear from the live one / the live one sends nonsense. (For example, you can use Kubernetes API server in this capacity). * Consensus systems where each actor plays the same role, while there's an "elected" master which deals with synchronization of the system state. (For example, you can use Etcd). In either case, it's the same program, but with a somewhat different state. It doesn't make sense to make different programs to deal with this problem because you will have double the amount of bugs for no practical gains. It's a lot more likely that two different programs will fail to communicate to each other than one program communicating to its own replica. Also, if you believe you were right the first time: why would you make the other one different? You will definitely want to choose the better of the two and have copies of that than have a better and a worse work together...
- barrucadu 3y ago
  How can you tell whether the problem is due to a software bug or due to a hardware fault though? The software could have thrown the "catastrophic failure, stop the world" exception due to memory corruption.

burntwater 3y ago

I'm wondering if the backup system could have a delayed queue; say, 30 seconds behind. If the primary fails, and exactly 30 seconds later the secondary system fails, you have reasonable assurance that it was queue input that caused the failure. Rollback to the last successful queue input, skip and flag the suspect input, and see if the next input is successful.

drachir91 3y ago
This looks to me like it could work, but would need a ready force of technicians always expecting something like that so they can troubleshoot it in a timely manner.