3 ms·

Early in my career when in a discussion with my manager at the time, he threw out the "planes with two engines have twice as many failures as single engine plan

Early in my career when in a discussion with my manager at the time, he threw out the "planes with two engines have twice as many failures as single engine planes" line. So in my now close to 30 years of experience later, working on systems expected to run 24/7/365, I can't tell you what percentage of failures and outages were caused by the redundancy/fail-over/etc software and hardware layered into a system, but its definitely a very significant part of the problem. Frequently because its just that a "layer" someone bolted on, even if that layer was some software engineer designing for "web scale". All the edge cases are frequently not completely thought out, and when you hit one its a lot harder to recover, than simply restarting a single service running on a single server.

amluto 5y ago
I once managed a thoroughly non-critical server that lived in a closet. In the couple years I managed it, we had quite a few failures due to the UPS tripping and no unscheduled downtime at all from any other reason. I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable. Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.
robocat 5y ago
> "planes with two engines have twice as many failures as single engine planes" There has to be a smart reply to that, like: 100% of pilots of single engine planes don’t make it home after a single engine failure. 100% of pilots of dual engine planes do make it home after a single engine failure.
- Jasper_ 5y ago
  Planes are natural gliders and you can recover from engine failure. It's not like they just fall out of the sky.
  - RobRivera 5y ago
    say for the zoomy lawn darts we call fighters. pushes glasses
- asmithmd1 5y ago
  If you lose an engine on a twin engine plane, you have to decide if you have an emergency or not. If you lose an engine on a single engine plane, you have to decide where you can make an emergency landing.
hef19898 5y ago
>> "planes with two engines have twice as many failures as single engine planes" Sounds like someone didn't understand redundancy...

amluto 5y ago

I once managed a thoroughly non-critical server that lived in a closet. In the couple years I managed it, we had quite a few failures due to the UPS tripping and no unscheduled downtime at all from any other reason. I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable. Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.

robocat 5y ago

> "planes with two engines have twice as many failures as single engine planes" There has to be a smart reply to that, like: 100% of pilots of single engine planes don’t make it home after a single engine failure. 100% of pilots of dual engine planes do make it home after a single engine failure.

Jasper_ 5y ago
Planes are natural gliders and you can recover from engine failure. It's not like they just fall out of the sky.
- RobRivera 5y ago
  say for the zoomy lawn darts we call fighters. pushes glasses
asmithmd1 5y ago
If you lose an engine on a twin engine plane, you have to decide if you have an emergency or not. If you lose an engine on a single engine plane, you have to decide where you can make an emergency landing.

hef19898 5y ago

>> "planes with two engines have twice as many failures as single engine planes" Sounds like someone didn't understand redundancy...