3 ms·

As a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds li

As a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds like a good idea, but the extra complexity kills it. One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and would trip and then immediately fail, dropping all supplied power in an instant. The system was dramatically more reliable with this unit simply disabled.

petschge 7y ago
I said the same thing about UPS causing more trouble than preventing it. Then I moved to Africa and quickly learned that a local UPS can be VERY helpful.
jniedrauer 7y ago
This has absolutely not been my experience, at least for well designed cloud infrastructure. High availability doesn't have to be complex or hard to understand. In most cases, automatic failovers and restarts on well designed architecture fix problems with zero end-user impact. It happens so seamlessly that my standard procedure these days is to fail over infrastructure if I want to restart it.
- downerending 7y ago
  What is this "well-designed infrastructure" of which you speak? ;-)
shoo 7y ago
> almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). I've seen similar things with rabbitmq, in situation where the load & number of events to process is tiny but HA is some checkbox item to deliver without a serious QA process to measure if the setup is actually working effectively. Number of events in production we genuinely needed multiple rabbitmq nodes in the cluster to cope with failure of one or more nodes: 0 . Number of times when rabbitmq got too excited processing large messages that it delayed heart beating & subsequently decided that there was a network partition when there was in fact no such thing, leading the prod support team to step in and manually recover the cluster: > 0
rjkennedy98 7y ago
Every? that seems a stretch. Plenty have very good and simple HA architectures: Cassandra, Kafka come to mind. Others of course don't.
- downerending 7y ago
  Well, "almost every" and by "seen" I mean in environments that I've had to support, in corporate and academic land. Perhaps I've just had bad luck, but my impression is that getting this right, both internally and in the field, is a lot harder than it looks. And when it breaks, it can be a real s___show compared to the simple, non-HA version.
thijsvandien 7y ago
At least for the relatively small businesses that I deal with, I basically always recommend to save the effort that would go into building a 'highly available' system, and instead spend it on a scenario for keeping the business running when tech fails—which it will to some extent at some point, no matter what. I suppose that line of thinking would do wonders for much larger businesses as well. When a system becomes too important, rather than trying to make it even more reliable, try making it less important first.
- downerending 7y ago
  Indeed. It shocks me when I see boo-coo bucks being spent on all sorts of bells and whistles, and then ask about their backup strategy. Oh, all of our drives are RAID...

petschge 7y ago

I said the same thing about UPS causing more trouble than preventing it. Then I moved to Africa and quickly learned that a local UPS can be VERY helpful.

jniedrauer 7y ago

This has absolutely not been my experience, at least for well designed cloud infrastructure. High availability doesn't have to be complex or hard to understand. In most cases, automatic failovers and restarts on well designed architecture fix problems with zero end-user impact. It happens so seamlessly that my standard procedure these days is to fail over infrastructure if I want to restart it.

downerending 7y ago
What is this "well-designed infrastructure" of which you speak? ;-)

shoo 7y ago

> almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). I've seen similar things with rabbitmq, in situation where the load & number of events to process is tiny but HA is some checkbox item to deliver without a serious QA process to measure if the setup is actually working effectively. Number of events in production we genuinely needed multiple rabbitmq nodes in the cluster to cope with failure of one or more nodes: 0 . Number of times when rabbitmq got too excited processing large messages that it delayed heart beating & subsequently decided that there was a network partition when there was in fact no such thing, leading the prod support team to step in and manually recover the cluster: > 0

rjkennedy98 7y ago

Every? that seems a stretch. Plenty have very good and simple HA architectures: Cassandra, Kafka come to mind. Others of course don't.

downerending 7y ago
Well, "almost every" and by "seen" I mean in environments that I've had to support, in corporate and academic land. Perhaps I've just had bad luck, but my impression is that getting this right, both internally and in the field, is a lot harder than it looks. And when it breaks, it can be a real s___show compared to the simple, non-HA version.

thijsvandien 7y ago

At least for the relatively small businesses that I deal with, I basically always recommend to save the effort that would go into building a 'highly available' system, and instead spend it on a scenario for keeping the business running when tech fails—which it will to some extent at some point, no matter what. I suppose that line of thinking would do wonders for much larger businesses as well. When a system becomes too important, rather than trying to make it even more reliable, try making it less important first.

downerending 7y ago
Indeed. It shocks me when I see boo-coo bucks being spent on all sorts of bells and whistles, and then ask about their backup strategy. Oh, all of our drives are RAID...