5 ms·

A Brief History of High Availability

59 points by EntICOnc 4y ago

jve 4y ago
> second, by placing computers under constant demand, they were more likely to fail Really? In what sense? Hardware fails or what? The complexity usually brings everything down. Anyway, looking forward to cockroachdb, it seems it solves some high-availability / multi-site problems.
- notacoward 4y ago
  I think what the authors are getting at is that many failure conditions are more likely to manifest under load. This includes various kinds of resource exhaustion, including those induced by leaks. It also includes things like timeouts due to long queues, and even race conditions. The higher the load, or the longer it's sustained, the more likely it becomes that such bugs will manifest. In every such system I've seen, from very early days of the internet or old-school HA to modern multi-thousand-machine clusters, there have been bugs that only showed up under heavy and/or sustained load. Both Erlang's supervisor trees and Recovery Oriented Computing (Berkeley/Stanford) are based on the observation that such failures will always be with us, and have influenced many subsequent systems - arguably including CockroachDB.
  - marcosdumay 4y ago
    Heat caused failures also increase with load, as does electromigration, disk stress, power failures... The GP has a very digital model of computers. It's not really useful for predicting failure.
- zamadatix 4y ago
  One concrete example involving DBs is the PostgreSQL TXID wraparound. It's basically 32 bit counter where if things go wrong or were built wrong it just keeps going up until it becomes a problem. If you're only doing <=1 request per second this won't ever be a problem since even if you ran it your whole life without freezing any old records it wouldn't approach max value. On the other hand if it's under a constant 1000 request per second load it can be the source of problems in about a month. You could argue to just call the TXID "complexity" but that'd come with a wide enough definition of what complexity is that it just becomes a replacement for saying "buggy code" which isn't very useful. Most leave complexity as being more like "grossly overengineered" to leave room to avoid that.
notacoward 4y ago
I was very unsurprised to see that this history starts a bit late. In 1992 I was working on HACMP/6000, which was a high-availability clustering product for IBM's AIX (though I worked at a company that was on contract to them). Databases were a primary use case, relying on shared storage instead of replication. We were hardly the first into that space BTW. DEC in particular had developed a lot of the concepts (e.g. locking models) that we and others at the time emulated, and the history in this space goes back even further than them. Were those systems as scalable as what we have now, or even as robust? Nowhere even close. But the erasure of that history still irks me. Modern distributed databases (and distributed systems in general) in their turn relied heavily on experience and concepts from those earlier systems, but rarely acknowledge the fact.
- macintux 4y ago
  Tandem started selling their NonStop systems in the mid-70s, so clearly high availability was important. At Basho we worked hard to acknowledge our debt to distributed systems pioneers. Not everyone is ignorant of that history.
  - notacoward 4y ago
    Yup, the Basho folks were/are very good about that. BTW hi, we've quite likely met, though I don't know which Basho alum you are. ;) And yes, even I was a pretty late entrant to the HA game. I never worked on Tandem systems, but I worked with Tandem people at the job I mentioned, after having worked with both Stratus and Sequoia systems previously. Those were all FT rather than HA, but the point remains that work on making systems survive failures through redundancy goes back much further than the OP mentions.
    - macintux 4y ago
      Oh, definitely, was trying to add to rather than detract from your point. I was mainly on the periphery at Basho, started as tech evangelist, migrated to engineering to help get 2.0 out the door. John Daily.
  - 4y ago
throwaway787544 4y ago
Fun facts about availability: 1) the more nodes, the higher probability of failure, 2) the more load, the higher probability of failure, 3) complex logic and/or nodes that aren't tested frequently have a higher probability of failure Basically, every person who has claimed "X is easy, I do it for my personal use and it's fine with one giant node, don't need HA" is true (for them) because it's literally less likely to fail. But if they had more load they'd see more failures, and probability doesn't mean they won't encounter the random failures that are much harder to recover when there's no backup system. HA means you're going to have more failure, which means more work to keep it running. But it's necessary work if total system failure is unacceptable. Two engines in an airplane is twice the maintenance, but it's better than the only engine conking out mid-flight.
- rospaya 4y ago
  > 1) the more nodes, the higher probability of failure I may be misreading your comment, but how is this true?
  - lhoff 4y ago
    If you have one node with the probability of x% that it fails within a year and you add a second one with the same probability to fail the total pobability that one of the two nodes fail within a year goes up.
    - deleted 4y ago
      [deleted]
  - macintux 4y ago
    Not OP, but two things I’ve found to be true: * the more complex the configuration required to provide high availability, the more likely something will go wrong and reduce availability * the more nodes you have, the more likely one of them will fail. Whether that reduces availability typically depends on the failure mode and control layer. For an example of the latter, with the distributed database Riak (and I imagine others as well) if a server is starting to fail (corrupt disk, bad network card, etc) you’re in a much better place if it stops working entirely. A gradual node failure is much much worse.

59 points by EntICOnc 4y ago

jve 4y ago

> second, by placing computers under constant demand, they were more likely to fail Really? In what sense? Hardware fails or what? The complexity usually brings everything down. Anyway, looking forward to cockroachdb, it seems it solves some high-availability / multi-site problems.

notacoward 4y ago
I think what the authors are getting at is that many failure conditions are more likely to manifest under load. This includes various kinds of resource exhaustion, including those induced by leaks. It also includes things like timeouts due to long queues, and even race conditions. The higher the load, or the longer it's sustained, the more likely it becomes that such bugs will manifest. In every such system I've seen, from very early days of the internet or old-school HA to modern multi-thousand-machine clusters, there have been bugs that only showed up under heavy and/or sustained load. Both Erlang's supervisor trees and Recovery Oriented Computing (Berkeley/Stanford) are based on the observation that such failures will always be with us, and have influenced many subsequent systems - arguably including CockroachDB.
- marcosdumay 4y ago
  Heat caused failures also increase with load, as does electromigration, disk stress, power failures... The GP has a very digital model of computers. It's not really useful for predicting failure.
zamadatix 4y ago
One concrete example involving DBs is the PostgreSQL TXID wraparound. It's basically 32 bit counter where if things go wrong or were built wrong it just keeps going up until it becomes a problem. If you're only doing <=1 request per second this won't ever be a problem since even if you ran it your whole life without freezing any old records it wouldn't approach max value. On the other hand if it's under a constant 1000 request per second load it can be the source of problems in about a month. You could argue to just call the TXID "complexity" but that'd come with a wide enough definition of what complexity is that it just becomes a replacement for saying "buggy code" which isn't very useful. Most leave complexity as being more like "grossly overengineered" to leave room to avoid that.

notacoward 4y ago

I was very unsurprised to see that this history starts a bit late. In 1992 I was working on HACMP/6000, which was a high-availability clustering product for IBM's AIX (though I worked at a company that was on contract to them). Databases were a primary use case, relying on shared storage instead of replication. We were hardly the first into that space BTW. DEC in particular had developed a lot of the concepts (e.g. locking models) that we and others at the time emulated, and the history in this space goes back even further than them. Were those systems as scalable as what we have now, or even as robust? Nowhere even close. But the erasure of that history still irks me. Modern distributed databases (and distributed systems in general) in their turn relied heavily on experience and concepts from those earlier systems, but rarely acknowledge the fact.

macintux 4y ago
Tandem started selling their NonStop systems in the mid-70s, so clearly high availability was important. At Basho we worked hard to acknowledge our debt to distributed systems pioneers. Not everyone is ignorant of that history.
- notacoward 4y ago
  Yup, the Basho folks were/are very good about that. BTW hi, we've quite likely met, though I don't know which Basho alum you are. ;) And yes, even I was a pretty late entrant to the HA game. I never worked on Tandem systems, but I worked with Tandem people at the job I mentioned, after having worked with both Stratus and Sequoia systems previously. Those were all FT rather than HA, but the point remains that work on making systems survive failures through redundancy goes back much further than the OP mentions.
  - macintux 4y ago
    Oh, definitely, was trying to add to rather than detract from your point. I was mainly on the periphery at Basho, started as tech evangelist, migrated to engineering to help get 2.0 out the door. John Daily.
- 4y ago

throwaway787544 4y ago

Fun facts about availability: 1) the more nodes, the higher probability of failure, 2) the more load, the higher probability of failure, 3) complex logic and/or nodes that aren't tested frequently have a higher probability of failure Basically, every person who has claimed "X is easy, I do it for my personal use and it's fine with one giant node, don't need HA" is true (for them) because it's literally less likely to fail. But if they had more load they'd see more failures, and probability doesn't mean they won't encounter the random failures that are much harder to recover when there's no backup system. HA means you're going to have more failure, which means more work to keep it running. But it's necessary work if total system failure is unacceptable. Two engines in an airplane is twice the maintenance, but it's better than the only engine conking out mid-flight.

rospaya 4y ago
> 1) the more nodes, the higher probability of failure I may be misreading your comment, but how is this true?
- lhoff 4y ago
  If you have one node with the probability of x% that it fails within a year and you add a second one with the same probability to fail the total pobability that one of the two nodes fail within a year goes up.
  - deleted 4y ago
    [deleted]
- macintux 4y ago
  Not OP, but two things I’ve found to be true: * the more complex the configuration required to provide high availability, the more likely something will go wrong and reduce availability * the more nodes you have, the more likely one of them will fail. Whether that reduces availability typically depends on the failure mode and control layer. For an example of the latter, with the distributed database Riak (and I imagine others as well) if a server is starting to fail (corrupt disk, bad network card, etc) you’re in a much better place if it stops working entirely. A gradual node failure is much much worse.