5 ms·

Hmm, I think potted history is more correct... There is no mention hardware here. HP, Compaq, IBM and Sun all produced hardware with HA ability. Meaning that n

Hmm, I think potted history is more correct... There is no mention hardware here. HP, Compaq, IBM and Sun all produced hardware with HA ability. Meaning that normal software could be run on two or more nodes, and should one break, it'd fall over with no loss of data, or outages. Here is a (contrived) video where they literally blow up a server stack: https://www.youtube.com/watch?v=qMCHpUtJnEI https://www.youtube.com/watch?v=qMCHpUtJnEI You can do this with VMware, and if AWS could see a way to make money, they'd do it to. (For a while AWS's platform was not capable, but that was 7+ years ago. its much more lucrative to make people build in HA and redundancy into software, as its requires more services to run)

arethuza 8y ago
IBM has done a huge amount in this area e.g. https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex
mattzito 8y ago
Hey, for once my former life designing HA infrastructure is actually useful! The reality is that hardware HA was almost always terrible. Of the platforms you described: - Sun had no hardware HA ever, down to the unfathomable design that all of their E-class machines had only one power cord. They had SOME hotswap hardware, but the rules were byzantine - it couldn't be the first processor board, and it couldn't be the last processor board, oh and by the way, if a processor or RAM went bad, the machine would crash, but when it came back up it would take the bad processor offline and if you were lucky enough the bad processor wasnt on the first or last board and then you could hotswap it. - The only HA hardware from IBM was the mainframe, and even there, it was definitely possible for a software fault to take down the entire thing. The P-series boxes had lots and lots of fancy sounding HA capabilities, but they would only certify a configuration as fault-tolerant if you bought two of them and clustered them with HA/CMP (i.e. software HA). - Compaq had the NonStop servers, based on the Tandem acquisition. As the downthread comment correctly pointed out, there was a ton of hardware redundancy in that platform, but I think it ran a proprietary OS. They also had their OpenVMS clustering, which offered amazing HA - but it was all delivered in software, and your app had to be either a) stateless or b) cluster-aware - HP bought compaq, but the HP-UX machines relied on software for their clustering. This all dates back to when people thought servers were special in some way, and needed to justify their insane price points with all sorts of fancy marketing features. Some were useful (I remember an early IBM linux box with RAM mirroring that actually kind of worked), but in the end, a Veritas cluster of decoupled nodes almost always worked better, more reliably, and faster than any hardware nonsense. EDIT: fixed my wrong assertion about Compaq tech.
- tyingq 8y ago
  "Compaq had the NonStop servers, which ran OpenVMS...but it was all delivered in software" NonStop/Guardian had nothing to do with OpenVMS. And the hardware was certainly purpose built for HA, including redundant, lockstepped CPUs, disk controllers, etc.
  - mattzito 8y ago
    You are 100% correct - in my haste, I conflated the old Tandem hardware line with the OpenVMS clustering platform. My mistake, I’ll see if I can edit my post.
- icedchai 8y ago
  I'm pretty sure I had an E3500, back in the late 90's, that had two power supplies. I may be misremembering... No, you are correct. That second power supply was apparently for "peripherals only", not the main CPUs... http://www.tech-specifications.com/SunServers/E3500.html http://www.tech-specifications.com/SunServers/E3500.html !
- rstupek 8y ago
  Compaq ServerPro had HA ability with a hot spare server connected to the primary with automatic failover well before the Tandem acquisition.
- SomeHacker44 8y ago
  What about true HA hardware, for example like Tandem, or the systems on the Space Shuttle, etc.? A discussion of those systems from the 70s, 80s and 90s seems appropriate (but I am not knowledgeable enough to start it).
  - dfox 8y ago
    The hardware failures that hardware-based HA systems solve without software assistance are extremely unlikely on any modern platform (eg. CPU failure). In this regard it is somewhat remarkable, that original Pentium supports glueless(!) high-reliability redundant CPU mode (2 cores execute same instruction stream and you get NMI when the results diverge). It is somewhat telling that probably world's first 2 votes out of 3 HA implemented in hardware computer (SAPO, first Czech computer) was motivated by the fact that used CPU implementation technology (mechanical relays) was highly unreliable. Both Tandem and Space Shuttle Flight Computer relied highly on software support for HA. And it is my experience that most approaches to bolting on redundacy on system that is originally designed to run on single node are actually detrimental to the resulting availability. Either because it introduces additional failure modes that have to do with the fail-over mechanism itself (too eager failover, various byzantine generals failures of the control layer...) or because it depends on global consistency assumptions that are not in fact met (case in point: one high-reliability soft-realtime system with redundant application CPUs I was involved with is built with the assumption that you can reboot the application CPU at any time and it will catch up with system state as long as at least one other CPU is still running correctly. This ends up to not be a case as there is initialization phase that has to happen for the CPU to know the overall system state which interferes with the overall system operation as it involves injecting test patterns into global system inputs.)
skyde 8y ago
I could not find anywhere an explanation of what HP nonstop is actually doing. The fact you say its like VMWare make me think you mean VMware vLockstep. Documentation say : "While both the primary and secondary VMs receive the same inputs, only the primary VM produces output such as disk writes and network transmits." The part I don't understand is how can the primary wait for the secondary to execute the same cpu instruction before producing output? this would be extremely slow even with very good network latency. Also it seem in vsphere6 VMware have rewritten how FT is handled, dropping the old “lock-step” approach in favour of a new “fast check-pointing” type method. If the memory content on the secondary is not exactly the same as what was on the primary before the crash. Then there is some kind of small dataloss. This is the kind of thing I assumed could only be solved by having the primary run a consensus protocol like Raft or Paxos for each network request it receive before producing response to the request.

arethuza 8y ago

IBM has done a huge amount in this area e.g. https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex

mattzito 8y ago

Hey, for once my former life designing HA infrastructure is actually useful! The reality is that hardware HA was almost always terrible. Of the platforms you described: - Sun had no hardware HA ever, down to the unfathomable design that all of their E-class machines had only one power cord. They had SOME hotswap hardware, but the rules were byzantine - it couldn't be the first processor board, and it couldn't be the last processor board, oh and by the way, if a processor or RAM went bad, the machine would crash, but when it came back up it would take the bad processor offline and if you were lucky enough the bad processor wasnt on the first or last board and then you could hotswap it. - The only HA hardware from IBM was the mainframe, and even there, it was definitely possible for a software fault to take down the entire thing. The P-series boxes had lots and lots of fancy sounding HA capabilities, but they would only certify a configuration as fault-tolerant if you bought two of them and clustered them with HA/CMP (i.e. software HA). - Compaq had the NonStop servers, based on the Tandem acquisition. As the downthread comment correctly pointed out, there was a ton of hardware redundancy in that platform, but I think it ran a proprietary OS. They also had their OpenVMS clustering, which offered amazing HA - but it was all delivered in software, and your app had to be either a) stateless or b) cluster-aware - HP bought compaq, but the HP-UX machines relied on software for their clustering. This all dates back to when people thought servers were special in some way, and needed to justify their insane price points with all sorts of fancy marketing features. Some were useful (I remember an early IBM linux box with RAM mirroring that actually kind of worked), but in the end, a Veritas cluster of decoupled nodes almost always worked better, more reliably, and faster than any hardware nonsense. EDIT: fixed my wrong assertion about Compaq tech.

tyingq 8y ago
"Compaq had the NonStop servers, which ran OpenVMS...but it was all delivered in software" NonStop/Guardian had nothing to do with OpenVMS. And the hardware was certainly purpose built for HA, including redundant, lockstepped CPUs, disk controllers, etc.
- mattzito 8y ago
  You are 100% correct - in my haste, I conflated the old Tandem hardware line with the OpenVMS clustering platform. My mistake, I’ll see if I can edit my post.
icedchai 8y ago
I'm pretty sure I had an E3500, back in the late 90's, that had two power supplies. I may be misremembering... No, you are correct. That second power supply was apparently for "peripherals only", not the main CPUs... http://www.tech-specifications.com/SunServers/E3500.html http://www.tech-specifications.com/SunServers/E3500.html !
rstupek 8y ago
Compaq ServerPro had HA ability with a hot spare server connected to the primary with automatic failover well before the Tandem acquisition.
SomeHacker44 8y ago
What about true HA hardware, for example like Tandem, or the systems on the Space Shuttle, etc.? A discussion of those systems from the 70s, 80s and 90s seems appropriate (but I am not knowledgeable enough to start it).
- dfox 8y ago
  The hardware failures that hardware-based HA systems solve without software assistance are extremely unlikely on any modern platform (eg. CPU failure). In this regard it is somewhat remarkable, that original Pentium supports glueless(!) high-reliability redundant CPU mode (2 cores execute same instruction stream and you get NMI when the results diverge). It is somewhat telling that probably world's first 2 votes out of 3 HA implemented in hardware computer (SAPO, first Czech computer) was motivated by the fact that used CPU implementation technology (mechanical relays) was highly unreliable. Both Tandem and Space Shuttle Flight Computer relied highly on software support for HA. And it is my experience that most approaches to bolting on redundacy on system that is originally designed to run on single node are actually detrimental to the resulting availability. Either because it introduces additional failure modes that have to do with the fail-over mechanism itself (too eager failover, various byzantine generals failures of the control layer...) or because it depends on global consistency assumptions that are not in fact met (case in point: one high-reliability soft-realtime system with redundant application CPUs I was involved with is built with the assumption that you can reboot the application CPU at any time and it will catch up with system state as long as at least one other CPU is still running correctly. This ends up to not be a case as there is initialization phase that has to happen for the CPU to know the overall system state which interferes with the overall system operation as it involves injecting test patterns into global system inputs.)

skyde 8y ago

I could not find anywhere an explanation of what HP nonstop is actually doing. The fact you say its like VMWare make me think you mean VMware vLockstep. Documentation say : "While both the primary and secondary VMs receive the same inputs, only the primary VM produces output such as disk writes and network transmits." The part I don't understand is how can the primary wait for the secondary to execute the same cpu instruction before producing output? this would be extremely slow even with very good network latency. Also it seem in vsphere6 VMware have rewritten how FT is handled, dropping the old “lock-step” approach in favour of a new “fast check-pointing” type method. If the memory content on the secondary is not exactly the same as what was on the primary before the crash. Then there is some kind of small dataloss. This is the kind of thing I assumed could only be solved by having the primary run a consensus protocol like Raft or Paxos for each network request it receive before producing response to the request.