8 ms·

Does someone know why these systems - Stratus, Tandem/HP NonStop, etc. - appear to be relics of the past? I can see how Google is better off spending money on

Does someone know why these systems - Stratus, Tandem/HP NonStop, etc. - appear to be relics of the past? I can see how Google is better off spending money on software engineers than on hardware (thus, fault-tolerant distributed systems); but there's lots of systems that don't need Google-scale computer power, nor Google-style ultra-low-cost-per-operation hardware, but that really need to stay up - utilities, banks, every system at the center of a big enterprise. Is the issue that modern software just isn't reliable enough to make hardware failures an important part of the downtime? Or is it just that systems like https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013428 https://kb.vmware.com/selfservice/microsites/search.do?langu... are cheaper today? (I guess those actually work, right? The level of complexity involved is frightening...)

brianwawok 10y ago
Worked at a big co switching off tandem. Went from giant tandem machine that cost 2 million dollars per year to rent to about 6 off the shelf Dell servers for 6k each one time fee. Performance at the end was about 3x the old system. Was a glorified billing system. If one node went down, who cares, the other 5 did the job. Most people don't value 99.999 vs 99.9999 reliability as worth millions of dollars per system per year. Space shuttles may disagree, but not billing platforms.
- z3t4 10y ago
  Five way redundancy look good on paper, put if you use the same vendor and hardware purchased at the same time, you have a one point of failure, aka when one server fails there's a high probability that the others will fail simultaneously.
  - davnn 10y ago
    Why?
    - smallnamespace 10y ago
      If the hardware failure is due to a manufacturing problem, then it affects every machine. If it's problem with the production batch, then again, unless you bought from different batches, you haven't reduced risk much. And if it's a software problem, then again the redundancy isn't helping you. Basically you haven't succeeded in getting 5 independent tries in the problem -- the failures are highly correlated. It's the same reasoning that led to the financial crisis -- you have a CDO or an MBS that has a bunch of different obligations so you think you've diversified away risk, but in actuality there is one big factor that causes everything to fail at once.
      - throwawayish 10y ago
        Also - same shipping way. If I buy four disks from the same shop, then those four disks came in the same delivery to the shop (so they were all dropped at the same time), are handled by the same picker, thrown into the same parcel and dropped a couple times the same way.
  - eeZah7Ux 10y ago
    By running statistics on a large enough infrastructure you can see how frequently 2 hard drives in the same raid fail within days, or even hours! Same brand, same batch, operating time, running temperature, even same vibration does trick. :(
    - chiph 10y ago
      We had 3 hard drives (the legendary IBM DeathStars) from the same production batch fail within hours of each other. Part of me was like "Yeah! Awesome quality control!" and a larger part was like "Oh shit." We ended up losing the array and had to restore from backup.
  - sllabres 10y ago
    It depends on what you are trying to protect and what options you have if one component fails (replacement). For highly critical systems that is an issue but it is being adressed. Take as example the space shuttle or the Airbus (3/2) primary/secondary flight computer [1] where the backup is being build by different companies with different processors ... [1] https://ifs.host.cs.st-andrews.ac.uk/Resources/CaseStudies/Airbus/Airbus-fcs.pdf https://ifs.host.cs.st-andrews.ac.uk/Resources/CaseStudies/A...
- trothamel 10y ago
  Note that the space shuttle didn't use processors running in lockstep, like the Tandem machines do (from what I understand). It used 5 single-core computers. Four of them would run the primary software, with a single control channel controlled by each of the four. For something like the elevons, control was accomplished by connecting 3 actuators to 3 of the channels, with voting being accomplished by physical force - if the 3 computers disagreed, the 2 would overpower the one. (Things like thrusters used electronic voting, close to the thruster itself.) This seems closer to modern architectures with multiple computers, than the mainframe idea of redundant hardware.
  - Trombone12 10y ago
    Note that all those computers shared the same bus, and so that the complete system wasn't as redundant as hoped as this story[1] retells: "At 12:12 GMT 13 May 2008, a space shuttle was loading its hypergolic fuel for mission STS-124 when a 3-1 disagreement occurred among its General Purpose Computers (GPC 4 disagreed with the other GPCs). Three seconds later, the split became 2-1-1 (GPC 2 now disagreed with GPC 4 and the other two GPCs). This required that the launch countdown be stopped. During the subsequent troubleshooting, the remaining two GPCs disagreed (1-1-1-1 split). This was a complete system disagreement. However, none of the GPCs were faulty. The fault was in the FA 2 Multiplexer Demultiplexer. This fault was a crack in a diode. This crack was perpendicular to the normal current flow and completely through the current path. As a crack opened up, it changed the diode into another type of component ... a capacitor. Because some of the bits in the signal are smaller than they should have been, some of the GPC receivers could not see these bits. The ability to see these bits depends on the sensitivity of the receiver, which is a function of manufacturing variances, temperature, and its power supply voltage. From the symptoms, it is apparent that the receiver in GPC 4 was the least sensitive and saw the errors before the other three GPC. This caused GPC 4 to disagree with the other three. Then, as the crack in the diode widened, the bits became shorter to the point where GPC 2 could no longer see these bits; which caused it to disagree with the other GPC. At this point, the set of messages that was received correctly by GPC 4 was different from the set of messages that was correctly received by GPC 2 which was different again from the set of messages that was correctly received by GPC 1 and GPC 3. This process continued until GPC 1 and GPC 3 also disagreed with all the other GPC." [1] Adapted from https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_split/ https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_s...
- newman314 10y ago
  So.... Years ago, I had a customer that had a Tandem which was quite exciting to me as I had yet to encounter one up to that point of my career. Imagine my letdown when I eventually discovered that the Tandem was used for FTP. ¯\_(ツ)_/¯
yellowapple 10y ago
Given the existence of things like Erlang/OTP, I doubt software is the big issue here. Rather, the issue is that individual machines are cheap enough that you can have machine-level redundancy at a much lower cost than trying to engineer a single machine with redundant components. The issue with machine-level redundancy is that they tend to run their own isolated operating systems (which is great from a fault-tolerance perspective, but is rather wasteful from a computational perspective). Operating systems like Plan 9 were billed to help bridge this gap and make clusters of machines communicating over 9P feel like a single unified whole, but they never seemed to catch on (besides maybe the concept of a Beowulf cluster).
VLM 10y ago
In the old days 100% of computers were doing civilization critical work, Air traffic control, Social Security checks, nuclear reactor monitoring. Computers are expensive and automation provides the most gain in those applications. There just aren't that many civilization critical systems. In the modern era we continue to have the same small number of civilization critical systems, we still have exactly one air traffic control system. However we have billions of, well, filler. Ringtones and social media and spam. Therefore rounded down 0% of modern computers are doing civilization critical work and safe secure systems superficially appear to be a thing of the past even if the number of systems and employees remain constant. In 1960 the average ALU was calculating your paycheck and it was rather important to get that done on time and correctly. In 2010 the average ALU is a countdown timer in your microwave oven, its your digital thermostat, its my cars transmission processor, maybe its video game system or other human interface device (desktop, laptop, phone) but probably numerically unlikely. Things are sloppier now because they're sloppy-tolerant applications.
- matt_wulfeck 10y ago
  Great explanation. In other words the median number of computers performing some kind of critical task has probably gone from 90% somewhere closer to 0.
  - zhte415 10y ago
    In absolute numbers they're still there. Air traffic control and baggage sorting applications still run APL, and your bank still runs COBOL. It has not gone away.
    - koja86 10y ago
      While hearing about COBOL in banks quite often all my friends actually working there say Java and Oracle and all related job adverts I have seen so far were Java or C++. Could someone please provide some relevant facts?
      - 10y ago
Elv13 10y ago
This issue has been fixed for good. If you need uptime, then virtualization is good enough. You can migrate VMs to other hosts on moment notice. As long as the failure isn't sudden, then you probably have enough time to migrate the VM elsewhere. Good servers, workstations and UPS have enough monitoring to warn you.
suprjami 10y ago
Modern software hasn't had the capability until recently. These Stratus VOS systems get updates by direct assembly patching over the running kernel. Linux ksplice and kpatch are relatively new, and AFAIK only Oracle support their use in production over their own UEK Linux Kernel. Source: I work with several ex-Stratus employees who have told me some of their war stories.
- sllabres 10y ago
  Redhat too, since 7.2 on x64 (with some support requirements). No experiences though.
zzzcpan 10y ago
> Is the issue that modern software just isn't reliable enough to make hardware failures an important part of the downtime? I think the issue is more about networks never being reliable enough for such reliable computers to ever make much sense. A data center simply cannot give you five nines availability over the internet, so it doesn't matter how reliable everything inside is, you would still need geo redundancy and all that fault-tolerance in a distributed system.
ChuckMcM 10y ago
They aren't, as VLM points out they just aren't something you see a lot of quantity of. For some applications you can achieve 100% uptime with networks and clusters, for others you still use doubly or triply redundant processor networks and various manufacturers have specialty chips for those markets (generally Health, Life, Safety (HLS) type systems). Sometimes people developed redundant but not non-stop systems. I talked with an VP of Citibank when working at NetApp and they had a number of systems which ran on schedules of alternate days, so one would process transaction records for a while, then another would take over and repeat. They had three identical systems where one was eseentially a hot standby for the other two, and new versions of code would be deployed on one which would run the same transaction records and they would check for the same output, so they could do a 'walking' upgrade of software. Back when Tandem's and big Sun iron ruled the roost those machines were too expensive to have an extra one which was essentially a spare. These days however its much more economical to do that.
tyingq 10y ago
I would think it's largely because the cost difference between a normal server and something like a Stratus grew significantly larger over time. You could more easily justify the cost when the upcharge wasn't so huge. And, of course, we all got better at reliable distributed systems with commodity servers. So aside from the cost gap, the uptime gap was closing as well.
- ethbro 10y ago
  I'd tune that a bit and guess that the price of non-Stratus hardware fell through the floor. In the mid-90s, data centers full of commodity hardware would have been cost prohibitive.
jlarocco 10y ago
They're not cost effective. It's possible to get close enough with off the shelf hardware and good system design, so nobody wants to pay 5-10x more for special hardware.
technion 10y ago
For my part (sample size of one means nothing of course)... The stratus server I worked with in around 2002 was, I believe, one of the first to move towards standard x86 configurations and Windows OS. They did so because, no matter how reliable your server, executives were jumping on Windows as an application server bandwagon and developers were targeting it. There were several driver and firmware bugs that would BSOD the whole server, and bring the entire fault tolerant OS down. Moreover, storage was a hack job. They couldn't make hardware RAID cards work in that configuration, and Windows software RAID was a joke, so you were stuck with Veritas storage software. Twice we applied Windows Service Packs only to find it would BSOD due to a Veritas software incompatibility. If it was a more average server I'd apply said update to a test server first, but no one could afford a test Stratus so you were always testing in production, which doesn't help reliability. Here's a picture of it. It became my homeserver for a few years but frankly it was less reliable than a desktop. http://imgur.com/a/7010z http://imgur.com/a/7010z
youdontknowtho 10y ago
"every system at the center of a big enterprise"...that's the thing. A lot of enterprises actually don't need this kind of uptime or can get by with other types of redundancy. Your comment about software, though, is a good point. The last relatively common OS that has that kind of uptime is OpenVMS.
youdontknowtho 10y ago
Oh yeah, Mainframes are still built this way.
rwallace 10y ago
One concern is that there is a limit to how reliable a single box can really be, no matter what sort of engineering you throw at it. Consider http://thedailywtf.com/articles/Designed-For-Reliability http://thedailywtf.com/articles/Designed-For-Reliability - because the box was designed to never ever go down, when it did go down, not only was there no failover, but it took twenty-four hours to reboot.

brianwawok 10y ago

Worked at a big co switching off tandem. Went from giant tandem machine that cost 2 million dollars per year to rent to about 6 off the shelf Dell servers for 6k each one time fee. Performance at the end was about 3x the old system. Was a glorified billing system. If one node went down, who cares, the other 5 did the job. Most people don't value 99.999 vs 99.9999 reliability as worth millions of dollars per system per year. Space shuttles may disagree, but not billing platforms.

z3t4 10y ago
Five way redundancy look good on paper, put if you use the same vendor and hardware purchased at the same time, you have a one point of failure, aka when one server fails there's a high probability that the others will fail simultaneously.
- davnn 10y ago
  Why?
  - smallnamespace 10y ago
    If the hardware failure is due to a manufacturing problem, then it affects every machine. If it's problem with the production batch, then again, unless you bought from different batches, you haven't reduced risk much. And if it's a software problem, then again the redundancy isn't helping you. Basically you haven't succeeded in getting 5 independent tries in the problem -- the failures are highly correlated. It's the same reasoning that led to the financial crisis -- you have a CDO or an MBS that has a bunch of different obligations so you think you've diversified away risk, but in actuality there is one big factor that causes everything to fail at once.
    - throwawayish 10y ago
      Also - same shipping way. If I buy four disks from the same shop, then those four disks came in the same delivery to the shop (so they were all dropped at the same time), are handled by the same picker, thrown into the same parcel and dropped a couple times the same way.
- eeZah7Ux 10y ago
  By running statistics on a large enough infrastructure you can see how frequently 2 hard drives in the same raid fail within days, or even hours! Same brand, same batch, operating time, running temperature, even same vibration does trick. :(
  - chiph 10y ago
    We had 3 hard drives (the legendary IBM DeathStars) from the same production batch fail within hours of each other. Part of me was like "Yeah! Awesome quality control!" and a larger part was like "Oh shit." We ended up losing the array and had to restore from backup.
- sllabres 10y ago
  It depends on what you are trying to protect and what options you have if one component fails (replacement). For highly critical systems that is an issue but it is being adressed. Take as example the space shuttle or the Airbus (3/2) primary/secondary flight computer [1] where the backup is being build by different companies with different processors ... [1] https://ifs.host.cs.st-andrews.ac.uk/Resources/CaseStudies/Airbus/Airbus-fcs.pdf https://ifs.host.cs.st-andrews.ac.uk/Resources/CaseStudies/A...
trothamel 10y ago
Note that the space shuttle didn't use processors running in lockstep, like the Tandem machines do (from what I understand). It used 5 single-core computers. Four of them would run the primary software, with a single control channel controlled by each of the four. For something like the elevons, control was accomplished by connecting 3 actuators to 3 of the channels, with voting being accomplished by physical force - if the 3 computers disagreed, the 2 would overpower the one. (Things like thrusters used electronic voting, close to the thruster itself.) This seems closer to modern architectures with multiple computers, than the mainframe idea of redundant hardware.
- Trombone12 10y ago
  Note that all those computers shared the same bus, and so that the complete system wasn't as redundant as hoped as this story[1] retells: "At 12:12 GMT 13 May 2008, a space shuttle was loading its hypergolic fuel for mission STS-124 when a 3-1 disagreement occurred among its General Purpose Computers (GPC 4 disagreed with the other GPCs). Three seconds later, the split became 2-1-1 (GPC 2 now disagreed with GPC 4 and the other two GPCs). This required that the launch countdown be stopped. During the subsequent troubleshooting, the remaining two GPCs disagreed (1-1-1-1 split). This was a complete system disagreement. However, none of the GPCs were faulty. The fault was in the FA 2 Multiplexer Demultiplexer. This fault was a crack in a diode. This crack was perpendicular to the normal current flow and completely through the current path. As a crack opened up, it changed the diode into another type of component ... a capacitor. Because some of the bits in the signal are smaller than they should have been, some of the GPC receivers could not see these bits. The ability to see these bits depends on the sensitivity of the receiver, which is a function of manufacturing variances, temperature, and its power supply voltage. From the symptoms, it is apparent that the receiver in GPC 4 was the least sensitive and saw the errors before the other three GPC. This caused GPC 4 to disagree with the other three. Then, as the crack in the diode widened, the bits became shorter to the point where GPC 2 could no longer see these bits; which caused it to disagree with the other GPC. At this point, the set of messages that was received correctly by GPC 4 was different from the set of messages that was correctly received by GPC 2 which was different again from the set of messages that was correctly received by GPC 1 and GPC 3. This process continued until GPC 1 and GPC 3 also disagreed with all the other GPC." [1] Adapted from https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_split/ https://c3.nasa.gov/dashlink/projects/79/wiki/test_stories_s...
newman314 10y ago
So.... Years ago, I had a customer that had a Tandem which was quite exciting to me as I had yet to encounter one up to that point of my career. Imagine my letdown when I eventually discovered that the Tandem was used for FTP. ¯\_(ツ)_/¯

yellowapple 10y ago

Given the existence of things like Erlang/OTP, I doubt software is the big issue here. Rather, the issue is that individual machines are cheap enough that you can have machine-level redundancy at a much lower cost than trying to engineer a single machine with redundant components. The issue with machine-level redundancy is that they tend to run their own isolated operating systems (which is great from a fault-tolerance perspective, but is rather wasteful from a computational perspective). Operating systems like Plan 9 were billed to help bridge this gap and make clusters of machines communicating over 9P feel like a single unified whole, but they never seemed to catch on (besides maybe the concept of a Beowulf cluster).

VLM 10y ago

In the old days 100% of computers were doing civilization critical work, Air traffic control, Social Security checks, nuclear reactor monitoring. Computers are expensive and automation provides the most gain in those applications. There just aren't that many civilization critical systems. In the modern era we continue to have the same small number of civilization critical systems, we still have exactly one air traffic control system. However we have billions of, well, filler. Ringtones and social media and spam. Therefore rounded down 0% of modern computers are doing civilization critical work and safe secure systems superficially appear to be a thing of the past even if the number of systems and employees remain constant. In 1960 the average ALU was calculating your paycheck and it was rather important to get that done on time and correctly. In 2010 the average ALU is a countdown timer in your microwave oven, its your digital thermostat, its my cars transmission processor, maybe its video game system or other human interface device (desktop, laptop, phone) but probably numerically unlikely. Things are sloppier now because they're sloppy-tolerant applications.

matt_wulfeck 10y ago
Great explanation. In other words the median number of computers performing some kind of critical task has probably gone from 90% somewhere closer to 0.
- zhte415 10y ago
  In absolute numbers they're still there. Air traffic control and baggage sorting applications still run APL, and your bank still runs COBOL. It has not gone away.
  - koja86 10y ago
    While hearing about COBOL in banks quite often all my friends actually working there say Java and Oracle and all related job adverts I have seen so far were Java or C++. Could someone please provide some relevant facts?
    - 10y ago

Elv13 10y ago

This issue has been fixed for good. If you need uptime, then virtualization is good enough. You can migrate VMs to other hosts on moment notice. As long as the failure isn't sudden, then you probably have enough time to migrate the VM elsewhere. Good servers, workstations and UPS have enough monitoring to warn you.

suprjami 10y ago

Modern software hasn't had the capability until recently. These Stratus VOS systems get updates by direct assembly patching over the running kernel. Linux ksplice and kpatch are relatively new, and AFAIK only Oracle support their use in production over their own UEK Linux Kernel. Source: I work with several ex-Stratus employees who have told me some of their war stories.

sllabres 10y ago
Redhat too, since 7.2 on x64 (with some support requirements). No experiences though.

zzzcpan 10y ago

> Is the issue that modern software just isn't reliable enough to make hardware failures an important part of the downtime? I think the issue is more about networks never being reliable enough for such reliable computers to ever make much sense. A data center simply cannot give you five nines availability over the internet, so it doesn't matter how reliable everything inside is, you would still need geo redundancy and all that fault-tolerance in a distributed system.

ChuckMcM 10y ago

They aren't, as VLM points out they just aren't something you see a lot of quantity of. For some applications you can achieve 100% uptime with networks and clusters, for others you still use doubly or triply redundant processor networks and various manufacturers have specialty chips for those markets (generally Health, Life, Safety (HLS) type systems). Sometimes people developed redundant but not non-stop systems. I talked with an VP of Citibank when working at NetApp and they had a number of systems which ran on schedules of alternate days, so one would process transaction records for a while, then another would take over and repeat. They had three identical systems where one was eseentially a hot standby for the other two, and new versions of code would be deployed on one which would run the same transaction records and they would check for the same output, so they could do a 'walking' upgrade of software. Back when Tandem's and big Sun iron ruled the roost those machines were too expensive to have an extra one which was essentially a spare. These days however its much more economical to do that.

tyingq 10y ago

I would think it's largely because the cost difference between a normal server and something like a Stratus grew significantly larger over time. You could more easily justify the cost when the upcharge wasn't so huge. And, of course, we all got better at reliable distributed systems with commodity servers. So aside from the cost gap, the uptime gap was closing as well.

ethbro 10y ago
I'd tune that a bit and guess that the price of non-Stratus hardware fell through the floor. In the mid-90s, data centers full of commodity hardware would have been cost prohibitive.

jlarocco 10y ago

They're not cost effective. It's possible to get close enough with off the shelf hardware and good system design, so nobody wants to pay 5-10x more for special hardware.

technion 10y ago

For my part (sample size of one means nothing of course)... The stratus server I worked with in around 2002 was, I believe, one of the first to move towards standard x86 configurations and Windows OS. They did so because, no matter how reliable your server, executives were jumping on Windows as an application server bandwagon and developers were targeting it. There were several driver and firmware bugs that would BSOD the whole server, and bring the entire fault tolerant OS down. Moreover, storage was a hack job. They couldn't make hardware RAID cards work in that configuration, and Windows software RAID was a joke, so you were stuck with Veritas storage software. Twice we applied Windows Service Packs only to find it would BSOD due to a Veritas software incompatibility. If it was a more average server I'd apply said update to a test server first, but no one could afford a test Stratus so you were always testing in production, which doesn't help reliability. Here's a picture of it. It became my homeserver for a few years but frankly it was less reliable than a desktop. http://imgur.com/a/7010z http://imgur.com/a/7010z

youdontknowtho 10y ago

"every system at the center of a big enterprise"...that's the thing. A lot of enterprises actually don't need this kind of uptime or can get by with other types of redundancy. Your comment about software, though, is a good point. The last relatively common OS that has that kind of uptime is OpenVMS.

youdontknowtho 10y ago

Oh yeah, Mainframes are still built this way.

rwallace 10y ago

One concern is that there is a limit to how reliable a single box can really be, no matter what sort of engineering you throw at it. Consider http://thedailywtf.com/articles/Designed-For-Reliability http://thedailywtf.com/articles/Designed-For-Reliability - because the box was designed to never ever go down, when it did go down, not only was there no failover, but it took twenty-four hours to reboot.