25 ms·

Stratus: Servers that won’t quit – The 24 year running computer

219 points by protomyth 10y ago

lallysingh 10y ago
Note: clocked-down (from 33Mhz) Intel RISC CPUs. It's reliable, sure, but at what cost / MIPS? Power, maintenance, finding spares, opportunity cost from adding new functionality, etc. Also, how does this compare to the oldest running satellites?
ftio 10y ago
Sounds like a modern day Ship of Theseus (https://en.wikipedia.org/wiki/Ship_of_Theseus https://en.wikipedia.org/wiki/Ship_of_Theseus). I wonder how many parts have been untouched since they were installed on Day 1.
- akeck 10y ago
  My favourite modern day example: http://www.smithsonianmag.com/smart-news/this-japanese-shrine-has-been-torn-down-and-rebuilt-every-20-years-for-the-past-millennium-575558/ http://www.smithsonianmag.com/smart-news/this-japanese-shrin...
- __s 10y ago
  Key differences being that this is more if the Ship of Theseus was going through the replacemnets, all the while remaining out at sea
  - gaius 10y ago
    Trigger's Broom was in active use.
- eps 10y ago
  80% of it is original [1] http://www.computerworld.com/article/3162416/data-center/booted-up-in-1993-this-server-still-runs-but-not-for-much-longer.html http://www.computerworld.com/article/3162416/data-center/boo...
jcoffland 10y ago
I'd be more interested in the computer system with the longest uptime. I've had Debian systems that have achieved over 1 year uptimes. Could have gone longer but kernel upgrades took precedence over uptime.
- nickpsecurity 10y ago
  OpenVMS or AS/400. Very common for admins to report systems running for 5+ years without crashes while doing real work. People occasionally forget where they're at and have to go looking for them. My company has an AS/400 employees never saw go down or get maintenance over 10 years. It's in use 24/7. They might have some fail-over setup or upgrade when nobody is using it due to traffic patterns. We've just never seen it fail. Their other benefit is they're very hands off with it nearly self-administering.
  - protomyth 10y ago
    Or that AS/400 calls the tech to fix it and s/he just does it while no one is around. Had a call once from IBM asking for directions since the AS/400 RAID cache battery needed replacing. No danger of losing data, the AS/400 was just running slower because it was bypassing the cache.
    - rashkov 10y ago
      So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.
      - jonknee 10y ago
        > that must cost a whole lot of money and is quite devious of IBM. You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.
nickpsecurity 10y ago
"The use of the cloud for server farms made of hundreds, thousands, and often more computers that are transparent to the user has achieved much the same goal, providing one’s connection to the cloud is also redundant. " This isn't true at all. The cloud offerings are more like traditional, high availability such as fail-over clusters. They might compete with VMS clusters by now with the right components. Maybe even exceed them except for longevity. Systems like NonStop and Stratus are fault-tolerant systems supposed to get five 9's availability with [hopefully] imperceptible moments of failure that [hopefully] recover immediately. I wouldn't trust a cloud platform for that. The latency alone would probably prevent the solution from matching a local, NonStop cluster.
__s 10y ago
Curious they have 2 pairs rather than a triplet
- VLM 10y ago
  One simple hardware mental model is running two processors in parallel with XORs on all pins with wired-OR outputs hooked up to the error-shutdown line. In practice that doesn't really work because irrelevant differences in PCB trace length and clock distribution mean both processors run just a little bit outta sync jitter so the XOR will be firing fairly constantly for a fraction of each clock cycle. You could play games with gating and sampling only when well settled.... A somewhat more realistic way to build it outta real hardware involves each processor runs every 1 outta X clock cycles. Reading between the lines of the story WRT "down clocked processors" I suspect this is what its doing. This also detects weird power spike problems where lightning miles away means every processor running is going to run 0xFFFF whatever opcode that happens to be, but a couple ns later the spike is gone and the other processors are back to normal. Or RFI/EMC, etc. This is all very nice other than a significant hit to performance if you run a large number of processors. Then again if your primary figure of merit for your system is extreme reliability and not raw MIPS... It would be interesting to hear if they run dual port RAM to get around the jitter XOR problem and have some alternative way to detect synchronous behavior. Dual port ram is weird but COTS. Triple port ram is not as COTS.
  - dfox 10y ago
    Comparing bus states of two CPU's running of same clock is perfectly feasible and there are no problems with glitches as long as the whole mechanism is synchronous (and the CPU is completely deterministic, which it should be, but then there are things like RNG-based cache eviction policies). In fact both original Pentium and some m68k processors have all the required hardware for this included and building error checking pair of them involves literally connecting all bus pins of two CPUs in parallel.
  - nickpsecurity 10y ago
    I read architectural details of NonStop and Stratus. Both were designed to compare I/O outputs instead of just catching everything at CPU level. The CPU-level redundancy is more for catching transient faults with fail-fast logic. Idea is problem shows up in obvious way by time it hits I/O buses. Past that I dont recall much.
JoachimSchipper 10y ago
Does someone know why these systems - Stratus, Tandem/HP NonStop, etc. - appear to be relics of the past? I can see how Google is better off spending money on software engineers than on hardware (thus, fault-tolerant distributed systems); but there's lots of systems that don't need Google-scale computer power, nor Google-style ultra-low-cost-per-operation hardware, but that really need to stay up - utilities, banks, every system at the center of a big enterprise. Is the issue that modern software just isn't reliable enough to make hardware failures an important part of the downtime? Or is it just that systems like https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013428 https://kb.vmware.com/selfservice/microsites/search.do?langu... are cheaper today? (I guess those actually work, right? The level of complexity involved is frightening...)
- brianwawok 10y ago
  Worked at a big co switching off tandem. Went from giant tandem machine that cost 2 million dollars per year to rent to about 6 off the shelf Dell servers for 6k each one time fee. Performance at the end was about 3x the old system. Was a glorified billing system. If one node went down, who cares, the other 5 did the job. Most people don't value 99.999 vs 99.9999 reliability as worth millions of dollars per system per year. Space shuttles may disagree, but not billing platforms.
  - z3t4 10y ago
    Five way redundancy look good on paper, put if you use the same vendor and hardware purchased at the same time, you have a one point of failure, aka when one server fails there's a high probability that the others will fail simultaneously.
    - davnn 10y ago
      Why?
      - smallnamespace 10y ago
nxc18 10y ago
I interned at a company that had given continuous service with a tandem/NonStop system since the 70s (80s latest) iirc. Its very cool to see such reliable systems in practice.
makethetick 10y ago
Stratus are still going (I'm assuming it's the same company, same product, same name) and making fault tolerant hardware like this. They also offer a lower end product based on two standard servers where all the redundancy is software based with virtual machines. It's clever kit and looks after a lot of critical infrastructure where downtime isn't an option.
- ghaff 10y ago
  It is although I believe all the FT is now done in software in some fashion. i.e. the CPU/Memory/Chipset/IO, etc. are standard components with the lockstep done through a software layer. The same is true of HP Integrity Nonstop AFAIK. I used to follow these when I was industry analyst but I've been away from this space for a few years.
  - makethetick 10y ago
    They still do both, here's their hardware offering which is hardware and software based: http://www.stratus.com/solutions/platforms/ftserver/ http://www.stratus.com/solutions/platforms/ftserver/ And this is there software based solution which runs on commodity hardware: http://www.stratus.com/solutions/software/everrun-enterprise-express/ http://www.stratus.com/solutions/software/everrun-enterprise... I only work with the latter but have heard the hardware offering is bulletproof.
eps 10y ago
mods, may want to merge this with [1] from yesterday. [1] https://news.ycombinator.com/item?id=13507972 https://news.ycombinator.com/item?id=13507972
myrandomcomment 10y ago
Quantum Link the online service for the Commodore 64 ran on this system. The first GUI online multiplayer world was on Q-link. Q-link became AOL. https://en.m.wikipedia.org/wiki/Quantum_Link https://en.m.wikipedia.org/wiki/Quantum_Link
cyberferret 10y ago
Reminds me of a day decades ago when I walked into a client site to do a server upgrade, and I noticed on their Novell Server, the uptime counter said it was close to 3 years of continuous uptime. I was flabbergasted. It was just an ordinary little small business with about 10 employees, and it was an old IBM server on a $100 UPS. I almost couldn't bring myself to shut it down to do the upgrade. I mean, I saw servers with 100 to 300 days uptime routinely, but not one that had never been interrupted for over 1000 days....
- Animats 10y ago
  Why not? I just looked at a dedicated server I have at a Codero colo site, and it says "System status: Database up for 516.23 days."
  - cyberferret 10y ago
    I guess some context might be in order - this server was back in the early 90's, in an outback town that is known for massive thunderstorm and power cuts that usually play havoc with computer systems. Being a machine literally in a cupboard in the back of an office, and not in a controlled data center, it was an achievement to have run for 3 years uninterrupted.
general_ai 10y ago
I wonder how it compares to a recent ARM-based Arduino in terms of computational power.
HillaryBriss 10y ago
> CPU check logic checks the results from each, and if there is a discrepancy, if one CPU comes up with a different result than the other, the system immediately disables that pair and uses the remaining pair. how does this work? how does the system know which pair is the faulty pair?
- empath75 10y ago
  More info than you probably wanted to know: https://www.google.co.in/patents/US5263034 https://www.google.co.in/patents/US5263034
Zenst 10y ago
I saw a demo of these systems back in the mid90's when I was working in the reinsurance industry. Was Impressive,fault tolerant stories of customers first knowing something needed to be replaced was when a spare part turned up, which the customer would fit. All apart from the hard drives which needed an engineer due to health and safety aspect due to weight of them. Was full diagnostics with any issue modem dialed up to stratus to report the fault and get replacement part shipped out. Was as if they thought of everything until I asked the question about how they handled heat if the aircon failed. Turned out back then the early systems did not have any temperature monitoring, so somewhat stymied the flow of good stories. But impressive kit, not cheap but at the time was one of the best choices outside the AS/400/IBM route and indeed Lloyds did purchase a few of these systems. More so given they lasted this long in use and even today I've had best in breed systems have faults that were not iditeified as soon as possible, recall some fancy IBMkit having a dieing PSU and could smell the electrical burn faintly without any diagnosstics flagging up any problem for that server to fail in a few days time. But today we tend not to have one large all singing and dancing stable system and often cheaper to have redundancy thru load balancing and with that able to pull a server out of a work load cluster. This along with Assured messaging systems (MQ etc) you negate so many hardware faults that can and did in the past put a halt to work.
dsfyu404ed 10y ago
This reminds me of a switch I heard about at UMaine. It was a 48-port Cisco something or other installed and configured to basically be a dumb switch in a telecom closet in some building around 2004. That building happened to have some things in it that needed to be on generator backup so it never lost power. 11yr later my friend went to go upgrade it.
googamooga 10y ago
First stock exchange in Russia called RTS was built originally on Stratus computers. First launch of the trades was done on Intel 960 based Stratus XA/R in 1995, then in late 1996 we migrated to two PA-RISC based Stratus Continuums, if memory servers me well. Both models provided quadruple redudancy and for the whole trades history we had no interruption related to hardware problems. Only bugs in our software. )) Continuums were working till 2003 when they were replaced with x86-based Stratus servers, which were double or triple redundant, depending on the model. They are still running at the RTS Exhange which is now part of Moscow Exchange (formely MICEX). The only problem with x86 models, at least the models produced back in mid-00 was that the backplane with synchroniztion clock was not redundant and if the clock failed the whole server went down.

219 points by protomyth 10y ago

Note: clocked-down (from 33Mhz) Intel RISC CPUs. It's reliable, sure, but at what cost / MIPS? Power, maintenance, finding spares, opportunity cost from adding new functionality, etc. Also, how does this compare to the oldest running satellites?

ftio 10y ago

Sounds like a modern day Ship of Theseus (https://en.wikipedia.org/wiki/Ship_of_Theseus https://en.wikipedia.org/wiki/Ship_of_Theseus). I wonder how many parts have been untouched since they were installed on Day 1.

akeck 10y ago
My favourite modern day example: http://www.smithsonianmag.com/smart-news/this-japanese-shrine-has-been-torn-down-and-rebuilt-every-20-years-for-the-past-millennium-575558/ http://www.smithsonianmag.com/smart-news/this-japanese-shrin...
__s 10y ago
Key differences being that this is more if the Ship of Theseus was going through the replacemnets, all the while remaining out at sea
- gaius 10y ago
  Trigger's Broom was in active use.
eps 10y ago
80% of it is original [1] http://www.computerworld.com/article/3162416/data-center/booted-up-in-1993-this-server-still-runs-but-not-for-much-longer.html http://www.computerworld.com/article/3162416/data-center/boo...

jcoffland 10y ago

I'd be more interested in the computer system with the longest uptime. I've had Debian systems that have achieved over 1 year uptimes. Could have gone longer but kernel upgrades took precedence over uptime.

nickpsecurity 10y ago
OpenVMS or AS/400. Very common for admins to report systems running for 5+ years without crashes while doing real work. People occasionally forget where they're at and have to go looking for them. My company has an AS/400 employees never saw go down or get maintenance over 10 years. It's in use 24/7. They might have some fail-over setup or upgrade when nobody is using it due to traffic patterns. We've just never seen it fail. Their other benefit is they're very hands off with it nearly self-administering.
- protomyth 10y ago
  Or that AS/400 calls the tech to fix it and s/he just does it while no one is around. Had a call once from IBM asking for directions since the AS/400 RAID cache battery needed replacing. No danger of losing data, the AS/400 was just running slower because it was bypassing the cache.
  - rashkov 10y ago
    So... a tech would just show up at your office, ready to fix the machine? On the one hand, that is some fantastic service when you're running mission critical software. On the other hand, that must cost a whole lot of money and is quite devious of IBM.
    - jonknee 10y ago
      > that must cost a whole lot of money and is quite devious of IBM. You have essentially described IBM's business model. You pay more, but if something goes wrong someone shows up.

nickpsecurity 10y ago

"The use of the cloud for server farms made of hundreds, thousands, and often more computers that are transparent to the user has achieved much the same goal, providing one’s connection to the cloud is also redundant. " This isn't true at all. The cloud offerings are more like traditional, high availability such as fail-over clusters. They might compete with VMS clusters by now with the right components. Maybe even exceed them except for longevity. Systems like NonStop and Stratus are fault-tolerant systems supposed to get five 9's availability with [hopefully] imperceptible moments of failure that [hopefully] recover immediately. I wouldn't trust a cloud platform for that. The latency alone would probably prevent the solution from matching a local, NonStop cluster.

__s 10y ago

Curious they have 2 pairs rather than a triplet

VLM 10y ago
One simple hardware mental model is running two processors in parallel with XORs on all pins with wired-OR outputs hooked up to the error-shutdown line. In practice that doesn't really work because irrelevant differences in PCB trace length and clock distribution mean both processors run just a little bit outta sync jitter so the XOR will be firing fairly constantly for a fraction of each clock cycle. You could play games with gating and sampling only when well settled.... A somewhat more realistic way to build it outta real hardware involves each processor runs every 1 outta X clock cycles. Reading between the lines of the story WRT "down clocked processors" I suspect this is what its doing. This also detects weird power spike problems where lightning miles away means every processor running is going to run 0xFFFF whatever opcode that happens to be, but a couple ns later the spike is gone and the other processors are back to normal. Or RFI/EMC, etc. This is all very nice other than a significant hit to performance if you run a large number of processors. Then again if your primary figure of merit for your system is extreme reliability and not raw MIPS... It would be interesting to hear if they run dual port RAM to get around the jitter XOR problem and have some alternative way to detect synchronous behavior. Dual port ram is weird but COTS. Triple port ram is not as COTS.
- dfox 10y ago
  Comparing bus states of two CPU's running of same clock is perfectly feasible and there are no problems with glitches as long as the whole mechanism is synchronous (and the CPU is completely deterministic, which it should be, but then there are things like RNG-based cache eviction policies). In fact both original Pentium and some m68k processors have all the required hardware for this included and building error checking pair of them involves literally connecting all bus pins of two CPUs in parallel.
- nickpsecurity 10y ago
  I read architectural details of NonStop and Stratus. Both were designed to compare I/O outputs instead of just catching everything at CPU level. The CPU-level redundancy is more for catching transient faults with fail-fast logic. Idea is problem shows up in obvious way by time it hits I/O buses. Past that I dont recall much.

JoachimSchipper 10y ago

Does someone know why these systems - Stratus, Tandem/HP NonStop, etc. - appear to be relics of the past? I can see how Google is better off spending money on software engineers than on hardware (thus, fault-tolerant distributed systems); but there's lots of systems that don't need Google-scale computer power, nor Google-style ultra-low-cost-per-operation hardware, but that really need to stay up - utilities, banks, every system at the center of a big enterprise. Is the issue that modern software just isn't reliable enough to make hardware failures an important part of the downtime? Or is it just that systems like https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013428 https://kb.vmware.com/selfservice/microsites/search.do?langu... are cheaper today? (I guess those actually work, right? The level of complexity involved is frightening...)

brianwawok 10y ago
Worked at a big co switching off tandem. Went from giant tandem machine that cost 2 million dollars per year to rent to about 6 off the shelf Dell servers for 6k each one time fee. Performance at the end was about 3x the old system. Was a glorified billing system. If one node went down, who cares, the other 5 did the job. Most people don't value 99.999 vs 99.9999 reliability as worth millions of dollars per system per year. Space shuttles may disagree, but not billing platforms.
- z3t4 10y ago
  Five way redundancy look good on paper, put if you use the same vendor and hardware purchased at the same time, you have a one point of failure, aka when one server fails there's a high probability that the others will fail simultaneously.
  - davnn 10y ago
    Why?
    - smallnamespace 10y ago

nxc18 10y ago

I interned at a company that had given continuous service with a tandem/NonStop system since the 70s (80s latest) iirc. Its very cool to see such reliable systems in practice.

makethetick 10y ago

Stratus are still going (I'm assuming it's the same company, same product, same name) and making fault tolerant hardware like this. They also offer a lower end product based on two standard servers where all the redundancy is software based with virtual machines. It's clever kit and looks after a lot of critical infrastructure where downtime isn't an option.

ghaff 10y ago
It is although I believe all the FT is now done in software in some fashion. i.e. the CPU/Memory/Chipset/IO, etc. are standard components with the lockstep done through a software layer. The same is true of HP Integrity Nonstop AFAIK. I used to follow these when I was industry analyst but I've been away from this space for a few years.
- makethetick 10y ago
  They still do both, here's their hardware offering which is hardware and software based: http://www.stratus.com/solutions/platforms/ftserver/ http://www.stratus.com/solutions/platforms/ftserver/ And this is there software based solution which runs on commodity hardware: http://www.stratus.com/solutions/software/everrun-enterprise-express/ http://www.stratus.com/solutions/software/everrun-enterprise... I only work with the latter but have heard the hardware offering is bulletproof.

eps 10y ago

mods, may want to merge this with [1] from yesterday. [1] https://news.ycombinator.com/item?id=13507972 https://news.ycombinator.com/item?id=13507972

myrandomcomment 10y ago

Quantum Link the online service for the Commodore 64 ran on this system. The first GUI online multiplayer world was on Q-link. Q-link became AOL. https://en.m.wikipedia.org/wiki/Quantum_Link https://en.m.wikipedia.org/wiki/Quantum_Link

cyberferret 10y ago

Reminds me of a day decades ago when I walked into a client site to do a server upgrade, and I noticed on their Novell Server, the uptime counter said it was close to 3 years of continuous uptime. I was flabbergasted. It was just an ordinary little small business with about 10 employees, and it was an old IBM server on a $100 UPS. I almost couldn't bring myself to shut it down to do the upgrade. I mean, I saw servers with 100 to 300 days uptime routinely, but not one that had never been interrupted for over 1000 days....

Animats 10y ago
Why not? I just looked at a dedicated server I have at a Codero colo site, and it says "System status: Database up for 516.23 days."
- cyberferret 10y ago
  I guess some context might be in order - this server was back in the early 90's, in an outback town that is known for massive thunderstorm and power cuts that usually play havoc with computer systems. Being a machine literally in a cupboard in the back of an office, and not in a controlled data center, it was an achievement to have run for 3 years uninterrupted.

general_ai 10y ago

I wonder how it compares to a recent ARM-based Arduino in terms of computational power.

HillaryBriss 10y ago

> CPU check logic checks the results from each, and if there is a discrepancy, if one CPU comes up with a different result than the other, the system immediately disables that pair and uses the remaining pair. how does this work? how does the system know which pair is the faulty pair?

empath75 10y ago
More info than you probably wanted to know: https://www.google.co.in/patents/US5263034 https://www.google.co.in/patents/US5263034

Zenst 10y ago

I saw a demo of these systems back in the mid90's when I was working in the reinsurance industry. Was Impressive,fault tolerant stories of customers first knowing something needed to be replaced was when a spare part turned up, which the customer would fit. All apart from the hard drives which needed an engineer due to health and safety aspect due to weight of them. Was full diagnostics with any issue modem dialed up to stratus to report the fault and get replacement part shipped out. Was as if they thought of everything until I asked the question about how they handled heat if the aircon failed. Turned out back then the early systems did not have any temperature monitoring, so somewhat stymied the flow of good stories. But impressive kit, not cheap but at the time was one of the best choices outside the AS/400/IBM route and indeed Lloyds did purchase a few of these systems. More so given they lasted this long in use and even today I've had best in breed systems have faults that were not iditeified as soon as possible, recall some fancy IBMkit having a dieing PSU and could smell the electrical burn faintly without any diagnosstics flagging up any problem for that server to fail in a few days time. But today we tend not to have one large all singing and dancing stable system and often cheaper to have redundancy thru load balancing and with that able to pull a server out of a work load cluster. This along with Assured messaging systems (MQ etc) you negate so many hardware faults that can and did in the past put a halt to work.

dsfyu404ed 10y ago

This reminds me of a switch I heard about at UMaine. It was a 48-port Cisco something or other installed and configured to basically be a dumb switch in a telecom closet in some building around 2004. That building happened to have some things in it that needed to be on generator backup so it never lost power. 11yr later my friend went to go upgrade it.

googamooga 10y ago

First stock exchange in Russia called RTS was built originally on Stratus computers. First launch of the trades was done on Intel 960 based Stratus XA/R in 1995, then in late 1996 we migrated to two PA-RISC based Stratus Continuums, if memory servers me well. Both models provided quadruple redudancy and for the whole trades history we had no interruption related to hardware problems. Only bugs in our software. )) Continuums were working till 2003 when they were replaced with x86-based Stratus servers, which were double or triple redundant, depending on the model. They are still running at the RTS Exhange which is now part of Moscow Exchange (formely MICEX). The only problem with x86 models, at least the models produced back in mid-00 was that the backplane with synchroniztion clock was not redundant and if the clock failed the whole server went down.