4 ms·

The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They

The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They also did rolling upgrades across both different OS versions and CPU architectures. The methods are public if any competitors want to match or exceed them: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04622808 https://support.hpe.com/hpsc/doc/public/display?docId=emr_na... These modern cloud services using Linux ecosystem are doing rolling outages instead. I'm amazed people keep making them dependencies. Maybe their local systems using similar technology went down more often. Maybe this is an improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was business-critical with lots of money on the line. I want it staying up, up, up. :)

mugsie 7y ago
Modern apps are designed to deal with part of them being down - it is the core upgrade strategy. VMS clusters were (and modern equivalents are) awesome, but they have a different speed of innovation / development. It is a trade off - perfect uptime, but that new feature could take a year, or 99.9% uptime, but it can be written and deployed tomorrow.
- nickpsecurity 7y ago
  Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation. In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security: https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/publication/p-safe-asynchronous-event-driven-programming/ https://www.microsoft.com/en-us/research/publication/p-safe-... Plus some of their other tools in various states of usability: https://rise4fun.com/ https://rise4fun.com/
- fortran77 7y ago
  Our Erlang nodes are never down. We do live code updates and have gone several years now with no downtime.

mugsie 7y ago

Modern apps are designed to deal with part of them being down - it is the core upgrade strategy. VMS clusters were (and modern equivalents are) awesome, but they have a different speed of innovation / development. It is a trade off - perfect uptime, but that new feature could take a year, or 99.9% uptime, but it can be written and deployed tomorrow.

nickpsecurity 7y ago
Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation. In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security: https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/project/ironclad/ https://www.microsoft.com/en-us/research/publication/p-safe-asynchronous-event-driven-programming/ https://www.microsoft.com/en-us/research/publication/p-safe-... Plus some of their other tools in various states of usability: https://rise4fun.com/ https://rise4fun.com/
fortran77 7y ago
Our Erlang nodes are never down. We do live code updates and have gone several years now with no downtime.