5 ms·

> It has since expanded into numerous other areas, such as web servers, and has achieved nine 9s of availability (31 milliseconds/year of downtime). I definite

> It has since expanded into numerous other areas, such as web servers, and has achieved nine 9s of availability (31 milliseconds/year of downtime). I definitely want to read more about this.

lucasmullens 10y ago
"nine 9s" comes from Erlang (or in this case BEAM): https://stackoverflow.com/questions/8426897/erlangs-99-9999999-nine-nines-reliability https://stackoverflow.com/questions/8426897/erlangs-99-99999...
kjksf 10y ago
This is how myths are created and perpetuated. This claims from a study of a particular model of Ericsson's ATM switch whose software was written in Erlang. A can tell you for sure that: * Ericsson has other switches and other telecom equipment whose software is written in C++ * other companies (Nokia, Cisco, Alcatel etc.) also built similar complex telecom equipment whose software was in C++ I'm going to bet that at least one of those devices was more reliable than that particular switch, because it's not the language that creates reliability but people writing and testing the software. Some languages make it easier than others but attributing some magic qualities from extrapolating a single case study of a telecom equipment is just silly. Telecom equipment is reliable because it has to be. If you're incapable of writing reliable software for telecom equipment then your company will go bankrupt and be replaced by one that can, even if the language they use is C++.
- planteen 10y ago
  Yeah exactly. To make something 9 9s reliable, there is lots of fault tolerance in the system and hardware architecture.
- brightball 10y ago
  I grabbed it from the Erlang Wikipedia for what it's worth. The language creates reliability by isolating running parts in millions of small heaps that can independently fail and immediately restart. It's not automatic, but the way that the language is designed makes these types of numbers much more feasible. https://en.wikipedia.org/wiki/Erlang_(programming_language) https://en.wikipedia.org/wiki/Erlang_(programming_language) It also had this along the same lines though: >As Tim Bray, director of Web Technologies at Sun Microsystems, expressed in his keynote at OSCON in July 2008: "If somebody came to me and wanted to pay me a lot of money to build a large scale message handling system that really had to be up all the time, could never afford to go down for years at a time, I would unhesitatingly choose Erlang to build it in."
- lobster_johnson 10y ago
  But suitability is a huge factor here. The point isn't that "nine nines" reliability cannot be achieved in C++ or whatever, it's that it's significantly harder and costlier to achieve. But also: Of course a language creates reliability. Null pointers are impossible in Erlang, whereas they are provable impossible to avoid in C++. So people bring this up because Erlang programs are pretty much reliable by default (assuming you stick to OTP and its design). To help it reach insane uptimes, Erlang also has a few features that are hard to replicate in C++, if not impossible. The main one is hot code replacement; being able to incrementally upgrade a system in a safe and controlled manner without restarting it is a huge benefit in such a system. It's also trivial to attach to a running Erlang program and introspect it as if you're truly running inside it. Goes hand in hand with hot code replacement. With C++, you get fairly crummy, low-level things like gdb, without the ability to actually write C++ code while in the debugger. Also, remember that crashes can occur in bug-free programs. Hardware failures can be hard to predict and handle.
  - gpderetta 10y ago
    > Null pointers are impossible in Erlang, whereas they are provable impossible to avoid in C++. They are trivially possible to avoid in C++ as you do not have to use raw pointers. References, value types, custom non-null smart_ptrs, etc.
    - jlouis 10y ago
      Don't underestimate the cost of trivial maintenance. In any project constrained by time, you risk people skipping on these things in part of the system. Erlang/Elixir is pretty well built for robustness, so a single library is less likely to affect the system as a whole. The same cannot be said for C++, where every part has to be close to correct for the operation to proceed as you expect. The trade-off you get however, is that the explicit control in C++ tend to produce systems which run faster under nominal operation. In practice, many Erlang systems contains some C/C++ parts at their core for fast operation. Depending on their complexity, you try to isolate them from the rest of the system as much as you can. In short, you use Erlang as a top-level control backplane for your C/C++ code. The advantage is you get the speed of C++ where needed, but the added productivity of Erlang everywhere else. Erlang programmers are usually practically minded in this regard. "functional purity" doesn't really matter that much if the problem isn't solved.
- pjmlp 10y ago
  Have you ever worked in telecom industry? I did, at one of the the companies you list there. We had outsourced and offshored code for modules on those switches you mentioned. Have you ever had the pleasure to review C++ code in such type of projects? I did, and hope to never do it again.
  - kinghajj 10y ago
    Remember any specific examples/patterns? Curious to see how bad it could be...
    - pjmlp 10y ago
      Basically: - Code either "C code compiled with C++ compiler" or C++ OOP Spaghetti like in the CORBA/DCOM days before JEE was a thing. - functions/methods that span several screens - barely any kind of testing - due to emphasis on C style programming, lots of fun tracking down pointer misuses - re-implementation of code that is already part of C++ standard library, even the ones from C - lots of copy-paste on the same file because devs don't understand neither source control nor modularity
      - wuschel 10y ago
        Many of these issues you list can be overcome by good coding practise. Although I understand that the framework/languages used allowed for these bad feats. I wonder how Rust with its focus on memory safety could play out in such scenarios. I need to have a look at Erlang/OTP. It is a bit of a hidden, alien gem.
        pjmlp 10y ago
        > Many of these issues you list can be overcome by good coding practise. The enterprise is not the space for it, specially among companies whose bread and butter is not to sell software. The only measures of quality management cares about, are "does it work" and "delivers what customers pay for", anything else are just costs that need cutting.
        10y ago
- taavi013 10y ago
  There has been nice study by Motorola exactly on this topic C++ vs. Erlang implementing telecom software. Study slides are online at http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for-motorola-telecoms-software http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for... Interesting result from this study - erlang implementation is not only more robust but also faster.
  - macintux 10y ago
    Thanks for the link. Jives with my intuition overall, but the speed comparison is jaw-dropping.
- eternalban 10y ago
  Not disagreeing with you, just adding a bit more info. I'm nearly sure that the fault-tolerance magic of Erlang happens at the library (OTP) level with supervisors.
  - macintux 10y ago
    That's certainly part of it, but really it's the overlap of multiple layers of the overall system. Lightweight processes, messaging, OTP, supervision all play a role when it comes to runtime fault tolerance. Erlang is a rare beast in that the language and its VM are both very opinionated and very focused on certain types of problems (such as telecom and fault tolerance). This is not Java and its VM trying to be all things to all people.
- pmarreck 10y ago
  If a rogue cosmic ray flips a bit in a running process handling phone calls (note: this exact thing apparently took down all of Amazon S3 once, http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html), in code that wasn't expected to error, 1) Go will crash or at minimum (arguably much worse) go into an unknown state. 2) C++ will crash, and ideally, restart, but lose all "live" state. (So calls get dropped, etc.) 3) Erlang/Elixir process will crash and then get instantly restarted by the supervisor. If the bit flip hit the supervisor instead, then its supervisor will restart it (most larger BEAM apps have a supervision hierarchy). The only way BEAM will go down is if the cosmic ray hit BEAM. Checkmate, non-BEAMs? ;)
  - ramchip 10y ago
    I'm not sure if you're being serious, but a random bit flip can easily put a process in an incorrect state rather than make it crash (say charge a user $1000000 instead of $1), or affect a pointer (of which Erlang uses a ton internally - imagine for a second how lists, maps, and code loading work) and make the whole VM crash when it tries to access it. Or it could just kill the OS itself. You would have to be incredibly lucky to get a nice, clean process crash.
    - pmarreck 10y ago
      Ah, you're right. Most bit flips wouldn't be detected under ANY language. I guess machine-level redundancy isn't going away anytime soon, then

> It has since expanded into numerous other areas, such as web servers, and has achieved nine 9s of availability (31 milliseconds/year of downtime). I definitely want to read more about this.

lucasmullens 10y ago

"nine 9s" comes from Erlang (or in this case BEAM): https://stackoverflow.com/questions/8426897/erlangs-99-9999999-nine-nines-reliability https://stackoverflow.com/questions/8426897/erlangs-99-99999...

kjksf 10y ago

This is how myths are created and perpetuated. This claims from a study of a particular model of Ericsson's ATM switch whose software was written in Erlang. A can tell you for sure that: * Ericsson has other switches and other telecom equipment whose software is written in C++ * other companies (Nokia, Cisco, Alcatel etc.) also built similar complex telecom equipment whose software was in C++ I'm going to bet that at least one of those devices was more reliable than that particular switch, because it's not the language that creates reliability but people writing and testing the software. Some languages make it easier than others but attributing some magic qualities from extrapolating a single case study of a telecom equipment is just silly. Telecom equipment is reliable because it has to be. If you're incapable of writing reliable software for telecom equipment then your company will go bankrupt and be replaced by one that can, even if the language they use is C++.

planteen 10y ago
Yeah exactly. To make something 9 9s reliable, there is lots of fault tolerance in the system and hardware architecture.
brightball 10y ago
I grabbed it from the Erlang Wikipedia for what it's worth. The language creates reliability by isolating running parts in millions of small heaps that can independently fail and immediately restart. It's not automatic, but the way that the language is designed makes these types of numbers much more feasible. https://en.wikipedia.org/wiki/Erlang_(programming_language) https://en.wikipedia.org/wiki/Erlang_(programming_language) It also had this along the same lines though: >As Tim Bray, director of Web Technologies at Sun Microsystems, expressed in his keynote at OSCON in July 2008: "If somebody came to me and wanted to pay me a lot of money to build a large scale message handling system that really had to be up all the time, could never afford to go down for years at a time, I would unhesitatingly choose Erlang to build it in."
lobster_johnson 10y ago
But suitability is a huge factor here. The point isn't that "nine nines" reliability cannot be achieved in C++ or whatever, it's that it's significantly harder and costlier to achieve. But also: Of course a language creates reliability. Null pointers are impossible in Erlang, whereas they are provable impossible to avoid in C++. So people bring this up because Erlang programs are pretty much reliable by default (assuming you stick to OTP and its design). To help it reach insane uptimes, Erlang also has a few features that are hard to replicate in C++, if not impossible. The main one is hot code replacement; being able to incrementally upgrade a system in a safe and controlled manner without restarting it is a huge benefit in such a system. It's also trivial to attach to a running Erlang program and introspect it as if you're truly running inside it. Goes hand in hand with hot code replacement. With C++, you get fairly crummy, low-level things like gdb, without the ability to actually write C++ code while in the debugger. Also, remember that crashes can occur in bug-free programs. Hardware failures can be hard to predict and handle.
- gpderetta 10y ago
  > Null pointers are impossible in Erlang, whereas they are provable impossible to avoid in C++. They are trivially possible to avoid in C++ as you do not have to use raw pointers. References, value types, custom non-null smart_ptrs, etc.
  - jlouis 10y ago
    Don't underestimate the cost of trivial maintenance. In any project constrained by time, you risk people skipping on these things in part of the system. Erlang/Elixir is pretty well built for robustness, so a single library is less likely to affect the system as a whole. The same cannot be said for C++, where every part has to be close to correct for the operation to proceed as you expect. The trade-off you get however, is that the explicit control in C++ tend to produce systems which run faster under nominal operation. In practice, many Erlang systems contains some C/C++ parts at their core for fast operation. Depending on their complexity, you try to isolate them from the rest of the system as much as you can. In short, you use Erlang as a top-level control backplane for your C/C++ code. The advantage is you get the speed of C++ where needed, but the added productivity of Erlang everywhere else. Erlang programmers are usually practically minded in this regard. "functional purity" doesn't really matter that much if the problem isn't solved.
pjmlp 10y ago
Have you ever worked in telecom industry? I did, at one of the the companies you list there. We had outsourced and offshored code for modules on those switches you mentioned. Have you ever had the pleasure to review C++ code in such type of projects? I did, and hope to never do it again.
- kinghajj 10y ago
  Remember any specific examples/patterns? Curious to see how bad it could be...
  - pjmlp 10y ago
    Basically: - Code either "C code compiled with C++ compiler" or C++ OOP Spaghetti like in the CORBA/DCOM days before JEE was a thing. - functions/methods that span several screens - barely any kind of testing - due to emphasis on C style programming, lots of fun tracking down pointer misuses - re-implementation of code that is already part of C++ standard library, even the ones from C - lots of copy-paste on the same file because devs don't understand neither source control nor modularity
    - wuschel 10y ago
      Many of these issues you list can be overcome by good coding practise. Although I understand that the framework/languages used allowed for these bad feats. I wonder how Rust with its focus on memory safety could play out in such scenarios. I need to have a look at Erlang/OTP. It is a bit of a hidden, alien gem.
      - pjmlp 10y ago
        > Many of these issues you list can be overcome by good coding practise. The enterprise is not the space for it, specially among companies whose bread and butter is not to sell software. The only measures of quality management cares about, are "does it work" and "delivers what customers pay for", anything else are just costs that need cutting.
      - 10y ago
taavi013 10y ago
There has been nice study by Motorola exactly on this topic C++ vs. Erlang implementing telecom software. Study slides are online at http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for-motorola-telecoms-software http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for... Interesting result from this study - erlang implementation is not only more robust but also faster.
- macintux 10y ago
  Thanks for the link. Jives with my intuition overall, but the speed comparison is jaw-dropping.
eternalban 10y ago
Not disagreeing with you, just adding a bit more info. I'm nearly sure that the fault-tolerance magic of Erlang happens at the library (OTP) level with supervisors.
- macintux 10y ago
  That's certainly part of it, but really it's the overlap of multiple layers of the overall system. Lightweight processes, messaging, OTP, supervision all play a role when it comes to runtime fault tolerance. Erlang is a rare beast in that the language and its VM are both very opinionated and very focused on certain types of problems (such as telecom and fault tolerance). This is not Java and its VM trying to be all things to all people.
pmarreck 10y ago
If a rogue cosmic ray flips a bit in a running process handling phone calls (note: this exact thing apparently took down all of Amazon S3 once, http://status.aws.amazon.com/s3-20080720.html http://status.aws.amazon.com/s3-20080720.html), in code that wasn't expected to error, 1) Go will crash or at minimum (arguably much worse) go into an unknown state. 2) C++ will crash, and ideally, restart, but lose all "live" state. (So calls get dropped, etc.) 3) Erlang/Elixir process will crash and then get instantly restarted by the supervisor. If the bit flip hit the supervisor instead, then its supervisor will restart it (most larger BEAM apps have a supervision hierarchy). The only way BEAM will go down is if the cosmic ray hit BEAM. Checkmate, non-BEAMs? ;)
- ramchip 10y ago
  I'm not sure if you're being serious, but a random bit flip can easily put a process in an incorrect state rather than make it crash (say charge a user $1000000 instead of $1), or affect a pointer (of which Erlang uses a ton internally - imagine for a second how lists, maps, and code loading work) and make the whole VM crash when it tries to access it. Or it could just kill the OS itself. You would have to be incredibly lucky to get a nice, clean process crash.
  - pmarreck 10y ago
    Ah, you're right. Most bit flips wouldn't be detected under ANY language. I guess machine-level redundancy isn't going away anytime soon, then