14 ms·

What I learned from Erlang about resiliency in systems design (2019)

142 points by srijan4 5y ago

einpoklum 5y ago
Apparently, what that guy learned was not to write system software, only containerized and supervised apps. Not sure if that's really the lesson of Erlang as a language, but it certainly sounds like the lesson you would learn if you work for a company selling VM management software (which the author does).
- pessimizer 5y ago
  You might have something there; I still remember Mitchell Hashimoto of Hashicorp/Vagrant fame as an erlang blogger.
dnautics 5y ago
I think it's important to mention that an important part of erlang's failure domain driven mentality is not just "let it crash", but also tying together failure domains: what should you also bring down with you when you crash. For example. Your data sync task encounter an http error in a connected SAAS that threatens data inconsistency. Perfect time to crash. But if you do so, also take down the database connection so that the connection can be returned to the connection pool and the transaction you're running in can be rolled back, ideally without having to try/catch and keep track/selectively recycle of all of cumulative responsibilities that have accrued in that task.
- continuational 5y ago
  This is so important. Too many asynchronous solution simply don't support cancellation, and instead just keep chugging along, doing undesirable things while wasting cpu & ram.
  - bullen 5y ago
    RAM yes, CPU should not waste while waiting for an async. non-blocking request. Apache web server also crashes safely when some PHP script leaks memory, but if you have a proper VM with GC this is something of the past. Fortunately most web systems use Java or it's copy C# at this point and that is not going to change since Erlang has a simple memory model that cannot do joint parallel tasks. Go has no VM, WASM has no GC, rust is too slow to compile... that leaves plain C with a C++ compiler but you don't want to have that on a server because assembly seg. faults. So on the server you have to use Java. Not EE but SE.
    - hnedeotes 5y ago
      > Erlang has a simple memory model that cannot do joint parallel tasks. What do you mean?
      - bullen 5y ago
        You have to copy memory before sharing it between threads. You cannot do atomic memory sharing between threads = threads cannot work on the same task at the "same" time efficiently.
        hnedeotes 5y ago
        That's actually much a feature. Many problems can still be done quite efficiently, for instance stream parsing a file you can have each N newlines being sent to different processes, and the same with many other problems that can be sliced, traversing nested collections, fetching batches of records from stores, etc. Sometimes you can also reformulate the problem, but yes not all problems fit. I would add though that whenever you want to write orchestration around that parallel work it's much easier in erlang than the alternatives.
        bullen 5y ago
        Maybe, but some problems, like making a MMO server cannot be solved with Erlang. So everyone that uses this inferior tech is wasting their and humanitys time. Erlang was designed to make many 1-on-1 phonecalls, it's terrible for almost everything else and today 1-on-1 does not exist just like obsolete TV is unidirectional. In the future everything is many 1-to-many (requires joint parallelism) and that is the final protocol just like the final medium is 3D! We need to sharpen up and make the best tools. C (client) and JavaSE (server) are unsuperceded for decades. it's time to stop fragmenting the programmers workspace and co-operate instead. If you use any other languages you are in the wrong and history will prove you to be a wasteful person.
- BoiledCabbage 5y ago
  The linked SO provides a great description of the philosophy/benefits of Erlang. > Erlang has several features that remove human working time as a source of downtime: > Hot code reloading. In an Erlang system, it is easy to compile and load a replacement module for an existing one. The BEAM emulator does the swap automatically without apparently stopping anything. There is doubtless some tiny amount of time during which this transfer happens, but it's happening automatically in computer time, rather than manually in human time. This makes it possible to do upgrades with essentially zero downtime. (You could have downtime if the replacement module has a bug which crashes the system, but that's why you test before deploying to production.) > Supervisors. Erlang's OTP library has a supervisory framework built into it which lets you define how the system should react if a module crashes. The standard action here is to restart the failed module. Assuming the restarted module doesn't immediately crash again, the total downtime charged against your system might be a matter of milliseconds. A solid system that hardly ever crashes might indeed accumulate only a fraction of a second of total downtime over the course of years of run time. > Processes. These correspond roughly to threads in other languages, except that they do not share state except through persistent data stores. Other than that, communication happens via message passing. Because Erlang processes are very inexpensive (far cheaper than OS threads) this encourages a loosely-coupled design, so that if a process dies, only one tiny part of the system experiences downtime. Typically, the supervisor restarts that one process, with little to no impact on the rest of the system. > Asynchronous message passing. When one process wants to tell another something, there is a first-class operator in the Erlang language that lets it do that. The message sending process doesn't have to wait for the receiver to process the message, and it doesn't have to coordinate ownership of data sent. The asynchronous functional nature of Erlang's message-passing system takes care of all that. This helps maintain high uptimes because it reduces the effect that downtime in one part of a system can have on other parts. > Clustering. This follows from the previous point: Erlang's message passing mechanism works transparently between machines on a network, so a sending process doesn't even have to care that the receiver is on a separate machine. This provides an easy mechanism for dividing a workload up among many machines, each of which can go down separately without harming overall system uptime. 1. https://stackoverflow.com/questions/8426897/erlangs-99-9999999-nine-nines-reliability https://stackoverflow.com/questions/8426897/erlangs-99-99999...
ezoe 5y ago
The hard problem is, it never crash so obviously like the author charmingly presented as echo c > /proc/sysrq-trigger. When something went bad(mostly hardware or rare edge case bug on software), it works mostly fine but randomly fails and no amount of hard coded sane checks can automate the detection. It either not crash(unusually slow or garbage result), or crash but it happens rarely enough to never go beyond the threshold of supervisor's restart strategy, so the supervisor keep restarting the unstable process rather than crash and let the parent supervisor know to remove itself from the cluster of nodes.
- svrtknst 5y ago
  "Crash" can mean a lot of things, and in Erlang, a crash occurs whenever your functions are unable to perform their task (for instance, there were no functions that match your input data), or when process calls time out. More obvious crashes exist as well - a remote connection hangs up, a computer catches on fire, etc. You could also use a variety of supervisor strategies or methods to detect what caused the crash.
- NavinF 5y ago
  Yep. Or one RPC call suddenly becomes slow because of a partial failure like a disk retrying sectors and that slows down the whole application to a crawl :(
heleninboodler 5y ago
I've always been a little confused about this "let it crash" philosophy. In my experience, the large scale services I have worked on almost never "crash." They throw 500 errors sometimes because they safely caught an exception. Would it be better to bring the whole server down and have the operating system restart it? I feel like I'm missing out on some kind of epiphany here, but it feels like the advice is to make my service crash where it wouldn't have before, which makes no sense to me. Can anyone explain it?
- vvanders 5y ago
  "Let it crash" is largely about restarting systems to known good state in an expanding scope. The Zen of Erlang[1] covers it really well, it's a longer read but well worth it if you want to understand a lot of the design choices of Erlang. Nearly all of the devices you use employ these approaches in one form or another. A watchdog timer[2] is a pretty simple and powerful version of this. Timeouts and retries follow a somewhat similar approach, Erlang just embraces that across the whole language. It really is a fascinating approach to a different design space(latency and reliability over throughput) through the requirements a telecom stack necessitated. [1] https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html [2] https://en.wikipedia.org/wiki/Watchdog_timer https://en.wikipedia.org/wiki/Watchdog_timer
  - 1propionyl 5y ago
    Perhaps another succinct phrasing that might be more easily grokked without spending time working on Erlang systems is: "Let it rollback and retry" The way one thinks about processes in Erlang is different than how one thinks about threads in most languages. The expected behavior when you kill a process is that it will be right back up with a known good state very quickly, and you won't have to do much about it yourself because it's handled in a supervisor far from your local process. It's subtle, but it makes a huge difference. In most languages one expects a thrown unhandled exception to wreak havoc. But in Erlang graceful failure and restarting is the norm, not the exception. It's the expected behavior. Moreover, the responsibility of maintaining the process tree integrity is delegated fully to specific processes. "Business intelligence" (to ape a phrase) nodes are very effectively isolated from having to care. If they don't know how to handle such a restart, you just let them crash/be killed and restarted too.
rvr_ 5y ago
And that is also why PHP is so robust: shared nothing. Preforking. Time and memory limits per request. Uncaught errors ends only the current request and so on...
- tannhaeuser 5y ago
  If you mean classic PHP CGI, sure. But almost all PHP sites today run on mod_php/PHP-FPM on Apache (or equivalents on nginx) which doesn't run a fresh CGI process-per-request, but multiplexes requests onto a single PHP process instead. The overhead of parsing PHP code for each request would be brutal.
  - rvr_ 5y ago
    Sure, classic CGI is one process per request, but even on php-fpm or apache prefork/mod_php each process will only work for a given maximum number of requests, thus refreshing periodicaly. Also, if any of these workers is killed (say, by the OOM-killer), none of the concurrent requests will be affected. Also, the overhead of parsing is almost zero when you have the opcache extension loaded. I'm not a huge fan of the language, but the PHP runtime wins me by its simplicity and robustness. I work daily on a codebase that has hundreds of millions of page hits per day and 100k QPS during peak hours running on LAMP with a very few servers. We face challenges like everyone else does, but we never had outages because of the languange/runtime.
- bruce343434 5y ago
  Ironic seeing PHP being mentioned as it almost never crashes and always attempt to chug along. Array index doesn't exist? Don't even 500, just chuck a warning out.
  - megous 5y ago
    You can make PHP throw on notices if you like.
  - rvr_ 5y ago
    The language/runtime is very configurable. Some codebases will set strict_type=1 plus error_reporting(E_ALL) eliminating many instances of these kind of problems / footguns.
kilodeca 5y ago
What's the purpose of having that picture at the beginning?
- bruce343434 5y ago
  Maybe it's a resilient tree?
deleted 5y ago
[deleted]

142 points by srijan4 5y ago

Apparently, what that guy learned was not to write system software, only containerized and supervised apps. Not sure if that's really the lesson of Erlang as a language, but it certainly sounds like the lesson you would learn if you work for a company selling VM management software (which the author does).

pessimizer 5y ago
You might have something there; I still remember Mitchell Hashimoto of Hashicorp/Vagrant fame as an erlang blogger.

dnautics 5y ago

I think it's important to mention that an important part of erlang's failure domain driven mentality is not just "let it crash", but also tying together failure domains: what should you also bring down with you when you crash. For example. Your data sync task encounter an http error in a connected SAAS that threatens data inconsistency. Perfect time to crash. But if you do so, also take down the database connection so that the connection can be returned to the connection pool and the transaction you're running in can be rolled back, ideally without having to try/catch and keep track/selectively recycle of all of cumulative responsibilities that have accrued in that task.

continuational 5y ago
This is so important. Too many asynchronous solution simply don't support cancellation, and instead just keep chugging along, doing undesirable things while wasting cpu & ram.
- bullen 5y ago
  RAM yes, CPU should not waste while waiting for an async. non-blocking request. Apache web server also crashes safely when some PHP script leaks memory, but if you have a proper VM with GC this is something of the past. Fortunately most web systems use Java or it's copy C# at this point and that is not going to change since Erlang has a simple memory model that cannot do joint parallel tasks. Go has no VM, WASM has no GC, rust is too slow to compile... that leaves plain C with a C++ compiler but you don't want to have that on a server because assembly seg. faults. So on the server you have to use Java. Not EE but SE.
  - hnedeotes 5y ago
    > Erlang has a simple memory model that cannot do joint parallel tasks. What do you mean?
    - bullen 5y ago
      You have to copy memory before sharing it between threads. You cannot do atomic memory sharing between threads = threads cannot work on the same task at the "same" time efficiently.
      - hnedeotes 5y ago
        That's actually much a feature. Many problems can still be done quite efficiently, for instance stream parsing a file you can have each N newlines being sent to different processes, and the same with many other problems that can be sliced, traversing nested collections, fetching batches of records from stores, etc. Sometimes you can also reformulate the problem, but yes not all problems fit. I would add though that whenever you want to write orchestration around that parallel work it's much easier in erlang than the alternatives.
        bullen 5y ago
        Maybe, but some problems, like making a MMO server cannot be solved with Erlang. So everyone that uses this inferior tech is wasting their and humanitys time. Erlang was designed to make many 1-on-1 phonecalls, it's terrible for almost everything else and today 1-on-1 does not exist just like obsolete TV is unidirectional. In the future everything is many 1-to-many (requires joint parallelism) and that is the final protocol just like the final medium is 3D! We need to sharpen up and make the best tools. C (client) and JavaSE (server) are unsuperceded for decades. it's time to stop fragmenting the programmers workspace and co-operate instead. If you use any other languages you are in the wrong and history will prove you to be a wasteful person.
BoiledCabbage 5y ago
The linked SO provides a great description of the philosophy/benefits of Erlang. > Erlang has several features that remove human working time as a source of downtime: > Hot code reloading. In an Erlang system, it is easy to compile and load a replacement module for an existing one. The BEAM emulator does the swap automatically without apparently stopping anything. There is doubtless some tiny amount of time during which this transfer happens, but it's happening automatically in computer time, rather than manually in human time. This makes it possible to do upgrades with essentially zero downtime. (You could have downtime if the replacement module has a bug which crashes the system, but that's why you test before deploying to production.) > Supervisors. Erlang's OTP library has a supervisory framework built into it which lets you define how the system should react if a module crashes. The standard action here is to restart the failed module. Assuming the restarted module doesn't immediately crash again, the total downtime charged against your system might be a matter of milliseconds. A solid system that hardly ever crashes might indeed accumulate only a fraction of a second of total downtime over the course of years of run time. > Processes. These correspond roughly to threads in other languages, except that they do not share state except through persistent data stores. Other than that, communication happens via message passing. Because Erlang processes are very inexpensive (far cheaper than OS threads) this encourages a loosely-coupled design, so that if a process dies, only one tiny part of the system experiences downtime. Typically, the supervisor restarts that one process, with little to no impact on the rest of the system. > Asynchronous message passing. When one process wants to tell another something, there is a first-class operator in the Erlang language that lets it do that. The message sending process doesn't have to wait for the receiver to process the message, and it doesn't have to coordinate ownership of data sent. The asynchronous functional nature of Erlang's message-passing system takes care of all that. This helps maintain high uptimes because it reduces the effect that downtime in one part of a system can have on other parts. > Clustering. This follows from the previous point: Erlang's message passing mechanism works transparently between machines on a network, so a sending process doesn't even have to care that the receiver is on a separate machine. This provides an easy mechanism for dividing a workload up among many machines, each of which can go down separately without harming overall system uptime. 1. https://stackoverflow.com/questions/8426897/erlangs-99-9999999-nine-nines-reliability https://stackoverflow.com/questions/8426897/erlangs-99-99999...

ezoe 5y ago

The hard problem is, it never crash so obviously like the author charmingly presented as echo c > /proc/sysrq-trigger. When something went bad(mostly hardware or rare edge case bug on software), it works mostly fine but randomly fails and no amount of hard coded sane checks can automate the detection. It either not crash(unusually slow or garbage result), or crash but it happens rarely enough to never go beyond the threshold of supervisor's restart strategy, so the supervisor keep restarting the unstable process rather than crash and let the parent supervisor know to remove itself from the cluster of nodes.

svrtknst 5y ago
"Crash" can mean a lot of things, and in Erlang, a crash occurs whenever your functions are unable to perform their task (for instance, there were no functions that match your input data), or when process calls time out. More obvious crashes exist as well - a remote connection hangs up, a computer catches on fire, etc. You could also use a variety of supervisor strategies or methods to detect what caused the crash.
NavinF 5y ago
Yep. Or one RPC call suddenly becomes slow because of a partial failure like a disk retrying sectors and that slows down the whole application to a crawl :(

heleninboodler 5y ago

I've always been a little confused about this "let it crash" philosophy. In my experience, the large scale services I have worked on almost never "crash." They throw 500 errors sometimes because they safely caught an exception. Would it be better to bring the whole server down and have the operating system restart it? I feel like I'm missing out on some kind of epiphany here, but it feels like the advice is to make my service crash where it wouldn't have before, which makes no sense to me. Can anyone explain it?

vvanders 5y ago
"Let it crash" is largely about restarting systems to known good state in an expanding scope. The Zen of Erlang[1] covers it really well, it's a longer read but well worth it if you want to understand a lot of the design choices of Erlang. Nearly all of the devices you use employ these approaches in one form or another. A watchdog timer[2] is a pretty simple and powerful version of this. Timeouts and retries follow a somewhat similar approach, Erlang just embraces that across the whole language. It really is a fascinating approach to a different design space(latency and reliability over throughput) through the requirements a telecom stack necessitated. [1] https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html [2] https://en.wikipedia.org/wiki/Watchdog_timer https://en.wikipedia.org/wiki/Watchdog_timer
- 1propionyl 5y ago
  Perhaps another succinct phrasing that might be more easily grokked without spending time working on Erlang systems is: "Let it rollback and retry" The way one thinks about processes in Erlang is different than how one thinks about threads in most languages. The expected behavior when you kill a process is that it will be right back up with a known good state very quickly, and you won't have to do much about it yourself because it's handled in a supervisor far from your local process. It's subtle, but it makes a huge difference. In most languages one expects a thrown unhandled exception to wreak havoc. But in Erlang graceful failure and restarting is the norm, not the exception. It's the expected behavior. Moreover, the responsibility of maintaining the process tree integrity is delegated fully to specific processes. "Business intelligence" (to ape a phrase) nodes are very effectively isolated from having to care. If they don't know how to handle such a restart, you just let them crash/be killed and restarted too.

rvr_ 5y ago

And that is also why PHP is so robust: shared nothing. Preforking. Time and memory limits per request. Uncaught errors ends only the current request and so on...

tannhaeuser 5y ago
If you mean classic PHP CGI, sure. But almost all PHP sites today run on mod_php/PHP-FPM on Apache (or equivalents on nginx) which doesn't run a fresh CGI process-per-request, but multiplexes requests onto a single PHP process instead. The overhead of parsing PHP code for each request would be brutal.
- rvr_ 5y ago
  Sure, classic CGI is one process per request, but even on php-fpm or apache prefork/mod_php each process will only work for a given maximum number of requests, thus refreshing periodicaly. Also, if any of these workers is killed (say, by the OOM-killer), none of the concurrent requests will be affected. Also, the overhead of parsing is almost zero when you have the opcache extension loaded. I'm not a huge fan of the language, but the PHP runtime wins me by its simplicity and robustness. I work daily on a codebase that has hundreds of millions of page hits per day and 100k QPS during peak hours running on LAMP with a very few servers. We face challenges like everyone else does, but we never had outages because of the languange/runtime.
bruce343434 5y ago
Ironic seeing PHP being mentioned as it almost never crashes and always attempt to chug along. Array index doesn't exist? Don't even 500, just chuck a warning out.
- megous 5y ago
  You can make PHP throw on notices if you like.
- rvr_ 5y ago
  The language/runtime is very configurable. Some codebases will set strict_type=1 plus error_reporting(E_ALL) eliminating many instances of these kind of problems / footguns.

kilodeca 5y ago

What's the purpose of having that picture at the beginning?

bruce343434 5y ago
Maybe it's a resilient tree?

deleted 5y ago

[deleted]