5 ms·

How does BEAM handle system calls? If I have a million BEAM processes and each one calls stat(), what happens?

ekimekim 6y ago
I can't speak directly for BEAM but in most languages with built-in green threads (like BEAM's processes), one of two things happens: 1. If the call is a short, syncronous operation (like stat() (well, mostly), open(), getpid()), it just calls it directly in your green thread. It occupies the entire OS-level thread while it runs. 2. For long, blocking operations (eg. nanosleep(), recv(), maybe read() depending on the nature of the fd), the runtime will translate your blocking call into a callback in the main event loop, informing your green thread when data is available to read, or your timer has expired, etc. Only once the runtime knows your operation will return immediately will it actually do the call. In the meantime this frees the OS thread to run a different green thread.
cuddlecake 6y ago
What do you expect to happen in other languages, when your program makes a million calls to `stat()`? Is that something that happens often? What kind of application needs that? Is there a language that solves this neatly? (I'm genuinely curious, independent of my newbie elixir evangelism)
- rcxdude 6y ago
  git, for example, can do this on a large repository when checking for updated files (because it needs to check the modification timestamp of every file in the repo), and for a lot of common operations it's actually the bottleneck (and one reason why it is often faster on linux: the VFS layer in linux puts a lot of effort into making such operations as fast as possible, somewhat appropriately Linus was the one who did a lot of that work originally). In C it works pretty much as expected, and most of the overhead is in the kernel.
- ridiculous_fish 6y ago
  This problem doesn't arise in a 1-1 language like C, because the thread spawning will either fail or have obviously terrible performance. These languages push you into using work queues, etc. immediately. But in languages with green threads (like Go), you CAN spawn a million threads and performance will be fine, until you make a syscall. An example of how this can happen: I once wrote a Go tool that walks the filesystem. I spawned a new thread for every directory, thinking that Go only has ~N kernel threads so performance will be fine. I was shocked to see that, in this scenario, it spawns a kernel thread for every green thread!
dmitriid 6y ago
(has been a while since I touched Erlang, so: grain of salt) There are several tricks that BEAM employs to work around these problems: - BEAM has dedicated threads to I/O, and system calls will happen on those dedicated threads. So, the green thread aka processes wil not be affected - (almost [1]) every call that happens "outside erlang" (that is it calls some code implemented in C inside the VM such as regexps etc.) is re-entrant. So the VM can and will re-prioritize tasks and put processes to sleep when needed and will re-start work when the process wakes up or some external work is done and the answer is received back So in theory all those stat() calls will be scheduled and queued on the separate prioritized I/O thread, the processes will be put to sleep until the result comes back, and then they will be awoken in turn. This may cause problems with the host OS though :) [2] [1] There are definitely places where code is not re-entrant yet, because you see updates in release notes from time to time, but most code is re-entrant because of reduction counting: https://news.ycombinator.com/item?id=14440205 https://news.ycombinator.com/item?id=14440205 and https://stackoverflow.com/questions/31751766/reductions-in-the-erlang-beam-machine https://stackoverflow.com/questions/31751766/reductions-in-t... and because schedulers can steal processes from each other: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [2] A slightly unrelated anecdote: At a previous job due to some improper coding the web server would slowly accumulate up to to a few 100s of GBs of data in memory due to some long-running processes. When the processes were done, the GC would kick in and release that chunk of memory back to the OS. The OS had trouble with quickly freeing and reclaiming that memory. BEAM was meanwhile happily chugging along as if nothing has happened :D

I can't speak directly for BEAM but in most languages with built-in green threads (like BEAM's processes), one of two things happens: 1. If the call is a short, syncronous operation (like stat() (well, mostly), open(), getpid()), it just calls it directly in your green thread. It occupies the entire OS-level thread while it runs. 2. For long, blocking operations (eg. nanosleep(), recv(), maybe read() depending on the nature of the fd), the runtime will translate your blocking call into a callback in the main event loop, informing your green thread when data is available to read, or your timer has expired, etc. Only once the runtime knows your operation will return immediately will it actually do the call. In the meantime this frees the OS thread to run a different green thread.

cuddlecake 6y ago

What do you expect to happen in other languages, when your program makes a million calls to `stat()`? Is that something that happens often? What kind of application needs that? Is there a language that solves this neatly? (I'm genuinely curious, independent of my newbie elixir evangelism)

rcxdude 6y ago
git, for example, can do this on a large repository when checking for updated files (because it needs to check the modification timestamp of every file in the repo), and for a lot of common operations it's actually the bottleneck (and one reason why it is often faster on linux: the VFS layer in linux puts a lot of effort into making such operations as fast as possible, somewhat appropriately Linus was the one who did a lot of that work originally). In C it works pretty much as expected, and most of the overhead is in the kernel.
ridiculous_fish 6y ago
This problem doesn't arise in a 1-1 language like C, because the thread spawning will either fail or have obviously terrible performance. These languages push you into using work queues, etc. immediately. But in languages with green threads (like Go), you CAN spawn a million threads and performance will be fine, until you make a syscall. An example of how this can happen: I once wrote a Go tool that walks the filesystem. I spawned a new thread for every directory, thinking that Go only has ~N kernel threads so performance will be fine. I was shocked to see that, in this scenario, it spawns a kernel thread for every green thread!

dmitriid 6y ago

(has been a while since I touched Erlang, so: grain of salt) There are several tricks that BEAM employs to work around these problems: - BEAM has dedicated threads to I/O, and system calls will happen on those dedicated threads. So, the green thread aka processes wil not be affected - (almost [1]) every call that happens "outside erlang" (that is it calls some code implemented in C inside the VM such as regexps etc.) is re-entrant. So the VM can and will re-prioritize tasks and put processes to sleep when needed and will re-start work when the process wakes up or some external work is done and the answer is received back So in theory all those stat() calls will be scheduled and queued on the separate prioritized I/O thread, the processes will be put to sleep until the result comes back, and then they will be awoken in turn. This may cause problems with the host OS though :) [2] [1] There are definitely places where code is not re-entrant yet, because you see updates in release notes from time to time, but most code is re-entrant because of reduction counting: https://news.ycombinator.com/item?id=14440205 https://news.ycombinator.com/item?id=14440205 and https://stackoverflow.com/questions/31751766/reductions-in-the-erlang-beam-machine https://stackoverflow.com/questions/31751766/reductions-in-t... and because schedulers can steal processes from each other: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html https://hamidreza-s.github.io/erlang/scheduling/real-time/pr... [2] A slightly unrelated anecdote: At a previous job due to some improper coding the web server would slowly accumulate up to to a few 100s of GBs of data in memory due to some long-running processes. When the processes were done, the GC would kick in and release that chunk of memory back to the OS. The OS had trouble with quickly freeing and reclaiming that memory. BEAM was meanwhile happily chugging along as if nothing has happened :D