6 ms·

BEAM is such an awesome and beautiful VM. Not only is concurrency at a scale unimaginable in other languages the norm for BEAM, but the isolation of running pro

by mikece 7y ago

BEAM is such an awesome and beautiful VM. Not only is concurrency at a scale unimaginable in other languages the norm for BEAM, but the isolation of running processes didn't hit me until I asked an Elixir dev about running BEAM in Docker. The confused look told me I was asking an odd question: processes in BEAM are as isolated from each other as console apps running in different VMs in VMWare server. These processes can send messages to each other but they cannot access each other's memory. It was truly a mind-blowing revelation!

kerkeslager 7y ago
> The confused look told me I was asking an odd question: processes in BEAM are as isolated from each other as console apps running in different VMs in VMWare server. These processes can send messages to each other but they cannot access each other's memory. It was truly a mind-blowing revelation! Not quite: the messages sent are kept in shared memory (which is reference-counted). Cycles can't occur due to immutability. But with everything else, you're correct. One reason for this, is that every thread has its own garbage collector! This allows multithreaded garbage collection without stop-the-world pauses. Source: Did a bunch of research into BEAM for implementing my own multithreaded GC. BEAM really is beautiful!
- vvanders 7y ago
  > This allows multithreaded garbage collection without stop-the-world pauses. This was one of the things that really floored me when I learned about how BEAM was designed. It's incredible that how all the seemingly small changes come together in a way that works really well for the problem domains Erlang was used for.
  - arthurcolle 7y ago
    Anyone interested in this subject would be extremely well-served by picking up a copy of the late, great Joe Armstrong's "Programming Erlang." Putting together a minimal "server" and then slowly evolving it into a GenServer is incredibly fun and rewarding. And then the kicker is you get hot-swapping for (almost) free! Learning about Erlang hot-swapping was probably one of my most favorite moments in my brief time using Erlang before moving onto Elixir, where you get it all right out of the box. BEAM is truly a modern engineering marvel.
  - macintux 7y ago
    This is exactly the theme of my favorite talk. Erlang has some cool features, but it's the way they enhance each other that's amazing. https://youtu.be/E18shi1qIHU https://youtu.be/E18shi1qIHU
- ramchip 7y ago
  > the messages sent are kept in shared memory (which is reference-counted) Only large binaries and slices of them use reference counting, other things are copied. And of course, if the process is remote, everything is copied :)
  - toast0 7y ago
    Not all refcounted binaries are large btw. It's pretty easy to construct a refc binary [1] Bin0 = <<0>>, %% 1 Bin1 = <<Bin0/binary,1,2,3>>, %% 2 Bin0 was on the process heap, but Bin1 is a refcounted binary with 256 bytes of storage. If you're putting Bin1 into ets/mnesia, use binary:copy/1 to get a clean copy that doesn't reference a lot of extra storage. [1] http://www1.erlang.org/doc/efficiency_guide/binaryhandling.html#id65352 http://www1.erlang.org/doc/efficiency_guide/binaryhandling.h...
  - kerkeslager 7y ago
    Interesting. I'm glad to be wrong because then I learn something! I'm not sure if you have answers, but I have questions: 1. How do they determine what size to start sharing/refcounting at? Now that I'm actually considering it, it's obvious that they'd copy small stuff--an obvious lower limit would be sizeof(<reference_counter_type>) + sizeof(<pointer_type>) because smaller than that, it actually is faster/less memory to copy. But the numbers you're mentioning are higher than that--is there some metadata being stored beyond the reference count and pointer? 2. Is copied data stored in thread storage and garbage collected by that thread's garbage collector? That would be my guess. Obviously the source of truth for all these questions is the code[1], but it's been a while since I dug in there. Maybe it's time to go again. [1] https://github.com/erlang/otp https://github.com/erlang/otp
    - elcritch 7y ago
      Regarding question 1, my understanding is that all non-binaries messages are copied regardless of size. Only binaries ever use shared memory. Early in BEAM development they debated optimizing shared memory for local messages but ended up not using it, partly due to the extra code complexity. Always copying messages means there’s little difference between local or remote messaging code paths and removes ref counting. Haven’t read the source myself regarding that, but it would be an interesting price to study. I’d bet optimizing local message passing would be a good feature of a BEAM implementation in Rust since the ref counting and other details would be less worrisome. Another research topic would be how BEAM would compare to say Java/Dotnet runtimes on cpu cores with lessened hardware shared memory guarantees and relying on message copying between cores instead.
onezerozeroone 7y ago
Wouldn't the point of running in Docker still be to guarantee version, environment, and config compatibility? Just because you run an Elixir app on your local machine doesn't mean it will work worry-free on Prod. Without Docker, you might have a different version of BEAM installed (or none at all), etc.
- marci 7y ago
  Isn't that the purpose of tools like https://erlang.mk/guide/getting_started.html#_getting_started_with_otp_releases https://erlang.mk/guide/getting_started.html#_getting_starte... ? Edit: and for cross-system dev (e.g. debian to windows) -> https://erlang.mk/guide/cross_compiling.html https://erlang.mk/guide/cross_compiling.html
  - walterstucco 7y ago
    Correct, but you still need to compile the release on a similar system. For example if your dev machine is Debian it won't work if the deploy machine is Alpine (similarly Windows builds won't work on Linux or Mac).
  - onezerozeroone 7y ago
    I think OTP Releases would still depend on the target host being configured correctly and having all dependencies installed that your application needs to run. Containers are all about consistency of the environment. For example, maybe your Elixir app wants to log out to something like Splunk or DataDog...if those agents aren't installed on the host, it doesn't matter what facilities BEAM provides. Or maybe your app depends on certain configs, libraries, or files being available at specific paths on the system. Things like that would be captured in the image your containers are running and guarantee consistency across machines and environments.
    - kungfooguru 7y ago
      Right, they do, that is why containers are still useful to releases for deployment.
didibus 7y ago
I'm not a BEAM user, so I may be wrong, but as I understand it, that's not really true. Two VMs are isolated at a much more fundamental level which matters for security and containment. For example, two BEAM process, while they don't have direct access to each other's memory in the sense of say global shared variables, still have access to the same physical memory, the same disk partition, the same file descriptors, the same sockets, etc.. So I'd imagine there's potential utility in still running inside docker, as a means to deploy the expected OS, the correct assets, the version of BEAM itself, etc. And there are reasons to isolate all that with VMs, to prevent sharing of machine resources like disk, file descriptors, sockets etc.
toast0 7y ago
> The confused look told me I was asking an odd question: processes in BEAM are as isolated from each other as console apps running in different VMs in VMWare server. These processes can send messages to each other but they cannot access each other's memory. The isolation is really not that high. erlang:process_info[1] allows one process to observe an awful lot about another. sys:get_status [2] often works (depending on software design choices) and provides even more data. I would not run two applications that needed isolation for security from each other in the same BEAM or as part of the dist network. [1] http://erlang.org/documentation/doc-5.7.4/erts-5.7.4/doc/html/erlang.html#process_info-2 http://erlang.org/documentation/doc-5.7.4/erts-5.7.4/doc/htm... [2] http://erlang.org/doc/man/sys.html#get_status-1 http://erlang.org/doc/man/sys.html#get_status-1

by mikece 7y ago

kerkeslager 7y ago

> The confused look told me I was asking an odd question: processes in BEAM are as isolated from each other as console apps running in different VMs in VMWare server. These processes can send messages to each other but they cannot access each other's memory. It was truly a mind-blowing revelation! Not quite: the messages sent are kept in shared memory (which is reference-counted). Cycles can't occur due to immutability. But with everything else, you're correct. One reason for this, is that every thread has its own garbage collector! This allows multithreaded garbage collection without stop-the-world pauses. Source: Did a bunch of research into BEAM for implementing my own multithreaded GC. BEAM really is beautiful!

vvanders 7y ago
> This allows multithreaded garbage collection without stop-the-world pauses. This was one of the things that really floored me when I learned about how BEAM was designed. It's incredible that how all the seemingly small changes come together in a way that works really well for the problem domains Erlang was used for.
- arthurcolle 7y ago
  Anyone interested in this subject would be extremely well-served by picking up a copy of the late, great Joe Armstrong's "Programming Erlang." Putting together a minimal "server" and then slowly evolving it into a GenServer is incredibly fun and rewarding. And then the kicker is you get hot-swapping for (almost) free! Learning about Erlang hot-swapping was probably one of my most favorite moments in my brief time using Erlang before moving onto Elixir, where you get it all right out of the box. BEAM is truly a modern engineering marvel.
- macintux 7y ago
  This is exactly the theme of my favorite talk. Erlang has some cool features, but it's the way they enhance each other that's amazing. https://youtu.be/E18shi1qIHU https://youtu.be/E18shi1qIHU
ramchip 7y ago
> the messages sent are kept in shared memory (which is reference-counted) Only large binaries and slices of them use reference counting, other things are copied. And of course, if the process is remote, everything is copied :)
- toast0 7y ago
  Not all refcounted binaries are large btw. It's pretty easy to construct a refc binary [1] Bin0 = <<0>>, %% 1 Bin1 = <<Bin0/binary,1,2,3>>, %% 2 Bin0 was on the process heap, but Bin1 is a refcounted binary with 256 bytes of storage. If you're putting Bin1 into ets/mnesia, use binary:copy/1 to get a clean copy that doesn't reference a lot of extra storage. [1] http://www1.erlang.org/doc/efficiency_guide/binaryhandling.html#id65352 http://www1.erlang.org/doc/efficiency_guide/binaryhandling.h...
- kerkeslager 7y ago
  Interesting. I'm glad to be wrong because then I learn something! I'm not sure if you have answers, but I have questions: 1. How do they determine what size to start sharing/refcounting at? Now that I'm actually considering it, it's obvious that they'd copy small stuff--an obvious lower limit would be sizeof(<reference_counter_type>) + sizeof(<pointer_type>) because smaller than that, it actually is faster/less memory to copy. But the numbers you're mentioning are higher than that--is there some metadata being stored beyond the reference count and pointer? 2. Is copied data stored in thread storage and garbage collected by that thread's garbage collector? That would be my guess. Obviously the source of truth for all these questions is the code[1], but it's been a while since I dug in there. Maybe it's time to go again. [1] https://github.com/erlang/otp https://github.com/erlang/otp
  - elcritch 7y ago
    Regarding question 1, my understanding is that all non-binaries messages are copied regardless of size. Only binaries ever use shared memory. Early in BEAM development they debated optimizing shared memory for local messages but ended up not using it, partly due to the extra code complexity. Always copying messages means there’s little difference between local or remote messaging code paths and removes ref counting. Haven’t read the source myself regarding that, but it would be an interesting price to study. I’d bet optimizing local message passing would be a good feature of a BEAM implementation in Rust since the ref counting and other details would be less worrisome. Another research topic would be how BEAM would compare to say Java/Dotnet runtimes on cpu cores with lessened hardware shared memory guarantees and relying on message copying between cores instead.

onezerozeroone 7y ago

Wouldn't the point of running in Docker still be to guarantee version, environment, and config compatibility? Just because you run an Elixir app on your local machine doesn't mean it will work worry-free on Prod. Without Docker, you might have a different version of BEAM installed (or none at all), etc.

marci 7y ago
Isn't that the purpose of tools like https://erlang.mk/guide/getting_started.html#_getting_started_with_otp_releases https://erlang.mk/guide/getting_started.html#_getting_starte... ? Edit: and for cross-system dev (e.g. debian to windows) -> https://erlang.mk/guide/cross_compiling.html https://erlang.mk/guide/cross_compiling.html
- walterstucco 7y ago
  Correct, but you still need to compile the release on a similar system. For example if your dev machine is Debian it won't work if the deploy machine is Alpine (similarly Windows builds won't work on Linux or Mac).
- onezerozeroone 7y ago
  I think OTP Releases would still depend on the target host being configured correctly and having all dependencies installed that your application needs to run. Containers are all about consistency of the environment. For example, maybe your Elixir app wants to log out to something like Splunk or DataDog...if those agents aren't installed on the host, it doesn't matter what facilities BEAM provides. Or maybe your app depends on certain configs, libraries, or files being available at specific paths on the system. Things like that would be captured in the image your containers are running and guarantee consistency across machines and environments.
  - kungfooguru 7y ago
    Right, they do, that is why containers are still useful to releases for deployment.

didibus 7y ago

I'm not a BEAM user, so I may be wrong, but as I understand it, that's not really true. Two VMs are isolated at a much more fundamental level which matters for security and containment. For example, two BEAM process, while they don't have direct access to each other's memory in the sense of say global shared variables, still have access to the same physical memory, the same disk partition, the same file descriptors, the same sockets, etc.. So I'd imagine there's potential utility in still running inside docker, as a means to deploy the expected OS, the correct assets, the version of BEAM itself, etc. And there are reasons to isolate all that with VMs, to prevent sharing of machine resources like disk, file descriptors, sockets etc.

toast0 7y ago

> The confused look told me I was asking an odd question: processes in BEAM are as isolated from each other as console apps running in different VMs in VMWare server. These processes can send messages to each other but they cannot access each other's memory. The isolation is really not that high. erlang:process_info[1] allows one process to observe an awful lot about another. sys:get_status [2] often works (depending on software design choices) and provides even more data. I would not run two applications that needed isolation for security from each other in the same BEAM or as part of the dist network. [1] http://erlang.org/documentation/doc-5.7.4/erts-5.7.4/doc/html/erlang.html#process_info-2 http://erlang.org/documentation/doc-5.7.4/erts-5.7.4/doc/htm... [2] http://erlang.org/doc/man/sys.html#get_status-1 http://erlang.org/doc/man/sys.html#get_status-1