3 ms·

> the messages sent are kept in shared memory (which is reference-counted) Only large binaries and slices of them use reference counting, other things are copi

> the messages sent are kept in shared memory (which is reference-counted) Only large binaries and slices of them use reference counting, other things are copied. And of course, if the process is remote, everything is copied :)

toast0 7y ago
Not all refcounted binaries are large btw. It's pretty easy to construct a refc binary [1] Bin0 = <<0>>, %% 1 Bin1 = <<Bin0/binary,1,2,3>>, %% 2 Bin0 was on the process heap, but Bin1 is a refcounted binary with 256 bytes of storage. If you're putting Bin1 into ets/mnesia, use binary:copy/1 to get a clean copy that doesn't reference a lot of extra storage. [1] http://www1.erlang.org/doc/efficiency_guide/binaryhandling.html#id65352 http://www1.erlang.org/doc/efficiency_guide/binaryhandling.h...
kerkeslager 7y ago
Interesting. I'm glad to be wrong because then I learn something! I'm not sure if you have answers, but I have questions: 1. How do they determine what size to start sharing/refcounting at? Now that I'm actually considering it, it's obvious that they'd copy small stuff--an obvious lower limit would be sizeof(<reference_counter_type>) + sizeof(<pointer_type>) because smaller than that, it actually is faster/less memory to copy. But the numbers you're mentioning are higher than that--is there some metadata being stored beyond the reference count and pointer? 2. Is copied data stored in thread storage and garbage collected by that thread's garbage collector? That would be my guess. Obviously the source of truth for all these questions is the code[1], but it's been a while since I dug in there. Maybe it's time to go again. [1] https://github.com/erlang/otp https://github.com/erlang/otp
- elcritch 7y ago
  Regarding question 1, my understanding is that all non-binaries messages are copied regardless of size. Only binaries ever use shared memory. Early in BEAM development they debated optimizing shared memory for local messages but ended up not using it, partly due to the extra code complexity. Always copying messages means there’s little difference between local or remote messaging code paths and removes ref counting. Haven’t read the source myself regarding that, but it would be an interesting price to study. I’d bet optimizing local message passing would be a good feature of a BEAM implementation in Rust since the ref counting and other details would be less worrisome. Another research topic would be how BEAM would compare to say Java/Dotnet runtimes on cpu cores with lessened hardware shared memory guarantees and relying on message copying between cores instead.
- ramchip 7y ago
  1. See: https://blog.stenmans.org/theBeamBook/#_the_tagging_scheme https://blog.stenmans.org/theBeamBook/#_the_tagging_scheme (search for "64 bytes") and code: https://github.com/erlang/otp/blob/OTP-22.0.7/erts/emulator/beam/erl_binary.h#L165 https://github.com/erlang/otp/blob/OTP-22.0.7/erts/emulator/... 2. See: https://www.erlang-solutions.com/blog/erlang-garbage-collector.html https://www.erlang-solutions.com/blog/erlang-garbage-collect... There's no thread-local GC, it's really per process. For instance a scheduler thread that's running a process and hits the need to do an expensive GC may delegate the task to a dirty scheduler (another type of thread, dedicated to long blocking tasks), and go back to running other processes in the meanwhile. It's frequent for processes to move around between schedulers, they steal work from each other.

toast0 7y ago

Not all refcounted binaries are large btw. It's pretty easy to construct a refc binary [1] Bin0 = <<0>>, %% 1 Bin1 = <<Bin0/binary,1,2,3>>, %% 2 Bin0 was on the process heap, but Bin1 is a refcounted binary with 256 bytes of storage. If you're putting Bin1 into ets/mnesia, use binary:copy/1 to get a clean copy that doesn't reference a lot of extra storage. [1] http://www1.erlang.org/doc/efficiency_guide/binaryhandling.html#id65352 http://www1.erlang.org/doc/efficiency_guide/binaryhandling.h...

kerkeslager 7y ago

Interesting. I'm glad to be wrong because then I learn something! I'm not sure if you have answers, but I have questions: 1. How do they determine what size to start sharing/refcounting at? Now that I'm actually considering it, it's obvious that they'd copy small stuff--an obvious lower limit would be sizeof(<reference_counter_type>) + sizeof(<pointer_type>) because smaller than that, it actually is faster/less memory to copy. But the numbers you're mentioning are higher than that--is there some metadata being stored beyond the reference count and pointer? 2. Is copied data stored in thread storage and garbage collected by that thread's garbage collector? That would be my guess. Obviously the source of truth for all these questions is the code[1], but it's been a while since I dug in there. Maybe it's time to go again. [1] https://github.com/erlang/otp https://github.com/erlang/otp

elcritch 7y ago
Regarding question 1, my understanding is that all non-binaries messages are copied regardless of size. Only binaries ever use shared memory. Early in BEAM development they debated optimizing shared memory for local messages but ended up not using it, partly due to the extra code complexity. Always copying messages means there’s little difference between local or remote messaging code paths and removes ref counting. Haven’t read the source myself regarding that, but it would be an interesting price to study. I’d bet optimizing local message passing would be a good feature of a BEAM implementation in Rust since the ref counting and other details would be less worrisome. Another research topic would be how BEAM would compare to say Java/Dotnet runtimes on cpu cores with lessened hardware shared memory guarantees and relying on message copying between cores instead.
ramchip 7y ago
1. See: https://blog.stenmans.org/theBeamBook/#_the_tagging_scheme https://blog.stenmans.org/theBeamBook/#_the_tagging_scheme (search for "64 bytes") and code: https://github.com/erlang/otp/blob/OTP-22.0.7/erts/emulator/beam/erl_binary.h#L165 https://github.com/erlang/otp/blob/OTP-22.0.7/erts/emulator/... 2. See: https://www.erlang-solutions.com/blog/erlang-garbage-collector.html https://www.erlang-solutions.com/blog/erlang-garbage-collect... There's no thread-local GC, it's really per process. For instance a scheduler thread that's running a process and hits the need to do an expensive GC may delegate the task to a dirty scheduler (another type of thread, dedicated to long blocking tasks), and go back to running other processes in the meanwhile. It's frequent for processes to move around between schedulers, they steal work from each other.