3 ms·
> Second, we passed one particularly large data structure from a manager to a pool of dedicated worker processes. This meant we were reincurring the memory cost
by davidw 4y ago
> Second, we passed one particularly large data structure from a manager to a pool of dedicated worker processes. This meant we were reincurring the memory cost of this data structure for each worker process. We couldn't eliminate the repetition, but reducing the data to its bare essentials before passing it down to the workers minimizes that cost.
Hard to say without knowing much about the data in question, but my recollection is that large Erlang/Elixir/BEAM "binaries" are actually not copied around. That might be a strategy for sharing larger things in some cases.
Marshalling data is pretty easy in Erlang:
2> Bin = erlang:term_to_binary([1, 2, 3]).
<<131,107,0,3,1,2,3>>
3> erlang:binary_to_term(Bin).
[1,2,3]
- realcorvus 4y agoIf the data does not change, persistent_term is useful as well
- lvass 4y agoJust confirmed it works. iex>term = for _ <- 0..1000000, into: [], do: :rand.uniform(0x1111111) iex>bin = :erlang.term_to_binary(term) iex>for _ <- 0..1000, do: spawn(fn -> x = bin; :timer.sleep(1000000) end) # Memory usage exploded in line below iex>for _ <- 0..1000, do: spawn(fn -> x = term; :timer.sleep(1000000) end)
- depr 4y ago"large binaries" is >64 bytes. https://www.erlang.org/doc/efficiency_guide/binaryhandling.html#refc-binaries https://www.erlang.org/doc/efficiency_guide/binaryhandling.h...