4 ms·

I've been learning Elixir and liking it very much. One question I have, though, coming from Clojure, is how to quantify the cost of sending large data structur

by lukev 11y ago

I've been learning Elixir and liking it very much. One question I have, though, coming from Clojure, is how to quantify the cost of sending large data structures between processes. I'm so used to shared-memory concurrency that it isn't clear to me how to efficiently write an Elixir/Erlang program in which multiple processes need to have (read-only) access to the same data. Are there any good practices for that, or ways of thinking about how much it will cost to copy the data between processes?

vezzy-fnord 11y ago
ETS?
- seiji 11y ago
  Yeah, ets is the solution when you need cross-process "global variables" in Erlang.
- ryeguy 11y ago
  ETS still gives you a copy of the data when you retrieve it, so it doesn't really help there.
deleted 11y ago
[deleted]
- lukev 11y ago
  Yes. They're also very isolated from eachother (compared to processes in most other languages) and, notably, do not share the same object heap. Hence my question.
  - adambrod 11y ago
    Oh gotcha, sorry didn't read it thoroughly enough :)
  - andrewmutz 11y ago
    I believe large objects are on a shared heap. Source: https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html https://hamidreza-s.github.io/erlang%20garbage%20collection%...
rubiquity 11y ago
Here's a few thoughts on sending large data between processes: 1) Don't unless you really have to. Does the receiving process really need all that data to do its job? 2) Define big? Most database adapters are multi-process so lots of message passing happens. I've had some large database rows (122 columns (sigh. I know) per row) and sending those between hasn't been a problem. 3) Is the large data a binary? In that case, binaries over 56 bytes are reference counted and not copied between processes. The binary will be GC'ed after all references are gone. 4) Measure measure measure. You don't know that you have a problem unless you measure.
andrewmutz 11y ago
I am not an Elixir expert, but as I understand it, passing data between processes is pass-by-value but copy-on-write, if the processes are within the same BEAM instance.
lectrick 11y ago
What's your impression of Elixir coming from Clojure, other than that? I wanted to get into Clojure but I really wanted a language that almost entirely forced me to forego mutability (due to the large class of bugs I've seen in the past because of it) and Elixir seemed to provide that, Clojure did not, it was more live-and-let-mutability-live... and I didn't like that. Regarding your question, it's my understanding that (again, due to everything being immutable), larger data sent between nodes in the same BEAM VM is not copied, it is reference-counted and garbage-collected and any changes are copy-on-write but only for that part of the data structure (due to implementation of Persistent Data Structures underneath). One thing I'm VERY curious about with regards to Clojure is Datomic. Elixir (IMHO) could really use a fully queryable, immutable DB. After I drank the immutable kool-aid, it was jarring to have to think about mutation in global state again whenever I touch a database, to the point that I'm considering foregoing SQL as long as possible and just dealing with Mnesia or Basho's Riak and writing implementation-agnostic wrappers around my datastores (the latter is probably a good idea anyway). Mutexes need to die in a fire. ;)

by lukev 11y ago

vezzy-fnord 11y ago

ETS?

seiji 11y ago
Yeah, ets is the solution when you need cross-process "global variables" in Erlang.
ryeguy 11y ago
ETS still gives you a copy of the data when you retrieve it, so it doesn't really help there.

deleted 11y ago

[deleted]

lukev 11y ago
Yes. They're also very isolated from eachother (compared to processes in most other languages) and, notably, do not share the same object heap. Hence my question.
- adambrod 11y ago
  Oh gotcha, sorry didn't read it thoroughly enough :)
- andrewmutz 11y ago
  I believe large objects are on a shared heap. Source: https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-garbage-collection-details-and-why-it-matters.html https://hamidreza-s.github.io/erlang%20garbage%20collection%...

rubiquity 11y ago

Here's a few thoughts on sending large data between processes: 1) Don't unless you really have to. Does the receiving process really need all that data to do its job? 2) Define big? Most database adapters are multi-process so lots of message passing happens. I've had some large database rows (122 columns (sigh. I know) per row) and sending those between hasn't been a problem. 3) Is the large data a binary? In that case, binaries over 56 bytes are reference counted and not copied between processes. The binary will be GC'ed after all references are gone. 4) Measure measure measure. You don't know that you have a problem unless you measure.

andrewmutz 11y ago

I am not an Elixir expert, but as I understand it, passing data between processes is pass-by-value but copy-on-write, if the processes are within the same BEAM instance.

lectrick 11y ago

What's your impression of Elixir coming from Clojure, other than that? I wanted to get into Clojure but I really wanted a language that almost entirely forced me to forego mutability (due to the large class of bugs I've seen in the past because of it) and Elixir seemed to provide that, Clojure did not, it was more live-and-let-mutability-live... and I didn't like that. Regarding your question, it's my understanding that (again, due to everything being immutable), larger data sent between nodes in the same BEAM VM is not copied, it is reference-counted and garbage-collected and any changes are copy-on-write but only for that part of the data structure (due to implementation of Persistent Data Structures underneath). One thing I'm VERY curious about with regards to Clojure is Datomic. Elixir (IMHO) could really use a fully queryable, immutable DB. After I drank the immutable kool-aid, it was jarring to have to think about mutation in global state again whenever I touch a database, to the point that I'm considering foregoing SQL as long as possible and just dealing with Mnesia or Basho's Riak and writing implementation-agnostic wrappers around my datastores (the latter is probably a good idea anyway). Mutexes need to die in a fire. ;)