5 ms·

Excuse my Erlang ignorance. When he sends the "{become, F}" message, and F is a function pointer or closure, how is that dispatched across a network? Do the co

by kitd 7y ago

Excuse my Erlang ignorance. When he sends the "{become, F}" message, and F is a function pointer or closure, how is that dispatched across a network? Do the code definitions need to be present remotely too? If so, I'm not sure that quite qualifies as a "universal" server, only a server for locally defined functions. Still cool though. BTW, I've used a similar technique to create a pool of worker goroutines in Golang. Had good performance from it too.

yetihehe 7y ago
That function's code is serialized and sent to remote node along with necessary data. In erlang, if you want, you can compile whole modules from source constructed at runtime and they behave just like normal code. Typical usage is hot-loading new modules from disk or from other nodes, you load your patch on one node and distribute it over network and if you don't make something stupid, your code is updated in whole system without dropping any tcp connections or the like. Everything happens automatically and every process using old code is updated as soon as possible to new version when possible.
- manthideaal 7y ago
  if you call a function in the form module:function-name then erlang detect and use the latest compiled version of the module, if you use function-name without the module then the old version is used. Is advised to use the module:function-name in tail call position.
  - toast0 7y ago
    Sure, but if you send fun Module:function-name/arity over the network, that will be sent as EXPORT_EXT [1], and only work if the module is available on the receiver. [1] http://erlang.org/doc/apps/erts/erl_ext_dist.html#export_ext http://erlang.org/doc/apps/erts/erl_ext_dist.html#export_ext
    - 3fe9a03ccd14ca5 7y ago
      In that scenario how do you deploy new code with this method? If you use a new module you must first preinstall it?
      - yetihehe 7y ago
        If you need to install module on other node, you make a fun which installs that module (typically one function call) and execute it on other node, that fun is automatically sent and executed where you want it. It's like using remote shell (ssh), but your program is automatically sent to remote server.
      - toast0 7y ago
        I wouldn't use function sending to deploy new code in a productionish environment. It's fun to play with, but it's easier to understand a production environment if you push the code/compiled beam files as traditional files through whatever means you like (rsync/packages/whatever), and then load the updated beam files. You could do the code loading by sending a function to the node that calls code:load(Module) and the like; code:soft_purge(Module) first is a good idea to make sure you don't kill lingering processes by accident. You could go all in and have (most of) your Erlang nodes be diskless Erlang, requesting the code to run from another Erlang server when they start, and that seems like a fun adventure, but I don't know if it adds a lot of business value. I'd be happy to play with that on your dime, contact in profile ;)
      - dnautics 7y ago
        Honestly, in most scenarios, you kill the vm and let it restart with new code (just like you would in a kubernetes pod, and in fact using kubernetes pods with BEAM languages is also a valid strategy). If you must keep some state preserved, there's even ways to do it without erlang's hot code reloading strategies, I would recommend watching Dan Azuma's fantastic video (stick around for the live demo): https://www.youtube.com/watch?v=nLApFANtkHs https://www.youtube.com/watch?v=nLApFANtkHs
- jerf 7y ago
  If you've ever looked at Erlang's data model briefly and gone "what the heck is with the immutability and lack of user-defined structures, etc.", have another look at it from this point of view of distribution. It's the secret decoder ring of "why did Erlang do it that way?" With another ~25 years of collective experience, if somebody took a shot at it today I think we could make some improvements, but it was pretty far ahead of its time, and can still pull some stunts almost nothing else can, or at least not without creating "an ad-hoc, informally-specified, bug-ridden, slow implementation of half of [Erlang]", to tweak Greenspun a bit.
  - afiori 7y ago
    There was a nice programming language (unison) posted here some weeks ago that was based on pushing this to the extreme https://news.ycombinator.com/item?id=9512955 https://news.ycombinator.com/item?id=9512955 What I understood was that objects/function are referred by hash of the source/dependency tree so that updates can be propagated transparently.
deleted 7y ago
[deleted]
manthideaal 7y ago
In erlang the code is compiled to an intermmediate representation for a virtual machine, BEAM (erlang virtual machine), usually the functions you define are very short and in this case the BEAM code is sent over the network, so the universal server receives the compiled or transpiled code and executed it using the BEAM. You can compare it to a lisp server in which you send the program as a list, in lisp code is data.
- OskarS 7y ago
  Can these functions be closures that close over a bunch of data that isn't hard-coded into the functions? I don't know erlang, but imagine the erlang version of this python code: def adder(v): return lambda x: v + x Can I then send the result of (say) "adder(5)" over the network and have it work?
  - manthideaal 7y ago
    See theses two answers: https://stackoverflow.com/questions/39396491/is-there-a-way-to-send-a-closure-to-remote-node-in-erlang https://stackoverflow.com/questions/39396491/is-there-a-way-... https://stackoverflow.com/questions/39255471/what-is-the-point-of-spawnnode-fun-on-erlang-if-node-has-to-have-the-same-mo https://stackoverflow.com/questions/39255471/what-is-the-poi...
  - toast0 7y ago
    Sure: On a erlang shell in node A, do: register('test', self()). receive F -> ok end. (this gives the shell a registered name, and then waits for a message) On a debug shell on node A do: AddrFactory = fun (X) -> fun (Y) -> Y + X end end. {test, NodeA} ! AddrFactory(5). Then, back on node A, do F(10). And you'll get 15 For more fun, you could have node A send the value to go into the closure, etc. NodeA is something like 'demo@hosta.example.org' NodeA would be started with something like erl -name demo@hosta.example.org -setcookie demo and NodeB started like erl -name demo@hostb.example.org -setcookie demo
    - OskarS 7y ago
avinassh 7y ago
> BTW, I've used a similar technique to create a pool of worker goroutines in Golang. Had good performance from it too. Do you mind telling more about this? Or link, if it’s on Github
- bananadonkey 7y ago
  Not the OP, but I have used https://github.com/dradtke/distchan https://github.com/dradtke/distchan in prod where portability outweighed the distribution capabilities of Erlang.
- kitd 7y ago
  Sorry, it's not public, but it was based on this: http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/ http://marcio.io/2015/07/handling-1-million-requests-per-min... but made more generic. The 'task' passed to each worker was a closure that was executed by the worker.
dnautics 7y ago
In erlang you can serialize anything. I can write a lambda and email it to you and you can put it in your VM and assuming you have the correct dependencies, it will probably work. Dependencies are also just atom (universal, global, enum) references, so there's no pointers involved.

by kitd 7y ago

yetihehe 7y ago

That function's code is serialized and sent to remote node along with necessary data. In erlang, if you want, you can compile whole modules from source constructed at runtime and they behave just like normal code. Typical usage is hot-loading new modules from disk or from other nodes, you load your patch on one node and distribute it over network and if you don't make something stupid, your code is updated in whole system without dropping any tcp connections or the like. Everything happens automatically and every process using old code is updated as soon as possible to new version when possible.

manthideaal 7y ago
if you call a function in the form module:function-name then erlang detect and use the latest compiled version of the module, if you use function-name without the module then the old version is used. Is advised to use the module:function-name in tail call position.
- toast0 7y ago
  Sure, but if you send fun Module:function-name/arity over the network, that will be sent as EXPORT_EXT [1], and only work if the module is available on the receiver. [1] http://erlang.org/doc/apps/erts/erl_ext_dist.html#export_ext http://erlang.org/doc/apps/erts/erl_ext_dist.html#export_ext
  - 3fe9a03ccd14ca5 7y ago
    In that scenario how do you deploy new code with this method? If you use a new module you must first preinstall it?
    - yetihehe 7y ago
      If you need to install module on other node, you make a fun which installs that module (typically one function call) and execute it on other node, that fun is automatically sent and executed where you want it. It's like using remote shell (ssh), but your program is automatically sent to remote server.
    - toast0 7y ago
      I wouldn't use function sending to deploy new code in a productionish environment. It's fun to play with, but it's easier to understand a production environment if you push the code/compiled beam files as traditional files through whatever means you like (rsync/packages/whatever), and then load the updated beam files. You could do the code loading by sending a function to the node that calls code:load(Module) and the like; code:soft_purge(Module) first is a good idea to make sure you don't kill lingering processes by accident. You could go all in and have (most of) your Erlang nodes be diskless Erlang, requesting the code to run from another Erlang server when they start, and that seems like a fun adventure, but I don't know if it adds a lot of business value. I'd be happy to play with that on your dime, contact in profile ;)
    - dnautics 7y ago
      Honestly, in most scenarios, you kill the vm and let it restart with new code (just like you would in a kubernetes pod, and in fact using kubernetes pods with BEAM languages is also a valid strategy). If you must keep some state preserved, there's even ways to do it without erlang's hot code reloading strategies, I would recommend watching Dan Azuma's fantastic video (stick around for the live demo): https://www.youtube.com/watch?v=nLApFANtkHs https://www.youtube.com/watch?v=nLApFANtkHs
jerf 7y ago
If you've ever looked at Erlang's data model briefly and gone "what the heck is with the immutability and lack of user-defined structures, etc.", have another look at it from this point of view of distribution. It's the secret decoder ring of "why did Erlang do it that way?" With another ~25 years of collective experience, if somebody took a shot at it today I think we could make some improvements, but it was pretty far ahead of its time, and can still pull some stunts almost nothing else can, or at least not without creating "an ad-hoc, informally-specified, bug-ridden, slow implementation of half of [Erlang]", to tweak Greenspun a bit.
- afiori 7y ago
  There was a nice programming language (unison) posted here some weeks ago that was based on pushing this to the extreme https://news.ycombinator.com/item?id=9512955 https://news.ycombinator.com/item?id=9512955 What I understood was that objects/function are referred by hash of the source/dependency tree so that updates can be propagated transparently.

deleted 7y ago

[deleted]

manthideaal 7y ago

In erlang the code is compiled to an intermmediate representation for a virtual machine, BEAM (erlang virtual machine), usually the functions you define are very short and in this case the BEAM code is sent over the network, so the universal server receives the compiled or transpiled code and executed it using the BEAM. You can compare it to a lisp server in which you send the program as a list, in lisp code is data.

OskarS 7y ago
Can these functions be closures that close over a bunch of data that isn't hard-coded into the functions? I don't know erlang, but imagine the erlang version of this python code: def adder(v): return lambda x: v + x Can I then send the result of (say) "adder(5)" over the network and have it work?
- manthideaal 7y ago
  See theses two answers: https://stackoverflow.com/questions/39396491/is-there-a-way-to-send-a-closure-to-remote-node-in-erlang https://stackoverflow.com/questions/39396491/is-there-a-way-... https://stackoverflow.com/questions/39255471/what-is-the-point-of-spawnnode-fun-on-erlang-if-node-has-to-have-the-same-mo https://stackoverflow.com/questions/39255471/what-is-the-poi...
- toast0 7y ago
  Sure: On a erlang shell in node A, do: register('test', self()). receive F -> ok end. (this gives the shell a registered name, and then waits for a message) On a debug shell on node A do: AddrFactory = fun (X) -> fun (Y) -> Y + X end end. {test, NodeA} ! AddrFactory(5). Then, back on node A, do F(10). And you'll get 15 For more fun, you could have node A send the value to go into the closure, etc. NodeA is something like 'demo@hosta.example.org' NodeA would be started with something like erl -name demo@hosta.example.org -setcookie demo and NodeB started like erl -name demo@hostb.example.org -setcookie demo
  - OskarS 7y ago

avinassh 7y ago

> BTW, I've used a similar technique to create a pool of worker goroutines in Golang. Had good performance from it too. Do you mind telling more about this? Or link, if it’s on Github

bananadonkey 7y ago
Not the OP, but I have used https://github.com/dradtke/distchan https://github.com/dradtke/distchan in prod where portability outweighed the distribution capabilities of Erlang.
kitd 7y ago
Sorry, it's not public, but it was based on this: http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/ http://marcio.io/2015/07/handling-1-million-requests-per-min... but made more generic. The 'task' passed to each worker was a closure that was executed by the worker.

dnautics 7y ago

In erlang you can serialize anything. I can write a lambda and email it to you and you can put it in your VM and assuming you have the correct dependencies, it will probably work. Dependencies are also just atom (universal, global, enum) references, so there's no pointers involved.