4 ms·

It's pretty clear that Joe Armstrong respecting that the speed of light is a thing and that data locality/data gravity are real is starting to pay off in big wa

It's pretty clear that Joe Armstrong respecting that the speed of light is a thing and that data locality/data gravity are real is starting to pay off in big ways. I do wonder if maybe streaming large data chunks over Erlang distribution might be a problem and a secondary data channel (e.g. over udp or sctp) might be worth playing with. Looking forward to NX transformations that take distributed training next level.

bmitc 4y ago
> It's pretty clear that Joe Armstrong respecting that the speed of light is a thing and that data locality/data gravity are real is starting to pay off in big ways. I'm familiar with Joe Armstrong and Erlang/Elixir, but do you have a particular reference in mind where he was specifically discussing this? Is it one of his papers or talks? Just looking for another interesting thing Joe Armstrong said or thought. :)
- OkayPhysicist 4y ago
  I think he said it a few times, but "The Mess We're In" features it prominently.
- throwawaymaths 4y ago
  I don't have a reference offhand, but I have seen it. It's mostly a vibe. Remember that Joe was a physicist before a programmer: the synchronicity problem is pervasive in the design of the platform. Local immediate access to data is generally a special cased situation via an escape hatch with tons of big red warning signs.
- POiNTx 4y ago
  Right here, but the entire talk is great: https://youtu.be/lKXe3HUG2l4?t=1431 https://youtu.be/lKXe3HUG2l4?t=1431
Ankhers 4y ago
> I do wonder if maybe streaming large data chunks over Erlang distribution might be a problem and a secondary data channel (e.g. over udp or sctp) might be worth playing with. You may want to take a look at the partisan[0] library written in Erlang. It is basically that, a reimagination of distributed Erlang, except that it can be multiplexed over multiple connections. [0] - https://github.com/lasp-lang/partisan/ https://github.com/lasp-lang/partisan/
- throwawaymaths 4y ago
  Yeah but partisan gives you a "ton of stuff you might not need" plus the point is to treat distribution as a control plane and separate concerns from the data plane. There used to be things to worry about using Erlang distribution in general -- irrespective of backend, iirc, like HOL blocking (I think those are resolved now).

bmitc 4y ago

> It's pretty clear that Joe Armstrong respecting that the speed of light is a thing and that data locality/data gravity are real is starting to pay off in big ways. I'm familiar with Joe Armstrong and Erlang/Elixir, but do you have a particular reference in mind where he was specifically discussing this? Is it one of his papers or talks? Just looking for another interesting thing Joe Armstrong said or thought. :)

OkayPhysicist 4y ago
I think he said it a few times, but "The Mess We're In" features it prominently.
throwawaymaths 4y ago
I don't have a reference offhand, but I have seen it. It's mostly a vibe. Remember that Joe was a physicist before a programmer: the synchronicity problem is pervasive in the design of the platform. Local immediate access to data is generally a special cased situation via an escape hatch with tons of big red warning signs.
POiNTx 4y ago
Right here, but the entire talk is great: https://youtu.be/lKXe3HUG2l4?t=1431 https://youtu.be/lKXe3HUG2l4?t=1431

Ankhers 4y ago

> I do wonder if maybe streaming large data chunks over Erlang distribution might be a problem and a secondary data channel (e.g. over udp or sctp) might be worth playing with. You may want to take a look at the partisan[0] library written in Erlang. It is basically that, a reimagination of distributed Erlang, except that it can be multiplexed over multiple connections. [0] - https://github.com/lasp-lang/partisan/ https://github.com/lasp-lang/partisan/

throwawaymaths 4y ago
Yeah but partisan gives you a "ton of stuff you might not need" plus the point is to treat distribution as a control plane and separate concerns from the data plane. There used to be things to worry about using Erlang distribution in general -- irrespective of backend, iirc, like HOL blocking (I think those are resolved now).