4 ms·

It's incredible for IO. It's not amazing at tight loop CPU intenstive tasks, but elixir and erlang do make it really easy to take something sequential and sprea

by el_oni 3y ago

It's incredible for IO. It's not amazing at tight loop CPU intenstive tasks, but elixir and erlang do make it really easy to take something sequential and spread it across all cores (or across a cluster if you're so inclined). It does have good FFI, like rustler for the elixir is great. It makes sure rust panics unroll into elixir exceptions and that takes some of the scariness of writing them. That plus dirty schedulers means you don't have to be a good citizen and can write cpu intensive code in a faster language.

nologic01 3y ago
a platform that would make distributed computing less of challenge would be well placed today with all the data science / ML / DL / AI focus but its not clear if erlang has ever had any ambitions in this space. the main reference that shows up if you search for erlang + hpc is from 2008 https://www.researchgate.net/publication/221211398_High-performance_technical_computing_with_erlang https://www.researchgate.net/publication/221211398_High-perf...
- zambal 3y ago
  As far as I know there are no Erlang projects with that ambition, but quite some interesting things are happening in the Elixir eco system: https://github.com/elixir-nx https://github.com/elixir-nx I'm a Elixir and Erlang developer who is just starting to explore ML, but these libraries seem solid and well designed.
- el_oni 3y ago
  I beleive there is currently a PhD project in collaboration with Jose Valim (Elixir's creator) which is looking at distributed machine learning. There is also the NX project (numerical elixir) which compiles tensors to GPU code making it much faster for those particular usecases. https://github.com/elixir-nx https://github.com/elixir-nx This has things like explorer which is elixir bindings to polars, livebook which is like jupyter notebooks (with enforced execution order thang god), aswell as some machine learning libraries in axon and scholar. There has also been some work with libraries like flow and broadway which allow pipelines to be constructed with the usual syntaxt `step1 |> step2 |> step3` but to be executed across cores (or cluster). I can imagine orgs using a combination of the above tools to do some interested distrubuted data pipelining and ML.

by el_oni 3y ago

nologic01 3y ago

a platform that would make distributed computing less of challenge would be well placed today with all the data science / ML / DL / AI focus but its not clear if erlang has ever had any ambitions in this space. the main reference that shows up if you search for erlang + hpc is from 2008 https://www.researchgate.net/publication/221211398_High-performance_technical_computing_with_erlang https://www.researchgate.net/publication/221211398_High-perf...

zambal 3y ago
As far as I know there are no Erlang projects with that ambition, but quite some interesting things are happening in the Elixir eco system: https://github.com/elixir-nx https://github.com/elixir-nx I'm a Elixir and Erlang developer who is just starting to explore ML, but these libraries seem solid and well designed.
el_oni 3y ago
I beleive there is currently a PhD project in collaboration with Jose Valim (Elixir's creator) which is looking at distributed machine learning. There is also the NX project (numerical elixir) which compiles tensors to GPU code making it much faster for those particular usecases. https://github.com/elixir-nx https://github.com/elixir-nx This has things like explorer which is elixir bindings to polars, livebook which is like jupyter notebooks (with enforced execution order thang god), aswell as some machine learning libraries in axon and scholar. There has also been some work with libraries like flow and broadway which allow pipelines to be constructed with the usual syntaxt `step1 |> step2 |> step3` but to be executed across cores (or cluster). I can imagine orgs using a combination of the above tools to do some interested distrubuted data pipelining and ML.