4 ms·

Flow/Broadway + how nice Ecto is already had me thinking, "Elixir is super interesting for data engineering." Explorer and Livebook have me thinking, "Elixir ha

Flow/Broadway + how nice Ecto is already had me thinking, "Elixir is super interesting for data engineering." Explorer and Livebook have me thinking, "Elixir has the best shot at unseating Python."

3jckd 4y ago
It's interesting what/when/if will unsettle Python. And what the adoption would be like. Julia has been <designed> to unsettle Python in the data space but to no avail. ¯\_(ツ)_/¯
- jasonpbecker 4y ago
  Julia was designed to unsettle a creeping set of python in the domain of Perl, C++, and Matlab in scientific computing and various quant analysis that’s pretty distant, IMO, from what industry data engineering, ML, and analytics work was doing.
tfp137 4y ago
I thought it would be Clojure, but that was 10 years ago and Python is still (inexplicably?) dominant.
sergiomattei 4y ago
Doubt it. Elixir is a leaky abstraction over Erlang, so OTP concepts are a huge barrier of entry for non-software engineers. e.g. to this day starting a small library requires learning supervisors, genservers, etc. Unless you’re doing a one off .exs, you’re going to be really confused by what “mix new” does. Saying this as a passionate Elixirist — there’s nothing wrong with the leaky abstraction part, it’s what makes it powerful — but dethroning Python isn’t happening anytime soon.
- josevalim 4y ago
  I am not sure I agree with the OTP bits. We often tend to compare having to learn OTP with “not having to learn OTP” but the reality is that we need to compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes). Of course, one can say: “I wouldn’t use threads in other languages” but that’s potentially a bigger problem, as you end up with software which can’t easily exploit concurrency (which is especially important in the data domain). I still think what you said has some truth to it, but I think the frame of reference is a bit off. I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :) PS: “mix new” does not generate supervision trees by default.
  - sergiomattei 4y ago
    > compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes) That’s a very fair point, but is the average data scientist going to be learning this or offloading that task to a software engineer? > I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :) Agreed —- I’ve been using Elixir for a bit and am only now learning about supervision trees and genservers. It’s such an exciting part of the language. > PS: “mix new” does not generate supervision trees by default. Yup and that was confusing for me as a beginner — I recall being confused over where the entrypoint of the app was. Coming from a language like Python. Hi José, huge fan. Thanks for dropping by this thread!
- ravi-delia 4y ago
  Maybe I've just forgotten what learning Elixir was like, but I feel like OTP never gave me all that much trouble? I guess it might be a little weird, but conceptually it's not that different from how objects work and presumably Python data people do fine with that. I feel like the sticking point might be the syntax. Elixir has a very developer-friendly syntax, but the flexibility (all the ways to call functions, implicit returns, arity stuff, collection syntax, even macros) makes it way harder to just learn "incantations" and be able to read other people's code.
rch 4y ago
Factor in Rust integration via Rustler and it starts to look like a 'when not if' dynamic.

Flow/Broadway + how nice Ecto is already had me thinking, "Elixir is super interesting for data engineering." Explorer and Livebook have me thinking, "Elixir has the best shot at unseating Python."

3jckd 4y ago

It's interesting what/when/if will unsettle Python. And what the adoption would be like. Julia has been <designed> to unsettle Python in the data space but to no avail. ¯\_(ツ)_/¯

jasonpbecker 4y ago
Julia was designed to unsettle a creeping set of python in the domain of Perl, C++, and Matlab in scientific computing and various quant analysis that’s pretty distant, IMO, from what industry data engineering, ML, and analytics work was doing.

tfp137 4y ago

I thought it would be Clojure, but that was 10 years ago and Python is still (inexplicably?) dominant.

sergiomattei 4y ago

Doubt it. Elixir is a leaky abstraction over Erlang, so OTP concepts are a huge barrier of entry for non-software engineers. e.g. to this day starting a small library requires learning supervisors, genservers, etc. Unless you’re doing a one off .exs, you’re going to be really confused by what “mix new” does. Saying this as a passionate Elixirist — there’s nothing wrong with the leaky abstraction part, it’s what makes it powerful — but dethroning Python isn’t happening anytime soon.

josevalim 4y ago
I am not sure I agree with the OTP bits. We often tend to compare having to learn OTP with “not having to learn OTP” but the reality is that we need to compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes). Of course, one can say: “I wouldn’t use threads in other languages” but that’s potentially a bigger problem, as you end up with software which can’t easily exploit concurrency (which is especially important in the data domain). I still think what you said has some truth to it, but I think the frame of reference is a bit off. I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :) PS: “mix new” does not generate supervision trees by default.
- sergiomattei 4y ago
  > compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes) That’s a very fair point, but is the average data scientist going to be learning this or offloading that task to a software engineer? > I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :) Agreed —- I’ve been using Elixir for a bit and am only now learning about supervision trees and genservers. It’s such an exciting part of the language. > PS: “mix new” does not generate supervision trees by default. Yup and that was confusing for me as a beginner — I recall being confused over where the entrypoint of the app was. Coming from a language like Python. Hi José, huge fan. Thanks for dropping by this thread!
ravi-delia 4y ago
Maybe I've just forgotten what learning Elixir was like, but I feel like OTP never gave me all that much trouble? I guess it might be a little weird, but conceptually it's not that different from how objects work and presumably Python data people do fine with that. I feel like the sticking point might be the syntax. Elixir has a very developer-friendly syntax, but the flexibility (all the ways to call functions, implicit returns, arity stuff, collection syntax, even macros) makes it way harder to just learn "incantations" and be able to read other people's code.

rch 4y ago

Factor in Rust integration via Rustler and it starts to look like a 'when not if' dynamic.