5 ms·

Explorer – A library to bring dataframes to Elixir

124 points by ahamez 4y ago

ahamez 4y ago
Added '(Elixir and Polars)' to the title as 'Introducing Explorer' alone is not clear enough in my opinion.
deleted 4y ago
[deleted]
peoplefromibiza 4y ago
Amazing! Rustler precompiled[1] is exactly what I was looking for to start writing NIFs in Rust. Having to bring in the entire Rust toolchain stopped me every time I thought about it - not mentioning that at my current day job I'm not completely at liberty of deciding what tools I can/cannot use. Combined with Livebook[2] this is the perfect combo to lure into the Elixir ecosystem my fellow data scientists colleagues. [1] https://dashbit.co/blog/rustler-precompiled https://dashbit.co/blog/rustler-precompiled [2] https://livebook.dev/ https://livebook.dev/
- pdimitar 4y ago
  > Having to bring in the entire Rust toolchain I relate to the general sentiment, mind you, but Rust is brainlessly easy to install. But even if you seriously don't want to even do that (two shell commands) you can still pull a Docker image and work inside it -- but that's a bit more involved, granted. I applaud the Rustler Precompiled effort because apparently it lowers the barrier to entry for many -- yourself included -- but at least for me installing Rust was never an actual problem. As for Explorer, I have a few friends that curse at Python and Pandas every week, I might be able to "sell" them this project instead!
  - peoplefromibiza 4y ago
    > I relate to the general sentiment, mind you, but Rust is brainlessly easy to install. True, my issue is with releasing libraries that need Rust. It forces the users of the library to install Rust, adding friction, and possibly deal with issues/errors that they are not used to. Especially on heterogeneous systems: library developed on Linux, user is on Mac/Windows. One example is my day job where the CI platform is controlled by another department and you can't create your own custom Dockerfile(s), because "security". On the developers' side Rust is completely fine, in fact IMO Python is much more problematic, despite being an order of magnitude more popular.
    - pdimitar 4y ago
      Oh yeah, agreed, distribution becomes little more involved. In these cases Rustler Precompiled is a complete bliss indeed. ^_^
  - umren 4y ago
    Your friends haven't tried R and Julia? I don't see how elixir can replace python any time soon because it's DS ecosystem is lacking rn sadly
    - pdimitar 4y ago
      Oh, he definitely tried R but was not very impressed. Says it's only moderately better than Python's otherwise excellent bindings to a lot of libraries, and that he had no patience for Julia. He isn't exactly a programmer, he's just a very practical guy that is open to learning some programming to get his stuff done quicker and with less mistakes but that's about it. He has a short fuse for ecosystem deficiencies -- tbf I think we the programmers have to learn from such people. :D And yeah, Elixir's scene in this area is just now starting. The NX project opened a lot of possible doors but the community just started stepping up. It's going to be a while, granted, but I am optimistic because Elixir's overall community values quality and centralization (with all the sometimes problematic collaboration efforts that entails) over rushing to have 10 competing implementations. Time will tell. I have no horse in the race either way, just sharing various observations really.
    - jasonpbecker 4y ago
      As an R user who can't stand working in Python, where Elixir I think has the most potential is not the data science part but data engineering. The whole world of ELT/ETL pipeline and workflow tools written in Python is a mess IMO, and Broadway/Flow + Ecto + what's going on with Explorer etc I think can quickly replace a whole mess of python/scala/EMR type workflows in a language and environment that feels way better suited to the problem space, in my experience.
      - pdimitar 4y ago
        Yeah, can confirm, I did a ton of Elixir work in the ETL space and it's amazingly well-suited for it. If you don't get an OCD over the fact that it obviously can't work as fast as Rust (realistically it's likely anywhere from 30x to 100x slower even) then the ergonomic and intuitive syntax combined with the transparent parallelism create an all-around win. So far I've never regretted using Elixir for ETL, but I do confess that for two projects I was mighty tempted to try a rewrite in Rust because ingesting several dozens of billions of records in a dynamic language isn't exactly fast. But, again, the ergonomics trump everything else 99% of the time for me. Plus for one-off ingestion tasks nobody cares if it takes 30 minutes or the whole night.
newskfm 4y ago
[flagged]
rapnie 4y ago
> we’ve been using Elixir in production since 2018. We switched over to LiveView in early 2020 and haven’t looked back. What does "switched to LiveView" mean? I thought LiveView was some collaborative, live documentation / interactive tutorial tool. Or does it provide a whole programming environment and/or development process?
- Existenceblinks 4y ago
  I think you mean https://livebook.dev/ https://livebook.dev/ which is built on top of LiveView. LV can be used instead of "Dead View" which is a non diff-tracking view.
  - rapnie 4y ago
    Ah stupid. Thank you. I will try to get my brain in higher gear today.
jasonpbecker 4y ago
Flow/Broadway + how nice Ecto is already had me thinking, "Elixir is super interesting for data engineering." Explorer and Livebook have me thinking, "Elixir has the best shot at unseating Python."
- 3jckd 4y ago
  It's interesting what/when/if will unsettle Python. And what the adoption would be like. Julia has been <designed> to unsettle Python in the data space but to no avail. ¯\_(ツ)_/¯
  - jasonpbecker 4y ago
    Julia was designed to unsettle a creeping set of python in the domain of Perl, C++, and Matlab in scientific computing and various quant analysis that’s pretty distant, IMO, from what industry data engineering, ML, and analytics work was doing.
- tfp137 4y ago
  I thought it would be Clojure, but that was 10 years ago and Python is still (inexplicably?) dominant.
- sergiomattei 4y ago
  Doubt it. Elixir is a leaky abstraction over Erlang, so OTP concepts are a huge barrier of entry for non-software engineers. e.g. to this day starting a small library requires learning supervisors, genservers, etc. Unless you’re doing a one off .exs, you’re going to be really confused by what “mix new” does. Saying this as a passionate Elixirist — there’s nothing wrong with the leaky abstraction part, it’s what makes it powerful — but dethroning Python isn’t happening anytime soon.
  - josevalim 4y ago
    I am not sure I agree with the OTP bits. We often tend to compare having to learn OTP with “not having to learn OTP” but the reality is that we need to compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes). Of course, one can say: “I wouldn’t use threads in other languages” but that’s potentially a bigger problem, as you end up with software which can’t easily exploit concurrency (which is especially important in the data domain). I still think what you said has some truth to it, but I think the frame of reference is a bit off. I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :) PS: “mix new” does not generate supervision trees by default.
pbowyer 4y ago
> Second, there are callbacks! The second argument in DataFrame.filter/2 takes a callback function against the dataframe. There’s some cool stuff we can do with that, and I think callbacks against the dataframe are a natural way to work. I'm not enamoured with how DataFrame.filter/2 reads compared to the dplyr example. I've written very little R but read code by others and find dpylr excellent for that; and I use a JS 'port' of dplyr [1] for projects. I'm in the "hate pandas" camp and find it painful to both read and write - and these callback functions are coming too close to Pandas for my liking. I hope an alternative can be found 1. https://pbeshai.github.io/tidy/ https://pbeshai.github.io/tidy/
- throwamon 4y ago
  For further inspiration, this is a pretty good-looking "dplyr for Python": https://github.com/machow/siuba https://github.com/machow/siuba There's precedent in Ecto for a "magic" operator (`^` if I'm not mistaken), so it wouldn't be a stretch to implement it here as well.

124 points by ahamez 4y ago

ahamez 4y ago

Added '(Elixir and Polars)' to the title as 'Introducing Explorer' alone is not clear enough in my opinion.

deleted 4y ago

[deleted]

peoplefromibiza 4y ago

Amazing! Rustler precompiled[1] is exactly what I was looking for to start writing NIFs in Rust. Having to bring in the entire Rust toolchain stopped me every time I thought about it - not mentioning that at my current day job I'm not completely at liberty of deciding what tools I can/cannot use. Combined with Livebook[2] this is the perfect combo to lure into the Elixir ecosystem my fellow data scientists colleagues. [1] https://dashbit.co/blog/rustler-precompiled https://dashbit.co/blog/rustler-precompiled [2] https://livebook.dev/ https://livebook.dev/

pdimitar 4y ago
> Having to bring in the entire Rust toolchain I relate to the general sentiment, mind you, but Rust is brainlessly easy to install. But even if you seriously don't want to even do that (two shell commands) you can still pull a Docker image and work inside it -- but that's a bit more involved, granted. I applaud the Rustler Precompiled effort because apparently it lowers the barrier to entry for many -- yourself included -- but at least for me installing Rust was never an actual problem. As for Explorer, I have a few friends that curse at Python and Pandas every week, I might be able to "sell" them this project instead!
- peoplefromibiza 4y ago
  > I relate to the general sentiment, mind you, but Rust is brainlessly easy to install. True, my issue is with releasing libraries that need Rust. It forces the users of the library to install Rust, adding friction, and possibly deal with issues/errors that they are not used to. Especially on heterogeneous systems: library developed on Linux, user is on Mac/Windows. One example is my day job where the CI platform is controlled by another department and you can't create your own custom Dockerfile(s), because "security". On the developers' side Rust is completely fine, in fact IMO Python is much more problematic, despite being an order of magnitude more popular.
  - pdimitar 4y ago
    Oh yeah, agreed, distribution becomes little more involved. In these cases Rustler Precompiled is a complete bliss indeed. ^_^
- umren 4y ago
  Your friends haven't tried R and Julia? I don't see how elixir can replace python any time soon because it's DS ecosystem is lacking rn sadly
  - pdimitar 4y ago
    Oh, he definitely tried R but was not very impressed. Says it's only moderately better than Python's otherwise excellent bindings to a lot of libraries, and that he had no patience for Julia. He isn't exactly a programmer, he's just a very practical guy that is open to learning some programming to get his stuff done quicker and with less mistakes but that's about it. He has a short fuse for ecosystem deficiencies -- tbf I think we the programmers have to learn from such people. :D And yeah, Elixir's scene in this area is just now starting. The NX project opened a lot of possible doors but the community just started stepping up. It's going to be a while, granted, but I am optimistic because Elixir's overall community values quality and centralization (with all the sometimes problematic collaboration efforts that entails) over rushing to have 10 competing implementations. Time will tell. I have no horse in the race either way, just sharing various observations really.
  - jasonpbecker 4y ago
    As an R user who can't stand working in Python, where Elixir I think has the most potential is not the data science part but data engineering. The whole world of ELT/ETL pipeline and workflow tools written in Python is a mess IMO, and Broadway/Flow + Ecto + what's going on with Explorer etc I think can quickly replace a whole mess of python/scala/EMR type workflows in a language and environment that feels way better suited to the problem space, in my experience.
    - pdimitar 4y ago
      Yeah, can confirm, I did a ton of Elixir work in the ETL space and it's amazingly well-suited for it. If you don't get an OCD over the fact that it obviously can't work as fast as Rust (realistically it's likely anywhere from 30x to 100x slower even) then the ergonomic and intuitive syntax combined with the transparent parallelism create an all-around win. So far I've never regretted using Elixir for ETL, but I do confess that for two projects I was mighty tempted to try a rewrite in Rust because ingesting several dozens of billions of records in a dynamic language isn't exactly fast. But, again, the ergonomics trump everything else 99% of the time for me. Plus for one-off ingestion tasks nobody cares if it takes 30 minutes or the whole night.

newskfm 4y ago

[flagged]

rapnie 4y ago

> we’ve been using Elixir in production since 2018. We switched over to LiveView in early 2020 and haven’t looked back. What does "switched to LiveView" mean? I thought LiveView was some collaborative, live documentation / interactive tutorial tool. Or does it provide a whole programming environment and/or development process?

Existenceblinks 4y ago
I think you mean https://livebook.dev/ https://livebook.dev/ which is built on top of LiveView. LV can be used instead of "Dead View" which is a non diff-tracking view.
- rapnie 4y ago
  Ah stupid. Thank you. I will try to get my brain in higher gear today.

jasonpbecker 4y ago

Flow/Broadway + how nice Ecto is already had me thinking, "Elixir is super interesting for data engineering." Explorer and Livebook have me thinking, "Elixir has the best shot at unseating Python."

3jckd 4y ago
It's interesting what/when/if will unsettle Python. And what the adoption would be like. Julia has been <designed> to unsettle Python in the data space but to no avail. ¯\_(ツ)_/¯
- jasonpbecker 4y ago
  Julia was designed to unsettle a creeping set of python in the domain of Perl, C++, and Matlab in scientific computing and various quant analysis that’s pretty distant, IMO, from what industry data engineering, ML, and analytics work was doing.
tfp137 4y ago
I thought it would be Clojure, but that was 10 years ago and Python is still (inexplicably?) dominant.
sergiomattei 4y ago
Doubt it. Elixir is a leaky abstraction over Erlang, so OTP concepts are a huge barrier of entry for non-software engineers. e.g. to this day starting a small library requires learning supervisors, genservers, etc. Unless you’re doing a one off .exs, you’re going to be really confused by what “mix new” does. Saying this as a passionate Elixirist — there’s nothing wrong with the leaky abstraction part, it’s what makes it powerful — but dethroning Python isn’t happening anytime soon.
- josevalim 4y ago
  I am not sure I agree with the OTP bits. We often tend to compare having to learn OTP with “not having to learn OTP” but the reality is that we need to compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes). Of course, one can say: “I wouldn’t use threads in other languages” but that’s potentially a bigger problem, as you end up with software which can’t easily exploit concurrency (which is especially important in the data domain). I still think what you said has some truth to it, but I think the frame of reference is a bit off. I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :) PS: “mix new” does not generate supervision trees by default.

pbowyer 4y ago

> Second, there are callbacks! The second argument in DataFrame.filter/2 takes a callback function against the dataframe. There’s some cool stuff we can do with that, and I think callbacks against the dataframe are a natural way to work. I'm not enamoured with how DataFrame.filter/2 reads compared to the dplyr example. I've written very little R but read code by others and find dpylr excellent for that; and I use a JS 'port' of dplyr [1] for projects. I'm in the "hate pandas" camp and find it painful to both read and write - and these callback functions are coming too close to Pandas for my liking. I hope an alternative can be found 1. https://pbeshai.github.io/tidy/ https://pbeshai.github.io/tidy/

throwamon 4y ago
For further inspiration, this is a pretty good-looking "dplyr for Python": https://github.com/machow/siuba https://github.com/machow/siuba There's precedent in Ecto for a "magic" operator (`^` if I'm not mistaken), so it wouldn't be a stretch to implement it here as well.