6 ms·

Scripting with Elixir

210 points by lawik 3y ago

gigatexal 3y ago
interesting. As a data engineer I have been indoctrinated to think about things as various transformations and so the functional approach to things is very nice. I've also been wanting to learn/play with Elixir. In the Python space, though not the same, the ergonomics of which kinda get you there, there's this: pipe https://pypi.org/project/pipe/ https://pypi.org/project/pipe/
- lawik 3y ago
  The pipes are really very nice. They are actually even nicer under Livebook: https://livebook.dev/ https://livebook.dev/ It can let you drag-drop reorder, enable/disable, steps in a pipeline: https://github.com/livebook-dev/livebook/blob/main/lib/livebook/notebook/learn/kino/intro_to_kino.livemd#dbg https://github.com/livebook-dev/livebook/blob/main/lib/liveb... It is really wild.
  - behnamoh 3y ago
    I like pipes but I'm not sure if it's pythonic. What if other people need to maintain/debug the code? Granted, it's not as much of a learning curve as personal Lisp macros.
    - deleted 3y ago
      [deleted]
- whalesalad 3y ago
  If you haven't played around with it - you might dig Livebook https://livebook.dev https://livebook.dev
- kaycey2022 3y ago
  Lost an opportunity to name it Pype
sb8244 3y ago
I haven't used Elixir for scripting yet, but I really like the idea of including deps at the top of the file. It's always been a pain to manage this in other scripting environments. I did scripts with Ruby in past and it didn't end up being easy to use because of external deps.
- tomjakubowski 3y ago
  good ol' nasty jupyter notebooks support this: %pip install foo~=1.5 %pip install bar~=1.7 import foo import bar
  - impulser_ 3y ago
    This would require you to install and run jupyter. Mix.install just requires Elixir being installed.
  - lawik 3y ago
    That's neat.
- radiospiel 3y ago
  TBF ruby lets you use inline bundler. See https://bundler.io/guides/bundler_in_a_single_file_ruby_script.html https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...
  - sb8244 3y ago
    Oh that's awesome. I didn't know about this (or not sure when it was released.) edit: it's been out since 2015 it looks like. Not sure why but I didn't know about this before. It would have been useful.
    - deleted 3y ago
      [deleted]
  - thibaut_barrere 3y ago
hospitalJail 3y ago
I didn't understand the use case. I had chatgpt try to find one, maybe dependency management(eh, I got conda for that) and parallel processing(eh, I got pool for that). Seems like a no-brainer to go with python for the long term. Why would anyone take the time to use this?
- knoebber 3y ago
  two reasons: 1. you prefer functional style languages, such as elixir 2. you want to avoid `pip install` ing dependencies
- OkayPhysicist 3y ago
  If you already write a lot of Elixir for it's application-level superpowers, then being able to hammer out a script in a language you're fresh on is nice. Combine that with better developer ergonomics (package handing in Python is inane, for one), and it's not a bad choice. No one should be learning Elixir solely for the purpose of writing short, one-off scripts, though.
  - thibaut_barrere 3y ago
    On the other hand, I find that starting to learn a language with one-off scripts is a nice gateway drug. This is how I started using Ruby initially!
- freedomben 3y ago
  If you know elixir, but don't know python, it seems like a no-brainer to choose language you know over language you don't. Plus the portability and dependency management of Python has traditionally been a nightmare. Elixir has great solutions to both of those problems for scripting contexts. Another case, if your code base is in Elixir, picking an unrelated different language (python) would (IMHO) be as silly a decision as picking elixir for scripts when you have a python app, and I would strongly question that person's judgment and fitness for making sustainable and maintainable technical decisions.
- tikhonj 3y ago
aeturnum 3y ago
I really want scripting in Elixir to be better - and the addition of Mix.install() is a great step forward - but there are still a lot of awkward elements. Elixir will require you to be in a module to define a function and then respect module namespaces when calling them - which makes sense in a normal context but I find it to be annoying while scripting. Likewise I don't think dialyzer, the Elixir / Erlang static type analyzer, will run on script files. That said I think the underlying lack of data mutability really increases the potential for code reuse and scripting.
- jolux 3y ago
  you can define functions outside of a module using fn blocks
  - skndr 3y ago
    There's also defining a module and then `import`ing it.
  - throwawaymaths 3y ago
    There's good odds it will come to elixir, since support just landed in Erlang
    - erszcz 3y ago
      To be specific, it has landed in the Erlang shell with Erlang/OTP 26. Defining inline functions in source code, on the other hand, has been possible for years (like 20+ years).
- thibaut_barrere 3y ago
  Agreed, I find this a bit annoying too. Makes me create a module earlier than I'd have wanted (but not too much trouble though). One can still reuse code with "Code.require_file", which is nice to have.
- jnsaff2 3y ago
  sum = fn (a, b) -> a + b end sum.(2, 3) Maybe a but clunky but no module required.
skndr 3y ago
I've been finding Livebook (https://livebook.dev/ https://livebook.dev/) really useful for iterating on a script or manipulating some data in a reproducible way. I'll often try out one solution and if that doesn't work, create a new section and collapse the old one to be able to go in a different direction.
thibaut_barrere 3y ago
I've been a big fan of "Mix.install" since it was released! I believe this made me use Elixir instead of Ruby (my natural scripting language) more and more. I use it on a regular basis for my work on https://transport.data.gouv.fr/?locale=en https://transport.data.gouv.fr/?locale=en (which is Elixir-based and open-source), as one can see at https://github.com/etalab/transport-site/tree/master/scripts https://github.com/etalab/transport-site/tree/master/scripts. What I like the most is that it helps me start ideas, experiments & data analysis with their own set of dependencies (without much care of "will this impact the main application?"), store those experiments in the same repo at the main application, and maybe later promote some of those experiments to the main application source code. Concrete examples include: - data wrangling https://github.com/etalab/transport-site/blob/master/scripts/api/filter_gtfs_rt_by_entity_types.exs https://github.com/etalab/transport-site/blob/master/scripts..., https://github.com/etalab/transport-site/blob/master/scripts/irve/irve.exs https://github.com/etalab/transport-site/blob/master/scripts... - exploring a new tool https://github.com/etalab/transport-site/blob/master/scripts/oban/oban_experiments.exs https://github.com/etalab/transport-site/blob/master/scripts... - gradually iterating to create scripts to generate XML queries https://github.com/etalab/transport-site/tree/master/scripts/siri https://github.com/etalab/transport-site/tree/master/scripts... (later incorporated in the application) - comparing GTFS (transportation data) files via scripting https://github.com/etalab/transport-site/blob/master/scripts/git_diff_experiment.exs https://github.com/etalab/transport-site/blob/master/scripts... - learning how to compute the checksum of the internal content of a zip file https://github.com/etalab/transport-site/blob/master/scripts/internal_zip_checksum_experiment.exs https://github.com/etalab/transport-site/blob/master/scripts... As mentioned in the article, I can definitely recommend to check out https://github.com/wojtekmach/mix_install_examples https://github.com/wojtekmach/mix_install_examples which has a long list of examples. One last tip is that you can also use "mix run script.exs" on a file not using "Mix.install", in order to rely on the same dependencies & configuration as the main application (e.g. to run a Ecto query https://github.com/etalab/transport-site/blob/master/scripts/subquery.exs https://github.com/etalab/transport-site/blob/master/scripts...).
nesarkvechnep 3y ago
I expected a mention of escripts.
nivertech 3y ago
Of course in Elixir[1] reduce is more efficient (at least memory-wise) than map/flat_map, but in case of scripting it usually doesn't matter (unless this JSON file is huge, but it's fully loaded into memory and parsed there anyway). I would rewrite the example from the post like this (NOTE: I didn't run it, so it might be slightly incorrect): #/usr/bin/env elixir "file.json" |> File.read!() |> Jason.decode!() |> Enum.flat_map(& &1["children"]) |> Enum.map(&{&1["id"], &1}) |> Enum.into(%{}) --- 1. although map is much easier to parallelize/vectorize (e.g. in Array PLs)
- rkangel 3y ago
  It's slightly more verbose but I have mostly stopped using the & notation to produce functions out of thin air[1]. I find the following code is much more easily readable - more welcoming to the less experienced, less brain load for the more experienced: #/usr/bin/env elixir "file.json" |> File.read!() |> Jason.decode!() |> Enum.flat_map(fn x -> x["children"] end) |> Enum.map(fn x -> {x["id"], x} end) |> Enum.into(%{}) [1] The `&a_function_with_odd_arguments(&2, &1)` type notation can still be very useful and is (subjectively) a bit less symbol-salad.
  - nivertech 3y ago
    I dislike Elixir's abuse of sigils (it's inhereted from Perl via Ruby). The capture operator syntax in Elixir isn't a very ergonomic design. It uses the same symbol for both fun and arguments, and also for MFAs (&M.f/a). It doesn't allow nesting, so in these cases there is no choice but to use the full notation. In case the intent isn't clear from the code, I'll also use the full notation. Also, I think it's much more acceptable to use shorthand/sugar in one-off scripts rather than in "real" code. NOTE: in your example you use "x" for argument, so it's not much better than "&1". Since you already using the full notation, it's better to meaningfull argument names, such as "item" and "subitem".
    - Lutger 3y ago
      For readers not used to Elixirs operators - which I am - "x" is much easier to understand than "&1" for two reasons. First, because the syntax of the full notation resembles how anonymous functions are done in almost every mainstream language that has them. Second, the character "x" is very often used as a name for the first parameter of an anonymous function, which reinforces the first point. If I see only only this: (x,y) -> ... I can guess quite well what x and y are used for, and I suspect I am not the only developer who recognizes this as a 'parameter list' partly because the names happen to use 'x' and 'y' which are otherwise non-descriptive. Nevertheless, your point still stands.

210 points by lawik 3y ago

interesting. As a data engineer I have been indoctrinated to think about things as various transformations and so the functional approach to things is very nice. I've also been wanting to learn/play with Elixir. In the Python space, though not the same, the ergonomics of which kinda get you there, there's this: pipe https://pypi.org/project/pipe/ https://pypi.org/project/pipe/

lawik 3y ago
The pipes are really very nice. They are actually even nicer under Livebook: https://livebook.dev/ https://livebook.dev/ It can let you drag-drop reorder, enable/disable, steps in a pipeline: https://github.com/livebook-dev/livebook/blob/main/lib/livebook/notebook/learn/kino/intro_to_kino.livemd#dbg https://github.com/livebook-dev/livebook/blob/main/lib/liveb... It is really wild.
- behnamoh 3y ago
  I like pipes but I'm not sure if it's pythonic. What if other people need to maintain/debug the code? Granted, it's not as much of a learning curve as personal Lisp macros.
  - deleted 3y ago
    [deleted]
whalesalad 3y ago
If you haven't played around with it - you might dig Livebook https://livebook.dev https://livebook.dev
kaycey2022 3y ago
Lost an opportunity to name it Pype

sb8244 3y ago

I haven't used Elixir for scripting yet, but I really like the idea of including deps at the top of the file. It's always been a pain to manage this in other scripting environments. I did scripts with Ruby in past and it didn't end up being easy to use because of external deps.

tomjakubowski 3y ago
good ol' nasty jupyter notebooks support this: %pip install foo~=1.5 %pip install bar~=1.7 import foo import bar
- impulser_ 3y ago
  This would require you to install and run jupyter. Mix.install just requires Elixir being installed.
- lawik 3y ago
  That's neat.
radiospiel 3y ago
TBF ruby lets you use inline bundler. See https://bundler.io/guides/bundler_in_a_single_file_ruby_script.html https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...
- sb8244 3y ago
  Oh that's awesome. I didn't know about this (or not sure when it was released.) edit: it's been out since 2015 it looks like. Not sure why but I didn't know about this before. It would have been useful.
  - deleted 3y ago
    [deleted]
- thibaut_barrere 3y ago

hospitalJail 3y ago

I didn't understand the use case. I had chatgpt try to find one, maybe dependency management(eh, I got conda for that) and parallel processing(eh, I got pool for that). Seems like a no-brainer to go with python for the long term. Why would anyone take the time to use this?

knoebber 3y ago
two reasons: 1. you prefer functional style languages, such as elixir 2. you want to avoid `pip install` ing dependencies
OkayPhysicist 3y ago
If you already write a lot of Elixir for it's application-level superpowers, then being able to hammer out a script in a language you're fresh on is nice. Combine that with better developer ergonomics (package handing in Python is inane, for one), and it's not a bad choice. No one should be learning Elixir solely for the purpose of writing short, one-off scripts, though.
- thibaut_barrere 3y ago
  On the other hand, I find that starting to learn a language with one-off scripts is a nice gateway drug. This is how I started using Ruby initially!
freedomben 3y ago
If you know elixir, but don't know python, it seems like a no-brainer to choose language you know over language you don't. Plus the portability and dependency management of Python has traditionally been a nightmare. Elixir has great solutions to both of those problems for scripting contexts. Another case, if your code base is in Elixir, picking an unrelated different language (python) would (IMHO) be as silly a decision as picking elixir for scripts when you have a python app, and I would strongly question that person's judgment and fitness for making sustainable and maintainable technical decisions.
tikhonj 3y ago

aeturnum 3y ago

I really want scripting in Elixir to be better - and the addition of Mix.install() is a great step forward - but there are still a lot of awkward elements. Elixir will require you to be in a module to define a function and then respect module namespaces when calling them - which makes sense in a normal context but I find it to be annoying while scripting. Likewise I don't think dialyzer, the Elixir / Erlang static type analyzer, will run on script files. That said I think the underlying lack of data mutability really increases the potential for code reuse and scripting.

jolux 3y ago
you can define functions outside of a module using fn blocks
- skndr 3y ago
  There's also defining a module and then `import`ing it.
- throwawaymaths 3y ago
  There's good odds it will come to elixir, since support just landed in Erlang
  - erszcz 3y ago
    To be specific, it has landed in the Erlang shell with Erlang/OTP 26. Defining inline functions in source code, on the other hand, has been possible for years (like 20+ years).
thibaut_barrere 3y ago
Agreed, I find this a bit annoying too. Makes me create a module earlier than I'd have wanted (but not too much trouble though). One can still reuse code with "Code.require_file", which is nice to have.
jnsaff2 3y ago
sum = fn (a, b) -> a + b end sum.(2, 3) Maybe a but clunky but no module required.

skndr 3y ago

I've been finding Livebook (https://livebook.dev/ https://livebook.dev/) really useful for iterating on a script or manipulating some data in a reproducible way. I'll often try out one solution and if that doesn't work, create a new section and collapse the old one to be able to go in a different direction.

thibaut_barrere 3y ago

I've been a big fan of "Mix.install" since it was released! I believe this made me use Elixir instead of Ruby (my natural scripting language) more and more. I use it on a regular basis for my work on https://transport.data.gouv.fr/?locale=en https://transport.data.gouv.fr/?locale=en (which is Elixir-based and open-source), as one can see at https://github.com/etalab/transport-site/tree/master/scripts https://github.com/etalab/transport-site/tree/master/scripts. What I like the most is that it helps me start ideas, experiments & data analysis with their own set of dependencies (without much care of "will this impact the main application?"), store those experiments in the same repo at the main application, and maybe later promote some of those experiments to the main application source code. Concrete examples include: - data wrangling https://github.com/etalab/transport-site/blob/master/scripts/api/filter_gtfs_rt_by_entity_types.exs https://github.com/etalab/transport-site/blob/master/scripts..., https://github.com/etalab/transport-site/blob/master/scripts/irve/irve.exs https://github.com/etalab/transport-site/blob/master/scripts... - exploring a new tool https://github.com/etalab/transport-site/blob/master/scripts/oban/oban_experiments.exs https://github.com/etalab/transport-site/blob/master/scripts... - gradually iterating to create scripts to generate XML queries https://github.com/etalab/transport-site/tree/master/scripts/siri https://github.com/etalab/transport-site/tree/master/scripts... (later incorporated in the application) - comparing GTFS (transportation data) files via scripting https://github.com/etalab/transport-site/blob/master/scripts/git_diff_experiment.exs https://github.com/etalab/transport-site/blob/master/scripts... - learning how to compute the checksum of the internal content of a zip file https://github.com/etalab/transport-site/blob/master/scripts/internal_zip_checksum_experiment.exs https://github.com/etalab/transport-site/blob/master/scripts... As mentioned in the article, I can definitely recommend to check out https://github.com/wojtekmach/mix_install_examples https://github.com/wojtekmach/mix_install_examples which has a long list of examples. One last tip is that you can also use "mix run script.exs" on a file not using "Mix.install", in order to rely on the same dependencies & configuration as the main application (e.g. to run a Ecto query https://github.com/etalab/transport-site/blob/master/scripts/subquery.exs https://github.com/etalab/transport-site/blob/master/scripts...).

nesarkvechnep 3y ago

I expected a mention of escripts.

nivertech 3y ago

Of course in Elixir[1] reduce is more efficient (at least memory-wise) than map/flat_map, but in case of scripting it usually doesn't matter (unless this JSON file is huge, but it's fully loaded into memory and parsed there anyway). I would rewrite the example from the post like this (NOTE: I didn't run it, so it might be slightly incorrect): #/usr/bin/env elixir "file.json" |> File.read!() |> Jason.decode!() |> Enum.flat_map(& &1["children"]) |> Enum.map(&{&1["id"], &1}) |> Enum.into(%{}) --- 1. although map is much easier to parallelize/vectorize (e.g. in Array PLs)

rkangel 3y ago
It's slightly more verbose but I have mostly stopped using the & notation to produce functions out of thin air[1]. I find the following code is much more easily readable - more welcoming to the less experienced, less brain load for the more experienced: #/usr/bin/env elixir "file.json" |> File.read!() |> Jason.decode!() |> Enum.flat_map(fn x -> x["children"] end) |> Enum.map(fn x -> {x["id"], x} end) |> Enum.into(%{}) [1] The `&a_function_with_odd_arguments(&2, &1)` type notation can still be very useful and is (subjectively) a bit less symbol-salad.
- nivertech 3y ago
  I dislike Elixir's abuse of sigils (it's inhereted from Perl via Ruby). The capture operator syntax in Elixir isn't a very ergonomic design. It uses the same symbol for both fun and arguments, and also for MFAs (&M.f/a). It doesn't allow nesting, so in these cases there is no choice but to use the full notation. In case the intent isn't clear from the code, I'll also use the full notation. Also, I think it's much more acceptable to use shorthand/sugar in one-off scripts rather than in "real" code. NOTE: in your example you use "x" for argument, so it's not much better than "&1". Since you already using the full notation, it's better to meaningfull argument names, such as "item" and "subitem".
  - Lutger 3y ago
    For readers not used to Elixirs operators - which I am - "x" is much easier to understand than "&1" for two reasons. First, because the syntax of the full notation resembles how anonymous functions are done in almost every mainstream language that has them. Second, the character "x" is very often used as a name for the first parameter of an anonymous function, which reinforces the first point. If I see only only this: (x,y) -> ... I can guess quite well what x and y are used for, and I suspect I am not the only developer who recognizes this as a 'parameter list' partly because the names happen to use 'x' and 'y' which are otherwise non-descriptive. Nevertheless, your point still stands.