7 ms·

Administrative Scripting with Julia

88 points by xgdgsc 4y ago

alhirzel 4y ago
I have created job schedulers/process managers in Julia. There are good facilities for these "systems" level tasks because Julia is based on the very portable libuv[1]. You're able to pipe processes around [2]. Interpolation into "shell command mode" has good ergonomics (like automatic safe escaping and quoting) and also solid semantics (for instance, you can write `-i$files` and it expands to `-ifile1 -ifile2` for an array [3]). I am unsure why there isn't a Cromwell [4] for Julia in the public domain yet... [1]: https://libuv.org/ https://libuv.org/ [2]: https://docs.julialang.org/en/v1/manual/running-external-programs/ https://docs.julialang.org/en/v1/manual/running-external-pro... [3]: https://docs.julialang.org/en/v1/manual/running-external-programs/#command-interpolation https://docs.julialang.org/en/v1/manual/running-external-pro... [4]: https://cromwell.readthedocs.io/en/stable/ https://cromwell.readthedocs.io/en/stable/
- Nanana909 4y ago
  Reduce Julia startup/JIT time for scripts with https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl
noisy_boy 4y ago
I appreciate the "Why You Shouldn't Use Julia for Administrative Scripts" section[0] which asked exactly the questions I would have asked. The choice of (non-Bash) language to write command line utilities is in a bit of odd spot right now. Python is basically almost everywhere installed but the dependency on runtime + venv oddities bring their own set of problems. Java has similar runtime needs though things might improve with initiatives regarding native binary compilation (though including the runtime may not produce exactly lightweight executables) - also, not super popular among the younger crowd. Perl used to be a hot favorite in this space but I don't think lot of people are writing new stuff in Perl even though it is still present by default almost everywhere. Go is almost perfect here except I don't want to deal with 3x the boilerplate. Personally I think Rust isn't a bad choice (libraries like clap hugely reduce the boilerplate) but the learning curve makes it a harder sell (even though for basic utilities, I don't think there would be too much wrestling with the borrow checker). Another choice that comes to mind is Nim; I think it is very well positioned except a lot of people don't know even about it so its a hard sell + even among those who know, everyone is looking at everyone else to take the initiative to adopt it in a corporate environment at a non-trivial scale. [0]: https://github.com/ninjaaron/administrative-scripting-with-julia#id4 https://github.com/ninjaaron/administrative-scripting-with-j...
- koito17 4y ago
  I wish there was something elaborated for scripts that run on Node. I've been using nbb[1] for scripting, and although it all runs through Node.js, it is fast to start up and quick to prototype/debug scripts. The best part is in CI I can simply `npx nbb path/to/script.cljs`. Things get clunky if I want to use anything outside of the Node stdlib (or the few things that come out of the box, like tools.cli for command-line argument parsing) though, since then you need the dreaded node_modules folder around. [1] https://github.com/babashka/nbb https://github.com/babashka/nbb
  - _0w8t 4y ago
    Why is nbb or anything else necessary to run node scripts? Compared with a Python Node runtime does not come with all batteries included, but is rich enough to replace shell scripts. It even supports things like file system change notifiers, while Python requires external dependencies for that. So I run the scripts directly with node.
- macrolocal 4y ago
  Haskell is useful to stop other people from breaking your scripts.
  - deleted 4y ago
    [deleted]
- 0cf8612b2e1e 4y ago
  > Python is basically almost everywhere installed but the dependency on runtime + venv oddities bring their own set of problems Not that it changes your core message, but if you need scripting dependencies, you should use pipx[0]. Pipx will manage isolated virtual environments for each utility. ‘pipx install black’ will create a dedicated venv just to hold black. Invoking ‘pipx upgrade-all’ will scan all venvs for out of date utilities. [0] https://pypa.github.io/pipx/ https://pypa.github.io/pipx/
  - noisy_boy 4y ago
contemplatter 4y ago
Julia seems like a nice programming language. Is it still worth learning, though, since ChatGPT can write all software now?
- usrbinbash 4y ago
  Prompt: "Write a software that does exactly what the Product Owner wants. Here is is email *copy pasted email*" The result was...disappointing.
- HexDecOctBin 4y ago
  Agree. Me think me learn english, but me too think ChatGPT come, then why learn English? So me not learn now, only wait for ChatGPT.
  - bryanrasmussen 4y ago
    Bizarro hate Superman, but ChatGPT hate Superman better, that mean Bizarro love Superman!
  - gjadi 4y ago
    > Agree. Me think me learn english, but me too think ChatGPT come, then why learn English? So me not learn now, only wait for ChatGPT. > write above proper english I agree. I thought about learning English, but I also thought since ChatGPT is available, why bother learning it? So I decided not to learn it now and just wait for ChatGPT.
- amval 4y ago
  Is it still worth posting, since ChatGPT can write all comments now?
- staunton 4y ago
  If you know Python, you can learn Julia in a day.
  - rsfern 4y ago
jhoechtl 4y ago
Would rather use Janet for that https://janet-lang.org/ https://janet-lang.org/
- xiaodai 4y ago
  reason?
  - kazinator 4y ago
    Not knowing about TXR Lisp, I would guess. :)
- Vaslo 4y ago
  Who?
Xcelerate 4y ago
As a data scientist, I’m constantly disappointed that Julia didn’t take off the way I was hoping (and thought) that it would when I was doing some combination of machine learning and HPC as a grad student back in 2013. It was just so pleasant and enjoyable to program in, and there were many things I was able to do with the language that would have been much more difficult in other languages (as one example, I wrote a generated function to compute the irreducible matrix representations of compact groups; Julia generated a unique function with optimized bytecode for each distinct matrix dimension). Like the post author, I also found Julia easy to use as a general purpose programming language. I’m still hoping adoption will gain momentum at some point. Python certainly took a while to get going.
- Lyngbakr 4y ago
  I'm really curious what a new language would need to have to start stealing users from Python in the data domain. I was also a bit surprised that Julia didn't get more traction, as I felt it was a more enjoyable experience, but obviously it doesn't have the extensive community and ecosystem that Python has so it was fighting an uphill battle from the start. Interop with Python (like R's reticulate) would be helpful when targeting Python users, so they could still draw in their favourite packages, but Julia has that already. In my new/current role, I primarily use Python but would prefer not to; however, it's a tough case to make against Python when it's become the default data language (along with SQL).
  - zaphar 4y ago
    It already has interop with python via: https://juliapackages.com/p/pythoncall https://juliapackages.com/p/pythoncall. It's pretty seamless for the most part I think.
  - Xcelerate 4y ago
    > I'm really curious what a new language would need to have to start stealing users from Python Backing of a large tech company. I tried to get something going at the one I worked at, but it went nowhere. To my surprise a coworker who was also one of the early top contributors to Julia mentioned to me that the language was unlikely to ever take off internally because the company had already invested so many resources into improving and building upon Python.
tpoacher 4y ago
Next step: using SHELL=julia in your makefiles. (/s)
skybrian 4y ago
The example of opening files within a do block could be explained better. [1] The article implies that the file gets closed when it goes out of scope, but doesn’t explain how it works. [1] https://github.com/ninjaaron/administrative-scripting-with-julia/blob/master/1-files.ipynb https://github.com/ninjaaron/administrative-scripting-with-j...

88 points by xgdgsc 4y ago

I have created job schedulers/process managers in Julia. There are good facilities for these "systems" level tasks because Julia is based on the very portable libuv[1]. You're able to pipe processes around [2]. Interpolation into "shell command mode" has good ergonomics (like automatic safe escaping and quoting) and also solid semantics (for instance, you can write `-i$files` and it expands to `-ifile1 -ifile2` for an array [3]). I am unsure why there isn't a Cromwell [4] for Julia in the public domain yet... [1]: https://libuv.org/ https://libuv.org/ [2]: https://docs.julialang.org/en/v1/manual/running-external-programs/ https://docs.julialang.org/en/v1/manual/running-external-pro... [3]: https://docs.julialang.org/en/v1/manual/running-external-programs/#command-interpolation https://docs.julialang.org/en/v1/manual/running-external-pro... [4]: https://cromwell.readthedocs.io/en/stable/ https://cromwell.readthedocs.io/en/stable/

Nanana909 4y ago
Reduce Julia startup/JIT time for scripts with https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl

noisy_boy 4y ago

I appreciate the "Why You Shouldn't Use Julia for Administrative Scripts" section[0] which asked exactly the questions I would have asked. The choice of (non-Bash) language to write command line utilities is in a bit of odd spot right now. Python is basically almost everywhere installed but the dependency on runtime + venv oddities bring their own set of problems. Java has similar runtime needs though things might improve with initiatives regarding native binary compilation (though including the runtime may not produce exactly lightweight executables) - also, not super popular among the younger crowd. Perl used to be a hot favorite in this space but I don't think lot of people are writing new stuff in Perl even though it is still present by default almost everywhere. Go is almost perfect here except I don't want to deal with 3x the boilerplate. Personally I think Rust isn't a bad choice (libraries like clap hugely reduce the boilerplate) but the learning curve makes it a harder sell (even though for basic utilities, I don't think there would be too much wrestling with the borrow checker). Another choice that comes to mind is Nim; I think it is very well positioned except a lot of people don't know even about it so its a hard sell + even among those who know, everyone is looking at everyone else to take the initiative to adopt it in a corporate environment at a non-trivial scale. [0]: https://github.com/ninjaaron/administrative-scripting-with-julia#id4 https://github.com/ninjaaron/administrative-scripting-with-j...

koito17 4y ago
I wish there was something elaborated for scripts that run on Node. I've been using nbb[1] for scripting, and although it all runs through Node.js, it is fast to start up and quick to prototype/debug scripts. The best part is in CI I can simply `npx nbb path/to/script.cljs`. Things get clunky if I want to use anything outside of the Node stdlib (or the few things that come out of the box, like tools.cli for command-line argument parsing) though, since then you need the dreaded node_modules folder around. [1] https://github.com/babashka/nbb https://github.com/babashka/nbb
- _0w8t 4y ago
  Why is nbb or anything else necessary to run node scripts? Compared with a Python Node runtime does not come with all batteries included, but is rich enough to replace shell scripts. It even supports things like file system change notifiers, while Python requires external dependencies for that. So I run the scripts directly with node.
macrolocal 4y ago
Haskell is useful to stop other people from breaking your scripts.
- deleted 4y ago
  [deleted]
0cf8612b2e1e 4y ago
> Python is basically almost everywhere installed but the dependency on runtime + venv oddities bring their own set of problems Not that it changes your core message, but if you need scripting dependencies, you should use pipx[0]. Pipx will manage isolated virtual environments for each utility. ‘pipx install black’ will create a dedicated venv just to hold black. Invoking ‘pipx upgrade-all’ will scan all venvs for out of date utilities. [0] https://pypa.github.io/pipx/ https://pypa.github.io/pipx/
- noisy_boy 4y ago

contemplatter 4y ago

Julia seems like a nice programming language. Is it still worth learning, though, since ChatGPT can write all software now?

usrbinbash 4y ago
Prompt: "Write a software that does exactly what the Product Owner wants. Here is is email *copy pasted email*" The result was...disappointing.
HexDecOctBin 4y ago
Agree. Me think me learn english, but me too think ChatGPT come, then why learn English? So me not learn now, only wait for ChatGPT.
- bryanrasmussen 4y ago
  Bizarro hate Superman, but ChatGPT hate Superman better, that mean Bizarro love Superman!
- gjadi 4y ago
  > Agree. Me think me learn english, but me too think ChatGPT come, then why learn English? So me not learn now, only wait for ChatGPT. > write above proper english I agree. I thought about learning English, but I also thought since ChatGPT is available, why bother learning it? So I decided not to learn it now and just wait for ChatGPT.
amval 4y ago
Is it still worth posting, since ChatGPT can write all comments now?
staunton 4y ago
If you know Python, you can learn Julia in a day.
- rsfern 4y ago

jhoechtl 4y ago

Would rather use Janet for that https://janet-lang.org/ https://janet-lang.org/

xiaodai 4y ago
reason?
- kazinator 4y ago
  Not knowing about TXR Lisp, I would guess. :)
Vaslo 4y ago
Who?

Xcelerate 4y ago

As a data scientist, I’m constantly disappointed that Julia didn’t take off the way I was hoping (and thought) that it would when I was doing some combination of machine learning and HPC as a grad student back in 2013. It was just so pleasant and enjoyable to program in, and there were many things I was able to do with the language that would have been much more difficult in other languages (as one example, I wrote a generated function to compute the irreducible matrix representations of compact groups; Julia generated a unique function with optimized bytecode for each distinct matrix dimension). Like the post author, I also found Julia easy to use as a general purpose programming language. I’m still hoping adoption will gain momentum at some point. Python certainly took a while to get going.

Lyngbakr 4y ago
I'm really curious what a new language would need to have to start stealing users from Python in the data domain. I was also a bit surprised that Julia didn't get more traction, as I felt it was a more enjoyable experience, but obviously it doesn't have the extensive community and ecosystem that Python has so it was fighting an uphill battle from the start. Interop with Python (like R's reticulate) would be helpful when targeting Python users, so they could still draw in their favourite packages, but Julia has that already. In my new/current role, I primarily use Python but would prefer not to; however, it's a tough case to make against Python when it's become the default data language (along with SQL).
- zaphar 4y ago
  It already has interop with python via: https://juliapackages.com/p/pythoncall https://juliapackages.com/p/pythoncall. It's pretty seamless for the most part I think.
- Xcelerate 4y ago
  > I'm really curious what a new language would need to have to start stealing users from Python Backing of a large tech company. I tried to get something going at the one I worked at, but it went nowhere. To my surprise a coworker who was also one of the early top contributors to Julia mentioned to me that the language was unlikely to ever take off internally because the company had already invested so many resources into improving and building upon Python.

tpoacher 4y ago

Next step: using SHELL=julia in your makefiles. (/s)

skybrian 4y ago

The example of opening files within a do block could be explained better. [1] The article implies that the file gets closed when it goes out of scope, but doesn’t explain how it works. [1] https://github.com/ninjaaron/administrative-scripting-with-julia/blob/master/1-files.ipynb https://github.com/ninjaaron/administrative-scripting-with-j...