6 ms·

Beyond Bash: Shell scripting in a typed, OO language

49 points by lihaoyi 11y ago

_ZeD_ 11y ago
If you don't mind using python, the sh package [0] can really get you a bash-style repl (dinamically) typed. .. [0]: http://amoffat.github.io/sh/ http://amoffat.github.io/sh/
- loevborg 11y ago
  Yes, in my mind this is the way to go (the other alternative, probably unviable, being scsh https://github.com/scheme/scsh https://github.com/scheme/scsh). I recently made a decision not to write any bash scripts that include a conditional or loop; for all those use cases python+sh is the way to go. The reason: http://mywiki.wooledge.org/BashPitfalls http://mywiki.wooledge.org/BashPitfalls I can't think of any environment where python would not be available. You can even ship sh.py as a single-file dependency along with your script, so `import cp from sh` will work as expected.
  - Johnny_Brahms 11y ago
    Haha! Just reading this thread I got the idea to replace bash with something scheme-like. You have a shell which works like it usually soes, but that easily let's you write any utilities in scheme. I use scheme to write most of my system stuff (most revently a custom automounter for my raspberry Pi) and thought it would have been a nice project. You could probably go macro galore and make it quite comfortable to do piping and integrate the regular system utilities in the language.
- kseistrup 11y ago
  There's also a bash-like Python REPL like Xonsh ⌘ http://xonsh.org/ http://xonsh.org/ Still in early development, but very useful.
manojlds 11y ago
That's Powershell in the Windows world.
- bentronic 11y ago
  Powershell is exactly what bash should be (and this person wants). Their time would have better spent using and improving the pash project (open source Powershell on Mono): https://github.com/Pash-Project/Pash https://github.com/Pash-Project/Pash
  - alloyed 11y ago
    Powershell is primarily a repl to a homogeneous .NET-based set of tools, which doesn't describe Unix very well. If you want a type-safe repl that interacts with all the handy tools written for the text-stream-only Unix world you're going to have to do some interpretation.
- GlennS 11y ago
  Powershell has some pitfalls of its own (I still much prefer it to Bash despite these): 1. Dynamic scoping 2. Squashing lists of size 0 to nulls and size 1 to single elements when they come out of pipes. 3. Incomprehensible API for XML handling. On the other hand, when a Powershell program starts getting too big, it's pretty trivial to port parts of it to one of the other .NET languages.
  - lihaoyi 11y ago
    > 3. Incomprehensible API for XML handling. If you think Scala has a better API for XML handling then you're in for a surprise...
  - manojlds 11y ago
    Very true pitfalls, and there are other ones like any non-captured value anywhere in a function becomes the output of the function.
joostdevries 11y ago
Li Haoyi has been doing a lot of cool open source Scala work. Ammonite is an example of that. You can find it here https://github.com/lihaoyi/Ammonite https://github.com/lihaoyi/Ammonite
- cmrdporcupine 11y ago
  I for one would not to pay the cost of starting the JVM up every time I want to invoke a shell script.
black_knight 11y ago
Some people do their shell scripting in a pure, functional, strictly typed language [0] — and I don't judge them! [0] http://www.haskellforall.com/2015/01/use-haskell-for-shell-scripting.html http://www.haskellforall.com/2015/01/use-haskell-for-shell-s...
- lihaoyi 11y ago
  That's cool! We should probably join forces. After all, Martin Odersky has stated that moving people onto Haskell was the primary motivation for inventing Scala, which will soon be renamed Hascalator https://twitter.com/myatlyuka/status/577625111575969792 https://twitter.com/myatlyuka/status/577625111575969792 At the same time, Typesafe is apparently being rename Reactive Banana, which is more evidence of the increased Haskell focus within the community https://www.reddit.com/r/scala/comments/36ho4u/breaking_typesafe_will_be_renamed_to_reactive/ https://www.reddit.com/r/scala/comments/36ho4u/breaking_type... EDIT: found a better slide for Hascalator
  - nine_k 11y ago
    My friends answered immediately: «Saying that Scala is a gateway drug to Haskell is like saying that poor-quality heroin cut with speed is а gateway drug to LSD».
eggie 11y ago
Shell languages are defined by their terseness. The author misunderstands how important this is: https://docs.google.com/presentation/d/11vZzXCfAA0aOFAuHA0nAvAzALGFGCH-dqHxx6XMgbk8/preview?sle=true&slide=id.ga54359adc_1_218 https://docs.google.com/presentation/d/11vZzXCfAA0aOFAuHA0nA... The author has created a radically "better" shell language which is twice as verbose as bash. In the space of human-computer interaction where shell languages lie, this is tantamount to total failure. The worst parts of shell are the flow control structures (if/else...). If only we could improve this without throwing out the entire system. Else we will lose the benefits of the ultra-terse and completely decentralized "language" that is unix shell.
- hyperpape 11y ago
  This is mostly true, and it doesn't look like this is the solution. My belief is any solution needs to be built from the ground up as a language for shell use, not adapting Python/Scala/Haskell to shell use. However, there is some non-zero amount of verbosity that would be worth trading for better control flow, composability, data structures, etc. As long as you can do simple operations and pipes with the same syntax | grep foo cp * path/ rm -rf it's ok if some operations are more verbose in exchange for that. Programmers don't live in bash, and deal with the pain points of switching between bash and an actual programming language on a regular basis. I know I'm there. A change that sometimes is more verbose but lets you do more things in shell would still have benefits. You might not win over the people who are bash experts, but you could still produce a better tool for many people.
- klibertp 11y ago
  I didn't check it out yet, but I think Tulip (http://www.jneen.net/posts/2015-07-29-tulip-language-updated http://www.jneen.net/posts/2015-07-29-tulip-language-updated) has a great potential in this space.
- sambe 11y ago
  The author explicitly states terseness is a goal, and compares alternatives' overhead to bash. However, he recognises there is a limit whilst remaining safe/close to Scala. I don't agree control structures are a major problem. There are subtle differences between correct and incorrect expressions, and they are not checked at "compile" time.
hello_there 11y ago
Cool stuff. I really like the idea, but why so many symbols? On slide "2.5 Piping" I find the description on the right to be far more readable than the examples on the left. Also the descriptions on the right are familiar, idiomatic Scala which is also a plus. The stuff on the left is impossible to decipher without some documentation and I don't see how it adds value except from saving a few chars. OTOH it introduces significant load on the developer that has to look them up (again and again) in the manual. If saving these chars are important then one might defend introducing a single new operator for some well thought-out abstraction, but introducing 7 new operators is just to much for anyone to remember. Especially the two last operators seems entirely redundant as the language already has quite compact syntax for function invocation (as does any language).
agumonkey 11y ago
When I see this unifying of language logic and shell ~logic, as a Lisp head, I can't help but to think of lisp machines: https://www.youtube.com/watch?v=o4-YnLpLgtk https://www.youtube.com/watch?v=o4-YnLpLgtk
jmount 11y ago
Wasn't this one of the points of Perl? New languages seem small, but tend to grow.
heinrich5991 11y ago
Path normalization does not work like that. If `/bin` is a symlink referring to `/usr/bin`, then `/bin/..` is the same as `/usr`, not `/` as the author suggests.
a3n 11y ago
Congratulations on making something cool and useful. But I think it's not Bash vs Ammonite, it's more accurately (Alternatives to Bash) vs (The Ammonite alternative to Bash). I've never worked in an environment that uses sh for large systems. My use is just my daily conversation with my OS and file system. I say things on the command line. Sometimes I say certain things in sequence often enough that I collect them into personal "jargon" scripts of two to twenty lines of Bash. For anything larger I generally use Python. My dividing line between Bash and Python (or any similar language) is conversation vs contract. A conversation is relatively informal. Anything larger requires negotiation and agreement by parties (me and the OS) using technically understood formal language (to strain the metaphor). I think Ammonite is more like the contract than the conversation, but maybe not as much as Python.
lihaoyi 11y ago
https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671 is a nice motivating example of why I think this project is useful
therealmarv 11y ago
Using a lot of shell can really be dangerous. When you are on the ops side you know how to avoid most of this pitfalls by using tools like Ansible, Puppet or Chef.
zimbatm 11y ago
If anything is going to displace bash at this point it's going to be something that brings substantial benefits to the user. Stricter syntax is good but not of the immediate concern to the user typing his 2x-longer command. The next evolution of shell is going to be something different. Something that doesn't depend on an antiquated TTY protocol, something that can display images and other constructs more complex that text, and more importantly something that allows the user to accumulate commands instead of typing the same thing over and over again. Maybe something like ipython/jupyter
- pekk 11y ago
  Using languages other than shell already brings lots of benefits to the user. That just doesn't matter to the people who are writing large programs in shell. It isn't like those people are basing their decision on rigorous user studies.
- Aleman360 11y ago
  PowerShell
- hyperpape 11y ago
  There's the shell language and the shell interface. The design of those things influence each other, but they're fundamentally separate even though bash/zsh/fish often make changes to both. The original article is really about languages, and I think that even with the best interface in the world, we want a better language.
  - zimbatm 11y ago
    Just changing the language is in the diminishing return area and is why it hasn't happened yet. Something else need to change. Any sort of shell programming is going to be full of side-effects. No matter how strict the language, building the script will still involve trying on the target system, maybe accumulating shell commands in a script, and then have these long full cycles where the script is tested on a fresh system to make sure it's still working. This is happening every time I do any sort of sysadmin-type of stuff. Even in ansible, chef, docker, ... To escape that something deeper needs to be changed.
tunesmith 11y ago
Pretty cool... more documentation and examples here: http://lihaoyi.github.io/Ammonite/ http://lihaoyi.github.io/Ammonite/
KuhlMensch 11y ago
I have a large multi-concern shell script for building a web-app. The "complexity" seems to impress my workmates, but I dislike it and avoid extending it if at all possible. Something like this might vastly improve the expressiveness, and scalability.

49 points by lihaoyi 11y ago

_ZeD_ 11y ago

If you don't mind using python, the sh package [0] can really get you a bash-style repl (dinamically) typed. .. [0]: http://amoffat.github.io/sh/ http://amoffat.github.io/sh/

loevborg 11y ago
Yes, in my mind this is the way to go (the other alternative, probably unviable, being scsh https://github.com/scheme/scsh https://github.com/scheme/scsh). I recently made a decision not to write any bash scripts that include a conditional or loop; for all those use cases python+sh is the way to go. The reason: http://mywiki.wooledge.org/BashPitfalls http://mywiki.wooledge.org/BashPitfalls I can't think of any environment where python would not be available. You can even ship sh.py as a single-file dependency along with your script, so `import cp from sh` will work as expected.
- Johnny_Brahms 11y ago
  Haha! Just reading this thread I got the idea to replace bash with something scheme-like. You have a shell which works like it usually soes, but that easily let's you write any utilities in scheme. I use scheme to write most of my system stuff (most revently a custom automounter for my raspberry Pi) and thought it would have been a nice project. You could probably go macro galore and make it quite comfortable to do piping and integrate the regular system utilities in the language.
kseistrup 11y ago
There's also a bash-like Python REPL like Xonsh ⌘ http://xonsh.org/ http://xonsh.org/ Still in early development, but very useful.

manojlds 11y ago

That's Powershell in the Windows world.

bentronic 11y ago
Powershell is exactly what bash should be (and this person wants). Their time would have better spent using and improving the pash project (open source Powershell on Mono): https://github.com/Pash-Project/Pash https://github.com/Pash-Project/Pash
- alloyed 11y ago
  Powershell is primarily a repl to a homogeneous .NET-based set of tools, which doesn't describe Unix very well. If you want a type-safe repl that interacts with all the handy tools written for the text-stream-only Unix world you're going to have to do some interpretation.
GlennS 11y ago
Powershell has some pitfalls of its own (I still much prefer it to Bash despite these): 1. Dynamic scoping 2. Squashing lists of size 0 to nulls and size 1 to single elements when they come out of pipes. 3. Incomprehensible API for XML handling. On the other hand, when a Powershell program starts getting too big, it's pretty trivial to port parts of it to one of the other .NET languages.
- lihaoyi 11y ago
  > 3. Incomprehensible API for XML handling. If you think Scala has a better API for XML handling then you're in for a surprise...
- manojlds 11y ago
  Very true pitfalls, and there are other ones like any non-captured value anywhere in a function becomes the output of the function.

joostdevries 11y ago

Li Haoyi has been doing a lot of cool open source Scala work. Ammonite is an example of that. You can find it here https://github.com/lihaoyi/Ammonite https://github.com/lihaoyi/Ammonite

cmrdporcupine 11y ago
I for one would not to pay the cost of starting the JVM up every time I want to invoke a shell script.

black_knight 11y ago

Some people do their shell scripting in a pure, functional, strictly typed language [0] — and I don't judge them! [0] http://www.haskellforall.com/2015/01/use-haskell-for-shell-scripting.html http://www.haskellforall.com/2015/01/use-haskell-for-shell-s...

lihaoyi 11y ago
That's cool! We should probably join forces. After all, Martin Odersky has stated that moving people onto Haskell was the primary motivation for inventing Scala, which will soon be renamed Hascalator https://twitter.com/myatlyuka/status/577625111575969792 https://twitter.com/myatlyuka/status/577625111575969792 At the same time, Typesafe is apparently being rename Reactive Banana, which is more evidence of the increased Haskell focus within the community https://www.reddit.com/r/scala/comments/36ho4u/breaking_typesafe_will_be_renamed_to_reactive/ https://www.reddit.com/r/scala/comments/36ho4u/breaking_type... EDIT: found a better slide for Hascalator
- nine_k 11y ago
  My friends answered immediately: «Saying that Scala is a gateway drug to Haskell is like saying that poor-quality heroin cut with speed is а gateway drug to LSD».

eggie 11y ago

Shell languages are defined by their terseness. The author misunderstands how important this is: https://docs.google.com/presentation/d/11vZzXCfAA0aOFAuHA0nAvAzALGFGCH-dqHxx6XMgbk8/preview?sle=true&slide=id.ga54359adc_1_218 https://docs.google.com/presentation/d/11vZzXCfAA0aOFAuHA0nA... The author has created a radically "better" shell language which is twice as verbose as bash. In the space of human-computer interaction where shell languages lie, this is tantamount to total failure. The worst parts of shell are the flow control structures (if/else...). If only we could improve this without throwing out the entire system. Else we will lose the benefits of the ultra-terse and completely decentralized "language" that is unix shell.

hyperpape 11y ago
This is mostly true, and it doesn't look like this is the solution. My belief is any solution needs to be built from the ground up as a language for shell use, not adapting Python/Scala/Haskell to shell use. However, there is some non-zero amount of verbosity that would be worth trading for better control flow, composability, data structures, etc. As long as you can do simple operations and pipes with the same syntax | grep foo cp * path/ rm -rf it's ok if some operations are more verbose in exchange for that. Programmers don't live in bash, and deal with the pain points of switching between bash and an actual programming language on a regular basis. I know I'm there. A change that sometimes is more verbose but lets you do more things in shell would still have benefits. You might not win over the people who are bash experts, but you could still produce a better tool for many people.
klibertp 11y ago
I didn't check it out yet, but I think Tulip (http://www.jneen.net/posts/2015-07-29-tulip-language-updated http://www.jneen.net/posts/2015-07-29-tulip-language-updated) has a great potential in this space.
sambe 11y ago
The author explicitly states terseness is a goal, and compares alternatives' overhead to bash. However, he recognises there is a limit whilst remaining safe/close to Scala. I don't agree control structures are a major problem. There are subtle differences between correct and incorrect expressions, and they are not checked at "compile" time.

hello_there 11y ago

Cool stuff. I really like the idea, but why so many symbols? On slide "2.5 Piping" I find the description on the right to be far more readable than the examples on the left. Also the descriptions on the right are familiar, idiomatic Scala which is also a plus. The stuff on the left is impossible to decipher without some documentation and I don't see how it adds value except from saving a few chars. OTOH it introduces significant load on the developer that has to look them up (again and again) in the manual. If saving these chars are important then one might defend introducing a single new operator for some well thought-out abstraction, but introducing 7 new operators is just to much for anyone to remember. Especially the two last operators seems entirely redundant as the language already has quite compact syntax for function invocation (as does any language).

agumonkey 11y ago

When I see this unifying of language logic and shell ~logic, as a Lisp head, I can't help but to think of lisp machines: https://www.youtube.com/watch?v=o4-YnLpLgtk https://www.youtube.com/watch?v=o4-YnLpLgtk

jmount 11y ago

Wasn't this one of the points of Perl? New languages seem small, but tend to grow.

heinrich5991 11y ago

Path normalization does not work like that. If `/bin` is a symlink referring to `/usr/bin`, then `/bin/..` is the same as `/usr`, not `/` as the author suggests.

a3n 11y ago

Congratulations on making something cool and useful. But I think it's not Bash vs Ammonite, it's more accurately (Alternatives to Bash) vs (The Ammonite alternative to Bash). I've never worked in an environment that uses sh for large systems. My use is just my daily conversation with my OS and file system. I say things on the command line. Sometimes I say certain things in sequence often enough that I collect them into personal "jargon" scripts of two to twenty lines of Bash. For anything larger I generally use Python. My dividing line between Bash and Python (or any similar language) is conversation vs contract. A conversation is relatively informal. Anything larger requires negotiation and agreement by parties (me and the OS) using technically understood formal language (to strain the metaphor). I think Ammonite is more like the contract than the conversation, but maybe not as much as Python.

lihaoyi 11y ago

https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671 is a nice motivating example of why I think this project is useful

therealmarv 11y ago

Using a lot of shell can really be dangerous. When you are on the ops side you know how to avoid most of this pitfalls by using tools like Ansible, Puppet or Chef.

zimbatm 11y ago

If anything is going to displace bash at this point it's going to be something that brings substantial benefits to the user. Stricter syntax is good but not of the immediate concern to the user typing his 2x-longer command. The next evolution of shell is going to be something different. Something that doesn't depend on an antiquated TTY protocol, something that can display images and other constructs more complex that text, and more importantly something that allows the user to accumulate commands instead of typing the same thing over and over again. Maybe something like ipython/jupyter

pekk 11y ago
Using languages other than shell already brings lots of benefits to the user. That just doesn't matter to the people who are writing large programs in shell. It isn't like those people are basing their decision on rigorous user studies.
Aleman360 11y ago
PowerShell
hyperpape 11y ago
There's the shell language and the shell interface. The design of those things influence each other, but they're fundamentally separate even though bash/zsh/fish often make changes to both. The original article is really about languages, and I think that even with the best interface in the world, we want a better language.
- zimbatm 11y ago
  Just changing the language is in the diminishing return area and is why it hasn't happened yet. Something else need to change. Any sort of shell programming is going to be full of side-effects. No matter how strict the language, building the script will still involve trying on the target system, maybe accumulating shell commands in a script, and then have these long full cycles where the script is tested on a fresh system to make sure it's still working. This is happening every time I do any sort of sysadmin-type of stuff. Even in ansible, chef, docker, ... To escape that something deeper needs to be changed.

tunesmith 11y ago

Pretty cool... more documentation and examples here: http://lihaoyi.github.io/Ammonite/ http://lihaoyi.github.io/Ammonite/

KuhlMensch 11y ago

I have a large multi-concern shell script for building a web-app. The "complexity" seems to impress my workmates, but I dislike it and avoid extending it if at all possible. Something like this might vastly improve the expressiveness, and scalability.