4 ms·

Shell Has a Forth-Like Quality (2017)

37 points by pcr910303 7y ago

dang 7y ago
Discussed at the time: https://news.ycombinator.com/item?id=13477842 https://news.ycombinator.com/item?id=13477842
- macintux 7y ago
  Thanks, that's some quality discussion. Really impressed with the depth and breadth of the commentary.
  - g82918 7y ago
    I really like oilshell and chubot for that. He has some really great posts on the shell language and parsing.
haolez 7y ago
I’ve done my share of shell tricks in the past and I love them, but I got very disappointed when I noticed that the performance sucks for anything more data intensive in the shell. At least in bash, which was the interpreter that I’ve used. I ended up porting my code to Python and gained at least 10x in performance. But I prefer the shell code for tasks interfacing with system tools. I wonder if there is a shell interpreter around that is really fast.
- chubot 7y ago
  If you have some realistic shell benchmarks, please file a bug and link the files: https://github.com/oilshell/oil https://github.com/oilshell/oil Oil isn't fast at the moment, but I'm currently optimizing it (auto-translating some code to C++, and eventually hand-optimizing other parts.) I run benchmarks on every release, so speed is definitely a goal. I haven't seen that many shell scripts that are "CPU bound" (as opposed to waiting for processes or I/O). (Although now I recall a shell-script-library bundler project that was.) Of course, one big problem with shell is that you often have to spawn an external process like sed or awk to do string manipulation that's just a library call in Python. That could easily be a three order-of-magnitude difference -- e.g. milliseconds vs. microseconds.
  - stephenr 7y ago
    I have a feeling the “bundler” lib you mentioned is mine - we discussed it a bit on lobste.rs a few years ago? Good to see you’re still plugging away on oil!
    - chubot 7y ago
      Yes! I remember it was a good workload because it was doing a lot of string manipulation in pure shell. I'm still interested in more feedback on Oil (e.g. does it still work on the latest version of shell-script-library? :) ) It's changed a lot, and the Oil language now exists! Latest post: You Can Now Try the Oil Language -- http://www.oilshell.org/blog/2019/10/04.html http://www.oilshell.org/blog/2019/10/04.html
      - stephenr 7y ago
        The testing setup for the lib is much improved, with actual unit tests and a built in capability for building/testing against a specific shell. Next time I’m working on it I’ll grab a release of oil and see how it fares!
- haolez 7y ago
  Great! I hope you manage to pull this off! It's a much needed tool in our IT world. I remember seeing something like what you are doing a while back, but it was such a small project that I can't recall its name or find it via Google. But I think it was a shell favoring functional programming style meant for plan9.
Ididntdothis 7y ago
When I see stuff like this I always wonder how people above about finding these capabilities. I don't use bash every day so I tend to forget a lot of "tricks". But then I get annoyed when I see people write about really neat stuff that can be done in bash but couldn't figure out myself.
- chubot 7y ago
  (author here) I agree a lot of this stuff hasn't been documented well in existing shell resources. I should write more about it, but I'm still busy working on the Oil shell implementation nearly 3 years later :) I should have written a post about xargs 2 years ago, which has nontrivial compositional patterns. Hopefully next year! I still recommend The Art of Unix Programming (mentioned in this post), which is more generally about Unix design, not about shell in particular. I also got a lot of ideas by reading DJB's shell and C code, and his papers (e.g. Ten Years of qmail Security). I probably learned about DJB 10 years ago from Hacker News. I would say there's a lot of "tribal knowledge" in people's heads, and online forums are where it gets disseminated. For me those forums were Usenet, the old JOS forum, and Hacker News and lobste.rs these days. ---- Neil Stephenson has a good quote about this "oral history" Unix, by contrast, is not so much a product as it is a painstakingly compiled oral history of the hacker subculture. It is our Gilgamesh epic. https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Command_Line https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Co... https://en.wikipedia.org/wiki/Epic_of_Gilgamesh https://en.wikipedia.org/wiki/Epic_of_Gilgamesh The next post has the "point-free" pipline pattern which is both useful and hard to find out about! hist() { sort | uniq -c | sort -n -r; } Pipelines Support Vectorized, Point-Free, and Imperative Style http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
- loopz 7y ago
  Scripts using ksh/bash are not very maintainable. If it's something simple that just works and never need to change, it's a really good RAD tool. It's good for initiating jobs or do stuff in between. Anything complex, will require looking up some arcane syntax, which the next person or you in 2 years, won't understand what actually does. So will you notice subtle bugs or be able to upgrade or fix the code? The result will be bitrot or adding complexity. The nice things with shell scripts is how fast you can pipe something through UNIX, and it's really really powerful. But regard it as temporary or throw-away code, unless it's just meant to sit there doing the same job, with the same subtle bugs that you won't notice if it's anything more complex.
chubot 7y ago
(author here) Oil has made a lot of progress in the last 3 years, with a few releases every month. http://www.oilshell.org/releases.html http://www.oilshell.org/releases.html The goal of the project is to replace bash! If you know shell and Python (or C++), I maintain a "help wanted" bug label: https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22 https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope... It's almost there, except for speed, but it needs more testing from users. The "should-run-this" label has some ideas: https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3Ashould-run-this https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope... There a lot of project updates on https://oilshell.zulipchat.com/ https://oilshell.zulipchat.com/ too (requires login)
- zomglings 7y ago
  Very curious - what kind of people use OSH and how many actually use Oil in OSH? I am hesitant to spend the time ramping up on something like this because, as much as possible, I try to write my shell scripts so that they can run without any additional packages in an Alpine docker container. Is this resistance that you have encountered before?
  - chubot 7y ago
    Right now it's mainly a few shell enthusiasts and early adopters, i.e. people who want a better shell and want to influence the design of the Oil language. [1] I've definitely seen people who want to limit themselves to either bash or POSIX shell on purpose, and that's fine. For simple shell scripts, I might lean that way too (at least until Oil is smaller/faster). One major reason for Oil is to give you an upgrade path after you've used and enhanced such a shell script for years. Then you will likely bump up into shell's limitations, and the extra dependency will be worth it. For example, all Linux distros are based on thousands of lines of shell scripts [2], and I think those scripts desperately need a new language. They're big and complex, and need a better language to be correct, robust, and fast. Here's a comment I wrote that might explain the design goals of the project more: https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj9/you_can_now_try_the_oil_language/f2htup3/ https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj... [1] Oil Language Design Notes #1 http://www.oilshell.org/blog/2019/08/22.html http://www.oilshell.org/blog/2019/08/22.html [2] Success with Aboriginal, Alpine, and Debian Linux http://www.oilshell.org/blog/2018/01/15.html http://www.oilshell.org/blog/2018/01/15.html

37 points by pcr910303 7y ago

dang 7y ago

Discussed at the time: https://news.ycombinator.com/item?id=13477842 https://news.ycombinator.com/item?id=13477842

macintux 7y ago
Thanks, that's some quality discussion. Really impressed with the depth and breadth of the commentary.
- g82918 7y ago
  I really like oilshell and chubot for that. He has some really great posts on the shell language and parsing.

haolez 7y ago

I’ve done my share of shell tricks in the past and I love them, but I got very disappointed when I noticed that the performance sucks for anything more data intensive in the shell. At least in bash, which was the interpreter that I’ve used. I ended up porting my code to Python and gained at least 10x in performance. But I prefer the shell code for tasks interfacing with system tools. I wonder if there is a shell interpreter around that is really fast.

chubot 7y ago
If you have some realistic shell benchmarks, please file a bug and link the files: https://github.com/oilshell/oil https://github.com/oilshell/oil Oil isn't fast at the moment, but I'm currently optimizing it (auto-translating some code to C++, and eventually hand-optimizing other parts.) I run benchmarks on every release, so speed is definitely a goal. I haven't seen that many shell scripts that are "CPU bound" (as opposed to waiting for processes or I/O). (Although now I recall a shell-script-library bundler project that was.) Of course, one big problem with shell is that you often have to spawn an external process like sed or awk to do string manipulation that's just a library call in Python. That could easily be a three order-of-magnitude difference -- e.g. milliseconds vs. microseconds.
- stephenr 7y ago
  I have a feeling the “bundler” lib you mentioned is mine - we discussed it a bit on lobste.rs a few years ago? Good to see you’re still plugging away on oil!
  - chubot 7y ago
    Yes! I remember it was a good workload because it was doing a lot of string manipulation in pure shell. I'm still interested in more feedback on Oil (e.g. does it still work on the latest version of shell-script-library? :) ) It's changed a lot, and the Oil language now exists! Latest post: You Can Now Try the Oil Language -- http://www.oilshell.org/blog/2019/10/04.html http://www.oilshell.org/blog/2019/10/04.html
    - stephenr 7y ago
      The testing setup for the lib is much improved, with actual unit tests and a built in capability for building/testing against a specific shell. Next time I’m working on it I’ll grab a release of oil and see how it fares!
haolez 7y ago
Great! I hope you manage to pull this off! It's a much needed tool in our IT world. I remember seeing something like what you are doing a while back, but it was such a small project that I can't recall its name or find it via Google. But I think it was a shell favoring functional programming style meant for plan9.

Ididntdothis 7y ago

When I see stuff like this I always wonder how people above about finding these capabilities. I don't use bash every day so I tend to forget a lot of "tricks". But then I get annoyed when I see people write about really neat stuff that can be done in bash but couldn't figure out myself.

chubot 7y ago
(author here) I agree a lot of this stuff hasn't been documented well in existing shell resources. I should write more about it, but I'm still busy working on the Oil shell implementation nearly 3 years later :) I should have written a post about xargs 2 years ago, which has nontrivial compositional patterns. Hopefully next year! I still recommend The Art of Unix Programming (mentioned in this post), which is more generally about Unix design, not about shell in particular. I also got a lot of ideas by reading DJB's shell and C code, and his papers (e.g. Ten Years of qmail Security). I probably learned about DJB 10 years ago from Hacker News. I would say there's a lot of "tribal knowledge" in people's heads, and online forums are where it gets disseminated. For me those forums were Usenet, the old JOS forum, and Hacker News and lobste.rs these days. ---- Neil Stephenson has a good quote about this "oral history" Unix, by contrast, is not so much a product as it is a painstakingly compiled oral history of the hacker subculture. It is our Gilgamesh epic. https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Command_Line https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Co... https://en.wikipedia.org/wiki/Epic_of_Gilgamesh https://en.wikipedia.org/wiki/Epic_of_Gilgamesh The next post has the "point-free" pipline pattern which is both useful and hard to find out about! hist() { sort | uniq -c | sort -n -r; } Pipelines Support Vectorized, Point-Free, and Imperative Style http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
loopz 7y ago
Scripts using ksh/bash are not very maintainable. If it's something simple that just works and never need to change, it's a really good RAD tool. It's good for initiating jobs or do stuff in between. Anything complex, will require looking up some arcane syntax, which the next person or you in 2 years, won't understand what actually does. So will you notice subtle bugs or be able to upgrade or fix the code? The result will be bitrot or adding complexity. The nice things with shell scripts is how fast you can pipe something through UNIX, and it's really really powerful. But regard it as temporary or throw-away code, unless it's just meant to sit there doing the same job, with the same subtle bugs that you won't notice if it's anything more complex.

chubot 7y ago

(author here) Oil has made a lot of progress in the last 3 years, with a few releases every month. http://www.oilshell.org/releases.html http://www.oilshell.org/releases.html The goal of the project is to replace bash! If you know shell and Python (or C++), I maintain a "help wanted" bug label: https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22 https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope... It's almost there, except for speed, but it needs more testing from users. The "should-run-this" label has some ideas: https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aopen+label%3Ashould-run-this https://github.com/oilshell/oil/issues?q=is%3Aissue+is%3Aope... There a lot of project updates on https://oilshell.zulipchat.com/ https://oilshell.zulipchat.com/ too (requires login)

zomglings 7y ago
Very curious - what kind of people use OSH and how many actually use Oil in OSH? I am hesitant to spend the time ramping up on something like this because, as much as possible, I try to write my shell scripts so that they can run without any additional packages in an Alpine docker container. Is this resistance that you have encountered before?
- chubot 7y ago
  Right now it's mainly a few shell enthusiasts and early adopters, i.e. people who want a better shell and want to influence the design of the Oil language. [1] I've definitely seen people who want to limit themselves to either bash or POSIX shell on purpose, and that's fine. For simple shell scripts, I might lean that way too (at least until Oil is smaller/faster). One major reason for Oil is to give you an upgrade path after you've used and enhanced such a shell script for years. Then you will likely bump up into shell's limitations, and the extra dependency will be worth it. For example, all Linux distros are based on thousands of lines of shell scripts [2], and I think those scripts desperately need a new language. They're big and complex, and need a better language to be correct, robust, and fast. Here's a comment I wrote that might explain the design goals of the project more: https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj9/you_can_now_try_the_oil_language/f2htup3/ https://www.reddit.com/r/ProgrammingLanguages/comments/ddhmj... [1] Oil Language Design Notes #1 http://www.oilshell.org/blog/2019/08/22.html http://www.oilshell.org/blog/2019/08/22.html [2] Success with Aboriginal, Alpine, and Debian Linux http://www.oilshell.org/blog/2018/01/15.html http://www.oilshell.org/blog/2018/01/15.html