10 ms·

Oil Shell: Success with Aboriginal, Alpine, and Debian Linux

125 points by chubot 9y ago

detuur 9y ago
Someone should push a patch to upstream bash that will make any script longer than 50 lines abort. If you're building something that takes 2500 lines of shell script, you're using the wrong tool for the job. You're trying to make a program, but in a syntax (I was going to say language, but shell scripts aren't a programming language) that is utterly unsuited for anything like it. A shell script has a very simple function: execute a number of tedious but common commands. That's it. Unfortunately someone decided they wanted control flow, so they wrote the horrifying [ and [[ programs, which I still consider to be a crime against humanity. Enter decades of write-once-read-never scripts that run with abysmal efficiency, kill maintainability, and neuter innovation. And somehow the entire Linux ecosystem depends on it and now every distro has to ship a shell with cruft dating back to 1970, with probably as many hidden vulnerabilities. And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered. Let shells be shells, let shell scripts be dumb as bricks, and please use python or even perl for the love of god if you're going to make a build system.
- xellisx 9y ago
  One could argue that HTML is a programming language. https://www.youtube.com/watch?v=4A2mWqLUpzw https://www.youtube.com/watch?v=4A2mWqLUpzw
  - tacon 9y ago
    Peter van Roy puts XML and S-expressions in the "descriptive declarative programming" paradigm, with data structures only (no Turing equivalence): https://www.info.ucl.ac.be/~pvr/paradigmsDIAGRAMeng108.jpg https://www.info.ucl.ac.be/~pvr/paradigmsDIAGRAMeng108.jpg
- chubot 9y ago
  I see this sentiment a lot, but it's not a useful one, as that ship has already sailed. Please read this comment: https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_release/dromj6n/ https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_... It's not a small amount of shell, and it's not even old in some cases. People are still writing big pieces of shell code. I point out at the end of this comment that Kubernetes, a brand new cluster manager from Google (2014 or so), has 48,000+ lines of shell in its repo. Shell is definitely a terrible language for many things, but that's what I'm trying to fix, in a way that doesn't force you to rewrite all your code. Some things will never be rewritten -- it's like saying "Hey why don't you rewrite Wikipedia in Python and get rid of all the PHP?" It will never happen, for fundamental reasons of economics. Also see this comment for more of the "why": https://www.reddit.com/r/commandline/comments/7c3f9f/osh_02_parsing_one_million_lines_of_shell/dposm7r/ https://www.reddit.com/r/commandline/comments/7c3f9f/osh_02_... I plan to write these up on the blog, as mentioned at the end of this post.
  - bch 9y ago
    > I see this sentiment a lot, but it’s not a useful one... Consider this response for entertainment purposes only. It’s easy to become insult(ed|ing) or get into a holy war, but I’m really not invested in any of that. That said: it is a useful sentiment, and a kubernetes script written in sh(1) is not the answer to the question. That one “can” does not mean one “should”. The reddit comment cites init scripts (I happen to use RC scripts, and haven’t viewed init scripts in a while) as an example of why shell is so great - everybody uses init! 1) of course everybody does. And it -happens- to be written in shell. So what? 2) this is fine! The scripts I see are 10s of lines long and practically flow top->bottom w maybe one or two conditionals. This is more akin to a job control language. A perfect application for sh(1). I just feel it (and you can too): sh(1) (or bash - nobody cares) features feel archaic and brittle. People have written impressive work in brainfuck[0][1], or C that just emits mov[2] instructions; is that a case for brainfuck projects or mov-only assembly? No. It’s just interesting. And kubernetes scaffolding or anything else big that happens (by some miracle) to work are not excuses for huge shell scripts. That’s just tautology - otherwise we’d be talking about the virtue of these scripts. For a boot sequence (rc/init) even if we ignore that the short scripts are actually suitable for the important job, I’d say their other grace is that sh(1) is demanded by POSIX, so you don’t need to wonder if it’s available in this nascent state of booting your OS - you’re guaranteed. The same couldn’t be said for ruby or python. From the “see this comment” comment: > Have you ever written a bash completion script yourself? I did a few years ago and I am scared to ever touch it again! I have a non-deterministic bug that I can't figure out, so I just restart my terminal every time completion gets borked. And I know bash very well. The comment admits what a warzone the space is... Sure, fix it, but we don’t need to promote it. The sentiment of discouraging huge shell scripts the GP expressed is useful. Resist. [0] https://en.wikipedia.org/wiki/Brainfuck https://en.wikipedia.org/wiki/Brainfuck [1] https://rosettacode.org/wiki/Category:Brainf*** https://rosettacode.org/wiki/Category:Brainf*** [2] https://youtu.be/R7EEoWg6Ekk https://youtu.be/R7EEoWg6Ekk
- LeoPanthera 9y ago
  > shell scripts aren't a programming language Please explain why. Because simple observation seems to prove the opposite.
- nerdponx 9y ago
  Tell that to the Zsh community. Or the community for any embedded scripting language -- Emacs Lisp, Vimscript, etc. Zsh, especially with 'set -eu', is a perfectly suitable, albeit slow, replacement for Perl, which is in my opinion more distasteful.
  - majewsky 9y ago
    Another example: Arch Linux is an entire distribution developed (at least 90%) in shell scripts. For example, the installer [1] is three shell scripts of 300-400 lines each. Packages are built with `makepkg`, which is a 2400-line shell script. It helps that the Arch developers are good at writing clean shell scripts. [1] In the package "arch-install-scripts".
    - chubot 9y ago
      Yes, Alpine's APKBUILD seems to be very much modelled after PKGBUILD, and "abuild" sounds analogous to "makepkg". It's also 100% shell, but it uses busybox ash rather than bash. I prefer pure hand-written shell to an unholy mix of auto-generated shell and Make. The latter is what I remember Debian's build system to be, although admittedly I haven't looked at it in awhile.
      - jhasse 9y ago
        Alpine's apk-build was also written in Shell, but rewritten in C: https://github.com/alpinelinux/apk-tools https://github.com/alpinelinux/apk-tools I wonder what the reason was.
        chubot 9y ago
        There is "abuild" to build packages, and "apk" for users to install packages. abuild is definitely still in shell (~2600 lines). I'm not sure if apk used to be in shell, but I wouldn't be surprised if it were. If you have a link to the old version I'm interested. I think the main reason not to use shell for the user side is the version solver. To install packages, you have to solve dependency constraints, which is actually an NP-complete problem! Those heuristics are best coded in C. In contrast, the build side doesn't have to do anything like that, or it can just shell out to "apk" if it does.
- effie 9y ago
  So much arrogance and negativity, so little evidence for the claims you make. > A shell script has a very simple function: execute a number of tedious but common commands. That's it. And why is that? What will happen to me if I write more complicated shell scripts? The functionality is there and one can write very expressive and readable programs in shell. I would definitely recommend shell for writing long programs (for example various system administration and web backend tasks which do not require complex libraries). Preferably bash since it is the most common one. > And somehow the entire Linux ecosystem depends on it and now every distro has to ship a shell with cruft dating back to 1970 The 'somehow' has quite a simple rationale, it isn't something that should outrage you. The family of Linux-based OSes take inspiration and compatibility target in unix systems on purpose. Unix systems are primarily operated via shell. The current standard of shell may be an old idea from 70's with some 'cruft', but what language does not have cruft? The shell has proven to be usable and worthy of continued use. The compatibility with the unix shell is a big part of the early success of GNU/Linux and majority of users need it. > And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered. I have no idea what you are talking about. > please use python or even perl for the love of god if you're going to make a build system OK, so you think perl and python are better for making a build system. Have you made one?
  - paulie_a 9y ago
    > What will happen to me if I write more complicated shell scripts? As part of my job currently, I am going to delete it. The same goes for poor python code, but I will take bad python over shell scripts any day of the week.
    - AlexCoventry 9y ago
      > As part of my job currently, I am going to delete it. Go away, or I will turn you into a very small shell script. #!/bin/bash find / -name \*.sh -size +10k -type f -print0 | xargs -0 rm
      - asfdsfggtfd 9y ago
- cozzyd 9y ago
  That would have the nice side effect of killing the systemd/init debate. (Disclaimer: I was foolish enough to write a continuous integration script in bash).
  - flukus 9y ago
    What's wrong with a continuous integration script in bash? A basic CI script just has to check source control periodically, run the build (make or something similar) and notify anyone if there's an error and maybe publish the result somewhere. I've done similar with the far less capable batch files. Sure it's not as great as a full on CI server, but it doesn't require the same investment either. For my personal projects I have bash scripts to handle basic CI (compiling and testing), given a bit of time I think I could string cron, make, bash and m4 into a CI server that fits 90%+ of my needs.
    - cozzyd 9y ago
      Oh it was great in the sense that it was very quick to write and it works... but it is not very easy to graft on new features.
  - chubot 9y ago
    FWIW I wrote a post that touched on that debate last year: http://www.oilshell.org/blog/2017/01/13.html http://www.oilshell.org/blog/2017/01/13.html (I didn't come down on either side of the debate -- it's more of an exploration.) Comments: https://news.ycombinator.com/item?id=13477842 https://news.ycombinator.com/item?id=13477842
- arca_vorago 9y ago
  As a sysadmin you can pry the ba(sh) scripts out of my cold dead hands. I'm not a dev, I'm not really a programmer, I'm the one who has to support all the crazy shit you devs are throwing at the wall to see what sticks. Bash is how I get shit done. I'm just tired of bash getting bashed everytime someone mentions shell scripting, based on what mostly seems to be bandwagon reasoning. Of course if you are writing more than $arbitrarycomplexity drop into perl, etc, but there is so much that can be done quickly, easily, and readably even for a non-programmer like myself that I think there are many benefits to using shell scripts that get far too often ignored. Also, a perfect example of a great bash script that's over 2k loc, and I've got plenty more were that came from: https://github.com/centminmod/centminmod/blob/master/centmin.sh https://github.com/centminmod/centminmod/blob/master/centmin...
  - agumonkey 9y ago
    Don't take this bad but when you're not a programmer it's hard to understand how 2k LoC are probably 50 LoC done wrong. It's a self fulfilling prophecy, since programmers spends their day finding abstraction to turn long and windy into concise, but for the rest of the world we're just lunatics. Now I'm not as pushy as OP, but it's far from baseless bandwagon.
    - majewsky 9y ago
      > it's hard to understand how 2k LoC are probably 50 LoC done wrong I challenge you to reimplement the shell script that grandparent linked in 50 LOC. In fact, shell script has the chance to be a lot more succinct than other languages for a lot of administrative tasks since the invocation of other programs is a first-class citizen. For example, compare this Go snippet: cmd := exec.Command("apt", append([]string{"install"}, dependencies...)) cmd.Stdin = os.Stdin cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr err := cmd.Run() if err != nil { fmt.Fprintln(os.Stderr, err.Error()) os.Exit(1) } to its equivalent in shell script: if ! apt install ${DEPENDENCIES[@]}; then echo "exit status: $!" >&2 exit 1 fi Or even just "apt install ${DEPENDENCIES[@]}" with "set -e". Finally, what is it with sysadmins maintaining large scripts while at the same time quipping that they are not programmers? A script is very much a program, and a scripting language is very much a programming language.
- majewsky 9y ago
  > And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered. That's patently false: https://fishshell.com https://fishshell.com
  - Chris2048 9y ago
    fish is praised a lot, but how often is it used? https://github.com/fish-shell/fish-shell/wiki/POSIX-compatibility-thoughts https://github.com/fish-shell/fish-shell/wiki/POSIX-compatib...
- teekert 9y ago
  That would be pretty hard to sell to the bioinformatics crowd. My Bash scripts are long, but easy to read. I'm just setting some environment variables, running some algorithms (many output to stdout and can be piped directly to other algorithms), moving the results around, write to a log here and there. Easily over 50 lines. Why would you force me into Python or what ever? I'm just gluing different programs together and moving files around.
- hzhou321 9y ago
  I have always been wondering why don't we have shell program that don't do scripts -- no flow control and fancy macros, and separate program that run scripts -- no readline or job control (while pipes and redirects are simply treated as syntax)?
  - geocar 9y ago
    We tried that. It was called the C-shell. Most people would use it on the command line and write scripts in Bourne shell, but some people kept trying to write scripts in C-shell for some incredibly stupid reason, and so it got banned.
    - darrenf 9y ago
      Indeed, I have vivid memories of reading the chapter in Unix Power Tools devoted to this very topic: https://docstore.mik.ua/orelly/unix2.1/upt/ch47_01.htm https://docstore.mik.ua/orelly/unix2.1/upt/ch47_01.htm
    - hzhou321 9y ago
      Good to learn! I always thought C-shell was aimed at scripting part (due to the sound of "C"); I guess I missed the boat.
- v_lisivka 9y ago
  Example script: #!/bin/bash trap '(( LINENO < 10+3 )) || { echo "Script executed more than 10 lines of code! Aborting."; exit 88; }' DEBUG echo 1 echo 2 echo 3 echo 4 echo 5 echo 6 echo 7 echo 8 echo 9 echo 10 echo 11 $ ./test.sh 1 2 3 4 5 6 7 8 9 10 Script executed more than 10 lines of code! Aborting.
eltoozero 9y ago
Sometimes - perhaps often - you are in situations where you have no control over the shell, in these cases writing "portable" shell scripts is helpful. I have been enjoying "Beginning Portable Shell Scripting"[0] by Peter Seebach for this very reason. [0]https://www.apress.com/us/book/9781430210436 https://www.apress.com/us/book/9781430210436
- majewsky 9y ago
  $ cat run-bash-script.sh #!/bin/sh which bash &>/dev/null && exec bash "$@" source /etc/os-release case "$ID" in debian) apt install bash || exit 1 ;; redhat|suse) yum install bash || exit 1 ;; arch) pacman -S bash || exit 1 ;; alpine) apk add bash || exit 1 ;; esac exec bash "$@" I don't know how to detect and handle the BSDs etc., but you get the idea. ;)
  - chubot 9y ago
    Also known as, "it's easier to port a shell than a shell script" :)
  - deadbunny 9y ago
    Now go try that on AIX boxes, in banks, where you don't get root/sudo.
    - oblio 9y ago
      You also don't get internet access, in case a bright bulb tries to go "curl ...". Though in many of these cases I'd just advise leaving the premises unless the salary is stellar (or there are other considerations such as family, etc).
      - deadbunny 9y ago
        On the DevOps side of things (Contracting in London) the day rate at banks is about 1.5x everywhere else which can be nice. Just as long as you're happy never getting anything done ever.
axlprose 9y ago
Would be interesting to test this out on SourceMage[0], seeing as how its package manager is written entirely in bash. [0] https://sourcemage.org/Intro https://sourcemage.org/Intro
- chubot 9y ago
  Interesting, I didn't know about SourceMage. It appears to be a source-based distro like Gentoo, and if it's package manager is written in bash, it's a perfect use case. Since I have my hands full with the distros I'm using, I may not be able to try it for awhile, but I saved this page for further reading. I still think distros are somewhat "broken", so alternative approaches are interesting to me. (Just one problem off the top of my head: packages are old, distros can't keep up with PyPI, npm, CRAN, etc.) If you know the SourceMage code, please try it with OSH :) Or if you know its developers, they might want to hear about a cleaner shell that's compatible with bash (although it's nascent) I spent some time last fall implementing some constructs that the Nix package manager uses, even though I don't use Nix. Nix uses bash heavily as well.
  - igravious 9y ago
    Hey, love your project's ambition but I think it's unhelpful if you say things like, “ I still think distros are somewhat "broken", so alternative approaches are interesting to me.” (Think of all the work you've put into your shell project, now think of how much integration work must be done to keep a distro going.) That said, I think you've highlighted _the_ problem that _all_ distros face that would get me to change distros if someone solved it. That is to say, “(Just one problem off the top of my head: packages are old, distros can't keep up with PyPI, npm, CRAN, etc.)” That is a doozy of a problem. One last thing–you mention four things: The trap builtin is unimplemented; alias is also unimplemented. set -h / hashall is a stub that does nothing. OSH builds are in a sense "shallow". Is there some way to get OSH to be un-shallow? For instance if I've invoked a shell script with OSH I'd probably like any shell script dialect that it can handle to have OSH invoked as well. Also, surely it would help you out with debugging OSH because you'd immediately way more lines of shell script! Implementing `trap', `alias', `set -h' seem like no-brainers prior to 0.4
jitl 9y ago
Here's what the plan seems to be from reading the blog and the README: 1. Re-implement Bash and similar shells in a modern programming environment with an eye to extensibility 2. Make good debugging tools because this is now easier because (1). 3. Create a new language that extends Bash, similar to how ES6 extends ES5, for a gentle migration of large shell scripts to a safer language What, other than static parsing, should this endgame language offer? Where does it fall, on the gamut from Bash to Haskell? We already have quite a few languages that offer a souped-up scripting experience. When my bash programs become too long, I usually reach for Ruby, which has convenient back-ticks syntax for shelling out, and many convenience methods on its built-in types for mashing strings and such. But like Bash, too much ruby eventually becomes an unmaintainable mess without strict standards enforcement. What will Oil offer for me, over Ruby? Here's how I'd order existing languages on a scale from Bash to Haskell: 1. Bash with `set -e` 2. Perl 3. TCL (has declared arguments, after all...) 4. Ruby 5. Python 6. Golang 7. Java 8. C++ 9. Ocaml 10. Haskell
- comstock 9y ago
  I agree, I personally can justify moving to a new shell. Bash is horrific to script in, compared to most everything else. It’s sole advantage for me is that it’s almost literally the default everywhere.
  - cturner 9y ago
    A fun problem I play with sometimes: eek out a small-but-super-powerful subset of standard shell tools (sh, awk). For example, if you could create a forth in a dozen lines of bash, then you could just recreate it whenever you needed it. Then you could start refactoring code to it. Easier than a rewrite. I got quite far along at building a message-driven paradigm in awk. The input is a stream of well-defined messages. As you read it, append new messages to the end of the stream. I found that the awk I was using (gnu I think) was complicated by IO buffering behaviour. Still, an interesting problem space. Lex may offer strong opportunities too, although is not part of the posix base.
    - 9y ago
dx034 9y ago
Hmm, relying on Python 2 for a program that's still in early stages and is supposed to have a long lifespan?
- chubot 9y ago
  I plan to cut the dependency on Python; the post called "the riskiest part of the project" which I linked describes that. http://www.oilshell.org/blog/2017/04/08.html http://www.oilshell.org/blog/2017/04/08.html There are some detail here in these posts: http://www.oilshell.org/blog/tags.html?tag=opy#opy http://www.oilshell.org/blog/tags.html?tag=opy#opy FWIW, Oil was in Python 3 at one point! But I ported it back to Python 2. There were a number of reasons, but unicode handling is one of them. See this comment: https://www.reddit.com/r/ProgrammingLanguages/comments/7elxlv/python_3_and_firefox_57_an_observation/dq6ixqu/ https://www.reddit.com/r/ProgrammingLanguages/comments/7elxl... Summary: The shell deals with strings from file systems, and file systems inherently have no encoding. So Python 3's model of unicode doesn't help with such programs. It just makes things more awkward.
  - earenndil 9y ago
    Can you just use binary strings?
    - chubot 9y ago
      It's possible (see the reddit comment), but there's no advantage to doing so.
sandGorgon 9y ago
You should integrate with Docker Linuxkit https://github.com/linuxkit/linuxkit/issues/161 https://github.com/linuxkit/linuxkit/issues/161

125 points by chubot 9y ago

Someone should push a patch to upstream bash that will make any script longer than 50 lines abort. If you're building something that takes 2500 lines of shell script, you're using the wrong tool for the job. You're trying to make a program, but in a syntax (I was going to say language, but shell scripts aren't a programming language) that is utterly unsuited for anything like it. A shell script has a very simple function: execute a number of tedious but common commands. That's it. Unfortunately someone decided they wanted control flow, so they wrote the horrifying [ and [[ programs, which I still consider to be a crime against humanity. Enter decades of write-once-read-never scripts that run with abysmal efficiency, kill maintainability, and neuter innovation. And somehow the entire Linux ecosystem depends on it and now every distro has to ship a shell with cruft dating back to 1970, with probably as many hidden vulnerabilities. And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered. Let shells be shells, let shell scripts be dumb as bricks, and please use python or even perl for the love of god if you're going to make a build system.

xellisx 9y ago
One could argue that HTML is a programming language. https://www.youtube.com/watch?v=4A2mWqLUpzw https://www.youtube.com/watch?v=4A2mWqLUpzw
- tacon 9y ago
  Peter van Roy puts XML and S-expressions in the "descriptive declarative programming" paradigm, with data structures only (no Turing equivalence): https://www.info.ucl.ac.be/~pvr/paradigmsDIAGRAMeng108.jpg https://www.info.ucl.ac.be/~pvr/paradigmsDIAGRAMeng108.jpg
chubot 9y ago
I see this sentiment a lot, but it's not a useful one, as that ship has already sailed. Please read this comment: https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_release/dromj6n/ https://www.reddit.com/r/linux/comments/7lsajn/oil_shell_03_... It's not a small amount of shell, and it's not even old in some cases. People are still writing big pieces of shell code. I point out at the end of this comment that Kubernetes, a brand new cluster manager from Google (2014 or so), has 48,000+ lines of shell in its repo. Shell is definitely a terrible language for many things, but that's what I'm trying to fix, in a way that doesn't force you to rewrite all your code. Some things will never be rewritten -- it's like saying "Hey why don't you rewrite Wikipedia in Python and get rid of all the PHP?" It will never happen, for fundamental reasons of economics. Also see this comment for more of the "why": https://www.reddit.com/r/commandline/comments/7c3f9f/osh_02_parsing_one_million_lines_of_shell/dposm7r/ https://www.reddit.com/r/commandline/comments/7c3f9f/osh_02_... I plan to write these up on the blog, as mentioned at the end of this post.
- bch 9y ago
  > I see this sentiment a lot, but it’s not a useful one... Consider this response for entertainment purposes only. It’s easy to become insult(ed|ing) or get into a holy war, but I’m really not invested in any of that. That said: it is a useful sentiment, and a kubernetes script written in sh(1) is not the answer to the question. That one “can” does not mean one “should”. The reddit comment cites init scripts (I happen to use RC scripts, and haven’t viewed init scripts in a while) as an example of why shell is so great - everybody uses init! 1) of course everybody does. And it -happens- to be written in shell. So what? 2) this is fine! The scripts I see are 10s of lines long and practically flow top->bottom w maybe one or two conditionals. This is more akin to a job control language. A perfect application for sh(1). I just feel it (and you can too): sh(1) (or bash - nobody cares) features feel archaic and brittle. People have written impressive work in brainfuck[0][1], or C that just emits mov[2] instructions; is that a case for brainfuck projects or mov-only assembly? No. It’s just interesting. And kubernetes scaffolding or anything else big that happens (by some miracle) to work are not excuses for huge shell scripts. That’s just tautology - otherwise we’d be talking about the virtue of these scripts. For a boot sequence (rc/init) even if we ignore that the short scripts are actually suitable for the important job, I’d say their other grace is that sh(1) is demanded by POSIX, so you don’t need to wonder if it’s available in this nascent state of booting your OS - you’re guaranteed. The same couldn’t be said for ruby or python. From the “see this comment” comment: > Have you ever written a bash completion script yourself? I did a few years ago and I am scared to ever touch it again! I have a non-deterministic bug that I can't figure out, so I just restart my terminal every time completion gets borked. And I know bash very well. The comment admits what a warzone the space is... Sure, fix it, but we don’t need to promote it. The sentiment of discouraging huge shell scripts the GP expressed is useful. Resist. [0] https://en.wikipedia.org/wiki/Brainfuck https://en.wikipedia.org/wiki/Brainfuck [1] https://rosettacode.org/wiki/Category:Brainf*** https://rosettacode.org/wiki/Category:Brainf*** [2] https://youtu.be/R7EEoWg6Ekk https://youtu.be/R7EEoWg6Ekk
LeoPanthera 9y ago
> shell scripts aren't a programming language Please explain why. Because simple observation seems to prove the opposite.
nerdponx 9y ago
Tell that to the Zsh community. Or the community for any embedded scripting language -- Emacs Lisp, Vimscript, etc. Zsh, especially with 'set -eu', is a perfectly suitable, albeit slow, replacement for Perl, which is in my opinion more distasteful.
- majewsky 9y ago
  Another example: Arch Linux is an entire distribution developed (at least 90%) in shell scripts. For example, the installer [1] is three shell scripts of 300-400 lines each. Packages are built with `makepkg`, which is a 2400-line shell script. It helps that the Arch developers are good at writing clean shell scripts. [1] In the package "arch-install-scripts".
  - chubot 9y ago
    Yes, Alpine's APKBUILD seems to be very much modelled after PKGBUILD, and "abuild" sounds analogous to "makepkg". It's also 100% shell, but it uses busybox ash rather than bash. I prefer pure hand-written shell to an unholy mix of auto-generated shell and Make. The latter is what I remember Debian's build system to be, although admittedly I haven't looked at it in awhile.
    - jhasse 9y ago
      Alpine's apk-build was also written in Shell, but rewritten in C: https://github.com/alpinelinux/apk-tools https://github.com/alpinelinux/apk-tools I wonder what the reason was.
      - chubot 9y ago
        There is "abuild" to build packages, and "apk" for users to install packages. abuild is definitely still in shell (~2600 lines). I'm not sure if apk used to be in shell, but I wouldn't be surprised if it were. If you have a link to the old version I'm interested. I think the main reason not to use shell for the user side is the version solver. To install packages, you have to solve dependency constraints, which is actually an NP-complete problem! Those heuristics are best coded in C. In contrast, the build side doesn't have to do anything like that, or it can just shell out to "apk" if it does.
effie 9y ago
So much arrogance and negativity, so little evidence for the claims you make. > A shell script has a very simple function: execute a number of tedious but common commands. That's it. And why is that? What will happen to me if I write more complicated shell scripts? The functionality is there and one can write very expressive and readable programs in shell. I would definitely recommend shell for writing long programs (for example various system administration and web backend tasks which do not require complex libraries). Preferably bash since it is the most common one. > And somehow the entire Linux ecosystem depends on it and now every distro has to ship a shell with cruft dating back to 1970 The 'somehow' has quite a simple rationale, it isn't something that should outrage you. The family of Linux-based OSes take inspiration and compatibility target in unix systems on purpose. Unix systems are primarily operated via shell. The current standard of shell may be an old idea from 70's with some 'cruft', but what language does not have cruft? The shell has proven to be usable and worthy of continued use. The compatibility with the unix shell is a big part of the early success of GNU/Linux and majority of users need it. > And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered. I have no idea what you are talking about. > please use python or even perl for the love of god if you're going to make a build system OK, so you think perl and python are better for making a build system. Have you made one?
- paulie_a 9y ago
  > What will happen to me if I write more complicated shell scripts? As part of my job currently, I am going to delete it. The same goes for poor python code, but I will take bad python over shell scripts any day of the week.
  - AlexCoventry 9y ago
    > As part of my job currently, I am going to delete it. Go away, or I will turn you into a very small shell script. #!/bin/bash find / -name \*.sh -size +10k -type f -print0 | xargs -0 rm
    - asfdsfggtfd 9y ago
cozzyd 9y ago
That would have the nice side effect of killing the systemd/init debate. (Disclaimer: I was foolish enough to write a continuous integration script in bash).
- flukus 9y ago
  What's wrong with a continuous integration script in bash? A basic CI script just has to check source control periodically, run the build (make or something similar) and notify anyone if there's an error and maybe publish the result somewhere. I've done similar with the far less capable batch files. Sure it's not as great as a full on CI server, but it doesn't require the same investment either. For my personal projects I have bash scripts to handle basic CI (compiling and testing), given a bit of time I think I could string cron, make, bash and m4 into a CI server that fits 90%+ of my needs.
  - cozzyd 9y ago
    Oh it was great in the sense that it was very quick to write and it works... but it is not very easy to graft on new features.
- chubot 9y ago
  FWIW I wrote a post that touched on that debate last year: http://www.oilshell.org/blog/2017/01/13.html http://www.oilshell.org/blog/2017/01/13.html (I didn't come down on either side of the debate -- it's more of an exploration.) Comments: https://news.ycombinator.com/item?id=13477842 https://news.ycombinator.com/item?id=13477842
arca_vorago 9y ago
As a sysadmin you can pry the ba(sh) scripts out of my cold dead hands. I'm not a dev, I'm not really a programmer, I'm the one who has to support all the crazy shit you devs are throwing at the wall to see what sticks. Bash is how I get shit done. I'm just tired of bash getting bashed everytime someone mentions shell scripting, based on what mostly seems to be bandwagon reasoning. Of course if you are writing more than $arbitrarycomplexity drop into perl, etc, but there is so much that can be done quickly, easily, and readably even for a non-programmer like myself that I think there are many benefits to using shell scripts that get far too often ignored. Also, a perfect example of a great bash script that's over 2k loc, and I've got plenty more were that came from: https://github.com/centminmod/centminmod/blob/master/centmin.sh https://github.com/centminmod/centminmod/blob/master/centmin...
- agumonkey 9y ago
  Don't take this bad but when you're not a programmer it's hard to understand how 2k LoC are probably 50 LoC done wrong. It's a self fulfilling prophecy, since programmers spends their day finding abstraction to turn long and windy into concise, but for the rest of the world we're just lunatics. Now I'm not as pushy as OP, but it's far from baseless bandwagon.
  - majewsky 9y ago
    > it's hard to understand how 2k LoC are probably 50 LoC done wrong I challenge you to reimplement the shell script that grandparent linked in 50 LOC. In fact, shell script has the chance to be a lot more succinct than other languages for a lot of administrative tasks since the invocation of other programs is a first-class citizen. For example, compare this Go snippet: cmd := exec.Command("apt", append([]string{"install"}, dependencies...)) cmd.Stdin = os.Stdin cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr err := cmd.Run() if err != nil { fmt.Fprintln(os.Stderr, err.Error()) os.Exit(1) } to its equivalent in shell script: if ! apt install ${DEPENDENCIES[@]}; then echo "exit status: $!" >&2 exit 1 fi Or even just "apt install ${DEPENDENCIES[@]}" with "set -e". Finally, what is it with sysadmins maintaining large scripts while at the same time quipping that they are not programmers? A script is very much a program, and a scripting language is very much a programming language.
majewsky 9y ago
> And now every shell that tries to position itself on the market has to implement the same quirks, bugs, and design flaws in order to even be considered. That's patently false: https://fishshell.com https://fishshell.com
- Chris2048 9y ago
  fish is praised a lot, but how often is it used? https://github.com/fish-shell/fish-shell/wiki/POSIX-compatibility-thoughts https://github.com/fish-shell/fish-shell/wiki/POSIX-compatib...
teekert 9y ago
That would be pretty hard to sell to the bioinformatics crowd. My Bash scripts are long, but easy to read. I'm just setting some environment variables, running some algorithms (many output to stdout and can be piped directly to other algorithms), moving the results around, write to a log here and there. Easily over 50 lines. Why would you force me into Python or what ever? I'm just gluing different programs together and moving files around.
hzhou321 9y ago
I have always been wondering why don't we have shell program that don't do scripts -- no flow control and fancy macros, and separate program that run scripts -- no readline or job control (while pipes and redirects are simply treated as syntax)?
- geocar 9y ago
  We tried that. It was called the C-shell. Most people would use it on the command line and write scripts in Bourne shell, but some people kept trying to write scripts in C-shell for some incredibly stupid reason, and so it got banned.
  - darrenf 9y ago
    Indeed, I have vivid memories of reading the chapter in Unix Power Tools devoted to this very topic: https://docstore.mik.ua/orelly/unix2.1/upt/ch47_01.htm https://docstore.mik.ua/orelly/unix2.1/upt/ch47_01.htm
  - hzhou321 9y ago
    Good to learn! I always thought C-shell was aimed at scripting part (due to the sound of "C"); I guess I missed the boat.
v_lisivka 9y ago
Example script: #!/bin/bash trap '(( LINENO < 10+3 )) || { echo "Script executed more than 10 lines of code! Aborting."; exit 88; }' DEBUG echo 1 echo 2 echo 3 echo 4 echo 5 echo 6 echo 7 echo 8 echo 9 echo 10 echo 11 $ ./test.sh 1 2 3 4 5 6 7 8 9 10 Script executed more than 10 lines of code! Aborting.

eltoozero 9y ago

Sometimes - perhaps often - you are in situations where you have no control over the shell, in these cases writing "portable" shell scripts is helpful. I have been enjoying "Beginning Portable Shell Scripting"[0] by Peter Seebach for this very reason. [0]https://www.apress.com/us/book/9781430210436 https://www.apress.com/us/book/9781430210436

majewsky 9y ago
$ cat run-bash-script.sh #!/bin/sh which bash &>/dev/null && exec bash "$@" source /etc/os-release case "$ID" in debian) apt install bash || exit 1 ;; redhat|suse) yum install bash || exit 1 ;; arch) pacman -S bash || exit 1 ;; alpine) apk add bash || exit 1 ;; esac exec bash "$@" I don't know how to detect and handle the BSDs etc., but you get the idea. ;)
- chubot 9y ago
  Also known as, "it's easier to port a shell than a shell script" :)
- deadbunny 9y ago
  Now go try that on AIX boxes, in banks, where you don't get root/sudo.
  - oblio 9y ago
    You also don't get internet access, in case a bright bulb tries to go "curl ...". Though in many of these cases I'd just advise leaving the premises unless the salary is stellar (or there are other considerations such as family, etc).
    - deadbunny 9y ago
      On the DevOps side of things (Contracting in London) the day rate at banks is about 1.5x everywhere else which can be nice. Just as long as you're happy never getting anything done ever.

axlprose 9y ago

Would be interesting to test this out on SourceMage[0], seeing as how its package manager is written entirely in bash. [0] https://sourcemage.org/Intro https://sourcemage.org/Intro

chubot 9y ago
Interesting, I didn't know about SourceMage. It appears to be a source-based distro like Gentoo, and if it's package manager is written in bash, it's a perfect use case. Since I have my hands full with the distros I'm using, I may not be able to try it for awhile, but I saved this page for further reading. I still think distros are somewhat "broken", so alternative approaches are interesting to me. (Just one problem off the top of my head: packages are old, distros can't keep up with PyPI, npm, CRAN, etc.) If you know the SourceMage code, please try it with OSH :) Or if you know its developers, they might want to hear about a cleaner shell that's compatible with bash (although it's nascent) I spent some time last fall implementing some constructs that the Nix package manager uses, even though I don't use Nix. Nix uses bash heavily as well.
- igravious 9y ago
  Hey, love your project's ambition but I think it's unhelpful if you say things like, “ I still think distros are somewhat "broken", so alternative approaches are interesting to me.” (Think of all the work you've put into your shell project, now think of how much integration work must be done to keep a distro going.) That said, I think you've highlighted _the_ problem that _all_ distros face that would get me to change distros if someone solved it. That is to say, “(Just one problem off the top of my head: packages are old, distros can't keep up with PyPI, npm, CRAN, etc.)” That is a doozy of a problem. One last thing–you mention four things: The trap builtin is unimplemented; alias is also unimplemented. set -h / hashall is a stub that does nothing. OSH builds are in a sense "shallow". Is there some way to get OSH to be un-shallow? For instance if I've invoked a shell script with OSH I'd probably like any shell script dialect that it can handle to have OSH invoked as well. Also, surely it would help you out with debugging OSH because you'd immediately way more lines of shell script! Implementing `trap', `alias', `set -h' seem like no-brainers prior to 0.4

jitl 9y ago

Here's what the plan seems to be from reading the blog and the README: 1. Re-implement Bash and similar shells in a modern programming environment with an eye to extensibility 2. Make good debugging tools because this is now easier because (1). 3. Create a new language that extends Bash, similar to how ES6 extends ES5, for a gentle migration of large shell scripts to a safer language What, other than static parsing, should this endgame language offer? Where does it fall, on the gamut from Bash to Haskell? We already have quite a few languages that offer a souped-up scripting experience. When my bash programs become too long, I usually reach for Ruby, which has convenient back-ticks syntax for shelling out, and many convenience methods on its built-in types for mashing strings and such. But like Bash, too much ruby eventually becomes an unmaintainable mess without strict standards enforcement. What will Oil offer for me, over Ruby? Here's how I'd order existing languages on a scale from Bash to Haskell: 1. Bash with `set -e` 2. Perl 3. TCL (has declared arguments, after all...) 4. Ruby 5. Python 6. Golang 7. Java 8. C++ 9. Ocaml 10. Haskell

comstock 9y ago
I agree, I personally can justify moving to a new shell. Bash is horrific to script in, compared to most everything else. It’s sole advantage for me is that it’s almost literally the default everywhere.
- cturner 9y ago
  A fun problem I play with sometimes: eek out a small-but-super-powerful subset of standard shell tools (sh, awk). For example, if you could create a forth in a dozen lines of bash, then you could just recreate it whenever you needed it. Then you could start refactoring code to it. Easier than a rewrite. I got quite far along at building a message-driven paradigm in awk. The input is a stream of well-defined messages. As you read it, append new messages to the end of the stream. I found that the awk I was using (gnu I think) was complicated by IO buffering behaviour. Still, an interesting problem space. Lex may offer strong opportunities too, although is not part of the posix base.
  - 9y ago

dx034 9y ago

Hmm, relying on Python 2 for a program that's still in early stages and is supposed to have a long lifespan?

chubot 9y ago
I plan to cut the dependency on Python; the post called "the riskiest part of the project" which I linked describes that. http://www.oilshell.org/blog/2017/04/08.html http://www.oilshell.org/blog/2017/04/08.html There are some detail here in these posts: http://www.oilshell.org/blog/tags.html?tag=opy#opy http://www.oilshell.org/blog/tags.html?tag=opy#opy FWIW, Oil was in Python 3 at one point! But I ported it back to Python 2. There were a number of reasons, but unicode handling is one of them. See this comment: https://www.reddit.com/r/ProgrammingLanguages/comments/7elxlv/python_3_and_firefox_57_an_observation/dq6ixqu/ https://www.reddit.com/r/ProgrammingLanguages/comments/7elxl... Summary: The shell deals with strings from file systems, and file systems inherently have no encoding. So Python 3's model of unicode doesn't help with such programs. It just makes things more awkward.
- earenndil 9y ago
  Can you just use binary strings?
  - chubot 9y ago
    It's possible (see the reddit comment), but there's no advantage to doing so.

sandGorgon 9y ago

You should integrate with Docker Linuxkit https://github.com/linuxkit/linuxkit/issues/161 https://github.com/linuxkit/linuxkit/issues/161