8 ms·

Effective Shell

188 points by signa11 4y ago

funnybookbinder 4y ago
Thanks for taking the time to do this and share it. I think it's a great resource for someone just starting out (this guy). Then more advanced materials can take it from there.
kazinator 4y ago
This is going to confuse the frankfurter out of a newbie. First it presents the idea that Windows has a shell, and even multiple shell programs. Screenshots 'n' everything. But then, nope: > Windows is not anything like Linux under the hood. So to get a shell working, we have three options: > Use a tool which provides common Linux tools which have been written to work with Windows > Use a "virtual machine" running Linux > Use the Windows Subsystem for Linux What "common Linux tools which have been written to work with Windows" is referring to is installing Cygwin. This a fairly hamfisted presentation of a subject matter that could be handled in a way that is simultaneously easier to grasp and factually correct. First of all, your goal is to teach about a specific command interpreter language, you really want to clarify how there are multiple ones which have different syntax, and brush that whole topic aside as fast as possible. > You can see what the version is of your shell by running: > bash --version Yikes. If we know we are running Bash, the way to know what version it is is: echo $BASH_VERSION and not to run another instance of Bash. Can this book teach something simple, like comments? > The shell ignores any text which follows a # hash symbol. Wrong: $ echo 'all of # this is seen' all of # this is seen "this is seen" looks like an example of "any text" to me. A hash comment marker is ignored if it is not quoted and escaped so as to be part of a word. The hash is also part of certain bits of syntax: $# (dollar hash) number of positional parameters ${variable#pat} expand the content of variable, chopping off a leading portion which matches the pat pattern. Example: $ echo $TERM xterm-256color $ echo ${TERM#xte} rm-256color
- aynawn 4y ago
  These are great comments. The code is open source. I'm sure the owner would love a pull request that would address these concerns.
  - kazinator 4y ago
    I don't want to discourage you, but if you know the subject matter and are inclined to work on a book, you should just write your own from scratch as the sole author: the normal way most books are written. (I personally wouldn't write a book about the shell. I would be too paralyzed by the thought that at any moment, Stéphane Chazelas might crack it open.)
    - basdp 4y ago
      The amount of people writing books on subjects as a way of learning something new, is too high. I’ve seen so much trash these days. Especially game design or programming books.
- chasil 4y ago
  The classic Windows shell has many flaws in its design, appears to have come from OS/2, and it's amazing that it works at all for how it has been used: https://blog.nullspace.io/batch.html https://blog.nullspace.io/batch.html Powershell is a new arrival, limited outside of Windows. I admit that I don't know it. The POSIX shell also has severe problems with ambiguity, and it is not an LR-parsed language: https://archive.fosdem.org/2018/schedule/event/code_parsing_posix_s_hell/ https://archive.fosdem.org/2018/schedule/event/code_parsing_... Use the right tool for the job, imperfect as they may be. It is unfortunate that awk didn't grow and supplant the shell, as for many years the BWK "One True Awk" did use straightforward lex and yacc grammars.
  - layer8 4y ago
    > The classic Windows shell […] appears to have come from OS/2 It’s actually based on the DOS command shell [1], which in turn was modeled after CP/M [2]. [1] https://en.m.wikipedia.org/wiki/COMMAND.COM https://en.m.wikipedia.org/wiki/COMMAND.COM [2] https://en.m.wikipedia.org/wiki/CP/M https://en.m.wikipedia.org/wiki/CP/M
rubicks 4y ago
Obligatory pedantry: Bash is A shell. It's definitely not "The Shell".
- colordrops 4y ago
  It kinda almost is though.
  - JonathonW 4y ago
    With Ubuntu using dash as its /bin/sh and macOS defaulting to zsh for interactive shells (I think their /bin/sh is still bash, but haven’t checked), this is probably the least true now that it’s been for a couple decades.
    - bigDinosaur 4y ago
      Also, as noted in another comment: windows has its own shell(s) which are entirely unrelated to bash.
  - hyperhopper 4y ago
    People have said the same thing about sh. And neither bash nor sh are that great, so no reason to put down the idea that they shouldn't be the "default"
  - chasil 4y ago
    Bash is a chameleon. When called as /bin/sh or if the POSIXLY_CORRECT environment variable is set, it will act as a POSIX shell. If not, it will behave very differently. The one major difference that I know is that aliases will be ignored in the execution of a script, so "alias p=printf" will fail (for example). This is not POSIX-compliant.
    - paskozdilar 4y ago
      Aliases seem to work on Bash 5.0.17: #!/bin/sh alias p=printf p ok
woodruffw 4y ago
For people who think that they're the intended audience of these kinds of guides: do you find them effective? I learned how to use a POSIX shell years before I learned how to program (outside of a shell, that is), and I've often wondered how to help people (particularly younger colleagues) who learned things the other way around. My experience so far is that a lot of people get stuck in an "experience trough," for lack of a better phrase: they're comfortable navigating around the filesystem and running basic commands, but struggle with some of the more powerful building blocks that can make someone a real shell power user (`xargs`, nontrivial uses of `find`, here-strings, subshells and blocks, process substitution, etc.).
- pozol 4y ago
  have you found any ways to get past that experience trough? I definitely feel like i’m there now
  - nerdponx 4y ago
    I did it by blocking out time to read the manual pages. Yes, it is slow and tedious. But the manuals make a lot more sense once you already know how the tools work at a basic level.
  - chasil 4y ago
    The author of the musl C library has a very poor opinion of the POSIX shell, and has published his "tricks" to force expected behavior. This is very useful for those who are stumped by a failing script. "I am a strong believer that Bourne-derived languages are extremely bad, on the same order of badness as Perl, for programming, and consider programming sh for any purpose other than as a super-portable, lowest-common-denominator platform for build or bootstrap scripts and the like, as an extremely misguided endeavor. As such you won’t see me spending many words on extensions particular to ksh, Bash, or whatever other shells may be popular." https://www.etalabs.net/sh_tricks.html https://www.etalabs.net/sh_tricks.html
  - woodruffw 4y ago
    For me, it helped to stop thinking of the shell as an efficient way to accomplish a task (it certainly is), but as a way to express my intent (the way I would in a normal programming language). That, combined with some automated tooling (shellcheck) and a handful of best practices (arrays over strings, avoiding the "cat pipe" pattern[1], etc.) has worked well for me. But I don't know how well that generalizes! [1]: https://www.linuxjournal.com/content/put-down-pipe https://www.linuxjournal.com/content/put-down-pipe
stevebmark 4y ago
Bash is so painful to use without vi mode (set -o vi).
- schoen 4y ago
  I've actively used bash and vi every day for decades now, but somehow never tried this mode. It's fascinating. I wonder if it will be worth it for the change of command-line editing habits. It certainly feels faster for most command-line edits.
  - tuxie_ 4y ago
    It changed my life. It's like your prompt becomes a mini editor now and you can copy/paste arguments around in your command. I love it.
  - massysett 4y ago
    I am a vi user and tried vi mode in zsh but never liked it. For some reason the modal aspect doesn’t fit for me in the context of the shell. I’ve learned enough emacs bindings to work the shell.
- paskozdilar 4y ago
  I've never heard of bash vi mode before. Thanks for this.
- urxvtcd 4y ago
  Heavy shell and vim user here, but I simply cannot imagine how this might be convenient. A bunch of readline-style shortcuts will do the job just fine. And they are everywhere. SSH, database prompts, even login screens support them.
  - stevebmark 4y ago
    Preemptively justifying something more efficient that you haven't used, because what you have is "fine," is a behavior I see from heavy Vim users when they justify not using IDEs, which are much more efficient than Vim for writing code. (I'm also a heavy Vim user, and IDEs are so much nicer for coding, especially efficiency). So, I'd say your story checks out.
alipitch 4y ago
Good guide to shell in general. Google shell style guide [0] was also a good read. I thought that the "When to use Shell" section is a section that is good for any kind of guide, not just for bash / shell. Also, maybe not so much a pitfall / bug, but something I had to deal with recently was that bash does not handle the EINTR when calling write() in the printf and echo builtins [1][2][3], etc. [0] https://google.github.io/styleguide/shellguide.html#s1.2-when-to-use-shell https://google.github.io/styleguide/shellguide.html#s1.2-whe... If you are writing a script that is more than 100 lines long, or that uses non-straightforward control flow logic, you should rewrite it in a more structured language now. Bear in mind that scripts grow. Rewrite your script early to avoid a more time-consuming rewrite at a later date. [1] https://unix.stackexchange.com/a/487260 https://unix.stackexchange.com/a/487260 handle the EINTR when calling write() in the printf and echo builtins. [2] https://github.com/torvalds/linux/blob/ca1fdab7fd27eb069df1384b2850dcd0c2bebe8d/Documentation/admin-guide/cgroup-v1/cgroups.rst https://github.com/torvalds/linux/blob/ca1fdab7fd27eb069df13... Q: what's up with this '/bin/echo' ? A: bash's builtin 'echo' command does not check calls to write() against errors. If you use it in the cgroup file system, you won't be able to tell whether a command succeeded or failed. [3] https://lists.gnu.org/archive/html/bug-bash/2018-01/msg00031.html https://lists.gnu.org/archive/html/bug-bash/2018-01/msg00031... write() not retried after EINTR in printf and echo
gravytron 4y ago
I will use the word `bash` for convenience. `bash` is powerful because it is convenient. `bash` is also powerful because it doesn't tell users how to behave or what to do. `bash` is all about empowering the user. That is what makes it great. `bash` is a meta language for the convenience and the empowerment of programmers. Over time, impossible tasks become trivial. You start off by using `bash` to create simple models of larger/more complex problems, then you adopt pipes and redirection, and eventually you don't need much handholding and you are 10x better at modeling problems, mocking designs, and general problem solving. Will there be 8 hour days to understand some simple concepts? Sure. Do they pay off? Yes. Absolutely yes. Undergraduate computer science background makes bash even more useful and helpful. Outside of college learning, you might be able to grok some things using man pages, tutorials, and various resources (like books).
e63f67dd-065b 4y ago
I can tell you put quite a bit of work into this, and good job! But there are some major inaccuracies, just from going through the first few pages: > When we talk about "The Shell", we're normally referring to the simple, text-based interface which is used to control a computer or a program. This is a bit nitpicky, but I don't like this definition. This is conflating a TUI, a shell, and a terminal. "The Shell" usually refers to `bash`, `zsh`, `fish`, `powershell`, or `cmd.exe`. The interface is the terminal (emulator), because programs like to dump ANSI control codes and the terminal takes those control codes and turn them into human-readable output with colours and such. > Windows is actually being updated at the time of writing to provide a Linux-like shell interface as part of the core operating system (this is known as the Windows Subsystem Linux No, no, no. This is just wrong. If this was WSL1, you would be at least partially correct, since it's a re-implementation of linux syscalls and runtime environment on Windows, kinda like reverse-wine. But with WSL2, windows now ships the entire linux kernel and virtualises it. The objective is not to provide a linux-like shell interface (whatever that means), but to let people run linux binaries on windows in a linux environment. Either way, the fact that `bash` (or any other shell) runs on WSL is a result of bringing along the rest of the linux runtime environment. > install instructions For the entire WSL section, I'm pretty sure that you can install wsl these days with just `wsl --install` on powershell. Maybe it only works on win11 or something, I'll have to look up the docs. Winget also has ubuntu2004 in the repos. But this is not a linux-like shell on windows, this is using linux, virtualised, on windows. Downloading homebrew just to install bash is ... interesting. Catalina is 3 years old, and I don't see any bash4/5 specific features that you're using (I had to google bash release notes lol). `zsh` is probably preferable anyways imo. Also as a sidenote, you can get pretty close to a linux-like environment by just installing busybox on windows. You'll have to be careful around things that powershell aliased away, but if you decide to just prefer busybox for everything then you're like 90% of the way there. > wget For some bizarre reason, macos doesn't ship with `wget` by default. Don't ask me why :/ > renaissance of the shell In my personal experience, shell usage has gone down, not up. What used to be shell scripts are now getting more and more specialised: sysv services became systemd, the rise of devops meant giant piles of Dockerfiles and YAML, etc. It's still a critical part of system administration and development, but its tentacles are slowly receding from poking into every corner of everything to slightly less pokes in everything.
dwmkerr 4y ago
"The best way to get the right answer on the Internet is not to ask a question, it's to post the wrong answer." - Ward Cunningham (possibly) Thanks to those who've commented with corrections/suggestions, I'll go through them try and get the content fixed up where possible. Will need to have a lie down and possibly an epidural before going through the comments again to pull out the changes needed.

188 points by signa11 4y ago

Thanks for taking the time to do this and share it. I think it's a great resource for someone just starting out (this guy). Then more advanced materials can take it from there.

kazinator 4y ago

This is going to confuse the frankfurter out of a newbie. First it presents the idea that Windows has a shell, and even multiple shell programs. Screenshots 'n' everything. But then, nope: > Windows is not anything like Linux under the hood. So to get a shell working, we have three options: > Use a tool which provides common Linux tools which have been written to work with Windows > Use a "virtual machine" running Linux > Use the Windows Subsystem for Linux What "common Linux tools which have been written to work with Windows" is referring to is installing Cygwin. This a fairly hamfisted presentation of a subject matter that could be handled in a way that is simultaneously easier to grasp and factually correct. First of all, your goal is to teach about a specific command interpreter language, you really want to clarify how there are multiple ones which have different syntax, and brush that whole topic aside as fast as possible. > You can see what the version is of your shell by running: > bash --version Yikes. If we know we are running Bash, the way to know what version it is is: echo $BASH_VERSION and not to run another instance of Bash. Can this book teach something simple, like comments? > The shell ignores any text which follows a # hash symbol. Wrong: $ echo 'all of # this is seen' all of # this is seen "this is seen" looks like an example of "any text" to me. A hash comment marker is ignored if it is not quoted and escaped so as to be part of a word. The hash is also part of certain bits of syntax: $# (dollar hash) number of positional parameters ${variable#pat} expand the content of variable, chopping off a leading portion which matches the pat pattern. Example: $ echo $TERM xterm-256color $ echo ${TERM#xte} rm-256color

aynawn 4y ago
These are great comments. The code is open source. I'm sure the owner would love a pull request that would address these concerns.
- kazinator 4y ago
  I don't want to discourage you, but if you know the subject matter and are inclined to work on a book, you should just write your own from scratch as the sole author: the normal way most books are written. (I personally wouldn't write a book about the shell. I would be too paralyzed by the thought that at any moment, Stéphane Chazelas might crack it open.)
  - basdp 4y ago
    The amount of people writing books on subjects as a way of learning something new, is too high. I’ve seen so much trash these days. Especially game design or programming books.
chasil 4y ago
The classic Windows shell has many flaws in its design, appears to have come from OS/2, and it's amazing that it works at all for how it has been used: https://blog.nullspace.io/batch.html https://blog.nullspace.io/batch.html Powershell is a new arrival, limited outside of Windows. I admit that I don't know it. The POSIX shell also has severe problems with ambiguity, and it is not an LR-parsed language: https://archive.fosdem.org/2018/schedule/event/code_parsing_posix_s_hell/ https://archive.fosdem.org/2018/schedule/event/code_parsing_... Use the right tool for the job, imperfect as they may be. It is unfortunate that awk didn't grow and supplant the shell, as for many years the BWK "One True Awk" did use straightforward lex and yacc grammars.
- layer8 4y ago
  > The classic Windows shell […] appears to have come from OS/2 It’s actually based on the DOS command shell [1], which in turn was modeled after CP/M [2]. [1] https://en.m.wikipedia.org/wiki/COMMAND.COM https://en.m.wikipedia.org/wiki/COMMAND.COM [2] https://en.m.wikipedia.org/wiki/CP/M https://en.m.wikipedia.org/wiki/CP/M

rubicks 4y ago

Obligatory pedantry: Bash is A shell. It's definitely not "The Shell".

colordrops 4y ago
It kinda almost is though.
- JonathonW 4y ago
  With Ubuntu using dash as its /bin/sh and macOS defaulting to zsh for interactive shells (I think their /bin/sh is still bash, but haven’t checked), this is probably the least true now that it’s been for a couple decades.
  - bigDinosaur 4y ago
    Also, as noted in another comment: windows has its own shell(s) which are entirely unrelated to bash.
- hyperhopper 4y ago
  People have said the same thing about sh. And neither bash nor sh are that great, so no reason to put down the idea that they shouldn't be the "default"
- chasil 4y ago
  Bash is a chameleon. When called as /bin/sh or if the POSIXLY_CORRECT environment variable is set, it will act as a POSIX shell. If not, it will behave very differently. The one major difference that I know is that aliases will be ignored in the execution of a script, so "alias p=printf" will fail (for example). This is not POSIX-compliant.
  - paskozdilar 4y ago
    Aliases seem to work on Bash 5.0.17: #!/bin/sh alias p=printf p ok

woodruffw 4y ago

For people who think that they're the intended audience of these kinds of guides: do you find them effective? I learned how to use a POSIX shell years before I learned how to program (outside of a shell, that is), and I've often wondered how to help people (particularly younger colleagues) who learned things the other way around. My experience so far is that a lot of people get stuck in an "experience trough," for lack of a better phrase: they're comfortable navigating around the filesystem and running basic commands, but struggle with some of the more powerful building blocks that can make someone a real shell power user (`xargs`, nontrivial uses of `find`, here-strings, subshells and blocks, process substitution, etc.).

pozol 4y ago
have you found any ways to get past that experience trough? I definitely feel like i’m there now
- nerdponx 4y ago
  I did it by blocking out time to read the manual pages. Yes, it is slow and tedious. But the manuals make a lot more sense once you already know how the tools work at a basic level.
- chasil 4y ago
  The author of the musl C library has a very poor opinion of the POSIX shell, and has published his "tricks" to force expected behavior. This is very useful for those who are stumped by a failing script. "I am a strong believer that Bourne-derived languages are extremely bad, on the same order of badness as Perl, for programming, and consider programming sh for any purpose other than as a super-portable, lowest-common-denominator platform for build or bootstrap scripts and the like, as an extremely misguided endeavor. As such you won’t see me spending many words on extensions particular to ksh, Bash, or whatever other shells may be popular." https://www.etalabs.net/sh_tricks.html https://www.etalabs.net/sh_tricks.html
- woodruffw 4y ago
  For me, it helped to stop thinking of the shell as an efficient way to accomplish a task (it certainly is), but as a way to express my intent (the way I would in a normal programming language). That, combined with some automated tooling (shellcheck) and a handful of best practices (arrays over strings, avoiding the "cat pipe" pattern[1], etc.) has worked well for me. But I don't know how well that generalizes! [1]: https://www.linuxjournal.com/content/put-down-pipe https://www.linuxjournal.com/content/put-down-pipe

stevebmark 4y ago

Bash is so painful to use without vi mode (set -o vi).

schoen 4y ago
I've actively used bash and vi every day for decades now, but somehow never tried this mode. It's fascinating. I wonder if it will be worth it for the change of command-line editing habits. It certainly feels faster for most command-line edits.
- tuxie_ 4y ago
  It changed my life. It's like your prompt becomes a mini editor now and you can copy/paste arguments around in your command. I love it.
- massysett 4y ago
  I am a vi user and tried vi mode in zsh but never liked it. For some reason the modal aspect doesn’t fit for me in the context of the shell. I’ve learned enough emacs bindings to work the shell.
paskozdilar 4y ago
I've never heard of bash vi mode before. Thanks for this.
urxvtcd 4y ago
Heavy shell and vim user here, but I simply cannot imagine how this might be convenient. A bunch of readline-style shortcuts will do the job just fine. And they are everywhere. SSH, database prompts, even login screens support them.
- stevebmark 4y ago
  Preemptively justifying something more efficient that you haven't used, because what you have is "fine," is a behavior I see from heavy Vim users when they justify not using IDEs, which are much more efficient than Vim for writing code. (I'm also a heavy Vim user, and IDEs are so much nicer for coding, especially efficiency). So, I'd say your story checks out.

alipitch 4y ago

Good guide to shell in general. Google shell style guide [0] was also a good read. I thought that the "When to use Shell" section is a section that is good for any kind of guide, not just for bash / shell. Also, maybe not so much a pitfall / bug, but something I had to deal with recently was that bash does not handle the EINTR when calling write() in the printf and echo builtins [1][2][3], etc. [0] https://google.github.io/styleguide/shellguide.html#s1.2-when-to-use-shell https://google.github.io/styleguide/shellguide.html#s1.2-whe... If you are writing a script that is more than 100 lines long, or that uses non-straightforward control flow logic, you should rewrite it in a more structured language now. Bear in mind that scripts grow. Rewrite your script early to avoid a more time-consuming rewrite at a later date. [1] https://unix.stackexchange.com/a/487260 https://unix.stackexchange.com/a/487260 handle the EINTR when calling write() in the printf and echo builtins. [2] https://github.com/torvalds/linux/blob/ca1fdab7fd27eb069df1384b2850dcd0c2bebe8d/Documentation/admin-guide/cgroup-v1/cgroups.rst https://github.com/torvalds/linux/blob/ca1fdab7fd27eb069df13... Q: what's up with this '/bin/echo' ? A: bash's builtin 'echo' command does not check calls to write() against errors. If you use it in the cgroup file system, you won't be able to tell whether a command succeeded or failed. [3] https://lists.gnu.org/archive/html/bug-bash/2018-01/msg00031.html https://lists.gnu.org/archive/html/bug-bash/2018-01/msg00031... write() not retried after EINTR in printf and echo

gravytron 4y ago

I will use the word `bash` for convenience. `bash` is powerful because it is convenient. `bash` is also powerful because it doesn't tell users how to behave or what to do. `bash` is all about empowering the user. That is what makes it great. `bash` is a meta language for the convenience and the empowerment of programmers. Over time, impossible tasks become trivial. You start off by using `bash` to create simple models of larger/more complex problems, then you adopt pipes and redirection, and eventually you don't need much handholding and you are 10x better at modeling problems, mocking designs, and general problem solving. Will there be 8 hour days to understand some simple concepts? Sure. Do they pay off? Yes. Absolutely yes. Undergraduate computer science background makes bash even more useful and helpful. Outside of college learning, you might be able to grok some things using man pages, tutorials, and various resources (like books).

e63f67dd-065b 4y ago

I can tell you put quite a bit of work into this, and good job! But there are some major inaccuracies, just from going through the first few pages: > When we talk about "The Shell", we're normally referring to the simple, text-based interface which is used to control a computer or a program. This is a bit nitpicky, but I don't like this definition. This is conflating a TUI, a shell, and a terminal. "The Shell" usually refers to `bash`, `zsh`, `fish`, `powershell`, or `cmd.exe`. The interface is the terminal (emulator), because programs like to dump ANSI control codes and the terminal takes those control codes and turn them into human-readable output with colours and such. > Windows is actually being updated at the time of writing to provide a Linux-like shell interface as part of the core operating system (this is known as the Windows Subsystem Linux No, no, no. This is just wrong. If this was WSL1, you would be at least partially correct, since it's a re-implementation of linux syscalls and runtime environment on Windows, kinda like reverse-wine. But with WSL2, windows now ships the entire linux kernel and virtualises it. The objective is not to provide a linux-like shell interface (whatever that means), but to let people run linux binaries on windows in a linux environment. Either way, the fact that `bash` (or any other shell) runs on WSL is a result of bringing along the rest of the linux runtime environment. > install instructions For the entire WSL section, I'm pretty sure that you can install wsl these days with just `wsl --install` on powershell. Maybe it only works on win11 or something, I'll have to look up the docs. Winget also has ubuntu2004 in the repos. But this is not a linux-like shell on windows, this is using linux, virtualised, on windows. Downloading homebrew just to install bash is ... interesting. Catalina is 3 years old, and I don't see any bash4/5 specific features that you're using (I had to google bash release notes lol). `zsh` is probably preferable anyways imo. Also as a sidenote, you can get pretty close to a linux-like environment by just installing busybox on windows. You'll have to be careful around things that powershell aliased away, but if you decide to just prefer busybox for everything then you're like 90% of the way there. > wget For some bizarre reason, macos doesn't ship with `wget` by default. Don't ask me why :/ > renaissance of the shell In my personal experience, shell usage has gone down, not up. What used to be shell scripts are now getting more and more specialised: sysv services became systemd, the rise of devops meant giant piles of Dockerfiles and YAML, etc. It's still a critical part of system administration and development, but its tentacles are slowly receding from poking into every corner of everything to slightly less pokes in everything.

dwmkerr 4y ago

"The best way to get the right answer on the Internet is not to ask a question, it's to post the wrong answer." - Ward Cunningham (possibly) Thanks to those who've commented with corrections/suggestions, I'll go through them try and get the content fixed up where possible. Will need to have a lie down and possibly an epidural before going through the comments again to pull out the changes needed.