8 ms·

Bash redirections cheat sheet

93 points by johndcook 14y ago

AceJohnny2 14y ago
the "cmd <<EOL ..\n..\n EOL" part has been mystical to me for ages, especially as I didn't know what to google to find out. I recently learned it was called a "Here Document". http://en.wikipedia.org/wiki/Here_document http://en.wikipedia.org/wiki/Here_document For extra credit, try using a "Here Document" as input for 'echo' and 'cat'. Explain the results.
- allstruck 14y ago
  What is so mystifying about this? Doing this: "cat << EOL line one line two EOL" Would be the same as this: "cat < file" That is if 'file' contained this: "line one line two" This doesn't work with the echo command because it does not use stdin like cat does.
- akldfgj 14y ago
  `man bash` , search for "<<" see the "Here Strings" section. You can also search https://www.google.com/search?q=bash https://www.google.com/search?q=bash and follow links to "Documentation" xargs echo <<EOL > line 1 > line 2 > EOL line 1 line 2 $ cat <<EOL > line 1 > line 2 > EOL line 1 line 2 line-terminators get eaten by xargs
darkstalker 14y ago
The PNG version is rendered with subpixel antialiasing, which is kinda annoying to see. Besides that, awesome job.
- jdpage 14y ago
  Is there a particular reason you'd go for the PNG version over the PDF version? It's a more suitable format for these "cheat sheet" type pages, since it'll use the platform text rendering, and also keep the vector information (meaning it'll print better if you want to hang it up next to your desk).
trusko 14y ago
Great! Thanks for sharing, very helpful.
dllthomas 14y ago
One thing worth noting is that "redirection of streams" is a poor metaphor. If I have two streams, 1 ---------- 1' 2 ---------- 2' and I redirect the 1>2 1 ---- \ 2 ---------- 2' it seems to work okay. But if you have three, 1 ---------- 1' 2 ---------- 2' 3 ---------- 3' and try "redirecting 1 to 2 then 2 to 3", the natural reading would be 1 - \ 2 ----- \ 3 ---------- 3' But that's not what you get, because you're not "redirecting streams" - you're copying file descriptors. This is partially fixed by informing people to do the operations backwards, but there are corner cases where that breaks down, and "copying file descriptors in the order they appear on the line" never does because that's genuinely what's going on. 1 -> 1' 2 -> 2' 3 -> 3' foo 1>2 2>3 This is "run foo copying fd 2 to 1 and 3 to 2" which gives 1 -> 2' 2 -> 3' 3 -> 3'
- wethesheeple 14y ago
  Is there a typo: s/genuinely/not geniuinely I guess what you're explaining is the process that happens as the kernel reserves fd's? And when you start your program there isn't necessarily any guarantee that a particular fd will be available, except for 0,1,2, right?
  - dllthomas 14y ago
    Copying file descriptors is genuinely what's going on, so understanding it that way always works. Edited for clarity: The way file descriptors work is that the kernel maintains, for each process, a table mapping from numerical file descriptors to structures describing the open file/socket/device associated with that descriptor. When the shell is setting up to spawn a new process (or subshell, for that matter), it processes the redirections in order, left to right. For redirections of the form "A>&B" this is a call to dup2(B, A), which has the kernel copy the entry in the table at B over the entry at A. Incidentally, you will note the direction of the symbol doesn't matter when cloning file descriptors: 5>&7 and 5<&7 mean the same thing - you're just copying a file descriptor. It does matter when opening a file, as the file will be opened for reading or writing before before being dup2'ed over the requested descriptor. Having said that, of course use the correct symbol - it's good documentation.
bajsejohannes 14y ago
I've always done (cmd 2>&1) > file to redirect stderr and stdout to a file. I always thought the parenthesis were necessary, but according to this I can just do cmd > file 2>&1 It seems very unintuitive, though. Does anyone know the rationale behind ordering of redirects? (That said, I'm going to use 'cmd &> file' that I just learned from now on)
- rcsorensen 14y ago
  cmd is going to output to STDOUT and STDERR, which are file descriptors 1 and 2. In a basic shell command execution, those route out to the console. Reading from left to right: cmd Run command cmd. File descriptors: {1: /dev/console, 2:/dev/console} > file Open a file descriptor for "file" and copy it into our first file descriptor (stdout). File descriptors: {1: fopen('file'), 2: /dev/console } 2>&1 Copy the file descriptor stored in 1 (stdout) into 2 (stderr). File descriptors: {1: fopen('file'), 2: fopen('file')}
- anon987 14y ago
  > Does anyone know the rationale behind ordering of redirects? I can't give you an exact rationale, but my old shell scripting books from the 90's use the cmd > file 2>&1 notation. My personal belief is that it was for situations when you would do something like this: cmd > output.txt 2> errors.txt
- dredmorbius 14y ago
  In the first case, you're actually executing two shells. (cmd 2>&1) runs in its own subshell, and copies stderr to stdout. >file is interpreted in the parent shell, and copies stdout to 'file'. In the second instance, you run a single shell, first copying stdout to 'file', then copying stderr to the (redirected) stdout, that is, 'file'.
- ralph 14y ago
  It's read as "cmd, stdout to file, stderr to the same place as stdout".
shabble 14y ago
I've found cmd <(another_cmd) to be particularly useful in the past for pushing generated data into something that doesn't accept - as an alias for STDIN. I'm not sure if it's a different way of expressing one of the listed items though; iirc it creates a temporary fifo
- daragh 14y ago
  That FIFO temporary file is the file descriptor of the subprocess' stdout. You're accessing it through a pipe. It's most useful when you want to diff the result of multiple commands, e.g.: diffing the contents of two directories: $ diff <(find /first/path | sort) <(find /second/path | sort)
  - akldfgj 14y ago
    Aha! In zsh "=(cmd)" does that, but "=" doesn't work in bash. "<(cmd)" works everywhere... why did zsh trick me with "="?
    - ralph 14y ago
      zsh had the idea first IIRC.
gwillen 14y ago
I think I previously had all the pieces required to do this, but only just figured out that it worked: Glenn-Willens-MacBook-Pro:~ gwillen$ exec 3<> /dev/tcp/<website>/80 Glenn-Willens-MacBook-Pro:~ gwillen$ echo "GET / HTTP/1.0" >&3 Glenn-Willens-MacBook-Pro:~ gwillen$ echo "" >&3 Glenn-Willens-MacBook-Pro:~ gwillen$ cat <& 3 HTTP/1.1 200 OK Date: Mon, 10 Sep 2012 21:43:33 GMT Server: Apache Last-Modified: Sun, 06 Sep 2009 18:44:42 GMT ETag: "30525-0-472ed20537a80" Accept-Ranges: bytes Content-Length: 0 Connection: close Content-Type: text/html Glenn-Willens-MacBook-Pro:~ gwillen$
michaelhoffman 14y ago
Thanks! Much more useful than the other recent alphabetic cheat sheets.
methoddk 14y ago
Very useful! Thank you for this.
dredmorbius 14y ago
The network pseudo filehandles (/dev/tcp/<host>/<port> and /dev/udp/<host>/<port>) are explicitly disabled in certain distros (notably Debian). Ubuntu appears to support it. Though not limited to just shell redirections, one of the true masters of bash (and zsh) is Larry Peek whose written/given multiple books, articles, and presentations on the topic.
- dfc 14y ago
  Do you know why debian did that? Any links to debian-devel threads?
  - dredmorbius 14y ago
    http://lists.debian.org/debian-user/2006/07/msg00204.html http://lists.debian.org/debian-user/2006/07/msg00204.html The feature is confusing (it uses virtual files visibly only to bash, and not extant on the filesystem) and overlaps existing functionality (netcat). Also: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464 http://lists.debian.org/debian-user/2003/04/msg01591.html http://lists.debian.org/debian-user/2003/04/msg01591.html http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu http://lists.debian.org/debian-user/2006/07/msg00204.html http://lists.debian.org/debian-user/2006/07/msg00204.html
luriel 14y ago
I can never remember the (ba)sh redirection syntax, and I used it for decades. Rc on the other hand is much more clean: http://rc.cat-v.org http://rc.cat-v.org
zwilliamson 14y ago
Where is EOF? I use that a lot in my Bash scripts. I think it should be on this cheat sheet.
dllthomas 14y ago
I like the mention of PIPESTATUS - $PIPESTATUS[*] is at the start of my PS1
dllthomas 14y ago
One thing missing is <<- heredocs, which strip (some) leading whitespace. It lets you keep up your indentation in a shell script without passing a bunch of whitespace to the commands.

93 points by johndcook 14y ago

the "cmd <<EOL ..\n..\n EOL" part has been mystical to me for ages, especially as I didn't know what to google to find out. I recently learned it was called a "Here Document". http://en.wikipedia.org/wiki/Here_document http://en.wikipedia.org/wiki/Here_document For extra credit, try using a "Here Document" as input for 'echo' and 'cat'. Explain the results.

allstruck 14y ago
What is so mystifying about this? Doing this: "cat << EOL line one line two EOL" Would be the same as this: "cat < file" That is if 'file' contained this: "line one line two" This doesn't work with the echo command because it does not use stdin like cat does.
akldfgj 14y ago
`man bash` , search for "<<" see the "Here Strings" section. You can also search https://www.google.com/search?q=bash https://www.google.com/search?q=bash and follow links to "Documentation" xargs echo <<EOL > line 1 > line 2 > EOL line 1 line 2 $ cat <<EOL > line 1 > line 2 > EOL line 1 line 2 line-terminators get eaten by xargs

darkstalker 14y ago

The PNG version is rendered with subpixel antialiasing, which is kinda annoying to see. Besides that, awesome job.

jdpage 14y ago
Is there a particular reason you'd go for the PNG version over the PDF version? It's a more suitable format for these "cheat sheet" type pages, since it'll use the platform text rendering, and also keep the vector information (meaning it'll print better if you want to hang it up next to your desk).

trusko 14y ago

Great! Thanks for sharing, very helpful.

dllthomas 14y ago

One thing worth noting is that "redirection of streams" is a poor metaphor. If I have two streams, 1 ---------- 1' 2 ---------- 2' and I redirect the 1>2 1 ---- \ 2 ---------- 2' it seems to work okay. But if you have three, 1 ---------- 1' 2 ---------- 2' 3 ---------- 3' and try "redirecting 1 to 2 then 2 to 3", the natural reading would be 1 - \ 2 ----- \ 3 ---------- 3' But that's not what you get, because you're not "redirecting streams" - you're copying file descriptors. This is partially fixed by informing people to do the operations backwards, but there are corner cases where that breaks down, and "copying file descriptors in the order they appear on the line" never does because that's genuinely what's going on. 1 -> 1' 2 -> 2' 3 -> 3' foo 1>2 2>3 This is "run foo copying fd 2 to 1 and 3 to 2" which gives 1 -> 2' 2 -> 3' 3 -> 3'

wethesheeple 14y ago
Is there a typo: s/genuinely/not geniuinely I guess what you're explaining is the process that happens as the kernel reserves fd's? And when you start your program there isn't necessarily any guarantee that a particular fd will be available, except for 0,1,2, right?
- dllthomas 14y ago
  Copying file descriptors is genuinely what's going on, so understanding it that way always works. Edited for clarity: The way file descriptors work is that the kernel maintains, for each process, a table mapping from numerical file descriptors to structures describing the open file/socket/device associated with that descriptor. When the shell is setting up to spawn a new process (or subshell, for that matter), it processes the redirections in order, left to right. For redirections of the form "A>&B" this is a call to dup2(B, A), which has the kernel copy the entry in the table at B over the entry at A. Incidentally, you will note the direction of the symbol doesn't matter when cloning file descriptors: 5>&7 and 5<&7 mean the same thing - you're just copying a file descriptor. It does matter when opening a file, as the file will be opened for reading or writing before before being dup2'ed over the requested descriptor. Having said that, of course use the correct symbol - it's good documentation.

bajsejohannes 14y ago

I've always done (cmd 2>&1) > file to redirect stderr and stdout to a file. I always thought the parenthesis were necessary, but according to this I can just do cmd > file 2>&1 It seems very unintuitive, though. Does anyone know the rationale behind ordering of redirects? (That said, I'm going to use 'cmd &> file' that I just learned from now on)

rcsorensen 14y ago
cmd is going to output to STDOUT and STDERR, which are file descriptors 1 and 2. In a basic shell command execution, those route out to the console. Reading from left to right: cmd Run command cmd. File descriptors: {1: /dev/console, 2:/dev/console} > file Open a file descriptor for "file" and copy it into our first file descriptor (stdout). File descriptors: {1: fopen('file'), 2: /dev/console } 2>&1 Copy the file descriptor stored in 1 (stdout) into 2 (stderr). File descriptors: {1: fopen('file'), 2: fopen('file')}
anon987 14y ago
> Does anyone know the rationale behind ordering of redirects? I can't give you an exact rationale, but my old shell scripting books from the 90's use the cmd > file 2>&1 notation. My personal belief is that it was for situations when you would do something like this: cmd > output.txt 2> errors.txt
dredmorbius 14y ago
In the first case, you're actually executing two shells. (cmd 2>&1) runs in its own subshell, and copies stderr to stdout. >file is interpreted in the parent shell, and copies stdout to 'file'. In the second instance, you run a single shell, first copying stdout to 'file', then copying stderr to the (redirected) stdout, that is, 'file'.
ralph 14y ago
It's read as "cmd, stdout to file, stderr to the same place as stdout".

shabble 14y ago

I've found cmd <(another_cmd) to be particularly useful in the past for pushing generated data into something that doesn't accept - as an alias for STDIN. I'm not sure if it's a different way of expressing one of the listed items though; iirc it creates a temporary fifo

daragh 14y ago
That FIFO temporary file is the file descriptor of the subprocess' stdout. You're accessing it through a pipe. It's most useful when you want to diff the result of multiple commands, e.g.: diffing the contents of two directories: $ diff <(find /first/path | sort) <(find /second/path | sort)
- akldfgj 14y ago
  Aha! In zsh "=(cmd)" does that, but "=" doesn't work in bash. "<(cmd)" works everywhere... why did zsh trick me with "="?
  - ralph 14y ago
    zsh had the idea first IIRC.

gwillen 14y ago

I think I previously had all the pieces required to do this, but only just figured out that it worked: Glenn-Willens-MacBook-Pro:~ gwillen$ exec 3<> /dev/tcp/<website>/80 Glenn-Willens-MacBook-Pro:~ gwillen$ echo "GET / HTTP/1.0" >&3 Glenn-Willens-MacBook-Pro:~ gwillen$ echo "" >&3 Glenn-Willens-MacBook-Pro:~ gwillen$ cat <& 3 HTTP/1.1 200 OK Date: Mon, 10 Sep 2012 21:43:33 GMT Server: Apache Last-Modified: Sun, 06 Sep 2009 18:44:42 GMT ETag: "30525-0-472ed20537a80" Accept-Ranges: bytes Content-Length: 0 Connection: close Content-Type: text/html Glenn-Willens-MacBook-Pro:~ gwillen$

michaelhoffman 14y ago

Thanks! Much more useful than the other recent alphabetic cheat sheets.

methoddk 14y ago

Very useful! Thank you for this.

dredmorbius 14y ago

The network pseudo filehandles (/dev/tcp/<host>/<port> and /dev/udp/<host>/<port>) are explicitly disabled in certain distros (notably Debian). Ubuntu appears to support it. Though not limited to just shell redirections, one of the true masters of bash (and zsh) is Larry Peek whose written/given multiple books, articles, and presentations on the topic.

dfc 14y ago
Do you know why debian did that? Any links to debian-devel threads?
- dredmorbius 14y ago
  http://lists.debian.org/debian-user/2006/07/msg00204.html http://lists.debian.org/debian-user/2006/07/msg00204.html The feature is confusing (it uses virtual files visibly only to bash, and not extant on the filesystem) and overlaps existing functionality (netcat). Also: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464 http://lists.debian.org/debian-user/2003/04/msg01591.html http://lists.debian.org/debian-user/2003/04/msg01591.html http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu http://lists.debian.org/debian-user/2006/07/msg00204.html http://lists.debian.org/debian-user/2006/07/msg00204.html

luriel 14y ago

I can never remember the (ba)sh redirection syntax, and I used it for decades. Rc on the other hand is much more clean: http://rc.cat-v.org http://rc.cat-v.org

zwilliamson 14y ago

Where is EOF? I use that a lot in my Bash scripts. I think it should be on this cheat sheet.

dllthomas 14y ago

I like the mention of PIPESTATUS - $PIPESTATUS[*] is at the start of my PS1

dllthomas 14y ago

One thing missing is <<- heredocs, which strip (some) leading whitespace. It lets you keep up your indentation in a shell script without passing a bunch of whitespace to the commands.