5 ms·

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipel

Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d '\t' | tr -d " ") In this case, perhaps awk would have absorbed 3 to 4 stages.

salmo 6y ago
I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but only tacks on commands with no refactoring. The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this. The fourth stage is rare. Some people start using weird file descriptors to chuck data around pipes, extensive process substitution, etc. This is the Perl of shell and should be avoided. The enlightened return back to terse readable scripts, but with greater wisdom. Also, I always hear of people hating awk and sed and preferring, say, python. Understanding awk and especially sed will make your python better. There really is a significant niche where the classic tools are a better fit than a full programming language.
- young_unixer 6y ago
  My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?
  - every 6y ago
    You can get a lot of mileage out of aliases, especially compound ones. But eventually you may run up against something they don't handle very well. For that a simple shell script might be the next logical step. And fortunately, they aren't that much of a leap...
  - DiggyJohnson 6y ago
    I absolutely think a lot of people are in this position. I am just now slightly "farther along" path as described by OP,and I'm starting to see how it can be worth it as long as you're pursuing the new information in an actually useful context. I definitely be interested in OP's answer. That sort of breakdown is often extremely valuable in any area of learning.
  - salmo 6y ago
    I may be a bad person to ask. I like history a lot. A lot of this stuff was originally picked up by secretaries and non-computer people working on manuals for Bell Labs. If you understand it, it will save you time. It will also make you use and understand Linux/Unix better, which will bleed into any other programming you do (assuming you write other code). The skill also comes up a lot in strange places like Dockerfiles, CI Pipelines, and a lot of that kind of surrounding infrastructure for applications. The video referenced in the article is actually pretty interesting, as is the book "The Unix Programming Environment". Both will refer to outdated technology like real terminals, but I think they help understand the intent of the systems we still use today. Any of the Bell Labs books are great for understanding Unix. Also do a minimal install of Linux or FreeBSD and read all of the man pages for commands in /bin, /usr/bin. Read the docs for the bash builtins. The old Useless Use of Cat is great: http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html But writing real scripts is probably the only way to learn. Next time you need to munge data, try the shell. When you read about a new command, like comm, try to use it to do something. Also force yourself into good practices: - revision control everything - provide -h output for every script you write (getopts is a good thing to be in the habit of using) - use shellcheck to beat bad habits out of you - try to do things in parallel when you can - assume everything you make is going to run at a much larger scale than you intend it to (more inputs, more servers, more everything) - after something works, see where you can reduce the number of commands/processes you run - write things like a moron is going to use them who will accidentally put spaces or hyphens or whatever in the worst place possible, not provide required arguments, and copy/paste broken things out of word - don't let scripts leave trash around, even when aborted: learn trap & signals The two things you should probably understand the most, but on which I'm the least help on are sed and awk. Think of sed as a way to have an automated text editor. Awk is about processing structured data (columns and fields). I learned these through trial and error and a lot of staring at examples. Understanding regular expressions is key to both and is, in general, an invaluable skill. Oh and if you are a vi(m) user, remember that the shell is always there to help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines 1-17, reverse numerical order on the second field) can save a ton of time. And even putting some shell in the middle of a file and running !!sh is super-handy to replace the current line that has some commands with the output of those commands.
  - asicsp 6y ago
    I have books on GNU grep/sed/awk [0] (currently free in spirit of quarantine learning) that teaches the command and features step by step using plenty of examples and exercises. There's a separate chapter for regular expressions too. And I have a list of curated resources for `bash` and Linux [1]. Here's what I'd recommend: * https://ryanstutorials.net/linuxtutorial/ https://ryanstutorials.net/linuxtutorial/ is a good place to start for beginners * https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide and the rest of this wonderful wonderful site is recommended to understand `bash` and its features and gotchas * https://explainshell.com/ https://explainshell.com/ gives you a quick help on the various parts of the command, including documentation * https://www.shellcheck.net/ https://www.shellcheck.net/ is a must have tool if you are writing shell scripts * https://linuxjourney.com/ https://linuxjourney.com/ is probably the most complete site to understand how things work in *nix environment [0] https://learnbyexample.github.io/books/ https://learnbyexample.github.io/books/ [1] https://github.com/learnbyexample/scripting_course/blob/master/Linux_curated_resources.md https://github.com/learnbyexample/scripting_course/blob/mast...
- viraptor 6y ago
  > Understanding awk and especially sed will make your python better. Why do you think so?
m0xte 6y ago
Every time I see that sort of stuff I end up | perl, still after 20 odd years. Sometimes i fine perl is not installed and I die a little.
- enriquto 6y ago
  In what unix system perl is not installed? It is in the default install on current versions of macos (where it comprises the majority of interpreted programs), openbsd, freebsd, ubuntu and fedora linux.
  - m0xte 6y ago
    Usually containers these days.
  - JdeBP 6y ago
    In FreeBSD, Perl was removed from base 17 years ago. * https://unix.stackexchange.com/a/568069/5132 https://unix.stackexchange.com/a/568069/5132
lordleft 6y ago
A little bit of grep / awk goes very, very far.
- jpxw 6y ago
  grep, awk, sed, cut, sort, uniq and join are the Swiss Army knife of working with tabulated data on the command line.
  - nojito 6y ago
    And yet they are absolutely useless if columns get swapped silently.
    - enriquto 6y ago
      Everything breaks down if columns are swapped silently (or database records are renamed silently). What do you mean?
      - nojito 6y ago
        Not with Powershell.
  - JdeBP 6y ago
    No, they are the opposites of an all-in-one tool. For a Swiss Army Knife, look at the likes of Miller. * https://johnkerl.org/miller/ https://johnkerl.org/miller/
  - anthk 6y ago
    You forgot paste.
dfinninger 6y ago
Oh man. That's when knowing more about the tools you are using comes in handy. Kubectl has native JSONPath support [0]. Or at the very least, use structured output with "-o json" and jq [1], like they mention in the article. I have always found that trying to parse JSON with native shell tools has been difficult and error-prone. [0] https://kubernetes.io/docs/reference/kubectl/jsonpath/ https://kubernetes.io/docs/reference/kubectl/jsonpath/ [1] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
ecnahc515 6y ago
This is a pretty bad example since kubectl can output JSON and using something like `jq` would have made it even simpler.
- majewsky 6y ago
  Better yet, kubectl can take a jsonpath expression to select from the JSON. For example, to just get a name: kubectl get something -o jsonpath='{.items[*].metadata.name}'
pwdisswordfish2 6y ago
The -E on the grep makes no sense. I do not see any ERE. Looks like everything here could be done with a single invocation of sed. Can anyone share some sample output of kubectl?

salmo 6y ago

I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but only tacks on commands with no refactoring. The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this. The fourth stage is rare. Some people start using weird file descriptors to chuck data around pipes, extensive process substitution, etc. This is the Perl of shell and should be avoided. The enlightened return back to terse readable scripts, but with greater wisdom. Also, I always hear of people hating awk and sed and preferring, say, python. Understanding awk and especially sed will make your python better. There really is a significant niche where the classic tools are a better fit than a full programming language.

young_unixer 6y ago
My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?
- every 6y ago
  You can get a lot of mileage out of aliases, especially compound ones. But eventually you may run up against something they don't handle very well. For that a simple shell script might be the next logical step. And fortunately, they aren't that much of a leap...
- DiggyJohnson 6y ago
  I absolutely think a lot of people are in this position. I am just now slightly "farther along" path as described by OP,and I'm starting to see how it can be worth it as long as you're pursuing the new information in an actually useful context. I definitely be interested in OP's answer. That sort of breakdown is often extremely valuable in any area of learning.
- salmo 6y ago
  I may be a bad person to ask. I like history a lot. A lot of this stuff was originally picked up by secretaries and non-computer people working on manuals for Bell Labs. If you understand it, it will save you time. It will also make you use and understand Linux/Unix better, which will bleed into any other programming you do (assuming you write other code). The skill also comes up a lot in strange places like Dockerfiles, CI Pipelines, and a lot of that kind of surrounding infrastructure for applications. The video referenced in the article is actually pretty interesting, as is the book "The Unix Programming Environment". Both will refer to outdated technology like real terminals, but I think they help understand the intent of the systems we still use today. Any of the Bell Labs books are great for understanding Unix. Also do a minimal install of Linux or FreeBSD and read all of the man pages for commands in /bin, /usr/bin. Read the docs for the bash builtins. The old Useless Use of Cat is great: http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html But writing real scripts is probably the only way to learn. Next time you need to munge data, try the shell. When you read about a new command, like comm, try to use it to do something. Also force yourself into good practices: - revision control everything - provide -h output for every script you write (getopts is a good thing to be in the habit of using) - use shellcheck to beat bad habits out of you - try to do things in parallel when you can - assume everything you make is going to run at a much larger scale than you intend it to (more inputs, more servers, more everything) - after something works, see where you can reduce the number of commands/processes you run - write things like a moron is going to use them who will accidentally put spaces or hyphens or whatever in the worst place possible, not provide required arguments, and copy/paste broken things out of word - don't let scripts leave trash around, even when aborted: learn trap & signals The two things you should probably understand the most, but on which I'm the least help on are sed and awk. Think of sed as a way to have an automated text editor. Awk is about processing structured data (columns and fields). I learned these through trial and error and a lot of staring at examples. Understanding regular expressions is key to both and is, in general, an invaluable skill. Oh and if you are a vi(m) user, remember that the shell is always there to help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines 1-17, reverse numerical order on the second field) can save a ton of time. And even putting some shell in the middle of a file and running !!sh is super-handy to replace the current line that has some commands with the output of those commands.
- asicsp 6y ago
  I have books on GNU grep/sed/awk [0] (currently free in spirit of quarantine learning) that teaches the command and features step by step using plenty of examples and exercises. There's a separate chapter for regular expressions too. And I have a list of curated resources for `bash` and Linux [1]. Here's what I'd recommend: * https://ryanstutorials.net/linuxtutorial/ https://ryanstutorials.net/linuxtutorial/ is a good place to start for beginners * https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide and the rest of this wonderful wonderful site is recommended to understand `bash` and its features and gotchas * https://explainshell.com/ https://explainshell.com/ gives you a quick help on the various parts of the command, including documentation * https://www.shellcheck.net/ https://www.shellcheck.net/ is a must have tool if you are writing shell scripts * https://linuxjourney.com/ https://linuxjourney.com/ is probably the most complete site to understand how things work in *nix environment [0] https://learnbyexample.github.io/books/ https://learnbyexample.github.io/books/ [1] https://github.com/learnbyexample/scripting_course/blob/master/Linux_curated_resources.md https://github.com/learnbyexample/scripting_course/blob/mast...
viraptor 6y ago
> Understanding awk and especially sed will make your python better. Why do you think so?

m0xte 6y ago

Every time I see that sort of stuff I end up | perl, still after 20 odd years. Sometimes i fine perl is not installed and I die a little.

enriquto 6y ago
In what unix system perl is not installed? It is in the default install on current versions of macos (where it comprises the majority of interpreted programs), openbsd, freebsd, ubuntu and fedora linux.
- m0xte 6y ago
  Usually containers these days.
- JdeBP 6y ago
  In FreeBSD, Perl was removed from base 17 years ago. * https://unix.stackexchange.com/a/568069/5132 https://unix.stackexchange.com/a/568069/5132

lordleft 6y ago

A little bit of grep / awk goes very, very far.

jpxw 6y ago
grep, awk, sed, cut, sort, uniq and join are the Swiss Army knife of working with tabulated data on the command line.
- nojito 6y ago
  And yet they are absolutely useless if columns get swapped silently.
  - enriquto 6y ago
    Everything breaks down if columns are swapped silently (or database records are renamed silently). What do you mean?
    - nojito 6y ago
      Not with Powershell.
- JdeBP 6y ago
  No, they are the opposites of an all-in-one tool. For a Swiss Army Knife, look at the likes of Miller. * https://johnkerl.org/miller/ https://johnkerl.org/miller/
- anthk 6y ago
  You forgot paste.

dfinninger 6y ago

Oh man. That's when knowing more about the tools you are using comes in handy. Kubectl has native JSONPath support [0]. Or at the very least, use structured output with "-o json" and jq [1], like they mention in the article. I have always found that trying to parse JSON with native shell tools has been difficult and error-prone. [0] https://kubernetes.io/docs/reference/kubectl/jsonpath/ https://kubernetes.io/docs/reference/kubectl/jsonpath/ [1] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/

ecnahc515 6y ago

This is a pretty bad example since kubectl can output JSON and using something like `jq` would have made it even simpler.

majewsky 6y ago
Better yet, kubectl can take a jsonpath expression to select from the JSON. For example, to just get a name: kubectl get something -o jsonpath='{.items[*].metadata.name}'

pwdisswordfish2 6y ago

The -E on the grep makes no sense. I do not see any ERE. Looks like everything here could be done with a single invocation of sed. Can anyone share some sample output of kubectl?