5 ms·

I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but onl

by salmo 6y ago

I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but only tacks on commands with no refactoring. The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this. The fourth stage is rare. Some people start using weird file descriptors to chuck data around pipes, extensive process substitution, etc. This is the Perl of shell and should be avoided. The enlightened return back to terse readable scripts, but with greater wisdom. Also, I always hear of people hating awk and sed and preferring, say, python. Understanding awk and especially sed will make your python better. There really is a significant niche where the classic tools are a better fit than a full programming language.

young_unixer 6y ago
My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?
- every 6y ago
  You can get a lot of mileage out of aliases, especially compound ones. But eventually you may run up against something they don't handle very well. For that a simple shell script might be the next logical step. And fortunately, they aren't that much of a leap...
- DiggyJohnson 6y ago
  I absolutely think a lot of people are in this position. I am just now slightly "farther along" path as described by OP,and I'm starting to see how it can be worth it as long as you're pursuing the new information in an actually useful context. I definitely be interested in OP's answer. That sort of breakdown is often extremely valuable in any area of learning.
- salmo 6y ago
  I may be a bad person to ask. I like history a lot. A lot of this stuff was originally picked up by secretaries and non-computer people working on manuals for Bell Labs. If you understand it, it will save you time. It will also make you use and understand Linux/Unix better, which will bleed into any other programming you do (assuming you write other code). The skill also comes up a lot in strange places like Dockerfiles, CI Pipelines, and a lot of that kind of surrounding infrastructure for applications. The video referenced in the article is actually pretty interesting, as is the book "The Unix Programming Environment". Both will refer to outdated technology like real terminals, but I think they help understand the intent of the systems we still use today. Any of the Bell Labs books are great for understanding Unix. Also do a minimal install of Linux or FreeBSD and read all of the man pages for commands in /bin, /usr/bin. Read the docs for the bash builtins. The old Useless Use of Cat is great: http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html But writing real scripts is probably the only way to learn. Next time you need to munge data, try the shell. When you read about a new command, like comm, try to use it to do something. Also force yourself into good practices: - revision control everything - provide -h output for every script you write (getopts is a good thing to be in the habit of using) - use shellcheck to beat bad habits out of you - try to do things in parallel when you can - assume everything you make is going to run at a much larger scale than you intend it to (more inputs, more servers, more everything) - after something works, see where you can reduce the number of commands/processes you run - write things like a moron is going to use them who will accidentally put spaces or hyphens or whatever in the worst place possible, not provide required arguments, and copy/paste broken things out of word - don't let scripts leave trash around, even when aborted: learn trap & signals The two things you should probably understand the most, but on which I'm the least help on are sed and awk. Think of sed as a way to have an automated text editor. Awk is about processing structured data (columns and fields). I learned these through trial and error and a lot of staring at examples. Understanding regular expressions is key to both and is, in general, an invaluable skill. Oh and if you are a vi(m) user, remember that the shell is always there to help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines 1-17, reverse numerical order on the second field) can save a ton of time. And even putting some shell in the middle of a file and running !!sh is super-handy to replace the current line that has some commands with the output of those commands.
- asicsp 6y ago
  I have books on GNU grep/sed/awk [0] (currently free in spirit of quarantine learning) that teaches the command and features step by step using plenty of examples and exercises. There's a separate chapter for regular expressions too. And I have a list of curated resources for `bash` and Linux [1]. Here's what I'd recommend: * https://ryanstutorials.net/linuxtutorial/ https://ryanstutorials.net/linuxtutorial/ is a good place to start for beginners * https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide and the rest of this wonderful wonderful site is recommended to understand `bash` and its features and gotchas * https://explainshell.com/ https://explainshell.com/ gives you a quick help on the various parts of the command, including documentation * https://www.shellcheck.net/ https://www.shellcheck.net/ is a must have tool if you are writing shell scripts * https://linuxjourney.com/ https://linuxjourney.com/ is probably the most complete site to understand how things work in *nix environment [0] https://learnbyexample.github.io/books/ https://learnbyexample.github.io/books/ [1] https://github.com/learnbyexample/scripting_course/blob/master/Linux_curated_resources.md https://github.com/learnbyexample/scripting_course/blob/mast...
viraptor 6y ago
> Understanding awk and especially sed will make your python better. Why do you think so?

by salmo 6y ago

young_unixer 6y ago

My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?

every 6y ago
You can get a lot of mileage out of aliases, especially compound ones. But eventually you may run up against something they don't handle very well. For that a simple shell script might be the next logical step. And fortunately, they aren't that much of a leap...
DiggyJohnson 6y ago
I absolutely think a lot of people are in this position. I am just now slightly "farther along" path as described by OP,and I'm starting to see how it can be worth it as long as you're pursuing the new information in an actually useful context. I definitely be interested in OP's answer. That sort of breakdown is often extremely valuable in any area of learning.
salmo 6y ago
I may be a bad person to ask. I like history a lot. A lot of this stuff was originally picked up by secretaries and non-computer people working on manuals for Bell Labs. If you understand it, it will save you time. It will also make you use and understand Linux/Unix better, which will bleed into any other programming you do (assuming you write other code). The skill also comes up a lot in strange places like Dockerfiles, CI Pipelines, and a lot of that kind of surrounding infrastructure for applications. The video referenced in the article is actually pretty interesting, as is the book "The Unix Programming Environment". Both will refer to outdated technology like real terminals, but I think they help understand the intent of the systems we still use today. Any of the Bell Labs books are great for understanding Unix. Also do a minimal install of Linux or FreeBSD and read all of the man pages for commands in /bin, /usr/bin. Read the docs for the bash builtins. The old Useless Use of Cat is great: http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html But writing real scripts is probably the only way to learn. Next time you need to munge data, try the shell. When you read about a new command, like comm, try to use it to do something. Also force yourself into good practices: - revision control everything - provide -h output for every script you write (getopts is a good thing to be in the habit of using) - use shellcheck to beat bad habits out of you - try to do things in parallel when you can - assume everything you make is going to run at a much larger scale than you intend it to (more inputs, more servers, more everything) - after something works, see where you can reduce the number of commands/processes you run - write things like a moron is going to use them who will accidentally put spaces or hyphens or whatever in the worst place possible, not provide required arguments, and copy/paste broken things out of word - don't let scripts leave trash around, even when aborted: learn trap & signals The two things you should probably understand the most, but on which I'm the least help on are sed and awk. Think of sed as a way to have an automated text editor. Awk is about processing structured data (columns and fields). I learned these through trial and error and a lot of staring at examples. Understanding regular expressions is key to both and is, in general, an invaluable skill. Oh and if you are a vi(m) user, remember that the shell is always there to help you and is just a ! away. Things like `:1,17!sort -nr -k2` (sort lines 1-17, reverse numerical order on the second field) can save a ton of time. And even putting some shell in the middle of a file and running !!sh is super-handy to replace the current line that has some commands with the output of those commands.
asicsp 6y ago
I have books on GNU grep/sed/awk [0] (currently free in spirit of quarantine learning) that teaches the command and features step by step using plenty of examples and exercises. There's a separate chapter for regular expressions too. And I have a list of curated resources for `bash` and Linux [1]. Here's what I'd recommend: * https://ryanstutorials.net/linuxtutorial/ https://ryanstutorials.net/linuxtutorial/ is a good place to start for beginners * https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide and the rest of this wonderful wonderful site is recommended to understand `bash` and its features and gotchas * https://explainshell.com/ https://explainshell.com/ gives you a quick help on the various parts of the command, including documentation * https://www.shellcheck.net/ https://www.shellcheck.net/ is a must have tool if you are writing shell scripts * https://linuxjourney.com/ https://linuxjourney.com/ is probably the most complete site to understand how things work in *nix environment [0] https://learnbyexample.github.io/books/ https://learnbyexample.github.io/books/ [1] https://github.com/learnbyexample/scripting_course/blob/master/Linux_curated_resources.md https://github.com/learnbyexample/scripting_course/blob/mast...

viraptor 6y ago

> Understanding awk and especially sed will make your python better. Why do you think so?