5 ms·

Shell scripting for processing text. You can often get so much done with so little code and effort. Also on a semi-related note, I think as a self taught progr

by nickjj 8y ago

Shell scripting for processing text. You can often get so much done with so little code and effort. Also on a semi-related note, I think as a self taught programmer, it's easy to get stuck on things that seem cool but are just procrastination enablers (I know, I've been guilty of it for 20 years). Like, if you're about to start a new project and you want to flesh out what it's about, you really don't need to spend 5 hours researching which mind map tool to use. Just open a text document and start writing, or get a piece of paper and a pen. It won't even take that long. I spent about 1.5 hours the other day planning a substantially sized web app. All I did was open a text file and type what came into my head. For fun I decided to record the whole process too[0]. I wish more people recorded their process for things like that because I find the journey more interesting than the destination most of the time. Like your journey of eventually finding message queues must have been quite fun and you probably learned a ton (after all, it lead you to message queues, so it was certainly time well spent). [0]: https://nickjanetakis.com/blog/live-demo-of-planning-a-real-world-web-application-from-scratch https://nickjanetakis.com/blog/live-demo-of-planning-a-real-...

Draiken 8y ago
This 1000x. I put of just getting by with shell script for some years and when I finally decided to get deeper into it, it's magical. A good series of piped commands with tools available basically everywhere can solve problems you had no idea could be so simple to solve.
- 0x262d 8y ago
  have any good resources for this?
  - fancyfish 8y ago
    Perhaps these are a good start: awk: - GNU Awk User Guide - https://www.gnu.org/software/gawk/manual/gawk.html#Getting-Started https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S... - Grymoire guide - http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Awk.html sed: - Grymoire guide - https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html - Official docs - http://sed.sourceforge.net/#docs http://sed.sourceforge.net/#docs
  - bromonkey 8y ago
    man bash man awk man sed man grep You can also do most of this stuff with Perl one liners if that suits your fancy. sed + awk - https://www.amazon.com/sed-awk-Dale-Dougherty/dp/1565922255 https://www.amazon.com/sed-awk-Dale-Dougherty/dp/1565922255 awk - https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI... general *nix text processing - https://www.tldp.org/LDP/abs/html/textproc.html https://www.tldp.org/LDP/abs/html/textproc.html
  - a_bonobo 7y ago
    There's a free O'Reilly book from last year: https://www.datascienceatthecommandline.com/ https://www.datascienceatthecommandline.com/ awk is pure magic.
  - markvdb 7y ago
    The IBM developerworks articles about this are old (2000!), but still incredibly useful and well written. Start here: https://www.ibm.com/developerworks/library/l-sed1/index.html https://www.ibm.com/developerworks/library/l-sed1/index.html https://developer.ibm.com/tutorials/l-awk1/ https://developer.ibm.com/tutorials/l-awk1/
m463 7y ago
Also, just get to know your shell. big timesaver: control-r basic but useful all over: for i in *.c; do cp "$i" "$i.bak"; done etc...
- hi41 7y ago
  What does double quotes dollar do in the command?
  - MHP47 7y ago
    $i is the variable i declared in the for loop. Quotes just wraps it, so that it's (somewhat) safe if the file has a space in the name
    - majewsky 7y ago
      What do you mean, "somewhat"? This looks safe for all I can tell.
  - akx 7y ago
    Ensures filenames with spaces in them get passed as a single argument, instead of being inadvertently expanded into several. Quoting and expansion issues are a pain in shell languages...
  - deleted 7y ago
    [deleted]
  - nickjj 7y ago
    You know how you should almost always eat your vegetables? You should almost always quote your Bash variables too. Here's why: https://nickjanetakis.com/blog/here-is-why-you-should-quote-your-variables-in-bash https://nickjanetakis.com/blog/here-is-why-you-should-quote-...
  - aoeusnth1 7y ago
    It prevents filenames with spaces from expanding into two arguments to the command.
slightwinder 7y ago
These days it might be better to just learn python. It's cleaner and scales better to complex code. And it's ons most system modern systems available out of the box where shells are available too. Shells are still good for simple oneliners, and knoting multiple processes together, but text-processing involves so many different commands, each with their own quirks, that a consistent simple language is IMHO superiour.
- chaostheory 7y ago
  Honestly this applies to Ruby & Javascript/Typescript as well and not just Python. I really don't see the value of learning shell scripting anymore when the newer languages are just was easy to learn, terse, and you can adapt better to changing conditions when needed with libraries.
  - latexr 7y ago
    I often find multi-line Python scripts with `import os` and others that could be a fraction in size (and just as clear) in bash. Even more ridiculous are the times I find a node script (published to npm, even) that is little more than a wrapper on a shell script. Inevitably someone will read these arguments and think “those are just bad programmers”, but your point was that you “don't see the value of learning shell scripting”. The value is in not spewing absurd code like that. Shell commands are fast and efficient. There isn’t an emphasis on libraries because instead you use tools. Is `grep` not enough? Try `the silver searcher`[1] or `ripgrep`[2]. Are shell scripts the best instrument for every job? No, but no tool is. [1]: https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher [2]: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
    - chaostheory 7y ago
      Just because someone knows shell scripting I'm not going to consider them "bad programmers". I know it myself and I've used it extensively before I learned python & ruby. My point is that the cost for learning and using shell scripts is just too high compared to just using a modern language that's just as terse and a lot more powerful and flexible. Context switching from one language to another isn't free either. imo the only time shell scripting was practical was when the only major programming languages were C, C++, and Java. imo even Perl5 is more practical than shell scripting. Also I doubt that the python program you mentioned was that much bigger than a shell script
chriswarbo 7y ago
I'd split your first point in two: - Shell scripting for running commands and managing files - Unix utilities for processing text These happen to complement each other nicely. It's not that bash is better at manipulating text than Python, it's that Python makes it painful to invoke commands (like those Unix utilities) and pipe data between them (e.g. https://news.ycombinator.com/item?id=17733865 https://news.ycombinator.com/item?id=17733865 ).

by nickjj 8y ago

Draiken 8y ago

This 1000x. I put of just getting by with shell script for some years and when I finally decided to get deeper into it, it's magical. A good series of piped commands with tools available basically everywhere can solve problems you had no idea could be so simple to solve.

0x262d 8y ago
have any good resources for this?
- fancyfish 8y ago
  Perhaps these are a good start: awk: - GNU Awk User Guide - https://www.gnu.org/software/gawk/manual/gawk.html#Getting-Started https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S... - Grymoire guide - http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Awk.html sed: - Grymoire guide - https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html - Official docs - http://sed.sourceforge.net/#docs http://sed.sourceforge.net/#docs
- bromonkey 8y ago
  man bash man awk man sed man grep You can also do most of this stuff with Perl one liners if that suits your fancy. sed + awk - https://www.amazon.com/sed-awk-Dale-Dougherty/dp/1565922255 https://www.amazon.com/sed-awk-Dale-Dougherty/dp/1565922255 awk - https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI... general *nix text processing - https://www.tldp.org/LDP/abs/html/textproc.html https://www.tldp.org/LDP/abs/html/textproc.html
- a_bonobo 7y ago
  There's a free O'Reilly book from last year: https://www.datascienceatthecommandline.com/ https://www.datascienceatthecommandline.com/ awk is pure magic.
- markvdb 7y ago
  The IBM developerworks articles about this are old (2000!), but still incredibly useful and well written. Start here: https://www.ibm.com/developerworks/library/l-sed1/index.html https://www.ibm.com/developerworks/library/l-sed1/index.html https://developer.ibm.com/tutorials/l-awk1/ https://developer.ibm.com/tutorials/l-awk1/

m463 7y ago

Also, just get to know your shell. big timesaver: control-r basic but useful all over: for i in *.c; do cp "$i" "$i.bak"; done etc...

hi41 7y ago
What does double quotes dollar do in the command?
- MHP47 7y ago
  $i is the variable i declared in the for loop. Quotes just wraps it, so that it's (somewhat) safe if the file has a space in the name
  - majewsky 7y ago
    What do you mean, "somewhat"? This looks safe for all I can tell.
- akx 7y ago
  Ensures filenames with spaces in them get passed as a single argument, instead of being inadvertently expanded into several. Quoting and expansion issues are a pain in shell languages...
- deleted 7y ago
  [deleted]
- nickjj 7y ago
  You know how you should almost always eat your vegetables? You should almost always quote your Bash variables too. Here's why: https://nickjanetakis.com/blog/here-is-why-you-should-quote-your-variables-in-bash https://nickjanetakis.com/blog/here-is-why-you-should-quote-...
- aoeusnth1 7y ago
  It prevents filenames with spaces from expanding into two arguments to the command.

slightwinder 7y ago

These days it might be better to just learn python. It's cleaner and scales better to complex code. And it's ons most system modern systems available out of the box where shells are available too. Shells are still good for simple oneliners, and knoting multiple processes together, but text-processing involves so many different commands, each with their own quirks, that a consistent simple language is IMHO superiour.

chaostheory 7y ago
Honestly this applies to Ruby & Javascript/Typescript as well and not just Python. I really don't see the value of learning shell scripting anymore when the newer languages are just was easy to learn, terse, and you can adapt better to changing conditions when needed with libraries.
- latexr 7y ago
  I often find multi-line Python scripts with `import os` and others that could be a fraction in size (and just as clear) in bash. Even more ridiculous are the times I find a node script (published to npm, even) that is little more than a wrapper on a shell script. Inevitably someone will read these arguments and think “those are just bad programmers”, but your point was that you “don't see the value of learning shell scripting”. The value is in not spewing absurd code like that. Shell commands are fast and efficient. There isn’t an emphasis on libraries because instead you use tools. Is `grep` not enough? Try `the silver searcher`[1] or `ripgrep`[2]. Are shell scripts the best instrument for every job? No, but no tool is. [1]: https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher [2]: https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
  - chaostheory 7y ago
    Just because someone knows shell scripting I'm not going to consider them "bad programmers". I know it myself and I've used it extensively before I learned python & ruby. My point is that the cost for learning and using shell scripts is just too high compared to just using a modern language that's just as terse and a lot more powerful and flexible. Context switching from one language to another isn't free either. imo the only time shell scripting was practical was when the only major programming languages were C, C++, and Java. imo even Perl5 is more practical than shell scripting. Also I doubt that the python program you mentioned was that much bigger than a shell script

chriswarbo 7y ago

I'd split your first point in two: - Shell scripting for running commands and managing files - Unix utilities for processing text These happen to complement each other nicely. It's not that bash is better at manipulating text than Python, it's that Python makes it painful to invoke commands (like those Unix utilities) and pipe data between them (e.g. https://news.ycombinator.com/item?id=17733865 https://news.ycombinator.com/item?id=17733865 ).