5 ms·

You just need to master a few commands: Find (take a look at -exec option) Cut (or awk) Sed (for simple text/string substitutions) Xargs Dd (that beast can

by eb0la 8y ago

You just need to master a few commands: Find (take a look at -exec option) Cut (or awk) Sed (for simple text/string substitutions) Xargs Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks) And bash of course :)

sidlls 8y ago
awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.
- flukus 8y ago
  I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html https://two-wrongs.com/awk-state-machine-parser-pattern.html I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.
  - cosmie 8y ago
    If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]! [1] https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
- cwilkes 8y ago
  An example here would be nice.
hoffbrau99 8y ago
Don't forget grep!
eanzenberg 8y ago
"master a few commands" "awk" brb 15 years
- ccozan 8y ago
  Might be true. The sed&awk from O'Reilly still lingers on my desk from time to time. And I bought it 20 years ago.
- buserror 8y ago
  Not really. awk is pretty simple, basically I just assume it's plain C with a huge regexp pattern 'switch' case at the top level, then I google again for the arguments to substr() :-)
  - heyoni 8y ago
    Brb in 20 years then.
pisky 8y ago
Also, 'parallel' if you've got any idle cores on your infrastructure.
foobarian 8y ago
And perl -pe or -ne ! :)
- Groxx 8y ago
  ^ the moment you do anything even slightly non-trivial with sed, abandon it and use perl instead. for even simple regexes with a couple non-"match this char" segments I regularly see almost 100x speedup, it's rather crazy. it's also a LOT more portable in my experience because the regex engine is so much more capable and consistent, even across fairly major perl versions.
tjr225 8y ago
Some of these have come up for me recently, particular sed, awk, and xargs. Xargs in particular is really gnarly and opens up a lot of possibilities from the command line when used in conjunction with other tools. Love it! Not a program, but another fun bash thing I've learned about recently is brace expansion: https://www.linuxjournal.com/content/bash-brace-expansion https://www.linuxjournal.com/content/bash-brace-expansion
jake_morrison 8y ago
One fun thing is that xargs supports parallelization. -P max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option with -P; otherwise chances are that only one exec will be done. -n max-args Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x option is given, in which case xargs will exit. So here is a "map/reduce" job which takes the log files in a directory and processes them in parallel on eight CPUs, then combines the results. find . -name "*.log" | sort | xargs -n 1 -P 8 agg_stats.py | sort | merge_periods.py
- adrianN 8y ago
  And for cases where xargs parallelism isn't flexible enough, there is GNU parallel.
- Groxx 8y ago
  xargs is one of my favorites, it's so easy to split something apart to speed it up by a few multiples. gotta SSH into 100 machines and ask them all a simple question? xargs will trivially speed that up by at least 10x, if not better, just by parallelizing the SSH handshakes.
- IndustrialJane 8y ago
  But that approach is potentially dangerous. See: https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...
throwaway5752 8y ago
'man' is probably the most indispensable.
- commandlinefan 8y ago
  That sounds interesting. Is it documented anywhere?
  - throwaway5752 8y ago
    No, it's something neckbeards pass along to junior sysadmins, and it's hidden very deep in /dev/null.
- jonhendry18 8y ago
  I find 'apropos' pretty indispensable, because it lets you find the thing that does the thing you need, so you can read its man page.
rickycook 8y ago
adding socat into the mix if you do network related things! it’s amazing what you can do when you replace bits and pieces of a pipe chain with network endpoints
crehn 8y ago
Honorable mention to jq. Also get comfy with Unix pipes and streams. e.g. testing in CI a JSON file is properly formatted: jq . < $f | diff -u $f -
- frou_dh 8y ago
  Personally, I'm not too interested in any commands that take string arguments in their own homegrown programming language[0]. I'd rather use something like this[1] than jq, to get some extra ROI out of one of the languages I already know. [0]: https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/ [1]: https://github.com/antonmedv/fx https://github.com/antonmedv/fx

by eb0la 8y ago

sidlls 8y ago

awk is so beautiful. A good awk one-liner can beat fancy pandas all day long in some cases.

flukus 8y ago
I hate how awk gets so much praise for one liners when it's so good at building things like state machines too: https://two-wrongs.com/awk-state-machine-parser-pattern.html https://two-wrongs.com/awk-state-machine-parser-pattern.html I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.
- cosmie 8y ago
  If anything, awk should get some praise for it's flexibility to go from one liners to Doom-style 3d games[1]! [1] https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
cwilkes 8y ago
An example here would be nice.

hoffbrau99 8y ago

Don't forget grep!

eanzenberg 8y ago

"master a few commands" "awk" brb 15 years

ccozan 8y ago
Might be true. The sed&awk from O'Reilly still lingers on my desk from time to time. And I bought it 20 years ago.
buserror 8y ago
Not really. awk is pretty simple, basically I just assume it's plain C with a huge regexp pattern 'switch' case at the top level, then I google again for the arguments to substr() :-)
- heyoni 8y ago
  Brb in 20 years then.

pisky 8y ago

Also, 'parallel' if you've got any idle cores on your infrastructure.

foobarian 8y ago

And perl -pe or -ne ! :)

Groxx 8y ago
^ the moment you do anything even slightly non-trivial with sed, abandon it and use perl instead. for even simple regexes with a couple non-"match this char" segments I regularly see almost 100x speedup, it's rather crazy. it's also a LOT more portable in my experience because the regex engine is so much more capable and consistent, even across fairly major perl versions.

tjr225 8y ago

Some of these have come up for me recently, particular sed, awk, and xargs. Xargs in particular is really gnarly and opens up a lot of possibilities from the command line when used in conjunction with other tools. Love it! Not a program, but another fun bash thing I've learned about recently is brace expansion: https://www.linuxjournal.com/content/bash-brace-expansion https://www.linuxjournal.com/content/bash-brace-expansion

jake_morrison 8y ago

One fun thing is that xargs supports parallelization. -P max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option with -P; otherwise chances are that only one exec will be done. -n max-args Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x option is given, in which case xargs will exit. So here is a "map/reduce" job which takes the log files in a directory and processes them in parallel on eight CPUs, then combines the results. find . -name "*.log" | sort | xargs -n 1 -P 8 agg_stats.py | sort | merge_periods.py

adrianN 8y ago
And for cases where xargs parallelism isn't flexible enough, there is GNU parallel.
Groxx 8y ago
xargs is one of my favorites, it's so easy to split something apart to speed it up by a few multiples. gotta SSH into 100 machines and ask them all a simple question? xargs will trivially speed that up by at least 10x, if not better, just by parallelizing the SSH handshakes.
IndustrialJane 8y ago
But that approach is potentially dangerous. See: https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...

throwaway5752 8y ago

'man' is probably the most indispensable.

commandlinefan 8y ago
That sounds interesting. Is it documented anywhere?
- throwaway5752 8y ago
  No, it's something neckbeards pass along to junior sysadmins, and it's hidden very deep in /dev/null.
jonhendry18 8y ago
I find 'apropos' pretty indispensable, because it lets you find the thing that does the thing you need, so you can read its man page.

rickycook 8y ago

adding socat into the mix if you do network related things! it’s amazing what you can do when you replace bits and pieces of a pipe chain with network endpoints

crehn 8y ago

Honorable mention to jq. Also get comfy with Unix pipes and streams. e.g. testing in CI a JSON file is properly formatted: jq . < $f | diff -u $f -

frou_dh 8y ago
Personally, I'm not too interested in any commands that take string arguments in their own homegrown programming language[0]. I'd rather use something like this[1] than jq, to get some extra ROI out of one of the languages I already know. [0]: https://stedolan.github.io/jq/manual/ https://stedolan.github.io/jq/manual/ [1]: https://github.com/antonmedv/fx https://github.com/antonmedv/fx