9 ms·
Bit of questionable advice: I've found using jq for all the argument mangling/postprocessing/glue tasks instead of sed, awk etc to be pretty workable lately. T
by xg15 5y ago
Bit of questionable advice: I've found using jq for all the argument mangling/postprocessing/glue tasks instead of sed, awk etc to be pretty workable lately.
The -R switch reads in arbitrary text instead of json and presents it as a sequence of json strings internally.
The capture() filter applies regexes to the string and extracts subgroups (which will be represented as json objects).
The gsub() filter performs search-and-replace on a string.
Finally, the -r switch writes the result back to stdout as a raw value, removing any json quoting and escaping.
By default, processing is linewise, but there are options to process the whole input at once instead. There is also supposed to be a way how you can have per-line processing with state tracking across lines (like sed's "holding space"), but I haven't tried that out yet.
I found this approach often more flexible than sed and easier to reason about. Also, I believe it may be faster, as more things are happening in a single process.
The one exception: grep on files is still orders of magnitude faster than anything else. I believe there also was an article about this on HN a while ago.
So if you have to find some data in a larger collection of files and then have to turn the data into something structured, a good approach is to make a "coarse" query with grep or grep -R first, to narrow down your set of candidates, then postprocess your remaining candidates with jq.
Also, apropos single process: Another questionable advice is that for/while loops in bash are subshells and you can use them as part of a pipe chain. This often lets you reduce the number of processes you need.
E.g. instead of
for f in $files; do foo "$f" | sed ...; done
You can often write:
for f in $files; do foo "$f"; done | sed ...;
Which will only launch a single sed process for all files instead of one per file.
But yeah, shell scripts do have many shortcomings and in particular the error handling situation seems extremely hard to fix. So actually sensible advice is still: Use python.
- dmurray 5y ago> The one exception: grep on files is still orders of magnitude faster than anything else. I believe there also was an article about this on HN a while ago. I'd like to see that article! Faster than jq for sure, but my experience is that ack/ag/rg are all orders of magnitude faster than grep for typical uses.
- xg15 5y agoI think it was this one: https://news.ycombinator.com/item?id=23923645 https://news.ycombinator.com/item?id=23923645 Also I think there is a difference between grep on files, i.e. "grep ... file1 file2" or "grep ... -R somedir" and grep on stdin, i.e. "something | grep ...". My understanding was that in the first case, grep can make use of random access APIs to control the size of the input chunks it processes or skip over parts that are known in advance not to match. That's not possible if the input is fed through stdin. I haven't seen any actual comparisons though apart from "subjective" speed. So things might be wrong.
- Annatar 5y agoPython is a horrible language for system administration (or designing systems as in system engineering, for that matter): it's big, it's bloated, it's over-complicated, it's slow, it's hard to debug. Shell is the native automation facility of the UNIX-like operating systems, and with 40+ years behind it, it is well understood, easy to write, easy to debug, and has no artificial dependencies (so, completely the opposite of Python). Combine shell with AWK and you have an unbeatable combination for most computer automation and even very large scale data (pre)processing and number crunching.
- dijit 5y agoI disagree but only about 50%. Python is not really very slow overall unless you’re on windows. When you get to serious computing then breaking out of python is possible, but systems administration is not the place to do heavy computation. As for debugability, I find python to be easier, set -/+x is great, but that’s not doing much more than printing out every line of execution, there’s no interactive debugger like python has afaik. The killer thing for me is that python has modules for basically everything, which is awesome. The thing that kills python for me is that I can’t be 100% sure if what version may exist on a system; and worse: if I actually use any modules then I have to somehow get them on the system. For systems administration, this is pretty close to a non-starter. Prevailing sysadmin knowledge is that the admin tools should not horn in excessive dependencies, since sysadmin tools should work when the system isn’t fully set up. This is why go is so great. But go is harder to debug than python in my experience.
- TomSwirly 5y ago> The thing that kills python for me is that I can’t be 100% sure if what version may exist on a system; pip handles that automatically - you set which version numbers of Python you can handle in your project specification. > and worse: if I actually use any modules then I have to somehow get them on the system. pip handles that too! ---- I'm not quite sure what the problems are, but between virtualenv/venv, pip, and modern package managers like poetry, this is really a solved problem - something you spend an hour setting up when you start a project and just never think of again. I do this so much I have a shell function, `nenv`, which creates a new virtualenv with a specific Python and then loads its dependencies into the virtualenv. It gives me tremendous freedom. For example, I can heavily instrument other modules' Python code inside the virtualenv (to find their bugs, e.g.) and then throw it all away and recreate it fresh in a few keystrokes. --- There are OK, well-known solutions to most of the common packaging and distribution problems with Python programs you might have. If you go back to it, you should spend a bit of time with this, it isn't that bad. (Poetry seems really cool - when I start something new it'll be with that.)