9 ms·
Benchmarking shell pipelines and the Unix “tools” philosophy
- Neil44 7y agoI got excited when I saw the 'f' and 'count' commands, but they're just scripts he has on his system. Like doing grep 'plover' blah.log | cut -d ' ' -f 11 | sort | uniq -c | sort -n. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system, but they are nice.
- amp108 7y agoThat's the whole point of shell scripting, to take a series of minimal programs and tie them together into something that does a more complex task. There's no reason to distrust a shell script simply because it is a script any more than there is to trust a binary simply because it's a binary.
- cholmon 7y agoSure, but relying on custom shell scripts as unix primitives can be problematic if you find yourself frequently managing/troubleshooting systems that you don't own, and you don't want to (or aren't allowed to) put those handy scripts in place. Then when you're on any given system, you forget whether you can use "f", or if you have to fall back on awk. I think it's less about not trusting custom scripts than it is about ensuring that your unix muscle memory doesn't atrophy.
- andrey_utkin 7y ago> if you find yourself frequently managing/troubleshooting systems that you don't own You start thinking about packaging.
- Galanwe 7y agoPackaging what? I would be mad at any admin that would dare to deploy his helper shell scripts like “addup” and “count” on a machine other than his laptop. And if you meant he could just have these things in his home, then it defeats the purpose of the original comment: trouble shooting and administering machines forces you to often switch user, machine, etc.
- dtoma 7y agoIf a company has recurring troubleshooting issues, eg. "we need to know which process is taking all the CPU/RAM/IO", why wouldn't they add a script when they provision their machines with puppet or whatever tool they use? Then instead of having to remember how to check all these things, they just run "find_resource_hogs.sh" and voila. It also enables other people to troubleshoot without specific knowledge. Of course you don't want to put anything in there just because it might save 10 seconds, but then https://xkcd.com/1205/ https://xkcd.com/1205/
- Galanwe 7y agoI agree 100% with you. I spend a lot of time moving around different machines, processes, configuration files, logs, etc. And I stopped maybe 10 years ago to use anything that is not available on a base system. I don’t use fancy shells, I don’t use aliases, I don’t write local shortcut scripts, etc. I just use regular bash, combine base utilities in one liners, and live with it. Maybe I loose 1s here and there when writing one liners compared to someone with a library of wrapper utilities. But that gives me an immense benefit: I am at home on any machine, any distribution, everywhere, without any configuration, with any user.
- blunte 7y agoThere's always something like Ansible (or even a lower-tech solution) to at least give you the basic toolset that helps you double or triple your performance.
- Galanwe 7y ago> that helps you double or triple your performance. Sure, in a quest of productivity, I should also use Vagrant and Packer to create docker images with a development environment so that I can run Serverless troubleshooting containers on all my machines. These scripts will surely help me triple the performance of my bash one liners. I’ve heard Eclipse has good shell completion and support for oh-my-zsh too.
- tasogare 7y agoThat’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution).
- tuldia 7y ago> That’s the theory but frankly the syntax is so cumbersome, irregular and needs so many googling for "easy" things like conditional, substring, etc. that I now use a real programming language if a script needs to be anything more than a list of commands without any logic (besides variables substitution). You are basically describing modern programming. Script Language (or scripting) is a programming language. And about the "real" programming language you can also trap yourself googling and installing yet another library (did you read the code?) and/or reimplementing existing tools from the unix programming environment.
- tasogare 7y agoYou are overly pedantic on a detail point that doesn’t matter: yes shell scripting is technically a programming language, but my point is that it is a terrible one worth ditching for any non trivial task. Perl was created precisely more than 3 decades ago to address this problem. Nowadays there are alternatives such as Python, Powershell or even scripting wrapper for compiled languages (such as C#) that allow to do the same job very well, with less surprising behavior and that can be refactored later more easily.
- vaingloriole 7y agoI disagree with you wholeheartedly and without condition. Not only are you discounting how much time it takes to learn how to program effectively in a real 'glue' language you are high handed in ignoring the ubiquity, working archive and efficacy of a shell script. Not to mischaracterize but I find this type of attitude most frequently in 'lead' individuals with less than 10 years experience: typically 20's and 30's in age. I'm curious if this is your case?
- juped 7y agoMost people who use Unix directly build up some stuff in ~/bin (often a misnomer because it's shell scripts and not binaries, although mine is less of a misnomer than most because so much is in C rather than shell). The trick is to build them out of the standard portable components that exist everywhere. (This means, among other things, no #!/bin/bash.)
- tuldia 7y agosed 's| no | not only |'
- juped 7y agoI disagree.
- tuldia 7y agoYou are also right. Now that I saw tingletech comment I got your point :)
- tingletech 7y ago/bin/bash won't usually ship with a BSDish OS because of the license, so it is not generally portable to use bash-isms. (HPUX, IRIX, SunOS, Solaris, etc. I don't reckon would have had bash either)
- chriswitts 7y ago
- tuldia 7y ago> [...] but they're just scripts he has on his system. Personally I'd prefer to use the ubiquitous commands that work everywhere than rely on having custom scripts on my system [...] Is okay for one to have their own tools. $ f() { printf "\$%s" "$1"; } $ echo a b c | awk '{ print $(f 2) }' His system is not very different from mine or yours. He just chose to combine the tools in a specific way.
- rmetzler 7y agoInstead of doing grep 'plover' blah.log | cut -d ' ' -f 11 I usually do awk '/plover/{print $11}' blah.log Less typing, but probably not as easy to understand.
- justinsaccount 7y ago'sort | uniq -c | sort -n' is an interesting pipeline. It will always work and does a great job with large cardinality data on low memory systems. However, if you have the ram, or know the data set has a low cardinality (like, http status codes or filesnames instead of ip addresses) then something that works in memory will be much more efficient. I threw 144,000,000 'hello' and 'world' into a file: justin@box:~$ ls -lh words -rw-r--r-- 1 justin justin 824M Jan 7 15:21 words justin@box:~$ wc -l words 144000000 words justin@box:~$ time (sort <words|uniq -c) 72000000 hello 72000000 world real 0m22.831s user 0m32.999s sys 0m4.675s Compared to doing it in memory with awk: justin@box:~$ time awk '{words[$1]++} END {for (w in words) printf("%s %d\n", w, words[w])}' < words hello 72000000 world 72000000 real 0m10.639s user 0m9.736s sys 0m0.876s so, half the time and 1/3 the cpu.
- tuldia 7y agoThis is because in the first example you are invoking two programs. The first one sort the content of the file, the second count how many lines are equal. While in the awk example it is creating a hash table with all words and incrementing by the key and then printing. There is no sorting plus printing may be buffered.
- justinsaccount 7y agoThanks for explaining my own comment to me.
- tuldia 7y agoNot explaining, trying to tell that you are comparing apples to oranges and making a conclusion based on that. Also, you don't need to spawn a subshell nor feed sort via stdin in the first example :)
- cperciva 7y agoHe's comparing apples to oranges and reaching the conclusion that... yes, apples and oranges are different things. He's quite aware of this, and even points out the tradeoff -- `sort | uniq -c` still works if your dataset doesn't fit into RAM.
- tuldia 7y agoThanks for this! Another nice thing about /usr/bin/time is the --verbose flag which gives: Command being timed: "ls" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 0% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 1912 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 112 Voluntary context switches: 1 Involuntary context switches: 1 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 :)
- yingw787 7y agoWow this looks amazing! I didn’t know time could track all of those!
- vaingloriole 7y agoman getrusage
- boyter 7y agoCan anyone comment why you can only use the verbose flag if you use the full path of time? time -v ls does not work but /usr/bin/time -v ls does? I don't have enough knowledge of either linux applications or bash to know whats happening to cause this.
- eesmith 7y agoYour shell has a built-in "time" command. man bash: If the time reserved word precedes a pipeline, the elapsed as well as user and system time consumed by its execution are reported when the pipeline terminates. man time: Some shells may provide a builtin time command which is similar or identical to this utility. Consult the builtin(1) manual page.
- 7y ago
- skywhopper 7y ago"What if Unix had less compositionality but I could use it with less memorized trivia? Would that be an improvement? I don't know." The answer is "no" here, because the alternative doesn't exist. Could it be created? Maybe in theory, but I suspect that the amount of stuff that you'd need to memorize (or learn to look up) to use it effectively would be about the same for any system that allowed a similar variety of work to be accomplished. If you are willing to trade off functionality for simplicity, then sure, it can be done. You can get it today by just not using all these tools at all, I suppose.
- jessant 7y agoThat is what the author says in the next paragraph. > I don't know. I rather suspect that there's no way to actually reach that hypothetical universe.
- nibbula 7y agoI'm already living in that universe, but with even more composability.
- wahern 7y agoThere would be less trivia to memorize if the command behaviors and options were more consistent. You may not be able to achieve that at the edges, where new commands and options are added, but you can always go back and clean things up. For example, the cut(1) command is intended to do precisely what his f script does. But it's inconvenient because unlike many other commands it (1) doesn't obey $IFS and (2) the -d delimiter option only takes a single character. This could and should be remediated with a new, simple option. I think the only thing preventing that change is that there's not enough interest in moving POSIX forward faster; certainly not like JavaScript. Another problem are GNU tools. They have many great features but OMG are they a nightmare of inconsistency. BSD extensions tend to be much better thought through, perhaps because GNU tools tend to be lead by a single developer while BSD tools tend to be more team oriented. So the way forward isn't to replace the organic evolution, it's to layer on processes that refine the proven extensions. And we already have some of those processes in place; we just need to imbue them with more authority, and that starts by not rolling our eyes at standardization and portability.
- 3xblah 7y agofor i in $(seq 1 20); do (run once and emit the total CPU time) done |addup "Here we don't actually care about the output (we never actually use $i) but it's a convenient way to get the for loop to run twenty times." This is slower than not running seq and just using builtins. n=1;while true;do test $n -le 20||break; (run once and emit the total CPU time) n=$((n+1)); done |addup
- deathanatos 7y agoThe author did say convenient, not fast. If you don't want the inefficiencies of seq, bash has: for (( expr1 ; expr2 ; expr3 )) ; do list ; done which is a lot more idiomatic than constructing a for loop out of a while loop.
- pimlottc 7y agoYou could also just use a bash brace expansion: for i in {1..20} do ... done https://wiki.bash-hackers.org/syntax/expansion/brace https://wiki.bash-hackers.org/syntax/expansion/brace
- deleted 7y ago[deleted]
- 3xblah 7y agoThe author refers to "the shell" in his "benchmark" but if he is a Linux user then sh -c 'f 11 access.2020-01-* ... is dash whereas f 11 access.2020-01* ... is bash Needless to say, one of these shells is much faster than the other and only one contains the "time" builtin command. https://wiki.ubuntu.com/DashAsBinSh https://wiki.ubuntu.com/DashAsBinSh Then there is the use of perl and its system command in "count". As with seq, why is perl needed. No explanation. Why not just put the entire pipeline into a perl system command.
- m463 7y agothe ACTUAL benchmark should be: 1) start timer 2) start deciding which commands to pipeline together 3) run the commands 4) stop timer a lot of times the decision is the long pole. in this authors case it included: 5) try a couple more variants of steps 2 and 3 6) write a blog post :)
- CGamesPlay 7y agoThis is a good point about the second half of the article (compositonality), but the author started the article by saying this was a command he "sometimes runs", presumably indicating he has it saved somewhere.
- m463 7y agoMy commands of this type are usually retrieved using control-r
- perfunctory 7y ago> The appearance of the TIME=… assignment at the start of the shell command disabled the shell's special builtin treatment of the keyword time, so it really did use /usr/bin/time. This computer stuff is amazingly complicated. I don't know how anyone gets anything done. Indeed.
- madacol 7y agoI haven't been able to open this site for days (Timeout error, thought it was down), until today that I decided to use tor