6 ms·

Every time I see things like this, I feel like the person must be unaware of awk. # the original one-liner to get unique IP addresses cut -d' ' -f 1 access

• zer8k 3y ago
  I don't understand the downvotes. This is a fair criticism. The author even points out "programs as pipelines" which is literally the UNIX philosophy. There are tools that already exist on UNIX-likes more people should use instead of reaching for a script. I can sympathize with the author w.r.t wanting to use a single language you like for everything. However, after decades I've found this to be untenable. There are languages that are just simply better for one-off scripting (Perl, Python), and languages that aren't (anything compiled). Trying to bolt an interpreter onto a compiled language from the outside seems like a lot of work for questionable gain.
  • karmakaze 3y ago
    The 'scripting' vs 'compiled' language is a false dichotomy. Awk, Perl, Python are compiled programs. What makes a 'scripting' language special? Dynamic typing? Lack of compile step/delay? I could imagine a lifetime of collecting scripting macros/libs in lisp to be as good or better.
    • tempusr 3y ago
      Python is not a compiled language. However, the reason Bash is so prolific amongst Sys Admins such as myself is the fact that they are portable and reliable to use across Debian, Arch or RHEL based distributions. You don't have to import extra libraries, ensure that you are running the proper python environment, or be certain that pip is properly installed and configured for whatever extra source code beyond what is included out of the box. Bash is the most consistent code you can write to perform any task you need when you have to work with Linux.
      • dragonwriter 3y ago
        > Python is not a compiled language. Python is (at least in the CPython implementation) compiled, to python byte code which runs on the python virtual machine. Its not compiled to native code. (Unless you use one of the compilers which do compile it to native code, though they tend to support only a subset of python.)
      • LeBit 3y ago
        Bash is fine for small scripts. Once you use it to manage complex data structures and flow, you are simply wasting time because you will have to rewrite it in Python or Go.
      • pdimitar 3y ago
        Another commenter beat me to it but still: sh / bash / zsh are quite fine up until certain complexity (say 500 lines), after which adding even a single small feature becomes a huge drag. We're talking hours for something that would take me 10 minutes in Golang and 15 in Rust.
  • ajross 3y ago
    > There are languages that are just simply better for one-off scripting (Perl, Python), and languages that aren't (anything compiled). Trying to bolt an interpreter onto a compiled language from the outside seems like a lot of work for questionable gain. One reason is deployment. Writing code in python/node/etc... implies the ability of the production environment to bootstrap a rather complicated installation tree for the elaborate runtimes required by the code and all its dependencies. And so there are elaborate tools (npm, venv, Docker, etc...) that have grown up around those requirements. Compiled languages (and Go in particular shines here) spit out a near-dependency-free[1] binary you can drop on the target without fuss. I deal with this in my day job pretty routinely. Chromebooks have an old python and limited ability to pull down dependencies for quick test runs. Static test binaries make things a lot easier. [1] Though there are shared libraries and runtime frameworks there too. You can't deploy a Gnome 3 app with the same freedom you can a TCP daemon, obviously.
    • LeBit 3y ago
      I agree with you. But I think for python you could also deploy a binary with pyinstaller.
    • nijave 3y ago
      > Compiled languages (and Go in particular shines here) spit out a near-dependency-free[1] binary you can drop on the target without fuss I think it's more accurate to say "static binaries" instead of "compiled languages." The same headaches exist with dynamically linked, compiled binaries (and sometimes they're worse since you don't have a dependency manager, unless you add one) > Static test binaries make things a lot easier I think this really depends on your target environment and how much control you have over it. If you're in a Ruby or Python shop, for example, all your servers already have the stack installed. If you're targeting end user devices, those can have a huge mess of different configs to account for
      • ajross 3y ago
        Meh. In my experience version churn with shared library dependencies is pretty minor. You have to worry about and work at it if you're doing stuff like deploying a single binary across a bunch of different linux distros. But the straightforward case of "build on your desktop and copy the file up to the host" is a routine thing you can expect to work. It's nothing like that with Python or Node. The rule there is that you get something working locally and then spend a while reverse engineering a pip/venv recipe or manifest or whatever to make it work somewhere else. It's decidedly non-trivial.
  • sgarland 3y ago
    > The author even points out "programs as pipelines" which is literally the UNIX philosophy. Yes, and if the thing I'm trying to do has a small input, it will only be done once, etc. I will often just pipe `grep` to `sort` or whatever, because it's less typing, it's generally clearer to a wider range of people, etc. But on larger inputs, or even things like doing a single pattern inversion mixed with a pattern match, I like awk.
  • tomcam 3y ago
    One reason the author could be doing this is to reduce dependencies. Maybe they deploy to Windows or to some other environment not guaranteed to have those utilities. Also testing probably gets simplified.
• ajross 3y ago
  And every time I see things like that, I feel like the person must be unaware of perl. I've made this point before, but I still find it hilarious. For more than a decade, awk was dead. Like, dead dead. There was nothing you could do in awk that wasn't cleaner and simpler and vastly more extensible in perl. And, yes, perl was faster than gawk, just like gawk is faster than shell pipelines. Then python got big, people decided that they didn't want to use perl for big projects[1], and so perl went out of vogue and got dropped even for the stuff it did (and continues to do) really well. Then a new generation came along having never learned perl, and... ... have apparently rediscovered awk? [1] Also the perl 5 tree stagnated[2] as all the stakeholders wandered off into the weeds to think about some new language. They're all still out there, AFAIK. [2] Around 2000-2005, perl was The Language to be seen writing your new stuff in, so e.g. bioinformatics landed there and not elsewhere. But by 2015, the TensorFlow people wouldn't be caught dead writing perl.
  • voidfunc 3y ago
    Perl never recovered from its many ways to do things label. It's a tired criticism of the language but it's lodged in the brains of a generation of programmers which is unfortunate. Also the classic sysadmin role which used to lean on Perl heavily sort of evolved with rise of The Cloud and automation tools like Chef, Puppet, and Ansible took over in that 2005-2015 time frame.
  • sgarland 3y ago
    That's a fair criticism. I know Perl can do pretty amazing things with text, but I've never bothered to learn it. EDIT: I decided to ask GPT-4 to translate the gawk script to Perl. I make zero claims that this is ideal (as stated, I don't know Perl at all), but it _does_ produce the same output, but slightly slower than the gawk script. $ hyperfine "perl -lane '\$ips{\$F[0]}++; END {print \"\$ips{\$_} \$_\" for (sort {\$ips{\$b} <=> \$ips{\$a}} keys %ips)[0..9]}' fake_log.txt" Benchmark 1: perl -lane '$ips{$F[0]}++; END {print "$ips{$_} $_" for (sort {$ips{$b} <=> $ips{$a}} keys %ips)[0..9]}' fake_log.txt Time (mean ± σ): 1.499 s ± 0.006 s [User: 1.447 s, System: 0.050 s] Range (min … max): 1.490 s … 1.507 s 10 runs
    • 3y ago
• jeffbee 3y ago
  The overwhelming cost of the first shell pipeline, at least on my machine, is caused by the default UTF-8 locale. As I have found in almost every other case, `LC_ALL=C` radically speeds this up. Original: 3.294s w/ LC_ALL=C: 1.055s w/ larger sort buffer `-S5%`: 0.780s Your gawk: 1.772s + LC_ALL=C: 1.772s By the way, these changes immediately suggested themselves after running the pipeline under `perf`. Profiling is always the first step in optimization.
  • sgarland 3y ago
    Collation aside (which is absolutely a huge boost in speed that I neglected to think about), I assumed that the rest of the difference was coming from the fact that the initial `cut` meant the rest of the pipeline had far less to deal with, whereas `awk` is processing every line. Benchmarking (and testing in `perf`) showed this to not be the case. I'd need to compile `awk` with debug symbols, I think, to know exactly where the slowdown is, but I'm going to assume it's mostly due to `sort` being extremely optimized for doing one thing, and doing it well. I did find one other interesting difference between BSD and GNU tools - BSD sort defaults to 90% for its buffer, GNU sort defaults to 1024 KiB. Combining all of these (and using GNU uniq - it was also faster), I was able to get down to 463 msec on the M1 Air: $ hyperfine "export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head" Benchmark 1: export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head Time (mean ± σ): 463.4 ms ± 3.3 ms [User: 965.5 ms, System: 93.3 ms] Range (min … max): 459.9 ms … 469.8 ms 10 runs TIL, thank you.
    • xvector 3y ago
      Could you elaborate on how you arrived at 5% for your buffer? Does specifying a buffer size really cause that much of a speed up?
      • sgarland 3y ago
        Parent poster used the 5% value, so I did as well. But yes, to answer your question. Shown here on a Debian 11 system that's fairly old: $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head Time (mean ± σ): 1.504 s ± 0.318 s [User: 2.833 s, System: 0.474 s] Range (min … max): 0.942 s … 1.937 s 10 runs $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head Time (mean ± σ): 3.847 s ± 0.093 s [User: 4.165 s, System: 0.613 s] Range (min … max): 3.591 s … 3.919 s 10 runs Setting the buffer value to ~half the file size (100 MB) resulted in a mean time of 2.291 seconds. Setting it to the size of the file resulted in a mean time of 1.549 seconds, which is close enough to the 5% to call it equal - not like this server isn't busy with other stuff, so it's hardly a good place for perfect benchmarking. Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.
• kermatt 3y ago
  Mawk can be even faster, although missing some features of GNU Awk 5.
• xdsdvsv 3y ago
  way to completely miss the point and turn this into a weird pissing competition (btw your "simple" awk example is super complicated and opaque to someone who doesn't have the awk man page open in front of them) The script package looks really cool and I'll definitely try it out, cause honestly even though I do a lot of bash scripting it's super painful for anything but something super simple.
  • sgarland 3y ago
    If someone doesn't know awk, then of course it'll be complicated and opaque - the same is true of practically any language. One-liners in general also tend to optimize for space. If you wanted it to be pretty-printed and with variable names that are more obvious: { PROCINFO["sorted_in"] = "@val_num_desc" top_ips[$1]++ } END { counter = 0 for (i in top_ips) { if (counter++ < 10) { print top_ips[i], i } } } But also, if you read further up in the thread, you'll see that another user correctly identified the bottlenecks in the original pipeline, and applying those optimizations made it about 3x as fast as the awk one. Arguably, if you weren't familiar with the tools (and their specific implementations, like how GNU sort and BSD sort have wildly different default buffer sizes), you'd still be facing the same problem. At least half of what people complain about with shell scripts can be solved by using ShellCheck [0], and understanding what it's asking you to do. I disagree with the common opinion of "anything beyond a few lines should be a Python script instead." If you're careful with variable scoping and error handling, bash is perfectly functional for many uses. [0]: https://www.shellcheck.net https://www.shellcheck.net
    • dharmab 3y ago
      > If someone doesn't know awk, then of course it'll be complicated and opaque - the same is true of practically any language I don't think this is true. Before I learned Go, I could follow along most Go programs pretty well, and learning Go well enough to get started took less than an hour. Every attempt I've made to learn more Awk, I've bounced off.
      • deleted 3y ago
        [deleted]
• tejtm 3y ago
  It is always "horses for courses" and there may be times when the five concurrent cores with the shell pipeline will beat the single core awk script.
• deleted 3y ago
  [deleted]