5 ms·

Command line tools are very terse, heavily optimized (ie fast), interactive, and easily chainable. I would never want to write a python program for quick analys

Command line tools are very terse, heavily optimized (ie fast), interactive, and easily chainable. I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. The other day my boss asked me to find out if there was any second where we processed more than 100 transactions a second on our server in the last 90 days. I needed to go through 100s of GB of data which mawk managed in no time and in a relatively simple one liner. I imagine my idiomatic python script would still be running Often times, analysis is an evolving thing. As you uncover one answer, new questions arise. It’s far easier imo to tweak my command line pipelines than to modify program flow. At the end of the day, general purpose programming languages are jacks of all trades and masters of none. They’re inherently held back by their flexibility when compared to DSLs. Text processing is so easy in awk b/c it’s a text processing domain specific language. If you really think that we should use on language for everything, you’re ignoring the value that specialization can provide you, whether in speed, power, or ease of use

umvi 7y ago
> I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. Yeah but I would wager very few people actually know awk. I would also wager 90+% of awk programs are pasted directly from StackOverflow. That's why I like the idea of pawk[1] - it's like awk, except with python syntax instead of some esoteric syntax that nobody knows. [1] https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk
- zwkrt 7y ago
  I guarantee that more people know awk syntax than pawk!
  - deleted 7y ago
    [deleted]
Too 7y ago
> It’s far easier imo to tweak my command line pipelines than to modify program flow. Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.
- mistahenry 7y ago
  A big factor is the speed with which I’m getting results (I’m generally processing 10gb compressed log files). I don’t want to wait around for results or spend time finding the bottleneck is that my string split is creating new strings. Secondly, these tools are much more powerful than programming languages in their domain. Through such a terse api, they can accomplish so much because the command line interface is encapsulating a well written, thoroughly tested complex program. There is no boilerplate in awk and all of the assumptions make sense for the line based text processing it’s built for. A grep|awk|sort|uniq -c pipeline is the workhorse of my production support via structured log files. The other day I needed to read images out of a database, resize them, and insert them back into the dB. It was trivially easy to ‘psql | xxd | convert | xxd | psql’ to extract, convert to hex, use image magick, convert back to hex, and store the new image. The solution flowed out of me within 1 minute. I guarantee you that doing that in a higher level language would require looking up the syntax for how to connect to the dB, how to convert a byte stream to hex, how to manage sub processes when calling to convert, or which library I should use for resizing images to avoid calling convert in the first place. On top of that, we’d have to perform the various necessary incantations, songs and dances, and boilerplate to like creating a dB connection, handling subprocess timeouts, etc. On top of all of this, there is the parallelism I get for free. Then there’s the intelligent use of writing to disk as opposed to running out of RAM when dealing with giant files (sort for example). You can of course use programming languages to do everything I’ve described. IMO the conceptual overhead is inherently higher b/c I’m left to think about many more factors.

umvi 7y ago

> I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. Yeah but I would wager very few people actually know awk. I would also wager 90+% of awk programs are pasted directly from StackOverflow. That's why I like the idea of pawk[1] - it's like awk, except with python syntax instead of some esoteric syntax that nobody knows. [1] https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk

zwkrt 7y ago
I guarantee that more people know awk syntax than pawk!
- deleted 7y ago
  [deleted]

Too 7y ago

> It’s far easier imo to tweak my command line pipelines than to modify program flow. Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.

mistahenry 7y ago
A big factor is the speed with which I’m getting results (I’m generally processing 10gb compressed log files). I don’t want to wait around for results or spend time finding the bottleneck is that my string split is creating new strings. Secondly, these tools are much more powerful than programming languages in their domain. Through such a terse api, they can accomplish so much because the command line interface is encapsulating a well written, thoroughly tested complex program. There is no boilerplate in awk and all of the assumptions make sense for the line based text processing it’s built for. A grep|awk|sort|uniq -c pipeline is the workhorse of my production support via structured log files. The other day I needed to read images out of a database, resize them, and insert them back into the dB. It was trivially easy to ‘psql | xxd | convert | xxd | psql’ to extract, convert to hex, use image magick, convert back to hex, and store the new image. The solution flowed out of me within 1 minute. I guarantee you that doing that in a higher level language would require looking up the syntax for how to connect to the dB, how to convert a byte stream to hex, how to manage sub processes when calling to convert, or which library I should use for resizing images to avoid calling convert in the first place. On top of that, we’d have to perform the various necessary incantations, songs and dances, and boilerplate to like creating a dB connection, handling subprocess timeouts, etc. On top of all of this, there is the parallelism I get for free. Then there’s the intelligent use of writing to disk as opposed to running out of RAM when dealing with giant files (sort for example). You can of course use programming languages to do everything I’ve described. IMO the conceptual overhead is inherently higher b/c I’m left to think about many more factors.