4 ms·

> It’s far easier imo to tweak my command line pipelines than to modify program flow. Why? Creating a throwaway file in the IDE and attaching the debugger to i

by Too 7y ago

> It’s far easier imo to tweak my command line pipelines than to modify program flow. Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.

mistahenry 7y ago
A big factor is the speed with which I’m getting results (I’m generally processing 10gb compressed log files). I don’t want to wait around for results or spend time finding the bottleneck is that my string split is creating new strings. Secondly, these tools are much more powerful than programming languages in their domain. Through such a terse api, they can accomplish so much because the command line interface is encapsulating a well written, thoroughly tested complex program. There is no boilerplate in awk and all of the assumptions make sense for the line based text processing it’s built for. A grep|awk|sort|uniq -c pipeline is the workhorse of my production support via structured log files. The other day I needed to read images out of a database, resize them, and insert them back into the dB. It was trivially easy to ‘psql | xxd | convert | xxd | psql’ to extract, convert to hex, use image magick, convert back to hex, and store the new image. The solution flowed out of me within 1 minute. I guarantee you that doing that in a higher level language would require looking up the syntax for how to connect to the dB, how to convert a byte stream to hex, how to manage sub processes when calling to convert, or which library I should use for resizing images to avoid calling convert in the first place. On top of that, we’d have to perform the various necessary incantations, songs and dances, and boilerplate to like creating a dB connection, handling subprocess timeouts, etc. On top of all of this, there is the parallelism I get for free. Then there’s the intelligent use of writing to disk as opposed to running out of RAM when dealing with giant files (sort for example). You can of course use programming languages to do everything I’ve described. IMO the conceptual overhead is inherently higher b/c I’m left to think about many more factors.

by Too 7y ago

mistahenry 7y ago

A big factor is the speed with which I’m getting results (I’m generally processing 10gb compressed log files). I don’t want to wait around for results or spend time finding the bottleneck is that my string split is creating new strings. Secondly, these tools are much more powerful than programming languages in their domain. Through such a terse api, they can accomplish so much because the command line interface is encapsulating a well written, thoroughly tested complex program. There is no boilerplate in awk and all of the assumptions make sense for the line based text processing it’s built for. A grep|awk|sort|uniq -c pipeline is the workhorse of my production support via structured log files. The other day I needed to read images out of a database, resize them, and insert them back into the dB. It was trivially easy to ‘psql | xxd | convert | xxd | psql’ to extract, convert to hex, use image magick, convert back to hex, and store the new image. The solution flowed out of me within 1 minute. I guarantee you that doing that in a higher level language would require looking up the syntax for how to connect to the dB, how to convert a byte stream to hex, how to manage sub processes when calling to convert, or which library I should use for resizing images to avoid calling convert in the first place. On top of that, we’d have to perform the various necessary incantations, songs and dances, and boilerplate to like creating a dB connection, handling subprocess timeouts, etc. On top of all of this, there is the parallelism I get for free. Then there’s the intelligent use of writing to disk as opposed to running out of RAM when dealing with giant files (sort for example). You can of course use programming languages to do everything I’ve described. IMO the conceptual overhead is inherently higher b/c I’m left to think about many more factors.