4 ms·
Each of the following is in the 1000 hours saved club (in no particular order): jq, perl, grep, gnu textutils, gnu parallel, bash, xargs, gkrellm, nload. Make
by hedora 4y ago
Each of the following is in the 1000 hours saved club (in no particular order):
jq, perl, grep, gnu textutils, gnu parallel, bash, xargs, gkrellm, nload.
Make deserves a special shout out. If you think it's only useful for building software, you've completely missed the point (similarly, if you think some language-specific tool is superior, you're doing things completely wrong):
Anyway, this document saved me at least 10,000 hours:
https://www.gnu.org/software/make/manual/html_node/index.html https://www.gnu.org/software/make/manual/html_node/index.htm...
Other peoples' python has the distinction of being the only software in my -1000 hours saved category. (And if you consider maintaining code written in the above "write only" languages, you'll realize how big of a feat that is!)
- jodrellblank 4y ago> "If you think it's only useful for building software, you've completely missed the point" I clicked on your link with 16 pages of just contents links to find what the point is, and found "the make utility automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them". What's the point you see?
- hedora 4y agoIt automatically parallelizes any large scale data processing pipeline (assuming that your data set is already broken into files). People use it for things like documentation generation. I've used it to process the results of web scrapers, large scale data cleaning tasks, etc, etc. It takes about an afternoon to set up NFS + SSH primitives that automatically distribute the computation across clusters of machines. Since it is restartable, it automatically tolerates hardware faults (up to dozens of machines, in practice). Basically, you get Map Reduce, but for arbitrary data processing DAGs, and it supports any language that works well on Unix-style operating systems.
- deleted 4y ago[deleted]
- rebelos 4y agoOkay but clearly none of this is "the point" of make, so your original comment is still baffling.
- hedora 4y agoThe point of make is that it's not only useful for building software. Take a look at the built-in implicit rules: https://www.gnu.org/software/make/manual/html_node/Catalogue-of-Rules.html#Catalogue-of-Rules https://www.gnu.org/software/make/manual/html_node/Catalogue... There are two things to note: (1) you can turn them off so they don't get in the way of other use cases, (2) they include non-build stuff like version control, linting, and document generation. There's also a whole chapter on ar file maintenance. This is mostly useful for linking these days, but ar is the precursor to tar, so this used to also be useful for running filesystem backups: https://www.gnu.org/software/make/manual/html_node/Archives.html#Archives https://www.gnu.org/software/make/manual/html_node/Archives....
- vehementi 4y agoYour point is that make has uses beyond that. That is not "the point of make"
- lelandbatey 4y agoNo, that absolutely is the point of `make`. You quoted the "overview of make" page[0] which does say "The make utility automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them." However, that same page, about 5 lines down, expands to say: > Indeed, 'make' is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change. Which is exactly what the original poster is claiming.
- jamespo 4y agoThat's the beauty of open ended UNIX tools
- capitalsigma 4y agoThe idea of working with a distributed data pipeline implemented via makefiles is nightmare fuel
- 2OEH8eoCRo0 4y ago$ make install
- rgavuliak 4y agoI think OP meant data analysis/one-offs as not building software that is technically true I guess
- athorax 4y agojq + parallel is a godsend. Take some command output, run another json-producing command in parallel on each line, and pipe it back into jq -s
- raffraffraff 4y agoCan you elaborate with a fake example?
- QuadrupleA 4y ago1000 hours each? Really? So literally half a year of full-time 40 hour weeks saved by each of these on your list, vs the next best alternative?
- xcdzvyn 4y agoYes. That statement was entirely literal; there is on reason to believe the statement "1000 hours" is anything but a precisely measured saving.
- fn1 4y ago> jq Add xmlstarlet for anything that has to do with xml.