3 ms·
I have long been an advocate of bashreduce. I even worked for a company that ran a cluster of storage machines that each had tiny amounts of ram and contained
by dangerlibrary 11y ago
I have long been an advocate of bashreduce.
I even worked for a company that ran a cluster of storage machines that each had tiny amounts of ram and contained enormous (many 100GB+) csv text files - this was right around the time hadoop was starting to be a thing, but the cluster was air-gapped for security and updates to external software were incredibly tedious. We'd write small 'cut | grep | sort | awk' sh commands and run them using a tiny perl script called 'prsh' - parallel rsh - to aggregate the results.