4 ms·

Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-t

by tacone 11y ago

Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.

tacone 11y ago
Excuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
nickpsecurity 11y ago
Oh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.
rakoo 11y ago
Let's also not forget GNU parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/)
- dagw 11y ago
  GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
  - nickpsecurity 11y ago
    Thanks for the tip!