3 ms·
That's a pretty sad result to get from 128 cores. I've seen amateur Beowulf clusters get better results.
by nickpsecurity 11y ago
That's a pretty sad result to get from 128 cores. I've seen amateur Beowulf clusters get better results.
- tacone 11y agoAlso GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
- tacone 11y agoExcuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
- nickpsecurity 11y agoOh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.
- rakoo 11y agoLet's also not forget GNU parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/)
- dagw 11y agoGNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
- nickpsecurity 11y agoThanks for the tip!