4 ms·
Two kinds of efficiency: (i) computing efficiency, (ii) developer efficiency If you are doing everything on one machine you can keep everything in RAM, if y
by PaulHoule 2y ago
Two kinds of efficiency: (i) computing efficiency, (ii) developer efficiency
If you are doing everything on one machine you can keep everything in RAM, if you have more than one machine you're going to have to serialize all the data that passes between the machines and send it over some communication link which is never going to be as fast as reading it out of RAM.
The simple case is that there is not a lot of coordination needed between the machines but the coordination overhead can easily be worse than O(N) where N is the number of machines.
Developer efficiency may be more important than computing efficiency in many cases. Hadoop, like Bitcoin, blindsided the distributed systems community because it was so much worse than MPI [1] but MPI applications were being developed by PhD students and postdocs and with Hadoop you could replace your
grep ... | awk ... | sort ... | uniq | grep ... | awk ...
pipeline with a simple Pig script with no talent at all and have it run pretty fast by most people's standards on a huge dataset... Even though the MPI geniuses could get it to run 1000x faster.
[1] https://en.wikipedia.org/wiki/Message_Passing_Interface https://en.wikipedia.org/wiki/Message_Passing_Interface