3 ms·

For some interesting thoughts on mapreduce frameworks check out this interview with Jonathan Ellis on the Cassandra project which is facebooks version of a dist

For some interesting thoughts on mapreduce frameworks check out this interview with Jonathan Ellis on the Cassandra project which is facebooks version of a distributed data store. http://techzinglive.com/?p=75 http://techzinglive.com/?p=75 From what I remember: Google built it's parallelizable stuff around it's already excellent distributed filesystem. In hadoop they are essentially cloning the filesystem and the rest of the stuff from scratch, which is a pretty hard thing to do, and a more leaner approach would be better.

sho 17y ago
Thanks, will check it out. Yeah, I appreciate what Hadoop is trying to do. I am just not sure they are going about it in the right way. I don't like their technology choices and do not get a good feeling from the few times I've looked at their issue tracking system (bureaucracy). I mean look, MogileFS is also a distributed FS. It works. It's very good, actually. It's in production in a lot of places. MogileFS is 268KB of Perl. Gearman is basically a distributed jobs broker, just like Hadoop. There are some complexities but it is not rocket science. They basically do the same thing. Gearman is 560KB of C, the original was 12KB of Perl. One of these solutions is literally 100 times larger than the other. I don't know about you, but for me .. that is not a good sign. I do not see what Hadoop can do that the Danga tools cannot. Hell, I have a hard time seeing what Hadoop can do that a couple hundred lines of Python can't. Sure, might not be enough to operate like Google does, but 90% of the time, quick and dirty gets it done ..
- prodigal_erik 17y ago
  As far as I can tell, right now Gearman is only a load balancer for small requests using a nonstandard RPC protocol. It's easy to be smaller when you don't yet have most of hadoop's major features: * use the whole cluster for large jobs, consuming all the data right where it's stored and combining output from every machine you own (what's the point of remote work on just one box?) * retry partial failures on different replicas of the same data (without losing successful work) * ship intermediate values between workers (without this you don't even have map/reduce) * roll new code and data beforehand (and clean it up after) * avoid job starvation

sho 17y ago

Thanks, will check it out. Yeah, I appreciate what Hadoop is trying to do. I am just not sure they are going about it in the right way. I don't like their technology choices and do not get a good feeling from the few times I've looked at their issue tracking system (bureaucracy). I mean look, MogileFS is also a distributed FS. It works. It's very good, actually. It's in production in a lot of places. MogileFS is 268KB of Perl. Gearman is basically a distributed jobs broker, just like Hadoop. There are some complexities but it is not rocket science. They basically do the same thing. Gearman is 560KB of C, the original was 12KB of Perl. One of these solutions is literally 100 times larger than the other. I don't know about you, but for me .. that is not a good sign. I do not see what Hadoop can do that the Danga tools cannot. Hell, I have a hard time seeing what Hadoop can do that a couple hundred lines of Python can't. Sure, might not be enough to operate like Google does, but 90% of the time, quick and dirty gets it done ..

prodigal_erik 17y ago
As far as I can tell, right now Gearman is only a load balancer for small requests using a nonstandard RPC protocol. It's easy to be smaller when you don't yet have most of hadoop's major features: * use the whole cluster for large jobs, consuming all the data right where it's stored and combining output from every machine you own (what's the point of remote work on just one box?) * retry partial failures on different replicas of the same data (without losing successful work) * ship intermediate values between workers (without this you don't even have map/reduce) * roll new code and data beforehand (and clean it up after) * avoid job starvation