5 ms·
Map Reduce implemented over HTTP
- icey 18y agoIt's a shame there are no comments to this yet. If you're relatively new to programming (say, a college student), this is a really cool chunk of code to copy into notepad and figure out. It really shows you how a tiny amount of code can do some pretty cool stuff.
- jrockway 18y agoM/R is the interesting part. Making it work over HTTP is just a few lines of Perl.
- deleted 18y ago[deleted]
- jey 18y agoAnd even M/R is just a framework for easily parallelizing "embarrassingly parallel" problems and incrementally aggregating the results (instead of in one step). A cool idea for sure, but not ground breaking -- these ideas are as old as parallel computing itself.
- bdotdub 18y agoNot a groundbreaking idea, but definitely takes the pain out of creating your own framework :)
- tlrobinson 18y agoPerhaps a simple version could be implemented in "a few lines of Perl" but the super redundant distributed version like Google's would be a bit more.
- nikils 18y agocan you show how it can be done in perl ?
- bdotdub 18y agoThe paper is a good read too: http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html
- mojuba 18y agoI'm glad to see this is not a SOAP opera and has no XML b.s. of any kind in it. In fact you can see one YAML file in the sources. Jeff Atwood knew something probably... http://www.codinghorror.com/blog/archives/001114.html http://www.codinghorror.com/blog/archives/001114.html
- omouse 18y agoDoesn't CouchDB already do this in some way?
- ntoshev 18y agoBuilt for Google App Engine? I think it will kill your request if handling it takes more than 8 seconds. Not useful yet.
- ropiku 18y agoThen just make a lot of small requests that take less then 8 seconds. Maybe Google will allow background jobs.
- st3fan 18y agoI've been playing with GridGain lately. It's Java, but their API is really sweet. You basically implement a simple interface that defines the map/reduce operations and then your code will be copied to the cluster nodes through a peer-to-peer classloader. In a recent test I validated a million image urls in less than a minute. In a small EC2 cluster running GridGain. It's certainly worth looking into if you are interested in that kind of stuff. http://www.gridgain.org http://www.gridgain.org