3 ms·
Other technologies to watch: 1. GraphLab2. Unlike Pregel's Bulk Synchronous Parallel Model, GraphLab2 it allows non-synchronous updates, which is more efficien
by yaroslavvb 14y ago
Other technologies to watch:
1. GraphLab2. Unlike Pregel's Bulk Synchronous Parallel Model, GraphLab2 it allows non-synchronous updates, which is more efficient for approximate quantities. For instance, for AltaVista's web graph, most nodes only need to be updated couple of times, while some nodes need more than 60 updates.
2. Flume: it's an abstraction on top of MapReduce, you program as if your data is contained in Java-like containers, and it turns your program into series of regular MapReduces
3. ScalOps (http://cs.markusweimer.com/pub/2012-DataEng.pdf http://cs.markusweimer.com/pub/2012-DataEng.pdf): that's a higher level abstraction prototyped in Yahoo Research, might get resurrected in Microsoft.
4. AllReduce
- AaronBBrown 14y agoFlume is only related to MapReduce in that it is able to write to HDFS. All Flume is is a transport mechanism to send log-like data from one place to another. It can be made fancy by adding decorators to manipulate the data along the way, but at its core, it just moves bytes around. Unfortunately, at this stage, it's highly unreliable with its fault tolerant design causing more problems than it solves. Hopefully FlumeNG will improve upon this. I speak as someone who runs Flume in production and has to deal with it constantly failing on me, usually silently. It's a great technology, but it just isn't there yet.
- yaroslavvb 14y agoOops, looks-like Flume is Google-only name, open-source implementation is called Crunch -- https://issues.apache.org/jira/browse/MAPREDUCE-1849 https://issues.apache.org/jira/browse/MAPREDUCE-1849