3 ms·
Check out Spark, a framework for writing parallel machine learning jobs (among lots of other things) on top of hadoop. http://www.cs.berkeley.edu/~matei/spark/
by joshhart 15y ago
Check out Spark, a framework for writing parallel machine learning jobs (among lots of other things) on top of hadoop. http://www.cs.berkeley.edu/~matei/spark/ http://www.cs.berkeley.edu/~matei/spark/
They need to keep data in memory and shuffle it around really fast. To do so, they use a bittorrent-variant algorithm. I think this is the paper that describes it - http://www.google.com/url?sa=t&source=web&cd=7&sqi=2&ved=0CEgQFjAG&url=http%3A%2F%2Fwww.cs.berkeley.edu%2F~agearh%2Fcs267.sp10%2Ffiles%2Fmosharaf-spark-bc-report-spring10.pdf&rct=j&q=spark%20mosharaf&ei=GeoPToGgE4-esQPdwtSoDg&usg=AFQjCNG6sVV12DfLM3iNBhLs6F66toBCwg&sig2=n9opoR0rGbQR0L0KShgZtw&cad=rja http://www.google.com/url?sa=t&source=web&cd=7&s...
- gruseom 15y agoInteresting. Thank you. They revise Bittorrent for use inside data centers, where block sizes can be much larger and there aren't any malicious nodes. They prefer Bittorrent to alternatives because, through its broadcast overhead is higher, it scales better with number of nodes. One thing I didn't glean from the paper is whether the advantages of BT are unique to their framework or whether they would apply just as well to MapReduce.