3 ms·
It wasn't obsolete 15 years ago. There were production mapreduces making double-digit improvements in key metrics (watch time, apps purchased) much more recent
by dekhn 3y ago
It wasn't obsolete 15 years ago. There were production mapreduces making double-digit improvements in key metrics (watch time, apps purchased) much more recently than that. The system I worked on, Sibyl, isn't well known outside of Google, but used MR as an engine to do extremely large-scale machine learning. we added a number of features and used MR in ways that would have been extremely challenging to reimplement while maintaining performance targets.
I'm not even sure the mapreduce code has been deleted from google3 yet.
To be fair, MR was definitely dated by the time I joined- 2007- and I'm surprised it lasted as long as it did. But it was really well-tuned and reliable.
Also the MR paper was never intended to stake Google's position as a data processing provider (that came far, far later). The MR, Bigtable, and GFS papers were written to attract software engineers to work on infra at google, to share some useful ideas with the world (specifically, the distributed shuffle in mapreduce, the bloom filter in bigtable, and the single-master-index-in-ram of GFS), and finally, to "show off".
- ithkuil 3y agoI remember the joke CL that deleted mapreduce from google3, even got an approval from Urs IIRC
- dieortin 3y agoAs an outsider, what is a “CL” and who is Urs?
- case 3y agohttps://en.wikipedia.org/wiki/Urs_H%C3%B6lzle https://en.wikipedia.org/wiki/Urs_H%C3%B6lzle
- ReactiveJelly 3y agoProbably Change List https://stackoverflow.com/questions/25716920/what-does-cl-mean-in-a-commit-message-what-does-it-stand-for https://stackoverflow.com/questions/25716920/what-does-cl-me...
- dmoy 3y agoCL is changelist, the perforce equivalent of e.g. a git pull request (sorta). Even though Google codebase broke perforce scaling long ago and doesn't use it anymore, the replacement still borrows a lot of perforce names and sort of API.
- calderwoodra 3y agoAre you sure you're not thinking of flume? I think flume is different from the original map-reduce.
- dekhn 3y agoYes, I'm not thinking of Flume. When I worked on sibyl we were in the middle of converting most of the data pipelines (which are now externalized in TFX) to Flume, but the core learner, along with many other prod jobs at google, still used MapReduce. But, by that time, the majority of google developers were using Flume instead of MR. Certainly even in 2013 MR was definitely being used; I launched a product at that time that ran an MR because we couldn't get similar performance out of Flume yet.