3 ms·
Surprised to read LinkedIn's "People You May Know" MapReduce pipeline takes 82 jobs. From what I've read Google's index runs in around 20 MapReduce jobs.
by KrisJordan 16y ago
Surprised to read LinkedIn's "People You May Know" MapReduce pipeline takes 82 jobs.
From what I've read Google's index runs in around 20 MapReduce jobs.
- rjurney 16y agoPagerank is a like a markov chain - you iterate the same dataset over and over until you're happy with the result. If 20 is good enough, its good enough. Good explanation here: http://www.iterativemapreduce.org/samples.html#Pagerank http://www.iterativemapreduce.org/samples.html#Pagerank Whereas if you include signals from multiple sources, the joins are each one MR job, never mind the calculations.