3 ms·

I don't get it. They were using Map/Reduce as a way to build the index, which they were able to query in mere milliseconds. This article claims that in order to

I don't get it. They were using Map/Reduce as a way to build the index, which they were able to query in mere milliseconds. This article claims that in order to facilitate Google Instant, they had to ditch the Map/Reduce-oriented updating of the index. How are these mutually exclusive? If you look at quotes like these: "Goal is to update the search index continuously, within seconds of content changing, without rebuilding the entire index from scratch using MapReduce." To me, this seems as if this change has nothing to do with Google Instant. This has more to do with being able to respond instantly to new content, instead of being able to query the index quickly. It sounds like they added support for distributed stored procedures on top of BigTable, which reminds me a bit of the way MongoDB implemented Map/Reduce. But I bet that they in no way at all have dumped Map/Reduce.

fizx 16y ago
> To me, this seems as if this change has nothing to do with Google Instant. This has more to do with being able to respond instantly to new content, instead of being able to query the index quickly. Right, this was part of the caffeine update, which happened months ago. > But I bet that they in no way at all have dumped Map/Reduce. The old way of doing calculations on the web graph was giant iterations on the adjacency matrix via map reduce. In the new system, they are probably doing local walks in the instantiated graph. These local walks are simple iterations, not map-reduce.
- Anon84 16y ago
  > In the new system, they are probably doing local walks in the instantiated graph. These local walks are simple iterations, not map-reduce. Actually it using Pregel, a variation on Map/Reduce: http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html http://googleresearch.blogspot.com/2009/06/large-scale-graph... http://horicky.blogspot.com/2010/07/google-pregel-graph-processing.html http://horicky.blogspot.com/2010/07/google-pregel-graph-proc... http://portal.acm.org/citation.cfm?id=1582716.1582723 http://portal.acm.org/citation.cfm?id=1582716.1582723
mark_l_watson 16y ago
I just blogged about Caffeine. To address your questions: they have moved to a continuous update indexing scheme that uses something like Big Table so their index is now in a data. From an end-user's perspective, calculating search results using this new data store and more caching is faster because search results are displayed while I type. I can't wait for technical papers on Caffeine to be released.
- stingraycharles 16y ago
  But how is this so different from the index servers? No map/reduce was required to lookup a query in the index servers, it was only used to /build/ to index servers. I can see that this new method allows for more rapid modifications of the index servers, but I don't see how it is related to Google Instant. Furthermore, if you look up an article TFA links to: http://www.theregister.co.uk/2010/09/09/google_caffeine_explained/ http://www.theregister.co.uk/2010/09/09/google_caffeine_expl... , you see quotes like this: "Google distinguished engineer Ben Gomes told The Reg that Caffeine was not built with Instant in mind, but that it helped deal with the added load placed on the system by the "streaming" search service. "Lipkovitz makes a point of saying that MapReduce is by no means dead. There are still cases where Caffeine uses batch processing, and MapReduce is still the basis for myriad other Google services. " I can't wait for the technical papers on Caffeine either, but to me it does not seem related to Google Instant in many ways, it's just part of Google's desire to be able to respond more rapidly to changing content on the internet. Which is totally OK with me. But then people shouldn't claim they dumped Map/Reduce in order to implement Google Instant.
  - mark_l_watson 16y ago
    Good points, thanks. Without more tech detail, it does look like the Big table type datastore is being updated while being used for search. In Lucene/Nutch/etc. apps, I think it s common to build one index offline, then swap for the live index (don't need to do so, but it makes sense to sometimes do this). Google is likely getting rid of the swapping index part of the process by having a large persistent datastore that is continually indexed.
joshu 16y ago
the assumption that you can understand anything about their infrastructure from the stuff that they talk about is a big mistake.

fizx 16y ago

> To me, this seems as if this change has nothing to do with Google Instant. This has more to do with being able to respond instantly to new content, instead of being able to query the index quickly. Right, this was part of the caffeine update, which happened months ago. > But I bet that they in no way at all have dumped Map/Reduce. The old way of doing calculations on the web graph was giant iterations on the adjacency matrix via map reduce. In the new system, they are probably doing local walks in the instantiated graph. These local walks are simple iterations, not map-reduce.

Anon84 16y ago
> In the new system, they are probably doing local walks in the instantiated graph. These local walks are simple iterations, not map-reduce. Actually it using Pregel, a variation on Map/Reduce: http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html http://googleresearch.blogspot.com/2009/06/large-scale-graph... http://horicky.blogspot.com/2010/07/google-pregel-graph-processing.html http://horicky.blogspot.com/2010/07/google-pregel-graph-proc... http://portal.acm.org/citation.cfm?id=1582716.1582723 http://portal.acm.org/citation.cfm?id=1582716.1582723

mark_l_watson 16y ago

I just blogged about Caffeine. To address your questions: they have moved to a continuous update indexing scheme that uses something like Big Table so their index is now in a data. From an end-user's perspective, calculating search results using this new data store and more caching is faster because search results are displayed while I type. I can't wait for technical papers on Caffeine to be released.

stingraycharles 16y ago
But how is this so different from the index servers? No map/reduce was required to lookup a query in the index servers, it was only used to /build/ to index servers. I can see that this new method allows for more rapid modifications of the index servers, but I don't see how it is related to Google Instant. Furthermore, if you look up an article TFA links to: http://www.theregister.co.uk/2010/09/09/google_caffeine_explained/ http://www.theregister.co.uk/2010/09/09/google_caffeine_expl... , you see quotes like this: "Google distinguished engineer Ben Gomes told The Reg that Caffeine was not built with Instant in mind, but that it helped deal with the added load placed on the system by the "streaming" search service. "Lipkovitz makes a point of saying that MapReduce is by no means dead. There are still cases where Caffeine uses batch processing, and MapReduce is still the basis for myriad other Google services. " I can't wait for the technical papers on Caffeine either, but to me it does not seem related to Google Instant in many ways, it's just part of Google's desire to be able to respond more rapidly to changing content on the internet. Which is totally OK with me. But then people shouldn't claim they dumped Map/Reduce in order to implement Google Instant.
- mark_l_watson 16y ago
  Good points, thanks. Without more tech detail, it does look like the Big table type datastore is being updated while being used for search. In Lucene/Nutch/etc. apps, I think it s common to build one index offline, then swap for the live index (don't need to do so, but it makes sense to sometimes do this). Google is likely getting rid of the swapping index part of the process by having a large persistent datastore that is continually indexed.

joshu 16y ago

the assumption that you can understand anything about their infrastructure from the stuff that they talk about is a big mistake.