4 ms·

I believe that when it comes to search, the aim should not be about competing with Google in it's current form. That is impossible. The aim should be to change

by majani 9y ago

I believe that when it comes to search, the aim should not be about competing with Google in it's current form. That is impossible. The aim should be to change the architecture of search such that it becomes a commodity that anyone can plug into. Like what open source CMSes did to the AOL/Time Warner giant. Then the owners of this commoditized search can charge for their premium API on a usage basis, transforming search into an ad-free service found anywhere

ChuckMcM 9y ago
I think that is an excellent vision but to understand the challenge of achieving it consider how a search engine has to operate. Typically it 'shards' the index of all available documents into chunks, where the necessary descriptive bits for a chunk can fit in the memory of a dense memory server. Then each server returns its "best" result which are collected, and from there the top 10 are selected. Search needs to be fast. When you click search you want the page to then pop up with your 10 blue links. The physics of light suggest that your index files all need to be close together, both physically and with high bandwidth to support a couple of round trips for distributing the query and returning the results. While I learned how search worked while I worked at Google, it was a brilliant engineer named Keith Peterson at Blekko (now at Google :-) who taught me some of the more subtle issues involved in balancing indexes, shards, and query processing in order to maximize utilization. That requirement of closeness and high bandwidth, makes federation hard. Even forwarding a search to Bing (via the Yahoo APIs) to check for additional "long tail" results really challenges your ability to deliver fast results. Unlike CMSes where they could run independently in parallel, search is a 'talk to everyone all the time' sort of problem, you don't know apriori which indices may have relevant documents (which is why it is called "search" ;-))
- sidcool 9y ago
  Good write up. Thanks

by majani 9y ago

ChuckMcM 9y ago

I think that is an excellent vision but to understand the challenge of achieving it consider how a search engine has to operate. Typically it 'shards' the index of all available documents into chunks, where the necessary descriptive bits for a chunk can fit in the memory of a dense memory server. Then each server returns its "best" result which are collected, and from there the top 10 are selected. Search needs to be fast. When you click search you want the page to then pop up with your 10 blue links. The physics of light suggest that your index files all need to be close together, both physically and with high bandwidth to support a couple of round trips for distributing the query and returning the results. While I learned how search worked while I worked at Google, it was a brilliant engineer named Keith Peterson at Blekko (now at Google :-) who taught me some of the more subtle issues involved in balancing indexes, shards, and query processing in order to maximize utilization. That requirement of closeness and high bandwidth, makes federation hard. Even forwarding a search to Bing (via the Yahoo APIs) to check for additional "long tail" results really challenges your ability to deliver fast results. Unlike CMSes where they could run independently in parallel, search is a 'talk to everyone all the time' sort of problem, you don't know apriori which indices may have relevant documents (which is why it is called "search" ;-))

sidcool 9y ago
Good write up. Thanks