15 ms·

In my opinion, yes, Google will keep crawling the web and indexing in a similar (but continually optimized) way for the foreseeable future. Here's why - let's

In my opinion, yes, Google will keep crawling the web and indexing in a similar (but continually optimized) way for the foreseeable future. Here's why - let's say you work at Google and come up with an entirely new way to do search. Here are your options: 1 - Go to Larry/Sergi with the proposal to completely rip and replace millions/billions of dollars of infrastructure and knowledge in favor of a completely unproven new idea. The cost and risk are so huge they'd never go for it. 2 - Go to an outside investor with the proposal to build it from scratch. If it's truly an amazing, innovative new approach, they might fund it and you'd be able to build it from scratch without the political nightmare of trying to rewrite the core product of a multi-billion dollar public company. This is exactly what both the Cuil and FriendFeed guys did (and yes, FriendFeed is very, very, very much a Google competitor). One thing I always think about is in 100 years (or 1000 years), will we still be opening a web browser, typing a few words into the same plain Google text box, and hitting the "Search" button to get a page with 20 blue links and a bunch of ads? I doubt it. So there's definitely a better way to do search, we just haven't discovered it. My personal opinion is that some combination of social search (i.e. FriendFeed) plus human-powered search (i.e. Wikia/Mahalo) plus semantic search (PowerSet) will be involved in the next evolution. Of course, if I knew exactly what that looked like, I'd be on a beach right now instead of hanging out here on Hacker News. :)

tom234 18y ago
Good reply. Thanks for replying. I personally think it ultimatily boils down to content that the search engine has. That is the index in this case for Google. I think the semantics and the semantic web will make a huge difference. How I look at it is that because Google's result for "c# string replace" http://www.google.com/search?hl=en&q=c%23+string+replace http://www.google.com/search?hl=en&q=c%23+string+replace is much better than Mahalo's http://www.mahalo.com/C#_string_replace http://www.mahalo.com/C#_string_replace, Google is good. I was thinking about how the search will be in the next 2-3 years. The main problem I see with is the discovery of new content in search engine, and I think Google basically brute forced the whole thing by trying to index everything and hope that the results are there and which works alright for the query above. It basically boils down to 100 thousand crawlers and huge index. If social search engine cannot discover this page http://msdn.microsoft.com/en-us/library/fk49wtc1.aspx http://msdn.microsoft.com/en-us/library/fk49wtc1.aspx for this query "c# string replace" for instance, it is game over. Google is alive because of these kinds of results. My main concern is how to discover new content and without the burden of updating the index and bruce forcing the whole thing by indexing every word on a webpage. I know Google indexes pieces of webpages but still it is ton of words to index. I just see huge problem with creating huge index like Google and maintaining that index which is also a lot of work. Also I don't have the resources (money) to create a huge index, which is another main reason. Good discussion by the way.
- kleneway 18y ago
  Yeah, I had some thoughts a while back on one way this could work using social search: http://tchblg.wordpress.com/2008/05/31/why-friendfeed-deserves-a-one-billion-dollar-valuation/ http://tchblg.wordpress.com/2008/05/31/why-friendfeed-deserv... Basically, you'd get results back that are from conversations posted by your friends, side-by-side with traditional results from a search engine (using something like Yahoo's BOSS). Then, if you still didn't get the answer you are looking for, you can broadcast the original query out to your online contact list to see if anyone within your social network knows the answer. You wouldn't get instant results, but it would likely be a very good, trusted result. This probably works better for things like opinions (what's the best Chinese restaurant in Seattle?) vs. a fact-based search like "C# string replace". I would seriously avoid trying to take on Google or MS or Yahoo by trying to out-index them. Cuil has millions in the bank and some of the world's foremost experts in search at the helm, and things aren't looking all that bright so far. There are lots of problems to solve in the world and lots of approaches on how to solve them - solving search by out-indexing Google should probably be pretty darn low on your "problems to solve today" list.
  - tom234 18y ago
    I will definitely check out your links. Out indexing Google or Microsoft live search is definitely out of the question for me just because of the amount of data among others. Social search with like Yahoo BOSS might work. I will look into fact-based search using social search and conventional search or some kind of combo.

tom234 18y ago

Good reply. Thanks for replying. I personally think it ultimatily boils down to content that the search engine has. That is the index in this case for Google. I think the semantics and the semantic web will make a huge difference. How I look at it is that because Google's result for "c# string replace" http://www.google.com/search?hl=en&q=c%23+string+replace http://www.google.com/search?hl=en&q=c%23+string+replace is much better than Mahalo's http://www.mahalo.com/C#_string_replace http://www.mahalo.com/C#_string_replace, Google is good. I was thinking about how the search will be in the next 2-3 years. The main problem I see with is the discovery of new content in search engine, and I think Google basically brute forced the whole thing by trying to index everything and hope that the results are there and which works alright for the query above. It basically boils down to 100 thousand crawlers and huge index. If social search engine cannot discover this page http://msdn.microsoft.com/en-us/library/fk49wtc1.aspx http://msdn.microsoft.com/en-us/library/fk49wtc1.aspx for this query "c# string replace" for instance, it is game over. Google is alive because of these kinds of results. My main concern is how to discover new content and without the burden of updating the index and bruce forcing the whole thing by indexing every word on a webpage. I know Google indexes pieces of webpages but still it is ton of words to index. I just see huge problem with creating huge index like Google and maintaining that index which is also a lot of work. Also I don't have the resources (money) to create a huge index, which is another main reason. Good discussion by the way.

kleneway 18y ago
Yeah, I had some thoughts a while back on one way this could work using social search: http://tchblg.wordpress.com/2008/05/31/why-friendfeed-deserves-a-one-billion-dollar-valuation/ http://tchblg.wordpress.com/2008/05/31/why-friendfeed-deserv... Basically, you'd get results back that are from conversations posted by your friends, side-by-side with traditional results from a search engine (using something like Yahoo's BOSS). Then, if you still didn't get the answer you are looking for, you can broadcast the original query out to your online contact list to see if anyone within your social network knows the answer. You wouldn't get instant results, but it would likely be a very good, trusted result. This probably works better for things like opinions (what's the best Chinese restaurant in Seattle?) vs. a fact-based search like "C# string replace". I would seriously avoid trying to take on Google or MS or Yahoo by trying to out-index them. Cuil has millions in the bank and some of the world's foremost experts in search at the helm, and things aren't looking all that bright so far. There are lots of problems to solve in the world and lots of approaches on how to solve them - solving search by out-indexing Google should probably be pretty darn low on your "problems to solve today" list.
- tom234 18y ago
  I will definitely check out your links. Out indexing Google or Microsoft live search is definitely out of the question for me just because of the amount of data among others. Social search with like Yahoo BOSS might work. I will look into fact-based search using social search and conventional search or some kind of combo.