4 ms·

its not a technical challenge to build an index of the entire web? certainly not easy, and not any easier than it was before as the web is much larger there i

by ashot 17y ago

its not a technical challenge to build an index of the entire web? certainly not easy, and not any easier than it was before as the web is much larger there is also freshness/timeliness which is the most difficult aspect and is becoming more and more important as more data is added to the web faster (one area where googlebot kills yahoo/msn)

andrewljohnson 17y ago
Yeah, I'd say it's not much of a technical challenge. Storage is cheap, indexing algorithms are well-known, and the problem space has been heavily explored. And it's misleading to talk about the entire web. Google may index billions upon billions of pages, but only a small percent actually get shown to the user, or stored in memory. And I would say that a smart kid with a few computers or a couple of bucks can build the index. Then the hard part starts. Machine learning to correct search placements, based on features like clicks and time in index. Algorithms to detect spam. Anticipation of user data needs. And all the nitty gritty of linguistic analysis to better sort your index. These types of things require new and innovative thoughts. In my mind, building a search index is just standing on the shoulders of the giants that came before. I'm not saying Google isn't amazing or that building a search index isn't a great homework problem, but the basic part of getting, storing, and serving the data has been engineered and can be engineered again at will.
- ashot 17y ago
  I'm think you are oversimplifying the problem. Web search is about the long tail you need breadth in your index. It will cost quite a bit more than a couple of bucks to do (storage is not that cheap, nor is cpu), and you'd have to do it repeatedly (and quickly as I mentioned above). And then of course you'll need enough computing power to perform the actual searches and run hadoop on your web index. There are only a handful of serious players in this space (check your site logs) for a reason
  - inerte 17y ago
    In his (her?) defense, he didn't say "search", he said "index". There's a difference! Building the index is easy(ier nowadys). Making it work for the users is the hard part :)
    - ashot 17y ago
      ha, well op said build a "search index".. its not much of a search index if you can't search it :)
      - inerte 17y ago
        Clarifying what I said, I meant "building a good search engine" is hard, while using an index to make something faster to find is easy(ier). I agree (with the previous poster) that indexing parts of the web and making it searchable is pretty much a solved technical problem. There are thousands of people who could pull this off. Heck, I probably could if you give me the hardware. Now doing it cheap, scalable, etc... it's hard. And making it a good experience for the user is pretty much a war on its own. Fighting spam must be pretty hard...
        litewulf 17y ago
        Indexing, in the trivial sense is easy. PageRank (which is old) is already a step beyond (How does one deal with spam? How does one handle redirects, or splash pages that forward you somewhere). Semantic analysis is a step further (what does apple mean?). And then theres everything else (What should happen when I search for "camera", am I shopping, or do I want to find out more about how a camera works?) In so far as building a search engine involves buying computers, yes, its easy, but then theres everything else. Any problem is trivializable in this sense. Amazon is just a storefront (except for the recommendations, the user experience, the search, and I'm sure all sorts of domain knowledge that I don't have at a glance)

by ashot 17y ago

andrewljohnson 17y ago

Yeah, I'd say it's not much of a technical challenge. Storage is cheap, indexing algorithms are well-known, and the problem space has been heavily explored. And it's misleading to talk about the entire web. Google may index billions upon billions of pages, but only a small percent actually get shown to the user, or stored in memory. And I would say that a smart kid with a few computers or a couple of bucks can build the index. Then the hard part starts. Machine learning to correct search placements, based on features like clicks and time in index. Algorithms to detect spam. Anticipation of user data needs. And all the nitty gritty of linguistic analysis to better sort your index. These types of things require new and innovative thoughts. In my mind, building a search index is just standing on the shoulders of the giants that came before. I'm not saying Google isn't amazing or that building a search index isn't a great homework problem, but the basic part of getting, storing, and serving the data has been engineered and can be engineered again at will.

ashot 17y ago
I'm think you are oversimplifying the problem. Web search is about the long tail you need breadth in your index. It will cost quite a bit more than a couple of bucks to do (storage is not that cheap, nor is cpu), and you'd have to do it repeatedly (and quickly as I mentioned above). And then of course you'll need enough computing power to perform the actual searches and run hadoop on your web index. There are only a handful of serious players in this space (check your site logs) for a reason
- inerte 17y ago
  In his (her?) defense, he didn't say "search", he said "index". There's a difference! Building the index is easy(ier nowadys). Making it work for the users is the hard part :)
  - ashot 17y ago
    ha, well op said build a "search index".. its not much of a search index if you can't search it :)
    - inerte 17y ago
      Clarifying what I said, I meant "building a good search engine" is hard, while using an index to make something faster to find is easy(ier). I agree (with the previous poster) that indexing parts of the web and making it searchable is pretty much a solved technical problem. There are thousands of people who could pull this off. Heck, I probably could if you give me the hardware. Now doing it cheap, scalable, etc... it's hard. And making it a good experience for the user is pretty much a war on its own. Fighting spam must be pretty hard...
      - litewulf 17y ago
        Indexing, in the trivial sense is easy. PageRank (which is old) is already a step beyond (How does one deal with spam? How does one handle redirects, or splash pages that forward you somewhere). Semantic analysis is a step further (what does apple mean?). And then theres everything else (What should happen when I search for "camera", am I shopping, or do I want to find out more about how a camera works?) In so far as building a search engine involves buying computers, yes, its easy, but then theres everything else. Any problem is trivializable in this sense. Amazon is just a storefront (except for the recommendations, the user experience, the search, and I'm sure all sorts of domain knowledge that I don't have at a glance)