4 ms·

I'm think you are oversimplifying the problem. Web search is about the long tail you need breadth in your index. It will cost quite a bit more than a couple o

by ashot 17y ago

I'm think you are oversimplifying the problem. Web search is about the long tail you need breadth in your index. It will cost quite a bit more than a couple of bucks to do (storage is not that cheap, nor is cpu), and you'd have to do it repeatedly (and quickly as I mentioned above). And then of course you'll need enough computing power to perform the actual searches and run hadoop on your web index. There are only a handful of serious players in this space (check your site logs) for a reason

inerte 17y ago
In his (her?) defense, he didn't say "search", he said "index". There's a difference! Building the index is easy(ier nowadys). Making it work for the users is the hard part :)
- ashot 17y ago
  ha, well op said build a "search index".. its not much of a search index if you can't search it :)
  - inerte 17y ago
    Clarifying what I said, I meant "building a good search engine" is hard, while using an index to make something faster to find is easy(ier). I agree (with the previous poster) that indexing parts of the web and making it searchable is pretty much a solved technical problem. There are thousands of people who could pull this off. Heck, I probably could if you give me the hardware. Now doing it cheap, scalable, etc... it's hard. And making it a good experience for the user is pretty much a war on its own. Fighting spam must be pretty hard...
    - litewulf 17y ago
      Indexing, in the trivial sense is easy. PageRank (which is old) is already a step beyond (How does one deal with spam? How does one handle redirects, or splash pages that forward you somewhere). Semantic analysis is a step further (what does apple mean?). And then theres everything else (What should happen when I search for "camera", am I shopping, or do I want to find out more about how a camera works?) In so far as building a search engine involves buying computers, yes, its easy, but then theres everything else. Any problem is trivializable in this sense. Amazon is just a storefront (except for the recommendations, the user experience, the search, and I'm sure all sorts of domain knowledge that I don't have at a glance)

by ashot 17y ago

inerte 17y ago

In his (her?) defense, he didn't say "search", he said "index". There's a difference! Building the index is easy(ier nowadys). Making it work for the users is the hard part :)

ashot 17y ago
ha, well op said build a "search index".. its not much of a search index if you can't search it :)
- inerte 17y ago
  Clarifying what I said, I meant "building a good search engine" is hard, while using an index to make something faster to find is easy(ier). I agree (with the previous poster) that indexing parts of the web and making it searchable is pretty much a solved technical problem. There are thousands of people who could pull this off. Heck, I probably could if you give me the hardware. Now doing it cheap, scalable, etc... it's hard. And making it a good experience for the user is pretty much a war on its own. Fighting spam must be pretty hard...
  - litewulf 17y ago
    Indexing, in the trivial sense is easy. PageRank (which is old) is already a step beyond (How does one deal with spam? How does one handle redirects, or splash pages that forward you somewhere). Semantic analysis is a step further (what does apple mean?). And then theres everything else (What should happen when I search for "camera", am I shopping, or do I want to find out more about how a camera works?) In so far as building a search engine involves buying computers, yes, its easy, but then theres everything else. Any problem is trivializable in this sense. Amazon is just a storefront (except for the recommendations, the user experience, the search, and I'm sure all sorts of domain knowledge that I don't have at a glance)