2 ms·
You can make a case that Web search is a natural monopoly. Running the crawlers and indexers and servers is freaking expensive, requiring monster capex and oper
by nl 6y ago
You can make a case that Web search is a natural monopoly. Running the crawlers and indexers and servers is freaking expensive, requiring monster capex and operational expenditure. It’s not obvious to me that the world needs more than one.
I wonder how true this is these days?
I understand that the "long tail" of web queries is very important, but if you were to skip that for version 1, and concentrate on crawling the top 1M English language sites how much would it really cost?
- xiphias2 6y agoEnglish is the language that needs the most resources, and where Google put the most effort. Actually any other language would be a smaller crawling/indexing investment, if really that was the hardest part. I don't know the current situation, but Yandex was better in Russia than Google a few years ago.
- vharuck 6y agoI'm not familiar with the mechanics of web crawlers, so hopefully somebody can shed some light. Would there be value in a search engines for specific topics? For example, movie reviews and blogs. Assuming you had a way to determine if a page is a review or blog, would it make indexing easier to ignore the rest? Could a tailored search engine be tweaked to give better results? I kind of think it would be useful, because some websites I use have really great search tools (e.g. ones with tags). But those are only for the website itself, so there's no crawling.