This is a problem of their own invention. Nobody said you have to index the entire web. The web would probably be a lot healthier if we had several small sear

This is a problem of their own invention. Nobody said you have to index the entire web. The web would probably be a lot healthier if we had several small search engines that focused on niches rather than 5 failed search engines that tried to index everything that was ever written and then ended up paying Bing.