The real problem is that a large majority of sites do permissions by bot, whitelisting only google and possibly bing while excluding all others. Further if you

The real problem is that a large majority of sites do permissions by bot, whitelisting only google and possibly bing while excluding all others. Further if you actually try to crawl a website you'll find yourself blackholed by a lot of websites pretty quickly.