A lot of the internet is behind semi-walled gardens and GoogleBot and BingBot have preferential access. All the large social sites, Cloudflare, and other CDNs

A lot of the internet is behind semi-walled gardens and GoogleBot and BingBot have preferential access. All the large social sites, Cloudflare, and other CDNs will just stop serving unknown bots after awhile, which is why there are spidering/scraping services that cycle thru IP addresses. See: https://news.ycombinator.com/item?id=29417061 https://news.ycombinator.com/item?id=29417061