At that point Cloudflare and Akamai are bigger problem. They are basically gate-keeping everything in the internet so you can't web-crawl the web anymore. Heck,

At that point Cloudflare and Akamai are bigger problem. They are basically gate-keeping everything in the internet so you can't web-crawl the web anymore. Heck, even 1 page sites are now 'protected'.