4 ms·

> or a search engine I would guess the crawling to be the easiest part.

• mdaniel 4y ago
  Maybe "easiest" in the "I can explain it to my mother" way, but as with many good things, the devil's in the details about that. That goes double now that Cloudflare have basically taken over some non-trivial portion of the Internet and they really do hate crawlers I have a theory that the big search engines have agreements in place for "side-channel" access to CF and Akamai POPs but I would guess none of the parties involved have any incentive to discuss such a thing in a place that I could read it
  • ddorian43 4y ago
    Cloudflare probably whitelists google/bing/etc. Hell, there are many sites that work with whitelists themselves. It's still the easiest part. > but I would guess none of the parties involved have any incentive to discuss such a thing in a place that I could read it This reminds me how all the tech sites banned people like Alex Jones simultaneously. It's collusion.