4 ms·
The problem with this is in the vein of `Requires immediate total cooperation from everybody at once` if it's going to replace googlebot. Everyone who only allo
by mullingitover 1y ago
The problem with this is in the vein of `Requires immediate total cooperation from everybody at once` if it's going to replace googlebot. Everyone who only allows googlebot would need to change and allow ccbot instead.
It's already the case that googlebot is the common denominator bot that's allowed everywhere, ccbot not so much.
- CPLX 1y agoI mean if it’s created as part of setting the global rules for the internet you could just make it opt out.
- xp84 1y agoWouldn’t a decent solution, if some action happened where Google was divesting the crawler stuff, be to just do like browser user agents have always done (in that case multiple times to comical degrees)? Something like ‘Googlebot/3.1 (successor, CommonCrawl 1.0)’
- toomuchtodo 1y agoLots of good replies to your comment already. I'd also offer up Cloudflare offering the option to crawl customer origins, with them shipping the compressed archives off to Common Crawl for storage. This gives site admins and owners control over the crawling, and reduces unnecessary load as someone like Cloudflare can manage the crawler worker queue and network shipping internally. (Cloudflare customer, no other affiliation)
- kzrdude 1y agoThat says that if google switches over to ccbot then the rest will follow.