2 ms·
The best part about "verified crawlers" is that there's no easy way to discover how to become one. Or if you need to become one.
by ccgreg 2y ago
The best part about "verified crawlers" is that there's no easy way to discover how to become one. Or if you need to become one.
- bigiain 2y agoEverybody knows how to become one. It's just like every "enterprise SaaS" out there. There's no 3 tier pricing plan with lists of features. You need to contact enterprise sales so they can work out how much you can afford to pay, then take all your money. And you _know_ if you need to become a "verified crawler", you just need to remember the developers you demoted or fired when they brought up the ethical problems of way you've configured your crawlers.
- ccgreg 2y agoHow does that second paragraph work? I run engineering at Common Crawl, and Common Crawl is ethical and has never fired a developer over ethics. During the End of Term 2024 crawl[1], we discovered a lot of blocking on US government websites. Many of these sites were also blocking the Internet Archive and the US National Archives. The US National Archives is a government agency. 1: https://eotarchive.org/ https://eotarchive.org/