4 ms·
As someone who has done a tiny bit of scraping, I've been surprised how difficult it is to be a good citizen. Are there packages you know of in the major langu
by yodon 2y ago
As someone who has done a tiny bit of scraping, I've been surprised how difficult it is to be a good citizen.
Are there packages you know of in the major language ecosystems that you can easily point at a both a robots.txt and an index.html (you need both to be compliant, most packages I've found only look at robots.txt) to get an answer on what you can and can't do at what rate?
The only one I'm aware of is a C# package which still requires you to do a lot of the heavy lifting[0].
[0]https://github.com/TurnerSoftware/RobotsExclusionTools https://github.com/TurnerSoftware/RobotsExclusionTools
- cavisne 2y agohttps://github.com/google/robotstxt https://github.com/google/robotstxt
- yodon 2y agoYou're proving my point. That's a reference C++ parser that only parses robots.txt files. It doesn't parse meta tags and doesn't help with writing crawlers running in any of the major language ecosystems used by crawler authors. It's much more difficult than it should be for scrapers who want to be compliant to to actually be compliant.
- mrweasel 2y agoYou obviously should read the robots.txt and follow it, but it's just as much about not just absolutely hammering a site over and over again, 24-7. Your scraper needs to learn about update patterns and adjust it's scraping interval. It needs to back off if the respons times increases. People are bringing up robots.txt because they want to ban AI scrapers, but if those same scrapers didn't constantly pound sites into the ground, it would be less of an issue.