3 ms·
Not every bot that ignores your robots.txt is necessarily using that data. What some bots do is they first scrape the whole site, then look at which parts are
by deadbabe 1y ago
Not every bot that ignores your robots.txt is necessarily using that data.
What some bots do is they first scrape the whole site, then look at which parts are covered by robots.txt, and then store that portion of the website under an “ignored” flag.
This way, if your robots.txt changes later, they don’t have to scrape the whole site again, they can just turn off the ignored flag.
- imtringued 1y agoYour post-rationalization just doubles down on the stance that these crawlers are abusive and poorly developed. You're also under the blatantly wrong misconception that people are worried about their data, when they are worried about the load of a poorly configured crawler. The crawler will scrape the whole website on a regular interval anyway, so what is the point of this "optimization" that optimizes for highly infrequent events?
- nvader 1y agoNot every intruder who enters your home is necessarily a burglar.
- cyphar 1y agoAh, so the NSA defence then -- "it's not bulk collection because it only counts as collection when we look at it".