4 ms·
Nothing in their rambling even remotely supports their argument. Yes, robots.txt is no magic bullet against ill-behaving crawlers (as proven by ArchiveTeam) bu
by datalist 10y ago
Nothing in their rambling even remotely supports their argument.
Yes, robots.txt is no magic bullet against ill-behaving crawlers (as proven by ArchiveTeam) but it never was supposed to be that.
You choose to ignore my specific wish not to be crawled by you? Fair enough, I'll return the favour and simply block your useragent
ArchiveTeam ArchiveBot/[DATECODE] (wpull [VERSION]) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[VERSION] Safari/537.36
In Apache for example
# Remove the ^ anchor in case text gets prepended
RewriteCond %{HTTP_USER_AGENT} ^ArchiveTeam
RewriteRule .* - [F,L]
and if possible your IP ranges.
- sgift 10y agoThat's okay. You and the three people on this planet who actually use a robots.txt because they don't want to be crawled are not the ones they care about. The 99% that just blindly copied some tutorial which said something bright like "only allow Googles bot" without thinking about it are.
- zamber 10y agoResistance is futile. Eventually you'll get a malicious crawler ignoring robots.txt, changing the UA for each request and querying you from a distributed network of different data centers.
- deleted 10y ago[deleted]
- krapp 10y agoYou're assuming ill-behaving crawlers wouldn't show you a completely mundane user-agent. I know they often don't... and I wonder why they don't, but there's no reason they should.
- deleted 10y ago[deleted]