4 ms·

It doesn’t. You can block their crawler with robots.txt and send DMCA takedown requests for archived pages as the domain owner which they will honor. Edit: I w

by oefrha 2y ago

It doesn’t. You can block their crawler with robots.txt and send DMCA takedown requests for archived pages as the domain owner which they will honor. Edit: I was under the wrong impression that if you specifically call out ia_archiver in robots.txt they would honor it. It’s been completely ignored since 2017.

CallMeMarc 2y ago
If I remember correctly they changed their policy quite some time ago and started to ignore the robots.txt, but not 100% sure about it
wl 2y ago
Their crawler now ignores robots.txt.
micromacrofoot 2y ago
robots.txt is a suggestion not a block
deleted 2y ago
[deleted]