4 ms·
That is not what the article says. It says Internet Archive had already started ignoring robots.txt on US government websites. Now (since 2017) they ignore it
by symfoniq 6y ago
That is not what the article says.
It says Internet Archive had already started ignoring robots.txt on US government websites.
Now (since 2017) they ignore it on all websites.