3 ms·
>may cause the Internet Archive to be banned from crawling some websites. It looks like Facebook banned ia_archiver (recently? I recall it worked a few weeks a
by r721 10y ago
>may cause the Internet Archive to be banned from crawling some websites.
It looks like Facebook banned ia_archiver (recently? I recall it worked a few weeks ago):
>User-agent: ia_archiver
>Disallow: /
https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt