4 ms·
no I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes
by npunt 3y ago
no I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes