3 ms·
IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.
by aero-002 8y ago
IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.