4 ms·
robots.txt means they shouldn't auto-scan your site. Any user though can go to the wayback machine and type in a URL and the wayback machine will read that URL.
by socalgal2 5mo ago
robots.txt means they shouldn't auto-scan your site. Any user though can go to the wayback machine and type in a URL and the wayback machine will read that URL. That was the intent of robots.txt (don't scan) not (don't read period). It's spelled out in the spec for robots.txt
- keane 5mo agoThe <meta name="robots"> tag and robots.txt serve different roles: robots.txt controls crawling, while the robots meta tag influences indexing and other behavior. https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/name/robots https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/... I wonder how archive.org_bot behaves when <meta name="robots" content="noindex, noarchive, nocache" /> is present.
- socalgal2 5mo agoThe person above those is complaining about entries in their logs from bots. A robot can't read a tag without first reading the document. So sure, if they're a good bot they might not store the results but the server's logs will still show the bot's GET request.