6 ms·

archive.org will only archive publicly visible content and it respects robots.txt

• tyingq 6y ago
  "and it respects robots.txt" Not since 2017. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... They now have a clunky manual process to exclude your site. https://help.archive.org/hc/en-us/articles/360004651732-Using-The-Wayback-Machine https://help.archive.org/hc/en-us/articles/360004651732-Usin... ("How can I exclude...") They don't spoof user agents, but blocking them actively doesn't remove their history.
  • Karunamon 6y ago
    When did this change? It used to be that adding robots.txt would retroactively remove archives for a domain.
    • tyingq 6y ago
      2017.
    • pseudalopex 6y ago
      Hide. Not remove.