4 ms·

There is some public discussion about why IA does not strictly adhere to robots.txt: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-do

• kodah 5y ago
  They're basically saying they're choosing to ignore a web convention that explicitly states that people don't want their websites archived or searchable because they want them to be. Sounds pretty unethical to me.
  • BenjiWiebe 5y ago
    When those people die and quit paying for hosting, the information on their website doesn't magically become useless. Perhaps other people still have a need for it. Thank goodness the IA doesn't blindly obey robots.txt.