For the second example it's pretty clear why it's excluded: https://www.amren.com/robots.txt https://www.amren.com/robots.txt User-agent: ia_archiver

For the second example it's pretty clear why it's excluded: https://www.amren.com/robots.txt https://www.amren.com/robots.txt User-agent: ia_archiver Disallow: / For the first example, it sounds like[1] she requested that her page be excluded. It sounds like[2] the Internet Archive allows you to request that. [1] https://www.foxnews.com/media/taylor-lorenz-twitter-page-excluded-internet-archive https://www.foxnews.com/media/taylor-lorenz-twitter-page-exc... [2] https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/ https://help.archive.org/help/how-do-i-request-to-remove-som...