3 ms·
Internet Archive Never Forgets
- J_Darnley 10y agoExcept for when someone gives them a robots.txt instruction. Then they forget whole websites.
- CM30 10y agoYeah, which is the Internet Archive's biggest issue. What site uses a domain name at one time isn't what site might use the same domain name at a different date. If the archive had more common sense, they'd note down when sites and robots.txt files change and only block pages by the site owner with the offending robots.txt file. So if say, Google went under and a new company started up, a robots.txt file from the latter would only block files added since the site switched over. That would solve 99.99% of these issues.
- db48x 10y agoThe IA doesn't forget whole websites. If the current robots.txt file excludes some website from crawling, IA hides (but doesn't delete), the excluded information.
- meshr 10y agoExcept for when someone’s government asks to remove content. For example, this link http://web.archive.org/web/20150618110931/http://ozpp.ru/pamyatka-potrebitelyam-pri-poseschenii-okkupirovannyh-territoriy/ http://web.archive.org/web/20150618110931/http://ozpp.ru/pam... was removed from Internet Archive machine by Russian government request It worked this way: Russia blocked IA for Russians => IA removed content for everyone => Russia unblocked IA for Russians.