13 ms·
End of Term Web Archive – Preserving the Transition of a Nation (2024)
- toomuchtodo 2y agohttps://eotarchive.org/ https://eotarchive.org/ > The Internet Archive has to date downloaded 500 terabytes of US government websites, which it crawls at the end of every presidential term. The whole archive is fully searchable. This effort's housed by a donation-funded nonprofit, not a branch of the US government. https://bsky.app/profile/adzebill.bsky.social/post/3lh3cnkqikk2j https://bsky.app/profile/adzebill.bsky.social/post/3lh3cnkqi...
- maxerickson 2y agoWhat if they update robots.txt?
- toomuchtodo 2y agoRespecting robots.txt can be overridden.
- deleted 2y ago[deleted]