4 ms·
https://eotarchive.org/ https://eotarchive.org/ > The Internet Archive has to date downloaded 500 terabytes of US government websites, which it crawls at the e
by toomuchtodo 2y ago
https://eotarchive.org/ https://eotarchive.org/
> The Internet Archive has to date downloaded 500 terabytes of US government websites, which it crawls at the end of every presidential term. The whole archive is fully searchable. This effort's housed by a donation-funded nonprofit, not a branch of the US government.
https://bsky.app/profile/adzebill.bsky.social/post/3lh3cnkqikk2j https://bsky.app/profile/adzebill.bsky.social/post/3lh3cnkqi...
- maxerickson 2y agoWhat if they update robots.txt?
- toomuchtodo 2y agoRespecting robots.txt can be overridden.