4 ms·
I can't speak formally for the Internet Archive, but the existing content and services are not going to disappear overnight: funding comes from several sources,
by bnewbold 8y ago
I can't speak formally for the Internet Archive, but the existing content and services are not going to disappear overnight: funding comes from several sources, thought has been put in to organizational structure, and things have been designed to keep core access and preservation infrastructure running with minimal cost and effort (eg, if the economy tanks).
Getting the content coverage people sometimes assume we already have is another matter. Additional funding (thanks for you donation!) go towards additional crawling and keeping up with the endless treadmill of media types and protocols. Eg, headless browser crawling development and deployment to capture javascript-heavy sites (https://github.com/internetarchive/brozzler); https://github.com/internetarchive/brozzler); this is much more expensive than "classic" crawling.
For more on increasing storage costs and the under-funded state of web archiving in general, I recommend David Rosenthal's blog, eg:
https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html
https://blog.dshr.org/2014/03/the-half-empty-archive.html https://blog.dshr.org/2014/03/the-half-empty-archive.html
Far more effective and robust than hoping the archive is "suck it up for us" is to upload snapshots/dumps/exports yourself! Anybody can create an archive.org account and upload content (recommend https://github.com/jjjake/internetarchive https://github.com/jjjake/internetarchive over the HTML form), within reasonable limits. Obviously, care needs to be taken to remove sensitive (and personal) information first.
- toomuchtodo 8y agoThese blog posts are fantastic. Thanks so much for sharing them.
- bryanrasmussen 8y agobrozzler looks great - do you know how they decide what are javascript heavy sites and how they queue them up?