3 ms·

Does anyone here work at Archive.org? Can you speak to how well-funded the organization is and what sort of measures are in place to keep it afloat? I think it'

Does anyone here work at Archive.org? Can you speak to how well-funded the organization is and what sort of measures are in place to keep it afloat? I think it's a fantastic service and I donate, but I worry that it could vanish the next day if funding suddenly dries up. I feel like a large corner of the Internet collectively takes the site for granted and don't bother doing their own in-house archiving because "TheArchive will just suck it up for us."

deleted 8y ago
[deleted]
ilaksh 8y ago
I don't work for them but I think the move towards decentralization is the best way to address those concerns.
- kinsomo 8y ago
  > I don't work for them but I think the move towards decentralization is the best way to address those concerns. How feasible would a reliable, distributed archive be; given how massive amount of data Archive.org has? After all, it was created precisely because the already-decentralized web was too ephemeral and unstable. I don't think decentralization is a panacea in this case. Some problems are best solved by institutions.
  - toomuchtodo 8y ago
    Yeah, spot on. Governance and durability through org structure is a thing. The way to solve this is to provide the Internet Archive with enough resources to build out a globally distributed storage system. Could you hack something together using their torrent tracker for every item served? Yes. But you don't hack together something made to preserve digital human culture in perpetuity.
    - kinsomo 8y ago
      > Yeah, spot on. Governance and durability through org structure is a thing. > The way to solve this is to provide the Internet Archive with enough resources to build out a globally distributed storage system. Yeah, I agree. I do see a space for other appropriate institutions (such as the Library of Congress, British Library, etc.) to pool resources and facilities with the Internet Archive to achieve that goal. Ultimately, it'd be awesome to see each national library run a semi-autonomous IA copy that synchronizes with all the others, but can continue to operate independently (scrapers and all), if need be.
  - ilaksh 8y ago
    Please research dat and similar systems more before you dismiss them as unscalable and ephemeral. That isn't the case.
- llao 8y ago
  Check out and contribute to https://www.archiveteam.org/index.php/INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php/INTERNETARCHIVE.BAK :)
dublinben 8y ago
You can read their IRS 990 forms to get a sense of what their financial health is. Ironically, I couldn't locate these anywhere on Archive.org 2016: http://990s.foundationcenter.org/990_pdf_archive/943/943242767/943242767_201612_990.pdf http://990s.foundationcenter.org/990_pdf_archive/943/9432427... more: http://990finder.foundationcenter.org/990results.aspx?action=Find&fn=&st=&ei=943242767 http://990finder.foundationcenter.org/990results.aspx?action...
bnewbold 8y ago
I can't speak formally for the Internet Archive, but the existing content and services are not going to disappear overnight: funding comes from several sources, thought has been put in to organizational structure, and things have been designed to keep core access and preservation infrastructure running with minimal cost and effort (eg, if the economy tanks). Getting the content coverage people sometimes assume we already have is another matter. Additional funding (thanks for you donation!) go towards additional crawling and keeping up with the endless treadmill of media types and protocols. Eg, headless browser crawling development and deployment to capture javascript-heavy sites (https://github.com/internetarchive/brozzler); https://github.com/internetarchive/brozzler); this is much more expensive than "classic" crawling. For more on increasing storage costs and the under-funded state of web archiving in general, I recommend David Rosenthal's blog, eg: https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html https://blog.dshr.org/2014/03/the-half-empty-archive.html https://blog.dshr.org/2014/03/the-half-empty-archive.html Far more effective and robust than hoping the archive is "suck it up for us" is to upload snapshots/dumps/exports yourself! Anybody can create an archive.org account and upload content (recommend https://github.com/jjjake/internetarchive https://github.com/jjjake/internetarchive over the HTML form), within reasonable limits. Obviously, care needs to be taken to remove sensitive (and personal) information first.
- toomuchtodo 8y ago
  These blog posts are fantastic. Thanks so much for sharing them.
- bryanrasmussen 8y ago
  brozzler looks great - do you know how they decide what are javascript heavy sites and how they queue them up?

deleted 8y ago

[deleted]

ilaksh 8y ago

I don't work for them but I think the move towards decentralization is the best way to address those concerns.

kinsomo 8y ago
> I don't work for them but I think the move towards decentralization is the best way to address those concerns. How feasible would a reliable, distributed archive be; given how massive amount of data Archive.org has? After all, it was created precisely because the already-decentralized web was too ephemeral and unstable. I don't think decentralization is a panacea in this case. Some problems are best solved by institutions.
- toomuchtodo 8y ago
  Yeah, spot on. Governance and durability through org structure is a thing. The way to solve this is to provide the Internet Archive with enough resources to build out a globally distributed storage system. Could you hack something together using their torrent tracker for every item served? Yes. But you don't hack together something made to preserve digital human culture in perpetuity.
  - kinsomo 8y ago
    > Yeah, spot on. Governance and durability through org structure is a thing. > The way to solve this is to provide the Internet Archive with enough resources to build out a globally distributed storage system. Yeah, I agree. I do see a space for other appropriate institutions (such as the Library of Congress, British Library, etc.) to pool resources and facilities with the Internet Archive to achieve that goal. Ultimately, it'd be awesome to see each national library run a semi-autonomous IA copy that synchronizes with all the others, but can continue to operate independently (scrapers and all), if need be.
- ilaksh 8y ago
  Please research dat and similar systems more before you dismiss them as unscalable and ephemeral. That isn't the case.
llao 8y ago
Check out and contribute to https://www.archiveteam.org/index.php/INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php/INTERNETARCHIVE.BAK :)

dublinben 8y ago

You can read their IRS 990 forms to get a sense of what their financial health is. Ironically, I couldn't locate these anywhere on Archive.org 2016: http://990s.foundationcenter.org/990_pdf_archive/943/943242767/943242767_201612_990.pdf http://990s.foundationcenter.org/990_pdf_archive/943/9432427... more: http://990finder.foundationcenter.org/990results.aspx?action=Find&fn=&st=&ei=943242767 http://990finder.foundationcenter.org/990results.aspx?action...

bnewbold 8y ago

I can't speak formally for the Internet Archive, but the existing content and services are not going to disappear overnight: funding comes from several sources, thought has been put in to organizational structure, and things have been designed to keep core access and preservation infrastructure running with minimal cost and effort (eg, if the economy tanks). Getting the content coverage people sometimes assume we already have is another matter. Additional funding (thanks for you donation!) go towards additional crawling and keeping up with the endless treadmill of media types and protocols. Eg, headless browser crawling development and deployment to capture javascript-heavy sites (https://github.com/internetarchive/brozzler); https://github.com/internetarchive/brozzler); this is much more expensive than "classic" crawling. For more on increasing storage costs and the under-funded state of web archiving in general, I recommend David Rosenthal's blog, eg: https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html https://blog.dshr.org/2018/05/longer-talk-at-msst2018.html https://blog.dshr.org/2014/03/the-half-empty-archive.html https://blog.dshr.org/2014/03/the-half-empty-archive.html Far more effective and robust than hoping the archive is "suck it up for us" is to upload snapshots/dumps/exports yourself! Anybody can create an archive.org account and upload content (recommend https://github.com/jjjake/internetarchive https://github.com/jjjake/internetarchive over the HTML form), within reasonable limits. Obviously, care needs to be taken to remove sensitive (and personal) information first.

toomuchtodo 8y ago
These blog posts are fantastic. Thanks so much for sharing them.
bryanrasmussen 8y ago
brozzler looks great - do you know how they decide what are javascript heavy sites and how they queue them up?