3 ms·
Archive Team is making a distributed backup of the Internet Archive. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php
by sp332 9y ago
Archive Team is making a distributed backup of the Internet Archive. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK Currently the method getting the most attention is to put the data into git-annex repos, and then have clients just download as many files as they have storage space for. But because of limitations with git, each repo can only handle about 100,000 files even if they are not "hydrated". http://git-annex.branchable.com/design/iabackup/ http://git-annex.branchable.com/design/iabackup/ If git performance were improved for files that have not been modified, this restriction could be lifted and the manual work of dividing collections up into repos could be a lot lower.
Edit: If you're interested in helping out, e.g. porting the client to Windows, stop by the IRC channel #internetarchive.bak on efnet.
- gcb0 9y agointernet archive sounds like the best ever use case for IPFS
- sp332 9y agoIt was considered but it just didn't get enough attention from anyone to get it done. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/ipfs_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/i...
- StavrosK 9y agoWouldn't IPFS be much, much more suitable for this purpose?
- jakeogh 9y agoCould bup be useful in addition to git-annex? https://github.com/bup/bup https://github.com/bup/bup