4 ms·
I still think that while backup solutions like this are technically very cool, the need for a working backup of the IA right now is so critical -- there is lite
by dmd 12y ago
I still think that while backup solutions like this are technically very cool, the need for a working backup of the IA right now is so critical -- there is literally no backup, none, at all -- that it is just kind of insane that they're unwilling to consider straining their principles at all and doing something more prosaic in the interim like using tape even when people say "hey, we'll donate all the hardware needed". Nope, tape's too proprietary for them.
- bonjourmonde 12y agoThis is not the Internet Archive's own project, but rather the idea of Archive Team, an unaffiliated group. The Internet Archive presumably backs up their data and I think they have some older copies in the Netherlands and the new Library of Alexandria in Egypt. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK
- deleted 12y ago[deleted]
- GauntletWizard 12y agoNot completely unaffiliated: The project is the brainchild of Jason Scott, who straddles both.
- db48x 12y agoI'm an Archive Team member. If you'll donate the hardware, I've got a spot for it in my living room.
- paulsutter 12y agoIf you want to talk about low overhead redundancy schemes based on Reed Solomon coding, let me know, we used them to halve disk space requirements in QFS as opposed to HDFS (with no loss in reliability), and it could be stretched arbitrarily far to get at least some redundancy with an arbitrarily small overhead.
- db48x 12y agoWe opted not to do any fancy encoding of the data, even though it would make it easier to store (we would need fewer volunteers). Instead we store individual files whole and unencoded on each volunteer's machine, allowing the volunteer to make use of them as they see fit; no one has to be able to contact other volunteers in order to get useful data. Of course, since each volunteer's client is downloading files at random, you may not end up with the exact file you wanted (to pick an example at random, perhaps you wanted the fifth reel of the soundex index of the 1880 US Census for the state of Tennessee, but you wanted a collection of Lois Armstrong records from the 1920s instead). This turns out not to be a problem, however, as anyone can use git annex to manually pull down any file in the backup that they want. It gets counted as a copy by the backup system along with all the rest, and increases the redundancy of the backup.
- dmd 12y agoAssuming you can mostly saturate a gigabit port, please contact me. Email in profile via keybase.
- db48x 12y agoAye, there's the rub of course. I can saturate a gigabit port, but sadly my DSL cannot. I appreciate the offer though; I'm sure if we really put our heads together we could make it work. Might be better for you to talk to IA though, rather than Archive Team. Pop in one Friday afternoon or send them an email. Archive Team is a very loose confederation of folk who get angry when a website full of user-created content goes offline. We like to say that our primary virtues are rage, paranoia, and kleptomania, and backing up IA comes under the headings of paranoia and whimsy (an optional virtue) more than anything else. IA is solid; they're not a startup to be acqui-hired, they're not Yahoo. We have time here for volunteers to step forward, each with a chunk of disk space and a desire to help out. We have time to invent something neat that can grow into a different way of doing things. In fact, we have time to do this in several different ways, all running parallel; I want to work on IPFS next.