4 ms·
I designed a system where you could say "donate this spare 2 TB of my disk space to the Internet Archive" and the IA would push 2 TB of data to you. This system
by stavros 2y ago
I designed a system where you could say "donate this spare 2 TB of my disk space to the Internet Archive" and the IA would push 2 TB of data to you. This system also has the property that it can be reconstructed if the IA (or whatever provider) goes away.
Unfortunately, when I talked to a few archival teams (including the IA) about whether they'd be interested in using it, I either got no response or a negative one.
- 4gotunameagain 2y agoWhy reinvent the wheel ? There are so many proven distributed archiving systems, a lot of which are mentioned in these comments.
- stavros 2y agoWhat system of these allows me to donate a bunch of disk space to a provider of my choosing, without thinking about it afterwards?
- whywhywhywhy 2y agoBecause the incentive will be archiving things they believe should be archived, you need the process to begin with what urls do you want to be archiving, then people will be incentivized for archiving the juicy stuff IA is used for and you just throw some stuff they didn't ask to archive in the remit of them storing what they want.
- stavros 2y agoI don't think that's necessarily true, I have a spare TB that I'd be glad to donate to the IA to store whatever they want in.
- boramalper 2y agoIs this open source or do you have any design docs? I love the idea and would love to learn more about it.
- stavros 2y agoThe idea is that it'll be open source, I have a rough design doc here: https://docs.google.com/document/d/1qKgIjUTef-I-BLWjn4sEIbYosqUtrepGF-WjFUx2DQA/edit?usp=drivesdk https://docs.google.com/document/d/1qKgIjUTef-I-BLWjn4sEIbYo... I'll write up a more detailed article on it, though, it'll be good to at least have the doc public somewhere.
- TZubiri 2y agoHosting something at a volunteer's drive, without any guarantees, is pretty useless. They can cease hosting, or have disk damage, and you lose data
- stavros 2y agoYou're essentially saying that having an extra copy of data is equally as reliable as not having an extra copy of data. I would encourage you to think about this a bit more.
- TZubiri 2y agoFor these parameters, yes. If you have a raid, then you have 2 copies with like 99.99% availability and 5 mean time years to failure. With a volunteer drive you have like ?% availability and ?% years to failure? You can't depend on it. Also the average value of data is very low, you don't want to be making many copies of for no reason.
- stavros 2y agoThat would mean that even with a million volunteer drives storing a file, you still wouldn't be able to depend on them, which is plainly wrong. > Also the average value of data is very low, you don't want to be making many copies of for no reason. The reason is that the value of that data is high to the archivist, since they want to preserve it.
- TZubiri 2y agoA million is out of the parameters of the case. Realistically you won't get enough volunteer-storage to cover one IA. And even if you did, it wouldn't satisfy the mission requirements, which is to store reliably for decades all of the data.
- stavros 2y agoThis isn't meant to be storage for IA, it's meant to be a distributed backup.