4 ms·
This is a neat idea. I have some random thoughts about doing this in a distributed fashion: ... In the wiki there's a mention of how the geocities torrent was
by makmanalp 12y ago
This is a neat idea. I have some random thoughts about doing this in a distributed fashion:
...
In the wiki there's a mention of how the geocities torrent was 900GB and very few people seeded it as a result. There should be a way and an incentive for clients to seed parts of files without having the whole thing. As long as there are enough copies of each chunk on the network, it's fine - we don't care that any one client has the whole thing.
...
Another cool thing is thinking about health metrics - you don't just care about how many people have a file, you care about recoverability too. This seems a bit similar to AWS glacier - you have to take into account how long it's going to retrieve data. You could have stats on how often a client responded and completed a transfer request, what its track record is, and assign a probability and timeframe for correctly retrieving a file.
...
One thing that comes to mind is that whatever the solution ends up being, it should be such that partial recovery should be possible. The example that comes to mind is stuff like a zip file or encrypted image that is completely ruined when a chunk is missing. So maybe it makes sense to have the smallest unit of information be large enough to be still intelligible on its own?
At first I was thinking of a model where chunks get randomly distributed and replicated, but then that made me wonder whether that's a bad idea.
...
And then what of malicious clients? I guess it's not hard to have a hash-based system and then just verify the contents of a file upon receiving it. But can I effectively destroy a file by controlling all the clients that are supposed to be keeping it up? How could you guard against that? Could you send a challenge to a client to send you the hash of a random byte sequence in a file to prove that they do indeed have it? What guarantees that they will send it even if they do?
...
And then what about search / indexing?! P2P search is an awesome problem domain that has so many problems that my mind is buzzing right now. Do you just index predetermined metadata? Do you allow for live searches that propagate through the system? Old systems like kademlia had issues with spammers falsifying search results to make you download viruses and stuff - how to guard against that? Searching by hash is safe but by name / data is not. Etc etc.
I wish this was my job! :)
- darkmighty 12y agoThis seems to have a lot of overlap with people coming up with distributed storage systems, perhaps Maidsafe [1] has solved some of those problems? (I'm not very familiar with it) [1] http://en.wikipedia.org/wiki/MaidSafe http://en.wikipedia.org/wiki/MaidSafe