5 ms·
I see things like this, and I wonder why the following software doesn't exist: I want a piece of software to which I can add a collection of files, say multipl
by stavros 4y ago
I see things like this, and I wonder why the following software doesn't exist:
I want a piece of software to which I can add a collection of files, say multiple TB. The software will then behave a bit like a BitTorrent tracker, and know which peer has which files. A peer joining this swarm will be able to say "I want to donate X GB of space", and the tracker would tell it "OK, then download and seed these files, which are the least seeded".
The peer would download the files from the rest of the swarm and make them available to it. Then, a request layer on top of the swarm could be used to request a file from the peer which had it. Adding/removing files to this collection would also need to be a feature.
Does anyone know if anything like this exists? If not, how easy would it be to make something like it out of BitTorrent? I might give it a go.
- er4hn 4y agoThis sounds very similar to ipfs. How does it differ?
- stavros 4y agoIPFS can't give you the least-seeded files. It can't give you any files, you have to manually pin the set you want, and you pin all of it, and it won't automatically change. It's not very convenient for archival, whereas the system I'm talking about would be (in my opinion, anyway).
- paskozdilar 4y ago(How) would you filter junk?
- stavros 4y agoYou'd trust the publisher, so you'd say "I want to help the Internet Archive with its archiving" and they'd be responsible for what files they pushed to your storage. This is basically an opportunistic, distributed filesystem that's designed to work with high latency links. Only the tracker can write to its nodes, but anyone can read.
- thanatos519 4y agoThis, and easy-to-configure bandwidth throttling was sorely missing from IPFS last time I tried it. Being able to specify "use 50% of storage to cache low-traffic things, and the other 50% for high-traffic things" would be amazing.
- Brian_K_White 4y agoNot part of the question or problem statement as I see it.
- deleted 4y ago[deleted]
- WhoIsSatoshi 4y agoI keep thinking in my mind this makes sense but then ponder about the "local" impact - what would a user "donating space" be storing locally and might risk? Where does the risk lay, is it with the uploader, the seeder, or the service provider? I would love to see this pickup but it feels as though it would devolve into one of the listed projects. It "sounds" as though it should use some kind of Distributed Ledger Technology, IPFS seem the most fit, maybe some kind of next layer that offers just this service (IPFS being GDPR compliant might mitigate some risk?)
- dchuk 4y agoHave always thought this would be a great way for many people to share well organized Plex libraries. Many semi-overlapping libraries basically creating a virtual Netflix, with some way to stream in Plex from the whole library no matter if you actually have the content locally or not
- 867-5309 4y ago>virtual Netflix this already exists with addons for Kodi
- sedatk 4y agoDonating space is only half of the equation. I think donating bandwidth is a more significant aspect, especially with ISP's like Comcast which provide very little upload bandwidth compared to download. You'd expect that uploads wouldn't impact download speeds, but it's not the case. A saturated upload bandwidth means, ACK packets getting delayed, which means connections would be established way more slower. So, it's not a feasible prospect unless the competition takes over.
- throwaway742 4y agoYou can always implement QoS or something like FQ_Codel.
- sedatk 4y agoI'm a senior software developer with experience in developing communications protocols and networking software. I haven't been able to set up QoS properly ever, especially in a way that addresses all my needs. Expecting end users to do it is way beyond the realm of possibility IMHO.
- throwaway742 4y agoIt is a huge pain in the ass to get it working right I will admit. Took me hours of reading docs and tweaking parameters. I'm still not sure I understand all of it, but I managed to get it working such that it at least meets my needs.
- sgtnoodle 4y agoIt's proprietary, but it sounds like you're describing the "Google File System". https://en.m.wikipedia.org/wiki/Google_File_System https://en.m.wikipedia.org/wiki/Google_File_System For your idea, once all the local storage everywhere is filled up with evenly distributed redundant copies, and then a new file is added, would peers arbitrarly choose other files to delete in order to make room for the new file?
- stavros 4y agoYes, it's very similar, though would work slightly differently (BitTorrent under the hood, whole files instead of chunks, etc).
- thrdbndndn 4y agoThat's literally how most of Japanese P2P software work. For example, Perfect Dark, Winny, and to less extent, Share (which is more similar to eDonkey/eMule).
- throwaway742 4y agoPrivate trackers do this somewhat via incentives. Files with less seeds earn you more bonus points. If you want to farm BP you can sort files by seeders and download poorly seeded torrents.
- npteljes 4y agoFreenet is built around similar ideas, combined with encryption an anonymization. Which hopefully adds the benefit of you not being legally liable of distributing CSAM. The gist of Freenet is that you can operate a freesite or upload files, and the content is redundantly dispersed in encrypted parts among a number of other Freenet peers. They don't know what they're hosting and neither do you, the client just fills up the allotted space and uses some bandwidth, that's all. https://en.wikipedia.org/wiki/Freenet https://en.wikipedia.org/wiki/Freenet Regarding of how easy to make something like this network, I'd wager it's pretty hard. There will be a lot of questions, even while establishing the happy path, for example how you manage the updates, especially when you update the protocol, not just the software, and how you effectively manage the volume of search requests, how you distribute the files etc. And then there's the abuse the network will inevitably get. How you handle spammers, CSAM, malware, ISPs that throttle/block you, the legal risk you put your clients up to, etc. Nice big can of worms. To begin opening it, I suggest a reading through Wikipedia's Peer to peer file sharing article, and especially the File sharing modal on the right, which nicely captures the ideas that have been tried so far. https://en.wikipedia.org/wiki/Peer-to-peer_file_sharing https://en.wikipedia.org/wiki/Peer-to-peer_file_sharing
- stavros 4y agoMy idea is similar but slightly different. You'd run your client and then choose whose provider's data to seed, eg you'd add the Internet Archive and Libgen to your datasets. Only those entities would be able to push data to you, nobody else, so if you trust the providers you specified, you should be good.
- npteljes 4y agoI remembered that I read about something like that before. I think this was it: https://en.wikipedia.org/wiki/Storage@home https://en.wikipedia.org/wiki/Storage@home Further this lead me to distributed data stores: https://en.wikipedia.org/wiki/Distributed_data_store#Peer_network_node_data_stores https://en.wikipedia.org/wiki/Distributed_data_store#Peer_ne...
- 4y ago
- DataWraith 4y agoSomething similar exists: iabackup[1][2]. It is designed to host an independent copy of (some of) the Internet Archive using git-annex. You tell it how much storage you want to donate and git-annex fills your disk with data from the least-seeded files IIRC. Its focus is on data backup, not data serving though. [1]: https://git-annex.branchable.com/design/iabackup/ https://git-annex.branchable.com/design/iabackup/ [2]: https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/git-annex_implementation https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK/g...
- zidel 4y agoI think BitTorrent has all the pieces needed for a fully distributed version of your idea. My initial thought is that you could publish a magnet link that points to a mutable DHT item, which in turn points to a torrent that has a JSON file with some metadata and a list of infohashes the publisher cares about. The client could then scrape the "leaf" torrents from multiple lists to get the peer counts and use that for local prioritization of what to store. By reusing existing torrents you could then share resources with standard torrent clients that are unaware of your system. The list idea could be extended to nested lists (stavros recommends Internet Archive) for discoverability and composition. If you go with v2 or hybrid torrents from the beginning you could deduplicate and cross seed files from different collections. The lists could also be modified to have torrents to exclude, possibly using some salt + rehash idea to make it hard to reverse into a list of e.g. CSAM you don't want to publish as is. Feels like a neat project that could interoperate nicely with existing torrents.
- stavros 4y agoThanks, that's exactly the feedback I was looking for! This sounds like it would work, though I'd have to see if it would scale to thousands or millions of files. Still, great for a PoC, thank you!