6 ms·

Devs need to be aware that bittorrent now(for sometime) have a DHT solution that allows to have "mutable" slots with the crypto public address tied up to it. S

Devs need to be aware that bittorrent now(for sometime) have a DHT solution that allows to have "mutable" slots with the crypto public address tied up to it. So if you own the private key you can write the payload, and just share your public-key address to the people you want to share the payload with. In the payload you can write a traditional immutable torrent manifest for instance, which is in essence a public-key crypto based update system. For a lot of cases i think it's a better approach than what IPFS and DAT provides, because you dont care about it being a global address. All you want is to share with a group of people, more in the p2p social/organic way. I was playing with it once, using the libtorrent library and the main bittorrent DHT, and it was a very nice experience.. it finds the payload pretty fast when you think its a DHT, and you are working in pure p2p fashion. The only single point of failure here is the DHT bootstrap peer. Im planing to use this feature to distribute binary images for clients that have my public key.

voltagex_ 8y ago
Is there anything I can play with for mutable torrents? I thought it was still only in research paper form.
- namibj 8y ago
  libtorrent has support for it [0], as far as I know. The "research paper" you probably mean [1] is the torrent equivalent of and RFC. AFAIK at least python uses something very similar. There is an apparently Node.js implementation [2] of something that can publish a given torrent to a mutable address, and also retrieve a mutable torrent from a given address. I do not know how well this is implemented in the usual clients, but if you want it in your docker, you might want to talk to libtorrent directly, and implementing BEP 46 yourself should not be hard with the things the library has to offer. A benefit could be that depending on how you handle it, you might be able to store the tarballs docker images seem to be in their unpacked form, and just keep some metadata about what the header(s) of the tarball were, along with some file offsets. This way you would be relieved of the unnecessary storage burden, and able to possibly use many more of your servers to seed at least part of the images, e.g. maybe only the parts that are not mutated when the software is running. E.g., download once, unpack, only offer to seed those files/pieces that did not get modified in the meantime, without trying to re-download the "broken" data. [0]: https://www.libtorrent.org/dht_store.html https://www.libtorrent.org/dht_store.html [1]: http://www.bittorrent.org/beps/bep_0046.html http://www.bittorrent.org/beps/bep_0046.html [2]: https://github.com/lmatteis/dmt https://github.com/lmatteis/dmt
brokenmachine 8y ago
Very interesting. Forgive my ignorance, but you mean it's kind of like mutable torrents that only the uploader can modify? Does a client just search the DHT for the public key? I thought torrent clients searched for the hash of the files. If it's searching for the public key then how does one person upload multiple different torrents, or do they create a new public key for each torrent? How does a client know which is the latest version if it has been updated multiple times? Are there any example projects using this?
- oscargrouch 8y ago
  @namibj has made a more elucidative comment with links to libtorrent library as also the torrent BEP that describe how the DHT are supposed to work in detail. > but you mean it's kind of like mutable torrents that only the uploader can modify? Yes, your public key(hash) is the DHT key which is the one that identifies the payload, and only the private key owner can modify the content in that particular slot. Thats why its cool, because you can have a p2p system that rely on trust between the parties, unlike the traditional torrent system. Also im not so sure, but lately using centralized trackers are discourage and i guess that magnet links must use something like a DHT to work the way they do. > I thought torrent clients searched for the hash of the files. You are correct, but the DHT is a BEP and is something more "on the corner", but its theres and at least in the time i've tried it was working great. > If it's searching for the public key then how does one person upload multiple different torrents, or do they create a new public key for each torrent? How does a client know which is the latest version if it has been updated multiple times? The rules are: You can create any slot you want, its just a matter of generating the public/private pair you want to use. (This would allow you even to use a forward-secrecy algorithm if you need one) The byte payload/value must be small, so you should use to give a manifest of something or to point to something else.. But lets not forget that Git just look at the HEAD record with only a hash to go on from there. Just pointing out to something else that can be a immutable resource, like a traditional torrent. So you can point to a torrent, download and have a small list of anything you like.. working as a catalog, and go from there.. anyway if you play the indirection game here right, theres no limit to what you can do. If you want the payload to be there you need to keep writing to it from time to time (the same value if you want), or it will expire and other peers will not be able to locate it anymore. What would i do? i would use the payload to point to something else.. like some torrent in the classic bittorrent network (you can use just a magnet link), or expose the whole torrent header. You can also point to some http resource or whatever. Need something more? how about point to a torrent that download a bootstrap program that start a RPC service over tcp.. or over a more simple HTTP interface.. than do something else from there.. I was thinking about how can i use this to create a update by using diff and patch, giving that by using a public key scheme you can create a trust relationship between parties and patch the binary with something coming from the 'mothership'. > Are there any example projects using this? I dont know any, but in my case i was playing with libtorrent implementation of DHT. And also as far as i know, is this kind of properties in the DHT that allow projects like IPFS to exist. The cool thing about using the torrent implementation is because the main DHT have a lot of nodes already so you can find something pretty fast.

voltagex_ 8y ago

Is there anything I can play with for mutable torrents? I thought it was still only in research paper form.

namibj 8y ago
libtorrent has support for it [0], as far as I know. The "research paper" you probably mean [1] is the torrent equivalent of and RFC. AFAIK at least python uses something very similar. There is an apparently Node.js implementation [2] of something that can publish a given torrent to a mutable address, and also retrieve a mutable torrent from a given address. I do not know how well this is implemented in the usual clients, but if you want it in your docker, you might want to talk to libtorrent directly, and implementing BEP 46 yourself should not be hard with the things the library has to offer. A benefit could be that depending on how you handle it, you might be able to store the tarballs docker images seem to be in their unpacked form, and just keep some metadata about what the header(s) of the tarball were, along with some file offsets. This way you would be relieved of the unnecessary storage burden, and able to possibly use many more of your servers to seed at least part of the images, e.g. maybe only the parts that are not mutated when the software is running. E.g., download once, unpack, only offer to seed those files/pieces that did not get modified in the meantime, without trying to re-download the "broken" data. [0]: https://www.libtorrent.org/dht_store.html https://www.libtorrent.org/dht_store.html [1]: http://www.bittorrent.org/beps/bep_0046.html http://www.bittorrent.org/beps/bep_0046.html [2]: https://github.com/lmatteis/dmt https://github.com/lmatteis/dmt

brokenmachine 8y ago

Very interesting. Forgive my ignorance, but you mean it's kind of like mutable torrents that only the uploader can modify? Does a client just search the DHT for the public key? I thought torrent clients searched for the hash of the files. If it's searching for the public key then how does one person upload multiple different torrents, or do they create a new public key for each torrent? How does a client know which is the latest version if it has been updated multiple times? Are there any example projects using this?

oscargrouch 8y ago
@namibj has made a more elucidative comment with links to libtorrent library as also the torrent BEP that describe how the DHT are supposed to work in detail. > but you mean it's kind of like mutable torrents that only the uploader can modify? Yes, your public key(hash) is the DHT key which is the one that identifies the payload, and only the private key owner can modify the content in that particular slot. Thats why its cool, because you can have a p2p system that rely on trust between the parties, unlike the traditional torrent system. Also im not so sure, but lately using centralized trackers are discourage and i guess that magnet links must use something like a DHT to work the way they do. > I thought torrent clients searched for the hash of the files. You are correct, but the DHT is a BEP and is something more "on the corner", but its theres and at least in the time i've tried it was working great. > If it's searching for the public key then how does one person upload multiple different torrents, or do they create a new public key for each torrent? How does a client know which is the latest version if it has been updated multiple times? The rules are: You can create any slot you want, its just a matter of generating the public/private pair you want to use. (This would allow you even to use a forward-secrecy algorithm if you need one) The byte payload/value must be small, so you should use to give a manifest of something or to point to something else.. But lets not forget that Git just look at the HEAD record with only a hash to go on from there. Just pointing out to something else that can be a immutable resource, like a traditional torrent. So you can point to a torrent, download and have a small list of anything you like.. working as a catalog, and go from there.. anyway if you play the indirection game here right, theres no limit to what you can do. If you want the payload to be there you need to keep writing to it from time to time (the same value if you want), or it will expire and other peers will not be able to locate it anymore. What would i do? i would use the payload to point to something else.. like some torrent in the classic bittorrent network (you can use just a magnet link), or expose the whole torrent header. You can also point to some http resource or whatever. Need something more? how about point to a torrent that download a bootstrap program that start a RPC service over tcp.. or over a more simple HTTP interface.. than do something else from there.. I was thinking about how can i use this to create a update by using diff and patch, giving that by using a public key scheme you can create a trust relationship between parties and patch the binary with something coming from the 'mothership'. > Are there any example projects using this? I dont know any, but in my case i was playing with libtorrent implementation of DHT. And also as far as i know, is this kind of properties in the DHT that allow projects like IPFS to exist. The cool thing about using the torrent implementation is because the main DHT have a lot of nodes already so you can find something pretty fast.