7 ms·

Updating Torrents via DHT Mutable Items

117 points by sktrdie 10y ago

Mizza 10y ago
This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this direction: http://www.bittorrent.org/beps/bep_0044.html http://www.bittorrent.org/beps/bep_0044.html although I don't think that it really goes far enough, as there is no standardization around how the data for searching will be structured, etc.
- sktrdie 10y ago
  Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
  - cocotino 10y ago
    An index? That could be huge. I think it's senseless. So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?
    - deleted 10y ago
      [deleted]
    - swolchok 10y ago
      Vuze (formerly known as Azureus) has (or at least had a prototype of) p2p search. I wrote a paper on building a torrent search index quickly using the DHT records that this feature emitted at the time. http://static.usenix.org/events/woot10/tech/full_papers/Wolchok.pdf http://static.usenix.org/events/woot10/tech/full_papers/Wolc...
      - sktrdie 10y ago
        Great work with this. Are you still in academia doing research?
    - niftich 10y ago
      The Mainline DHT (the more-supported DHT vs. Azureus/Vuze DHT) traditionally only contains (iirc, hashes of) torrent infohases mapped to peer lists. With BEP 44 (arbitrary data in DHT) or some other enhancement, this may change, but right now even if you traverse the DHT there's no "interesting" metadata about the keys (or, y'know, the actual content of the .torrent file) inside the DHT. From the sibling comment (posted by swolchok)'s paper, a relevant quote: "We chose to demonstrate our proof of concept only on Vuze due to the significant additional complexity of supporting two DHTs in the crawler. While Mainline does not contain torrent descriptions, its peer lists are keyed by the torrent infohashes themselves, so no additional machinery would be needed to discover infohashes. However, some method would be needed to discover torrent names, such as downloading .torrent files directly from peers using the metadata exchange protocol. To explore this possibility, we built a prototype .torrent crawler. Since we are operating in the context of the Vuze DHT, our DHT crawler needs to obtain both a torrent description and a peer list before the .torrent crawler can contact the appropriate peers with the correct infohash. However, the .torrent files contain useful additional data, including the full listing of files contained in the torrent"
    - xorcist 10y ago
      Kademlia search is also rather useless as it is flooded with false results. This is of interest not only to malware writers but also as a service to sell to rights holders.
  - Mizza 10y ago
    You don't see a problem with that..? You still need to have the entry point - that's exactly where the censorship is occurring. Try to find a copy of the KickAssTorrents database on Google right now and tell me how easy it is. The approach you're talking about is essentially what http://bitcannon.io/ http://bitcannon.io/ is trying to do - but that requires running a separate app, a local MongoDB server, downloading many gigabytes of database data, and half an hour to import the torrent DB into your local instance, and that STILL requires you to be able to acquire the Torrent database or related infohash, which isn't easy because of the very censorship we're trying to sidestep. Quite frankly, that's a really dumb solution to this censorship and it's naive to say that it's useful. This will never be used in the fashion you're describing and it does nothing for anti-censorship. It makes so much more sense just to have the DHT be actually useful in surfacing content.
    - wyager 10y ago
      > You still need to have the entry point Which will now be a short hexadecimal code that is easily shared and can't be taken down.
    - sktrdie 10y ago
      If KickAss decides to share their dump using their public key, it becomes much harder to censor because of how the DHT system works. Authorities would need to shut down all the nodes that republish that public key on the DHT, which is incredibly hard to do. Much harder than asking ISPs to shutdown a domain name in DNS.
- icebraining 10y ago
  Tribler is a research project of Delft University of Technology. (...) Tribler is the first client which continuously improves upon the aging BitTorrent protocol from 2001 and addresses its flaws. We expanded it with, amongst others, streaming from magnet links, keyword search for content, channels and reputation-management. All these features are implemented in a completely distributed manner, not relying on any centralized component. Still, Tribler manages to remain fully backwards compatible with BitTorrent. https://www.tribler.org/about.html https://www.tribler.org/about.html
  - Mizza 10y ago
    I worked on an anonymous P2P research project many many years ago now, and we were shocked when we saw how much funding they received compared to us (3 mil EUR!) - but I'm super happy to see that their project is still alive and kicking! I guess the money was put to good use.
- aninhumer 10y ago
  I remember someone setting up a bittorrent search engine based on DHT scraping. It's probably been taken down, but I think it was called DHT Dig, and there's apparently a library with that name. Something like that would make it easy to create centralised search engines for casual users, and allow power users to run local databases.
  - rakoo 10y ago
    There was btdigg at some point (https://btdigg.org/ https://btdigg.org/) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that: * In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/..., slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/Wolchok/DEFCON-18-Wolchok-Crawling-Bittorrent-DHTS.pdf https://www.defcon.org/images/defcon-18/dc-18-presentations/... * In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf Basically all you have to do is sit there, people will come to you asking for content, you'll ask as a standard DHT node but note that the infohash exists, and get more information (name, number of peers, ...) about it in the background. All you need is a few IPs.
    - aninhumer 10y ago
      Ah yes, BTDigg was the name. I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.
      - rakoo 10y ago
        Yep, that's the idea: if you join the DHT network with multpile, far away ids, you get more requests coming in. Unfortunately there is BEP 42 (http://www.bittorrent.org/beps/bep_0042.html http://www.bittorrent.org/beps/bep_0042.html) which restricts the number of IDs you can have on the same IP (the restriction is there for a good reason, so it's a good think to follow it).
niftich 10y ago
This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience. This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3]. [1] https://ipfs.io/ https://ipfs.io/ [2] http://www.bittorrent.org/beps/bep_0044.html http://www.bittorrent.org/beps/bep_0044.html [3] https://gist.github.com/substack/eadd13302d785dc13aac#file-results-markdown https://gist.github.com/substack/eadd13302d785dc13aac#file-r...
- Mizza 10y ago
  IPFS still doesn't have a content discovery mechanism, which I found very disappointing. It's not a killer app without that.
  - niftich 10y ago
    Well, to be fair, neither does HTTP, short of either name-services (DNS), or crawling-and-indexing. IFPS has IPNS [1], the name services (the feature most-directly relevant to the Bittorrent enhancement in question). To date, I'm unaware of an IFPS crawler-and-indexer search engine. [1] https://github.com/ipfs/faq/issues/16 https://github.com/ipfs/faq/issues/16
    - Mizza 10y ago
      Why is that fair? They're designed for different purposes. IPFS falls far short of being useful in a transformative way because it lacks this important feature. Otherwise, it'll be a kind of gimmicky alternative to HTTP, "auto-archiving HTTP", that never got used for anything real.
      - niftich 10y ago
        I don't disagree that having content discovery would be nice, but IFPS makes a pretty good case [1] about what it brings to the table over HTTP. My point was that HTTP also has "zero out-of-the-box support" for content discovery, yet once search engines came out, we were fine. [1] https://ipfs.io/#why https://ipfs.io/#why
eximius 10y ago
Wow, the first thing I think of is that this could be a fantastic vector for malware. Anyone who implements this should make it very clear the underlying data can change.
- DarkLinkXXXX 10y ago
  To be fair, torrents were always a fantastic vector for malware.
  - pessimizer 10y ago
    To be fair, the internet has always been a fantastic vector for malware.
    - jokoon 10y ago
      To be fair, human cells has always been a fantastic vector for infection.
- pepijndevos 10y ago
  At least now you can hope that popular torrents with positive comments are maybe okay. With this the author could simply add the malware after it has become popular.
  - Senji 10y ago
    Not a problem. The tracker will mark the torrent as changed from its previous version.
- rakoo 10y ago
  You don't stumble randomly on such a mutable torrent. If you know git's data model, usual torrents are like commits, and those mutable torrents are like branches: If you give someone a commit there is trust that it won't move, and if you give them a branch name, there is expectation that the actual commit pointed to will change.
beefield 10y ago
So, what is stopping this to be the ultimate distributed social media platform? I mean, I should be able to publish a relatively simple sqlite database that contains: - my public posts - my public encryption key - list of my friends - my private posts encrypted with the friends' public keys Then I should seed at least my and my friends databases, so even if some of my friend is offline, his/hers database is still available.
- niftich 10y ago
  Network effects and lack of off-the-shelf implementations.
  - beefield 10y ago
    Network effects are clearly a major issue, but wrapping the concept to a webpage and/or mobile app should not be a major hurdle. If I have not missed something. If I only had some more time on my hands...
    - niftich 10y ago
      Although the subject matter is different, GitTorrent [1] actually uses techniques you propose, by using BEP 44 to store Git revisions in the Bittorrent Mainline DHT, and Bitcoin to store cryptographically-signed usernames in the Bitcoin Blockchain. You can look at how that implementation was done and what issues they encountered to see what it would take to implement a distributed [something], where that [something] in your case is a social network. [1] https://github.com/cjb/GitTorrent https://github.com/cjb/GitTorrent
- rakoo 10y ago
  Twister (https://en.wikipedia.org/wiki/Twister_(software) https://en.wikipedia.org/wiki/Twister_(software)) partly uses bittorrent, for long-term storage of posts and messages. It's not purely standard bittorrent, though.

117 points by sktrdie 10y ago

Mizza 10y ago

This is an okay idea, but it doesn't really solve the big problem facing BT right now: content discovery. Mutable Items are good for updating content, but that doesn't matter if nobody can find it. There needs to be a BEP for a way to host, serve and search metadata about torrents, not just their info-hashes. This should be priority #1 for BT devs. Fortunately, there is at least some related thinking in this direction: http://www.bittorrent.org/beps/bep_0044.html http://www.bittorrent.org/beps/bep_0044.html although I don't think that it really goes far enough, as there is no standardization around how the data for searching will be structured, etc.

sktrdie 10y ago
Torrent sites could publish their database dumps via this extension, and consumers would automatically download their updated index. A torrent can be anything; it can also be a list of other torrents ;) So it does help with content discovery.
- cocotino 10y ago
  An index? That could be huge. I think it's senseless. So many p2p protocols have support for p2p search (to my mind comes Kademlia), why can't BitTorrent have p2p search?
  - deleted 10y ago
    [deleted]
  - swolchok 10y ago
    Vuze (formerly known as Azureus) has (or at least had a prototype of) p2p search. I wrote a paper on building a torrent search index quickly using the DHT records that this feature emitted at the time. http://static.usenix.org/events/woot10/tech/full_papers/Wolchok.pdf http://static.usenix.org/events/woot10/tech/full_papers/Wolc...
    - sktrdie 10y ago
      Great work with this. Are you still in academia doing research?
  - niftich 10y ago
    The Mainline DHT (the more-supported DHT vs. Azureus/Vuze DHT) traditionally only contains (iirc, hashes of) torrent infohases mapped to peer lists. With BEP 44 (arbitrary data in DHT) or some other enhancement, this may change, but right now even if you traverse the DHT there's no "interesting" metadata about the keys (or, y'know, the actual content of the .torrent file) inside the DHT. From the sibling comment (posted by swolchok)'s paper, a relevant quote: "We chose to demonstrate our proof of concept only on Vuze due to the significant additional complexity of supporting two DHTs in the crawler. While Mainline does not contain torrent descriptions, its peer lists are keyed by the torrent infohashes themselves, so no additional machinery would be needed to discover infohashes. However, some method would be needed to discover torrent names, such as downloading .torrent files directly from peers using the metadata exchange protocol. To explore this possibility, we built a prototype .torrent crawler. Since we are operating in the context of the Vuze DHT, our DHT crawler needs to obtain both a torrent description and a peer list before the .torrent crawler can contact the appropriate peers with the correct infohash. However, the .torrent files contain useful additional data, including the full listing of files contained in the torrent"
  - xorcist 10y ago
    Kademlia search is also rather useless as it is flooded with false results. This is of interest not only to malware writers but also as a service to sell to rights holders.
- Mizza 10y ago
  You don't see a problem with that..? You still need to have the entry point - that's exactly where the censorship is occurring. Try to find a copy of the KickAssTorrents database on Google right now and tell me how easy it is. The approach you're talking about is essentially what http://bitcannon.io/ http://bitcannon.io/ is trying to do - but that requires running a separate app, a local MongoDB server, downloading many gigabytes of database data, and half an hour to import the torrent DB into your local instance, and that STILL requires you to be able to acquire the Torrent database or related infohash, which isn't easy because of the very censorship we're trying to sidestep. Quite frankly, that's a really dumb solution to this censorship and it's naive to say that it's useful. This will never be used in the fashion you're describing and it does nothing for anti-censorship. It makes so much more sense just to have the DHT be actually useful in surfacing content.
  - wyager 10y ago
    > You still need to have the entry point Which will now be a short hexadecimal code that is easily shared and can't be taken down.
  - sktrdie 10y ago
    If KickAss decides to share their dump using their public key, it becomes much harder to censor because of how the DHT system works. Authorities would need to shut down all the nodes that republish that public key on the DHT, which is incredibly hard to do. Much harder than asking ISPs to shutdown a domain name in DNS.
icebraining 10y ago
Tribler is a research project of Delft University of Technology. (...) Tribler is the first client which continuously improves upon the aging BitTorrent protocol from 2001 and addresses its flaws. We expanded it with, amongst others, streaming from magnet links, keyword search for content, channels and reputation-management. All these features are implemented in a completely distributed manner, not relying on any centralized component. Still, Tribler manages to remain fully backwards compatible with BitTorrent. https://www.tribler.org/about.html https://www.tribler.org/about.html
- Mizza 10y ago
  I worked on an anonymous P2P research project many many years ago now, and we were shocked when we saw how much funding they received compared to us (3 mil EUR!) - but I'm super happy to see that their project is still alive and kicking! I guess the money was put to good use.
aninhumer 10y ago
I remember someone setting up a bittorrent search engine based on DHT scraping. It's probably been taken down, but I think it was called DHT Dig, and there's apparently a library with that name. Something like that would make it easy to create centralised search engines for casual users, and allow power users to run local databases.
- rakoo 10y ago
  There was btdigg at some point (https://btdigg.org/ https://btdigg.org/) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that: * In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/..., slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/Wolchok/DEFCON-18-Wolchok-Crawling-Bittorrent-DHTS.pdf https://www.defcon.org/images/defcon-18/dc-18-presentations/... * In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf Basically all you have to do is sit there, people will come to you asking for content, you'll ask as a standard DHT node but note that the infohash exists, and get more information (name, number of peers, ...) about it in the background. All you need is a few IPs.
  - aninhumer 10y ago
    Ah yes, BTDigg was the name. I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.
    - rakoo 10y ago
      Yep, that's the idea: if you join the DHT network with multpile, far away ids, you get more requests coming in. Unfortunately there is BEP 42 (http://www.bittorrent.org/beps/bep_0042.html http://www.bittorrent.org/beps/bep_0042.html) which restricts the number of IDs you can have on the same IP (the restriction is there for a good reason, so it's a good think to follow it).

niftich 10y ago

This is an intriguing addition to Bittorrent; this kind of feature exists in IPFS [1], but of course Bittorrent has a much, much larger install base, so this progressive enhancement could bring similar or near-equivalent features to a wider audience. This implementation relies on BEP 44 (Storing arbitrary data in the DHT) [2], which, together with the design of the DHT, does have some security implications [3]. [1] https://ipfs.io/ https://ipfs.io/ [2] http://www.bittorrent.org/beps/bep_0044.html http://www.bittorrent.org/beps/bep_0044.html [3] https://gist.github.com/substack/eadd13302d785dc13aac#file-results-markdown https://gist.github.com/substack/eadd13302d785dc13aac#file-r...

Mizza 10y ago
IPFS still doesn't have a content discovery mechanism, which I found very disappointing. It's not a killer app without that.
- niftich 10y ago
  Well, to be fair, neither does HTTP, short of either name-services (DNS), or crawling-and-indexing. IFPS has IPNS [1], the name services (the feature most-directly relevant to the Bittorrent enhancement in question). To date, I'm unaware of an IFPS crawler-and-indexer search engine. [1] https://github.com/ipfs/faq/issues/16 https://github.com/ipfs/faq/issues/16
  - Mizza 10y ago
    Why is that fair? They're designed for different purposes. IPFS falls far short of being useful in a transformative way because it lacks this important feature. Otherwise, it'll be a kind of gimmicky alternative to HTTP, "auto-archiving HTTP", that never got used for anything real.
    - niftich 10y ago
      I don't disagree that having content discovery would be nice, but IFPS makes a pretty good case [1] about what it brings to the table over HTTP. My point was that HTTP also has "zero out-of-the-box support" for content discovery, yet once search engines came out, we were fine. [1] https://ipfs.io/#why https://ipfs.io/#why

eximius 10y ago

Wow, the first thing I think of is that this could be a fantastic vector for malware. Anyone who implements this should make it very clear the underlying data can change.

DarkLinkXXXX 10y ago
To be fair, torrents were always a fantastic vector for malware.
- pessimizer 10y ago
  To be fair, the internet has always been a fantastic vector for malware.
  - jokoon 10y ago
    To be fair, human cells has always been a fantastic vector for infection.
pepijndevos 10y ago
At least now you can hope that popular torrents with positive comments are maybe okay. With this the author could simply add the malware after it has become popular.
- Senji 10y ago
  Not a problem. The tracker will mark the torrent as changed from its previous version.
rakoo 10y ago
You don't stumble randomly on such a mutable torrent. If you know git's data model, usual torrents are like commits, and those mutable torrents are like branches: If you give someone a commit there is trust that it won't move, and if you give them a branch name, there is expectation that the actual commit pointed to will change.

beefield 10y ago

So, what is stopping this to be the ultimate distributed social media platform? I mean, I should be able to publish a relatively simple sqlite database that contains: - my public posts - my public encryption key - list of my friends - my private posts encrypted with the friends' public keys Then I should seed at least my and my friends databases, so even if some of my friend is offline, his/hers database is still available.

niftich 10y ago
Network effects and lack of off-the-shelf implementations.
- beefield 10y ago
  Network effects are clearly a major issue, but wrapping the concept to a webpage and/or mobile app should not be a major hurdle. If I have not missed something. If I only had some more time on my hands...
  - niftich 10y ago
    Although the subject matter is different, GitTorrent [1] actually uses techniques you propose, by using BEP 44 to store Git revisions in the Bittorrent Mainline DHT, and Bitcoin to store cryptographically-signed usernames in the Bitcoin Blockchain. You can look at how that implementation was done and what issues they encountered to see what it would take to implement a distributed [something], where that [something] in your case is a social network. [1] https://github.com/cjb/GitTorrent https://github.com/cjb/GitTorrent
rakoo 10y ago
Twister (https://en.wikipedia.org/wiki/Twister_(software) https://en.wikipedia.org/wiki/Twister_(software)) partly uses bittorrent, for long-term storage of posts and messages. It's not purely standard bittorrent, though.