12 ms·

> However, to get access to the DHT in the first place, you need to connect to a bootstrapping server, such as router.bittorrent.com:6881 or router.utorrent.com

by the8472 10y ago

> However, to get access to the DHT in the first place, you need to connect to a bootstrapping server, such as router.bittorrent.com:6881 or router.utorrent.com:6881 This is a common misunderstanding. You do not need to use those nodes to bootstrap. Most clients simply choose to because it is the most convenient way to do so on the given substrate (the internet). DHTs are in no way limited to specific bootstrap nodes, any node that can be contacted can be used to join the network, the protocol itself is truly distributed. If the underlying network provides some hop-limited multicast or anycast a DHT could easily bootstrap via such queries. In fact, bittorrent clients already implement multicast neighbor discovery which under some circumstances can result in joining the DHT without any hardcoded bootstrap node.

glasz 10y ago
to me it always sounds like approaches like dht are the solution but i'm having difficulties diving into it for the purpose of implementing it for my own apps. are there any noteworthy resources for non-academics to get started?
- the8472 10y ago
  Well, for an in-depth understanding you will ultimately have to read the academic papers on specific DHT algorithms, but you don't have to be an academic to read academic papers, no? Besides that there are the usual resources for higher-level overview or gleaning some details: wikipedia, protocol specifications, toy implementations on github, stack overflow, various blog posts/articles that can be found via google. But a DHT is usually just a low-level building block in more complex p2p systems. As its name says it's simply a distributed hash table. A data structure on a network. It just gives you a distributed key-value pair store where the values are often required to be small. In itself it doesn't give you trust, two-way communication, discovery or anything like that. Those are often either tacked on as ad-hoc features, handled by separate protocols or require some tricky cryptography.
- jude- 10y ago
  Speaking as an academic who studies distributed systems, my advise is to stay away from anything that relies on a public DHT to work correctly. They're vulnerable to node churn, Sybil attacks, and routing attacks. The last two are particularly devastating. Even if the peers had a key/value whitelist and hashes (e.g. like a .torrent file), an adversary can still insert itself into the routing tables of honest nodes and prevent peers from ever discovering your key/value pairs. Moreover, they can easily spy on everyone who tries to access them. It is estimated [1] that 300,000 of the BitTorrent DHT's nodes are Sybils, for example. [1] https://www.cl.cam.ac.uk/~lw525/publications/security.pdf https://www.cl.cam.ac.uk/~lw525/publications/security.pdf
  - the8472 10y ago
    In practice none of those attacks have yet reached a level of concern for bittorrent developers to deploy serious countermeasures. Torrents generally are considered public data, especially those made available through the DHT, and provide peer exchange which allows near-complete extraction of peer lists anyway, so it hardly introduces any new privacy leaks. Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. [0] http://bittorrent.org/beps/bep_0042.html http://bittorrent.org/beps/bep_0042.html
    - jude- 10y ago
      > Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. If I'm "in" on the sharing, then I learn the IP addresses (and ISPs and proximate locations) of the other people downloading the shared file. Moreover, if I control the right hash buckets in the DHT's key space, I can learn from routing queries who's looking for the content (even if they haven't begun to share it yet). Encryption alone does not make file-sharing a private affair. > BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. It also does not appear to solve the problem. The attacker only needs to get control of hash buckets to launch routing attacks. Even with a small number of unchanging node IDs, the attacker is still free to insert a pathological sequence of key/value pairs to bump hash buckets from other nodes to them. > All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. Are you suggesting that high-value apps should not rely on a DHT, then?
cjbprime 10y ago
I think you're being uncharitable in attributing a misunderstanding. The OP used the phrase "in the first place", and it's mostly correct that if you don't have any cached nodes (hence that phrase), the bootstrap nodes do in fact act as a single point of failure for you. The multicast neighbor discovery is a neat idea. I wonder what percentage of clients/connections it results in successful bootstrapping for.
- the8472 10y ago
  You can ship a client with a long list of "cached" nodes that were verified to be long-lived. I mean you need to obtain the client at some point, you can gather a fresh list of nodes along with it. From that point onward you keep your own cache fresh. You could also run your own bootstrap node on an always-up server if downtimes making the lists stale is a concern. You can also inject contacts when starting the client, you would have to obtain them out-of-band from somewhere of course, but it still does not require anything centralized. If you're desperate you could also just sweep allocated IPv4 blocks and DHT-ping port 6881, you'll probably find one relatively fast. Of course that doesn't work with v6. So there is no centralization and no single point of failure. > The multicast neighbor discovery is a neat idea. I wonder what percentage of clients/connections it results in successful bootstrapping for. It could work on a college campus, some conference network or occasionally some open wifi. Additionally there are some corporate bittorrent deployments where peer discovery via multicast can make sense. If I understand TFA correctly scuttlebutt assumes(?) roaming through wifis and LANs. Those circumstances are ideal for multicast bootstrapping, so in principle the DHT can perform just as well as scuttlebutt, probably even better because once it has bootstrapped it can use the global DHT to keep contact with the network even if there is no lan-local peer to be discovered.
  - cjbprime 10y ago
    These are excellent ideas. Are any of them implemented? If I download e.g. uTorrent today and firewall off the hardcoded public bootstrap nodes, will it bootstrap?
    - 10y ago

by the8472 10y ago

glasz 10y ago

to me it always sounds like approaches like dht are the solution but i'm having difficulties diving into it for the purpose of implementing it for my own apps. are there any noteworthy resources for non-academics to get started?

the8472 10y ago
Well, for an in-depth understanding you will ultimately have to read the academic papers on specific DHT algorithms, but you don't have to be an academic to read academic papers, no? Besides that there are the usual resources for higher-level overview or gleaning some details: wikipedia, protocol specifications, toy implementations on github, stack overflow, various blog posts/articles that can be found via google. But a DHT is usually just a low-level building block in more complex p2p systems. As its name says it's simply a distributed hash table. A data structure on a network. It just gives you a distributed key-value pair store where the values are often required to be small. In itself it doesn't give you trust, two-way communication, discovery or anything like that. Those are often either tacked on as ad-hoc features, handled by separate protocols or require some tricky cryptography.
jude- 10y ago
Speaking as an academic who studies distributed systems, my advise is to stay away from anything that relies on a public DHT to work correctly. They're vulnerable to node churn, Sybil attacks, and routing attacks. The last two are particularly devastating. Even if the peers had a key/value whitelist and hashes (e.g. like a .torrent file), an adversary can still insert itself into the routing tables of honest nodes and prevent peers from ever discovering your key/value pairs. Moreover, they can easily spy on everyone who tries to access them. It is estimated [1] that 300,000 of the BitTorrent DHT's nodes are Sybils, for example. [1] https://www.cl.cam.ac.uk/~lw525/publications/security.pdf https://www.cl.cam.ac.uk/~lw525/publications/security.pdf
- the8472 10y ago
  In practice none of those attacks have yet reached a level of concern for bittorrent developers to deploy serious countermeasures. Torrents generally are considered public data, especially those made available through the DHT, and provide peer exchange which allows near-complete extraction of peer lists anyway, so it hardly introduces any new privacy leaks. Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. [0] http://bittorrent.org/beps/bep_0042.html http://bittorrent.org/beps/bep_0042.html
  - jude- 10y ago
    > Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. If I'm "in" on the sharing, then I learn the IP addresses (and ISPs and proximate locations) of the other people downloading the shared file. Moreover, if I control the right hash buckets in the DHT's key space, I can learn from routing queries who's looking for the content (even if they haven't begun to share it yet). Encryption alone does not make file-sharing a private affair. > BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. It also does not appear to solve the problem. The attacker only needs to get control of hash buckets to launch routing attacks. Even with a small number of unchanging node IDs, the attacker is still free to insert a pathological sequence of key/value pairs to bump hash buckets from other nodes to them. > All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. Are you suggesting that high-value apps should not rely on a DHT, then?

cjbprime 10y ago

I think you're being uncharitable in attributing a misunderstanding. The OP used the phrase "in the first place", and it's mostly correct that if you don't have any cached nodes (hence that phrase), the bootstrap nodes do in fact act as a single point of failure for you. The multicast neighbor discovery is a neat idea. I wonder what percentage of clients/connections it results in successful bootstrapping for.

the8472 10y ago
You can ship a client with a long list of "cached" nodes that were verified to be long-lived. I mean you need to obtain the client at some point, you can gather a fresh list of nodes along with it. From that point onward you keep your own cache fresh. You could also run your own bootstrap node on an always-up server if downtimes making the lists stale is a concern. You can also inject contacts when starting the client, you would have to obtain them out-of-band from somewhere of course, but it still does not require anything centralized. If you're desperate you could also just sweep allocated IPv4 blocks and DHT-ping port 6881, you'll probably find one relatively fast. Of course that doesn't work with v6. So there is no centralization and no single point of failure. > The multicast neighbor discovery is a neat idea. I wonder what percentage of clients/connections it results in successful bootstrapping for. It could work on a college campus, some conference network or occasionally some open wifi. Additionally there are some corporate bittorrent deployments where peer discovery via multicast can make sense. If I understand TFA correctly scuttlebutt assumes(?) roaming through wifis and LANs. Those circumstances are ideal for multicast bootstrapping, so in principle the DHT can perform just as well as scuttlebutt, probably even better because once it has bootstrapped it can use the global DHT to keep contact with the network even if there is no lan-local peer to be discovered.
- cjbprime 10y ago
  These are excellent ideas. Are any of them implemented? If I download e.g. uTorrent today and firewall off the hardcoded public bootstrap nodes, will it bootstrap?
  - 10y ago