5 ms·

to me it always sounds like approaches like dht are the solution but i'm having difficulties diving into it for the purpose of implementing it for my own apps.

by glasz 10y ago

to me it always sounds like approaches like dht are the solution but i'm having difficulties diving into it for the purpose of implementing it for my own apps. are there any noteworthy resources for non-academics to get started?

the8472 10y ago
Well, for an in-depth understanding you will ultimately have to read the academic papers on specific DHT algorithms, but you don't have to be an academic to read academic papers, no? Besides that there are the usual resources for higher-level overview or gleaning some details: wikipedia, protocol specifications, toy implementations on github, stack overflow, various blog posts/articles that can be found via google. But a DHT is usually just a low-level building block in more complex p2p systems. As its name says it's simply a distributed hash table. A data structure on a network. It just gives you a distributed key-value pair store where the values are often required to be small. In itself it doesn't give you trust, two-way communication, discovery or anything like that. Those are often either tacked on as ad-hoc features, handled by separate protocols or require some tricky cryptography.
jude- 10y ago
Speaking as an academic who studies distributed systems, my advise is to stay away from anything that relies on a public DHT to work correctly. They're vulnerable to node churn, Sybil attacks, and routing attacks. The last two are particularly devastating. Even if the peers had a key/value whitelist and hashes (e.g. like a .torrent file), an adversary can still insert itself into the routing tables of honest nodes and prevent peers from ever discovering your key/value pairs. Moreover, they can easily spy on everyone who tries to access them. It is estimated [1] that 300,000 of the BitTorrent DHT's nodes are Sybils, for example. [1] https://www.cl.cam.ac.uk/~lw525/publications/security.pdf https://www.cl.cam.ac.uk/~lw525/publications/security.pdf
- the8472 10y ago
  In practice none of those attacks have yet reached a level of concern for bittorrent developers to deploy serious countermeasures. Torrents generally are considered public data, especially those made available through the DHT, and provide peer exchange which allows near-complete extraction of peer lists anyway, so it hardly introduces any new privacy leaks. Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. [0] http://bittorrent.org/beps/bep_0042.html http://bittorrent.org/beps/bep_0042.html
  - jude- 10y ago
    > Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. If I'm "in" on the sharing, then I learn the IP addresses (and ISPs and proximate locations) of the other people downloading the shared file. Moreover, if I control the right hash buckets in the DHT's key space, I can learn from routing queries who's looking for the content (even if they haven't begun to share it yet). Encryption alone does not make file-sharing a private affair. > BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. It also does not appear to solve the problem. The attacker only needs to get control of hash buckets to launch routing attacks. Even with a small number of unchanging node IDs, the attacker is still free to insert a pathological sequence of key/value pairs to bump hash buckets from other nodes to them. > All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. Are you suggesting that high-value apps should not rely on a DHT, then?
    - the8472 10y ago
      > Encryption alone does not make file-sharing a private affair. Someone who is "in" on encrypted content can observe the swarm anyway, thus gains very little from performing snooping on a DHT. On the other hand a passive DHT observer who is not "in" will be hampered by not knowing what content is shared, he only sees participation in opaque hashes. Additionally payload encryption adds deniability because anyone can transfer the ciphertext but participants won't know whether others have the necessary keys to decrypt it. What I'm saying is that any information leakage via the DHT (compared to public trackers and PEX) is quite small, and this small loss can be more than made up by adding payload encryption. > the attacker is still free to insert a pathological sequence of key/value pairs to bump hash buckets from other nodes to them. There is no bumping in kademlia with unbounded node storage. And clients with limited storage can make bumping very hard for others with oldest-first and one-per-subnet policies, i.e. bumping the attackers instead of genuine keys. > Are you suggesting that high-value apps should not rely on a DHT, then? No, they should use DHT as a bootstrap mechanism of easy-to-replicate, difficult-to-disrupt small bits of information (e.g. peer contacts as in bittorrent) which then run their own content-specific gossip network for the critical content. In some contexts it can also make sense to make reverse lookups difficult, so attackers won't know what to disrupt unless they're already part of some group.
- password_reset 10y ago
  absolutely. ssb avoids using DHT for these reasons (and to prove you can build something interesting without using them) also note: DHT: hash table, BlockChain: linked list. but there are a lot more datastructures than that!
- antocv 10y ago
  What do you think of S/Kademlia, or the invite-based DHT Persea?
  - jude- 10y ago
    S/Kademlia does not solve this problem; it simply slows down the rate at which an adversary can attack the system by a small amount (i.e. by making node ID creation more expensive and increasing a key/value pair's number of replicas). There are several DHT papers that talk about bootstrapping DHTs off of social networks. They all fail to solve the Sybil problem in the same way: an adversary simply attacks the social network by pretending to be many people.

by glasz 10y ago

the8472 10y ago

Well, for an in-depth understanding you will ultimately have to read the academic papers on specific DHT algorithms, but you don't have to be an academic to read academic papers, no? Besides that there are the usual resources for higher-level overview or gleaning some details: wikipedia, protocol specifications, toy implementations on github, stack overflow, various blog posts/articles that can be found via google. But a DHT is usually just a low-level building block in more complex p2p systems. As its name says it's simply a distributed hash table. A data structure on a network. It just gives you a distributed key-value pair store where the values are often required to be small. In itself it doesn't give you trust, two-way communication, discovery or anything like that. Those are often either tacked on as ad-hoc features, handled by separate protocols or require some tricky cryptography.

jude- 10y ago

Speaking as an academic who studies distributed systems, my advise is to stay away from anything that relies on a public DHT to work correctly. They're vulnerable to node churn, Sybil attacks, and routing attacks. The last two are particularly devastating. Even if the peers had a key/value whitelist and hashes (e.g. like a .torrent file), an adversary can still insert itself into the routing tables of honest nodes and prevent peers from ever discovering your key/value pairs. Moreover, they can easily spy on everyone who tries to access them. It is estimated [1] that 300,000 of the BitTorrent DHT's nodes are Sybils, for example. [1] https://www.cl.cam.ac.uk/~lw525/publications/security.pdf https://www.cl.cam.ac.uk/~lw525/publications/security.pdf

the8472 10y ago
In practice none of those attacks have yet reached a level of concern for bittorrent developers to deploy serious countermeasures. Torrents generally are considered public data, especially those made available through the DHT, and provide peer exchange which allows near-complete extraction of peer lists anyway, so it hardly introduces any new privacy leaks. Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. [0] http://bittorrent.org/beps/bep_0042.html http://bittorrent.org/beps/bep_0042.html
- jude- 10y ago
  > Although maintaining secrecy while exchanging data over public infrastructure is desirable, that can be achieved by encrypting the payload instead of obscuring the fact that you participated in the network at all. If I'm "in" on the sharing, then I learn the IP addresses (and ISPs and proximate locations) of the other people downloading the shared file. Moreover, if I control the right hash buckets in the DHT's key space, I can learn from routing queries who's looking for the content (even if they haven't begun to share it yet). Encryption alone does not make file-sharing a private affair. > BEP42[0] has been implemented by many clients and yet nobody has felt the need to actually switching to enforcement mode. It also does not appear to solve the problem. The attacker only needs to get control of hash buckets to launch routing attacks. Even with a small number of unchanging node IDs, the attacker is still free to insert a pathological sequence of key/value pairs to bump hash buckets from other nodes to them. > All that is the result of the bittorrent DHT being a low-value target. It does not contain any juicy information and is just one of multiple peer discovery mechanisms, so there's some redundancy too. Are you suggesting that high-value apps should not rely on a DHT, then?
  - the8472 10y ago
    > Encryption alone does not make file-sharing a private affair. Someone who is "in" on encrypted content can observe the swarm anyway, thus gains very little from performing snooping on a DHT. On the other hand a passive DHT observer who is not "in" will be hampered by not knowing what content is shared, he only sees participation in opaque hashes. Additionally payload encryption adds deniability because anyone can transfer the ciphertext but participants won't know whether others have the necessary keys to decrypt it. What I'm saying is that any information leakage via the DHT (compared to public trackers and PEX) is quite small, and this small loss can be more than made up by adding payload encryption. > the attacker is still free to insert a pathological sequence of key/value pairs to bump hash buckets from other nodes to them. There is no bumping in kademlia with unbounded node storage. And clients with limited storage can make bumping very hard for others with oldest-first and one-per-subnet policies, i.e. bumping the attackers instead of genuine keys. > Are you suggesting that high-value apps should not rely on a DHT, then? No, they should use DHT as a bootstrap mechanism of easy-to-replicate, difficult-to-disrupt small bits of information (e.g. peer contacts as in bittorrent) which then run their own content-specific gossip network for the critical content. In some contexts it can also make sense to make reverse lookups difficult, so attackers won't know what to disrupt unless they're already part of some group.
password_reset 10y ago
absolutely. ssb avoids using DHT for these reasons (and to prove you can build something interesting without using them) also note: DHT: hash table, BlockChain: linked list. but there are a lot more datastructures than that!
antocv 10y ago
What do you think of S/Kademlia, or the invite-based DHT Persea?
- jude- 10y ago
  S/Kademlia does not solve this problem; it simply slows down the rate at which an adversary can attack the system by a small amount (i.e. by making node ID creation more expensive and increasing a key/value pair's number of replicas). There are several DHT papers that talk about bootstrapping DHTs off of social networks. They all fail to solve the Sybil problem in the same way: an adversary simply attacks the social network by pretending to be many people.