4 ms·

> That it's ok to be selfish because there are enough compliant nodes to compensate for it? Yes. Of course, I do not have data nor the foresight to claim that

> That it's ok to be selfish because there are enough compliant nodes to compensate for it? Yes. Of course, I do not have data nor the foresight to claim that its short or long term effects are negligible, but I really doubt if it's not the case. > That it's ok to misbehave because there are papers describing how to do it? Not because of that, but I was pointing out that the technique is nothing new, and probably numerous real attacks of much greater scale (with the intention of actually disrupting the network) using the same techniques must have been launched many times before magnetico. If BitTorrent DHT survived those, surely it is more resilient than you tend to think. ---- I would love to hear the opinion of a BitTorrent developer though. I'll try to e-mail them tonight and ask for their opinion. =)

the8472 10y ago
> Yes. Of course, I do not have data nor the foresight to claim that its short or long term effects are negligible, but I really doubt if it's not the case. And what if people see your solution and try to build an end-user thing out of it so that millions of users could run it at home? When building something in a P2P network you always should consider whether it would still work if the majority of the network behaved like that. > If BitTorrent DHT survived those, surely it is more resilient than you tend to think. Yes, the rain forest is also quite resilient. Nothing wrong with logging a few more trees, right? I'm saying your kind of reasoning is responsible for tragedy of the commons outcomes. You're not even trying to make your nodes a good citizen in the network because "others do it too". -- Anyway, such behavior can be detected and if it becomes more common other clients will probably implement blacklisting for it.
- boramalper 10y ago
  > DHT indexing already is possible and done in practice by passively observing get_peers queries. But that approach is inefficient, favoring indexers with lots of unique IP addresses at their disposal. It also incentivizes bad behavior such as spoofing node IDs and attempting to pollute other nodes' routing tables. Source: DHT Infohash Indexing < http://www.bittorrent.org/beps/bep_0051.html http://www.bittorrent.org/beps/bep_0051.html > Figurative speech aside, bittorrent.org acknowledges the need & the issue, and calls it bad, not "egregious". The paper[0] mentioned in the BEP 51 proposal describes more or less how magnetico works ("horizontal attack") and states in conclusion that > Through an extensive measurement study since December 2010, we have identified that both of these attacks are happening in the real network. We have analyzed their exact behavior through honeypots and have shown the scale of the on-going activities. We must stress that we have no concrete proof of actual malicious activities; our work only shows that the scale of attacks is large enough for this to be a concern. Repeating for the last time (as I feel this slowly creeps into a flame war): 1. As I have said, these technique is already in use. This means that the network is more resilient than you claim and the chances that a few hundred (or thousand, in future) magnetico instances will harm the network is near zero. I could answer in a more direct way if you did not use rain forest metaphor and addressed the issues you see more directly. 2. magnetico uses a similar but also a lot different technique. The horizontal (Sybil) attack requires the offending node to maintain its identity: "First, the attacker only has to answer two types of messages, `PING` and `FIND_NODE`; he can ignore every other message."[0] magneticod, in contrast, answers only `GET_PEERS` and `ANNOUNCE_PEER` queries, which are essential for the remote node to register itself as a peer downloading the torrent. magneticod will send hundreds of `FIND_NODE` queries per second[1] using forged node IDs, and that's all! According to the BitTorrent DHT protocol[2], "after 15 minutes of inactivity, a node becomes questionable." Hence, presumably, magneticod should be forgotten & removed from the routing table after it does not answer several `PING` queries. Consider that `FIND_NODE` queries that magnetico makes are targetting (at every single time) totally random nodes, so same set of nodes will not be targeted ever (statistically speaking). The problem is, BEP 51 is still a draft and AFAIK no clients in the wild implements it, so it's simply impractical to write a DHT search-engine depending on this particular behaviour. Once it becomes more widespread (hopefully together with BEP 42), magnetico will (and has to) drop its current technique and support the new and proper one. [0]: https://www.cl.cam.ac.uk/~lw525/publications/security.pdf https://www.cl.cam.ac.uk/~lw525/publications/security.pdf [1]: https://github.com/boramalper/magnetico/blob/257912a12f862d9f4ca3c5970df57f3a62d29c66/magneticod/magneticod/dht.py#L225 https://github.com/boramalper/magnetico/blob/257912a12f862d9... [2]: http://www.bittorrent.org/beps/bep_0005.html http://www.bittorrent.org/beps/bep_0005.html
  - the8472 10y ago
    > bittorrent.org acknowledges the need & the issue It does not acknowledge or imply that what you are doing (forged replies) is necessary. It only acknowledges that indexing happens in the wild and that when it is happening nodes are currently incentivized to misbehave, that does not mean they need to misbehave to achieve their goals. > and calls it bad, not "egregious". That's nitpicking words, a spec is obviously written in a more conservative tone. Your implementation is far outside specs, intentionally so to manipulate the behavior of other clients. > This means that the network is more resilient than you claim and the chances that a few hundred (or thousand, in future) magnetico instances will harm the network is near zero. I have not denied that the network has some resilience or claimed that a single instance of this code running will destroy the network. I am saying that your code is misbehaving and greedy, which makes it a bad citizen in a p2p environment. > BEP 51 is still a draft Hardly relevant, many widely deployed BEPs are in draft status. > AFAIK no clients in the wild implements it That's not entirely true, but even assuming that it is it is not relevant either because BEP51 is not the only approach to be a good citizen while indexing the DHT.
  - rakoo 10y ago
    Probably a bit late to the party, but: > Once [BEP51] becomes more widespread (hopefully together with BEP 42), magnetico will (and has to) drop its current technique and support the new and proper one. Why not be part of the solution then ? Make Magnetico respond to BEP51 requests properly, and you'll help migrate the DHT towards this.
- c3833174 10y ago
  You haven't taken into account the resources required to do so. 1) Open ports, alredy rules out a large part of those millions 2) Willingness to sacrifice bandwidth for something that will still take a long time to populate a large enough database when run on a home ISP connection 3) The average user will stop using it when too much UDP crashes the router 4) The "Crawling BitTorrent DHTs for Fun and Profit" paper only takes into account the Vuze-only DHT 5) A completely distributed torrent database system with a pretty nice search function like Tribler alredy exists and doesn't seem to be that popular.
  - the8472 10y ago
    I am using "does this approach work if every node in the network behaves like this" as a yard stick to gauge whether some p2p node is a good implementation. You're not just running the software on your own machine, you're running in a network with everyone. If the network would fail if a specific implementation becomes dominant then it's badly engineered. But that's just the point where harm becomes obvious. That doesn't mean there's no harm when they're only a minority of nodes. They're still making things less efficient for other nodes, drain resources or might make lookups more unreliable. This might make the network less attractive for end-users for example. So the extrapolation simply helps making an already-present net drain on the network resources more apparent.
    - belorn 10y ago
      Could you explain the harm in a bit more details? How many false identities (that don't respond) is needed to flood the routing table and cause a noticeable impact, and further, to completely make a resources inaccessible? Are we talking about 1:1 for each fake vs real, 10^2:1, or does it more or less need to fill up limited resources such as memory or network?
      - the8472 10y ago
        I'm not sure if you have understood my argument. The precise ratio at which it becomes noticeable or at which things start breaking down is not relevant to gauging whether an implementation is well-behaved or not. Fundamentally a p2p network can only work if averaged over the nodes the following holds: resources consumed <= resources provided. If everyone tries to consume more than they provide (possibly zero) then the network does not work. This can be extended a bit further by also looking at whether some node behavior impacts how much other nodes consume. Ultimately means selfish nodes are harmful to the network on a fundamental level. They simply cannot satisfy that inequality. Even a single misbehaving node will inflict an - admittedly miniscule - amount of harm on the network. It's akin to stealing a dollar from your wallet, it probably won't even impact your day. And just like stealing dollars gets worse and worse as you take more of them from people the noticeable impact in a network gets worse and worse as the fraction of misbehaving nodes increases. It starts with wasting everyone's bandwidth (metered internet!), then increases the latency of lookups because you need to visit more hops and then less strict implementations start to fail lookups. The most strict, defensive implementations gradually failing more and more of their lookups is just the last and most visible kind of harm. To give a more concrete example, I have observed utorrent and libtorrent DHT implementations having polluted routing tables near their home buckets since they don't sanitize fake nodes aggressively enough. This adds maybe a dozen entries to their routing tables, it's not much. But any lookup coming their way will experience significantly increased latency because the terminal phase of a lookup should be exhaustive, so an implementation has to try all the bogus entries. There are counter-measures of course, but they increase implementation complexity and result more maintenance traffic than otherwise necessary. Those fake nodes come from only a few thousand IPs and yet they cause a negative impact for everyone because they have indirect effects. So in the real world the impact depends on a) fraction of bad nodes b) aggressiveness of bad nodes (not all bad behavior has the same impact) c) implementation quality of those who are trying to be good citizens.

the8472 10y ago

> Yes. Of course, I do not have data nor the foresight to claim that its short or long term effects are negligible, but I really doubt if it's not the case. And what if people see your solution and try to build an end-user thing out of it so that millions of users could run it at home? When building something in a P2P network you always should consider whether it would still work if the majority of the network behaved like that. > If BitTorrent DHT survived those, surely it is more resilient than you tend to think. Yes, the rain forest is also quite resilient. Nothing wrong with logging a few more trees, right? I'm saying your kind of reasoning is responsible for tragedy of the commons outcomes. You're not even trying to make your nodes a good citizen in the network because "others do it too". -- Anyway, such behavior can be detected and if it becomes more common other clients will probably implement blacklisting for it.

boramalper 10y ago
> DHT indexing already is possible and done in practice by passively observing get_peers queries. But that approach is inefficient, favoring indexers with lots of unique IP addresses at their disposal. It also incentivizes bad behavior such as spoofing node IDs and attempting to pollute other nodes' routing tables. Source: DHT Infohash Indexing < http://www.bittorrent.org/beps/bep_0051.html http://www.bittorrent.org/beps/bep_0051.html > Figurative speech aside, bittorrent.org acknowledges the need & the issue, and calls it bad, not "egregious". The paper[0] mentioned in the BEP 51 proposal describes more or less how magnetico works ("horizontal attack") and states in conclusion that > Through an extensive measurement study since December 2010, we have identified that both of these attacks are happening in the real network. We have analyzed their exact behavior through honeypots and have shown the scale of the on-going activities. We must stress that we have no concrete proof of actual malicious activities; our work only shows that the scale of attacks is large enough for this to be a concern. Repeating for the last time (as I feel this slowly creeps into a flame war): 1. As I have said, these technique is already in use. This means that the network is more resilient than you claim and the chances that a few hundred (or thousand, in future) magnetico instances will harm the network is near zero. I could answer in a more direct way if you did not use rain forest metaphor and addressed the issues you see more directly. 2. magnetico uses a similar but also a lot different technique. The horizontal (Sybil) attack requires the offending node to maintain its identity: "First, the attacker only has to answer two types of messages, `PING` and `FIND_NODE`; he can ignore every other message."[0] magneticod, in contrast, answers only `GET_PEERS` and `ANNOUNCE_PEER` queries, which are essential for the remote node to register itself as a peer downloading the torrent. magneticod will send hundreds of `FIND_NODE` queries per second[1] using forged node IDs, and that's all! According to the BitTorrent DHT protocol[2], "after 15 minutes of inactivity, a node becomes questionable." Hence, presumably, magneticod should be forgotten & removed from the routing table after it does not answer several `PING` queries. Consider that `FIND_NODE` queries that magnetico makes are targetting (at every single time) totally random nodes, so same set of nodes will not be targeted ever (statistically speaking). The problem is, BEP 51 is still a draft and AFAIK no clients in the wild implements it, so it's simply impractical to write a DHT search-engine depending on this particular behaviour. Once it becomes more widespread (hopefully together with BEP 42), magnetico will (and has to) drop its current technique and support the new and proper one. [0]: https://www.cl.cam.ac.uk/~lw525/publications/security.pdf https://www.cl.cam.ac.uk/~lw525/publications/security.pdf [1]: https://github.com/boramalper/magnetico/blob/257912a12f862d9f4ca3c5970df57f3a62d29c66/magneticod/magneticod/dht.py#L225 https://github.com/boramalper/magnetico/blob/257912a12f862d9... [2]: http://www.bittorrent.org/beps/bep_0005.html http://www.bittorrent.org/beps/bep_0005.html
- the8472 10y ago
  > bittorrent.org acknowledges the need & the issue It does not acknowledge or imply that what you are doing (forged replies) is necessary. It only acknowledges that indexing happens in the wild and that when it is happening nodes are currently incentivized to misbehave, that does not mean they need to misbehave to achieve their goals. > and calls it bad, not "egregious". That's nitpicking words, a spec is obviously written in a more conservative tone. Your implementation is far outside specs, intentionally so to manipulate the behavior of other clients. > This means that the network is more resilient than you claim and the chances that a few hundred (or thousand, in future) magnetico instances will harm the network is near zero. I have not denied that the network has some resilience or claimed that a single instance of this code running will destroy the network. I am saying that your code is misbehaving and greedy, which makes it a bad citizen in a p2p environment. > BEP 51 is still a draft Hardly relevant, many widely deployed BEPs are in draft status. > AFAIK no clients in the wild implements it That's not entirely true, but even assuming that it is it is not relevant either because BEP51 is not the only approach to be a good citizen while indexing the DHT.
- rakoo 10y ago
  Probably a bit late to the party, but: > Once [BEP51] becomes more widespread (hopefully together with BEP 42), magnetico will (and has to) drop its current technique and support the new and proper one. Why not be part of the solution then ? Make Magnetico respond to BEP51 requests properly, and you'll help migrate the DHT towards this.
c3833174 10y ago
You haven't taken into account the resources required to do so. 1) Open ports, alredy rules out a large part of those millions 2) Willingness to sacrifice bandwidth for something that will still take a long time to populate a large enough database when run on a home ISP connection 3) The average user will stop using it when too much UDP crashes the router 4) The "Crawling BitTorrent DHTs for Fun and Profit" paper only takes into account the Vuze-only DHT 5) A completely distributed torrent database system with a pretty nice search function like Tribler alredy exists and doesn't seem to be that popular.
- the8472 10y ago
  I am using "does this approach work if every node in the network behaves like this" as a yard stick to gauge whether some p2p node is a good implementation. You're not just running the software on your own machine, you're running in a network with everyone. If the network would fail if a specific implementation becomes dominant then it's badly engineered. But that's just the point where harm becomes obvious. That doesn't mean there's no harm when they're only a minority of nodes. They're still making things less efficient for other nodes, drain resources or might make lookups more unreliable. This might make the network less attractive for end-users for example. So the extrapolation simply helps making an already-present net drain on the network resources more apparent.
  - belorn 10y ago
    Could you explain the harm in a bit more details? How many false identities (that don't respond) is needed to flood the routing table and cause a noticeable impact, and further, to completely make a resources inaccessible? Are we talking about 1:1 for each fake vs real, 10^2:1, or does it more or less need to fill up limited resources such as memory or network?
    - the8472 10y ago
      I'm not sure if you have understood my argument. The precise ratio at which it becomes noticeable or at which things start breaking down is not relevant to gauging whether an implementation is well-behaved or not. Fundamentally a p2p network can only work if averaged over the nodes the following holds: resources consumed <= resources provided. If everyone tries to consume more than they provide (possibly zero) then the network does not work. This can be extended a bit further by also looking at whether some node behavior impacts how much other nodes consume. Ultimately means selfish nodes are harmful to the network on a fundamental level. They simply cannot satisfy that inequality. Even a single misbehaving node will inflict an - admittedly miniscule - amount of harm on the network. It's akin to stealing a dollar from your wallet, it probably won't even impact your day. And just like stealing dollars gets worse and worse as you take more of them from people the noticeable impact in a network gets worse and worse as the fraction of misbehaving nodes increases. It starts with wasting everyone's bandwidth (metered internet!), then increases the latency of lookups because you need to visit more hops and then less strict implementations start to fail lookups. The most strict, defensive implementations gradually failing more and more of their lookups is just the last and most visible kind of harm. To give a more concrete example, I have observed utorrent and libtorrent DHT implementations having polluted routing tables near their home buckets since they don't sanitize fake nodes aggressively enough. This adds maybe a dozen entries to their routing tables, it's not much. But any lookup coming their way will experience significantly increased latency because the terminal phase of a lookup should be exhaustive, so an implementation has to try all the bogus entries. There are counter-measures of course, but they increase implementation complexity and result more maintenance traffic than otherwise necessary. Those fake nodes come from only a few thousand IPs and yet they cause a negative impact for everyone because they have indirect effects. So in the real world the impact depends on a) fraction of bad nodes b) aggressiveness of bad nodes (not all bad behavior has the same impact) c) implementation quality of those who are trying to be good citizens.