3 ms·
I mean, what I'm describing is the same thing that BEP51 mentions as a motivation: > DHT indexing already is possible and done in practice by passively observi
by derefr 3y ago
I mean, what I'm describing is the same thing that BEP51 mentions as a motivation:
> DHT indexing already is possible and done in practice by passively observing get_peers queries. But that approach is inefficient, favoring indexers with lots of unique IP addresses at their disposal. It also incentivizes bad behavior such as spoofing node IDs and attempting to pollute other nodes' routing tables.
If you have a lot of IP addresses (from e.g. AWS Lambda) then you can partition DHT keyspace across a large-N number of nodes and then very quickly discover everything in the keyspace.
The trick is that, since BEP51 exists, you don't need to have all these nodes register themselves into the hash-ring (at arbitrary spoofed positions) to listen. You can just have all these nodes independently probing the hash-ring "from the outside" — just making short-lived connections to registered nodes (without first registering themselves); handshaking that connection as a spoofed node ID; and then firing off one `sample_infohashes` request, getting a response, and disconnectting. The lack of registration shouldn't make any difference, as long as they don't want anyone to try connecting to them.
Which is why I say that these are just "crawler agents", not "nodes" per se. They don't start up P2P at all — to them, this is a one-shot client/server RPC conversation, like a regular web crawler making HTTP requests!
- the8472 3y agoOh, I already have implemented something[0] like that. It doesn't need lambdas or anything "cloud scale" like that. You "just" need a few dozen to a hundred IP addresses assigned to one machine and run a multi-homed DHT node on that to passively observe traffic from multiple points in the keyspace. But neither of these approaches is what I'd call a "super simple DHT crawler CLI tool" that the initial comment was asking about. BEP51 is intended to make crawling simple enough that it can run on a single home internet connection, but a proper implementation still isn't trivial. [0] https://github.com/the8472/mldht https://github.com/the8472/mldht