4 ms·

I remember someone setting up a bittorrent search engine based on DHT scraping. It's probably been taken down, but I think it was called DHT Dig, and there's ap

I remember someone setting up a bittorrent search engine based on DHT scraping. It's probably been taken down, but I think it was called DHT Dig, and there's apparently a library with that name. Something like that would make it easy to create centralised search engines for casual users, and allow power users to run local databases.

rakoo 10y ago
There was btdigg at some point (https://btdigg.org/ https://btdigg.org/) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that: * In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/..., slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/Wolchok/DEFCON-18-Wolchok-Crawling-Bittorrent-DHTS.pdf https://www.defcon.org/images/defcon-18/dc-18-presentations/... * In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf Basically all you have to do is sit there, people will come to you asking for content, you'll ask as a standard DHT node but note that the infohash exists, and get more information (name, number of peers, ...) about it in the background. All you need is a few IPs.
- aninhumer 10y ago
  Ah yes, BTDigg was the name. I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.
  - rakoo 10y ago
    Yep, that's the idea: if you join the DHT network with multpile, far away ids, you get more requests coming in. Unfortunately there is BEP 42 (http://www.bittorrent.org/beps/bep_0042.html http://www.bittorrent.org/beps/bep_0042.html) which restricts the number of IDs you can have on the same IP (the restriction is there for a good reason, so it's a good think to follow it).

rakoo 10y ago

There was btdigg at some point (https://btdigg.org/ https://btdigg.org/) but for some reason it's down today. The approach is not particularly complicated, there even were papers about that: * In Vuze: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/..., slides at https://www.defcon.org/images/defcon-18/dc-18-presentations/Wolchok/DEFCON-18-Wolchok-Crawling-Bittorrent-DHTS.pdf https://www.defcon.org/images/defcon-18/dc-18-presentations/... * In Mainline: https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf https://arxiv.org/ftp/arxiv/papers/1009/1009.3681.pdf Basically all you have to do is sit there, people will come to you asking for content, you'll ask as a standard DHT node but note that the infohash exists, and get more information (name, number of peers, ...) about it in the background. All you need is a few IPs.

aninhumer 10y ago
Ah yes, BTDigg was the name. I'd imagine to do more thorough scraping you'd want to reconnect to the network repeatedly with different peer hashes, otherwise you'd only see the infohashes "near" one point in the network.
- rakoo 10y ago
  Yep, that's the idea: if you join the DHT network with multpile, far away ids, you get more requests coming in. Unfortunately there is BEP 42 (http://www.bittorrent.org/beps/bep_0042.html http://www.bittorrent.org/beps/bep_0042.html) which restricts the number of IDs you can have on the same IP (the restriction is there for a good reason, so it's a good think to follow it).