7 ms·

Show HN: Magnetico – self-hosted BitTorrent DHT search engine suite

95 points by boramalper 10y ago

tscs37 10y ago
This looks rather nice though two things seem to be missing: * A docker-compose file for easy testing * Screenshots of the WebUI
- boramalper 10y ago
  I have no experience with docker so I couldn't (nor can) create docker-compose file but the installation is pretty straightforward: pip3 install magneticod magneticow --user Also, I added some screenshots, thanks for the suggestion! =)
  - helb 10y ago
    "Dockerizing" Python apps is pretty straightforward, too. This might help if you ever decide to try it: https://runnable.com/docker/python/dockerize-your-python-application https://runnable.com/docker/python/dockerize-your-python-app...
- simonvc 10y ago
  Here ya go, i made a simple Dockerfile that works for me. https://gist.github.com/simonvc/8fb20a5823955645806ff0783c75f37f https://gist.github.com/simonvc/8fb20a5823955645806ff0783c75...
  - tscs37 10y ago
    Thanks a lot!
wut42 10y ago
Nice project! Last year I made a similar PoC on Elixir based on the paper "Crawling DHTs for fun and profit"[1]. In that paper the researcher uses a pool of IPs to introduce multiple nodes (to cover the hash space). On my PoC I used less IPs but multiple ports per IP. How does this project manages to cover that ? [1]: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.172.8202 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.172....
- boramalper 10y ago
  I too read that paper, but magnetico crawls Mainline DHT (which is the de facto standard) whilst the paper is on Vuze DHT, though the idea is more or less the same. This project uses only one instance of crawler running at the same time, so one IP and one port for crawling the DHT. Whenever it receives a `GET_PEERS` query from another peer in the network, it forges itself a new ID that has the first 15 bytes in common with the info hash in the query, so that the querying node would think that it found the closest (or a close enough) node to announce. After responding back, most of the times the same querying node sends a `ANNOUNCE_PEER` query to register itself as a peer downloading that particular torrent. Upon receiving `ANNOUNCE_PEER` query, magneticod then connects the peer using BitTorrent (TCP) protocol, receives the complete metadata, and closes the connection. Keep repeating these and you'll end up with thousands of torrents at the end of each day. Possibly, utilizing multiple IPs might increase the throughput and prevent IP based bans, but I didn't bother to be honest, as the current solution seems to work well enough.
barrystaes 10y ago
Its not really decentralized. Readme says: " magnetico liberates BitTorrent from the yoke of centralised trackers & web-sites and makes it truly decentralised. Finally! " Source says: ( https://github.com/boramalper/magnetico/blob/42b2d64d200e2aff106eb0ce4241d7bcc411ceb5/magneticod/magneticod/dht.py#L31-L34 https://github.com/boramalper/magnetico/blob/42b2d64d200e2af... ) BOOTSTRAPPING_NODES = [ ("router.bittorrent.com", 6881), ("dht.transmissionbt.com", 6881) ] Also, i think its going to be very difficult to find anything in the DHT (spam) chaos without some of the pre-filtering that exists by using torrent sites. Fun project though..
- setra 10y ago
  >Its not really decentralized Every DHT has to be bootstrapped from something. If that makes it not distributed then i'm not sure anything is. You can use whatever bootstrap nodes you want by changing the values.
  - barrystaes 10y ago
    Readme says "truly decentralised" which is the point, not "not distributed". It certainly is distributed, just not truly. It requires bootstrapping with prefixed domain names. Bootstrapping is not bad (when done securely), but the false claim is.
    - antocv 10y ago
      Please explain how you would do truly decentralized bootstrapping?
    - boramalper 10y ago
      Well, it is decentralised but not truly/completely distributed.[0] I guess a distributed system would allow clients to search the network right from the querying machine instead of a server. The adjective truly meant that you can actually start using it right away. Starting a traditional torrent website/tracker is not hard and there must be open source implementations as well, but the problem is (as in all social systems), they suffer from the trap of initialisation: there is no content because you have no users, and there are no users because you don't have anything to offer them, and so on. magnetico discovers the torrents so it's a truly decentralised solution that you can actually start using right away. [0]: https://commons.wikimedia.org/wiki/File:Centralised-decentralised-distributed.png https://commons.wikimedia.org/wiki/File:Centralised-decentra...
the8472 10y ago
Skimming over the code it seems like it's doing fairly egregious bittorrent spec violations. In other words its very selfish and other clients may blacklist it if they detect such behavior.
- boramalper 10y ago
  > Skimming over the code it seems like it's doing fairly egregious bittorrent spec violations. It's true and even understated: magneticod is not only selfish but also incredibly aggressive. It will literally contact a thousand (and if your network allows, even more) nodes every single second and deceive them by forging thousands of ids to create the impression that it is "very close" (in terms of XOR-distance used in DHT Kademlia) to the every single node. Please have a look at this website (although it is a bit old): https://dsn.tm.kit.edu/english/2936.php https://dsn.tm.kit.edu/english/2936.php There are on average 7,000,000 DHT peers in the network and less than a quarter of the nodes stay longer than 12 hours in the network (i.e. the node you contacted this morning might be long gone at the evening). There are quite a few public papers[0] describing various Sybil-attacks on BitTorrent DHT, so it's not something unknown. BEP 42[1] tries to mitigate the issue by making node IDs dependent on the IP address of the node, but as far as I know not many clients (if any) implements it. I spoke too much but one last point: These attacks -given the scale of BitTorrent DHT- I doubt to be disruptive, but they are more dangerous for the privacy of the peers. One can gather thousands of IP addresses each day as easily using the same technique. (See http://opentracker.blog.h3q.com/2007/02/12/perfect-deniability/ http://opentracker.blog.h3q.com/2007/02/12/perfect-deniabili...) [0]: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/... [1]: http://www.bittorrent.org/beps/bep_0042.html http://www.bittorrent.org/beps/bep_0042.html
  - the8472 10y ago
    I'm not sure what you're trying to argue here. That it's ok to misbehave because there are papers describing how to do it? That it's ok to be selfish because there are enough compliant nodes to compensate for it?
NoGravitas 10y ago
A couple of thoughts on this. 1. I assume it is "dangerous" to run this on an IP address that is easily traceable back to you, since it claims, to various peers, to be downloading all the things, even though it doesn't. Is this correct? 2. A site, skytorrents.in, was posted to Show HN a while ago, that appears to be similar to a centrally hosted version of this, in that it builds its index of magnet links by trawling the DHT rather than relying on user submissions.
- c3833174 10y ago
  1. I'd expect any decent logging system to actually attempt transferring a file before marking an IP 2. BTDigg is/was the most known one, dozens of other followed.
  - number6 10y ago
    I expect the content mafia just to sue whoever they can

95 points by boramalper 10y ago

This looks rather nice though two things seem to be missing: * A docker-compose file for easy testing * Screenshots of the WebUI

boramalper 10y ago
I have no experience with docker so I couldn't (nor can) create docker-compose file but the installation is pretty straightforward: pip3 install magneticod magneticow --user Also, I added some screenshots, thanks for the suggestion! =)
- helb 10y ago
  "Dockerizing" Python apps is pretty straightforward, too. This might help if you ever decide to try it: https://runnable.com/docker/python/dockerize-your-python-application https://runnable.com/docker/python/dockerize-your-python-app...
simonvc 10y ago
Here ya go, i made a simple Dockerfile that works for me. https://gist.github.com/simonvc/8fb20a5823955645806ff0783c75f37f https://gist.github.com/simonvc/8fb20a5823955645806ff0783c75...
- tscs37 10y ago
  Thanks a lot!

wut42 10y ago

Nice project! Last year I made a similar PoC on Elixir based on the paper "Crawling DHTs for fun and profit"[1]. In that paper the researcher uses a pool of IPs to introduce multiple nodes (to cover the hash space). On my PoC I used less IPs but multiple ports per IP. How does this project manages to cover that ? [1]: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.172.8202 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.172....

boramalper 10y ago
I too read that paper, but magnetico crawls Mainline DHT (which is the de facto standard) whilst the paper is on Vuze DHT, though the idea is more or less the same. This project uses only one instance of crawler running at the same time, so one IP and one port for crawling the DHT. Whenever it receives a `GET_PEERS` query from another peer in the network, it forges itself a new ID that has the first 15 bytes in common with the info hash in the query, so that the querying node would think that it found the closest (or a close enough) node to announce. After responding back, most of the times the same querying node sends a `ANNOUNCE_PEER` query to register itself as a peer downloading that particular torrent. Upon receiving `ANNOUNCE_PEER` query, magneticod then connects the peer using BitTorrent (TCP) protocol, receives the complete metadata, and closes the connection. Keep repeating these and you'll end up with thousands of torrents at the end of each day. Possibly, utilizing multiple IPs might increase the throughput and prevent IP based bans, but I didn't bother to be honest, as the current solution seems to work well enough.

barrystaes 10y ago

Its not really decentralized. Readme says: " magnetico liberates BitTorrent from the yoke of centralised trackers & web-sites and makes it truly decentralised. Finally! " Source says: ( https://github.com/boramalper/magnetico/blob/42b2d64d200e2aff106eb0ce4241d7bcc411ceb5/magneticod/magneticod/dht.py#L31-L34 https://github.com/boramalper/magnetico/blob/42b2d64d200e2af... ) BOOTSTRAPPING_NODES = [ ("router.bittorrent.com", 6881), ("dht.transmissionbt.com", 6881) ] Also, i think its going to be very difficult to find anything in the DHT (spam) chaos without some of the pre-filtering that exists by using torrent sites. Fun project though..

setra 10y ago
>Its not really decentralized Every DHT has to be bootstrapped from something. If that makes it not distributed then i'm not sure anything is. You can use whatever bootstrap nodes you want by changing the values.
- barrystaes 10y ago
  Readme says "truly decentralised" which is the point, not "not distributed". It certainly is distributed, just not truly. It requires bootstrapping with prefixed domain names. Bootstrapping is not bad (when done securely), but the false claim is.
  - antocv 10y ago
    Please explain how you would do truly decentralized bootstrapping?
  - boramalper 10y ago
    Well, it is decentralised but not truly/completely distributed.[0] I guess a distributed system would allow clients to search the network right from the querying machine instead of a server. The adjective truly meant that you can actually start using it right away. Starting a traditional torrent website/tracker is not hard and there must be open source implementations as well, but the problem is (as in all social systems), they suffer from the trap of initialisation: there is no content because you have no users, and there are no users because you don't have anything to offer them, and so on. magnetico discovers the torrents so it's a truly decentralised solution that you can actually start using right away. [0]: https://commons.wikimedia.org/wiki/File:Centralised-decentralised-distributed.png https://commons.wikimedia.org/wiki/File:Centralised-decentra...

the8472 10y ago

Skimming over the code it seems like it's doing fairly egregious bittorrent spec violations. In other words its very selfish and other clients may blacklist it if they detect such behavior.

boramalper 10y ago
> Skimming over the code it seems like it's doing fairly egregious bittorrent spec violations. It's true and even understated: magneticod is not only selfish but also incredibly aggressive. It will literally contact a thousand (and if your network allows, even more) nodes every single second and deceive them by forging thousands of ids to create the impression that it is "very close" (in terms of XOR-distance used in DHT Kademlia) to the every single node. Please have a look at this website (although it is a bit old): https://dsn.tm.kit.edu/english/2936.php https://dsn.tm.kit.edu/english/2936.php There are on average 7,000,000 DHT peers in the network and less than a quarter of the nodes stay longer than 12 hours in the network (i.e. the node you contacted this morning might be long gone at the evening). There are quite a few public papers[0] describing various Sybil-attacks on BitTorrent DHT, so it's not something unknown. BEP 42[1] tries to mitigate the issue by making node IDs dependent on the IP address of the node, but as far as I know not many clients (if any) implements it. I spoke too much but one last point: These attacks -given the scale of BitTorrent DHT- I doubt to be disruptive, but they are more dangerous for the privacy of the peers. One can gather thousands of IP addresses each day as easily using the same technique. (See http://opentracker.blog.h3q.com/2007/02/12/perfect-deniability/ http://opentracker.blog.h3q.com/2007/02/12/perfect-deniabili...) [0]: https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf https://www.usenix.org/legacy/event/woot10/tech/full_papers/... [1]: http://www.bittorrent.org/beps/bep_0042.html http://www.bittorrent.org/beps/bep_0042.html
- the8472 10y ago
  I'm not sure what you're trying to argue here. That it's ok to misbehave because there are papers describing how to do it? That it's ok to be selfish because there are enough compliant nodes to compensate for it?

NoGravitas 10y ago

A couple of thoughts on this. 1. I assume it is "dangerous" to run this on an IP address that is easily traceable back to you, since it claims, to various peers, to be downloading all the things, even though it doesn't. Is this correct? 2. A site, skytorrents.in, was posted to Show HN a while ago, that appears to be similar to a centrally hosted version of this, in that it builds its index of magnet links by trawling the DHT rather than relying on user submissions.

c3833174 10y ago
1. I'd expect any decent logging system to actually attempt transferring a file before marking an IP 2. BTDigg is/was the most known one, dozens of other followed.
- number6 10y ago
  I expect the content mafia just to sue whoever they can