10 ms·

Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent search

497 points by KoftaBob 3y ago

johng 3y ago
I've been running this for about 5 days in Docker. I've had to restart bitmagnet a handful of times as it seems to crash. The web interface is available but the number of torrents indexed never increases. I'm currently at 180,744 indexed and I'm waiting to see how high it goes. I check on it a few times a day.
thelastparadise 3y ago
Very interesting. Does this approach actually work in practice? Also what happens if illegal content gets scooped up into the index?
- KoftaBob 3y ago
  Based on my understanding of how the torrent DHT works, all that’s happening is that youre requesting metadata on various torrent info hashes, but that’s not the same thing as actually downloading/seeding the content in the torrent itself.
  - no_time 3y ago
    >but that’s not the same thing as actually downloading/seeding the content in the torrent itself. The question is whether Law Enforcement and "Intellectual Property" watchdogs make a meaningful distinction between the two in their monitoring tools.
    - justinclift 3y ago
      Seeing as they get so much other stuff wrong, it's highly likely they don't care.
- Akashic101 3y ago
  Considering the screenshot on the linked page shows The Flash among other movies I dont think the author is too concerned about that. I believe its similar to the area that Plex and Jellyfin operate in, mainly that they just provide the framework and tools, what the user does with them is not in their control
  - EGreg 3y ago
    How will governments police it when MaidSAFE and other systems distribute totally encrypted content?
    - colinsane 3y ago
      BitTorrent already supports encrypted peer connections.
      - mixmastamyk 3y ago
        Doesn’t seem to work, our ISP has sent nastygrams despite this and site on https.
        colinsane 3y ago
        > How will governments police it when MaidSAFE and other systems distribute totally encrypted content? so isn't the answer then "they'll continue to police it the way they already do"? i don't know what a MaidSAFE is, but the context of this discussion is the DHT, and so public (indexable) torrents, and so however you encrypt the content doesn't matter because you have to provide the decryption method to anyone who asks for any of the previous context (public torrents/indexes) to make any sense.
        mixmastamyk 3y ago
        Encrypted connections shouldn’t decrypt for anyone who asks, otherwise they have no reason to exist. The weak point seems to be the tracker or filename, but been told https hides that so not sure.
- lobsterslive 3y ago
  It works well in practice. The DHT protocol includes announce messages that broadcast when new files are shared on BitTorrent. It then includes a "geometric" way to find people who are sharing those files. It doesn't include the files themselves, just the torrents which include a file list and location hashes. If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over time), this will include commercial movies and such.
  - grepfru_it 3y ago
    >It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually hours later) that your supposed pirated download is actually just a 2hour loop of Rick Astley (still piracy though, so you are still winning.. i think?).
    - SparkyMcUnicorn 3y ago
      But you can still pick the option with the most seeders, which should get you what you're looking for most of the time. The spam problem isn't nonexistent within the centralized services either.
      - grepfru_it 3y ago
        Hehe in a popular P2P client from the '03-'05 period, we said the same thing. Turns out there are groups with large amounts of funding which will provide a fake seed count. Either just faking metadata making it seem there was a high seed count but bogus nodes which would refuse connections (which was actual behavior from clients with bad ISPs - which we saw valid cases in asia or east europe) or would actually stream data (and some of them were on good hosts seeding multi mbps of bad data) What i'm saying is it becomes a numbers game and those fake seeders usually have deep pockets financed by the content creators themselves
- quickthrower2 3y ago
  I think illegal (unlawful?) content is its raison d’etre. With the odd exceptions of thinks like linux distros, LLM weights etc. to take pressure off centralized servers.
fallat 3y ago
It'd be so nice to have a super simple DHT crawler CLI tool, in both implementation and interface.
- the8472 3y ago
  These things need uptime of hours and days to do it properly and also to stay up to date. There are millions of nodes and torrents and to be non-abusive you have to issue requests at a somewhat sedate pace. And activity kind of moves with the sun due to people who run torrent clients on their home machines. And there are lots of buggy or malicious implementations out there that you have to deal with. So you'd want to run it as a daemon. The CLI would have to be a frontend to the daemon or its database. The UI could be simple. I'm skeptical whether an implementation could be both good and simple.
  - derefr 3y ago
    That's if you're imagining a single node to discover the whole DHT. What if you want to fire off a map-reduce of limited-run DHT explorations starting from different DHT ring positions, where each agent just crawls and emits what it finds on stdout as it finds it? (In a sense, I suppose this would still be a "daemon", but that daemon would be the map-reduce infrastructure.)
    - the8472 3y ago
      I don't quite understand what you're proposing here. Generally you only control and operate ~1 node per IPv4 address or per IPv6 /64. All other nodes are operated by someone else, so they don't cooperate on anything beyond what the protocol specifies. Which means everyone is their own little silo. If you want a list of all currently active torrents (millions) then you have to do it with 1 or a handful of nodes, depending on how many IPs you have. DHTs are not arbitrary-distributed-compute frameworks, they're a quite restrictive get/put service. BEP51[0] does let you query other nodes for a sample of their keys (infohashes) but what they can offer is limited by their vantage point of the network so you need to go around and ask all those millions of nodes. And since it's all random you can't really "search" for anything, you can only sample. And that just gives you 20-byte keys. Afterwards you need to do a lot of additional work to turn those into human-readable metadata. [0] http://bittorrent.org/beps/bep_0051.html http://bittorrent.org/beps/bep_0051.html
Fnoord 3y ago
See also: Magnetissimo [1], Torrentinim [2], and Spotweb [3]. [1] https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo [2] https://github.com/sergiotapia/torrentinim https://github.com/sergiotapia/torrentinim [3] https://github.com/spotweb/spotweb https://github.com/spotweb/spotweb
- e12e 3y ago
  Any comments on how these compare? Especially in relation to sibling comment about BEP51?
  - the8472 3y ago
    From a brief look at each it seems like they're scraping things like torrent websites, usenet or maybe RSS feeds. Not the DHT.
  - diggan 3y ago
    From https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11 > The DHT implementation was largely borrowed from Magnetico (https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico) a popular and widely used app which is likewise unable to service these requests.
  - Fnoord 3y ago
    You want Torznab support. That is basically metadata you want to export, to import it into your application which holds the database of what you are after. If it is a match, it should attempt to download it via your download client (BitTorrent client). Torrentinim is the successor of Magnetissimo but it lacks Radarr/Sonarr integration (there is a pull request for Torznab support for both). Spotweb has Newznab support [1] but at Black Friday (soon) there's usually tons of deals available for Newznab indexes. I don't care about BEP51 as I don't have huge upload. That is also why I prefer Usenet over torrents. But torrents are a useful and sometimes required backup mode. Just not my preferred one. [1] https://github.com/Spotweb/Spotweb/wiki/Spotweb-als-Newznab-Provider https://github.com/Spotweb/Spotweb/wiki/Spotweb-als-Newznab-...
- 3y ago
the8472 3y ago
It seems like every single of these things always cut corners and don't implement proper, spec-compliant nodes that provide the same services as they use. You know, the "peer" in p2p. BEP51 was designed to make it easier to not trample on the commons, and yet...
- e12e 3y ago
  Appreciate that you took the time to file an issue: https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11
- zolland 3y ago
  Without the peers all we have is to!
- mgdigital 3y ago
  Author here. FWIW I wasn't intending this to make it onto HN, having posted about this on Lemmy looking for beta testers. The current version of the app is very much a preview. There's much further work to be done and this will include as far as possible ensuring Bitmagnet is a "good citizen". The suggestions made on the GH issue look largely feasible and I'll get round to looking at them as soon as I can. The issue and my response on GH: https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11
  - Y-bar 3y ago
    Hey! Don't know if the Github repo or here is the best way to ask since the Discussions on the repo are not active. I have started the docker-compose.yml file in WSL and it has been running for an hour slowly accumulating a few megabytes of redis data running at about 5% CPU usage. Inspecting it shows magnet links. It appears to be working. But visiting the the web interface at localhost:3333 just yields "Firefox can’t establish a connection to the server at localhost:3333." after a 30 second timeout. Would you have a guess why?
doakes 3y ago
What kind of bandwidth usage should be expected from a DHT crawler like this?
- KennyBlanken 3y ago
  After ~30-60 minutes of running, still less than 100kB/sec combined in and out. However, as others have noted, nodes don't communicate much with nodes that haven't been up for a while (days.) It's using roughly 6% CPU time for the crawler and another 1-2% for postgres, on a second-gen i7. As a datapoint to set expectations: 4000 torrents have been captured so far, and somewhat surprisingly, they're not very current results, necessarily. For example, a certain wildly popular TV series about samurai in space swinging very hot swords around which just had its season ending episode last night (I think)...that ep isn't in my list so far, but the episode prior to it, and the first two episodes, are. There's a ton of random, low-seed torrents, so it's actually kind of interesting to search by type, year, etc and see what comes up.
chmod775 3y ago
> The DHT crawler is not quite unique to bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. Heh. That was one of my first projects when I was still learning to code back in 2012: https://github.com/laino/shiny-adventure https://github.com/laino/shiny-adventure The DHT crawler/worker lived seperately, and I eventually put it here to rescue it from a dying HDD: https://github.com/laino/DHT-Torrent-database-Worker https://github.com/laino/DHT-Torrent-database-Worker The code is abhorrent and you absolutely shouldn't use it, but it worked. At least the crawler did - the frontend was never completed. Since the first implementation of mainline DHT appeared in 2005 and crawling that network is really quite an obvious idea, I doubt we (a friend was working on it as well) were first either.
- JP44 3y ago
  Nothing substantial, I chuckled when I saw the commit history on your linked projects. I do not mean to belittle you (or the purpose/goal of the projects), genuinely enjoyed the distraction and 'results' from it: Today was the first commit after 11 (9 oct 2012) and 5 years (24 nov 2018), respectively, on the projects. I think your repo might be part of some sort of oldest 'active'- or 'not ported to another repo'-repo For what I've found in ~10 min (google/gpt), excluding git projects existing before spring 2008 (couldn't get a quick consensus on feb vs april of that year), there's not a lot (I'll edit this part if sources are requested)
  - Fatnino 3y ago
    I recently committed to an old repo of mine after a 9 year gap. It holds several one file python script experiments and toys that I lumped into one place to get them off my hdd and make them available from wherever. Recently remembered it existed and added another one. And while I was in there I also ran 2to3 on the ones that needed it and polished the results up.
askiiart 3y ago
And the predecessor to this: https://github.com/mgdigital/rarbg-selfhosted https://github.com/mgdigital/rarbg-selfhosted It's now archived due to effort being redirected to Bitmagnet.
coolspot 3y ago
Please remove copyrighted movies from the screenshot on your website. It provides evidence that this program is designed for violating copyright, which makes DCMA takedown trivial.
- gymbeaux 3y ago
  I don’t see any screenshots on the website
  - davidcollantes 3y ago
    There was, and were removed.
- rsync 3y ago
  Please determine if these images fall under fair-use provisions and, if so, leave them in place. Bad actors - whoever they may be - need to see your rights constantly reasserted.
  - KennyBlanken 3y ago
    The MPAA and other organizations use screenshots that show copyrighted material as "proof" that the tools are used for copyright violation, and then DMCA them. If you want to help pay for lawyers to fight those DMCA notices with counterclaims and lawsuits, put up or shut up; the FSF, EFF and ACLU have been noticeably disinterested in doing so.
  - crazygringo 3y ago
    This has nothing whatsoever to do with fair use. It's about arguments in court about the intention of the software if sued. Images of copyrighted content indicate intent to infringe copyright. Without those, you can argue it's only meant to find and index Linux image torrents or whatever. Fair use doesn't enter the picture at all.
- pipes 3y ago
  And they probably want to remove text like "It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed."
LeoPanthera 3y ago
> Pipe dream features > In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.
- Scion9066 3y ago
  It might be able to do this with the Bittorent V2 format as the hashes would be per file.
  - chhs 3y ago
    V2 definitely makes it easier, but it's possible to identify files with V1 by iterating through metadata looking for files with the same length as a local file, then checking each piece against the local file. If they all match, then it's the same file. For boundary pieces where it consists of multiple files, I think it's safe enough to ignore if all of the remaining pieces match the local file, but you could do a more complex search looking for sets of files that all have the same length, and then compare the piece hash with the local files.
- miffe 3y ago
  dc++ does it, but it's not torrent based.
- orbisvicis 3y ago
  I've thought about writing a fuse system that tracks rename / move / delete and updates the torrent client. Never had the time, though.
- chhs 3y ago
  I wrote a project to do this using libtorrent a while ago, but unfortunately libtorrent crashes when seeding thousands of torrents at once, which is the current blocker. I haven't been working on it since. https://github.com/chhs1/content-seeder https://github.com/chhs1/content-seeder
r3trohack3r 3y ago
Have been playing around with DHT crawling for a while now, curious how you're getting around the "tiers" of the DHT? IIUC peers favor nodes they've had longer relationships with to provide stable routes through the DHT. This means short-lived nodes receive very little traffic, nobody is routing much traffic through fresh nodes, they choose nodes they’ve had longer relationships with. The longer you stay up, the more you start seeing. At least this is what I've observed in my projects. The only way I've been able to get anything interesting out of the DHT in the last ~5 years has been to put up a node and leave it up for a long time. If I spin up something, the first day I usually only find a handful of resolvable hashes. Not to mention it seems the BitTorrent DHT is very lax in what it will route compared to other DHTs (like IPFS) meaning many of the hashes you receive aren't for torrents at all.
complianceowl 3y ago
Guys, please educate me: I want to use torrents, but the thought of downloading something inappropriate by clicking on a deceptive link terrifies me (e.g., a download with the title of an action movie, but it turns out to be something else). How do you guys handle that risk?
- shepherdjerred 3y ago
  If you're downloading "normal" stuff, then you aren't likely to run into an issue. Stick to reputable sites; reddit can help you figure out what those are.
  - ShadowBanThis01 3y ago
    What sites? Isn't the purpose of the software discussed here to make tracker sites unnecessary?
    - iinnPP 3y ago
      It's not the same thing.
      - ShadowBanThis01 3y ago
        What's not the same thing as what?
- askiiart 3y ago
  Just use reputable sites and run a good adblocker, like Unlock Origin.
- dvdkon 3y ago
  Unless you're also afraid of clicking links on the web, you should be fine with torrents. Maybe you could not seed by default and turn it on only after verifying the data is authentic, that way you're actually only downloading and the analogy is complete.
KennyBlanken 3y ago
I've been unable to get this running; I gave it a postgres user and database, granted it ownership and all permissions on said DB, and there's nothing in the database. Edit: found the init schema and things seem to be working now: https://github.com/bitmagnet-io/bitmagnet/blob/main/migrations/00001_init.sql https://github.com/bitmagnet-io/bitmagnet/blob/main/migratio... It would be really nice to be able to sort by header (size, seeders) and/or some filters for seed/downloaders (for example, filtering out anything with less than X seeds.)
RIMR 3y ago
This is really neat. I'll need to check it out. A couple years ago I ran my own instance of Magnetico (https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico), but this project looks a lot more polished.
forgetm3 3y ago
Isn't this much the same as btdig.com which is based on: https://github.com/btdig/dhtcrawler2 https://github.com/btdig/dhtcrawler2 I use this service to do security research a fair bit. It'd be nice if there was a higher quality self-hosted version I could use so I'll be watching this project with interest!
- executesorder66 3y ago
  I use btdig for finding torrents myself. But I am curious what you mean when you say you use it "to do security research"? Are you just looking for security information that is available in torrents, or does btdig have some other features that I am unaware of?
JohnDeHope 3y ago
Why isn't this built into thick desktop BT clients? Is it just a little too early for it, and next year it will be? Or is there some reason I'm missing?
- austhrow743 3y ago
  People who are in to torrenting enough to consider self hosting a bitttorrent indexer largely dont use desktop BT clients. They chain different software together on a home media server or seedbox such that the whole concept of torrents, seeds, trackers, etc is unseen plumbing and instead you have interfaces focused on the content. Descriptions, trailers, actor lists, etc. You subscribe to a tv show for example and then episodes are just automatically downloaded as they air according to your preset quality criteria, then they appear in your own personal netflix style app.
- themoonisachees 3y ago
  What the other person said, but also: you have to have your crawler up for several days, ideally near 100% uptime for this to be effective. Thick clients can have high uptime but they also can not.
- themoonisachees 3y ago
  What the other person said, but also: you have to have your crawler up for several days, ideally near 100% uptime for this to be effective. Thick clients can have high uptime but they also can not.
m3kw9 3y ago
How’s this different from piratebay
- KiwiJohnno 3y ago
  It's self-hosted.
drakenot 3y ago
> Something that looks like a decentralized private tracker; by this I probably mean something that’s based partly on personal trust and manually weeding out any bad actors; I’d be wary of creating something that looks a bit like Tribler, which while an interesting project seems to have demonstrated that implementing trust, reputation and privacy at the protocol level carries too much overhead to be a compelling alternative to plain old BitTorrent, for all its imperfections I've thought about this problem a lot. Having a federated / distributed tracker but with some form of trust based, or opt-in curation would be amazing.
- synctext 3y ago
  Tribler developer here. The "Trust framework" that everybody is after: lightness of tit-for-tat with little overhead. With trust that actually can make The Internet a nice place again! Curation is indeed the problem discovered by HN practitioners such yourself. However, nobody in academia is working on this. As one old scientific article is titled: "Curation, Curation, Curation"
ShadowBanThis01 3y ago
Strange: I tried to give it a star on Github, but the star control just resets every time. And I am logged in.
pepe56 3y ago
Does this also share back DHT information like a "server"? I am not downloading torrents, but I always wished to have a daemon running on my servers to help the overall DHT networks health. Is there anything like this out there? A DHT server, that only collects, stores and gives back information?
- zidel 3y ago
  > Does this also share back DHT information like a "server"? No, it ignores all incoming requests. You don't need special software to help out, just run any normal Bittorrent client in the background (no need to download or share anything) and it will help out. Just make sure you forward the right port if you are behind a NAT. Traffic will slowly increase over time, and drop quickly when offline, so leaving it up for days at a time is better when possible.
alexpotato 3y ago
For those of you interested in the intersection of "make vs buy" decisions, self hosting torrent servers and working at a hedge fund, you might get a kick out of this Twitter thread: https://twitter.com/alexpotato/status/1661334108180566016 https://twitter.com/alexpotato/status/1661334108180566016
Alifatisk 3y ago
> This is how bitmagnet’s DHT crawler works works You wrote "works" twice

497 points by KoftaBob 3y ago

I've been running this for about 5 days in Docker. I've had to restart bitmagnet a handful of times as it seems to crash. The web interface is available but the number of torrents indexed never increases. I'm currently at 180,744 indexed and I'm waiting to see how high it goes. I check on it a few times a day.

thelastparadise 3y ago

Very interesting. Does this approach actually work in practice? Also what happens if illegal content gets scooped up into the index?

KoftaBob 3y ago
Based on my understanding of how the torrent DHT works, all that’s happening is that youre requesting metadata on various torrent info hashes, but that’s not the same thing as actually downloading/seeding the content in the torrent itself.
- no_time 3y ago
  >but that’s not the same thing as actually downloading/seeding the content in the torrent itself. The question is whether Law Enforcement and "Intellectual Property" watchdogs make a meaningful distinction between the two in their monitoring tools.
  - justinclift 3y ago
    Seeing as they get so much other stuff wrong, it's highly likely they don't care.
Akashic101 3y ago
Considering the screenshot on the linked page shows The Flash among other movies I dont think the author is too concerned about that. I believe its similar to the area that Plex and Jellyfin operate in, mainly that they just provide the framework and tools, what the user does with them is not in their control
- EGreg 3y ago
  How will governments police it when MaidSAFE and other systems distribute totally encrypted content?
  - colinsane 3y ago
    BitTorrent already supports encrypted peer connections.
    - mixmastamyk 3y ago
      Doesn’t seem to work, our ISP has sent nastygrams despite this and site on https.
      - colinsane 3y ago
        > How will governments police it when MaidSAFE and other systems distribute totally encrypted content? so isn't the answer then "they'll continue to police it the way they already do"? i don't know what a MaidSAFE is, but the context of this discussion is the DHT, and so public (indexable) torrents, and so however you encrypt the content doesn't matter because you have to provide the decryption method to anyone who asks for any of the previous context (public torrents/indexes) to make any sense.
        mixmastamyk 3y ago
        Encrypted connections shouldn’t decrypt for anyone who asks, otherwise they have no reason to exist. The weak point seems to be the tracker or filename, but been told https hides that so not sure.
lobsterslive 3y ago
It works well in practice. The DHT protocol includes announce messages that broadcast when new files are shared on BitTorrent. It then includes a "geometric" way to find people who are sharing those files. It doesn't include the files themselves, just the torrents which include a file list and location hashes. If you listen to BitTorrent's DHT network, you'll build an index of everything shared on BitTorrent (over time), this will include commercial movies and such.
- grepfru_it 3y ago
  >It works well in practice. Hi, I worked on gnutella and lots of P2P systems in the early 00s. This will devolve into noise and spam as the number of users who adopt this feature pass a critical mass. With a fully decentralized system, there are no gatekeepers, and as such, there is no way to filter counterfiet items. While your client will present with you the data you are searching for, you will find out (usually hours later) that your supposed pirated download is actually just a 2hour loop of Rick Astley (still piracy though, so you are still winning.. i think?).
  - SparkyMcUnicorn 3y ago
    But you can still pick the option with the most seeders, which should get you what you're looking for most of the time. The spam problem isn't nonexistent within the centralized services either.
    - grepfru_it 3y ago
      Hehe in a popular P2P client from the '03-'05 period, we said the same thing. Turns out there are groups with large amounts of funding which will provide a fake seed count. Either just faking metadata making it seem there was a high seed count but bogus nodes which would refuse connections (which was actual behavior from clients with bad ISPs - which we saw valid cases in asia or east europe) or would actually stream data (and some of them were on good hosts seeding multi mbps of bad data) What i'm saying is it becomes a numbers game and those fake seeders usually have deep pockets financed by the content creators themselves
quickthrower2 3y ago
I think illegal (unlawful?) content is its raison d’etre. With the odd exceptions of thinks like linux distros, LLM weights etc. to take pressure off centralized servers.

fallat 3y ago

It'd be so nice to have a super simple DHT crawler CLI tool, in both implementation and interface.

the8472 3y ago
These things need uptime of hours and days to do it properly and also to stay up to date. There are millions of nodes and torrents and to be non-abusive you have to issue requests at a somewhat sedate pace. And activity kind of moves with the sun due to people who run torrent clients on their home machines. And there are lots of buggy or malicious implementations out there that you have to deal with. So you'd want to run it as a daemon. The CLI would have to be a frontend to the daemon or its database. The UI could be simple. I'm skeptical whether an implementation could be both good and simple.
- derefr 3y ago
  That's if you're imagining a single node to discover the whole DHT. What if you want to fire off a map-reduce of limited-run DHT explorations starting from different DHT ring positions, where each agent just crawls and emits what it finds on stdout as it finds it? (In a sense, I suppose this would still be a "daemon", but that daemon would be the map-reduce infrastructure.)
  - the8472 3y ago
    I don't quite understand what you're proposing here. Generally you only control and operate ~1 node per IPv4 address or per IPv6 /64. All other nodes are operated by someone else, so they don't cooperate on anything beyond what the protocol specifies. Which means everyone is their own little silo. If you want a list of all currently active torrents (millions) then you have to do it with 1 or a handful of nodes, depending on how many IPs you have. DHTs are not arbitrary-distributed-compute frameworks, they're a quite restrictive get/put service. BEP51[0] does let you query other nodes for a sample of their keys (infohashes) but what they can offer is limited by their vantage point of the network so you need to go around and ask all those millions of nodes. And since it's all random you can't really "search" for anything, you can only sample. And that just gives you 20-byte keys. Afterwards you need to do a lot of additional work to turn those into human-readable metadata. [0] http://bittorrent.org/beps/bep_0051.html http://bittorrent.org/beps/bep_0051.html

Fnoord 3y ago

See also: Magnetissimo [1], Torrentinim [2], and Spotweb [3]. [1] https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo [2] https://github.com/sergiotapia/torrentinim https://github.com/sergiotapia/torrentinim [3] https://github.com/spotweb/spotweb https://github.com/spotweb/spotweb

e12e 3y ago
Any comments on how these compare? Especially in relation to sibling comment about BEP51?
- the8472 3y ago
  From a brief look at each it seems like they're scraping things like torrent websites, usenet or maybe RSS feeds. Not the DHT.
- diggan 3y ago
  From https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11 > The DHT implementation was largely borrowed from Magnetico (https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico) a popular and widely used app which is likewise unable to service these requests.
- Fnoord 3y ago
  You want Torznab support. That is basically metadata you want to export, to import it into your application which holds the database of what you are after. If it is a match, it should attempt to download it via your download client (BitTorrent client). Torrentinim is the successor of Magnetissimo but it lacks Radarr/Sonarr integration (there is a pull request for Torznab support for both). Spotweb has Newznab support [1] but at Black Friday (soon) there's usually tons of deals available for Newznab indexes. I don't care about BEP51 as I don't have huge upload. That is also why I prefer Usenet over torrents. But torrents are a useful and sometimes required backup mode. Just not my preferred one. [1] https://github.com/Spotweb/Spotweb/wiki/Spotweb-als-Newznab-Provider https://github.com/Spotweb/Spotweb/wiki/Spotweb-als-Newznab-...
3y ago

the8472 3y ago

It seems like every single of these things always cut corners and don't implement proper, spec-compliant nodes that provide the same services as they use. You know, the "peer" in p2p. BEP51 was designed to make it easier to not trample on the commons, and yet...

e12e 3y ago
Appreciate that you took the time to file an issue: https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11
zolland 3y ago
Without the peers all we have is to!
mgdigital 3y ago
Author here. FWIW I wasn't intending this to make it onto HN, having posted about this on Lemmy looking for beta testers. The current version of the app is very much a preview. There's much further work to be done and this will include as far as possible ensuring Bitmagnet is a "good citizen". The suggestions made on the GH issue look largely feasible and I'll get round to looking at them as soon as I can. The issue and my response on GH: https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11
- Y-bar 3y ago
  Hey! Don't know if the Github repo or here is the best way to ask since the Discussions on the repo are not active. I have started the docker-compose.yml file in WSL and it has been running for an hour slowly accumulating a few megabytes of redis data running at about 5% CPU usage. Inspecting it shows magnet links. It appears to be working. But visiting the the web interface at localhost:3333 just yields "Firefox can’t establish a connection to the server at localhost:3333." after a 30 second timeout. Would you have a guess why?

doakes 3y ago

What kind of bandwidth usage should be expected from a DHT crawler like this?

KennyBlanken 3y ago
After ~30-60 minutes of running, still less than 100kB/sec combined in and out. However, as others have noted, nodes don't communicate much with nodes that haven't been up for a while (days.) It's using roughly 6% CPU time for the crawler and another 1-2% for postgres, on a second-gen i7. As a datapoint to set expectations: 4000 torrents have been captured so far, and somewhat surprisingly, they're not very current results, necessarily. For example, a certain wildly popular TV series about samurai in space swinging very hot swords around which just had its season ending episode last night (I think)...that ep isn't in my list so far, but the episode prior to it, and the first two episodes, are. There's a ton of random, low-seed torrents, so it's actually kind of interesting to search by type, year, etc and see what comes up.

chmod775 3y ago

> The DHT crawler is not quite unique to bitmagnet; another open-source project, magnetico was first (as far as I know) to implement a usable DHT crawler, and was a crucial reference point for implementing this feature. Heh. That was one of my first projects when I was still learning to code back in 2012: https://github.com/laino/shiny-adventure https://github.com/laino/shiny-adventure The DHT crawler/worker lived seperately, and I eventually put it here to rescue it from a dying HDD: https://github.com/laino/DHT-Torrent-database-Worker https://github.com/laino/DHT-Torrent-database-Worker The code is abhorrent and you absolutely shouldn't use it, but it worked. At least the crawler did - the frontend was never completed. Since the first implementation of mainline DHT appeared in 2005 and crawling that network is really quite an obvious idea, I doubt we (a friend was working on it as well) were first either.

JP44 3y ago
Nothing substantial, I chuckled when I saw the commit history on your linked projects. I do not mean to belittle you (or the purpose/goal of the projects), genuinely enjoyed the distraction and 'results' from it: Today was the first commit after 11 (9 oct 2012) and 5 years (24 nov 2018), respectively, on the projects. I think your repo might be part of some sort of oldest 'active'- or 'not ported to another repo'-repo For what I've found in ~10 min (google/gpt), excluding git projects existing before spring 2008 (couldn't get a quick consensus on feb vs april of that year), there's not a lot (I'll edit this part if sources are requested)
- Fatnino 3y ago
  I recently committed to an old repo of mine after a 9 year gap. It holds several one file python script experiments and toys that I lumped into one place to get them off my hdd and make them available from wherever. Recently remembered it existed and added another one. And while I was in there I also ran 2to3 on the ones that needed it and polished the results up.

askiiart 3y ago

And the predecessor to this: https://github.com/mgdigital/rarbg-selfhosted https://github.com/mgdigital/rarbg-selfhosted It's now archived due to effort being redirected to Bitmagnet.

coolspot 3y ago

Please remove copyrighted movies from the screenshot on your website. It provides evidence that this program is designed for violating copyright, which makes DCMA takedown trivial.

gymbeaux 3y ago
I don’t see any screenshots on the website
- davidcollantes 3y ago
  There was, and were removed.
rsync 3y ago
Please determine if these images fall under fair-use provisions and, if so, leave them in place. Bad actors - whoever they may be - need to see your rights constantly reasserted.
- KennyBlanken 3y ago
  The MPAA and other organizations use screenshots that show copyrighted material as "proof" that the tools are used for copyright violation, and then DMCA them. If you want to help pay for lawyers to fight those DMCA notices with counterclaims and lawsuits, put up or shut up; the FSF, EFF and ACLU have been noticeably disinterested in doing so.
- crazygringo 3y ago
  This has nothing whatsoever to do with fair use. It's about arguments in court about the intention of the software if sued. Images of copyrighted content indicate intent to infringe copyright. Without those, you can argue it's only meant to find and index Linux image torrents or whatever. Fair use doesn't enter the picture at all.
pipes 3y ago
And they probably want to remove text like "It then further enriches this metadata by attempting to classify it and associate it with known pieces of content, such as movies and TV shows. It then allows you to search everything it has indexed."

LeoPanthera 3y ago

> Pipe dream features > In-place seeding: identify files on your computer that are part of an indexed torrent, and allow them to be seeded in place after having moved, renamed or deleted parts of the torrent Does anything do this already? It would be amazing to point a client at a folder of unstructured junk and have it magically find the right parts.

Scion9066 3y ago
It might be able to do this with the Bittorent V2 format as the hashes would be per file.
- chhs 3y ago
  V2 definitely makes it easier, but it's possible to identify files with V1 by iterating through metadata looking for files with the same length as a local file, then checking each piece against the local file. If they all match, then it's the same file. For boundary pieces where it consists of multiple files, I think it's safe enough to ignore if all of the remaining pieces match the local file, but you could do a more complex search looking for sets of files that all have the same length, and then compare the piece hash with the local files.
miffe 3y ago
dc++ does it, but it's not torrent based.
orbisvicis 3y ago
I've thought about writing a fuse system that tracks rename / move / delete and updates the torrent client. Never had the time, though.
chhs 3y ago
I wrote a project to do this using libtorrent a while ago, but unfortunately libtorrent crashes when seeding thousands of torrents at once, which is the current blocker. I haven't been working on it since. https://github.com/chhs1/content-seeder https://github.com/chhs1/content-seeder

r3trohack3r 3y ago

Have been playing around with DHT crawling for a while now, curious how you're getting around the "tiers" of the DHT? IIUC peers favor nodes they've had longer relationships with to provide stable routes through the DHT. This means short-lived nodes receive very little traffic, nobody is routing much traffic through fresh nodes, they choose nodes they’ve had longer relationships with. The longer you stay up, the more you start seeing. At least this is what I've observed in my projects. The only way I've been able to get anything interesting out of the DHT in the last ~5 years has been to put up a node and leave it up for a long time. If I spin up something, the first day I usually only find a handful of resolvable hashes. Not to mention it seems the BitTorrent DHT is very lax in what it will route compared to other DHTs (like IPFS) meaning many of the hashes you receive aren't for torrents at all.

complianceowl 3y ago

Guys, please educate me: I want to use torrents, but the thought of downloading something inappropriate by clicking on a deceptive link terrifies me (e.g., a download with the title of an action movie, but it turns out to be something else). How do you guys handle that risk?

shepherdjerred 3y ago
If you're downloading "normal" stuff, then you aren't likely to run into an issue. Stick to reputable sites; reddit can help you figure out what those are.
- ShadowBanThis01 3y ago
  What sites? Isn't the purpose of the software discussed here to make tracker sites unnecessary?
  - iinnPP 3y ago
    It's not the same thing.
    - ShadowBanThis01 3y ago
      What's not the same thing as what?
askiiart 3y ago
Just use reputable sites and run a good adblocker, like Unlock Origin.
dvdkon 3y ago
Unless you're also afraid of clicking links on the web, you should be fine with torrents. Maybe you could not seed by default and turn it on only after verifying the data is authentic, that way you're actually only downloading and the analogy is complete.

KennyBlanken 3y ago

I've been unable to get this running; I gave it a postgres user and database, granted it ownership and all permissions on said DB, and there's nothing in the database. Edit: found the init schema and things seem to be working now: https://github.com/bitmagnet-io/bitmagnet/blob/main/migrations/00001_init.sql https://github.com/bitmagnet-io/bitmagnet/blob/main/migratio... It would be really nice to be able to sort by header (size, seeders) and/or some filters for seed/downloaders (for example, filtering out anything with less than X seeds.)

RIMR 3y ago

This is really neat. I'll need to check it out. A couple years ago I ran my own instance of Magnetico (https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico), but this project looks a lot more polished.

forgetm3 3y ago

Isn't this much the same as btdig.com which is based on: https://github.com/btdig/dhtcrawler2 https://github.com/btdig/dhtcrawler2 I use this service to do security research a fair bit. It'd be nice if there was a higher quality self-hosted version I could use so I'll be watching this project with interest!

executesorder66 3y ago
I use btdig for finding torrents myself. But I am curious what you mean when you say you use it "to do security research"? Are you just looking for security information that is available in torrents, or does btdig have some other features that I am unaware of?

JohnDeHope 3y ago

Why isn't this built into thick desktop BT clients? Is it just a little too early for it, and next year it will be? Or is there some reason I'm missing?

austhrow743 3y ago
People who are in to torrenting enough to consider self hosting a bitttorrent indexer largely dont use desktop BT clients. They chain different software together on a home media server or seedbox such that the whole concept of torrents, seeds, trackers, etc is unseen plumbing and instead you have interfaces focused on the content. Descriptions, trailers, actor lists, etc. You subscribe to a tv show for example and then episodes are just automatically downloaded as they air according to your preset quality criteria, then they appear in your own personal netflix style app.
themoonisachees 3y ago
What the other person said, but also: you have to have your crawler up for several days, ideally near 100% uptime for this to be effective. Thick clients can have high uptime but they also can not.
themoonisachees 3y ago
What the other person said, but also: you have to have your crawler up for several days, ideally near 100% uptime for this to be effective. Thick clients can have high uptime but they also can not.

m3kw9 3y ago

How’s this different from piratebay

KiwiJohnno 3y ago
It's self-hosted.

drakenot 3y ago

> Something that looks like a decentralized private tracker; by this I probably mean something that’s based partly on personal trust and manually weeding out any bad actors; I’d be wary of creating something that looks a bit like Tribler, which while an interesting project seems to have demonstrated that implementing trust, reputation and privacy at the protocol level carries too much overhead to be a compelling alternative to plain old BitTorrent, for all its imperfections I've thought about this problem a lot. Having a federated / distributed tracker but with some form of trust based, or opt-in curation would be amazing.

synctext 3y ago
Tribler developer here. The "Trust framework" that everybody is after: lightness of tit-for-tat with little overhead. With trust that actually can make The Internet a nice place again! Curation is indeed the problem discovered by HN practitioners such yourself. However, nobody in academia is working on this. As one old scientific article is titled: "Curation, Curation, Curation"

ShadowBanThis01 3y ago

Strange: I tried to give it a star on Github, but the star control just resets every time. And I am logged in.

pepe56 3y ago

Does this also share back DHT information like a "server"? I am not downloading torrents, but I always wished to have a daemon running on my servers to help the overall DHT networks health. Is there anything like this out there? A DHT server, that only collects, stores and gives back information?

zidel 3y ago
> Does this also share back DHT information like a "server"? No, it ignores all incoming requests. You don't need special software to help out, just run any normal Bittorrent client in the background (no need to download or share anything) and it will help out. Just make sure you forward the right port if you are behind a NAT. Traffic will slowly increase over time, and drop quickly when offline, so leaving it up for days at a time is better when possible.

alexpotato 3y ago

For those of you interested in the intersection of "make vs buy" decisions, self hosting torrent servers and working at a hedge fund, you might get a kick out of this Twitter thread: https://twitter.com/alexpotato/status/1661334108180566016 https://twitter.com/alexpotato/status/1661334108180566016

Alifatisk 3y ago

> This is how bitmagnet’s DHT crawler works works You wrote "works" twice