10 ms·

Ask HN: I've Built a DHT Torrent Sniffer and Search Engine. Should I Release?

165 points by sylwester 4y ago

Recently I was researching about DHTs and developed a DHT Sniffer in Go which connects to some known DHT Routers and sniffs all the annoucements. I've quickly added ZincSearch and it is now basically a search engine which can search for hashes, name or files contained in the torrents. It is able to index around 5-10k annoucements per second, so the index grows quite fast. Now, I am thinking about releasing it as open-source for others to study, but not sure if I should, because it might be used for "evil".

compressedgas 4y ago
Consider that such already is available as open source in Go even.
navjack27 4y ago
That would be very useful... Just release the code and building instructions.
2Gkashmiri 4y ago
yes please. you are not responsible for any "Evil" users of the software might do. This is not even about enabling bad stuff, this is just natural progression of technology.
- rrwo 4y ago
  Nuclear and biological weapons are also part of the "natural progression of technology" but it's widely agreed that they shouldn't be released to anybody and everybody, and the people distributing those technologies have a moral responsibility. If you think your software would be of more use to "evil" than not, then don't release it widely.
  - loa_in_ 4y ago
    Yeah, it is everyone's moral responsibility to consider effects of our actions as far as we're able to.
qualudeheart 4y ago
Which evil usages are you concerned about? I think it would be very useful for the public.
- seqastian 4y ago
  copyright holder bots
  - Ysx 4y ago
    DHT has existed for 17 years - the cat's out of the bag. The anti-piracy companies have built their own crawlers.
    - aliqot 4y ago
      They match swarm IP's and then notify ISP's who voluntarily hassle you on their behalf. An IP doesn't equal a person in USA.
      - netsharc 4y ago
        And outside of the US? In Germany there's an "efficient" business of law companies acting on behalf of the copyright holders, subpoenaing the ISP to get the owner of the IP/connection and sending them a cease-and-desist with a hefty fee. There's the next part, law companies specializing in "Here's the letter you have to send to say you agree to pay them without admitting guilt, that'll be € please!".
        TillE 4y ago
        I think the demand is usually around 1000 Euro, so it's way cheaper to just pay for a VPN service, configure qBittorrent to only use your VPN interface, and torrent whatever the hell you want.
0dayz 4y ago
I would recommend writing down the worst and best case scenarios that could happen with your software, then determine if you notice either that through severity or quantity the software outweighs the positives, don't release it.
tmtvl 4y ago
Just AGPL it, I hear it's an effective ward against Alphabet.
- Eduard 4y ago
  Very good suggestion, considering the sniffer could be used by copyright claims lawyers.
  - MattPalmer1086 4y ago
    Can't see how AGPL would stop anyone using it. Doesn't it just prevent them from modifying the source code without sharing their changes. ?
brobinson 4y ago
Let it rip. DHT has been around for so long now that whatever bad actors/evil use cases you're imagining have already happened. It sounds like a cool project, and I'd be interested to see it.
- sshine 4y ago
  I accidentally read this as "Let it R.I.P." I totally agree, unroll it! While the file sharing is distributed, the centralised web-based indexing is still a game of whack-a-mole.
  - MuffinFlavored 4y ago
    how is what OP made not basically going to be the p1rat3 bay 2.0?
pdimitar 4y ago
I imagine any bad actors who store IPs of torrent seeders have done so a long time ago already so your software will not do any harm that hasn't been done already. Go for it and open-source it.
sascha_sl 4y ago
I mean, it already exists. [1] Always fun to see what my neighbors behind the same NAT download. [1]: https://iknowwhatyoudownload.com/ https://iknowwhatyoudownload.com/
- oefrha 4y ago
  Ah, haven’t checked that one for a while. Surprised to find a tech-minded neighbor who has downloaded kali-linux-2022.3-live-everything-amd64.iso. And no pr0n this time round.
  - IYasha 4y ago
    You'd better have a good firewall since you have such a tech-minded neighbor. )
- insane_dreamer 4y ago
  I thought streaming had spelled the demise of torrenting, so was v surprised to check out my local hood (quiet suburban, skewing older few ppl <40yrs) and see how it's very much alive and well. I guess I was naive. (Not condemning anyone, as I did my share of peering back in the day, but streaming services are convenient and honestly don't cost that much all things considered. Torrents (tho I prefer IRC) good for finding obscure stuff that's unavailable anywhere else, but that's not what my neighbors are downloading; and it's not like any of them can't afford to pay. Interesting dynamic.
- Klaster_1 4y ago
  It gives quite fascinating results for me. Surprisingly, among dozens of torrents only a couple are porn. Most of my neighbors using BitTorrent are Russians and Ukrainians (I'm in Cyprus), judging by files names and seeder distribution, a couple of English-speaking people (probably brits) and a single person seeding content in Greek. At least half of Russian-speaking neighbors seed IT, engineering and English learning materials. And I seed more than everyone together on the same subnet Wish I could message them.
- imhoguy 4y ago
  Checked it for my Transmission host static IP which doesn't download anything for a while and it has some completely random stuff I never downloaded or searched for. Even shows items from today which for sure I haven't requested. Could be some DHT spoofing or my Transmission is relaying some DHT requests?
synctext 4y ago
- dhaavi 4y ago
  Tribler is an interesting project, but not directly comparable to what the OP is trying to achieve.
  - synctext 4y ago
    indeed, bit different. lots of people worked in this field and made these tools. Around 2014 the state of the art was 20k DHT responses per second, see '100 million DHT replies'. https://doi.org/10.1109/P2P.2014.6934318 https://doi.org/10.1109/P2P.2014.6934318
    - dhaavi 4y ago
      Thanks for the pointer! I just read some interesting proposals in the GH issues and recognized your username. This is some nice piece of work - will definitely return to read more about the underlying concepts. We are working in a similar area with similar problems: Building a new Tor/VPN-like privacy network. See https://safing.io/spn/ https://safing.io/spn/
      - simjnd 4y ago
        Taking this chance to say thank you for Safing Portmaster it is awesome!
- pharmakom 4y ago
  Who is funding it and why?
  - someotherperson 4y ago
    If you follow through to the site linked in the Github: > Tribler is a research project of Delft University of Technology > Work on Tribler has been supported by multiple Internet research European grants. In total we received 3,538,609 Euro in funding for our open source self-organising systems research. Roughly 10 to 15 scientists and engineers work on it full-time. Our ambition is to make darknet technology, security and privacy the default for all Internet users. > Vision & Mission ... "Push the boundaries of self-organising systems, robust reputation systems and craft collaborative systems with millions of active participants under continuous attack from spammers and other adversarial entities."
BLKNSLVR 4y ago
I'll add to the chorus of people saying "yes, release it". If you're worried about blowback as a result of "evil" uses / users, is there a way to release it (somewhat) anonymously, so it's difficult to be traced back to you?
icpmoles 4y ago
Is it basically btdig.com ?
- storrgie 4y ago
  An advantage here would be people self hosting their own based on this project.
- encryptluks2 4y ago
  Btdig is awesome. If this is like an open source version in Go then that is a huge contribution.
  - steinuil 4y ago
    Isn't btdig also open source? I see a "fork me" button on the front page and it leads to a repo: https://github.com/btdigg-org/dhtcrawler2 https://github.com/btdigg-org/dhtcrawler2
    - encryptluks2 4y ago
      It is 9 years old and in Erlang.
berry_sortoro 4y ago
Proven 4y ago
thinkmcfly 4y ago
I think you should. From my understanding, use of DHT is already dead in the eyes of most torrenters
the8472 4y ago
Having seen other indexers before I suspect your implementation isn't spec-compliant or well-behaved (perhaps spoofing node-IDs? causing more traffic than necessary?) If you want to build an indexer you should write a normal implementation and then use http://bittorrent.org/beps/bep_0051.html http://bittorrent.org/beps/bep_0051.html
- rahimnathwani 4y ago
  Related comment: https://news.ycombinator.com/item?id=14022107 https://news.ycombinator.com/item?id=14022107
gwnywg 4y ago
I was planning to start learning GO, I'd be definitely interested to learn from your project :)
boramalper 4y ago
I had been working on this successfully for a couple years in the past before I got tired of it and moved on. I still think it's a magnificent idea, to be able to host your own torrent site and to decentralise the last centralised bit of BitTorrent. https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico
- hardwaresofton 4y ago
  Thanks for writing that note on the repository -- it was thoughtful and well explained -- reading through your small manifesto that was linked and that's fun too.
  - kevinwang 4y ago
    I think the manifesto was Aaron Swartz's
    - hardwaresofton 4y ago
      oh you're right I didn't even notice -- aaronsw.com... First time I've read it.
- rakoo 4y ago
  I've built a small cli utility to search the database: https://sr.ht/~rakoo/magneticos/ https://sr.ht/~rakoo/magneticos/. It also asks the DHT and known trackers for the liveness of the swarms, so I can sort by seeders. It's simpler and more useful than running a full webserver with ports and all when I only need an ssh access. It has served me well and will be more than enough for a long time
- thomastjeffery 4y ago
  The main thing that seems to be missing, in my experience, is moderation. If someone can manage to make a decentralized forum-like moderation list, then we wouldn't have any use for centralized trackers anymore. I think it could probably be done by using public GPG keys as identities, then keeping changes in a torrent-distributed git repo where user-generated data is saved in GPG signed git commits. Then the only centralization left would be what branch/torrent people introduce to new users.
mutant 4y ago
https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico Someone else did this a while back, universe continues to exist.
majestic5762 4y ago
I wrote a similar solution 8 years ago. I repurposed the system to identify IPs owned by the government, and notified them if a malicious copy of Windows (but not limited to) was seeded by them. Meaning there was a chance that an unknown actor had a backdoor in my government's network. If you wanna discuss, I'm happy to talk and even contribute towards a commercial solution majestic.hn@fastmail.com. I didn't pursue this opportunity for money at that time, but I had my fair share of "shoutz". Won't be bad to pick this up again
zelienople 4y ago
- computerfriend 4y ago
  I don't know why you think the OP wants to be regarded as a professional. But you'd do well to adopt a little professionalism in your communication too.
jrm4 4y ago
Please do. This old-ish timer is kind of blown away by the idea that you shouldn't. Nothing evil about being a modern archivist/librarian, despite what big companies would tell you.
hardwaresofton 4y ago
Would you mind explainng why you chose ZincSearch? Curious on why you picked it over some of the other non-ElasticSearch/OpenSearch alternatives (Meilisearch, Typesense)
keroro 4y ago
torrent-paradise [0] is a go project which seems to do the same as your project and has existed since 2019. It’s since gone down but remains up on IPFS [1] but its index hasn’t updated since January. [0] https://github.com/urbanguacamole/torrent-paradise https://github.com/urbanguacamole/torrent-paradise [1] https://cloudflare-ipfs.com/ipfs/QmQjsKamNFZRvCMXDvZXQmRYjsmSkmZG5pBCTY4LtMj8hs/about.html https://cloudflare-ipfs.com/ipfs/QmQjsKamNFZRvCMXDvZXQmRYjsm...
- mdaniel 4y ago
  > remains up on IPFS [1] but its index hasn’t updated since January. But only the "about" page remains up, right? Because both https://cloudflare-ipfs.com/ipns/torrent-paradise.ml https://cloudflare-ipfs.com/ipns/torrent-paradise.ml and https://cloudflare-ipfs.com/ipns/12D3KooWB3GY1u6zMLqnf3MJ8zhX3SS1oBj7VXk3xp6sJJiFGZXp https://cloudflare-ipfs.com/ipns/12D3KooWB3GY1u6zMLqnf3MJ8zh... show a command-line(!) error If my mental model of IPFS is correct, it means no one is seeding the underlying IPFS ... CIDs? ... that those links point to
hombre_fatal 4y ago
This is a good example of our hubris as developers. We like to think our project will have some impact on the world when in reality you’re extremely lucky if anybody notices much less cares. ;)
arthurcolle 4y ago
Magnetissimo did this too, and he's still around.
r3trohack3r 4y ago
You should absolutely open source this. Working with the DHT is a lot of fun, and a great learning exercise. I adapted my local “torrent roulette” application to an electron app that can be shared. My local version downloads the files, but the one I share only fetches the torrents metadata (easy to adapt it back to my roulette approach). I call it Taboo: https://github.com/retrohacker/taboo https://github.com/retrohacker/taboo The amount of “evil” on the DHT is pretty low. I’ve run mine for a long time, and very rarely get anything evil. I suspect it’s because of how poorly BitTorrent plays with privacy tools like VPN and Tor, IIUC it’s easy to leak identifying information with BT and its high bandwidth. Nearly every “evil” file I’ve found is either: a honeypot with not-evil content or password encrypted (maybe to remove the plausible deniability of a random download?). I don’t know if the encrypted files actually contain evil content, I don’t bother trying to crack them and promptly gshred them. What you will find: * a lot of content illegal under US copyright law * a lot of porn (also illegal under US copyright law) * a metric tonne of fascinating content from other cultures you’d otherwise not be exposed to (also probably illegal under US copyright law) There is very little “legal” content on the DHT, but most of it is falling on the wrong side of intellectual property law. Sometimes password/credit card dumps. I once found some very sketchy schematics of Eastern European military equipment. But that stuff is also really rare in my experience. If you do play with these systems, I’ll leave the same warning I left on the Taboo repo: > Note: I AM NOT A LAWYER! To my knowledge, there aren't any other systems doing this that you can run on your laptop. I suspect the nuance of how Taboo works isn't going to be appreciated by your local law enforcement. If you don't want to test the legality of Taboo in court, I'd strongly recommend either: running a VPN (less safe) or not using Taboo (most safe). If you want to use Taboo with a VPN, may I suggest putting some money in an envelope and sending it to Mullvad? Adding this on after reading your comments elsewhere in the threads: If you're worried about Intellectual Property enforcers using this for evil, I wouldn't worry too much about it. BTDigg already exists. Not that what you're doing isn't novel or exciting (great work on this BTW) - but DHT indexing is an art thats already being practiced. Cat is out of the bag.
- Onanymous 4y ago
  I like how clearly you distinguish "evil" and "not legal" :)
ehPReth 4y ago
please do!
effnorwood 4y ago
yieldcrv 4y ago
yeah release it
retonato 4y ago
I have been operating a BitTorrent search engine for a few years (it is closed now), here is some advice if you want to launch it as a public website: 1. There is no absence of people opening the kind of torrent indexers, which you have in mind. In 2010s there were 5-10 such sites launched each year, even nowadays there are at least a few new indexers yearly. Almost all are closed within a year or two (at most). There are reasons for that, some are less obvious than others. 2. Most likely you will close your site after some time as well, here is why: 2.1. It will be difficult to find a hosting, which will tolerate it. Forget about Linode or Digital Ocean or any similar hosting providers - they will suspend your server (first) and account (later) after receiving a few automated DCMA emails. You can use some "second-tier" providers for some time, but eventually even they will get tired of you, you will be out the moment they receive the first paper letter from some law company, which represents some movie company or alike. You can use proxy servers or some other arrangement for some time, but eventually you will decide that it isn't worth the effort. 2.2. The same thing is true for domain provider. Yes, there are a few, which are more resilient, than others, but don't expect that they will protect you and your domain indefinitely for $15 per year. If your site is popular enough - the only option may remain to register a new domain every few months and hop between them. Eventually you will get tired of that. It may take half a year or a few years, but you will. 2.3. There are no money in torrenting nowadays. Forget about ads or donations - even thepiratebay cannot earn money that way, you will not be able to do that either. 2.4. Sadly, torrents are not that popular nowadays as they were 5-10-15 years ago. Some people still use them, but in most developed countries that's more like an exception, than the rule. The rule is netflix, spotify and alike. As a result, if you imagine large and happy community of users around your site - just don't, most likely there won't be any. 2.5. Don't expect just to launch a site and rest. You will spend at least 5-10-20 hours per week on its maintenance (fixing bugs, importing/cleaning data, adding new features, etc). It will feel fine at first, but more and more tiring as the months and years go by. Eventually you will get bored and stop any maintenance. Users don't like old/unmaintained sites, so they will be less and less interested in it. Eventually you will decide, that it isn't worth the effort to run it at all.
- retonato 4y ago
  Don't underestimate possible legal problems. Tech people tend to simply throw them aside ("I will use a VPN/Cloudflare/Onion, no one will find me"), but here are my 5 cents: 1. Most likely - you site will not gain any significant popularity and you will decide to close it on your own. Meanwhile you will just be getting some automated DMCA emails/complaints from time to time, that's it. 2. Less likely - you site will be somewhat popular, it will become an attractive target for copyright owners and law firms (not as attractive as thepiratebay, but still). It usually takes just a month or two for some interested law enforcement agency to find out who you are (forget about all privacy protection methods you know, it is very easy to slip and eventually you will slip; you need to be lucky every time, they need to be lucky just once). Depending on what country you live, you will end up with a few thousand dollar fine (best case) or tens/hundreds of thousands + a small jail sentence + a criminal record. Not bad for a hobby torrent site with no income and a minimal community, is it? Just to reiterate this one more time - it is VERY easy to find out who you are for any interested government agency [in your country], most of those 1-2 months will be spent waiting for replies for their letters and other similar bureaucracy, not for some CSI stuff. Here is how it might go. First they will check the site domain and IP address and ask your hosting/domain providers to reveal you payment details (name/address on credit card). That's enough in 95% of all cases. Even if you use Cloudflare to hide your site IP and even if you use bitcoins to pay for everything - that will help only for some time, eventually Cloudflare will reveal your server IP and bitcoin transaction will be tracked back to your credit card. That's just a question of time and whether you are "wanted" enough or not.
acehw 4y ago
it's a tool. What people use it for is their business.
joeman1000 4y ago
Please release it. We are in dire need of good torrent search without ad-trackers or other nasty stuff.
bArray 4y ago
At the very least, please do a write-up for us on how you're achieving 5-10k announcements a second.
gloosx 4y ago
You should def release it, what else can you do with it? Just let it rot and fade away?
dontbenebby 4y ago
>Now, I am thinking about releasing it as open-source for others to study, but not sure if I should, because it might be used for "evil". For evil? I wouldn't worry about that. Not now. You should put prominent warnings it's not "consumer grade" or whatever, but I think there's more value in sharing your code than there is risk someone will perform an attack they otherwise couldn't. Conversely, the second amendment wasn't just meant to apply to guns -- in America, "arms" can absolutely mean "cyber". Now, to be fair... the NRA is basically a way for boomers old enough to get "reoccurring income" to write off donations to the GRU on their taxes at this point, but there was a time in this country[0] when they used to give people who had a re-occurring subscription were sent a video called "Stop! Don't touch!!" (or something to that effect), which was meant to be their first lesson on guns -- and make no mistake, when you "cyber", you're reaching into the toolbox. There was a period in the 2000s where it was EXTREMELY difficult to get some of these tools up and running, and then, in parallel, you could also experience hardware or driver issues, and people quite rightfully used to scare the everloving shit out of the type of person who would bring esoteric knowledge to light too quickly. Even if you knew exactly which commands to run, in which order, you had to deal with stuff like the fact the drivers for Airport literally won't go into monitor mode, which is when you tell the radio in your laptop to store not just the packets addressed to it, but any packets that happen to... drift on by. I really cannot emphasize enough how hilarious it was to me that it took until about twenty goddamn twenty[-1] for some folks to realize the main benefit to "cyber" is that it's remote -- since I was about twelve years old, I've run into people who do... very rude things with the computer, paired with wielding the fact that in the united states, possession of what is now being called "CSAM" was what's called a "strict liability" offense -- pair that with gatekeeping access to the title "security researcher" and it led to a very uneven playing field. (They'd also do things like say oh, gee, you can't get a security clearance if you engage in software piracy, while also telling folks it's probably better to just torrent stuff than use some shady torrent site... and mentioning they have a security clearance in another thread or whatever.) Now, if this program was, say, a script that spiders through a hard drive using the Luhn algorithm[1] to suss out if the drive contains PII or automates bringing down one's wireless interface, changing the MAC address, then bringing it back up... that might be something you might not want to give to your enemies by putting onto Github or whatever. Myself, I usually still stick to the Pirate Bay -- I've still got a backlog of stuff I haven't watched... it feels like just a short time ago I was getting all emo I had no one to watch "Cats"[2] with, ha-HA!! But this? I think you're good to go, and I thank you for taking the time to learn the version control system and share your code. If anything, you'll be aiding public health. Streaming services are abusive... I own very few movies since I've moved around so much -- it became a running gag with my exes -- but it's like back when cable briefly wasn't going to have commercials, then within pretty much one generation they added them right back in addition to collecting the subscription money[3]... those sorts of people should be shown that video from the 80s of the one KGB defector explaining that America is unique, and there is nowhere else to defect to. (He was right, and if you abuse your access folks might be unwelcoming.) -- [-1] I purposefully waited about two years to make that joke... at least two times. [0] (I'm posting from my home) [1] I was told it's used to verify something is a CC # but apparently it can get some false positives -- that wasn't mentioned last time I looked it up, hehe: https://en.wikipedia.org/wiki/Luhn_algorithm https://en.wikipedia.org/wiki/Luhn_algorithm [2] Did you people forget that they called it Redphone because it was supposed to REDUCE tensions? [3] https://web.archive.org/web/20150501092025/nytimes.com/1981/07/26/arts/will-cable-tv-be-invaded-by-commercials.html https://web.archive.org/web/20150501092025/nytimes.com/1981/...