19 ms·
Bitmagnet Allows People to Run Their Own Decentralized Torrent Indexer Locally
- ranger_danger 3y agoIsn't this dangerous? I thought that in order to get details of any torrent from the DHT you must connect to it. This would automatically set off DMCA complaints from MPAA/RIAA etc. right? (also how do they ever prove you actually downloaded any usable part of a torrent?!)
- anacrolix 3y agoYou must connect to peers yes, but metadata is separate to torrent data.
- ranger_danger 3y agobut copyright holders do not need to show you actually DID anything besides connect to the torrent in order to accuse you, that's all they ever do...
- jMyles 3y agoInteresting. I want to seamlessly distribute my music through bittorrent, but since I have a small fan base (and thus, a very small potential seeding pool), I found it difficult to connect all the moving parts. I'll give Bitmagnet a try for an Indexer. I did find that IPFS, with a pinning service (I won't shill the one I used in particular), was a bit easier and just worked for everyone who tried to use it. But I'd like to get my bittorrent presence up to the "just works" level also.
- 8organicbits 3y agoHave you considered setting up a web seed? https://www.ubuntubuzz.com/2021/06/how-to-add-web-seeds-to-torrent-download.html?m=1 https://www.ubuntubuzz.com/2021/06/how-to-add-web-seeds-to-t...
- jMyles 3y agoNo I hadn't! This seems exciting. Have you done it successfully?
- evulhotdog 3y agoIt is pretty common for open source distribution of operating systems, and it works well.
- deleted 3y ago[deleted]
- KoftaBob 3y agoWell for your use case of distributing your own music, Bitmagnet wouldn't be necessary. What a DHT crawler like Bitmagnet does is the following: 1. Take a few initial "bootstrap" torrents and ping them to see which IP addresses are seeding that file 2. Ping those IP addresses and ask what other files/torrents they're seeding 3. Ping those torrents to see which IP addresses are seeding that torrent Rinse repeat. So to distribute your music to fans, you'd just want to put magnet links to your music on your site.
- gkbrk 3y agoYou don't need to ping anyone to crawl the DHT. You can passively wait and you'll get DHT queries in the form of "I'm looking for people seeding XYZ. Do you have a list?". You can just save those somewhere and you'll accumulate a list of active and new torrents. Writing a DHT crawler is super fun, I suggest everyone to get a cheap VM and write/run one.
- pluto_modadic 3y agothat'd be the slow mode, though
- jMyles 3y agoIt's kinda like the owl with the tootsie pop: How many cups of coffee does it take to internalize hamming distance? :-)
- the8472 3y agoThat's not correct. Torrent swarms and the DHT are separate. Each torrent basically forms its own small network of TCP connections to exchange data specific to that one torrent. While the DHT is a network shared by all clients that speak the protocol and it's carried over short-lived UDP query-response exchanges.
- pessimizer 3y agoYou have to be participating in a torrent swarm in order to bootstrap yourself into DHT at all. Bittorrent's DHT is not a network independent of torrent swarms. You need the address of a peer who is already part of the network in order to join it, and you have to get a list those addresses from somewhere.
- johnchristopher 3y agoWon't you run into the problem that you will have to generate torrents every time you update your music collection ?
- rakoo 3y agoYou don't need an indexer for that. All you need is a static list with all your music, and people can click on links to download your music. Give them qbittorrent and it'll be running smoothly.
- jMyles 3y agoSure, but I'd like to go beyond magnet links and make it properly decentralized, with the discovery happening in the DHT and torrent swarm.
- anacrolix 3y agoMaybe check out how it's done here: https://archive.org/details/the-fanimatrix-div-x-5.1-hq https://archive.org/details/the-fanimatrix-div-x-5.1-hq. They don't seem to provide magnet links, but basically you create a torrent from your music. Host it somewhere (Cloudflare R2 would be good for free egress) in the right structure. Add the webseed endpoint to your torrent file and create a magnet link. Put all this stuff on a website in the download section. Let your users download it however they want.
- rakoo 3y agoThere is no point in running your own indexer if it's only for your own music though. If your goal is to run a more "communal" indexing point, it also makes sense to build it cooperatively with other artists, listing only content you all care about and want to distribute (rather than doing a filtering after the fact with a general indexer) I like your idea of giving its name back to a technology that just works better, and I think it pairs really well with giving back a humane touch to how we use the internet, computers, and putting people back at the center rather than thinking about tools first !
- CharlesW 3y ago> I want to seamlessly distribute my music through bittorrent, but since I have a small fan base (and thus, a very small potential seeding pool), I found it difficult to connect all the moving parts. Out of curiosity, why not just distribute via R2 or similar, or archive.org if you don't need AuthN/AuthZ? What's the complexity (to you and listeners) of BitTorrent buy you?
- jMyles 3y agoWell, I do distribute via some centralized platforms (including some odious ones like Spotify). But I'd like to put forward a practice that demonstrates that the tools that have been smeared as anti-artist (chiefly but not only bittorrent) are actually compelling tools for independent distribution.
- CharlesW 3y agoThanks, I love it!
- felixg3 3y agoPost your magnet link and I’ll host it on my Amsterdam-based server
- throwing_away 3y agoYou can also encourage people to host your content with IPFS, removing the need for a pinning service if there's enough of them.
- factormeta 3y agoTry https://webtorrent.io https://webtorrent.io
- rakoo 3y agoAnother alternative is https://github.com/the8472/mldht https://github.com/the8472/mldht which, contrary to magnetico, strives to be a nice citizen (its author is active in the bittorrent community AFAIU)
- mgdigital 3y agoI have worked with the8472 to get Bitmagnet's BEP5 & BEP51 implementations working and ensure it's a good citizen on the network - there is more to be done and more protocols to be implemented, but unlike Magnetico, BM is not simply scraping without responding to incoming requests.
- the8472 3y agoWe had a discussion here https://github.com/bitmagnet-io/bitmagnet/issues/11 https://github.com/bitmagnet-io/bitmagnet/issues/11 With the related changes bitmagnet shouldn't have the blatant misbehavior of magnetico (anymore). Though I haven't looked at its in-the-wild behavior, so I can't vouch for how spec-compliant the implementation plays in practice.
- mgdigital 3y agoI'm also open to more feedback, if there are specific areas that need attention let me know :)
- swayvil 3y agoGoogled the bittorrent community with the acronym "AFAIU". Lol.
- bangonkeyboard 3y agoWho remembers Skytorrents (https://news.ycombinator.com/item?id=13423629 https://news.ycombinator.com/item?id=13423629)? Posted as a "Show HN" here, it was a DHT-sourced index and stack written in C with no JavaScript, no cookies, no ads, no tracking. Skytorrents was unbelievably fast, friendly, and complete, and this translated into rapid adoption and traffic growth that caused the site to shut down due to server costs after just a year (https://torrentfreak.com/skytorrents-dumps-massive-torrent-database-and-shuts-down180221/ https://torrentfreak.com/skytorrents-dumps-massive-torrent-d...). It was a shame that the technology behind Skytorrents was never open-sourced; it was the best torrent crawler and site I've ever seen, and I would have liked to see how it worked so well.
- mgdigital 3y agoI guess the takeaway is that a public torrent site can (probably) never be sustainable while acting in users' best interests?
- evanjrowley 3y agoSkytorrentz was great, but I never knew it had been coded in C nor presented here. Thanks for the background info.
- deleted 3y ago[deleted]
- bawolff 3y agoI feel like writing a fast torrent tracker isn't particularly hard. The hard part is the legal risk.
- nadermx 3y agoThis can't be underestimated. Legal is by and large the largest risk of a torrent site. Maybe one day..
- 3abiton 3y agoNever heard of it, what an interesting read
- chopsuey5540 3y agoThis looks interesting but I’m a bit worried about the CSAM / illegal stuff part, could a user get in trouble because he has traces of that in his crawled index? Also, how large is the index after indexing for a few months?
- rakoo 3y agoAn indexer doesn't download content. The only information you'll have is the name of a torrent, potentially its files, and who is interested in those files. But that's the technical view, what happens in court might be totally different.
- Scion9066 3y agoIn order to get the information such as the name of the torrent and its files from the hash you do need to connect to someone in the swarm to download that metadata. You won't know what it is until after you've already connected.
- toyg 3y agoConnecting to an unknown machine and asking what they have, is like knocking on a stranger's door and asking what they're selling. Them mentioning something nefarious and you leaving in response, is very obviously not a crime.
- rakoo 3y agoThere probably are nefarious content you can see just from the filenames but not everything is like that. Moreover, you "only" know they distribute it, you don't do it yourself.
- MomoXenosaga 3y agoConsidering many countries block torrent sites I wouldn't chance it.
- ZoomZoomZoom 3y ago
- belorn 3y agoThe really interesting part for me is what this technology might lead towards in the end, which is decentralized community based curation. An index with white listed curation would be indistinguishable from a website, but it would not need a domain name nor a ip address to function.
- imtringued 3y agoThe problem with decentralised software is that you don't want to host other people's illegal content. I once tried out zeronet, which downloads the entire decentralised website and anyone can post things to it. Although I have not found CSAM directly on their Reddit equivalent, there are people posting advertisements to zeronet CSAM sites. The idea that I am downloading and automatically redistributing content like that is disturbing and zeronet is dead for a good reason. It's a pool that is asking to be peed in, even if the abusers themselves are a tiny minority.
- mgdigital 3y agoBitmagnet may download metadata about CSAM content, which is automatically deleted with fairly high accuracy. You would never be redistributing it. No outgoing peer protocol is currently implemented. This is planned but it will give users control of what they're sharing rather than indiscriminately sharing everything.
- anacrolix 3y agohttps://www.coveapp.info/ https://www.coveapp.info/ approaches this in a similar fashion. I don't believe there's any legal issues in collecting metadata in an automated fashion from a public network. So creating a search index for personal use from this is fine. However if you do click and start seeding questionable content, then it becomes an issue. https://www.coveapp.info/#dht-indexer https://www.coveapp.info/#dht-indexer
- jesprenj 3y agoNice! I made my own DHT crawler in C for a science project in high school. git: http://ni.4a.si./sijanec/travnik/tree/src/dht.c http://ni.4a.si./sijanec/travnik/tree/src/dht.c
- jesprenj 3y agoImplementing sample_infohashes opens your torrent client to abuse. UDP responses are much larger than queries and this protocol (BEP51) allows attackers that can spoof source IP addresses to use a large number of clients as mules for amplified distributed denial of service attacks.
- mgdigital 3y agoBitmagnet has rate limiting on incoming UDP requests (both overall and per-IP) so I don't know that it would be vulnerable; if there's anything else that should be done to mitigate any risk I'd like to know.
- pluto_modadic 3y agooutgoing rate limits, and also it could exhaust the overall rate with useless traffic.
- mgdigital 3y agoIncoming and outgoing are both limited, I think the worst such an attack could do is prevent responding to legitimate incoming queries - this shouldn't slow down the DHT crawler in a noticeable way.
- jesprenj 3y agoI do not mean bitmagnet here, I mean other bittorrent clients that respond to sample_infohashes.
- p1mrx 3y agoHas anyone tried to fix this by making queries bigger, and limiting the response to the size of the query?
- the8472 3y agoIndividual DHT nodes should only see a trickle of packets. A few kilobytes per second. Even less per remote IP. So they can set fairly strict rate limits. And ISPs are in a much better position to solve this problem anyway. They should ask for higher peering fees from peers that don't do source-filtering.
- UberFly 3y ago"Unlike well-moderated torrent sites, Bitmagnet adds almost any torrent it finds to its database. This includes mislabeled files, malware-ridden releases, and potentially illegal content. The software tries to limit abuse by filtering metadata for CSAM content, however." This is by far the biggest hurdle to something like this. You'll eventually have to end up again with a centralized curator.
- mgdigital 3y agoOr connect with trusted curators over a decentralized network, similar to the fediverse?
- UberFly 3y agoOr a baked-in a use-based rating system. That would be cool if it could be protected from abuse.
- paulmd 3y agoThen you have a user reputation problem instead… me and my 5 million alts say this is the REAL korn_blind.mp3.exe. Nothing is new under the sun since the internet was created, these are the same problems that email has tried and failed to solve for 50 years.
- idiotsecant 3y agoIts going to be an unpopular take, but crypto solves this. As in currency. Make people attest to things with locking a very small amount of money.
- paulmd 3y agoBitcoin originally evolved from the "hashcash" proof-of-work system, which was intended to be a scalable anti-spam measure (force an attacker to generate a unique proof that is expensive to generate but cheap to verify). And you are basically describing proof-of-stake (force participants to stake, and if they misbehave you slash their stake). I wasn't kidding about "nothing is new under the sun" ;) In general though others are right that the problem often is that attackers are more willing to play these games than individuals - you are not willing to put up 10 bucks to post on a web forum, but an attacker might think $10 to get a wonderful commercial offer delivered to 1000 people sounds really good! You won't spend 10c to upvote each individual song, but the RIAA will! Etc. It needs to be highly asymmetrical, and ideally have minimal/zero cost to "honest" users with an exponential penalty for attackers. Obviously we are not living in a post-email-spam world unfortunately, and what we have is basically the "lightning network" with gatekeepers offloading and centralizing a lot of the problem so users don't have to deal with it. But you aren't the first person to make the observation that these are related problems!
- lupusreal 3y agoThe web UI is super slow. It's slow with the default view of 10 results per page and unbearably slower at 100. These aren't big numbers and my computer isn't old, something was poorly done here and setting the default number of results per page to 10 clearly is not the correct fix. (For the record it only has a mere thousand results so far..) Also, the content classification is so bad it might as well not exist. How does a torrent with "Playboy" in the title get classified as 'Unknown' instead of 'XXX'? Even torrents with "Porn" or "XXX" in the title get classified as Unknown. A simple Bayesian classifier should have this covered, it's not a task that needs heavy duty AI to solve. This whole thing seems way too half-baked for me.
- anacrolix 3y agoTry https://www.coveapp.info/ https://www.coveapp.info/, I mentioned it here: https://news.ycombinator.com/item?id=39425381 https://news.ycombinator.com/item?id=39425381. I wrote a custom search implementation for torrents that uses keyword matching and boosts search scores based on popularity of torrents on the DHT. It feels very much like TPB search but not so aggressively focused on seeder counts (i.e. better matching on file names can outrank a higher seeder count).
- schemathings 3y agoWhy is there an ffmpeg dependency if the app is only working with metadata?
- anacrolix 3y agoIt uses ffprobe for metadata, and ffmpeg for transcoding.
- schemathings 3y agoMakese sense, thanks.
- viraptor 3y ago
- infogulch 3y agoHow does Bitmagnet compare to Aquatic? https://github.com/greatest-ape/aquatic https://github.com/greatest-ape/aquatic
- anacrolix 3y agoI'm the author of https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent (started in 2013) and https://github.com/anacrolix/dht https://github.com/anacrolix/dht (started in 2015). I have a DHT indexer implementation I developed in 2021. It's currently closed source but available for use as part of https://www.coveapp.info/ https://www.coveapp.info/. I have found that after several hours the search is excellent and stays up to date with ease.
- mgdigital 3y agoSome of the features in your screenshots look interesting, but why is Cove closed source?
- anacrolix 3y agoMainly because I am not ready to share my DHT indexing implementation. It's the culmination of years of working with BitTorrent and DHTs and I'd like to get something back for it one day. However I do dogfood Cove and want people to have that experience too.
- pdimitar 3y agoI would think the only thing you would get out of such work is a sternly worded cease and desist from a big entity so you might as well open-source this and let somebody else take the torch in case you are shut down. I mean I'd use such software (and huge DBs like that of Skytorrentz) for research purposes because to me distribution is hugely interesting as a hobby but many courts won't see it that way, and we know a lot of them are influenced by copyrights holders.
- anacrolix 3y agoThanks for your insight!
- aaron695 3y ago[dead]