9 ms·

I keep wondering all the time why torrent search is based on websites (centralized), which can be taken down, etc., while once you have a torrent file or a magn

I keep wondering all the time why torrent search is based on websites (centralized), which can be taken down, etc., while once you have a torrent file or a magnet/hash everything is distributed. Is there a main reason why there isn't (AFAIK, even though I haven't really researched) a distributed search that wouldn't have these problems? Is it a tech problem that literally can't be solved? Or it just hasn't been done? It seems like search is the obvious weak link, since the websites keep disappearing or taken down or blocked by governments and ISPs, etc.

dmos62 3y ago
That's a good question, but the answers are somewhat sad. I see the other commenters saying it's not about the search, but about the curation. Curation or identification (as in, "who's the author?") is essential, but decentralized search is non-existent too. Yes, there is unmaintained, resource-hungry, locally-run, unscalable software out there that you can use, but there really isn't a public search engine for this. Which is a shame. I really hope someone will tell me I missed something, but I'm not holding my breath :( Edit: found mentions of https://btdig.com https://btdig.com and https://bt4g.org https://bt4g.org. I wasn't aware of the latter. A problem with the former is that it doesn't track number of peers.
drexlspivey 3y ago
How is it distributed? Don’t you need a torrent tracker? How do you know which peer has which files to share?
- zolland 3y ago
  Here is a good resource https://wiki.theory.org/BitTorrentSpecification https://wiki.theory.org/BitTorrentSpecification
grishka 3y ago
Decentralized search is a complex problem and there exist different approaches to it with different degree of centralization and resource requirements. DC++ is a bit more decentralized than BitTorrent. There still are central servers ("hubs"), but they don't even host any metadata. Search works by the hub broadcasting all search queries to all online peers and them replying with results if they have any. The file transfers themselves are p2p. I have an idea that's kind of more decentralized. Initially envisioned as a missing global search feature for the fediverse, but can be adapter for anything that has a similar network structure. A server has a number of peers already established because of the ActivityPub federation. Each server would send to its peers some kind of bloom filter that determines the tags or keywords that this server has results for. Then, when searching, your server would find the peers who are likely to have what you want, and only send your search query to them. If there aren't any, then it would send your query to the peers that have most users (with some random bias for load balancing purposes) because they're likely to have more connectivity, and they would point you where to look based on their own peers and their bloom filters. There would also need to be some kind of reputation system (centralized server lists? p2p exchange of scores/reports?) so that servers that return spam or intentionally wrong results would get punished. This could probably be made to work in a fully-decentralized p2p network, but I imagine it would be too easy to abuse. Getting a new domain costs money, yet getting a new IP or public key is free and easy.
- boredcaveman 3y ago
  boredcaveman (at] tutanota d0t com
r3trohack3r 3y ago
Mining the DHT for content isn’t particularly difficult, just time consuming and noisy. For example, here is a personal DHT monitor where you can view what’s being announced on the DHT: https://github.com/retrohacker/taboo https://github.com/retrohacker/taboo
hoschicz 3y ago
You can easily decentralize search, but not fetching seed/leech counts essentially
chakintosh 3y ago
r/DataHoarder are sharing their backups of Magnet links and torrent files they backed up for years and are trying to bring the site back from the dead. For torrent sites, nothing stops you really from scraping the site for torrents. As long as the file itself is being seeded, it'll be impossible to take down. It IS decentralized.
Tuna-Fish 3y ago
The problem is not search, it's curation. Any system that accepts everything will be overrun with things you do not want (fakes, *very* illegal things), and so there needs to be some authority that determines what is allowed. At that point, you are centralized, and hosting a website doesn't significantly hurt you. This is also the problem with all distributed social networks. In the end, your options are formal centralization, and informal centralization, because absolutely nobody wants to live in true decentralization.
- throwawayiddqd2 3y ago
  Most of the time when I search for something, I sort by more seeders first. Some type of signature can be used too.
  - ryandrake 3y ago
    Sorting by number of seeders will probably get you a file that is fast to download because of the number of seeders, but it will rarely get you quality. Let's go take a stroll over to a popular public torrent site and search for, say, a recent superhero movie. I see 100 (!) results for that movie's name. 49 of them have zero seeders at all. I don't know what even is the point. 29 of the results have one seeder. So already, 78% of the results are pure crap. Let's look at the top result with 338 seeders: File is 3GB, H.264 video, 1080p, but with a crappy stereo AAC audio encoding... arrggh why?? Number 2 result with 84 seeders: 1.43GB, H.264 video, 720p, no word on the quality of the audio encoding. Even more worthless. Number 3 result with 17 seeders: HEVC format, 2160p, audio streams include TrueHD Atmos 7.1, DTS-HD, Dolby Digital 5.1, stereo, and three non-English language streams. But, with an eye-watering download size of 61GB. Holy shit! Nice, but wow, what a download. You have to go a few more down the list to find a good balance of high quality video and audio encoding, but with a reasonable file size. By that point you're in the single digit number of seeders. Don't get me wrong, it's great to have a few choices and quality trade-offs. I guess there's someone out there who doesn't care about the stereo audio because they watch their movies with laptop speakers. But 100 results, with the vast majority of them either unseeded, poorly-seeded, or flawed in some way. I agree with OP: You definitely want some curation, not just search!
maeln 3y ago
There is many search engine that use the DHT to retrieve the metadata and share torrent via the magnet. The main issue here is not the tech, it's the trust and discoverability/curating and also not spreading the seeding capabilities. When downloading from "reputable torrent tracker XYZ" you can trust the quality of the torrent, that it is virus free, etc ... It is also usually make searching for particular torrent easier (less like searching for a nail in a hay stack) and you avoid spreading the seeding potential to hundreds of similar torrents. As a extreme example, BakaBT (a private torrent for anime/manga related torrent) has a strict "no duplicate torrent" policy. This means that if you are searching for the OST of a specific show, you will have usually only one result and it's the most up-to-date, highest quality version. Since it is the only option, everyone seed this one. It really diminish the issue of abandoned torrents. To "replace" and existing torrent, you have to provide a strictly better version. A decentralized torrent search engine could not do that. The real value of torrent tracker are the community. That is also why decentralized software like eMule/eDonkey lost a lot of popularity to torrent tracker: Lots of duplication, very dodgy download, no curation, virus, ....
- deleted 3y ago
  [deleted]
- SergeAx 3y ago
  > reputable torrent tracker I once visited RarBG with uBlock Origin turned off by accident. The intensity of the shitstorm of fake links, transparent GIFs on top of the content and other stuff like that was overwhelming. I believe that it is universally stupid to trust torrent sites with anything.
- Solvency 3y ago
  This makes sense but...I'm being fully serious and earnest here and revealing my own naïveté: Couldn't some form of blockchain work here? Like couldn't some form of distributed/democratized community curation and moderation happen by using the blockchain to manage the arbitration of new torrents (and their successors, like when the community decides New Random Anime X encoding to be a superior copy)? Plus you have proof of stake or whatever the leading mechanism is to help combat and filter out fakes/illegal activity (etc)? Then you'd have blockchain managing the trackers and torrents managing the file sharing.
rakoo 3y ago
You can't build a decentralized search because there's no way to trust whatever results you get until you actually build it. If you don't want to rely on a specific community, your best bet is to crawl yourself and search locally: you can do that with magnetico (https://github.com/boramalper/magnetico/ https://github.com/boramalper/magnetico/). Don't be frightened by the fact that it is archived, it works. The problem then will be, how do you make sure your content is legit? There's no magic way here, the best thing you can do is compare the number of seeders and aim for the highest. If a torrent is fake, people will delete it and it won't be seeded. I have a thingy for that: https://sr.ht/~rakoo/magneticos/ https://sr.ht/~rakoo/magneticos/ The problem then becomes, number of seeders naturally selects towards popular content. It doesn't ensure viability of content. But I don't think there's a technical answer to that.
- xenago 3y ago
  Very interesting, thanks for sharing your project. Magnetico is also integrated into Jackett I believe, which might be helpful to some people who use that

dmos62 3y ago

That's a good question, but the answers are somewhat sad. I see the other commenters saying it's not about the search, but about the curation. Curation or identification (as in, "who's the author?") is essential, but decentralized search is non-existent too. Yes, there is unmaintained, resource-hungry, locally-run, unscalable software out there that you can use, but there really isn't a public search engine for this. Which is a shame. I really hope someone will tell me I missed something, but I'm not holding my breath :( Edit: found mentions of https://btdig.com https://btdig.com and https://bt4g.org https://bt4g.org. I wasn't aware of the latter. A problem with the former is that it doesn't track number of peers.

drexlspivey 3y ago

How is it distributed? Don’t you need a torrent tracker? How do you know which peer has which files to share?

zolland 3y ago
Here is a good resource https://wiki.theory.org/BitTorrentSpecification https://wiki.theory.org/BitTorrentSpecification

grishka 3y ago

Decentralized search is a complex problem and there exist different approaches to it with different degree of centralization and resource requirements. DC++ is a bit more decentralized than BitTorrent. There still are central servers ("hubs"), but they don't even host any metadata. Search works by the hub broadcasting all search queries to all online peers and them replying with results if they have any. The file transfers themselves are p2p. I have an idea that's kind of more decentralized. Initially envisioned as a missing global search feature for the fediverse, but can be adapter for anything that has a similar network structure. A server has a number of peers already established because of the ActivityPub federation. Each server would send to its peers some kind of bloom filter that determines the tags or keywords that this server has results for. Then, when searching, your server would find the peers who are likely to have what you want, and only send your search query to them. If there aren't any, then it would send your query to the peers that have most users (with some random bias for load balancing purposes) because they're likely to have more connectivity, and they would point you where to look based on their own peers and their bloom filters. There would also need to be some kind of reputation system (centralized server lists? p2p exchange of scores/reports?) so that servers that return spam or intentionally wrong results would get punished. This could probably be made to work in a fully-decentralized p2p network, but I imagine it would be too easy to abuse. Getting a new domain costs money, yet getting a new IP or public key is free and easy.

boredcaveman 3y ago
boredcaveman (at] tutanota d0t com

r3trohack3r 3y ago

Mining the DHT for content isn’t particularly difficult, just time consuming and noisy. For example, here is a personal DHT monitor where you can view what’s being announced on the DHT: https://github.com/retrohacker/taboo https://github.com/retrohacker/taboo

hoschicz 3y ago

You can easily decentralize search, but not fetching seed/leech counts essentially

chakintosh 3y ago

r/DataHoarder are sharing their backups of Magnet links and torrent files they backed up for years and are trying to bring the site back from the dead. For torrent sites, nothing stops you really from scraping the site for torrents. As long as the file itself is being seeded, it'll be impossible to take down. It IS decentralized.

Tuna-Fish 3y ago

The problem is not search, it's curation. Any system that accepts everything will be overrun with things you do not want (fakes, *very* illegal things), and so there needs to be some authority that determines what is allowed. At that point, you are centralized, and hosting a website doesn't significantly hurt you. This is also the problem with all distributed social networks. In the end, your options are formal centralization, and informal centralization, because absolutely nobody wants to live in true decentralization.

throwawayiddqd2 3y ago
Most of the time when I search for something, I sort by more seeders first. Some type of signature can be used too.
- ryandrake 3y ago
  Sorting by number of seeders will probably get you a file that is fast to download because of the number of seeders, but it will rarely get you quality. Let's go take a stroll over to a popular public torrent site and search for, say, a recent superhero movie. I see 100 (!) results for that movie's name. 49 of them have zero seeders at all. I don't know what even is the point. 29 of the results have one seeder. So already, 78% of the results are pure crap. Let's look at the top result with 338 seeders: File is 3GB, H.264 video, 1080p, but with a crappy stereo AAC audio encoding... arrggh why?? Number 2 result with 84 seeders: 1.43GB, H.264 video, 720p, no word on the quality of the audio encoding. Even more worthless. Number 3 result with 17 seeders: HEVC format, 2160p, audio streams include TrueHD Atmos 7.1, DTS-HD, Dolby Digital 5.1, stereo, and three non-English language streams. But, with an eye-watering download size of 61GB. Holy shit! Nice, but wow, what a download. You have to go a few more down the list to find a good balance of high quality video and audio encoding, but with a reasonable file size. By that point you're in the single digit number of seeders. Don't get me wrong, it's great to have a few choices and quality trade-offs. I guess there's someone out there who doesn't care about the stereo audio because they watch their movies with laptop speakers. But 100 results, with the vast majority of them either unseeded, poorly-seeded, or flawed in some way. I agree with OP: You definitely want some curation, not just search!

maeln 3y ago

There is many search engine that use the DHT to retrieve the metadata and share torrent via the magnet. The main issue here is not the tech, it's the trust and discoverability/curating and also not spreading the seeding capabilities. When downloading from "reputable torrent tracker XYZ" you can trust the quality of the torrent, that it is virus free, etc ... It is also usually make searching for particular torrent easier (less like searching for a nail in a hay stack) and you avoid spreading the seeding potential to hundreds of similar torrents. As a extreme example, BakaBT (a private torrent for anime/manga related torrent) has a strict "no duplicate torrent" policy. This means that if you are searching for the OST of a specific show, you will have usually only one result and it's the most up-to-date, highest quality version. Since it is the only option, everyone seed this one. It really diminish the issue of abandoned torrents. To "replace" and existing torrent, you have to provide a strictly better version. A decentralized torrent search engine could not do that. The real value of torrent tracker are the community. That is also why decentralized software like eMule/eDonkey lost a lot of popularity to torrent tracker: Lots of duplication, very dodgy download, no curation, virus, ....

deleted 3y ago
[deleted]
SergeAx 3y ago
> reputable torrent tracker I once visited RarBG with uBlock Origin turned off by accident. The intensity of the shitstorm of fake links, transparent GIFs on top of the content and other stuff like that was overwhelming. I believe that it is universally stupid to trust torrent sites with anything.
Solvency 3y ago
This makes sense but...I'm being fully serious and earnest here and revealing my own naïveté: Couldn't some form of blockchain work here? Like couldn't some form of distributed/democratized community curation and moderation happen by using the blockchain to manage the arbitration of new torrents (and their successors, like when the community decides New Random Anime X encoding to be a superior copy)? Plus you have proof of stake or whatever the leading mechanism is to help combat and filter out fakes/illegal activity (etc)? Then you'd have blockchain managing the trackers and torrents managing the file sharing.

rakoo 3y ago

You can't build a decentralized search because there's no way to trust whatever results you get until you actually build it. If you don't want to rely on a specific community, your best bet is to crawl yourself and search locally: you can do that with magnetico (https://github.com/boramalper/magnetico/ https://github.com/boramalper/magnetico/). Don't be frightened by the fact that it is archived, it works. The problem then will be, how do you make sure your content is legit? There's no magic way here, the best thing you can do is compare the number of seeders and aim for the highest. If a torrent is fake, people will delete it and it won't be seeded. I have a thingy for that: https://sr.ht/~rakoo/magneticos/ https://sr.ht/~rakoo/magneticos/ The problem then becomes, number of seeders naturally selects towards popular content. It doesn't ensure viability of content. But I don't think there's a technical answer to that.

xenago 3y ago
Very interesting, thanks for sharing your project. Magnetico is also integrated into Jackett I believe, which might be helpful to some people who use that