12 ms·

Show HN: Magnetissimo – Self-hosted torrent search engine written in Elixir

403 points by sergiotapia 10y ago

deleted 10y ago
[deleted]
dopkew 10y ago
Are there other programs that offer this same function?
- bharani_m 10y ago
  Not a self-hosted program. But, Skytorrents [1] was featured on HN recently [2]. [1] https://www.skytorrents.in/ https://www.skytorrents.in/ [2] https://news.ycombinator.com/item?id=13423629 https://news.ycombinator.com/item?id=13423629
  - patrickk 10y ago
    Another "inspirational" list thanks to the US government, from a HN comment[1] [1] https://news.ycombinator.com/item?id=13237667 https://news.ycombinator.com/item?id=13237667 [2] https://ustr.gov/sites/default/files/2016-Out-of-Cycle-Review-Notorious-Markets.pdf https://ustr.gov/sites/default/files/2016-Out-of-Cycle-Revie...
mrmondo 10y ago
Looks really interesting, I love goal 1: no JavaScript - YES! Thank you! And elixir is a very exciting language, especially to an ops person where when I read elixir code it actually makes sense to me and seems elegant. Ignoring my technical praise the objective of the product is solid, this is something I've been interested in for a while, I have a solution for usenet but not so much for torrents.
- marvel_boy 10y ago
  Absolutetly. +1 for the "no javascript" goal.
  - mrmondo 10y ago
    I'm so sick of 'designers' thinking they're programmers and smashing out some NodeJS or JavaScript heavy crap with no regards to performance or security. Elixir seems to attract some smart minds that think 'big picture' - obviously I'm generalising here and I'm not stating that this particular project is brilliant or anything but it is a trend I've observed.
    - huduo 10y ago
      NodeJS is a dependency in the usage instructions for Debian 7 and CentOS.
      - swiley 10y ago
        Ew, what for?
        luckystartup 10y ago
        The creators of Phoenix didn't want to write their own code to manage Javascript and CSS (e.g. rewriting the Rails "asset pipeline" in Elixir), so they just integrated Brunch [1], which is built on Node.js and NPM. I think this was an excellent decision. (Node.js is only needed during development and deployment of your assets. It is not used in production.) [1] http://brunch.io http://brunch.io
        dingdingdang 10y ago
        That is still pretty awful imo - think quite a few of the people considering Elixir and Phoenix are doing so explicitly to get away from having to Node.js close to their computers in the first place.
nikcub 10y ago
I'll be trying this out, it could work well with Cloud Torrent. I've largely moved away from Torrents tho and switched to using NZB's and apps that automatically download tv shows and films[0] Here is my stack: 1. NZB downloader - NZBGet - http://nzbget.net/ http://nzbget.net/ 2. TV Shows - Sonarr - https://sonarr.tv/ https://sonarr.tv/ 3. Films - CouchPotato - https://couchpota.to/ https://couchpota.to/ 4. NZB Search - NZBHydra - https://github.com/theotherp/nzbhydra https://github.com/theotherp/nzbhydra (I contribute to this project) 5. Plex - https://www.plex.tv/ https://www.plex.tv/ Add Torrent support: 6. Client - rtorrent - https://rakshasa.github.io/rtorrent/ https://rakshasa.github.io/rtorrent/ 7. Web Interface - ruTorrent (warning: PHP) - https://github.com/Novik/ruTorrent https://github.com/Novik/ruTorrent 8. Proxy private trackers to XML-RPC - Jackett - https://github.com/Jackett/Jackett https://github.com/Jackett/Jackett 9. Cloud Torrent - web interface - https://github.com/jpillora/cloud-torrent https://github.com/jpillora/cloud-torrent Additional: 10. Plex stats, analytics and user management - PlexPy - https://github.com/JonnyWong16/plexpy https://github.com/JonnyWong16/plexpy 11. Plex request - allow users to request content - https://github.com/ngovil21/PlexRequestChannel.bundle https://github.com/ngovil21/PlexRequestChannel.bundle Hardware is a pair of HP Proliant Gen8 microservers, Ubuntu 14, Docker, nginx and LetsEncrypt. There is no real easy way to set this all up, you have to do each part of the stack yourself (a docker-compose file would go a long way to simplifying it) [0] I spend over $200 a month on content subscriptions so I don't feel bad about utilizing the conveniance of NZB downloads + Plex
- patrickk 10y ago
  This is a insane setup, congrats ;) I thought I was doing well running flexget[1] on Ubuntu on a crappy old netbook. HTPC manager[2] might be worth a look too. [1] Setup guide: http://www.htpcbeginner.com/blueprint-flexget-setup-guide/ http://www.htpcbeginner.com/blueprint-flexget-setup-guide/ [2] http://htpc.io/ http://htpc.io/
- mrmondo 10y ago
  I'm the same but I don't torrent and I'm using sickrage rather than sonarr just because I find mono so unreliable, but I hear sonarr is better at finding older series.
- 10y ago
yoavm 10y ago
Looks like a useful tool, but I kinda wish this kind of programs had a CLI. It seems to be doing one main thing (searching), and it could be super useful to be able to pipe the results to a downloading program. Some programs can benefit vastly from a GUI, this... I'm just not sure what's the point. Kudos to the developer nevertheless for the tool itself and for the choice to go with Elixir.
- smcl 10y ago
  Sounds like you found yourself a wee evening task - hack a command line interface for this tool :D (edit: only messing - I was just thinking that if I wanted to learn Elixir hacking on some existing project is a good way)
sergiotapia 10y ago
Hope you guys like this and use it! If it's hard to use, tell me. My goal is to make this simple to run anywhere. Part of that will be to use Distillery to create compiled executables. I rewrote this after an initial version that used Redis as a queue to process. Asking end users to install Redis was a step too far I think, so I stepped back and thought about how to solve this without external dependencies. I ended up with a simple Elixir and Erlang queue implementation. It works much simpler, just as fast, and no freaky deps. GenServer starts up a worker for each crawler, and it schedules work by itself. It's really strange to use a language so complete, it kind of feels like cheating. Programming in easy-mode. Pull requests very welcome to the crawler folder, it's super easy to write a crawler for your favorite site! https://github.com/sergiotapia/magnetissimo/tree/master/lib/crawler https://github.com/sergiotapia/magnetissimo/tree/master/lib/...
- memla 10y ago
  If the goal is to make it simple to run, why not support SQLite?
  - brightball 10y ago
    Seems like the goal is avoiding dependencies and he's got 3 built in database choices for that already. DETS or Mnesia would probably work well for this.
    - sergiotapia 10y ago
      Thank you, I'll take a look at what those are. If there's a way to remove Postgres from the equation, I'm all for it.
      - macintux 10y ago
        DETS (and ETS, the in-memory equivalent) are quite simple for basic K/V-ish storage needs. If you need a relational database, Mnesia should do the trick.
      - 10y ago
brian-armstrong 10y ago
When someone posts about a GPL violation, people are quick to decry it, and rightly so. But this repo demonstrates, right in the first image, this tool being used to violate copyright, and we applaud that. Why do we have this double standard?
- arianvanp 10y ago
  They're the exact opposite from my ethic point of view. Gpl violation is a violation where the person didn't share information. Torrentting 'illegally' is the act of sharing information. If you're all for sharing an free flow of information.. I don't see how these two conflict I my ethical point of view.
  - digi_owl 10y ago
    The tricky bit is that GPL is built on top of copyright. In effect it use the legal precedence of copyright against itself.
    - cyborgx7 10y ago
      They weren't arguing from a legal, but from an ethics point of view.
    - eeZah7Ux 10y ago
      It's not "tricky", it's by design.
  - dibstern 10y ago
    In both scenarios the person's property is being stolen. Their property might be in the form of information, sure. But one product being more widely distributed does not make any ethical difference. The fundamental is that the information/property has been stolen in both cases.
    - dozzie 10y ago
      It's not a property in the first place. It's nowhere near like physical belongings and it shouldn't be treated as such.
deleted 10y ago
[deleted]
tmalsburg2 10y ago
After many years, Bittorrent doesn't cease to amaze me. Such a fantastic technology. Sad, though, that it is primarily known for illegitimate uses. This gives it a bad reputation and makes it hard to promote Bittorrent for more worthwhile applications. For instance, Bittorrent would be a perfect platform for sharing scientific data but when I suggest that to colleagues, they don't take it seriously because Bittorrent is perceived to be that tool for downloading pirated movies. I wonder how this barrier for adoption in academia could be overcome. Perhaps, with a separate network that is targeted at academic uses (think alt-coins). Call this network ResearchTorrent and researchers could use the technology without ever having to make contact with the whole ecosystem around pirated material. Universities could support this network without risking bad PR and getting in conflict with the music/film industry.
- tom_mellior 10y ago
  http://academictorrents.com/ http://academictorrents.com/ claims to have 16.33 TB of research data.
  - tmalsburg2 10y ago
    Last time I checked (3 years ago) this project didn't seem very active in terms of submission numbers and it appears that this hasn't really changed (7 submissions in Jan 2017). Not sure why, but the UI may be one reason.
    - tom_mellior 10y ago
      I don't think it's fair to expect scientific output to be comparable to Hollywood. Anyway, your colleagues think this is a pirate site?
- jzelinskie 10y ago
  I work on a BitTorrent tracker on the side[0] that's used for deploying software at a few companies, but has the unique feature of a middleware layer. Right now, there are two different middleware that require a signature of the infohash (files being shared) before allowing content to be shared. One is for JWTs[1] and upstreamed in the project and the other is for a blockchain and hasn't been upstreamed yet. Using features like this, they can run their tracker on the public internet and not worry about people sharing illegal content. I had never considered it, but this sounds like a great use case for academia, too. [0]: https://chihaya.io https://chihaya.io [1]: https://jwt.io https://jwt.io
fenollp 10y ago
You should look for sitemaps. They are often listed in robots.txt. Will save you lots of time! These sites have sitemaps: http://torrentproject.se/robots.txt http://torrentproject.se/robots.txt https://www.torrentdownloads.me/robots.txt https://www.torrentdownloads.me/robots.txt https://thepiratebay.org/robots.txt https://thepiratebay.org/robots.txt http://goldtorrents.com/robots.txt http://goldtorrents.com/robots.txt https://bitsnoop.com/robots.txt https://bitsnoop.com/robots.txt Enough with the hype: "high performance" What?? Your program is not concurrent: you have one process per website and go through URLs one at a time. You don't even use Bloom filters.
luckystartup 10y ago
I might have to start an anonymous GitHub account so that I can contribute to this. It would be fantastic if there was a way to start a download directly from the web UI. Many torrent clients can monitor an RSS feed, so that would probably be the best solution. I would add the ability to create a user account, and each user could set up a few RSS feeds. In my case, I would like to have one RSS feed with the label "Movies", and one for "TV Shows". Then the main search page would have buttons to add the magnet link to either RSS feed. I already use http://showrss.info http://showrss.info, which provides an RSS feed for all of my favorite shows. So it would be great to manage my own RSS feed for movies and specific episodes.
- the8472 10y ago
  > It would be fantastic if there was a way to start a download directly from the web UI. There are browser addons that can send magnets and torrents straight to the remote APIs of torrent clients.
listic 10y ago
Can anyone suggest a good torrent tracker suitable to run a torrent site? Opentracker [0] seems to be abandoned and the author doesn't reply. [0] http://erdgeist.org/arts/software/opentracker/ http://erdgeist.org/arts/software/opentracker/
- the8472 10y ago
  opentracker is what is used by the high-throughput trackers. the protocol is fairly simple and doesn't change a lot and the implementation is fairly minimal, so they don't need to update the codebase often, it does it's job, it's mature. for smaller sites there are more specialized trackers that allow more control, e.g. chihaya and gazelle.
kenning 10y ago
Has anyone hosted this somewhere yet? I was hoping I could just run it locally but you have to crawl all those sites every time, and it's not concurrent...
- uhryks 10y ago
  > leveraging Elixir's GenServer and Erlang's BEAM VM. > it's not concurrent (。´･ω･)?
  - sergiotapia 10y ago
    It's concurrent in that multiple crawlers run at the same time. But I do not concurrently crawl single websites. An initial version had that, and Magnetissimo easily crawled 600 torrents every page 3 seconds (not hyperbole). It just pounded websites and I didn't want that. It was bad karma.
- omnicoder 10y ago
  I just setup a test site based on it http://v.reni.com:4000/ http://v.reni.com:4000/
giridhar50 10y ago
very useful tool

403 points by sergiotapia 10y ago

deleted 10y ago

[deleted]

dopkew 10y ago

Are there other programs that offer this same function?

bharani_m 10y ago
Not a self-hosted program. But, Skytorrents [1] was featured on HN recently [2]. [1] https://www.skytorrents.in/ https://www.skytorrents.in/ [2] https://news.ycombinator.com/item?id=13423629 https://news.ycombinator.com/item?id=13423629
- patrickk 10y ago
  Another "inspirational" list thanks to the US government, from a HN comment[1] [1] https://news.ycombinator.com/item?id=13237667 https://news.ycombinator.com/item?id=13237667 [2] https://ustr.gov/sites/default/files/2016-Out-of-Cycle-Review-Notorious-Markets.pdf https://ustr.gov/sites/default/files/2016-Out-of-Cycle-Revie...

mrmondo 10y ago

Looks really interesting, I love goal 1: no JavaScript - YES! Thank you! And elixir is a very exciting language, especially to an ops person where when I read elixir code it actually makes sense to me and seems elegant. Ignoring my technical praise the objective of the product is solid, this is something I've been interested in for a while, I have a solution for usenet but not so much for torrents.

marvel_boy 10y ago
Absolutetly. +1 for the "no javascript" goal.
- mrmondo 10y ago
  I'm so sick of 'designers' thinking they're programmers and smashing out some NodeJS or JavaScript heavy crap with no regards to performance or security. Elixir seems to attract some smart minds that think 'big picture' - obviously I'm generalising here and I'm not stating that this particular project is brilliant or anything but it is a trend I've observed.
  - huduo 10y ago
    NodeJS is a dependency in the usage instructions for Debian 7 and CentOS.
    - swiley 10y ago
      Ew, what for?
      - luckystartup 10y ago
        The creators of Phoenix didn't want to write their own code to manage Javascript and CSS (e.g. rewriting the Rails "asset pipeline" in Elixir), so they just integrated Brunch [1], which is built on Node.js and NPM. I think this was an excellent decision. (Node.js is only needed during development and deployment of your assets. It is not used in production.) [1] http://brunch.io http://brunch.io
        dingdingdang 10y ago
        That is still pretty awful imo - think quite a few of the people considering Elixir and Phoenix are doing so explicitly to get away from having to Node.js close to their computers in the first place.

nikcub 10y ago

I'll be trying this out, it could work well with Cloud Torrent. I've largely moved away from Torrents tho and switched to using NZB's and apps that automatically download tv shows and films[0] Here is my stack: 1. NZB downloader - NZBGet - http://nzbget.net/ http://nzbget.net/ 2. TV Shows - Sonarr - https://sonarr.tv/ https://sonarr.tv/ 3. Films - CouchPotato - https://couchpota.to/ https://couchpota.to/ 4. NZB Search - NZBHydra - https://github.com/theotherp/nzbhydra https://github.com/theotherp/nzbhydra (I contribute to this project) 5. Plex - https://www.plex.tv/ https://www.plex.tv/ Add Torrent support: 6. Client - rtorrent - https://rakshasa.github.io/rtorrent/ https://rakshasa.github.io/rtorrent/ 7. Web Interface - ruTorrent (warning: PHP) - https://github.com/Novik/ruTorrent https://github.com/Novik/ruTorrent 8. Proxy private trackers to XML-RPC - Jackett - https://github.com/Jackett/Jackett https://github.com/Jackett/Jackett 9. Cloud Torrent - web interface - https://github.com/jpillora/cloud-torrent https://github.com/jpillora/cloud-torrent Additional: 10. Plex stats, analytics and user management - PlexPy - https://github.com/JonnyWong16/plexpy https://github.com/JonnyWong16/plexpy 11. Plex request - allow users to request content - https://github.com/ngovil21/PlexRequestChannel.bundle https://github.com/ngovil21/PlexRequestChannel.bundle Hardware is a pair of HP Proliant Gen8 microservers, Ubuntu 14, Docker, nginx and LetsEncrypt. There is no real easy way to set this all up, you have to do each part of the stack yourself (a docker-compose file would go a long way to simplifying it) [0] I spend over $200 a month on content subscriptions so I don't feel bad about utilizing the conveniance of NZB downloads + Plex

patrickk 10y ago
This is a insane setup, congrats ;) I thought I was doing well running flexget[1] on Ubuntu on a crappy old netbook. HTPC manager[2] might be worth a look too. [1] Setup guide: http://www.htpcbeginner.com/blueprint-flexget-setup-guide/ http://www.htpcbeginner.com/blueprint-flexget-setup-guide/ [2] http://htpc.io/ http://htpc.io/
mrmondo 10y ago
I'm the same but I don't torrent and I'm using sickrage rather than sonarr just because I find mono so unreliable, but I hear sonarr is better at finding older series.
10y ago

yoavm 10y ago

Looks like a useful tool, but I kinda wish this kind of programs had a CLI. It seems to be doing one main thing (searching), and it could be super useful to be able to pipe the results to a downloading program. Some programs can benefit vastly from a GUI, this... I'm just not sure what's the point. Kudos to the developer nevertheless for the tool itself and for the choice to go with Elixir.

smcl 10y ago
Sounds like you found yourself a wee evening task - hack a command line interface for this tool :D (edit: only messing - I was just thinking that if I wanted to learn Elixir hacking on some existing project is a good way)

sergiotapia 10y ago

Hope you guys like this and use it! If it's hard to use, tell me. My goal is to make this simple to run anywhere. Part of that will be to use Distillery to create compiled executables. I rewrote this after an initial version that used Redis as a queue to process. Asking end users to install Redis was a step too far I think, so I stepped back and thought about how to solve this without external dependencies. I ended up with a simple Elixir and Erlang queue implementation. It works much simpler, just as fast, and no freaky deps. GenServer starts up a worker for each crawler, and it schedules work by itself. It's really strange to use a language so complete, it kind of feels like cheating. Programming in easy-mode. Pull requests very welcome to the crawler folder, it's super easy to write a crawler for your favorite site! https://github.com/sergiotapia/magnetissimo/tree/master/lib/crawler https://github.com/sergiotapia/magnetissimo/tree/master/lib/...

memla 10y ago
If the goal is to make it simple to run, why not support SQLite?
- brightball 10y ago
  Seems like the goal is avoiding dependencies and he's got 3 built in database choices for that already. DETS or Mnesia would probably work well for this.
  - sergiotapia 10y ago
    Thank you, I'll take a look at what those are. If there's a way to remove Postgres from the equation, I'm all for it.
    - macintux 10y ago
      DETS (and ETS, the in-memory equivalent) are quite simple for basic K/V-ish storage needs. If you need a relational database, Mnesia should do the trick.
    - 10y ago

brian-armstrong 10y ago

When someone posts about a GPL violation, people are quick to decry it, and rightly so. But this repo demonstrates, right in the first image, this tool being used to violate copyright, and we applaud that. Why do we have this double standard?

arianvanp 10y ago
They're the exact opposite from my ethic point of view. Gpl violation is a violation where the person didn't share information. Torrentting 'illegally' is the act of sharing information. If you're all for sharing an free flow of information.. I don't see how these two conflict I my ethical point of view.
- digi_owl 10y ago
  The tricky bit is that GPL is built on top of copyright. In effect it use the legal precedence of copyright against itself.
  - cyborgx7 10y ago
    They weren't arguing from a legal, but from an ethics point of view.
  - eeZah7Ux 10y ago
    It's not "tricky", it's by design.
- dibstern 10y ago
  In both scenarios the person's property is being stolen. Their property might be in the form of information, sure. But one product being more widely distributed does not make any ethical difference. The fundamental is that the information/property has been stolen in both cases.
  - dozzie 10y ago
    It's not a property in the first place. It's nowhere near like physical belongings and it shouldn't be treated as such.

deleted 10y ago

[deleted]

tmalsburg2 10y ago

After many years, Bittorrent doesn't cease to amaze me. Such a fantastic technology. Sad, though, that it is primarily known for illegitimate uses. This gives it a bad reputation and makes it hard to promote Bittorrent for more worthwhile applications. For instance, Bittorrent would be a perfect platform for sharing scientific data but when I suggest that to colleagues, they don't take it seriously because Bittorrent is perceived to be that tool for downloading pirated movies. I wonder how this barrier for adoption in academia could be overcome. Perhaps, with a separate network that is targeted at academic uses (think alt-coins). Call this network ResearchTorrent and researchers could use the technology without ever having to make contact with the whole ecosystem around pirated material. Universities could support this network without risking bad PR and getting in conflict with the music/film industry.

tom_mellior 10y ago
http://academictorrents.com/ http://academictorrents.com/ claims to have 16.33 TB of research data.
- tmalsburg2 10y ago
  Last time I checked (3 years ago) this project didn't seem very active in terms of submission numbers and it appears that this hasn't really changed (7 submissions in Jan 2017). Not sure why, but the UI may be one reason.
  - tom_mellior 10y ago
    I don't think it's fair to expect scientific output to be comparable to Hollywood. Anyway, your colleagues think this is a pirate site?
jzelinskie 10y ago
I work on a BitTorrent tracker on the side[0] that's used for deploying software at a few companies, but has the unique feature of a middleware layer. Right now, there are two different middleware that require a signature of the infohash (files being shared) before allowing content to be shared. One is for JWTs[1] and upstreamed in the project and the other is for a blockchain and hasn't been upstreamed yet. Using features like this, they can run their tracker on the public internet and not worry about people sharing illegal content. I had never considered it, but this sounds like a great use case for academia, too. [0]: https://chihaya.io https://chihaya.io [1]: https://jwt.io https://jwt.io

fenollp 10y ago

You should look for sitemaps. They are often listed in robots.txt. Will save you lots of time! These sites have sitemaps: http://torrentproject.se/robots.txt http://torrentproject.se/robots.txt https://www.torrentdownloads.me/robots.txt https://www.torrentdownloads.me/robots.txt https://thepiratebay.org/robots.txt https://thepiratebay.org/robots.txt http://goldtorrents.com/robots.txt http://goldtorrents.com/robots.txt https://bitsnoop.com/robots.txt https://bitsnoop.com/robots.txt Enough with the hype: "high performance" What?? Your program is not concurrent: you have one process per website and go through URLs one at a time. You don't even use Bloom filters.

luckystartup 10y ago

I might have to start an anonymous GitHub account so that I can contribute to this. It would be fantastic if there was a way to start a download directly from the web UI. Many torrent clients can monitor an RSS feed, so that would probably be the best solution. I would add the ability to create a user account, and each user could set up a few RSS feeds. In my case, I would like to have one RSS feed with the label "Movies", and one for "TV Shows". Then the main search page would have buttons to add the magnet link to either RSS feed. I already use http://showrss.info http://showrss.info, which provides an RSS feed for all of my favorite shows. So it would be great to manage my own RSS feed for movies and specific episodes.

the8472 10y ago
> It would be fantastic if there was a way to start a download directly from the web UI. There are browser addons that can send magnets and torrents straight to the remote APIs of torrent clients.

listic 10y ago

Can anyone suggest a good torrent tracker suitable to run a torrent site? Opentracker [0] seems to be abandoned and the author doesn't reply. [0] http://erdgeist.org/arts/software/opentracker/ http://erdgeist.org/arts/software/opentracker/

the8472 10y ago
opentracker is what is used by the high-throughput trackers. the protocol is fairly simple and doesn't change a lot and the implementation is fairly minimal, so they don't need to update the codebase often, it does it's job, it's mature. for smaller sites there are more specialized trackers that allow more control, e.g. chihaya and gazelle.

kenning 10y ago

Has anyone hosted this somewhere yet? I was hoping I could just run it locally but you have to crawl all those sites every time, and it's not concurrent...

uhryks 10y ago
> leveraging Elixir's GenServer and Erlang's BEAM VM. > it's not concurrent (。´･ω･)?
- sergiotapia 10y ago
  It's concurrent in that multiple crawlers run at the same time. But I do not concurrently crawl single websites. An initial version had that, and Magnetissimo easily crawled 600 torrents every page 3 seconds (not hyperbole). It just pounded websites and I didn't want that. It was bad karma.
omnicoder 10y ago
I just setup a test site based on it http://v.reni.com:4000/ http://v.reni.com:4000/

giridhar50 10y ago

very useful tool