5 ms·

Show HN: Torrentinim – low memory-footprint, API-only torrent search engine

117 points by sergiotapia 5y ago

belval 5y ago
Interesting project, what are some similar self-hosted solutions that allow you to search torrents on public trackers? To the author: How do you personally use it, do you have a custom interface to search the aggregated results or do you just plug it into a bigger content consumption system?
- LeoPanthera 5y ago
  > what are some similar self-hosted solutions that allow you to search torrents on public trackers? Jackett. https://github.com/Jackett/Jackett https://github.com/Jackett/Jackett
  - nirav72 5y ago
    also Prowlarr https://github.com/Prowlarr/Prowlarr https://github.com/Prowlarr/Prowlarr
    - arthurcolle 5y ago
      also Magnetissimo https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo implemented using Elixir and Phoenix. Cool little project
      - laurentlbm 5y ago
        It's a former project by OP, who stopped work on it: > I've paused development of Magnetissimo to focus on Torrentinim.
        arthurcolle 5y ago
        Good catch, didn't notice that!
- rnhmjoj 5y ago
  Magnetico[1] is an easy-to-use, self-hosted DHT crawler. It has a web UI that shows statistics and gives you a search engine (it uses sqlite full-text search). It needs quite a bit of bandwidth and some storage space, but worked well so far, for me. I've been running it for a couple of years and it indexed around 12.5M torrents (36GB of uncompressed database). [1]: https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico
  - ComputerGuru 5y ago
    Do you mind sharing what sample daily/monthly bandwidth during discovery looks like?
    - rnhmjoj 5y ago
      You can somewhat control the bandwith by limiting the number of simultaneous peers: in my case the usage seems to be around 5 Mbps up/down. If you're running it from home, the real problem is not the bandwidth but the number of open connections the router has to keep track (assuming the network is NATed). The default settings will likely crash any cheap consumer router, so I would run the daemon from a VPS or a machine that isn't behind a NAT.
- sergiotapia 5y ago
  I use it to search for DODI releases and x265 releases, it's faster than looking through websites. Once I finish Radarr/Sonnar support, I'll point my instances at Torrentinim.
sergiotapia 5y ago
I wanted a very small, efficient program that anyone could run without ceremony on their machine. Nim made it very easy for me to do. It looks like Python/Ruby, runs really fast, and uses very little resources. I love Nim!
- FnControlOption 5y ago
  > It looks like Python/Ruby, runs really fast, and uses very little resources. You should also check out Crystal, which had its 1.0 release earlier this year: https://crystal-lang.org/ https://crystal-lang.org/ (Although, no Windows support yet.)
- elcritch 5y ago
  Impressive! > High performance, extremely low memory footprint. > RAM usage (avg.): 21.5MB > Binary application size: 600KB
  - nix23 5y ago
    Application size here[1] is 1.8MiB, Ram 131MiB (release build -> make build) [1] Freebsd13 x64, Nim Compiler Version 1.4.8
rmm 5y ago
As an aside, does anyone know of a light-weight torrent tracker that can be used for personal use? I always wanted to setup a tracker for documents and files etc, without the need for huge member management systems etc.
- sa1 5y ago
  If you want a hosted tracker instead, I’ve used privtracker.com for personal files with success. If you want to set one up, chihaya or udpt might do the job.
- dicknuckle 5y ago
  what about Resilio Sync?
  - Wowfunhappy 5y ago
    ^ Or it's open source version, SyncThing.
- ComputerGuru 5y ago
  OpenTracker and XBT are the most famous for lightweight use.
  - nix23 5y ago
    OpenTracker is absolutely great and under one of the best licenses -> Beerware https://erdgeist.org/arts/software/opentracker/ https://erdgeist.org/arts/software/opentracker/
- Wowfunhappy 5y ago
  Hard to get simpler than BitStorm. A single PHP file you can drop on a web server: https://web.archive.org/web/20111101104813id_/https://ck3r.org/tracker/source.txt https://web.archive.org/web/20111101104813id_/https://ck3r.o...
reilly3000 5y ago
This looks nice! Is there a way to configure it to exclude certain categories of content (such as nsfw)? Do you tend to run this behind a proxy?
- squarefoot 5y ago
  After a brief look at the code it doesn't seem so, however there's a procedure (insert_torrent() in src/torrents.nim) in which a retrieved torrent is added to the database, of which one column is the name of the torrent, so it would be possible to insert a filter based on a list of nsfw words that would discard the torrent if one or more are found. Of course it wouldn't work on torrents that hide their nature, which hopefully happens very rarely.
halfmatthalfcat 5y ago
I've been looking for something like this but on the JVM (specifically Java or Scala if possible)
wiradikusuma 5y ago
What do people use this for? I mean beside creating another pirate bay (CIMMW, that's the only thing comes to mind when I see "torrent search engine").
- asxd 5y ago
  Dumb question I'm sure, but what does CIMMW stand for? (search results didn't turn anything up for me)
dmos62 5y ago
This queries the tracker websites, but I was hoping this would be a distributed hash table (DHT) search engine, like bitcq. Bitcq is indispensible for obscure torrents, but its interface could be better. Edit: wait, looking at the source, this doesn't search torrents on the mentioned trackers, it keeps an eye on feeds of newest or most popular (in case of tpb) torrents and caches the tracker metadata. This doesn't seem very useful to me. Over a long time you'll have a big view of the tracker's database, but not older content than you started tracking, and, in the case of tpb, you'll just have a view of torrents that have been in the top 100 since you started crawling. Weird. Also, peer counts don't seem to get updated, I think.
teitoklien 5y ago
Does anyone have a book recommendation to learn and implement the bittorrent protocol myself ? Ive been looking for a good book on the protocol.
- eliasson 5y ago
  Not a book but I wrote an article about implementing the basics of a BitTorrent client in Python a few years ago. Maybe you'll find that useful. https://markuseliasson.se/article/bittorrent-in-python/ https://markuseliasson.se/article/bittorrent-in-python/
  - teitoklien 5y ago
    Thanks a lot ! Really appreciate it
nix23 5y ago
I liked magnetissimo (from the same author) much more, but its kind of sleeping: https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo

117 points by sergiotapia 5y ago

Interesting project, what are some similar self-hosted solutions that allow you to search torrents on public trackers? To the author: How do you personally use it, do you have a custom interface to search the aggregated results or do you just plug it into a bigger content consumption system?

LeoPanthera 5y ago
> what are some similar self-hosted solutions that allow you to search torrents on public trackers? Jackett. https://github.com/Jackett/Jackett https://github.com/Jackett/Jackett
- nirav72 5y ago
  also Prowlarr https://github.com/Prowlarr/Prowlarr https://github.com/Prowlarr/Prowlarr
  - arthurcolle 5y ago
    also Magnetissimo https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo implemented using Elixir and Phoenix. Cool little project
    - laurentlbm 5y ago
      It's a former project by OP, who stopped work on it: > I've paused development of Magnetissimo to focus on Torrentinim.
      - arthurcolle 5y ago
        Good catch, didn't notice that!
rnhmjoj 5y ago
Magnetico[1] is an easy-to-use, self-hosted DHT crawler. It has a web UI that shows statistics and gives you a search engine (it uses sqlite full-text search). It needs quite a bit of bandwidth and some storage space, but worked well so far, for me. I've been running it for a couple of years and it indexed around 12.5M torrents (36GB of uncompressed database). [1]: https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico
- ComputerGuru 5y ago
  Do you mind sharing what sample daily/monthly bandwidth during discovery looks like?
  - rnhmjoj 5y ago
    You can somewhat control the bandwith by limiting the number of simultaneous peers: in my case the usage seems to be around 5 Mbps up/down. If you're running it from home, the real problem is not the bandwidth but the number of open connections the router has to keep track (assuming the network is NATed). The default settings will likely crash any cheap consumer router, so I would run the daemon from a VPS or a machine that isn't behind a NAT.
sergiotapia 5y ago
I use it to search for DODI releases and x265 releases, it's faster than looking through websites. Once I finish Radarr/Sonnar support, I'll point my instances at Torrentinim.

sergiotapia 5y ago

I wanted a very small, efficient program that anyone could run without ceremony on their machine. Nim made it very easy for me to do. It looks like Python/Ruby, runs really fast, and uses very little resources. I love Nim!

FnControlOption 5y ago
> It looks like Python/Ruby, runs really fast, and uses very little resources. You should also check out Crystal, which had its 1.0 release earlier this year: https://crystal-lang.org/ https://crystal-lang.org/ (Although, no Windows support yet.)
elcritch 5y ago
Impressive! > High performance, extremely low memory footprint. > RAM usage (avg.): 21.5MB > Binary application size: 600KB
- nix23 5y ago
  Application size here[1] is 1.8MiB, Ram 131MiB (release build -> make build) [1] Freebsd13 x64, Nim Compiler Version 1.4.8

rmm 5y ago

As an aside, does anyone know of a light-weight torrent tracker that can be used for personal use? I always wanted to setup a tracker for documents and files etc, without the need for huge member management systems etc.

sa1 5y ago
If you want a hosted tracker instead, I’ve used privtracker.com for personal files with success. If you want to set one up, chihaya or udpt might do the job.
dicknuckle 5y ago
what about Resilio Sync?
- Wowfunhappy 5y ago
  ^ Or it's open source version, SyncThing.
ComputerGuru 5y ago
OpenTracker and XBT are the most famous for lightweight use.
- nix23 5y ago
  OpenTracker is absolutely great and under one of the best licenses -> Beerware https://erdgeist.org/arts/software/opentracker/ https://erdgeist.org/arts/software/opentracker/
Wowfunhappy 5y ago
Hard to get simpler than BitStorm. A single PHP file you can drop on a web server: https://web.archive.org/web/20111101104813id_/https://ck3r.org/tracker/source.txt https://web.archive.org/web/20111101104813id_/https://ck3r.o...

reilly3000 5y ago

This looks nice! Is there a way to configure it to exclude certain categories of content (such as nsfw)? Do you tend to run this behind a proxy?

squarefoot 5y ago
After a brief look at the code it doesn't seem so, however there's a procedure (insert_torrent() in src/torrents.nim) in which a retrieved torrent is added to the database, of which one column is the name of the torrent, so it would be possible to insert a filter based on a list of nsfw words that would discard the torrent if one or more are found. Of course it wouldn't work on torrents that hide their nature, which hopefully happens very rarely.

halfmatthalfcat 5y ago

I've been looking for something like this but on the JVM (specifically Java or Scala if possible)

wiradikusuma 5y ago

What do people use this for? I mean beside creating another pirate bay (CIMMW, that's the only thing comes to mind when I see "torrent search engine").

asxd 5y ago
Dumb question I'm sure, but what does CIMMW stand for? (search results didn't turn anything up for me)

dmos62 5y ago

This queries the tracker websites, but I was hoping this would be a distributed hash table (DHT) search engine, like bitcq. Bitcq is indispensible for obscure torrents, but its interface could be better. Edit: wait, looking at the source, this doesn't search torrents on the mentioned trackers, it keeps an eye on feeds of newest or most popular (in case of tpb) torrents and caches the tracker metadata. This doesn't seem very useful to me. Over a long time you'll have a big view of the tracker's database, but not older content than you started tracking, and, in the case of tpb, you'll just have a view of torrents that have been in the top 100 since you started crawling. Weird. Also, peer counts don't seem to get updated, I think.

teitoklien 5y ago

Does anyone have a book recommendation to learn and implement the bittorrent protocol myself ? Ive been looking for a good book on the protocol.

eliasson 5y ago
Not a book but I wrote an article about implementing the basics of a BitTorrent client in Python a few years ago. Maybe you'll find that useful. https://markuseliasson.se/article/bittorrent-in-python/ https://markuseliasson.se/article/bittorrent-in-python/
- teitoklien 5y ago
  Thanks a lot ! Really appreciate it

nix23 5y ago

I liked magnetissimo (from the same author) much more, but its kind of sleeping: https://github.com/sergiotapia/magnetissimo https://github.com/sergiotapia/magnetissimo