5 ms·
Bitmagnet -https://bitmagnet.io/ https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much,
by hdgr 3mo ago
Bitmagnet -https://bitmagnet.io/ https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.
- NegativeLatency 3mo agoI've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO
- drdexebtjl 3mo agoI disabled mine because it was constantly writing to my SSD.
- felooboolooomba 3mo agoI solved it by storing the data on /dev/null
- Modified3019 3mo agoIn case you haven’t considered, HDDs work great for this, since write wear out isn’t really a factor for those. ZFS can also help batch together TXG’s until they hit time or size limits before commit it at once to the disk.
- qingcharles 3mo agoAre you running it at home? I built one with a nice TUI to run on a VPS so I can try and find rare magazine torrents, but Hetzner were upset about it. I need to find it a new home. It was a very good citizen, but it still raised too many flags.
- k4rli 3mo agoRuns fine at home. I've indexed 20M+ torrents in last few months running it during the day. With Prowlarr (or similar) it could easily replace other indexers.
- NoMoreNicksLeft 3mo agoWhich magazines?
- qingcharles 3mo agoAnything I don't have! Sometimes I'll find a torrent and no seeds/peers and I'll wonder if there is another torrent out there that has the same files in it somewhere that I can find. The other day it was trying to track down some older High Times issues that were torrented but the torrent is dead. Last night it was a mag titled Films & Filming which I know is scanned, but I can't find anywhere.
- toomuchtodo 3mo agoCan I get a copy for the Internet Archive? Will take as much of the corpus as you’re willing to provide. (no affiliation with them)
- qingcharles 3mo agoI'm starting to upload everything very soon. I have >4million magazines here so far I think. Feel free to email me on my profile :)
- 3mo ago
- plusfour 3mo agoSame here, for over a year. how many torrents has yours indexed?
- NegativeLatency 3mo ago28M
- plusfour 3mo ago27. it's creating about 20k a day.
- gonzalohm 3mo agoHow much space do you need to store the index?
- plusfour 3mo ago160gb for 27m torrents
- knowaveragejoe 3mo agoThe factor that determines this most is how much of the files metadata for each torrent you choose to keep, which is configurable.
- qingcharles 3mo agoThis is a good tip, thanks. I'll probably replace my home-grown scanner for this one.
- muyuu 3mo agoprobably worth adding some ML filter to it because yeah, most of the bulky stuff in bittorrent is always going to be garbage - a lot like the internet generally, the value is in filtering the good stuff out
- infinite_spin 3mo agoout of curiosity, what kind of junk/garbage is typical?
- yeeeloit 3mo agoungodly amounts of porn.
- deleted 3mo ago[deleted]
- muyuu 3mo agothere's certainly massive amounts of porn, esp. if you look at active traffic, but perhaps even worse than that is random crap collected for no good reason and "backyard genius" trash content pooled together by the gigabyte quality and traffic are grossly uncorrelated, so finding stuff is a huge challenge
- naikrovek 3mo agowhy the hell does that require a postgres database instance? has no one heard of sqlite? "It's just `docker compose up` dude." that's not excuse for unnecessary complexity. software for a single person should be a single binary that anyone can run how they choose. If they want to use Docker, great. If they want to run it in a terminal, great. If they want to run it on a server, great. If they want to run it on their phone, great. developers: avoid limiting people artificially like this. There is no way that Sqlite is insufficient for this.
- knowaveragejoe 3mo agoThe database gets quite large and the developers put in a lot of effort for tuning it for searchability.
- naikrovek 3mo agoPeople don't understand how fast SQLite is, it seems. You have to know what you're doing, but that's the same requirement that Postgres has if you want to get the most out of it.