3 ms·
Right now I am pulling from other sources like Kickass and other torrent sites to ensure I gain new torrents along with uploads. Right now there are 6+ million
by Prefinem 12y ago
Right now I am pulling from other sources like Kickass and other torrent sites to ensure I gain new torrents along with uploads. Right now there are 6+ million torrents that are being scraped/updated with more being added
- tokenizerrr 12y agoWhat do you mean with updated? Are you scraping the individual pages for all those torrents on Kickass (and others)? Some more technical details would be lovely.
- Prefinem 12y agoSeveral sites provide apis or daily dumps of their torrents, so I am loading them, indexing them, and then scraping the trackers for their information. How technical do you want?
- tokenizerrr 12y agoAs technical as you're willing to get :)
- Prefinem 12y agoDaily dumps are pulled from torrent sites. I grab each hash and title and store those. Then all the new torrents, I grab the hash, and scrape the demonii tracker to get seeder/leecher information. Also at the same time, I am constantly scraping the demonii tracker for seeder/leecher information for older torrents based on age. It currently takes 5 days to update all the torrents, but I will be implementing an auto-pruning system for torrents that are dead for so long, negative votes, etc etc