6 ms·

Show HN: Lightweight Torrent Search Engine

13 points by Prefinem 12y ago

sysk 12y ago
How do you keep your magnet link database fresh? Do you crawl the web for those links or do you rely exclusively on people uploading them? Sorry if the question doesn't make sense, I'm not very familiar with how BitTorrent works.
- Prefinem 12y ago
  Right now I am pulling from other sources like Kickass and other torrent sites to ensure I gain new torrents along with uploads. Right now there are 6+ million torrents that are being scraped/updated with more being added
  - tokenizerrr 12y ago
    What do you mean with updated? Are you scraping the individual pages for all those torrents on Kickass (and others)? Some more technical details would be lovely.
    - Prefinem 12y ago
      Several sites provide apis or daily dumps of their torrents, so I am loading them, indexing them, and then scraping the trackers for their information. How technical do you want?
      - tokenizerrr 12y ago
        As technical as you're willing to get :)
        Prefinem 12y ago
        Daily dumps are pulled from torrent sites. I grab each hash and title and store those. Then all the new torrents, I grab the hash, and scrape the demonii tracker to get seeder/leecher information. Also at the same time, I am constantly scraping the demonii tracker for seeder/leecher information for older torrents based on age. It currently takes 5 days to update all the torrents, but I will be implementing an auto-pruning system for torrents that are dead for so long, negative votes, etc etc
egwynn 12y ago
s/dcma/dmca/
- Prefinem 12y ago
  Wow, thanks! Can't believe I missed that....
forgotpasswd3x 12y ago
What does clicking on the username do?
- Prefinem 12y ago
  Gives you the hash that generates your user ID. This is only important if you upload.
emic 12y ago
This is a great site, but one thing bothers me - your policy when it comes to DMCA takedown requests. If this becomes well-known, you'll get so many DMCAs that there'll be nothing worth downloading. Is that just there to keep the movie/music industries happy, or will you actually be responding to every takedown request?
- Prefinem 12y ago
  I had it in my mind to try and respond to every takedown request, but I have very specific instructions that should hinder mass takedowns. Also, if there is a flood of takedowns, response time will take longer. And, there is nothing stopping from someone uploading the same file under another torrent. (although this will hurt seeders/leechers) But we will see. I don't think this will become too popular so hopefully I don't have to worry to much

13 points by Prefinem 12y ago

sysk 12y ago

How do you keep your magnet link database fresh? Do you crawl the web for those links or do you rely exclusively on people uploading them? Sorry if the question doesn't make sense, I'm not very familiar with how BitTorrent works.

Prefinem 12y ago
Right now I am pulling from other sources like Kickass and other torrent sites to ensure I gain new torrents along with uploads. Right now there are 6+ million torrents that are being scraped/updated with more being added
- tokenizerrr 12y ago
  What do you mean with updated? Are you scraping the individual pages for all those torrents on Kickass (and others)? Some more technical details would be lovely.
  - Prefinem 12y ago
    Several sites provide apis or daily dumps of their torrents, so I am loading them, indexing them, and then scraping the trackers for their information. How technical do you want?
    - tokenizerrr 12y ago
      As technical as you're willing to get :)
      - Prefinem 12y ago
        Daily dumps are pulled from torrent sites. I grab each hash and title and store those. Then all the new torrents, I grab the hash, and scrape the demonii tracker to get seeder/leecher information. Also at the same time, I am constantly scraping the demonii tracker for seeder/leecher information for older torrents based on age. It currently takes 5 days to update all the torrents, but I will be implementing an auto-pruning system for torrents that are dead for so long, negative votes, etc etc

egwynn 12y ago

s/dcma/dmca/

Prefinem 12y ago
Wow, thanks! Can't believe I missed that....

forgotpasswd3x 12y ago

What does clicking on the username do?

Prefinem 12y ago
Gives you the hash that generates your user ID. This is only important if you upload.

emic 12y ago

This is a great site, but one thing bothers me - your policy when it comes to DMCA takedown requests. If this becomes well-known, you'll get so many DMCAs that there'll be nothing worth downloading. Is that just there to keep the movie/music industries happy, or will you actually be responding to every takedown request?

Prefinem 12y ago
I had it in my mind to try and respond to every takedown request, but I have very specific instructions that should hinder mass takedowns. Also, if there is a flood of takedowns, response time will take longer. And, there is nothing stopping from someone uploading the same file under another torrent. (although this will hurt seeders/leechers) But we will see. I don't think this will become too popular so hopefully I don't have to worry to much