4 ms·

Currently, users have to manually download archives from torrent sites that provide them (Kickass and Demonoid are 2 I know work with BitCannon) I know this is

Currently, users have to manually download archives from torrent sites that provide them (Kickass and Demonoid are 2 I know work with BitCannon) I know this isn't ideal, so I intend to implement an auto updating function to periodically auto download and import updates. You can see the beginnings of this in the config file in the current release.

caractacus 12y ago
BitSnoop has a db of 24m+ torrents available, though it's (i) badly categorised and (ii) offline atm. http://bitsnoop.com/api.html http://bitsnoop.com/api.html So if I installed this, I would (for the moment) currently need to grab the dump from KAT or Demonoid or wherever, extract it, and add it to the database...? How does it deal with duplicates from the same site - or the same infohash found on different sites?
- Stephen304 12y ago
  Yes, that's the current process for importing torrents. Making an automatic import / update is high on my priorities. It uses a unique key on the infohash, so it skips duplicates at the moment. I've thought about concatenating the categories info when duplicates are found, so a torrent on kickass in Movies and Demonoid in Anime will merge to be 1 torrent with [Movies, Anime], but it isn't too high on my priorities considering I still have to add a UDP tracker scraper.
  - caractacus 12y ago
    Okay, so installed and added today's KAT db in. A few initial thoughts: 1. It took 2-3 hours to add in the whole KAT db. No problems in parsing the data from what I could tell, but a long time to do it. Maybe that's my machine but it's not ridiculously slow. I'm not going to be adding in the bitsnoop db of 24m+ torrents at this rate. 2. Pulling up 'browse' takes 45-50s just with the KAT stuff indexed and it takes that long every time I click on browse. Could you add some kind of caching to that page? 3. There's no paging. It loads a page of results and no more. 4. Search is very very fast. Would be nice to be able to specific the category when searching. The main thing it's missing that makes other sites good is all the ratings, seed/leech data, comments, meta data. I wondered, maybe you could do a large scrape of KAT every now and again that grabs all of that and provide it to people as a kick-start to their database....? Then they only need grab updated files every day or something.
    - fortytw2 12y ago
      I think @Stephen304 is working on a way to pull seed/leech data from the DHT. The main problem with pulling the rest of the data is that you land up with 150+ GBs of metadata rather easily
  - caractacus 12y ago
    Thanks. I'll give this a go tomorrow and see how it looks.

caractacus 12y ago

BitSnoop has a db of 24m+ torrents available, though it's (i) badly categorised and (ii) offline atm. http://bitsnoop.com/api.html http://bitsnoop.com/api.html So if I installed this, I would (for the moment) currently need to grab the dump from KAT or Demonoid or wherever, extract it, and add it to the database...? How does it deal with duplicates from the same site - or the same infohash found on different sites?

Stephen304 12y ago
Yes, that's the current process for importing torrents. Making an automatic import / update is high on my priorities. It uses a unique key on the infohash, so it skips duplicates at the moment. I've thought about concatenating the categories info when duplicates are found, so a torrent on kickass in Movies and Demonoid in Anime will merge to be 1 torrent with [Movies, Anime], but it isn't too high on my priorities considering I still have to add a UDP tracker scraper.
- caractacus 12y ago
  Okay, so installed and added today's KAT db in. A few initial thoughts: 1. It took 2-3 hours to add in the whole KAT db. No problems in parsing the data from what I could tell, but a long time to do it. Maybe that's my machine but it's not ridiculously slow. I'm not going to be adding in the bitsnoop db of 24m+ torrents at this rate. 2. Pulling up 'browse' takes 45-50s just with the KAT stuff indexed and it takes that long every time I click on browse. Could you add some kind of caching to that page? 3. There's no paging. It loads a page of results and no more. 4. Search is very very fast. Would be nice to be able to specific the category when searching. The main thing it's missing that makes other sites good is all the ratings, seed/leech data, comments, meta data. I wondered, maybe you could do a large scrape of KAT every now and again that grabs all of that and provide it to people as a kick-start to their database....? Then they only need grab updated files every day or something.
  - fortytw2 12y ago
    I think @Stephen304 is working on a way to pull seed/leech data from the DHT. The main problem with pulling the rest of the data is that you land up with 150+ GBs of metadata rather easily
- caractacus 12y ago
  Thanks. I'll give this a go tomorrow and see how it looks.