7 ms·

BitCannon – A torrent index archiving, browsing, and backup tool

54 points by fortytw2 12y ago

jfree 12y ago
Hi, excellent idea but each time I try to import a file, I've got file parsing ended. New: 0 Dup:xxxxxx. In the site, there is no torrent at all. It looks like the import works but think that it is already imported? I've got a Win8.1 64 bits and I tried the 64 and 32 bits .exe. I run the latest version of MongoDB 2.6. Any idea of what I'm doing wrong? Thanks!
jfree 12y ago
Hi, excellent idea but each time I try to import a file, I've got file parsing ended. New: 0 Dup:xxxxxx. In the site, there is no torrent at all. It looks like the import works but think that it is already imported? I've got a Win8.1 64 bits and I tried the 64 and 32 bits .exe. I run the latest version of MongoDB 2.6. Any idea of what I'm doing wrong? Thanks!
americandane 12y ago
BitCannon Windows Guide with autostart script for the curious http://www.htpcguides.com/install-bitcannon-personal-kickasstorrents-windows/ http://www.htpcguides.com/install-bitcannon-personal-kickass...
chrisabrams 12y ago
This is an interesting concept because until reading this, it had never occurred to me that the torrent indexes are typically centralized as opposed to the decentralized nature of torrenting.
- Stephen304 12y ago
  That was exactly my goal. I was thinking how easy it is for people to download the backups that a couple torrent sites provide, but hardly anybody does so because the backups are hard to use. I also hope that, in addition to encouraging people to download torrent archives, that I may also help encourage more torrent sites to provide archive downloads of their torrent database.
  - toomuchtodo 12y ago
    > that I may also help encourage more torrent sites to provide archive downloads of their torrent database. Could the underlying archive be managed with git? For distributed versioning, sha verificiation, etc?
  - Maakuth 12y ago
    Would it be possible to populate it just by indexing everything tracked in DHT? There was a research paper featured here recently, but I can't find it right now. They had a way of getting a decent chunk of all the DHT torrents and did some analysis based on that data.
    - Stephen304 12y ago
      I think that would be a nice way to import torrents without relying on a site that could be taken down. Unfortunately, I don't know a ton about DHT and the BitTorrent protocols. I'm still trying to figure out how to scrape a UDP tracker in go to get seed / leech counts. I like this idea though!
fdsary 12y ago
A torrent index that I can view thru an emacs mode would be the most useful. If the data format is just a text tile with each torrent taking one line would be amazing. It would be long, but it's just text so memory consumption is negligible. Kinda same style as Org-mode, I imagine. You could just search like usual, open a magnet with Return, click 'c' for it's comments. When opening the file, the mode would check for seeders/leechers and display it inline.
- Stephen304 12y ago
  That's an interesting idea, and I'm sure people familiar with emacs would find it very useful. BitCannon has a pretty simple API (at least I think) so feel free to tinker around and make alternative clients for it!
touristtam 12y ago
So you have a NodeJs server for the web interface and a Martini server for the API part? Would it not have made more sense to have chosen either NodeJs or Martini for the whole project?
- Stephen304 12y ago
  Initially it was separated, but I thought the same thing so I embedded the web app in the binary and Martini now serves the whole thing.
mikegioia 12y ago
This is fantastic work and I can't wait to get this running on my home server. Thank you!
- Stephen304 12y ago
  You're welcome! Let me know what you think, and if you have any ideas for improving it, feel free to email me or open an issue on github!
daveloyall 12y ago
@Stephen304, what are the mechanisms for gathering index content? Does the user manually download those from centralized sites? How do the centralized sites get their index content?
- Stephen304 12y ago
  Currently, users have to manually download archives from torrent sites that provide them (Kickass and Demonoid are 2 I know work with BitCannon) I know this isn't ideal, so I intend to implement an auto updating function to periodically auto download and import updates. You can see the beginnings of this in the config file in the current release.
  - caractacus 12y ago
    BitSnoop has a db of 24m+ torrents available, though it's (i) badly categorised and (ii) offline atm. http://bitsnoop.com/api.html http://bitsnoop.com/api.html So if I installed this, I would (for the moment) currently need to grab the dump from KAT or Demonoid or wherever, extract it, and add it to the database...? How does it deal with duplicates from the same site - or the same infohash found on different sites?
    - Stephen304 12y ago
      Yes, that's the current process for importing torrents. Making an automatic import / update is high on my priorities. It uses a unique key on the infohash, so it skips duplicates at the moment. I've thought about concatenating the categories info when duplicates are found, so a torrent on kickass in Movies and Demonoid in Anime will merge to be 1 torrent with [Movies, Anime], but it isn't too high on my priorities considering I still have to add a UDP tracker scraper.
      - caractacus 12y ago
        Okay, so installed and added today's KAT db in. A few initial thoughts: 1. It took 2-3 hours to add in the whole KAT db. No problems in parsing the data from what I could tell, but a long time to do it. Maybe that's my machine but it's not ridiculously slow. I'm not going to be adding in the bitsnoop db of 24m+ torrents at this rate. 2. Pulling up 'browse' takes 45-50s just with the KAT stuff indexed and it takes that long every time I click on browse. Could you add some kind of caching to that page? 3. There's no paging. It loads a page of results and no more. 4. Search is very very fast. Would be nice to be able to specific the category when searching. The main thing it's missing that makes other sites good is all the ratings, seed/leech data, comments, meta data. I wondered, maybe you could do a large scrape of KAT every now and again that grabs all of that and provide it to people as a kick-start to their database....? Then they only need grab updated files every day or something.
alfg 12y ago
Cool idea. Any screenshots or a demo instance of the web interface?
- Stephen304 12y ago
  I've added a couple screenshots under the features section of the website to show what it basically looks like. (I've been trying to add screenshots all day but I've been in and out of class)
eberfreitas 12y ago
I would love if that was a desktop client. I don't want to mess with Mongo and Node for my own torrent index. Maybe I could persist the index on dropbox or something and point my client to it. NW.js seems like a good option for this!
- Stephen304 12y ago
  I'm not using Node in the web app itself, just for dependency management and development. Once the app is built, it's just angular and regular js. Give it a try, I think that will clear up any confusion. All you need is MongoDB and the BitCannon binary. (ie You only need Go/Node/bower/grunt if you wish to compile it yourself)

54 points by fortytw2 12y ago

jfree 12y ago

Hi, excellent idea but each time I try to import a file, I've got file parsing ended. New: 0 Dup:xxxxxx. In the site, there is no torrent at all. It looks like the import works but think that it is already imported? I've got a Win8.1 64 bits and I tried the 64 and 32 bits .exe. I run the latest version of MongoDB 2.6. Any idea of what I'm doing wrong? Thanks!

jfree 12y ago

americandane 12y ago

BitCannon Windows Guide with autostart script for the curious http://www.htpcguides.com/install-bitcannon-personal-kickasstorrents-windows/ http://www.htpcguides.com/install-bitcannon-personal-kickass...

chrisabrams 12y ago

This is an interesting concept because until reading this, it had never occurred to me that the torrent indexes are typically centralized as opposed to the decentralized nature of torrenting.

Stephen304 12y ago
That was exactly my goal. I was thinking how easy it is for people to download the backups that a couple torrent sites provide, but hardly anybody does so because the backups are hard to use. I also hope that, in addition to encouraging people to download torrent archives, that I may also help encourage more torrent sites to provide archive downloads of their torrent database.
- toomuchtodo 12y ago
  > that I may also help encourage more torrent sites to provide archive downloads of their torrent database. Could the underlying archive be managed with git? For distributed versioning, sha verificiation, etc?
- Maakuth 12y ago
  Would it be possible to populate it just by indexing everything tracked in DHT? There was a research paper featured here recently, but I can't find it right now. They had a way of getting a decent chunk of all the DHT torrents and did some analysis based on that data.
  - Stephen304 12y ago
    I think that would be a nice way to import torrents without relying on a site that could be taken down. Unfortunately, I don't know a ton about DHT and the BitTorrent protocols. I'm still trying to figure out how to scrape a UDP tracker in go to get seed / leech counts. I like this idea though!

fdsary 12y ago

A torrent index that I can view thru an emacs mode would be the most useful. If the data format is just a text tile with each torrent taking one line would be amazing. It would be long, but it's just text so memory consumption is negligible. Kinda same style as Org-mode, I imagine. You could just search like usual, open a magnet with Return, click 'c' for it's comments. When opening the file, the mode would check for seeders/leechers and display it inline.

Stephen304 12y ago
That's an interesting idea, and I'm sure people familiar with emacs would find it very useful. BitCannon has a pretty simple API (at least I think) so feel free to tinker around and make alternative clients for it!

touristtam 12y ago

So you have a NodeJs server for the web interface and a Martini server for the API part? Would it not have made more sense to have chosen either NodeJs or Martini for the whole project?

Stephen304 12y ago
Initially it was separated, but I thought the same thing so I embedded the web app in the binary and Martini now serves the whole thing.

mikegioia 12y ago

This is fantastic work and I can't wait to get this running on my home server. Thank you!

Stephen304 12y ago
You're welcome! Let me know what you think, and if you have any ideas for improving it, feel free to email me or open an issue on github!

daveloyall 12y ago

@Stephen304, what are the mechanisms for gathering index content? Does the user manually download those from centralized sites? How do the centralized sites get their index content?

Stephen304 12y ago
Currently, users have to manually download archives from torrent sites that provide them (Kickass and Demonoid are 2 I know work with BitCannon) I know this isn't ideal, so I intend to implement an auto updating function to periodically auto download and import updates. You can see the beginnings of this in the config file in the current release.
- caractacus 12y ago
  BitSnoop has a db of 24m+ torrents available, though it's (i) badly categorised and (ii) offline atm. http://bitsnoop.com/api.html http://bitsnoop.com/api.html So if I installed this, I would (for the moment) currently need to grab the dump from KAT or Demonoid or wherever, extract it, and add it to the database...? How does it deal with duplicates from the same site - or the same infohash found on different sites?
  - Stephen304 12y ago
    Yes, that's the current process for importing torrents. Making an automatic import / update is high on my priorities. It uses a unique key on the infohash, so it skips duplicates at the moment. I've thought about concatenating the categories info when duplicates are found, so a torrent on kickass in Movies and Demonoid in Anime will merge to be 1 torrent with [Movies, Anime], but it isn't too high on my priorities considering I still have to add a UDP tracker scraper.
    - caractacus 12y ago
      Okay, so installed and added today's KAT db in. A few initial thoughts: 1. It took 2-3 hours to add in the whole KAT db. No problems in parsing the data from what I could tell, but a long time to do it. Maybe that's my machine but it's not ridiculously slow. I'm not going to be adding in the bitsnoop db of 24m+ torrents at this rate. 2. Pulling up 'browse' takes 45-50s just with the KAT stuff indexed and it takes that long every time I click on browse. Could you add some kind of caching to that page? 3. There's no paging. It loads a page of results and no more. 4. Search is very very fast. Would be nice to be able to specific the category when searching. The main thing it's missing that makes other sites good is all the ratings, seed/leech data, comments, meta data. I wondered, maybe you could do a large scrape of KAT every now and again that grabs all of that and provide it to people as a kick-start to their database....? Then they only need grab updated files every day or something.

alfg 12y ago

Cool idea. Any screenshots or a demo instance of the web interface?

Stephen304 12y ago
I've added a couple screenshots under the features section of the website to show what it basically looks like. (I've been trying to add screenshots all day but I've been in and out of class)

eberfreitas 12y ago

I would love if that was a desktop client. I don't want to mess with Mongo and Node for my own torrent index. Maybe I could persist the index on dropbox or something and point my client to it. NW.js seems like a good option for this!

Stephen304 12y ago
I'm not using Node in the web app itself, just for dependency management and development. Once the app is built, it's just angular and regular js. Give it a try, I think that will clear up any confusion. All you need is MongoDB and the BitCannon binary. (ie You only need Go/Node/bower/grunt if you wish to compile it yourself)