5 ms·

Which tracker is this data from?

banana_giraffe 5y ago
It's from a DHT/BitTorrent crawler that runs around downloading all the torrent metadata it can find I've been running for ages.
- pbronez 5y ago
  cool. Did you write it yourself or are you using an open source project of some kind? Would be interested to run one of these myself.
  - banana_giraffe 5y ago
    I stumbled across a DHT crawler, and started running it. Over time I added random things to it, like IPv6 and the ability to download metadata. I was mostly interested in using it to learn how DHT worked both in theory and in practice, and for that, it did it's job. The page for the crawler I started with is long gone, but this appears to be a reasonable copy on github: https://github.com/zlzlovezl/simDHT-1 https://github.com/zlzlovezl/simDHT-1 I'm sure you can find other copies, with features added. Looking around, it seems like quite a few people have taken this and made it their own over time. It's far from perfect in this state, but a fun toy to play with. I will say, from a point of painful experience: The way this crawler works tends to make some of the various people that track such things think you're sharing pretty much all the things. If that's an issue where you live, don't run it from your home connection.
- sitkack 5y ago
  Do you make visualizations of the traffic?
  - banana_giraffe 5y ago
    Some different views. For instance, to give some sense of a scale for the 250 or so v2 torrents seen every day is, here's a view of the total number of torrents it discovers each day [1]. (Or, more properly, this is the number of torrents it gathered complete metadata and validated against the infohash each day). One of these days I should find somewhere to put all these bits of metadata I've been downloading. [1] https://imgur.com/a/8pWhjxD https://imgur.com/a/8pWhjxD
    - sitkack 5y ago
      Neat! I don’t have it handy, but years ago I scrapped the states traffic page and then made time lapses from them. One of the road segments had a bad sensor that would toggle into a stuck state. Keep it up!
    - toomuchtodo 5y ago
      Consider creating a collection at the Internet Archive for these data sets.
      - banana_giraffe 5y ago
        Sure: https://archive.org/details/torrent_metadata_archive_sample https://archive.org/details/torrent_metadata_archive_sample I'll start uploading monthly archives.
        toomuchtodo 5y ago
        I genuinely appreciate that you took the time to do this. Thank you.
        banana_giraffe 5y ago
        For anyone looking to download the data: Click on my name in archive.org to see the monthly archives. I'm slowly working my way through the months, though there may be breaks in the work as other work takes priority. The first few months will be the biggest, then it should settle down to something like 15-30gb a month.

It's from a DHT/BitTorrent crawler that runs around downloading all the torrent metadata it can find I've been running for ages.

pbronez 5y ago
cool. Did you write it yourself or are you using an open source project of some kind? Would be interested to run one of these myself.
- banana_giraffe 5y ago
  I stumbled across a DHT crawler, and started running it. Over time I added random things to it, like IPv6 and the ability to download metadata. I was mostly interested in using it to learn how DHT worked both in theory and in practice, and for that, it did it's job. The page for the crawler I started with is long gone, but this appears to be a reasonable copy on github: https://github.com/zlzlovezl/simDHT-1 https://github.com/zlzlovezl/simDHT-1 I'm sure you can find other copies, with features added. Looking around, it seems like quite a few people have taken this and made it their own over time. It's far from perfect in this state, but a fun toy to play with. I will say, from a point of painful experience: The way this crawler works tends to make some of the various people that track such things think you're sharing pretty much all the things. If that's an issue where you live, don't run it from your home connection.
sitkack 5y ago
Do you make visualizations of the traffic?
- banana_giraffe 5y ago
  Some different views. For instance, to give some sense of a scale for the 250 or so v2 torrents seen every day is, here's a view of the total number of torrents it discovers each day [1]. (Or, more properly, this is the number of torrents it gathered complete metadata and validated against the infohash each day). One of these days I should find somewhere to put all these bits of metadata I've been downloading. [1] https://imgur.com/a/8pWhjxD https://imgur.com/a/8pWhjxD
  - sitkack 5y ago
    Neat! I don’t have it handy, but years ago I scrapped the states traffic page and then made time lapses from them. One of the road segments had a bad sensor that would toggle into a stuck state. Keep it up!
  - toomuchtodo 5y ago
    Consider creating a collection at the Internet Archive for these data sets.
    - banana_giraffe 5y ago
      Sure: https://archive.org/details/torrent_metadata_archive_sample https://archive.org/details/torrent_metadata_archive_sample I'll start uploading monthly archives.
      - toomuchtodo 5y ago
        I genuinely appreciate that you took the time to do this. Thank you.
      - banana_giraffe 5y ago
        For anyone looking to download the data: Click on my name in archive.org to see the monthly archives. I'm slowly working my way through the months, though there may be breaks in the work as other work takes priority. The first few months will be the biggest, then it should settle down to something like 15-30gb a month.