6 ms·

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I mis

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, even partially complete. In super extra option the app could even notify me to load DVD because somebody asked for a file which I indexed and advertised previously. For years my program preferences changed, file locations changed - I have moved files around, made them offline, burned on DVDs, deleted some parts of torrent just to keep interesting stuff. Now these torrents are lost, at least my seeding contribution is gone. But I almost never change the content of these files, their checksum stays the same forever, so they could be still discoverable. The digital preservation needs better distribution system.

londons_explore 4y ago
The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.
- chhs 4y ago
  A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there's no lookup mechanism for a "pieces root" to know that torrent B exists and contains file C. If you were to make a v2 torrent of your entire drive, other clients won't know to download from your torrent. They'd need to have the contents of the metadata to know it would contain a file they are interested in, and have no way of knowing which metadata contains the desired "pieces root" entries without downloading all of them. I'm very interested in this problem space, if you are aware of clients/mechanisms that allow for this I would love to hear them.
  - justshowpost 4y ago
    This. Also, advertising every file in torrent (of unsuspecting user) to entire world would be a HUGE privacy flaw.
    - londons_explore 4y ago
      But you already advertise (via the DHT, which offers no secrecy) the torrent file... and that contains all the data needed to get all the files within the torrent. So this data is already being published, just in a less queryable form.
  - londons_explore 4y ago
    > there's no lookup mechanism for a "pieces root" t I guess they'd just have to publish each pieces root in the DHT. And obviously you would let the user decide which directories to share... I guess that'll have to wait for torrent v3...
  - jychang 4y ago
    I don't think you can solve this problem, without destroying the system by releasing too much data. If I find a torrent of a file, and can look up all the seeders across all torrents for this file... then so can the FBI.
- Retr0id 4y ago
  Hm, is there a security concern here? Someone could use this to A) Remotely check the presence of any specific file on your machine. B) Exfiltrate the contents of any file they know the hash of (or possibly more specifically, the hash of each piece? I don't know the protocol details). Fine if you have a dedicated "I expect the contents to be public" drive or directory, but not something I'd want to do on my OS drive.
  - autoexec 4y ago
    I have seen people obviously sharing their entire hard drive on P2P platforms, and it's usually been someone's C drive and a massive security issue. I don't think some clients do enough to make it clear to people that they need to be careful not to share more than they intend to or should.
    - londons_explore 4y ago
      On modern windows, ideally there are no sensitive files accessible to a user program, because everything should be in the secret store/protectedstorage. Obviously a lot of programs still store cached session keys and passwords in plaintext somewhere in your user directory though...
      - imhoguy 4y ago
        and assuming nobody puts their passwords.txt on desktop and keeps unprotected wallet.dat around.
      - crtasm 4y ago
        besides that a user's Photos and Documents directories can be rather sensitive
      - patrick451 4y ago
        There's a lot more to be concerned about than session keys and passwords. Tax documents, health records, photos, private journal documents, just to name a few.
chhs 4y ago
I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file without having all of the other files in the torrent. I built it with libtorrent and after loading in all of the torrents (multiple TBs of data), it would promptly and routinely crashed. I couldn't find the cause of the error, it doesn't seem it was designed to run with thousands of torrents. One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I haven't been able to find a publicly available database of torrent metadata. If you have an info hash then itorrents.org will give you the metadata, if it exists. I started scraping metadata via DHT announcements, but it's not exactly fast, and each client would have to do this unless they can share the database of metadata between them (I have an idea on how to accomplish this via BEP 46).
- 3np 4y ago
  Would you mind sharing the source? Sounds like something others could build on.
  - chhs 4y ago
    The source is available here: https://github.com/chhs1/content-seeder https://github.com/chhs1/content-seeder
- trevyn 4y ago
  >One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I think BEP 51 followed by BEP 9 is all you need.
  - chhs 4y ago
    This is how I was originally achieving this. As I said, it's very slow. I don't think it would be a good solution on its own because it would require that every client be constantly sampling all DHT nodes, and downloading all metadata and indexing it for a potential future lookup. It's a huge amount of additional load on the DHT. I think a better solution would be some way for clients to query the DHT for a specific "pieces root", but I don't know if all clients publishing "pieces root" for the torrents they know about would also be a good idea. Some kind of distributed metadata database where clients can query would be ideal.
justshowpost 4y ago
But why? The current Kademlia implementation built in eMule/aMule does exactly this. Primary data advertised is file name string, there is also a support for several meta fields. Transfer proto sucks, but I don't think it really matters in the case of ultra-rare content.
- imhoguy 4y ago
  That `ultra-rare content`, eMule is mess if you need collection of files, ofc you can share ZIP but they get somehow repackaged too often. Torrents nicely group the stuff - it is kind of album. I've dreamed of federated meta-client which mixes all available p2p networks in a wild and can download missing torrent part from eMule or Soulseek.
dangerface 4y ago
> I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. That was the problem with p2p no one bothered to actually package their releases instead just slinging it onto the internet no care given as to if it was complete or even correctly labeled. Sites like pirate bay had everything, torrents are a super fast and error resistant way to distribute content its the perfect system. The problem was that the content is illegal so it was nuked by companies with more money and power than any one can fight against. The problem has never been technical or distribution the issue is the law.

londons_explore 4y ago

The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.

chhs 4y ago
A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there's no lookup mechanism for a "pieces root" to know that torrent B exists and contains file C. If you were to make a v2 torrent of your entire drive, other clients won't know to download from your torrent. They'd need to have the contents of the metadata to know it would contain a file they are interested in, and have no way of knowing which metadata contains the desired "pieces root" entries without downloading all of them. I'm very interested in this problem space, if you are aware of clients/mechanisms that allow for this I would love to hear them.
- justshowpost 4y ago
  This. Also, advertising every file in torrent (of unsuspecting user) to entire world would be a HUGE privacy flaw.
  - londons_explore 4y ago
    But you already advertise (via the DHT, which offers no secrecy) the torrent file... and that contains all the data needed to get all the files within the torrent. So this data is already being published, just in a less queryable form.
- londons_explore 4y ago
  > there's no lookup mechanism for a "pieces root" t I guess they'd just have to publish each pieces root in the DHT. And obviously you would let the user decide which directories to share... I guess that'll have to wait for torrent v3...
- jychang 4y ago
  I don't think you can solve this problem, without destroying the system by releasing too much data. If I find a torrent of a file, and can look up all the seeders across all torrents for this file... then so can the FBI.
Retr0id 4y ago
Hm, is there a security concern here? Someone could use this to A) Remotely check the presence of any specific file on your machine. B) Exfiltrate the contents of any file they know the hash of (or possibly more specifically, the hash of each piece? I don't know the protocol details). Fine if you have a dedicated "I expect the contents to be public" drive or directory, but not something I'd want to do on my OS drive.
- autoexec 4y ago
  I have seen people obviously sharing their entire hard drive on P2P platforms, and it's usually been someone's C drive and a massive security issue. I don't think some clients do enough to make it clear to people that they need to be careful not to share more than they intend to or should.
  - londons_explore 4y ago
    On modern windows, ideally there are no sensitive files accessible to a user program, because everything should be in the secret store/protectedstorage. Obviously a lot of programs still store cached session keys and passwords in plaintext somewhere in your user directory though...
    - imhoguy 4y ago
      and assuming nobody puts their passwords.txt on desktop and keeps unprotected wallet.dat around.
    - crtasm 4y ago
      besides that a user's Photos and Documents directories can be rather sensitive
    - patrick451 4y ago
      There's a lot more to be concerned about than session keys and passwords. Tax documents, health records, photos, private journal documents, just to name a few.

chhs 4y ago

I did work on a proof of concept program to accomplish this for my own content library. It would scan a directory to find files and compare them with locally stored metadata. For v2 torrents this is trivial to do via a "pieces root" lookup, for v1 torrents it involves basically checking that each piece matches, and since pieces may not align with the file then it's not possible to guarantee that it's the same file without having all of the other files in the torrent. I built it with libtorrent and after loading in all of the torrents (multiple TBs of data), it would promptly and routinely crashed. I couldn't find the cause of the error, it doesn't seem it was designed to run with thousands of torrents. One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I haven't been able to find a publicly available database of torrent metadata. If you have an info hash then itorrents.org will give you the metadata, if it exists. I started scraping metadata via DHT announcements, but it's not exactly fast, and each client would have to do this unless they can share the database of metadata between them (I have an idea on how to accomplish this via BEP 46).

3np 4y ago
Would you mind sharing the source? Sounds like something others could build on.
- chhs 4y ago
  The source is available here: https://github.com/chhs1/content-seeder https://github.com/chhs1/content-seeder
trevyn 4y ago
>One problem that I've yet to build a solution for is finding the metadata to use for the lookup phase. I think BEP 51 followed by BEP 9 is all you need.
- chhs 4y ago
  This is how I was originally achieving this. As I said, it's very slow. I don't think it would be a good solution on its own because it would require that every client be constantly sampling all DHT nodes, and downloading all metadata and indexing it for a potential future lookup. It's a huge amount of additional load on the DHT. I think a better solution would be some way for clients to query the DHT for a specific "pieces root", but I don't know if all clients publishing "pieces root" for the torrents they know about would also be a good idea. Some kind of distributed metadata database where clients can query would be ideal.

justshowpost 4y ago

But why? The current Kademlia implementation built in eMule/aMule does exactly this. Primary data advertised is file name string, there is also a support for several meta fields. Transfer proto sucks, but I don't think it really matters in the case of ultra-rare content.

imhoguy 4y ago
That `ultra-rare content`, eMule is mess if you need collection of files, ofc you can share ZIP but they get somehow repackaged too often. Torrents nicely group the stuff - it is kind of album. I've dreamed of federated meta-client which mixes all available p2p networks in a wild and can download missing torrent part from eMule or Soulseek.

dangerface 4y ago

> I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. That was the problem with p2p no one bothered to actually package their releases instead just slinging it onto the internet no care given as to if it was complete or even correctly labeled. Sites like pirate bay had everything, torrents are a super fast and error resistant way to distribute content its the perfect system. The problem was that the content is illegal so it was nuked by companies with more money and power than any one can fight against. The problem has never been technical or distribution the issue is the law.