5 ms·

Distribyted: Torrent client with on-demand file downloading as a filesystem

75 points by mastabadtomm 6y ago

Pet_Ant 6y ago
I was actually sketching out an idea like this. Except that each file would have a header with a list of entries encrypted using public keys. When you found an entry it that you could decrypt, it would tell you how to reconstruct the desired data from rest of the file. The best part of it is that you have no idea what any of the other entries could produce. Your cat gif could contain a whistle blower's document.
- Jtsummers 6y ago
  You may be interested in Freenet. Data is distributed across nodes but the nodes don't know what they have. An individual file isn't necessarily entirely on one node, it could (and most likely is) distributed across many nodes. In order to retrieve data you need to know the key for that data. At that point you can start requesting the appropriate blocks from the network and reconstitute the file(s).
Sean-Der 6y ago
Super cool. It would be really awesome to get this in the web browser as well. I would love to have access to my music/movie collection on anything with a browser. You also wouldn't need to worry about setting up DNS/Public IP for the host. github.com/anacrolix/torrent supports WebTorrent so it shouldn't be that hard.
- ignoramous 6y ago
  Not over bit-torrent but this exists: https://syncthing.net/ https://syncthing.net/
  - remram 6y ago
    Syncthing does not work in the browser and does not provide on-demand access to files. It syncs directories.
    - toomuchtodo 6y ago
      Not yet, but it could.
      - 67868018 6y ago
        The constant "Out of Sync" errors in Syncthing make me think it's not so great at it's first job.
      - remram 6y ago
        On-demand access to files is a completely different workflow than syncing. It would also clash with the name, and require a change in technologies to run in the browser, as well as transport protocols.
        toomuchtodo 6y ago
        I don't agree. Dropbox syncs files across all of my devices, but I can also get files out using a web interface and mobile device apps (which, coincidently, is why I don't use syncthing). Regardless of the name, I think on demand access would be complementary to it's primary purpose (syncing), and serving up the content over http would be trivial (as well as over bittorrent/webtorrent, considering existing library support [1] [2]). At it's core, syncthing is an object distribution orchestration system, therefore other transports besides native syncing should always be considered (imho). [1] https://news.ycombinator.com/item?id=23818659 https://news.ycombinator.com/item?id=23818659 (HN: Libtorrent adds support for the WebTorrent protocol) [2] https://www.bittorrent.org/beps/bep_0019.html https://www.bittorrent.org/beps/bep_0019.html (WebSeed - HTTP/FTP Seeding (GetRight style))
        remram 6y ago
        You know what, if you are willing to spend this much time proving that you are "not technically wrong", I will give it to you. Sure, everything can be turned into anything, given enough effort. And Syncthing does show similarities if you squint really hard. It doesn't change the fact that the "have access to my music/movie collection on anything with a browser" mentioned by the grand-parent is not what Syncthing does, and is not even close to what it does now or what the team is interested in implementing [1]. While it is a great piece of software (that I personally use), this recommendation was misguiding. Have a good rest of your week. [1]: https://github.com/syncthing/syncthing/issues/6753#issuecomment-645510165 https://github.com/syncthing/syncthing/issues/6753#issuecomm...
toomim 6y ago
Is this like IPFS? People have long described it as "Bittorrent inside a filesystem."
- X6S1x6Okd1st 6y ago
  It appears to just be a torrent client with an smart FUSE capability.
BuildTheRobots 6y ago
Somewhat off topic, but is anyone aware of a bittorrent proxy? I regularly end up downloading torrents on a remote box, and then have to copy them to my local network. In an ideal world, I could run a torrent client at home and have it proxy though my vps so it doesn't leak my home IP. The other way I can see it working is being able to add a torrent to both my vps and server at home, and have the torrent client at home only ever talk to the single remote box so I still end up with the torrent at home without leaking my home IP and without having to download the finished files as a second step.
- deleted 6y ago
  [deleted]
- corobo 6y ago
  I've heard decent things about put.io for the remote side of things, then using `rclone move` [1] to pull the content down at frequent intervals (put.io doesn't show the files till they're done downloading so no worries of pulling half files) If you're not a fan of the full setup there you could look at just the rclone part and have your torrent client move completed downloads into the folder you pulling using rclone [1]: https://rclone.org/commands/rclone_move/ https://rclone.org/commands/rclone_move/ I realise this isn't quite what you've asked for but the direct answer would be to set up either a VPN or a proxy yourself, of which I have no suggestions or recommendations software/host-wise
  - mmastrac 6y ago
    put.io has rclone support? I've been using aria in the past and it's not bad but not super convenient. EDIT: https://help.put.io/en/articles/3480094-plex-rclone https://help.put.io/en/articles/3480094-plex-rclone
    - corobo 6y ago
      Aye. This is a Plex-centric guide but you can ignore the plex part https://help.put.io/en/articles/3480094-plex-rclone https://help.put.io/en/articles/3480094-plex-rclone I'd personally go off-guide and use rclone move rather than mounting it just to avoid having to keep bumping up storage space on put.io e: hah you already got it
dndvr 6y ago
Sounds a bit like PiperNet
rambojazz 6y ago
Is this read only? Can files be edited?

75 points by mastabadtomm 6y ago

I was actually sketching out an idea like this. Except that each file would have a header with a list of entries encrypted using public keys. When you found an entry it that you could decrypt, it would tell you how to reconstruct the desired data from rest of the file. The best part of it is that you have no idea what any of the other entries could produce. Your cat gif could contain a whistle blower's document.

Jtsummers 6y ago
You may be interested in Freenet. Data is distributed across nodes but the nodes don't know what they have. An individual file isn't necessarily entirely on one node, it could (and most likely is) distributed across many nodes. In order to retrieve data you need to know the key for that data. At that point you can start requesting the appropriate blocks from the network and reconstitute the file(s).

Sean-Der 6y ago

Super cool. It would be really awesome to get this in the web browser as well. I would love to have access to my music/movie collection on anything with a browser. You also wouldn't need to worry about setting up DNS/Public IP for the host. github.com/anacrolix/torrent supports WebTorrent so it shouldn't be that hard.

ignoramous 6y ago
Not over bit-torrent but this exists: https://syncthing.net/ https://syncthing.net/
- remram 6y ago
  Syncthing does not work in the browser and does not provide on-demand access to files. It syncs directories.
  - toomuchtodo 6y ago
    Not yet, but it could.
    - 67868018 6y ago
      The constant "Out of Sync" errors in Syncthing make me think it's not so great at it's first job.
    - remram 6y ago
      On-demand access to files is a completely different workflow than syncing. It would also clash with the name, and require a change in technologies to run in the browser, as well as transport protocols.
      - toomuchtodo 6y ago
        I don't agree. Dropbox syncs files across all of my devices, but I can also get files out using a web interface and mobile device apps (which, coincidently, is why I don't use syncthing). Regardless of the name, I think on demand access would be complementary to it's primary purpose (syncing), and serving up the content over http would be trivial (as well as over bittorrent/webtorrent, considering existing library support [1] [2]). At it's core, syncthing is an object distribution orchestration system, therefore other transports besides native syncing should always be considered (imho). [1] https://news.ycombinator.com/item?id=23818659 https://news.ycombinator.com/item?id=23818659 (HN: Libtorrent adds support for the WebTorrent protocol) [2] https://www.bittorrent.org/beps/bep_0019.html https://www.bittorrent.org/beps/bep_0019.html (WebSeed - HTTP/FTP Seeding (GetRight style))
        remram 6y ago
        You know what, if you are willing to spend this much time proving that you are "not technically wrong", I will give it to you. Sure, everything can be turned into anything, given enough effort. And Syncthing does show similarities if you squint really hard. It doesn't change the fact that the "have access to my music/movie collection on anything with a browser" mentioned by the grand-parent is not what Syncthing does, and is not even close to what it does now or what the team is interested in implementing [1]. While it is a great piece of software (that I personally use), this recommendation was misguiding. Have a good rest of your week. [1]: https://github.com/syncthing/syncthing/issues/6753#issuecomment-645510165 https://github.com/syncthing/syncthing/issues/6753#issuecomm...

toomim 6y ago

Is this like IPFS? People have long described it as "Bittorrent inside a filesystem."

X6S1x6Okd1st 6y ago
It appears to just be a torrent client with an smart FUSE capability.

BuildTheRobots 6y ago

Somewhat off topic, but is anyone aware of a bittorrent proxy? I regularly end up downloading torrents on a remote box, and then have to copy them to my local network. In an ideal world, I could run a torrent client at home and have it proxy though my vps so it doesn't leak my home IP. The other way I can see it working is being able to add a torrent to both my vps and server at home, and have the torrent client at home only ever talk to the single remote box so I still end up with the torrent at home without leaking my home IP and without having to download the finished files as a second step.

deleted 6y ago
[deleted]
corobo 6y ago
I've heard decent things about put.io for the remote side of things, then using `rclone move` [1] to pull the content down at frequent intervals (put.io doesn't show the files till they're done downloading so no worries of pulling half files) If you're not a fan of the full setup there you could look at just the rclone part and have your torrent client move completed downloads into the folder you pulling using rclone [1]: https://rclone.org/commands/rclone_move/ https://rclone.org/commands/rclone_move/ I realise this isn't quite what you've asked for but the direct answer would be to set up either a VPN or a proxy yourself, of which I have no suggestions or recommendations software/host-wise
- mmastrac 6y ago
  put.io has rclone support? I've been using aria in the past and it's not bad but not super convenient. EDIT: https://help.put.io/en/articles/3480094-plex-rclone https://help.put.io/en/articles/3480094-plex-rclone
  - corobo 6y ago
    Aye. This is a Plex-centric guide but you can ignore the plex part https://help.put.io/en/articles/3480094-plex-rclone https://help.put.io/en/articles/3480094-plex-rclone I'd personally go off-guide and use rclone move rather than mounting it just to avoid having to keep bumping up storage space on put.io e: hah you already got it

dndvr 6y ago

Sounds a bit like PiperNet

rambojazz 6y ago

Is this read only? Can files be edited?