10 ms·

BTFS: BitTorrent Filesystem

386 points by pyinstallwoes 2y ago

pyinstallwoes 2y ago
Submitting because I'm surprised why this isn't used more... couldn't we build a virtualmachine/OS's as an overlay on BTFS? Seems like an interesting direction.
- idle_zealot 2y ago
  I'm not sure I see the point. A read-only filesystem that downloads files on-the-fly is neat, but doesn't sound practical in most situations.
  - crest 2y ago
    It can be an essential component, but for on-site replication you need to coordinate your caches to make the most of your available capacity. There're efforts to implement this on top IPFS to have mutually trusted nodes elect a leader deciding who should pin what to ensure you keep enough intact copies of everything in the distributed cache, but like so many things IPFS it started out interesting, died from feature creep and "visions" instead of working code.
  - pyinstallwoes 2y ago
    Imagine that any computation is a hash, then every possible thing becomes memoized not distinguishing between data/code. Then as a consequence you have durability, cache, security to an extent, verifiability through peers (could be trusted or degrees away from peers you trust).
    - gameman144 2y ago
      Is every computation worth memoizing? I can think of very few computations I do that others would care about, and in those cases there's already a much more efficient caching layer fronting the data anyway.
      - pyinstallwoes 2y ago
        Why not? I think there is some interesting research here at the computationally level / distributed that could lead to some interesting architecture and discoveries. Fully distributed OS's/Virtual Machines/LLM's/Neural Networks If LLM's are token predictors for language, what happens when you do token prediction for computation across a distributed network? Then run a NN on the cache and clustering itself? Lots of potential use cases.
  - cyanydeez 2y ago
    If you were a billionaire and you wanted some software update, you could log into your super computer and have every shell mount the same torrent and it should be the fastest upload.
- retzkek 2y ago
  CVMFS is a mature entry in that space, heavily used in the physics community to distribute software and container images, allowing simple and efficient sharing of computing resources. https://cernvm.cern.ch/fs/ https://cernvm.cern.ch/fs/
- Jhsto 2y ago
  Just the other week I used Nix on my laptop to derive PXE boot images, uploaded those to IPFS, and netbooted my server in another country over a public IPFS mirror. The initrd gets mounted as read-only overlayfs on boot. My configs are public: https://github.com/jhvst/nix-config https://github.com/jhvst/nix-config I plan to write documentation of the IPFS process including the PXE router config later at https://github.com/majbacka-labs/nixos.fi https://github.com/majbacka-labs/nixos.fi -- we might also run a small public build server for peoples Flake configs, who are interested in trying out this process.
  - jquaint 2y ago
    This is really cool. Plan to take some inspiration from your config!
  - __MatrixMan__ 2y ago
    I laughed when I saw that your readme jumps straight into some category theory. FYI others might cry instead. You're doing some cool things here.
  - vmfunction 2y ago
    >A prominent direction in the Linux distribution scene lately has been the concept of immutable desktop operating systems. Recent examples include Fedora Silverblue (2018) and Vanilla OS (2022). But, on my anecdotal understanding of the timelines concerning Linux distros, both are spiritual successors to CoreOS (2013). Remember in the late 90's booting server off a CD-ROM was the thing.
    - hnarn 2y ago
      Booting off USB sticks is still done all the time and it's (almost) literally the same thing. Doing it in combination with encrypted persistence support as available in Debian for example can be really nice.
- infogulch 2y ago
  It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol to deliver docker images to large clusters. https://github.com/uber/kraken https://github.com/uber/kraken
  - XorNot 2y ago
    The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull. But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers. Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).
    - infogulch 2y ago
      Maybe something like this? https://github.com/containers/fuse-overlayfs https://github.com/containers/fuse-overlayfs
  - apichat 2y ago
    https://github.com/uber/kraken?tab=readme-ov-file#comparison-with-other-projects https://github.com/uber/kraken?tab=readme-ov-file#comparison... "Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."
- thesuitonym 2y ago
  Every once in a while, someone reinvents Plan 9 from Bell Labs.
Solvency 2y ago
so is this like a Dropbox alternative using the bittorrent protocol?
- VyseofArcadia 2y ago
  No,this is being able to interact with a .torrent file as if it's a directory. The only usecase I see for this is as an alternative to a more traditional bittorrent client.
  - Solvency 2y ago
    got it. but now i kind of want a torrent-based dropbox. i have five workstations. would be great to be able to utilize them as my own miniature distributed file system without a corporate server.
    - Zambyte 2y ago
      IPFS?
    - ggop 2y ago
      You could try Syncthing https://docs.syncthing.net/intro/getting-started.html https://docs.syncthing.net/intro/getting-started.html
      - jtriangle 2y ago
        Syncthing experience is greatly improved if you also host your own discovery server, and if you can port forward. Pretty minor to do, but, it's a big speed increase.
      - jszymborski 2y ago
        Syncthing is great (I use it daily!) but I'm not sure it does the Dropbox/NextCloud thing that BTFS does where you can see remote files and download them on access. Syncthing rather just syncs folders as far as I can tell.
    - 2y ago
ChrisArchitect 2y ago
Thoughts from 4 years ago: https://news.ycombinator.com/item?id=23576063 https://news.ycombinator.com/item?id=23576063
mdaniel 2y ago
the top comment <https://news.ycombinator.com/item?id=23580334 https://news.ycombinator.com/item?id=23580334> by saurik (yes, that one) on the previous 121 comment thread back in 2020 sums up my feelings about the situation: BTFS is a "one CID at a time" version of IPFS I do think IPFS is awesome, but is going to take some major advances in at least 3 areas before it becomes something usable day-to-day: 1. not running a local node proxy (I hear that Brave has some built-in WebTorrent support, so maybe that's the path, but since I don't use Brave I can't say whether they are "WebTorrent in name only" or what 2. related to that, the swarm/peer resolution latency suffers in the same way that "web3 crypto tomfoolery" does, and that latency makes "browsing" feel like the old 14.4k modem days 3. IPFS is absolutely fantastic for infrequently changing but super popular content, e.g. wikipedia, game releases, MDN content, etc, but is a super PITA to replace "tip" or "main" (if one thinks of browsing a git repo) with the "updated" version since (to the best of my knowledge) the only way folks have to resolve that newest CID is IPNS and DNS is never, ever going to be a "well, that's a good mechanism and surely doesn't contribute to one of the N things any outage always involves" I'm aware that I have spent an inordinate amount of words talking about a filesystem other than the one you submitted, but unlike BTFS, which I would never install, I think that those who click on this and are interested in the idea of BTFS may enjoy reading further into IPFS, but should bear in mind my opinion of its current shortcomings
SuperNinKenDo 2y ago
Cool concept. I assume that it seeds if and while the files are present on your device? Tried to read the manpage but unformatted manpage markdown on a phone was too difficult to read.
xmichael909 2y ago
No security at all?
- mixmastamyk 2y ago
  Give her some funked-up muzak, she treats you nice Feed her some hungry reggae, she'll love you twice
- rakoo 2y ago
  What security would be interesting here ?
- blueflow 2y ago
  What would be something that needs to be protected against?
stevefan1999 2y ago
I found it might be quite useful for huge LLMs since those are now hosted on BitTorrent. Of course it is not going to be as practical as IPFS since IPFS is content addressable and easier to do random access.
- kimixa 2y ago
  Is there much use for a partially resident LLM though? Then I don't see much of an advantage over just vanilla bittorrent - if you realistically need a full local copy to even start working anyway.
Kerbonut 2y ago
I dream of having a BTFS that will fix my "damaged" media files. E.g. ones I media shift, if my disk was scratched and portions are missing, or if the codec options I picked suck, it could download the "damaged" portions of my media and fix it seamlessly.
- gosub100 2y ago
  another use of this is to share media after I've imported it into my library. if I voluntarily scan hashes of all my media, if a smart torrent client could offer those files only (so a partial torrent because I always remove the superfluous files) it would help seed a lot of rare media files.
  - Stephen304 2y ago
    This happens to be one of the pipe dream roadmap milestones for bitmagnet: https://bitmagnet.io/#pipe-dream-features https://bitmagnet.io/#pipe-dream-features I used to use magnetico and wanted to make something that would use crawled info hashes to fetch the metadata and retrieve the file listing, then search a folder for any matching files. You'd probably want to pre-hash everything in the folder and cache the hashes. I hope bitmagnet gets that ability, it would be super cool
  - jonhohle 2y ago
    I’ve done a lot of archival of CD-ROM based games, and it’s not clear to me this is possible without a lot of coordination and consistency (there are like 7 programs that use AccurateRip, )and those only deal with audio). I have found zero instances where a bin/cue I’ve downloaded online perfectly matches (hashes) to the same disc I’ve ripped locally. I’ve had some instances where different pressings if the same content hash differently. I’ve written tools to inspect content (say in an ISO file system), and those will hash to the same value (so different sector data but the same resulting file system). Audio converted to CDDA (16-bit PCM) will hash as well. If audio is transcoded into anything else, there’s no way it would hash the same. At my last job I did something similar for build artifacts. You need the same compiler, same version, same settings, the ability to look inside the final artifact and avoid all the variable information (e.g. time). That requires a bit of domain specific information to get right.
dang 2y ago
Related: BTFS – mount any .torrent file or magnet link as directory - https://news.ycombinator.com/item?id=23576063 https://news.ycombinator.com/item?id=23576063 - June 2020 (121 comments) BitTorrent file system - https://news.ycombinator.com/item?id=10826154 https://news.ycombinator.com/item?id=10826154 - Jan 2016 (33 comments)
Cieric 2y ago
I've thought about using this for my media server in the past, but in the end I ran into to many issues trying to automate it. Then there's all the normal issues, slow downloads can wreck havoc on some programs expecting the whole thing to be there and I couldn't move files around without breaking connections. It was interesting to mess with, but in the end I just decided it would be a fun challenge to write my own in zig so I could have something "easy" to hack on in the future.
- Cieric 2y ago
  I guess giving it some visibility wouldn't be the worst idea. https://github.com/ookami125/Zig-Torrent https://github.com/ookami125/Zig-Torrent
  - all2 2y ago
    Thank you. This answers the first question I have to folks who say "I made X that is related to this".
skeledrew 2y ago
Pretty cool on the surface, but 2 things I can think of right now that mostly kills it for me (and maybe many others). 1) the things one usually use Bittorrent for tend to need the complete files to be all there to be useful, and internet speeds is a limiter in that regard. 2) seeder count tends to break down pretty quickly, as people delete or even just move the files elsewhere for whatever reason, so availability falls.
- elevenz 2y ago
  A file index tool can be add to torrent clients, letting it scan all files in a user-selected folder and tag them with torrent info. Then how can it find the torrent info for a given file? Maybe we need a central server, maybe some hacker will invent a distributed reverse torrent search network, to list what playlists has a specific song. I don't know if someone has already invented this. I think it can be the final answer to any distributed file system driven by user uploads.
  - WhiteOwlLion 2y ago
    Aren't torrents block based so file boundaries are not observed without hacks?
    - rakoo 2y ago
      torrents are file-based but in v1 the edge of a file doesn't map with the edge of a piece, so you can't easily find file's hashes. In v2 this is solved and it is possible to easily know the hash of each file in the torrent, so you can search for it in other torrents
  - skeledrew 2y ago
    IPFS has resolved the indexing issue, with content addressing. Overall I'd say it has all the pros of bittorrent and fewer of the cons, from a technical perspective. However it's more complex, which may be a reason why it isn't more adopted.
ceritium 2y ago
I did something similar some years ago, https://github.com/ceritium/fuse-torrent https://github.com/ceritium/fuse-torrent I had no idea what I was doing, most of the hard work IS done by the torrent-stream node package
Maakuth 2y ago
This is the perfect client for accessing Internet Archive content! Each IA item automatically has a torrent that has IA's web seeds. Try Big Buck Bunny: btfs https://archive.org/download/BigBuckBunny_124/BigBuckBunny_124_archive.torrent https://archive.org/download/BigBuckBunny_124/BigBuckBunny_1... mountpoint
- rnhmjoj 2y ago
  Even better, try this: btplay https://archive.org/download/BigBuckBunny_124/BigBuckBunny_124_archive.torrent
  - Maakuth 2y ago
    What is that tool?
    - palata 2y ago
      I guess this: https://github.com/johang/btfs/blob/master/scripts/btplay https://github.com/johang/btfs/blob/master/scripts/btplay
- haunter 2y ago
  I don’t know the internal workings of IA and the bittorent architecture but if an archive has too many items the torrent file won’t have them all. I see this all the time with ROM packs and magazine archives for example. +1000 items, the torrent will only have the first ~200 or so available
  - sumtechguy 2y ago
    I think for some reason IA limits the torrent size. I have seen as low as 50 with a 1000+ item archive.
sktrdie 2y ago
Or even better store data as an sqlite file that is full-text-search indexed. Then you can full-text search the torrent on demand: https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent
rwmj 2y ago
Here's a similar idea for block devices: https://libguestfs.org/nbdkit-torrent-plugin.1.html https://libguestfs.org/nbdkit-torrent-plugin.1.html It lets you create a block device (/dev/nbd0) backed by a torrent. Blocks are fetched in the background, or on demand (by prioritizing blocks according to recent read requests). In practice it works - you can even boot a VM from it - but it's quite slow unless you have lots of seeds. There's a danger, particularly with VMs, that you can hit time outs waiting for a block to be read, unless you adjust some guest kernel parameters. There are some bootable examples in that page if you want to try it.
- efrecon 2y ago
  That's really cool
apichat 2y ago
This tool should be upgrade to use Bittorrent v2 new functions. https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/ Especially merkle hash trees which enable : - per-file hash trees - directory structure
- bardan 2y ago
  somebody check the test functions...
  - logrot 2y ago
    Especially the corrupt torrent test file.
- extraduder_ire 2y ago
  Ever since I learned of bittorrent V2, I've been hoping for some per-file index to show up online. I'd love to be able to tag most or all of the bulk media files on my drives with how many copies exist in bittorrent swarms at any time. That way, I could more easily pick out big files for deletion knowing I could easily get another identical copy back later. I tried doing something like this in the past using ipfs, but I wasn't very successful and didn't find any duplicates.
090rf 2y ago
[dead]
bArray 2y ago
Couldn't help but think it would be epic if it went the other way - you throw files into a folder, it's set as read-only and then share that as a torrent.
- throwme_123 2y ago
  Someone should integrate: 1. https://github.com/ipfs/kubo/blob/master/docs/fuse.md https://github.com/ipfs/kubo/blob/master/docs/fuse.md 2. https://ipfscluster.io/ https://ipfscluster.io/
  - bArray 2y ago
    I'm not sure it's the same, IPFS would be another (valuable) thing entirely. I would be tempted to say that the torrent filesystem should be kept strictly in the scope of torrents.
anacrolix 2y ago
https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent has a fuse driver since 2013. I'm in the early stages of removing it. There are WebDAV, 3rd party FUSE, and HTTP wrappers of the client all doing similar things: serving magnet links, infohashes, and torrent files like an immutable filesystem. BitTorrent v2 support is currently in master.
raffraffraff 2y ago
So is there a server program to partner this with? Something that acts as a torrent file builder, tracker and simple file server for the torrent? I can imagine in a large org you could store a gigantic quantity of public data on a server that creates a torrent whenever the data changes, serves the.torrent file over http and also acts as a tracker. You could wrap the FUSE client in some basic logic that detects newer torrents on the server and reloads/remounts. Many moons ago I created a Linux distribution for a bank. It was based on Ubuntu NetBoot with minimal packages for their branch desktop. As the branches were serverless, the distro was self-seeding. You could walk into a building with one of them and use it to build hundreds of clones in a pretty short time. All you needed was wake-on-lan and PXE configured on the switches. The new clones could also act as seeds. Under the hood it served a custom Ubuntu repo on nginx and ran tftp/inetd and wackamole (which used libspread, neither have been maintained for years). Once a machine got built, it pulled a torrent off the "server" and added it to transmission. Once that was completed the machine could also act as a seed, so it would start up wackamole, inetd, nginx, tracker etc. At first you seed 10 machines reliably, but once they were all up, you could wake machines in greater numbers. Across hundreds of bank branches I deployed the image onto 8000 machines in a few weeks (most of the delays due to change control and staged rollout plan). Actually the hardest part was getting the first seed downloaded to the branches via their old Linux build, and using one of them to act as a seed machine. That was done in 350+ branches, over inadequate network connections (some were 256kbps ISDN)
- mdaniel 2y ago
  This may interest you, although as far as I know only AWS's S3 implements it: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTorrent.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjec... I actually have never used it in order to know if AWS puts their own trackers in the resulting .torrent or what
  - raffraffraff 2y ago
    hmmm, pretty cool. Of course you have to pay the egress tax, but still...
- 2y ago
eternityforest 2y ago
For a long time I've been thinking there should be a P2P alternative to the snap package store. This looks like it's already got 90% of the work on the technical side to implement it.

386 points by pyinstallwoes 2y ago

Submitting because I'm surprised why this isn't used more... couldn't we build a virtualmachine/OS's as an overlay on BTFS? Seems like an interesting direction.

idle_zealot 2y ago
I'm not sure I see the point. A read-only filesystem that downloads files on-the-fly is neat, but doesn't sound practical in most situations.
- crest 2y ago
  It can be an essential component, but for on-site replication you need to coordinate your caches to make the most of your available capacity. There're efforts to implement this on top IPFS to have mutually trusted nodes elect a leader deciding who should pin what to ensure you keep enough intact copies of everything in the distributed cache, but like so many things IPFS it started out interesting, died from feature creep and "visions" instead of working code.
- pyinstallwoes 2y ago
  Imagine that any computation is a hash, then every possible thing becomes memoized not distinguishing between data/code. Then as a consequence you have durability, cache, security to an extent, verifiability through peers (could be trusted or degrees away from peers you trust).
  - gameman144 2y ago
    Is every computation worth memoizing? I can think of very few computations I do that others would care about, and in those cases there's already a much more efficient caching layer fronting the data anyway.
    - pyinstallwoes 2y ago
      Why not? I think there is some interesting research here at the computationally level / distributed that could lead to some interesting architecture and discoveries. Fully distributed OS's/Virtual Machines/LLM's/Neural Networks If LLM's are token predictors for language, what happens when you do token prediction for computation across a distributed network? Then run a NN on the cache and clustering itself? Lots of potential use cases.
- cyanydeez 2y ago
  If you were a billionaire and you wanted some software update, you could log into your super computer and have every shell mount the same torrent and it should be the fastest upload.
retzkek 2y ago
CVMFS is a mature entry in that space, heavily used in the physics community to distribute software and container images, allowing simple and efficient sharing of computing resources. https://cernvm.cern.ch/fs/ https://cernvm.cern.ch/fs/
Jhsto 2y ago
Just the other week I used Nix on my laptop to derive PXE boot images, uploaded those to IPFS, and netbooted my server in another country over a public IPFS mirror. The initrd gets mounted as read-only overlayfs on boot. My configs are public: https://github.com/jhvst/nix-config https://github.com/jhvst/nix-config I plan to write documentation of the IPFS process including the PXE router config later at https://github.com/majbacka-labs/nixos.fi https://github.com/majbacka-labs/nixos.fi -- we might also run a small public build server for peoples Flake configs, who are interested in trying out this process.
- jquaint 2y ago
  This is really cool. Plan to take some inspiration from your config!
- __MatrixMan__ 2y ago
  I laughed when I saw that your readme jumps straight into some category theory. FYI others might cry instead. You're doing some cool things here.
- vmfunction 2y ago
  >A prominent direction in the Linux distribution scene lately has been the concept of immutable desktop operating systems. Recent examples include Fedora Silverblue (2018) and Vanilla OS (2022). But, on my anecdotal understanding of the timelines concerning Linux distros, both are spiritual successors to CoreOS (2013). Remember in the late 90's booting server off a CD-ROM was the thing.
  - hnarn 2y ago
    Booting off USB sticks is still done all the time and it's (almost) literally the same thing. Doing it in combination with encrypted persistence support as available in Debian for example can be really nice.
infogulch 2y ago
It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol to deliver docker images to large clusters. https://github.com/uber/kraken https://github.com/uber/kraken
- XorNot 2y ago
  The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull. But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers. Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).
  - infogulch 2y ago
    Maybe something like this? https://github.com/containers/fuse-overlayfs https://github.com/containers/fuse-overlayfs
- apichat 2y ago
  https://github.com/uber/kraken?tab=readme-ov-file#comparison-with-other-projects https://github.com/uber/kraken?tab=readme-ov-file#comparison... "Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."
thesuitonym 2y ago
Every once in a while, someone reinvents Plan 9 from Bell Labs.

Solvency 2y ago

so is this like a Dropbox alternative using the bittorrent protocol?

VyseofArcadia 2y ago
No,this is being able to interact with a .torrent file as if it's a directory. The only usecase I see for this is as an alternative to a more traditional bittorrent client.
- Solvency 2y ago
  got it. but now i kind of want a torrent-based dropbox. i have five workstations. would be great to be able to utilize them as my own miniature distributed file system without a corporate server.
  - Zambyte 2y ago
    IPFS?
  - ggop 2y ago
    You could try Syncthing https://docs.syncthing.net/intro/getting-started.html https://docs.syncthing.net/intro/getting-started.html
    - jtriangle 2y ago
      Syncthing experience is greatly improved if you also host your own discovery server, and if you can port forward. Pretty minor to do, but, it's a big speed increase.
    - jszymborski 2y ago
      Syncthing is great (I use it daily!) but I'm not sure it does the Dropbox/NextCloud thing that BTFS does where you can see remote files and download them on access. Syncthing rather just syncs folders as far as I can tell.
  - 2y ago

ChrisArchitect 2y ago

Thoughts from 4 years ago: https://news.ycombinator.com/item?id=23576063 https://news.ycombinator.com/item?id=23576063

mdaniel 2y ago

the top comment <https://news.ycombinator.com/item?id=23580334 https://news.ycombinator.com/item?id=23580334> by saurik (yes, that one) on the previous 121 comment thread back in 2020 sums up my feelings about the situation: BTFS is a "one CID at a time" version of IPFS I do think IPFS is awesome, but is going to take some major advances in at least 3 areas before it becomes something usable day-to-day: 1. not running a local node proxy (I hear that Brave has some built-in WebTorrent support, so maybe that's the path, but since I don't use Brave I can't say whether they are "WebTorrent in name only" or what 2. related to that, the swarm/peer resolution latency suffers in the same way that "web3 crypto tomfoolery" does, and that latency makes "browsing" feel like the old 14.4k modem days 3. IPFS is absolutely fantastic for infrequently changing but super popular content, e.g. wikipedia, game releases, MDN content, etc, but is a super PITA to replace "tip" or "main" (if one thinks of browsing a git repo) with the "updated" version since (to the best of my knowledge) the only way folks have to resolve that newest CID is IPNS and DNS is never, ever going to be a "well, that's a good mechanism and surely doesn't contribute to one of the N things any outage always involves" I'm aware that I have spent an inordinate amount of words talking about a filesystem other than the one you submitted, but unlike BTFS, which I would never install, I think that those who click on this and are interested in the idea of BTFS may enjoy reading further into IPFS, but should bear in mind my opinion of its current shortcomings

SuperNinKenDo 2y ago

Cool concept. I assume that it seeds if and while the files are present on your device? Tried to read the manpage but unformatted manpage markdown on a phone was too difficult to read.

xmichael909 2y ago

No security at all?

mixmastamyk 2y ago
Give her some funked-up muzak, she treats you nice Feed her some hungry reggae, she'll love you twice
rakoo 2y ago
What security would be interesting here ?
blueflow 2y ago
What would be something that needs to be protected against?

stevefan1999 2y ago

I found it might be quite useful for huge LLMs since those are now hosted on BitTorrent. Of course it is not going to be as practical as IPFS since IPFS is content addressable and easier to do random access.

kimixa 2y ago
Is there much use for a partially resident LLM though? Then I don't see much of an advantage over just vanilla bittorrent - if you realistically need a full local copy to even start working anyway.

Kerbonut 2y ago

I dream of having a BTFS that will fix my "damaged" media files. E.g. ones I media shift, if my disk was scratched and portions are missing, or if the codec options I picked suck, it could download the "damaged" portions of my media and fix it seamlessly.

gosub100 2y ago
another use of this is to share media after I've imported it into my library. if I voluntarily scan hashes of all my media, if a smart torrent client could offer those files only (so a partial torrent because I always remove the superfluous files) it would help seed a lot of rare media files.
- Stephen304 2y ago
  This happens to be one of the pipe dream roadmap milestones for bitmagnet: https://bitmagnet.io/#pipe-dream-features https://bitmagnet.io/#pipe-dream-features I used to use magnetico and wanted to make something that would use crawled info hashes to fetch the metadata and retrieve the file listing, then search a folder for any matching files. You'd probably want to pre-hash everything in the folder and cache the hashes. I hope bitmagnet gets that ability, it would be super cool
- jonhohle 2y ago
  I’ve done a lot of archival of CD-ROM based games, and it’s not clear to me this is possible without a lot of coordination and consistency (there are like 7 programs that use AccurateRip, )and those only deal with audio). I have found zero instances where a bin/cue I’ve downloaded online perfectly matches (hashes) to the same disc I’ve ripped locally. I’ve had some instances where different pressings if the same content hash differently. I’ve written tools to inspect content (say in an ISO file system), and those will hash to the same value (so different sector data but the same resulting file system). Audio converted to CDDA (16-bit PCM) will hash as well. If audio is transcoded into anything else, there’s no way it would hash the same. At my last job I did something similar for build artifacts. You need the same compiler, same version, same settings, the ability to look inside the final artifact and avoid all the variable information (e.g. time). That requires a bit of domain specific information to get right.

dang 2y ago

Related: BTFS – mount any .torrent file or magnet link as directory - https://news.ycombinator.com/item?id=23576063 https://news.ycombinator.com/item?id=23576063 - June 2020 (121 comments) BitTorrent file system - https://news.ycombinator.com/item?id=10826154 https://news.ycombinator.com/item?id=10826154 - Jan 2016 (33 comments)

Cieric 2y ago

I've thought about using this for my media server in the past, but in the end I ran into to many issues trying to automate it. Then there's all the normal issues, slow downloads can wreck havoc on some programs expecting the whole thing to be there and I couldn't move files around without breaking connections. It was interesting to mess with, but in the end I just decided it would be a fun challenge to write my own in zig so I could have something "easy" to hack on in the future.

Cieric 2y ago
I guess giving it some visibility wouldn't be the worst idea. https://github.com/ookami125/Zig-Torrent https://github.com/ookami125/Zig-Torrent
- all2 2y ago
  Thank you. This answers the first question I have to folks who say "I made X that is related to this".

skeledrew 2y ago

Pretty cool on the surface, but 2 things I can think of right now that mostly kills it for me (and maybe many others). 1) the things one usually use Bittorrent for tend to need the complete files to be all there to be useful, and internet speeds is a limiter in that regard. 2) seeder count tends to break down pretty quickly, as people delete or even just move the files elsewhere for whatever reason, so availability falls.

elevenz 2y ago
A file index tool can be add to torrent clients, letting it scan all files in a user-selected folder and tag them with torrent info. Then how can it find the torrent info for a given file? Maybe we need a central server, maybe some hacker will invent a distributed reverse torrent search network, to list what playlists has a specific song. I don't know if someone has already invented this. I think it can be the final answer to any distributed file system driven by user uploads.
- WhiteOwlLion 2y ago
  Aren't torrents block based so file boundaries are not observed without hacks?
  - rakoo 2y ago
    torrents are file-based but in v1 the edge of a file doesn't map with the edge of a piece, so you can't easily find file's hashes. In v2 this is solved and it is possible to easily know the hash of each file in the torrent, so you can search for it in other torrents
- skeledrew 2y ago
  IPFS has resolved the indexing issue, with content addressing. Overall I'd say it has all the pros of bittorrent and fewer of the cons, from a technical perspective. However it's more complex, which may be a reason why it isn't more adopted.

ceritium 2y ago

I did something similar some years ago, https://github.com/ceritium/fuse-torrent https://github.com/ceritium/fuse-torrent I had no idea what I was doing, most of the hard work IS done by the torrent-stream node package

Maakuth 2y ago

This is the perfect client for accessing Internet Archive content! Each IA item automatically has a torrent that has IA's web seeds. Try Big Buck Bunny: btfs https://archive.org/download/BigBuckBunny_124/BigBuckBunny_124_archive.torrent https://archive.org/download/BigBuckBunny_124/BigBuckBunny_1... mountpoint

rnhmjoj 2y ago
Even better, try this: btplay https://archive.org/download/BigBuckBunny_124/BigBuckBunny_124_archive.torrent
- Maakuth 2y ago
  What is that tool?
  - palata 2y ago
    I guess this: https://github.com/johang/btfs/blob/master/scripts/btplay https://github.com/johang/btfs/blob/master/scripts/btplay
haunter 2y ago
I don’t know the internal workings of IA and the bittorent architecture but if an archive has too many items the torrent file won’t have them all. I see this all the time with ROM packs and magazine archives for example. +1000 items, the torrent will only have the first ~200 or so available
- sumtechguy 2y ago
  I think for some reason IA limits the torrent size. I have seen as low as 50 with a 1000+ item archive.

sktrdie 2y ago

Or even better store data as an sqlite file that is full-text-search indexed. Then you can full-text search the torrent on demand: https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent

rwmj 2y ago

Here's a similar idea for block devices: https://libguestfs.org/nbdkit-torrent-plugin.1.html https://libguestfs.org/nbdkit-torrent-plugin.1.html It lets you create a block device (/dev/nbd0) backed by a torrent. Blocks are fetched in the background, or on demand (by prioritizing blocks according to recent read requests). In practice it works - you can even boot a VM from it - but it's quite slow unless you have lots of seeds. There's a danger, particularly with VMs, that you can hit time outs waiting for a block to be read, unless you adjust some guest kernel parameters. There are some bootable examples in that page if you want to try it.

efrecon 2y ago
That's really cool

apichat 2y ago

This tool should be upgrade to use Bittorrent v2 new functions. https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/ Especially merkle hash trees which enable : - per-file hash trees - directory structure

bardan 2y ago
somebody check the test functions...
- logrot 2y ago
  Especially the corrupt torrent test file.
extraduder_ire 2y ago
Ever since I learned of bittorrent V2, I've been hoping for some per-file index to show up online. I'd love to be able to tag most or all of the bulk media files on my drives with how many copies exist in bittorrent swarms at any time. That way, I could more easily pick out big files for deletion knowing I could easily get another identical copy back later. I tried doing something like this in the past using ipfs, but I wasn't very successful and didn't find any duplicates.

090rf 2y ago

[dead]

bArray 2y ago

Couldn't help but think it would be epic if it went the other way - you throw files into a folder, it's set as read-only and then share that as a torrent.

throwme_123 2y ago
Someone should integrate: 1. https://github.com/ipfs/kubo/blob/master/docs/fuse.md https://github.com/ipfs/kubo/blob/master/docs/fuse.md 2. https://ipfscluster.io/ https://ipfscluster.io/
- bArray 2y ago
  I'm not sure it's the same, IPFS would be another (valuable) thing entirely. I would be tempted to say that the torrent filesystem should be kept strictly in the scope of torrents.

anacrolix 2y ago

https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent has a fuse driver since 2013. I'm in the early stages of removing it. There are WebDAV, 3rd party FUSE, and HTTP wrappers of the client all doing similar things: serving magnet links, infohashes, and torrent files like an immutable filesystem. BitTorrent v2 support is currently in master.

raffraffraff 2y ago

So is there a server program to partner this with? Something that acts as a torrent file builder, tracker and simple file server for the torrent? I can imagine in a large org you could store a gigantic quantity of public data on a server that creates a torrent whenever the data changes, serves the.torrent file over http and also acts as a tracker. You could wrap the FUSE client in some basic logic that detects newer torrents on the server and reloads/remounts. Many moons ago I created a Linux distribution for a bank. It was based on Ubuntu NetBoot with minimal packages for their branch desktop. As the branches were serverless, the distro was self-seeding. You could walk into a building with one of them and use it to build hundreds of clones in a pretty short time. All you needed was wake-on-lan and PXE configured on the switches. The new clones could also act as seeds. Under the hood it served a custom Ubuntu repo on nginx and ran tftp/inetd and wackamole (which used libspread, neither have been maintained for years). Once a machine got built, it pulled a torrent off the "server" and added it to transmission. Once that was completed the machine could also act as a seed, so it would start up wackamole, inetd, nginx, tracker etc. At first you seed 10 machines reliably, but once they were all up, you could wake machines in greater numbers. Across hundreds of bank branches I deployed the image onto 8000 machines in a few weeks (most of the delays due to change control and staged rollout plan). Actually the hardest part was getting the first seed downloaded to the branches via their old Linux build, and using one of them to act as a seed machine. That was done in 350+ branches, over inadequate network connections (some were 256kbps ISDN)

mdaniel 2y ago
This may interest you, although as far as I know only AWS's S3 implements it: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTorrent.html https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjec... I actually have never used it in order to know if AWS puts their own trackers in the resulting .torrent or what
- raffraffraff 2y ago
  hmmm, pretty cool. Of course you have to pay the egress tax, but still...
2y ago

eternityforest 2y ago

For a long time I've been thinking there should be a P2P alternative to the snap package store. This looks like it's already got 90% of the work on the technical side to implement it.