5 ms·

Submitting because I'm surprised why this isn't used more... couldn't we build a virtualmachine/OS's as an overlay on BTFS? Seems like an interesting direction.

idle_zealot 2y ago
I'm not sure I see the point. A read-only filesystem that downloads files on-the-fly is neat, but doesn't sound practical in most situations.
- crest 2y ago
  It can be an essential component, but for on-site replication you need to coordinate your caches to make the most of your available capacity. There're efforts to implement this on top IPFS to have mutually trusted nodes elect a leader deciding who should pin what to ensure you keep enough intact copies of everything in the distributed cache, but like so many things IPFS it started out interesting, died from feature creep and "visions" instead of working code.
- pyinstallwoes 2y ago
  Imagine that any computation is a hash, then every possible thing becomes memoized not distinguishing between data/code. Then as a consequence you have durability, cache, security to an extent, verifiability through peers (could be trusted or degrees away from peers you trust).
  - gameman144 2y ago
    Is every computation worth memoizing? I can think of very few computations I do that others would care about, and in those cases there's already a much more efficient caching layer fronting the data anyway.
    - pyinstallwoes 2y ago
      Why not? I think there is some interesting research here at the computationally level / distributed that could lead to some interesting architecture and discoveries. Fully distributed OS's/Virtual Machines/LLM's/Neural Networks If LLM's are token predictors for language, what happens when you do token prediction for computation across a distributed network? Then run a NN on the cache and clustering itself? Lots of potential use cases.
- cyanydeez 2y ago
  If you were a billionaire and you wanted some software update, you could log into your super computer and have every shell mount the same torrent and it should be the fastest upload.
retzkek 2y ago
CVMFS is a mature entry in that space, heavily used in the physics community to distribute software and container images, allowing simple and efficient sharing of computing resources. https://cernvm.cern.ch/fs/ https://cernvm.cern.ch/fs/
Jhsto 2y ago
Just the other week I used Nix on my laptop to derive PXE boot images, uploaded those to IPFS, and netbooted my server in another country over a public IPFS mirror. The initrd gets mounted as read-only overlayfs on boot. My configs are public: https://github.com/jhvst/nix-config https://github.com/jhvst/nix-config I plan to write documentation of the IPFS process including the PXE router config later at https://github.com/majbacka-labs/nixos.fi https://github.com/majbacka-labs/nixos.fi -- we might also run a small public build server for peoples Flake configs, who are interested in trying out this process.
- jquaint 2y ago
  This is really cool. Plan to take some inspiration from your config!
- __MatrixMan__ 2y ago
  I laughed when I saw that your readme jumps straight into some category theory. FYI others might cry instead. You're doing some cool things here.
- vmfunction 2y ago
  >A prominent direction in the Linux distribution scene lately has been the concept of immutable desktop operating systems. Recent examples include Fedora Silverblue (2018) and Vanilla OS (2022). But, on my anecdotal understanding of the timelines concerning Linux distros, both are spiritual successors to CoreOS (2013). Remember in the late 90's booting server off a CD-ROM was the thing.
  - hnarn 2y ago
    Booting off USB sticks is still done all the time and it's (almost) literally the same thing. Doing it in combination with encrypted persistence support as available in Debian for example can be really nice.
infogulch 2y ago
It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol to deliver docker images to large clusters. https://github.com/uber/kraken https://github.com/uber/kraken
- XorNot 2y ago
  The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull. But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers. Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).
  - infogulch 2y ago
    Maybe something like this? https://github.com/containers/fuse-overlayfs https://github.com/containers/fuse-overlayfs
- apichat 2y ago
  https://github.com/uber/kraken?tab=readme-ov-file#comparison-with-other-projects https://github.com/uber/kraken?tab=readme-ov-file#comparison... "Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."
thesuitonym 2y ago
Every once in a while, someone reinvents Plan 9 from Bell Labs.

idle_zealot 2y ago

I'm not sure I see the point. A read-only filesystem that downloads files on-the-fly is neat, but doesn't sound practical in most situations.

crest 2y ago
It can be an essential component, but for on-site replication you need to coordinate your caches to make the most of your available capacity. There're efforts to implement this on top IPFS to have mutually trusted nodes elect a leader deciding who should pin what to ensure you keep enough intact copies of everything in the distributed cache, but like so many things IPFS it started out interesting, died from feature creep and "visions" instead of working code.
pyinstallwoes 2y ago
Imagine that any computation is a hash, then every possible thing becomes memoized not distinguishing between data/code. Then as a consequence you have durability, cache, security to an extent, verifiability through peers (could be trusted or degrees away from peers you trust).
- gameman144 2y ago
  Is every computation worth memoizing? I can think of very few computations I do that others would care about, and in those cases there's already a much more efficient caching layer fronting the data anyway.
  - pyinstallwoes 2y ago
    Why not? I think there is some interesting research here at the computationally level / distributed that could lead to some interesting architecture and discoveries. Fully distributed OS's/Virtual Machines/LLM's/Neural Networks If LLM's are token predictors for language, what happens when you do token prediction for computation across a distributed network? Then run a NN on the cache and clustering itself? Lots of potential use cases.
cyanydeez 2y ago
If you were a billionaire and you wanted some software update, you could log into your super computer and have every shell mount the same torrent and it should be the fastest upload.

retzkek 2y ago

CVMFS is a mature entry in that space, heavily used in the physics community to distribute software and container images, allowing simple and efficient sharing of computing resources. https://cernvm.cern.ch/fs/ https://cernvm.cern.ch/fs/

Jhsto 2y ago

Just the other week I used Nix on my laptop to derive PXE boot images, uploaded those to IPFS, and netbooted my server in another country over a public IPFS mirror. The initrd gets mounted as read-only overlayfs on boot. My configs are public: https://github.com/jhvst/nix-config https://github.com/jhvst/nix-config I plan to write documentation of the IPFS process including the PXE router config later at https://github.com/majbacka-labs/nixos.fi https://github.com/majbacka-labs/nixos.fi -- we might also run a small public build server for peoples Flake configs, who are interested in trying out this process.

jquaint 2y ago
This is really cool. Plan to take some inspiration from your config!
__MatrixMan__ 2y ago
I laughed when I saw that your readme jumps straight into some category theory. FYI others might cry instead. You're doing some cool things here.
vmfunction 2y ago
>A prominent direction in the Linux distribution scene lately has been the concept of immutable desktop operating systems. Recent examples include Fedora Silverblue (2018) and Vanilla OS (2022). But, on my anecdotal understanding of the timelines concerning Linux distros, both are spiritual successors to CoreOS (2013). Remember in the late 90's booting server off a CD-ROM was the thing.
- hnarn 2y ago
  Booting off USB sticks is still done all the time and it's (almost) literally the same thing. Doing it in combination with encrypted persistence support as available in Debian for example can be really nice.

infogulch 2y ago

It's not an overlay provider itself, but uber/kraken is a "P2P Docker registry capable of distributing TBs of data in seconds". It uses the bittorrent protocol to deliver docker images to large clusters. https://github.com/uber/kraken https://github.com/uber/kraken

XorNot 2y ago
The problem with being a docker registry is that you're still having to double-dip: distribute to the registry, then docker pull. But you shouldn't need to: you should be able to do the same thing with a docker graph driver, so there is no registry - even daemon should perceive the local registry as "already available", even though in reality it's going to just download the parts it needs as it overlay mounts the image layers. Which would actually potentially save a ton of bandwidth, since the stuff in an image is usually quite different to the stuff any given application needs (i.e. I usually base off Ubuntu, but if I'm only throwing a Go binary in there plus wanting debugging tools maybe available, then in most executions the actual image pulled to the local disk would be very small).
- infogulch 2y ago
  Maybe something like this? https://github.com/containers/fuse-overlayfs https://github.com/containers/fuse-overlayfs
apichat 2y ago
https://github.com/uber/kraken?tab=readme-ov-file#comparison-with-other-projects https://github.com/uber/kraken?tab=readme-ov-file#comparison... "Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more control over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is to reduce global max download time and communication overhead in a stable environment, while BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more copies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we hope to make it compatible with BitTorrent again."

thesuitonym 2y ago

Every once in a while, someone reinvents Plan 9 from Bell Labs.