4 ms·

Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the

by mhx77 6y ago

Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.

jodrellblank 6y ago
> "taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive" I imagine these days you have more than 300GB hard disk space, making this all moot?
- jandrese 6y ago
  256GB SSDs are still everywhere.
simcop2387 6y ago
Funny thing about it is that I've got a similar problem powering https://perl.bot/ https://perl.bot/ (and the associated irc bot). I don't have as many installs as you currently but It's not far off and I want to add more compile time settings to them. I'd need to setup a full build server/system though because I need to regularly update them with new modules. How opposed would you be to this being reworked to being able to be mainline kernel support too?
- mhx77 6y ago
  > How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that aren't even shipped by a lot of distributions (folly & fbthrift). And, tbh, I don't see much benefit given that FUSE these days doesn't seem to be significantly worse in terms of performance. > I'd need to setup a full build server/system though because I need to regularly update them with new modules. Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past.
  - aarchi 6y ago
    Is it possible to rebuild a DwarFS fs to incorporate changes from an overlay fs without decompressing, then recompressing? It seems feasible that a second DwarFS fs could be built from an overlay/DwarFS, then delete the original overlay/DwarFS fs. That would require 2N storage as the new DwarFS is being built. Is it possible to patch an existing DwarFS? By overlay, are you referring to overlayfs [0]? https://wiki.archlinux.org/index.php/Overlay_filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem
    - mhx77 6y ago
      Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#setting-up-a-writable-file-system-on-top-of-a-dwarfs-image https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
  - JoshTriplett 6y ago
    > Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past. It would be nice to be able to build a new read-only filesystem in incremental mode: given a compressed filesystem and some new uncompressed data, incorporate the uncompressed data without completely re-doing all the work.

by mhx77 6y ago

Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.

jodrellblank 6y ago

> "taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive" I imagine these days you have more than 300GB hard disk space, making this all moot?

jandrese 6y ago
256GB SSDs are still everywhere.

simcop2387 6y ago

Funny thing about it is that I've got a similar problem powering https://perl.bot/ https://perl.bot/ (and the associated irc bot). I don't have as many installs as you currently but It's not far off and I want to add more compile time settings to them. I'd need to setup a full build server/system though because I need to regularly update them with new modules. How opposed would you be to this being reworked to being able to be mainline kernel support too?

mhx77 6y ago
> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on libraries that aren't even shipped by a lot of distributions (folly & fbthrift). And, tbh, I don't see much benefit given that FUSE these days doesn't seem to be significantly worse in terms of performance. > I'd need to setup a full build server/system though because I need to regularly update them with new modules. Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past.
- aarchi 6y ago
  Is it possible to rebuild a DwarFS fs to incorporate changes from an overlay fs without decompressing, then recompressing? It seems feasible that a second DwarFS fs could be built from an overlay/DwarFS, then delete the original overlay/DwarFS fs. That would require 2N storage as the new DwarFS is being built. Is it possible to patch an existing DwarFS? By overlay, are you referring to overlayfs [0]? https://wiki.archlinux.org/index.php/Overlay_filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem
  - mhx77 6y ago
    Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#setting-up-a-writable-file-system-on-top-of-a-dwarfs-image https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
- JoshTriplett 6y ago
  > Overlay the mounted read-only fs with a read-write fs. Then you can install modules as you like and if you want to start fresh, just throw away the read-write fs. That's what I've done in the past. It would be nice to be able to build a new read-only filesystem in incremental mode: given a compressed filesystem and some new uncompressed data, incorporate the uncompressed data without completely re-doing all the work.