4 ms·

I have this same kind of thing, for running https://perl.bot/ https://perl.bot/ and related services. I'm using BTRFS however (due to a long story and actual r

I have this same kind of thing, for running https://perl.bot/ https://perl.bot/ and related services. I'm using BTRFS however (due to a long story and actual reasons) and use it to dedupe, compress and discard a file backed filesystem for them. I'm at a logical store of about 46GB with an on disk size of 25GB. Most of that is the fact that I have several hundred (550ish last i counted) libraries and modules installed into each install of perl. This means that they dedupe incredibly well since they're basically perfect copies, and most of it is text based so it compresses well too. I've looked at DwarFS before for this same use case but the fact that it's read only makes it more difficult to handle since I'd have to have an uncompressed version sitting out elsewhere. Though I've now got the hardware to actually put that all into a CI/CD that generates the image. I might actually work on that once I get my turing pi 2 boards since I want to port this whole setup to ARM as well as x86_64. I might also put it on my RISC-V board too but it's too slow so I think it won't be as useful. EDIT: fixed the tbd size, it was taking a while to calculate.

liuliu 4y ago
Can you have OverlayFS on top of it to make it writable and periodically generate new base DwarFS from that overlay?
- simcop2387 4y ago
  I'd imagine so, that might be a good strategy for doing it and you could measure the newly setup stuff fairly easily.
- pdimitar 4y ago
  People do something similar with e.g. Raspberry Pi machines with a microSD card; they make an overlayfs setup where the OS logs et. al. get written to memory and are only being synced to the main microSD card once an hour or so -- to avoid amortizing it too quickly. So I'd imagine making something like that but with DwarFS below would be quite easy although it'd require you to set it up by hand. Still, once done it'll likely be a rock-solid setup for a long time.

liuliu 4y ago

Can you have OverlayFS on top of it to make it writable and periodically generate new base DwarFS from that overlay?

simcop2387 4y ago
I'd imagine so, that might be a good strategy for doing it and you could measure the newly setup stuff fairly easily.
pdimitar 4y ago
People do something similar with e.g. Raspberry Pi machines with a microSD card; they make an overlayfs setup where the OS logs et. al. get written to memory and are only being synced to the main microSD card once an hour or so -- to avoid amortizing it too quickly. So I'd imagine making something like that but with DwarFS below would be quite easy although it'd require you to set it up by hand. Still, once done it'll likely be a rock-solid setup for a long time.