5 ms·

Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage dr

by Oxodao 5mo ago

Docker already fills up my dev machines yet they decided for this insane solution: > The containerd image store uses more disk space than the legacy storage drivers for the same images. This is because containerd stores images in both compressed and uncompressed formats, while the legacy drivers stored only the uncompressed layers. Why ?

ElevenLathe 5mo ago
Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.
- deleted 5mo ago
  [deleted]
- Filligree 5mo ago
  Why would I need the compressed layers?
  - NewJazz 5mo ago
    Pushing
    - mort96 5mo ago
      What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...
  - cryptonym 5mo ago
    To save disk space /s
  - XYen0n 5mo ago
    The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
    - flakes 5mo ago
      Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.
      - XYen0n 5mo ago
        You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.
        flakes 5mo ago
        Thats true. And regardless of compressed vs regular tar, I think the OCI format working with opaque archives is extremely limiting. I hope the industry will eventually redesign to use content addressable storage per file and have metadata to describe the layer/disk layout instead. That would allow per file deduplication, and we can use tar for just bulk transfer over the wire, rather than using tar for the data at rest.
        cpuguy83 5mo ago
        containerd 2.3 has support for erofs which does a direct import of the layer. It can even convert the tar based layers to erofs, faster than extracting the tar normally. Also looking at block-based content store so that blocks can be deduped across images.
      - cpuguy83 5mo ago
        That is not correct. You would have to use the same compression tool (and likely version) for this to match. Old docker discarded the compressed bits but kept some metadata about the the so it can at least recreate the tar. It also recreated the manifest o push.
- freedomben 5mo ago
  did you mean the first "compressed" to be "uncompressed" ?
- colechristensen 5mo ago
  I'm not sure about the fastest macbook disk access, but even with NVMe storage I've found lz4 to be faster than the disk. That is (it's hard to say this exactly correct) compressed content gets read/written FASTER than uncompressed content because fewer bytes need to transit the disk interface and the CPU is able to compress/decompress significantly faster than data is able to go through whatever disk bus you've got.
  - fpoling 5mo ago
    On my 2 years old ThinkPad laptop SSD is faster than lz4. On a fat EC2 server lz4 is faster. So one really has to test a particular config.
    - colechristensen 5mo ago
      Yeah, I'm not surprised the PCIe 5.0 transfer speeds matched with top tier SSD chips win that race. It still bothers me that the fastest most performant computer I have access to is almost always my laptop, and that by a considerable margin. Someone should do some lz4 vs. ssd benchmarks across hardware to make my argument more solid and the boundaries clear.
      - UltraSane 5mo ago
        You can get AWS instances with very fast local NVMe drives.
        colechristensen 5mo ago
        If you have many cores and have the right optimizations in place the bottleneck for lz4 decompression is RAM throughput which is always going to beat whatever fancy disk setup you have. But yes on the extreme end absolutely there's a point where lz4 stops making sense, but also most of us aren't trying to max out a 128 core postgres server or whatever.
- awestroke 5mo ago
  But if it stores the uncompressed layers, why store the compressed ones too? Why both at the same time?
sschueller 5mo ago
[flagged]
- mschuster91 5mo ago
  > It is shameful for apple to hard solder their disks. There is no benefit to the user Actually, it is. The speed and latency difference does matter, that is how even an 8GB RAM MacBook feels snappier than many a 32GB Windows machine - it can use the disk as swap.
  - newsoftheday 5mo ago
    I had to work on a Mac M3 for a year, it sucked, it did not feel snappier than any Windows or Linux machine (including this one) that I've ever used and that is going back to the 1980's.
    - stingraycharles 5mo ago
      I suggest you judge based on benchmarks rather than vibes. If you believe the latest M3 does not perform better than machines you’ve used in the 80s, I have no idea how to even start a reasonable discussion about this.
      - newsoftheday 5mo ago
        > If you believe the latest M3 does not perform better than machines you’ve used in the 80s That wasn't what I was trying to say, I apologize, I should have been clearer. What I intended to say was that I've been using various, many computers since the 1980's so I have a wide and deep sampling of experiences with them and to that end...the M3 did NOT feel to me like it performed better. Regardless the benchmarks, I know how the machine should feel and I know M3 did not feel any better than any other machine I've used (and that is a lot of laptops).
        stingraycharles 5mo ago
giobox 5mo ago
> https://docs.docker.com/reference/cli/docker/system/prune/ https://docs.docker.com/reference/cli/docker/system/prune/ Just in case - I'm always amazed how many Docker users don't know about the prune command for cleaning up the caches and deleting unused container images and just slowly let their docker image cache eat their disk.
- johannes1234321 5mo ago
  Prune is nice, but if you have a bunch of containers which run shirt time for a build step or similar prune would collect those, too. A filter "last used a few months ago" would be useful.
  - deleted 5mo ago
    [deleted]
  - giobox 5mo ago
    I think you can filter on last created, but agree last used would be helpful: docker image prune -a --filter "until=24h" > https://docs.docker.com/reference/cli/docker/image/prune/#filter https://docs.docker.com/reference/cli/docker/image/prune/#fi...
  - kevinmgranger 5mo ago
    Any reason why those containers can't be run ephemerally?
  - Oxodao 5mo ago
    I do prune all the time, but working on a lot of different projects it fills up as quickly as it empties
black3r 5mo ago
Also this doesn't just mean more disk space usage, but also longer local build times... for the app I'm working on `exporting to image` takes 71.5 seconds with containerd, without containerd it's 4.3s (the rest of the build takes ~180 seconds). And that's just a 5.76GB image.
- cpuguy83 5mo ago
  That's almost certainly nothing to do with the change. Please report this if you can with commands used. Buildkit isn't changing behavior here. Internally in docker there is a shim to make the legacy storage behave like containerd snapshotters (as well as it can, anyway--not perfect due to hard to resolve issues in the old storage). But it still kept both the compressed and uncompressed versions of images.

by Oxodao 5mo ago

ElevenLathe 5mo ago

Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.

deleted 5mo ago
[deleted]
Filligree 5mo ago
Why would I need the compressed layers?
- NewJazz 5mo ago
  Pushing
  - mort96 5mo ago
    What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...
- cryptonym 5mo ago
  To save disk space /s
- XYen0n 5mo ago
  The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
  - flakes 5mo ago
    Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.
    - XYen0n 5mo ago
      You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.
      - flakes 5mo ago
        Thats true. And regardless of compressed vs regular tar, I think the OCI format working with opaque archives is extremely limiting. I hope the industry will eventually redesign to use content addressable storage per file and have metadata to describe the layer/disk layout instead. That would allow per file deduplication, and we can use tar for just bulk transfer over the wire, rather than using tar for the data at rest.
        cpuguy83 5mo ago
        containerd 2.3 has support for erofs which does a direct import of the layer. It can even convert the tar based layers to erofs, faster than extracting the tar normally. Also looking at block-based content store so that blocks can be deduped across images.
    - cpuguy83 5mo ago
      That is not correct. You would have to use the same compression tool (and likely version) for this to match. Old docker discarded the compressed bits but kept some metadata about the the so it can at least recreate the tar. It also recreated the manifest o push.
freedomben 5mo ago
did you mean the first "compressed" to be "uncompressed" ?
colechristensen 5mo ago
I'm not sure about the fastest macbook disk access, but even with NVMe storage I've found lz4 to be faster than the disk. That is (it's hard to say this exactly correct) compressed content gets read/written FASTER than uncompressed content because fewer bytes need to transit the disk interface and the CPU is able to compress/decompress significantly faster than data is able to go through whatever disk bus you've got.
- fpoling 5mo ago
  On my 2 years old ThinkPad laptop SSD is faster than lz4. On a fat EC2 server lz4 is faster. So one really has to test a particular config.
  - colechristensen 5mo ago
    Yeah, I'm not surprised the PCIe 5.0 transfer speeds matched with top tier SSD chips win that race. It still bothers me that the fastest most performant computer I have access to is almost always my laptop, and that by a considerable margin. Someone should do some lz4 vs. ssd benchmarks across hardware to make my argument more solid and the boundaries clear.
    - UltraSane 5mo ago
      You can get AWS instances with very fast local NVMe drives.
      - colechristensen 5mo ago
        If you have many cores and have the right optimizations in place the bottleneck for lz4 decompression is RAM throughput which is always going to beat whatever fancy disk setup you have. But yes on the extreme end absolutely there's a point where lz4 stops making sense, but also most of us aren't trying to max out a 128 core postgres server or whatever.
awestroke 5mo ago
But if it stores the uncompressed layers, why store the compressed ones too? Why both at the same time?

sschueller 5mo ago

[flagged]

mschuster91 5mo ago
> It is shameful for apple to hard solder their disks. There is no benefit to the user Actually, it is. The speed and latency difference does matter, that is how even an 8GB RAM MacBook feels snappier than many a 32GB Windows machine - it can use the disk as swap.
- newsoftheday 5mo ago
  I had to work on a Mac M3 for a year, it sucked, it did not feel snappier than any Windows or Linux machine (including this one) that I've ever used and that is going back to the 1980's.
  - stingraycharles 5mo ago
    I suggest you judge based on benchmarks rather than vibes. If you believe the latest M3 does not perform better than machines you’ve used in the 80s, I have no idea how to even start a reasonable discussion about this.
    - newsoftheday 5mo ago
      > If you believe the latest M3 does not perform better than machines you’ve used in the 80s That wasn't what I was trying to say, I apologize, I should have been clearer. What I intended to say was that I've been using various, many computers since the 1980's so I have a wide and deep sampling of experiences with them and to that end...the M3 did NOT feel to me like it performed better. Regardless the benchmarks, I know how the machine should feel and I know M3 did not feel any better than any other machine I've used (and that is a lot of laptops).
      - stingraycharles 5mo ago

giobox 5mo ago

> https://docs.docker.com/reference/cli/docker/system/prune/ https://docs.docker.com/reference/cli/docker/system/prune/ Just in case - I'm always amazed how many Docker users don't know about the prune command for cleaning up the caches and deleting unused container images and just slowly let their docker image cache eat their disk.

johannes1234321 5mo ago
Prune is nice, but if you have a bunch of containers which run shirt time for a build step or similar prune would collect those, too. A filter "last used a few months ago" would be useful.
- deleted 5mo ago
  [deleted]
- giobox 5mo ago
  I think you can filter on last created, but agree last used would be helpful: docker image prune -a --filter "until=24h" > https://docs.docker.com/reference/cli/docker/image/prune/#filter https://docs.docker.com/reference/cli/docker/image/prune/#fi...
- kevinmgranger 5mo ago
  Any reason why those containers can't be run ephemerally?
- Oxodao 5mo ago
  I do prune all the time, but working on a lot of different projects it fills up as quickly as it empties

black3r 5mo ago

Also this doesn't just mean more disk space usage, but also longer local build times... for the app I'm working on `exporting to image` takes 71.5 seconds with containerd, without containerd it's 4.3s (the rest of the build takes ~180 seconds). And that's just a 5.76GB image.

cpuguy83 5mo ago
That's almost certainly nothing to do with the change. Please report this if you can with commands used. Buildkit isn't changing behavior here. Internally in docker there is a shim to make the legacy storage behave like containerd snapshotters (as well as it can, anyway--not perfect due to hard to resolve issues in the old storage). But it still kept both the compressed and uncompressed versions of images.