4 ms·

Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of co

Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.

deleted 6mo ago
[deleted]
Filligree 6mo ago
Why would I need the compressed layers?
- NewJazz 6mo ago
  Pushing
  - mort96 6mo ago
    What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...
- cryptonym 6mo ago
  To save disk space /s
- XYen0n 6mo ago
  The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
  - flakes 6mo ago
    Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.
    - XYen0n 6mo ago
      You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.
      - flakes 6mo ago
        Thats true. And regardless of compressed vs regular tar, I think the OCI format working with opaque archives is extremely limiting. I hope the industry will eventually redesign to use content addressable storage per file and have metadata to describe the layer/disk layout instead. That would allow per file deduplication, and we can use tar for just bulk transfer over the wire, rather than using tar for the data at rest.
        cpuguy83 6mo ago
        containerd 2.3 has support for erofs which does a direct import of the layer. It can even convert the tar based layers to erofs, faster than extracting the tar normally. Also looking at block-based content store so that blocks can be deduped across images.
    - cpuguy83 6mo ago
      That is not correct. You would have to use the same compression tool (and likely version) for this to match. Old docker discarded the compressed bits but kept some metadata about the the so it can at least recreate the tar. It also recreated the manifest o push.
      - flakes 6mo ago
        Thanks for the correction. I did mean given the same tooling version/parameters, but (as you and others pointed out) preserving and recreating that state is not at all straightforward.
  - mort96 6mo ago
    If that's the purpose, couldn't you store the hash and throw away the compressed image? (As others said, compression is deterministic for the same algorithm, parameters and input data)
    - a_t48 6mo ago
      Zstd for example only promises determinism on the same version of the library. I've personally seen the hashes mutate between pull and export. Things like tar padding also make a difference. Really, the thing to do is to hash on the _uncompressed_ data and let compression be a transport/registry detail. That's what I've done, at least.
      - mort96 6mo ago
        I didn't know that about zstd, that's a bit unfortunate. Tar isn't related here though, we're talking about compression not archival formats
        thaJeztah 6mo ago
        Yes, compression being part of the OCI image's digest was (in hindsight) a poor decision. _Technically_ OCI images allow uncompressed layers, and the layers could be included without compression (and transport compression to be used); this would allow layers to be fully reproducible. We explored some options to do this (and made some preparations; https://github.com/containerd/containerd/pull/8166 https://github.com/containerd/containerd/pull/8166), but also discovered that various implementations of registry clients didn't handle transport-compression correctly (https://github.com/distribution/distribution/pull/3754 https://github.com/distribution/distribution/pull/3754), which could result in client either pulling the full, uncompressed, content, or image validation failing.
        a_t48 6mo ago
        For my registry fork/custom pull client I hash on the uncompressed content and store as compressed under the uncompressed digest. This lets me have my cake and eat it, too - compression free digests, smaller storage costs, be able to set consistent compression settings, have the ability to spend extra CPU to recompress on the backend without breaking hashes, etc. I control both pull client and registry, so it works.
freedomben 6mo ago
did you mean the first "compressed" to be "uncompressed" ?
colechristensen 6mo ago
I'm not sure about the fastest macbook disk access, but even with NVMe storage I've found lz4 to be faster than the disk. That is (it's hard to say this exactly correct) compressed content gets read/written FASTER than uncompressed content because fewer bytes need to transit the disk interface and the CPU is able to compress/decompress significantly faster than data is able to go through whatever disk bus you've got.
- fpoling 6mo ago
  On my 2 years old ThinkPad laptop SSD is faster than lz4. On a fat EC2 server lz4 is faster. So one really has to test a particular config.
  - colechristensen 6mo ago
    Yeah, I'm not surprised the PCIe 5.0 transfer speeds matched with top tier SSD chips win that race. It still bothers me that the fastest most performant computer I have access to is almost always my laptop, and that by a considerable margin. Someone should do some lz4 vs. ssd benchmarks across hardware to make my argument more solid and the boundaries clear.
    - UltraSane 6mo ago
      You can get AWS instances with very fast local NVMe drives.
      - colechristensen 6mo ago
        If you have many cores and have the right optimizations in place the bottleneck for lz4 decompression is RAM throughput which is always going to beat whatever fancy disk setup you have. But yes on the extreme end absolutely there's a point where lz4 stops making sense, but also most of us aren't trying to max out a 128 core postgres server or whatever.
awestroke 6mo ago
But if it stores the uncompressed layers, why store the compressed ones too? Why both at the same time?

Sounds like a straightforward time-space tradeoff: if you have the compressed layers sitting around when you need them, you can avoid the expense and time of compressing them.

deleted 6mo ago

[deleted]

Filligree 6mo ago

Why would I need the compressed layers?

NewJazz 6mo ago
Pushing
- mort96 6mo ago
  What about pushing? Computers are fast enough to compress stuff as it's being transmitted, you don't need to store the compressed copy anywhere...
cryptonym 6mo ago
To save disk space /s
XYen0n 6mo ago
The OCI manifest references the hashes of these compressed layers, and re-compressing them does not guarantee obtaining the same hash
- flakes 6mo ago
  Recompressing should be guaranteed deterministic. It’s the packing/unpacking of tar archives to/from directories on disk that leads to the non-determinism (such as timestamps and ownership metadata). If the tar is left intact, both zstd and gzip should produce byte for byte identical outputs given the same compression parameters.
  - XYen0n 6mo ago
    You are correct; I confused archiving with compression. However, even considering only the compression process, same compression parameters cannot be guaranteed, as it is unknown which compression parameters the image publisher used.
    - flakes 6mo ago
      Thats true. And regardless of compressed vs regular tar, I think the OCI format working with opaque archives is extremely limiting. I hope the industry will eventually redesign to use content addressable storage per file and have metadata to describe the layer/disk layout instead. That would allow per file deduplication, and we can use tar for just bulk transfer over the wire, rather than using tar for the data at rest.
      - cpuguy83 6mo ago
        containerd 2.3 has support for erofs which does a direct import of the layer. It can even convert the tar based layers to erofs, faster than extracting the tar normally. Also looking at block-based content store so that blocks can be deduped across images.
  - cpuguy83 6mo ago
    That is not correct. You would have to use the same compression tool (and likely version) for this to match. Old docker discarded the compressed bits but kept some metadata about the the so it can at least recreate the tar. It also recreated the manifest o push.
    - flakes 6mo ago
      Thanks for the correction. I did mean given the same tooling version/parameters, but (as you and others pointed out) preserving and recreating that state is not at all straightforward.
- mort96 6mo ago
  If that's the purpose, couldn't you store the hash and throw away the compressed image? (As others said, compression is deterministic for the same algorithm, parameters and input data)
  - a_t48 6mo ago
    Zstd for example only promises determinism on the same version of the library. I've personally seen the hashes mutate between pull and export. Things like tar padding also make a difference. Really, the thing to do is to hash on the _uncompressed_ data and let compression be a transport/registry detail. That's what I've done, at least.
    - mort96 6mo ago
      I didn't know that about zstd, that's a bit unfortunate. Tar isn't related here though, we're talking about compression not archival formats
      - thaJeztah 6mo ago
        Yes, compression being part of the OCI image's digest was (in hindsight) a poor decision. _Technically_ OCI images allow uncompressed layers, and the layers could be included without compression (and transport compression to be used); this would allow layers to be fully reproducible. We explored some options to do this (and made some preparations; https://github.com/containerd/containerd/pull/8166 https://github.com/containerd/containerd/pull/8166), but also discovered that various implementations of registry clients didn't handle transport-compression correctly (https://github.com/distribution/distribution/pull/3754 https://github.com/distribution/distribution/pull/3754), which could result in client either pulling the full, uncompressed, content, or image validation failing.
        a_t48 6mo ago
        For my registry fork/custom pull client I hash on the uncompressed content and store as compressed under the uncompressed digest. This lets me have my cake and eat it, too - compression free digests, smaller storage costs, be able to set consistent compression settings, have the ability to spend extra CPU to recompress on the backend without breaking hashes, etc. I control both pull client and registry, so it works.

freedomben 6mo ago

did you mean the first "compressed" to be "uncompressed" ?

colechristensen 6mo ago

I'm not sure about the fastest macbook disk access, but even with NVMe storage I've found lz4 to be faster than the disk. That is (it's hard to say this exactly correct) compressed content gets read/written FASTER than uncompressed content because fewer bytes need to transit the disk interface and the CPU is able to compress/decompress significantly faster than data is able to go through whatever disk bus you've got.

fpoling 6mo ago
On my 2 years old ThinkPad laptop SSD is faster than lz4. On a fat EC2 server lz4 is faster. So one really has to test a particular config.
- colechristensen 6mo ago
  Yeah, I'm not surprised the PCIe 5.0 transfer speeds matched with top tier SSD chips win that race. It still bothers me that the fastest most performant computer I have access to is almost always my laptop, and that by a considerable margin. Someone should do some lz4 vs. ssd benchmarks across hardware to make my argument more solid and the boundaries clear.
  - UltraSane 6mo ago
    You can get AWS instances with very fast local NVMe drives.
    - colechristensen 6mo ago
      If you have many cores and have the right optimizations in place the bottleneck for lz4 decompression is RAM throughput which is always going to beat whatever fancy disk setup you have. But yes on the extreme end absolutely there's a point where lz4 stops making sense, but also most of us aren't trying to max out a 128 core postgres server or whatever.

awestroke 6mo ago

But if it stores the uncompressed layers, why store the compressed ones too? Why both at the same time?