4 ms·

It is sad how much waste will occur because OCI used Docker's flat layer list instead of appc's dependency graph. This will lead to a tremendous amount of unnec

It is sad how much waste will occur because OCI used Docker's flat layer list instead of appc's dependency graph. This will lead to a tremendous amount of unnecessary network traffic and disk usage globally. I think this is a Betamax vs VHS moment on a much smaller scale.

ejholmes 9y ago
If we distribute OCI images via IPFS, problem solved.
cpuguy83 9y ago
Images don't need to be stored in this format on disk, the format is for transport/interchange. But out of curiosity, why do you say that a dependency graph would be better here? In terms of on disk storage and transport of layers, the layers are already content addressed.
- dlor 9y ago
  Not the OP so I'm just speculating, but being able to "mount" content-addressable blobs into different locations in the resulting rootfs would enable more incrementality and sharing of blobs. With a flat list, the same logical set of files will have a different checksum depending on where in the final rootfs it needs to end up, right?
  - cpuguy83 9y ago
    These blobs are already shared. (In Docker) Since each layer is only the diff from the layer before it, the content address is only from the change set. Technically you can share blobs even between an image that is `FROM centos` and one that is `FROM ubuntu` as long as the layer hashes are the same. If you have a hierarchical dependency chain, and addressing content based on the content that comes before it in the chain, you can only share blobs between images if they have a common ancestry.
    - dlor 9y ago
      Here's the example I'm thinking of: One image has a large file at a specific path, added via something like: "COPY largefile /etc/path1". Another image wants to add this same file to a different location: "COPY largefile /etc/path2/". My understanding is that these two blobs will have a different hash and can't be shared because the changeset includes both the file itself, and the destination location in the image. Is that correct?
      - cpuguy83 9y ago
        Yes, however I'm not sure this would be shared in any format right now.
stevvooe 9y ago
There is no real advantage to a dependency graph versus a list of layers. Originally, Docker images were actually fetched in sequence, similar to the ACI approach, but this blocked parallelization on each subsequent step. What has become OCI uses a list of layers to avoid this bottleneck. As long as images share layers, network traffic would be the same in either scenario. Reduced transfers will be achieved by formats that have a more granular filesystem representation. Right now, we are really held back by the continued use of tar, but that is another problem.
- philips 9y ago
  Yes, I agree that the method that OCI v1.0 & Docker v2.2 has of telling you ahead of time "you must fetch all of these objects and they will be this size" in a compact single metadata only format is the best option. Then it is up to the fetching code to choose the best way to pipeline or parallelize.
- clhodapp 9y ago
  I think that lack of parallelism has almost nothing to do with graphs vs lists. It would be pretty easy to either store the entire dependency graph's fetching-critical metadata in each image or to make it possible to fetch the metadata separately from the core data. Both of those would allow you to parallelize downloading of image data in the graph-based system. Having more granular layers can help a bit but in practice it hits its limits very quickly because higher layers will necessarily get swept up and have to be rebuilt when layers below them that they don't actually depend on get rebuilt. This rules out having a core image and a set of modules that each downstream image may or may not need (to be mixed in or not per-image). Thus, when you use the layer-based system in practice you usually end up with very little layer sharing between images, outside of the core Linux distro layers.
  - stevvooe 9y ago
    Note that I did not say "more granular layers" but more "more granular filesystem representation". This is a key point in achieving better data sharing. In fact, these use cases are not ruled out, but bolstered by OCIs approach. The introduction of alternative systems will be much easier to do when OCI is broadly implemented. As far as metadata is concerned, your suggestion is exactly what OCI does. The problem with ACI is that it embeds a large part of the metadata into a tar file, which has to be fetched in its entirety. OCI is mostly metadata scaffolding, made up of indexes, manifests and configs that can all be fetched without large bandwidth requirements. The compositional aspect that you've brought up has been explored and it doesn't make a whole lot of sense to cram that into images. Typically, such a system requires composing container filesystems through named references, allowing components to be independently rebuilt. Because this composition often relies on details of the target deployment media (orchestration system, container runtime, specific operating system, etc.), especially in how it deals with the security of name resolution, baking it into an image format leads to massive inflexibility. Moving it up the stack works much better and avoid the technical complexities that come with doing it within images. We can see this image composition in action in k8s PODs, docker stacks and other systems. Such constructions can be distributed through OCI images, through media types, but they are based on the compositional capabilities of the target system.

ejholmes 9y ago

If we distribute OCI images via IPFS, problem solved.

cpuguy83 9y ago

Images don't need to be stored in this format on disk, the format is for transport/interchange. But out of curiosity, why do you say that a dependency graph would be better here? In terms of on disk storage and transport of layers, the layers are already content addressed.

dlor 9y ago
Not the OP so I'm just speculating, but being able to "mount" content-addressable blobs into different locations in the resulting rootfs would enable more incrementality and sharing of blobs. With a flat list, the same logical set of files will have a different checksum depending on where in the final rootfs it needs to end up, right?
- cpuguy83 9y ago
  These blobs are already shared. (In Docker) Since each layer is only the diff from the layer before it, the content address is only from the change set. Technically you can share blobs even between an image that is `FROM centos` and one that is `FROM ubuntu` as long as the layer hashes are the same. If you have a hierarchical dependency chain, and addressing content based on the content that comes before it in the chain, you can only share blobs between images if they have a common ancestry.
  - dlor 9y ago
    Here's the example I'm thinking of: One image has a large file at a specific path, added via something like: "COPY largefile /etc/path1". Another image wants to add this same file to a different location: "COPY largefile /etc/path2/". My understanding is that these two blobs will have a different hash and can't be shared because the changeset includes both the file itself, and the destination location in the image. Is that correct?
    - cpuguy83 9y ago
      Yes, however I'm not sure this would be shared in any format right now.

stevvooe 9y ago

There is no real advantage to a dependency graph versus a list of layers. Originally, Docker images were actually fetched in sequence, similar to the ACI approach, but this blocked parallelization on each subsequent step. What has become OCI uses a list of layers to avoid this bottleneck. As long as images share layers, network traffic would be the same in either scenario. Reduced transfers will be achieved by formats that have a more granular filesystem representation. Right now, we are really held back by the continued use of tar, but that is another problem.

philips 9y ago
Yes, I agree that the method that OCI v1.0 & Docker v2.2 has of telling you ahead of time "you must fetch all of these objects and they will be this size" in a compact single metadata only format is the best option. Then it is up to the fetching code to choose the best way to pipeline or parallelize.
clhodapp 9y ago
I think that lack of parallelism has almost nothing to do with graphs vs lists. It would be pretty easy to either store the entire dependency graph's fetching-critical metadata in each image or to make it possible to fetch the metadata separately from the core data. Both of those would allow you to parallelize downloading of image data in the graph-based system. Having more granular layers can help a bit but in practice it hits its limits very quickly because higher layers will necessarily get swept up and have to be rebuilt when layers below them that they don't actually depend on get rebuilt. This rules out having a core image and a set of modules that each downstream image may or may not need (to be mixed in or not per-image). Thus, when you use the layer-based system in practice you usually end up with very little layer sharing between images, outside of the core Linux distro layers.
- stevvooe 9y ago
  Note that I did not say "more granular layers" but more "more granular filesystem representation". This is a key point in achieving better data sharing. In fact, these use cases are not ruled out, but bolstered by OCIs approach. The introduction of alternative systems will be much easier to do when OCI is broadly implemented. As far as metadata is concerned, your suggestion is exactly what OCI does. The problem with ACI is that it embeds a large part of the metadata into a tar file, which has to be fetched in its entirety. OCI is mostly metadata scaffolding, made up of indexes, manifests and configs that can all be fetched without large bandwidth requirements. The compositional aspect that you've brought up has been explored and it doesn't make a whole lot of sense to cram that into images. Typically, such a system requires composing container filesystems through named references, allowing components to be independently rebuilt. Because this composition often relies on details of the target deployment media (orchestration system, container runtime, specific operating system, etc.), especially in how it deals with the security of name resolution, baking it into an image format leads to massive inflexibility. Moving it up the stack works much better and avoid the technical complexities that come with doing it within images. We can see this image composition in action in k8s PODs, docker stacks and other systems. Such constructions can be distributed through OCI images, through media types, but they are based on the compositional capabilities of the target system.