4 ms·

Without complete asset transparency, these issues will only grow worse. Off and on I have pitched folks the idea that we need some concept of "universal asset

Without complete asset transparency, these issues will only grow worse. Off and on I have pitched folks the idea that we need some concept of "universal asset accounting" or a "universal asset graph". A federatable body of data about assets at various levels of detail, abstraction and age. For example: I should be able to have the digest for a file and be able to see anywhere it has ever showed up. I should be able to zoom in and out from that pivot point: to debian repositories, to container layers, to dependency graphs in NPM, to Helm charts. I should be able to see where it lives in my own systems, and exactly how it came to be there, from the network operations in my own systems all the way back to tickets and commits. For any given asset I should be able to discover everything that was assembled, transformed, transported and inspected in its scope.

icefrakker 6y ago
I want is not the same thing as I should be able to.
13security 6y ago
The best description of this I've seen is "chain of custody": Like with evidence in criminal proceedings. Docker and Git's use of content-addressable hashes helps in this regard, but there is still considerable work to be done when tracking third-party dependencies.
- jacques_chester 6y ago
  We used "chain of custody" as our analogy for buildpacks when I was working on them. It's not completely accurate, though, for the same reason as "supply chain". There isn't a single linear sequence of agents, actions and assets for a given asset. It's a graph in which things can appear many times in many permutations.
- kitplummer 6y ago
  Like In-Toto? https://landscape.cncf.io/selected=in-toto https://landscape.cncf.io/selected=in-toto
lukeschlather 6y ago
From this viewpoint, Helm is really kind of broken-by-design. It takes a perfectly good system (Kubernetes Manifests) which allows you to exactly specify the exact versions of all the containers you want on your cluster, and it wraps it in an inscrutable "helm install" invocation which can potentially pull containers from anywhere at deploy time. I've used helm template which generates manifests but unfortunately this isn't really the common use case, and it doesn't give you that much compared to your favorite text templating language.
- jacques_chester 6y ago
  I think that an accounting system which encodes uncertainty ("it could be any one of these two dozen images") will help create pressure on the upstream, as well as downstream visibility into who is affected.
- theptip 6y ago
  I’d agree with this, particularly Helm v1/2 which installed an agent (Tiller) to inject imperative statefulness back into your deployments, this obviating some of the benefits of the elegant declarative design of k8s. As of Helm 3 Tiller is gone and it’s much better, but you’re also right that the templating is horrible. I think you have clearly gone wrong when you have a template operator to indent your yaml the correct amount when composing reusable chunks into your manifests, white space debugging in yaml is not what we should be spending our time on in 2021. I think the kustomize approach of layering JSON patches is also not generalizable enough, though it’s better than Helm for simple cases. It does come with some good dev tooling in skaffold and so that’s what I’m using for now. I have long thought that simple code objects in your language of choice are likely to be the long-term winner, operator frameworks, etc. - the OpenAPI generated code isn’t that good in Python (I suspect it’s better in Go) but Pulumi or Pykube are examples of this approach done well (though neither are suitable for every use case at this time). Sadly Helm is the current default and it’s where the community has standardized so it will take a lot of work to map all the accumulated wisdom from the Helm charts into some other approach.

icefrakker 6y ago

I want is not the same thing as I should be able to.

13security 6y ago

The best description of this I've seen is "chain of custody": Like with evidence in criminal proceedings. Docker and Git's use of content-addressable hashes helps in this regard, but there is still considerable work to be done when tracking third-party dependencies.

jacques_chester 6y ago
We used "chain of custody" as our analogy for buildpacks when I was working on them. It's not completely accurate, though, for the same reason as "supply chain". There isn't a single linear sequence of agents, actions and assets for a given asset. It's a graph in which things can appear many times in many permutations.
kitplummer 6y ago
Like In-Toto? https://landscape.cncf.io/selected=in-toto https://landscape.cncf.io/selected=in-toto

lukeschlather 6y ago

From this viewpoint, Helm is really kind of broken-by-design. It takes a perfectly good system (Kubernetes Manifests) which allows you to exactly specify the exact versions of all the containers you want on your cluster, and it wraps it in an inscrutable "helm install" invocation which can potentially pull containers from anywhere at deploy time. I've used helm template which generates manifests but unfortunately this isn't really the common use case, and it doesn't give you that much compared to your favorite text templating language.

jacques_chester 6y ago
I think that an accounting system which encodes uncertainty ("it could be any one of these two dozen images") will help create pressure on the upstream, as well as downstream visibility into who is affected.
theptip 6y ago
I’d agree with this, particularly Helm v1/2 which installed an agent (Tiller) to inject imperative statefulness back into your deployments, this obviating some of the benefits of the elegant declarative design of k8s. As of Helm 3 Tiller is gone and it’s much better, but you’re also right that the templating is horrible. I think you have clearly gone wrong when you have a template operator to indent your yaml the correct amount when composing reusable chunks into your manifests, white space debugging in yaml is not what we should be spending our time on in 2021. I think the kustomize approach of layering JSON patches is also not generalizable enough, though it’s better than Helm for simple cases. It does come with some good dev tooling in skaffold and so that’s what I’m using for now. I have long thought that simple code objects in your language of choice are likely to be the long-term winner, operator frameworks, etc. - the OpenAPI generated code isn’t that good in Python (I suspect it’s better in Go) but Pulumi or Pykube are examples of this approach done well (though neither are suitable for every use case at this time). Sadly Helm is the current default and it’s where the community has standardized so it will take a lot of work to map all the accumulated wisdom from the Helm charts into some other approach.