6 ms·

How .NET container images are maintained

141 points by alexis2b 6y ago

ImJasonH 6y ago
container-diff[0], mentioned in the article, is a really useful image release tool, that I don't think gets enough love. 0: https://github.com/GoogleContainerTools/container-diff https://github.com/GoogleContainerTools/container-diff
- akx 6y ago
  If only it didn't do its thing by extracting the containers into dotdirectories...
- dastx 6y ago
  Also Dive is a good tool to analise docker containers.
  - jon-wood 6y ago
    I wouldn’t normally nitpick spelling, but “analyse” (or “analyze” if you’re in the US I guess) is the word you’re looking for. “Analise” has a very different meaning, one that you probably don’t want to Google on a corporate computer.
    - mads 6y ago
      I will tell corporate I am Portugese.
    - dastx 6y ago
      You are right. I never knew analise was a word, but I know analyse is the right spelling. I'm gonna chalk this up to auto complete error (really I don't know why I spelled it as analise).
raziel2p 6y ago
Wanted to know more about the tools they use for maintaining and scanning, but I guess many of them are internal and not open source?
- mthalman 6y ago
  For the scanning, we (.NET team) use the scanning services provided in Azure Container Registry (ACR). This is an internal ACR and the results of that are internal as mentioned in the post. All the other tooling we use is open source. You can find our build infrastructure at https://github.com/dotnet/docker-tools https://github.com/dotnet/docker-tools. There's a tool there called image-builder that provides much of the functionality. I've written a blog post on how we use Azure Pipelines to manage the builds: https://devblogs.microsoft.com/dotnet/how-the-net-team-uses-azure-pipelines-to-produce-docker-images/ https://devblogs.microsoft.com/dotnet/how-the-net-team-uses-.... Between image-builder and the pipelines, there's some automation that automatically rebuilds our images whenever a parent image changes.
jonjonsonjr 6y ago
> Comparing image digests won’t work; they will never match. This is a strong assertion with no further explanation. It reads like a generic truth about container images, but it's certainly possible to achieve this, as referenced later: > Sidebar: Various folks in the container ecosystem are looking at enabling deterministic images. We welcome that. See Building deterministic Docker images with Bazel and DETERMINISTIC DOCKER IMAGES WITH GO MICROSERVICES. I'll agree that docker makes it _really_ difficult to build and consume reproducible images (for a variety of reasons, see https://github.com/google/go-containerregistry/issues/895#issuecomment-753521526 https://github.com/google/go-containerregistry/issues/895#is... and https://twitter.com/lorenc_dan/status/1343921451792003073 https://twitter.com/lorenc_dan/status/1343921451792003073 for a sampling of interesting ones), but there is more to the container ecosystem than docker or Dockerfiles. Shameless plug: I help maintain ko (https://github.com/google/ko https://github.com/google/ko), which can achieve reproducible builds for go projects without much fuss. It also leans heavily on go's excellent support for cross-compilation to produce multi-platform images, trivially. > There are two cases where the container-diff tool will report that the registry and local images that you are comparing are the same (in terms of Docker history lines), but will be misleading because the images are actually different. While container-diff is great, it can obscure what's really going on a bit. If you're interested in uncovering exactly why the digest of the image you built is different from what was published, please forgive another shameless plug for crane (https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md https://github.com/google/go-containerregistry/blob/main/cmd...), a tool I wrote to expose most of the functionality of go-containerregistry (https://github.com/google/go-containerregistry https://github.com/google/go-containerregistry), which is the library both container-diff and ko use under the hood. Forgive the sparse documentation, but it should be relatively straightforward for anyone familiar with the registry API and data structures, as the commands map pretty directly to registry functionality. Using crane, you can easily inspect the image in the registry directly to compare the manifests and blobs that make up an image. For example, one reason that the digests might never match is that these images are somewhat strangely wrapped as singleton manifest lists: https://gist.github.com/jonjohnsonjr/ffba104ca504b5bb4a1f22749e624abf https://gist.github.com/jonjohnsonjr/ffba104ca504b5bb4a1f227... It makes some sense to me that they might want to do this to prevent folks from pulling this on windows, but usually you would only encounter manifest lists for multi-platform images. Even if these builds were reproducible, you would have to compare the digest of what you built with sha256:9a210bb9cbbdba5ae2199b659551959cd01e0299419f4118d111f8443971491a -- not the sha256:fb1a43b50c7047e5f28e309268a8f5425abc9cb852124f6828dcb0e4f859a4a1 that docker outputs, as shown in the article. The tag used for this example (mcr.microsoft.com/dotnet/sdk:5.0-alpine) has since been updated. Comparing this with the original using container-diff just tells us that the size changed: https://gist.github.com/jonjohnsonjr/90c2def551833c8cacf3264248825855 https://gist.github.com/jonjohnsonjr/90c2def551833c8cacf3264... But looking at the actual manifests, config blobs, and layers using crane is often faster and more interesting: https://gist.github.com/jonjohnsonjr/283eab27d996b2f4cc045530283b40be https://gist.github.com/jonjohnsonjr/283eab27d996b2f4cc04553... My intention with crane is to be easily composable so that you can use familiar tools like tar, sort, diff, jq, etc. (To be fair to container-diff, you can use the -t flag to show similar things.) I realize this is not really the point of the article, but it's a huge pet peeve of mine that everyone has just given up on understanding what's going on with their images because the tooling UX makes everything so opaque. If the digest of something doesn't match, you should know why! It's as if `git push --force` was on by default and everyone has just accepted that reality. Now to read the rest of the article :)
bambam24 6y ago
Why uses it is the question? NY State?
qbasic_forever 6y ago
I am really curious, who is actually using Windows containers in a way that's critical to their business? (i.e. your main web app, not giving some janky legacy Windows thing a place to live while you migrate away from it) I have to imagine at some point probably very soon all of the time spent maintaining and hacking in the NT kernel to work in a container world is just throwing good money at a vanishingly small segment of the market. kudos to this team for being on the ball and supporting ARM and ARM64 images though. Now when can we actually run ARM64 stuff on Azure....
- vyrotek 6y ago
  There are large and successful companies with very modern enterprise apps communicating with old stuff like AS400 mainframe systems. These and many other companies also have generations of (not quite as old) Windows applications that aren't going anywhere anytime soon either. These companies want to manage it all in a modern way. Sometimes, it's part of the migration strategy.
- foepys 6y ago
  Hacker News represents only a pretty small fraction of the IT world. There are many, many companies out there doing things Hacker News never heard of because it's not related to web or startups. What I do at my day job would be considered legacy tech here on HN but my company is a multi million dollar profitable business working in the B2B sector. Our customers don't want flashy websites, they want software that works over many years with minimal maintenance burden running on their own hardware. Microsoft Windows is still dominant in the business world because the often external IT knows how to work on it and most of the time it just works.
- runjake 6y ago
  I don’t know anyone using Windows Containers in production. Even Microsoft is using Linux containers for production dotnet deployments.
  - pjmlp 6y ago
    I would pay to see SharePoint and Dynamics running on Linux containers. Don't judge what Microsoft is doing, with the .NET Core teams are marketing for.

141 points by alexis2b 6y ago

container-diff[0], mentioned in the article, is a really useful image release tool, that I don't think gets enough love. 0: https://github.com/GoogleContainerTools/container-diff https://github.com/GoogleContainerTools/container-diff

akx 6y ago
If only it didn't do its thing by extracting the containers into dotdirectories...
dastx 6y ago
Also Dive is a good tool to analise docker containers.
- jon-wood 6y ago
  I wouldn’t normally nitpick spelling, but “analyse” (or “analyze” if you’re in the US I guess) is the word you’re looking for. “Analise” has a very different meaning, one that you probably don’t want to Google on a corporate computer.
  - mads 6y ago
    I will tell corporate I am Portugese.
  - dastx 6y ago
    You are right. I never knew analise was a word, but I know analyse is the right spelling. I'm gonna chalk this up to auto complete error (really I don't know why I spelled it as analise).

raziel2p 6y ago

Wanted to know more about the tools they use for maintaining and scanning, but I guess many of them are internal and not open source?

mthalman 6y ago
For the scanning, we (.NET team) use the scanning services provided in Azure Container Registry (ACR). This is an internal ACR and the results of that are internal as mentioned in the post. All the other tooling we use is open source. You can find our build infrastructure at https://github.com/dotnet/docker-tools https://github.com/dotnet/docker-tools. There's a tool there called image-builder that provides much of the functionality. I've written a blog post on how we use Azure Pipelines to manage the builds: https://devblogs.microsoft.com/dotnet/how-the-net-team-uses-azure-pipelines-to-produce-docker-images/ https://devblogs.microsoft.com/dotnet/how-the-net-team-uses-.... Between image-builder and the pipelines, there's some automation that automatically rebuilds our images whenever a parent image changes.

jonjonsonjr 6y ago

> Comparing image digests won’t work; they will never match. This is a strong assertion with no further explanation. It reads like a generic truth about container images, but it's certainly possible to achieve this, as referenced later: > Sidebar: Various folks in the container ecosystem are looking at enabling deterministic images. We welcome that. See Building deterministic Docker images with Bazel and DETERMINISTIC DOCKER IMAGES WITH GO MICROSERVICES. I'll agree that docker makes it _really_ difficult to build and consume reproducible images (for a variety of reasons, see https://github.com/google/go-containerregistry/issues/895#issuecomment-753521526 https://github.com/google/go-containerregistry/issues/895#is... and https://twitter.com/lorenc_dan/status/1343921451792003073 https://twitter.com/lorenc_dan/status/1343921451792003073 for a sampling of interesting ones), but there is more to the container ecosystem than docker or Dockerfiles. Shameless plug: I help maintain ko (https://github.com/google/ko https://github.com/google/ko), which can achieve reproducible builds for go projects without much fuss. It also leans heavily on go's excellent support for cross-compilation to produce multi-platform images, trivially. > There are two cases where the container-diff tool will report that the registry and local images that you are comparing are the same (in terms of Docker history lines), but will be misleading because the images are actually different. While container-diff is great, it can obscure what's really going on a bit. If you're interested in uncovering exactly why the digest of the image you built is different from what was published, please forgive another shameless plug for crane (https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md https://github.com/google/go-containerregistry/blob/main/cmd...), a tool I wrote to expose most of the functionality of go-containerregistry (https://github.com/google/go-containerregistry https://github.com/google/go-containerregistry), which is the library both container-diff and ko use under the hood. Forgive the sparse documentation, but it should be relatively straightforward for anyone familiar with the registry API and data structures, as the commands map pretty directly to registry functionality. Using crane, you can easily inspect the image in the registry directly to compare the manifests and blobs that make up an image. For example, one reason that the digests might never match is that these images are somewhat strangely wrapped as singleton manifest lists: https://gist.github.com/jonjohnsonjr/ffba104ca504b5bb4a1f22749e624abf https://gist.github.com/jonjohnsonjr/ffba104ca504b5bb4a1f227... It makes some sense to me that they might want to do this to prevent folks from pulling this on windows, but usually you would only encounter manifest lists for multi-platform images. Even if these builds were reproducible, you would have to compare the digest of what you built with sha256:9a210bb9cbbdba5ae2199b659551959cd01e0299419f4118d111f8443971491a -- not the sha256:fb1a43b50c7047e5f28e309268a8f5425abc9cb852124f6828dcb0e4f859a4a1 that docker outputs, as shown in the article. The tag used for this example (mcr.microsoft.com/dotnet/sdk:5.0-alpine) has since been updated. Comparing this with the original using container-diff just tells us that the size changed: https://gist.github.com/jonjohnsonjr/90c2def551833c8cacf3264248825855 https://gist.github.com/jonjohnsonjr/90c2def551833c8cacf3264... But looking at the actual manifests, config blobs, and layers using crane is often faster and more interesting: https://gist.github.com/jonjohnsonjr/283eab27d996b2f4cc045530283b40be https://gist.github.com/jonjohnsonjr/283eab27d996b2f4cc04553... My intention with crane is to be easily composable so that you can use familiar tools like tar, sort, diff, jq, etc. (To be fair to container-diff, you can use the -t flag to show similar things.) I realize this is not really the point of the article, but it's a huge pet peeve of mine that everyone has just given up on understanding what's going on with their images because the tooling UX makes everything so opaque. If the digest of something doesn't match, you should know why! It's as if `git push --force` was on by default and everyone has just accepted that reality. Now to read the rest of the article :)

bambam24 6y ago

Why uses it is the question? NY State?

qbasic_forever 6y ago

I am really curious, who is actually using Windows containers in a way that's critical to their business? (i.e. your main web app, not giving some janky legacy Windows thing a place to live while you migrate away from it) I have to imagine at some point probably very soon all of the time spent maintaining and hacking in the NT kernel to work in a container world is just throwing good money at a vanishingly small segment of the market. kudos to this team for being on the ball and supporting ARM and ARM64 images though. Now when can we actually run ARM64 stuff on Azure....

vyrotek 6y ago
There are large and successful companies with very modern enterprise apps communicating with old stuff like AS400 mainframe systems. These and many other companies also have generations of (not quite as old) Windows applications that aren't going anywhere anytime soon either. These companies want to manage it all in a modern way. Sometimes, it's part of the migration strategy.
foepys 6y ago
Hacker News represents only a pretty small fraction of the IT world. There are many, many companies out there doing things Hacker News never heard of because it's not related to web or startups. What I do at my day job would be considered legacy tech here on HN but my company is a multi million dollar profitable business working in the B2B sector. Our customers don't want flashy websites, they want software that works over many years with minimal maintenance burden running on their own hardware. Microsoft Windows is still dominant in the business world because the often external IT knows how to work on it and most of the time it just works.
runjake 6y ago
I don’t know anyone using Windows Containers in production. Even Microsoft is using Linux containers for production dotnet deployments.
- pjmlp 6y ago
  I would pay to see SharePoint and Dynamics running on Linux containers. Don't judge what Microsoft is doing, with the .NET Core teams are marketing for.