5 ms·

Preventing Kubernetes from pulling the pause image from the internet

97 points by meatmanek 11mo ago

cmckn 11mo ago
It bugs me that this implementation detail of containerd has leaked to such an extent. This should be part of the containerd distribution, and should not be pulled at runtime. Instead of just swapping out the registry, try baking it into your machine image.
- chupasaurus 11mo ago
  It's implementation of cri plugin. > This should be part of the containerd distribution containerd is not the only CRI runtime out there.
  - cmckn 11mo ago
    > It's implementation of cri plugin. Right, that’s the point. A user of the CRI should not have to care about this implementation detail. > containerd is not the only CRI runtime out there. Any CRI that needs a pause executable should come with one.
    - chupasaurus 11mo ago
      > A user of the CRI should not have to care about this implementation detail. This detail is implemented by K8s, not a container runtime. User has to care about the source of it due to supply chain attacks, though. > Any CRI that needs a pause executable should come with one. Which would introduce a simultaneous update across different projects, which is a problem harder than a line of config.
- spwa4 11mo ago
  More general one would wish that Kubernetes had a few extra ways to get images, so you could grow on a scale from "minimal infrastructure" to "fully CI/CD". Starting with just sending the image in the RPC itself or even just on local disk (you figure out how to get it there), all the way up to registries with tightly controlled versioning.
  - p_l 11mo ago
    It's possible to do that, as kubernetes only passes the image information to CRI. You can also setup a separate service to "push" images directly to your container runtime, someone even demoed one in Show HN post some time ago I think.
- kawsper 11mo ago
  Relying on an hosted image also caused some disruptions for Nomad (the scheduler from Hashicorp), because the default pause image was hosted at gcr.io which google killed, and it moved to registry.k8s.io. The nomad team made this configurable afterwards.
  - verdverm 11mo ago
    That nomad was hit with this after years of notice and deprecation extension, seems a sign of serious maintenance issues
- yencabulator 11mo ago
  It's almost as if Docker (the company) wants it this way.
fred_is_fred 11mo ago
I've used k8s before a lot and at several companies. I am convinced that 99.9% of the people who use it should not be. But it's more fun than deploying VM images at least.
- kachapopopow 11mo ago
  I use k3s for my home and for dev envs I think it's completely fine especially when it comes to deployment documentation. I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot. Well... it's what I would have said until bitnami pulled the rug and pretty much ruined the entire ecosystem as now you don't have a way to pull something that you know is trusted with similar configuration and all from a single repository which makes deployments a pain in the ass. However, on the plus side I've just been creating my own every time I need one with the help of claude using bitnami as reference and honestly it doesn't take that much more time and keeping them up to date is relatively easy as well with ci automations.
  - Imustaskforhelp 11mo ago
    > I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot. Thoughts on Tmux-resurrect[1] , it can even resurrect programs running inside of it as well. It feels like it can as such reduce complexity from something like k3s back to tmux. What are your thoughts on it? [1]:https://github.com/tmux-plugins/tmux-resurrect?tab=readme-ov-file https://github.com/tmux-plugins/tmux-resurrect?tab=readme-ov...
    - kachapopopow 11mo ago
      I had it break enough times to where I just don't bother.
      - Imustaskforhelp 11mo ago
        Well firstly I would love to know more about your workflow where it actually broke etc. because I feel like tmux-ressurect team could help or something for sure. I haven't used the tool itself so I am curious as I was thinking of a similar workflow as well sometime ago Now please answer the above questions but also I am going to assume that you are right about tmux-ressurect, even then there are other ways of doing the same thing as well. https://www.baeldung.com/linux/process-save-restore https://www.baeldung.com/linux/process-save-restore This mentions either Criu if you want a process to persist after a shutdown, or the shutdown utility's flags if you want to temporarily do it. I have played around with Criu and docker, docker can even use criu with things like docker checkpoint and I have played with that as well (I used it to shutdown mid compression of a large file and recontinue compression exactly from where I left) What are your thoughts on using criu+docker/criu + termux, I think that it itself might be an easier thing than k3s for your workflow. Plus, I have seen some people mention vps where they are running the processes for 300 days or even more without a single shutdown iirc and I feel like modern VPS providers are insanely good at uptime, even more so than sometimes cloud providers.
nneonneo 11mo ago
O/T, but I'm getting a cert error on this page - wonder if it's just me or if this site is just serving a weird cert. Looks like it's signed by some Fortinet appliance - maybe I'm getting MITMed? Would be kind of exciting/frightening if so. EDIT: I loaded the page from a cloud box, and wow, I'm getting MITMed! Seems to only be for this site, wonder if it's some kind of sensitivity to the .family TLD.
- gregoryl 11mo ago
  Ooft. If it helps, this is the PEM I'm getting. LetEncrypt signed. -----BEGIN CERTIFICATE----- MIIFAjCCA+qgAwIBAgISBZR6PR4jNhx4fBFvqKwzJWx4MA0GCSqGSIb3DQEBCwUA MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD EwNSMTMwHhcNMjUwOTE4MTM1OTEwWhcNMjUxMjE3MTM1OTA5WjAeMRwwGgYDVQQD ExNreWxlLmNhc2NhZGUuZmFtaWx5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA55JknkVzyq5QGaRXn2TAzaOGYTHUVxl89lGOFgEEaWEvH5pcZL7xkqfv Edee7l5MeRKuK1zJ+ISPQQaEjGTk51y1aXXfOKs62NiNy6QQUbzQ+euecqrKsJVN l3PC3EYlEGibKI1gZ2x/ht8WJU9o4KiswCLqHrY7nC7BeEByv/ehiYyRTTxAXJsr 2X4LgPX6MQ1Iu10S2Bp9jnOlEV7n4RCTPFeWtfQ0CdXH45ykuwL/zrTaD111oNQE BQPNq7Ig7OihLZcJQo8TMJ3FUgzDI9z6kMy7QHNR1I8uODVUohQCO6E7A29x8nRJ UBV5DN1as3aHYFJ4FbX9s2tuLwCTiwIDAQABo4ICIzCCAh8wDgYDVR0PAQH/BAQD AgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAA MB0GA1UdDgQWBBTXwJ21Mudr9rplbA970jxJk44pEDAfBgNVHSMEGDAWgBTnq58P LDOgU9NeT3jIsoQOO9aSMzAzBggrBgEFBQcBAQQnMCUwIwYIKwYBBQUHMAKGF2h0 dHA6Ly9yMTMuaS5sZW5jci5vcmcvMB4GA1UdEQQXMBWCE2t5bGUuY2FzY2FkZS5m YW1pbHkwEwYDVR0gBAwwCjAIBgZngQwBAgEwLwYDVR0fBCgwJjAkoCKgIIYeaHR0 cDovL3IxMy5jLmxlbmNyLm9yZy8xMjEuY3JsMIIBAwYKKwYBBAHWeQIEAgSB9ASB 8QDvAHYApELFBklgYVSPD9TqnPt6LSZFTYepfy/fRVn2J086hFQAAAGZXVTEhwAA BAMARzBFAiAVfJZ/XSqNq0sdf49o/8Mhs1uG9H/iPAHynYubtxfw4wIhAPiDa5S5 DoawcZlWePa+uKZRiIaZwlVVOigiZEfm+75VAHUAzPsPaoVxCWX+lZtTzumyfCLp hVwNl422qX5UwP5MDbAAAAGZXVTEmAAABAMARjBEAiAJTtUg1SkZlRsuvXiWbeon ehJiRiOvQBBjCrDhPk+EmAIgRy7+96Uq7sFF2iQqlDbBJTbfxqVxsLAKKsv/4mUQ 76gwDQYJKoZIhvcNAQELBQADggEBADwJpGkcEI9YQJBcBqJ1k4lkUBI4zdhjYYuv Z2kbXFRkl041ulyel48qZZW20T9myTL4dI/2kqLP4VSrz+vk4xTzyXtTCJZHDeke dzoQ7lplxTfZRwDVx19PkJIFPIShHyS/Ia0XTqHC8F81PmwXULRAgMYrBS3sKLXg aIyf00xq7W6s0uPd0XDn5CsmJgHzEcBZ0F423V42iedwgGNv6GnlgzKP3Q8fkf21 4KdRYBgyYBfi33jQFf5fuMuSTtFak++BYe/ZWVAoehlw0gLh5BBmBXtCFrVFZc+q uXXe4q5MVQmDRa0A+QtKbwkyZxIiwJ8Xi+eBTKQSscpdINy5bUs= -----END CERTIFICATE-----
nodesocket 11mo ago
Nice to know, though I wonder how many companies are really using all private images? I've certainly had a client running their own Harbor instance, but almost all others pulled from Docker Hub or Github (ghcr.io).
- redrove 11mo ago
  Pretty much all enterprises are using their own ECR/GCR/ACR.
- j-scott 11mo ago
  I work on the container registry team at my current company running a custom container registry service!
  - deleted 11mo ago
    [deleted]
  - hanikesn 11mo ago
    How does this require a whole team? Unless you're working at a hyperscaler
    - hshdhdhehd 11mo ago
      Maybe they work for docker
    - fragmede 11mo ago
      Please describe their system for us, including system throughput, the hardware they're on, networking constraints, and how many people are allowed to be needed to operate it.
    - j-scott 11mo ago
      Not a hyperscaler, but we’re multi-cloud and probably one to two steps down. My team’s service implements a number of performance and functionality improvements on top of your typical registry to support the company’s needs. I can’t say much more than that sadly.
nevon 11mo ago
Just to save someone 5 minutes of research, if you are using the EKS AMIs based on AL2023 or Bottlerocket, this is already done for you by pointing to an image on ECR. At least on Bottlerocket, I haven't checked AL2023, the image is baked into the AMI so you don't even need to pull it from ECR.
- Chilinot 11mo ago
  Thank you, was just about to task my team with figuring out how affected we are by this.
- cmckn 11mo ago
  We removed the image registry dependency on AL2023 as well. :) https://github.com/awslabs/amazon-eks-ami/pull/2000 https://github.com/awslabs/amazon-eks-ami/pull/2000
rcarmo 11mo ago
I believe this has been patched time and time again in on-premises variants like OpenShift. Curious to check if it’s there in small variants like microk8s, k3s, etc., as I’m considering moving a few offline services to Talos.
- anttiharju 11mo ago
  Talos' KubeSpan is backed by Sidero-hosted disovery service that cannot be self-hosted without a commercial license
  - rcarmo 11mo ago
    Hmmm. So I need to find something equivalent as a base OS. Maybe it's time to do Fedora Core.
    - anttiharju 11mo ago
      Flatcar Linux has looked decent
hexo 11mo ago
Easy. Dont use kubernetes. You'll thank me later.
preisschild 11mo ago
Yeah that pause image was really annoying when I was hosting a k8s cluster on Hetzner, since the `registry.k8s.io` -registry was blocking some Hetzner IPs, since its hosted on Google.
ralgozino 11mo ago
I went down this rabbit hole not so long ago too. There was a discussion open on containerd's GitHub on removing the dependency on the pause image but it has been closed as won't fix: https://github.com/containerd/containerd/issues/10505 https://github.com/containerd/containerd/issues/10505 Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration: https://github.com/kubernetes/kubeadm/issues/2020 https://github.com/kubernetes/kubeadm/issues/2020
thiht 11mo ago
This is insane, am I the only one being shocked to learn about this? This reeks of bad engineering, is there at least a plan to make this go away by embedding "pause" with the distribution?

97 points by meatmanek 11mo ago

cmckn 11mo ago

It bugs me that this implementation detail of containerd has leaked to such an extent. This should be part of the containerd distribution, and should not be pulled at runtime. Instead of just swapping out the registry, try baking it into your machine image.

chupasaurus 11mo ago
It's implementation of cri plugin. > This should be part of the containerd distribution containerd is not the only CRI runtime out there.
- cmckn 11mo ago
  > It's implementation of cri plugin. Right, that’s the point. A user of the CRI should not have to care about this implementation detail. > containerd is not the only CRI runtime out there. Any CRI that needs a pause executable should come with one.
  - chupasaurus 11mo ago
    > A user of the CRI should not have to care about this implementation detail. This detail is implemented by K8s, not a container runtime. User has to care about the source of it due to supply chain attacks, though. > Any CRI that needs a pause executable should come with one. Which would introduce a simultaneous update across different projects, which is a problem harder than a line of config.
spwa4 11mo ago
More general one would wish that Kubernetes had a few extra ways to get images, so you could grow on a scale from "minimal infrastructure" to "fully CI/CD". Starting with just sending the image in the RPC itself or even just on local disk (you figure out how to get it there), all the way up to registries with tightly controlled versioning.
- p_l 11mo ago
  It's possible to do that, as kubernetes only passes the image information to CRI. You can also setup a separate service to "push" images directly to your container runtime, someone even demoed one in Show HN post some time ago I think.
kawsper 11mo ago
Relying on an hosted image also caused some disruptions for Nomad (the scheduler from Hashicorp), because the default pause image was hosted at gcr.io which google killed, and it moved to registry.k8s.io. The nomad team made this configurable afterwards.
- verdverm 11mo ago
  That nomad was hit with this after years of notice and deprecation extension, seems a sign of serious maintenance issues
yencabulator 11mo ago
It's almost as if Docker (the company) wants it this way.

fred_is_fred 11mo ago

I've used k8s before a lot and at several companies. I am convinced that 99.9% of the people who use it should not be. But it's more fun than deploying VM images at least.

kachapopopow 11mo ago
I use k3s for my home and for dev envs I think it's completely fine especially when it comes to deployment documentation. I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot. Well... it's what I would have said until bitnami pulled the rug and pretty much ruined the entire ecosystem as now you don't have a way to pull something that you know is trusted with similar configuration and all from a single repository which makes deployments a pain in the ass. However, on the plus side I've just been creating my own every time I need one with the help of claude using bitnami as reference and honestly it doesn't take that much more time and keeping them up to date is relatively easy as well with ci automations.
- Imustaskforhelp 11mo ago
  > I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot. Thoughts on Tmux-resurrect[1] , it can even resurrect programs running inside of it as well. It feels like it can as such reduce complexity from something like k3s back to tmux. What are your thoughts on it? [1]:https://github.com/tmux-plugins/tmux-resurrect?tab=readme-ov-file https://github.com/tmux-plugins/tmux-resurrect?tab=readme-ov...
  - kachapopopow 11mo ago
    I had it break enough times to where I just don't bother.
    - Imustaskforhelp 11mo ago
      Well firstly I would love to know more about your workflow where it actually broke etc. because I feel like tmux-ressurect team could help or something for sure. I haven't used the tool itself so I am curious as I was thinking of a similar workflow as well sometime ago Now please answer the above questions but also I am going to assume that you are right about tmux-ressurect, even then there are other ways of doing the same thing as well. https://www.baeldung.com/linux/process-save-restore https://www.baeldung.com/linux/process-save-restore This mentions either Criu if you want a process to persist after a shutdown, or the shutdown utility's flags if you want to temporarily do it. I have played around with Criu and docker, docker can even use criu with things like docker checkpoint and I have played with that as well (I used it to shutdown mid compression of a large file and recontinue compression exactly from where I left) What are your thoughts on using criu+docker/criu + termux, I think that it itself might be an easier thing than k3s for your workflow. Plus, I have seen some people mention vps where they are running the processes for 300 days or even more without a single shutdown iirc and I feel like modern VPS providers are insanely good at uptime, even more so than sometimes cloud providers.

nneonneo 11mo ago

O/T, but I'm getting a cert error on this page - wonder if it's just me or if this site is just serving a weird cert. Looks like it's signed by some Fortinet appliance - maybe I'm getting MITMed? Would be kind of exciting/frightening if so. EDIT: I loaded the page from a cloud box, and wow, I'm getting MITMed! Seems to only be for this site, wonder if it's some kind of sensitivity to the .family TLD.

gregoryl 11mo ago
Ooft. If it helps, this is the PEM I'm getting. LetEncrypt signed. -----BEGIN CERTIFICATE----- MIIFAjCCA+qgAwIBAgISBZR6PR4jNhx4fBFvqKwzJWx4MA0GCSqGSIb3DQEBCwUA MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD EwNSMTMwHhcNMjUwOTE4MTM1OTEwWhcNMjUxMjE3MTM1OTA5WjAeMRwwGgYDVQQD ExNreWxlLmNhc2NhZGUuZmFtaWx5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA55JknkVzyq5QGaRXn2TAzaOGYTHUVxl89lGOFgEEaWEvH5pcZL7xkqfv Edee7l5MeRKuK1zJ+ISPQQaEjGTk51y1aXXfOKs62NiNy6QQUbzQ+euecqrKsJVN l3PC3EYlEGibKI1gZ2x/ht8WJU9o4KiswCLqHrY7nC7BeEByv/ehiYyRTTxAXJsr 2X4LgPX6MQ1Iu10S2Bp9jnOlEV7n4RCTPFeWtfQ0CdXH45ykuwL/zrTaD111oNQE BQPNq7Ig7OihLZcJQo8TMJ3FUgzDI9z6kMy7QHNR1I8uODVUohQCO6E7A29x8nRJ UBV5DN1as3aHYFJ4FbX9s2tuLwCTiwIDAQABo4ICIzCCAh8wDgYDVR0PAQH/BAQD AgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAA MB0GA1UdDgQWBBTXwJ21Mudr9rplbA970jxJk44pEDAfBgNVHSMEGDAWgBTnq58P LDOgU9NeT3jIsoQOO9aSMzAzBggrBgEFBQcBAQQnMCUwIwYIKwYBBQUHMAKGF2h0 dHA6Ly9yMTMuaS5sZW5jci5vcmcvMB4GA1UdEQQXMBWCE2t5bGUuY2FzY2FkZS5m YW1pbHkwEwYDVR0gBAwwCjAIBgZngQwBAgEwLwYDVR0fBCgwJjAkoCKgIIYeaHR0 cDovL3IxMy5jLmxlbmNyLm9yZy8xMjEuY3JsMIIBAwYKKwYBBAHWeQIEAgSB9ASB 8QDvAHYApELFBklgYVSPD9TqnPt6LSZFTYepfy/fRVn2J086hFQAAAGZXVTEhwAA BAMARzBFAiAVfJZ/XSqNq0sdf49o/8Mhs1uG9H/iPAHynYubtxfw4wIhAPiDa5S5 DoawcZlWePa+uKZRiIaZwlVVOigiZEfm+75VAHUAzPsPaoVxCWX+lZtTzumyfCLp hVwNl422qX5UwP5MDbAAAAGZXVTEmAAABAMARjBEAiAJTtUg1SkZlRsuvXiWbeon ehJiRiOvQBBjCrDhPk+EmAIgRy7+96Uq7sFF2iQqlDbBJTbfxqVxsLAKKsv/4mUQ 76gwDQYJKoZIhvcNAQELBQADggEBADwJpGkcEI9YQJBcBqJ1k4lkUBI4zdhjYYuv Z2kbXFRkl041ulyel48qZZW20T9myTL4dI/2kqLP4VSrz+vk4xTzyXtTCJZHDeke dzoQ7lplxTfZRwDVx19PkJIFPIShHyS/Ia0XTqHC8F81PmwXULRAgMYrBS3sKLXg aIyf00xq7W6s0uPd0XDn5CsmJgHzEcBZ0F423V42iedwgGNv6GnlgzKP3Q8fkf21 4KdRYBgyYBfi33jQFf5fuMuSTtFak++BYe/ZWVAoehlw0gLh5BBmBXtCFrVFZc+q uXXe4q5MVQmDRa0A+QtKbwkyZxIiwJ8Xi+eBTKQSscpdINy5bUs= -----END CERTIFICATE-----

nodesocket 11mo ago

Nice to know, though I wonder how many companies are really using all private images? I've certainly had a client running their own Harbor instance, but almost all others pulled from Docker Hub or Github (ghcr.io).

redrove 11mo ago
Pretty much all enterprises are using their own ECR/GCR/ACR.
j-scott 11mo ago
I work on the container registry team at my current company running a custom container registry service!
- deleted 11mo ago
  [deleted]
- hanikesn 11mo ago
  How does this require a whole team? Unless you're working at a hyperscaler
  - hshdhdhehd 11mo ago
    Maybe they work for docker
  - fragmede 11mo ago
    Please describe their system for us, including system throughput, the hardware they're on, networking constraints, and how many people are allowed to be needed to operate it.
  - j-scott 11mo ago
    Not a hyperscaler, but we’re multi-cloud and probably one to two steps down. My team’s service implements a number of performance and functionality improvements on top of your typical registry to support the company’s needs. I can’t say much more than that sadly.

nevon 11mo ago

Just to save someone 5 minutes of research, if you are using the EKS AMIs based on AL2023 or Bottlerocket, this is already done for you by pointing to an image on ECR. At least on Bottlerocket, I haven't checked AL2023, the image is baked into the AMI so you don't even need to pull it from ECR.

Chilinot 11mo ago
Thank you, was just about to task my team with figuring out how affected we are by this.
cmckn 11mo ago
We removed the image registry dependency on AL2023 as well. :) https://github.com/awslabs/amazon-eks-ami/pull/2000 https://github.com/awslabs/amazon-eks-ami/pull/2000

rcarmo 11mo ago

I believe this has been patched time and time again in on-premises variants like OpenShift. Curious to check if it’s there in small variants like microk8s, k3s, etc., as I’m considering moving a few offline services to Talos.

anttiharju 11mo ago
Talos' KubeSpan is backed by Sidero-hosted disovery service that cannot be self-hosted without a commercial license
- rcarmo 11mo ago
  Hmmm. So I need to find something equivalent as a base OS. Maybe it's time to do Fedora Core.
  - anttiharju 11mo ago
    Flatcar Linux has looked decent

hexo 11mo ago

Easy. Dont use kubernetes. You'll thank me later.

preisschild 11mo ago

Yeah that pause image was really annoying when I was hosting a k8s cluster on Hetzner, since the `registry.k8s.io` -registry was blocking some Hetzner IPs, since its hosted on Google.

ralgozino 11mo ago

I went down this rabbit hole not so long ago too. There was a discussion open on containerd's GitHub on removing the dependency on the pause image but it has been closed as won't fix: https://github.com/containerd/containerd/issues/10505 https://github.com/containerd/containerd/issues/10505 Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration: https://github.com/kubernetes/kubeadm/issues/2020 https://github.com/kubernetes/kubeadm/issues/2020

thiht 11mo ago

This is insane, am I the only one being shocked to learn about this? This reeks of bad engineering, is there at least a plan to make this go away by embedding "pause" with the distribution?