3 ms·

I mean... one certainly could do this. And then as soon as anything breaks with that container, the person trying to fix it will curse the day you were born. Pe

by bigstrat2003 2y ago

I mean... one certainly could do this. And then as soon as anything breaks with that container, the person trying to fix it will curse the day you were born. People already go way too far with stripping container images, and it's awful for troubleshooting. The last thing we need as an industry is taking that silly trend even bother.

wmf 2y ago
If you're going to use minimal container images you definitely need to know how to inject tools into the container.
- mschuster91 2y ago
  Yeah and it's a true pain. Double that if your container security policy requires running in fully rootless mode (i.e. not just the usual CAP_ALL drop but also container uid != 0), quadruple if you're in a Kubernetes environment, and even more if you can't access the nodes themselves directly (AWS EKS).
- tremon 2y ago
  I'd say you need to know how to properly instrument your containers so that you'll never need to get inside. If your team is on the level of "we need tools inside the container for troubleshooting", then don't use minimized containers like these.
jpalomaki 2y ago
It's about security and reducing the probability of getting CVEs for your running containers. Less software you bundle there, less security issues you will have. And this in turn means less work on maintenance as you don't need to update, test and deploy new versions of images so often.
bitwize 2y ago
When the cow gets sick, you shoot it.
- mschuster91 2y ago
  You still want to know why the cow got sick, in order to prevent all other cows in the stable from falling sick as well. And ideally, you want to keep the cow alive, at least if you fed it artificially.
  - bitwize 2y ago
    You're not looking at this from a DevOps perspective. In real-world cloud environments, logging into the container and expecting to have a complete Unix environment to poke around in just isn't an option. You might not even have the required permissions to do so, as a matter of corporate policy. You need to build observability directly into the application -- or you haven't built a complete cloud application.
    - Dylan16807 2y ago
      > You need to build observability directly into the application This is not objectively better than external observability. If the "DevOps perspective" requires it, then it's a pointlessly inflexible perspective. > just isn't an option You don't get to blame the system when you made the system. The option is there.
    - mschuster91 2y ago
      That's the key thing why DevOps is frankly a pile of bullshit and why so many companies ridiculously overpay on cloud infrastructure. When you got clueless morons who were in a 6 week nodejs bootcamp now "developing" containers, with no one even having the permissions to troubleshoot crashes - that may very well originate from someone trying to get remote code execution - it is absolutely no surprise that hacks, ransomware, infrastructure so brittle that it only doesn't fail because EBS / EKS keeps restarting the pile of dung every minute and infrastructure pieced together from 40 AWS services where one classic EC2 VM would be sufficient are plentiful.
hifromwork 2y ago
Developers having shell access to production container is not the best practice, to say the least. For debugging locally, it's very easy to nsenter your container.
sbarre 2y ago
We use Google's distroless containers for production, but we also have alternate Dockerfiles available in our CI/CD for building a more traditional container that can be logged into if there are application issues to debug. It's not an exact copy of prod, but so far (a few years, and thousands of containers) it hasn't really been an issue as far as I've heard. We don't do anything fancy though. Node, Go, Java and Python apps..
__MatrixMan__ 2y ago
I think the point is that you should have a plan (which may or may not require debug tooling in the image) and you should include what you need for that plan and nothing more. Finding something useful that the distro left lying around but that wasn't part of the plan might feel like a win here and there, it's just as likely to be a win for your adversary as it is for your friend.

wmf 2y ago

If you're going to use minimal container images you definitely need to know how to inject tools into the container.

mschuster91 2y ago
Yeah and it's a true pain. Double that if your container security policy requires running in fully rootless mode (i.e. not just the usual CAP_ALL drop but also container uid != 0), quadruple if you're in a Kubernetes environment, and even more if you can't access the nodes themselves directly (AWS EKS).
tremon 2y ago
I'd say you need to know how to properly instrument your containers so that you'll never need to get inside. If your team is on the level of "we need tools inside the container for troubleshooting", then don't use minimized containers like these.

jpalomaki 2y ago

It's about security and reducing the probability of getting CVEs for your running containers. Less software you bundle there, less security issues you will have. And this in turn means less work on maintenance as you don't need to update, test and deploy new versions of images so often.

bitwize 2y ago

When the cow gets sick, you shoot it.

mschuster91 2y ago
You still want to know why the cow got sick, in order to prevent all other cows in the stable from falling sick as well. And ideally, you want to keep the cow alive, at least if you fed it artificially.
- bitwize 2y ago
  You're not looking at this from a DevOps perspective. In real-world cloud environments, logging into the container and expecting to have a complete Unix environment to poke around in just isn't an option. You might not even have the required permissions to do so, as a matter of corporate policy. You need to build observability directly into the application -- or you haven't built a complete cloud application.
  - Dylan16807 2y ago
    > You need to build observability directly into the application This is not objectively better than external observability. If the "DevOps perspective" requires it, then it's a pointlessly inflexible perspective. > just isn't an option You don't get to blame the system when you made the system. The option is there.
  - mschuster91 2y ago
    That's the key thing why DevOps is frankly a pile of bullshit and why so many companies ridiculously overpay on cloud infrastructure. When you got clueless morons who were in a 6 week nodejs bootcamp now "developing" containers, with no one even having the permissions to troubleshoot crashes - that may very well originate from someone trying to get remote code execution - it is absolutely no surprise that hacks, ransomware, infrastructure so brittle that it only doesn't fail because EBS / EKS keeps restarting the pile of dung every minute and infrastructure pieced together from 40 AWS services where one classic EC2 VM would be sufficient are plentiful.

hifromwork 2y ago

Developers having shell access to production container is not the best practice, to say the least. For debugging locally, it's very easy to nsenter your container.

sbarre 2y ago

We use Google's distroless containers for production, but we also have alternate Dockerfiles available in our CI/CD for building a more traditional container that can be logged into if there are application issues to debug. It's not an exact copy of prod, but so far (a few years, and thousands of containers) it hasn't really been an issue as far as I've heard. We don't do anything fancy though. Node, Go, Java and Python apps..

__MatrixMan__ 2y ago

I think the point is that you should have a plan (which may or may not require debug tooling in the image) and you should include what you need for that plan and nothing more. Finding something useful that the distro left lying around but that wasn't part of the plan might feel like a win here and there, it's just as likely to be a win for your adversary as it is for your friend.