4 ms·

I prefer to run my service containers without a shell or much of an operating system. YMMV but in this corporate world where every container is scanned for ever

by oooyay 3y ago

I prefer to run my service containers without a shell or much of an operating system. YMMV but in this corporate world where every container is scanned for every CVE and some binaries are deep inspected my list of vulnerabilities on a more full fledged container or VM can become a task that spirals out of control. The context being that many (maybe even most) of those CVEs probably don't pertain to how I use the software or package and that it's far easier to patch than file for an exemption. Thus, I spend the investment up front in getting log streaming working, my logs are concise, I implement application monitoring, and I demand host monitoring from the platforms I use. If I check all of those boxes, I generally don't have anything I need to do with a shell.

ericbarrett 3y ago
I fully agree with you about vuln scans, but as a counterpoint there have been dozens of times when I've saved hours of debugging with a well-applied strace or tcpdump. Logging and monitoring are great and necessary, but they'll only catch things you thought of ahead of time; using them to debug something ongoing is basically printf() debugging where compile time = the full length of your CI/CD pipeline.
- oooyay 3y ago
  Yeah, that's definitely a valid take depending on your setup. If I have those kinds of problems with a container then I generally jump into the underlying VM or metal to use those tools, but that also implies a lot of knowledge around how a host system incorporates container networking, which arguably makes hard troubleshooting even harder. On headless systems they usually come with some sort of privileged "admin" container, so the setup is the same. Second to that is that I have dev stages that are built with containers that do have those tools, and generally if I run into those kinds of problems I see them in dev first.
- lifty 3y ago
  Depending on how you run your containers, you should be able to run a debug container in the same namespace as your target container. That way you can keep your images lean and bundle all the debugging tools in a different image, which you run only when you need to.

by oooyay 3y ago

ericbarrett 3y ago

I fully agree with you about vuln scans, but as a counterpoint there have been dozens of times when I've saved hours of debugging with a well-applied strace or tcpdump. Logging and monitoring are great and necessary, but they'll only catch things you thought of ahead of time; using them to debug something ongoing is basically printf() debugging where compile time = the full length of your CI/CD pipeline.

oooyay 3y ago
Yeah, that's definitely a valid take depending on your setup. If I have those kinds of problems with a container then I generally jump into the underlying VM or metal to use those tools, but that also implies a lot of knowledge around how a host system incorporates container networking, which arguably makes hard troubleshooting even harder. On headless systems they usually come with some sort of privileged "admin" container, so the setup is the same. Second to that is that I have dev stages that are built with containers that do have those tools, and generally if I run into those kinds of problems I see them in dev first.
lifty 3y ago
Depending on how you run your containers, you should be able to run a debug container in the same namespace as your target container. That way you can keep your images lean and bundle all the debugging tools in a different image, which you run only when you need to.