9 ms·

Container security best practices: Ultimate guide

221 points by knoxa2511 5y ago

gui77aume 5y ago
I'm always a bit confused about the CPU limit (for the pod), some guides (and tools) advice to always set one, but this one [0] doesn't. Ops people I worked with almost always want to lower that limit and I have to insist for raising it (no way they disable it). Is there an ultimate best practice for that? [0] https://learnk8s.io/production-best-practices https://learnk8s.io/production-best-practices
- jeffbee 5y ago
  CPU limits are harmful if they strand resources that could have been applied. I usually skip them for batch deployments, use them for latency-sensitive services. Doesn’t seem like a security topic though.
  - dilyevsky 5y ago
    They are actually even worse for latency sensitive workload because cfs with 100ms default period will cause crap tail latency (especially for multithreaded processes such as most go programs)
    - jeffbee 5y ago
      Yes, but I put limits on LS workloads because I expect them to have a capacity plan, stick to it, and not abusively starve out batch workloads.
      - dilyevsky 5y ago
        I think this is backwards. How are you planning on “sticking to it” when you’re serving unpredictable user traffic? If requests are set appropriately everywhere then it won’t really starve batch as kernel would just scale everything to their respective cpu.shares when cpu is fully saturated. This would allow you to weather spiky load with minimum latency impact and minimize spend
        jeffbee 5y ago
        It's weird that apparently you are a borg user from google, according to other discussions we have exchanged, but you question the value of hard-capping for latency-sensitive processes.
        dilyevsky 5y ago
        Borg sre even ;) (former) and yes i do question them. For one borg aint using 100ms cfs period and it wasn’t even standard cfs if i recall so yes i do question that outside of limited borg usecase
    - gui77aume 5y ago
      Interesting. It's my impression too. I understand that CPU limit will artificially throttle CPU, when not necessarily needed, wasting CPU cycles I could use. (Java programs in my case but I imagine it's comparable to Go ones) Do you recommend to disable CPU limit? In the general case.
      - dilyevsky 5y ago
        We don’t set them anywhere in prod and generally didn’t have any issues. We always set cpu requests and alert if those are exceeded for prolonged periods and always set memory req=limit
dpedu 5y ago
Perhaps I overlooked it, but it seems strange there's nothing about making containers immutable and read-only. This is a powerful tool IMO. https://cloud.google.com/architecture/best-practices-for-operating-containers#ensure_that_your_containers_are_stateless_and_immutable https://cloud.google.com/architecture/best-practices-for-ope...
- kjs3 5y ago
  I would assume that's because that mitigation isn't what sysdig does.
- raesene9 5y ago
  Yep I've always had read only root filesystems down as a good control and one that's often not too tough to implement. Another favourite of mine would be using multi-stage builds and minimal base images in production (FROM Scratch, where possible). having limited or no tooling in the running container makes an attackers life trickier for sure.
  - travisd 5y ago
    The distroless static images are pretty good. It’s essentially scratch plus certificate authority roots of trust.
- capableweb 5y ago
  It seems that Sysdig doesn't have a blog post about making containers immutable and read-only, nor offer a service that enables that, so probably not worth mentioning for them.
  - capitangolo 5y ago
    Hmm, that seems like a weird miss from my side. i.e. We covered this across several articles like this one about tags: https://sysdig.com/blog/toctou-tag-mutability/ https://sysdig.com/blog/toctou-tag-mutability/ This other one about file integrity monitoring (Disclaimer: A rather commercial one) https://sysdig.com/blog/file-integrity-monitoring/ https://sysdig.com/blog/file-integrity-monitoring/ And I recall others more explicit on the read-only part, but I’m away from my laptop now. Edit: Found it (point 1.3 in https://sysdig.com/blog/dockerfile-best-practices/ https://sysdig.com/blog/dockerfile-best-practices/ ) Thanks for pointing it out. Definitely it should be more explicit.
OrvalWintermute 5y ago
Unfortunately, this reads like a 100 foot marketing document for Sysdig, not actual container security best practices. If you want to look at actual container security best practices, check out CIS [1] & DISA [2], and NSA [3], with some theory at NIST [4], as well as the documentation from your preferred cloud vendors, be it AWS, Azure, GCP, or other, as well as the specific container security practices. [1] https://www.cisecurity.org/ https://www.cisecurity.org/ [2] https://public.cyber.mil/stigs/downloads/ https://public.cyber.mil/stigs/downloads/ [3] https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/0/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/0/CTR... [4] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S...
- adamgordonbell 5y ago
  Container security should start with image security. Instead of runtime security stuff, you can statically analysis images before they are running somewhere and find what known exploits might exist in them. This is also easier to scale. Nist gets it right by starting there.
  - thinkharderdev 5y ago
    One of the hardest things to get any dev organization to start taking seriously is supply chain security. That first scan which lights up like a Christmas Tree is always such a daunting obstacle to get over. It's a shame because it is probably the highest value SDLC practice that many are not doing.
    - dijit 5y ago
      Yet, the base Debian image _does_ light up like a Christmas tree when you run a snyk scan. Mostly with incorrect issues (version number causes a flag but the fix is backported) or are considered low priority and thus WONTFIX by upstream. If you’re writing software against, say, dotnet3 (which has a docker image based on Debian) then you’re basically noised out.
danjc 5y ago
Curious to know whether anyone here can speak to how much safer Hyper V isolation[1] is than process isolation and whether it negates some of the concerns in the article. 1. https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container https://docs.microsoft.com/en-us/virtualization/windowsconta...
- invokestatic 5y ago
  Virtualization-backed container technologies are a definite security improvement over traditional containers (including Hyper-V), but most of the measures in this article are still important. Remember, security-in-depth. Virtualization mainly protects against zero-day kernel exploits, limiting the "blast radius" to a single container. You still need to monitor dependencies, isolation, signing, scanning, and have a vulnerability management program, among other things.
- raesene9 5y ago
  Microsoft's guidance (last I looked) was that Windows containers (e.g. the non Hyper-V ones) were not a security boundary, only Hyper-V based Windows containers should be considered to provide isolation. That has the slight clash with the fact that Hyper-V containers are not currently supported under Kubernetes (https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/ https://kubernetes.io/docs/setup/production-environment/wind...):) For more depth on the challenges of securing Process isolation containers with Windows https://googleprojectzero.blogspot.com/2021/04/who-contains-containers.html https://googleprojectzero.blogspot.com/2021/04/who-contains-... is a great read.
  - OrvalWintermute 5y ago
    To add to above: Virtualization & Containerization security depends a great deal on the security of the underlying platform. Hyper-V can be used on endpoints [1], similar to VMware Workstation. It can also be installed as a role on top of Windows Server [2], and, used as bootable OS of its own[3] (likely deprecated in the future, so no hyper-v server past server 2019). Related to this is the type of Windows server install, as it touches on attack surface also [4], but I believe there are constraints for the very small installs. This matters because attack surface is likely to be, from smallest to largest: hyper-v server < Windows Server < Windows Endpoint [1] https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/ https://docs.microsoft.com/en-us/virtualization/hyper-v-on-w... [2] https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/get-started/install-the-hyper-v-role-on-windows-server https://docs.microsoft.com/en-us/windows-server/virtualizati... [3] https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2019 https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-... [4] https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/mt588479(v=vs.85) https://docs.microsoft.com/en-us/previous-versions/windows/d...
eatonphil 5y ago
The thing that kills me about all of this is how hard it is to do it right. I wish there were a dumbed down version of containers and orchestrators for people trying to do basic multi-tenant compute in a SaaS and don't care a ton about the best performance. Would I be generally ok if I use gvisor to give a shell environment to customers and just keep the host up to date? Or is using containers just relatively pointless for multitenant compute in a SaaS compared to giving customers virtual machines? If you can't imagine the kind of SaaS I'm talking about, think something along the lines of Github's new online IDE, CodeSpaces.
- raesene9 5y ago
  Multi-Tenant Kubernetes is straight up difficult to do well, especially where you're talking hard multi-tenancy for external customers. There was a good report that covered a lot of the risks and mitigations here https://raw.githubusercontent.com/salesforce/kubernetes-control-plane-assessment/master/Atredis%20Partners-Salesforce-Kubernetes%20Control%20Plane%20Vulnerability%20Assessment-v1.1.pdf https://raw.githubusercontent.com/salesforce/kubernetes-cont... But even then that had limited scope and didn't cover things like networking.
- simonebrunozzi 5y ago
  Reinventing the wheel, all the time. Early VMware (with VMs) had a much better sense of product than Google had with K8S.
- paxys 5y ago
  I think "dumbed down" and "multi-tenant compute" aren't compatible. No company needs to do multi-tenant compute by default. If you do, you are in the cloud hosting/infrastructure business (whether you like it or not) and should be expected to have the knowledge necessary to run such an operation.
  - eatonphil 5y ago
    While that's a common sentiment in some topics in tech; I think the general intent, if not actual result, of progress in tech is to make things faster, more secure and easier. So forgive me for asking. :)
w7 5y ago
My home k8s cluster is now "locked down" using micro-vms (kata-containers[0]), pod level firewalling (cilium[1]), permission-limited container users, mostly immutable environments, and distroless[2] base images (not even a shell is inside!). Given how quickly I rolled this out; the tools to enhance cluster environment security seem more accessible now than my previous research a few years ago. I know it's not exactly a production setup, but I really do feel that it's atleast the most secure runtime environment I've ever had accessible at home. Probably more so than my desktops, which you could argue undermines most of my effort, but I like to think I'm pretty careful. In the beginning I was very skeptical, but being able to just build a docker/OCI image and then manage its relationships with other services with "one pane of glass" that I can commit to git is so much simpler to me than my previous workflows. My previous setup involved messing with a bunch of tools like packer, cloud-init, terraform, ansible, libvirt, whatever firewall frontend was on the OS, and occasionally sshing in for anything not covered. And now I can feel even more comfortable than when I was running a traditional VM+VLAN per exposed service. [0] https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers [1] https://github.com/cilium/cilium https://github.com/cilium/cilium [2] https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless
- OrvalWintermute 5y ago
  I read alot on home setups and yours seems to balance both high security and maintainability very well. Care to share about the details of the security services side of your stack too? Cheers
  - w7 5y ago
    Sure, hopefully I understand what you mean. For network observability I'm using Cilium's Hubble, which I will soon figure out how to get into a greylog setup or something. For container image vulnerability interrogation I'm running Harbor with Trivy enabled, initial motivation was to have an effective pull through cache for multiple registries because I got rate limited by AWS ECR (due to a misconfigured CI pipeline, oops), but it ended up killing two birds with 1 stone. Next on my list is writing an admission controller to modify supported registry targets to match my pull through cache configuration. Is there something more specific you wanted?
hsbauauvhabzb 5y ago
Calling your guide the ‘ultimate guide’ is disingenuous marketing. No single guide can cover all security concepts in all contexts. Every time I see that sorta wording I just assume the writer doesn’t actually know what they’re talking about
- hsbauauvhabzb 5y ago
  Continued: and given the writer seems to be all about tools the article fails to highlight that static (and automated dynamic) tools are limited in their ability to detect some classes of vulnerabilities and need to be backed with experience manual testing. This almost feels like it’s been written by a devops engineer who has a vague understanding about containerisation doesn’t have a clue about real and practical mechanisms to secure applications and services hosted inside containers. I’m not saying the article is totally bad, but calling it an ‘Ultimate Guide’ makes the author a charlatan.
_8j50 5y ago
Production host root fs should be mounted ro. Check out Linux IMA and how to only allow specific executables by hash. Centrally forward container logs. Use a VCS for container/workload templates and routinely audit for misconfig. Sysdig/falco and related tools are nice, but containers and their prod hosts are easier to harden

221 points by knoxa2511 5y ago

I'm always a bit confused about the CPU limit (for the pod), some guides (and tools) advice to always set one, but this one [0] doesn't. Ops people I worked with almost always want to lower that limit and I have to insist for raising it (no way they disable it). Is there an ultimate best practice for that? [0] https://learnk8s.io/production-best-practices https://learnk8s.io/production-best-practices

jeffbee 5y ago
CPU limits are harmful if they strand resources that could have been applied. I usually skip them for batch deployments, use them for latency-sensitive services. Doesn’t seem like a security topic though.
- dilyevsky 5y ago
  They are actually even worse for latency sensitive workload because cfs with 100ms default period will cause crap tail latency (especially for multithreaded processes such as most go programs)
  - jeffbee 5y ago
    Yes, but I put limits on LS workloads because I expect them to have a capacity plan, stick to it, and not abusively starve out batch workloads.
    - dilyevsky 5y ago
      I think this is backwards. How are you planning on “sticking to it” when you’re serving unpredictable user traffic? If requests are set appropriately everywhere then it won’t really starve batch as kernel would just scale everything to their respective cpu.shares when cpu is fully saturated. This would allow you to weather spiky load with minimum latency impact and minimize spend
      - jeffbee 5y ago
        It's weird that apparently you are a borg user from google, according to other discussions we have exchanged, but you question the value of hard-capping for latency-sensitive processes.
        dilyevsky 5y ago
        Borg sre even ;) (former) and yes i do question them. For one borg aint using 100ms cfs period and it wasn’t even standard cfs if i recall so yes i do question that outside of limited borg usecase
  - gui77aume 5y ago
    Interesting. It's my impression too. I understand that CPU limit will artificially throttle CPU, when not necessarily needed, wasting CPU cycles I could use. (Java programs in my case but I imagine it's comparable to Go ones) Do you recommend to disable CPU limit? In the general case.
    - dilyevsky 5y ago
      We don’t set them anywhere in prod and generally didn’t have any issues. We always set cpu requests and alert if those are exceeded for prolonged periods and always set memory req=limit

dpedu 5y ago

Perhaps I overlooked it, but it seems strange there's nothing about making containers immutable and read-only. This is a powerful tool IMO. https://cloud.google.com/architecture/best-practices-for-operating-containers#ensure_that_your_containers_are_stateless_and_immutable https://cloud.google.com/architecture/best-practices-for-ope...

kjs3 5y ago
I would assume that's because that mitigation isn't what sysdig does.
raesene9 5y ago
Yep I've always had read only root filesystems down as a good control and one that's often not too tough to implement. Another favourite of mine would be using multi-stage builds and minimal base images in production (FROM Scratch, where possible). having limited or no tooling in the running container makes an attackers life trickier for sure.
- travisd 5y ago
  The distroless static images are pretty good. It’s essentially scratch plus certificate authority roots of trust.
capableweb 5y ago
It seems that Sysdig doesn't have a blog post about making containers immutable and read-only, nor offer a service that enables that, so probably not worth mentioning for them.
- capitangolo 5y ago
  Hmm, that seems like a weird miss from my side. i.e. We covered this across several articles like this one about tags: https://sysdig.com/blog/toctou-tag-mutability/ https://sysdig.com/blog/toctou-tag-mutability/ This other one about file integrity monitoring (Disclaimer: A rather commercial one) https://sysdig.com/blog/file-integrity-monitoring/ https://sysdig.com/blog/file-integrity-monitoring/ And I recall others more explicit on the read-only part, but I’m away from my laptop now. Edit: Found it (point 1.3 in https://sysdig.com/blog/dockerfile-best-practices/ https://sysdig.com/blog/dockerfile-best-practices/ ) Thanks for pointing it out. Definitely it should be more explicit.

OrvalWintermute 5y ago

Unfortunately, this reads like a 100 foot marketing document for Sysdig, not actual container security best practices. If you want to look at actual container security best practices, check out CIS [1] & DISA [2], and NSA [3], with some theory at NIST [4], as well as the documentation from your preferred cloud vendors, be it AWS, Azure, GCP, or other, as well as the specific container security practices. [1] https://www.cisecurity.org/ https://www.cisecurity.org/ [2] https://public.cyber.mil/stigs/downloads/ https://public.cyber.mil/stigs/downloads/ [3] https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/0/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/0/CTR... [4] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S...

adamgordonbell 5y ago
Container security should start with image security. Instead of runtime security stuff, you can statically analysis images before they are running somewhere and find what known exploits might exist in them. This is also easier to scale. Nist gets it right by starting there.
- thinkharderdev 5y ago
  One of the hardest things to get any dev organization to start taking seriously is supply chain security. That first scan which lights up like a Christmas Tree is always such a daunting obstacle to get over. It's a shame because it is probably the highest value SDLC practice that many are not doing.
  - dijit 5y ago
    Yet, the base Debian image _does_ light up like a Christmas tree when you run a snyk scan. Mostly with incorrect issues (version number causes a flag but the fix is backported) or are considered low priority and thus WONTFIX by upstream. If you’re writing software against, say, dotnet3 (which has a docker image based on Debian) then you’re basically noised out.

danjc 5y ago

Curious to know whether anyone here can speak to how much safer Hyper V isolation[1] is than process isolation and whether it negates some of the concerns in the article. 1. https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container https://docs.microsoft.com/en-us/virtualization/windowsconta...

invokestatic 5y ago
Virtualization-backed container technologies are a definite security improvement over traditional containers (including Hyper-V), but most of the measures in this article are still important. Remember, security-in-depth. Virtualization mainly protects against zero-day kernel exploits, limiting the "blast radius" to a single container. You still need to monitor dependencies, isolation, signing, scanning, and have a vulnerability management program, among other things.
raesene9 5y ago
Microsoft's guidance (last I looked) was that Windows containers (e.g. the non Hyper-V ones) were not a security boundary, only Hyper-V based Windows containers should be considered to provide isolation. That has the slight clash with the fact that Hyper-V containers are not currently supported under Kubernetes (https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/ https://kubernetes.io/docs/setup/production-environment/wind...):) For more depth on the challenges of securing Process isolation containers with Windows https://googleprojectzero.blogspot.com/2021/04/who-contains-containers.html https://googleprojectzero.blogspot.com/2021/04/who-contains-... is a great read.
- OrvalWintermute 5y ago
  To add to above: Virtualization & Containerization security depends a great deal on the security of the underlying platform. Hyper-V can be used on endpoints [1], similar to VMware Workstation. It can also be installed as a role on top of Windows Server [2], and, used as bootable OS of its own[3] (likely deprecated in the future, so no hyper-v server past server 2019). Related to this is the type of Windows server install, as it touches on attack surface also [4], but I believe there are constraints for the very small installs. This matters because attack surface is likely to be, from smallest to largest: hyper-v server < Windows Server < Windows Endpoint [1] https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/ https://docs.microsoft.com/en-us/virtualization/hyper-v-on-w... [2] https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/get-started/install-the-hyper-v-role-on-windows-server https://docs.microsoft.com/en-us/windows-server/virtualizati... [3] https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2019 https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-... [4] https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/mt588479(v=vs.85) https://docs.microsoft.com/en-us/previous-versions/windows/d...

eatonphil 5y ago

The thing that kills me about all of this is how hard it is to do it right. I wish there were a dumbed down version of containers and orchestrators for people trying to do basic multi-tenant compute in a SaaS and don't care a ton about the best performance. Would I be generally ok if I use gvisor to give a shell environment to customers and just keep the host up to date? Or is using containers just relatively pointless for multitenant compute in a SaaS compared to giving customers virtual machines? If you can't imagine the kind of SaaS I'm talking about, think something along the lines of Github's new online IDE, CodeSpaces.

raesene9 5y ago
Multi-Tenant Kubernetes is straight up difficult to do well, especially where you're talking hard multi-tenancy for external customers. There was a good report that covered a lot of the risks and mitigations here https://raw.githubusercontent.com/salesforce/kubernetes-control-plane-assessment/master/Atredis%20Partners-Salesforce-Kubernetes%20Control%20Plane%20Vulnerability%20Assessment-v1.1.pdf https://raw.githubusercontent.com/salesforce/kubernetes-cont... But even then that had limited scope and didn't cover things like networking.
simonebrunozzi 5y ago
Reinventing the wheel, all the time. Early VMware (with VMs) had a much better sense of product than Google had with K8S.
paxys 5y ago
I think "dumbed down" and "multi-tenant compute" aren't compatible. No company needs to do multi-tenant compute by default. If you do, you are in the cloud hosting/infrastructure business (whether you like it or not) and should be expected to have the knowledge necessary to run such an operation.
- eatonphil 5y ago
  While that's a common sentiment in some topics in tech; I think the general intent, if not actual result, of progress in tech is to make things faster, more secure and easier. So forgive me for asking. :)

w7 5y ago

My home k8s cluster is now "locked down" using micro-vms (kata-containers[0]), pod level firewalling (cilium[1]), permission-limited container users, mostly immutable environments, and distroless[2] base images (not even a shell is inside!). Given how quickly I rolled this out; the tools to enhance cluster environment security seem more accessible now than my previous research a few years ago. I know it's not exactly a production setup, but I really do feel that it's atleast the most secure runtime environment I've ever had accessible at home. Probably more so than my desktops, which you could argue undermines most of my effort, but I like to think I'm pretty careful. In the beginning I was very skeptical, but being able to just build a docker/OCI image and then manage its relationships with other services with "one pane of glass" that I can commit to git is so much simpler to me than my previous workflows. My previous setup involved messing with a bunch of tools like packer, cloud-init, terraform, ansible, libvirt, whatever firewall frontend was on the OS, and occasionally sshing in for anything not covered. And now I can feel even more comfortable than when I was running a traditional VM+VLAN per exposed service. [0] https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers [1] https://github.com/cilium/cilium https://github.com/cilium/cilium [2] https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless

OrvalWintermute 5y ago
I read alot on home setups and yours seems to balance both high security and maintainability very well. Care to share about the details of the security services side of your stack too? Cheers
- w7 5y ago
  Sure, hopefully I understand what you mean. For network observability I'm using Cilium's Hubble, which I will soon figure out how to get into a greylog setup or something. For container image vulnerability interrogation I'm running Harbor with Trivy enabled, initial motivation was to have an effective pull through cache for multiple registries because I got rate limited by AWS ECR (due to a misconfigured CI pipeline, oops), but it ended up killing two birds with 1 stone. Next on my list is writing an admission controller to modify supported registry targets to match my pull through cache configuration. Is there something more specific you wanted?

hsbauauvhabzb 5y ago

Calling your guide the ‘ultimate guide’ is disingenuous marketing. No single guide can cover all security concepts in all contexts. Every time I see that sorta wording I just assume the writer doesn’t actually know what they’re talking about

hsbauauvhabzb 5y ago
Continued: and given the writer seems to be all about tools the article fails to highlight that static (and automated dynamic) tools are limited in their ability to detect some classes of vulnerabilities and need to be backed with experience manual testing. This almost feels like it’s been written by a devops engineer who has a vague understanding about containerisation doesn’t have a clue about real and practical mechanisms to secure applications and services hosted inside containers. I’m not saying the article is totally bad, but calling it an ‘Ultimate Guide’ makes the author a charlatan.

_8j50 5y ago

Production host root fs should be mounted ro. Check out Linux IMA and how to only allow specific executables by hash. Centrally forward container logs. Use a VCS for container/workload templates and routinely audit for misconfig. Sysdig/falco and related tools are nice, but containers and their prod hosts are easier to harden