15 ms·

NSA Kubernetes Hardening Guidance [pdf]

• pletnes 5y ago
  What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations?
  • tamalsaha001 5y ago
    Both for different reasons!
  • raesene9 5y ago
    with the usual caveat that it depends on your threat model, I'd say that having separate clusters is likely to provide better segregation. Of course that reduces the benefits of Kubernetes from a cost perspective, and increases administrative overhead, so it's a trade-off.
  • joncp 5y ago
    Hmm... seems to call for a tool for a cluster-of-clusters. If "kubernetes" is Greek for the ship's pilot, what's the name for the captain or maybe the admiral of the armada?
    • adolph 5y ago
      This is how GKE Anthos works on prem
    • sascha_sl 5y ago
      It exists, sort of. https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/
      • antonvs 5y ago
        More directly relevant is Cluster Federation: https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
    • cmckn 5y ago
      Rancher.
    • boilerupnc 5y ago
      Cluster-as-a-Service :-) Hive for OpenShift [0] - Provisioning of the K8s Fleet Advanced Cluster Management [1] - Management of the K8s Fleet [0] https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-service https://cloud.redhat.com/blog/openshift-hive-cluster-as-a-se... [1] https://cloud.redhat.com/products/advanced-cluster-management https://cloud.redhat.com/products/advanced-cluster-managemen... [Disclosure: I'm an IBMer]
    • baq 5y ago
      "Nauarch, in ancient Greece, an admiral or supreme commander of the navy, used as an official title primarily in Sparta in the late 5th and early 4th centuries bc." - google cites britannica (!)
    • antonvs 5y ago
      One of the several tools in this space is called Admiralty: https://caylent.com/kubernetes-cluster-federation-with-admiralty https://caylent.com/kubernetes-cluster-federation-with-admir... Google says the Greek for that is Ναυαρχείο. Now if you see a product named Navarxeio or Navarcheio, you'll know what it is.
    • CorralPeltzer 5y ago
      https://github.com/karmada-io/karmada https://github.com/karmada-io/karmada
  • bsamuels 5y ago
    If I could go back, single cluster. Any benefits you get from going multi-cluster can be achieved by configuring a single cluster correctly.
    • agilob 5y ago
      >a single cluster correctly Can you elaborate?
      • antonvs 5y ago
        At a high level, almost anything you would want to use multiple clusters for can be done on a single cluster, using e.g. node pools, affinity, and taints to ensure that workloads only run on the machines you want them to. As a simple example, you can set up a separate node pool for production, and use node affinity and/or taints to ensure that only production workloads can run there. One exception, as other have mentioned, is blast radius - with a single cluster, a problem with Kubernetes itself could take down everything.
        • dharmab 5y ago
          Another issue is scaling limits. We've found a few dozen ways to break a cluster by scaling along a certain axis. (Most are not related to "vanilla" Kubernetes but the backing cloud provider or specific add-on components.)
        • allset_ 5y ago
          Other management tasks are easier when you have separate clusters, such as applying environment-specific OPA policies and not having to filter them based on labels or annotations you hope everyone is using correctly.
    • kelseyhightower 5y ago
      The one benefit you get is protection from bugs in Kubernetes itself and a reduced blast radius. Even if you could produce a secure and H/A cluster, you still leave yourself open to Kubernetes bugs and configuration mistakes such as adding a network policy that blocks all communication across all namespaces. Multiple clusters protects you from these types of configuration mistakes by reducing the blast radius and providing an additional landing zone to roll out changes over time.
  • dharmab 5y ago
    At our very large org we do both. At least two clusters per region to isolate platform changes, all hardened to the same standards using automated tooling.
  • zzyzxd 5y ago
    You can't skip "spending a ton of time hardening one cluster" anyways. Having multiple clusters may help reduce the blast radius of _certain_ attacks, to some degree. However, managing multiple clusters is a lot more difficult than managing one, and you will potentially replicate bad practices, vulnerabilities to multiple places and increase maintenance burden.
  • outworlder 5y ago
    > What yields the lowest risk - spending a ton of time hardening one cluster, or building multiple clusters to reduce the blast radius of bugs and misconfigurations? Not sure this is a valid dichotomy. If you are spinning up multiple clusters, you are presumably doing so in an automated fashion. If so, then the effort of hardening is very similar. It doesn't really matter where you do it. Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk.
    • kortilla 5y ago
      > If so, then the effort of hardening is very similar. It doesn't really matter where you do it. Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. This is why cloud providers don’t stick you on the same k8s cluster as another tenant. > Multiple clusters may have a smaller blast radius, but will have a larger attack surface. Things may be shared between them (accounts? network tunnels? credentials to a shared service?) in which case an intrusion in one puts everyone else at risk. It’s not really clear what you’re trying to say here. If someone compromises credentials shared between all clusters that’s the same as compromising credentials used by one mega cluster.
      • RandomThrow321 5y ago
        > Nope. If the clusters are separate it limits how damaging a compromise of the cluster is. But if the clusters are configured similarly, a flaw in one is likely present in the others. GPs point is that if you invest in hardening, you can easily apply it to multiple clusters. > It’s not really clear what you’re trying to say here. I assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.).
        • kortilla 5y ago
          > But if the clusters are configured similarly, a flaw in one is likely present in the others. That doesn’t matter. The point is that you isolate applications/tenants into different clusters. So if someone exploits their own, they haven’t gained access to some other application. > assume they mean having more clusters present means there are more opportunities to be compromised (e.g. more credentials to leak, more API servers to target, possible version skew, etc.). That doesn’t even make sense though. In our strawman scenario these are cookie cutter things. Many is not more vulnerable than one in this case.
  • kenm47 5y ago
    what about tooling that hardens it across clusters?
• raesene9 5y ago
  Some useful guidance here, although worth noting that some of it is a bit dated (k8s security can move quickly). Most notably from a scan through, they're mentioning PodSecurityPolicy, but that's deprecated and scheduled to be removed in 1.25. There will be an in-tree replacement but it won't work the same way. Out of tree open source options would be things like OPA, Kyverno, jsPolicy, k-rail or Kubewarden.
  • xxpor 5y ago
    > Some useful guidance here, although worth noting that some of it is a bit dated. Is there any digital security guidance from the feds that doesn't apply to? :)
    • sslayer 5y ago
      Everybody wants small gov, until they don't.
      • xxpor 5y ago
        This is why I think big vs little government is really missing the forest for the trees in a lot of contexts (unless your overall goal is to minimize taxes and regulations at all costs). It's really a debate about the nature of bureaucracy. Process vs nimble. You can organize things to promote either, depending on your actual goals. Unfortunately small government activists have recognized this and have enacted policies that promote incompetence as much as possible. "Good enough for government work" is a choice, not an inevitability.
        • herodoturtle 5y ago
          I wonder if there's a third option, a decentralized government of small nodes, which can orchestrate their activity to rapidly scale in the need of large resource projects.
          • smichel17 5y ago
• soheil 5y ago
  I keep forgetting NSA's job is to protect instead of maliciously eavesdropping on Americans. Given their prior probability of being a bad actor I'd take any security "guidance" they issue with a huge grain of salt.
  • buggeryorkshire 5y ago
    Reasonable. But then again they did come up with selinux and I haven't seen any backdoors in that.
    • reacharavindh 5y ago
      Yet.
  • lazide 5y ago
    Like many large organizations, The government has many groups, often with many of them working to some extent against each other. AES also has been blessed by the NSA, and I bet you use that extensively too - if you want to or not?
  • wvh 5y ago
    It's perfectly possible, laudable even, to read things with a healthy dose of salt and still expand your understanding. I think this document gives a good general overview, often missing in the fast-paced, crowded and noisy Kubernetes landscape.
  • kgarten 5y ago
    not sure why you are downvoted. Completely agreed. Keeping 0-days for yourself and making security standards weaker, will just weaken your standing. Even if this information might be useful (and without backdoors or bad advice), I just cannot trust them (so I won't click on the link). In terms of security, I trust the hacker community much more (going by ccc or other groups advice is definitely better).
• kchoudhu 5y ago
  Somehow the text is not just... kubectl -n my-ns delete pod,svc --all
  • deleted 5y ago
    [deleted]
• debarshri 5y ago
  This is really helpful. I wonder if there is curated list of k8s hardening guidelines for various organisations.
  • adolph 5y ago
    Just search for ‘* awesome list’ https://github.com/magnologan/awesome-k8s-security https://github.com/magnologan/awesome-k8s-security (Unaffiliated with above, just popped up for k8s hardening awesome list)
• flerovium 5y ago
  > A powerful example is the use of hypervisors to provide container isolation. Hypervisors rely on hardware to enforce the virtualization boundary rather than the operating system. Hypervisor isolation is more secure than traditional container isolation. The more things change, the more they stay the same.
• legrande 5y ago
  We all know it's the National Insecurity Agency[0], and that the NSA hoards & stockpiles 0day. They very rarely release tools and research papers designed to strengthen our IT infra, since they sit on so much 0day. There's no balance. I don't buy that they're 50% red team, and 50% blue team. More like 99% red team and 1% blue team. [0] https://en.wikipedia.org/wiki/Doublespeak https://en.wikipedia.org/wiki/Doublespeak
  • texasbigdata 5y ago
    Have absolutely zero background knowledge here, but just to be pendantic your argument is structured as a logical fallacy [1]. While we maybe could estimate the relative sizes of the groups you mention and compare them relative to each other to guess the strategy/policy/tactics it's not clear that would be accurate; or maybe we could infer based on some heuristic or metric (like budget being a proxy for headcount), and even then it's not clear how certain that guess would be, so it's not obvious how "we all know" it's 99/1 vs 50/50, vs any other permutation. Push come to shove would probably agree with your premise and conclusion, and really have no idea, so apologies for being nitpicky; without a background on the technical details it's likely I'm wrong. [1] https://www.logicallyfallacious.com/logicalfallacies/Alleged-Certainty https://www.logicallyfallacious.com/logicalfallacies/Alleged...
    • bredren 5y ago
      To add to this, GP: it is enough to simply state facts and how they influence your opinion. No individual can speak for all readers here on how they view this agency. Attempting to weakens the comment.
    • kgarten 5y ago
      Yes, there is a logical fallacy here. Yet, that does not mean that the initial comment doesn't have a point. (that's another logical fallacy ...) https://www.logicallyfallacious.com/logicalfallacies/Argument-from-Fallacy https://www.logicallyfallacious.com/logicalfallacies/Argumen...
  • Closi 5y ago
    > We all know it's the National Insecurity Agency[0], and that the NSA hoards & stockpiles 0day. They very rarely release tools and research papers designed to strengthen our IT infra, since they sit on so much 0day. There's no balance. Well if the NSA does have loads of 0day then it's still better for them to give good security advice to strengthen infra, because it will limit the access adversary's have while they still have all the 0day's anyway. i.e. they are advanced enough to not need to walk through an open door, so they might as well encourage others to close the doors because that will increase national security (while presumably not limiting their own access).
• andrewmcwatters 5y ago
  Well that’s… curious. Not sure I’ve ever read the NSA providing hardening guidance on anything before.
  • antonvs 5y ago
    They've been doing that for at least a decade, but probably quite a bit longer. Here are their hardening guidelines for RHEL 5, from 2011: https://apps.nsa.gov/iaarchive/library/ia-guidance/security-configuration/operating-systems/guide-to-the-secure-configuration-of-red-hat-enterprise.cfm https://apps.nsa.gov/iaarchive/library/ia-guidance/security-... They have similar guidance for Windows, web browsers, industrial control systems, etc.
    • andrewmcwatters 5y ago
      Interesting! Thank you for sharing this.
    • deleted 5y ago
      [deleted]
  • beprogrammed 5y ago
    It's fine to trust them right up until they give you a magic number.
    • tablespoon 5y ago
      > It's fine to trust them right up until they give you a magic number. IIRC, DES had NSA-provided magic numbers in it that made it more secure against a then-not-publicly-known cryptoanalytic attack.
  • sdmike1 5y ago
    In general the NSA functions more like 2 agencies, one focused on the "red" side (hacking, breaking crypto, sigint stuff) and one focused on the "blue" side (protecting US assets from being hacked, developing better/new crypto, providing guidance on security). Both sides are good at their jobs and for what it's worth, my understanding is that the blue side really does want to keep your shit from being hacked.
• beprogrammed 5y ago
  - Scan containers and Pods for vulnerabilities or misconfigurations. - Run containers and Pods with the least privileges possible. - Use network separation to control the amount of damage a compromise can cause. - Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality. - Use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface. - Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity. - Periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied.
  • haolez 5y ago
    Who scans the vulnerability scanners? Genuine question. How does the community/ecosystem solve this problem of auditability?
    • tinco 5y ago
      If your threat profile says you need to audit your vulnerability scanners, you audit your vulnerability scanners. There's not really a problem there right?
      • haolez 5y ago
        I've never had to. I wanted feedback from people who have.
      • Pokepokalypse 5y ago
        NIST also says: if your scanner finds a vulnerability, it's up to you to VALIDATE that it's not a false-positive. False-positives abound on these scanners.
    • knownjorbist 5y ago
      For anyone who hasn't read it: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
• __app_dev__ 5y ago
  I used to study and focus on security a lot more and keep up with trends. After several interviews this year I realize a lot of jobs prioritize leetcode over everything else. It's pretty annoying and makes me wonder if the focus for tech works is leetcode above all else then no wonder so many companies have insecure apps and servers.
  • tra3 5y ago
    If you can't reverse a doubly linked binary prefix tree in O(1) then how can you be trusted with security?! /s :(
    • herodoturtle 5y ago
      Yes but in most circumstances, quick security is better than linear security; not sure about bubble security though.
      • dkersten 5y ago
        Bubble security sounds like a good idea. You know, put everything into its own isolated little bubble.
      • sellyme 5y ago
        Best I can do is Bogo security
  • stillbourne 5y ago
    I applied for a job that wanted someone who has experience with SAML. I've actually written my own hobby IDP, and I can diagram the handshake off the top of my head. I've spent a lot of time learning how to write custom decorators to handle access restrictions. I failed my interview because they wanted me to leetcode some shit with 3d geometric volumes. I'm sorry but what does this have to do with SAML or security?
    • Gene_Parmesan 5y ago
• kenm47 5y ago
  so... a lot of this can be done with Fairwind's OSS tool Polaris... https://github.com/FairwindsOps/polaris https://github.com/FairwindsOps/polaris feels good that we've been addressing this for a bit already tbh. (disclaimer, I work for fairwinds)
  • herodoturtle 5y ago
    How did you (/they) come up with the name Polaris?
    • EamonnMR 5y ago
      If I had to guess it's a nautical theme, following Kubes. Fair Winds (sailing), Polaris (North Star, used for navigation.)
      • kenm47 5y ago
        basically this. yes.
• nwmcsween 5y ago
  The elephant in the room here is almost all containers according to artifacthub.io, etc are a complete tire fire
  • kenm47 5y ago
    there are good free/oss container scanners. check out Trivy.—no reason not to use one.
  • nonameiguess 5y ago
    The DoD maintains its own registry of hardened container images they call the Iron Bank. I guess they can't issue guidelines to the general public that you should use these, but the DoD has to use them. Which kind of sucks, because they may be hardened, but they also break all the time because the people responsible for hardening them can't possibly understand all the myriad subleties involved in building and deploying software packaged with dependencies in the same way the actual software vendors do. They make some serious rookie mistakes, like just straight copying executables out of a Fedora image into a UBI images, which works perfectly fine when a brand-new UBI release happens and it's on the same glibc as Fedora, then immediately stops working and all your containers break when Fedora updates.
    • throwaway984393 5y ago
      They may suck at building containers, but this also sounds like a release management issue. Both the producers and consumers of the release need a test suite to validate the new artifacts before they can make it into a pipeline to eventually deliver to a customer use case. (But also they should 100% not be copying random binaries) For what it's worth I've seen worse from corporations. Bad hires lead to bad systems.
    • spectre013 5y ago
      I work on Platform one and we use and deploy new versions of these containers weekly and have never had them break in that way. In the Beginning when I was on the Kubernetes team we struggled with the containers just not working at all but they have gotten better. Now I work on deploying and we run every container from IB and have few issues. If you find them report the images and they will fix them pretty quick.
• Rd6n6 5y ago
  Do they have a version for ordinary web app servers?
• kgarten 5y ago
  How do I know that this advice is useful and does not put me in danger? Example: NSA recommends to use RSA encryption. https://www.theverge.com/2013/12/20/5231006/nsa-paid-10-million-for-a-back-door-into-rsa-encryption-according-to https://www.theverge.com/2013/12/20/5231006/nsa-paid-10-mill...
  • m1keil 5y ago
    You don't use this guide as a bible but take it into account and compare with other common security advice in the field. If you get similar results it most likely a good list of advice.
  • jdubs 5y ago
    This isn't for regular people, this is telling third parties what they need, in order for them to try to sell something to the nsa.
• pulketo 5y ago
  A guide from somebody who hates not knowing everything about you... Tengo mis dudas
• rob_c 5y ago
  A lot of this applies to containers in general. Not complaining, it's well written but wish they would break out the none kube container stuff into a general container-sec advice for people.
  • asymptosis 5y ago
    This is a great point. And containers don't even really exist in the first place, so really there should be (at least one of) a family of docs about securing the various namespaces, cgroups etc in modern Linux releases, and a doc about how to secure them in combination with each other.
• neop1x 5y ago
  First you should configure some kind of authentication. It is fun to remember this 3 years old Tesla example [1]: Publicly accessible Kubernetes Dashboard. [1] https://www.zdnet.com/article/tesla-systems-used-by-hackers-to-mine-cryptocurrency/ https://www.zdnet.com/article/tesla-systems-used-by-hackers-...