7 ms·

> and encryption to protect confidentiality Probably the hardest part about this. Private networks with private domains. Who runs the private CA, updates DNS r

> and encryption to protect confidentiality Probably the hardest part about this. Private networks with private domains. Who runs the private CA, updates DNS records, issues certs, revokes, keeps the keys secure, embeds the keychain in every container's cert stack, and enforces validation? That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. How many teams are actually going to do this, versus just terminating at the load balancer and everything in the cluster running plaintext?

devy 5y ago
> That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. Agree! This is why that "Kubernetes Hardening Guidance" is for NSA, not for startups. Resource needs aside, keeping basic AppSec/InfoSec hygiene is a strong recommendation. Also there are tons of startups that are trying to provide solutions/services to solve that also. A lot of times, it's worth the money.
- throwaway984393 5y ago
  I wonder about that. What are the attack vectors within a K8s cluster to necessitate inter-cluster transport encryption?
  - smarterclayton 5y ago
    Most (but not all) overlay networks are implemented in kernel. If you compromise one node in a cluster, you can fairly trivially snoop traffic, bias other nodes to send traffic through you, or listen via various mechanisms such that you can intercept traffic flowing between workloads not actually located on the compromised node. So always encrypt everything unless you’re in a very rare environment with central network control that cannot be compromised or intercepted from a given machine.
    - killjoywashere 5y ago
      And if you've been in that kind of rare environment, those people encrypt everything. They'd encrypt their license plate if they could. You want paranoid, look up laser microphones.
    - throwaway984393 5y ago
      This would be less of a concern if the cluster's pods were Firecrackers, yes? AWS EKS on Fargate has a dedicated ENI and kernel per pod; the only way to intercept the traffic is when it crosses a network, or with flow control logs. Or if somebody hacked the control plane, but that's always "Game over man, game over!"
    - raesene9 5y ago
      Although with many clusters, compromising a single node is likely to lead to cluster compromise as it allows for all the service account tokens assigned to workloads running on the compromised node to be used by the attacker :)
- zeckalpha 5y ago
  This guidance is provided by the NSA, not for the NSA.
  - ROARosen 5y ago
    From the doc: >It includes hardening strategies to avoid common misconfigurations and guide system administrators and developers of National Security Systems on how to deploy Kubernetes...
    - zeckalpha 5y ago
      Also: > Purpose > NSA and CISA developed this document in furtherance of their respective cybersecurity missions, including their responsibilities to develop and issue cybersecurity specifications and mitigations. This information may be shared broadly to reach all appropriate stakeholders. NSA has multiple mandates and many stakeholders.
      - westurner 5y ago
        Looks like there's actually a "summary of the key recommendations from each section" on page 2. > Works cited: > [1] Center for Internet Security, "Kubernetes," 2021. [Online]. Available: https://cisecurity.org/resources/?type=benchmark&search=kubernetes https://cisecurity.org/resources/?type=benchmark&search=kube... . > [2] DISA, "Kubernetes STIG," 2021. [Online]. Available: https://dl.dod.cyber.mil.wp- https://dl.dod.cyber.mil.wp- content/uploads/stigs/zip/U_Kubernetes_V1R1_STIG.zip. [Accessed 8 July 2021] > [3] The Linux Foundation, "Kubernetes Documentation," 2021. [Online]. Available: https://kubernetes.io/docs/home/ https://kubernetes.io/docs/home/ . [Accessed 8 July 2021]. > [4] The Linux Foundation, "11 Ways (Not) to Get Hacked," 18 07 2018. [Online]. Available: https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/#10-scan-images-and-run-ids https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hac... . [Accessed 8 July 2021]. > [5] MITRE, "Unsecured Credentials: Cloud Instance Metadata API." MITRE ATT&CK, 2021. [Online]. Available: https://attack.mitre.org/techniques/T1552/005/ https://attack.mitre.org/techniques/T1552/005/. [Accessed 8 July 2021]. > [6] CISA, "Analysis Report (AR21-013A): Strengthening Security Configurations to Defend Against Attackers Targeting Cloud Services." Cybersecurity and Infrastructure Security Agency, 14 January 2021. [Online]. Available:https://us- https://us- cert.cisa.gov/ncas/analysis-reports/ar21-013a [Accessed 8 July 2021]. How can k8s and zero-trust cooccur? > CISA encourages administrators and organizations review NSA’s guidance on Embracing a Zero Trust Security Model to help secure sensitive data, systems, and services. "Embracing a Zero Trust Security Model" (2021, as well) https://media.defense.gov/2021/Feb/25/2002588479/-1/-1/0/CSI_EMBRACING_ZT_SECURITY_MODEL_UOO115131-21.PDF https://media.defense.gov/2021/Feb/25/2002588479/-1/-1/0/CSI... In addition to "zero [trust]", I also looked for the term "SBOM". From p.32//39: > As updates are deployed, administrators should also keep up with removing any old components that are no longer needed from the environment. Using a managed Kubernetes service can help to automate upgrades and patches for Kubernetes, operating systems, and networking protocols. *However, administrators must still patch and upgrade their containerized applications.* "Existing artifact vuln scanners, databases, and specs?" https://github.com/google/osv/issues/55 https://github.com/google/osv/issues/55
__app_dev__ 5y ago
I agree there is a shit-ton of stuff to set up but based on the recent Pipeline and other Hacks where the Companies had to pay millions I would expect more companies to take stuff like this seriously.
sdenton4 5y ago
For historical context: https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html https://www.washingtonpost.com/world/national-security/nsa-i... (Google has since added encryption to its internal entworks.)
- adolph 5y ago
  Entwork uses a tree topology with stepping roots
zaat 5y ago
Encryption today is pretty much requirement for any regulated businesses and required practice for any sane shop, with or without Kubernetes. The only difference is that those services are communicating within the cluster internal network and not across different machines in the servers vlans. If anything, setting up the whole things within Kubernetes ecosystem can be much easier with the available operators and automation frameworks like cert manager and/or Istio.
_spoonman 5y ago
That sounds like work for a prime contractor with at least two subs under it. And consultants to help them implement SAFe AGILE.
spectre013 5y ago
We do this right now in a totally disconnected env. We have process in place to get images and manifests into our env. All containers have to go through scanning pipelines and have to be approved through a process. We also for any container that makes requests that does not have the mechanisms for adding certificates we have to rebuild the containers in the disconnected env to insert certificates to allow communication. Makes daily life really interesting
- moochmooch 5y ago
  Do you use a proprietary tool to filter containers lacking the certificate mechanisms or is this handled as part of the manifests?
  - spectre013 5y ago
    It's all handled as part of the manifests as well as we can have our clusters pulled if we are caught using non approved containers and they are all scanned when they are brought into the disconnected environment.
adolph 5y ago
Opportunity is knocking
awsthro00945 5y ago
For as fundamental and important as encryption-in-transit is, it's always baffled me that there isn't a simpler, easier solution to accomplishing it on private networks. Everyone knows its important, and everyone wants to do it, but it's just such a pain in the ass and so prone to error that even some top security leaders will tell you not to bother because it's such a footgun. We really need something to help make the process simpler, like how Let's Encrypt made public HTTPS so much easier to do for even the smallest of websites.
- spydum 5y ago
  Smallstep attempts this, but I agree with everything you have said.
- amarshall 5y ago
  In some senses it’s differently complex, but WireGuard or similar may be simpler since it’s lower on the OSI and every application gets it “for free”.
  - EthanHeilman 5y ago
    I would argue that if you have services then the right place to put encryption and authentication is at the service level. Building secure channels between IP addresses is all good, but do you really want to map roles/identities/privileges to specific IP addresses if those roles/identities/privileges really represent services?
    - amarshall 5y ago
      Why not? The IPs are cryptographically mapped to a specific client. Doesn’t stop one from using DNS to find the IP. Like I said: differently complex but it’s a general solution to the problem and doesn’t require changing more “inner” things as much.
      - EthanHeilman 5y ago
        What if you end up spinning more than one container for that service? How are these containers getting the different secrets they need to identify themselves? Are you attaching IAM roles to them to get secrets from some secret store?
3np 5y ago
Service mesh solutions like Istio / Consul Connect+Vault can help a lot with this. Depending on the existing size and complexity of your stack those months can be cut down to weeks or even days. I don't mean to trivialize the time and expertise needed to set up and manage, but if you can afford to run a microservice architecture on k8s already it's definitely not untenable.
calvinmorrison 5y ago
Isn't this exactly what hashicorps "consul" can do? Specific services, and setup keys/certs so that all internal traffic is also 'blindly' encrypted? End point services don't care or know about it because it's transparent, but over the internal network it's encrypted?
xyst 5y ago
TLS terminating is fine for most use cases (basic web services), but if you are protecting PII information you need to/should protect from external and internal threats. Maybe a disgruntled sys admin decides to capture data coming in from the load balancer between the service(s) and sell it to the highest bidder. If traffic is encrypted between the load balancer and underlying service, it makes it much harder to do.
- fomine3 5y ago
  like protecting from three letter agency

devy 5y ago

> That is a shit-ton of stuff to set up (and potentially screw up) which will take a small team probably months to complete. Agree! This is why that "Kubernetes Hardening Guidance" is for NSA, not for startups. Resource needs aside, keeping basic AppSec/InfoSec hygiene is a strong recommendation. Also there are tons of startups that are trying to provide solutions/services to solve that also. A lot of times, it's worth the money.

throwaway984393 5y ago
I wonder about that. What are the attack vectors within a K8s cluster to necessitate inter-cluster transport encryption?
- smarterclayton 5y ago
  Most (but not all) overlay networks are implemented in kernel. If you compromise one node in a cluster, you can fairly trivially snoop traffic, bias other nodes to send traffic through you, or listen via various mechanisms such that you can intercept traffic flowing between workloads not actually located on the compromised node. So always encrypt everything unless you’re in a very rare environment with central network control that cannot be compromised or intercepted from a given machine.
  - killjoywashere 5y ago
    And if you've been in that kind of rare environment, those people encrypt everything. They'd encrypt their license plate if they could. You want paranoid, look up laser microphones.
  - throwaway984393 5y ago
    This would be less of a concern if the cluster's pods were Firecrackers, yes? AWS EKS on Fargate has a dedicated ENI and kernel per pod; the only way to intercept the traffic is when it crosses a network, or with flow control logs. Or if somebody hacked the control plane, but that's always "Game over man, game over!"
  - raesene9 5y ago
    Although with many clusters, compromising a single node is likely to lead to cluster compromise as it allows for all the service account tokens assigned to workloads running on the compromised node to be used by the attacker :)
zeckalpha 5y ago
This guidance is provided by the NSA, not for the NSA.
- ROARosen 5y ago
  From the doc: >It includes hardening strategies to avoid common misconfigurations and guide system administrators and developers of National Security Systems on how to deploy Kubernetes...
  - zeckalpha 5y ago
    Also: > Purpose > NSA and CISA developed this document in furtherance of their respective cybersecurity missions, including their responsibilities to develop and issue cybersecurity specifications and mitigations. This information may be shared broadly to reach all appropriate stakeholders. NSA has multiple mandates and many stakeholders.
    - westurner 5y ago
      Looks like there's actually a "summary of the key recommendations from each section" on page 2. > Works cited: > [1] Center for Internet Security, "Kubernetes," 2021. [Online]. Available: https://cisecurity.org/resources/?type=benchmark&search=kubernetes https://cisecurity.org/resources/?type=benchmark&search=kube... . > [2] DISA, "Kubernetes STIG," 2021. [Online]. Available: https://dl.dod.cyber.mil.wp- https://dl.dod.cyber.mil.wp- content/uploads/stigs/zip/U_Kubernetes_V1R1_STIG.zip. [Accessed 8 July 2021] > [3] The Linux Foundation, "Kubernetes Documentation," 2021. [Online]. Available: https://kubernetes.io/docs/home/ https://kubernetes.io/docs/home/ . [Accessed 8 July 2021]. > [4] The Linux Foundation, "11 Ways (Not) to Get Hacked," 18 07 2018. [Online]. Available: https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/#10-scan-images-and-run-ids https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hac... . [Accessed 8 July 2021]. > [5] MITRE, "Unsecured Credentials: Cloud Instance Metadata API." MITRE ATT&CK, 2021. [Online]. Available: https://attack.mitre.org/techniques/T1552/005/ https://attack.mitre.org/techniques/T1552/005/. [Accessed 8 July 2021]. > [6] CISA, "Analysis Report (AR21-013A): Strengthening Security Configurations to Defend Against Attackers Targeting Cloud Services." Cybersecurity and Infrastructure Security Agency, 14 January 2021. [Online]. Available:https://us- https://us- cert.cisa.gov/ncas/analysis-reports/ar21-013a [Accessed 8 July 2021]. How can k8s and zero-trust cooccur? > CISA encourages administrators and organizations review NSA’s guidance on Embracing a Zero Trust Security Model to help secure sensitive data, systems, and services. "Embracing a Zero Trust Security Model" (2021, as well) https://media.defense.gov/2021/Feb/25/2002588479/-1/-1/0/CSI_EMBRACING_ZT_SECURITY_MODEL_UOO115131-21.PDF https://media.defense.gov/2021/Feb/25/2002588479/-1/-1/0/CSI... In addition to "zero [trust]", I also looked for the term "SBOM". From p.32//39: > As updates are deployed, administrators should also keep up with removing any old components that are no longer needed from the environment. Using a managed Kubernetes service can help to automate upgrades and patches for Kubernetes, operating systems, and networking protocols. *However, administrators must still patch and upgrade their containerized applications.* "Existing artifact vuln scanners, databases, and specs?" https://github.com/google/osv/issues/55 https://github.com/google/osv/issues/55

__app_dev__ 5y ago

I agree there is a shit-ton of stuff to set up but based on the recent Pipeline and other Hacks where the Companies had to pay millions I would expect more companies to take stuff like this seriously.

sdenton4 5y ago

For historical context: https://www.washingtonpost.com/world/national-security/nsa-infiltrates-links-to-yahoo-google-data-centers-worldwide-snowden-documents-say/2013/10/30/e51d661e-4166-11e3-8b74-d89d714ca4dd_story.html https://www.washingtonpost.com/world/national-security/nsa-i... (Google has since added encryption to its internal entworks.)

adolph 5y ago
Entwork uses a tree topology with stepping roots

zaat 5y ago

Encryption today is pretty much requirement for any regulated businesses and required practice for any sane shop, with or without Kubernetes. The only difference is that those services are communicating within the cluster internal network and not across different machines in the servers vlans. If anything, setting up the whole things within Kubernetes ecosystem can be much easier with the available operators and automation frameworks like cert manager and/or Istio.

_spoonman 5y ago

That sounds like work for a prime contractor with at least two subs under it. And consultants to help them implement SAFe AGILE.

spectre013 5y ago

We do this right now in a totally disconnected env. We have process in place to get images and manifests into our env. All containers have to go through scanning pipelines and have to be approved through a process. We also for any container that makes requests that does not have the mechanisms for adding certificates we have to rebuild the containers in the disconnected env to insert certificates to allow communication. Makes daily life really interesting

moochmooch 5y ago
Do you use a proprietary tool to filter containers lacking the certificate mechanisms or is this handled as part of the manifests?
- spectre013 5y ago
  It's all handled as part of the manifests as well as we can have our clusters pulled if we are caught using non approved containers and they are all scanned when they are brought into the disconnected environment.

adolph 5y ago

Opportunity is knocking

awsthro00945 5y ago

For as fundamental and important as encryption-in-transit is, it's always baffled me that there isn't a simpler, easier solution to accomplishing it on private networks. Everyone knows its important, and everyone wants to do it, but it's just such a pain in the ass and so prone to error that even some top security leaders will tell you not to bother because it's such a footgun. We really need something to help make the process simpler, like how Let's Encrypt made public HTTPS so much easier to do for even the smallest of websites.

spydum 5y ago
Smallstep attempts this, but I agree with everything you have said.
amarshall 5y ago
In some senses it’s differently complex, but WireGuard or similar may be simpler since it’s lower on the OSI and every application gets it “for free”.
- EthanHeilman 5y ago
  I would argue that if you have services then the right place to put encryption and authentication is at the service level. Building secure channels between IP addresses is all good, but do you really want to map roles/identities/privileges to specific IP addresses if those roles/identities/privileges really represent services?
  - amarshall 5y ago
    Why not? The IPs are cryptographically mapped to a specific client. Doesn’t stop one from using DNS to find the IP. Like I said: differently complex but it’s a general solution to the problem and doesn’t require changing more “inner” things as much.
    - EthanHeilman 5y ago
      What if you end up spinning more than one container for that service? How are these containers getting the different secrets they need to identify themselves? Are you attaching IAM roles to them to get secrets from some secret store?

3np 5y ago

Service mesh solutions like Istio / Consul Connect+Vault can help a lot with this. Depending on the existing size and complexity of your stack those months can be cut down to weeks or even days. I don't mean to trivialize the time and expertise needed to set up and manage, but if you can afford to run a microservice architecture on k8s already it's definitely not untenable.

calvinmorrison 5y ago

Isn't this exactly what hashicorps "consul" can do? Specific services, and setup keys/certs so that all internal traffic is also 'blindly' encrypted? End point services don't care or know about it because it's transparent, but over the internal network it's encrypted?

xyst 5y ago

TLS terminating is fine for most use cases (basic web services), but if you are protecting PII information you need to/should protect from external and internal threats. Maybe a disgruntled sys admin decides to capture data coming in from the load balancer between the service(s) and sell it to the highest bidder. If traffic is encrypted between the load balancer and underlying service, it makes it much harder to do.

fomine3 5y ago
like protecting from three letter agency