3 ms·
Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1
by theptip 10y ago
Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted.
That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear.
There are a couple of pieces of hard-to-find config required to achieve TLS-to-the-pod with Ingress:
1) You need to enable ssl-passthrough on your nginx ingress; this is a simple annotation: https://github.com/kubernetes/contrib/issues/1854 https://github.com/kubernetes/contrib/issues/1854. This will use nginx's streaming mode to route requests with SNI without terminating the TLS connection.
2) Now you'll need a way of getting your certs into the pod; kube-lego attaches the certs to the Ingress pod, which is not what you want for TLS-to-the-pod. https://github.com/PalmStoneGames/kube-cert-manager/ https://github.com/PalmStoneGames/kube-cert-manager/ lets you do this in an automated way, by creating k8s secrets containing the letsencrypt certs.
3) Your pods will need an SSL proxy to terminate the TLS connection. I use a modified version of https://github.com/GoogleCloudPlatform/nginx-ssl-proxy https://github.com/GoogleCloudPlatform/nginx-ssl-proxy.
4) You'll want a way to dynamically create DNS entries; Mate is a good approach here. Note that once you enable automatic DNS names for your Services, then it becomes less important to share a single public IP using SNI. You can actually abandon the Ingress, and have Mate set up your generated DNS records to point to the Service's LoadBalancer IP.
(As an aside, if you stick with Nginx Ingress, you can connect it to the outside world using a Kubernetes loadbalancer, instead of having to use a Terraform LB; the (hard-to-find and fairly new) config flag for that is `publish-service` (https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/controller/launch.go#L49 https://github.com/kubernetes/ingress/blob/master/core/pkg/i...).
- conradev 10y agoWould an overlay network with a shared secret for encryption of pod-to-pod networking be another solution to this problem? I feel like the ideal should involve keeping the key material in as few places as possible.
- theptip 10y agoYes, that's an option -- the simplest option would be a single overlay network, with the node-node tunnels encrypted using IPSec or similar (https://github.com/coreos/flannel/issues/6 https://github.com/coreos/flannel/issues/6 or https://www.weave.works/documentation/net-latest-how-it-works/net-latest-encryption/ https://www.weave.works/documentation/net-latest-how-it-work... or https://github.com/projectcalico/felix/issues/997 https://github.com/projectcalico/felix/issues/997). I think this would be tricky to configure in GKE though. With a secret-per-pod, your key material lives in in the etcd on the API server, and gets mounted in a tmpfs on each pod that is given the secret. Only Pods in the Secret's namespace can access the Secret, so if you have RBAC configured correctly it should be possible to lock this down tightly to only the code that needs the Secret. (I'm not sure how to do this in GKE; I'm currently treating each cluster as a single security domain).
- A-F1V3 10y agoIf you are open to using an (awesome) vendor for this kind of thing, I would highly recommend taking a look at Backplane (https://www.backplane.io/ https://www.backplane.io/). It's as simple as running a sidecar Backplane agent container along side any container that you want to route http traffic to and then shape the traffic via their API (it becomes your load balancer as well). They automatically provision Lets Encrypt certificates for your endpoints as well, so you don't have to worry about any of that. We have been using it at Mux in production for months and have been very happy with the results. Backplane also has some other nice built in features like blue/green deploys and built in OAuth support that are really nice to have out of the box.
- SEJeff 10y agoYes and no. You're sending traffic over google's SDN in the clear, which is still encrypted by google if you believe: https://cloud.google.com/security/security-design/ https://cloud.google.com/security/security-design/
- theptip 10y agoI am fairly sure that traffic between nodes is not encrypted; I have dug into this in some detail with GCP staff. if you can quote some sources to the contrary I'd be interested to see them. (The linked security design doc mentions 'encrypted in transit _to the data center_', but it doesn't address traffic inside the DC, last time I read through it in detail).
- SEJeff 10y agoNo I suspect you're right even though the linked security design doc does say they encapsulate all app level traffic (such as HTTP) in their own RPC.
- lobster_johnson 10y agoI wonder how much of a vulnerability that really is. The SDN encapsulates everything and is supposedly IP-spoofing-secure, so in principle there's no way for anyone else in the same DC to get your traffic. Of course, you could have a local attacker get in through other means, and then access local DC traffic within your SDN. But if you get to that point, you probably have bigger problem than terminating SSL.
- theptip 10y agoYour main attack vectors are: 1) A disgruntled employee sets up a surreptitious tap on the network to see if any secret material comes through. A high value target would be `Authorization: Bearer` in HTTP headers, but there are plenty of other things to slurp up. 2) A normally honest employee running an unrelated network tap to diagnose an issue with the SDN spots your Authorization headers (or other secrets), and knowing that they have a legitimate reason to have the wire capture, copies out the key material. This is very hard to prevent, since network admins can and should be tapping the network from time to time. I'm not particularly concerned about someone from outside Google breaking into the SDN fabric, though a hypervisor breach could leak network traffic from other tenants on your instance (if you are sharing).