3 ms·

so do you make a point of changing SVC.cluster.local to your own TLD?

jmillikin 4y ago
This might be a longer answer than you expected/wanted; sorry. Let's say you have a bunch of machines, mounted in racks, installed in datacenters. You might have a naming convention like `m3r5dls1.acme-prod.com` to identify "machine #3 in rack #5 in datacenter #1 in The Dalles, Oregon". The acme-prod.com TLD is used to avoid cross-contamination between internal and public-facing identities. You'll have remote management services running on those machines, which identify themselves with an mTLS certificate issued to the machine hostname. For services it's a little more complex, because there are at least three common use cases: 1. You have a service like `my-namespace/hello-world`, and you'd like to be able to send it RPCs. The clients don't care where the service backends are located, they just need to be able to resolve that service name to a set of (IP, port) pairs. 2. The same service as above, but now there's a locality requirement -- you want to send RPCs to a service in a specific place. Maybe for latency (closer = faster), maybe cost (local datacenter = no egress = cheaper), maybe regulatory ("all customers in India must have their data fetched from the Mumbai datacenter"). 3. You have a specific instance of a service, for example when tracking the progress of a long-running operation. In Kubernetes this would be a specific pod. It would also be nice if the services' mTLS certs are served from the same CA as machine certs, since that allows you to write tools that send RPCs without having to do their own parsing of the destination address. The `{service_name}.cluster.local` format is not flexible enough to accommodate these goals, but if you're willing to use longer service identities then there's no problem: 1. `{service_name}.{namespace}.any.acme-prod.com` can resolve to any instance of that service, regardless of location. 2. `{service_name}.{namespace}.dls1.acme-prod.com` can resolve to any instance of that service in the datacenter `dls1`. 3. `{pod_id}.{service_name}.{namespace}.dls1.acme-prod.com` resolves to a specific instance. Now you have consistent identities and universal mTLS authentication, so you can write authorization policies that grant services specific permissions they need to do their work.

This might be a longer answer than you expected/wanted; sorry. Let's say you have a bunch of machines, mounted in racks, installed in datacenters. You might have a naming convention like `m3r5dls1.acme-prod.com` to identify "machine #3 in rack #5 in datacenter #1 in The Dalles, Oregon". The acme-prod.com TLD is used to avoid cross-contamination between internal and public-facing identities. You'll have remote management services running on those machines, which identify themselves with an mTLS certificate issued to the machine hostname. For services it's a little more complex, because there are at least three common use cases: 1. You have a service like `my-namespace/hello-world`, and you'd like to be able to send it RPCs. The clients don't care where the service backends are located, they just need to be able to resolve that service name to a set of (IP, port) pairs. 2. The same service as above, but now there's a locality requirement -- you want to send RPCs to a service in a specific place. Maybe for latency (closer = faster), maybe cost (local datacenter = no egress = cheaper), maybe regulatory ("all customers in India must have their data fetched from the Mumbai datacenter"). 3. You have a specific instance of a service, for example when tracking the progress of a long-running operation. In Kubernetes this would be a specific pod. It would also be nice if the services' mTLS certs are served from the same CA as machine certs, since that allows you to write tools that send RPCs without having to do their own parsing of the destination address. The `{service_name}.cluster.local` format is not flexible enough to accommodate these goals, but if you're willing to use longer service identities then there's no problem: 1. `{service_name}.{namespace}.any.acme-prod.com` can resolve to any instance of that service, regardless of location. 2. `{service_name}.{namespace}.dls1.acme-prod.com` can resolve to any instance of that service in the datacenter `dls1`. 3. `{pod_id}.{service_name}.{namespace}.dls1.acme-prod.com` resolves to a specific instance. Now you have consistent identities and universal mTLS authentication, so you can write authorization policies that grant services specific permissions they need to do their work.