7 ms·

Teleport v3.0 introduces Kubernetes support

61 points by twakefield 8y ago

aberoham 8y ago
Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. At KubeCon CPH, when I asked Tim Hockin about the long-awaited debug containers feature, aka 'kubectl debug', he said that "I really do feel that every time someone has to SSH into a node a ferry loses its wings." :) Dropping out of or below the cluster layer inherently means there is something in K8s that is missing. But no matter what is added to its APIs, Kubernetes will experience a long tail of weird corner cases that just can’t be covered. Reportedly the same is true with Borg jobs internally within Google. Teleport's K8s protocol implementation mates its certificate-based SSH auth with Kubernetes certificate signing request API, plus Teleport acts as a full recording MITM for all K8s apiserver requests. How exactly end-users will leverage this remains to be seen, but we're super excited to see where the community takes this (and related tricky K8s vs provider IAM issues) as it continues to expand. Full disclosure -- I work at Gravitational, worked closely with Sasha on this feature[1], and am a fan of Kelsey Hightower's live demo keynotes[2] where he frequently quips about the inherent struggle between procedural Dev+Ops encountering declarative APIs such as K8s. [1] https://github.com/gravitational/teleport/issues/1986 https://github.com/gravitational/teleport/issues/1986 [2] https://www.youtube.com/watch?v=07jq-5VbBVQ https://www.youtube.com/watch?v=07jq-5VbBVQ
- outworlder 8y ago
  > Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. Agreed. In general, every time you need to SSH anywhere, it means that there is some tooling you are missing. In very rare cases, you do need to break the glass to deal with exceptional conditions, but this should be rare.
- malkia 8y ago
  AFAIR, "debugging", at least in java allowed you to set a "breakpoint" in codesearch, but essentially something magically was happening behind, where if the CPU goes through this line you've marked as breakpoint it'll remember the surroundings (like variables in that scope, etc). Furthermore, one could declare (a bit like "live" coding) a one-liner (say in java) that would get called if that lines passes - so you can log something in better fashion, etc. But it had limitations. First I didn't get it, then it sunk into me - once you have thousands of jobs - it's pointless to try and debug one of them (thousands, as in MapReduce thousands). Not that I've used the feature a lot, but tried it several times and worked fine. It had nice integration with codesearch, where you can, given your "namespace" and "job" name, can set the codesearch to be "synced" to the CL it built your job + cherrypicks (I guess) - so the code you are looking is the code that your binary got compiled (another benefit of monolith depots, is that such things are easier there, or maybe it'll take a bit more work to get it working with multiple repos). E.g. magic :)
feydaykyn 8y ago
There's no pricing available for the enterprise edition, does someone know (roughly) how much we are talking about? Thanks!
- outworlder 8y ago
  I don't have a clue and would also like to know. Whenever there is a 'contact us' pricing page, I don't even bother. I can't even tell my boss to look into it if I don't know what the required budget will be, and I won't give out information to get contacted by sales teams and waste my time on a project that I wasn't even allocated time for. Plus it won't even be only my time, it will be finance and maybe even legal. So now there are multiple people involved to get a quote before it's even figured out if the solution fits the budget or if it is even a fit for the problem (unless there is some sort of trial). Whenever a project does this, it had better be the only thing capable of dousing a burning fire to get even considered.
  - gk1 8y ago
    Enterprise software is not one-size-fits-all like SaaS, and therefore can't have one-size-fits-all pricing like SaaS. If seeing "contact us" scares you then you're probably not the target audience.
    - outworlder 8y ago
      It doesn't 'scare' me as such. It just upgrades the task from 'can I quickly evaluate this, determine possible fit, and forward the recommendation to higher ups' to assembling a small task force. EDIT: I understand that this is a common practice and a legitimate business model, so it's not aimed at Gravitational specifically. It just causes people to sometimes dismiss offerings that they would be legitimately interested in, otherwise. For instance, Slack offers paid versions that are not yet at the Enterprise level(so less features), but you can at least get an idea, and also what is the pricing model (is it per seat? per server? per gb of storage? etc).
madjam002 8y ago
How does this compare to the OIDC integration with Kubernetes which is supported natively out of the box? You can hook up an enterprise IdP like AD FS with that, without any additional software.
- aberoham 8y ago
  The biggest difference is that Teleport is a full MITM for all apiserver traffic, leveraging the cluster-local certificate authority for short-lived creds (similar to how K8s internal bits such as the kubelet speaks TLS to the apiserver). Notably this also works with completely air-gapped clusters or end-users, or clusters where you want access to apiserver to be via egress (hidden bastion) only. K8s authentication methods that go inline with the apiserver, such as oidc or dex, achieve the same goals but are a bit of a different design flavor and arguably less clean from a threat or failure model perspective.
  - madjam002 8y ago
    How is OIDC less clean from a threat/failure model perspective? If your organisation has implemented an OIDC IdP, you can have the API server behind a Bastion like you suggest, and accessible through an authenticating reverse proxy (Beyondcorp style) WAF. You can even have token bound access/ID tokens which are backed by hardware TPMs if you're REALLY paranoid (upcoming spec). Plus because it's an open standard which K8s heavily embraces, you get rich group/user claims mapping in K8s allowing for logging and auditing.
- old-gregg 8y ago
  The obvious benefit is that synchronizes Kubernetes auth with SSH access, i.e. you don't have to authenticate twice if your organization uses SSH-based tools and k8s-based tooling, moreover it extends the same RBAC rules to both SSH and k8s, i.e. "developers must not touch production data" will be enforced for both "ssh production-host" and "kubectl exec production-pod". The second, perhaps less obvious, benefit is Teleport's built-in audit as shown in the video, i.e. all your interactions with all Kubernetes clusters are logged in one centralized place for audit purposes (and, in the case of the interactive sessions) can be replayed in a Youtube-like player. But yes, if you don't need the audit, and you don't use SSH, and if you only have one k8s cluster, this may be an overkill.
raesene9 8y ago
I'll be interested to see how this plays out. AuthN for users in k8s clusters is a bit of a pain point from what I've seen in a number of reviews I've done. The in-built options (Basic auth, token auth, client cert) are not really scalable for a larger number of users, and the other options (webhook, OIDC) can be complex to setup. So any relatively easy to use options for this would be welcome.
- SEJeff 8y ago
  OIDC tied to a LDAP directory is generally how you setup authentication for larger numbers of users. It really isn't that hard and has a dozen or so solutions with sensible config and operational footprints. Here is an example of how to do this (OIDC). Notice the short and concise documentation on how to set this up. It isn't too bad at all if you're running your own apiserver already: https://github.com/heptiolabs/gangway https://github.com/heptiolabs/gangway

61 points by twakefield 8y ago

Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. At KubeCon CPH, when I asked Tim Hockin about the long-awaited debug containers feature, aka 'kubectl debug', he said that "I really do feel that every time someone has to SSH into a node a ferry loses its wings." :) Dropping out of or below the cluster layer inherently means there is something in K8s that is missing. But no matter what is added to its APIs, Kubernetes will experience a long tail of weird corner cases that just can’t be covered. Reportedly the same is true with Borg jobs internally within Google. Teleport's K8s protocol implementation mates its certificate-based SSH auth with Kubernetes certificate signing request API, plus Teleport acts as a full recording MITM for all K8s apiserver requests. How exactly end-users will leverage this remains to be seen, but we're super excited to see where the community takes this (and related tricky K8s vs provider IAM issues) as it continues to expand. Full disclosure -- I work at Gravitational, worked closely with Sasha on this feature[1], and am a fan of Kelsey Hightower's live demo keynotes[2] where he frequently quips about the inherent struggle between procedural Dev+Ops encountering declarative APIs such as K8s. [1] https://github.com/gravitational/teleport/issues/1986 https://github.com/gravitational/teleport/issues/1986 [2] https://www.youtube.com/watch?v=07jq-5VbBVQ https://www.youtube.com/watch?v=07jq-5VbBVQ

outworlder 8y ago
> Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. Agreed. In general, every time you need to SSH anywhere, it means that there is some tooling you are missing. In very rare cases, you do need to break the glass to deal with exceptional conditions, but this should be rare.
malkia 8y ago
AFAIR, "debugging", at least in java allowed you to set a "breakpoint" in codesearch, but essentially something magically was happening behind, where if the CPU goes through this line you've marked as breakpoint it'll remember the surroundings (like variables in that scope, etc). Furthermore, one could declare (a bit like "live" coding) a one-liner (say in java) that would get called if that lines passes - so you can log something in better fashion, etc. But it had limitations. First I didn't get it, then it sunk into me - once you have thousands of jobs - it's pointless to try and debug one of them (thousands, as in MapReduce thousands). Not that I've used the feature a lot, but tried it several times and worked fine. It had nice integration with codesearch, where you can, given your "namespace" and "job" name, can set the codesearch to be "synced" to the CL it built your job + cherrypicks (I guess) - so the code you are looking is the code that your binary got compiled (another benefit of monolith depots, is that such things are easier there, or maybe it'll take a bit more work to get it working with multiple repos). E.g. magic :)

feydaykyn 8y ago

There's no pricing available for the enterprise edition, does someone know (roughly) how much we are talking about? Thanks!

outworlder 8y ago
I don't have a clue and would also like to know. Whenever there is a 'contact us' pricing page, I don't even bother. I can't even tell my boss to look into it if I don't know what the required budget will be, and I won't give out information to get contacted by sales teams and waste my time on a project that I wasn't even allocated time for. Plus it won't even be only my time, it will be finance and maybe even legal. So now there are multiple people involved to get a quote before it's even figured out if the solution fits the budget or if it is even a fit for the problem (unless there is some sort of trial). Whenever a project does this, it had better be the only thing capable of dousing a burning fire to get even considered.
- gk1 8y ago
  Enterprise software is not one-size-fits-all like SaaS, and therefore can't have one-size-fits-all pricing like SaaS. If seeing "contact us" scares you then you're probably not the target audience.
  - outworlder 8y ago
    It doesn't 'scare' me as such. It just upgrades the task from 'can I quickly evaluate this, determine possible fit, and forward the recommendation to higher ups' to assembling a small task force. EDIT: I understand that this is a common practice and a legitimate business model, so it's not aimed at Gravitational specifically. It just causes people to sometimes dismiss offerings that they would be legitimately interested in, otherwise. For instance, Slack offers paid versions that are not yet at the Enterprise level(so less features), but you can at least get an idea, and also what is the pricing model (is it per seat? per server? per gb of storage? etc).

madjam002 8y ago

How does this compare to the OIDC integration with Kubernetes which is supported natively out of the box? You can hook up an enterprise IdP like AD FS with that, without any additional software.

aberoham 8y ago
The biggest difference is that Teleport is a full MITM for all apiserver traffic, leveraging the cluster-local certificate authority for short-lived creds (similar to how K8s internal bits such as the kubelet speaks TLS to the apiserver). Notably this also works with completely air-gapped clusters or end-users, or clusters where you want access to apiserver to be via egress (hidden bastion) only. K8s authentication methods that go inline with the apiserver, such as oidc or dex, achieve the same goals but are a bit of a different design flavor and arguably less clean from a threat or failure model perspective.
- madjam002 8y ago
  How is OIDC less clean from a threat/failure model perspective? If your organisation has implemented an OIDC IdP, you can have the API server behind a Bastion like you suggest, and accessible through an authenticating reverse proxy (Beyondcorp style) WAF. You can even have token bound access/ID tokens which are backed by hardware TPMs if you're REALLY paranoid (upcoming spec). Plus because it's an open standard which K8s heavily embraces, you get rich group/user claims mapping in K8s allowing for logging and auditing.
old-gregg 8y ago
The obvious benefit is that synchronizes Kubernetes auth with SSH access, i.e. you don't have to authenticate twice if your organization uses SSH-based tools and k8s-based tooling, moreover it extends the same RBAC rules to both SSH and k8s, i.e. "developers must not touch production data" will be enforced for both "ssh production-host" and "kubectl exec production-pod". The second, perhaps less obvious, benefit is Teleport's built-in audit as shown in the video, i.e. all your interactions with all Kubernetes clusters are logged in one centralized place for audit purposes (and, in the case of the interactive sessions) can be replayed in a Youtube-like player. But yes, if you don't need the audit, and you don't use SSH, and if you only have one k8s cluster, this may be an overkill.

raesene9 8y ago

I'll be interested to see how this plays out. AuthN for users in k8s clusters is a bit of a pain point from what I've seen in a number of reviews I've done. The in-built options (Basic auth, token auth, client cert) are not really scalable for a larger number of users, and the other options (webhook, OIDC) can be complex to setup. So any relatively easy to use options for this would be welcome.

SEJeff 8y ago
OIDC tied to a LDAP directory is generally how you setup authentication for larger numbers of users. It really isn't that hard and has a dozen or so solutions with sensible config and operational footprints. Here is an example of how to do this (OIDC). Notice the short and concise documentation on how to set this up. It isn't too bad at all if you're running your own apiserver already: https://github.com/heptiolabs/gangway https://github.com/heptiolabs/gangway