4 ms·

Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. At KubeCon CPH,

Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. At KubeCon CPH, when I asked Tim Hockin about the long-awaited debug containers feature, aka 'kubectl debug', he said that "I really do feel that every time someone has to SSH into a node a ferry loses its wings." :) Dropping out of or below the cluster layer inherently means there is something in K8s that is missing. But no matter what is added to its APIs, Kubernetes will experience a long tail of weird corner cases that just can’t be covered. Reportedly the same is true with Borg jobs internally within Google. Teleport's K8s protocol implementation mates its certificate-based SSH auth with Kubernetes certificate signing request API, plus Teleport acts as a full recording MITM for all K8s apiserver requests. How exactly end-users will leverage this remains to be seen, but we're super excited to see where the community takes this (and related tricky K8s vs provider IAM issues) as it continues to expand. Full disclosure -- I work at Gravitational, worked closely with Sasha on this feature[1], and am a fan of Kelsey Hightower's live demo keynotes[2] where he frequently quips about the inherent struggle between procedural Dev+Ops encountering declarative APIs such as K8s. [1] https://github.com/gravitational/teleport/issues/1986 https://github.com/gravitational/teleport/issues/1986 [2] https://www.youtube.com/watch?v=07jq-5VbBVQ https://www.youtube.com/watch?v=07jq-5VbBVQ

outworlder 8y ago
> Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. Agreed. In general, every time you need to SSH anywhere, it means that there is some tooling you are missing. In very rare cases, you do need to break the glass to deal with exceptional conditions, but this should be rare.
malkia 8y ago
AFAIR, "debugging", at least in java allowed you to set a "breakpoint" in codesearch, but essentially something magically was happening behind, where if the CPU goes through this line you've marked as breakpoint it'll remember the surroundings (like variables in that scope, etc). Furthermore, one could declare (a bit like "live" coding) a one-liner (say in java) that would get called if that lines passes - so you can log something in better fashion, etc. But it had limitations. First I didn't get it, then it sunk into me - once you have thousands of jobs - it's pointless to try and debug one of them (thousands, as in MapReduce thousands). Not that I've used the feature a lot, but tried it several times and worked fine. It had nice integration with codesearch, where you can, given your "namespace" and "job" name, can set the codesearch to be "synced" to the CL it built your job + cherrypicks (I guess) - so the code you are looking is the code that your binary got compiled (another benefit of monolith depots, is that such things are easier there, or maybe it'll take a bit more work to get it working with multiple repos). E.g. magic :)

outworlder 8y ago

> Kubernetes projects leads would argue that any time a user has to SSH into one of their K8s workers, they should file a bug against Kubernetes. Agreed. In general, every time you need to SSH anywhere, it means that there is some tooling you are missing. In very rare cases, you do need to break the glass to deal with exceptional conditions, but this should be rare.

malkia 8y ago

AFAIR, "debugging", at least in java allowed you to set a "breakpoint" in codesearch, but essentially something magically was happening behind, where if the CPU goes through this line you've marked as breakpoint it'll remember the surroundings (like variables in that scope, etc). Furthermore, one could declare (a bit like "live" coding) a one-liner (say in java) that would get called if that lines passes - so you can log something in better fashion, etc. But it had limitations. First I didn't get it, then it sunk into me - once you have thousands of jobs - it's pointless to try and debug one of them (thousands, as in MapReduce thousands). Not that I've used the feature a lot, but tried it several times and worked fine. It had nice integration with codesearch, where you can, given your "namespace" and "job" name, can set the codesearch to be "synced" to the CL it built your job + cherrypicks (I guess) - so the code you are looking is the code that your binary got compiled (another benefit of monolith depots, is that such things are easier there, or maybe it'll take a bit more work to get it working with multiple repos). E.g. magic :)