3 ms·

Debug containers have been a bit of a let down in UX. It's rather difficult to do sort of basic stuff, and a lot of stuff is hidden behind flags. E.g., if you

Debug containers have been a bit of a let down in UX. It's rather difficult to do sort of basic stuff, and a lot of stuff is hidden behind flags. E.g., if you just sort of roll with the defaults, you're dropped into a pod in a very confused state: `ps -ef` says nothing in your debugee is running, and the filesystem of the debugee is nowhere to be found. You can work around both of those (the first is --target, but the latter requires an intricate SO answer¹) but its the sort of thing that would be nicer out of the box? The node debugging mode is a bit better: by default, puts you in at least the host pidns, and mounts the host FS. ¹https://stackoverflow.com/questions/73355970/how-to-get-access-to-filesystem-with-kubectl-debug-ephemeral-containers https://stackoverflow.com/questions/73355970/how-to-get-acce...

bravetraveler 3y ago
As someone who doesn't do much (any) k8s... Seems only marginally better than using nsenter on a privileged container to just go muck with the host
freedomben 3y ago
I've also been disappointed with debug containers. They are often not useful for debugging trick production-only issues because so many of those issues are related to container state, which can be (often is) different inside the container. Certain languages/platforms and developer discipline are better about this than others, like if you're using functional/immutable languages then it's less of an issue. For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.
solatic 3y ago
> UX I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments: * Rotating secrets without exposing the secret to the shell history file (hint: kubectl apply -f can take - to signify atdin, but not kubectl patch!) * Ensuring your edits to a ConfigMap pass application-level validation (i.e. your configuration changes won't crash your app, not just that it's a valid ConfigMap) * Anything to do with user auth or RBAC * Scaling the default persistent volume size of a StatefulSet The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.
- arve0 3y ago
  > without exposing the secret to the shell history file Any command in shell with a space before it will be omitted from history. Agree it should take input from stdin.
  - sirn 3y ago
    This requires `setopt HIST_IGNORE_SPACE` (zsh) or `HISTCONTROL=ignorespace`/`HISTCONTROL=ignoreboth` (bash/ksh) to be set and may not be enabled by default in many distros (e.g. NixOS doesn't, Alpine doesn’t, etc). Always check your shell before assuming this will work!

bravetraveler 3y ago

As someone who doesn't do much (any) k8s... Seems only marginally better than using nsenter on a privileged container to just go muck with the host

freedomben 3y ago

I've also been disappointed with debug containers. They are often not useful for debugging trick production-only issues because so many of those issues are related to container state, which can be (often is) different inside the container. Certain languages/platforms and developer discipline are better about this than others, like if you're using functional/immutable languages then it's less of an issue. For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.

solatic 3y ago

> UX I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments: * Rotating secrets without exposing the secret to the shell history file (hint: kubectl apply -f can take - to signify atdin, but not kubectl patch!) * Ensuring your edits to a ConfigMap pass application-level validation (i.e. your configuration changes won't crash your app, not just that it's a valid ConfigMap) * Anything to do with user auth or RBAC * Scaling the default persistent volume size of a StatefulSet The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.

arve0 3y ago
> without exposing the secret to the shell history file Any command in shell with a space before it will be omitted from history. Agree it should take input from stdin.
- sirn 3y ago
  This requires `setopt HIST_IGNORE_SPACE` (zsh) or `HISTCONTROL=ignorespace`/`HISTCONTROL=ignoreboth` (bash/ksh) to be set and may not be enabled by default in many distros (e.g. NixOS doesn't, Alpine doesn’t, etc). Always check your shell before assuming this will work!