6 ms·
Kubesafe: Never run Kubernetes commands on the wrong cluster again
- robertlagrant 2y agoThis seems good, but can it also be done via ACLs in vanilla Kubernetes?
- Telemaco019 2y agoThanks Robert! Yes, you can achieve this with ACLs in Kubernetes, but it requires setting up multiple Roles and contexts. Even then, you might accidentally switch to a higher-permission Role and accidentally run a risky command, thinking you're in a different cluster or using a low-permission user. Kubesafe is just an extra safety net to prevent those kind of accidents :)
- robertlagrant 2y agoThat makes sense - thanks for the reply.
- JohnMakin 2y agoI am not trying to shit on this, sorry - but can't you achieve the same thing with rudimentary automation, and barring that, rudimentary scripting? This seems to just be adding y/n prompts to certain contexts. How's that different than a bash wrapper script that does something like this? context=$(grep "current-context:" ~/.kube/config | grep "*prod*") if [[ -z ${context} ]] then # do the command else # do a y/n prompt fi Am I missing something?
- Telemaco019 2y agoThanks for the feedback John! You're right, that's pretty much it :) I developed kubesafe because (1) I was tired of tinkering with shell aliases and scripts (especially when I wanted to define protected commands) and (2) I needed something that worked smoothly with all Kubernetes tools like kubectl, helm, kubecolor, etc. Kubesafe is just a convenient way to manage protected commands and contexts. Nothing too fancy! Btw - I also found a kubectl plugin written in Bash that’s similar to what you mentioned, in case you're interested: https://github.com/jordanwilson230/kubectl-plugins/blob/krew/kubectl-prompt https://github.com/jordanwilson230/kubectl-plugins/blob/krew...
- JohnMakin 2y agothanks for the explanation, I like the idea
- Telemaco019 2y agoYou're welcome! And thanks again for the feedback!
- thewisenerd 2y agohaha i added the following to my bashrc a few days ago for similar reasons; this forces me to be explicit about the cluster; now i mess up the wrong namespace instead :) if [[ -e "/opt/homebrew/bin/kubectl" ]]; then /opt/homebrew/bin/kubectl config unset current-context >/dev/null fi
- ed_mercer 2y agoI got burned by this recently and came to the conclusion that the concept of a current context is evil. Now I always specify —-context when running kubectl commands.
- cyberpunk 2y agoI actually go a step further and keep multiple kubeconfigs and have a load of shell aliases for managing them. Active one is in $PS1 somewhere.
- lukaslalinsky 2y agoI also got burned by this, pretty badly, and ever since it happened, I don't even have a default kubeconfig, have to specify it for every single kubectl run.
- vdm 2y agoThis is the way. Same for awscli profiles.
- cryptonector 2y agoI never even set up a default context. I sussed out that problem from the get-go and always use `--context`. But that's not really enough if you use shell history, or if your clusters differ in few letters that are easy to typo.
- physicles 2y agoCheck out direnv, and use a shell alias for kubectl. And yeah, current context is evil.
- 8organicbits 2y agoThis is a good suggestion, but keep in mind that you can accidentally run a command in the wrong directory. I've certainly done that too, with painful results.
- evnix 2y agoHave been burnt by this, I have to deal with close to 8 clusters and it is very easy to make a mistake. Would highly recommend kubie, it allows you to switch and shows you the name of the cluster in the prompt. It's probably a more visual way of solving the same problem. https://github.com/sbstp/kubie https://github.com/sbstp/kubie
- Yasuraka 2y agoI do this with kubectx to switch and kube-ps1 with ohmyzsh to display cluster/namespace in my usual prompt
- glitchcrab 2y ago'close to 8 clusters' is a strange turn of phrase. So you manage 6 or 7?
- glitchcrab 2y agoI toyed with the idea of having a kubeconfig per cluster some time ago, but I work with 10s of clusters on a daily basis (often with multiple terminals targeting the same cluster) and having to auth every single time would have been too much of a pain. Instead I went with kubeswitch which still gives you a different kubeconfig per terminal but allows you to re-use existing sessions. https://github.com/danielfoehrKn/kubeswitch https://github.com/danielfoehrKn/kubeswitch
- arccy 2y agowhether a reauth is necessary depends on your k8s setup a lot of the cloud ones only configure kubeconfig to call an external command, which can share auth state between terminals
- renewiltord 2y agoSomeone here showed me this cool technique with `fzf`: #!/usr/bin/env bash set -e context=$(kubectl config get-contexts | awk '{print $2;}' | grep -v NAME | fzf --preview 'kubectl config use-context {} && kubectl get namespaces') kubectl config use-context $context You get a two-pane window with the context on the left and the namespaces on the right. That's all I need to find what I'm looking at. It's destructive, though.
- smlx 2y agoI came up with a simpler solution that keeps kube contexts separated per terminal. https://smlx.dev/posts/kubectl-global-state/ https://smlx.dev/posts/kubectl-global-state/
- cduzz 2y agoIn the early 1990s I ran a math department's 4 servers and 50 workstations and (with a few exceptions) only ever did administrative actions through scripts. I've worked in lots of places since and the world's matured from scripts and rsync to ansible and puppet and similar. Have we regressed to the point where we've turned big clusters of systems back into "oops I ran a command as superuser in the wrong directory" ?
- physicles 2y agoWe’ve basically solved this where I work, with these steps: - Each environment gets its own directory. We use kustomize to share config between environments. - direnv “sets” the current context when you cd under a cluster’s directory (it sets an environment variable that a kubectl alias uses. Nobody calls kubectl directly; it wouldn’t work because we’ve banned the current context from the yaml files). You switch clusters by changing to that cluster’s directory. - most of the time, the only command you run is ‘make’ which just does kubectl kustomize apply (or whatever). 100% of cluster config is checked into git (with git-crypt for secrets), so the worst that can happen is that you reapply something that’s already there. I’ve also colored the command prompt according to the current cluster. But anyway it’s essentially impossible to apply a config change to the wrong cluster. I haven’t worried about this in years.
- noja 2y agowhat if you forget to run direnv the second time?
- ijustlovemath 2y agoI imagine they use an alias or bash function for cd, which uses direnv under the hood
- physicles 2y agoYep. Direnv inserts itself into your prompt function.
- Gooblebrai 2y agoYou don't have to run anything. The point of direnv is that loads/unloads automatically when your enter/leave a directory
- decasia 2y agoI like to print the k8s context and current namespace in the shell prompt. It's still possible I could mess something up with kubectl, but it provides constant reminders of what I'm working with.
- Telemaco019 2y agoI also have it in my zsh config, but that didn’t stop me from screwing up in the past. Having an active confirmation prompt for potentially risky commands is what works best for me
- coding123 2y agoAnother option, just give prod's creds to CI only.
- Telemaco019 2y agoI think it’s a tradeoff between safety and speed. Having only the CI/CD with production access can significantly slow you down, especially in the early stages when you’re focused on the product and still building out your tooling/infrastructure.
- acedTrex 2y agoI handle this by never keeping production kubeconfigs on my local device. i pull them down on demand.
- adolph 2y agoDon't keep anything in the default .kube/config. Set KUBECONFIG envar instead. Keep every cluster in separate config. Set an indicator in PS1. Helm et al follow the envar. Roast my zsh: k8x() { export env=$1; # exit if no param if [ -z $1 ]; then if [ -z ${KUBECONFIG+x} ]; then echo "Need param of a k8s environment"; return 1 else echo "Removing KUBECONFIG variable"; PS1="$(echo "$PS1" | sed -e 's;^([^)]*) ;;')"; unset KUBECONFIG; return 0 fi fi; # exit if no file for param cfgPath="$HOME/.kube/config.${env}"; if [ ! -f $cfgPath ]; then echo "A config does not exist"; return 1 fi; PS1="$(echo "$PS1" | sed -e 's;^([^)]*) ;;' -e 's;^;('$env') ;')"; export KUBECONFIG="$cfgPath"; }
- millerm 2y agoHah! I accidentally deleted a production deployment the other day, because I thought it was mucking with my local Colima Kubernetes's cluster. I forgot that I had my context set to one of my AWS clusters. I had been meaning to write a command to wrap helm and kubectrl to prompt me with info before committing, so I will have to take a peek at this.
- jasonhansel 2y agoCan you use this with kubecolor? https://github.com/kubecolor/kubecolor https://github.com/kubecolor/kubecolor Incidentally: I have no idea why something like kubecolor isn't built in to kubectl itself.
- Telemaco019 2y agoAbsolutely! kubesafe is simply a wrapper, so you can use it with any Kubernetes tool by passing the tool as the first argument to kubesafe. Example with kubecolor: `kubesafe kubecolor get pods --all-namespaces`