5 ms·

Kubectl – Configuration Guide

72 points by lukasbar 6y ago

mlthoughts2018 6y ago
My strong advice is do not ever rely on use-context or any similar abilities to change implicit state. You should absolutely always be typing out huge, verbose commands like kubectl —context my-context -n my-namespace <some commands> ... The opportunity for tragic error when you have implicit context or namespace settings applied is just too great.
- longcommonname 6y ago
  Or you can signify the context using shell prompt and use kube rbac to give read only permissions instead of letting people mess with your prod cluster.
  - mlthoughts2018 6y ago
    Managing shell tooling to edit ps1 for this is a bad idea, because there are already many other much longer established mutations people use that for (like git branch or activated Python environment), kubernetes can’t come along decades later and suggest to use the tool effectively one has to break those long standing workflows. It’s always an available option for people who want it, but many many use cases are poorly served by it. The RBAC thing is different. I am 100% for those RBAC limitation as long as I have total authority to make a devops admin do what I need them to do right when I need them to do it. If my team is going to get alerted on build failures or service errors that require kubernetes mutation actions or exec to debug or resolve (99% of the time they do), then my team needs full admin access. If rbac prevents access, then you better route the alerts to someone else or give us authority to compel instant triage responses. In the 3 different companies I’ve worked in that use large kubernetes clusters, rbac has been a miserable problem across the board, because SREs don’t want to be responsible for triaging or resolving application team issues, and application teams will not honor alerts that are not actionable because of poorly conceived permissions issues where people mistakenly think they are setting useful access control policy but really they aren’t.
    - DasIch 6y ago
      You could have a mechanism to grant yourself temporary write access on an as-needed basis. At Zalando everyone has read access (except to secrets). You can request write access with a command line tool another employee can then approve it. There is also an option to request access with an incident ticket, in that case you immediately get write access without approval by someone else. Write access expires after 1h. Access to the underlying AWS account uses the same mechanism.
      - mlthoughts2018 6y ago
        But then when you’ve unlocked access, you could just make the same fat-fingered mistakes due to implicit use-context settings. You might use use-context to set yourself to that restricted production context at the start of an incident. 30 minutes later you resolved the incident, but the access is still active and whoops you delete a deployment you thought you were deleting from a stage experiment or something. All that does is add an extra hoop to jump through. People need to stop believing that adding extra bureaucratic hoops offers any type of safety - it doesn’t. If you want to restrict access then you need to actually restrict it and convert on-call alert responsibilities to a central devops team that, like it or not, is responsible for solving everyone else’s problems. If you don’t want a central devops team that can be paged and on the hook for everyone else’s systems, then you cannot have write access controls. There’s just no way out of this dilemma. Temporary access that can be self-granted == no access control. Temporary access that requires admin approval == admin team is on pager duty for every other team.
- rtempaccount1 6y ago
  kube-ps1, kubectx and kubens can help quite a bit here. kube-ps1 to give you a visual indication of current context and the others to make switching easier. You can also explicitly add namespaces to manifests, which help avoid applying to the wrong namespace.
  - jdellinger 6y ago
    While kube-ps1 helps, remember that the context is set session-wide. So even if your last command says "development" in one terminal, another terminal may have modified the context by now. So make sure to spam the return key before deleting stuff :D
    - minaguib 6y ago
      Agreed, FWIW this should be a habit for any PS1-modifiers (like git repo status)
- joeskyyy 6y ago
  Agreed. With autocomplete I find myself being able to make those long commands painless while making sure I'm being explicit. Only took me one time running the wrong command against the wrong cluster to drill that one into my brain forever haha
kissgyorgy 6y ago
I find the kubectx and kubens tools invaluable for switching contexts and namespaces quickly: https://github.com/ahmetb/kubectx https://github.com/ahmetb/kubectx
- echelon 6y ago
  These are amazing! Thanks for sharing.
- neximo64 6y ago
  I've always been confused with kubectx. How do I actually add a seperate yaml file (say the one I download with my browser from DigitalOcean) into the context?
  - kissgyorgy 6y ago
    You should read the article: https://knowledgepill.it/posts/kubernetes-kubectl-client-config/#multiple-config-files-and-kubeconfig-variable https://knowledgepill.it/posts/kubernetes-kubectl-client-con...
- remram 6y ago
  I've just been using this function in my bashrc: kubea(){ if [ "$#" = 1 ]; then KUBECONFIG=~/.kube/configs/"$1" else ls -1 ~/.kube/configs/ fi } That way I can "activate" a specific config the same way I activate my Python virtualenv, for a specific terminal. I also add the basename of the config to my prompt. kubea myproject-PROD
  - Nullabillity 6y ago
    Or, to avoid the persistent state.. function dkube env KUBECONFIG=$HOME/.kube/config-dev $argv end function dkctl dkube kubectl $argv end
    - remram 6y ago
nunez 6y ago
Does anyone else hate that we have to deal with Kubeconfigs at all?
- freedomben 6y ago
  It's a mixed bag. On one hand it's a pain when you have to switch back and forth. On the other hand it's awesome that all the state is in one, human-readable file and can easily be duplicated or pointed somewhere else, etc. I primarily use OpenShift which fixes part of it, and with a few bash aliases the rest are pretty painless as well.
- pgwhalen 6y ago
  I’m a little confused, what would the alternative be?
  - kobalsky 6y ago
    What I have been doing is a directory per namespace/context. I have a common Makefile that I link there which creates the namespace and does all the basic initialization stuff like inserting deny-all netpol rules, credentials for our docker registry. It also runs a puppet task that creates scripts to use kubectl and helm with the correct context. An optional config file let's me change the defaults, like pointing it to a different cluster. I also keep context specific notes and files, scripts, etc. in this directory. Everything under version control, except for the autogenerated stuff. For normal usage I always just run ./kubectl.sh. I switch between contexts by changing directories and the commands to get pods and talk with the cluster don't change. I just go nuts with ctrl+r. I also can run commands on different contexts easily, lets me pipe the output from one context into another just running: namespace1/kubectl.sh exec ... | namespace2/kubectl2.sh run ... namespace1/mysqldump.sh mydb | namespace2/mysql.sh myotherdb Each kubectl.sh points to the correct version of kubectl for each cluster, so you don't get weird kubectl errors from mixing stable clusters with more modern ones. The config file being a puppet script is a very powerful tool. I use it to for example generate the config files for our software and the yaml files for helm. Also for doing the vendor specific stuff like allocating volumes, ips and configuring DNS servers.
  - 6y ago
jaimehrubiks 6y ago
I like to have one config file per cluster and point kubeconfig to all these files. Then I use a bash function to select the cluster by name. My zsh shell previews the cluster name on the right when I start typing k, kubectl, helm and similar... So it is harder for me to make a mistake
- jml78 6y ago
  starship.rs will show your current context all the time. Personally I have that and the git plugins configured
- mastegizmo 6y ago
  I think this is the Best method - it is quite safe, I’m doing the same
anurag 6y ago
Kubie makes it easier (and safer) to manage multiple contexts because it makes each shell independent: https://github.com/sbstp/kubie https://github.com/sbstp/kubie
pepemon 6y ago
What does exactly this post add to what have been already described in the official documentation[1]? [1] https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ https://kubernetes.io/docs/tasks/access-application-cluster/...
deleted 6y ago
[deleted]

72 points by lukasbar 6y ago

mlthoughts2018 6y ago

My strong advice is do not ever rely on use-context or any similar abilities to change implicit state. You should absolutely always be typing out huge, verbose commands like kubectl —context my-context -n my-namespace <some commands> ... The opportunity for tragic error when you have implicit context or namespace settings applied is just too great.

longcommonname 6y ago
Or you can signify the context using shell prompt and use kube rbac to give read only permissions instead of letting people mess with your prod cluster.
- mlthoughts2018 6y ago
  Managing shell tooling to edit ps1 for this is a bad idea, because there are already many other much longer established mutations people use that for (like git branch or activated Python environment), kubernetes can’t come along decades later and suggest to use the tool effectively one has to break those long standing workflows. It’s always an available option for people who want it, but many many use cases are poorly served by it. The RBAC thing is different. I am 100% for those RBAC limitation as long as I have total authority to make a devops admin do what I need them to do right when I need them to do it. If my team is going to get alerted on build failures or service errors that require kubernetes mutation actions or exec to debug or resolve (99% of the time they do), then my team needs full admin access. If rbac prevents access, then you better route the alerts to someone else or give us authority to compel instant triage responses. In the 3 different companies I’ve worked in that use large kubernetes clusters, rbac has been a miserable problem across the board, because SREs don’t want to be responsible for triaging or resolving application team issues, and application teams will not honor alerts that are not actionable because of poorly conceived permissions issues where people mistakenly think they are setting useful access control policy but really they aren’t.
  - DasIch 6y ago
    You could have a mechanism to grant yourself temporary write access on an as-needed basis. At Zalando everyone has read access (except to secrets). You can request write access with a command line tool another employee can then approve it. There is also an option to request access with an incident ticket, in that case you immediately get write access without approval by someone else. Write access expires after 1h. Access to the underlying AWS account uses the same mechanism.
    - mlthoughts2018 6y ago
      But then when you’ve unlocked access, you could just make the same fat-fingered mistakes due to implicit use-context settings. You might use use-context to set yourself to that restricted production context at the start of an incident. 30 minutes later you resolved the incident, but the access is still active and whoops you delete a deployment you thought you were deleting from a stage experiment or something. All that does is add an extra hoop to jump through. People need to stop believing that adding extra bureaucratic hoops offers any type of safety - it doesn’t. If you want to restrict access then you need to actually restrict it and convert on-call alert responsibilities to a central devops team that, like it or not, is responsible for solving everyone else’s problems. If you don’t want a central devops team that can be paged and on the hook for everyone else’s systems, then you cannot have write access controls. There’s just no way out of this dilemma. Temporary access that can be self-granted == no access control. Temporary access that requires admin approval == admin team is on pager duty for every other team.
rtempaccount1 6y ago
kube-ps1, kubectx and kubens can help quite a bit here. kube-ps1 to give you a visual indication of current context and the others to make switching easier. You can also explicitly add namespaces to manifests, which help avoid applying to the wrong namespace.
- jdellinger 6y ago
  While kube-ps1 helps, remember that the context is set session-wide. So even if your last command says "development" in one terminal, another terminal may have modified the context by now. So make sure to spam the return key before deleting stuff :D
  - minaguib 6y ago
    Agreed, FWIW this should be a habit for any PS1-modifiers (like git repo status)
joeskyyy 6y ago
Agreed. With autocomplete I find myself being able to make those long commands painless while making sure I'm being explicit. Only took me one time running the wrong command against the wrong cluster to drill that one into my brain forever haha

kissgyorgy 6y ago

I find the kubectx and kubens tools invaluable for switching contexts and namespaces quickly: https://github.com/ahmetb/kubectx https://github.com/ahmetb/kubectx

echelon 6y ago
These are amazing! Thanks for sharing.
neximo64 6y ago
I've always been confused with kubectx. How do I actually add a seperate yaml file (say the one I download with my browser from DigitalOcean) into the context?
- kissgyorgy 6y ago
  You should read the article: https://knowledgepill.it/posts/kubernetes-kubectl-client-config/#multiple-config-files-and-kubeconfig-variable https://knowledgepill.it/posts/kubernetes-kubectl-client-con...
remram 6y ago
I've just been using this function in my bashrc: kubea(){ if [ "$#" = 1 ]; then KUBECONFIG=~/.kube/configs/"$1" else ls -1 ~/.kube/configs/ fi } That way I can "activate" a specific config the same way I activate my Python virtualenv, for a specific terminal. I also add the basename of the config to my prompt. kubea myproject-PROD
- Nullabillity 6y ago
  Or, to avoid the persistent state.. function dkube env KUBECONFIG=$HOME/.kube/config-dev $argv end function dkctl dkube kubectl $argv end
  - remram 6y ago

nunez 6y ago

Does anyone else hate that we have to deal with Kubeconfigs at all?

freedomben 6y ago
It's a mixed bag. On one hand it's a pain when you have to switch back and forth. On the other hand it's awesome that all the state is in one, human-readable file and can easily be duplicated or pointed somewhere else, etc. I primarily use OpenShift which fixes part of it, and with a few bash aliases the rest are pretty painless as well.
pgwhalen 6y ago
I’m a little confused, what would the alternative be?
- kobalsky 6y ago
  What I have been doing is a directory per namespace/context. I have a common Makefile that I link there which creates the namespace and does all the basic initialization stuff like inserting deny-all netpol rules, credentials for our docker registry. It also runs a puppet task that creates scripts to use kubectl and helm with the correct context. An optional config file let's me change the defaults, like pointing it to a different cluster. I also keep context specific notes and files, scripts, etc. in this directory. Everything under version control, except for the autogenerated stuff. For normal usage I always just run ./kubectl.sh. I switch between contexts by changing directories and the commands to get pods and talk with the cluster don't change. I just go nuts with ctrl+r. I also can run commands on different contexts easily, lets me pipe the output from one context into another just running: namespace1/kubectl.sh exec ... | namespace2/kubectl2.sh run ... namespace1/mysqldump.sh mydb | namespace2/mysql.sh myotherdb Each kubectl.sh points to the correct version of kubectl for each cluster, so you don't get weird kubectl errors from mixing stable clusters with more modern ones. The config file being a puppet script is a very powerful tool. I use it to for example generate the config files for our software and the yaml files for helm. Also for doing the vendor specific stuff like allocating volumes, ips and configuring DNS servers.
- 6y ago

jaimehrubiks 6y ago

I like to have one config file per cluster and point kubeconfig to all these files. Then I use a bash function to select the cluster by name. My zsh shell previews the cluster name on the right when I start typing k, kubectl, helm and similar... So it is harder for me to make a mistake

jml78 6y ago
starship.rs will show your current context all the time. Personally I have that and the git plugins configured
mastegizmo 6y ago
I think this is the Best method - it is quite safe, I’m doing the same

anurag 6y ago

Kubie makes it easier (and safer) to manage multiple contexts because it makes each shell independent: https://github.com/sbstp/kubie https://github.com/sbstp/kubie

pepemon 6y ago

What does exactly this post add to what have been already described in the official documentation[1]? [1] https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/ https://kubernetes.io/docs/tasks/access-application-cluster/...

deleted 6y ago

[deleted]