3 ms·

For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations t

For less complex options in modern k8s, you could configure Pod Security Admission on a namespace basis. It's not too complex as you just add some annotations to a namespace to activate it. The baseline policy will disable hostPath volumes which helps a bit, alongside quite a few other restrictions.

vbezhenar 3y ago
I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things. I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking. One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it. There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it. https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/ https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about. It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.
- raesene9 3y ago
  yeah if you give a user namespace edit access, then PSA is not going to help you (apart from that default config option you listed, but then even that is tricky as you need to know the namespaces to exempt like kube-system and others) Kubeadm is a pretty barebones option, which I think needs work to scale (you're on the hook to add all the bells and whistles to create a full k8s environment) There's validating admission policy coming along (https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/ https://kubernetes.io/docs/reference/access-authn-authz/vali...) beta in 1.28, but you still need the policies to apply. Really for a more "batteries included" kind of cluster, you'd need one of the larger distributions that tries to do more out of the box. Obviously OpenShift is the ultimate example of that (for on-prem) but it massively increases the complexity of the environment.

vbezhenar 3y ago

I'm aware about that, but the problem is: it's just some kind of good will. First of all: you need to actually add those annotations to the namespace. If someone adds his own namespace, it won't be there. Second: anyone can just remove those annotations (without complicated RBAC rules) and do hostpath and other things. I'm not talking about hackers, but just some less experienced people who might read that advice on forums or chats and decided to just proceed without asking. One can use kyverno to enforce those namespace, but it's so heavyweight solution, I actually deployed it and it was terrible decision, so now I have to plan to decommission it. There's mechanism to actually enforce default baseline policy, so even if it's not set on a namespace, it'll be "baseline". However to configure that mechanism one need to configure admission controller. But good luck configuring it with kubeadm. It requires some cryptic YAMLs, YAML patches and whatnot. You can't just edit some configmap in the cluster and be done with it. https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/ https://kubernetes.io/docs/tasks/configure-pod-container/enf... this is what I'm talking about. It's more issue with kubeadm, I guess, which really makes things harder than they should be. I can't just run kubeadm join control-plane, I need to craft configuration, add this AdmissionConfiguration to it, remember to update it on all control nodes if I decide to change it later.

raesene9 3y ago
yeah if you give a user namespace edit access, then PSA is not going to help you (apart from that default config option you listed, but then even that is tricky as you need to know the namespaces to exempt like kube-system and others) Kubeadm is a pretty barebones option, which I think needs work to scale (you're on the hook to add all the bells and whistles to create a full k8s environment) There's validating admission policy coming along (https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/ https://kubernetes.io/docs/reference/access-authn-authz/vali...) beta in 1.28, but you still need the policies to apply. Really for a more "batteries included" kind of cluster, you'd need one of the larger distributions that tries to do more out of the box. Obviously OpenShift is the ultimate example of that (for on-prem) but it massively increases the complexity of the environment.