5 ms·

> Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without produ

by tpetry 4y ago

> Don't get me wrong, I still like it but I wouldn't want to run it with little expertise at hand. It's not something I would trust someone to run without production experience, which is difficult because there are very few people out there who are battle hardened past trivial home deployments and tiny little stacks. That‘s the problem. Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. But when anything does not work correctly the only approach is to kill the complete machine and start a new cluster on a server. Have fun with statefull data which needs to be copied…

iasay 4y ago
We sort of waltzed around that one with PV's on EBS on Amazon and also shifted the control plane to them. But there are still some serious problems in that space.
- sleepybrett 4y ago
  A limitation of EBS on amazon that I've run into a few times is that EBS volumes must connect to EC2 in their zone only. So if your k8s cluster has nodes across multiple AZs (which obv. important) your pod that mounts that pv will always be zone locked. This can also be problematic if you write a pod that mounts claims that are in different zones, that will never work. There are also amazon limitations to how many volumes per node and I used to see problems with ebs volumes 'unmounting' from nodes getting stuck. The later was always problematic and required an admin with a hammer. However i've never seens a 'kube' problem per se, they've all been aws problems.
  - nijave 4y ago
    Resource zone affinity is annoying but usually works fine if you have 1 ASG per zone (Azure and afaik GCP are the same way). You mainly need to be careful during initial creation to make sure the volumes are spread out WaitForFirstConsumer is very helpful, too (you basically guarantee the pod can be scheduled at least once before creating the resources which greatly improves the likelihood it can get rescheduled in the future without getting stuck) Don't remember seeing any issues on AWS but I've seen Azure CSI take about 7.5 minutes to unmount a volume from 1 and mount to another (so each pod in a statefulset can take around 8-10 minutes)
throwaway894345 4y ago
It's not like you can let someone run bare VMs without any production experience either. You need a PaaS for something like that, even then... > Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. This hasn't been my experience, and certainly the Kubernetes project doesn't advertise the software as "so simple anyone can do it" or any such thing. Kubernetes definitely requires experience--it's not a PaaS, but a framework on which something like a PaaS could be built.

by tpetry 4y ago

iasay 4y ago

We sort of waltzed around that one with PV's on EBS on Amazon and also shifted the control plane to them. But there are still some serious problems in that space.

sleepybrett 4y ago
A limitation of EBS on amazon that I've run into a few times is that EBS volumes must connect to EC2 in their zone only. So if your k8s cluster has nodes across multiple AZs (which obv. important) your pod that mounts that pv will always be zone locked. This can also be problematic if you write a pod that mounts claims that are in different zones, that will never work. There are also amazon limitations to how many volumes per node and I used to see problems with ebs volumes 'unmounting' from nodes getting stuck. The later was always problematic and required an admin with a hammer. However i've never seens a 'kube' problem per se, they've all been aws problems.
- nijave 4y ago
  Resource zone affinity is annoying but usually works fine if you have 1 ASG per zone (Azure and afaik GCP are the same way). You mainly need to be careful during initial creation to make sure the volumes are spread out WaitForFirstConsumer is very helpful, too (you basically guarantee the pod can be scheduled at least once before creating the resources which greatly improves the likelihood it can get rescheduled in the future without getting stuck) Don't remember seeing any issues on AWS but I've seen Azure CSI take about 7.5 minutes to unmount a volume from 1 and mount to another (so each pod in a statefulset can take around 8-10 minutes)

throwaway894345 4y ago

It's not like you can let someone run bare VMs without any production experience either. You need a PaaS for something like that, even then... > Everyone used something like minicube to bring a kubernetes cluster online and believes it is simple. This hasn't been my experience, and certainly the Kubernetes project doesn't advertise the software as "so simple anyone can do it" or any such thing. Kubernetes definitely requires experience--it's not a PaaS, but a framework on which something like a PaaS could be built.