7 ms·

It was year 2016, kubernetes did not have jobs, cronjobs, statefulsets. Pods would get stuck in terminating state or container creating state. Networking in kub

It was year 2016, kubernetes did not have jobs, cronjobs, statefulsets. Pods would get stuck in terminating state or container creating state. Networking in kubernetes was wonky. AWS did not have support for EKS. It used be painful. It is year 2021, 1000s of new startups around kubernetes, more features, more resource types. Pods would still get stuck in terminating state or container creating state. It still pretty painful.

HelloNurse 5y ago
> Networking in kubernetes was wonky. Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?
- krab 5y ago
  Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.
- jrockway 5y ago
  You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html; https://docs.aws.amazon.com/eks/latest/userguide/pod-network... each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).
- kaidon 5y ago
  Please take a seat and let the joys of k8s networking overwhelm your senses: https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n... And yes... Kubernetes network configuration is on a whole different level from docker networking.
  - theptip 5y ago
    To be fair, multi-node networking of any sort is on a different level than single-host docker networking. If you ever tried to use Docker Swarm to network multiple nodes, god help you. Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff. It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you. Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.
    - kazen44 5y ago
      > Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes. How is routing from the internet to multiple servers a problem? usually, you have either one of these setups: - you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP). - you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider. - you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you. Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).
      - theptip 5y ago
        I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2. I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.
- kazen44 5y ago
  the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology. Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance). Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).
jrockway 5y ago
I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to start start, and all the pods we intend to kill die -- even with autoscaling that provisions new instance types. Our biggest failure mode is TLS certificates failing to provision through Let's Encrypt, but that has nothing to do with Kubernetes (that is a layer above; what we run in Kubernetes). EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.
- shaicoleman 5y ago
  What are the EKS pain points?
  - jrockway 5y ago
    The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays. There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice. Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)
dehrmann 5y ago
Around 2015, I was at Spotify, and we were using a container orchestrator build in-house named Helios. They didn't build it because Kubernetes wasn't invented there; they built it because Kubernetes didn't exist, yet.
- lacion 5y ago
  kubernetes was released in 2014, in late 2015 I already had a cluster marked as released candidate that was put into production early 2016. so im guessing here that spotify wrote their own becouse they had a spesific requirement? nomad was also early days in 2015
  - dehrmann 5y ago
    I'm pretty sure the work started before Kubernetes was released, and even then, it wasn't clear that was going to be the de factor orchestrator.
halfmatthalfcat 5y ago
I have used Kubernetes extensively over the past couple years and have never seen pods stuck in terminating or creating state that didn't have to do with errors in container creation (your Dockerfile/bootstrapping is messed up) or issues with healthchecks.
CSDude 5y ago
We spawn thousands of pods per day for jobs and never get those stuck and it was not the case in 2018 either. Not sure what is it you are doing causes this.

HelloNurse 5y ago

> Networking in kubernetes was wonky. Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?

krab 5y ago
Yes, Kubernetes (actually its add-ons) provide a virtual network that unifies communication within the cluster so you don't need to care on which computer your service runs.
jrockway 5y ago
You can make it as complicated as you want it to be; part of setting up a cluster is picking the networking system ("CNI"). Cloud providers often have their own IPAM (i.e. on Amazon, you get this: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html; https://docs.aws.amazon.com/eks/latest/userguide/pod-network... each Pod gets an IP from your VPC, resulting in weird limits like 17 pods per instance because that's how many IP addresses you can have for that particular instance type throughout EC2).
kaidon 5y ago
Please take a seat and let the joys of k8s networking overwhelm your senses: https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/cluster-administration/n... And yes... Kubernetes network configuration is on a whole different level from docker networking.
- theptip 5y ago
  To be fair, multi-node networking of any sort is on a different level than single-host docker networking. If you ever tried to use Docker Swarm to network multiple nodes, god help you. Also worth noting that almost all users of K8s don't actually need to operate a cluster, the hosted offerings handle all of that for you. You just need to understand the Service object, and maybe Ingress if you're trying to do some more advanced cert management or API gateway stuff. It's a common meme around here to point in horror to the complexity that is abstracted away under the K8s cluster API, and claim that k8s is really hard to use. I think that's mostly misguided, the hosted offerings like GKE really do a good job of hiding away all that complexity from you. Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes.
  - kazen44 5y ago
    > Honestly I think that it's defensible to say that the k8s networking model is in most cases _simpler_ than what you'd end up configuring in AWS / GCP to route traffic from the internet to multiple VM nodes. How is routing from the internet to multiple servers a problem? usually, you have either one of these setups: - you run a loadbalancer that distributes traffic across your nodes. (This loadbalancer could even be distributed thanks to BGP). - you either run your own firewall or have a managed one, in which you either announce your IP prefix yourself, or they are announced for you by your uplink provider. - you run an anycast setup (for, for example, globally distributed DNS). and announce multiples of the same prefix across the globe. Routing in the DFZ does the rest for you. Streched L2 across the globe/internet is also possible (although not very performant) either by doing IPsec tunneling, or by buying/setting up L2VPN services. (either MPLS or VXLAN based).
    - theptip 5y ago
      I didn't say it was a problem. My claim was just that it's easier in GKE than in GCE/EC2. I only mentioned multi-node because exposing a single VM to the internet is trivial -- just give it a public IP -- and thus is not an apples-to-apples comparison with the multi-node load balancing that you get from the entry-level k8s configuration of Service > Pod < Deployment.
kazen44 5y ago
the largest issue with kubernetes networking seems to be the lack of integration with modern datacenter networking technology. Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance). Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).

jrockway 5y ago

I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to start start, and all the pods we intend to kill die -- even with autoscaling that provisions new instance types. Our biggest failure mode is TLS certificates failing to provision through Let's Encrypt, but that has nothing to do with Kubernetes (that is a layer above; what we run in Kubernetes). EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.

shaicoleman 5y ago
What are the EKS pain points?
- jrockway 5y ago
  The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays. There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice. Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)

dehrmann 5y ago

Around 2015, I was at Spotify, and we were using a container orchestrator build in-house named Helios. They didn't build it because Kubernetes wasn't invented there; they built it because Kubernetes didn't exist, yet.

lacion 5y ago
kubernetes was released in 2014, in late 2015 I already had a cluster marked as released candidate that was put into production early 2016. so im guessing here that spotify wrote their own becouse they had a spesific requirement? nomad was also early days in 2015
- dehrmann 5y ago
  I'm pretty sure the work started before Kubernetes was released, and even then, it wasn't clear that was going to be the de factor orchestrator.

halfmatthalfcat 5y ago

I have used Kubernetes extensively over the past couple years and have never seen pods stuck in terminating or creating state that didn't have to do with errors in container creation (your Dockerfile/bootstrapping is messed up) or issues with healthchecks.

CSDude 5y ago

We spawn thousands of pods per day for jobs and never get those stuck and it was not the case in 2018 either. Not sure what is it you are doing causes this.