5 ms·

I came here to post pretty much the same thing. Networking in k8s is the poorest designed aspect of Kubernetes. So much so, I'm honestly surprised it didn't ki

by nvarsj 6y ago

I came here to post pretty much the same thing. Networking in k8s is the poorest designed aspect of Kubernetes. So much so, I'm honestly surprised it didn't kill off K8s early on. You still can't get proper service layer load balancing out of the box w/ iptables. I'm not sure the history - it's the simplest thing possible, I guess, but doesn't perform very well at the most basic of tasks. To try and solve this, we have the current mess of service meshes, CNI, etc. It's going back to the middleware days of yore - which I think anyone with operational experience should know we really need to avoid. Having multiple layers of network proxies and masquerading between service calls, on an internal network, is just ridiculous and difficult to debug or operate at scale.

intricatedetail 6y ago
What rescued kubernetes was adoption by AWS and Google. If you want to host Kubernetes yourself it is still a waste of time broadly speaking.
- joana035 6y ago
  Kubernetes didn't need to be rescued in the first place. If you know how to manage any Linux box, setting up kubernetes is as easy as "# apt install kubelet kubectl". Broadly speaking kubernetes saves time by providing an api for existing Linux technology. For example, iptables, ipvs, storage, self-healing, virtual ips, etc via yaml. Kubernetes was created to give anyone "Google like, production grade" infrastructure without the vendor lock-in of the cloud. And kubernetes on physical machines is where it shines most. EDIT: it is very easy to complain that kubernetes is complex, but try to do virtual ips with keepalived, corosync, pacemaker to understand how much time kubernetes saves by providing the same capability in a generic way which is easy to use. I have the feeling that people take kubernetes for granted without knowing how much manual work one has to go through in order to build a similar experience using the standard building blocks available. EDIT 2: straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google" that is a lot of saved time just by using kubernetes.
  - deleted 6y ago
    [deleted]
  - geofft 6y ago
    Kubernetes "builds upon" Google's "experience," but it isn't what Google runs and doesn't totally reflect what Google actually does: https://twitter.com/copyconstruct/status/1253968037579030532 https://twitter.com/copyconstruct/status/1253968037579030532 https://twitter.com/copyconstruct/status/1254273432847446016 https://twitter.com/copyconstruct/status/1254273432847446016 If you happen to have the problems Kubernetes solves, it's great. But even Google doesn't have exactly those problems. (And there is a huge difference between setting up Kubernetes and hosting Kubernetes. If you know how to sit in a chair, you can be in a plane's pilot seat just fine, but piloting is an entirely different skill.)
    - joana035 6y ago
      Correct, kubernetes is a destilation of those 15 years of experience acquired running something else that also has network, storage and compute concepts. I didn't said Google uses kubernetes, though :) Still, kubernetes makes a wonderful job abstracting all the hard work of using the underlying tools. Look how simple is to create a ipvs round-robin by just defining a "kind: Service" object, try to do the same using the bare tools in a way that is synchronized across N computers.
      - gabereiser 6y ago
        I would argue it doesn’t go far enough, especially in networking/discovery. Kubernetes is incredibly useful but I personally want more out of the box than just abstractions over existing linux technologies.
        joana035 6y ago
        Networking just works. Discovery via DNS is pretty standard too. Mind to elaborate on what you would like to see in those areas?
api 6y ago
K8s networking is just typical networking. Networking always has to be made as complex and clunky as possible. Obvious solutions like unified IPv6 address space must be avoided as they would simplify the system and avoid the need for NAT. I’m being snarky but it is a long time observation of mine. When I see a network config I am more often than not shocked by its needless complexity.
- skywhopper 6y ago
  Folks who say stuff like this aren’t taking into account the number of real world constraints on networking at a large scale. There are bandwidth, reliability, security, management, cost, and performance considerations behind the complexity, just as there are at other layers of the stack. Folks higher up the stack want a simpler abstraction, but you can’t short-circuit physical limitations with a unified IP space.
  - fulafel 6y ago
    To take the first property you mention, as an example: How do you feel bandwidth is helped by the proxying / NAT approach compared to traditional IP routing and Internet addressing?
    - 0xEFF 6y ago
      Caching proxies and traffic controlling NAT.
      - deleted 6y ago
        [deleted]
      - api 6y ago
        Why do you need NAT to have caching or traffic control?
- siliconmountain 6y ago
  "K8s networking is just typical networking." K8 networking is far more complex than typical networking and with less visibility

by nvarsj 6y ago

intricatedetail 6y ago

What rescued kubernetes was adoption by AWS and Google. If you want to host Kubernetes yourself it is still a waste of time broadly speaking.

joana035 6y ago
Kubernetes didn't need to be rescued in the first place. If you know how to manage any Linux box, setting up kubernetes is as easy as "# apt install kubelet kubectl". Broadly speaking kubernetes saves time by providing an api for existing Linux technology. For example, iptables, ipvs, storage, self-healing, virtual ips, etc via yaml. Kubernetes was created to give anyone "Google like, production grade" infrastructure without the vendor lock-in of the cloud. And kubernetes on physical machines is where it shines most. EDIT: it is very easy to complain that kubernetes is complex, but try to do virtual ips with keepalived, corosync, pacemaker to understand how much time kubernetes saves by providing the same capability in a generic way which is easy to use. I have the feeling that people take kubernetes for granted without knowing how much manual work one has to go through in order to build a similar experience using the standard building blocks available. EDIT 2: straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google" that is a lot of saved time just by using kubernetes.
- deleted 6y ago
  [deleted]
- geofft 6y ago
  Kubernetes "builds upon" Google's "experience," but it isn't what Google runs and doesn't totally reflect what Google actually does: https://twitter.com/copyconstruct/status/1253968037579030532 https://twitter.com/copyconstruct/status/1253968037579030532 https://twitter.com/copyconstruct/status/1254273432847446016 https://twitter.com/copyconstruct/status/1254273432847446016 If you happen to have the problems Kubernetes solves, it's great. But even Google doesn't have exactly those problems. (And there is a huge difference between setting up Kubernetes and hosting Kubernetes. If you know how to sit in a chair, you can be in a plane's pilot seat just fine, but piloting is an entirely different skill.)
  - joana035 6y ago
    Correct, kubernetes is a destilation of those 15 years of experience acquired running something else that also has network, storage and compute concepts. I didn't said Google uses kubernetes, though :) Still, kubernetes makes a wonderful job abstracting all the hard work of using the underlying tools. Look how simple is to create a ipvs round-robin by just defining a "kind: Service" object, try to do the same using the bare tools in a way that is synchronized across N computers.
    - gabereiser 6y ago
      I would argue it doesn’t go far enough, especially in networking/discovery. Kubernetes is incredibly useful but I personally want more out of the box than just abstractions over existing linux technologies.
      - joana035 6y ago
        Networking just works. Discovery via DNS is pretty standard too. Mind to elaborate on what you would like to see in those areas?

api 6y ago

K8s networking is just typical networking. Networking always has to be made as complex and clunky as possible. Obvious solutions like unified IPv6 address space must be avoided as they would simplify the system and avoid the need for NAT. I’m being snarky but it is a long time observation of mine. When I see a network config I am more often than not shocked by its needless complexity.

skywhopper 6y ago
Folks who say stuff like this aren’t taking into account the number of real world constraints on networking at a large scale. There are bandwidth, reliability, security, management, cost, and performance considerations behind the complexity, just as there are at other layers of the stack. Folks higher up the stack want a simpler abstraction, but you can’t short-circuit physical limitations with a unified IP space.
- fulafel 6y ago
  To take the first property you mention, as an example: How do you feel bandwidth is helped by the proxying / NAT approach compared to traditional IP routing and Internet addressing?
  - 0xEFF 6y ago
    Caching proxies and traffic controlling NAT.
    - deleted 6y ago
      [deleted]
    - api 6y ago
      Why do you need NAT to have caching or traffic control?
siliconmountain 6y ago
"K8s networking is just typical networking." K8 networking is far more complex than typical networking and with less visibility