5 ms·

I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes. I don't really see Kubernetes as being

by kev009 2y ago

I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes. I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work. But it also seems to come with a magnitude of accidental and ornamental complexity. I would imply the same about microservices versus, say, figuring out your repository, language, and deployment pipelines to provide a smooth developer and operator experience. Too much of this industry is fashion and navel gazing instead of thinking about the core problems and standing behind a methodology that works for the business. Unless google moves its own infrastructure to Kubernetes, then maybe there's something to be had that couldn't reasonably be done otherwise :)

hggigg 2y ago
Agree with this on all points. We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including staff. This was all driven on the back of "we must have microservices to scale", something we have failed entirely to do. It's a complete own goal.
- threeseed 2y ago
  That's because you've moved to AWS which is an expensive cloud. You could have easily run Kubernetes on your virtualised servers.
  - hggigg 2y ago
    Define easily?
    - stackskipton 2y ago
      Still requires SRE but alot easier than it used to be: https://www.talos.dev/v1.8/ https://www.talos.dev/v1.8/ Also, k3s. https://k3s.io/ https://k3s.io/
  - baq 2y ago
    IME nothing easy about maintaining your own storage cluster!
    - threeseed 2y ago
      Nothing easy in general about running a distributed platform for highly-available applications.
      - baq 2y ago
        True but compute nodes are basically stateless in comparison to anything storage.
- wkat4242 2y ago
  I think it mainly works for startups which can build it as a greenfield and are extremely sensitive to scaling (either they are exploding within 2 years or they're dead within 2 years). So they plan to scale from 0 to millions quickly. For that kinda business it makes sense because once the exploding growth comes it's super easy to scale up and even do that automatically. That's the power of Kubernetes. For an existing business not in a hyper-growth phase with lots of restrictions and policies and processes and legacy stuff it's just extra work with no gain. It's just a round peg in a square hole. Wrong tool for the job.
kunley 2y ago
Apart from a sober and common-sense thinking in your comment, I like the term "ornamental complexity" :)
threeseed 2y ago
Kubernetes has unquestionably been a game changer. If you look at most enterprises today you will see it deployed everywhere. And most of the complexity has been abstracted away by the cloud providers so all you're left with is a system that can handle all manner of different applications and deployment scenarios in a consistent way.
- krageon 2y ago
  > Kubernetes has unquestionably been a game changer. > If you look at most enterprises today you will see it deployed everywhere. This doesn't mean it's a game changer. It just means it has a big cargo-cult and people keep using it regardless of whether they need it. > most of the complexity has been abstracted away by the cloud providers The abstractions that people add make things more complex, not less. Unless of course you don't care to understand what you're running, which is precisely the problem with the culture around most folks that use kubernetes (and more broadly the US cloud providers as an entity).
  - martinbaun 2y ago
    You're probably going to be downvoted to oblivion but I 100% agree with you. It seems like they tried to remove all the complexity, and just made new complexity.
    - stackskipton 2y ago
      Modern systems are complex generally. If it's not Kubernetes with YAML files, it's a bunch of servers in VMware which is own ball of wax followed by extensive Ansible/Puppet/Chef setup and SREs who keeping entire architecture in their head since they don't have time to write it down. Obviously, there is exact opposite with stuff like fly.io but they can be extremely constraining.
      - martinbaun 2y ago
        Yeah, I really liked Heroku and I used it for so many years, but it also killed itself kinda. It could never reduce prices as all machines "was the same" and thus a reduction in price meant lose of revenue and now they're just super expensive comparably. Now I just dump Caddyserver and install things bare metals servers. I avoid all dependencies like the plague
vilunov 2y ago
Kubernetes is absolutely not an accidental complexity. Microservices are not an accidental complexity, and they are not a replacement for a proper repository. Kubernetes solves administration of a cluster of Linux machines, as opposed to administering a single Linux machine. It abstracts away the concept of a machine, because it automates application scheduling, scaling across different machines, rolling updates of applications, adding/removing machines to the cluster all at the same time. There are no instruments like that for applications, the closest to them are something like Spark and Hadoop for data engineering tasks (not general applications). Microservices are also used to solve a very specific problem – independent deployments of parts of the system. You can dance with your repository and your code directories as much as you want, if you're not in a very specific runtime (e.g. BEAM VM), you will not achieve independent deployments of parts of your service. The ability to "scale independently" (which tbh is mostly bullshit) is an accidental consequence of using HTTP RPC for microservice communication, which is also not the only way, but it allows reuse of the HTTP ecosystem.

by kev009 2y ago

hggigg 2y ago

Agree with this on all points. We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including staff. This was all driven on the back of "we must have microservices to scale", something we have failed entirely to do. It's a complete own goal.

threeseed 2y ago
That's because you've moved to AWS which is an expensive cloud. You could have easily run Kubernetes on your virtualised servers.
- hggigg 2y ago
  Define easily?
  - stackskipton 2y ago
    Still requires SRE but alot easier than it used to be: https://www.talos.dev/v1.8/ https://www.talos.dev/v1.8/ Also, k3s. https://k3s.io/ https://k3s.io/
- baq 2y ago
  IME nothing easy about maintaining your own storage cluster!
  - threeseed 2y ago
    Nothing easy in general about running a distributed platform for highly-available applications.
    - baq 2y ago
      True but compute nodes are basically stateless in comparison to anything storage.
wkat4242 2y ago
I think it mainly works for startups which can build it as a greenfield and are extremely sensitive to scaling (either they are exploding within 2 years or they're dead within 2 years). So they plan to scale from 0 to millions quickly. For that kinda business it makes sense because once the exploding growth comes it's super easy to scale up and even do that automatically. That's the power of Kubernetes. For an existing business not in a hyper-growth phase with lots of restrictions and policies and processes and legacy stuff it's just extra work with no gain. It's just a round peg in a square hole. Wrong tool for the job.

kunley 2y ago

Apart from a sober and common-sense thinking in your comment, I like the term "ornamental complexity" :)

threeseed 2y ago

Kubernetes has unquestionably been a game changer. If you look at most enterprises today you will see it deployed everywhere. And most of the complexity has been abstracted away by the cloud providers so all you're left with is a system that can handle all manner of different applications and deployment scenarios in a consistent way.

krageon 2y ago
> Kubernetes has unquestionably been a game changer. > If you look at most enterprises today you will see it deployed everywhere. This doesn't mean it's a game changer. It just means it has a big cargo-cult and people keep using it regardless of whether they need it. > most of the complexity has been abstracted away by the cloud providers The abstractions that people add make things more complex, not less. Unless of course you don't care to understand what you're running, which is precisely the problem with the culture around most folks that use kubernetes (and more broadly the US cloud providers as an entity).
- martinbaun 2y ago
  You're probably going to be downvoted to oblivion but I 100% agree with you. It seems like they tried to remove all the complexity, and just made new complexity.
  - stackskipton 2y ago
    Modern systems are complex generally. If it's not Kubernetes with YAML files, it's a bunch of servers in VMware which is own ball of wax followed by extensive Ansible/Puppet/Chef setup and SREs who keeping entire architecture in their head since they don't have time to write it down. Obviously, there is exact opposite with stuff like fly.io but they can be extremely constraining.
    - martinbaun 2y ago
      Yeah, I really liked Heroku and I used it for so many years, but it also killed itself kinda. It could never reduce prices as all machines "was the same" and thus a reduction in price meant lose of revenue and now they're just super expensive comparably. Now I just dump Caddyserver and install things bare metals servers. I avoid all dependencies like the plague

vilunov 2y ago

Kubernetes is absolutely not an accidental complexity. Microservices are not an accidental complexity, and they are not a replacement for a proper repository. Kubernetes solves administration of a cluster of Linux machines, as opposed to administering a single Linux machine. It abstracts away the concept of a machine, because it automates application scheduling, scaling across different machines, rolling updates of applications, adding/removing machines to the cluster all at the same time. There are no instruments like that for applications, the closest to them are something like Spark and Hadoop for data engineering tasks (not general applications). Microservices are also used to solve a very specific problem – independent deployments of parts of the system. You can dance with your repository and your code directories as much as you want, if you're not in a very specific runtime (e.g. BEAM VM), you will not achieve independent deployments of parts of your service. The ability to "scale independently" (which tbh is mostly bullshit) is an accidental consequence of using HTTP RPC for microservice communication, which is also not the only way, but it allows reuse of the HTTP ecosystem.