5 ms·

Did JVM reinvent VM? Is it a VM or a container? No one (not enough people?) asked (cared?) for this about the JVM for like 25 years till Linux Cgroups came aro

Did JVM reinvent VM? Is it a VM or a container? No one (not enough people?) asked (cared?) for this about the JVM for like 25 years till Linux Cgroups came around and Docker capitalized on it. Its not VM v/s container question that we care about. (I am aware this is a very poor comparison but so is comparing a vm to a container, which is my point) K8S brings complexity yes, and in a consistent, predictable way. The biggest difference between a k8s based shop vs a DIY VM/Container shop is the former can (in theory) hire some people with intermediate K8S experience, and have them sort out the mess that the people without that experience (or more typically the ones who don't have clear requirements) may have created. With k8s it becomes more predictable to find the boundaries and start pulling things apart in production and make incremental changes. Without k8s, making changes to a production setup is a nightmare that folks who have done that sort of thing don't ever want to repeat.

aprdm 4y ago
That’s true if your shop didn’t create a bunch of custom controllers or apis inside the k8s world… or forked it.
- koffiezet 4y ago
  Any decent k8s engineer worth his salt should find it pretty easy to navigate this, since how you deal with that is pretty standardised. I've written custom operators and submitted upstream patches to existing-ones, it's not that hard once you know what you're dealing with conceptually.
  - x86x87 4y ago
    So your point is that once you learn how to use the tool it's easier to use the tool? Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
    - koffiezet 4y ago
      > So your point is that once you learn how to use the tool it's easier to use the tool? Kubernetes is not just "a tool", it's a platform. So if you know the k8s fundamentals, being dropped in an unfamiliar environment should not be a problem. It's the same as knowing your way around a Linux system, you might not be familiar with that specific distribution you're thrown into, but you should be able to explore the system and figure stuff out. > Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer? Any engineer who claims to know a bit more than the basics of k8s? It being a software, "devops", SRE, "platform", ... engineer doesn't really matter, I've been labeled all of the above. I'll never claim k8s is perfect, I have plenty of issues with it, but very few of them are an architectural problem, and I've yet to encounter a platform that attempts to address all the issues k8s does. But you seem to be very critical of it, so what would you propose as a viable alternative?
x86x87 4y ago
Are you running the k8s control plane yourself? Are you keeping it up-to-date?
- dilyevsky 4y ago
  We do (using Cluster API) and yes we keep it now within 2 releases from latest. Most of the work required to keep it up to date is not on k8s side tho… Edit: I can answer questions on how we do it or maybe work on blog post if you’re seriously asking
  - x86x87 4y ago
    I am asking because I've seen the work that goes into keeping a control plane (for anything other than toy applications) running. It's non-trivial and requires a lot of know-how about many not well documented k8s features. You may have figured a better way of doing it or maybe you have reduced complexity by having a more opioninated setup. How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services) Are you running your workloads in the cloud or on dedicated bare metal machines?
    - dilyevsky 4y ago
      I’d love to hear more details/chat about what aspects you found most difficult specifically with running control plane (not on gke/eks/aks). I’m looking at how I could solve this more generally for businesses ;) > How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services) Up to upper hundreds of nodes/tens of thousands of cores (beefy nodes) per cluster in biggest ones (several geo/cloud-distributed clusters). Tbh what’s giving me more grief is number and diversity (different clouds/different setups) of clusters not individual cluster sizes. Original team that built/proved the design was only 3-4 devs but k8s was only part of what we did so it’s more like 1-2 if full time. Number of engineers that deploy there is probably close to 50 now with at least twice as many services (not all actively developed or in-house). > Are you running your workloads in the cloud or on dedicated bare metal machines? GCP and AWS VMs. We managed to stay away from anything managed except block and object storage so baremetal is in store for us just wasn’t the time yet. One of my main goals developing our current design was to keep this door open when the time comes and the setup will be largely the same as in cloud (with a few additions like 3rd party CSI for ISCI/NVMeOf storage)

aprdm 4y ago

That’s true if your shop didn’t create a bunch of custom controllers or apis inside the k8s world… or forked it.

koffiezet 4y ago
Any decent k8s engineer worth his salt should find it pretty easy to navigate this, since how you deal with that is pretty standardised. I've written custom operators and submitted upstream patches to existing-ones, it's not that hard once you know what you're dealing with conceptually.
- x86x87 4y ago
  So your point is that once you learn how to use the tool it's easier to use the tool? Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
  - koffiezet 4y ago
    > So your point is that once you learn how to use the tool it's easier to use the tool? Kubernetes is not just "a tool", it's a platform. So if you know the k8s fundamentals, being dropped in an unfamiliar environment should not be a problem. It's the same as knowing your way around a Linux system, you might not be familiar with that specific distribution you're thrown into, but you should be able to explore the system and figure stuff out. > Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer? Any engineer who claims to know a bit more than the basics of k8s? It being a software, "devops", SRE, "platform", ... engineer doesn't really matter, I've been labeled all of the above. I'll never claim k8s is perfect, I have plenty of issues with it, but very few of them are an architectural problem, and I've yet to encounter a platform that attempts to address all the issues k8s does. But you seem to be very critical of it, so what would you propose as a viable alternative?

x86x87 4y ago

Are you running the k8s control plane yourself? Are you keeping it up-to-date?

dilyevsky 4y ago
We do (using Cluster API) and yes we keep it now within 2 releases from latest. Most of the work required to keep it up to date is not on k8s side tho… Edit: I can answer questions on how we do it or maybe work on blog post if you’re seriously asking
- x86x87 4y ago
  I am asking because I've seen the work that goes into keeping a control plane (for anything other than toy applications) running. It's non-trivial and requires a lot of know-how about many not well documented k8s features. You may have figured a better way of doing it or maybe you have reduced complexity by having a more opioninated setup. How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services) Are you running your workloads in the cloud or on dedicated bare metal machines?
  - dilyevsky 4y ago
    I’d love to hear more details/chat about what aspects you found most difficult specifically with running control plane (not on gke/eks/aks). I’m looking at how I could solve this more generally for businesses ;) > How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services) Up to upper hundreds of nodes/tens of thousands of cores (beefy nodes) per cluster in biggest ones (several geo/cloud-distributed clusters). Tbh what’s giving me more grief is number and diversity (different clouds/different setups) of clusters not individual cluster sizes. Original team that built/proved the design was only 3-4 devs but k8s was only part of what we did so it’s more like 1-2 if full time. Number of engineers that deploy there is probably close to 50 now with at least twice as many services (not all actively developed or in-house). > Are you running your workloads in the cloud or on dedicated bare metal machines? GCP and AWS VMs. We managed to stay away from anything managed except block and object storage so baremetal is in store for us just wasn’t the time yet. One of my main goals developing our current design was to keep this door open when the time comes and the setup will be largely the same as in cloud (with a few additions like 3rd party CSI for ISCI/NVMeOf storage)