6 ms·

To those who have used K8s extensively... 1. Is it really so complicated? 2. Is that complexity incidental or essential? 3. Could we get away with a simpler

To those who have used K8s extensively... 1. Is it really so complicated? 2. Is that complexity incidental or essential? 3. Could we get away with a simpler set of abstractions for 90% of applications?

lapser 4y ago
1. I used to use it at an old company in the early days of K8s. We ran our own setup, as EKS and AKS didn't exist. GCP did, but we were on AWS. It really is very complicated, however, with EKS, GCP, and AKS, it makes it a lot easier. Note, for users it's a lot simpler than the alternatives. Sure it's no heroku, but it vastly makes things easier compared to running on AWS, GCP, Azure, or worse, on bare metal. 2. Essential. K8s solves a problem that's quite complex. You can't really solve it in a simple manner. 3. Probably, but that 10% will require the additional abstractions and complications anyway, and it'll be easier to manage one system rather than 2.
canucklady 4y ago
k8s is actually very simple as a user. It's complicated to operate it yourself without EKS< GKE, etc. But from an end user perspective you write some delcarative manifests, they get put into an event bus, and then the state of the world is reconciled with your manifests. Easy peasy.
- api 4y ago
  > It's complicated to operate it yourself without EKS< GKE, etc. ... and we're now fully back to the mainframe era with the people in white coats who "run" the computer. The cloud truly is mainframe 2.0.
mixxit 4y ago
imagine trying to create an infinitely scalable system that is bound by aws account limits
- threeseed 4y ago
  a) AWS account limits are very flexible if you have spend enough money with them. b) Kubernetes clusters can span multiple accounts, clouds etc.
harpratap 4y ago
> 1. Is it really so complicated? Depends. Are you a >500 Developer org with many services? Then it's easy compared to what's out there. Anything less than that I'd say it's complex and you'd be better off using a PaaS > 2. Is that complexity incidental or essential? Depends. If you're going to do simple things forever then it's an overkill. But if you expect to grow in unknown ways in the future and don't want to waste your time doing bunch of migrations in the future then it's essential. > 3. Could we get away with a simpler set of abstractions for 90% of applications? Maybe? Heroku, AppEngine, CloudFoundry tried, but didn't go too far. Let's see what new crop of PaaS offerings are able to do
- threeseed 4y ago
  > Anything less than that I'd say it's complex and you'd be better off using a PaaS Kubernetes is available as a managed service in AWS, Azure, Google etc and this is likely to be the most popular deployment model. By any definition this is a PaaS and if you add in custom monitoring, logging, security, ingress etc. is going to be just as simple and significantly cheaper than using a managed solution. If you're just building a basic website then sure it's an overkill but fewer people are building those these days.
  - throwawaycuriou 4y ago
    it really comes down to what you're building. many web app startups would be better served paying for PaaS that manage this for them. as an example: Netlify/Vercel. if you need a database add FaunaDB to that. if that sounds risky or expensive, consider the cost of building a DevOps team.
deleted 4y ago
[deleted]
ekidd 4y ago
In my experience, Kubernetes can be straightforward if: 1. You read the O'Reilly book first (or another good book). There are a few unexpected abstractions (replicas, services, deployments, etc) which the book explains nicely. 2. You pay for a hosted Kubernetes. Google's is great. EKS is workable, but you may need to spend more time configuring it. 3. You don't mess with the networking system, and nothing goes horribly wrong. Our clusters peak out at close to 400 CPUs, and Kubernetes generally does what it says it will do. One caveat: If your app can be deployed using a "platform as a service" (Heroku, Render, etc), that's usually a better idea than Kubernetes. Kubernetes makes sense when a PaaS starts feeling too limited.
- bluehatbrit 4y ago
  On the PaaS front, I've also found that for smaller applications / new startups the pricing between starting with k8s or starting with a PaaS are pretty similar.
  - oneplane 4y ago
    Definitely. Works for FaaS too up to some extent (especially if you need hot-standby/zero-latency startup). Problem is mostly that once you scale beyond the "look it says hello world" levels the price goes up so fast you can essentially pay someone to "make it cheaper by running it elsewhere" and still be better off. Most questions seem to revolve around a tiny part of the puzzle, or a small "just starting out" phase and completely forgets about the lifecycle of the business process that it is built for, and the existing systems it needs to interact with. Even a startup will have that problem considering most are trying to get bought which essentially means being absorbed into a legacy company. So even starting out with no legacy to worry about is just a stay of execution.
    - p_l 4y ago
      If you are based somewhere else than in USA and without VC backing, the pricing wall hits you even faster :V
atmosx 4y ago
Do you write code in python? Is it really complicated to write a script to fetch some data, extract info, upload it someplace else? It’s not. But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many? See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening. As another commentator said, there is a level of knowledge that is required to things in production reliably.
- twelvedogs 4y ago
  I feel like you might be listing a bunch of edge cases that won't affect most people, or at least are as likely as a bunch of other edge cases with completely different optional solutions
mountainriver 4y ago
It’s not any more complicated than doing it other ways if you want control of the full infra stack. Advantage of k8s is you get a hardened unified API that works everywhere. > 3. Could we get away with a simpler set of abstractions for 90% of applications? Yeah but you can do that in k8s too, check out knative serving for example. K8s encourages the creation of higher level abstractions, with the advantage that you always have the break glass to dig into the primitives, which you don’t get with a lot of other systems.
stonemetal12 4y ago
1) No, the problem is it is all or nothing. Knowing a little k8s means your stuff doesn't work. 2) necessary at scale, incidental before then. 3) yes.
roflyear 4y ago
K8s does not have to be complex. If all your doing is, hosting a bunch of various web services, it is really simple. Actually, doesn't have to be just web: you can host services that only can communicate within kubernetes, or services that monitor and manage some XYZ resource, etc.. and that will all be really simple. Even hosting redis etc.. is really straight forward. It is funny, but the complexity starts to happen where you want kubernetes to handle other stuff: like hosting databases, or other storage resources, and if you want to for some reason I will never understand have your external services essentially communicate directly with kubernetes rather than have some middleware service you pay for do that for you (like a load balancer, etc.) One thing I did have an issue with was setting up SSL... that was surprisingly stupid. Should have been much easier to do that with LetsEncrypt.
- standyro 4y ago
  The problem is that it's often simple at first until you dive into the management of the cluster. Then you run into a litany of issues with networking (like you mentioned SSL termination) and stateful apps or databases. Even in this thread, someone mentioned how Redis defaults lead to a lot of issues in containers.
oneplane 4y ago
It's too dynamic to have universal answers.. but I'll give it a shot: 1. It's only as complicated as you make it. Kubernetes is essentially PKI (which is a must in any case), a REST API, and a scheduler. It stores some stuff somewhere, and you can add more stuff for it do have more features. I wouldn't call that complicated and it's essentially what Swarm and Mesos do as well (minus the PKI part). 2. PKI is essential. If you think that's complicated that's a whole different problem. Everything else is incidental. If adding more OpenAPIV3 schemas or REST API seem complex, again, not really a Kubernetes thing, mostly a general software development thing. 3. Yes, as 90% of applications really only exist as mediocre CRUD viewers you could run on a potato. Also, 90% of applications don't need to be as highly available or scalable as people might think. Then again, ecosystem complexity in software development combined with the lack of general knowledge (i.e. how to use an RDBMS properly) makes that while the software is simple and could be run as a single statically compiled binary, it generally is a mess, requiring more messes to make it run. But since that is cheaper (less developer time spent, more cheaper developers available to do that type of work), that is where we end up.
unity1001 4y ago
1 - No. People try to implement old, complex, stateful apps on K8 by just slapping on some stuff. That creates problems. 2- See 1. 3 - If you can containerize your app in a simple way, then yes. Note that a stateful app that would require attention in a bare metal server or a singular VM would still require that kind of attention on K8 as well. K8 just removes the need to manage the VM infra. And makes running your infra as code much easier. If you need to run a stateful app in a highly available manner, you can do it in K8 and it would be good - however you will spend a similar effort for maintaining the high available services like you do in other venues. Ie, if your stateful app requires a Percona cluster and a NFS cluster off of K8, you will still need to launch and maintain those services. K8 operators make these a lot easier to launch and maintain. But its still maintenance nonetheless. Using managed, hosted databases can work for the database part. But they are expensive. So launching a database cluster via a K8 operator would be cheaper to maintain. NFS is a problematic thing across all platforms. So if you need it, you either launch a rook-ceph cluster to provide a shared filesystem or use a hosted service like Google File Store.

lapser 4y ago

1. I used to use it at an old company in the early days of K8s. We ran our own setup, as EKS and AKS didn't exist. GCP did, but we were on AWS. It really is very complicated, however, with EKS, GCP, and AKS, it makes it a lot easier. Note, for users it's a lot simpler than the alternatives. Sure it's no heroku, but it vastly makes things easier compared to running on AWS, GCP, Azure, or worse, on bare metal. 2. Essential. K8s solves a problem that's quite complex. You can't really solve it in a simple manner. 3. Probably, but that 10% will require the additional abstractions and complications anyway, and it'll be easier to manage one system rather than 2.

canucklady 4y ago

k8s is actually very simple as a user. It's complicated to operate it yourself without EKS< GKE, etc. But from an end user perspective you write some delcarative manifests, they get put into an event bus, and then the state of the world is reconciled with your manifests. Easy peasy.

api 4y ago
> It's complicated to operate it yourself without EKS< GKE, etc. ... and we're now fully back to the mainframe era with the people in white coats who "run" the computer. The cloud truly is mainframe 2.0.

mixxit 4y ago

imagine trying to create an infinitely scalable system that is bound by aws account limits

threeseed 4y ago
a) AWS account limits are very flexible if you have spend enough money with them. b) Kubernetes clusters can span multiple accounts, clouds etc.

harpratap 4y ago

> 1. Is it really so complicated? Depends. Are you a >500 Developer org with many services? Then it's easy compared to what's out there. Anything less than that I'd say it's complex and you'd be better off using a PaaS > 2. Is that complexity incidental or essential? Depends. If you're going to do simple things forever then it's an overkill. But if you expect to grow in unknown ways in the future and don't want to waste your time doing bunch of migrations in the future then it's essential. > 3. Could we get away with a simpler set of abstractions for 90% of applications? Maybe? Heroku, AppEngine, CloudFoundry tried, but didn't go too far. Let's see what new crop of PaaS offerings are able to do

threeseed 4y ago
> Anything less than that I'd say it's complex and you'd be better off using a PaaS Kubernetes is available as a managed service in AWS, Azure, Google etc and this is likely to be the most popular deployment model. By any definition this is a PaaS and if you add in custom monitoring, logging, security, ingress etc. is going to be just as simple and significantly cheaper than using a managed solution. If you're just building a basic website then sure it's an overkill but fewer people are building those these days.
- throwawaycuriou 4y ago
  it really comes down to what you're building. many web app startups would be better served paying for PaaS that manage this for them. as an example: Netlify/Vercel. if you need a database add FaunaDB to that. if that sounds risky or expensive, consider the cost of building a DevOps team.

deleted 4y ago

[deleted]

ekidd 4y ago

In my experience, Kubernetes can be straightforward if: 1. You read the O'Reilly book first (or another good book). There are a few unexpected abstractions (replicas, services, deployments, etc) which the book explains nicely. 2. You pay for a hosted Kubernetes. Google's is great. EKS is workable, but you may need to spend more time configuring it. 3. You don't mess with the networking system, and nothing goes horribly wrong. Our clusters peak out at close to 400 CPUs, and Kubernetes generally does what it says it will do. One caveat: If your app can be deployed using a "platform as a service" (Heroku, Render, etc), that's usually a better idea than Kubernetes. Kubernetes makes sense when a PaaS starts feeling too limited.

bluehatbrit 4y ago
On the PaaS front, I've also found that for smaller applications / new startups the pricing between starting with k8s or starting with a PaaS are pretty similar.
- oneplane 4y ago
  Definitely. Works for FaaS too up to some extent (especially if you need hot-standby/zero-latency startup). Problem is mostly that once you scale beyond the "look it says hello world" levels the price goes up so fast you can essentially pay someone to "make it cheaper by running it elsewhere" and still be better off. Most questions seem to revolve around a tiny part of the puzzle, or a small "just starting out" phase and completely forgets about the lifecycle of the business process that it is built for, and the existing systems it needs to interact with. Even a startup will have that problem considering most are trying to get bought which essentially means being absorbed into a legacy company. So even starting out with no legacy to worry about is just a stay of execution.
  - p_l 4y ago
    If you are based somewhere else than in USA and without VC backing, the pricing wall hits you even faster :V

atmosx 4y ago

Do you write code in python? Is it really complicated to write a script to fetch some data, extract info, upload it someplace else? It’s not. But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many? See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening. As another commentator said, there is a level of knowledge that is required to things in production reliably.

twelvedogs 4y ago
I feel like you might be listing a bunch of edge cases that won't affect most people, or at least are as likely as a bunch of other edge cases with completely different optional solutions

mountainriver 4y ago

It’s not any more complicated than doing it other ways if you want control of the full infra stack. Advantage of k8s is you get a hardened unified API that works everywhere. > 3. Could we get away with a simpler set of abstractions for 90% of applications? Yeah but you can do that in k8s too, check out knative serving for example. K8s encourages the creation of higher level abstractions, with the advantage that you always have the break glass to dig into the primitives, which you don’t get with a lot of other systems.

stonemetal12 4y ago

1) No, the problem is it is all or nothing. Knowing a little k8s means your stuff doesn't work. 2) necessary at scale, incidental before then. 3) yes.

roflyear 4y ago

K8s does not have to be complex. If all your doing is, hosting a bunch of various web services, it is really simple. Actually, doesn't have to be just web: you can host services that only can communicate within kubernetes, or services that monitor and manage some XYZ resource, etc.. and that will all be really simple. Even hosting redis etc.. is really straight forward. It is funny, but the complexity starts to happen where you want kubernetes to handle other stuff: like hosting databases, or other storage resources, and if you want to for some reason I will never understand have your external services essentially communicate directly with kubernetes rather than have some middleware service you pay for do that for you (like a load balancer, etc.) One thing I did have an issue with was setting up SSL... that was surprisingly stupid. Should have been much easier to do that with LetsEncrypt.

standyro 4y ago
The problem is that it's often simple at first until you dive into the management of the cluster. Then you run into a litany of issues with networking (like you mentioned SSL termination) and stateful apps or databases. Even in this thread, someone mentioned how Redis defaults lead to a lot of issues in containers.

oneplane 4y ago

It's too dynamic to have universal answers.. but I'll give it a shot: 1. It's only as complicated as you make it. Kubernetes is essentially PKI (which is a must in any case), a REST API, and a scheduler. It stores some stuff somewhere, and you can add more stuff for it do have more features. I wouldn't call that complicated and it's essentially what Swarm and Mesos do as well (minus the PKI part). 2. PKI is essential. If you think that's complicated that's a whole different problem. Everything else is incidental. If adding more OpenAPIV3 schemas or REST API seem complex, again, not really a Kubernetes thing, mostly a general software development thing. 3. Yes, as 90% of applications really only exist as mediocre CRUD viewers you could run on a potato. Also, 90% of applications don't need to be as highly available or scalable as people might think. Then again, ecosystem complexity in software development combined with the lack of general knowledge (i.e. how to use an RDBMS properly) makes that while the software is simple and could be run as a single statically compiled binary, it generally is a mess, requiring more messes to make it run. But since that is cheaper (less developer time spent, more cheaper developers available to do that type of work), that is where we end up.

unity1001 4y ago

1 - No. People try to implement old, complex, stateful apps on K8 by just slapping on some stuff. That creates problems. 2- See 1. 3 - If you can containerize your app in a simple way, then yes. Note that a stateful app that would require attention in a bare metal server or a singular VM would still require that kind of attention on K8 as well. K8 just removes the need to manage the VM infra. And makes running your infra as code much easier. If you need to run a stateful app in a highly available manner, you can do it in K8 and it would be good - however you will spend a similar effort for maintaining the high available services like you do in other venues. Ie, if your stateful app requires a Percona cluster and a NFS cluster off of K8, you will still need to launch and maintain those services. K8 operators make these a lot easier to launch and maintain. But its still maintenance nonetheless. Using managed, hosted databases can work for the database part. But they are expensive. So launching a database cluster via a K8 operator would be cheaper to maintain. NFS is a problematic thing across all platforms. So if you need it, you either launch a rook-ceph cluster to provide a shared filesystem or use a hosted service like Google File Store.