6 ms·

Some of the arguments against Kubernetes and for simpler systems/approaches, like Docker Compose, systemd or others actually gave me the push i needed to collec

Some of the arguments against Kubernetes and for simpler systems/approaches, like Docker Compose, systemd or others actually gave me the push i needed to collect some of my thoughts and past experiences in a blog post "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chaos-to-order-a-tale-of-legacy-code-services-and-containers https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao... (i'm linking it as a top level comment here, since responding to multiple comments would feel spammy, but the discussion is interesting) In short: - Docker Compose is lovely for local development, but isn't necessarily the only option to do containers easily - Docker Swarm is what i chose and has been working wonderfully with software like Portainer - K3s is a good Kubernetes distro that cuts down on the resource usage and allows you to work slightly more easily with it - that said, Kubernetes isn't always the right tool for the job, with which i agree, especially for smaller teams - systemd services can also be perfectly passable, but only as long as you control everything around the service itself (e.g. the environment with the updates etc.) - regardless of whether you use containers or not, there are benefits to configuration management systems like Ansible, which i suggest you look into if you haven't already

laurowyn 5y ago
>Kubernetes isn't always the right tool for the job, with which i agree, especially for smaller teams I think this is the key point. Kubernetes is great when used correctly, in the right scenario. And I feel that most of the hate comes from situations where it's not suitable. Kubernetes is just one of the many platforms available to run your code. My point extends to more systems, like Ansible, Chef and all the others. TL;DR it's not a tool for developers, it's a tool for operations and systems integrators. A developer working on an application shouldn't care about the platform being run on, only defining its requirements - think big enterprise system that just consumes data, processes and stores it. The moment an engineer starts thinking "I need X platform" they're not developing the app anymore, they're integrating it too. The system integrator should be able to take those requirements and match it to the right platform - choose the OS, native package vs container, scalability, fault tolerance etc. But the system integrator doesn't need to care about the long term maintenance of the platform, just how to integrate the application onto the platform. k8s can be considered here for long term support, ease of integration, portability etc. The systems maintenance are responsible for ensuring the system has the resources to run the application and fix any faults that appear in the infrastructure - this is where k8s shines the most, because I can drain a node, update it and uncordon it without worrying about where I'm going to move the containers running on the node because k8s will do that for me. Gone are the days where we need to plan taking a machine offline for an hour, making sure the right failover procedures are in place, because k8s should be demonstrating that regularly. Essentially, kubernetes/Ansible/etc. should not be a part of the development process. It doesn't solve problems that an application developer should care about. It's an operations tool, able to monitor and manage deployed applications. The hate occurs when the wrong people are using the tools, or the right people are using the tools incorrectly. And of course, these kinds of separations of concern can only really happen in larger teams, on larger projects. Dedicating a few people to maintaining a deployment, a different group to continuous integration, and a 3rd group to actually developing the functionality. That's just not possible in every case, and the most hate seems to come from those in-between scenarios where there's too many people to call it a small team, but too few to enable this separation of concern. Meanwhile, the most love seems to come from those that can differentiate between the many hats they need to wear in a small team.
- KronisLV 5y ago
  > A developer working on an application shouldn't care about the platform being run on, only defining its requirements - think big enterprise system that just consumes data, processes and stores it. The moment an engineer starts thinking "I need X platform" they're not developing the app anymore, they're integrating it too. This feels like a statement that begs more discussion because it's an interesting argument, albeit probably a polarizing one as well. My experience has been the complete opposite to this - when developers also think about how the app will run, the outcomes will be the best. No system runs in a vacuum or exists separate of its runtime environments, so attempts to decouple those could lead to countless problems and inconsistencies down the road. For example, if you create a Java app that expects configuration to come from XML files, this may cause difficulty with containerization and won't follow the best practices anymore. If you create an app that stores session data in memory instead of Redis, then it won't be horizontally scalable. Same for shared data, like using the file system for something - that will create a lot of problems with multiple instances vs using something like S3. Even if you don't pick the platform yourself, you should make sure that the app integrates with it smoothly, hence the 12 Factor App principles can help for many of these cases. > Essentially, kubernetes/Ansible/etc. should not be a part of the development process. It doesn't solve problems that an application developer should care about. It's an operations tool, able to monitor and manage deployed applications. The hate occurs when the wrong people are using the tools, or the right people are using the tools incorrectly. In my experience, this leads to knowledge silos, problematic communication, not thinking about Ops concerns during development either due to a missing set of skills or just not caring due to a lack of ownership about the outcome, and just generally a slowed down pace of development. Now, there probably is a lot of benefit in specialization (e.g. Ops specialists that consult across different teams and projects to make everything more consistent), but in my eyes everyone should know a bit of everything and communicate freely. If you don't do that, developers use ORMs in their apps and create N+1 problems or large amounts of DB calls due to a lack of joins or no consideration for indices because they're not DBAs after all, other times people don't bother with instrumentation in their apps and introduce problems with proxying configuration due to not caring about the ingress will look like etc. I actually recall seeing an issue at work about an Ops related problem, recommended a few steps to take care of it, whereas the responsible dev simply told me: "That's not in my job description, i just write Java." Now, not making people learn about an extremely vast array of topics and spend evenings/weekends doing so would also be nice, but then again, that person didn't solve the issue, i did. The end result is what matters, optimizing the road to it is up to the people in control. That said, "rockstar devs" or "DevOps" specialists (or whatever the current term is) also carry risks in regards to a low bus factor (everything hinging on them in certain orgs, vs spreading the knowledge), so it's all probably pretty situational and scalable to different degrees.
MuffinFlavored 5y ago
what is good to use if you need to do a rolling deploy midday without interruption for clients?
- otterley 5y ago
  Unfortunately not Kubernetes, because it decoupled pod lifecycles from load balancers by design. K8S has no idea how to coordinate with load balancers to ensure new requests are flowing to new pods and to allow in-flight requests on moribund pods to complete before terminating them. This leads to at least a small number of 5xx errors being returned to clients on almost every deployment unless you implement workarounds like delaying pod shutdowns with arbitrary sleep times.
  - dharmab 5y ago
    You can do this in Kubernetes by combining Envoy's draining support and graceful shutdown in the backend Pod (via handling SIGTERM or adding a preStop hook to delay termination). The big users of Envoy+Kubernetes are mostly running their own custom ingress controllers to coordinate this across the ingress system (easier than it sounds). https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... Source: I helped build a system like this for a major tech company that handles many tens of billions request daily.
    - otterley 5y ago
      That sounds like a great deal of work that ought to have been needless. And Envoy is a second system, not part of K8S proper.
  - magicalhippo 5y ago
    In our current non-web application we have a launcher, which allows users to select between the current and a few previous versions. This helps us iterate faster due to less show-stoppers. If a user experiences a blocking bug in current version, they log out and back into an older version. For our web version I was thinking of doing something similar by having a launcher page, and then having each version run in its own container on a subdomain. So far I tested this using Docker Compose with Traefik routing the requests to the appropriate app container, and it worked fine. My deployment script modifies the docker-compose.yml config file to add a new container, and then Compose brings the additional container online. Current users get a notification that a new version is available but can continue working in the current version, and new logins have the new version selected by default (but can be overridden as mentioned). Would something like this not work with Kubernetes? I'm not yet familiar with Kubernetes but I've been assuming we might end up using it.

laurowyn 5y ago

>Kubernetes isn't always the right tool for the job, with which i agree, especially for smaller teams I think this is the key point. Kubernetes is great when used correctly, in the right scenario. And I feel that most of the hate comes from situations where it's not suitable. Kubernetes is just one of the many platforms available to run your code. My point extends to more systems, like Ansible, Chef and all the others. TL;DR it's not a tool for developers, it's a tool for operations and systems integrators. A developer working on an application shouldn't care about the platform being run on, only defining its requirements - think big enterprise system that just consumes data, processes and stores it. The moment an engineer starts thinking "I need X platform" they're not developing the app anymore, they're integrating it too. The system integrator should be able to take those requirements and match it to the right platform - choose the OS, native package vs container, scalability, fault tolerance etc. But the system integrator doesn't need to care about the long term maintenance of the platform, just how to integrate the application onto the platform. k8s can be considered here for long term support, ease of integration, portability etc. The systems maintenance are responsible for ensuring the system has the resources to run the application and fix any faults that appear in the infrastructure - this is where k8s shines the most, because I can drain a node, update it and uncordon it without worrying about where I'm going to move the containers running on the node because k8s will do that for me. Gone are the days where we need to plan taking a machine offline for an hour, making sure the right failover procedures are in place, because k8s should be demonstrating that regularly. Essentially, kubernetes/Ansible/etc. should not be a part of the development process. It doesn't solve problems that an application developer should care about. It's an operations tool, able to monitor and manage deployed applications. The hate occurs when the wrong people are using the tools, or the right people are using the tools incorrectly. And of course, these kinds of separations of concern can only really happen in larger teams, on larger projects. Dedicating a few people to maintaining a deployment, a different group to continuous integration, and a 3rd group to actually developing the functionality. That's just not possible in every case, and the most hate seems to come from those in-between scenarios where there's too many people to call it a small team, but too few to enable this separation of concern. Meanwhile, the most love seems to come from those that can differentiate between the many hats they need to wear in a small team.

KronisLV 5y ago
> A developer working on an application shouldn't care about the platform being run on, only defining its requirements - think big enterprise system that just consumes data, processes and stores it. The moment an engineer starts thinking "I need X platform" they're not developing the app anymore, they're integrating it too. This feels like a statement that begs more discussion because it's an interesting argument, albeit probably a polarizing one as well. My experience has been the complete opposite to this - when developers also think about how the app will run, the outcomes will be the best. No system runs in a vacuum or exists separate of its runtime environments, so attempts to decouple those could lead to countless problems and inconsistencies down the road. For example, if you create a Java app that expects configuration to come from XML files, this may cause difficulty with containerization and won't follow the best practices anymore. If you create an app that stores session data in memory instead of Redis, then it won't be horizontally scalable. Same for shared data, like using the file system for something - that will create a lot of problems with multiple instances vs using something like S3. Even if you don't pick the platform yourself, you should make sure that the app integrates with it smoothly, hence the 12 Factor App principles can help for many of these cases. > Essentially, kubernetes/Ansible/etc. should not be a part of the development process. It doesn't solve problems that an application developer should care about. It's an operations tool, able to monitor and manage deployed applications. The hate occurs when the wrong people are using the tools, or the right people are using the tools incorrectly. In my experience, this leads to knowledge silos, problematic communication, not thinking about Ops concerns during development either due to a missing set of skills or just not caring due to a lack of ownership about the outcome, and just generally a slowed down pace of development. Now, there probably is a lot of benefit in specialization (e.g. Ops specialists that consult across different teams and projects to make everything more consistent), but in my eyes everyone should know a bit of everything and communicate freely. If you don't do that, developers use ORMs in their apps and create N+1 problems or large amounts of DB calls due to a lack of joins or no consideration for indices because they're not DBAs after all, other times people don't bother with instrumentation in their apps and introduce problems with proxying configuration due to not caring about the ingress will look like etc. I actually recall seeing an issue at work about an Ops related problem, recommended a few steps to take care of it, whereas the responsible dev simply told me: "That's not in my job description, i just write Java." Now, not making people learn about an extremely vast array of topics and spend evenings/weekends doing so would also be nice, but then again, that person didn't solve the issue, i did. The end result is what matters, optimizing the road to it is up to the people in control. That said, "rockstar devs" or "DevOps" specialists (or whatever the current term is) also carry risks in regards to a low bus factor (everything hinging on them in certain orgs, vs spreading the knowledge), so it's all probably pretty situational and scalable to different degrees.

MuffinFlavored 5y ago

what is good to use if you need to do a rolling deploy midday without interruption for clients?

otterley 5y ago
Unfortunately not Kubernetes, because it decoupled pod lifecycles from load balancers by design. K8S has no idea how to coordinate with load balancers to ensure new requests are flowing to new pods and to allow in-flight requests on moribund pods to complete before terminating them. This leads to at least a small number of 5xx errors being returned to clients on almost every deployment unless you implement workarounds like delaying pod shutdowns with arbitrary sleep times.
- dharmab 5y ago
  You can do this in Kubernetes by combining Envoy's draining support and graceful shutdown in the backend Pod (via handling SIGTERM or adding a preStop hook to delay termination). The big users of Envoy+Kubernetes are mostly running their own custom ingress controllers to coordinate this across the ingress system (easier than it sounds). https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... Source: I helped build a system like this for a major tech company that handles many tens of billions request daily.
  - otterley 5y ago
    That sounds like a great deal of work that ought to have been needless. And Envoy is a second system, not part of K8S proper.
- magicalhippo 5y ago
  In our current non-web application we have a launcher, which allows users to select between the current and a few previous versions. This helps us iterate faster due to less show-stoppers. If a user experiences a blocking bug in current version, they log out and back into an older version. For our web version I was thinking of doing something similar by having a launcher page, and then having each version run in its own container on a subdomain. So far I tested this using Docker Compose with Traefik routing the requests to the appropriate app container, and it worked fine. My deployment script modifies the docker-compose.yml config file to add a new container, and then Compose brings the additional container online. Current users get a notification that a new version is available but can continue working in the current version, and new logins have the new version selected by default (but can be overridden as mentioned). Would something like this not work with Kubernetes? I'm not yet familiar with Kubernetes but I've been assuming we might end up using it.