4 ms·

The risk with using Docker in production is that you can end up building your own bad version of Kubernetes over time. K8s is fairly complex but it solves a lot

The risk with using Docker in production is that you can end up building your own bad version of Kubernetes over time. K8s is fairly complex but it solves a lot of useful problems such as zero downtime upgrades, service discovery, declarative config etc

zzbzq 5y ago
I was a big advocate for Docker-as-deployment-packaging for a long time, and cautious about Kubernetes. I imagined building something like Ably describes would be the most practical. I was wrong. What I didn't understand is how easy kubernetes is to use for the application developers. It's the most natural, seamless way to do Docker-as-deployment-packaging. If you're going to have infrastructure guys at all, might as well have them maintain k8s.
dolni 5y ago
You don't build your own Kubernetes, though. You run EC2 instances that are sized for the container they run and let autoscaling take care of it. Load balancers distribute the load. Kubernetes makes use of a lot of the same stuff, but with even more complexity. Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.
- gizdan 5y ago
  > Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide. Sure it's not needed! But cloud providers aren't needed in much the same way.
  - dolni 5y ago
    Uh... what? Apples and oranges. Whatever convenience you think Kubernetes provides over EC2/autoscaling (which Kubernetes uses, by the way) is several orders of magnitude less than the convenience of using a cloud provider. That you would draw on-prem versus cloud as an equivalency to Kube vs other deployment methods reeks of inexperience, to me. edit: Oh no, I seem to triggered a drive by Kube fanboy or two. Yes, stealth downvote because you disagree without defending your position. You will do so much to show how right you are.
    - gizdan 5y ago
      Right, my point is, you can always run your infra much the same AWS runs their infra. Cloud providers give you certain advantages that on prem doesn't have, and vice versa. Equally, running container orchestration gives you advantages AWS cannot give you (as well as k8s does anyway), and vice versa. I'm a definitely fan of K8s, but I'm not defending it here, however, to saying orchestration isn't needed is silly. In a way what AWS provides is orchestration, it's just for VMs instead of containers. As a devops engineer I've worked with a lot of individuals and with a lot of tooling, and so far I can only say my opinion for container orchestation has only grown stronger. I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places. Compare that with a Dockerfile, a pipeline, and some manifest in the same repo. I've seen teams deploy a completely new application from first commit to production in less than a week, with next to zero knowledge of K8s when they started. The same teams, who weren't pros but had a bit of experience with AWS, took several weeks to deploy services of similar complexity on EC2. Saving two weeks of Developer's time and headaches is a lot, considering they're one of the most expensive expenses a company has.
      - throwdbaaway 5y ago
        > I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places. That sounds horrible. You can actually do all that from a single repository, using just ansible alone, with one cli invocation: - use packer to build an AMI => ansible - provisioned using Chef => ansible - terraform the relevant services => ansible - use cloud-init to pull configuration values => baked into the image
- ojhughes 5y ago
  You are still orchestrating though, you still need to provision nodes and related resources, ensure containers are running on the node and are healthy, mount volumes, clean up old docker resources. Plus you are now completely locked in to EC2, which may not be a concern for you but it is if your product or service needs to run on different cloud platforms like ours, or the CTO decides to change cloud providers one day because they got a sweet deal. I’m glad it works for you but after working with k8s for 2 years I never want to write another Ansible / Terraform script to provision IaaS resources
davewritescode 5y ago
This 100% People think that Kubernetes is just for service orchestration and it's true, it's very good for that. But what really sets it apart for me is the ability to extend kubernetes through operators and hooks that enables platform teams to really start to abstract away most of the underlying platform. A good example is the custom resources we've created that heavily abstract away what's happening underneath. The apps gives us a docker image and describe how they want to deploy and it's all done.
- handrous 5y ago
  The two things that have scared me away from k8s: 1) Churn + upgrade horror stories, 2) Apparent heavy reliance on 3rd party projects to use it well and sanely which can (in my experience, in other ecosystems) seriously hamper one's ability to resist the constant-upgrade train to mitigate problem #1. Basically, I'm afraid it's the Nodejs (or React Native—folks who've been there will know what I mean) of cluster management, not the... I dunno, Debian, reliable and slowly changing and sits-there-and-does-its-job. Does it seem that way to you and is just so good it's worth it anyway, or am I misunderstanding the situation?
  - ojhughes 5y ago
    The K8s API is very conservative with backwards compatability and graceful deprecation. In general, cluster upgrades should have minimal impact on your workloads. The horror stories are usually from a cluster admin point of view and devs are usually shielded from that
gizdan 5y ago
Came here to say this. It sounds like Ably is slowly moving towards creating their own orchestration tooling. Before you know it they'll need a bunch of additional features such as scaling up an individual service that runs on the same EC2 as 5 other services. Or permissions. Or whatever else. Kubernetes has it all solved. With the current abstractions given by major cloud providers such as EKS and GKE, it makes running it a lot simpler than it used to be. All of this effort they've spent on doing this would have likely taken a lot less effort than their current setup.
robfig 5y ago
I see this argument made pretty frequently, and it might be correct for a team that's just starting out. But service discovery and zero downtime upgrades are not that hard to implement and maintain. Our company's implementations of those things from 12 years ago have required ~zero maintenance and work completely fine. Sure, it uses Zookeeper which is out of vogue, but that also has been "just working" with very little attention. A thought experiment where we had instead been running a Kubernetes cluster for 12 years comes out pretty lopsided in favor of the path we took for which one minimizes the effort spent and complexity of the overall system.
- dolni 5y ago
  You don't even need to implement service discovery. AWS load balancers do that for you.
stewartmcgown 5y ago
Using Docker Swarm as an alternative to K8s in prod has been incredibly helpful for us as a growing startup. You get 90% of the advertised K8s goodies for almost zero extra complexity. Using deployment scripts + docker alone would be insane, even at our small scale.

zzbzq 5y ago

I was a big advocate for Docker-as-deployment-packaging for a long time, and cautious about Kubernetes. I imagined building something like Ably describes would be the most practical. I was wrong. What I didn't understand is how easy kubernetes is to use for the application developers. It's the most natural, seamless way to do Docker-as-deployment-packaging. If you're going to have infrastructure guys at all, might as well have them maintain k8s.

dolni 5y ago

You don't build your own Kubernetes, though. You run EC2 instances that are sized for the container they run and let autoscaling take care of it. Load balancers distribute the load. Kubernetes makes use of a lot of the same stuff, but with even more complexity. Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.

gizdan 5y ago
> Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide. Sure it's not needed! But cloud providers aren't needed in much the same way.
- dolni 5y ago
  Uh... what? Apples and oranges. Whatever convenience you think Kubernetes provides over EC2/autoscaling (which Kubernetes uses, by the way) is several orders of magnitude less than the convenience of using a cloud provider. That you would draw on-prem versus cloud as an equivalency to Kube vs other deployment methods reeks of inexperience, to me. edit: Oh no, I seem to triggered a drive by Kube fanboy or two. Yes, stealth downvote because you disagree without defending your position. You will do so much to show how right you are.
  - gizdan 5y ago
    Right, my point is, you can always run your infra much the same AWS runs their infra. Cloud providers give you certain advantages that on prem doesn't have, and vice versa. Equally, running container orchestration gives you advantages AWS cannot give you (as well as k8s does anyway), and vice versa. I'm a definitely fan of K8s, but I'm not defending it here, however, to saying orchestration isn't needed is silly. In a way what AWS provides is orchestration, it's just for VMs instead of containers. As a devops engineer I've worked with a lot of individuals and with a lot of tooling, and so far I can only say my opinion for container orchestation has only grown stronger. I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places. Compare that with a Dockerfile, a pipeline, and some manifest in the same repo. I've seen teams deploy a completely new application from first commit to production in less than a week, with next to zero knowledge of K8s when they started. The same teams, who weren't pros but had a bit of experience with AWS, took several weeks to deploy services of similar complexity on EC2. Saving two weeks of Developer's time and headaches is a lot, considering they're one of the most expensive expenses a company has.
    - throwdbaaway 5y ago
      > I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places. That sounds horrible. You can actually do all that from a single repository, using just ansible alone, with one cli invocation: - use packer to build an AMI => ansible - provisioned using Chef => ansible - terraform the relevant services => ansible - use cloud-init to pull configuration values => baked into the image
ojhughes 5y ago
You are still orchestrating though, you still need to provision nodes and related resources, ensure containers are running on the node and are healthy, mount volumes, clean up old docker resources. Plus you are now completely locked in to EC2, which may not be a concern for you but it is if your product or service needs to run on different cloud platforms like ours, or the CTO decides to change cloud providers one day because they got a sweet deal. I’m glad it works for you but after working with k8s for 2 years I never want to write another Ansible / Terraform script to provision IaaS resources

davewritescode 5y ago

This 100% People think that Kubernetes is just for service orchestration and it's true, it's very good for that. But what really sets it apart for me is the ability to extend kubernetes through operators and hooks that enables platform teams to really start to abstract away most of the underlying platform. A good example is the custom resources we've created that heavily abstract away what's happening underneath. The apps gives us a docker image and describe how they want to deploy and it's all done.

handrous 5y ago
The two things that have scared me away from k8s: 1) Churn + upgrade horror stories, 2) Apparent heavy reliance on 3rd party projects to use it well and sanely which can (in my experience, in other ecosystems) seriously hamper one's ability to resist the constant-upgrade train to mitigate problem #1. Basically, I'm afraid it's the Nodejs (or React Native—folks who've been there will know what I mean) of cluster management, not the... I dunno, Debian, reliable and slowly changing and sits-there-and-does-its-job. Does it seem that way to you and is just so good it's worth it anyway, or am I misunderstanding the situation?
- ojhughes 5y ago
  The K8s API is very conservative with backwards compatability and graceful deprecation. In general, cluster upgrades should have minimal impact on your workloads. The horror stories are usually from a cluster admin point of view and devs are usually shielded from that

gizdan 5y ago

Came here to say this. It sounds like Ably is slowly moving towards creating their own orchestration tooling. Before you know it they'll need a bunch of additional features such as scaling up an individual service that runs on the same EC2 as 5 other services. Or permissions. Or whatever else. Kubernetes has it all solved. With the current abstractions given by major cloud providers such as EKS and GKE, it makes running it a lot simpler than it used to be. All of this effort they've spent on doing this would have likely taken a lot less effort than their current setup.

robfig 5y ago

I see this argument made pretty frequently, and it might be correct for a team that's just starting out. But service discovery and zero downtime upgrades are not that hard to implement and maintain. Our company's implementations of those things from 12 years ago have required ~zero maintenance and work completely fine. Sure, it uses Zookeeper which is out of vogue, but that also has been "just working" with very little attention. A thought experiment where we had instead been running a Kubernetes cluster for 12 years comes out pretty lopsided in favor of the path we took for which one minimizes the effort spent and complexity of the overall system.

dolni 5y ago
You don't even need to implement service discovery. AWS load balancers do that for you.

stewartmcgown 5y ago

Using Docker Swarm as an alternative to K8s in prod has been incredibly helpful for us as a growing startup. You get 90% of the advertised K8s goodies for almost zero extra complexity. Using deployment scripts + docker alone would be insane, even at our small scale.