5 ms·

What do you recommend to these people instead of k8s, so that they can have all the convenience of orchestration and containerization?

cogman10 6y ago
Why containerize to start with? Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with. The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
- p_l 6y ago
  PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s. Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
  - cogman10 6y ago
    While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline. Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
    - p_l 6y ago
      That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions. And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
- bronson 6y ago
  Lambdas get pricey quick. PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on). Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting. Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers. That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
- AYBABTME 6y ago
  I mean, containerizing isn't harder than anything else. It makes distributing your artifacts much easier, and building across people's computers easier too. It takes O(minute) to containerize something, so I don't see why not do it.
tlarkworthy 6y ago
I do Terraform + VMs + Serverless. It's the same thing with a layer of indirection cut out and platform native cloud UIs.
kristianpaul 6y ago
You can always use systemd and pick an orchestration tool like ansible,cdk or terraform
- eeZah7Ux 6y ago
  You don't even need the "orchestration tool"
jadams3 6y ago
Started small with k8s / gke, suffered a lot, coming to appreciate it as we grow. I think over the last 1-2 years there weren't a lot of other choices. Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible. For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
zenlot 6y ago
Docker Swarm. It's not dead and you can set it up in no time.
deleted 6y ago
[deleted]
cwyers 6y ago
ECS and Cloud Run are options to get containers without k8s.
- harpratap 6y ago
  Cloud Run is basically Custom Resource on top of Kubernetes API. The yaml definition is pretty much the same - https://cloud.google.com/run/docs/deploying#yaml https://cloud.google.com/run/docs/deploying#yaml What you mean is probably without vanilla k8s
  - cwyers 6y ago
    Well, sorta. I'm not surprised it's k8s under the hood. But it's not a managed k8s product like GKE -- you don't have to _care_ that it's k8s under the hood. To the consumer, it presents like ECS or Lambda.
dolni 6y ago
Run EC2 instances in an autoscaling group. Provision them with whatever you're comfortable with. Ansible is nice because you can point it at a node to test the deploy out without having to always respin the node to try again. Have ansible fire up a docker container on your host. That's it, done. You don't need Kubernetes. Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors. While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in. If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity. Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
- lkrubner 6y ago
  This can go a very long way. Using Ansible and Packer to provision AMIs, and then Terraform to spin up what's needed, goes a long way, without introducing a new layer of technology.
- AYBABTME 6y ago
  But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
  - dolni 6y ago
    > But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-refresh-for-ec2-auto-scaling/ https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one. Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes? > I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1. The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently. There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through. None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.

cogman10 6y ago

Why containerize to start with? Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with. The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.

p_l 6y ago
PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s. Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
- cogman10 6y ago
  While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline. Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
  - p_l 6y ago
    That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions. And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
bronson 6y ago
Lambdas get pricey quick. PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on). Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting. Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers. That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
AYBABTME 6y ago
I mean, containerizing isn't harder than anything else. It makes distributing your artifacts much easier, and building across people's computers easier too. It takes O(minute) to containerize something, so I don't see why not do it.

tlarkworthy 6y ago

I do Terraform + VMs + Serverless. It's the same thing with a layer of indirection cut out and platform native cloud UIs.

kristianpaul 6y ago

You can always use systemd and pick an orchestration tool like ansible,cdk or terraform

eeZah7Ux 6y ago
You don't even need the "orchestration tool"

jadams3 6y ago

Started small with k8s / gke, suffered a lot, coming to appreciate it as we grow. I think over the last 1-2 years there weren't a lot of other choices. Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible. For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.

zenlot 6y ago

Docker Swarm. It's not dead and you can set it up in no time.

deleted 6y ago

[deleted]

cwyers 6y ago

ECS and Cloud Run are options to get containers without k8s.

harpratap 6y ago
Cloud Run is basically Custom Resource on top of Kubernetes API. The yaml definition is pretty much the same - https://cloud.google.com/run/docs/deploying#yaml https://cloud.google.com/run/docs/deploying#yaml What you mean is probably without vanilla k8s
- cwyers 6y ago
  Well, sorta. I'm not surprised it's k8s under the hood. But it's not a managed k8s product like GKE -- you don't have to _care_ that it's k8s under the hood. To the consumer, it presents like ECS or Lambda.

dolni 6y ago

Run EC2 instances in an autoscaling group. Provision them with whatever you're comfortable with. Ansible is nice because you can point it at a node to test the deploy out without having to always respin the node to try again. Have ansible fire up a docker container on your host. That's it, done. You don't need Kubernetes. Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors. While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in. If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity. Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.

lkrubner 6y ago
This can go a very long way. Using Ansible and Packer to provision AMIs, and then Terraform to spin up what's needed, goes a long way, without introducing a new layer of technology.
AYBABTME 6y ago
But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
- dolni 6y ago
  > But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-refresh-for-ec2-auto-scaling/ https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one. Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes? > I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1. The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently. There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through. None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.