4 ms·

I think one of the underrated parts of using something like Kubernetes early (or even w/ simpler orchestrators like swarm or rancher), is that it encourages (an

I think one of the underrated parts of using something like Kubernetes early (or even w/ simpler orchestrators like swarm or rancher), is that it encourages (and sometimes enforces) architecture best practices from the start. IE, you won't be storing state locally, you'll be able to handle servers being randomly killed, you'll already have horizontal scaling, etc. In my experience the hard part of migrating to containers in a legacy app is when they break those constraints, especially around local state and special servers. It's easier to do these things sooner rather than later, and the constraints kubernetes places on you aren't that hard to work around if you design it in from the start. I good reason not to use kubernetes though is if you know your app is probably never going to scale, or if it's the kind of thing that can scale very well on a single machine, or if it's not primarily based on http communication. (I wouldn't write a real time game server in kubernetes, for instance, because I doubt it'd really help with a primarily UDP workload that is likely going to be attached to one server)

throwaway894345 6y ago
I’m not avidly opposed to k8s by any means, but you can get these same properties from any of a variety of easier-to-use schedulers such as Fargate, Heroku, or even EC2 autoscaling groups. Of course, there are probably Kubernetes distributions that lower the threshold of using Kubernetes (and if there aren’t, there really should be) by providing solutions for logging, monitoring, certificate management, Functions (a la AWS Lambda), load balancing / ingress, state management (databases as a service), etc preconfigured out of the box (similar to what you get with AWS or Heroku).
- threeseed 6y ago
  The problem with your approach is that you're firmly locked in to the vendor. And in the case of Fargate, Heroku etc you're paying significantly more than if you had made use of Spot instances or shopped around for a cheaper vendor.
  - sukilot 6y ago
    Heroku costs about a nickel an hour per server, yeah? A dollar a day? So I'll worry about that price when I get to at least 10 servers
    - JamesSwift 6y ago
      A nickel an hour sounds fine/miniscule, but when you realize thats 30$/month vs an alternative VPS of $5/month then I think the cost premium is pretty clear.
  - throwaway894345 6y ago
    Vendor lock-in concerns are overblown. Unless there’s a real chance you’ll need to pick up and move, don’t worry about it. Your savings by not building/operating everything yourself will dwarf other costs (unless your business has huge scale and you have a world class internal cloud capability which you probably don’t and if you do, you can probably just negotiate a better deal from a cloud provider a la Netflix and Amazon). To that end, you only “save money” by managing everything yourself if you write off the cost of engineering time and talent, which is to say you lose money by doing it yourself because you don’t have the scale or talent to compete with Amazon even with their markup (certainly not when you account for opportunity cost).
    - threeseed 6y ago
      Nobody is saying you have to build everything yourself. But you can just use cloud providers for their hardware and not needlessly tie yourself to their software. For example you can use a managed K8s solution like EKS but then have all of your monitoring, logging, databases etc all be self-hosted. And it's not just about cost but also about being able to take advantage of other cloud provider's unique strengths or being truly resilient to outages.
      - throwaway894345 6y ago
        The same arguments apply to hardware and software. Cloud providers’ core competency is cloud infrastructure and services and they have the scale to economize their offerings. Your business very likely can’t compete with them, so to the extent that you’re owning things that cloud providers could sell you, you’re throwing away money and that figure very likely dwarfs the risk adjusted cost of maybe having to migrate to another provider one day. (Of course, there are services that are overpriced here and there, but the general principle holds).
- overgard 6y ago
  So, I think the downside of those systems is that they tend to be specific to a provider, so you get some lock in, and not all of those things are easy/possible to run locally, whereas minikube or docker swarm can replicate a cluster locally, which I think is really important. Otherwise yeah, those systems can be used to accomplish the same goals. One thing that I think is non-obvious: there are easy ways to set up a kubernetes cluster, it's just that there's so many options out there that the easier solutions tend to get lost in the noise. If I had to manage kubernetes nodes manually I'd probably never do it, but setting up something like AKS/EKS/GKS -- they all have their own issues, but they're pretty easy to get started with.
- EdwardDiego 6y ago
  > you can get these same properties from any of a variety of easier-to-use schedulers such as Fargate, Heroku, or even EC2 autoscaling group Serious question, as I don't have any experience of those, can you scale based on a custom metric? It's probably the one feature of k8s I appreciate the most.
  - compumike 6y ago
    On Heroku, I have written a very small autoscaling script that is called once every few minutes by Heroku Scheduler (like a cronjob). The autoscaling script pulls the custom metric, does some arithmetic, and calls the Heroku API to scale the dyno count up/down as desired. (That script was probably shorter and more readable than this comment.)
    - EdwardDiego 6y ago
      Thanks :)
moomin 6y ago
Seriously though, the world needs a solution for Kubernetes like problems that involve UDP and multicast.
- aliswe 6y ago
  Do you mean that UDP capabilities are lacking on Kubernetes workloads? Serious question.
  - moomin 6y ago
    Multicast is definitely lacking (well, it’s completely absent if you’re using Calico), but even for regular UDP there’s a big problem if you care about latency (and if you don’t, why are you using UDP?) Anecdotally, I’ve heard people complain that UDP dropout on their clusters is just too high to actually support the kind of things they want to do. Equally, UDP based products like Tibco FTL just literally aren’t supported on Kubernetes environments. None of this means I have any answers, I’m afraid.
secondcoming 6y ago
> I good reason not to use kubernetes though is if you know your app is probably never going to scale Suggesting that you can only scale with Kubernetes just isn't true. Our tech stack originally ran on AWS in multi-regions. Then we moved to GCP. The architecture was the same for both: - public facing Load Balancer - instance groups attached to the LB - add health checks so unresponsive instances get killed - add scaling rules based on whatever metric you want (AWS's offering was far better than GCP's) - choose your instance type(s). And for the record, we handle ~15 billion HTTP requests per day (traffic is seasonal). Our busiest data centre is us-east, which has a max of 50 instances (16 vCPU, 31GB RAM), and we rarely need them all. You can also mix spot and non-spot instances too, so we know we have a least one or two machines that GCP won't kill. Rolling out updates involves building a debian package, building a new VM image, and then clicking a 'deploy' button (GCP makes this easier than AWS).
- lallysingh 6y ago
  That isn't what parent suggested. The quotation is a reason not to use it.
  - secondcoming 6y ago
    The quotation has a double negative.

throwaway894345 6y ago

I’m not avidly opposed to k8s by any means, but you can get these same properties from any of a variety of easier-to-use schedulers such as Fargate, Heroku, or even EC2 autoscaling groups. Of course, there are probably Kubernetes distributions that lower the threshold of using Kubernetes (and if there aren’t, there really should be) by providing solutions for logging, monitoring, certificate management, Functions (a la AWS Lambda), load balancing / ingress, state management (databases as a service), etc preconfigured out of the box (similar to what you get with AWS or Heroku).

threeseed 6y ago
The problem with your approach is that you're firmly locked in to the vendor. And in the case of Fargate, Heroku etc you're paying significantly more than if you had made use of Spot instances or shopped around for a cheaper vendor.
- sukilot 6y ago
  Heroku costs about a nickel an hour per server, yeah? A dollar a day? So I'll worry about that price when I get to at least 10 servers
  - JamesSwift 6y ago
    A nickel an hour sounds fine/miniscule, but when you realize thats 30$/month vs an alternative VPS of $5/month then I think the cost premium is pretty clear.
- throwaway894345 6y ago
  Vendor lock-in concerns are overblown. Unless there’s a real chance you’ll need to pick up and move, don’t worry about it. Your savings by not building/operating everything yourself will dwarf other costs (unless your business has huge scale and you have a world class internal cloud capability which you probably don’t and if you do, you can probably just negotiate a better deal from a cloud provider a la Netflix and Amazon). To that end, you only “save money” by managing everything yourself if you write off the cost of engineering time and talent, which is to say you lose money by doing it yourself because you don’t have the scale or talent to compete with Amazon even with their markup (certainly not when you account for opportunity cost).
  - threeseed 6y ago
    Nobody is saying you have to build everything yourself. But you can just use cloud providers for their hardware and not needlessly tie yourself to their software. For example you can use a managed K8s solution like EKS but then have all of your monitoring, logging, databases etc all be self-hosted. And it's not just about cost but also about being able to take advantage of other cloud provider's unique strengths or being truly resilient to outages.
    - throwaway894345 6y ago
      The same arguments apply to hardware and software. Cloud providers’ core competency is cloud infrastructure and services and they have the scale to economize their offerings. Your business very likely can’t compete with them, so to the extent that you’re owning things that cloud providers could sell you, you’re throwing away money and that figure very likely dwarfs the risk adjusted cost of maybe having to migrate to another provider one day. (Of course, there are services that are overpriced here and there, but the general principle holds).
overgard 6y ago
So, I think the downside of those systems is that they tend to be specific to a provider, so you get some lock in, and not all of those things are easy/possible to run locally, whereas minikube or docker swarm can replicate a cluster locally, which I think is really important. Otherwise yeah, those systems can be used to accomplish the same goals. One thing that I think is non-obvious: there are easy ways to set up a kubernetes cluster, it's just that there's so many options out there that the easier solutions tend to get lost in the noise. If I had to manage kubernetes nodes manually I'd probably never do it, but setting up something like AKS/EKS/GKS -- they all have their own issues, but they're pretty easy to get started with.
EdwardDiego 6y ago
> you can get these same properties from any of a variety of easier-to-use schedulers such as Fargate, Heroku, or even EC2 autoscaling group Serious question, as I don't have any experience of those, can you scale based on a custom metric? It's probably the one feature of k8s I appreciate the most.
- compumike 6y ago
  On Heroku, I have written a very small autoscaling script that is called once every few minutes by Heroku Scheduler (like a cronjob). The autoscaling script pulls the custom metric, does some arithmetic, and calls the Heroku API to scale the dyno count up/down as desired. (That script was probably shorter and more readable than this comment.)
  - EdwardDiego 6y ago
    Thanks :)

moomin 6y ago

Seriously though, the world needs a solution for Kubernetes like problems that involve UDP and multicast.

aliswe 6y ago
Do you mean that UDP capabilities are lacking on Kubernetes workloads? Serious question.
- moomin 6y ago
  Multicast is definitely lacking (well, it’s completely absent if you’re using Calico), but even for regular UDP there’s a big problem if you care about latency (and if you don’t, why are you using UDP?) Anecdotally, I’ve heard people complain that UDP dropout on their clusters is just too high to actually support the kind of things they want to do. Equally, UDP based products like Tibco FTL just literally aren’t supported on Kubernetes environments. None of this means I have any answers, I’m afraid.

secondcoming 6y ago

> I good reason not to use kubernetes though is if you know your app is probably never going to scale Suggesting that you can only scale with Kubernetes just isn't true. Our tech stack originally ran on AWS in multi-regions. Then we moved to GCP. The architecture was the same for both: - public facing Load Balancer - instance groups attached to the LB - add health checks so unresponsive instances get killed - add scaling rules based on whatever metric you want (AWS's offering was far better than GCP's) - choose your instance type(s). And for the record, we handle ~15 billion HTTP requests per day (traffic is seasonal). Our busiest data centre is us-east, which has a max of 50 instances (16 vCPU, 31GB RAM), and we rarely need them all. You can also mix spot and non-spot instances too, so we know we have a least one or two machines that GCP won't kill. Rolling out updates involves building a debian package, building a new VM image, and then clicking a 'deploy' button (GCP makes this easier than AWS).

lallysingh 6y ago
That isn't what parent suggested. The quotation is a reason not to use it.
- secondcoming 6y ago
  The quotation has a double negative.