7 ms·

I don't know what your internal architecture looks like, but on the face of it, arguably you should be running Dev and QA loads on the same cluster. Either you

I don't know what your internal architecture looks like, but on the face of it, arguably you should be running Dev and QA loads on the same cluster. Either you have an organization where an SRE team is responsible for running clusters for teams, in which case why not run Dev and QA on the same cluster, or you are responsible for last-line-of-support for teams responsible for running their own clusters, in which case you say, here's a new version of Kubernetes (e.g. a Terraform module) and in this sprint you are responsible for deploying it through dev, QA, pre-prod, and production clusters. Especially if you have separate Kubernetes clusters per product line (do you really need separate dev Kubernetes clusters per product line?).

dalyons 3y ago
unless you have some strict security reasons, why even have separate prod clusters per product? We definitely don’t. seems to kinda miss the point of k8s - aka running all kinds of mixed workloads with separation within a cluster if you need it.
- solatic 3y ago
  Agreed. Separate prod clusters per product usually fixes an organizational problem (lack of trust that running shared workloads would/could be safe, versus giving each team their own servers) first. A lot of organizations, sadly, prefer to pay for separate clusters than to set up RBAC, ResourceQuotas, etc. Hypothetically, since the scaling limit for Kubernetes clusters is ~10,000 nodes (last I checked), you could have multiple product lines that took up 10k+ nodes each. Then there's no reason why not to split by product line. But in the beginning it should be fine. There's also edge cases - Kubernetes doesn't support setting resource requests or limits for networking or I/O, which you usually solve by setting up taints/tolerations/affinity to manually schedule those workloads onto nodes where you've manually run the numbers. But still not usually a reason to prefer separate production clusters.
  - p_l 3y ago
    Arguably, the design of kubernetes is for clusters to be based on physical clusters - to the point of a cluster per aisle (or multiple aisles), and resources of those clusters then being used to deploy applications even across clusters. Or in the small, like Chick-Fil-A, with kubernetes running locally at every restaurant.
    - solatic 3y ago
      Yup, this is how you scale an application past the 10k node soft limit.
      - p_l 3y ago
        Not just past 10k node limit - it's also because the k8s design has unsaid assumption about being used to arrange physical sets of machines, similar to how Borg paper would talk about cluster being equivalent to an aisle in datacenter.
        solatic 3y ago
        Are you just referring to how you assign labels and taints to nodes, which in a physical datacenter could include labels like which aisle the baremetal server is in? Of course that's feasible, to tell the Kubernetes scheduler that you only want to schedule the workload on a very specific server, but that's not really best-practice...
        p_l 3y ago
        No, I'm referring to bundling machines into clusters across physical lines, with bare metal deployments.
- JackSlateur 3y ago
  Easier to track expenses (for cost reporting etc) : each products are billed separately Easier to manage because less dependencies : you are the sole user, so you can do whatever you want This is the greatness of public cloud, where multi team synchronisation can be reduced a lot For very small projects, one cluster per product is expensive, so mutualisation makes sense. Past a certain size, this is not the case
  - solatic 3y ago
    > separate billing IME mixing financial concerns with engineering concerns puts you into an architectural corner. Suddenly the business wants an integration between the two projects - which cluster does the integration run on? Do you spin up a whole new cluster / account just for the integration? Inside a project, you need to run dedicated infrastructure for large customers. How do you report that for billing? Just set up tagging correctly. > For very small projects, one cluster per product is expensive, so mutualisation makes sense. If you're not working for a BigCo, every project starts out very small ;) You shouldn't optimize prematurely.
    - JackSlateur 3y ago
      I meant "small" as in "can live on a 3*4GB RAM cluster" About integration, I do not understand what you say You have a provider and a consumer, each of them build (and pay) their own parts
      - solatic 3y ago
        I've worked on ETL integrations in the past where there's an API for service A that's read-only and an API for service B that's write-only, service A is provided by a third-party vendor (so you're not exactly going to get the vendor's code to alter or fork it) and politically the people who work on the integration are different from the people who are responsible for service B; the integrators neither have permission to make alterations to service B nor will they get service B engineers to take backlog items to integrate read functionality or otherwise integrate with the vendor for service A directly. The integrators simply need to build a completely separate integration that lives outside of service A and service B to ETL from service A into service B. If service A is in one cluster, and service B is in another cluster, where billing is per-cluster, in which cluster do you put the ETL integration? It belongs to either both or neither, depending on how Finance wants to bill it.
- JohnMakin 3y ago
  Sometimes certain use cases require clusters running in different regions of the world, or have to run in a specific region. There's also fail over. Personally I always strive to use a "mono" cluster unless I can't avoid otherwise. It does happen.
- natbobc 3y ago
  we have hundreds of clients most on k8s or openshift. Many of them are shifting to smaller team or division oriented clusters so that they can move at their own pace in consuming capabilities, tools, security practises, etc. It also simplifies distribution of operational expenses, yea there’s tools out there to help with that but it’s a whole lot easier to hand someone a sub-account in AWS and call it a day. Additionally it reduces your fault boundary. It isn’t as sexy as saying you have a 5000 node cluster with 100k+ pods running but it can make life a whole lot less stressful when it comes to upgrades and changes.
  - solatic 3y ago
    So basically, doing to infrastructure the microservices approach? a) if you're going to intentionally make the trade-off to have higher costs in exchange for reducing ops load and simplifying administration within an account, where services make network calls across accounts, then adopting ECS/Fargate is probably a significant improvement over Kubernetes. b) The underlying engineering / financial reality is still there and very much a leaky abstraction that will show up on your AWS bill. Cross-VPC, cross-region, and cross-AZ networking costs are very real overhead that you must consider; you either still have centralized network planning with shared VPCs and subnets or you basically decide to give up and let AWS send you the bill when teams create their own VPCs, their own subnets, etc. c) setting up DNS / service discovery and network controls within a single cluster is simple. Doing so across account boundaries is not, and deciding to set up solutions like AWS Transit Gateway incur their own costs. I'm sure it simplifies some things for some people, but Finance is going to get upset pretty quickly. Once you start to go down that path, it's very difficult to back out.
    - natbobc 3y ago
      Everything’s benefits and tradeoffs. With this model there’s a clear ownership model. I’m not suggesting it’s right for everyone but it is a trend we’ve noticed with a number of clients over the last 2-3 years.
  - dalyons 3y ago
    imo you've just traded one set of centralized costs (developing tools, security practices that work for all and educating people) for a distributed set of costs with everyone managing their own clusters and duplicating all this work. You've just pushed the cost onto other teams - which can be a fine approach for reducing the central teams stress, but is net more costly to the business. EDIT: Im not saying its necessarily a bad choice, just much more expensive.

dalyons 3y ago

unless you have some strict security reasons, why even have separate prod clusters per product? We definitely don’t. seems to kinda miss the point of k8s - aka running all kinds of mixed workloads with separation within a cluster if you need it.

solatic 3y ago
Agreed. Separate prod clusters per product usually fixes an organizational problem (lack of trust that running shared workloads would/could be safe, versus giving each team their own servers) first. A lot of organizations, sadly, prefer to pay for separate clusters than to set up RBAC, ResourceQuotas, etc. Hypothetically, since the scaling limit for Kubernetes clusters is ~10,000 nodes (last I checked), you could have multiple product lines that took up 10k+ nodes each. Then there's no reason why not to split by product line. But in the beginning it should be fine. There's also edge cases - Kubernetes doesn't support setting resource requests or limits for networking or I/O, which you usually solve by setting up taints/tolerations/affinity to manually schedule those workloads onto nodes where you've manually run the numbers. But still not usually a reason to prefer separate production clusters.
- p_l 3y ago
  Arguably, the design of kubernetes is for clusters to be based on physical clusters - to the point of a cluster per aisle (or multiple aisles), and resources of those clusters then being used to deploy applications even across clusters. Or in the small, like Chick-Fil-A, with kubernetes running locally at every restaurant.
  - solatic 3y ago
    Yup, this is how you scale an application past the 10k node soft limit.
    - p_l 3y ago
      Not just past 10k node limit - it's also because the k8s design has unsaid assumption about being used to arrange physical sets of machines, similar to how Borg paper would talk about cluster being equivalent to an aisle in datacenter.
      - solatic 3y ago
        Are you just referring to how you assign labels and taints to nodes, which in a physical datacenter could include labels like which aisle the baremetal server is in? Of course that's feasible, to tell the Kubernetes scheduler that you only want to schedule the workload on a very specific server, but that's not really best-practice...
        p_l 3y ago
        No, I'm referring to bundling machines into clusters across physical lines, with bare metal deployments.
JackSlateur 3y ago
Easier to track expenses (for cost reporting etc) : each products are billed separately Easier to manage because less dependencies : you are the sole user, so you can do whatever you want This is the greatness of public cloud, where multi team synchronisation can be reduced a lot For very small projects, one cluster per product is expensive, so mutualisation makes sense. Past a certain size, this is not the case
- solatic 3y ago
  > separate billing IME mixing financial concerns with engineering concerns puts you into an architectural corner. Suddenly the business wants an integration between the two projects - which cluster does the integration run on? Do you spin up a whole new cluster / account just for the integration? Inside a project, you need to run dedicated infrastructure for large customers. How do you report that for billing? Just set up tagging correctly. > For very small projects, one cluster per product is expensive, so mutualisation makes sense. If you're not working for a BigCo, every project starts out very small ;) You shouldn't optimize prematurely.
  - JackSlateur 3y ago
    I meant "small" as in "can live on a 3*4GB RAM cluster" About integration, I do not understand what you say You have a provider and a consumer, each of them build (and pay) their own parts
    - solatic 3y ago
      I've worked on ETL integrations in the past where there's an API for service A that's read-only and an API for service B that's write-only, service A is provided by a third-party vendor (so you're not exactly going to get the vendor's code to alter or fork it) and politically the people who work on the integration are different from the people who are responsible for service B; the integrators neither have permission to make alterations to service B nor will they get service B engineers to take backlog items to integrate read functionality or otherwise integrate with the vendor for service A directly. The integrators simply need to build a completely separate integration that lives outside of service A and service B to ETL from service A into service B. If service A is in one cluster, and service B is in another cluster, where billing is per-cluster, in which cluster do you put the ETL integration? It belongs to either both or neither, depending on how Finance wants to bill it.
JohnMakin 3y ago
Sometimes certain use cases require clusters running in different regions of the world, or have to run in a specific region. There's also fail over. Personally I always strive to use a "mono" cluster unless I can't avoid otherwise. It does happen.
natbobc 3y ago
we have hundreds of clients most on k8s or openshift. Many of them are shifting to smaller team or division oriented clusters so that they can move at their own pace in consuming capabilities, tools, security practises, etc. It also simplifies distribution of operational expenses, yea there’s tools out there to help with that but it’s a whole lot easier to hand someone a sub-account in AWS and call it a day. Additionally it reduces your fault boundary. It isn’t as sexy as saying you have a 5000 node cluster with 100k+ pods running but it can make life a whole lot less stressful when it comes to upgrades and changes.
- solatic 3y ago
  So basically, doing to infrastructure the microservices approach? a) if you're going to intentionally make the trade-off to have higher costs in exchange for reducing ops load and simplifying administration within an account, where services make network calls across accounts, then adopting ECS/Fargate is probably a significant improvement over Kubernetes. b) The underlying engineering / financial reality is still there and very much a leaky abstraction that will show up on your AWS bill. Cross-VPC, cross-region, and cross-AZ networking costs are very real overhead that you must consider; you either still have centralized network planning with shared VPCs and subnets or you basically decide to give up and let AWS send you the bill when teams create their own VPCs, their own subnets, etc. c) setting up DNS / service discovery and network controls within a single cluster is simple. Doing so across account boundaries is not, and deciding to set up solutions like AWS Transit Gateway incur their own costs. I'm sure it simplifies some things for some people, but Finance is going to get upset pretty quickly. Once you start to go down that path, it's very difficult to back out.
  - natbobc 3y ago
    Everything’s benefits and tradeoffs. With this model there’s a clear ownership model. I’m not suggesting it’s right for everyone but it is a trend we’ve noticed with a number of clients over the last 2-3 years.
- dalyons 3y ago
  imo you've just traded one set of centralized costs (developing tools, security practices that work for all and educating people) for a distributed set of costs with everyone managing their own clusters and duplicating all this work. You've just pushed the cost onto other teams - which can be a fine approach for reducing the central teams stress, but is net more costly to the business. EDIT: Im not saying its necessarily a bad choice, just much more expensive.