11 ms·

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like

• redis_mlc 6y ago
  > For instance, we had several outages when upgrading the kubernetes version in our clusters. Yes, this was a big problem in 2018-2019. Are things smoother now in your experience? What magnified the problems where I was working at was that the India team would silently do the k8s upgrade overnight, and when the US office opened in the morning we'd find problems several hours later. So several hours of downtime.
• superbcarrot 6y ago
  Sounds like madness. Should we expect a new tool that orchestrates all of your Kubernetes clusters?
  • bo0tzz 6y ago
    https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api
    • seneca 6y ago
      CAPI is for creating clusters only. It doesn't offer any orchestration.
      • moondev 6y ago
        I'm curious what you mean here, since CAPI allows you to manage and scale clusters like any other Kubernetes object. You can literally "kubectl scale --replicas" a cluster control-plane and worker nodes.
  • bcrosby95 6y ago
    A Kubernetes for Kubernetes! But what about when we want to upgrade that?
    • dabeeeenster 6y ago
      I mean thats simply Kubernetes for Kubernetes for Kubernetes. Lets go!
      • testplzignore 6y ago
        I need a Kubernetes Kubernetes Kubernetes Kubernetes Kubernetes emulator so I can run Civilization 14 on my iPhone 35XSXS.
        • deleted 6y ago
          [deleted]
      • higeorge13 6y ago
        Inception!
    • deleted 6y ago
      [deleted]
    • verst 6y ago
      In a way this reminds me of Kubernetes Virtual Clusters. Each virtual cluster has its own tenant control plane, namespaces. Multiple virtual clusters exist in a super cluster. https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster https://github.com/kubernetes-sigs/multi-tenancy/tree/master...
    • tutfbhuf 6y ago
      I think at a certain point up in the stack you don't want to have another orchestrator for the orchestrator. You would just tell each team to keep their cluster up-to-date without creating a (dangerous) master button to update them all at once.
    • 6y ago
  • thirdlamp 6y ago
    it's kubernetes all the way down
  • codeulike 6y ago
    Kubermeta
  • mullingitover 6y ago
    Rancher does this nicely.
  • YawningAngel 6y ago
    You end up needing this anyway, because Kubernetes clusters are confined to single geographic regions and you need some way of managing the separate cluster(s) you have in each region
    • WJW 6y ago
      Not saying you should, but it's entirely possible to have nodes all over the world managed by a single set of master nodes. The system is not really designed with that use case in mind though, the added latencies will hurt a lot in getting etcd consensus.
      • tutfbhuf 6y ago
        While it is certainly possible to have mixed region clusters, from my experience it is better to have one cluster per region. Do your synchronous communication within the cluster / within the datacenter with sub millisecond latencies and do asynchronous communication across cluster boundaries with up to triple digit millisecond latencies.
  • verdverm 6y ago
    KubeFed - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
  • theptip 6y ago
    It used to be called Ubernetes, shame that name didn't stick.
  • jknoepfler 6y ago
    I operate one at work, actually. It's not so bad.
  • yongjik 6y ago
    > There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare
    • ex_amazon_sde 6y ago
      Ex Amazon SDE here. I would pick the first method a hundred times. People would be surprised at how simple the internal infra is, given the fleet size, compared to stuff like k8s. (I'm talking about the infra that runs on bare metal, not AWS)
      • paulryanrogers 6y ago
        You mean the infra behind non-AWS services like retail Amazon.com? Or the infra underlying AWS itself?
        • ex_amazon_sde 6y ago
          The first one, but I'm trying not to divulge too many details on the architecture of Amazon - as you can imagine.
          • paulryanrogers 6y ago
            Fascinating because I would have thought the whole idea is AWS is dog-fooded by retail. Yet in truth AWS is itself too expensive and / or complex for Amazon itself! (Or at least the original part of it.)
            • ex_amazon_sde 6y ago
              The majority of retail is dog-fooding AWS, but a lot of code was written before AWS existed or before it provided many of the required services.
      • 6y ago
  • Aeolun 6y ago
    I think we’re already there? Maybe we’re better off looking for a cluster orchestrator orchestrator.
  • Turbots 6y ago
    What, like the [Cluster API](https://cluster-api.sigs.k8s.io/ https://cluster-api.sigs.k8s.io/), [RedHat Advanced Cluster Mgmt](https://www.redhat.com/en/resources/advanced-cluster-management-FAQ https://www.redhat.com/en/resources/advanced-cluster-managem...), [Rancher](https://rancher.com/ https://rancher.com/), [VMware Tanzu Mission Control](https://tanzu.vmware.com/mission-control https://tanzu.vmware.com/mission-control)? Most large enterprises that have been using Kubernetes for a while, have many dozens or hundreds of Kubernetes clusters running. Clusters for different Business Groups, environments or clouds. Something needs to manage User Access, Network policies, storage policies/quotas, Security Policies across those clusters. Something needs to provision, scale, patch and upgrade those clusters in an automated way. There are 4 major releases of Kubernetes a year, probably 20-30 minor releases and patch versions. Enterprises need to keep those up to date. Terraform, Ansible or Chef/Puppet won't solve that problem without spending hundreds of mandays per year on it.
  • deleted 6y ago
    [deleted]
  • seneca 6y ago
    That's what Federation in Kubernetes was for, sort of, but it has never really come to fruition. The closest I've seen is Admiralty, but I'm not sure how it actually works out in reality.
    • p_l 6y ago
      Federation v2 is nice, but not many places use it. Unfortunately, all my clusters are either confined locally anyway, or are deliberately supposed to be separated in all ways :(
  • 6nf 6y ago
    Agreed, this is absolute insanity and people can't see it from the weeds.
  • jachee 6y ago
    Isn't that basically what helmfile[0] does? It lets you declaratively define helm deployments, which in turn orchestrate k8s infrastructure. 0 - https://github.com/roboll/helmfile https://github.com/roboll/helmfile
  • nyellin 6y ago
    Kubernetes solves a problem. It isn't appropriate for all use cases, but the real problem isn't kubernetes but rather the knowledge gap around new technology.
• colllectorof 6y ago
  The tech churn cycle is getting more and more insane. It's the same process repeating endlessly. 1. Identify one problem you want to fix and ignore everything else. 2. Make a tool to manage the problem while still ignoring everything else. 3. Hype the tool up and shove it in every niche and domain possible. 4. Observer how "everything else" bites you in the ass. 5. Identify the worst problem from #4, use it to start the whole process again. Microservices will save us! Oh no, they make things complicated. Well, I will solve that with containers! On no, managing containers is a pain. Container orchestration! Oh no, our clusters fail. Meanwhile, the complexity of our technology goes up and its reliability in practice goes down. Plus, the concepts we operate with are less and less tethered to reality, which makes even talking about certain issues really hard.
  • Karunamon 6y ago
    I've got a different read on this. It's always been complicated, it's just that each.. I don't want to say "generation", but roughly the same concept, grew up with and internalized and knew about the complexities of the tech stacks they learned, and so when something comes about that moves the abstraction one level higher than what people are used to, it's seen as unstable crap. This isn't some ageist kids-these-days thing, I mean that the complexity has always existed, and it's always abstracted one level higher over time. Kubernetes doesn't exist because Google and Google-adjacent engineers wanted to foist off "ooh, shiny" on the world, it exists because it solves a problem with containers at scale. Containers solve a problem with resource usage at scale, which are really just an evolution of VMs and a response to their inefficient resource usage and brittleness at scale, and even they were once seen as the new hot fancy overly-complicated thing. For context, I entered the IT workforce just as virtual machines were being introduced into the average enterprise, and a distressingly high amount of the complaints I hear about containers and k8s were being used against VMs as well. I'm curious to see where we go from k8s. What the next level of abstraction up will be.
    • Lucasoato 6y ago
      The fact that this is similar to something already happened reassures me a little. May Lambda and other FAAS frameworks be considered the next layer of abstraction? (even if cloud providers are using containers underneath)
• avereveard 6y ago
  What are the best practices for transactional data storage in multi cluster environments, just ditch databases and go for distributed, raft based, nosql variants?
  • pojzon 6y ago
    Thats a very good question Im also interested in knowing the answer. Currently I have seen that every company I had a pleasure to work with had to settle with „eventual” consistency of replicated masters. This is ofcourse totally unacceptable in case of some industries. Thats why I would like to know whether someone found a holy grail of multi-region database design and is willing to share it for free!
    • tutfbhuf 6y ago
      Eventual consistency happens when you allow your write transaction to finish, before the data has been spread across all masters. It allows you to have fast writes, but it's actually cheating (in my opinion), because only one master (the nearest) has seen your write, while the others don't know about it yet. If you want to have a strong consistent multi-region database (or other form of storage), then you have to deal with slow writes, because your transaction is only allowed to terminate successfully when all masters have written the new data and are in a synchronous state. I very much prefer slow strong consistent writes instead of fast eventual consistent writes.
    • h43k3r 6y ago
      Lookup Google's Spanner. It's a distributed transactional SQL database with whatever consistency levels you are interested in Disclaimer- Google employee who is heavily using Spanner
      • tutfbhuf 6y ago
        Yes, but Google Spanner is not open source, so it's not so easy to actually look at it. But what is known to public is that Spanner relies on perfectly synchronized atomic clocks within their datacenters, called TrueTime. This solution is very special and does not work for other databases in general.
• hinkley 6y ago
  There's a fun one I hadn't thought about that was mentioned here sometime last summer: What do you do if your Raft data is in zookeeper and you want to move it to etcd or consul? We need 'migrations' for our non-db data too.
• xiwenc 6y ago
  Are you implying we need a manager1 (k8s) to manage containers? This manager1 cannot be trusted so introduce manager2 to manage manager1? What’s next? We need to look forward to manager3? I dont like where this is going
• flurdy 6y ago
  Which is why GCP's decision last year [1] to encourage people to only use 1 cluster with GKE for all of your minor, internal, exploring, testing, side and hobby projects was so frustrating. For your main production load at work, paying management fees for many clusters is not that relevant costs. So not an issue with my clients' projects. But for my many hobby-projects, dev and few self-employed business workloads balancing them all on as few clusters as possible is a major pain. Before that I/you scaled it out to separation of concerns, maybe only grouping a few workloads. And then quite happily scrapping whole clusters all the time. Now any lifecycle issues with them affects lots of independent applications. They went from cattle to pets over night. [1] https://www.reddit.com/r/kubernetes/comments/fdgblk/google_gke_to_introduce_management_fees_by_june/ https://www.reddit.com/r/kubernetes/comments/fdgblk/google_g...