3 ms·

I'm seeing this "we started out with 2 and now there's dozens" pattern with k8s everywhere, and just like in this example the cause is a one-two-punch combo of

I'm seeing this "we started out with 2 and now there's dozens" pattern with k8s everywhere, and just like in this example the cause is a one-two-punch combo of proliferation of service complexity and geographic replication demands (be they legal or performance). kube was originally meant to abstract the datacenter, is there anyone who's abstracting the globe/internet yet? I'd love a geo-kube abstraction where based on annotations or something I could provision deployments/pods to arbitrary geographic locations. I want a scale-to-zero node-pool in every region, and one kube master api for the world.

gravypod 5y ago
There's active work on a standard called kubefed [0] that is being worked on. > I want a scale-to-zero node-pool in every region, and one kube master api for the world. Personally, I'd generalize this to: "I want to describe the reliability requirements and configuration for my software and have an automated system solve for where, how many, when, and how to route to it" I want to have something where I can say "I need to have high availability, lowest latency, and X GB of RAM and Y cores" and have a system automatically schedule me wherever compute is cheapest while also intelligently routing traffic to my servers based on client origins. [0] - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
- cagenut 5y ago
  does it have to be mulitple federated clusters? can etcd and kubelet not just handle 100ms? (or be adapted to).
  - p_l 5y ago
    While laten ies can get pretty good these days, I'd rather not plan on having 100ms ceiling for multiple DCs without dedicated lines and considerably close geographic locations.
    - cagenut 5y ago
      us east-west coast is in the 60's and us to western eu is in the 70s. hell I'd be content to run the etcd cluster across a tighter range, us-east-1/us-east-2/ca-central-1 are all within 30ms of each other. so I guess its really about kublet and whatever comes after that I'm not thinking of (networking).
  - gravypod 5y ago
    If you have multiple federated clusters you can tolerate issues that affect an entire cluster of machines. Having 3x clusters each with 3x master nodes that do not depend on each other, and Volume drivers that are aware of having replication across regions, you can make some really reliable things. For most use cases though what you're talking about will work and you can do it today with just setting up a VPN to have a flat internal network topology.

gravypod 5y ago

There's active work on a standard called kubefed [0] that is being worked on. > I want a scale-to-zero node-pool in every region, and one kube master api for the world. Personally, I'd generalize this to: "I want to describe the reliability requirements and configuration for my software and have an automated system solve for where, how many, when, and how to route to it" I want to have something where I can say "I need to have high availability, lowest latency, and X GB of RAM and Y cores" and have a system automatically schedule me wherever compute is cheapest while also intelligently routing traffic to my servers based on client origins. [0] - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed

cagenut 5y ago
does it have to be mulitple federated clusters? can etcd and kubelet not just handle 100ms? (or be adapted to).
- p_l 5y ago
  While laten ies can get pretty good these days, I'd rather not plan on having 100ms ceiling for multiple DCs without dedicated lines and considerably close geographic locations.
  - cagenut 5y ago
    us east-west coast is in the 60's and us to western eu is in the 70s. hell I'd be content to run the etcd cluster across a tighter range, us-east-1/us-east-2/ca-central-1 are all within 30ms of each other. so I guess its really about kublet and whatever comes after that I'm not thinking of (networking).
- gravypod 5y ago
  If you have multiple federated clusters you can tolerate issues that affect an entire cluster of machines. Having 3x clusters each with 3x master nodes that do not depend on each other, and Volume drivers that are aware of having replication across regions, you can make some really reliable things. For most use cases though what you're talking about will work and you can do it today with just setting up a VPN to have a flat internal network topology.