6 ms·

Slightly aside of BGP on an internet scale - We’ve recently switched to using BGP internally within our network for kubernetes + load balancer (nginx) routing,

Slightly aside of BGP on an internet scale - We’ve recently switched to using BGP internally within our network for kubernetes + load balancer (nginx) routing, (announcement / discovery) amongst other things and I can tell you - it’s one of the best moves we’ve ever made. The simplicity, performance and reliability is brilliant. https://www.projectcalico.org https://www.projectcalico.org

deleted 9y ago
[deleted]
_nalply 9y ago
Do you have loops internally?
- noselasd 9y ago
  I would think people deploy BGP to automate routing and routing policies, not primarily to avoid loops.
  - mrmondo 9y ago
    Indeed, completely for fast, simple routing (rather than higher level, more complex failover). Slightly off topic, but in my experience most (general) internal networking loops stem from mis configured devices bridging networks (desktop level, network device level or VM / SDN)
- bartvanH 9y ago
  I may be old, but if you have a network loop inside your own infrastructure wouldn't it be way more sane to handle that on layer 2? for instance with properly configured spanning tree?
  - woah 9y ago
    A routing protocol will avoid loops more efficiently
    - jethro_tell 9y ago
      Also, it's allows loops to be used as multi paths from different points in the net. Say you have a rack at south and north and there is a loop to a rack in the middle. For failover, you'll take any path that is up. But for normal operation, you'd prefer to take the shortest path.
      - deleted 9y ago
        [deleted]
      - lima 9y ago
        Layer 2 technologies like MC-LAG do that as well (though I agree that it's better to route, if possible)
    - scheesman 9y ago
      Routing is a Layer 3 issue. Loops are a Layer 2 issue. Multiple paths on a Layer 3 network provide redundancy when used with a routing protocol such as BGP. Multiple paths on a Layer 2 network provide redundancy when used with a protocol such as LACP. Without proper design, 2 network segments connected with multiple links will trigger Spanning Tree and shut down one of the links. And depending on the configuration, it may or may not unshut if the active link pair goes down.
  - elihu 9y ago
    Spanning tree just prunes links from the network to eliminate loops. Sometimes the link that got pruned happened to be the fastest path from point A to point B. So, sometimes you can get a more efficient network if you leave those links in, but that requires more sophisticated routing.
merb 9y ago
OT: is this really "simple" to setup or do you mean that once it's running that it's simple? or what do you mean with simplicity. I actually looking forward to use k8s in our network, too. But I think the simplest solution for HA k8s is using keepalived paired with nginx/haproxy on the master level and add a keepalived-vip for every service. (at least for our 3 node master, 3 node worker setup) (I evaluated the calico + bgp option, too but it looked way to complex, as a k8s starter)
- mrmondo 9y ago
  Yes, it was very simple to setup, deploy and understand on an ongoing process, BGP is incredibly simple by design. I’m heading to bed now (AEDST), but check out: - https://docs.projectcalico.org/latest/getting-started/kubernetes/tutorials/simple-policy https://docs.projectcalico.org/latest/getting-started/kubern... - https://kubernetes.io/docs/tasks/administer-cluster/calico-network-policy/ https://kubernetes.io/docs/tasks/administer-cluster/calico-n... And there’s plenty of other resources out there. IMO if you’re designing a highly available platform, BGP is one of the simplest parts of that complex system. In my experience you’re likely to run into much more finicky problems with higher layer systems including failover delays, reliance on casting systems and state drift.
- scurvy 9y ago
  Kubernetes is very complicated, but BGP is not. If the curve for BGP is too steep, you can do OSPF inside your racks with BGP announcements to spine. This is pretty common as OSPF can use area 0 with little to zero configuration. BGP leaf to spine is standard is layer 3 networks.
- murali-reddy 9y ago
  As an alternative approach may I suggest to look at the way kube-router does. It basically leverages BGP and ECMP to provide HA. https://cloudnativelabs.github.io/post/2017-11-01-kube-high-available-ingress/ https://cloudnativelabs.github.io/post/2017-11-01-kube-high-...
SEJeff 9y ago
We did the same thing but opted for the much simpler kube-router, which is more k8s specific and doesn't have nearly as many moving parts (easier to troubleshoot as a result). Bonus point is uses LVS for service proxy load balancing: https://www.kube-router.io/ https://www.kube-router.io/ https://github.com/cloudnativelabs/kube-router https://github.com/cloudnativelabs/kube-router
- deleted 9y ago
  [deleted]
- lima 9y ago
  Depends on your use case, the network policy features in Calico are very important for many real-world use cases.
  - SEJeff 9y ago
    What does calico do (Ingress and Egress policy) that kube-router does't? https://cloudnativelabs.github.io/post/2017-05-1-kube-network-policies/ https://cloudnativelabs.github.io/post/2017-05-1-kube-networ...

deleted 9y ago

[deleted]

_nalply 9y ago

Do you have loops internally?

noselasd 9y ago
I would think people deploy BGP to automate routing and routing policies, not primarily to avoid loops.
- mrmondo 9y ago
  Indeed, completely for fast, simple routing (rather than higher level, more complex failover). Slightly off topic, but in my experience most (general) internal networking loops stem from mis configured devices bridging networks (desktop level, network device level or VM / SDN)
bartvanH 9y ago
I may be old, but if you have a network loop inside your own infrastructure wouldn't it be way more sane to handle that on layer 2? for instance with properly configured spanning tree?
- woah 9y ago
  A routing protocol will avoid loops more efficiently
  - jethro_tell 9y ago
    Also, it's allows loops to be used as multi paths from different points in the net. Say you have a rack at south and north and there is a loop to a rack in the middle. For failover, you'll take any path that is up. But for normal operation, you'd prefer to take the shortest path.
    - deleted 9y ago
      [deleted]
    - lima 9y ago
      Layer 2 technologies like MC-LAG do that as well (though I agree that it's better to route, if possible)
  - scheesman 9y ago
    Routing is a Layer 3 issue. Loops are a Layer 2 issue. Multiple paths on a Layer 3 network provide redundancy when used with a routing protocol such as BGP. Multiple paths on a Layer 2 network provide redundancy when used with a protocol such as LACP. Without proper design, 2 network segments connected with multiple links will trigger Spanning Tree and shut down one of the links. And depending on the configuration, it may or may not unshut if the active link pair goes down.
- elihu 9y ago
  Spanning tree just prunes links from the network to eliminate loops. Sometimes the link that got pruned happened to be the fastest path from point A to point B. So, sometimes you can get a more efficient network if you leave those links in, but that requires more sophisticated routing.

merb 9y ago

OT: is this really "simple" to setup or do you mean that once it's running that it's simple? or what do you mean with simplicity. I actually looking forward to use k8s in our network, too. But I think the simplest solution for HA k8s is using keepalived paired with nginx/haproxy on the master level and add a keepalived-vip for every service. (at least for our 3 node master, 3 node worker setup) (I evaluated the calico + bgp option, too but it looked way to complex, as a k8s starter)

mrmondo 9y ago
Yes, it was very simple to setup, deploy and understand on an ongoing process, BGP is incredibly simple by design. I’m heading to bed now (AEDST), but check out: - https://docs.projectcalico.org/latest/getting-started/kubernetes/tutorials/simple-policy https://docs.projectcalico.org/latest/getting-started/kubern... - https://kubernetes.io/docs/tasks/administer-cluster/calico-network-policy/ https://kubernetes.io/docs/tasks/administer-cluster/calico-n... And there’s plenty of other resources out there. IMO if you’re designing a highly available platform, BGP is one of the simplest parts of that complex system. In my experience you’re likely to run into much more finicky problems with higher layer systems including failover delays, reliance on casting systems and state drift.
scurvy 9y ago
Kubernetes is very complicated, but BGP is not. If the curve for BGP is too steep, you can do OSPF inside your racks with BGP announcements to spine. This is pretty common as OSPF can use area 0 with little to zero configuration. BGP leaf to spine is standard is layer 3 networks.
murali-reddy 9y ago
As an alternative approach may I suggest to look at the way kube-router does. It basically leverages BGP and ECMP to provide HA. https://cloudnativelabs.github.io/post/2017-11-01-kube-high-available-ingress/ https://cloudnativelabs.github.io/post/2017-11-01-kube-high-...

SEJeff 9y ago

We did the same thing but opted for the much simpler kube-router, which is more k8s specific and doesn't have nearly as many moving parts (easier to troubleshoot as a result). Bonus point is uses LVS for service proxy load balancing: https://www.kube-router.io/ https://www.kube-router.io/ https://github.com/cloudnativelabs/kube-router https://github.com/cloudnativelabs/kube-router

deleted 9y ago
[deleted]
lima 9y ago
Depends on your use case, the network policy features in Calico are very important for many real-world use cases.
- SEJeff 9y ago
  What does calico do (Ingress and Egress policy) that kube-router does't? https://cloudnativelabs.github.io/post/2017-05-1-kube-network-policies/ https://cloudnativelabs.github.io/post/2017-05-1-kube-networ...