6 ms·

It's a cool hack because you probably already have bought the hardware to do this. Most decent switches these days can run BGP (or at least OSPF, which is capab

It's a cool hack because you probably already have bought the hardware to do this. Most decent switches these days can run BGP (or at least OSPF, which is capable of the same thing). And switches are usually way cheaper than hardware load balancers, at least per-port. Sure it's not perfect but for a startup trying not to spend a lot it can get you pretty far.

peterwwillis 12y ago
I don't know. From an infrastructure perspective I don't like the idea of having too many eggs in one basket, like combining the router/switch with the lb vips. Ideally you'd get a couple commodity boxes and configure them with LVS or pfsense or something. That way things like maintenance and access control of different parts of your network are separated based on the resource, and stability of one component won't necessarily affect another. It would also probably be cheaper to buy a couple servers than buy a couple routers/switches for redundancy.
- mprovost 12y ago
  Right, but you already have to buy two switches for redundancy right, what else are you going to plug those servers into? I get what you're saying about stability etc but BGP is a core function of a networking operating system. I do agree that this is not a typical setup and probably not for everyone but it is a clever use of existing protocols which made for an interesting read. I think for some people it could solve a real problem until they have time to do something better.
- phil21 12y ago
  You're not understanding how this works at all. Your router/switch doesn't "combine" the vips. Ignore the running of bgpd on the webserver - that's really an extremely specific use-case that is not useful for most folks. However, imagine your scenario where you have two routers, two switches, and two load balancers configured in failover (LVS per your example) - with webservers behind that stack. Now you need more than a single load balancer worth of capacity? How do you scale it? Generally, you're pretty much stuck doing DNS RR to load balance across VIPs, and you add at least one VIP per load balancer you have. Need to do maintenance? Good luck not directing traffic to the load balancer you want to take out of service :) You can wait 3 days for all the DNS caches in the world to purge, or you're going to be killing sessions when you fail that VIP over. Now consider instead running BGPd vs. DNS RR. You have a single VIP, and as many load balancers as you like. I enjoy HAProxy, so I'll use it here. haproxy01 1.1.1.1 haproxy02 1.1.1.2 haproxy03 1.1.1.3 All these machines advertise the VIP to the switch they are connected to. You set up path cost on your network so all these VIPs share an equal cost at your routers, and your router will ECMP to each. Doesn't matter what switch or rack your HAProxy box is connected to in your network (I suggest paying attention for traffic management reasons) - as long as it can speak BGP to the switch/router, the traffic flows. Need to do maint? Kill bgpd on one of the HAProxy boxes. Current sessions stay up, you go get a coffee, and when you come back you have an empty session table and are free to do whatever you like to the machine. Turn it back on by starting up bgpd, and watch your traffic instantly rebalance. You are completely correct that if you don't need more than one (in a pair for HA) load balancer worth of capacity, this solution is likely overkill. But once you need to start scaling, you're quickly going to learn that it will either be prohibitively expensive, or come with lots of downsides. There pretty much are no downsides to this architecture, other than needing someone with a small amount of clue to operate it.
  - staunch 12y ago
    You don't know about keepalived or something? Your wait 3 days example is actually just bring the VIP up on another haproxy in one second. You can get essentially identical behavior like this. Kill keepalived and other box takes over IP instantly. VRRP.
    - phil21 12y ago
      I haven't kept up on it, when did they add TCP state syncing? Edit: Glanced over the docs, I don't see this listed as an obvious feature. So I'm missing what your point is. If you mean you can failover a VIP instantly, sure I agree. But you're dropping the TCP session for all those clients utilizing that VIP at the moment. I was illustrating how with ECMP you can just wait a few minutes and the sessions migrate away/timeout then you do your maint without impacting a single TCP flow. Might not matter to you, matters to some folks.
      - todayismyday2 12y ago
        Then use state "EQUAL" and nopreempt options. This will make your VIP failover only when one of the hosts dies and keep it there regardless of whether the new host came back up.
  - peterwwillis 12y ago
    What I was saying with that "combining" is that your router is now essentially the vip, in the sense that it is the load balancer and the peers are chosen and routed to from it. As opposed to a normal router which is merely passing routed traffic into the network and letting a different device handle load balancing. The idea being that different devices are used for different purposes, and separation of their functions may improve overall stability and increase flexibility of your network services. One downside here is ECMP assumes all paths cost the same, which is ridiculous in real-world load balancing. One of your haproxies is going to get overloaded and then traffic to your site is going to intermittently suck balls as sessions stream in to both under-loaded and over-loaded boxes. Of course, you have the same problem with round-robin DNS to load balancers, but in the case of a DR LVS load balancer for example, at least it's just starting the connection and handing it off to the appropriate proxy instead of randomly pinning sessions to specific interfaces. With DR it's the backend proxy that determines its return path; the LVS VIP isn't in the path. With LVS it can pick a destination path based on real-world load. The other downside that you seem to gloss over with regard to scaling is the maximum of 16 ECMP addresses in the forwarding table. I'm sure we'll never need more than 16 of those, though....... (For reference: the company I used to work for had up to 23 proxies just for one application... might cause some hiccups with this set-up) Doing maintenance on a vip address and doing maintenance on one of these bgp peers work about the same. You stop accepting new connections, let old connections expire, then take down the vip. As far as changing DNS records, instead of that you can either add a hot-spare vip with the IP of the one you want to maintain, or add the ip of that vip to an existing load balancer.

peterwwillis 12y ago

I don't know. From an infrastructure perspective I don't like the idea of having too many eggs in one basket, like combining the router/switch with the lb vips. Ideally you'd get a couple commodity boxes and configure them with LVS or pfsense or something. That way things like maintenance and access control of different parts of your network are separated based on the resource, and stability of one component won't necessarily affect another. It would also probably be cheaper to buy a couple servers than buy a couple routers/switches for redundancy.

mprovost 12y ago
Right, but you already have to buy two switches for redundancy right, what else are you going to plug those servers into? I get what you're saying about stability etc but BGP is a core function of a networking operating system. I do agree that this is not a typical setup and probably not for everyone but it is a clever use of existing protocols which made for an interesting read. I think for some people it could solve a real problem until they have time to do something better.
phil21 12y ago
You're not understanding how this works at all. Your router/switch doesn't "combine" the vips. Ignore the running of bgpd on the webserver - that's really an extremely specific use-case that is not useful for most folks. However, imagine your scenario where you have two routers, two switches, and two load balancers configured in failover (LVS per your example) - with webservers behind that stack. Now you need more than a single load balancer worth of capacity? How do you scale it? Generally, you're pretty much stuck doing DNS RR to load balance across VIPs, and you add at least one VIP per load balancer you have. Need to do maintenance? Good luck not directing traffic to the load balancer you want to take out of service :) You can wait 3 days for all the DNS caches in the world to purge, or you're going to be killing sessions when you fail that VIP over. Now consider instead running BGPd vs. DNS RR. You have a single VIP, and as many load balancers as you like. I enjoy HAProxy, so I'll use it here. haproxy01 1.1.1.1 haproxy02 1.1.1.2 haproxy03 1.1.1.3 All these machines advertise the VIP to the switch they are connected to. You set up path cost on your network so all these VIPs share an equal cost at your routers, and your router will ECMP to each. Doesn't matter what switch or rack your HAProxy box is connected to in your network (I suggest paying attention for traffic management reasons) - as long as it can speak BGP to the switch/router, the traffic flows. Need to do maint? Kill bgpd on one of the HAProxy boxes. Current sessions stay up, you go get a coffee, and when you come back you have an empty session table and are free to do whatever you like to the machine. Turn it back on by starting up bgpd, and watch your traffic instantly rebalance. You are completely correct that if you don't need more than one (in a pair for HA) load balancer worth of capacity, this solution is likely overkill. But once you need to start scaling, you're quickly going to learn that it will either be prohibitively expensive, or come with lots of downsides. There pretty much are no downsides to this architecture, other than needing someone with a small amount of clue to operate it.
- staunch 12y ago
  You don't know about keepalived or something? Your wait 3 days example is actually just bring the VIP up on another haproxy in one second. You can get essentially identical behavior like this. Kill keepalived and other box takes over IP instantly. VRRP.
  - phil21 12y ago
    I haven't kept up on it, when did they add TCP state syncing? Edit: Glanced over the docs, I don't see this listed as an obvious feature. So I'm missing what your point is. If you mean you can failover a VIP instantly, sure I agree. But you're dropping the TCP session for all those clients utilizing that VIP at the moment. I was illustrating how with ECMP you can just wait a few minutes and the sessions migrate away/timeout then you do your maint without impacting a single TCP flow. Might not matter to you, matters to some folks.
    - todayismyday2 12y ago
      Then use state "EQUAL" and nopreempt options. This will make your VIP failover only when one of the hosts dies and keep it there regardless of whether the new host came back up.
- peterwwillis 12y ago
  What I was saying with that "combining" is that your router is now essentially the vip, in the sense that it is the load balancer and the peers are chosen and routed to from it. As opposed to a normal router which is merely passing routed traffic into the network and letting a different device handle load balancing. The idea being that different devices are used for different purposes, and separation of their functions may improve overall stability and increase flexibility of your network services. One downside here is ECMP assumes all paths cost the same, which is ridiculous in real-world load balancing. One of your haproxies is going to get overloaded and then traffic to your site is going to intermittently suck balls as sessions stream in to both under-loaded and over-loaded boxes. Of course, you have the same problem with round-robin DNS to load balancers, but in the case of a DR LVS load balancer for example, at least it's just starting the connection and handing it off to the appropriate proxy instead of randomly pinning sessions to specific interfaces. With DR it's the backend proxy that determines its return path; the LVS VIP isn't in the path. With LVS it can pick a destination path based on real-world load. The other downside that you seem to gloss over with regard to scaling is the maximum of 16 ECMP addresses in the forwarding table. I'm sure we'll never need more than 16 of those, though....... (For reference: the company I used to work for had up to 23 proxies just for one application... might cause some hiccups with this set-up) Doing maintenance on a vip address and doing maintenance on one of these bgp peers work about the same. You stop accepting new connections, let old connections expire, then take down the vip. As far as changing DNS records, instead of that you can either add a hot-spare vip with the IP of the one you want to maintain, or add the ip of that vip to an existing load balancer.