4 ms·

I've been using Caddy in a 3 node HA cluster sharing an anycast bgp address for about 18 months now and it's been fantastic. Certs "just work" across the clust

by 0xEFF 4y ago

I've been using Caddy in a 3 node HA cluster sharing an anycast bgp address for about 18 months now and it's been fantastic. Certs "just work" across the cluster once consul is wired up. I recently added IPv6 which also "just works." greenpau/caddy-security is fantastic and "just works" for OIDC sso. mholt, thanks for recently adding the ability to bind to multiple specific IP addresses by default, this help me conserve precious public IPv4 addresses.

jabart 4y ago
This is a thing? Any docs on this as the top search is this comment.
- francislavoie 4y ago
  For which part? The comment you replied to mentions a bunch of different features, and a plugin.
  - jabart 4y ago
    The HA Cluster with an Anycast IP.
    - francislavoie 4y ago
      They're essentially just saying they run multiple instances of Caddy, and those instances are reached by clients via Anycast. This isn't anything specific to Caddy, that's networking-level stuff, in front of Caddy.
      - jabart 4y ago
        TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. Most anycast setups I have seen have been geographic which means unless you move you won't get routed to the wrong endpoint. *http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.
        lossolo 4y ago
        > TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. It's called BGP protocol, as OP mentioned. This is basically how cloudflare works, you have the same IP for dozens of datacenters around the world. If routing changes and suddenly you are routed to different POP then your TCP session is broken but in most cases this is rare. You could "fix" this by routing TCP packets to POP B when detecting on POP A that session origin was POP B but this would incur additional latency for the rest of the session etc. But this has nothing to do with Caddy, you could use any other software same way using anycast IP.
        tptacek 4y ago
        BGP just makes sure your packets get to the right location. If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.
        lossolo 4y ago
        > BGP just makes sure your packets get to the right location. I think I wasn't clear enough. Yes, BGP will route the packet to the nearest POP for that IP if it was announced in different locations, that's how "anycast IP" works, thanks to how BGP works. OP asked: > or something in front of Caddy So I've described what is it and how it works in simple terms. > If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off. If you are talking only about BGP then of course you are right but I was not talking about solving this using only BGP. You can tunnel the packet to your other POP which was origin, as I mentioned in my answer. So if you have a session established after 3 way handshake in POP B client <-> POP B and suddenly you are routed via POP A then just tunel all packets for that session: client <-> POP A <-> POP B or you could use MPTCP to solve this if supported etc.
    - 0xEFF 4y ago
      This is all in my basement but I tried to build it like I would a rack in a datacenter the stack is: 1: EdgeRouter4 connected to ISP. This operates as a typical router, nothing special other than some static routes to item 2. 2: Pair of VyOS routers running in VMs. I think of them as top of rack L3 switches in a datacenter. Their primary purpose is for ECMP [1]. They also use VRRP so the can be upgraded without downtime. 3: Three node caddy cluster I mentioned. Running plain old Debian. anycast-healthchecker configures the anycast IP on each VM's loopback interface. Bird advertises the address to the two VyOS routers via BGP. 4: Caddy binds to each address. All of the hosts in this network are configured with a default route to the VRRP address on the VyOS VM's. That's pretty much it. I've recently started using Calico in k8s on some separate VM's, and they work the exact same way, advertising Services of type LoadBalancer to the VyOS routers which does ECMP across the nodes. I learned quite a bit configuring anycast-healthchecker with Caddy, then comparing it to how Calico works. [1]: https://codecave.cc/multipath-routing-in-linux-part-2.html https://codecave.cc/multipath-routing-in-linux-part-2.html
      - jabart 4y ago
        Got it, ECMP was the missing link to that and makes sense. Thanks!

by 0xEFF 4y ago

jabart 4y ago

This is a thing? Any docs on this as the top search is this comment.

francislavoie 4y ago
For which part? The comment you replied to mentions a bunch of different features, and a plugin.
- jabart 4y ago
  The HA Cluster with an Anycast IP.
  - francislavoie 4y ago
    They're essentially just saying they run multiple instances of Caddy, and those instances are reached by clients via Anycast. This isn't anything specific to Caddy, that's networking-level stuff, in front of Caddy.
    - jabart 4y ago
      TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. Most anycast setups I have seen have been geographic which means unless you move you won't get routed to the wrong endpoint. *http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.
      - lossolo 4y ago
        > TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. It's called BGP protocol, as OP mentioned. This is basically how cloudflare works, you have the same IP for dozens of datacenters around the world. If routing changes and suddenly you are routed to different POP then your TCP session is broken but in most cases this is rare. You could "fix" this by routing TCP packets to POP B when detecting on POP A that session origin was POP B but this would incur additional latency for the rest of the session etc. But this has nothing to do with Caddy, you could use any other software same way using anycast IP.
        tptacek 4y ago
        BGP just makes sure your packets get to the right location. If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.
        lossolo 4y ago
        > BGP just makes sure your packets get to the right location. I think I wasn't clear enough. Yes, BGP will route the packet to the nearest POP for that IP if it was announced in different locations, that's how "anycast IP" works, thanks to how BGP works. OP asked: > or something in front of Caddy So I've described what is it and how it works in simple terms. > If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off. If you are talking only about BGP then of course you are right but I was not talking about solving this using only BGP. You can tunnel the packet to your other POP which was origin, as I mentioned in my answer. So if you have a session established after 3 way handshake in POP B client <-> POP B and suddenly you are routed via POP A then just tunel all packets for that session: client <-> POP A <-> POP B or you could use MPTCP to solve this if supported etc.
  - 0xEFF 4y ago
    This is all in my basement but I tried to build it like I would a rack in a datacenter the stack is: 1: EdgeRouter4 connected to ISP. This operates as a typical router, nothing special other than some static routes to item 2. 2: Pair of VyOS routers running in VMs. I think of them as top of rack L3 switches in a datacenter. Their primary purpose is for ECMP [1]. They also use VRRP so the can be upgraded without downtime. 3: Three node caddy cluster I mentioned. Running plain old Debian. anycast-healthchecker configures the anycast IP on each VM's loopback interface. Bird advertises the address to the two VyOS routers via BGP. 4: Caddy binds to each address. All of the hosts in this network are configured with a default route to the VRRP address on the VyOS VM's. That's pretty much it. I've recently started using Calico in k8s on some separate VM's, and they work the exact same way, advertising Services of type LoadBalancer to the VyOS routers which does ECMP across the nodes. I learned quite a bit configuring anycast-healthchecker with Caddy, then comparing it to how Calico works. [1]: https://codecave.cc/multipath-routing-in-linux-part-2.html https://codecave.cc/multipath-routing-in-linux-part-2.html
    - jabart 4y ago
      Got it, ECMP was the missing link to that and makes sense. Thanks!