4 ms·

> However, while anycast works well in the ingress direction, it can't operate on egress. Establishing an outgoing connection from an anycast IP won't work. Con

by xg15 4y ago

> However, while anycast works well in the ingress direction, it can't operate on egress. Establishing an outgoing connection from an anycast IP won't work. Consider the response packet. It's likely to be routed back to a wrong place - a data center geographically closest to the sender, not necessarily the source data center! Slightly OT question, but why wouldn't this be a problem with ingress, too? E.g. suppose I want to send a request to https://1.2.3.4 https://1.2.3.4. What I don't know is that 1.2.3.4 is an anycast address. So my client sends a SYN packet to 1.2.3.4:443 to open the connection. The packet is routed to data center #1. The data center duly replies with a SYN/ACK packet, which my client answers with an ACK packet. However, due to some bad luck, the ACK packet is routed to data center #2 which is also a destination for the anycast address. Of course, data center #2 doesn't know anything about my connection, so it just drops the ACK or replies with a RST. In the best case, I can eventually resend my ACK and reach the right data center (with multi-second delay), in the worst case, the connection setup will fail. Why does this not happen on ingress, but is a problem for egress? Even if the handshake uses SYN cookies and got through on data center #2, what would keep subsequent packets that I send on that connection from being routed to random data centers that don't know anything about the connection?

matsur 4y ago
This is a problem in theory. In practice (and through experience) we see very little routing instability in the way you describe.
- xg15 4y ago
  You mean, it's just luck?
  - rizky05 4y ago
    calculated luck
  - Brian_K_White 4y ago
    right? also seems like load should or at least could be changing all the time. geo or hops proximity is really the only things that decide a route? not load also? But although I would be surprised if load were not also part of the route picker, I would also be surprised if the routers didn't have some association or state tracking to actively ensure related packets get the same route. But I guess this is saying exactly that, that it's relying on luck and happenstance. It may be doing the job well enough that not enough people complain, but I wouldn't be proud of it myself.
    - remram 4y ago
      Anycast is implemented by BGP and doesn't take load into account in any way. You will reach the closest location announcing that address (well, prefix).
      - ignoramous 4y ago
        TFA claims that Anycast is an advantage when dealing with DDoS because it helps spread the load? A regional DDoS (where it consistently hits a small set of DCs) is not a common scenario, I guess?
        csande17 4y ago
        Basically yes. Large-scale DDoS attacks rely on compromising random servers and devices, either directly with malware or indirectly with reflection attacks. Those hosts aren't all going to be located in the same place. An attacker could choose to only compromise devices located near a particular data center, but that would really reduce the amount of traffic they could generate, and also other data centers would stay online and serve requests from users in other places.
    - richieartoul 4y ago
      Why shouldn’t they be proud of a massive system like Cloudflare that works extremely well? As a commentor below described, it’s not luck or happenstance, it’s a natural consequence of how BGP works. Seems pretty elegant to me.
  - deleted 4y ago
    [deleted]
tonyb 4y ago
It works because the route to 1.2.3.4 is relatively stable. The routes would only change and end up at data center #2 if data center #1 stopped announcing the routes. In that case the connection would just re-negotiate to data center #2.
- xg15 4y ago
  Ah, ok, that makes sense. So for a given point of origin, anycast generally routes to the same server?
  - majke 4y ago
    Correct. From a single place, you're likely to BGP-reach one Cloudflare location, and it doesn't change often.
grogers 4y ago
Yep, it can happen that your packet gets routed to a different DC from a prior packet. But the routers in between the client and the anycast destination will do the same thing if the environment is the same. So to get routed to a new location, you would usually need either: * A new (usually closer) DC comes online. That will probably be your destination from now on. * The prior DC (or a critical link on the path to it) goes down. The crucial thing is that the client will typically be routed to the closest destination to it. In the egress case the current DC may not be the closest DC to the server it is trying to reach so the return traffic would go to the wrong place. This system of identifying a server with unique IP/port(s) means that CF's network can forward the return traffic to the correct place.
- 9935c101ab17a66 4y ago
  Dang this stuff is super interesting. As someone who knows very little about networking and DNS, both your comment and the parent comment are incredible insightful and illuminating! Thanks for sharing.
ignoramous 4y ago
Yes, as others have mentioned, route flapping is a problem. But, in practice, not as big a problem as DNS-based routing. - See: https://news.ycombinator.com/item?id=10636547 https://news.ycombinator.com/item?id=10636547 - And: https://news.ycombinator.com/item?id=17904663 https://news.ycombinator.com/item?id=17904663 Besides, SCTP / QUIC aware load balancers (or proxies) are detached from IPs and should continue to hum along just fine regardless of which server IP the packet ends up at.
ratorx 4y ago
As others have mentioned, this is not often a problem because routing is normally fairly stable (at least compared to the lifetime of a typical connection). For longer lived connections (e.g. video uploads), it’s more of a problem. Also, there are a fair number of ASes that attempt to load balance traffic between multiple peering points, without hashing (or only using the src/dst address and not the port). This will also cause the problem you described. In practice it’s possible to handle this by keeping track of where the connections for an IP address typically ingress and sending packets there instead of handling them locally. Again, since it’s a few ASes that cause problems for typical connections, is also possible to figure out which IP prefixes experience the most instability and only turn on this overlay for them.
error503 4y ago
In addition to what others have said about route stability, there is possibly a more relevant point to make: for 'ingress' traffic, the anycast IP is the destination, and for 'egress' traffic it is the source. This distinction is important because the system doesn't have any reasonable way to anticipate where the reply traffic to the anycast IP will actually route to. A packet is launched to the Internet from an anycast IP, and the replies could come to anywhere that IP is announced - most likely not the same place the connection originated. Contrast to when the (unicast) client makes the first move, and the anycast node that gets the initial packets is highly likely to get any follow up packets as well, so in the vast majority of cases, this works just fine.

by xg15 4y ago

matsur 4y ago

This is a problem in theory. In practice (and through experience) we see very little routing instability in the way you describe.

xg15 4y ago
You mean, it's just luck?
- rizky05 4y ago
  calculated luck
- Brian_K_White 4y ago
  right? also seems like load should or at least could be changing all the time. geo or hops proximity is really the only things that decide a route? not load also? But although I would be surprised if load were not also part of the route picker, I would also be surprised if the routers didn't have some association or state tracking to actively ensure related packets get the same route. But I guess this is saying exactly that, that it's relying on luck and happenstance. It may be doing the job well enough that not enough people complain, but I wouldn't be proud of it myself.
  - remram 4y ago
    Anycast is implemented by BGP and doesn't take load into account in any way. You will reach the closest location announcing that address (well, prefix).
    - ignoramous 4y ago
      TFA claims that Anycast is an advantage when dealing with DDoS because it helps spread the load? A regional DDoS (where it consistently hits a small set of DCs) is not a common scenario, I guess?
      - csande17 4y ago
        Basically yes. Large-scale DDoS attacks rely on compromising random servers and devices, either directly with malware or indirectly with reflection attacks. Those hosts aren't all going to be located in the same place. An attacker could choose to only compromise devices located near a particular data center, but that would really reduce the amount of traffic they could generate, and also other data centers would stay online and serve requests from users in other places.
  - richieartoul 4y ago
    Why shouldn’t they be proud of a massive system like Cloudflare that works extremely well? As a commentor below described, it’s not luck or happenstance, it’s a natural consequence of how BGP works. Seems pretty elegant to me.
- deleted 4y ago
  [deleted]

tonyb 4y ago

It works because the route to 1.2.3.4 is relatively stable. The routes would only change and end up at data center #2 if data center #1 stopped announcing the routes. In that case the connection would just re-negotiate to data center #2.

xg15 4y ago
Ah, ok, that makes sense. So for a given point of origin, anycast generally routes to the same server?
- majke 4y ago
  Correct. From a single place, you're likely to BGP-reach one Cloudflare location, and it doesn't change often.

grogers 4y ago

Yep, it can happen that your packet gets routed to a different DC from a prior packet. But the routers in between the client and the anycast destination will do the same thing if the environment is the same. So to get routed to a new location, you would usually need either: * A new (usually closer) DC comes online. That will probably be your destination from now on. * The prior DC (or a critical link on the path to it) goes down. The crucial thing is that the client will typically be routed to the closest destination to it. In the egress case the current DC may not be the closest DC to the server it is trying to reach so the return traffic would go to the wrong place. This system of identifying a server with unique IP/port(s) means that CF's network can forward the return traffic to the correct place.

9935c101ab17a66 4y ago
Dang this stuff is super interesting. As someone who knows very little about networking and DNS, both your comment and the parent comment are incredible insightful and illuminating! Thanks for sharing.

ignoramous 4y ago

Yes, as others have mentioned, route flapping is a problem. But, in practice, not as big a problem as DNS-based routing. - See: https://news.ycombinator.com/item?id=10636547 https://news.ycombinator.com/item?id=10636547 - And: https://news.ycombinator.com/item?id=17904663 https://news.ycombinator.com/item?id=17904663 Besides, SCTP / QUIC aware load balancers (or proxies) are detached from IPs and should continue to hum along just fine regardless of which server IP the packet ends up at.

ratorx 4y ago

As others have mentioned, this is not often a problem because routing is normally fairly stable (at least compared to the lifetime of a typical connection). For longer lived connections (e.g. video uploads), it’s more of a problem. Also, there are a fair number of ASes that attempt to load balance traffic between multiple peering points, without hashing (or only using the src/dst address and not the port). This will also cause the problem you described. In practice it’s possible to handle this by keeping track of where the connections for an IP address typically ingress and sending packets there instead of handling them locally. Again, since it’s a few ASes that cause problems for typical connections, is also possible to figure out which IP prefixes experience the most instability and only turn on this overlay for them.

error503 4y ago

In addition to what others have said about route stability, there is possibly a more relevant point to make: for 'ingress' traffic, the anycast IP is the destination, and for 'egress' traffic it is the source. This distinction is important because the system doesn't have any reasonable way to anticipate where the reply traffic to the anycast IP will actually route to. A packet is launched to the Internet from an anycast IP, and the replies could come to anywhere that IP is announced - most likely not the same place the connection originated. Contrast to when the (unicast) client makes the first move, and the anycast node that gets the initial packets is highly likely to get any follow up packets as well, so in the vast majority of cases, this works just fine.