5 ms·

TCP Over IP Anycast – Pipe Dream or Reality?

81 points by r4um 11y ago

jakozaur 11y ago
Would be interested to read how CloudFlare solve flapping problem with their Anycast network: https://blog.cloudflare.com/a-brief-anycast-primer/ https://blog.cloudflare.com/a-brief-anycast-primer/
- samwillis 11y ago
  I may be wrong but the impression I get is that CloudFlare have many many more POP much closer to the visitor, quite often in the major consumer ISPs own data centres. This should reduce the probability of what LinkedIn saw happening.
Animats 11y ago
What is LinkedIn sending that generates so much traffic that they need this? They're not Netflix. (Never mind, I just looked at LinkedIn, which I don't use much. With their new layout, I had to scroll through three screens of ads to get to anything that affected me.)
- creshal 11y ago
  But the ads are on (untrusted, ill-maintained) third-party CDNs anyway…
- forty 11y ago
  This is to improve latency, not throughput.
- deleted 11y ago
  [deleted]
- _ao789 11y ago
  It's a pity this study was done on LinkedIn and not something more bandwidth hungry, say a video site maybe?
- k33n 11y ago
  Looks like they're trying to shave MS off of user requests any way they can. Was probably a pretty fun project. Beats sitting and staring at an empty JIRA task list for 8 hours.
- ape4 11y ago
  This article :)
- riteshm 11y ago
  This was for improving performance of our site, not throughput.
pyvpx 11y ago
it has been a reality for many folks for years. real-world anycast can be tricky, but it is certainly deployable and usable.
SixSigma 11y ago
Plan9 can turn any protocol into TCP via 9p. Connect to a remote plan9 machine by whatever means using 9p, import /net from the remote machine into your local namespace, et viola you now have all the network connections of that remote machine available, including the tcp stack.
- detaro 11y ago
  What does this have to do with the article?
  - SixSigma 11y ago
    Any protocol over any protocol. Never a pipe dream.
    - detaro 11y ago
      Anycast is not a specific protocol, but about how (normally IP-)packets are routed. Transporting protocols in other protocols has nothing to do with it and doesn't help.
sspies 11y ago
We are working on this as well at http://datapath.io http://datapath.io. You can use multiple iaas providers using global elastic IP addresses that you can announce from multiple locations at the same time. If you want to participate in our beta please write to beta@datapath.io Best, Sebastian
paulsutter 11y ago
We had the same experience at Quantcast. Better results with anycast within continents, and DNS to distribute a per-continent anycast addresses. Some know-it-all busybodies like to proclaim that anycast can't work for TCP because the route can change during the session, but the reality is that: - route changes are infrequent - individual sessions are typically much shorter than route change intervals - we never experienced proposed problems with sustained route flapping - route changes are readily detected and handled as (rare) dead connections. The nicest bit is that transferred live connections can get rejected and closed immediately by the new server allowing for rapid recovery (unlike DNS changes which leaves things hanging for a long time). We had extremely fast datacenter failover using anycast (100% of traffic moved as soon as 5 seconds). Contrasted to the minutes-to-hours nightmare delays that characterize DNS failover. (The DNS time-to-live is an advisory at best, and outright ignored by many clients) It's too bad more hosting providers don't allow route announcements.
- sspies 11y ago
  I can fully agree with your experience using DNS failover vs anycast failover. Made a video of how I implemented anycast https://www.youtube.com/watch?v=tsXpQHi7Udo https://www.youtube.com/watch?v=tsXpQHi7Udo
  - _ao789 11y ago
    Interesting video, thanks for sharing
- federico3 11y ago
  > individual sessions are typically much shorter than route change intervals True for web traffic, not true for large file transfers, many multiplayer games, VPNs. > live connections can get rejected and closed immediately by the new server allowing for rapid recovery ...if the client protocol allows recovery.
- dsl 11y ago
  Same here. The only people dismissing TCP anycast at this point are folks who haven't been running anycast deployments. It also helps to respond to DNS with multiple VIPs within your anycast /24. Browser retry logic is pretty nifty and solves most of the real world problems.
CodingGuy 11y ago
Most important - how much has this work improved linkedin's site performance? 30%? 40%? 50%?
- BillinghamJ 11y ago
  And perhaps they should also compare that improvement with the improvement garnered by turning off javascript on LinkedIn (assuming it even works).
  - _ao789 11y ago
    And all so that you can read about people talking about themselves and how amazing they think they are. Oh, what game changers the linkedin users are..
    - stephengillie 11y ago
      It's just another echo chamber. Don't hate the players, hate the game.
transitorykris 11y ago
Sometimes you don't have as much control over a client (many non-HTTP TCP applications). A middle ground is to use anycast for authoritative DNS and return a set of records for unicast IPs local to that POP. You get the benefits of anycast and the stickiness of unicast. It's a short jump to overriding decisions to correct certain clients, or to shed traffic from servers or POPs.
- jzwinck 11y ago
  Doesn't this leave you with the problem mentioned in the article where some users have DNS hosts rather far away from themselves?
davidu 11y ago
Having done TCP Anycast for more than 8 years in production now, with extensive testing, and third-party monitoring, there has never been an issue with it. When a route flap happens that causes an AS path selection change, the user almost universally routes back to the exact same node they would have routed to prior to the flap. The theoretical "IT person in texas doing per packet load balancing on a Sprint T1 to Atlanta and an ATT T1 to Denver" doesn't actually exist.
mcguire 11y ago
"Anycast has historically not proven to be useful for stateful protocols in the internet because of the inherent instability of the internet." Inherent instability meaning packet-switched networking, a.k.a. the most basic mechanism of how the internet works.
t0mas88 11y ago
This is exactly how EdgeCast CDN (now Verizon) has worked for many years, down to the ip per region approach. So this is known for a long time to be a good way to do global content delivery. I'm curious why linkedin didn't just use a commercially available CDN service for this? Could save them a lot of time and maintenance, while probably getting a better result because CDNs have this down to the details.

81 points by r4um 11y ago

Would be interested to read how CloudFlare solve flapping problem with their Anycast network: https://blog.cloudflare.com/a-brief-anycast-primer/ https://blog.cloudflare.com/a-brief-anycast-primer/

samwillis 11y ago
I may be wrong but the impression I get is that CloudFlare have many many more POP much closer to the visitor, quite often in the major consumer ISPs own data centres. This should reduce the probability of what LinkedIn saw happening.

Animats 11y ago

What is LinkedIn sending that generates so much traffic that they need this? They're not Netflix. (Never mind, I just looked at LinkedIn, which I don't use much. With their new layout, I had to scroll through three screens of ads to get to anything that affected me.)

creshal 11y ago
But the ads are on (untrusted, ill-maintained) third-party CDNs anyway…
forty 11y ago
This is to improve latency, not throughput.
deleted 11y ago
[deleted]
_ao789 11y ago
It's a pity this study was done on LinkedIn and not something more bandwidth hungry, say a video site maybe?
k33n 11y ago
Looks like they're trying to shave MS off of user requests any way they can. Was probably a pretty fun project. Beats sitting and staring at an empty JIRA task list for 8 hours.
ape4 11y ago
This article :)
riteshm 11y ago
This was for improving performance of our site, not throughput.

pyvpx 11y ago

it has been a reality for many folks for years. real-world anycast can be tricky, but it is certainly deployable and usable.

SixSigma 11y ago

Plan9 can turn any protocol into TCP via 9p. Connect to a remote plan9 machine by whatever means using 9p, import /net from the remote machine into your local namespace, et viola you now have all the network connections of that remote machine available, including the tcp stack.

detaro 11y ago
What does this have to do with the article?
- SixSigma 11y ago
  Any protocol over any protocol. Never a pipe dream.
  - detaro 11y ago
    Anycast is not a specific protocol, but about how (normally IP-)packets are routed. Transporting protocols in other protocols has nothing to do with it and doesn't help.

sspies 11y ago

We are working on this as well at http://datapath.io http://datapath.io. You can use multiple iaas providers using global elastic IP addresses that you can announce from multiple locations at the same time. If you want to participate in our beta please write to beta@datapath.io Best, Sebastian

paulsutter 11y ago

We had the same experience at Quantcast. Better results with anycast within continents, and DNS to distribute a per-continent anycast addresses. Some know-it-all busybodies like to proclaim that anycast can't work for TCP because the route can change during the session, but the reality is that: - route changes are infrequent - individual sessions are typically much shorter than route change intervals - we never experienced proposed problems with sustained route flapping - route changes are readily detected and handled as (rare) dead connections. The nicest bit is that transferred live connections can get rejected and closed immediately by the new server allowing for rapid recovery (unlike DNS changes which leaves things hanging for a long time). We had extremely fast datacenter failover using anycast (100% of traffic moved as soon as 5 seconds). Contrasted to the minutes-to-hours nightmare delays that characterize DNS failover. (The DNS time-to-live is an advisory at best, and outright ignored by many clients) It's too bad more hosting providers don't allow route announcements.

sspies 11y ago
I can fully agree with your experience using DNS failover vs anycast failover. Made a video of how I implemented anycast https://www.youtube.com/watch?v=tsXpQHi7Udo https://www.youtube.com/watch?v=tsXpQHi7Udo
- _ao789 11y ago
  Interesting video, thanks for sharing
federico3 11y ago
> individual sessions are typically much shorter than route change intervals True for web traffic, not true for large file transfers, many multiplayer games, VPNs. > live connections can get rejected and closed immediately by the new server allowing for rapid recovery ...if the client protocol allows recovery.
dsl 11y ago
Same here. The only people dismissing TCP anycast at this point are folks who haven't been running anycast deployments. It also helps to respond to DNS with multiple VIPs within your anycast /24. Browser retry logic is pretty nifty and solves most of the real world problems.

CodingGuy 11y ago

Most important - how much has this work improved linkedin's site performance? 30%? 40%? 50%?

BillinghamJ 11y ago
And perhaps they should also compare that improvement with the improvement garnered by turning off javascript on LinkedIn (assuming it even works).
- _ao789 11y ago
  And all so that you can read about people talking about themselves and how amazing they think they are. Oh, what game changers the linkedin users are..
  - stephengillie 11y ago
    It's just another echo chamber. Don't hate the players, hate the game.

transitorykris 11y ago

Sometimes you don't have as much control over a client (many non-HTTP TCP applications). A middle ground is to use anycast for authoritative DNS and return a set of records for unicast IPs local to that POP. You get the benefits of anycast and the stickiness of unicast. It's a short jump to overriding decisions to correct certain clients, or to shed traffic from servers or POPs.

jzwinck 11y ago
Doesn't this leave you with the problem mentioned in the article where some users have DNS hosts rather far away from themselves?

davidu 11y ago

Having done TCP Anycast for more than 8 years in production now, with extensive testing, and third-party monitoring, there has never been an issue with it. When a route flap happens that causes an AS path selection change, the user almost universally routes back to the exact same node they would have routed to prior to the flap. The theoretical "IT person in texas doing per packet load balancing on a Sprint T1 to Atlanta and an ATT T1 to Denver" doesn't actually exist.

mcguire 11y ago

"Anycast has historically not proven to be useful for stateful protocols in the internet because of the inherent instability of the internet." Inherent instability meaning packet-switched networking, a.k.a. the most basic mechanism of how the internet works.

t0mas88 11y ago

This is exactly how EdgeCast CDN (now Verizon) has worked for many years, down to the ip per region approach. So this is known for a long time to be a good way to do global content delivery. I'm curious why linkedin didn't just use a commercially available CDN service for this? Could save them a lot of time and maintenance, while probably getting a better result because CDNs have this down to the details.