5 ms·
Ha, I guess this is one way of summarizing it :) Author here. I wanted to share more subtleties of the design, but maybe I failed. Indeed, the starting point i
by majke 4y ago
Ha, I guess this is one way of summarizing it :) Author here. I wanted to share more subtleties of the design, but maybe I failed.
Indeed, the starting point is sharing IP's across servers with port-ranges.
But there is more:
* awesome performance allowed by anycast.
* ability to route /32 instead of /24 per datacenter.
Generally, with this tech we can have much better IP usage density, without sacrificing reliability or performance. You can call it "global anycast-based stateless NAT" but that often implies some magic router configuration, which we don't have.
Here's one example of problems we run into - the lack of connectx() syscall on Linux - makes it hard to actually select port range to originate connections from:
https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem...
- chatmasta 4y agoI was surprised IPv6 was only briefly mentioned! Is that something you're looking at next, or are you already running an IPv6 egress network? Of course not every destination is an IPv6 host, so IPv4 remains necessary, but at least IPv6 can avoid the need for port slicing, since you can encode the same bucketing information in the IP address itself. I've seen this idea used as a cool trick [0] to implement a SOCKS proxy that randomizes outbound IPv6 address to be within a publicly routed prefix for the host (commonly a /64). I guess as long as you need to support IPv4, then port slicing is a requirement and IPv6 won't confer much benefit. (Maybe it could help alleviate port exhaustion if IPv6 addresses can use dynamic ports from any slice?) Either way, thanks for the blog post, I enjoyed it! [0] https://github.com/blacklanternsecurity/TREVORproxy https://github.com/blacklanternsecurity/TREVORproxy
- miyuru 4y agoI was also interested to know how this was handled for IPv6, but it was only briefly mentioned. Probably they didn't need to do much work with IPv6, since half of the post is solving IPv4 exhaustion problems.
- chriscappuccio 4y agoCloudflare wants to make money. The IPv6 features can come second as v6 usage increases.
- dknecht 4y agoAll of Cloudflare services ship with IPv6 day 1. IPv6 not an issue as we have enough IPv6 for each machine to have own IPs.
- pencilcode 4y agoIs the geofencing country level only? So if, using warp, I use trip advisor and go and see nearby restaurants it will have no idea of what city I’m in? Guessing that’s not so but wondering how it works
- aeyes 4y agoThis blog post has some info: https://blog.cloudflare.com/geoexit-improving-warp-user-experience-larger-network/ https://blog.cloudflare.com/geoexit-improving-warp-user-expe... Warp uses its own set of egress IPs and their geolocation is close to your real location.
- remram 4y agoFrom your article it seemed that your use of anycast was more accident than feature, due to the limit of BGP prefix sizes. If you could route those IPs to their correct destination, you would, you only go to the closest data center and route again because you have no choice. Maybe this ends up reducing cost on customers though, because the international transit happens in your backbone network rather than on the internet (customer-side).