14 ms·

Cloudflare servers don't own IPs anymore so how do they connect to the internet?

505 points by jgrahamc 4y ago

deleted 4y ago
[deleted]
remram 4y ago
TLDR: > To avoid geofencing issues, we need to choose specific egress addresses tagged with an appropriate country, depending on WARP user location. (...) Instead of having one or two egress IP addresses for each server, now we require dozens, and IPv4 addresses aren't cheap. > Instead of assigning one /32 IPv4 address for each server, we devised a method of assigning a /32 IP per data center, and then sharing it among physical servers (...) splitting an egress IP across servers by a port range.
- elp 4y ago
  In english they now do carrier grade nat.
  - cm2187 4y ago
    well, vanilla NAT really.
    - deleted 4y ago
      [deleted]
  - dilyevsky 4y ago
    right? "We don't like distributed NAT" couple paragraphs later - "hey check out this awesome dist NAT implementation".
- majke 4y ago
  Ha, I guess this is one way of summarizing it :) Author here. I wanted to share more subtleties of the design, but maybe I failed. Indeed, the starting point is sharing IP's across servers with port-ranges. But there is more: * awesome performance allowed by anycast. * ability to route /32 instead of /24 per datacenter. Generally, with this tech we can have much better IP usage density, without sacrificing reliability or performance. You can call it "global anycast-based stateless NAT" but that often implies some magic router configuration, which we don't have. Here's one example of problems we run into - the lack of connectx() syscall on Linux - makes it hard to actually select port range to originate connections from: https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem...
  - chatmasta 4y ago
    I was surprised IPv6 was only briefly mentioned! Is that something you're looking at next, or are you already running an IPv6 egress network? Of course not every destination is an IPv6 host, so IPv4 remains necessary, but at least IPv6 can avoid the need for port slicing, since you can encode the same bucketing information in the IP address itself. I've seen this idea used as a cool trick [0] to implement a SOCKS proxy that randomizes outbound IPv6 address to be within a publicly routed prefix for the host (commonly a /64). I guess as long as you need to support IPv4, then port slicing is a requirement and IPv6 won't confer much benefit. (Maybe it could help alleviate port exhaustion if IPv6 addresses can use dynamic ports from any slice?) Either way, thanks for the blog post, I enjoyed it! [0] https://github.com/blacklanternsecurity/TREVORproxy https://github.com/blacklanternsecurity/TREVORproxy
    - miyuru 4y ago
      I was also interested to know how this was handled for IPv6, but it was only briefly mentioned. Probably they didn't need to do much work with IPv6, since half of the post is solving IPv4 exhaustion problems.
      - chriscappuccio 4y ago
        Cloudflare wants to make money. The IPv6 features can come second as v6 usage increases.
        dknecht 4y ago
        All of Cloudflare services ship with IPv6 day 1. IPv6 not an issue as we have enough IPv6 for each machine to have own IPs.
  - pencilcode 4y ago
    Is the geofencing country level only? So if, using warp, I use trip advisor and go and see nearby restaurants it will have no idea of what city I’m in? Guessing that’s not so but wondering how it works
superkuh 4y ago
Yikes. More cloudflare breakage of the internet model. Pretty soon we might as well all just live within cloudflare's WAN entirely.
- majke 4y ago
  Author here, I know this is a dismissive comment, but I'll bite anyway. As far as I understand the history of the IP protocol, initially an IP address pointed to a host. (/etc/hosts file seems that way) Then it was realized a single entity might have multiple network interfaces, and an IP started to point to a network card on a host. (a host can have many IP's). Then all the VRF, dummy devices, tuntaps, VETH and containers. I guess an IP is now pointing to a container or VM. But there is more. For performance you can (almost should!) have an unique IP address per NUMA node. Or even logical CPU. In modern internet a server IP: points to a single CPU on a container in a VM on a host. Then consider Anycast, like 1.1.1.1 or 8.8.8.8. An IP means something else... it means a resource. On the "client" side we have customer NAT's. CG NAT's and VPN's. An IP means similarly little. The IP's are really expensive, so in some cases there is a strong advantage to save them. Take a look at https://blog.cloudflare.com/addressing-agility/ https://blog.cloudflare.com/addressing-agility/ "So, test we did. From a /20 address set, to a /24 and then, from June 2021, to an address set of one /32, and equivalently a /128 (Ao1). It doesn’t just work. It really works" We're able to serve "all cloudflare" from /32. There is this whole trend of getting denser and denser IP usage. It's not avoidable. It's not "breaking the Internet" in any way more than "NAT's are breaking the Internet". The network evolves, because it has to. And for one, I don't think this is inherently bad.
  - superkuh 4y ago
    >It's not avoidable. It's not "breaking the Internet" in any way more than "NAT's are breaking the Internet". I agree. NATs, particularly the Carrier NAT that smartphone users are behind, has broken the internet. It's made it so most people do not have ports and cannot participate in the internet. So now software developers cannot write software that uses the internet (without depending on third parties). This is bad. So is what you've done. Someday ipv6 will save us.
subarctic 4y ago
Pretty interesting article. TLDR: they're now using anycast for egress, not just ingress. Each data center has a single IP for each country code (so that they can make outgoing requests that are geolocated in any country). In order to achieve that, they have a /24 or larger range for each country, and announce it from all their data centers, and then they route the traffic over their backbone to the appropriate data center for that IP. Then in the data center, they share the single IP across all their servers by giving each server a range of TCP/UDP port space (instead of doing stateful NAT).
- ec109685 4y ago
  It’s not a single IP address per data center. Otherwise they’d only be able to make 64k simultaneous egress connections, nor would their scheme of different ip addresses per “geo” and product work. Edit: based on this part of blog post: “With a port slice of say 2,048 ports, we can share one IP among 31 servers. However, there is always a possibility of running out of ports. To address this, we've worked hard to be able to reuse the egress ports efficiently.”
  - grogers 4y ago
    I guess for large data centers they could use multiple IPs and the overall scheme stays essentially the same, but the article seems to strongly imply that it is one single egress IP per DC. The 64k simultaneous connections limitation is only per origin server IP.
  - icehawk 4y ago
    Do you mean 64k simultaneous egress connections per origin server IP? That's still a reasonable number of connections.
    - ec109685 4y ago
      I might not be understanding, but it needs to know how to route the packets back to the server and it does that with ports.
  - 4y ago
xg15 4y ago
> However, while anycast works well in the ingress direction, it can't operate on egress. Establishing an outgoing connection from an anycast IP won't work. Consider the response packet. It's likely to be routed back to a wrong place - a data center geographically closest to the sender, not necessarily the source data center! Slightly OT question, but why wouldn't this be a problem with ingress, too? E.g. suppose I want to send a request to https://1.2.3.4 https://1.2.3.4. What I don't know is that 1.2.3.4 is an anycast address. So my client sends a SYN packet to 1.2.3.4:443 to open the connection. The packet is routed to data center #1. The data center duly replies with a SYN/ACK packet, which my client answers with an ACK packet. However, due to some bad luck, the ACK packet is routed to data center #2 which is also a destination for the anycast address. Of course, data center #2 doesn't know anything about my connection, so it just drops the ACK or replies with a RST. In the best case, I can eventually resend my ACK and reach the right data center (with multi-second delay), in the worst case, the connection setup will fail. Why does this not happen on ingress, but is a problem for egress? Even if the handshake uses SYN cookies and got through on data center #2, what would keep subsequent packets that I send on that connection from being routed to random data centers that don't know anything about the connection?
- matsur 4y ago
  This is a problem in theory. In practice (and through experience) we see very little routing instability in the way you describe.
  - xg15 4y ago
    You mean, it's just luck?
    - rizky05 4y ago
      calculated luck
    - Brian_K_White 4y ago
      right? also seems like load should or at least could be changing all the time. geo or hops proximity is really the only things that decide a route? not load also? But although I would be surprised if load were not also part of the route picker, I would also be surprised if the routers didn't have some association or state tracking to actively ensure related packets get the same route. But I guess this is saying exactly that, that it's relying on luck and happenstance. It may be doing the job well enough that not enough people complain, but I wouldn't be proud of it myself.
Terretta 4y ago
I quite like what CloudFlare has done here. There's a fourth way to resolve this, that works for the core use case, is less engineering, and was in production 20 years ago, but I can't fit it within the margins of this comment box. // CF's approach has additional feature advantages though.
danrl 4y ago
As an industry, we are bad at deprecating old protocols like IPv4. This is a genius hack for a problem we have due to IPv6 not being adopted widely enough so that serving legacy IP users becomes a dropable liability to the business. The ROI is still high enough for us to “innovate” here. I applaud the solution but mourn the fact that we still need this. I guess ingress is next, then? Two layers of Unimog to achieve stability before TCP/TLS termination maybe.
- dopylitty 4y ago
  I've been thinking a lot about this in my own enterprise and I've increasingly come to the conclusion that IP itself is the wrong abstraction for how the majority of modern networked compute works. IPv6, as a (quite old itself) iteration on top of IPv4 with a bunch of byzantine processes and acronyms tacked on is solving the wrong problem. Originally IP was a way to allow discrete physical computers in different locations owned by different organizations to find each other and exchange information autonomously. These days most compute actually doesn't look like that. All my compute is in AWS. Rather than being autonomous it is controlled by a single global control plane and uniquely identified within that control plane. So when I want my services to connect to each-other within AWS why am I still dealing with these complex routing algorithms and obtuse numbering schemes? AWS knows exactly which physical hosts my processes are running on and could at a control plane level connect them directly. And I, as someone running a business, could focus on the higher level problem of 'service X is allowed to connect to service Y' rather than figuring out how to send IP packets across subnets/TGWs and where to configure which ports in NACLs and security groups to allow the connection. Similarly my ISP knows exactly where Amazon and CloudFlare's nearest front doors are so instead of 15 hops and DNS resolutions my laptop could just make a request to Service X on AWS. My ISP could drop the message in AWS' nearest front door and AWS could figure out how to drop the message on the right host however they want to. I know there's a lot of legacy cruft and also that there are benefits of the autonomous/decentralized model vs central control for the internet as a whole but given the centralized reality we're in, especially within the enterprise, I think it's worth reevaluating how we approach networking and whether the continuing focus on IP is the best use of of our time.
  - 4y ago
jesuspiece 4y ago
uvdn7 4y ago
This is a wonderful article. Thanks for sharing. As always, Cloudflare blog posts do not disappoint. It’s very interesting that they are essentially treating IP addresses as “data”. Once looking at the problem from a distributed system lens, the solution here can be mapped to distributed systems almost perfectly. - Replicating a piece of data on every host in the fleet is expensive, but fast and reliable. The compromise is usually to keep one replica in a region; same as how they share a single /32 IP address in a region. - “sending datagram to IP X” is no different than “fetching data X from a distributed system”. This is essentially the underlying philosophy of the soft-unicast. Just like data lives in a distributed system/cloud, you no longer know where is an IP address located. It’s ingenious. They said they don’t like stateful NAT, which is understandable. But the load balancer has to be stateful still to perform the routing correctly. It would be an interesting follow up blog post talking about how they coordinate port/data movements (moving a port from server A to server B), as it’s state management (not very different from moving data in a distributed system again).
- remram 4y ago
  I have a lot of trouble mapping your comment to the content of the article. It is about the egress addresses, the ones CloudFlare use as source when fetching from origin servers. Those addresses need to be separated by the region of the end-user ("eyeball"/browser) and the CloudFlare service they are using (CDN or WARP). The cost they are working around is the cost of IPv4 addresses, versus the combinatorial explosion in their allocation scheme (they need number of services * number of regions * whatever dimension they add next, because IP addresses are nothing like data). I am not sure where you see data replication in this scheme?
  - uvdn7 4y ago
    It's not meant to be a perfect analogy. The replication analogy is mostly talking about the tradeoff between performance and cost. So it's less about "replicating" the ip addresses (which is not happening). On that front, maybe distribution would be a better term. Instead of storing a single piece of data on a single host (unicast), they are distributing it to a set of hosts. Overall, it seems like they are treating ip addresses as data essentially, which becomes most obvious when they talk about soft-unicast. Anyway, I just found it interesting to look at this through this lens.
dfawcus 4y ago
What they describe sounds a lot like a distributed static RSIP scheme. https://en.wikipedia.org/wiki/Realm-Specific_IP https://en.wikipedia.org/wiki/Realm-Specific_IP With port ranges rather than being 'leased', being allocated on the the basis of per server within a locale. So the IP goes to the locale, the port range is the the static RSIP to the server within that locale.
- majke 4y ago
  There is more: * pcp https://datatracker.ietf.org/doc/html/rfc6887 https://datatracker.ietf.org/doc/html/rfc6887 * ds-lite port allocation mode https://support.huawei.com/enterprise/en/doc/EDOC1100125799/2ef058/configuring-basic-ds-lite-functions#EN-US_TASK_0172374694 https://support.huawei.com/enterprise/en/doc/EDOC1100125799/...
martinohansen 4y ago
Am I missing something here or did they just reinvent a NAT gateway with static rules? I understand that they started using anycast for the egress IPs as well, but thats unrelated to the NAT problem.
- deleted 4y ago
  [deleted]
ronnier 4y ago
Spammers are exploiting cloudflare by creating thousands of new domains on the free tld (like .ml) and hosting the sites behind cloudflare and spamming social media apps with links to scam dating sites. CPA scammers. If anyone from CF sees this, I can work with you and give you data on this. I’m dealing with this at one of the large social media companies. Here's an example, this is NSFW - https://atragcara.ga https://atragcara.ga
- sschueller 4y ago
  Same goes for DDoS attacks. I am not sure how they do it but we get hit by CF IPs with synfloods etc.
  - slothsarecool 4y ago
    You do not get attacked from Cloudflare with TCP attacks. Somebody is spoofing the IP header and make it seem like Cloudflare is DDoSing you. The only way for somebody to DDoS from Cloudflare would be using workers, however, this isn't practical as workers have a very limited IP Range.
    - fncivivue7 4y ago
      Why are you assuming amplification attacks aren't a thing? I think you're probably right about the spoofing but it comes off a little dismissive when the possibility of a site that queries other sites, could be tricked into doing something it shouldn't, is always going to be in the realm of a possibility.
    - cmeacham98 4y ago
      The reason people do this, by the way, is because it's common if you're hosting via CF to whitelist their IPs and block the rest. This allows their SYN flood to bypass that.
    - patricklorio 4y ago
      I run a fairly popular service and have received DDoS attacks from Cloudflare's IP range (~20gbps). I can confirm they respond to SYN+ACK with an ACK to complete the TCP handshake. Through some investigating it seems like a botnet using Cloudflare WARP (their VPN service).
Ptchd 4y ago
If you don't need an IP to be connected to the internet, sign me up... I think they are full of it though... Even if you only have one IP.... you still have an IP > PING cloudflare.com (104.16.133.229) 56(84) bytes of data. > 64 bytes from 104.16.133.229 (104.16.133.229): icmp_seq=1 ttl=52 time=10.6 ms With a ping like this, you know that I am not using Musk's Internet....
immibis 4y ago
This is a horrible way to avoid upgrading the world to IPv6.
- Animats 4y ago
  I'm surprised that Cloudflare isn't all IPv6 when Cloudflare is the client. That would solve their address problems. Maybe charge more if your servers can't talk IPv6. Or require it for the free tier. It's useful that they use client side certificates. (They call this "authenticated origin pull", but it seems to be client side certs.
  - ec109685 4y ago
    They also have to egress to third party servers since they are a CDN and support things like serverless functions
  - amluto 4y ago
    Sadly Cloudflare seems to treat client certificates as an optional nifty feature as opposed to the critical feature that it is. And even some of the settings that look secure aren’t: https://medium.com/@ss23/leveraging-cloudflares-authenticated-origin-pulls-for-pentesting-565c562ef1bb https://medium.com/@ss23/leveraging-cloudflares-authenticate... Authenticated origin pulls should not be “useful”. They should be on and configured securely by default, and any insecure setting should get a loud warning.
    - xPaw 4y ago
      I have tried using cloudflare's client certificate, but it doesn't play well with nginx ssl cache due to increased size on a relatively high load website.
  - majke 4y ago
    Hi there! In the article I only briefly mentioned IPv6, it was long enough. Indeed, we do happy-eyeballs where we can and _strongly_ prefer IPv6 when origin host resolves AAAA. However, we still need IPv4, since there is still a big chunk of traffic that only works IPv4. If the client gives us a chance, we'll strongly prefer IPv6.
Thorentis 4y ago
The fact that we haven't yet adopted IPv6 tells me that IPv6 isn't actually that great of a solution. We need an Internet Protocol that solves modern problems and that has a good migration path.
- zekica 4y ago
  Where do they say that they haven't adopted IPv6? All their offerings support IPv6.
- wpietri 4y ago
  40% of Google's traffic comes via IPv6. Up from 1% a decade ago. https://www.google.com/intl/en/ipv6/statistics.html https://www.google.com/intl/en/ipv6/statistics.html If you think you can do better than that, I look forward to hearing your plan. Personally, I think that's huge progress.
- growse 4y ago
  You need an IPv4 src address to connect out to an IPv4 origin.
- eastdakota 4y ago
  Fun fact: the first product we announced to celebrate Cloudflare’s launch day anniversary was a IPv4<->IPv6 gateway: https://blog.cloudflare.com/introducing-cloudflares-automatic-ipv6-gatewa/ https://blog.cloudflare.com/introducing-cloudflares-automati... The success of that convinced us we should do something to improve the Internet every year to celebrate our “birthday.” Over time we ended up with more than one product that met that criteria and timing, so it went from a day of celebration to a week. That became our Birthday Week. Then we saw how well bundling a set of announcements into a week was so we decided to do it other times of the year. And that’s how Cloudflare Innovation Weeks got started, explicitly with us delivering IPv6 support back in 2011.
inopinatus 4y ago
TLDR: Cloudflare is using five bits from the port number as a subnetting & routing scheme, with optional content policy semantics, for hosts behind anycast addressing and inside their network boundary.
- huggingmouth 4y ago
  By far the best summary I've come across so far. Thank you!
mike256 4y ago
Wouldn't it be better when all those big CDNs just switch off IPv4 and force the sleeping ISPs to enable IPv6? Maybe we should introduce some IPv6 only days as a first step...
cesarb 4y ago
All this wonderful complexity, just because a few servers insist on behaving as if the location of the IP address and the location of the user should always match.
Rasbora 4y ago
Whenever I see the name Marek Majkowski come up, I know the blog post is going to be good. I had to solve this exact problem a year ago when attempting to build an anycast forward proxy, quickly came to the conclusion that it'd be impossible without a massive infrastructure presence. Ironically I was using CF connections to debug how they might go about this problem, when I realized they were just using local unicast routes for egress traffic I stopped digging any deeper. Maintaining a routing table in unimog to forward lopsided egress connections to the correct DC is brilliant and shows what is possible when you have a global network to play with, however I wonder if this opens up an attack vector where previously distributed connections are now being forwarded & centralized at a single DC, especially if they are all destined for the same port slice...
- jxhdkdhdks 4y ago
  you wonder about an attack and everyone else wonder if that was the main point of all this.
- deftoast 4y ago
  I think this is no worse than using unicast IPs - they can probably reuse the same firewall technology to guard against attacks as before.
pm2222 4y ago
Packet headers contain essentially info that labels a packet. In v4 world they are running out of bits so they are maneuvering in my view that’s cool yet at the same time boring to me. Perhaps they should go v6 internally and implement one v4-v6 gateway to put all these tricks behind.
soumendrak 4y ago
Cloudflare and Akamai: competition
mcint 4y ago
Given the discussion under https://news.ycombinator.com/item?id=33743935 https://news.ycombinator.com/item?id=33743935 about opening a connection to 1.2.3.4:443 (I'm prompted by the same curiosity about ingress load-balancing, statelessly)... How does the ingress "router" load-balance incoming connections, which it must (even if the "router" is a default host or cluster)? CF isn't opening TCP, then HTTP just to send redirects to another IP for the same cluster. I guess hashing, on IP and port, is already readily used in routing decisions, so eyeball-addr and -port of the inbound packets, 4-tuple (CF-addr [fixed, shared], CF-port [443], eyeball-addr, eyeball-port), provides a consistent. I guess that this is good for 99.9 percent of connections, which are short-lived, and opportunistically kept open or reused. I suppose other long-lived connections might take place in the context of an application that tracks data above and outside of TCP-alone. I'm grasping for a missing middle, in size of use case, and can't quickly name things that people might proxy but need stable connections. CloudFlare's reverse proxying to web servers would count, if the web-fronting had to traverse someone else's proxy layer. What are the rough edges here? What's are next challenges here to build around?
BlueTemplar 4y ago
Related : Deploying IPv6-mostly access networks : https://news.ycombinator.com/item?id=33694293 https://news.ycombinator.com/item?id=33694293 (I can't for the life of me find the comment chain where this link was posted ~4 days ago ??) RFC 8925 – IPv6-Only Preferred Option for DHCPv4 (2020) : https://news.ycombinator.com/item?id=33697978 https://news.ycombinator.com/item?id=33697978 (posted on HN by me)
benlivengood 4y ago
It's strange/sad how 50% of their problem is caused by geofencing which is caused by archaic laws. 40% is caused by archaic protocols that don't allow enough addresses or local efficient P2P or ISP caching of public content (sort of what IPFS aimed to do), which would alleviate much of the need for CDNs in the first place. The remaining 10% is just solving hard problems. I'm a little surprised that splitting by port number gives servers enough connections; maybe they are connection-pooling some of them between their egress and final destination. If there was truly a 1:1 mapping of all user requests to TCP connections then I'd expect there to be ~thousands of simultaneous connections to, say, Walmart today (black Friday) which is also probably on an anycast address, limiting the number of unique src:dst ip and port tuples to 65K per cloudflare egress IP address. Maybe that ends up being not so bad and DCs can scale by adding new IPs? https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem... covers a lot of the details in solving that and similar problems.
zeckalpha 4y ago
Concerningly, they discuss routing traffic between jurisdictions when it is meant to be local. This is something state actors use to get around domestic spying rules. Neat technical solution though!
tobsterr 4y ago
As a networking noob, can someone explain to me how this solution plays nicely with the geofencing constraint described in the post? How does the IP/port combination map to a CIDR block defined in the geofeed?

505 points by jgrahamc 4y ago

deleted 4y ago

[deleted]

TLDR: > To avoid geofencing issues, we need to choose specific egress addresses tagged with an appropriate country, depending on WARP user location. (...) Instead of having one or two egress IP addresses for each server, now we require dozens, and IPv4 addresses aren't cheap. > Instead of assigning one /32 IPv4 address for each server, we devised a method of assigning a /32 IP per data center, and then sharing it among physical servers (...) splitting an egress IP across servers by a port range.

elp 4y ago
In english they now do carrier grade nat.
- cm2187 4y ago
  well, vanilla NAT really.
  - deleted 4y ago
    [deleted]
- dilyevsky 4y ago
  right? "We don't like distributed NAT" couple paragraphs later - "hey check out this awesome dist NAT implementation".
majke 4y ago
Ha, I guess this is one way of summarizing it :) Author here. I wanted to share more subtleties of the design, but maybe I failed. Indeed, the starting point is sharing IP's across servers with port-ranges. But there is more: * awesome performance allowed by anycast. * ability to route /32 instead of /24 per datacenter. Generally, with this tech we can have much better IP usage density, without sacrificing reliability or performance. You can call it "global anycast-based stateless NAT" but that often implies some magic router configuration, which we don't have. Here's one example of problems we run into - the lack of connectx() syscall on Linux - makes it hard to actually select port range to originate connections from: https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem...
- chatmasta 4y ago
  I was surprised IPv6 was only briefly mentioned! Is that something you're looking at next, or are you already running an IPv6 egress network? Of course not every destination is an IPv6 host, so IPv4 remains necessary, but at least IPv6 can avoid the need for port slicing, since you can encode the same bucketing information in the IP address itself. I've seen this idea used as a cool trick [0] to implement a SOCKS proxy that randomizes outbound IPv6 address to be within a publicly routed prefix for the host (commonly a /64). I guess as long as you need to support IPv4, then port slicing is a requirement and IPv6 won't confer much benefit. (Maybe it could help alleviate port exhaustion if IPv6 addresses can use dynamic ports from any slice?) Either way, thanks for the blog post, I enjoyed it! [0] https://github.com/blacklanternsecurity/TREVORproxy https://github.com/blacklanternsecurity/TREVORproxy
  - miyuru 4y ago
    I was also interested to know how this was handled for IPv6, but it was only briefly mentioned. Probably they didn't need to do much work with IPv6, since half of the post is solving IPv4 exhaustion problems.
    - chriscappuccio 4y ago
      Cloudflare wants to make money. The IPv6 features can come second as v6 usage increases.
      - dknecht 4y ago
        All of Cloudflare services ship with IPv6 day 1. IPv6 not an issue as we have enough IPv6 for each machine to have own IPs.
- pencilcode 4y ago
  Is the geofencing country level only? So if, using warp, I use trip advisor and go and see nearby restaurants it will have no idea of what city I’m in? Guessing that’s not so but wondering how it works

superkuh 4y ago

Yikes. More cloudflare breakage of the internet model. Pretty soon we might as well all just live within cloudflare's WAN entirely.

majke 4y ago
Author here, I know this is a dismissive comment, but I'll bite anyway. As far as I understand the history of the IP protocol, initially an IP address pointed to a host. (/etc/hosts file seems that way) Then it was realized a single entity might have multiple network interfaces, and an IP started to point to a network card on a host. (a host can have many IP's). Then all the VRF, dummy devices, tuntaps, VETH and containers. I guess an IP is now pointing to a container or VM. But there is more. For performance you can (almost should!) have an unique IP address per NUMA node. Or even logical CPU. In modern internet a server IP: points to a single CPU on a container in a VM on a host. Then consider Anycast, like 1.1.1.1 or 8.8.8.8. An IP means something else... it means a resource. On the "client" side we have customer NAT's. CG NAT's and VPN's. An IP means similarly little. The IP's are really expensive, so in some cases there is a strong advantage to save them. Take a look at https://blog.cloudflare.com/addressing-agility/ https://blog.cloudflare.com/addressing-agility/ "So, test we did. From a /20 address set, to a /24 and then, from June 2021, to an address set of one /32, and equivalently a /128 (Ao1). It doesn’t just work. It really works" We're able to serve "all cloudflare" from /32. There is this whole trend of getting denser and denser IP usage. It's not avoidable. It's not "breaking the Internet" in any way more than "NAT's are breaking the Internet". The network evolves, because it has to. And for one, I don't think this is inherently bad.
- superkuh 4y ago
  >It's not avoidable. It's not "breaking the Internet" in any way more than "NAT's are breaking the Internet". I agree. NATs, particularly the Carrier NAT that smartphone users are behind, has broken the internet. It's made it so most people do not have ports and cannot participate in the internet. So now software developers cannot write software that uses the internet (without depending on third parties). This is bad. So is what you've done. Someday ipv6 will save us.

subarctic 4y ago

Pretty interesting article. TLDR: they're now using anycast for egress, not just ingress. Each data center has a single IP for each country code (so that they can make outgoing requests that are geolocated in any country). In order to achieve that, they have a /24 or larger range for each country, and announce it from all their data centers, and then they route the traffic over their backbone to the appropriate data center for that IP. Then in the data center, they share the single IP across all their servers by giving each server a range of TCP/UDP port space (instead of doing stateful NAT).

ec109685 4y ago
It’s not a single IP address per data center. Otherwise they’d only be able to make 64k simultaneous egress connections, nor would their scheme of different ip addresses per “geo” and product work. Edit: based on this part of blog post: “With a port slice of say 2,048 ports, we can share one IP among 31 servers. However, there is always a possibility of running out of ports. To address this, we've worked hard to be able to reuse the egress ports efficiently.”
- grogers 4y ago
  I guess for large data centers they could use multiple IPs and the overall scheme stays essentially the same, but the article seems to strongly imply that it is one single egress IP per DC. The 64k simultaneous connections limitation is only per origin server IP.
- icehawk 4y ago
  Do you mean 64k simultaneous egress connections per origin server IP? That's still a reasonable number of connections.
  - ec109685 4y ago
    I might not be understanding, but it needs to know how to route the packets back to the server and it does that with ports.
- 4y ago

xg15 4y ago

> However, while anycast works well in the ingress direction, it can't operate on egress. Establishing an outgoing connection from an anycast IP won't work. Consider the response packet. It's likely to be routed back to a wrong place - a data center geographically closest to the sender, not necessarily the source data center! Slightly OT question, but why wouldn't this be a problem with ingress, too? E.g. suppose I want to send a request to https://1.2.3.4 https://1.2.3.4. What I don't know is that 1.2.3.4 is an anycast address. So my client sends a SYN packet to 1.2.3.4:443 to open the connection. The packet is routed to data center #1. The data center duly replies with a SYN/ACK packet, which my client answers with an ACK packet. However, due to some bad luck, the ACK packet is routed to data center #2 which is also a destination for the anycast address. Of course, data center #2 doesn't know anything about my connection, so it just drops the ACK or replies with a RST. In the best case, I can eventually resend my ACK and reach the right data center (with multi-second delay), in the worst case, the connection setup will fail. Why does this not happen on ingress, but is a problem for egress? Even if the handshake uses SYN cookies and got through on data center #2, what would keep subsequent packets that I send on that connection from being routed to random data centers that don't know anything about the connection?

matsur 4y ago
This is a problem in theory. In practice (and through experience) we see very little routing instability in the way you describe.
- xg15 4y ago
  You mean, it's just luck?
  - rizky05 4y ago
    calculated luck
  - Brian_K_White 4y ago
    right? also seems like load should or at least could be changing all the time. geo or hops proximity is really the only things that decide a route? not load also? But although I would be surprised if load were not also part of the route picker, I would also be surprised if the routers didn't have some association or state tracking to actively ensure related packets get the same route. But I guess this is saying exactly that, that it's relying on luck and happenstance. It may be doing the job well enough that not enough people complain, but I wouldn't be proud of it myself.

Terretta 4y ago

I quite like what CloudFlare has done here. There's a fourth way to resolve this, that works for the core use case, is less engineering, and was in production 20 years ago, but I can't fit it within the margins of this comment box. // CF's approach has additional feature advantages though.

danrl 4y ago

As an industry, we are bad at deprecating old protocols like IPv4. This is a genius hack for a problem we have due to IPv6 not being adopted widely enough so that serving legacy IP users becomes a dropable liability to the business. The ROI is still high enough for us to “innovate” here. I applaud the solution but mourn the fact that we still need this. I guess ingress is next, then? Two layers of Unimog to achieve stability before TCP/TLS termination maybe.

dopylitty 4y ago
I've been thinking a lot about this in my own enterprise and I've increasingly come to the conclusion that IP itself is the wrong abstraction for how the majority of modern networked compute works. IPv6, as a (quite old itself) iteration on top of IPv4 with a bunch of byzantine processes and acronyms tacked on is solving the wrong problem. Originally IP was a way to allow discrete physical computers in different locations owned by different organizations to find each other and exchange information autonomously. These days most compute actually doesn't look like that. All my compute is in AWS. Rather than being autonomous it is controlled by a single global control plane and uniquely identified within that control plane. So when I want my services to connect to each-other within AWS why am I still dealing with these complex routing algorithms and obtuse numbering schemes? AWS knows exactly which physical hosts my processes are running on and could at a control plane level connect them directly. And I, as someone running a business, could focus on the higher level problem of 'service X is allowed to connect to service Y' rather than figuring out how to send IP packets across subnets/TGWs and where to configure which ports in NACLs and security groups to allow the connection. Similarly my ISP knows exactly where Amazon and CloudFlare's nearest front doors are so instead of 15 hops and DNS resolutions my laptop could just make a request to Service X on AWS. My ISP could drop the message in AWS' nearest front door and AWS could figure out how to drop the message on the right host however they want to. I know there's a lot of legacy cruft and also that there are benefits of the autonomous/decentralized model vs central control for the internet as a whole but given the centralized reality we're in, especially within the enterprise, I think it's worth reevaluating how we approach networking and whether the continuing focus on IP is the best use of of our time.
- 4y ago

jesuspiece 4y ago

uvdn7 4y ago

This is a wonderful article. Thanks for sharing. As always, Cloudflare blog posts do not disappoint. It’s very interesting that they are essentially treating IP addresses as “data”. Once looking at the problem from a distributed system lens, the solution here can be mapped to distributed systems almost perfectly. - Replicating a piece of data on every host in the fleet is expensive, but fast and reliable. The compromise is usually to keep one replica in a region; same as how they share a single /32 IP address in a region. - “sending datagram to IP X” is no different than “fetching data X from a distributed system”. This is essentially the underlying philosophy of the soft-unicast. Just like data lives in a distributed system/cloud, you no longer know where is an IP address located. It’s ingenious. They said they don’t like stateful NAT, which is understandable. But the load balancer has to be stateful still to perform the routing correctly. It would be an interesting follow up blog post talking about how they coordinate port/data movements (moving a port from server A to server B), as it’s state management (not very different from moving data in a distributed system again).

remram 4y ago
I have a lot of trouble mapping your comment to the content of the article. It is about the egress addresses, the ones CloudFlare use as source when fetching from origin servers. Those addresses need to be separated by the region of the end-user ("eyeball"/browser) and the CloudFlare service they are using (CDN or WARP). The cost they are working around is the cost of IPv4 addresses, versus the combinatorial explosion in their allocation scheme (they need number of services * number of regions * whatever dimension they add next, because IP addresses are nothing like data). I am not sure where you see data replication in this scheme?
- uvdn7 4y ago
  It's not meant to be a perfect analogy. The replication analogy is mostly talking about the tradeoff between performance and cost. So it's less about "replicating" the ip addresses (which is not happening). On that front, maybe distribution would be a better term. Instead of storing a single piece of data on a single host (unicast), they are distributing it to a set of hosts. Overall, it seems like they are treating ip addresses as data essentially, which becomes most obvious when they talk about soft-unicast. Anyway, I just found it interesting to look at this through this lens.

dfawcus 4y ago

What they describe sounds a lot like a distributed static RSIP scheme. https://en.wikipedia.org/wiki/Realm-Specific_IP https://en.wikipedia.org/wiki/Realm-Specific_IP With port ranges rather than being 'leased', being allocated on the the basis of per server within a locale. So the IP goes to the locale, the port range is the the static RSIP to the server within that locale.

majke 4y ago
There is more: * pcp https://datatracker.ietf.org/doc/html/rfc6887 https://datatracker.ietf.org/doc/html/rfc6887 * ds-lite port allocation mode https://support.huawei.com/enterprise/en/doc/EDOC1100125799/2ef058/configuring-basic-ds-lite-functions#EN-US_TASK_0172374694 https://support.huawei.com/enterprise/en/doc/EDOC1100125799/...

martinohansen 4y ago

Am I missing something here or did they just reinvent a NAT gateway with static rules? I understand that they started using anycast for the egress IPs as well, but thats unrelated to the NAT problem.

deleted 4y ago
[deleted]

ronnier 4y ago

Spammers are exploiting cloudflare by creating thousands of new domains on the free tld (like .ml) and hosting the sites behind cloudflare and spamming social media apps with links to scam dating sites. CPA scammers. If anyone from CF sees this, I can work with you and give you data on this. I’m dealing with this at one of the large social media companies. Here's an example, this is NSFW - https://atragcara.ga https://atragcara.ga

sschueller 4y ago
Same goes for DDoS attacks. I am not sure how they do it but we get hit by CF IPs with synfloods etc.
- slothsarecool 4y ago
  You do not get attacked from Cloudflare with TCP attacks. Somebody is spoofing the IP header and make it seem like Cloudflare is DDoSing you. The only way for somebody to DDoS from Cloudflare would be using workers, however, this isn't practical as workers have a very limited IP Range.
  - fncivivue7 4y ago
    Why are you assuming amplification attacks aren't a thing? I think you're probably right about the spoofing but it comes off a little dismissive when the possibility of a site that queries other sites, could be tricked into doing something it shouldn't, is always going to be in the realm of a possibility.
  - cmeacham98 4y ago
    The reason people do this, by the way, is because it's common if you're hosting via CF to whitelist their IPs and block the rest. This allows their SYN flood to bypass that.
  - patricklorio 4y ago
    I run a fairly popular service and have received DDoS attacks from Cloudflare's IP range (~20gbps). I can confirm they respond to SYN+ACK with an ACK to complete the TCP handshake. Through some investigating it seems like a botnet using Cloudflare WARP (their VPN service).

Ptchd 4y ago

If you don't need an IP to be connected to the internet, sign me up... I think they are full of it though... Even if you only have one IP.... you still have an IP > PING cloudflare.com (104.16.133.229) 56(84) bytes of data. > 64 bytes from 104.16.133.229 (104.16.133.229): icmp_seq=1 ttl=52 time=10.6 ms With a ping like this, you know that I am not using Musk's Internet....

immibis 4y ago

This is a horrible way to avoid upgrading the world to IPv6.

Animats 4y ago
I'm surprised that Cloudflare isn't all IPv6 when Cloudflare is the client. That would solve their address problems. Maybe charge more if your servers can't talk IPv6. Or require it for the free tier. It's useful that they use client side certificates. (They call this "authenticated origin pull", but it seems to be client side certs.
- ec109685 4y ago
  They also have to egress to third party servers since they are a CDN and support things like serverless functions
- amluto 4y ago
  Sadly Cloudflare seems to treat client certificates as an optional nifty feature as opposed to the critical feature that it is. And even some of the settings that look secure aren’t: https://medium.com/@ss23/leveraging-cloudflares-authenticated-origin-pulls-for-pentesting-565c562ef1bb https://medium.com/@ss23/leveraging-cloudflares-authenticate... Authenticated origin pulls should not be “useful”. They should be on and configured securely by default, and any insecure setting should get a loud warning.
  - xPaw 4y ago
    I have tried using cloudflare's client certificate, but it doesn't play well with nginx ssl cache due to increased size on a relatively high load website.
- majke 4y ago
  Hi there! In the article I only briefly mentioned IPv6, it was long enough. Indeed, we do happy-eyeballs where we can and _strongly_ prefer IPv6 when origin host resolves AAAA. However, we still need IPv4, since there is still a big chunk of traffic that only works IPv4. If the client gives us a chance, we'll strongly prefer IPv6.

Thorentis 4y ago

The fact that we haven't yet adopted IPv6 tells me that IPv6 isn't actually that great of a solution. We need an Internet Protocol that solves modern problems and that has a good migration path.

zekica 4y ago
Where do they say that they haven't adopted IPv6? All their offerings support IPv6.
wpietri 4y ago
40% of Google's traffic comes via IPv6. Up from 1% a decade ago. https://www.google.com/intl/en/ipv6/statistics.html https://www.google.com/intl/en/ipv6/statistics.html If you think you can do better than that, I look forward to hearing your plan. Personally, I think that's huge progress.
growse 4y ago
You need an IPv4 src address to connect out to an IPv4 origin.
eastdakota 4y ago
Fun fact: the first product we announced to celebrate Cloudflare’s launch day anniversary was a IPv4<->IPv6 gateway: https://blog.cloudflare.com/introducing-cloudflares-automatic-ipv6-gatewa/ https://blog.cloudflare.com/introducing-cloudflares-automati... The success of that convinced us we should do something to improve the Internet every year to celebrate our “birthday.” Over time we ended up with more than one product that met that criteria and timing, so it went from a day of celebration to a week. That became our Birthday Week. Then we saw how well bundling a set of announcements into a week was so we decided to do it other times of the year. And that’s how Cloudflare Innovation Weeks got started, explicitly with us delivering IPv6 support back in 2011.

inopinatus 4y ago

TLDR: Cloudflare is using five bits from the port number as a subnetting & routing scheme, with optional content policy semantics, for hosts behind anycast addressing and inside their network boundary.

huggingmouth 4y ago
By far the best summary I've come across so far. Thank you!

mike256 4y ago

Wouldn't it be better when all those big CDNs just switch off IPv4 and force the sleeping ISPs to enable IPv6? Maybe we should introduce some IPv6 only days as a first step...

cesarb 4y ago

All this wonderful complexity, just because a few servers insist on behaving as if the location of the IP address and the location of the user should always match.

Rasbora 4y ago

Whenever I see the name Marek Majkowski come up, I know the blog post is going to be good. I had to solve this exact problem a year ago when attempting to build an anycast forward proxy, quickly came to the conclusion that it'd be impossible without a massive infrastructure presence. Ironically I was using CF connections to debug how they might go about this problem, when I realized they were just using local unicast routes for egress traffic I stopped digging any deeper. Maintaining a routing table in unimog to forward lopsided egress connections to the correct DC is brilliant and shows what is possible when you have a global network to play with, however I wonder if this opens up an attack vector where previously distributed connections are now being forwarded & centralized at a single DC, especially if they are all destined for the same port slice...

jxhdkdhdks 4y ago
you wonder about an attack and everyone else wonder if that was the main point of all this.
deftoast 4y ago
I think this is no worse than using unicast IPs - they can probably reuse the same firewall technology to guard against attacks as before.

pm2222 4y ago

Packet headers contain essentially info that labels a packet. In v4 world they are running out of bits so they are maneuvering in my view that’s cool yet at the same time boring to me. Perhaps they should go v6 internally and implement one v4-v6 gateway to put all these tricks behind.

soumendrak 4y ago

Cloudflare and Akamai: competition

mcint 4y ago

Given the discussion under https://news.ycombinator.com/item?id=33743935 https://news.ycombinator.com/item?id=33743935 about opening a connection to 1.2.3.4:443 (I'm prompted by the same curiosity about ingress load-balancing, statelessly)... How does the ingress "router" load-balance incoming connections, which it must (even if the "router" is a default host or cluster)? CF isn't opening TCP, then HTTP just to send redirects to another IP for the same cluster. I guess hashing, on IP and port, is already readily used in routing decisions, so eyeball-addr and -port of the inbound packets, 4-tuple (CF-addr [fixed, shared], CF-port [443], eyeball-addr, eyeball-port), provides a consistent. I guess that this is good for 99.9 percent of connections, which are short-lived, and opportunistically kept open or reused. I suppose other long-lived connections might take place in the context of an application that tracks data above and outside of TCP-alone. I'm grasping for a missing middle, in size of use case, and can't quickly name things that people might proxy but need stable connections. CloudFlare's reverse proxying to web servers would count, if the web-fronting had to traverse someone else's proxy layer. What are the rough edges here? What's are next challenges here to build around?

BlueTemplar 4y ago

Related : Deploying IPv6-mostly access networks : https://news.ycombinator.com/item?id=33694293 https://news.ycombinator.com/item?id=33694293 (I can't for the life of me find the comment chain where this link was posted ~4 days ago ??) RFC 8925 – IPv6-Only Preferred Option for DHCPv4 (2020) : https://news.ycombinator.com/item?id=33697978 https://news.ycombinator.com/item?id=33697978 (posted on HN by me)

benlivengood 4y ago

It's strange/sad how 50% of their problem is caused by geofencing which is caused by archaic laws. 40% is caused by archaic protocols that don't allow enough addresses or local efficient P2P or ISP caching of public content (sort of what IPFS aimed to do), which would alleviate much of the need for CDNs in the first place. The remaining 10% is just solving hard problems. I'm a little surprised that splitting by port number gives servers enough connections; maybe they are connection-pooling some of them between their egress and final destination. If there was truly a 1:1 mapping of all user requests to TCP connections then I'd expect there to be ~thousands of simultaneous connections to, say, Walmart today (black Friday) which is also probably on an anycast address, limiting the number of unique src:dst ip and port tuples to 65K per cloudflare egress IP address. Maybe that ends up being not so bad and DCs can scale by adding new IPs? https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem... covers a lot of the details in solving that and similar problems.

zeckalpha 4y ago

Concerningly, they discuss routing traffic between jurisdictions when it is meant to be local. This is something state actors use to get around domestic spying rules. Neat technical solution though!

tobsterr 4y ago

As a networking noob, can someone explain to me how this solution plays nicely with the geofencing constraint described in the post? How does the IP/port combination map to a CIDR block defined in the geofeed?