3 ms·

ISPs more likely to throttle netizens who connect through CG-NAT: Cloudflare

76 points by throw0101a 11mo ago

lxgr 11mo ago
> Because CGNAT is more prominent, and more heavily used, in Africa and Asia […] Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data? I’ve also had DOCSIS connections, i.e., fixed lines, with only CG-NAT in Europe years ago.
- deleted 11mo ago
  [deleted]
- mcpherrinm 11mo ago
  I wonder if that's not very visible in Cloudflare's data because those mobile devices will likely use IPv6 to connect to Cloudflare-hosted sites.
  - trollbridge 11mo ago
    That’s what I was thinking. Anyone coming from Cloudflare will end up getting there via IPv6.
    - joecool1029 11mo ago
      I use cloudflare to make my weather station available over T-Mobile. They don’t filter inbound ipv6 on regular phone lines (they do for TMHI) so you can host a simple page on ipv6, only set the AAAA record in cloudflare, and they will proxy it for ipv4 users so I can ignore being CGNAT’d for ipv4. Make sure if you do this setup with a tool like ddclient to keep the record current as T-mobile rotates ipv6 frequently
      - trollbridge 11mo ago
        Well, what I mean to say was “anyone coming from Verizon will get to Cloudflare via IPv6”. My current endpoint lacks IPv6, so I use Cloudflare so IPv6 clients can get to it. Verizon’s IPv6 is noticeably faster than their IPv4 CG-NAT.
  - globalnode 11mo ago
    mobile devices dont get ip6 do they? last i looked my cheapo gateway only provided v4 cgnat
    - wmf 11mo ago
      Phones have been on IPv6 for years.
- userbinator 11mo ago
  AFAIK all mobile networks use NAT unless you pay a lot more for a special service with a public static IP.
- 0134340 11mo ago
  For those of us who don't have many options other than satellite internet, it generally uses CG-NAT, specifically Starlink.
- vitorgrs 11mo ago
  In Brazil I think basically more than half of fixed broadband should be CGNAT. Basically only one single ISP don't use CGNAT... Would be interesting if Cloudflare could give this info!
- throw0101a 11mo ago
  > Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data? T-Mobile, for one, has had their handsets IPv6-only for a few years, so if your Android/iOS does a DNS lookup and gets an AAAA record back, it will skip CG-NAT. T-Mobile presenting at NANOG in 2018 on IPv6: * https://www.youtube.com/watch?v=d6oBCYHzrTA https://www.youtube.com/watch?v=d6oBCYHzrTA And Rocky Mountain IPv6 Taskforce in 2017: * https://www.youtube.com/watch?v=nNMNglk_CvE https://www.youtube.com/watch?v=nNMNglk_CvE Further data: > Chances are if you use the Internet on your smartphone, you are connecting via IPv6. According to the Internet Society’s 2018 State of IPv6 Deployment,[1] 80% of smartphones in the US on the major cellular network operators use IPv6 and major mobile networks are driving IPv6 adoption with Verizon Wireless at 84%, Sprint at 70%, T-Mobile USA at 93%, and AT&T Wireless at 57%. Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs. * https://www.arin.net/blog/2020/01/16/mobile-edge-of-the-internet-is-rapidly-moving-to-ipv6/ https://www.arin.net/blog/2020/01/16/mobile-edge-of-the-inte... Though some folks aren't happy with the implementation: > But from my own experience, neither T-Mobile nor AT&T allows inbound traffic to the phone's IPv6 address. This negates some of the advantages of having a globally routable IPv6 address. * https://isc.sans.edu/diary/27814 https://isc.sans.edu/diary/27814
  - lxgr 11mo ago
    > Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs. I think this has also already happened. You don't really notice it as a user, because the phone itself can act as a 4-to-6 translator (similar to how DS-Lite in DOCSIS transports CG-NATted v4 traffic in v6) for both native apps doing incompatible things with literal v4 addresses and tethering clients.
mcpherrinm 11mo ago
The Register is adding very little on top of https://blog.cloudflare.com/detecting-cgn-to-reduce-collateral-damage/ https://blog.cloudflare.com/detecting-cgn-to-reduce-collater... Previously discussed (a bit) at https://news.ycombinator.com/item?id=45746509 https://news.ycombinator.com/item?id=45746509
daft_pink 11mo ago
They need to come up with an ip solution that is useful enough that people actually want to upgrade to it. When you compare it to other technologies like https, tls1.3, unicode, 5g cellular, wifi 6, wifi 5 or bluetooth versions, etc. It’s clear that ipv6 adoption is not what it should be if they launched a protocol with clearer benefits to the end user.
- ronsor 11mo ago
  It's the Internet protocol. End users are not supposed to interact with it directly. What exactly would replace IPv6? It's just an implementation detail, but an important one if you want to make the rest of the stack suck less.
- theamk 11mo ago
  Yeah, IPv6 is heavily tuned to the needs of the large-scale network operators, and is actively worse for the regular user and small networks. From user/small admin standpoint, the goal is to re-use as much admin knowledge as possible - and what's on the wire does not really matter. So the ideal IPv4 upgrade _for users_ is IPv4 with larger addresses, but otherwise behaving identically. Ideally all the admin tooling stays the same, and the software needs changing some struct names, and tweaking IP regex. And sure, it'll all be different on the wire and all the OS'es need to be upgraded - but that is not a problem, consumer OS'es live only for a few years anyway. From large network operator standpoint, the goal is improve efficiency of the huge networks. So lets eliminate NAT everywhere, completely redo host addressing, get rid of DHCP, and so on - redesign everything from scratch so it's "better". Sure, it's a huge learning curve but they have departments full of network engineers, they can do it. They are not some part-time sysadmins who just want their network to keep functioning.
  - OCTAGRAM 11mo ago
    It does not behave identically. I have MultiWAN on OPNsense. My PC IP is always 192.168.0.12. My router decides which upstream it should go. If I go full IPv6, router should derive double IPv6 from both WANs and if main upstream goes down, stop advertising IPv6 from main upstream. Or stop advertising gateway. I don't know what is the right IPv6 way of doing MultiWAN. Not only PC may change IP, but also servers. Legacy IPv4 DNS can be extended to IPv6, but that mechanical action is not flexible enough. With IPv6 we need to be able to mass replace IPv6 /64 prefix leaving all suffixes intact. We probably need /64 prefix alias system. Software is not prepared for this. In IPv4 SNAT and DNAT were being these "aliases". If NAT is not an option anymore, then DNS must step in. For many server software it just not possible to listen on multiple IPv6 address. Last time I tried MySQL, it just could not listen on multiple addresses. I could not make it listen on IPv4 and IPv6, specifying two addresses. MySQL server wanted just one address. This address could be [::], which means all interfaces and all protocols. And Linux implements some stupid hack to accept IPv4 connections to IPv6 socket. And Windows Vista also adopted this brainrot. But this is all wrong. Servers have to learn to listen to multiple IPs. This is normal. And for good IPv6 servers should learn to not only listen on multiple IPs, most wanted multiple IPv6, but also rebind listeners on the fly. If I got disconnected from ISP, reconnected by DHCPv6, and ISP assigned another IPv6 prefix, then DynDNS should update all my zones to new /64 prefix, and all servers in my network should rebind listeners. Or else we may abandon all that TRUE IPv6 philosophy and do SNAT in DNAT in IPv6 just like in IPv4, but with wider address space. But then again, software (another software) is not quite ready for this. Software is expecting public IPv6 address to be just reachable. And private IPv6 address to be just unreachable.
bethekidyouwant 11mo ago
The same applies to us rich folk on mobile. Not sure what the point of this article is.
- mrbluecoat 11mo ago
  I agree. A bunch of platforms use CGNAT, like Tailscale: https://tailscale.com/kb/1015/100.x-addresses https://tailscale.com/kb/1015/100.x-addresses
  - gertburger 11mo ago
    My understanding is that they use the IP range assigned for CGNAT as their private address range to avoid conflicts, but do they use CGNAT?
chrismorgan 11mo ago
I grew up in Australia, and have spent a fair bit of time in India for over a decade, and now live in India (1⅓ years). Every ISP that I have experienced, mobile and broadband, is using CGNAT. The easiest way I’ve seen this on broadband is https://iknowwhatyoudownload.com/ https://iknowwhatyoudownload.com/ showing several movie downloads per day. Cloudflare isn’t the only problem, but they are the worst, probably by dint of popularity. I get blocked outright occasionally (presented dishonestly as because my request matched attack patterns due to things like SQL injection in query string parameters, when I’m actually just trying to load any regular page), and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding”, though a few like blender.org customise it). It’s draining. In Cloudflare’s actual article, they claim their bot detection to be resilient to CGNAT <https://blog.cloudflare.com/detecting-cgn-to-reduce-collateral-damage/#users-behind-cgnat-are-more-likely-to-be-rate-limited https://blog.cloudflare.com/detecting-cgn-to-reduce-collater...>. Frankly, if it is so, I wonder if they just have a rule that amounts to “is user in India”. I definitely feel prejudged and discriminated against. I am idly curious if leasing a static IP from my ISP would help anything, in the short or long term. In Australia, I think I experienced Cloudflare’s blocking page once in my life, and no others.
- gruez 11mo ago
  >and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding” Isn't that from cloudflare, not hcaptcha?
  - chrismorgan 11mo ago
    I meant, in the context: blocked by Cloudflare with hCAPTCHA recourse. But as I consider it more carefully, I don’t think they don’t use hCAPTCHA in their challenges any more. They used reCAPTCHA at first, moved to hCAPTCHA around 2020, then they made their own thing Turnstile in ~2022 and migrated challenges to that at probably? the same time.
- protocolture 11mo ago
protocolture 11mo ago
The usual story is: 1. Add an IP that has been freed from another use to a CG-NAT Pool. 2. Get complaints from customers about being hard banned from things like Netflix, Sport Streaming and VPNs or other utilities. 3. Investigate, no IP reputation issues. Find some random GEO IP database that has a side business in selling lists of VPNS or other geo breakout tools. They have listed this IP for some random reason. Almost never nefarious. 4. Give it 3 weeks for the Geoip nard to update from the wrong classification (harmful) to some kind of also wrong but unharmful classification like "Datacentre" 5. Customers can stream The Witcher again. Yay. Really while ipv6 should be a solution here, another very good solution would be the removal of such useless middlemen from the face of the earth.
- kotaKat 11mo ago
  What I'd give at #2 to have some kind of backdoor ISP contact to these vendors to be able to ask "what exactly the hell is going on here?". Having to troubleshoot things like Playstation Network bans ("or is it a ban?") behind CGNAT is an interesting adventure that typically leads nowhere.
  - protocolture 11mo ago
    Few if any of them permit outside contact. I remember having to get a customer to escalate their PSN case after removing the GEOIP configuration because PSN didnt reflect the change immediately. Its really difficult from a customer service perspective, because the error messages tend to pin the blame on the ISP, despite it wholly being a circumstance of the content owner. You do get lucky, IIRC theres a few blog posts where people go into detail about how to contact some providers, but I found that some of them rotated those emails pretty regularly.
- 7bit 11mo ago
  As someone who read all books and played the third game: People should totally be banned from watching Netflix's Witcher. On a more serious note, I used GeoIP when it was free and it was a godsend to reduce malicious connection attempts to my webserver without impacting 99 % of my "clients" (not paying customers). These kind of services *are* helpful. Wehterh you should rely on them when you have millions of customers is a different story altogether.
winstonwinston 11mo ago
I’ve had IPv4 CG-NAT on mobile LTE since ever and for a decade on residential cable in europe. Cloudflare is being lazy, Google too. I get served “Captacha’s” at least 10 times a day. Even when i have IPv6 assigned, iOS and macOS seem to prefer A TXT RR and proceed just using IPv4 almost always. On LAN mDNS link-local IPv6 is always prefered.
Havoc 11mo ago
Isn’t that just a correlation with crappy internet? If you’re on non cg Nat it’s likely a pretty high end connection
xacky 11mo ago
It's almost impossible to edit Wikipedia on mobile networks due to ip hopping vandals on CGNAT.

76 points by throw0101a 11mo ago

lxgr 11mo ago

> Because CGNAT is more prominent, and more heavily used, in Africa and Asia […] Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data? I’ve also had DOCSIS connections, i.e., fixed lines, with only CG-NAT in Europe years ago.

deleted 11mo ago
[deleted]
mcpherrinm 11mo ago
I wonder if that's not very visible in Cloudflare's data because those mobile devices will likely use IPv6 to connect to Cloudflare-hosted sites.
- trollbridge 11mo ago
  That’s what I was thinking. Anyone coming from Cloudflare will end up getting there via IPv6.
  - joecool1029 11mo ago
    I use cloudflare to make my weather station available over T-Mobile. They don’t filter inbound ipv6 on regular phone lines (they do for TMHI) so you can host a simple page on ipv6, only set the AAAA record in cloudflare, and they will proxy it for ipv4 users so I can ignore being CGNAT’d for ipv4. Make sure if you do this setup with a tool like ddclient to keep the record current as T-mobile rotates ipv6 frequently
    - trollbridge 11mo ago
      Well, what I mean to say was “anyone coming from Verizon will get to Cloudflare via IPv6”. My current endpoint lacks IPv6, so I use Cloudflare so IPv6 clients can get to it. Verizon’s IPv6 is noticeably faster than their IPv4 CG-NAT.
- globalnode 11mo ago
  mobile devices dont get ip6 do they? last i looked my cheapo gateway only provided v4 cgnat
  - wmf 11mo ago
    Phones have been on IPv6 for years.
userbinator 11mo ago
AFAIK all mobile networks use NAT unless you pay a lot more for a special service with a public static IP.
0134340 11mo ago
For those of us who don't have many options other than satellite internet, it generally uses CG-NAT, specifically Starlink.
vitorgrs 11mo ago
In Brazil I think basically more than half of fixed broadband should be CGNAT. Basically only one single ISP don't use CGNAT... Would be interesting if Cloudflare could give this info!
throw0101a 11mo ago
> Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data? T-Mobile, for one, has had their handsets IPv6-only for a few years, so if your Android/iOS does a DNS lookup and gets an AAAA record back, it will skip CG-NAT. T-Mobile presenting at NANOG in 2018 on IPv6: * https://www.youtube.com/watch?v=d6oBCYHzrTA https://www.youtube.com/watch?v=d6oBCYHzrTA And Rocky Mountain IPv6 Taskforce in 2017: * https://www.youtube.com/watch?v=nNMNglk_CvE https://www.youtube.com/watch?v=nNMNglk_CvE Further data: > Chances are if you use the Internet on your smartphone, you are connecting via IPv6. According to the Internet Society’s 2018 State of IPv6 Deployment,[1] 80% of smartphones in the US on the major cellular network operators use IPv6 and major mobile networks are driving IPv6 adoption with Verizon Wireless at 84%, Sprint at 70%, T-Mobile USA at 93%, and AT&T Wireless at 57%. Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs. * https://www.arin.net/blog/2020/01/16/mobile-edge-of-the-internet-is-rapidly-moving-to-ipv6/ https://www.arin.net/blog/2020/01/16/mobile-edge-of-the-inte... Though some folks aren't happy with the implementation: > But from my own experience, neither T-Mobile nor AT&T allows inbound traffic to the phone's IPv6 address. This negates some of the advantages of having a globally routable IPv6 address. * https://isc.sans.edu/diary/27814 https://isc.sans.edu/diary/27814
- lxgr 11mo ago
  > Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs. I think this has also already happened. You don't really notice it as a user, because the phone itself can act as a 4-to-6 translator (similar to how DS-Lite in DOCSIS transports CG-NATted v4 traffic in v6) for both native apps doing incompatible things with literal v4 addresses and tethering clients.

mcpherrinm 11mo ago

The Register is adding very little on top of https://blog.cloudflare.com/detecting-cgn-to-reduce-collateral-damage/ https://blog.cloudflare.com/detecting-cgn-to-reduce-collater... Previously discussed (a bit) at https://news.ycombinator.com/item?id=45746509 https://news.ycombinator.com/item?id=45746509

daft_pink 11mo ago

They need to come up with an ip solution that is useful enough that people actually want to upgrade to it. When you compare it to other technologies like https, tls1.3, unicode, 5g cellular, wifi 6, wifi 5 or bluetooth versions, etc. It’s clear that ipv6 adoption is not what it should be if they launched a protocol with clearer benefits to the end user.

ronsor 11mo ago
It's the Internet protocol. End users are not supposed to interact with it directly. What exactly would replace IPv6? It's just an implementation detail, but an important one if you want to make the rest of the stack suck less.
theamk 11mo ago
Yeah, IPv6 is heavily tuned to the needs of the large-scale network operators, and is actively worse for the regular user and small networks. From user/small admin standpoint, the goal is to re-use as much admin knowledge as possible - and what's on the wire does not really matter. So the ideal IPv4 upgrade _for users_ is IPv4 with larger addresses, but otherwise behaving identically. Ideally all the admin tooling stays the same, and the software needs changing some struct names, and tweaking IP regex. And sure, it'll all be different on the wire and all the OS'es need to be upgraded - but that is not a problem, consumer OS'es live only for a few years anyway. From large network operator standpoint, the goal is improve efficiency of the huge networks. So lets eliminate NAT everywhere, completely redo host addressing, get rid of DHCP, and so on - redesign everything from scratch so it's "better". Sure, it's a huge learning curve but they have departments full of network engineers, they can do it. They are not some part-time sysadmins who just want their network to keep functioning.
- OCTAGRAM 11mo ago
  It does not behave identically. I have MultiWAN on OPNsense. My PC IP is always 192.168.0.12. My router decides which upstream it should go. If I go full IPv6, router should derive double IPv6 from both WANs and if main upstream goes down, stop advertising IPv6 from main upstream. Or stop advertising gateway. I don't know what is the right IPv6 way of doing MultiWAN. Not only PC may change IP, but also servers. Legacy IPv4 DNS can be extended to IPv6, but that mechanical action is not flexible enough. With IPv6 we need to be able to mass replace IPv6 /64 prefix leaving all suffixes intact. We probably need /64 prefix alias system. Software is not prepared for this. In IPv4 SNAT and DNAT were being these "aliases". If NAT is not an option anymore, then DNS must step in. For many server software it just not possible to listen on multiple IPv6 address. Last time I tried MySQL, it just could not listen on multiple addresses. I could not make it listen on IPv4 and IPv6, specifying two addresses. MySQL server wanted just one address. This address could be [::], which means all interfaces and all protocols. And Linux implements some stupid hack to accept IPv4 connections to IPv6 socket. And Windows Vista also adopted this brainrot. But this is all wrong. Servers have to learn to listen to multiple IPs. This is normal. And for good IPv6 servers should learn to not only listen on multiple IPs, most wanted multiple IPv6, but also rebind listeners on the fly. If I got disconnected from ISP, reconnected by DHCPv6, and ISP assigned another IPv6 prefix, then DynDNS should update all my zones to new /64 prefix, and all servers in my network should rebind listeners. Or else we may abandon all that TRUE IPv6 philosophy and do SNAT in DNAT in IPv6 just like in IPv4, but with wider address space. But then again, software (another software) is not quite ready for this. Software is expecting public IPv6 address to be just reachable. And private IPv6 address to be just unreachable.

bethekidyouwant 11mo ago

The same applies to us rich folk on mobile. Not sure what the point of this article is.

mrbluecoat 11mo ago
I agree. A bunch of platforms use CGNAT, like Tailscale: https://tailscale.com/kb/1015/100.x-addresses https://tailscale.com/kb/1015/100.x-addresses
- gertburger 11mo ago
  My understanding is that they use the IP range assigned for CGNAT as their private address range to avoid conflicts, but do they use CGNAT?

chrismorgan 11mo ago

I grew up in Australia, and have spent a fair bit of time in India for over a decade, and now live in India (1⅓ years). Every ISP that I have experienced, mobile and broadband, is using CGNAT. The easiest way I’ve seen this on broadband is https://iknowwhatyoudownload.com/ https://iknowwhatyoudownload.com/ showing several movie downloads per day. Cloudflare isn’t the only problem, but they are the worst, probably by dint of popularity. I get blocked outright occasionally (presented dishonestly as because my request matched attack patterns due to things like SQL injection in query string parameters, when I’m actually just trying to load any regular page), and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding”, though a few like blender.org customise it). It’s draining. In Cloudflare’s actual article, they claim their bot detection to be resilient to CGNAT <https://blog.cloudflare.com/detecting-cgn-to-reduce-collateral-damage/#users-behind-cgnat-are-more-likely-to-be-rate-limited https://blog.cloudflare.com/detecting-cgn-to-reduce-collater...>. Frankly, if it is so, I wonder if they just have a rule that amounts to “is user in India”. I definitely feel prejudged and discriminated against. I am idly curious if leasing a static IP from my ISP would help anything, in the short or long term. In Australia, I think I experienced Cloudflare’s blocking page once in my life, and no others.

gruez 11mo ago
>and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding” Isn't that from cloudflare, not hcaptcha?
- chrismorgan 11mo ago
  I meant, in the context: blocked by Cloudflare with hCAPTCHA recourse. But as I consider it more carefully, I don’t think they don’t use hCAPTCHA in their challenges any more. They used reCAPTCHA at first, moved to hCAPTCHA around 2020, then they made their own thing Turnstile in ~2022 and migrated challenges to that at probably? the same time.
protocolture 11mo ago

protocolture 11mo ago

The usual story is: 1. Add an IP that has been freed from another use to a CG-NAT Pool. 2. Get complaints from customers about being hard banned from things like Netflix, Sport Streaming and VPNs or other utilities. 3. Investigate, no IP reputation issues. Find some random GEO IP database that has a side business in selling lists of VPNS or other geo breakout tools. They have listed this IP for some random reason. Almost never nefarious. 4. Give it 3 weeks for the Geoip nard to update from the wrong classification (harmful) to some kind of also wrong but unharmful classification like "Datacentre" 5. Customers can stream The Witcher again. Yay. Really while ipv6 should be a solution here, another very good solution would be the removal of such useless middlemen from the face of the earth.

kotaKat 11mo ago
What I'd give at #2 to have some kind of backdoor ISP contact to these vendors to be able to ask "what exactly the hell is going on here?". Having to troubleshoot things like Playstation Network bans ("or is it a ban?") behind CGNAT is an interesting adventure that typically leads nowhere.
- protocolture 11mo ago
  Few if any of them permit outside contact. I remember having to get a customer to escalate their PSN case after removing the GEOIP configuration because PSN didnt reflect the change immediately. Its really difficult from a customer service perspective, because the error messages tend to pin the blame on the ISP, despite it wholly being a circumstance of the content owner. You do get lucky, IIRC theres a few blog posts where people go into detail about how to contact some providers, but I found that some of them rotated those emails pretty regularly.
7bit 11mo ago
As someone who read all books and played the third game: People should totally be banned from watching Netflix's Witcher. On a more serious note, I used GeoIP when it was free and it was a godsend to reduce malicious connection attempts to my webserver without impacting 99 % of my "clients" (not paying customers). These kind of services *are* helpful. Wehterh you should rely on them when you have millions of customers is a different story altogether.

winstonwinston 11mo ago

I’ve had IPv4 CG-NAT on mobile LTE since ever and for a decade on residential cable in europe. Cloudflare is being lazy, Google too. I get served “Captacha’s” at least 10 times a day. Even when i have IPv6 assigned, iOS and macOS seem to prefer A TXT RR and proceed just using IPv4 almost always. On LAN mDNS link-local IPv6 is always prefered.

Havoc 11mo ago

Isn’t that just a correlation with crappy internet? If you’re on non cg Nat it’s likely a pretty high end connection

xacky 11mo ago

It's almost impossible to edit Wikipedia on mobile networks due to ip hopping vandals on CGNAT.