9 ms·
This happens because there's NAT (network address translation) happening somewhere. Without NAT the only 2 parties that need to know anything about a TCP conne
by petethepig 6y ago
This happens because there's NAT (network address translation) happening somewhere.
Without NAT the only 2 parties that need to know anything about a TCP connection are client and server.
With NAT you have this problem where the router now also has to keep track of opened TCP connections.
E.g if you have a router with local IP 10.0.0.1 and external IP 30.0.0.1 and you are 10.0.0.2:55000 connecting to 230.0.0.1:443 router will have to allocate a port on it's external interface (let's say 56000) and remember it (this is the key part). So the connection will look like this:
10.0.0.2:55000 <-> NATing router 10.0.0.1 - 30.0.0.1:56000 <-> 230.0.0.1:443
When router receives packets on 30.0.0.1:56000 it has to remember to redirect them to 10.0.0.2:55000.
Memory is a limited resource so you can't just have an unlimited number of these opened connections floating around. This also makes your router vulnerable to an attack where an attacker can just open a bunch of connections and never close them, making your router eventually run out of memory.
So the classic solution to this problem is to use an LRU cache. So when your router is close to running out of space you just drop the connection that has been idling the longest.
Unfortunately, a) some routers are less sophisticated and will still drop your connections even if you do keep-alives and such, b) no matter what you do, memory is a finite resource and if the router doesn't have a lot of RAM, connections will be dropped.
¯\_(ツ)_/¯
- inetknght 6y agoAT&T's home gateways have a maximum NAT translation table of 1024^H^H^H^H8192 connections. Some websites will go past that. A torrent client almost certainly will. And, now that people are working from home, there's a good chance that having multiple computers will only make that 1024 table limit even more laughable. EDIT: okay I'm wrong. It's 8192 connections, not 1024 connections. But still ridiculously low
- MichaelApproved 6y ago> Some websites will go past that. Do you literally mean a website? Using a browser? What’s an example website that would go past that?
- taf2 6y agowouldn’t websockets be impacted by this limit?
- DoctorOW 6y agoYes but I've yet to see a website use more than 10 simultaneous websocket connections, let alone 1000.
- codebje 6y agoThere's something like a 256 count limit on total websockets, and 30 per domain, in Chromium. A malicious website could open up 256 websockets and as many HTTP connections as the browser allows, and that might be enough to swamp cheaper NATs. See https://bugs.chromium.org/p/chromium/issues/detail?id=12066 https://bugs.chromium.org/p/chromium/issues/detail?id=12066 for some 2009 discussion about people having troubles using the web when background tabs held connections open for polling. That wasn't a NAT issue, but it does highlight that a decade or two ago we all thought we only needed to manage tens of connections for a host to be online but that rapidly spiralled into hundreds.
- codebje 6y agoYou can overwhelm a NAT in several ways. UDP is connectionless, but typically a UDP communication is bidirectional. This means a NAT needs to inspect UDP packets and retain a mapping to direct incoming UDP packets to the right place. With no connection information this can only be done as an LRU cache or similar. TCP is connection oriented, and a NAT might rapidly free up resources when a connection is closed (ie, when the final FIN has been ACKed). But if there's no FIN, the NAT is in the same case as it is with UDP. Making a lot of connections without closing them fills up NAT buffers. When you have a home NAT and a carrier-grade NAT you may get an impedance mismatch of sorts. The CGNAT might have insufficient ports allocated to your service to keep up with your home NAT, resulting in timeouts or dropped mappings. Your home NAT will have one set of mappings and the CGNAT another, and the two sets probably won't be exactly the same. This means some portion of the mappings held in memory are useless. As a specific example, many years ago Google Maps would routinely trigger failures. Using Maps would load many tile images, which could overwhelm a NAT or CGNAT. The result was a map with holes in it where some tiles failed to load. Browsers have long had limits on concurrent connections per domain. Total concurrent connection limits are also old news, but are not quite as old as per-domain limits. You probably can't make a NAT choke with just simple web requests (even AJAX) any more. You might be able to do it using, eg, WebRTC APIs, though I would be surprised if those aren't also subject to limits.
- userbinator 6y ago1k certainly seems absurdly small considering how much RAM routers likely have, the fact that they can use most of it, and the amount of data needed for a single connection table entry (2 bytes external port, 2 bytes internal port, 4 bytes internal IP adds up to 8 bytes per entry, even being very generous at 16 bytes including overhead, that's still only 16K --- on a device that likely has several MB if not more, and whose primary function is likely NAT.
- Rapzid 6y agoSome providers do this to force you to upgrade to business plans. Comcast business though, at least a while back, still had a limit too low for the office I worked. We switched to ATT business fiber and used our own GW.
- l2p 6y agoJust as a FYI/aside, it is fairly trivial to root AT&T home gateways, pull the certs and use your own hardware to authenticate to the network, removing their hardware off your stack entirely except for the ONT. (goodbye internet downtime due to random uncontrolled gateway "upgrades"). You just need a router capable of 802.1x client auth. Throughput both ways actually gets really close to what I am paying for with this configuration, where as before with the default gateway (regardless of configuration), I was lucky to see 1/2 of the gigabit speeds I have been paying for.
- conk 6y agoThis is true for existing installs. But recently ATT moved to XGPON gateways with integrated ONT. You can no longer bypass these gateways. Also to my knowledge you can’t extract the certs from Pace gateways.
- inetknght 6y agoAnd, these gateways use NAT even when in "bridged mode"
- BrianGraggg 6y agoYou can request to go into bridge mode which will bypass the internal residential gateway (NAT).
- userbinator 6y agoIf an ISP is NAT'ing everyone (which I've heard of referred to as an "InterNAT Service Provider"), does "bridge mode" mean you get a real public IP? How does that work with everyone else still behind the NAT? (I have an actual end-to-end-connectable public IP from my ISP, which from the general discussion seems like an increasingly rare thing --- they keep pestering me to "upgrade" to outrageously faster yet slightly cheaper plans with a "free router included", so I suspect they are trying to get me to give up that IP...)
- 6y ago
- bryanrasmussen 6y agoI'm pretty sure this guy is not using anything from AT&T though, as the chances seem really good he's in Denmark.
- iforgotpassword 6y agoI know it's not 2002 anymore, but I'm pretty sure no website on this planet would even come close to 1000 open connections, unless it actively tries to achieve just that, but even then I think browsers still have a limit on number of concurrent open connections, per tab and maybe total.
- dmerrick 6y agoI just want to say thank you, this was a very concise explanation of a very complex concept. I've been working with NATs for years and your comment helped me "click" and understand them at a different level.
- Defenestresque 6y agoSimilarly, I found that OP's article provided an excellent primer on many concepts -- it certainly clarified the relationship between NAT and firewalls: that is, the latter being somewhat of an unintended consequence of the former. Stumbling upon a great blog post that makes something click is always a pleasant experience.
- anderstrier 6y agoThanks! I guess I should keep blogging then :)
- colejohnson66 6y agoMe too. I’ve always wondered how a NAT knows where to route traffic. I figured it would use a lookup table, but I never know what the “keys” were. For some reason, using different ports for each device behind the NAT never crossed my mind! I knew it couldn’t be done by adding routing data to the packets (which is what IPv6 ended up doing) because that isn’t sustainable over multiple NATs. A port based routing with a table makes so much sense! It also explains why idle sessions are dropped.
- tssva 6y ago"Without NAT the only 2 parties that need to know anything about a TCP connection are client and server." Even without NAT there may be multiple devices between a client and server which need to know about the TCP connection. Stateful firewalls, WAN accelerators, and load balancers are some examples.
- globular-toast 6y agoThey should all be under your control, though. Once you hand off to your ISP it should be nothing but routers all the way.
- tssva 6y agoI wish they were all under my control.
- sleepydog 6y agoIt's not just NATs that cause this. Stateful firewalls must also keep track connections to allow the responses for outbound requests that would not otherwise be allowed into the network. E.g. when you make a request to www.example.com:443 From source port 12345, and you or your isp has a firewall that blocks everything that isn't explicitly allowed (this is common in corporate networks), the response could be allowed using firewall rules such as iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT This has the benefit of being general, but the drawback is that the firewall now needs to track the connection, with similar consequences to the NAT example you have. It's also more likely for the firewalls to time out connections rather than use some kind of LRU scheme. In my opinion the time-based eviction is more predictable, so I prefer it. (Of course once you run out of memory you still need to evict "live" connections)
- globular-toast 6y agoIndeed. It's fairly common to mix up stateful firewalls with NAT. You can have a stateful firewall without NAT, but you can't have NAT without a firewall. It's actually the firewall that is keeping track of connections. The big difference here, though, is carrier-grade NAT. That means the firewall is not under your control and might have a tiny state table. NAT is bad enough as it is, but CGN should never have happened. It's just depressing to think about, to be honest. Even with IPv6 many ISPs are still doing it wrong. They'll give subscribers dynamic prefixes which means having to use unique local addresses (ULAs) in addition to their Internet routable addresses because the latter keep changing. This kind of stupidity makes people at home want to hang on to their IPv4 LANs because they seem more under their control. If only I could get an ISP like Hurricane Electric to provide me with a DSL line at home for a reasonable price. Consumer-grade ones are all hopelessly bad.
- dspillett 6y ago> but you can't have NAT without a firewall While it is true that most NAT arrangements are provided by firewalls, it is quite possible for a device to provide NAT with no other firewalling features at all, so not be considered a firewall. In this case the device would just be a router that provides NAT. Some confuse NAT and firewalling because NAT effectively implements a default-deny-all-not-initiated-here rule in one direction which is what most home users want in a firewall.
- samoa42 6y agoas noted ITT: set ServerAliveInterval in your ssh_config to avoid this (default uset)