10 ms·
When can two TCP sockets share a local address?
- loeg 4y agoNice drgn shout-out 2/3 of the way down the page! https://drgn.readthedocs.io/en/latest/index.html https://drgn.readthedocs.io/en/latest/index.html
- pm2222 4y agoWas hoping to see a summary, is there one? Another case is with nat at internet boundary. As far as I can tell, Cisco/palo firewall only track local_ip:port so one public ip only gives you 64k connections top.
- majke 4y agoLet me try. You could expect that on linux host you can have 64K concurrent connections to single target ip. If you had two target ips you could expect 128k concurrent conns. This is not how it works. Especially when doing bind-before-connect trick to set source ip. Linux internally tracks local ports in a hashtable and often forbids their reuse in surprising ways. The most surprising is ordering. If you do bind-before-connect then on that port later connect() will not work. The effect is that its very hard to achieve the 128k conncurrent in our two targets scenario
- toast0 4y agoI've done similar exercises with FreeBSD. The tldr is, if you want to approach the connection limits[1], you need to use bind-before-connect for the bulk connections, and it's useful to have connect on a separate (small) port range, for the ancillary connections your machine might have. There's lots of other things you might want to do, such as divide your outgoing ports by thread/cpu so there's no conflicts; and if you're dividing by cpu, you should probably calculate the recieve side scaling (RSS) hash, most likely Toeplitz of the returning packets, so you work the socket from userspace on the same CPU it's going to be worked in kernelspace when it comes in. Modern FreeBSD and presumably Linux are very good at scaling inbound connections, but if you need to scale outbound connections, you've got to do more of the work. [1] Also, consider if you're in the client seat, you're more likely to be the one closing the connections, so you've got TIME_WAIT states on your side; depending on your rate of connection closing, you may have a significant number of those, clogging up your ports; or it may not be significant at all.
- hinkley 4y agoThe NAT box is proxying all the traffic. To the server you're contacting, the NAT box is the address. Which leads to fun things when you try to embed the callback address into the line protocol between the machines. It doesn't work. So the NAT box gets 64k local ports for the entire organization, not for your one computer. That's still a lot, but if you have one or two popular remotes you talk to, you only get less than 2x64k combinations (say, 64k + 32k). If you have 10 employees that's not a problem. If you have 50k and each service makes 2 connections, you can run out quickly.
- toast0 4y agoIt could be that the NAT box isn't so great, and just limits itself to 64k outgoing connections per IP. Really depends on how hard the developers wanted to work. TCP itself doesn't have that limitation, but implementations of it may.
- hinkley 4y agoNAT can be a feature or the whole product. I would certainly hope there are boxes out there that present multiple IP addresses to the 'outside' network and avoid these problems. It's an old technology and there's been plenty of room for improvement. I haven't caught any consumer products (eg, access points, routers, firewalls) doing this, and a lot of small businesses... well sometimes my work network is better maintained than my home network, and sometimes not so much.
- deleted 4y ago[deleted]
- nh23423fefe 4y agomeaningless seo quantum keyword
- jeffbee 4y agoOne of the best things about user-space TCP implementations is never needing to be an archaeologist, never trying to figure out the path-dependent sequence of accidents that leads to surprising and undocumented kernel behaviors.
- FPGAhacker 4y agoWhat are arguments against user-space TCP?
- jeffbee 4y agoIt doesn't work out-of-the-box with any existing software, so it only applies to things you are building from scratch, or that are structured in such a way that adding an alternate network mode, or that would work with a dynamic library that shims the entire sockets API. Your user-mode stack won't be observable by any existing monitoring tools. Also, your process needs to execute with CAP_NET_RAW.
- chatmasta 4y agoOne argument is that it requires building on top of a raw socket, which can open you to all sorts of ancient vulnerabilities that have been patched in the battle-tested code running in the kernel, e.g. this recent ICMP remote code execution vulnerability [0] ("An attacker could send a low-level protocol error containing a fragmented IP packet inside another ICMP packet in its header to the target machine. To trigger the vulnerable code path, an application on the target must be bound to a raw socket") [1]. [0] Discussion: https://old.reddit.com/r/netsec/comments/11s80zo/cve202323415_icmp_remote_code_execution/ https://old.reddit.com/r/netsec/comments/11s80zo/cve20232341... [1] Advisory: https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-23415 https://msrc.microsoft.com/update-guide/vulnerability/CVE-20...
- ratorx 4y agoIf you are able, could you explain this in more detail? I find the description unparseable. Reading the words I do understand, the raw socket aspect seems to be irrelevant right? The vulnerable code would be the the parser that incorrectly runs code based on invalid input? It might require raw socket to trigger the vulnerability, but perhaps it would not exist if it was not written in kernel C but rather in user space garbage collected code with a good type system (not sure what the vulnerability actually is).
- chatmasta 4y agoI'm surprised the blog post does not mention Cloudflare's own library, tubular [0], the "BSD socket API on steroids": > The control plane for BPF socket lookup. Steers traffic that arrives via the tubes of the Internet to processes running on the machine. Its much more flexible than traditional BSD bind semantics: > * You can bind to all ports on an IP > * You can bind to a subnet instead of an IP > * You can bind to all ports on a subnet I played with it once and found it to be pretty awesome. [0] https://github.com/cloudflare/tubular https://github.com/cloudflare/tubular
- majke 4y agoThanks for reminder. This was a blog post after we hit the connectx() troubles :) Tubular/sk_lookup is about ingress. This blog post is about connected sockets on “egress”. Actually, bpf sk lookup could be used on egress, but its not quite yet implemented
- jmholla 4y agoI didn't really like the format of this article. The upfront quiz without much context or insight makes it feel like you're in class for an exam you didn't study for. The answers they provide are largely just Yes or No. When you get to the meat of the article, the quiz itself is forgotten until the conclusion. In fact, they even mention how useless the quiz is to provide insight at the beginning of the instructive content. > Is it all clear now? Well, probably no. It feels like reverse engineering a black box. So what is happening behind the scenes? Let's take a look. Keeping the quiz format, I think it would've been better to have it at the end then have the answers tie into the concepts you've learned. Or have the questions littered throughout the article and used to build upon the concepts it is teaching you. In its current state, I don't see this article's information having much staying power in my brain without concentrated study of the article.
- AstixAndBelix 4y agoThis article is meant for tech people and the purpose of the quiz is make you realize how little you know about the subject even as an expert. This is not a pop-sci youtube video
- ilyt 4y agoAnd it fails at that, all the format is is being annoying, and having to go thru links to random github gists is extra annoying if you didn't know the answer and want to just learn
- deleted 4y ago[deleted]
- dcow 4y agoSame. I don't want to go run python code to play your little quiz game, sorry. I think the article would be better structured as an explanation of what a connection is, its 5-tuple, and then a dive into any surprising exceptions to the rule that there can be no ambiguity in where a packet is delivered, it must go to exactly one spot on a host. So as long as you can find an unambiguous subset of available 5-tuples, you're allowed to bind/connect.
- wyldfire 4y agoThe identity of a TCP/IP connection is a quintuple containing { src addr, src port, dest addr, dest port, proto }. It feels like the article would be better if before or after all the network stack spelunking and quizzing that it was mentioned somewhere. We don't need to rely on the implementation, though there's always interesting corner cases worth exploring.
- loeg 4y agoI agree that is super important and maybe worth mentioning but the point of the quiz is to demonstrate that Linux's implementation is actually more constrained than the traditional "unique src addr/port dst addr/port 4-tuple" (for TCP).
- DSMan195276 4y agoYeah I feel like people are missing the point that it's not a 4-tuple thing, it's an ordering issue. Since the source port (sometimes) gets picked before the interface or destination is, you can get an EADDRNOTAVAIL result even when there's technically a potential for it to end-up as a unique 4-tuple. Doing the assignment in a different order or more explicitly can allow it to work by making sure that the kernel always knows it will be unique.
- grayfaced 4y agoIt's interesting design decision to preemptively fail when the connect might later be ambiguous. However in memory allocation, Linux will gladly allocate memory it doesn't have, return success and then later when you attempt to use it, it will kill the process with OOM killer.
- loeg 4y ago> However in memory allocation, Linux will gladly allocate memory it doesn't have, return success and then later when you attempt to use it, it will kill the process with OOM killer. In userspace, with overcommit enabled, yes. In the kernel, you will often do the opposite -- speculatively allocate memory first, then take locks or other serialization primitives, then attempt to insert the object into some container, but fail if there was a collision. (The idea is to keep the relatively slow memory allocation outside of the locked region.)
- waynesonfire 4y agoCan someone TLDR this? interesting question but not worth reading all this to find out. Can two IPs bind to the same IP / port on the server? Even if they come from the same client, my intuition says no.
- toast0 4y agoI'm having a hard time understanding your question. But let me try. If none of these answer your question, you're going to need to provide some more concrete details. If your sever has IPs 192.0.2.1 and 192.0.2.2. You can bind to 192.0.2.1:80 on one socket and 192.0.2.2:80 on another and listen for connections on both (or use it for outgoing connections, whatever). Or you could bind to 0.0.0.0:80 and listen for connections on either, as long as they don't have more specific bindings. If your client has IP 198.51.100.1, it can connect from tcp 198.51.100.1:18490 to 192.0.2.1:80 in one socket, and from tcp 198.51.100.1:18490 to 192.0.2.2:80 in another. You could also do the same on UDP, UDP's ports are orthogonal to TCP's. If you have another client with IP 198.51.100.2, it can also use local port 18490 to connect to both servers on port 80. A server can accept a virtually unlimited number of connections on a given tcp port, but a maximum of 64k per client IP, and your operating system will have some limit on the number of sockets; you can often raise that to some function of physical memory (FreeBSD limits you to a maximum of one fd for each four pages of physical memory); although, it takes a very specific load to be able to max out the FDs. Something like HAProxy in tcp forwarding mode can do it, but most other applications are going to use enough CPU and memory that you'll run out of those before you run out of FDs.
- waynesonfire 4y agoi was thinking of "scenario #1" in the article. you explained in well and i was wrong. i dug in a bit more and it looks like the linux kernel represents a connection as 4-tuple: source ip, source port, destination ip, destination port which is what allows the kernel to accept multiple connections on the listening port.