5 ms·

Was hoping to see a summary, is there one? Another case is with nat at internet boundary. As far as I can tell, Cisco/palo firewall only track local_ip:port so

by pm2222 4y ago

Was hoping to see a summary, is there one? Another case is with nat at internet boundary. As far as I can tell, Cisco/palo firewall only track local_ip:port so one public ip only gives you 64k connections top.

majke 4y ago
Let me try. You could expect that on linux host you can have 64K concurrent connections to single target ip. If you had two target ips you could expect 128k concurrent conns. This is not how it works. Especially when doing bind-before-connect trick to set source ip. Linux internally tracks local ports in a hashtable and often forbids their reuse in surprising ways. The most surprising is ordering. If you do bind-before-connect then on that port later connect() will not work. The effect is that its very hard to achieve the 128k conncurrent in our two targets scenario
- toast0 4y ago
  I've done similar exercises with FreeBSD. The tldr is, if you want to approach the connection limits[1], you need to use bind-before-connect for the bulk connections, and it's useful to have connect on a separate (small) port range, for the ancillary connections your machine might have. There's lots of other things you might want to do, such as divide your outgoing ports by thread/cpu so there's no conflicts; and if you're dividing by cpu, you should probably calculate the recieve side scaling (RSS) hash, most likely Toeplitz of the returning packets, so you work the socket from userspace on the same CPU it's going to be worked in kernelspace when it comes in. Modern FreeBSD and presumably Linux are very good at scaling inbound connections, but if you need to scale outbound connections, you've got to do more of the work. [1] Also, consider if you're in the client seat, you're more likely to be the one closing the connections, so you've got TIME_WAIT states on your side; depending on your rate of connection closing, you may have a significant number of those, clogging up your ports; or it may not be significant at all.
hinkley 4y ago
The NAT box is proxying all the traffic. To the server you're contacting, the NAT box is the address. Which leads to fun things when you try to embed the callback address into the line protocol between the machines. It doesn't work. So the NAT box gets 64k local ports for the entire organization, not for your one computer. That's still a lot, but if you have one or two popular remotes you talk to, you only get less than 2x64k combinations (say, 64k + 32k). If you have 10 employees that's not a problem. If you have 50k and each service makes 2 connections, you can run out quickly.
- toast0 4y ago
  It could be that the NAT box isn't so great, and just limits itself to 64k outgoing connections per IP. Really depends on how hard the developers wanted to work. TCP itself doesn't have that limitation, but implementations of it may.
  - hinkley 4y ago
    NAT can be a feature or the whole product. I would certainly hope there are boxes out there that present multiple IP addresses to the 'outside' network and avoid these problems. It's an old technology and there's been plenty of room for improvement. I haven't caught any consumer products (eg, access points, routers, firewalls) doing this, and a lot of small businesses... well sometimes my work network is better maintained than my home network, and sometimes not so much.
deleted 4y ago
[deleted]

by pm2222 4y ago

majke 4y ago

Let me try. You could expect that on linux host you can have 64K concurrent connections to single target ip. If you had two target ips you could expect 128k concurrent conns. This is not how it works. Especially when doing bind-before-connect trick to set source ip. Linux internally tracks local ports in a hashtable and often forbids their reuse in surprising ways. The most surprising is ordering. If you do bind-before-connect then on that port later connect() will not work. The effect is that its very hard to achieve the 128k conncurrent in our two targets scenario

toast0 4y ago
I've done similar exercises with FreeBSD. The tldr is, if you want to approach the connection limits[1], you need to use bind-before-connect for the bulk connections, and it's useful to have connect on a separate (small) port range, for the ancillary connections your machine might have. There's lots of other things you might want to do, such as divide your outgoing ports by thread/cpu so there's no conflicts; and if you're dividing by cpu, you should probably calculate the recieve side scaling (RSS) hash, most likely Toeplitz of the returning packets, so you work the socket from userspace on the same CPU it's going to be worked in kernelspace when it comes in. Modern FreeBSD and presumably Linux are very good at scaling inbound connections, but if you need to scale outbound connections, you've got to do more of the work. [1] Also, consider if you're in the client seat, you're more likely to be the one closing the connections, so you've got TIME_WAIT states on your side; depending on your rate of connection closing, you may have a significant number of those, clogging up your ports; or it may not be significant at all.

hinkley 4y ago

The NAT box is proxying all the traffic. To the server you're contacting, the NAT box is the address. Which leads to fun things when you try to embed the callback address into the line protocol between the machines. It doesn't work. So the NAT box gets 64k local ports for the entire organization, not for your one computer. That's still a lot, but if you have one or two popular remotes you talk to, you only get less than 2x64k combinations (say, 64k + 32k). If you have 10 employees that's not a problem. If you have 50k and each service makes 2 connections, you can run out quickly.

toast0 4y ago
It could be that the NAT box isn't so great, and just limits itself to 64k outgoing connections per IP. Really depends on how hard the developers wanted to work. TCP itself doesn't have that limitation, but implementations of it may.
- hinkley 4y ago
  NAT can be a feature or the whole product. I would certainly hope there are boxes out there that present multiple IP addresses to the 'outside' network and avoid these problems. It's an old technology and there's been plenty of room for improvement. I haven't caught any consumer products (eg, access points, routers, firewalls) doing this, and a lot of small businesses... well sometimes my work network is better maintained than my home network, and sometimes not so much.

deleted 4y ago

[deleted]