6 ms·
It would be very unusual for an ISP to drop idle connections. This implies all your connections are going through a layer 4 router. More likely you have a sta
by LinuxBender 6y ago
It would be very unusual for an ISP to drop idle connections. This implies all your connections are going through a layer 4 router. More likely you have a statefull firewall in the path somewhere. home router, server ISP firewall, etc...
[Edit: ] Or in this case, an ISP in Denmark that is trying to minimize ipv4 cost by using LSN (carrier grade nat) which also has many other drawbacks.
A SSH session does not generate any traffic
This does not have to be true. You can enable TCP keepalive in the server and client configuration.
Client via ~/.ssh/config:
TCPKeepAlive yes
ServerAliveInterval 60
ServerAliveCountMax 2
Server via /etc/ssh/sshd_config:
TCPKeepAlive yes
ClientAliveInterval 60
ClientAliveCountMax 2
Why are the TCP keepalives only sent after 2 hours?
Each OS has a default time set for keepalives. If you do not specify it in the ssh config, it will use the OS default. In Linux, you can set this in /etc/sysctl.conf:
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 2
After adding this, run sysctl -e -p
You can see the timers on your established connections with:
ss -emoian | grep tim
Note: TCP timers are not the same as ssh client and sshd server tcp keepalive packets. These are two distinctly different mechanisms that can accomplish the same thing. Not all applications support TCP socket keepalive. You can wrap applications with a library called libkeepalived to add support without code changes by using LD_PRELOAD.
In Windows this is set in the registry [1] On mac this is set via sysctl similar to Linux.
After you have adjusted your client and server config, restarted sshd on the server, then ssh to your server using the flag -vv and you will see the keepalive packets.
[1] - https://serverfault.com/questions/735515/tcp-timeout-for-established-connections-in-windows https://serverfault.com/questions/735515/tcp-timeout-for-est...
- anderstrier 6y agoI describe those workarounds in my post as well. But that only solves the problem for me. Making my ISP fix the underlying issue - that their TCP connection idle-timeout is too short - will make sure all their customers won't have to encounter this problem.
- LinuxBender 6y agoEdit: I missed the part that their network used LSN.
- anderstrier 6y agoPlease read the post. My ISP already confirmed the problem, and told me that they expect to roll out a fix this week. I live in Denmark, and here it is fairly common that ISPs do Carrier-grade NAT.
- LinuxBender 6y agoI missed that part. I would not have expected that in Denmark. LSN is awful. You will be sharing source port depletion limitations with others in your network. That also means you can't host any servers unless you use port forwarding services or reverse vpns like hamachi. It also means you are sharing a SNAT with others on your network which means that malicious traffic from others could be attributed to you. Glad they are fixing it for you. If they didn't, then one would hope there were other ISP options. Any ISP using LSN will have low NAT timeouts because it takes memory on their routers to track sessions and state. I would be surprised if your ISP remove timeouts unless they are letting it fall back to FIFO pruning on your segment. Did they tell you what they are changing?
- toast0 6y agoIt sounds like he's paid his ISP for a (dedicated) public IP, so it should be 1:1 NAT, which doesn't really need connection tracking. For the rest of the customers that don't pay extra for a public IP, all the crappy things you mention do apply. Hopefully, the ISP does native IPv6? And, while 60 minute timeouts violate the RFC, it's a whole lot better than I expected. Usually CGN timeouts are around 15 minutes for nice ones, and I've seen 10 seconds at the bottom end. I wish the longer ones would probe both ends of the connection to see if it's still live a minute or so before they intend to kill it.
- eznzt 6y agoWhat you say sounds very dramatic, but the truth is that CGNAT is good enough for 99.9% of users.
- aMadMan 6y agoServerAliveInterval and/or ClientAliveInterval fix that behaviour just fine, for everyone, if they use it.
- theamk 6y agoCorrection: TCP keepalives, ssh server keepalives, and ssh client keepalives are three distinct and independent mechanisms. You only need one. I usually just do client keepalives as they are easiest to set up. Server keepalives are good if you are worried about “forgotten” clients. TCP keepalives are usually not worth it IMHO.
- gregmac 6y agoI also changed to using client keepalives after something in our office network changed: they installed new switches and access points and suddenly my ssh sessions wouldn't stay open. After getting nowhere with IT (mainly just a low priority issue to them) it was just less frustrating to enable keepalives and the problem disappeared, so that's my default config everywhere ever since.
- GekkePrutser 6y agoI think TCP keepalives are conceptually the best though. As your problem occurs at the transport level, not the application layer. This way you solve it where the issue occurs, and with the added benefit that it works for all TCP connections, not just SSH. However I haven't had this issue. My isp is pretty ok in this regard and I supply my own router. So I don't know if there's issues with this in real life.
- dfox 6y agoSome NAT implementations ignore TCP keepalives. Alcatel branded ADSL modem/router I had used in 2005-ish certainly did and IIRC some more recent Zyxel ones do the same.
- amelius 6y agoDo you know of some mechanism that makes ssh sessions survive a power-suspend (on a Linux desktop)?
- lytedev 6y agoYou should check out mosh.
- dfox 6y agoThat is pretty much an inverse problem ;) If you care about that you probably should use mosh as that does solve that by design and not by random chance. On the other hand using VPN with fixed tunneled endpoint IPs causes idle TCP connections going through it to remain connected pretty much indefinitely.
- jeffbee 6y agoJust don’t use keep-alive feature. Without keep-alive traffic the peers have no way to tell the interface was transiently unavailable.
- jusssi 6y agoThis is needed, in addition to the machine getting the same IP address back after resume (static assignment or long DHCP leases).
- deleted 6y ago[deleted]