20 ms·
My ISP Is Killing My Idle SSH Sessions
- LinuxBender 6y agoIt would be very unusual for an ISP to drop idle connections. This implies all your connections are going through a layer 4 router. More likely you have a statefull firewall in the path somewhere. home router, server ISP firewall, etc... [Edit: ] Or in this case, an ISP in Denmark that is trying to minimize ipv4 cost by using LSN (carrier grade nat) which also has many other drawbacks. A SSH session does not generate any traffic This does not have to be true. You can enable TCP keepalive in the server and client configuration. Client via ~/.ssh/config: TCPKeepAlive yes ServerAliveInterval 60 ServerAliveCountMax 2 Server via /etc/ssh/sshd_config: TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 2 Why are the TCP keepalives only sent after 2 hours? Each OS has a default time set for keepalives. If you do not specify it in the ssh config, it will use the OS default. In Linux, you can set this in /etc/sysctl.conf: net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_keepalive_intvl = 60 net.ipv4.tcp_keepalive_probes = 2 After adding this, run sysctl -e -p You can see the timers on your established connections with: ss -emoian | grep tim Note: TCP timers are not the same as ssh client and sshd server tcp keepalive packets. These are two distinctly different mechanisms that can accomplish the same thing. Not all applications support TCP socket keepalive. You can wrap applications with a library called libkeepalived to add support without code changes by using LD_PRELOAD. In Windows this is set in the registry [1] On mac this is set via sysctl similar to Linux. After you have adjusted your client and server config, restarted sshd on the server, then ssh to your server using the flag -vv and you will see the keepalive packets. [1] - https://serverfault.com/questions/735515/tcp-timeout-for-established-connections-in-windows https://serverfault.com/questions/735515/tcp-timeout-for-est...
- anderstrier 6y agoI describe those workarounds in my post as well. But that only solves the problem for me. Making my ISP fix the underlying issue - that their TCP connection idle-timeout is too short - will make sure all their customers won't have to encounter this problem.
- LinuxBender 6y agoEdit: I missed the part that their network used LSN.
- anderstrier 6y agoPlease read the post. My ISP already confirmed the problem, and told me that they expect to roll out a fix this week. I live in Denmark, and here it is fairly common that ISPs do Carrier-grade NAT.
- LinuxBender 6y agoI missed that part. I would not have expected that in Denmark. LSN is awful. You will be sharing source port depletion limitations with others in your network. That also means you can't host any servers unless you use port forwarding services or reverse vpns like hamachi. It also means you are sharing a SNAT with others on your network which means that malicious traffic from others could be attributed to you. Glad they are fixing it for you. If they didn't, then one would hope there were other ISP options. Any ISP using LSN will have low NAT timeouts because it takes memory on their routers to track sessions and state. I would be surprised if your ISP remove timeouts unless they are letting it fall back to FIFO pruning on your segment. Did they tell you what they are changing?
- toast0 6y agoIt sounds like he's paid his ISP for a (dedicated) public IP, so it should be 1:1 NAT, which doesn't really need connection tracking. For the rest of the customers that don't pay extra for a public IP, all the crappy things you mention do apply. Hopefully, the ISP does native IPv6? And, while 60 minute timeouts violate the RFC, it's a whole lot better than I expected. Usually CGN timeouts are around 15 minutes for nice ones, and I've seen 10 seconds at the bottom end. I wish the longer ones would probe both ends of the connection to see if it's still live a minute or so before they intend to kill it.
- eznzt 6y ago
- theamk 6y agoCorrection: TCP keepalives, ssh server keepalives, and ssh client keepalives are three distinct and independent mechanisms. You only need one. I usually just do client keepalives as they are easiest to set up. Server keepalives are good if you are worried about “forgotten” clients. TCP keepalives are usually not worth it IMHO.
- gregmac 6y agoI also changed to using client keepalives after something in our office network changed: they installed new switches and access points and suddenly my ssh sessions wouldn't stay open. After getting nowhere with IT (mainly just a low priority issue to them) it was just less frustrating to enable keepalives and the problem disappeared, so that's my default config everywhere ever since.
- GekkePrutser 6y agoI think TCP keepalives are conceptually the best though. As your problem occurs at the transport level, not the application layer. This way you solve it where the issue occurs, and with the added benefit that it works for all TCP connections, not just SSH. However I haven't had this issue. My isp is pretty ok in this regard and I supply my own router. So I don't know if there's issues with this in real life.
- dfox 6y agoSome NAT implementations ignore TCP keepalives. Alcatel branded ADSL modem/router I had used in 2005-ish certainly did and IIRC some more recent Zyxel ones do the same.
- amelius 6y agoDo you know of some mechanism that makes ssh sessions survive a power-suspend (on a Linux desktop)?
- lytedev 6y agoYou should check out mosh.
- dfox 6y agoThat is pretty much an inverse problem ;) If you care about that you probably should use mosh as that does solve that by design and not by random chance. On the other hand using VPN with fixed tunneled endpoint IPs causes idle TCP connections going through it to remain connected pretty much indefinitely.
- jeffbee 6y agoJust don’t use keep-alive feature. Without keep-alive traffic the peers have no way to tell the interface was transiently unavailable.
- jusssi 6y agoThis is needed, in addition to the machine getting the same IP address back after resume (static assignment or long DHCP leases).
- deleted 6y ago[deleted]
- rachelbythebay 6y agoIPv6. Push for it.
- pacamara619 6y agoI see an IPv6 address on a computer screen. I want to connect to it from another computer. Copy pasting doesn't work between computers. Am I supposed to type it letter after letter? Am I supposed to send somehow? What if I don't have internet access on the first computer? Do I need to go find a USB stick so I can transfer a file with the IPv6 address? Am I supposed to set up a DNS server of some kind? I'll keep my IPv4 thank you very much.
- p1mrx 6y agoI use https://dns.he.net https://dns.he.net, with an hourly cron job that runs 'curl' to keep the address updated.
- deleted 6y ago[deleted]
- lmm 6y agoDo you seriously ssh to raw IPv4 addresses? Everywhere I ssh to (including my home server) has a DNS address and that's how I connect to it.
- Mediterraneo10 6y agoWhenever I have got a new Raspberry Pi or PinePhone and connect it to the home network, I always SSH to the raw IP first. (Sure, at some point I’ll configure the router’s DHCP settings to ensure the new device gets a stable IP address, and then I can just use an SSH alias.) I would imagine that this is a very common use case.
- fsh 6y agoAny decent home router will automatically add the hostname of all DHCP(v6)-configured devices to its DNS service. You shouldn't need raw addresses at all.
- Causality1 6y agoInteresting. I hadn't really been familiar with carrier-level NAT. How does that work with say, a home server that other people connect to using only an IP address and a port number?
- jepler 6y agoIt doesn't.
- Causality1 6y agoSo one day I could wake up to find it's impossible to host a Minecraft server at home, with nothing I can do about it?
- welterde 6y agoPerhaps - depends on the growth rate of the ISP and how many IPv4 addresses they already have (and how much money they are willing to spend to acquire more). But the real fix is to push your ISP to deploy IPv6. No need for the ISP to run carrier-grade NAT and you can host as many services at home as you want.
- BlueTemplar 6y agoYeah, several of people I played with using various games over the years had this issue. I even had it too when I used cell Internet, but cell ISPs have a better excuse. We generally found someone else able to host, except for some 1vs1 situations.
- peter_d_sherman 6y ago>"A SSH session does not generate any traffic, unless there’s new output or input. The same is true for TCP. That is why, after the TCP and SSH sessions have been established, no more packages are sent for a long time ."
- eznzt 6y agoI haven't read TFA, but it happened to me on an ISP which had a CGNAT, probably as a way of garbage collecting open connections. The solution was to use an application-level keepalive, which winscp and putty support.
- arghwhat 6y ago*workaround. The solution is for the ISP to fix their misconfigured NAT.
- eznzt 6y agoTaking into account they had to take extra steps to enable this "feature", they probably don't consider it to be "misconfigured", at least from their point of view.
- bob1029 6y agoNAT is fundamentally not the solution.
- tenebrisalietum 6y agoThe solution is IPv6.
- api 6y agoA NAT is going to have to have a timeout, otherwise it will gradually leak and run out of ports. All protocols that operate behind NAT must implement keepalive. The solution is IPv6. Then you don't need your ISP to maintain a stateful connection table.
- _wldu 6y agoRunning tmux or screen with top in one window should fix it.
- deleted 6y ago[deleted]
- asdff 6y agoWhat happens when it doesn't? I've messed with my ssh config and TCP keepalive settings. I've made sure to set my computer to not go to sleep when I have open network connections. No matter what I do, if I leave the terminal window open overnight and come back in the morning, I get a broken pipe and have to log back into my cluster, and reattach my tmux session. I've pretty much just given up on finding a solution.
- lrossi 6y agoOr just display the clock in the tmux status bar.
- dmerrick 6y agoBy default `<prefix> t` shows a clock in the current pane, really handsome looking and it scales nicely as the pane changes shape.
- tantalor 6y agoThe very first thing I do whenever I ssh, $ screen -x If you don't do this or something similar, then perhaps you should start now. https://www.gnu.org/software/screen/ https://www.gnu.org/software/screen/
- mankyd 6y agoI've always use `-D -R`. I just checked `man screen` and it actually states "This is the author's favorite."
- lhoff 6y agoThat doesn't really help when transfering a file. That was one of the user cases where the author had problem with.
- tantalor 6y agoYes it would because the file transfer would continue inside the screen session instead of dying with the ssh session.
- jimmaswell 6y agoOr keeping an SSH session open for proxy tunneling when PuTTy is only capable of automatically reconnecting on certain disconnects and not others for some stupid reason.
- lazyjones 6y agorsync over ssh solves this... Just start it again and transfer the remaining part.
- throwaway888abc 6y agoSame issue from one of client location. Didn't bothered with debugging but switching connection to over Wireguard vpn resolved it.Just 2c in case someone has similar issue. EDIT: Also see Mosh (https://mosh.org/ https://mosh.org/)
- oandrew 6y ago[mosh or eternal terminal] + tmux combo solves the problem (and adds other cool functionality) https://mosh.org/ https://mosh.org/ https://eternalterminal.dev/ https://eternalterminal.dev/ https://github.com/tmux/tmux/wiki https://github.com/tmux/tmux/wiki
- nh2 6y agoIf you require SSH features (e.g. port forwarding) or want to continue using your terminal's native scroll functionality, here's another alternative that a friend and I devised: https://mazzo.li/posts/autoscreen.html https://mazzo.li/posts/autoscreen.html
- hansvs 6y agoThis is great, thanks!
- FunnyLookinHat 6y agoMosh is great - it drastically improved my ability to work remotely over my DSL connection. Adding byobu made it effectively impervious.
- perryizgr8 6y agoI hate mosh because it nukes scrolling. I hate tmux because I just can't remember the keybindings. I've been using mosh+byobu since the keybindings are a bit easier for me. But I'd love regular ssh to be resilient like mosh, or maybe mosh can start supporting scrolling.
- shakna 6y agoI use mosh + screen, because you can tell screen to act like it has normal scrolling with a simple config and then the only keybinding you need to remember is disconnect. defscrollback 500000 scrollback 500000 termcapinfo xterm* ti@:te@ There you go, normal scrolling!
- 6y ago
- CamperBob2 6y agoFrankly I've never seen any connection that didn't use keepalives stay up for very long on my LAN, much less the Internet at large. Keepalives are a basic requirement for persistent connections. It's literally what they're for. Harmless enough solution.
- CountSessine 6y agoThat’s a bit strange though, don’t you think? TCP state only lives in the endpoints, unless you have something awful like NAT in between. Without NAT, why are keepalives a basic requirement for persistent connections?
- CamperBob2 6y agoI run under NAT, like most Internet users in the US, although my gateway has a static IP. No clue why the LAN drops connections without keepalive traffic. I need to get up to speed on using WireShark to diagnose dropped connections one of these days, as I actually have a couple of dropped-connection issues that need troubleshooting. Typically I'll get a connection timeout every few days, even with keepalives, and even with a hardwired DHCP address for the MAC in question. Connections with no traffic tend to get 'cleaned up' by somebody after a few hours at most.
- sonotmyname 6y agoWithout timeouts, one of the endpoints would have to maintain dead sessions indefinitely. One cannot rely on protocols to close connections properly - stuff happens.
- innocenat 6y agoBack before I know about SSH KeepAlive setting, I have had SSH connection going on for entire workday (~8 hours or so) on my LAN to local server.
- syntaxing 6y agoThis + VPN idle killing sessions has forced me to get much better at tmux. I highly recommend for others who use SSH often, especially with WFH, to give it a try. The pane structure has made me much more efficient at my work nowadays.
- swalladge 6y ago+1 to tmux, or even screen if you can't get tmux installed. It also protects against flaky network connections and accidentally closing the local terminal emulator.
- whimsicalism 6y agotmux is legitimately one of my most useful tools in day to day work.
- aftbit 6y agoWhen my company started, we used tmux in PROD as a process supervisor. We had tons of lines like: while true; do ./runserver; sleep 10; done in a shell script that would start a new server with one window per process. It got called from /etc/rc.local on boot IIRC. Deploys meant pulling the new code on the server (or maybe just editing it in vim right there), then just Ctrl-C in every terminal (or when I got lazier, `killall runserver`). This was back in 2010 or so... I have professionally come a long way in the intervening decade, and no longer have any PROD services running in tmux panes, but I definitely learned to love that tool.
- rovr138 6y agoI've done things similar and do it for quick and dirty. Currently I have something like that on a RPi but instead of while true; I have, while [[ ! -f "$DIRECTORY"/backup_exit.stop ]]; do sleep 3600 # sleep an hour time /home/ubuntu/.pyenv/versions/backup/bin/python main.py done Whenever I want the process to stop, `touch backup_exit.stop` . This waits until the run finishes and exists on the next loop. Anyway, that is saved to a 'backup.sh' Then I have also, #!/bin/bash session="test" backup="/path/to/backup.sh" #create detached session named test tmux new-session -d -s ${session} # Create windows tmux rename-window -t :0 'backup' #rename the first one tmux new-window -n 'htop' # Run processes tmux send-keys -t 'backup' "$backup" ENTER tmux send-keys -t 'htop' 'htop' ENTER And this is run automatically on @reboot via cron.
- timmorgan 6y agoThanks for sharing this. I’d always wondered how NAT works and this explanation worked well for me. Nice bit of investigation too!
- f430 6y agoUnrelated but when I went to university I was ssh'd into my webhost to build a website and some Ken (male version of Karen) reported me to the administration for "hacking" and I could no longer access my server without a VPN. He kept a frown on his face and asked me what I was doing. I told him none of your fucking business and he assumed the worst.
- astrea 6y agoSounds like you both could use a lesson in interpersonal conduct
- notretarded 6y agoKyle not Ken
- axaxs 6y agoOf all the shitty things 2020 will be known for, this stupid calling everyone Karen/Ken is the absolute dumbest. Please, for the sake of our species, stop.
- f430 6y agocalm down Ken
- BlueTemplar 6y agoJust a repeat of the "Kevin" a few years later. I'm afraid this is never going to stop, though we have to fight it.
- petethepig 6y agoThis happens because there's NAT (network address translation) happening somewhere. Without NAT the only 2 parties that need to know anything about a TCP connection are client and server. With NAT you have this problem where the router now also has to keep track of opened TCP connections. E.g if you have a router with local IP 10.0.0.1 and external IP 30.0.0.1 and you are 10.0.0.2:55000 connecting to 230.0.0.1:443 router will have to allocate a port on it's external interface (let's say 56000) and remember it (this is the key part). So the connection will look like this: 10.0.0.2:55000 <-> NATing router 10.0.0.1 - 30.0.0.1:56000 <-> 230.0.0.1:443 When router receives packets on 30.0.0.1:56000 it has to remember to redirect them to 10.0.0.2:55000. Memory is a limited resource so you can't just have an unlimited number of these opened connections floating around. This also makes your router vulnerable to an attack where an attacker can just open a bunch of connections and never close them, making your router eventually run out of memory. So the classic solution to this problem is to use an LRU cache. So when your router is close to running out of space you just drop the connection that has been idling the longest. Unfortunately, a) some routers are less sophisticated and will still drop your connections even if you do keep-alives and such, b) no matter what you do, memory is a finite resource and if the router doesn't have a lot of RAM, connections will be dropped. ¯\_(ツ)_/¯
- inetknght 6y agoAT&T's home gateways have a maximum NAT translation table of 1024^H^H^H^H8192 connections. Some websites will go past that. A torrent client almost certainly will. And, now that people are working from home, there's a good chance that having multiple computers will only make that 1024 table limit even more laughable. EDIT: okay I'm wrong. It's 8192 connections, not 1024 connections. But still ridiculously low
- MichaelApproved 6y ago> Some websites will go past that. Do you literally mean a website? Using a browser? What’s an example website that would go past that?
- 6y ago
- dn3500 6y agoTo transfer a multi-hundred-GB file I'd use something restartable like rsync or wget instead of nc.
- kayodelycaon 6y agoThe issue in the article could have been solved by running a long-running command with no output in screen or tmux. If ssh disconnects, just reattach.
- cat199 6y agono, because this example was site->site data transfer across the disconnecting link
- BossHamster 6y agoyes, because the issue was not the site data transfer, but the idle SSH session(s) running the nc commands. When the idle SSH session got dropped, it took nc down with it and that stopped the transfer.
- the8472 6y agono need for tmux, nohup would have done the job.
- screamingninja 6y agohttps://unix.stackexchange.com/questions/3026/what-options-serveraliveinterval-and-clientaliveinterval-in-sshd-config-exac#3027 https://unix.stackexchange.com/questions/3026/what-options-s...
- centimeter 6y agoWireguard will help you work around this. Just tunnel ssh through it. You will still have nat failures but wg will reconnect and things will look stable to ssh.
- shmerl 6y agoIt's really a problem that many ISPs are lagging with transition to IPv6 still.
- bcrl 6y agoIt's not surprising. Try dealing with consumer routers and their varying degrees of support for dynamic assignment of IPv6 addresses. Seriously, it's a support nightmare.
- shmerl 6y agoSurprisingly, the problem is often their backend support for it, not so much the end user issues.
- KirillPanov 6y agoIt's not surprising because IPv6 completely ignored transition planning: https://cr.yp.to/djbdns/ipv6mess.html https://cr.yp.to/djbdns/ipv6mess.html Twenty years later, it still hasn't displaced IPv4. IPv6 was designed by a bunch of telephone-company guys who were used to Ma Bell being able to declare a flag day. That doesn't scale up to the Internet, and we're all suffering for their failure to understand backward compatibility.
- bcrl 6y agoThis. When I (as an ISP) was looking into the consumer router situation 10 years ago, the protocols required to handle dynamic IPv6 assignment to end users in conjunction with PPPoE simply didn't exist. Specifically, without NAT, the ISP has to somehow delegate an IPv6 subnet to the customer for their local network. The mix of IPV6CP and DHCPv6 simply weren't able to do that, and if you've ever had to deal with this for more than a handful of customers, you know that getting $joerandomenduser to manually configure an IPv6 subnet just isn't going to work. Sure, if you can control what router every customer uses you can come up with a procedure for that, but going through the wholesale network of an incumbent means you don't have that luxury.
- the8472 6y ago
- shk1338 6y agoSame for me. To avoid the problem I'm using tmux inside SSH session.
- floatingatoll 6y agoHas anyone implemented OpenSSH over QUIC yet? If ever there was a match made in heaven for pairing an asynchronous channels protocol encapsulated over TCP with an asynchronous channels protocol encapsulated over UDP, this seems like one. EDIT: Yep, there's `draft-bider-ssh-quic-09` and a localhost proxycommand that offers both-sides OpenSSH integration: https://datatracker.ietf.org/doc/draft-bider-ssh-quic/ https://datatracker.ietf.org/doc/draft-bider-ssh-quic/ https://github.com/moul/quicssh https://github.com/moul/quicssh
- KirillPanov 6y ago> If ever there was a match made in heaven I think SIP+RTP over QUIC might be a contender for that title. No more NAT traversal wackiness caused by needing both a TCP connection and a UDP connection. I still can't figure out why SIP didn't use TCP-over-UDP for the low-bandwidth control packets. QUIC has protocol-level support for running both a reliable (TCP-like) and datagram (UDP-like) substream over the same QUIC connection. Cannot wait for SIP-over-QUIC!
- floatingatoll 6y agoWhen SIP first came out, UDP wasn’t safe through NAT like it is now. Lots of cheap home routers would just wreck it. It’s gotten better since then.
- ddtaylor 6y agoIt's worth mentioning if you have a large job that will need to finish you should disown the processor or use setsid: setsid somecommand --blah &
- blazor 6y agoMy wireless hub is doing that not my ISP.
- henearkr 6y agoFor this reason, I use autossh. I recommend it.
- boneitis 6y agoAs do I. Every time I need to set it up, I go through this series of very short and digestible articles. https://news.ycombinator.com/item?id=10937277 https://news.ycombinator.com/item?id=10937277 (discussion for the third in a 4-part series). Never thought to look for an HN discussion on it, but it turns out that there is one. IIRC, one thing the series doesn't mention is that a particular option needs to be explicitly specified in order to maintain port tunnels. By default, as long as the SSH connection succeeds during an AutoSSH reconnect, it will chug along happily without the port-forward if it's still blocked from the previous connection before the drop.
- flaxton 6y agoYou should try tmux over ssh over mosh. Your session lives on thanks to tmux. And mosh reconnects you automatically, even when your IP address changes ;-)
- hinkley 6y agoThe good ol’ days, when you had to keep something sending a steady packets across the ssh session to keep the connection from dropping.
- xaduha 6y agoIPv6 is pretty nice. Once it's working natively on both sides that is.
- donretag 6y ago"I documented my findings, and sent an email to my ISP. I quickly got a response back acknowledging that this is a bug on their side, and thanking me for my research." I am actually shocked that: a) the ISP has an email or any sort of asynchronous communication. Most in the US have at best a "talk to a bot" functionality b) they acknowledge the behavior and did not simply respond "have you tried rebooting the router?"
- dotdi 6y agoI'm working on a product that requires to keep the same TCP connection open for a long time, and my company found the same: After a lot of resources invested in debugging, switching out hardware, and interminable log files we suspected the ISP is to blame, so we built an MVP that was connected directly to what the ISP provides, and a client on a different ISP that wasn't closing connections. We saw the connection close at exactly the same time as before.
- scraft 6y agoJust a little FWIW, I am with Hyperoptic in the UK (pretty common if you live in an apartment in a big city centre). They provide nice fast, cheap, fibre broadband (1Gbit/sec in both directions). I started allowing remote connections on my Plex (media server) but couldn't get it to work, which is puzzling as I have been setting up put forwarding for 25 years, so thought I had run out of unsolved mysteries. Then I read about CGNAT. Then I found out Hyperoptic uses CGNAT. I felt like I had been swindled, an internet connection which ample bandwidth for hosting services with no way to have any incoming ports (as no matter what portforwarding I set up on my router, the ISPs CGNAT router doesn't let me set any portforwarding up as it is out of my control). I spoke to Hyperoptic, they said I could have a fixed (non CGNAT) IP for five months for free and then £5 per month thereafter. After five months I noticed they had started charging me £1.25 per month. I am not sure if that will increase to £5 at some point, but either way, I am happy to pay to have a "proper" internet connection. I share this as perhaps others will be in similar situations but not realize some ISPs will let you escape the CGNAT. They switched me after a five minute chat and within 2 hours (considerably less but they said up to 2 hours) I restarted my router and I was good to go.
- strictfp 6y agoMaybe they could offer you IPv6 cheaper?
- fsh 6y agoMy ISP in Germany charges 5€/month for a non-GCN IPv4, but static addresses are unobtainium outside expensive business contracts. At least most ISPs have good IPv6 support nowadays, even on mobile.
- cyberdelica 6y agoThanks for this, as I am thinking about switching to Hyperoptic, and am interested in self hosting several services.
- selfhoster11 6y agoHyperoptic is amazing. I cannot recommend them enough. The only annoying thing about them is that not even a dynamic IPv4 is included in the price.
- logarithm69 6y agoJust ping it. Ping it. Ping it Ping it All you got to do is ping it. https://stackoverflow.com/questions/13628517/is-it-possible-to-write-a-script-to-avoid-vpn-from-getting-timeout https://stackoverflow.com/questions/13628517/is-it-possible-...
- unixhero 6y agoTry using Mosh instead.
- knorker 6y agoWait, this person is in Denmark and can't change ISP? If this happened to me then I'd change ISP. NAT on your home Internet? What is this, the US? I thought if you were in Europe you'd pretty much always be able to get public IPv4, IPv6, and IPv6-PD. I know I can.
- Daho0n 6y agoHe can but instead he contacted his ISP and got it fixed, which is much better. I don't know which ISP he is using but sounds like it is on the awful (by Danish standards) TDC network. Good thing there's great fiber almost everywhere.
- msh 6y agoTDC does not by standard use CGNAT, they give out dynamic IPv4.
- XelNika 6y agoCGNAT is standard for all smaller ISPs in Denmark. I'm not sure if it's mainly to stay competitive or due to limited supply, but they offer public IPv4 addresses upon request, sometimes for free. It makes sense really; if you default to CGNAT, which is fine for 99% of users, there will be more public addresses for the people who need them which keeps costs down for both segments. It's not the ISP's fault the standard is outdated.
- BlueTemplar 6y agoI take issue that it's "fine for 99% of users" or that it's not a big deal : https://news.ycombinator.com/item?id=25744675 https://news.ycombinator.com/item?id=25744675
- XelNika 6y agoThat is not a realistic scenario. Firstly, there's no waking up to find your server is unresponsive. It's part of the deal when you sign up. Secondly, public IPs are always an option. Thirdly, the ISPs I have used have always had relevant FAQ/help articles. If any of those points do not apply to your ISP, that's not because of CGNAT, it's just a shit company. I've used four different ISPs in the past four years and getting rid of CGNAT has not been a problem once. IIRC only one of them used public addresses by default, two offered free dynamic IPs upon request and my current ISP offers paid static IPs for $3. Tons of my friends and family have had CGNAT and never known. It's just not a big deal for most content consumers.
- beagle3 6y agoOne should set both ServerAliveInterval and ClientAliveInterval settings on ssh to a reasonably small time (e.g. 1 min or 5 min). Note that there's another setting {Server,Client}AliveCountMax which multiplies this to find the actual "connection is dead" determination time. The tradeoff is between a network disconnect-reconnect killing your connection when it shouldn't (because you've noticed the disconnect but you wouldn't if you didn't send heartbeats), and discovering the network disconnect or dead peer (which you wouldn't if you didn't send anything, thinking it's still there). Personally, I prefer to know the connection is dead sooner than later even if it comes back on its own.
- hienyimba 6y agoI discovered this same issue with my ISP. Solution: I’ve found that simply executing “top” generates enough activity to keep the session alive until you get back to it.
- deleted 6y ago[deleted]
- jscholes 6y ago> This happened one day while I was transferring a VM image in the hundreds of gigabytes from one server to another using netcat like so: > ... > my SSH connections had died yet again. And sshd had taken netcat down with it, killing the transfer midway. As somebody who has experienced the described behaviour, I can empathise because it is super annoying. But this isn't a particular good point with which to illustrate it. If you're doing any sort of work on one or more remote servers that shouldn't be accidentally terminated like a large data transfer, you should be backgrounding it with screen or similar. Even for users without the idel connection dropping as described, the Internet can still fail.