3 ms·

I have a couple of FreeBSD VMs on Vultr. For quite some time sshd regularly dies on one of them and I am not sure why. A couple of theories I’ve had is that ma

I have a couple of FreeBSD VMs on Vultr. For quite some time sshd regularly dies on one of them and I am not sure why. A couple of theories I’ve had is that maybe a) my VM was compromised and there is a persistent rootkit installed that kills sshd, or b) file corruption after previous unclean shutdown has left some file needed by sshd corrupted and it leads to this behaviour, or c) maybe it’s running out of memory sometimes Each time I want to ssh into the machine I usually have to first connect with the VNC from the vultr dashboard to start sshd up again. It’s running the latest FreeBSD, as every now and then I log in and do an upgrade on it some time after a new version has been released. A persistent rootkit may have been installed if it was compromised between when some vulnerability became known and when I later upgraded next time. If a file was corrupted in an unclean shutdown in the past maybe it’s a file that has not been changed between FreeBSD versions so even though upgrades replace some files maybe it’s the same corrupted file all along. Ideally I’d just reinstall the machine, but that’s always more of a hassle than it should be so I continue running the VM in this broken state where sshd keeps dying every now and then.

cyberpunk 2mo ago
Well, you can at least run "freebsd-update IDS" on this system to verify the base system, "pkg check -s -a" would check the integrity of files installed via pkg also. It will only take a few min..
- QuantumNomad_ 2mo ago
  Neat! Haven’t tried either of those before. While I’m at it I also took a quick look now at output of `top` and it’s sitting at 27 MB free RAM lol. So from that, out of memory is very likely the reason I keep having sshd die on me.
  - silisili 2mo ago
    Not familiar with BSD but does it not syslog oom kills like linux?
    - QuantumNomad_ 2mo ago
      After rebooting the VM now, sshd died even when there was hundreds of megabytes free RAM available. So it seems I spoke too soon when I said it seemed to be for that reason. Previously I haven't seen much detailed reason for why it dies in system messages. But this time it said something very specific: > sshd[2036]: fatal: pack_hostkeys: serialize hostkey private: string is too large Which kind of sounds like one of the sshd hostkey files might be corrupt? And maybe it only triggers after a while becuase it happens when scanners try to connect to it and during ssh negotiation sshd ends up selecting a different hostkey type than the one it uses when I connect to the machine myself? I'm going to regenerate all of the three hostkey files on the server, and after that also disable the two that I can do without anyway.
  - ilikecode 2mo ago
    Try adding this to your /etc/rc.conf: sshd_oomprotect=YES Then run service sshd restart
    - deleted 2mo ago
      [deleted]
toyg 2mo ago
I have run OpenBSD on Vultr for probably a decade at this point, and never seen that behaviour.
faded242 2mo ago
Sounds like a you problem I've had multiple FreeBSD systems on vultr for over a decade and never had sshd issues. Of course I also use an alternate port and active fail2ban to block those who insist on any brute force for any protocol. 128K active blocks currently.

cyberpunk 2mo ago

Well, you can at least run "freebsd-update IDS" on this system to verify the base system, "pkg check -s -a" would check the integrity of files installed via pkg also. It will only take a few min..

QuantumNomad_ 2mo ago
Neat! Haven’t tried either of those before. While I’m at it I also took a quick look now at output of `top` and it’s sitting at 27 MB free RAM lol. So from that, out of memory is very likely the reason I keep having sshd die on me.
- silisili 2mo ago
  Not familiar with BSD but does it not syslog oom kills like linux?
  - QuantumNomad_ 2mo ago
    After rebooting the VM now, sshd died even when there was hundreds of megabytes free RAM available. So it seems I spoke too soon when I said it seemed to be for that reason. Previously I haven't seen much detailed reason for why it dies in system messages. But this time it said something very specific: > sshd[2036]: fatal: pack_hostkeys: serialize hostkey private: string is too large Which kind of sounds like one of the sshd hostkey files might be corrupt? And maybe it only triggers after a while becuase it happens when scanners try to connect to it and during ssh negotiation sshd ends up selecting a different hostkey type than the one it uses when I connect to the machine myself? I'm going to regenerate all of the three hostkey files on the server, and after that also disable the two that I can do without anyway.
- ilikecode 2mo ago
  Try adding this to your /etc/rc.conf: sshd_oomprotect=YES Then run service sshd restart
  - deleted 2mo ago
    [deleted]

toyg 2mo ago

I have run OpenBSD on Vultr for probably a decade at this point, and never seen that behaviour.

faded242 2mo ago

Sounds like a you problem I've had multiple FreeBSD systems on vultr for over a decade and never had sshd issues. Of course I also use an alternate port and active fail2ban to block those who insist on any brute force for any protocol. 128K active blocks currently.