3 ms·

If you have a large amount of memory, are doing decent amounts of IO, and linux OOMs, the system becomes unresponsive for many minutes before killing any proces

If you have a large amount of memory, are doing decent amounts of IO, and linux OOMs, the system becomes unresponsive for many minutes before killing any process. At which point, ssh sessions timeout endlessly. A serial console stands a chance. There is then also any case where you're debugging AMI builds and need to fix grub, or the init system without waiting 20 minutes for a new AMI build each time. Also, the existing console log feature in AWS is insultingly not real time. It doesn't typically update at all unless you're within minutes of boot or trigger a reboot and it only buffers something like 4kb so a reboot can easily fully replace the logs. This really sucks when you're trying to get the debug console output, so this feature finally solves that.

herpderperator 6y ago
Why would linux trigger OOM if you have a large amount of memory available? Or, what did you mean by "large amount of memory"? Also, why would an SSH session, which is entirely in memory, time out because of I/O thrashing? You can disconnect the hard drive that sshd and/or the OS is running from and your SSH connections to that machine won't break. If you run some commands that aren't cached in memory you'll naturally get critical I/O errors, but it won't cause a disconnect on the SSH layer.
- paulfurtado 6y ago
  By large amount of memory, I meant systems that have large amounts of memory that is nearly full. For example, a server with 256GB of memory and 255.9GB in use. SSH is purely in memory, however, in order to allocate memory for it, linux will pull "free" memory out of whatever heavily fragmented corners it can find them in. And, it may even need to perform disk I/O to free memory that was tied up in various disk caches. People refer to this as a "livelock", where Linux is going crazy doing lots of stuff but from userspace the system is completely frozen. Facebook developed OOMD, a userspace oom killer to deal with this issue, their release blog post references the 30 minute livelocks they face: https://engineering.fb.com/2018/07/19/production-engineering/oomd/ https://engineering.fb.com/2018/07/19/production-engineering... They actually have gone so far as to submit kernel patches for newer PSI (pressure stall information) interfaces which they use in oomd to better detect stalls due to this thrashing

herpderperator 6y ago

Why would linux trigger OOM if you have a large amount of memory available? Or, what did you mean by "large amount of memory"? Also, why would an SSH session, which is entirely in memory, time out because of I/O thrashing? You can disconnect the hard drive that sshd and/or the OS is running from and your SSH connections to that machine won't break. If you run some commands that aren't cached in memory you'll naturally get critical I/O errors, but it won't cause a disconnect on the SSH layer.

paulfurtado 6y ago
By large amount of memory, I meant systems that have large amounts of memory that is nearly full. For example, a server with 256GB of memory and 255.9GB in use. SSH is purely in memory, however, in order to allocate memory for it, linux will pull "free" memory out of whatever heavily fragmented corners it can find them in. And, it may even need to perform disk I/O to free memory that was tied up in various disk caches. People refer to this as a "livelock", where Linux is going crazy doing lots of stuff but from userspace the system is completely frozen. Facebook developed OOMD, a userspace oom killer to deal with this issue, their release blog post references the 30 minute livelocks they face: https://engineering.fb.com/2018/07/19/production-engineering/oomd/ https://engineering.fb.com/2018/07/19/production-engineering... They actually have gone so far as to submit kernel patches for newer PSI (pressure stall information) interfaces which they use in oomd to better detect stalls due to this thrashing