4 ms·

Not gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.

rsstack 6y ago
If something like this happens to an instance, I don't trust the VM's state even after recovery and I'd retire & replace regardless. But I'd love to have serial access to do a root-cause analysis to prevent the issue from occurring in the future.
pickle-wizard 6y ago
Most of the time, that is what I'd suggest to do. However this is like the 4 wheel drive on my truck. I usually don't need it, but when I need it, I really in a bind and glad to have it.
boulos 6y ago
Disclosure: I used to work on GCE (and even helped push our serial console access years ago). That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill. At this point, without interactive serial port access, you get to replace whatever you’ve got on the box with more logging statements. That’s a totally reasonable approach, but with interactive serial ports you can poke at it and root cause a lot faster. Edit: Also, Linux seems to always kill sshd first. (Part of this is survivorship bias, of course).
- herpderperator 6y ago
  I don't understand your last statement. Linux wouldn't kill sshd first unless it somehow was the highest memory consumer. [0] [0] https://unix.stackexchange.com/a/153586 https://unix.stackexchange.com/a/153586
  - boulos 6y ago
    The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause. Amusingly, this finally forced me to find bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1071290 https://bugzilla.redhat.com/show_bug.cgi?id=1071290 (All processes started under a remote shell get adjustment -1000, which is basically never shoot me). There are a few related to setting up the sshd adjustment itself as well. So, looks like a config problem! Thanks for pointing this out.
    - herpderperator 6y ago
      You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this: server ~ # grep oom_score_adj /usr/sbin/sshd grep: /usr/sbin/sshd: binary file matches ...which is fascinating and clever. That's when I checked the source code linked at "[0]". I now finally have an answer as to why I've seen dmesg memory stat dumps display different oom_score_adj values for sshd. I always thought _something_ was smart enough to know that we don't want to risk killing sshd, but I didn't know what that _something_ was. It turns out it was the daemon itself. Oct 13 23:20:38 server kernel: Mem-Info: ... Oct 13 23:20:38 server kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Oct 13 23:20:38 server kernel: [ 2455] 60 2455 1778653 273114 710 10 0 0 mysqld ... Oct 13 23:20:38 server kernel: [12085] 207 12085 22574 673 32 3 0 0 tlsmgr Oct 13 23:20:38 server kernel: [ 4238] 0 4238 9234 518 20 3 0 -1000 systemd-udevd Oct 13 23:20:38 server kernel: [12278] 0 12278 88107 5597 136 4 0 0 apache2 Oct 13 23:20:38 server kernel: [17222] 0 17222 1258983 142035 505 8 0 0 qemu-system-x86 ... Oct 13 23:20:38 server kernel: [21069] 0 21069 5033 487 14 4 0 0 bash Oct 13 23:20:38 server kernel: [15935] 0 15935 7081 487 16 3 0 -1000 sshd ... In retrospect it makes a lot of sense, especially considering sshd runs as root -- it has complete ability to do that. And it's not like anything else would know the importance of sshd except for itself. However I still don't understand your comment about the Linux OOM killer wanting to kill sshd "first" (or _ever_ based on these renewed findings!) Can you elaborate? [0] https://github.com/openssh/openssh-portable/blob/e51dc7fab61df36e43f3bc64b673f88d388cab91/openbsd-compat/port-linux.c#L280 https://github.com/openssh/openssh-portable/blob/e51dc7fab61...
- sargun 6y ago
  I did a lot of kernel development with GCP instances. Having the serial port enabled me to use a remote debugger and made it super great.
saurik 6y ago
It makes debugging the construction of instances way more annoying, particular if you are dealing with any local networking on the machine. The turnaround time for testing something goes from tens seconds to multiple minutes (at best) without a serial console.
rantwasp 6y ago
that’s the right answer. unless you are doing something super-specialized or esoteric you should not touch this. and as a matter of fact if you are doing things properly you also don’t have ssh access. remember? cattle not pets!

rsstack 6y ago

If something like this happens to an instance, I don't trust the VM's state even after recovery and I'd retire & replace regardless. But I'd love to have serial access to do a root-cause analysis to prevent the issue from occurring in the future.

pickle-wizard 6y ago

Most of the time, that is what I'd suggest to do. However this is like the 4 wheel drive on my truck. I usually don't need it, but when I need it, I really in a bind and glad to have it.

boulos 6y ago

Disclosure: I used to work on GCE (and even helped push our serial console access years ago). That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill. At this point, without interactive serial port access, you get to replace whatever you’ve got on the box with more logging statements. That’s a totally reasonable approach, but with interactive serial ports you can poke at it and root cause a lot faster. Edit: Also, Linux seems to always kill sshd first. (Part of this is survivorship bias, of course).

herpderperator 6y ago
I don't understand your last statement. Linux wouldn't kill sshd first unless it somehow was the highest memory consumer. [0] [0] https://unix.stackexchange.com/a/153586 https://unix.stackexchange.com/a/153586
- boulos 6y ago
  The OOM score isn’t strictly ordered by memory usage. The oom score adjustment is usually the cause. Amusingly, this finally forced me to find bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1071290 https://bugzilla.redhat.com/show_bug.cgi?id=1071290 (All processes started under a remote shell get adjustment -1000, which is basically never shoot me). There are a few related to setting up the sshd adjustment itself as well. So, looks like a config problem! Thanks for pointing this out.
  - herpderperator 6y ago
    You're right that it's not _strictly_ memory consumption and that other criteria and overrides exist, but memory consumption is highly weighted. Regarding SSH, if you enable sshd debug logging you can see that sshd sets its own score to the minimum possible [0] which is why your comment about sshd being targeted still doesn't make sense to me. I actually didn't know it was sshd doing this on its own till I ran this: server ~ # grep oom_score_adj /usr/sbin/sshd grep: /usr/sbin/sshd: binary file matches ...which is fascinating and clever. That's when I checked the source code linked at "[0]". I now finally have an answer as to why I've seen dmesg memory stat dumps display different oom_score_adj values for sshd. I always thought _something_ was smart enough to know that we don't want to risk killing sshd, but I didn't know what that _something_ was. It turns out it was the daemon itself. Oct 13 23:20:38 server kernel: Mem-Info: ... Oct 13 23:20:38 server kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Oct 13 23:20:38 server kernel: [ 2455] 60 2455 1778653 273114 710 10 0 0 mysqld ... Oct 13 23:20:38 server kernel: [12085] 207 12085 22574 673 32 3 0 0 tlsmgr Oct 13 23:20:38 server kernel: [ 4238] 0 4238 9234 518 20 3 0 -1000 systemd-udevd Oct 13 23:20:38 server kernel: [12278] 0 12278 88107 5597 136 4 0 0 apache2 Oct 13 23:20:38 server kernel: [17222] 0 17222 1258983 142035 505 8 0 0 qemu-system-x86 ... Oct 13 23:20:38 server kernel: [21069] 0 21069 5033 487 14 4 0 0 bash Oct 13 23:20:38 server kernel: [15935] 0 15935 7081 487 16 3 0 -1000 sshd ... In retrospect it makes a lot of sense, especially considering sshd runs as root -- it has complete ability to do that. And it's not like anything else would know the importance of sshd except for itself. However I still don't understand your comment about the Linux OOM killer wanting to kill sshd "first" (or _ever_ based on these renewed findings!) Can you elaborate? [0] https://github.com/openssh/openssh-portable/blob/e51dc7fab61df36e43f3bc64b673f88d388cab91/openbsd-compat/port-linux.c#L280 https://github.com/openssh/openssh-portable/blob/e51dc7fab61...
sargun 6y ago
I did a lot of kernel development with GCP instances. Having the serial port enabled me to use a remote debugger and made it super great.

saurik 6y ago

It makes debugging the construction of instances way more annoying, particular if you are dealing with any local networking on the machine. The turnaround time for testing something goes from tens seconds to multiple minutes (at best) without a serial console.

rantwasp 6y ago

that’s the right answer. unless you are doing something super-specialized or esoteric you should not touch this. and as a matter of fact if you are doing things properly you also don’t have ssh access. remember? cattle not pets!