4 ms·

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.

growse 6y ago
Probably shouldn't have erased their SSH keys then.
- monocasa 6y ago
  I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another. "Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).
koolba 6y ago
To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.
- monocasa 6y ago
  Is that strictly true? Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.
  - wahern 6y ago
    Depends on how getty (or other local TTY manager) is setup. Most (if not all) Linux and BSD distributions attach login(1) to console and serial TTYs by default, and login(1) will normally require password authentication. Though, whether login prompts for a password is (I think) also a function of flags in /etc/passwd (or /etc/shadow or equivalent), so you may be able to login as `root` or `$USER` without entering a password if the account was setup that way.
    - tingletech 6y ago
      if you have serial console access, can't you key-combo it to boot into single user mode?
      - wahern 6y ago
        Not by default as far as I'm aware. Though, it's been a long time since I've used a desktop Linux distribution. My experience is mostly with serial access for servers--Linux, BSD, and Solaris. I much prefer serial ports for backup administrative access as there's little chance of a misconfiguration. Whereas w/ port failover and other IPMI, BMC, etc non-sense if you disconnect the ethernet cable from the dedicated network management port you may still have admin/admin access live on the network.
        tingletech 6y ago
        once upon a time in another life it seems like, we had a modem hooked to a serial multiplexer in the "core room" in the library, and I swear once I used that to reboot a solaris machine to single user mode, but I probably telnet'ed in (this was before ssh) and then did the init 1 or whatever and then dialed in to the serial multiplexer.
        wahern 6y ago
        I once had a second-hand Sun Sparc system, and several Sun amd64 systems (discounted for startups), and I remember Sun having a fairly sophisticated serial-attached firmware console. Though a fail-open default configuration that permitted trivially bypassing authentication seems more like something one would see w/ an x86 BIOS or bootloader. I wouldn't claim the serial-attached bootloaders on my x86 lab machines don't somehow permit bypassing login(1) authentication. But at least w/ EC2 and other hypervisors that element should be out of the equation.
- steffan 6y ago
  My understanding from the post was that it requires a system root account, not ssh. The former implies the latter, but you could (and usually should) disable root ssh access.
petercooper 6y ago
You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around. I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.

growse 6y ago

Probably shouldn't have erased their SSH keys then.

monocasa 6y ago
I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another. "Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).

koolba 6y ago

To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.

monocasa 6y ago
Is that strictly true? Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.
- wahern 6y ago
  Depends on how getty (or other local TTY manager) is setup. Most (if not all) Linux and BSD distributions attach login(1) to console and serial TTYs by default, and login(1) will normally require password authentication. Though, whether login prompts for a password is (I think) also a function of flags in /etc/passwd (or /etc/shadow or equivalent), so you may be able to login as `root` or `$USER` without entering a password if the account was setup that way.
  - tingletech 6y ago
    if you have serial console access, can't you key-combo it to boot into single user mode?
    - wahern 6y ago
      Not by default as far as I'm aware. Though, it's been a long time since I've used a desktop Linux distribution. My experience is mostly with serial access for servers--Linux, BSD, and Solaris. I much prefer serial ports for backup administrative access as there's little chance of a misconfiguration. Whereas w/ port failover and other IPMI, BMC, etc non-sense if you disconnect the ethernet cable from the dedicated network management port you may still have admin/admin access live on the network.
      - tingletech 6y ago
        once upon a time in another life it seems like, we had a modem hooked to a serial multiplexer in the "core room" in the library, and I swear once I used that to reboot a solaris machine to single user mode, but I probably telnet'ed in (this was before ssh) and then did the init 1 or whatever and then dialed in to the serial multiplexer.
        wahern 6y ago
        I once had a second-hand Sun Sparc system, and several Sun amd64 systems (discounted for startups), and I remember Sun having a fairly sophisticated serial-attached firmware console. Though a fail-open default configuration that permitted trivially bypassing authentication seems more like something one would see w/ an x86 BIOS or bootloader. I wouldn't claim the serial-attached bootloaders on my x86 lab machines don't somehow permit bypassing login(1) authentication. But at least w/ EC2 and other hypervisors that element should be out of the equation.
steffan 6y ago
My understanding from the post was that it requires a system root account, not ssh. The former implies the latter, but you could (and usually should) disable root ssh access.

petercooper 6y ago

You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around. I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.