3 ms·

Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.

by koolba 6y ago

monocasa 6y ago
They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.
- growse 6y ago
  Probably shouldn't have erased their SSH keys then.
  - monocasa 6y ago
    I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another. "Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).
- koolba 6y ago
  To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.
  - monocasa 6y ago
    Is that strictly true? Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.
    - wahern 6y ago
      Depends on how getty (or other local TTY manager) is setup. Most (if not all) Linux and BSD distributions attach login(1) to console and serial TTYs by default, and login(1) will normally require password authentication. Though, whether login prompts for a password is (I think) also a function of flags in /etc/passwd (or /etc/shadow or equivalent), so you may be able to login as `root` or `$USER` without entering a password if the account was setup that way.
      - tingletech 6y ago
        if you have serial console access, can't you key-combo it to boot into single user mode?
        wahern 6y ago
        Not by default as far as I'm aware. Though, it's been a long time since I've used a desktop Linux distribution. My experience is mostly with serial access for servers--Linux, BSD, and Solaris. I much prefer serial ports for backup administrative access as there's little chance of a misconfiguration. Whereas w/ port failover and other IPMI, BMC, etc non-sense if you disconnect the ethernet cable from the dedicated network management port you may still have admin/admin access live on the network.
        tingletech 6y ago
        once upon a time in another life it seems like, we had a modem hooked to a serial multiplexer in the "core room" in the library, and I swear once I used that to reboot a solaris machine to single user mode, but I probably telnet'ed in (this was before ssh) and then did the init 1 or whatever and then dialed in to the serial multiplexer.
- petercooper 6y ago
  You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around. I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.
commandlinefan 6y ago
That's what I was thinking... I mean, this is cool and all, but the only reason I'd ever muck around at this level is if I was actually responsible for the physical hardware and there was a cost associated with replacing it. The whole point of moving to the cloud is to pay somebody else to have to worry about anything I can comprehend needing this for.
- monocasa 6y ago
  A lot of medium sized companies moved to the cloud without understanding the tradeoffs. And they have big pockets when it matters.
  - bombcar 6y ago
    I suspect the percentage of “cloud servers” that are just standard Linux boxes running relatively standard applications is quite high.
- toast0 6y ago
  The FreeBSD kernel debugger is available via serial console and is really useful when it's useful. If you manage to have a system that breaks your kernel, chances are switching instances won't help, and figuring it out will help. If it crashes, you can get a core dump, but if it loops in a bad place and that only breaks some things, the debugger can show things quickly. Also serial console is pretty handy when you push bad firewall rules and don't want to throw away your instance.
saurik 6y ago
So, a lot of database systems aren't using EBS for many reasons: that's why the local I/O instances exist. And so, yes: you don't build out like that unless but you are prepared to just scrap the instance and rebuild it from WAL logs or have replication or whatever, but knowing "this is a 30 second fix with a serial console" really makes taking all of that cost in stride painful.

Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.

by koolba 6y ago

Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.

growse 6y ago
Probably shouldn't have erased their SSH keys then.
- monocasa 6y ago
  I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another. "Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).
koolba 6y ago
To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.
- monocasa 6y ago
  Is that strictly true? Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.
  - wahern 6y ago
    Depends on how getty (or other local TTY manager) is setup. Most (if not all) Linux and BSD distributions attach login(1) to console and serial TTYs by default, and login(1) will normally require password authentication. Though, whether login prompts for a password is (I think) also a function of flags in /etc/passwd (or /etc/shadow or equivalent), so you may be able to login as `root` or `$USER` without entering a password if the account was setup that way.
    - tingletech 6y ago
      if you have serial console access, can't you key-combo it to boot into single user mode?
      - wahern 6y ago
        Not by default as far as I'm aware. Though, it's been a long time since I've used a desktop Linux distribution. My experience is mostly with serial access for servers--Linux, BSD, and Solaris. I much prefer serial ports for backup administrative access as there's little chance of a misconfiguration. Whereas w/ port failover and other IPMI, BMC, etc non-sense if you disconnect the ethernet cable from the dedicated network management port you may still have admin/admin access live on the network.
        tingletech 6y ago
        once upon a time in another life it seems like, we had a modem hooked to a serial multiplexer in the "core room" in the library, and I swear once I used that to reboot a solaris machine to single user mode, but I probably telnet'ed in (this was before ssh) and then did the init 1 or whatever and then dialed in to the serial multiplexer.
petercooper 6y ago
You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around. I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.

commandlinefan 6y ago

That's what I was thinking... I mean, this is cool and all, but the only reason I'd ever muck around at this level is if I was actually responsible for the physical hardware and there was a cost associated with replacing it. The whole point of moving to the cloud is to pay somebody else to have to worry about anything I can comprehend needing this for.

monocasa 6y ago
A lot of medium sized companies moved to the cloud without understanding the tradeoffs. And they have big pockets when it matters.
- bombcar 6y ago
  I suspect the percentage of “cloud servers” that are just standard Linux boxes running relatively standard applications is quite high.
toast0 6y ago
The FreeBSD kernel debugger is available via serial console and is really useful when it's useful. If you manage to have a system that breaks your kernel, chances are switching instances won't help, and figuring it out will help. If it crashes, you can get a core dump, but if it loops in a bad place and that only breaks some things, the debugger can show things quickly. Also serial console is pretty handy when you push bad firewall rules and don't want to throw away your instance.

saurik 6y ago

So, a lot of database systems aren't using EBS for many reasons: that's why the local I/O instances exist. And so, yes: you don't build out like that unless but you are prepared to just scrap the instance and rebuild it from WAL logs or have replication or whatever, but knowing "this is a 30 second fix with a serial console" really makes taking all of that cost in stride painful.