10 ms·
EC2 Serial Console
- petethepig 6y agoThis seems to be such an obvious feature that I'm surprised they are only adding this now. I'm not an expert in hypervisors or anything like that and so I'm wondering what was stopping them from adding it in the past?
- cperciva 6y agoThe biggest reason is probably security. This is not something you want to take chances with.
- syncsynchalt 6y agoThat's my take too. They've gotten so far without this functionality that I have to wonder what finally tipped the balance into their offering it.
- pwarner 6y agoI get this sense they are very conservative in their feature set there? Nested VMs for example are supposed in GCP, Azure and Oracle clouds I think, but not AWS. VM migration too I think exists in GCP at least, but not AWS. It's interesting.
- oblio 6y agoAWS is super slow to deprecate or remove features and services. A few people were noting that SimpleDB has been deprecated, it's not listed in the AWS web console, you can't find docs for it anymore, but if you have a running instance, your service API calls still work. And I think there have been many deprecation warnings since, plus migration messages. But they don't want to break existing clients. I'm guessing this is a similar case where they want to be really, really sure that it's worth offering the service.
- ignoramous 6y ago> ...you can't find docs for it anymore Here are the docs: https://aws.amazon.com/simpledb/ https://aws.amazon.com/simpledb/
- oblio 6y agoAh, my bad. But the service is no longer listed in the AWS Console menus.
- chousuke 6y agoLikely they just had no need for it themselves. Might be that they didn't prioritize having a feature over the risk of someone taking over their hypervisors thanks to a buggy serial port emulator. Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and they can't trust the users.
- bigiain 6y agoThey probably don't need it because they are much more likely to follow best practise "treat servers as cattle, not pets". If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me). I'm not sure I'd be at all comfortable having irreplaceable single points of failure in AWS. (Though I do recognise that people use it that way all the time...)
- sargun 6y agoHow do you debug broken instances?
- x3n0ph3n3 6y agoShut them down and mount the volume on a new host.
- somethingAlex 6y agoTypically the goal is to architect the system such that you don't really care. If it's stateful, there's some other replica. Promote that and then spin up a new replica from a backup and roll it forward. If it's stateless then just kill it and spin up another.
- sargun 6y agoBut what if there is a bug that’s repeatable / pops up on the regular?
- amelius 6y agoPerhaps patents in HP's iLO system?
- technion 6y agoLower end VPS providers have offered full console access for years. For example: https://www.linode.com/docs/guides/using-the-linode-shell-lish/ https://www.linode.com/docs/guides/using-the-linode-shell-li...
- monocasa 6y agoOh thank god. I had a customer once that erased their SSH keys, and had a running database cluster on EC2 that they couldn't get access too anymore. That was... fun. This is a long time coming.
- koolba 6y agoSure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.
- monocasa 6y agoThey could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice. There were a whole lot of questionable prior decisions that had been made that did not help. You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.
- growse 6y agoProbably shouldn't have erased their SSH keys then.
- monocasa 6y agoI mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another. "Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).
- koolba 6y agoTo use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones. One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.
- commandlinefan 6y agoNot gonna lie... if I ever got into a situation where I needed serial access to an EC2 instance, I'd just retire the EC2 instance and spin up another one.
- rsstack 6y agoIf something like this happens to an instance, I don't trust the VM's state even after recovery and I'd retire & replace regardless. But I'd love to have serial access to do a root-cause analysis to prevent the issue from occurring in the future.
- pickle-wizard 6y agoMost of the time, that is what I'd suggest to do. However this is like the 4 wheel drive on my truck. I usually don't need it, but when I need it, I really in a bind and glad to have it.
- boulos 6y agoDisclosure: I used to work on GCE (and even helped push our serial console access years ago). That’s a good default posture. What sucks is when you’re trying to debug a system that has OOM-killed sshd and then is behaving generally poorly. If you replace your instance with another one, you just get another OOM kill. At this point, without interactive serial port access, you get to replace whatever you’ve got on the box with more logging statements. That’s a totally reasonable approach, but with interactive serial ports you can poke at it and root cause a lot faster. Edit: Also, Linux seems to always kill sshd first. (Part of this is survivorship bias, of course).
- herpderperator 6y agoI don't understand your last statement. Linux wouldn't kill sshd first unless it somehow was the highest memory consumer. [0] [0] https://unix.stackexchange.com/a/153586 https://unix.stackexchange.com/a/153586
- boulos 6y ago
- sargun 6y agoThis is amazing.
- bombcar 6y agoI wonder if this is implemented at the hypervisor level (gives you access to the "console") or if this is actually implemented as a serial port.
- rwmj 6y agoWhat would be the difference exactly? The console of a Linux VM is either the emulated serial port or the emulated VGA device + emulated keyboard. I once implemented a tool called virt-dmesg which read out the log_buf from a running Linux kernel (surprisingly useful for those tricky crashes, but difficult from a maintenance point view so the tool is now abandoned). I suppose that's the closest you could get to a "real" console at the hypervisor level.
- bombcar 6y agoA "text console" is almost always there but a serial console has to be enabled on the instance itself. Based on a quick SSH it looks like it's a serial thing: root 1221 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 root 1229 /sbin/agetty -o -p -- \u --noclear tty1 linux
- deleted 6y ago[deleted]
- JdeBP 6y ago... or the parallel port. A lot of people forget that one. * http://jdebp.uk./Softwares/nosh/guide/commands/linux-console.xml http://jdebp.uk./Softwares/nosh/guide/commands/linux-console...
- Daviey 6y agoThis is super useful if you are in the game of building images, or highly tweeking init systems and/or the kernel. For general consumers, not much value IMHO.
- whalesalad 6y agoI see a ton of value when you have instances on a private network with no shell access and want to debug them without setting up a bastion instance.
- Spivak 6y agoYep, this might very well enable whole deployments where there is no SSH access anywhere.
- cj 6y agoI manage the AWS infrastructure for a medium sized SaaS company and we have no SSH access to any of our ~20 servers. The only access is through AWS Systems Manager Session Manager. Not managing keys is major upside.
- sciurus 6y agoAWS's (poorly named) SSM Session Manager service already allows that. https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
- whalesalad 6y agoIt requires an agent.
- deleted 6y ago[deleted]
- edoceo 6y ago
- Waterluvian 6y agoIs this the cloud equivalent to hooking up a monitor and keyboard to a server? I remember having an EC2 terminal in the browser years ago and recently I went back and it seemed far more locked down.
- syncsynchalt 6y agoSort of, a few key differences: - usually these are running with very few dependencies in the userspace stack, such as a getty directly spawned by init (or the modern equivalent arrangement). This means that it's accessible even if your networking stack is not working, or if you screw up your firewall config. You can even make sure your init / getty / bash are statically linked so that not even ld.so breakage will stop you. - it looks like they're also enabling Linux's Magic SysRq features, which gives you some very raw hooks into the kernel itself.
- banana_giraffe 6y agoI suppose this isn't much of a surprise, but it's kinda sorta pointless for Windows instances. I was hoping it would be something, but it's just a dump of the standard Windows system log output. And adding to the fun, watching it on an initial instance bootup seems to block the process AWS uses to grab the encrypted password out of the log. So, it's not useful, and makes the instance a bit hard to remote into.
- my123 6y agoYou can enable the SAC for Windows. Then, you can have a command prompt too over serial.
- banana_giraffe 6y agoFair enough, found the documentation: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/troubleshooting-sac.html https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/troub... It does require connecting to the working instance first and setting things up, which is less than ideal, but I get why it's the case. I guess I'm off to configure the Windows instances I care about, so I have a way to troubleshoot things in the future.
- nn3 6y ago"Accessing your serial console is only 0.04$/per character, except if you bid on peak console then ..." I made that up, but it would totally not surprise anyone, would it?
- WrtCdEvrydy 6y agoI did certifications and couldn't believe that it wasn't $5/missed question the first time around :D
- saurik 6y agoI have waited 10 years for this :D. https://forums.aws.amazon.com/thread.jspa?messageID=979374 https://forums.aws.amazon.com/thread.jspa?messageID=979374
- anilshanbhag 6y agoWow. Out of curiosity just checked if the other cloud providers have it - both Azure and GCP have this feature already. Azure got it ~ Feb 2020, GCP Feb 20201.
- boulos 6y agoDisclosure: I used to work on GCE (and was adjacent to the serial port work). IIRC, we launched interactive serial port access sometime in late 2014. For example, mbrukman answered a SO question on Jan 2, 2015 with connect-to-serial-port [1]. I don’t recall when we gained fancier IAM controls for it, but we’ve had it forever (and I think getting / view only was there at public launch). [1] https://stackoverflow.com/questions/27734763/how-do-you-access-the-console-of-a-gce-vm-instance https://stackoverflow.com/questions/27734763/how-do-you-acce...
- MikeHalcrow 6y agoI did the security review of the GCE serial port back when it first came into existence. We probably know each other. The tech industry really is tiny.
- jsolson 6y agoI will almost guarantee you two know each other (as someone who knows both of you from Google :)
- yellowapple 6y agoPictured: three Google engineers happily doxxing each other (c. 2021, colorized)
- waynesonfire 6y agoCloud's catching up. :tada:
- Sebguer 6y agoOh good, I've always wanted to be able to raise the elephants in the cloud...
- llbeansandrice 6y agoWhat's the difference between this and the SSM agent? Can I replace SSM agent configs for this?
- gregdunn 6y agoDisclaimer: I work at AWS, primarily in the compute space, but I'm not speaking in an official capacity. I would say they fulfill different purposes. The SSM agent has quite a bit of additional functionality, even within the Session Manager portion. It's more of your solution for online, general day to day access. Serial console will let you fix issues when you have lost the ability to boot an instance, or network connectivity has failed. When SSH or Session Manager are available, I personally would opt to utilize them over the serial console. But if I have an instance that I can't reach via those, am unable to replace it for whatever reason, and need to bring it back online, serial console would be what I would reach for.
- herpderperator 6y agoWas this really not available before? It is such a basic requirement for virtual machines. Other providers like Linode have had this since 2009 if not earlier.
- digianarchist 6y agoProxmox has had this for quite a while. https://pve.proxmox.com/wiki/Serial_Terminal https://pve.proxmox.com/wiki/Serial_Terminal
- somethingAlex 6y agoI find it interesting that half of the comments indicate that people don't really understand why this would be needed and the other half are surprised that is hasn't been offered before. Personally, I'm in the first camp. I'm used to taking instances for granted on the rare occasion that a low-level issue arises and just promoting a replica or trashing it if it's stateless. I'm assuming the use case where you care about fixing the type of issues this feature helps debug is fairly esoteric?
- jake_morrison 6y agoI would have loved to have that when working on building AMIs for custom Linux systems, e.g. https://github.com/cogini/buildroot_ec2 https://github.com/cogini/buildroot_ec2 and https://github.com/cogini/nerves_system_ec2 https://github.com/cogini/nerves_system_ec2 I spent a lot of time looking at console screenshots of machines that would not boot and iterating to figure out the problem.
- roomey 6y agoAm I missing something? This is for Nitro instances right, not EC2? Nitro is when you get the whole bare metal server and you need to run your own Hypervisor/ OS (which is why they mentioned VMware). This instance hasn't been available to the public very long (like a year or two). Maybe I am missing something here but I think a lot of comments seem to be mis understanding what this is
- fguerraz 6y agoAll modern instances families (regardless of their type) are powered by Nitro.
- deleted 6y ago[deleted]
- gregdunn 6y agoDisclaimer: I work at AWS, primarily in the compute space, but I'm not speaking in an official capacity. As fguerraz mentioned, modern AWS instance families are basically all powered by Nitro, which refers to the ecosystem around the hypervisor and hardware acceleration cards utilized. https://aws.amazon.com/ec2/nitro/ https://aws.amazon.com/ec2/nitro/
- roomey 6y agoThank you very much! I thought nitro referred to the bare metal offering only.
- Simulacra 6y agoIt's about time. Wow.