4 ms·
> Due to some bizarre breakage in EC2 — which I've been complaining about for ten years — the serial console is very "laggy". If you find that you're not gettin
by zdw 5y ago
> Due to some bizarre breakage in EC2 — which I've been complaining about for ten years — the serial console is very "laggy". If you find that you're not getting any output, wait five minutes and try again.
This is the single most frustrating bug with EC2 I've experienced. In my case, I have Jenkins spinning up temporary executors in EC2, then getting the SSH host key from serial log to verify the connection (I appreciate the "importance of SSH host keys compared with flossing" comparison in this article...)
Machines take <1m to boot, then sit there for 4-8m just waiting for the log message to show up.
- paulddraper 5y agoYou should add the fingerprint as an EC2 tag or something from a boot script.
- kelnos 5y agoDoesn't that require giving the instance elevated privileges around setting tags? Some people might not want that.
- paulddraper 5y agoIt does, though you can limit it to certain tag name like "SSH Fingerprint". And if course you need the AWS CLI installed (which AWS built AMIs have). All in all seems fine for a Jenkins worker
- cperciva 5y agoTwo problems with doing this: 1. If you provide an EC2 Role which allows setting that tag, any process on the instance which can access the Instance Metadata Store can record a different SSH fingerprint. 2. You can only have one EC2 Role attached at once, so doing this prevents you from using other roles.
- klohto 5y ago1. You can limit access to IMS to certain processes inside linux 2. Just combine the policies If you’re worried about malicious processes on the EC2 altering the fingerprint, you got bigger issues. You can also sign the fingerprint with gpg.
- jen20 5y agoIt’s worse than that with respect to (1): any process on _any_ instance which has tag setting permissions can set the tag for _any other_ instance, since conditions don’t support scoping to instance ID. Re (2) technically you cannot have any roles attached directly, but instead attach an instance profile (the distinction is clearer via the API than the console). The shape of the API for an instance profile clearly was designed to support multiple roles, but in practice is limited to one. It’s typical to create a role with many policies attached for each functional type of instance, so in practice it does’t matter too much.
- klohto 5y ago1) CreateTags does support scoping using StringEquals with ec2:SourceInstanceARN and comparing that with the aws:arn "aws:ARN": "${ec2:SourceInstanceARN}" It will give you a warning when creating through web console, but the condition works.
- jen20 5y agoTIL - thanks for the heads up. Is this a documented substitution?
- klohto 5y agoI don't think so... I have been trying to find it for a few years now, but it just won't turn up anywhere. FYI, I personally use resource tags as well on top this, just to limit the scope (in case it magically disappears one day).
- paulddraper 5y ago
- oppositelock 5y agoThere's a new console feature which addresses this, but yes, the previous incarnation was very frustrating. I'd like to see this on all instance types. https://aws.amazon.com/about-aws/whats-new/2021/03/introducing-ec2-serial-console/ https://aws.amazon.com/about-aws/whats-new/2021/03/introduci...
- znpy 5y agoYou might want to look at SSH certificate authority and stuff like that. No more checking SSH host keys, validation is automatic and stuff only fail if something is wrong.