7 ms·

AWS Session Manager: less infrastructure, more features

199 points by jon918 7y ago

jon918 7y ago
I'd love to learn how you're using Session Manager or what other features/integrations you'd like to see us explore. Also if the terraform module packaging is useful. There are additional Session Manager features like port forwarding that I plan to write about soon.
- sandGorgon 7y ago
  1. does it work with hardware tokens ? we have some regulatory requirements that require us to use hardware tokens for 2FA access to servers. 2. what about SSH tunnels ?
  - jon918 7y ago
    It does work with hardware tokens, IF you get your AWS IAM credentials using a hardware token. If you're using AWS IAM users then here are instructions: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... If you're doing a federated login with Okta or another provider, you need to set up the hardware MFA there. There is SSH tunneling support as well, will add an update on that soon.
    - nubs 7y ago
      Our organization recently looked into AWS Session Manager for tunneling but couldn't find documentation on how to make it work for our usecase. We were trying to tunnel into our VPC in order to be able to connect to an Amazon DocumentDB cluster. We don't have any EC2 instances which seems to be the only thing Session Manager has support for. Despite the callouts that Session Manager replaces bastion servers, that didn't seem to be the case for us. Did we miss something in our research?
      - russellendicott 7y ago
        Last I checked the "tunneling" only works to redirect traffic to a different port on the same SSM managed instance. The tunnel cannot be established with another box in the same VPC. So I don't think you can call it tunneling until they add that feature. Here's the GitHub issue where they discuss the limitation and a workaround: https://github.com/aws/amazon-ssm-agent/issues/208 https://github.com/aws/amazon-ssm-agent/issues/208
- skb4 7y ago
  Can you write one about port forwarding? Specifically, I would like to understand how various web interfaces on EMR cluster can be accessed through Sessions Manager. (Ganglia, Spark history server, etc.)
  - jon918 7y ago
    Cool, will do!
  - mullingitover 7y ago
    We'd love to use Session Manager, but we're running into the same issue mentioned here: "Tunnel created using SSM only allows single connection to destination port" - https://forums.aws.amazon.com/thread.jspa?threadID=314882&tstart=0 https://forums.aws.amazon.com/thread.jspa?threadID=314882&ts... This has been sitting open in the support forums unanswered for over two months :/
- klohto 7y ago
  I have bunch of questions that stop me from deploying SSM into real world production scenarios. 1) Is logging for access from CLI finally supported? 2) Can I setup which shell is used? 3) Are logs readable when I switch to something else than sh? 4) Is U2F supported (awscli question) Once all of these are fixed, then it can be possible to claim that SSM solves these issues. Otherwise it’s nothing more than for adhoc usage.
derefr 7y ago
Are they basically trying to emulate GCP’s OS Login (https://cloud.google.com/compute/docs/instances/managing-instance-access https://cloud.google.com/compute/docs/instances/managing-ins...) feature here? We’ve been using that for a while, and it’s been a big relief.
- jon918 7y ago
  Yeah, this is the same deal. Session Manager will log your sessions which is pretty cool.
- idunno246 7y ago
  os login is probably a little closer to ec2 instance connect because you still need ssh inbound access right? whereas aws provides a bastion here
  - aPoCoMiLogin 7y ago
    There is a proxy called IAP [1] which is used to create SSH tunnel over HTTPS to instances without public IP. [1] https://cloud.google.com/compute/docs/instances/connecting-advanced#cloud_iap https://cloud.google.com/compute/docs/instances/connecting-a...
  - WaxProlix 7y ago
    You're right in a sense, but there's no aws-managed bastion. Session manager communicates with your instance via an outbound-created websocket connection. Inputs and outputs are piped through it.
    - idunno246 7y ago
      yea, i was trying to keep things simplified, but it has to proxy through something behind the vpc endpoint. could also say its not technically ssh
      - zokier 7y ago
        Though you can actually get ssh through ssm: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
yasyfm 7y ago
This is awesome! How can I install the the agent if I'm not using Amazon Linux?
- gamache 7y ago
  Amazon installs it on some other AMIs (notably, Ubuntu 16.04 and 18.04), but for other OSes, install instructions are here: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
- WaxProlix 7y ago
  Note that you won't get this feature at the free tier for non-EC2 instances.
jcims 7y ago
IAM is easy to mess up. Would be interesting to lock down the session manager agent (if possible) so that the only way to privileged access is through sudo-like priv esc that uses 2fa.
- jon918 7y ago
  You can do this but it depends on your setup as to how. If you have AWS IAM users (not federated), then you can use MFA conditions in your policies as documented here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_configure-api-require.html#MFAProtectedAPI-overview https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... For federation you need to rely on the config in whatever your identity provider is, like Okta.
- NikolaeVarius 7y ago
  Its fairly trivial to lock down AWS via a require MFA policy
  - jcims 7y ago
    I'm talking about on the host, so if you mess up your IAM policy there is still an authorization layer on the host to get privileged access.
    - jbergknoff 7y ago
      As far as I know, SSH over SSM doesn't do anything regarding user management. It just establishes an SSH connection. Management of users on the host, authorized SSH keys, etc. is totally out of scope for SSM. So if you already have access control setup on your host, then SSM doesn't do anything to undermine it. If you don't have it, you'll still need to add it.
peterwwillis 7y ago
It's great for managing active SSH sessions, but not so much for the other purpose for bastions: fine-grained network access control+routing. It would be cool if they made a more specific version of this just for network traffic without the SSH component.
- mdaniel 7y ago
  FWIW, the project is open source, so you could build a modified agent for your purposes and inject it via cloud-init or your favorite config management tool: https://github.com/aws/amazon-ssm-agent https://github.com/aws/amazon-ssm-agent
gregmac 7y ago
I never see mention of Windows with Session Manager. I have a mixed infrastructure with a number of Windows (IIS) app servers running various things. We currently connect via SSH to a bastionhost, then tunnel from there to various systems, which allows connecting to SSH (linux instances), RDP (Windows), or basically any other network services like Redis or a database. I ended up writing some scripts to automate all this, so as long as you have the right certificates and IAM permissions, you can connect with a single command -- for Windows instances, it even retrieves the randomized password from the EC2 API. The end result is for any EC2 instances you're instantly popped into a shell/RDP session without having to enter credentials. I'd love to replace this with something better (eg Session Manager), but I've not seen how to do this for RDP, and haven't had the time to go experimenting on my own to see if it's even possible. If I can't 100% replace the bastionhosts, having two entirely different connection methods doesn't solve anything (and in fact makes it worse, because it's harder to use).
- gregoryl 7y ago
  Have a google, I was using SSM for remote access to Windows instances, specifically headless instances.
feydaykyn 7y ago
Does anyone know of it works with Ansible ? Thanks!
mishappen 7y ago
Be careful with SSM in general. The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances you want to access via Session Manager. This role grants read/write to all S3 buckets in your account (amongst other things). See this article for better steps and unavoidable risky things: https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/
- jon918 7y ago
  Good call to watch out for this stuff. The examples in the repo we set up use the AmazonSSMManagedInstanceCore managed policy, which does not grant any S3 permissions, just various ssm, ssmmessages, and ec2messages permissions.
- jcrites 7y ago
  > The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances Which documentation do you mean? The article mentions the policy AmazonSSMManagedInstanceCore, which is the same as what's mentioned in the SSM setup guide: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
  - x3n0ph3n3 7y ago
    AmazonSSMManagedInstanceCore is still too much access, it has unscoped ssm:GetParameter! I hope you weren't trying to protect any secrets in ParameterStore!
    - acdha 7y ago
      Doesn’t that still fail if you don’t also have the corresponding decrypt KMS permission?
  - mishappen 7y ago
    Thanks for clarifying, I didn’t recheck since we rolled out SSM in mid-2019 and then scrambled when we realised we’d granted account wide S3 permissions. The article I linked to also has a recommended minimal IAM policy for Run Command and SSM. I’ll update my comment to mention this.
jadell 7y ago
Does anyone know how this works with other utils that use SSH protocol, like rsync? What about tunneling other services to or from a local host? I'd love to have fewer hosts to maintain and a smaller network/attack surface, but we use SSH for more than just gaining commandline access to our instances.
- WaxProlix 7y ago
  It does. https://aws.amazon.com/about-aws/whats-new/2019/07/session-manager-launches-tunneling-support-for-ssh-and-scp/ https://aws.amazon.com/about-aws/whats-new/2019/07/session-m...
bogomipz 7y ago
The author states: >"No more bastion hosts required! Session Manager uses AWS APIs to communicate with your instances, so you can remove the administrative burden of maintaining bastion hosts." Does this presume the EC2 instances have a public IP or is there a way this would also work with EC2 instances on private subnets?
- jhinds 7y ago
  We've been using the Session Manager with instances in private subnets without issue, works like a charm.
  - bogomipz 7y ago
    Is there anything special that needs to be configured to get this to work on private subnets? Currently I have an EKS cluster accessible only on private subnets. It would be wonderful to to be able to access this without OpenVPN in the mix.
    - exidy 7y ago
      The instances establish an outbound connection to the API SSM API, so as long as they can hit that, Session Manager will work. Connectivity from a private subnet to the AWS API could be (a) NAT gateway (b) HTTP proxy (c) PrivateLink VPC endpoint.
jon918 7y ago
I wrote a follow up post to this on SSH tunneling: https://news.ycombinator.com/item?id=22665037 https://news.ycombinator.com/item?id=22665037
shurco 7y ago
Hey, what about the Werbot solution - werbot.com? Now it is very relevant.

199 points by jon918 7y ago

I'd love to learn how you're using Session Manager or what other features/integrations you'd like to see us explore. Also if the terraform module packaging is useful. There are additional Session Manager features like port forwarding that I plan to write about soon.

sandGorgon 7y ago
1. does it work with hardware tokens ? we have some regulatory requirements that require us to use hardware tokens for 2FA access to servers. 2. what about SSH tunnels ?
- jon918 7y ago
  It does work with hardware tokens, IF you get your AWS IAM credentials using a hardware token. If you're using AWS IAM users then here are instructions: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... If you're doing a federated login with Okta or another provider, you need to set up the hardware MFA there. There is SSH tunneling support as well, will add an update on that soon.
  - nubs 7y ago
    Our organization recently looked into AWS Session Manager for tunneling but couldn't find documentation on how to make it work for our usecase. We were trying to tunnel into our VPC in order to be able to connect to an Amazon DocumentDB cluster. We don't have any EC2 instances which seems to be the only thing Session Manager has support for. Despite the callouts that Session Manager replaces bastion servers, that didn't seem to be the case for us. Did we miss something in our research?
    - russellendicott 7y ago
      Last I checked the "tunneling" only works to redirect traffic to a different port on the same SSM managed instance. The tunnel cannot be established with another box in the same VPC. So I don't think you can call it tunneling until they add that feature. Here's the GitHub issue where they discuss the limitation and a workaround: https://github.com/aws/amazon-ssm-agent/issues/208 https://github.com/aws/amazon-ssm-agent/issues/208
skb4 7y ago
Can you write one about port forwarding? Specifically, I would like to understand how various web interfaces on EMR cluster can be accessed through Sessions Manager. (Ganglia, Spark history server, etc.)
- jon918 7y ago
  Cool, will do!
- mullingitover 7y ago
  We'd love to use Session Manager, but we're running into the same issue mentioned here: "Tunnel created using SSM only allows single connection to destination port" - https://forums.aws.amazon.com/thread.jspa?threadID=314882&tstart=0 https://forums.aws.amazon.com/thread.jspa?threadID=314882&ts... This has been sitting open in the support forums unanswered for over two months :/
klohto 7y ago
I have bunch of questions that stop me from deploying SSM into real world production scenarios. 1) Is logging for access from CLI finally supported? 2) Can I setup which shell is used? 3) Are logs readable when I switch to something else than sh? 4) Is U2F supported (awscli question) Once all of these are fixed, then it can be possible to claim that SSM solves these issues. Otherwise it’s nothing more than for adhoc usage.

derefr 7y ago

Are they basically trying to emulate GCP’s OS Login (https://cloud.google.com/compute/docs/instances/managing-instance-access https://cloud.google.com/compute/docs/instances/managing-ins...) feature here? We’ve been using that for a while, and it’s been a big relief.

jon918 7y ago
Yeah, this is the same deal. Session Manager will log your sessions which is pretty cool.
idunno246 7y ago
os login is probably a little closer to ec2 instance connect because you still need ssh inbound access right? whereas aws provides a bastion here
- aPoCoMiLogin 7y ago
  There is a proxy called IAP [1] which is used to create SSH tunnel over HTTPS to instances without public IP. [1] https://cloud.google.com/compute/docs/instances/connecting-advanced#cloud_iap https://cloud.google.com/compute/docs/instances/connecting-a...
- WaxProlix 7y ago
  You're right in a sense, but there's no aws-managed bastion. Session manager communicates with your instance via an outbound-created websocket connection. Inputs and outputs are piped through it.
  - idunno246 7y ago
    yea, i was trying to keep things simplified, but it has to proxy through something behind the vpc endpoint. could also say its not technically ssh
    - zokier 7y ago
      Though you can actually get ssh through ssm: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html https://docs.aws.amazon.com/systems-manager/latest/userguide...

yasyfm 7y ago

This is awesome! How can I install the the agent if I'm not using Amazon Linux?

gamache 7y ago
Amazon installs it on some other AMIs (notably, Ubuntu 16.04 and 18.04), but for other OSes, install instructions are here: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
WaxProlix 7y ago
Note that you won't get this feature at the free tier for non-EC2 instances.

jcims 7y ago

IAM is easy to mess up. Would be interesting to lock down the session manager agent (if possible) so that the only way to privileged access is through sudo-like priv esc that uses 2fa.

jon918 7y ago
You can do this but it depends on your setup as to how. If you have AWS IAM users (not federated), then you can use MFA conditions in your policies as documented here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_configure-api-require.html#MFAProtectedAPI-overview https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credenti... For federation you need to rely on the config in whatever your identity provider is, like Okta.
NikolaeVarius 7y ago
Its fairly trivial to lock down AWS via a require MFA policy
- jcims 7y ago
  I'm talking about on the host, so if you mess up your IAM policy there is still an authorization layer on the host to get privileged access.
  - jbergknoff 7y ago
    As far as I know, SSH over SSM doesn't do anything regarding user management. It just establishes an SSH connection. Management of users on the host, authorized SSH keys, etc. is totally out of scope for SSM. So if you already have access control setup on your host, then SSM doesn't do anything to undermine it. If you don't have it, you'll still need to add it.

peterwwillis 7y ago

It's great for managing active SSH sessions, but not so much for the other purpose for bastions: fine-grained network access control+routing. It would be cool if they made a more specific version of this just for network traffic without the SSH component.

mdaniel 7y ago
FWIW, the project is open source, so you could build a modified agent for your purposes and inject it via cloud-init or your favorite config management tool: https://github.com/aws/amazon-ssm-agent https://github.com/aws/amazon-ssm-agent

gregmac 7y ago

I never see mention of Windows with Session Manager. I have a mixed infrastructure with a number of Windows (IIS) app servers running various things. We currently connect via SSH to a bastionhost, then tunnel from there to various systems, which allows connecting to SSH (linux instances), RDP (Windows), or basically any other network services like Redis or a database. I ended up writing some scripts to automate all this, so as long as you have the right certificates and IAM permissions, you can connect with a single command -- for Windows instances, it even retrieves the randomized password from the EC2 API. The end result is for any EC2 instances you're instantly popped into a shell/RDP session without having to enter credentials. I'd love to replace this with something better (eg Session Manager), but I've not seen how to do this for RDP, and haven't had the time to go experimenting on my own to see if it's even possible. If I can't 100% replace the bastionhosts, having two entirely different connection methods doesn't solve anything (and in fact makes it worse, because it's harder to use).

gregoryl 7y ago
Have a google, I was using SSM for remote access to Windows instances, specifically headless instances.

feydaykyn 7y ago

Does anyone know of it works with Ansible ? Thanks!

mishappen 7y ago

Be careful with SSM in general. The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances you want to access via Session Manager. This role grants read/write to all S3 buckets in your account (amongst other things). See this article for better steps and unavoidable risky things: https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/

jon918 7y ago
Good call to watch out for this stuff. The examples in the repo we set up use the AmazonSSMManagedInstanceCore managed policy, which does not grant any S3 permissions, just various ssm, ssmmessages, and ec2messages permissions.
jcrites 7y ago
> The documentation suggests adding the AmazonEC2RoleforSSM policy to the role of the EC2 instances Which documentation do you mean? The article mentions the policy AmazonSSMManagedInstanceCore, which is the same as what's mentioned in the SSM setup guide: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
- x3n0ph3n3 7y ago
  AmazonSSMManagedInstanceCore is still too much access, it has unscoped ssm:GetParameter! I hope you weren't trying to protect any secrets in ParameterStore!
  - acdha 7y ago
    Doesn’t that still fail if you don’t also have the corresponding decrypt KMS permission?
- mishappen 7y ago
  Thanks for clarifying, I didn’t recheck since we rolled out SSM in mid-2019 and then scrambled when we realised we’d granted account wide S3 permissions. The article I linked to also has a recommended minimal IAM policy for Run Command and SSM. I’ll update my comment to mention this.

jadell 7y ago

Does anyone know how this works with other utils that use SSH protocol, like rsync? What about tunneling other services to or from a local host? I'd love to have fewer hosts to maintain and a smaller network/attack surface, but we use SSH for more than just gaining commandline access to our instances.

WaxProlix 7y ago
It does. https://aws.amazon.com/about-aws/whats-new/2019/07/session-manager-launches-tunneling-support-for-ssh-and-scp/ https://aws.amazon.com/about-aws/whats-new/2019/07/session-m...

bogomipz 7y ago

The author states: >"No more bastion hosts required! Session Manager uses AWS APIs to communicate with your instances, so you can remove the administrative burden of maintaining bastion hosts." Does this presume the EC2 instances have a public IP or is there a way this would also work with EC2 instances on private subnets?

jhinds 7y ago
We've been using the Session Manager with instances in private subnets without issue, works like a charm.
- bogomipz 7y ago
  Is there anything special that needs to be configured to get this to work on private subnets? Currently I have an EKS cluster accessible only on private subnets. It would be wonderful to to be able to access this without OpenVPN in the mix.
  - exidy 7y ago
    The instances establish an outbound connection to the API SSM API, so as long as they can hit that, Session Manager will work. Connectivity from a private subnet to the AWS API could be (a) NAT gateway (b) HTTP proxy (c) PrivateLink VPC endpoint.

jon918 7y ago

I wrote a follow up post to this on SSH tunneling: https://news.ycombinator.com/item?id=22665037 https://news.ycombinator.com/item?id=22665037

shurco 7y ago

Hey, what about the Werbot solution - werbot.com? Now it is very relevant.