4 ms·

Down the rabbit hole: Implementing SSH port forwarding over AWS Session Manager

59 points by saligrama 2y ago

rpicard 2y ago
Funny to read this after seeing Fly.io’s recent blog about VS Code’s remote SSH agent: https://fly.io/blog/vscode-ssh-wtf/ https://fly.io/blog/vscode-ssh-wtf/ I’m surprised this hasn’t come up more frequently. I’d expect a lot of security products to flag it as very much looking like malware.
- scarface_74 2y ago
  The System Manager Agent on EC2 instances is meant to have enough privileges to manage fleet of instances at scale. System Manager Session Manager (yes that’s the real name) is a part of it. What do you suggest as an alternative? Session Manager is IAM controlled and much more secure and easily managed than the traditional means involving direct access via SSH and opening a port through the security group for a certain IP address or a jump box server.
ChoHag 2y ago
[dead]
nicornk 2y ago
If you just want to enable ssh to ec2 instances (through SSM) using ssh i-… you can add the following lines to your ssh config https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0c16c https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0... host i-* IdentityFile ~/.ssh/id_rsa TCPKeepAlive yes ServerAliveInterval 120 User ec2-user ProxyCommand sh -c "aws ec2 start-instances --instance-ids %h ; aws ec2 wait instance-running --instance-ids %h ; aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user %r --ssh-public-key 'file://~/.ssh/id_rsa.pub' --availability-zone $(aws ec2 describe-instances --instance-ids %h --query 'Reservations[0].Instances[0].Placement.AvailabilityZone') ; aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" This will also allow VSCode remote development.
- deleted 2y ago
  [deleted]
- Galanwe 2y ago
  My variation is to use a custom script as `ProxyCommand` that resolves private route53 DNS names to instance ids, because remembering instance IDs is insane.
  - smackeyacky 2y ago
    Mine is to run a Tailscale node on a tiny ec2 instance. Not only enabling ssh but direct access to database instances, s3 buckets that are blocked from public access etc
    - scarface_74 2y ago
      How are S3 buckets blocked from public access? I mean I know there is literally a “Block public access” feature that keeps S3 buckets from being read or written by unauthenticated users. But as far as I know without some really weird bucket ACLs you can still access S3 buckets if you have the IAM credentials. Before anyone well actually’s me. Yes I know you can also route S3 via AWS internal network with VPC Endpoints between AWS services.
      - Galanwe 2y ago
        You essentially add a policy that limits the access to only come from your VPC endpoint.
      - nijave 2y ago
        In general, condition keys https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p... And https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html#amazons3-policy-keys https://docs.aws.amazon.com/service-authorization/latest/ref... Specifically the vpce one as the other poster mentioned but there's other like IP limits Another way is an IdP that supports network or device rules. For instance, Cloudflare Access and Okta you can add policies where they'll only let you auth if you meet device or network requirements which achieved the same thing
ryan-duve 2y ago
Does using SSH this way solves the Ctrl-Y problem in AWS Session Manager, which kills the session?
zanderhill 2y ago
Very excited about the fork's improvements given the lack of responsiveness in core repo. I've hit some of these bugs including the registry issue. Thank you for open sourcing it. I've used a combination of ProxyCommand directive in ssh config + a script it calls w/ the `%h` (host) to unpack what the correct instance-id is (like @galanwe). For the proxycommand, you can embed an `aws ec2-instance-connect send-ssh-public-key` for pushing a key valid for 60s followed by activating the SSM session. The downside is it adds ~20-30s delay in connection due to the API requests, but if you're making repeated rapid requests to same instance, I recommend looking into ssh's ControlPath, ControlMaster and ControlPersist to keep a longer lived session that's re-used for client re-connections (ref: https://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing/ https://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing...) [Edit to add that I've hit the registry bug myself]
nijave 2y ago
Was wondering if it's possible to use sshuttle with SSM but didn't realize SSM had its own mixing protocol. An SSM client library is super helpful for embedding the tunneling into the client app. At a previous employer, we'd access jumphosts from AWS Session Manager and GCP Identity Aware Proxy and were looking to embed that connection flow into some cli management utilities to ease use. At the time, we ended up starting the tunneling processes separately inside the code but it would have been much nicer and easier with a library.

59 points by saligrama 2y ago

Funny to read this after seeing Fly.io’s recent blog about VS Code’s remote SSH agent: https://fly.io/blog/vscode-ssh-wtf/ https://fly.io/blog/vscode-ssh-wtf/ I’m surprised this hasn’t come up more frequently. I’d expect a lot of security products to flag it as very much looking like malware.

scarface_74 2y ago
The System Manager Agent on EC2 instances is meant to have enough privileges to manage fleet of instances at scale. System Manager Session Manager (yes that’s the real name) is a part of it. What do you suggest as an alternative? Session Manager is IAM controlled and much more secure and easily managed than the traditional means involving direct access via SSH and opening a port through the security group for a certain IP address or a jump box server.

ChoHag 2y ago

[dead]

nicornk 2y ago

If you just want to enable ssh to ec2 instances (through SSM) using ssh i-… you can add the following lines to your ssh config https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0c16c https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0... host i-* IdentityFile ~/.ssh/id_rsa TCPKeepAlive yes ServerAliveInterval 120 User ec2-user ProxyCommand sh -c "aws ec2 start-instances --instance-ids %h ; aws ec2 wait instance-running --instance-ids %h ; aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user %r --ssh-public-key 'file://~/.ssh/id_rsa.pub' --availability-zone $(aws ec2 describe-instances --instance-ids %h --query 'Reservations[0].Instances[0].Placement.AvailabilityZone') ; aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" This will also allow VSCode remote development.

deleted 2y ago
[deleted]
Galanwe 2y ago
My variation is to use a custom script as `ProxyCommand` that resolves private route53 DNS names to instance ids, because remembering instance IDs is insane.
- smackeyacky 2y ago
  Mine is to run a Tailscale node on a tiny ec2 instance. Not only enabling ssh but direct access to database instances, s3 buckets that are blocked from public access etc
  - scarface_74 2y ago
    How are S3 buckets blocked from public access? I mean I know there is literally a “Block public access” feature that keeps S3 buckets from being read or written by unauthenticated users. But as far as I know without some really weird bucket ACLs you can still access S3 buckets if you have the IAM credentials. Before anyone well actually’s me. Yes I know you can also route S3 via AWS internal network with VPC Endpoints between AWS services.
    - Galanwe 2y ago
      You essentially add a policy that limits the access to only come from your VPC endpoint.
    - nijave 2y ago
      In general, condition keys https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p... And https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html#amazons3-policy-keys https://docs.aws.amazon.com/service-authorization/latest/ref... Specifically the vpce one as the other poster mentioned but there's other like IP limits Another way is an IdP that supports network or device rules. For instance, Cloudflare Access and Okta you can add policies where they'll only let you auth if you meet device or network requirements which achieved the same thing

ryan-duve 2y ago

Does using SSH this way solves the Ctrl-Y problem in AWS Session Manager, which kills the session?

zanderhill 2y ago

Very excited about the fork's improvements given the lack of responsiveness in core repo. I've hit some of these bugs including the registry issue. Thank you for open sourcing it. I've used a combination of ProxyCommand directive in ssh config + a script it calls w/ the `%h` (host) to unpack what the correct instance-id is (like @galanwe). For the proxycommand, you can embed an `aws ec2-instance-connect send-ssh-public-key` for pushing a key valid for 60s followed by activating the SSM session. The downside is it adds ~20-30s delay in connection due to the API requests, but if you're making repeated rapid requests to same instance, I recommend looking into ssh's ControlPath, ControlMaster and ControlPersist to keep a longer lived session that's re-used for client re-connections (ref: https://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing/ https://blog.scottlowe.org/2015/12/11/using-ssh-multiplexing...) [Edit to add that I've hit the registry bug myself]

nijave 2y ago

Was wondering if it's possible to use sshuttle with SSM but didn't realize SSM had its own mixing protocol. An SSM client library is super helpful for embedding the tunneling into the client app. At a previous employer, we'd access jumphosts from AWS Session Manager and GCP Identity Aware Proxy and were looking to embed that connection flow into some cli management utilities to ease use. At the time, we ended up starting the tunneling processes separately inside the code but it would have been much nicer and easier with a library.