17 ms·

Emacs Tramp over AWS SSM APIs

116 points by zshev 5y ago

taeric 5y ago
Tramp is by far the most magical feeling trick in the emacs toolbox.
- Scarbutt 5y ago
  vscode ate everyone's lunch in this department.
  - taeric 5y ago
    How? Tramp has been working at this level for well over a decade, if I'm not mistaken. And you don't have to have anything installed in the "host" that you are connecting to.
    - Scarbutt 5y ago
      When your dev environment is on a remote server, the DX of vscode is superior, everything just works(like all your plugins), it's seamless and fast. Tramp is great for editing some remote files here and there, but to match vscode you will have to put a lot of effort to make everything feel equally fast and make all your packages work. Even then it won't feel as seamless as vscode because it "cheats" by installing a remote component, and I don't find that to be a valid complaint since you are already installing your whole dev environment in the remote server. Having said that (I'm not a vscode user), what I always do is use Emacs on the remote server inside tmux. For me that's better and superior to the vscode remote plugin, my dev environment is local to my editor.
      - taeric 5y ago
        If you install any auxiliary dev tools on the remote, it will work just as seamlessly. Git, lsp, etc. I confess that you need a solid ssh connection. But for the most part it has been great for a long time.
      - ungamedplayer 5y ago
        I hate hearing the 'everything just works', because it 'just works' until it doesn't. Apple users use the same term until their software no longer just works.
    - squiddev 5y ago
      If you're working on very slow network connections, or the network just dies entirely, it's not uncommon for Emacs just to hang entirely. You then have to kill Emacs from another terminal. At the time I was running exwm (an Emacs window manager), which made the whole thing even more painful. Emacs is powerful, but polish is not its strong point. That said, this was a few years ago now. Things may have improved in 26.1 when threads were introduced, and async got even easier.
      - taeric 5y ago
        I can concur that a slow connection is bad. It shouldn't hang entirely, as C-g should still get you response back. That said, it will hiccup bad and a save needs a round trip, regardless.
    - ithrow 5y ago
      Every time you save your edits Tramp makes a new connection to the remote server, it's slow(1sec vs 1ms) and becomes annoying waiting for the save all the time. For doing quick edits it doesn't matter but for doing dev all day it does.
      - hibbelig 5y ago
        Tramp offers different "connection methods" with different characteristics. For example, the scp connection method uses scp to copy files to/from the remote machine, and this implies a new ssh connection each time. But the ssh connection method transfers the files inline, using base64 or uu encoding, and then you do not need a new connection each time the file is read or written.
      - kstrauser 5y ago
        Configuring OpenSSH's ControlMaster setting here makes an enormous difference. Summary: it keeps a connection open for a while in case you want to connect to the same machine again. If you do, it reuses that connection so the new one is nearly instant.
        Scarbutt 5y ago
        I'll give it a try, thanks.
        kstrauser 5y ago
        Examples, starting with a new connection to a server I haven't accessed recently: ᐅ time ssh myserver exit Executed in 1.66 secs Now visiting it again uses the existing connection: ᐅ time ssh mastodon exit Executed in 55.89 millis
        kstrauser 5y ago
        (Assume the hostname is identical in both cases. I edited one and not the other. Oops!)
    - criddell 5y ago
      > How? Microsoft has done a lot better job promoting VSCode than GNU has promoting Emacs for the past few years. More mindshare among influential developers / evangelists has lead to massive increases in adoption which leads to better extensions which in turn fuels more adoption. It probably doesn't hurt that VSCode uses the MIT license.
      - ithrow 5y ago
        The vscode remote plugin is developed by microsoft and it's closed source. I can see why, it gives them a competitive advantage over all rival IDEs.
  - ak217 5y ago
    Yes, the vscode remote development plugin is a game changer. It's the new benchmark for how client-server IDEs should work. I am (and more importantly, my team is) no longer constrained to the terminal and memorizing incredibly obscure emacs or vi commands to get stuff done on a remote instance. There is no input lag because vscode keeps all the IDE UI local while doing all the heavy lifting remotely. And to the article's point, it treats the remote as "cattle not pets": all of my vscode settings and preferences are local and synced to github, and any time I connect to a new instance, it's able to reinitialize all of my vscode remote state from scratch. Tramp may have been able to do some of that before... but its accessibility was lacking.
    - dangom 5y ago
      I'd love to see a workflow comparison between emacs, vim (with remote work via neovim's tcp support + neovide) and vscode. I'm currently using Emacs and have a pretty decent setup for remote work with jupyter-emacs and tramp, and it's pretty much 0 overhead to run the same code on multiple remotes, or have the same remote run code stored in multiple places. With that said, all abstractions end if my SSH connection breaks, since remote stuff dies on disconnect. With neovim the remote can run inside of a tmux pane, so disconnections are not really a problem, but my vim skills are as of yet not as great. I haven't used VS code yet, simply because of lack of time in relearning another editor. In which particular way do you feel like VS code remote plugin is superior to the alternatives? And is there anything lacking in your VS code experience as of today?
      - ak217 5y ago
        I don't think the level of integration between tramp and other essential emacs plugins (flycheck, jedi, magit, etc.) is comparable to vscode. In vscode, entire plugins responsible for this stuff are sent over to run on the remote, leaving the local to run the UI - but also to keep all the settings and state. In emacs I have to either add pretty complicated scripts to my .emacs just to get stuff to play together if it's even possible at all, or stay in the terminal and run it all on the remote (and put up with the lag, and re-mount/upload my configuration when a new instance starts). For the longest time I used emacs on the remote and pycharm/jetbrains locally (and was a vscode skeptic) - that changed once I saw what the remote dev plugin was capable of (jetbrains doesn't have an equivalent). I still use emacs in the terminal on remotes for quick text editing, but for project work vscode works better specifically because it's easier to resume on disconnect (one-click restore of all state) and easier to configure. I use tmux in the vscode terminal to resume remote shell sessions. More importantly, it's a lot easier to onboard others to vscode because the IDE as a whole is more discoverable, more user-friendly, and follows platform conventions more closely compared to emacs or vi. The one big feature that I miss in vscode is tab key behavior/intelligent indentation. Emacs does this way better - tab just does what I mean, instead of inserting a useless literal tab or spaces.
  - daptaq 5y ago
    Modulo being a proprietary extention.
hibbelig 5y ago
If ssm-session-manager-plugin gives you a shell, then it should not be too hard to extend Tramp to use it directly. (I know nothing about ssm-session-manager-plugin.) Tramp does not need scp to transfer files, it can just as easily multiplex them over the shell connection by using base64 or uu encoding.
tyingq 5y ago
"Perhaps more interesting, though, is that for the last couple of years AWS has supported tunneling the SSH protocol over their SSM APIs if you use the SSM “document” called AWS-StartSSHSession." That's interesting. I know some places go to great lengths to keep developers from accessing production without some sort of break-glass procedure through a jump host. I'm curious if they all know about this sort of loophole.
- WaxProlix 5y ago
  As the article states, it's completely controlled by IAM and whatever federated identity management you hook up to AWS, and the events are auditable via cloudtrail etc.
- staticassertion 5y ago
  SSM is much preferred to a jump host for a number of reasons. 1. You don't have to expose a jump host at all, which is one less exposed asset to manage and worry about. 2. Your security team should already be collecting Cloudtrail logs, so they get auditing of SSM/SSH "for free". 3. You can control SSM access via your SSO provider, which means you can trivially enforce a bunch of policies all in one place vs having to configure SSHD. 4. You can control SSM access via IAM. 5. You can limit session duration easily. 6. No more SSH agent hijacking, at least I don't think. I also wouldn't call this a loophole, you have to explicitly have permissions to use SSM.
  - tyingq 5y ago
    >I also wouldn't call this a loophole, you have to explicitly have permissions to use SSM. Perhaps not the best wording on my part. I was aware of SSM, but not aware of the SSH tunneling features. I'm wondering if that's common. Is the SSH tunneling controlled separately, or on by default if SSM is on?
    - awsthro00945 5y ago
      It is "on" by default, but the user still has to have the 'ssm:StartSession' permission (and probably others) to open the SSM session, and for some(?) operations you also still need to have the appropriate credentials (ssh keypair or a password) to login via SSH. SSM Session Manager is one of the (if not the) preferred way to manage SSH access to instances in AWS. It's kinda hairy to set up, but it removes the need for bastion hosts/jump boxes for most use cases. From my experience I would say it is quite common.
thamer 5y ago
TIL: TRAMP (Transparent Remote Access, Multiple Protocols) is a package for editing remote files [...] Whereas the others use FTP to connect to the remote host and to transfer the files, TRAMP uses a remote shell connection (rlogin, telnet, ssh). https://www.emacswiki.org/emacs/TrampMode https://www.emacswiki.org/emacs/TrampMode
- throwawayboise 5y ago
  Tramp can use FTP if you tell it to. And it's still transferring files. It's not remotely editing.
- tptacek 5y ago
  Tramp is one of those "reasons to use Emacs in the first place" packages. I've been using Emacs since the 1990s, when someone impressed me with syntax highlighting in Lucid Emacs, and I only picked up Tramp last year. In the last 6-9 months or so, almost all of my development has been over Tramp. What's particularly impressive about Tramp is that other Emacs packages tend to work well with it. For instance, you can Magit over Tramp --- or, better put: Magit just works in Tramp buffers. Same with language server stuff. It's kind of wild when you think about what's happening under the hood.
gunapologist99 5y ago
> For a good wee while now, AWS SSM (or AWS Systems Manager as I see they are calling it nowadays) has arguably been the most secure way to permit controlled and audited access to an EC2 instance. SSM is definitely not the most secure way[0]. SSM is super complex and super-integrated into the rest of AWS, and also isn't cross-cloud to GCP, Azure, DO, etc, so now everyone needs an account just to log into a Linux server. Worse, IAM roles are powerful but easy to misconfigure, and that's before getting into how hard they are to apply with any granularity because of the policy length limitations[1], so you're likely giving everyone access to log into every instance without even knowing it. 0. https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ 1. https://aws.amazon.com/premiumsupport/knowledge-center/iam-increase-policy-size/ https://aws.amazon.com/premiumsupport/knowledge-center/iam-i...
- nijave 5y ago
  SSM supports BYO doesn't it? Can't you install the agent on any machine to enroll it in SSM or does that limit what you can do?
- tptacek 5y ago
  What does being cross-cloud have to do with whether SSM is the most secure way to SSH into an AWS instance?
  - gunapologist99 5y ago
    Because everyone will need a (possibly misconfigured) AWS IAM account just to log into any Linux server.. this increases complexity and reduces isolation, compartmentalization, separation of concerns, least privilege, etc. I was mentioning that particular misfeature because it was a personal annoyance of mine. Oh well, I suppose everything is about customer lock-in these days.
    - tptacek 5y ago
      It sounds like you don't think AWS is the most secure place to host an application. That's not the argument being made here; the argument stipulates AWS.

116 points by zshev 5y ago

taeric 5y ago

Tramp is by far the most magical feeling trick in the emacs toolbox.

Scarbutt 5y ago
vscode ate everyone's lunch in this department.
- taeric 5y ago
  How? Tramp has been working at this level for well over a decade, if I'm not mistaken. And you don't have to have anything installed in the "host" that you are connecting to.
  - Scarbutt 5y ago
    When your dev environment is on a remote server, the DX of vscode is superior, everything just works(like all your plugins), it's seamless and fast. Tramp is great for editing some remote files here and there, but to match vscode you will have to put a lot of effort to make everything feel equally fast and make all your packages work. Even then it won't feel as seamless as vscode because it "cheats" by installing a remote component, and I don't find that to be a valid complaint since you are already installing your whole dev environment in the remote server. Having said that (I'm not a vscode user), what I always do is use Emacs on the remote server inside tmux. For me that's better and superior to the vscode remote plugin, my dev environment is local to my editor.
    - taeric 5y ago
      If you install any auxiliary dev tools on the remote, it will work just as seamlessly. Git, lsp, etc. I confess that you need a solid ssh connection. But for the most part it has been great for a long time.
    - ungamedplayer 5y ago
      I hate hearing the 'everything just works', because it 'just works' until it doesn't. Apple users use the same term until their software no longer just works.
  - squiddev 5y ago
    If you're working on very slow network connections, or the network just dies entirely, it's not uncommon for Emacs just to hang entirely. You then have to kill Emacs from another terminal. At the time I was running exwm (an Emacs window manager), which made the whole thing even more painful. Emacs is powerful, but polish is not its strong point. That said, this was a few years ago now. Things may have improved in 26.1 when threads were introduced, and async got even easier.
    - taeric 5y ago
      I can concur that a slow connection is bad. It shouldn't hang entirely, as C-g should still get you response back. That said, it will hiccup bad and a save needs a round trip, regardless.
  - ithrow 5y ago
    Every time you save your edits Tramp makes a new connection to the remote server, it's slow(1sec vs 1ms) and becomes annoying waiting for the save all the time. For doing quick edits it doesn't matter but for doing dev all day it does.
    - hibbelig 5y ago
      Tramp offers different "connection methods" with different characteristics. For example, the scp connection method uses scp to copy files to/from the remote machine, and this implies a new ssh connection each time. But the ssh connection method transfers the files inline, using base64 or uu encoding, and then you do not need a new connection each time the file is read or written.
    - kstrauser 5y ago
      Configuring OpenSSH's ControlMaster setting here makes an enormous difference. Summary: it keeps a connection open for a while in case you want to connect to the same machine again. If you do, it reuses that connection so the new one is nearly instant.
      - Scarbutt 5y ago
        I'll give it a try, thanks.
        kstrauser 5y ago
        Examples, starting with a new connection to a server I haven't accessed recently: ᐅ time ssh myserver exit Executed in 1.66 secs Now visiting it again uses the existing connection: ᐅ time ssh mastodon exit Executed in 55.89 millis
        kstrauser 5y ago
        (Assume the hostname is identical in both cases. I edited one and not the other. Oops!)
  - criddell 5y ago
    > How? Microsoft has done a lot better job promoting VSCode than GNU has promoting Emacs for the past few years. More mindshare among influential developers / evangelists has lead to massive increases in adoption which leads to better extensions which in turn fuels more adoption. It probably doesn't hurt that VSCode uses the MIT license.
    - ithrow 5y ago
      The vscode remote plugin is developed by microsoft and it's closed source. I can see why, it gives them a competitive advantage over all rival IDEs.
- ak217 5y ago
  Yes, the vscode remote development plugin is a game changer. It's the new benchmark for how client-server IDEs should work. I am (and more importantly, my team is) no longer constrained to the terminal and memorizing incredibly obscure emacs or vi commands to get stuff done on a remote instance. There is no input lag because vscode keeps all the IDE UI local while doing all the heavy lifting remotely. And to the article's point, it treats the remote as "cattle not pets": all of my vscode settings and preferences are local and synced to github, and any time I connect to a new instance, it's able to reinitialize all of my vscode remote state from scratch. Tramp may have been able to do some of that before... but its accessibility was lacking.
  - dangom 5y ago
    I'd love to see a workflow comparison between emacs, vim (with remote work via neovim's tcp support + neovide) and vscode. I'm currently using Emacs and have a pretty decent setup for remote work with jupyter-emacs and tramp, and it's pretty much 0 overhead to run the same code on multiple remotes, or have the same remote run code stored in multiple places. With that said, all abstractions end if my SSH connection breaks, since remote stuff dies on disconnect. With neovim the remote can run inside of a tmux pane, so disconnections are not really a problem, but my vim skills are as of yet not as great. I haven't used VS code yet, simply because of lack of time in relearning another editor. In which particular way do you feel like VS code remote plugin is superior to the alternatives? And is there anything lacking in your VS code experience as of today?
    - ak217 5y ago
      I don't think the level of integration between tramp and other essential emacs plugins (flycheck, jedi, magit, etc.) is comparable to vscode. In vscode, entire plugins responsible for this stuff are sent over to run on the remote, leaving the local to run the UI - but also to keep all the settings and state. In emacs I have to either add pretty complicated scripts to my .emacs just to get stuff to play together if it's even possible at all, or stay in the terminal and run it all on the remote (and put up with the lag, and re-mount/upload my configuration when a new instance starts). For the longest time I used emacs on the remote and pycharm/jetbrains locally (and was a vscode skeptic) - that changed once I saw what the remote dev plugin was capable of (jetbrains doesn't have an equivalent). I still use emacs in the terminal on remotes for quick text editing, but for project work vscode works better specifically because it's easier to resume on disconnect (one-click restore of all state) and easier to configure. I use tmux in the vscode terminal to resume remote shell sessions. More importantly, it's a lot easier to onboard others to vscode because the IDE as a whole is more discoverable, more user-friendly, and follows platform conventions more closely compared to emacs or vi. The one big feature that I miss in vscode is tab key behavior/intelligent indentation. Emacs does this way better - tab just does what I mean, instead of inserting a useless literal tab or spaces.
- daptaq 5y ago
  Modulo being a proprietary extention.

hibbelig 5y ago

If ssm-session-manager-plugin gives you a shell, then it should not be too hard to extend Tramp to use it directly. (I know nothing about ssm-session-manager-plugin.) Tramp does not need scp to transfer files, it can just as easily multiplex them over the shell connection by using base64 or uu encoding.

tyingq 5y ago

"Perhaps more interesting, though, is that for the last couple of years AWS has supported tunneling the SSH protocol over their SSM APIs if you use the SSM “document” called AWS-StartSSHSession." That's interesting. I know some places go to great lengths to keep developers from accessing production without some sort of break-glass procedure through a jump host. I'm curious if they all know about this sort of loophole.

WaxProlix 5y ago
As the article states, it's completely controlled by IAM and whatever federated identity management you hook up to AWS, and the events are auditable via cloudtrail etc.
staticassertion 5y ago
SSM is much preferred to a jump host for a number of reasons. 1. You don't have to expose a jump host at all, which is one less exposed asset to manage and worry about. 2. Your security team should already be collecting Cloudtrail logs, so they get auditing of SSM/SSH "for free". 3. You can control SSM access via your SSO provider, which means you can trivially enforce a bunch of policies all in one place vs having to configure SSHD. 4. You can control SSM access via IAM. 5. You can limit session duration easily. 6. No more SSH agent hijacking, at least I don't think. I also wouldn't call this a loophole, you have to explicitly have permissions to use SSM.
- tyingq 5y ago
  >I also wouldn't call this a loophole, you have to explicitly have permissions to use SSM. Perhaps not the best wording on my part. I was aware of SSM, but not aware of the SSH tunneling features. I'm wondering if that's common. Is the SSH tunneling controlled separately, or on by default if SSM is on?
  - awsthro00945 5y ago
    It is "on" by default, but the user still has to have the 'ssm:StartSession' permission (and probably others) to open the SSM session, and for some(?) operations you also still need to have the appropriate credentials (ssh keypair or a password) to login via SSH. SSM Session Manager is one of the (if not the) preferred way to manage SSH access to instances in AWS. It's kinda hairy to set up, but it removes the need for bastion hosts/jump boxes for most use cases. From my experience I would say it is quite common.

thamer 5y ago

TIL: TRAMP (Transparent Remote Access, Multiple Protocols) is a package for editing remote files [...] Whereas the others use FTP to connect to the remote host and to transfer the files, TRAMP uses a remote shell connection (rlogin, telnet, ssh). https://www.emacswiki.org/emacs/TrampMode https://www.emacswiki.org/emacs/TrampMode

throwawayboise 5y ago
Tramp can use FTP if you tell it to. And it's still transferring files. It's not remotely editing.
tptacek 5y ago
Tramp is one of those "reasons to use Emacs in the first place" packages. I've been using Emacs since the 1990s, when someone impressed me with syntax highlighting in Lucid Emacs, and I only picked up Tramp last year. In the last 6-9 months or so, almost all of my development has been over Tramp. What's particularly impressive about Tramp is that other Emacs packages tend to work well with it. For instance, you can Magit over Tramp --- or, better put: Magit just works in Tramp buffers. Same with language server stuff. It's kind of wild when you think about what's happening under the hood.

gunapologist99 5y ago

> For a good wee while now, AWS SSM (or AWS Systems Manager as I see they are calling it nowadays) has arguably been the most secure way to permit controlled and audited access to an EC2 instance. SSM is definitely not the most secure way[0]. SSM is super complex and super-integrated into the rest of AWS, and also isn't cross-cloud to GCP, Azure, DO, etc, so now everyone needs an account just to log into a Linux server. Worse, IAM roles are powerful but easy to misconfigure, and that's before getting into how hard they are to apply with any granularity because of the policy length limitations[1], so you're likely giving everyone access to log into every instance without even knowing it. 0. https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ 1. https://aws.amazon.com/premiumsupport/knowledge-center/iam-increase-policy-size/ https://aws.amazon.com/premiumsupport/knowledge-center/iam-i...

nijave 5y ago
SSM supports BYO doesn't it? Can't you install the agent on any machine to enroll it in SSM or does that limit what you can do?
tptacek 5y ago
What does being cross-cloud have to do with whether SSM is the most secure way to SSH into an AWS instance?
- gunapologist99 5y ago
  Because everyone will need a (possibly misconfigured) AWS IAM account just to log into any Linux server.. this increases complexity and reduces isolation, compartmentalization, separation of concerns, least privilege, etc. I was mentioning that particular misfeature because it was a personal annoyance of mine. Oh well, I suppose everything is about customer lock-in these days.
  - tptacek 5y ago
    It sounds like you don't think AWS is the most secure place to host an application. That's not the argument being made here; the argument stipulates AWS.