9 ms·

Zero-Trust RDP and SSH Access to VMs on Google Cloud

185 points by confuseddeputy 5y ago

OrvalWintermute 5y ago
In reading through this on the surface, it appears as though there is a mix of trust relationships that pre-exist, and credential issuances that occur on the fly. Also, it also appears there is no privilege tiering aka, enterprise access model, applied to the example. Did I see this wrong? I'd be interested in seeing what credentials in toto are there, and which ones are ephemeral, and susceptibility to lateral traversal. Could you respond on the merits of the critique?
- CaliforniaKarl 5y ago
  For Linux systems at least, IAP doesn’t deal with privilege tiering. Instead, OS Login handles mapping a user’s Google account to a local account. There is also a program that queries a user’s SSH key from OS Login, and passes it to sshd when asked. OS Login defines two IAM roles, one for “Can I log in?” and one for “Can I sudo?”. Those are implemented on the system via PAM, so you can add whatever additional restrictions you’d like. Fetching of user information via OS Login is implemented via a NSS module. POSIX attributes can be customized via the Google Directory API. And I believe Google Groups can be mapped to POSIX supplemental groups, but I’m not certain.
CaliforniaKarl 5y ago
I make use of IAP and OS Login today, to log in to a Compute Engine Linux VM. The VM has Internet access via NAT, and has no public IP. Logging in is via `gcloud compute ssh`. Authenticating `gcloud` involves a corporate login which uses a client certificate and two-step. For all the components involved, it works pretty well!
- pritambaral 5y ago
  > ... which uses a client certificate ... Can you encrypt the client private key on disk and use sth like ssh-agent?
  - 0xEFF 5y ago
    Yes, simply add the key gcloud generates as normal using ssh-add ~/.ssh/google_compute_engine
    - pritambaral 5y ago
      No, I meant the client private key that gcloud uses to authenticate itself (on your behalf) to Google's servers, not you to your servers. That wouldn't be an SSH key, probably TLS or hand-rolled crypto. ---- Also, now that you mention it, even if I encrypted the generated SSH key, wouldn't running a `gcloud ...` command again just ... re-generate the key, in unencrypted form?
      - CaliforniaKarl 5y ago
        Sorry, I should clarify: The client key is used in our corporate login. When I log in to `gcloud`, that goes through our corporate login. Corporate login uses a client certificate and two-step.
- sillystuff 5y ago
  You can also add it to your ~/.ssh/config, so you can just ssh hostname, scp hostname, etc. without a public IP on the VM. Host myhost ProxyCommand gcloud compute ssh user@myhost --zone=myzone --tunnel-through-iap --command="nc 0.0.0.0 22" -- -o "UserKnownHostsFile /dev/null" -o "StrictHostKeyChecking no"
etaioinshrdlu 5y ago
I didn't know big tech companies were still capable of making GUI's that look like this.
- unixhero 5y ago
  It looks really useful!
  - vbezhenar 5y ago
    It's gorgeous. Like a breath of fresh air after all that electron nightmare.
    - michaelcampbell 5y ago
      I don't see the correlation. I get that people don't like electron bloat, but that is orthogonal to UI/UX, no?
      - Isthatablackgsd 5y ago
        The issue with Electron is that devs tries to mimic the browsers UI in a desktop environment. Our desktop OS need to have a desktop UI instead of the mobile webpage with Material/Flat UI design (no visual box line, no separators, no multi-windows, pushing the setting menu as a sidebar, etc). It should be treated as a desktop app with the benefits of desktop, not a mobile app which is the issue.
- hmottestad 5y ago
  It’s based on visual studio.
  - scq 5y ago
    It's not. It's using WinForms with the DockPanel Suite library, with a VS2015 theme.
  - electroly 5y ago
unixhero 5y ago
How much work would it be to make this general purpose? To not only work for Google-cloud...
- mwadhwa 5y ago
  Checkout this guide I published today. It walks through the code to do the secure tunneling part in ~20 lines of Rust, using Ockam a library to create end-to-end encrypted secure channels https://github.com/ockam-network/ockam/tree/develop/documentation/use-cases/secure-remote-access-tunnels#readme https://github.com/ockam-network/ockam/tree/develop/document...
- PLG88 5y ago
  Here open source, general purpose version for access to any cloud or resource in fact. This is specifically for SSH but can support any protocol or app with the vanilla SDKs and tunnelers - https://ziti.dev/blog/zitifying-ssh/ https://ziti.dev/blog/zitifying-ssh/
Ruhrbaron 5y ago
Would be nice to have a Mac version of that.
- PLG88 5y ago
  See PLG88 comment above, this can work for MAC - https://apps.apple.com/app/id1460484572 https://apps.apple.com/app/id1460484572
nickysielicki 5y ago
I'm sorry if this sounds completely ridiculous to some people, but what do people use RDP/Windows server for in 2021? Given that ASP/Dotnet is portable to linux, what are people building that isn't better deployed to linux? It can't just be the legacy use-case, can it?
- cprecioso 5y ago
  I use Windows VMs on Google Cloud to install and play games in the cloud - my Mac can’t play any
  - crakenzak 5y ago
    hows the performance/input lag on that? How much are you paying & is it worth?
    - cprecioso 5y ago
      It really depends on how far away from the data center you are. I used to live close to Amsterdam and that felt basically local, even with a not-amazing internet connection. Right now I live in the Valencia (Spain), with a local and not very well paired ISP. There is lag, but it is just about acceptable for platformers and action/adventure games. I wouldn't recommend it for shooters. The cost ended up being around 1€/hr when the instance was on, and 0.06€/day for the storage. I am sure this could be optimized (e.g. use snapshots and spot instance) - but I just couldn't be bothered to. For me it is worth it.
  - yoz-y 5y ago
    This is a very novel use case to me. Could you elaborate why use this solution instead of a local VM, boot camp, or something like Stadia? Seems that it would work quite well for turn based games (including auto save)
    - cprecioso 5y ago
      I don't use a local VM or bootcamp because I just don't have a powerful enough computer to run AAA games from the last 5 years or so - even earlier if the game is badly optimized. Now with an M1 they're not even compatible. Stadia, GeForce NOW, Xbox Game Streaming and PlayStation Now: I've tried them all, and GeForce NOW specifically is very good. But, they only allow me to play a subset of my large pre-existing game library, and/or make me buy games again, which I'm not interested in. Stadia is specially bad at this, having to purchase on a game-by-game basis, not being compatible with any existing ecosystem (Steam, PS, Xbox have cloud saves and friend lists), and then trusting Google to run my game forever on their servers after paying a one-time cost.
pledess 5y ago
This sounds like a deal breaker for some use cases: https://github.com/GoogleCloudPlatform/iap-desktop/wiki/Troubleshooting https://github.com/GoogleCloudPlatform/iap-desktop/wiki/Trou... "Because of the way IAP Desktop tunnels RDP connections, it always uses NTLM for authentication and can't use Kerberos." There may be environments that lose the security benefits of Kerberos over NTLMv2 (e.g., mutual authentication) because they've been forced into a new compliance mandate that dictates adoption of Zero Trust in all available contexts.
- dvdkon 5y ago
  Looks like Microsoft's decision to go with increasingly elaborate challenge-response schemes instead of properly encrypting the whole connection (like SSL/SSH) will be haunting us for a while yet. I don't understand why RDP/SMB/... with plaintext auth over SSL hasn't been a thing for at least a decade, does Microsoft just not care about transport security?
  - zaphirplane 5y ago
    Isn’t it plausible that an interactive gui over ssl didn’t perform well specially for VMs or the affect of the renegotiation
    - franga2000 5y ago
      RDP over SSH already performs very well, so any in-protocol implementation would only be faster (less overhead).
thowawaypets 5y ago
Why are we still building tools to hand manage VMs in 2021? Am I missing something or is this for raising pets instead of heading cattle[0]? 0. http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ http://cloudscaling.com/blog/cloud-computing/the-history-of-...
- pjmlp 5y ago
  Because not everyone is on the latest fashion.
- e12e 5y ago
  Farmers keep both sheep dogs and sheep...
- corty 5y ago
  Because when your herd of cattle is sick, you need to grab one and have a vet look at it before your whole herd dies from the plague.
  - thowawaypets 5y ago
    No, part of managing a heard is having the right tools in place. Like monitoring, logging, and observability tools. There is nothing I can learn from accessing a VM in production that I can't learn from my monitoring system. In prod where I work, if someone logs into a production VM we mark it tainted and replace it with a fresh instance. This keeps things nice and consistent. Of you need an interactive session on a prod machine you are missing tools.
    - verdverm 5y ago
      What about cloud based developer VMs?
    - corty 5y ago
      Plugging all our cattle into a heart- and bloodpressure monitor and doing frequent blood draws from every cow "just in case" is wasteful and unnecessary. There is a balance between sensible general always-available monitoring and special-case-debugging a problem. My rule for that is: more than once a year or more than 6h? Automate and tool it. Less? SSH or other special-case tools are fine.
e12e 5y ago
> IAP Desktop is a Windows application that allows you to manage multiple Remote Desktop and SSH connections to VM instances that run on Google Cloud. Is there a Linux client too?
technological 5y ago
Is this similar to AWS workspaces ?
hardwaresofton 5y ago
Inspired by Fly.io's post a while ago[0] I also did something similar to this on my small k8s cluster with the help of stunnel, sslh, and traefik[1]. Weirdly enough I thought this was the ability to provision a wireguard-esque proxy to any machine you want, operated at the edge of the cloud, but it seems like it's really TCP-over-HTTPS. It's easy to imagine doing the former (dynamic wireguard proxy surfacing) too though -- wireguard sidecar container with shared network namespace with the workload in question + open-to-the-world port somewhere and you'd theoretically have access to any port you wanted on said machine as well. Feels like an easy set up to trust as wireguard is pretty reliable/sound. [0]: https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ [1]: https://vadosware.io/post/stuffing-both-ssh-and-https-on-port-443-with-stunnel-ssh-and-traefik/ https://vadosware.io/post/stuffing-both-ssh-and-https-on-por...
- chucky_z 5y ago
  I run IAP. It's TCP-over-HTTPS but it works remarkably well, connects to all kinds of things, and for users it really is just "login with google, proceed as normal." I use a JWT proxy + ghostunnel within GKE with a VIP so it's not quite their reference setup but it's extremely "just works" outside GKE being weird and eating its own routes. BTW, side-note but try out ghostunnel over stunnel! I've really enjoyed using it and it's been fantastic to debug and work with.
  - hardwaresofton 5y ago
    > I run IAP. It's TCP-over-HTTPS but it works remarkably well, connects to all kinds of things, and for users it really is just "login with google, proceed as normal." Yeah that's really amazing, with client-side software like they've already made and I've seen from other vendors (whether GUI or TUI) the interfaces IaaS/PaaS companies can build are really slick. Looks like they'll be able to cut down on dashboard fatigue/complexity people are wrangling quite a bit. > I use a JWT proxy + ghostunnel within GKE with a VIP so it's not quite their reference setup but it's extremely "just works" outside GKE being weird and eating its own routes. Interesting, so JWT proxy (or any other auth mechanism that is viable over HTTPS) -> ghostunnel machine w/ public VIP -> Target machine ? Or ghostunnel directly running on the Target machine which holds the public VIP? Or does the JWT proxy take the public IP and the ghostunnel machine keep the private VIP? Apologies just want to be able to picture your solution clearly. > BTW, side-note but try out ghostunnel over stunnel! I've really enjoyed using it and it's been fantastic to debug and work with. Thanks for the recommendation of ghostunnel, will use it in the future over stunnel next time I hack together something like this. BTW: super-side note, breath of fire III avatar was a blast from the past, instantly recognized it.
deleted 5y ago
[deleted]
floatingatoll 5y ago
> IAP Desktop is an open-source project and not an officially supported Google product.
ngcc_hk 5y ago
Amazing. Obviously need audit. But so far so good.

In reading through this on the surface, it appears as though there is a mix of trust relationships that pre-exist, and credential issuances that occur on the fly. Also, it also appears there is no privilege tiering aka, enterprise access model, applied to the example. Did I see this wrong? I'd be interested in seeing what credentials in toto are there, and which ones are ephemeral, and susceptibility to lateral traversal. Could you respond on the merits of the critique?

CaliforniaKarl 5y ago
For Linux systems at least, IAP doesn’t deal with privilege tiering. Instead, OS Login handles mapping a user’s Google account to a local account. There is also a program that queries a user’s SSH key from OS Login, and passes it to sshd when asked. OS Login defines two IAM roles, one for “Can I log in?” and one for “Can I sudo?”. Those are implemented on the system via PAM, so you can add whatever additional restrictions you’d like. Fetching of user information via OS Login is implemented via a NSS module. POSIX attributes can be customized via the Google Directory API. And I believe Google Groups can be mapped to POSIX supplemental groups, but I’m not certain.

CaliforniaKarl 5y ago

I make use of IAP and OS Login today, to log in to a Compute Engine Linux VM. The VM has Internet access via NAT, and has no public IP. Logging in is via `gcloud compute ssh`. Authenticating `gcloud` involves a corporate login which uses a client certificate and two-step. For all the components involved, it works pretty well!

pritambaral 5y ago
> ... which uses a client certificate ... Can you encrypt the client private key on disk and use sth like ssh-agent?
- 0xEFF 5y ago
  Yes, simply add the key gcloud generates as normal using ssh-add ~/.ssh/google_compute_engine
  - pritambaral 5y ago
    No, I meant the client private key that gcloud uses to authenticate itself (on your behalf) to Google's servers, not you to your servers. That wouldn't be an SSH key, probably TLS or hand-rolled crypto. ---- Also, now that you mention it, even if I encrypted the generated SSH key, wouldn't running a `gcloud ...` command again just ... re-generate the key, in unencrypted form?
    - CaliforniaKarl 5y ago
      Sorry, I should clarify: The client key is used in our corporate login. When I log in to `gcloud`, that goes through our corporate login. Corporate login uses a client certificate and two-step.
sillystuff 5y ago
You can also add it to your ~/.ssh/config, so you can just ssh hostname, scp hostname, etc. without a public IP on the VM. Host myhost ProxyCommand gcloud compute ssh user@myhost --zone=myzone --tunnel-through-iap --command="nc 0.0.0.0 22" -- -o "UserKnownHostsFile /dev/null" -o "StrictHostKeyChecking no"

etaioinshrdlu 5y ago

I didn't know big tech companies were still capable of making GUI's that look like this.

unixhero 5y ago
It looks really useful!
- vbezhenar 5y ago
  It's gorgeous. Like a breath of fresh air after all that electron nightmare.
  - michaelcampbell 5y ago
    I don't see the correlation. I get that people don't like electron bloat, but that is orthogonal to UI/UX, no?
    - Isthatablackgsd 5y ago
      The issue with Electron is that devs tries to mimic the browsers UI in a desktop environment. Our desktop OS need to have a desktop UI instead of the mobile webpage with Material/Flat UI design (no visual box line, no separators, no multi-windows, pushing the setting menu as a sidebar, etc). It should be treated as a desktop app with the benefits of desktop, not a mobile app which is the issue.
hmottestad 5y ago
It’s based on visual studio.
- scq 5y ago
  It's not. It's using WinForms with the DockPanel Suite library, with a VS2015 theme.
- electroly 5y ago

unixhero 5y ago

How much work would it be to make this general purpose? To not only work for Google-cloud...

mwadhwa 5y ago
Checkout this guide I published today. It walks through the code to do the secure tunneling part in ~20 lines of Rust, using Ockam a library to create end-to-end encrypted secure channels https://github.com/ockam-network/ockam/tree/develop/documentation/use-cases/secure-remote-access-tunnels#readme https://github.com/ockam-network/ockam/tree/develop/document...
PLG88 5y ago
Here open source, general purpose version for access to any cloud or resource in fact. This is specifically for SSH but can support any protocol or app with the vanilla SDKs and tunnelers - https://ziti.dev/blog/zitifying-ssh/ https://ziti.dev/blog/zitifying-ssh/

Ruhrbaron 5y ago

Would be nice to have a Mac version of that.

PLG88 5y ago
See PLG88 comment above, this can work for MAC - https://apps.apple.com/app/id1460484572 https://apps.apple.com/app/id1460484572

nickysielicki 5y ago

I'm sorry if this sounds completely ridiculous to some people, but what do people use RDP/Windows server for in 2021? Given that ASP/Dotnet is portable to linux, what are people building that isn't better deployed to linux? It can't just be the legacy use-case, can it?

cprecioso 5y ago
I use Windows VMs on Google Cloud to install and play games in the cloud - my Mac can’t play any
- crakenzak 5y ago
  hows the performance/input lag on that? How much are you paying & is it worth?
  - cprecioso 5y ago
    It really depends on how far away from the data center you are. I used to live close to Amsterdam and that felt basically local, even with a not-amazing internet connection. Right now I live in the Valencia (Spain), with a local and not very well paired ISP. There is lag, but it is just about acceptable for platformers and action/adventure games. I wouldn't recommend it for shooters. The cost ended up being around 1€/hr when the instance was on, and 0.06€/day for the storage. I am sure this could be optimized (e.g. use snapshots and spot instance) - but I just couldn't be bothered to. For me it is worth it.
- yoz-y 5y ago
  This is a very novel use case to me. Could you elaborate why use this solution instead of a local VM, boot camp, or something like Stadia? Seems that it would work quite well for turn based games (including auto save)
  - cprecioso 5y ago
    I don't use a local VM or bootcamp because I just don't have a powerful enough computer to run AAA games from the last 5 years or so - even earlier if the game is badly optimized. Now with an M1 they're not even compatible. Stadia, GeForce NOW, Xbox Game Streaming and PlayStation Now: I've tried them all, and GeForce NOW specifically is very good. But, they only allow me to play a subset of my large pre-existing game library, and/or make me buy games again, which I'm not interested in. Stadia is specially bad at this, having to purchase on a game-by-game basis, not being compatible with any existing ecosystem (Steam, PS, Xbox have cloud saves and friend lists), and then trusting Google to run my game forever on their servers after paying a one-time cost.

pledess 5y ago

This sounds like a deal breaker for some use cases: https://github.com/GoogleCloudPlatform/iap-desktop/wiki/Troubleshooting https://github.com/GoogleCloudPlatform/iap-desktop/wiki/Trou... "Because of the way IAP Desktop tunnels RDP connections, it always uses NTLM for authentication and can't use Kerberos." There may be environments that lose the security benefits of Kerberos over NTLMv2 (e.g., mutual authentication) because they've been forced into a new compliance mandate that dictates adoption of Zero Trust in all available contexts.

dvdkon 5y ago
Looks like Microsoft's decision to go with increasingly elaborate challenge-response schemes instead of properly encrypting the whole connection (like SSL/SSH) will be haunting us for a while yet. I don't understand why RDP/SMB/... with plaintext auth over SSL hasn't been a thing for at least a decade, does Microsoft just not care about transport security?
- zaphirplane 5y ago
  Isn’t it plausible that an interactive gui over ssl didn’t perform well specially for VMs or the affect of the renegotiation
  - franga2000 5y ago
    RDP over SSH already performs very well, so any in-protocol implementation would only be faster (less overhead).

thowawaypets 5y ago

Why are we still building tools to hand manage VMs in 2021? Am I missing something or is this for raising pets instead of heading cattle[0]? 0. http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ http://cloudscaling.com/blog/cloud-computing/the-history-of-...

pjmlp 5y ago
Because not everyone is on the latest fashion.
e12e 5y ago
Farmers keep both sheep dogs and sheep...
corty 5y ago
Because when your herd of cattle is sick, you need to grab one and have a vet look at it before your whole herd dies from the plague.
- thowawaypets 5y ago
  No, part of managing a heard is having the right tools in place. Like monitoring, logging, and observability tools. There is nothing I can learn from accessing a VM in production that I can't learn from my monitoring system. In prod where I work, if someone logs into a production VM we mark it tainted and replace it with a fresh instance. This keeps things nice and consistent. Of you need an interactive session on a prod machine you are missing tools.
  - verdverm 5y ago
    What about cloud based developer VMs?
  - corty 5y ago
    Plugging all our cattle into a heart- and bloodpressure monitor and doing frequent blood draws from every cow "just in case" is wasteful and unnecessary. There is a balance between sensible general always-available monitoring and special-case-debugging a problem. My rule for that is: more than once a year or more than 6h? Automate and tool it. Less? SSH or other special-case tools are fine.

e12e 5y ago

> IAP Desktop is a Windows application that allows you to manage multiple Remote Desktop and SSH connections to VM instances that run on Google Cloud. Is there a Linux client too?

technological 5y ago

Is this similar to AWS workspaces ?

hardwaresofton 5y ago

Inspired by Fly.io's post a while ago[0] I also did something similar to this on my small k8s cluster with the help of stunnel, sslh, and traefik[1]. Weirdly enough I thought this was the ability to provision a wireguard-esque proxy to any machine you want, operated at the edge of the cloud, but it seems like it's really TCP-over-HTTPS. It's easy to imagine doing the former (dynamic wireguard proxy surfacing) too though -- wireguard sidecar container with shared network namespace with the workload in question + open-to-the-world port somewhere and you'd theoretically have access to any port you wanted on said machine as well. Feels like an easy set up to trust as wireguard is pretty reliable/sound. [0]: https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ https://fly.io/blog/ssh-and-user-mode-ip-wireguard/ [1]: https://vadosware.io/post/stuffing-both-ssh-and-https-on-port-443-with-stunnel-ssh-and-traefik/ https://vadosware.io/post/stuffing-both-ssh-and-https-on-por...

chucky_z 5y ago
I run IAP. It's TCP-over-HTTPS but it works remarkably well, connects to all kinds of things, and for users it really is just "login with google, proceed as normal." I use a JWT proxy + ghostunnel within GKE with a VIP so it's not quite their reference setup but it's extremely "just works" outside GKE being weird and eating its own routes. BTW, side-note but try out ghostunnel over stunnel! I've really enjoyed using it and it's been fantastic to debug and work with.
- hardwaresofton 5y ago
  > I run IAP. It's TCP-over-HTTPS but it works remarkably well, connects to all kinds of things, and for users it really is just "login with google, proceed as normal." Yeah that's really amazing, with client-side software like they've already made and I've seen from other vendors (whether GUI or TUI) the interfaces IaaS/PaaS companies can build are really slick. Looks like they'll be able to cut down on dashboard fatigue/complexity people are wrangling quite a bit. > I use a JWT proxy + ghostunnel within GKE with a VIP so it's not quite their reference setup but it's extremely "just works" outside GKE being weird and eating its own routes. Interesting, so JWT proxy (or any other auth mechanism that is viable over HTTPS) -> ghostunnel machine w/ public VIP -> Target machine ? Or ghostunnel directly running on the Target machine which holds the public VIP? Or does the JWT proxy take the public IP and the ghostunnel machine keep the private VIP? Apologies just want to be able to picture your solution clearly. > BTW, side-note but try out ghostunnel over stunnel! I've really enjoyed using it and it's been fantastic to debug and work with. Thanks for the recommendation of ghostunnel, will use it in the future over stunnel next time I hack together something like this. BTW: super-side note, breath of fire III avatar was a blast from the past, instantly recognized it.

deleted 5y ago

[deleted]

floatingatoll 5y ago

> IAP Desktop is an open-source project and not an officially supported Google product.

ngcc_hk 5y ago

Amazing. Obviously need audit. But so far so good.