12 ms·

Remote access to production infrastructure (death to the VPN)

236 points by sullivanmatt 7y ago

throwaway3157 7y ago
Hey Matt. I appreciate images in articles, but GIFs are very distracting while trying to read. May I suggest static images next time?
- deleted 7y ago
  [deleted]
- lolc 7y ago
  I second this. I had to zoom way in so I could scroll between the animations for undisturbed reading.
- low_key 7y ago
  This article actually got me to go track down the firefox pref "image.animation_mode". "none" is a very nice choice.
- sullivanmatt 7y ago
  Thanks all for your feedback. I have removed the images to improve readability, especially for mobile users. The post was originally written for an internal blog where we have a GIF-heavy communication culture, and I probably should have cleaned it up a bit more for general public consumption.
fulafel 7y ago
The critique against VPNs is exactly right, they're such garbage compared to the standard we otherwise hold SSH, TLS etc to, and the access granularity is too wide, and there's no transparency on how wide the access is configured from VPNs. And they're very often on the wrong side of the it dept vs devops responsibility split so often misconfigured.
- jschwartzi 7y ago
  The biggest issue I have with replacing VPNs is in server management, but not the way the author is talking. My job entails doing software development for over 1000 devices that are all fielded behind enterprise firewalls, and the PCI compliance requirements dictate that no unnecessary access be provided into those firewalls. What this means in practice is that no connections may be established which originate from a device on the internet to a device behind the firewall. We don't control the firewall as it is under the control of our customers. What we used VPN for is allowing us to establish SSH connections to the equipment. I would really, really like a low-resource mechanism to replace this but everyone wants to deploy their solution in a 90+ megabyte Docker container, or a Snap, which is about 1.5 times as large as the entire Linux system image for our oldest equipment. So these are great solutions for when you control the entire network path from the server(or when you are using a server!) to the Internet including the firewall, but they suck terribly for eliminating VPN in cases where you can't just open an inbound port on the firewall. As it is I'm trying to figure out how to configure an OpenSSH client to punch out through the firewall to an OpenSSH server, then immediately turn around and provide a shell to the server. This seems to be entirely contradictory to how OpenSSH is designed, but I'm hopeful I can hack something together.
  - xorcist 7y ago
    Are you just looking for "ssh -fNT -R 10022:127.0.0.1:22 remote"? If so, then that's not as much a hack as a pretty standard reverse ssh.
  - sk5t 7y ago
    Does a VPN afford your customers belief in the fiction that the source device is no longer on the internet?
  - jiveturkey 7y ago
    > We don't control the firewall as it is under the control of our customers. > As it is I'm trying to figure out how to configure an OpenSSH client to punch out through the firewall to an OpenSSH server, then immediately turn around and provide a shell to the server. This seems to be entirely contradictory to how OpenSSH is designed, but I'm hopeful I can hack something together. This is trivial. But if you don't control the firewall, how will you get the outbound SSH access? PCI requires that both inbound and outbound traffic from the secure zone (CDE) be controlled. If you can impose upon the customer that they punch an outbound hole, you can impose inbound requirements as well. Your inbound connection does not come from "the public internet", it comes from your managed in-scope network.
nurettin 7y ago
> One of my biggest pet peeves about VPNs is that they hijack all your network traffic. They can be configured not to, but our customers and security controls like NIST 800-53 SC-7(7) typically require that they do. VPN is dead because some customers want you to route the internet interfaces of all machines through the VPN server. How does this even make any sense?
- sullivanmatt 7y ago
  As I alluded to in the post, it's a legacy viewpoint. These customers hand you a 300+ security questionnaire that hasn't been updated in 10 years. When you tick the 'VPN' boxes, alarms sound, but they are thinking about the term 'VPN' in a different context, like employees accessing a network file share. But what we really have is a Bastion host (aka jump box), which is fundamentally different. By not saying you run VPN software, the conversations shift significantly, especially when dealing with the F100 banks and the like that may not be as familiar with modern cloud architectures.
- whatthesmack 7y ago
  Split tunneling is the answer here. Many folks configure their VPN solutions that way. That's exactly what I do at my present employer... traffic meant for VPN goes over VPN, and everything else gets routed through the client's internet connection.
  - sullivanmatt 7y ago
    The control I listed, NIST 800-53 SC-7(7) [which is a part of the FedRAMP Moderate suite of controls], specifically requires you implement a technical control such that your users cannot split tunnel.
    - whatthesmack 7y ago
      Apologies... didn't read into the NIST requirement. Out of curiosity, what do you normally implement that meets that requirement? Forcing all traffic when there is no security benefit (what is the advantage of getting to https://news.ycombinator.com https://news.ycombinator.com through the VPN?) seems ripe for a compensating control.
Trias11 7y ago
Agree on VPNs. They need to die. Let elect ZeroTier to be the president of remote, secure access :)
- jedieaston 7y ago
  Nebula is very nice too, licensed MIT and has DNS support, something ZeroTier still hasn’t added outside of the ztdns server that someone wrote that I never did get to work properly... hopefully ZeroTier makes some strides in 2.0.
  - jyrkesh 7y ago
    Oh my god, thank you for mentioning this project. I just deployed ZeroTier for some local infra, but in doing investigation, I was desparately trying to find the Nebula project name and GitHub. I was looking for "beacon", "lighthouse", "bastion", "ZeroTier compete" and everything in between. I even knew it created at some prominent tech company (kept thinking Netflix, of course it was Slack). I was starting to think that I'd concocted a false memory and it didn't really exist. So now I gotta go decide if I want to rip out all my existing ZT infra or not.
emperor_ 7y ago
Interesting read! A couple of days ago I setup a Cloudflare’s access with argotunnels and a CA. A very cool beyondcorp service which looks similar to okta.
whatthesmack 7y ago
This doesn't make any sense. The people that make Okta ASA (formerly ScaleFT) -- Okta themselves -- must VPN into production. I wouldn't use ASA or the like without VPN if the people that make it won't even do that. Besides... it's an extra layer of security. Don't have the requisite certificate, username-password combo, and additional factor? Then you can't even get to port 22 to start with.
- samcat116 7y ago
  How do you know they don’t use it for production?
meowface 7y ago
Any suggestions for a good inexpensive or open source zero trust auth solution that supports both HTTP and SSH? I considered Cloudflare Access, but you need to pay extra for Argo Tunnel if you want it to work with SSH. The main open source option I'm aware of now with support is Pritunl Zero. Was going to actually stand that up today before I read the article.
- johnmarcus 7y ago
  I love Pritunl. Have been using it for 5 years. It's super easy to setup, maybe takes an hour the first time and like 15 minutes once you have done it.
- jyrkesh 7y ago
  Not sure if you need SSO (it doesn't do it), but if you can bootstrap with a cert or key of some kind, I've been loving Zerotier.
- jiveturkey 7y ago
  CFA is not ZT. It is "simply" moving the auth point to the CF gateway. It's still a VPN (or bastion, if you will). ZT is when you move [strong] authn and authz to the endpoint itself.
exabrial 7y ago
Pure Zero Trust is just as ridiculous as using Pure-VPN-around-a-garden. The first gives an attacker unlimited retries, and the second gives an attacker full system access once they breach the outer wall. The correct solution is somewhere in the middle: block everything by default to get you to an inner courtyard, where the zero trust model is deployed... (which ironically he suggests by deploying port knocking (port knocking is a bad idea (TCP/UDP ports are sent in the clear and the "key" is never rotated))) The best model is probably "block everything" by default, then allow access to the inner courtyard via a VPN, where then the Zero Trust model is deployed. You remove the ability for an attacker to have unlimited retries, but access to resources still requires individual authentication.
- bronco21016 7y ago
  Why does pure zero trust give unlimited retries? If you use rate limiting, good password policy, and strong 2FA then most motivated attackers are going to seek a different entrance.
  - nitrogen 7y ago
    The vulnerability in your stack may be before your limits/policies are checked.
  - exabrial 7y ago
    Rate limiting across distributed systems is a notoriously hard problem to solve.
- packetslave 7y ago
  He doesn't ACTUALLY suggest port knocking, just the concept ("do something in order to open a firewall hole"). The proposed solution using Lambda w/ 2FA is actually pretty cool.
  - exabrial 7y ago
    Correct, but his solution is a hair improvement at best over port knocking. A VPN means an individual connection is authorized into the interior courtyard. A Lambda with 2FA to whitelist an IP, then a cron job to cleanup means everyone at your local cafe wireless access point is also authorized into the inner courtyard.
bob1029 7y ago
In my experience, the best way to eliminate the VPN is to expose your various internal business services as websites w/ TLS1.2 & multi-factor authentication. Obviously, this isn't practical for everything. But, if the thing you were using VPN for is already a web application, you are basically halfway there. Ideally, you just directly expose a secure web application to clients, but in some cases (i.e. very old legacy systems) you probably want to put an nginx box in front and then put the authentication at that level. Web access has a huge range of benefits. Users are scoped directly to the system of concern rather than an entire network of hosts. You can take security to the next level with server-side rendering of web content in order to avoid additional required channels of communication or revealing of implementation secrets to the client (e.g. SPA client source). We are at a point of placing our actual application servers directly on the public internet (with TLS1.2/MFA/ACLs/etc). Hiding behind VPNs or layers of reverse proxies seems to cause more harm than good.
- cameronbrown 7y ago
  > Obviously, this isn't practical for everything If you have the engineering resources to back it up, it definitely can be. Internal services at Google usually trust the office network the same as any other -- well documented in the BeyondCorp paper if you're interested.
  - LinuxBender 7y ago
    My understanding based on the folks I know at Google is that BeyondCorp paper was a PoC that was implemented in part of their corp network, that is called Production, not to be confused with the production network that hosts their search site. That network still requires a VPN and a hardened Linux laptop to access. Not every service has been modified to implement the RPC calls / authenticated protobuf code changes. Someone at Google please feel free to correct me on this.
    - asdfasgasdgasdg 7y ago
      BeyondCorp is not just a proof of concept. Everything at Google is accessed through it, including production-production (through a proxy maybe? Not sure the details). You're right about the requirement to have a hardened device -- which acts sort of like a token (as described in the BeyondCorp whitepapers). But it can be Windows, Linux, Mac, Chromebook, Android or iPhone. I never use VPN and I work on production stuff from outside the office all the time. (FWIW, I don't think there's anything secret here. This stuff is very explicitly described in the whitepapers.)
nimbusblack 7y ago
This is an apples to orange comparison. Most network admins are lazy and just provide full L3 access so they don't have deal with any access issues. Most users use VPNs as an access mechanism rather than to secure anything. Most vendors you mentioned in the article can also control access at application level like ssh.
kodablah 7y ago
Pardon my ignorance, so this updates a security group (or multiple) which presumably have access to several internal things? One wonders if you can take it a step further and for web-based services (i.e. doesn't apply to SSH access), at the conclusion of the authentication, it updates the security group for just that webapp. With how e.g. oauth2 SSO automatically auths via redirects, if the update to the security group is atomic/fast, access can be given one webapp a time. Also, are there any concerns about IP timeout vs explicit VPN disconnect? Obviously the latter works better in shared environments (e.g. shared terminals, wifi's that reuse IPs frequently, large NATs that have many devices behind a single IP).
- sullivanmatt 7y ago
  There is one network entry point per deployment of our app infrastructure (eg. US and EU deploys), so the lambda does go and update both security groups simultaneously to allow the requestor's IP to hit either if they would like. If you wanted to, you could certainly make it more fine-grained than that, but the goal was simply to cut off the majority of the Internet from these mechanisms as an extra protection layer. There are all the other protection mechanisms (e.g. the mutual certs) to actually protect and authenticate the connection itself. For web apps, we simply front using an OAuth2-aware proxy. Back in the day, we used this: https://mattslifebytes.com/2018/08/07/protecting-internal-applications-with-a-saml-aware-reverse-proxy-a-tutorial/ https://mattslifebytes.com/2018/08/07/protecting-internal-ap... Now, we utilize Kubernetes for hosting most production internal apps, so we run the oauth2-proxy Helm chart (https://github.com/helm/charts/tree/master/stable/oauth2-proxy https://github.com/helm/charts/tree/master/stable/oauth2-pro...) to handle verification of identity before sending traffic back to its destination service. Conceptually similar, as auth has to be completed before the request is sent to the back-end.
bdesimone 7y ago
If you are interested in BeyondCorp-style access, I put together a collection of curated resources. https://github.com/pomerium/awesome-zero-trust https://github.com/pomerium/awesome-zero-trust PRs welcome.
irl_ 7y ago
If you wanted to do this "enterprise port knocking" on OpenBSD, you could use pf to do this. https://www.openbsd.org/faq/pf/authpf.html https://www.openbsd.org/faq/pf/authpf.html The FAQ entry is about building an authenticated gateway, but the same technique can be applied to open individual ports.
kylek 7y ago
AWS Systems Manager provides a neat solution to do this [0], permission would be managed via IAM. [0] https://github.com/elpy1/ssh-over-ssm https://github.com/elpy1/ssh-over-ssm <-- not made by me, but a good example
sscarduzio 7y ago
Authenticating the source IP address on the fly (as detected from the browser) is definitely not the way to go for many reasons: 1. With NAT and metropolitan area networks, hundreds of thousands of devices could share the same public IP. 2. Large networks with many devices often connect to the public network through trunking (load balance the connections through multiple routers), so the HTTP connection between OKTA and my browser can VERY well originate from a different IP address than my SSH session, and I would never be able to connect. 3. Many devices are mobile, and they can change their IP address when they pass from WiFi to LTE for example. This would force an unnecessary re-auth.
- sullivanmatt 7y ago
  I want to be clear about the use case that the enterprise port knocking solution is trying to solve: it's an additional control that would not normally even be in place. In most setups, you are exposing some sort of relay to the internet, through which your users can access the services after authentication - such as an SSH bastion host or a VPN. The IP based whitelisting mechanism is simply a layer to allow you to not have to compromise and expose anything to the world wide internet. The actual authentication and authorization mechanism is the certificate-based set of SSH connections.
- BlueTemplar 7y ago
  NAT and single IP adresses for multiple users are going away with IPv6 ?
  - zamadatix 7y ago
    Eventually, for now v4 CGNAT exists too though.
johnmarcus 7y ago
Meh, this sounds like it's paid for by okta. It also doesn't cover the real use case scenario of vpns - non-technical folks need to access internal services. What's presented is a reasonable approach for ssh control. Oh, and I'm pretty sure Cloud Passage has a port-knocking based solution in the real world which also gives 2fa access for ssh.
- sullivanmatt 7y ago
  I have no way to prove to you that I am not some paid shill :) , but I have no relationship with Okta outside of being a customer through my employer, and I was not compensated or gifted anything for the creation of my post. I blog about things that I encounter at work and find interesting. That happens to often be a cross-section of infrastructure and identity!
  - breput 7y ago
    I can verify that he isn't an Okta shill but he also glosses over some of the problems and limitations that I've experienced with the ScaleFT product compared to our co-existing OpenVPN solution. We have used multiple OpenVPN servers with password protected cerificates and TOTP. Even if someone were to obtain access to my credentials and certs, they wouldn't be able to access the production services without also obtaining access to the authenticator device. Once your machine is enrolled in ScaleFT and while you're authenticated with your identity provider, malware or just a malicious coworker could access the production services with a single command line. There are upsides to ScaleFT as well, though. As long as you're all in on Okta or can federate with it, user management is a no brainer. And having the IdP integration is much more user (and malware) friendly and is likely more reliable for server to server use compared to OpenVPN. Limiting access to particular services is likely easier, too. Downsides with this product include having all sorts of reoccuring configuration problems where a server just disappears from the list of available services, which requires ops involement to restore access. If you're using macOS and RDP (I just outed myself to Matt...) you have to use the sub-par FreeRDP client. And ultimately you're tunnelling TCP over TCP, which works ok in the office but which might not always work as well in mobile or higher latency network situations.
ex_amazon_sde 7y ago
> Amazon Linux 2-based EC2 instances, meaning the attack surface is extraordinarily small WHAT?
BlueTemplar 7y ago
Hmm, static IP suffixes are a thing of the past anyway with IPv6 ? "Tens of millions" ? More like quintillions, and that's for a single IP prefix more typical of a home connection !
- sullivanmatt 7y ago
  Sorry, the tens of millions of IP addresses is referencing the IP space of AWS (one of which these network access points occupies at any given point in time).
mleonhard 7y ago
TLDR: Use a complicated SSH proxy instead of a VPN. This has some serious downsides for non-SSH applications. For example, to connect to a production database cluster, one would need to ssh through the proxy to a bastion host, and then set up port forwarding from the bastion host to the database. Setting up a simple database connection now requires shell access to a production server. This is less secure and more complex than using a traditional VPN.
- sullivanmatt 7y ago
  A great point. It does depend on your use case, and your dependence on manual operations. For our organization, almost all database interactions and maintenance are performed in code; if somebody is connecting manually, something pretty bad has happened. So for us, we are not really impacted by having to perform port forwarding like this on rare occasion. I completely agree that it could be much more impactful to other organizations. I'm curious: why is utilizing port forwarding over these mutually authenticated SSH tunnels less secure than employing a VPN? From my perspective, port forwarding still adds a level of intentionality which reduces the likelihood of an incident/accident.
  - mleonhard 7y ago
    Good VPNs are mutually authenticated. Intentionality is good, but in your example it comes at a cost of complexity. Simplicity is paramount for security. If intentionality is desired, one can use per-server VPNs.
forkexec 7y ago
No, no, no. The whole point of a VPN or SSH jumpbox is to airgap critical infrastructure with unknown vulnerabilities behind a hardened point of access. Putting production infrastructure on the public internet is beyond idiotic and regressive, and an invitation to be hacked by an unlimited and unknown number of exploits. It took forever to get departmental firewalls at a big name university where I worked because systems put in before my time like nutrition/meal planner, housing lottery draw, facilities management system and retail POS systems were getting owned left-and-right by remote malware. I'll keep using fwknop-protected OpenSSH on OpenBSD and WireGuard, others can do whatever they want without thinking about the security vs. convenience.
- jjoonathan 7y ago
  So, they have drawn their lines of defense in a position you are not used to, and therefore they are beyond idiotic and regressive? Really?
  - forkexec 7y ago
    Yeap. There is a night and day difference in attack surfaces between isolating access to a single (or HA pair) jumpbox and N boxes on the internet with no real DMZ or private admin network. Feelings and fashions don't make stupid configurations better. If you have a problem with honest opinions from someone with 25 years of experience, I think you need thicker skin or I can choose to simply not comment and let stupid fashions propagate.
    - sullivanmatt 7y ago
      I wish you had come into this discussion with constructive criticism, instead of simply swinging a hammer. I, for one, am happy to learn from somebody with a number of years of experience. However showing up on a thread and spewing negativity and name calling isn't a great way to earn respect in this industry.
    - icedchai 7y ago
      Yep. Putting everything directly on the public Internet is 90's style. I remember it well. Whole offices with public IP addresses. No firewall. It's amazing anyone ever considered this sane, but it was a different time.
neurobashing 7y ago
Does anyone happen to know what this costs? They’re predictably quiet about it.
cordite 7y ago
I'm surprised to see no mention of Mutual TLS (MTLS), PIV cards, or the like
- bb611 7y ago
  I believe in this case he's talking about MTLS: > OASA also protects these hops by issuing client certificates with 10-minute expirations after first verifying your identity through our single sign-on provider, and then also verifying you are on a pre-enrolled (and approved) trusted company device.
  - sullivanmatt 7y ago
    Basically. Since we are focusing on SSH in this post (and keep in mind that SSH is its own protocol, separate from TLS), it's conceptually the same: client has a certificate and a key, signed by a trusted certificate authority, and the client is also in possession of the server certificate authority. So then you have a bi-directional trust established. The certificates are short-lived, and issued after a successful authentication + dial request to the OASA service.
brentis 7y ago
I work in this space who has a commercial offering. Ill just leave this CSA spec here for reference. Please also look up SPA - newer than port knocking, but based on same premise. https://downloads.cloudsecurityalliance.org/initiatives/sdp/SDP_Specification_1.0.pdf https://downloads.cloudsecurityalliance.org/initiatives/sdp/...
- fulafel 7y ago
  This talks about air-gapped networks?
justlexi93 7y ago
In 99.95% of cases, VPNs are set up to: Bridge a network device – such as a laptop or even another server … into a larger network of servers – such as in the cloud or on-prem … across the Internet – protected with an additional layer of encryption
theptip 7y ago
If you're in GCP, then you can use their Identity Aware Proxy to achieve most of this. (https://cloud.google.com/iap/docs https://cloud.google.com/iap/docs) IAP supports HTTP and TCP connections, so you can put it in front of your website (say an internal admin webapp), or use it to tunnel SSH onto a machine that doesn't have a public IP, using your IAM roles. If you're running Kubernetes in GKE, you can also wire IAP up to an Ingress, to protect any TCP/HTTP services you have in your k8s cluster. This one is a bit tricky to configure, but is very nice once you have it up and running.
znpy 7y ago
I've recently started using socks proxies via ssh and I was surprised about how far you can go with such solution. Unless you want permanent connection with routing and everything, ssh socks proxy work awesomely. Point a firefox profile to use it and you can really act as if you were in a different subnet: it can proxy dns resolution too.
peterwwillis 7y ago
Mainly this is a poor idea because of the whole allowing-an-IP-thing. Do not rely on IPs or ports for security, ever. Ever. Evereverevereverever. They are not secure. They do not contribute to security. Do not make your security dependent on them. Ever. Ok? Thanks. AWS already has a vastly superior solution for this, called AWS Systems Manager Session Manager (it's quite a mouthful). You create a session with AWS using a federated login service (SAML-based SSO) and then craft IAM policies to allow a single user to ssh into a single server, over the AWS API. Not only will this be more secure, you don't have to maintain a wacky custom solution. Logging into servers is an anti-pattern, and wherever possible you should be running away from it. Get metrics out of the server and analyze them, run commands remotely using some kind of persistent system agent, stop storing state on your servers. I know this is not the point of the article, but I want to remind people of it so they can can avoid the ssh trap early.

236 points by sullivanmatt 7y ago

Hey Matt. I appreciate images in articles, but GIFs are very distracting while trying to read. May I suggest static images next time?

deleted 7y ago
[deleted]
lolc 7y ago
I second this. I had to zoom way in so I could scroll between the animations for undisturbed reading.
low_key 7y ago
This article actually got me to go track down the firefox pref "image.animation_mode". "none" is a very nice choice.
sullivanmatt 7y ago
Thanks all for your feedback. I have removed the images to improve readability, especially for mobile users. The post was originally written for an internal blog where we have a GIF-heavy communication culture, and I probably should have cleaned it up a bit more for general public consumption.

fulafel 7y ago

The critique against VPNs is exactly right, they're such garbage compared to the standard we otherwise hold SSH, TLS etc to, and the access granularity is too wide, and there's no transparency on how wide the access is configured from VPNs. And they're very often on the wrong side of the it dept vs devops responsibility split so often misconfigured.

jschwartzi 7y ago
The biggest issue I have with replacing VPNs is in server management, but not the way the author is talking. My job entails doing software development for over 1000 devices that are all fielded behind enterprise firewalls, and the PCI compliance requirements dictate that no unnecessary access be provided into those firewalls. What this means in practice is that no connections may be established which originate from a device on the internet to a device behind the firewall. We don't control the firewall as it is under the control of our customers. What we used VPN for is allowing us to establish SSH connections to the equipment. I would really, really like a low-resource mechanism to replace this but everyone wants to deploy their solution in a 90+ megabyte Docker container, or a Snap, which is about 1.5 times as large as the entire Linux system image for our oldest equipment. So these are great solutions for when you control the entire network path from the server(or when you are using a server!) to the Internet including the firewall, but they suck terribly for eliminating VPN in cases where you can't just open an inbound port on the firewall. As it is I'm trying to figure out how to configure an OpenSSH client to punch out through the firewall to an OpenSSH server, then immediately turn around and provide a shell to the server. This seems to be entirely contradictory to how OpenSSH is designed, but I'm hopeful I can hack something together.
- xorcist 7y ago
  Are you just looking for "ssh -fNT -R 10022:127.0.0.1:22 remote"? If so, then that's not as much a hack as a pretty standard reverse ssh.
- sk5t 7y ago
  Does a VPN afford your customers belief in the fiction that the source device is no longer on the internet?
- jiveturkey 7y ago
  > We don't control the firewall as it is under the control of our customers. > As it is I'm trying to figure out how to configure an OpenSSH client to punch out through the firewall to an OpenSSH server, then immediately turn around and provide a shell to the server. This seems to be entirely contradictory to how OpenSSH is designed, but I'm hopeful I can hack something together. This is trivial. But if you don't control the firewall, how will you get the outbound SSH access? PCI requires that both inbound and outbound traffic from the secure zone (CDE) be controlled. If you can impose upon the customer that they punch an outbound hole, you can impose inbound requirements as well. Your inbound connection does not come from "the public internet", it comes from your managed in-scope network.

nurettin 7y ago

> One of my biggest pet peeves about VPNs is that they hijack all your network traffic. They can be configured not to, but our customers and security controls like NIST 800-53 SC-7(7) typically require that they do. VPN is dead because some customers want you to route the internet interfaces of all machines through the VPN server. How does this even make any sense?

sullivanmatt 7y ago
As I alluded to in the post, it's a legacy viewpoint. These customers hand you a 300+ security questionnaire that hasn't been updated in 10 years. When you tick the 'VPN' boxes, alarms sound, but they are thinking about the term 'VPN' in a different context, like employees accessing a network file share. But what we really have is a Bastion host (aka jump box), which is fundamentally different. By not saying you run VPN software, the conversations shift significantly, especially when dealing with the F100 banks and the like that may not be as familiar with modern cloud architectures.
whatthesmack 7y ago
Split tunneling is the answer here. Many folks configure their VPN solutions that way. That's exactly what I do at my present employer... traffic meant for VPN goes over VPN, and everything else gets routed through the client's internet connection.
- sullivanmatt 7y ago
  The control I listed, NIST 800-53 SC-7(7) [which is a part of the FedRAMP Moderate suite of controls], specifically requires you implement a technical control such that your users cannot split tunnel.
  - whatthesmack 7y ago
    Apologies... didn't read into the NIST requirement. Out of curiosity, what do you normally implement that meets that requirement? Forcing all traffic when there is no security benefit (what is the advantage of getting to https://news.ycombinator.com https://news.ycombinator.com through the VPN?) seems ripe for a compensating control.

Trias11 7y ago

Agree on VPNs. They need to die. Let elect ZeroTier to be the president of remote, secure access :)

jedieaston 7y ago
Nebula is very nice too, licensed MIT and has DNS support, something ZeroTier still hasn’t added outside of the ztdns server that someone wrote that I never did get to work properly... hopefully ZeroTier makes some strides in 2.0.
- jyrkesh 7y ago
  Oh my god, thank you for mentioning this project. I just deployed ZeroTier for some local infra, but in doing investigation, I was desparately trying to find the Nebula project name and GitHub. I was looking for "beacon", "lighthouse", "bastion", "ZeroTier compete" and everything in between. I even knew it created at some prominent tech company (kept thinking Netflix, of course it was Slack). I was starting to think that I'd concocted a false memory and it didn't really exist. So now I gotta go decide if I want to rip out all my existing ZT infra or not.

emperor_ 7y ago

Interesting read! A couple of days ago I setup a Cloudflare’s access with argotunnels and a CA. A very cool beyondcorp service which looks similar to okta.

whatthesmack 7y ago

This doesn't make any sense. The people that make Okta ASA (formerly ScaleFT) -- Okta themselves -- must VPN into production. I wouldn't use ASA or the like without VPN if the people that make it won't even do that. Besides... it's an extra layer of security. Don't have the requisite certificate, username-password combo, and additional factor? Then you can't even get to port 22 to start with.

samcat116 7y ago
How do you know they don’t use it for production?

meowface 7y ago

Any suggestions for a good inexpensive or open source zero trust auth solution that supports both HTTP and SSH? I considered Cloudflare Access, but you need to pay extra for Argo Tunnel if you want it to work with SSH. The main open source option I'm aware of now with support is Pritunl Zero. Was going to actually stand that up today before I read the article.

johnmarcus 7y ago
I love Pritunl. Have been using it for 5 years. It's super easy to setup, maybe takes an hour the first time and like 15 minutes once you have done it.
jyrkesh 7y ago
Not sure if you need SSO (it doesn't do it), but if you can bootstrap with a cert or key of some kind, I've been loving Zerotier.
jiveturkey 7y ago
CFA is not ZT. It is "simply" moving the auth point to the CF gateway. It's still a VPN (or bastion, if you will). ZT is when you move [strong] authn and authz to the endpoint itself.

exabrial 7y ago

Pure Zero Trust is just as ridiculous as using Pure-VPN-around-a-garden. The first gives an attacker unlimited retries, and the second gives an attacker full system access once they breach the outer wall. The correct solution is somewhere in the middle: block everything by default to get you to an inner courtyard, where the zero trust model is deployed... (which ironically he suggests by deploying port knocking (port knocking is a bad idea (TCP/UDP ports are sent in the clear and the "key" is never rotated))) The best model is probably "block everything" by default, then allow access to the inner courtyard via a VPN, where then the Zero Trust model is deployed. You remove the ability for an attacker to have unlimited retries, but access to resources still requires individual authentication.

bronco21016 7y ago
Why does pure zero trust give unlimited retries? If you use rate limiting, good password policy, and strong 2FA then most motivated attackers are going to seek a different entrance.
- nitrogen 7y ago
  The vulnerability in your stack may be before your limits/policies are checked.
- exabrial 7y ago
  Rate limiting across distributed systems is a notoriously hard problem to solve.
packetslave 7y ago
He doesn't ACTUALLY suggest port knocking, just the concept ("do something in order to open a firewall hole"). The proposed solution using Lambda w/ 2FA is actually pretty cool.
- exabrial 7y ago
  Correct, but his solution is a hair improvement at best over port knocking. A VPN means an individual connection is authorized into the interior courtyard. A Lambda with 2FA to whitelist an IP, then a cron job to cleanup means everyone at your local cafe wireless access point is also authorized into the inner courtyard.

bob1029 7y ago

In my experience, the best way to eliminate the VPN is to expose your various internal business services as websites w/ TLS1.2 & multi-factor authentication. Obviously, this isn't practical for everything. But, if the thing you were using VPN for is already a web application, you are basically halfway there. Ideally, you just directly expose a secure web application to clients, but in some cases (i.e. very old legacy systems) you probably want to put an nginx box in front and then put the authentication at that level. Web access has a huge range of benefits. Users are scoped directly to the system of concern rather than an entire network of hosts. You can take security to the next level with server-side rendering of web content in order to avoid additional required channels of communication or revealing of implementation secrets to the client (e.g. SPA client source). We are at a point of placing our actual application servers directly on the public internet (with TLS1.2/MFA/ACLs/etc). Hiding behind VPNs or layers of reverse proxies seems to cause more harm than good.

cameronbrown 7y ago
> Obviously, this isn't practical for everything If you have the engineering resources to back it up, it definitely can be. Internal services at Google usually trust the office network the same as any other -- well documented in the BeyondCorp paper if you're interested.
- LinuxBender 7y ago
  My understanding based on the folks I know at Google is that BeyondCorp paper was a PoC that was implemented in part of their corp network, that is called Production, not to be confused with the production network that hosts their search site. That network still requires a VPN and a hardened Linux laptop to access. Not every service has been modified to implement the RPC calls / authenticated protobuf code changes. Someone at Google please feel free to correct me on this.
  - asdfasgasdgasdg 7y ago
    BeyondCorp is not just a proof of concept. Everything at Google is accessed through it, including production-production (through a proxy maybe? Not sure the details). You're right about the requirement to have a hardened device -- which acts sort of like a token (as described in the BeyondCorp whitepapers). But it can be Windows, Linux, Mac, Chromebook, Android or iPhone. I never use VPN and I work on production stuff from outside the office all the time. (FWIW, I don't think there's anything secret here. This stuff is very explicitly described in the whitepapers.)

nimbusblack 7y ago

This is an apples to orange comparison. Most network admins are lazy and just provide full L3 access so they don't have deal with any access issues. Most users use VPNs as an access mechanism rather than to secure anything. Most vendors you mentioned in the article can also control access at application level like ssh.

kodablah 7y ago

Pardon my ignorance, so this updates a security group (or multiple) which presumably have access to several internal things? One wonders if you can take it a step further and for web-based services (i.e. doesn't apply to SSH access), at the conclusion of the authentication, it updates the security group for just that webapp. With how e.g. oauth2 SSO automatically auths via redirects, if the update to the security group is atomic/fast, access can be given one webapp a time. Also, are there any concerns about IP timeout vs explicit VPN disconnect? Obviously the latter works better in shared environments (e.g. shared terminals, wifi's that reuse IPs frequently, large NATs that have many devices behind a single IP).

sullivanmatt 7y ago
There is one network entry point per deployment of our app infrastructure (eg. US and EU deploys), so the lambda does go and update both security groups simultaneously to allow the requestor's IP to hit either if they would like. If you wanted to, you could certainly make it more fine-grained than that, but the goal was simply to cut off the majority of the Internet from these mechanisms as an extra protection layer. There are all the other protection mechanisms (e.g. the mutual certs) to actually protect and authenticate the connection itself. For web apps, we simply front using an OAuth2-aware proxy. Back in the day, we used this: https://mattslifebytes.com/2018/08/07/protecting-internal-applications-with-a-saml-aware-reverse-proxy-a-tutorial/ https://mattslifebytes.com/2018/08/07/protecting-internal-ap... Now, we utilize Kubernetes for hosting most production internal apps, so we run the oauth2-proxy Helm chart (https://github.com/helm/charts/tree/master/stable/oauth2-proxy https://github.com/helm/charts/tree/master/stable/oauth2-pro...) to handle verification of identity before sending traffic back to its destination service. Conceptually similar, as auth has to be completed before the request is sent to the back-end.

bdesimone 7y ago

If you are interested in BeyondCorp-style access, I put together a collection of curated resources. https://github.com/pomerium/awesome-zero-trust https://github.com/pomerium/awesome-zero-trust PRs welcome.

irl_ 7y ago

If you wanted to do this "enterprise port knocking" on OpenBSD, you could use pf to do this. https://www.openbsd.org/faq/pf/authpf.html https://www.openbsd.org/faq/pf/authpf.html The FAQ entry is about building an authenticated gateway, but the same technique can be applied to open individual ports.

kylek 7y ago

AWS Systems Manager provides a neat solution to do this [0], permission would be managed via IAM. [0] https://github.com/elpy1/ssh-over-ssm https://github.com/elpy1/ssh-over-ssm <-- not made by me, but a good example

sscarduzio 7y ago

Authenticating the source IP address on the fly (as detected from the browser) is definitely not the way to go for many reasons: 1. With NAT and metropolitan area networks, hundreds of thousands of devices could share the same public IP. 2. Large networks with many devices often connect to the public network through trunking (load balance the connections through multiple routers), so the HTTP connection between OKTA and my browser can VERY well originate from a different IP address than my SSH session, and I would never be able to connect. 3. Many devices are mobile, and they can change their IP address when they pass from WiFi to LTE for example. This would force an unnecessary re-auth.

sullivanmatt 7y ago
I want to be clear about the use case that the enterprise port knocking solution is trying to solve: it's an additional control that would not normally even be in place. In most setups, you are exposing some sort of relay to the internet, through which your users can access the services after authentication - such as an SSH bastion host or a VPN. The IP based whitelisting mechanism is simply a layer to allow you to not have to compromise and expose anything to the world wide internet. The actual authentication and authorization mechanism is the certificate-based set of SSH connections.
BlueTemplar 7y ago
NAT and single IP adresses for multiple users are going away with IPv6 ?
- zamadatix 7y ago
  Eventually, for now v4 CGNAT exists too though.

johnmarcus 7y ago

Meh, this sounds like it's paid for by okta. It also doesn't cover the real use case scenario of vpns - non-technical folks need to access internal services. What's presented is a reasonable approach for ssh control. Oh, and I'm pretty sure Cloud Passage has a port-knocking based solution in the real world which also gives 2fa access for ssh.

sullivanmatt 7y ago
I have no way to prove to you that I am not some paid shill :) , but I have no relationship with Okta outside of being a customer through my employer, and I was not compensated or gifted anything for the creation of my post. I blog about things that I encounter at work and find interesting. That happens to often be a cross-section of infrastructure and identity!
- breput 7y ago
  I can verify that he isn't an Okta shill but he also glosses over some of the problems and limitations that I've experienced with the ScaleFT product compared to our co-existing OpenVPN solution. We have used multiple OpenVPN servers with password protected cerificates and TOTP. Even if someone were to obtain access to my credentials and certs, they wouldn't be able to access the production services without also obtaining access to the authenticator device. Once your machine is enrolled in ScaleFT and while you're authenticated with your identity provider, malware or just a malicious coworker could access the production services with a single command line. There are upsides to ScaleFT as well, though. As long as you're all in on Okta or can federate with it, user management is a no brainer. And having the IdP integration is much more user (and malware) friendly and is likely more reliable for server to server use compared to OpenVPN. Limiting access to particular services is likely easier, too. Downsides with this product include having all sorts of reoccuring configuration problems where a server just disappears from the list of available services, which requires ops involement to restore access. If you're using macOS and RDP (I just outed myself to Matt...) you have to use the sub-par FreeRDP client. And ultimately you're tunnelling TCP over TCP, which works ok in the office but which might not always work as well in mobile or higher latency network situations.

ex_amazon_sde 7y ago

> Amazon Linux 2-based EC2 instances, meaning the attack surface is extraordinarily small WHAT?

BlueTemplar 7y ago

Hmm, static IP suffixes are a thing of the past anyway with IPv6 ? "Tens of millions" ? More like quintillions, and that's for a single IP prefix more typical of a home connection !

sullivanmatt 7y ago
Sorry, the tens of millions of IP addresses is referencing the IP space of AWS (one of which these network access points occupies at any given point in time).

mleonhard 7y ago

TLDR: Use a complicated SSH proxy instead of a VPN. This has some serious downsides for non-SSH applications. For example, to connect to a production database cluster, one would need to ssh through the proxy to a bastion host, and then set up port forwarding from the bastion host to the database. Setting up a simple database connection now requires shell access to a production server. This is less secure and more complex than using a traditional VPN.

sullivanmatt 7y ago
A great point. It does depend on your use case, and your dependence on manual operations. For our organization, almost all database interactions and maintenance are performed in code; if somebody is connecting manually, something pretty bad has happened. So for us, we are not really impacted by having to perform port forwarding like this on rare occasion. I completely agree that it could be much more impactful to other organizations. I'm curious: why is utilizing port forwarding over these mutually authenticated SSH tunnels less secure than employing a VPN? From my perspective, port forwarding still adds a level of intentionality which reduces the likelihood of an incident/accident.
- mleonhard 7y ago
  Good VPNs are mutually authenticated. Intentionality is good, but in your example it comes at a cost of complexity. Simplicity is paramount for security. If intentionality is desired, one can use per-server VPNs.

forkexec 7y ago

No, no, no. The whole point of a VPN or SSH jumpbox is to airgap critical infrastructure with unknown vulnerabilities behind a hardened point of access. Putting production infrastructure on the public internet is beyond idiotic and regressive, and an invitation to be hacked by an unlimited and unknown number of exploits. It took forever to get departmental firewalls at a big name university where I worked because systems put in before my time like nutrition/meal planner, housing lottery draw, facilities management system and retail POS systems were getting owned left-and-right by remote malware. I'll keep using fwknop-protected OpenSSH on OpenBSD and WireGuard, others can do whatever they want without thinking about the security vs. convenience.

jjoonathan 7y ago
So, they have drawn their lines of defense in a position you are not used to, and therefore they are beyond idiotic and regressive? Really?
- forkexec 7y ago
  Yeap. There is a night and day difference in attack surfaces between isolating access to a single (or HA pair) jumpbox and N boxes on the internet with no real DMZ or private admin network. Feelings and fashions don't make stupid configurations better. If you have a problem with honest opinions from someone with 25 years of experience, I think you need thicker skin or I can choose to simply not comment and let stupid fashions propagate.
  - sullivanmatt 7y ago
    I wish you had come into this discussion with constructive criticism, instead of simply swinging a hammer. I, for one, am happy to learn from somebody with a number of years of experience. However showing up on a thread and spewing negativity and name calling isn't a great way to earn respect in this industry.
  - icedchai 7y ago
    Yep. Putting everything directly on the public Internet is 90's style. I remember it well. Whole offices with public IP addresses. No firewall. It's amazing anyone ever considered this sane, but it was a different time.

neurobashing 7y ago

Does anyone happen to know what this costs? They’re predictably quiet about it.

cordite 7y ago

I'm surprised to see no mention of Mutual TLS (MTLS), PIV cards, or the like

bb611 7y ago
I believe in this case he's talking about MTLS: > OASA also protects these hops by issuing client certificates with 10-minute expirations after first verifying your identity through our single sign-on provider, and then also verifying you are on a pre-enrolled (and approved) trusted company device.
- sullivanmatt 7y ago
  Basically. Since we are focusing on SSH in this post (and keep in mind that SSH is its own protocol, separate from TLS), it's conceptually the same: client has a certificate and a key, signed by a trusted certificate authority, and the client is also in possession of the server certificate authority. So then you have a bi-directional trust established. The certificates are short-lived, and issued after a successful authentication + dial request to the OASA service.

brentis 7y ago

I work in this space who has a commercial offering. Ill just leave this CSA spec here for reference. Please also look up SPA - newer than port knocking, but based on same premise. https://downloads.cloudsecurityalliance.org/initiatives/sdp/SDP_Specification_1.0.pdf https://downloads.cloudsecurityalliance.org/initiatives/sdp/...

fulafel 7y ago
This talks about air-gapped networks?

justlexi93 7y ago

In 99.95% of cases, VPNs are set up to: Bridge a network device – such as a laptop or even another server … into a larger network of servers – such as in the cloud or on-prem … across the Internet – protected with an additional layer of encryption

theptip 7y ago

If you're in GCP, then you can use their Identity Aware Proxy to achieve most of this. (https://cloud.google.com/iap/docs https://cloud.google.com/iap/docs) IAP supports HTTP and TCP connections, so you can put it in front of your website (say an internal admin webapp), or use it to tunnel SSH onto a machine that doesn't have a public IP, using your IAM roles. If you're running Kubernetes in GKE, you can also wire IAP up to an Ingress, to protect any TCP/HTTP services you have in your k8s cluster. This one is a bit tricky to configure, but is very nice once you have it up and running.

znpy 7y ago

I've recently started using socks proxies via ssh and I was surprised about how far you can go with such solution. Unless you want permanent connection with routing and everything, ssh socks proxy work awesomely. Point a firefox profile to use it and you can really act as if you were in a different subnet: it can proxy dns resolution too.

peterwwillis 7y ago

Mainly this is a poor idea because of the whole allowing-an-IP-thing. Do not rely on IPs or ports for security, ever. Ever. Evereverevereverever. They are not secure. They do not contribute to security. Do not make your security dependent on them. Ever. Ok? Thanks. AWS already has a vastly superior solution for this, called AWS Systems Manager Session Manager (it's quite a mouthful). You create a session with AWS using a federated login service (SAML-based SSO) and then craft IAM policies to allow a single user to ssh into a single server, over the AWS API. Not only will this be more secure, you don't have to maintain a wacky custom solution. Logging into servers is an anti-pattern, and wherever possible you should be running away from it. Get metrics out of the server and analyze them, run commands remotely using some kind of persistent system agent, stop storing state on your servers. I know this is not the point of the article, but I want to remind people of it so they can can avoid the ssh trap early.