7 ms·

In which I agree with the federal government and bash VPNs for fun and profit

49 points by EthanHeilman 5y ago

bradknowles 5y ago
TLDR: Bastion Zero let’s you log into individual hosts, not networks. But it obviously doesn’t go down to the app level, which is what the original government document is actually saying we should do. Unless you can guarantee that one host belongs to one and only one app, then restricting things to the host level doesn’t really help you that much beyond segmented VPNs. So, I don’t really see how this is a huge improvement.
- EthanHeilman 5y ago
  Disclaimer I helped design the bastion-zero protocol. Sorry for the wall of text, I just care a lot about this protocol and I think it is really neat so I'm always excited to talk about it. > But it obviously doesn’t go down to the app level, which is what the original government document is actually saying we should do. Bastion-zero does go down to the app level. Bastion Zero lets you log into roles in applications not mere hosts or ports. This is one of the big advantages of bastion-zero over say tailscale or segmented VPNs. With kube, bastion-zero you could enforce a policy that a particular user identity, say "Alice@example.com", can only access the control plane of certain kube clusters with the kube role 'dev-user'. Not only is this policy enforced at the application and role level but logs are application and identity aware as well. Look at an example of using bastion-zero for shell access. Alice's gsuite identity is Alice@example.com and lets say she has a policy in bastion-zero which allows her to login to the linux shell as 'qa-user' on the host 'qa-dev-123'. Without bastion-zero logs on 'qa-dev-123' would only capture the 'qa-user' performed some action so: 1. Application role: 'qa-user', 2. hostname: 'qa-dev-123', 3. Action: 'rm -rf /' That isn't helpful if multiple people can sign in as 'qa-user'. Additionally since the logs are stored on the host, an attacker might be able to delete or tamper with them after the fact. With bastion-zero, the bastion captures the logs off-the-wire preventing tampering and it associates each action Alice takes with: 1. OpenID Connect auth session: 'a specific authorization token issued by gsuite and associated with a particular machine', 2. Alice's OpenID Connect identity: 'Alice@example.com', 3. Application role: 'qa-user', 4. hostname: 'qa-dev-123', 5. cryptographic identity of host: 'pubkey:0x4e32...' 6. Action: 'rm -rf /' This way we can tell, oh Alice's OpenID Connect identity token has been compromised on her laptop. Revoke that token. What other actions where taken under that token? Did the attacker successfully answer an MFA challenge from the Bastion? Yes? we know Alice's MFA secret has been compromised. Our goal with basiton-zero: * all parties have pubkeys which are strongly bound to their identity forever, but only usable over short periods of time * all pubkeys are short lived and ephemeral you are expected to generate new ones, you don't need to remember old ones * all the content of all messages is digitally signed by public keys producing logs which can be cryptographically attributed * no single points of trust or compromise * policy and logging are identity and application aware
gigel82 5y ago
Yeah right, the folks that write our intranet tools are the ones not qualified enough to work on a shipping product (I know this is harsh but the few exceptions just prove the rule). No way in hell we'd expose their stuff directly to the internet, especially since these internal tools deal with things like benefits, paystubs, asset allocations, access control, etc.
- EthanHeilman 5y ago
  The government's point here is that qualified people should be writing those intranet tools. It is no longer ok to write insecure janky software because "don't worry we can secure later by putting it behind a VPN/Firewall". That is fixing bad engineering with duck tape. The correct approach is to engineer it correctly. Write your software as if it was going to be directly exposed to the internet. While VPNs do provide protection, they do not provide sufficient protection. If software isn't safe to use without a VPN, it isn't safe to use with a VPN either.
- zackbloom 5y ago
  You could use something like Cloudflare Access to secure it, while it's on the Internet. There's no reason you need to build the security just because you build the tool.
kevwil 5y ago
I read this as "... bash (the shell) VPNs for fun and profit" and thought, "what a cursed title!". Now I get it. LOL
codeflo 5y ago
I agree with the goal, but I have trouble imagining reaching it in a typical enterprise/government IT environment. I agree that VPN is a crutch. The alternative is that every little legacy intranet application needs a fully staffed and security-aware maintenance team. That team needs to work full time to keep the technology up to date so that everything can be patched, forever. Many little internal tools wouldn't exist in such an environment. Maybe that's not entirely a bad thing, but it's very different from today's expectations.
- EthanHeilman 5y ago
  I think you are right. Resources spread too thin is a major reason why intranet applications are hard to secure. I'd argue that if you can't secure a legacy application figure out some way to move to a SaaS and let someone else run it for you. Only run stuff yourself that you have the resources to secure. It is why I'm bullish on Saas, it is eating the intranet.
- Jtsummers 5y ago
  US gov't systems have already been pushing out the little bespoke apps for a while now anyways. They're a security nightmare with or without a VPN. Plus, the majority of them seem to cover three things: 1. Task management (JIRA clones, but worse) and coordination, possibly with external user (contractor) access which opens up a hole in the system anyways. 2. Custom workflows (BPMN-ish, but not disciplined and not, easily, connectable to each other), again with external user (contractor, citizen, applicants, whatever) access. 3. Document sharing (SharePoint, but more like a skin over a network share), again with external user access. All of those can be covered by other systems, sometimes simpler ones than presently used (3 is the core of what HTTP was meant to serve, they "just" need access control on top of a basic HTTP server which is a solved problem these days). A bigger problem with the bespoke apps is that they often have their own access control mechanisms, which don't always play well with the larger organization access control systems. They have their own user names and passwords, they may or may not connect to a CAC-authenticated backend. Or they do stupid things (had to fix one of these once) like not authenticating the CAC but using trust-on-first-use, and then the user's access had to be reset when they got a new CAC. Brilliant. These aren't good systems, they are serviceable systems. There are other rapid application development tools (see Appian and others) that can cover (1) and (2) but provide a unified backend (which means the workflows can now be composed, too). SharePoint and other document management systems can cover (3). SharePoint can also cover (1) and (2) (though I've barely tolerated it for either use, it is at least better than dropping $1 million+ a year on hardware and software licenses for a one-off product supporting a half dozen users that doesn't integrate into the rest of the system).
tptacek 5y ago
There are two subtexts behind these arguments against VPNs: 1. Your IT infrastructure is mature enough that all application access is directed through a series of authenticating proxies, so that you're not making individual app-by-app configuration changes to enforce SSO/MFA/device-context rules. 2. Your VPNs have strictly iptables-style network access control (if that). The latter subtext is true of most existing VPNs and is responsible, I think, for almost all of the flak access control VPNs take. Until pretty recently, the state of the art for access VPNs was stuff like OpenVPN, which is so painful to set up and maintain that most installations have no access control at all. The former subtext is true almost nowhere, and that's a problem I have with arguments like this. I like the BastionZero people, but, obviously, most engineering teams aren't using BastionZero, or anything remotely like it. I also don't believe the dust has settled on what products/solutions like this are going to look like. If you can't get that first subtext to hold true for your team, and you nevertheless adopt the logic of this post, you are going to be boned; you'll effectively be setting your security posture to a mid-oughts level, when VPNs were themselves a novelty at small engineering teams, and debug instances of apps hanging out on the Internet were a norm. Meanwhile, you can mostly break the subtext about VPNs by using a modern VPN like (sorry, I have to say it again) Tailscale, which has fine-grained per-app ACLs based on role-based access control. A major complaint the "Zero Trust" people have about modern VPNs is that app-layer proxies can do device-context authentication --- so "employer-owned device, up-to-date patches" is a valid access rule, and that's harder to do right now with things like Tailscale. Fair enough. But almost nobody is doing anything like that today, and I'm not sure that when the dust settles it'll be a market win for app-layer ZTA proxies. You should do whatever you can to limit the scope of exposure for your networks, and you should not put off network access controls you can deploy today because you think you should just wait to see how ZTA products mature. I think Tailscale is probably the most pragmatic next step for most companies.
- d4mi3n 5y ago
  > A major complaint the "Zero Trust" people have about modern VPNs is that app-layer proxies can do device-context authentication --- so "employer-owned device, up-to-date patches" is a valid access rule, and that's harder to do right now with things like Tailscale. Fair enough. But almost nobody is doing anything like that today, and I'm not sure that when the dust settles it'll be a market win for app-layer ZTA proxies. I may be missing the mark, but Okta and competing IdP (identity platform) or CASB (cloud access security broker) products do stuff like this. SSH access is one piece of the puzzle when you talk about a VPN-less model; generally orgs need this capability around any corporate system that is web/ssh/vpn/api accessible.
reincarnate0x14 5y ago
The goal of removing the excuse of "oh it's behind a VPN" is in and of itself not bad, but I have a hard time getting behind the methodology. Modern VPNs aren't just ways to get around fucked up NAT disasters in organizations that can't be bothered to learn IPv6, they also have user, protocol, and application awareness that can be utilized to make fairly granular distinctions about who can access what without ever specifying a ip/mask range. In practical terms this new model is demanding a 5-10x increase in the number of security focused people at any subject organization, and there frankly aren't that many people out to be hired that are actually any good at it. (and the "cybersecurity" organizational mentality does not overlap as much as might be imagined with security research or forward-thinking computer science, those people would be bored to suicide filling out the forms all day that they know no one ever reads) Most organizations are going to find a way to out-source this to things like "government clouds" so it's mostly a major revenue windfall for existing larger players in the same vein as other defense and government contracting.
- joshuamorton 5y ago
  > they also have user, protocol, and application awareness that can be utilized to make fairly granular distinctions about who can access what without ever specifying a ip/mask range. Doesn't this usually require the VPN to do weird packet inspection things? Like you can't, afaik, have a VPN do application or request aware auth of an HSTS enabled endpoint. But with the beyondcorp model, you can. So you don't need to sacrifice this, you just sacrifice the bad parts of it.
  - DarylZero 5y ago
    > Like you can't, afaik, have a VPN do application or request aware auth of an HSTS enabled endpoint. The application can be made aware of the VPN identity.
  - reincarnate0x14 5y ago
    There is nothing weird about packet inspection, and if the VPN or whatever gateway is keyed off the auth of the end application (and not vice versa) it doesn't really matter if it can't necessarily unveil all the TLS traffic. The main concern I have is not that the same sort of protections can't exist at the application endpoint, it's that vendors and developers and IT departments have all been absolutely terrible at doing this. They're not going to get better, they're going to find ways around it. Whereas, just to pick an example and not as any sort of vendor endorsement, I can have a Palo Alto "VPN" that isn't even using private address ranges but sends user tagging data and performs traffic inspection to stop a compromised but authenticated user endpoint from performing attack attempts on the application endpoint. Comedy example: The PAs we put it as part of a protection scheme for wide area renewable energy products blocked attempts to update some shitty GE SCADA software because GE was too lazy to renew their signing certificate and was somehow still OK with shipping builds. All the end users had been trained to simply click the "WTF YES I DON'T CARE I HAVE TO WORK TO DO" button. The network control policy did not give them the option. Application layer proxies can do the same thing, sure, but that fundamentally isn't really any different other than it moves responsibility around.

49 points by EthanHeilman 5y ago

TLDR: Bastion Zero let’s you log into individual hosts, not networks. But it obviously doesn’t go down to the app level, which is what the original government document is actually saying we should do. Unless you can guarantee that one host belongs to one and only one app, then restricting things to the host level doesn’t really help you that much beyond segmented VPNs. So, I don’t really see how this is a huge improvement.

EthanHeilman 5y ago
Disclaimer I helped design the bastion-zero protocol. Sorry for the wall of text, I just care a lot about this protocol and I think it is really neat so I'm always excited to talk about it. > But it obviously doesn’t go down to the app level, which is what the original government document is actually saying we should do. Bastion-zero does go down to the app level. Bastion Zero lets you log into roles in applications not mere hosts or ports. This is one of the big advantages of bastion-zero over say tailscale or segmented VPNs. With kube, bastion-zero you could enforce a policy that a particular user identity, say "Alice@example.com", can only access the control plane of certain kube clusters with the kube role 'dev-user'. Not only is this policy enforced at the application and role level but logs are application and identity aware as well. Look at an example of using bastion-zero for shell access. Alice's gsuite identity is Alice@example.com and lets say she has a policy in bastion-zero which allows her to login to the linux shell as 'qa-user' on the host 'qa-dev-123'. Without bastion-zero logs on 'qa-dev-123' would only capture the 'qa-user' performed some action so: 1. Application role: 'qa-user', 2. hostname: 'qa-dev-123', 3. Action: 'rm -rf /' That isn't helpful if multiple people can sign in as 'qa-user'. Additionally since the logs are stored on the host, an attacker might be able to delete or tamper with them after the fact. With bastion-zero, the bastion captures the logs off-the-wire preventing tampering and it associates each action Alice takes with: 1. OpenID Connect auth session: 'a specific authorization token issued by gsuite and associated with a particular machine', 2. Alice's OpenID Connect identity: 'Alice@example.com', 3. Application role: 'qa-user', 4. hostname: 'qa-dev-123', 5. cryptographic identity of host: 'pubkey:0x4e32...' 6. Action: 'rm -rf /' This way we can tell, oh Alice's OpenID Connect identity token has been compromised on her laptop. Revoke that token. What other actions where taken under that token? Did the attacker successfully answer an MFA challenge from the Bastion? Yes? we know Alice's MFA secret has been compromised. Our goal with basiton-zero: * all parties have pubkeys which are strongly bound to their identity forever, but only usable over short periods of time * all pubkeys are short lived and ephemeral you are expected to generate new ones, you don't need to remember old ones * all the content of all messages is digitally signed by public keys producing logs which can be cryptographically attributed * no single points of trust or compromise * policy and logging are identity and application aware

gigel82 5y ago

Yeah right, the folks that write our intranet tools are the ones not qualified enough to work on a shipping product (I know this is harsh but the few exceptions just prove the rule). No way in hell we'd expose their stuff directly to the internet, especially since these internal tools deal with things like benefits, paystubs, asset allocations, access control, etc.

EthanHeilman 5y ago
The government's point here is that qualified people should be writing those intranet tools. It is no longer ok to write insecure janky software because "don't worry we can secure later by putting it behind a VPN/Firewall". That is fixing bad engineering with duck tape. The correct approach is to engineer it correctly. Write your software as if it was going to be directly exposed to the internet. While VPNs do provide protection, they do not provide sufficient protection. If software isn't safe to use without a VPN, it isn't safe to use with a VPN either.
zackbloom 5y ago
You could use something like Cloudflare Access to secure it, while it's on the Internet. There's no reason you need to build the security just because you build the tool.

kevwil 5y ago

I read this as "... bash (the shell) VPNs for fun and profit" and thought, "what a cursed title!". Now I get it. LOL

codeflo 5y ago

I agree with the goal, but I have trouble imagining reaching it in a typical enterprise/government IT environment. I agree that VPN is a crutch. The alternative is that every little legacy intranet application needs a fully staffed and security-aware maintenance team. That team needs to work full time to keep the technology up to date so that everything can be patched, forever. Many little internal tools wouldn't exist in such an environment. Maybe that's not entirely a bad thing, but it's very different from today's expectations.

EthanHeilman 5y ago
I think you are right. Resources spread too thin is a major reason why intranet applications are hard to secure. I'd argue that if you can't secure a legacy application figure out some way to move to a SaaS and let someone else run it for you. Only run stuff yourself that you have the resources to secure. It is why I'm bullish on Saas, it is eating the intranet.
Jtsummers 5y ago
US gov't systems have already been pushing out the little bespoke apps for a while now anyways. They're a security nightmare with or without a VPN. Plus, the majority of them seem to cover three things: 1. Task management (JIRA clones, but worse) and coordination, possibly with external user (contractor) access which opens up a hole in the system anyways. 2. Custom workflows (BPMN-ish, but not disciplined and not, easily, connectable to each other), again with external user (contractor, citizen, applicants, whatever) access. 3. Document sharing (SharePoint, but more like a skin over a network share), again with external user access. All of those can be covered by other systems, sometimes simpler ones than presently used (3 is the core of what HTTP was meant to serve, they "just" need access control on top of a basic HTTP server which is a solved problem these days). A bigger problem with the bespoke apps is that they often have their own access control mechanisms, which don't always play well with the larger organization access control systems. They have their own user names and passwords, they may or may not connect to a CAC-authenticated backend. Or they do stupid things (had to fix one of these once) like not authenticating the CAC but using trust-on-first-use, and then the user's access had to be reset when they got a new CAC. Brilliant. These aren't good systems, they are serviceable systems. There are other rapid application development tools (see Appian and others) that can cover (1) and (2) but provide a unified backend (which means the workflows can now be composed, too). SharePoint and other document management systems can cover (3). SharePoint can also cover (1) and (2) (though I've barely tolerated it for either use, it is at least better than dropping $1 million+ a year on hardware and software licenses for a one-off product supporting a half dozen users that doesn't integrate into the rest of the system).

tptacek 5y ago

There are two subtexts behind these arguments against VPNs: 1. Your IT infrastructure is mature enough that all application access is directed through a series of authenticating proxies, so that you're not making individual app-by-app configuration changes to enforce SSO/MFA/device-context rules. 2. Your VPNs have strictly iptables-style network access control (if that). The latter subtext is true of most existing VPNs and is responsible, I think, for almost all of the flak access control VPNs take. Until pretty recently, the state of the art for access VPNs was stuff like OpenVPN, which is so painful to set up and maintain that most installations have no access control at all. The former subtext is true almost nowhere, and that's a problem I have with arguments like this. I like the BastionZero people, but, obviously, most engineering teams aren't using BastionZero, or anything remotely like it. I also don't believe the dust has settled on what products/solutions like this are going to look like. If you can't get that first subtext to hold true for your team, and you nevertheless adopt the logic of this post, you are going to be boned; you'll effectively be setting your security posture to a mid-oughts level, when VPNs were themselves a novelty at small engineering teams, and debug instances of apps hanging out on the Internet were a norm. Meanwhile, you can mostly break the subtext about VPNs by using a modern VPN like (sorry, I have to say it again) Tailscale, which has fine-grained per-app ACLs based on role-based access control. A major complaint the "Zero Trust" people have about modern VPNs is that app-layer proxies can do device-context authentication --- so "employer-owned device, up-to-date patches" is a valid access rule, and that's harder to do right now with things like Tailscale. Fair enough. But almost nobody is doing anything like that today, and I'm not sure that when the dust settles it'll be a market win for app-layer ZTA proxies. You should do whatever you can to limit the scope of exposure for your networks, and you should not put off network access controls you can deploy today because you think you should just wait to see how ZTA products mature. I think Tailscale is probably the most pragmatic next step for most companies.

d4mi3n 5y ago
> A major complaint the "Zero Trust" people have about modern VPNs is that app-layer proxies can do device-context authentication --- so "employer-owned device, up-to-date patches" is a valid access rule, and that's harder to do right now with things like Tailscale. Fair enough. But almost nobody is doing anything like that today, and I'm not sure that when the dust settles it'll be a market win for app-layer ZTA proxies. I may be missing the mark, but Okta and competing IdP (identity platform) or CASB (cloud access security broker) products do stuff like this. SSH access is one piece of the puzzle when you talk about a VPN-less model; generally orgs need this capability around any corporate system that is web/ssh/vpn/api accessible.

reincarnate0x14 5y ago

The goal of removing the excuse of "oh it's behind a VPN" is in and of itself not bad, but I have a hard time getting behind the methodology. Modern VPNs aren't just ways to get around fucked up NAT disasters in organizations that can't be bothered to learn IPv6, they also have user, protocol, and application awareness that can be utilized to make fairly granular distinctions about who can access what without ever specifying a ip/mask range. In practical terms this new model is demanding a 5-10x increase in the number of security focused people at any subject organization, and there frankly aren't that many people out to be hired that are actually any good at it. (and the "cybersecurity" organizational mentality does not overlap as much as might be imagined with security research or forward-thinking computer science, those people would be bored to suicide filling out the forms all day that they know no one ever reads) Most organizations are going to find a way to out-source this to things like "government clouds" so it's mostly a major revenue windfall for existing larger players in the same vein as other defense and government contracting.

joshuamorton 5y ago
> they also have user, protocol, and application awareness that can be utilized to make fairly granular distinctions about who can access what without ever specifying a ip/mask range. Doesn't this usually require the VPN to do weird packet inspection things? Like you can't, afaik, have a VPN do application or request aware auth of an HSTS enabled endpoint. But with the beyondcorp model, you can. So you don't need to sacrifice this, you just sacrifice the bad parts of it.
- DarylZero 5y ago
  > Like you can't, afaik, have a VPN do application or request aware auth of an HSTS enabled endpoint. The application can be made aware of the VPN identity.
- reincarnate0x14 5y ago
  There is nothing weird about packet inspection, and if the VPN or whatever gateway is keyed off the auth of the end application (and not vice versa) it doesn't really matter if it can't necessarily unveil all the TLS traffic. The main concern I have is not that the same sort of protections can't exist at the application endpoint, it's that vendors and developers and IT departments have all been absolutely terrible at doing this. They're not going to get better, they're going to find ways around it. Whereas, just to pick an example and not as any sort of vendor endorsement, I can have a Palo Alto "VPN" that isn't even using private address ranges but sends user tagging data and performs traffic inspection to stop a compromised but authenticated user endpoint from performing attack attempts on the application endpoint. Comedy example: The PAs we put it as part of a protection scheme for wide area renewable energy products blocked attempts to update some shitty GE SCADA software because GE was too lazy to renew their signing certificate and was somehow still OK with shipping builds. All the end users had been trained to simply click the "WTF YES I DON'T CARE I HAVE TO WORK TO DO" button. The network control policy did not give them the option. Application layer proxies can do the same thing, sure, but that fundamentally isn't really any different other than it moves responsibility around.