4 ms·

Things like this are why mTLS internally are so important. If a hole is found in your firewall, services still don't trust each other until they have a valid T

Things like this are why mTLS internally are so important. If a hole is found in your firewall, services still don't trust each other until they have a valid TLS certificate.

freedomben 7y ago
Totally agree. I've been rolling every service out with mTLS. It was a huge PITA tho without a service mesh (which we can't use for different reasons), so I built a drop-in solution for use with any Kubernetes. I'm still developing on it a bit but my solution is open source [1]. If anybody want to use this I'm happy to provide answers to questions, and quick bug fixes (as this directly benefits my work right now). If you're using kubernetes this is a pretty easy drop in for your pod. It's part of our default setup now. [1] https://github.com/FreedomBen/metals https://github.com/FreedomBen/metals
- ForHackernews 7y ago
  I think Istio gives you mTLS for free if you add it to your kubernetes cluster.
  - freedomben 7y ago
    > I think Istio gives you mTLS for free if you add it to your kubernetes cluster. Yes, Istio was the service mesh I referenced above that we can't install for different reasons: >> It was a huge PITA tho without a service mesh (which we can't use for different reasons) If you have Istio then you don't need MeTaLS (unless your client comes from outside the cluster or something, and even then I think there are ways to make it work). I don't know that I would agree that it is "for free" as Istio still needs to be configured, and it isn't trivial from my experience. I could also see situations where something like MeTaLS where you place a few env vars for certs and you're done is nice to have. I would definitely recommend Istio if you can use it though.
Nextgrid 7y ago
I wish there was an easy way to do mutual TLS auth with pre-shared keys that can be stored and copy/pasted just like normal passwords or API keys without having to maintain a CA and handle certificate issuing & renewal (sure, technically forever-lived certificates aren't as secure, but even those would already be a major upgrade compared to the status quo).
- freedomben 7y ago
  Shameless plug (I really did not intend to promote this yet but it can help so figured I'd mention it in case you are interested), but if you are using Kubernetes you could get pretty darn close with MeTaLS[1]. Generate self-signed certs that last for 10 or more years and copy/paste them into environment variables for your service, and you've got it. Of course I don't recommend that as it's not as secure as doing things "the right" way but it's definitely better than no mTLS at all (as you pointed out). MeTaLS won't provide you with client-related stuff, but most clients and client libs make it easy to set a certificate/key with a request. [1]: https://github.com/FreedomBen/metals https://github.com/FreedomBen/metals
- jrockway 7y ago
  I personally use cert-manager to run the CA, then create a cert for each app and have k8s inject it. It is more manual than a service mesh, but many applications support this strategy out of the box. (I will say that creating a certificate resource is a lot easier than the old days of some directory of shell scripts acting as your CA, though. And the same code manages my letsencrypt certs.) For example, I have Grafana backed by Postgres, and they both understand this authentication scheme out of the box. Postgres is happy to be provided with a cert to present to connecting applications, and is happy to check the cert that applications present against the CA cert. The main problem with my setup is that I use a ClusterIssuer CA, so really anyone in the cluster can get a valid certificate. This is not amazingly secure and things like Istio do a bit more provenance checking of the application before issuing a cert, which I like. But this is simple, and does protect against the attack this article covers -- as long as you don't go out of your way to present the application's cert when proxying a connection. (Which is probably an easy mistake to make, so be careful.)
- 7y ago
tptacek 7y ago
Any kind of inter-service authentication, really. And for lots of reasons, not just SSRF. But regardless: coherent inter-service authentication is not a norm. If you're exclusively interested in mitigating SSRF, a more targeted solution is to run your connections (HTTP or TCP) through a proxy that enforces network-level rules. That seems like it would have worked here. For HTTP SSRF, Stripe has a good tool, Smokescreen.
empath75 7y ago
I worked for a large tech company that was hacked by Russians at least twice, and they had a not only unencrypted, but unauthenticated sql front end and api to their inventory management system that listed every server on the network and every piece of software and version installed on it, as all the user accounts on it and privileges as well as sysadmin contact info and everything you’d need for social engineering. I realized how bad it was when I was using it to find all the servers at the company that needed to be patched for heartbleed. Anybody on any server in our network could get to it, or just someone who plugged a laptop in at the office or got in to our vpn, and you’d have the keys to the kingdom. I told the head of security and he said it would break too many things to put it behind authentication.
paxys 7y ago
Funny enough Slack itself recently open-sourced https://github.com/slackhq/nebula https://github.com/slackhq/nebula which does exactly that.

Things like this are why mTLS internally are so important. If a hole is found in your firewall, services still don't trust each other until they have a valid TLS certificate.

freedomben 7y ago

Totally agree. I've been rolling every service out with mTLS. It was a huge PITA tho without a service mesh (which we can't use for different reasons), so I built a drop-in solution for use with any Kubernetes. I'm still developing on it a bit but my solution is open source [1]. If anybody want to use this I'm happy to provide answers to questions, and quick bug fixes (as this directly benefits my work right now). If you're using kubernetes this is a pretty easy drop in for your pod. It's part of our default setup now. [1] https://github.com/FreedomBen/metals https://github.com/FreedomBen/metals

ForHackernews 7y ago
I think Istio gives you mTLS for free if you add it to your kubernetes cluster.
- freedomben 7y ago
  > I think Istio gives you mTLS for free if you add it to your kubernetes cluster. Yes, Istio was the service mesh I referenced above that we can't install for different reasons: >> It was a huge PITA tho without a service mesh (which we can't use for different reasons) If you have Istio then you don't need MeTaLS (unless your client comes from outside the cluster or something, and even then I think there are ways to make it work). I don't know that I would agree that it is "for free" as Istio still needs to be configured, and it isn't trivial from my experience. I could also see situations where something like MeTaLS where you place a few env vars for certs and you're done is nice to have. I would definitely recommend Istio if you can use it though.

Nextgrid 7y ago

I wish there was an easy way to do mutual TLS auth with pre-shared keys that can be stored and copy/pasted just like normal passwords or API keys without having to maintain a CA and handle certificate issuing & renewal (sure, technically forever-lived certificates aren't as secure, but even those would already be a major upgrade compared to the status quo).

freedomben 7y ago
Shameless plug (I really did not intend to promote this yet but it can help so figured I'd mention it in case you are interested), but if you are using Kubernetes you could get pretty darn close with MeTaLS[1]. Generate self-signed certs that last for 10 or more years and copy/paste them into environment variables for your service, and you've got it. Of course I don't recommend that as it's not as secure as doing things "the right" way but it's definitely better than no mTLS at all (as you pointed out). MeTaLS won't provide you with client-related stuff, but most clients and client libs make it easy to set a certificate/key with a request. [1]: https://github.com/FreedomBen/metals https://github.com/FreedomBen/metals
jrockway 7y ago
I personally use cert-manager to run the CA, then create a cert for each app and have k8s inject it. It is more manual than a service mesh, but many applications support this strategy out of the box. (I will say that creating a certificate resource is a lot easier than the old days of some directory of shell scripts acting as your CA, though. And the same code manages my letsencrypt certs.) For example, I have Grafana backed by Postgres, and they both understand this authentication scheme out of the box. Postgres is happy to be provided with a cert to present to connecting applications, and is happy to check the cert that applications present against the CA cert. The main problem with my setup is that I use a ClusterIssuer CA, so really anyone in the cluster can get a valid certificate. This is not amazingly secure and things like Istio do a bit more provenance checking of the application before issuing a cert, which I like. But this is simple, and does protect against the attack this article covers -- as long as you don't go out of your way to present the application's cert when proxying a connection. (Which is probably an easy mistake to make, so be careful.)
7y ago

tptacek 7y ago

Any kind of inter-service authentication, really. And for lots of reasons, not just SSRF. But regardless: coherent inter-service authentication is not a norm. If you're exclusively interested in mitigating SSRF, a more targeted solution is to run your connections (HTTP or TCP) through a proxy that enforces network-level rules. That seems like it would have worked here. For HTTP SSRF, Stripe has a good tool, Smokescreen.

empath75 7y ago

I worked for a large tech company that was hacked by Russians at least twice, and they had a not only unencrypted, but unauthenticated sql front end and api to their inventory management system that listed every server on the network and every piece of software and version installed on it, as all the user accounts on it and privileges as well as sysadmin contact info and everything you’d need for social engineering. I realized how bad it was when I was using it to find all the servers at the company that needed to be patched for heartbleed. Anybody on any server in our network could get to it, or just someone who plugged a laptop in at the office or got in to our vpn, and you’d have the keys to the kingdom. I told the head of security and he said it would break too many things to put it behind authentication.

paxys 7y ago

Funny enough Slack itself recently open-sourced https://github.com/slackhq/nebula https://github.com/slackhq/nebula which does exactly that.