11 ms·

We abused Slack's TURN servers to gain access to internal services

381 points by reader_1000 6y ago

kylek 6y ago
tldr- November 2017: added TURN abuse to our stunner toolset December 2017: discovered and reported TURN vulnerability in private customer of Enable Security February 2018: briefly tested Slack and discovered the vulnerability April 2018: submitted our report to Slack, helped them reproduce and address the issue through various rounds of testing May 2018: Slack pushed patch to live servers which was retested by Enable Security January 2020: asked to publish report February 2020: disclosure delayed by HackerOne/Slack March 2020: report published
- lonelappde 6y ago
  Don't use indentation for formatting linebreaks. It beaks HN layout. Just add extra linebreaks
  - dang 6y ago
    I've fixed the formatting now.
- jsjddbbwj 6y ago
  Stop using code blocks if you're not posting code
  - rvz 6y ago
    Sorry big brother.
wrkronmiller 6y ago
As a complete novice in this area, I don't understand the advantage of using a proxy-like service such as TURN. What is the advantage over simply routing the media streams through application servers (i.e. user A connects to server which links to user B) which can then perform application-specific authentication, enforce restrictions on payloads, etc... Performance?
- detaro 6y ago
  Edit: nvm, confused STUN and TURN
  - wrkronmiller 6y ago
    It sounds like the TURN server is effectively acting like an (open) proxy. Wouldn't that mean the operator still has to have the infrastructure to handle the connections + traffic? I'm assuming, perhaps incorrectly, that most of these RTC connections are happening over NAT and therefore usually go over TURN rather than by connecting directly. Even if that's not the case, why not try direct p2p connection first then fall back to routing through an application-specific proxy, which can have tighter controls on who connects to who and what payloads they send?
    - detaro 6y ago
      Sorry, my bad, I indeed confused STUN and TURN.
- aclavelle 6y ago
  My knowledge is about 2 years old on this but I can try to explain: TURN/STUN are to facilitate users communicating behind NAT and firewalls. TURN routes all traffic through a central server and pushes it to clients which it has an established connection with, thus getting around NAT/Firewall. STUN is a bit more lightweight in that it really just helps users to negotiate a normal P2P connection and then they send messages directly to eachother.
  - wrkronmiller 6y ago
    Thanks! That's in-line with what I thought was going on. It sounds like TURN is very close to being an open proxy. Rather than falling back from p2p to STUN to TURN, why not replace TURN with something more application/protocol-specific? Perhaps a webrtc-only proxy that performs authentication and can perform authorization along the lines of: user A is (only) allowed to connect to user B using protocol WebRTC.
BoorishBears 6y ago
Timeline— November 2017: added TURN abuse to our stunner toolset December 2017: discovered and reported TURN vulnerability in private customer of Enable Security February 2018: briefly tested Slack and discovered the vulnerability April 2018: submitted our report to Slack, helped them reproduce and address the issue through various rounds of testing May 2018: Slack pushed patch to live servers which was retested by Enable Security January 2020: asked to publish report February 2020: disclosure delayed by HackerOne/Slack March 2020: report published
gfodor 6y ago
Huh, I happen to be knee-deep in this stuff right now. This article noted that Slack seemed to be running an old TURN server (pre-coturn): https://webrtchacks.com/slack-webrtc-slacking/ https://webrtchacks.com/slack-webrtc-slacking/ Given that the latest coturn has this vulnerability mitigated by default, perhaps all this boils down to is "Slack runs outdated software, we exploited it."?
- jackewiehose 6y ago
  It's not really a bug in old coturn, just a feature in the protocol. According to the article newer versions just disable routing to 127.0.0.1 by default but there are still other network addresses you might have to consider (see article for a recommended list of "denied-peer-ips").
  - gfodor 6y ago
    Yup, sorry if my phrasing implied it's a bug. It's just better defaults, and there's evidence to support the idea slack was just running the software with the old defaults.
    - StillBored 6y ago
      But its not necessarily the defaults. The article isn't clear if the private services are on 127/8 or the TURN server has access to other things in the DMZ/VPC/whatever. I'm guessing its actually the latter as everyone is so fond of the 1:1 VM/container->service model. Meaning its likely a config problem with the denied-peer-ips the parent here links.
  - jerf 6y ago
    You may already know this, but it's worth getting the word out. Do not just deny routing to 127.0.0.1. 127.0.0.1 is merely the conventional "localhost" address; however, ALL 127.x.x.x is "localhost". You can check this now on your local command line with "ping 127.1.2.3". (This just seems to be one of those bugs that every proxy goes through at some point, just like pretty much any attempt to write a web server that serves files off disk will have at least one directory traversal bug.)
enrichp 6y ago
scs@cd
enrichpu 6y ago
cdcd
jrockway 6y ago
Things like this are why mTLS internally are so important. If a hole is found in your firewall, services still don't trust each other until they have a valid TLS certificate.
- freedomben 6y ago
  Totally agree. I've been rolling every service out with mTLS. It was a huge PITA tho without a service mesh (which we can't use for different reasons), so I built a drop-in solution for use with any Kubernetes. I'm still developing on it a bit but my solution is open source [1]. If anybody want to use this I'm happy to provide answers to questions, and quick bug fixes (as this directly benefits my work right now). If you're using kubernetes this is a pretty easy drop in for your pod. It's part of our default setup now. [1] https://github.com/FreedomBen/metals https://github.com/FreedomBen/metals
  - ForHackernews 6y ago
    I think Istio gives you mTLS for free if you add it to your kubernetes cluster.
    - freedomben 6y ago
      > I think Istio gives you mTLS for free if you add it to your kubernetes cluster. Yes, Istio was the service mesh I referenced above that we can't install for different reasons: >> It was a huge PITA tho without a service mesh (which we can't use for different reasons) If you have Istio then you don't need MeTaLS (unless your client comes from outside the cluster or something, and even then I think there are ways to make it work). I don't know that I would agree that it is "for free" as Istio still needs to be configured, and it isn't trivial from my experience. I could also see situations where something like MeTaLS where you place a few env vars for certs and you're done is nice to have. I would definitely recommend Istio if you can use it though.
- Nextgrid 6y ago
  I wish there was an easy way to do mutual TLS auth with pre-shared keys that can be stored and copy/pasted just like normal passwords or API keys without having to maintain a CA and handle certificate issuing & renewal (sure, technically forever-lived certificates aren't as secure, but even those would already be a major upgrade compared to the status quo).
russellbeattie 6y ago
So Slack's VoIP uses WebRTC, which connects via UDP/TCP to always send SRTP packets through a TURN proxy (which extends STUN via ICE) to work around usual NAT problems. These guys scanned the TURN and found an SSRF which allowed them to connect to Slack's VPC on AWS using IAM temporary credentials. Interesting. For fun, read that last paragraph out loud to a non-techy near by and watch their eyes...
- lwb 6y ago
  Even a techy who isn't familiar with networking protocols would start to glaze over!
  - ct520 6y ago
    Not so much networking protocols but WebRTC maybe? I did a hello world type of implementation of WEBRTC awhile back and it made perfect sense to me.
    - dep_b 6y ago
      Yeah there's the WebRTC you can hack in an afternoon and there's the WebRTC that covers the most common edge cases. The problem that the solution to one problem (for example your websocket going disconnecting mid call while WebRTC still soldiers on) can create three new problems when a different edge case arises.
- tandr 6y ago
  This is a nice summary actually. (btw, you can read it to techy-but-not-in-the-field and still get the same look. I am not sure if I should be sad or proud from the fact that I understood 90% of what you have said without google-fu...)
  - anthk 6y ago
    If you are into SIP this is pretty well known.
Diggsey 6y ago
Instead of sending traffic anywhere, why don't they have the destination address first send a (slack-authenticated) request to the TURN server saying "I'm happy to receive traffic from [SOURCE]" and then a temporary window is opened for [SOURCE] to open a connection to that specific destination.
realchucknorris 6y ago
am i wrong or security researchers aren't paid well. i mean not sure how much this bug is wort but def. $3500 looks like a small number.
- anotheraccountf 6y ago
  Yeah, I had the same thought. For something as big as this? Should be at least 2 more zeros imo.
  - imtringued 6y ago
    I don't understand what's so big about this. It's akin to telling someone that they forgot to use passwords on their mongodb database. Does that really deserve $350k compensation?
    - anotheraccountf 6y ago
      Depending on what a black hat could do with the data in your database, it might absolutely be worth it. I understand that 350k is way more than bug bounties usually pay, but 3.5k is taking advantage of people's ethics to outsource your security. Let's put it another way: The team who discovered this has skills WELL worth 350k for a year's worth a work. How many security issues would they have to catch for it to be "worth it"? Maybe more than 1, but 100 show stopping vulnerabilities for 350k is crazy to me. edit: ESPECIALLY slack, if it was possible to use this to get access to any chat logs.
      - tptacek 6y ago
        No, none of this is how vulnerability research compensation works.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
lardikanna 6y ago
Now many companies are changing their schedule and style of work. Remote access is an important part of business success
hodoj 6y ago
Thanka a lot . It is intresting for me
lardikanna 6y ago
In order for employees to be able to fully work in remote access, the company installs the necessary software on employees' computers. It's enough to https://www.action1.com/ https://www.action1.com/. This software gives control to endpoints in real time, software deployment, installation of fixes from the cloud
ChrisArchitect 6y ago
kind of important almost title-edit-worthy to note this is an exploit and research that went on late-2017 until about mid-2018 no? Not that this is some current thing
- jackewiehose 6y ago
  Published March 2020. It's not about some Slack issue that is irrelevant now. It's about misconfigured TURN-servers and at least for me it's a current thing ;-)

381 points by reader_1000 6y ago

tldr- November 2017: added TURN abuse to our stunner toolset December 2017: discovered and reported TURN vulnerability in private customer of Enable Security February 2018: briefly tested Slack and discovered the vulnerability April 2018: submitted our report to Slack, helped them reproduce and address the issue through various rounds of testing May 2018: Slack pushed patch to live servers which was retested by Enable Security January 2020: asked to publish report February 2020: disclosure delayed by HackerOne/Slack March 2020: report published

lonelappde 6y ago
Don't use indentation for formatting linebreaks. It beaks HN layout. Just add extra linebreaks
- dang 6y ago
  I've fixed the formatting now.
jsjddbbwj 6y ago
Stop using code blocks if you're not posting code
- rvz 6y ago
  Sorry big brother.

wrkronmiller 6y ago

As a complete novice in this area, I don't understand the advantage of using a proxy-like service such as TURN. What is the advantage over simply routing the media streams through application servers (i.e. user A connects to server which links to user B) which can then perform application-specific authentication, enforce restrictions on payloads, etc... Performance?

detaro 6y ago
Edit: nvm, confused STUN and TURN
- wrkronmiller 6y ago
  It sounds like the TURN server is effectively acting like an (open) proxy. Wouldn't that mean the operator still has to have the infrastructure to handle the connections + traffic? I'm assuming, perhaps incorrectly, that most of these RTC connections are happening over NAT and therefore usually go over TURN rather than by connecting directly. Even if that's not the case, why not try direct p2p connection first then fall back to routing through an application-specific proxy, which can have tighter controls on who connects to who and what payloads they send?
  - detaro 6y ago
    Sorry, my bad, I indeed confused STUN and TURN.
aclavelle 6y ago
My knowledge is about 2 years old on this but I can try to explain: TURN/STUN are to facilitate users communicating behind NAT and firewalls. TURN routes all traffic through a central server and pushes it to clients which it has an established connection with, thus getting around NAT/Firewall. STUN is a bit more lightweight in that it really just helps users to negotiate a normal P2P connection and then they send messages directly to eachother.
- wrkronmiller 6y ago
  Thanks! That's in-line with what I thought was going on. It sounds like TURN is very close to being an open proxy. Rather than falling back from p2p to STUN to TURN, why not replace TURN with something more application/protocol-specific? Perhaps a webrtc-only proxy that performs authentication and can perform authorization along the lines of: user A is (only) allowed to connect to user B using protocol WebRTC.

BoorishBears 6y ago

Timeline— November 2017: added TURN abuse to our stunner toolset December 2017: discovered and reported TURN vulnerability in private customer of Enable Security February 2018: briefly tested Slack and discovered the vulnerability April 2018: submitted our report to Slack, helped them reproduce and address the issue through various rounds of testing May 2018: Slack pushed patch to live servers which was retested by Enable Security January 2020: asked to publish report February 2020: disclosure delayed by HackerOne/Slack March 2020: report published

gfodor 6y ago

Huh, I happen to be knee-deep in this stuff right now. This article noted that Slack seemed to be running an old TURN server (pre-coturn): https://webrtchacks.com/slack-webrtc-slacking/ https://webrtchacks.com/slack-webrtc-slacking/ Given that the latest coturn has this vulnerability mitigated by default, perhaps all this boils down to is "Slack runs outdated software, we exploited it."?

jackewiehose 6y ago
It's not really a bug in old coturn, just a feature in the protocol. According to the article newer versions just disable routing to 127.0.0.1 by default but there are still other network addresses you might have to consider (see article for a recommended list of "denied-peer-ips").
- gfodor 6y ago
  Yup, sorry if my phrasing implied it's a bug. It's just better defaults, and there's evidence to support the idea slack was just running the software with the old defaults.
  - StillBored 6y ago
    But its not necessarily the defaults. The article isn't clear if the private services are on 127/8 or the TURN server has access to other things in the DMZ/VPC/whatever. I'm guessing its actually the latter as everyone is so fond of the 1:1 VM/container->service model. Meaning its likely a config problem with the denied-peer-ips the parent here links.
- jerf 6y ago
  You may already know this, but it's worth getting the word out. Do not just deny routing to 127.0.0.1. 127.0.0.1 is merely the conventional "localhost" address; however, ALL 127.x.x.x is "localhost". You can check this now on your local command line with "ping 127.1.2.3". (This just seems to be one of those bugs that every proxy goes through at some point, just like pretty much any attempt to write a web server that serves files off disk will have at least one directory traversal bug.)

enrichp 6y ago

scs@cd

enrichpu 6y ago

cdcd

jrockway 6y ago

Things like this are why mTLS internally are so important. If a hole is found in your firewall, services still don't trust each other until they have a valid TLS certificate.

freedomben 6y ago
Totally agree. I've been rolling every service out with mTLS. It was a huge PITA tho without a service mesh (which we can't use for different reasons), so I built a drop-in solution for use with any Kubernetes. I'm still developing on it a bit but my solution is open source [1]. If anybody want to use this I'm happy to provide answers to questions, and quick bug fixes (as this directly benefits my work right now). If you're using kubernetes this is a pretty easy drop in for your pod. It's part of our default setup now. [1] https://github.com/FreedomBen/metals https://github.com/FreedomBen/metals
- ForHackernews 6y ago
  I think Istio gives you mTLS for free if you add it to your kubernetes cluster.
  - freedomben 6y ago
    > I think Istio gives you mTLS for free if you add it to your kubernetes cluster. Yes, Istio was the service mesh I referenced above that we can't install for different reasons: >> It was a huge PITA tho without a service mesh (which we can't use for different reasons) If you have Istio then you don't need MeTaLS (unless your client comes from outside the cluster or something, and even then I think there are ways to make it work). I don't know that I would agree that it is "for free" as Istio still needs to be configured, and it isn't trivial from my experience. I could also see situations where something like MeTaLS where you place a few env vars for certs and you're done is nice to have. I would definitely recommend Istio if you can use it though.
Nextgrid 6y ago
I wish there was an easy way to do mutual TLS auth with pre-shared keys that can be stored and copy/pasted just like normal passwords or API keys without having to maintain a CA and handle certificate issuing & renewal (sure, technically forever-lived certificates aren't as secure, but even those would already be a major upgrade compared to the status quo).

russellbeattie 6y ago

So Slack's VoIP uses WebRTC, which connects via UDP/TCP to always send SRTP packets through a TURN proxy (which extends STUN via ICE) to work around usual NAT problems. These guys scanned the TURN and found an SSRF which allowed them to connect to Slack's VPC on AWS using IAM temporary credentials. Interesting. For fun, read that last paragraph out loud to a non-techy near by and watch their eyes...

lwb 6y ago
Even a techy who isn't familiar with networking protocols would start to glaze over!
- ct520 6y ago
  Not so much networking protocols but WebRTC maybe? I did a hello world type of implementation of WEBRTC awhile back and it made perfect sense to me.
  - dep_b 6y ago
    Yeah there's the WebRTC you can hack in an afternoon and there's the WebRTC that covers the most common edge cases. The problem that the solution to one problem (for example your websocket going disconnecting mid call while WebRTC still soldiers on) can create three new problems when a different edge case arises.
tandr 6y ago
This is a nice summary actually. (btw, you can read it to techy-but-not-in-the-field and still get the same look. I am not sure if I should be sad or proud from the fact that I understood 90% of what you have said without google-fu...)
- anthk 6y ago
  If you are into SIP this is pretty well known.

Diggsey 6y ago

Instead of sending traffic anywhere, why don't they have the destination address first send a (slack-authenticated) request to the TURN server saying "I'm happy to receive traffic from [SOURCE]" and then a temporary window is opened for [SOURCE] to open a connection to that specific destination.

realchucknorris 6y ago

am i wrong or security researchers aren't paid well. i mean not sure how much this bug is wort but def. $3500 looks like a small number.

anotheraccountf 6y ago
Yeah, I had the same thought. For something as big as this? Should be at least 2 more zeros imo.
- imtringued 6y ago
  I don't understand what's so big about this. It's akin to telling someone that they forgot to use passwords on their mongodb database. Does that really deserve $350k compensation?
  - anotheraccountf 6y ago
    Depending on what a black hat could do with the data in your database, it might absolutely be worth it. I understand that 350k is way more than bug bounties usually pay, but 3.5k is taking advantage of people's ethics to outsource your security. Let's put it another way: The team who discovered this has skills WELL worth 350k for a year's worth a work. How many security issues would they have to catch for it to be "worth it"? Maybe more than 1, but 100 show stopping vulnerabilities for 350k is crazy to me. edit: ESPECIALLY slack, if it was possible to use this to get access to any chat logs.
    - tptacek 6y ago
      No, none of this is how vulnerability research compensation works.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

lardikanna 6y ago

Now many companies are changing their schedule and style of work. Remote access is an important part of business success

hodoj 6y ago

Thanka a lot . It is intresting for me

lardikanna 6y ago

In order for employees to be able to fully work in remote access, the company installs the necessary software on employees' computers. It's enough to https://www.action1.com/ https://www.action1.com/. This software gives control to endpoints in real time, software deployment, installation of fixes from the cloud

ChrisArchitect 6y ago

kind of important almost title-edit-worthy to note this is an exploit and research that went on late-2017 until about mid-2018 no? Not that this is some current thing

jackewiehose 6y ago
Published March 2020. It's not about some Slack issue that is irrelevant now. It's about misconfigured TURN-servers and at least for me it's a current thing ;-)