10 ms·

Slack account takeovers using HTTP Request Smuggling

459 points by bartkappenburg 7y ago

Lex-2008 7y ago
Greatly detailed report and impressive resolution speed, indeed, but sadly it fell behind the cracks regarding disclosure.
- andrekorol 7y ago
  What do you mean by "it fell behind the cracks regarding disclosure"?
  - harrier 7y ago
    After the issue was resolved the reporter was asked to wait before disclosure. The reporter waited three months before asking about it again. After that there was response but it took another month the approve the disclosure.
    - andrekorol 7y ago
      Oh, now I get it. Thanks for the clarification.
Bhilai 7y ago
Original Paper for those interested - https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request...
wpietri 7y ago
I was unfamiliar with request smuggling; here's an explainer: https://portswigger.net/web-security/request-smuggling https://portswigger.net/web-security/request-smuggling The first in-band signalling attack I came across was the blue box [1], invented in the late 1960s. It still occasionally worked in the 1980s on older phone systems. It amazes me that we're still creating new systems vulnerable to in-band attacks. [1] https://en.wikipedia.org/wiki/Blue_box https://en.wikipedia.org/wiki/Blue_box
- sixstringtheory 7y ago
  Wow, I was fascinated by phreaking when I was younger and first getting into programming. Been working on mac/iOS for years and never knew that the guys responsible for the most enjoyable part of my career, the Steves, built and sold a blue box!
  - at-fates-hands 7y ago
    Of all the stories I heard, Woz was way more interested in using them. When Jobs found it out it was illegal, he dropped selling them and distanced himself from the other phreakers he was hanging out with, too afraid of the legal ramifications of getting caught with this equipment. Most of the books I've read basically paint Jobs as a goody-two-shoes type who cringed at the illegality of phreaking - while saying he probably was thinking ahead, knowing he already wanted to start his own company and didn't want stuff like this to come back and haunt him. It was an interesting dichotomy to me.
- abhishekjha 7y ago
  Is it what Steve Jobs and his friend used to hijack telephone lines and call the pope, free of cost?
  - acruns 7y ago
    Yes. https://www.inc.com/glenn-leibowitz/in-a-rare-23-year-old-interview-steve-jobs-said-this-1-pivotal-experience-inspired-him-to-start-apple-computer.html https://www.inc.com/glenn-leibowitz/in-a-rare-23-year-old-in...
  - AnIdiotOnTheNet 7y ago
    Of all the places on the internet to hear Steve Wozniak referred to simply as Steve Jobs's friend...
    - abhishekjha 7y ago
      Sorry, I didn't realise that other friend in this incient was Steve Wozniak specifically. I saw his interview after his return to Apple.
fiberoptick 7y ago
Wow, an ATO exploit that only received a $6,500 bounty? This signals to grey-/black-hat researchers that their research efforts or Slack bug disclosures are best directed elsewhere..
- edoceo 7y ago
  How much should they have been paid? This seems like a week of pay for a pretty good software dev. That's not fair comp?
  - penagwin 7y ago
    > This seems like a week of pay for a pretty good software dev. Wait, 6500 * 4 * 12 = 312,000/yr Might be a bit high for a week :P EDIT: Okay turns out that if you live in the Bay area this isn't unheard of - the rest of us make 4x to 5x less then that (Saying this as a mid-level software dev from West Michigan).
    - randlet 7y ago
      Salary information on HN is skewed by Bay area engineers where $312k is not out of the realm of possibility for a senior developer afaik.
    - moneromoney 7y ago
      In Germany you will be VERY lucky making 100,000 euro / year. Germany has the lowest software dev. salary / average country salary ratio of all countries on planet.
      - foepys 7y ago
        Unless you are doing anything more or less related to SAP. Then you can make $150k/a easily. If only other companies would realize that that's one of the reasons why SAP is Germany's no. 1 software company.
    - sciurus 7y ago
arkadiyt 7y ago
Protecting against request smuggling: - If you don't have a proxy fronting traffic, no action required - If you're behind Fastly/Cloudflare [1] or Akamai [2], no action required / they protect against this attack - If you're behind AWS Cloudfront, no action required / they protect against this attack - If you're behind AWS ALB, you're vulnerable by default but can opt-in to protection by enabling the "routing.http.drop_invalid_header_fields.enabled" attribute [3]. They initially had it on by default but it broke customers - If you have a different proxy (e.g. some other provider or your own nginx, haproxy before 2.0.6 [2], etc), you might be vulnerable [1]: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request... [2]: https://portswigger.net/research/http-desync-attacks-what-happened-next https://portswigger.net/research/http-desync-attacks-what-ha... [3]: https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_LoadBalancerAttribute.html https://docs.aws.amazon.com/elasticloadbalancing/latest/APIR...
- judge2020 7y ago
  Regarding Cloudflare/fastly, you do need to make sure you're only allowing requests that originate from the proxy, either via IP-based firewall rules or something like CF's authenticated origin pulls [0]. Otherwise someone could find your origin server's IP and potentially perform this attack (and generally bypass your security settings). 0: https://support.cloudflare.com/hc/en-us/articles/204899617-Authenticated-Origin-Pulls https://support.cloudflare.com/hc/en-us/articles/204899617-A...
  - tialaramex 7y ago
    Allowing Connections to your backend directly might make you vulnerable to certain types of attack but it doesn't impact Request Smuggling. The trick in Request Smuggling is that you're trusting an intermediary (in this case a frontend reverse proxy) to mingle everybody's requests into a single pile for you to process and they don't agree with you about how to do this. Chuck thus gets to submit a request which is mingled with Alice's and you end up letting Chuck modify Alice's request. Oops. But Chuck sending requests directly to your backend doesn't allow him to do this. You're definitely not going to think Chuck's weird garbled nonsense is part of Alice's request when it isn't even on the same TLS connection.
gavingmiller 7y ago
Anyone know if the `smuggler` tool used is available online? Can't find any reference to it in github or elsewhere, and I'm not familiar with it. Edit: Found it here: https://github.com/gwen001/pentest-tools/blob/master/smuggler.py https://github.com/gwen001/pentest-tools/blob/master/smuggle...
- dt3ft 7y ago
  Thank you, I was looking for this as well.
anonfunction 7y ago
> Re: disclosure - a redacted disclosure will be fine, but we'll need to hold off for a little bit while we perform our investigation. We'll keep you updated in the meantime, and once we've concluded there wasn't a customer impact we can disclose this. Thanks for your patience! Does this mean they wouldn't want to disclose it if the same exploit was used against users by another hacker?
- jaywalk 7y ago
  No, it means they'd want affected customers to hear it from them first.
- jessaustin 7y ago
  Maybe they could have had similar vulnerabilities in other parts of their stack?
mcherm 7y ago
> I did not expect for this finding to go from submit to fix/bounty in a matter of 24 hours. I didn't expect that either. I am very pleasantly surprised. I hope my own company would do as well as Slack did here, but I am not certain whether we would.
- caymanjim 7y ago
  This was probably a one-line code fix in the end. I'm sure they added extra tests and guards, but at its core, this is a trivial change. I've worked at less-than-agile companies before, but if the turnaround on something like this is more than 24 hours, that's pretty bad.
phonebucket 7y ago
I'm impressed by how competent and professional some independent security researchers are. How do people learn this stuff? Are there any resources that anyone here can recommend?
- JMTQp8lwXL 7y ago
  In this particular instance, it's a savvy understanding of the HTTP protocol. Both the 'Transfer-Encoding' and 'Content-Length' headers have opposite goals: one says how many bytes of response body data there is; the other signals that it is limitless until (IIRC) an empty newline is transmitted. Realizing that different systems resolve the question of "Well, what should I (the system) do when I get both headers?" and you realize you can break the atomicity of the HTTP request-- which opens up a really interesting (and severe) class of exploits. I'm not very good at finding vulnerabilities, beyond the obvious sql-injection attack types that we are generally trained to avoid. I imagine, however, getting good at finding these is a skill that can be learned with time. Alternatively, finding vulnerabilities might be more akin to pharmaceutical developments: with a 1,000 swings, you're missing at least 950 of them, or more. A dash of luck in there.
  - tptacek 7y ago
    It probably helps to know that this particular bug was a major announcement at Black Hat last year, by James Kettle, who is a vulnerability research celebrity. So lots of people are looking for this particular bug. Black Hat talks are eventually published online for free; here's this one: https://www.youtube.com/watch?v=upEMlJeU_Ik https://www.youtube.com/watch?v=upEMlJeU_Ik
    - JMTQp8lwXL 7y ago
      Given how widely publicized request smuggling was, I'm surprised it's still a problem for apps with large user bases like Slack.
      - lonelappde 7y ago
lala26in 7y ago
I read it and didn't understand now questioning my software engineering career choices.
londons_explore 7y ago
And the real lesson here is HTTP probably isn't a good protocol between your proxies and your backends, and you should probably use HTTP/3 to fully eliminate this entire class of bug.
- SahAssar 7y ago
  HTTP/2 also works to fix this, right?
  - kevin_thibedeau 7y ago
    Proxies would be forced to keep track of headers.
deleted 7y ago
[deleted]
ec109685 7y ago
The fact that you can steal cookies like this is such a flaw in the way the web works. We need to move to tokens that the browser is in control of that provides this device has access to this resource.
_em_ 7y ago
is there any website which teaches these kind of hacking? I am interested in it but not as a full time job.

Lex-2008 7y ago

Greatly detailed report and impressive resolution speed, indeed, but sadly it fell behind the cracks regarding disclosure.

andrekorol 7y ago
What do you mean by "it fell behind the cracks regarding disclosure"?
- harrier 7y ago
  After the issue was resolved the reporter was asked to wait before disclosure. The reporter waited three months before asking about it again. After that there was response but it took another month the approve the disclosure.
  - andrekorol 7y ago
    Oh, now I get it. Thanks for the clarification.

Bhilai 7y ago

Original Paper for those interested - https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request...

wpietri 7y ago

I was unfamiliar with request smuggling; here's an explainer: https://portswigger.net/web-security/request-smuggling https://portswigger.net/web-security/request-smuggling The first in-band signalling attack I came across was the blue box [1], invented in the late 1960s. It still occasionally worked in the 1980s on older phone systems. It amazes me that we're still creating new systems vulnerable to in-band attacks. [1] https://en.wikipedia.org/wiki/Blue_box https://en.wikipedia.org/wiki/Blue_box

sixstringtheory 7y ago
Wow, I was fascinated by phreaking when I was younger and first getting into programming. Been working on mac/iOS for years and never knew that the guys responsible for the most enjoyable part of my career, the Steves, built and sold a blue box!
- at-fates-hands 7y ago
  Of all the stories I heard, Woz was way more interested in using them. When Jobs found it out it was illegal, he dropped selling them and distanced himself from the other phreakers he was hanging out with, too afraid of the legal ramifications of getting caught with this equipment. Most of the books I've read basically paint Jobs as a goody-two-shoes type who cringed at the illegality of phreaking - while saying he probably was thinking ahead, knowing he already wanted to start his own company and didn't want stuff like this to come back and haunt him. It was an interesting dichotomy to me.
abhishekjha 7y ago
Is it what Steve Jobs and his friend used to hijack telephone lines and call the pope, free of cost?
- acruns 7y ago
  Yes. https://www.inc.com/glenn-leibowitz/in-a-rare-23-year-old-interview-steve-jobs-said-this-1-pivotal-experience-inspired-him-to-start-apple-computer.html https://www.inc.com/glenn-leibowitz/in-a-rare-23-year-old-in...
- AnIdiotOnTheNet 7y ago
  Of all the places on the internet to hear Steve Wozniak referred to simply as Steve Jobs's friend...
  - abhishekjha 7y ago
    Sorry, I didn't realise that other friend in this incient was Steve Wozniak specifically. I saw his interview after his return to Apple.

fiberoptick 7y ago

Wow, an ATO exploit that only received a $6,500 bounty? This signals to grey-/black-hat researchers that their research efforts or Slack bug disclosures are best directed elsewhere..

edoceo 7y ago
How much should they have been paid? This seems like a week of pay for a pretty good software dev. That's not fair comp?
- penagwin 7y ago
  > This seems like a week of pay for a pretty good software dev. Wait, 6500 * 4 * 12 = 312,000/yr Might be a bit high for a week :P EDIT: Okay turns out that if you live in the Bay area this isn't unheard of - the rest of us make 4x to 5x less then that (Saying this as a mid-level software dev from West Michigan).
  - randlet 7y ago
    Salary information on HN is skewed by Bay area engineers where $312k is not out of the realm of possibility for a senior developer afaik.
  - moneromoney 7y ago
    In Germany you will be VERY lucky making 100,000 euro / year. Germany has the lowest software dev. salary / average country salary ratio of all countries on planet.
    - foepys 7y ago
      Unless you are doing anything more or less related to SAP. Then you can make $150k/a easily. If only other companies would realize that that's one of the reasons why SAP is Germany's no. 1 software company.
  - sciurus 7y ago

arkadiyt 7y ago

Protecting against request smuggling: - If you don't have a proxy fronting traffic, no action required - If you're behind Fastly/Cloudflare [1] or Akamai [2], no action required / they protect against this attack - If you're behind AWS Cloudfront, no action required / they protect against this attack - If you're behind AWS ALB, you're vulnerable by default but can opt-in to protection by enabling the "routing.http.drop_invalid_header_fields.enabled" attribute [3]. They initially had it on by default but it broke customers - If you have a different proxy (e.g. some other provider or your own nginx, haproxy before 2.0.6 [2], etc), you might be vulnerable [1]: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn https://portswigger.net/research/http-desync-attacks-request... [2]: https://portswigger.net/research/http-desync-attacks-what-happened-next https://portswigger.net/research/http-desync-attacks-what-ha... [3]: https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_LoadBalancerAttribute.html https://docs.aws.amazon.com/elasticloadbalancing/latest/APIR...

judge2020 7y ago
Regarding Cloudflare/fastly, you do need to make sure you're only allowing requests that originate from the proxy, either via IP-based firewall rules or something like CF's authenticated origin pulls [0]. Otherwise someone could find your origin server's IP and potentially perform this attack (and generally bypass your security settings). 0: https://support.cloudflare.com/hc/en-us/articles/204899617-Authenticated-Origin-Pulls https://support.cloudflare.com/hc/en-us/articles/204899617-A...
- tialaramex 7y ago
  Allowing Connections to your backend directly might make you vulnerable to certain types of attack but it doesn't impact Request Smuggling. The trick in Request Smuggling is that you're trusting an intermediary (in this case a frontend reverse proxy) to mingle everybody's requests into a single pile for you to process and they don't agree with you about how to do this. Chuck thus gets to submit a request which is mingled with Alice's and you end up letting Chuck modify Alice's request. Oops. But Chuck sending requests directly to your backend doesn't allow him to do this. You're definitely not going to think Chuck's weird garbled nonsense is part of Alice's request when it isn't even on the same TLS connection.

gavingmiller 7y ago

Anyone know if the `smuggler` tool used is available online? Can't find any reference to it in github or elsewhere, and I'm not familiar with it. Edit: Found it here: https://github.com/gwen001/pentest-tools/blob/master/smuggler.py https://github.com/gwen001/pentest-tools/blob/master/smuggle...

dt3ft 7y ago
Thank you, I was looking for this as well.

anonfunction 7y ago

> Re: disclosure - a redacted disclosure will be fine, but we'll need to hold off for a little bit while we perform our investigation. We'll keep you updated in the meantime, and once we've concluded there wasn't a customer impact we can disclose this. Thanks for your patience! Does this mean they wouldn't want to disclose it if the same exploit was used against users by another hacker?

jaywalk 7y ago
No, it means they'd want affected customers to hear it from them first.
jessaustin 7y ago
Maybe they could have had similar vulnerabilities in other parts of their stack?

mcherm 7y ago

> I did not expect for this finding to go from submit to fix/bounty in a matter of 24 hours. I didn't expect that either. I am very pleasantly surprised. I hope my own company would do as well as Slack did here, but I am not certain whether we would.

caymanjim 7y ago
This was probably a one-line code fix in the end. I'm sure they added extra tests and guards, but at its core, this is a trivial change. I've worked at less-than-agile companies before, but if the turnaround on something like this is more than 24 hours, that's pretty bad.

phonebucket 7y ago

I'm impressed by how competent and professional some independent security researchers are. How do people learn this stuff? Are there any resources that anyone here can recommend?

JMTQp8lwXL 7y ago
In this particular instance, it's a savvy understanding of the HTTP protocol. Both the 'Transfer-Encoding' and 'Content-Length' headers have opposite goals: one says how many bytes of response body data there is; the other signals that it is limitless until (IIRC) an empty newline is transmitted. Realizing that different systems resolve the question of "Well, what should I (the system) do when I get both headers?" and you realize you can break the atomicity of the HTTP request-- which opens up a really interesting (and severe) class of exploits. I'm not very good at finding vulnerabilities, beyond the obvious sql-injection attack types that we are generally trained to avoid. I imagine, however, getting good at finding these is a skill that can be learned with time. Alternatively, finding vulnerabilities might be more akin to pharmaceutical developments: with a 1,000 swings, you're missing at least 950 of them, or more. A dash of luck in there.
- tptacek 7y ago
  It probably helps to know that this particular bug was a major announcement at Black Hat last year, by James Kettle, who is a vulnerability research celebrity. So lots of people are looking for this particular bug. Black Hat talks are eventually published online for free; here's this one: https://www.youtube.com/watch?v=upEMlJeU_Ik https://www.youtube.com/watch?v=upEMlJeU_Ik
  - JMTQp8lwXL 7y ago
    Given how widely publicized request smuggling was, I'm surprised it's still a problem for apps with large user bases like Slack.
    - lonelappde 7y ago

lala26in 7y ago

I read it and didn't understand now questioning my software engineering career choices.

londons_explore 7y ago

And the real lesson here is HTTP probably isn't a good protocol between your proxies and your backends, and you should probably use HTTP/3 to fully eliminate this entire class of bug.

SahAssar 7y ago
HTTP/2 also works to fix this, right?
- kevin_thibedeau 7y ago
  Proxies would be forced to keep track of headers.

deleted 7y ago

[deleted]

ec109685 7y ago

The fact that you can steal cookies like this is such a flaw in the way the web works. We need to move to tokens that the browser is in control of that provides this device has access to this resource.

_em_ 7y ago

is there any website which teaches these kind of hacking? I am interested in it but not as a full time job.