9 ms·

Hijacking Email with Cloudflare Email Routing

262 points by albertpedersen 4y ago

sjatkins 4y ago
Nice catch! Frankly I think this points to a real danger of any central widely used service of which Cloudflare is an example. Any vulnerability or exploitation of such a point jeopardizes far too much.
albertpedersen 4y ago
Here's the write-up of a simple but severe exploit I found in Cloudflare's email forwarding service.
- LinuxBender 4y ago
  That is a good write-up and a good find. Thankyou for publishing it and for the responsible disclosure timeline.
  - DanAtC 4y ago
    Over 7 months after it was fixed? Why not disclose immediately after?
    - LinuxBender 4y ago
      Disclosing a vulnerability immediately after it is discovered has a few problems. One is a risk to the customers, as script kiddies will create git repos full of tools to automation exploitation of the vulnerability. Another risk is that people will jump to conclusions without a proper root cause analysis being performed that determines how this happened, what is required to prevent it from happening and if there may be more aspects to this vulnerability than was were originally thought to exist. Another reason to not disclose immediately would be that in most cases it will violate the agreement the penetration tester or security researcher has with the bug bounty program. Disclosing immediately would mean they do not get paid for their discovery. This payment for bugs concept provides an incentive for people to help a company fix their bugs that their own developers and QA teams may have overlooked.
      - bluehatbrit 4y ago
        I think the parent comment was asking why they didn't disclose immediately after it was fixed.
        LinuxBender 4y ago
        In that case I would have to defer to @albertpedersen
        albertpedersen 4y ago
        The bug was initially reported to Cloudflare's private bug bounty program since there was no public program at the time. Like it or not, the private program does not have the same disclosure policy as the public program does. In early July I asked if the report could be disclosed, seeing as things had changed since the bug was originally reported. Cloudflare agreed and the report was then moved to the public program. As to why it was disclosed now rather than in February when the public program launched, that was my fault for not asking earlier.
upupandup 4y ago
wow this is insane. who knows how many emails were skimmed on cloudflare? definitely will not be trusting this service because how many other vulnerabilities are not yet discovered?
- ejcx 4y ago
  I lead Product Security at Cloudflare (and I'm one of Albert's biggest fans, he's contributed a lot to our bug bounty, thank you Albert). Once he reported the issue we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us. We disclosed the issue here earlier this week: https://hackerone.com/reports/1419341 https://hackerone.com/reports/1419341
- ctippett 4y ago
  I place greater trust in organisations that are proactive about their security posture, versus an organisation where this type of vulnerability would never have been publicly disclosed.
  - upupandup 4y ago
    Proactive but not preventive. You know only after an incident occurs. While I appreciate it, the risk isn't mitigated at all unless you don't use the said service. ex. Heroku
js2 4y ago
> The restrictions were all client-side. :-( I've been in this industry since 1996. Must every programmer make this mistake for themselves before they learn? I see this over, and over, and over again, at company after company after company. I'm really surprised a company as seemingly competent as Cloudflare would make this mistake, to say nothing of the larger error of allowing forwarding to be setup on an unverified domain. I weep. Edit: I'd appreciate a response from Cloudflare on how this slipped through the cracks and what changes they are making to prevent such mistakes in the future. Not trusting user input is among the most basic thing I'd expect a programmer, especially one working at CF, to know in 2022, so I'm assuming this was some sort of miscommunication between teams.
- vxNsr 4y ago
  Yea this was a surprise.
- jgrahamc 4y ago
  It wasn't mean to be client-side only, it was meant to be client-side and server-side. Unfortunately, the server-side check wasn't happening in the way intended allowing Albert to find this vulnerability. We have a special document that describes bug classes that need special attention and this particular incident has been added to it.
  - OJFord 4y ago
    I think you're giving too much weight to this angle - if OP had happened to be in the beta, that part wouldn't have been found or mentioned at all. Bypassing a beta feature toggle isn't really a vulnerability IMO, it just allowed OP to find one in this case.
    - mox1 4y ago
      What!? So nobody here can think of any possible scenario where bypassing a server-side check for 'Account X can access Feature Y' could directly lead to a security issue? This is absolutely 100% a vulnerability - insomuch as that CloudFlare should have an explicit policy that ALL account features are enabled / verified server-side, not client side. Think of all the spam that would have happened, had this been discovered on underground black-hat forums.
nop_slide 4y ago
Only a $6000 reward for being able to intercept someone's email?!
- happyopossum 4y ago
  Likely impacted by the fact that this was in a private beta and not a publicly available product. Betas are “supposed to” have bugs.
  - r1ch 4y ago
    A well-run bug bounty should encourage early reporting of issues. Should he have waited until public release so the impact was bigger?
    - rsstack 4y ago
      It's a moot point because this hasn't happened. Cloudflare didn't lower the payout because it was in beta.
      - r1ch 4y ago
        Yeah I didn't realize CF's payouts were so low.
  - Bytewave81 4y ago
    A "private" beta enforced solely by a clientside check, notably.
    - rsstack 4y ago
      Sure, but the client-side check completely limits the exposure Cloudflare have because big enterprise corporates don't get themselves into a private beta by playing with client-side checks. Only a few companies were using this feature at the time.
- jtokoph 4y ago
jmull3n 4y ago
Good find. I can't believe it was that easy, tests should have caught this.
powerhour 4y ago
> The bug has since been fixed and Cloudflare has kindly allowed me to publish this write-up. Seems wild that they can deny you the opportunity to publish your findings for 7 months after the fix for a beta product went live. What is that about? Was that a requirement to get the bug bounty? Seems like a great way to encourage finding another buyer instead.
- Aaron2222 4y ago
  https://news.ycombinator.com/item?id=32333099 https://news.ycombinator.com/item?id=32333099 > Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any.
deleted 4y ago
[deleted]
ejcx 4y ago
I lead Product Security at Cloudflare, thanks for the writeup Albert and the fantastic security research throughout the past year. Once this issue was fixed we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us. Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any. We disclosed the issue here earlier this week once Albert told us he was writing a blog: https://hackerone.com/reports/1419341 https://hackerone.com/reports/1419341
- alphager 4y ago
  What's missing from your statement is how you plan to prevent this kind of thing (a programming oversight so simple it shouldn't have landed in production) in the future. Something like mandatory security reviews.
- dreadlordbone 4y ago
  Any info about your devs doing only client side verification of the beta flag? Seems like a pretty bad practice.
  - Operyl 4y ago
    Usually rollouts like that are slow to prevent a flood of people using it at once, and to slowly open the flood gates. Not many people will manually change the flag, either. I don't think the fact that the feature flag was just client side is a "smoking gun".
    - brightball 4y ago
      Agreed. Feature flags can be complicated to implement both client side and server side. Ideally you want both but it doesn't always work out that way.
megraf 4y ago
Hey Albert, awesome first post. You made a great deal to include the _right_ amount of detail, and your writing style is sufficient enough to keep me engaged through the entire post. Keep going!
lizardactivist 4y ago
Why would anyone even allow the biggest man-in-the-middle on the Internet to route their e-mail in the first place? It's a situation that will never be private and secure.
- fjni 4y ago
  Elephant in the room.
- JimWestergren 4y ago
  A smaller man-in-the-middle would be more secure? I mean if email forward is really necessary (in my case yes) and successful delivery of email is really important (so I don't want to use my own VPS and the headache it brings) are there any better options for me than CloudFlare? My domains already use CloudFlare for their CDN etc.
  - lizardactivist 4y ago
    If it's smaller and not connected to the US intelligence agencies, it certainly would. But if you use CloudFlare to begin with then there is no absolute notion of security and privacy, but this is of course a viable option if one doesn't do critical, important stuff.
theunixbeard 4y ago
Awesome work, Albert! Looks like you are crushing it on HackerOne, over $37K in bounties? https://hackerone.com/albertspedersen?type=user https://hackerone.com/albertspedersen?type=user You've obviously got a strong career in Security in the future. Have you looked at any Crypto projects? Seems like there are some massive bounties on https://immunefi.com https://immunefi.com and similar sites.
- sneak 4y ago
  Security professionals of this caliber often make $37k in monthly compensation, each and every month. That's only $230/hour. If you can do work like this, your consulting rate is at least that for penetration testing. Bug bounty programs are a bad deal for researchers. The payout for this bug is absurdly low.
  - BeefWellington 4y ago
    Yep. On the hiring side, you can absolutely see this when you get someone's resume. A person with in-industry experience will often not list their HackerOne profile (if they even have one), while students mostly do in my experience. Payouts are a joke and progress is slow. It wasn't that long ago people were overwhelmingly just arrested or threatened for reporting these kinds of things but thankfully that's becoming rarer. The amounts for these bounties though seem to be a token gesture and not much else, especially considering the damage someone could have caused with this.
- sammy2244 4y ago
boredpudding 4y ago
Is the 'Burp' mentioned in this, the 'Burp Suite' that can be downloaded here? https://portswigger.net/burp/communitydownload https://portswigger.net/burp/communitydownload Am new to this kinda stuff but would love to play with it. Always love reading up on responsible disclosures.
- albertpedersen 4y ago
  Yup, 'Burp' refers to the free version of 'Burp Suite'. I don't use Burp Suite anymore though. Some months ago I started using mitmproxy (https://github.com/mitmproxy/mitmproxy https://github.com/mitmproxy/mitmproxy) due to it's Python scripting API. I have never looked back since then.
  - zegerius 4y ago
    This is also my goto. I work with it to do reverse engineering of APIs for apps on Android icw Frida.
dustinmoris 4y ago
Wow did Cloudflare outsource all their development work to some cheap agency or how come they develop features like this with zero security and pretty damn naive, simplistic and childish bugs you wouldn’t expect from a company that routes half of the internet’s traffic through their servers.
wizofaus 4y ago
So now email is unsafe for MFA/password reset messages too? (actually I've long argued it's not really any safer than SMS, at least in countries where there are decent regulations around porting numbers).
JimWestergren 4y ago
Albert, you seem to work a lot with or for CloudFlare in regards to security. Would you trust CloudFlare handling your important email forwards going forward? I recently switched to them for my forwards a few weeks ago, I would feel better if your answer is a yes .... :)

Nice catch! Frankly I think this points to a real danger of any central widely used service of which Cloudflare is an example. Any vulnerability or exploitation of such a point jeopardizes far too much.

albertpedersen 4y ago

Here's the write-up of a simple but severe exploit I found in Cloudflare's email forwarding service.

LinuxBender 4y ago
That is a good write-up and a good find. Thankyou for publishing it and for the responsible disclosure timeline.
- DanAtC 4y ago
  Over 7 months after it was fixed? Why not disclose immediately after?
  - LinuxBender 4y ago
    Disclosing a vulnerability immediately after it is discovered has a few problems. One is a risk to the customers, as script kiddies will create git repos full of tools to automation exploitation of the vulnerability. Another risk is that people will jump to conclusions without a proper root cause analysis being performed that determines how this happened, what is required to prevent it from happening and if there may be more aspects to this vulnerability than was were originally thought to exist. Another reason to not disclose immediately would be that in most cases it will violate the agreement the penetration tester or security researcher has with the bug bounty program. Disclosing immediately would mean they do not get paid for their discovery. This payment for bugs concept provides an incentive for people to help a company fix their bugs that their own developers and QA teams may have overlooked.
    - bluehatbrit 4y ago
      I think the parent comment was asking why they didn't disclose immediately after it was fixed.
      - LinuxBender 4y ago
        In that case I would have to defer to @albertpedersen
        albertpedersen 4y ago
        The bug was initially reported to Cloudflare's private bug bounty program since there was no public program at the time. Like it or not, the private program does not have the same disclosure policy as the public program does. In early July I asked if the report could be disclosed, seeing as things had changed since the bug was originally reported. Cloudflare agreed and the report was then moved to the public program. As to why it was disclosed now rather than in February when the public program launched, that was my fault for not asking earlier.

upupandup 4y ago

wow this is insane. who knows how many emails were skimmed on cloudflare? definitely will not be trusting this service because how many other vulnerabilities are not yet discovered?

ejcx 4y ago
I lead Product Security at Cloudflare (and I'm one of Albert's biggest fans, he's contributed a lot to our bug bounty, thank you Albert). Once he reported the issue we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us. We disclosed the issue here earlier this week: https://hackerone.com/reports/1419341 https://hackerone.com/reports/1419341
ctippett 4y ago
I place greater trust in organisations that are proactive about their security posture, versus an organisation where this type of vulnerability would never have been publicly disclosed.
- upupandup 4y ago
  Proactive but not preventive. You know only after an incident occurs. While I appreciate it, the risk isn't mitigated at all unless you don't use the said service. ex. Heroku

js2 4y ago

> The restrictions were all client-side. :-( I've been in this industry since 1996. Must every programmer make this mistake for themselves before they learn? I see this over, and over, and over again, at company after company after company. I'm really surprised a company as seemingly competent as Cloudflare would make this mistake, to say nothing of the larger error of allowing forwarding to be setup on an unverified domain. I weep. Edit: I'd appreciate a response from Cloudflare on how this slipped through the cracks and what changes they are making to prevent such mistakes in the future. Not trusting user input is among the most basic thing I'd expect a programmer, especially one working at CF, to know in 2022, so I'm assuming this was some sort of miscommunication between teams.

vxNsr 4y ago
Yea this was a surprise.
jgrahamc 4y ago
It wasn't mean to be client-side only, it was meant to be client-side and server-side. Unfortunately, the server-side check wasn't happening in the way intended allowing Albert to find this vulnerability. We have a special document that describes bug classes that need special attention and this particular incident has been added to it.
- OJFord 4y ago
  I think you're giving too much weight to this angle - if OP had happened to be in the beta, that part wouldn't have been found or mentioned at all. Bypassing a beta feature toggle isn't really a vulnerability IMO, it just allowed OP to find one in this case.
  - mox1 4y ago
    What!? So nobody here can think of any possible scenario where bypassing a server-side check for 'Account X can access Feature Y' could directly lead to a security issue? This is absolutely 100% a vulnerability - insomuch as that CloudFlare should have an explicit policy that ALL account features are enabled / verified server-side, not client side. Think of all the spam that would have happened, had this been discovered on underground black-hat forums.

nop_slide 4y ago

Only a $6000 reward for being able to intercept someone's email?!

happyopossum 4y ago
Likely impacted by the fact that this was in a private beta and not a publicly available product. Betas are “supposed to” have bugs.
- r1ch 4y ago
  A well-run bug bounty should encourage early reporting of issues. Should he have waited until public release so the impact was bigger?
  - rsstack 4y ago
    It's a moot point because this hasn't happened. Cloudflare didn't lower the payout because it was in beta.
    - r1ch 4y ago
      Yeah I didn't realize CF's payouts were so low.
- Bytewave81 4y ago
  A "private" beta enforced solely by a clientside check, notably.
  - rsstack 4y ago
    Sure, but the client-side check completely limits the exposure Cloudflare have because big enterprise corporates don't get themselves into a private beta by playing with client-side checks. Only a few companies were using this feature at the time.
jtokoph 4y ago

jmull3n 4y ago

Good find. I can't believe it was that easy, tests should have caught this.

powerhour 4y ago

> The bug has since been fixed and Cloudflare has kindly allowed me to publish this write-up. Seems wild that they can deny you the opportunity to publish your findings for 7 months after the fix for a beta product went live. What is that about? Was that a requirement to get the bug bounty? Seems like a great way to encourage finding another buyer instead.

Aaron2222 4y ago
https://news.ycombinator.com/item?id=32333099 https://news.ycombinator.com/item?id=32333099 > Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any.

deleted 4y ago

[deleted]

ejcx 4y ago

I lead Product Security at Cloudflare, thanks for the writeup Albert and the fantastic security research throughout the past year. Once this issue was fixed we investigated all prior email routing configurations to ensure that this had only been found as part of Albert's responsible disclosure to us. Since some comments are addressing that this happened 7 months ago. Our disclosure policy is to allow researchers to write about us once the issue is fixed, but give us a week heads up before they publish so we aren't surprised, can coordinate any public comms we want to make, FAQs that need to be written for inbound questions from customers, and can tailor our response to the issue at hand. Can answer other questions if you have any. We disclosed the issue here earlier this week once Albert told us he was writing a blog: https://hackerone.com/reports/1419341 https://hackerone.com/reports/1419341

alphager 4y ago
What's missing from your statement is how you plan to prevent this kind of thing (a programming oversight so simple it shouldn't have landed in production) in the future. Something like mandatory security reviews.
dreadlordbone 4y ago
Any info about your devs doing only client side verification of the beta flag? Seems like a pretty bad practice.
- Operyl 4y ago
  Usually rollouts like that are slow to prevent a flood of people using it at once, and to slowly open the flood gates. Not many people will manually change the flag, either. I don't think the fact that the feature flag was just client side is a "smoking gun".
  - brightball 4y ago
    Agreed. Feature flags can be complicated to implement both client side and server side. Ideally you want both but it doesn't always work out that way.

megraf 4y ago

Hey Albert, awesome first post. You made a great deal to include the _right_ amount of detail, and your writing style is sufficient enough to keep me engaged through the entire post. Keep going!

lizardactivist 4y ago

Why would anyone even allow the biggest man-in-the-middle on the Internet to route their e-mail in the first place? It's a situation that will never be private and secure.

fjni 4y ago
Elephant in the room.
JimWestergren 4y ago
A smaller man-in-the-middle would be more secure? I mean if email forward is really necessary (in my case yes) and successful delivery of email is really important (so I don't want to use my own VPS and the headache it brings) are there any better options for me than CloudFlare? My domains already use CloudFlare for their CDN etc.
- lizardactivist 4y ago
  If it's smaller and not connected to the US intelligence agencies, it certainly would. But if you use CloudFlare to begin with then there is no absolute notion of security and privacy, but this is of course a viable option if one doesn't do critical, important stuff.

theunixbeard 4y ago

Awesome work, Albert! Looks like you are crushing it on HackerOne, over $37K in bounties? https://hackerone.com/albertspedersen?type=user https://hackerone.com/albertspedersen?type=user You've obviously got a strong career in Security in the future. Have you looked at any Crypto projects? Seems like there are some massive bounties on https://immunefi.com https://immunefi.com and similar sites.

sneak 4y ago
Security professionals of this caliber often make $37k in monthly compensation, each and every month. That's only $230/hour. If you can do work like this, your consulting rate is at least that for penetration testing. Bug bounty programs are a bad deal for researchers. The payout for this bug is absurdly low.
- BeefWellington 4y ago
  Yep. On the hiring side, you can absolutely see this when you get someone's resume. A person with in-industry experience will often not list their HackerOne profile (if they even have one), while students mostly do in my experience. Payouts are a joke and progress is slow. It wasn't that long ago people were overwhelmingly just arrested or threatened for reporting these kinds of things but thankfully that's becoming rarer. The amounts for these bounties though seem to be a token gesture and not much else, especially considering the damage someone could have caused with this.
sammy2244 4y ago

boredpudding 4y ago

Is the 'Burp' mentioned in this, the 'Burp Suite' that can be downloaded here? https://portswigger.net/burp/communitydownload https://portswigger.net/burp/communitydownload Am new to this kinda stuff but would love to play with it. Always love reading up on responsible disclosures.

albertpedersen 4y ago
Yup, 'Burp' refers to the free version of 'Burp Suite'. I don't use Burp Suite anymore though. Some months ago I started using mitmproxy (https://github.com/mitmproxy/mitmproxy https://github.com/mitmproxy/mitmproxy) due to it's Python scripting API. I have never looked back since then.
- zegerius 4y ago
  This is also my goto. I work with it to do reverse engineering of APIs for apps on Android icw Frida.

dustinmoris 4y ago

Wow did Cloudflare outsource all their development work to some cheap agency or how come they develop features like this with zero security and pretty damn naive, simplistic and childish bugs you wouldn’t expect from a company that routes half of the internet’s traffic through their servers.

wizofaus 4y ago

So now email is unsafe for MFA/password reset messages too? (actually I've long argued it's not really any safer than SMS, at least in countries where there are decent regulations around porting numbers).

JimWestergren 4y ago

Albert, you seem to work a lot with or for CloudFlare in regards to security. Would you trust CloudFlare handling your important email forwards going forward? I recently switched to them for my forwards a few weeks ago, I would feel better if your answer is a yes .... :)