8 ms·

Cloudflare Attack Post Mortem; Apparent Google Apps/Gmail Vulnerability

118 points by demonfly77 14y ago

demonfly77 14y ago
this is the tweet where CosmoTheGod states that UGNazi hacked Cloudflare and 4chan https://twitter.com/CosmoTheGod/status/208656791708508160 https://twitter.com/CosmoTheGod/status/208656791708508160
lomegor 14y ago
This is really important news. It was mostly social engineering hacking, but it seems to have been very clever. It's important to notice that it wasn't a flaw in two factor authentication itself, but on the recovery options. I really hate the recovery options. I don't know what they are there. If I forgot my password, my recovery e-mail, I don't want to answer questions to be sure it was me. Who knows how this exactly worked, but it sure seems to be a problem with that logic.
- joelhaasnoot 14y ago
  So it seems that there was a bug in the two factor authentication affecting "some accounts" during the recovery procedure (see the update to the article). Slightly worrying that the two factor authentication can be bypassed when resetting a password.
  - boundlessdreamz 14y ago
    But even after resetting the password, two factor authentication is needed to log in right? How was that bypassed?
    - monopede 14y ago
      The blog author didn't actually have two factor authentication enabled. The headline is wrong.
      - Strom 14y ago
        You are incorrect, quote from the article: all CloudFlare.com accounts use two-factor authentication. We are still working with Google to understand how the hacker was able to reset the password without providing a valid two-factor authentication token.
      - boundlessdreamz 14y ago
        He didn't have it for his personal account. But he did have it for cloudfare.com accounts.
quarterto 14y ago
WHERE IS YOUR GOD NOW, JEFF? [1] [1]: http://www.codinghorror.com/blog/2012/04/make-your-email-hacker-proof.html http://www.codinghorror.com/blog/2012/04/make-your-email-hac...
- stephengillie 14y ago
  If only that admin had followed this suggestion for his personal account...
- stack0v3erfl0w 14y ago
  Have you read the article ? the 2 factor authentication wasn't compromised, it was mostly a social engineering attack. Even if it was the 2 factor authentication that was compromised, how would this be Jeff's fault ?
  - eastdakota 14y ago
    There was a flaw in the system that Google uses to allow the transfer of control from an account. Two-factor authentication wasn't compromised itself, but the attacker was able to bypass it to access it. That flaw, they tell us, has since been patched on their end.
    - boundlessdreamz 14y ago
      Can you explain this? After the attacker reset the password of cloudfare.com email address what exactly did they do? How exactly did they login?
citricsquid 14y ago
Cloudflare supports hacking group by providing them services -> gets hacked by them. Reminds me very much of what happens in every episode of 24, the terrorists kill their accomplices after they're finished with them, it never works out for anyone involved.
- daeken 14y ago
  Which "hacking group" was supported by Cloudflare?
  - redouane 14y ago
    lulzsec
  - citricsquid 14y ago
    Lulzsec was the most prominent, they used Cloudflare to keep their sites online, Cloudflare used it as a marketing tool, here's a blog post about it: http://blog.cloudflare.com/58611873 http://blog.cloudflare.com/58611873
  - tedivm 14y ago
    Cloudflare doesn't necessarily "support" hacker groups- although their CEO did a great talk at Defcon last year about keeping Lulsec up (which I consider more of an activist group than a hacker group, personally). However, they do very little to keep their network safe- they have no dedicated abuse team (or, if they do, they're impossible to contact directly), are slow to response to abuse requests (if they respond at all), and even when they do respond take only the most basic steps to filter the issues (which takes literally minutes for their account holders to work around, and they don't actually disable the accounts). Dozens, if not hundreds, of hacker and malware related sites use Cloudflare to mask their origin and essentially use the other shared sites as hostages. If you block one of the malicious sites then you also end up blocking legitimate sites, and since Cloudflare goes out of their way to not take down sites (even those hosting malicious programs- including the drive by exploits that install them) it makes it very difficult to trace the malware back to it's source or actually take it down and stop people from getting infected.
ashconnor 14y ago
The headline of this submission is misleading. tl;dr; Hacker used social engineering to add a secondary recovery email address to the victims Google account. 2-factor wasn't even enabled on the victims personal account but the hacker didn't need a 2-factor code to reset the password on the victims company account. From the article: Google reports that they discovered a "subtle flaw affecting not 2-step verification itself, but the account recovery flow for some accounts. We've now blocked that attack vector to prevent further abuse."
- stephengillie 14y ago
  The title of the article doesn't even mention 2-factor: Post Mortem: Today's Attack; Apparent Google Apps/Gmail Vulnerability; and How to Protect Yourself Don't commingle business and personal accounts.
- Strom 14y ago
  It's not misleading. It should not be possible to recover your way past 2-factor auth, which was enabled for the company account.
  - deleted 14y ago
    [deleted]
- Havoc 14y ago
  Thank you. I was skeptical about that headline but didn't feel like reading that wall of text just to make sure its BS.
- kinleyd 14y ago
  Yes, it qualifies as social engineering rather than cracking. The original title is more appropriate.
  - deleted 14y ago
    [deleted]
- eli 14y ago
StavrosK 14y ago
If I understand it correctly, the attacker managed to reset the password by adding a secondary, fraudulent email account as a recovery option, and Google just logged them in after the password was reset, not requiring a token from the 2-factor system. That's quite the attack.
demonfly77 14y ago
as stated on Softpedia, UGNazi got access to all Cloudflare customers' IPs and IDs, and they will sell these informations http://news.softpedia.com/news/CloudFlare-Reports-Breach-UGNazi-Takes-Credit-273334.shtml http://news.softpedia.com/news/CloudFlare-Reports-Breach-UGN...
rdl 14y ago
There are a lot of good reasons not to use gmail or google hosted apps (performance, especially, but apparently also security). Two factor auth as google implements it goes partway, but email really is the skeleton key to all accounts. I'm not sure how meaningful Google Authenticator is as two factor auth when you're actually using the iPhone or iPad to access mail much of the time. Google Authenticator protects against untargeted mass attacks, but if you are specifically a target, it's a pretty mediocre two factor solution. I think it's fair to consider administrators of any major SaaS app to be specific targets. Unfortunately, there really isn't a great alternative. The next best thing I've found is to just run your own mail server, which is pretty obnoxious if you have any non-technical users to support. Kerio is the thing which most Mac shops I know use (since people want directory services and the other stuff you can't easily implement just from open source). Yet, there's better search on server on gmail than from iOS to IMAP (i.e. Kerio), so that's a sacrifice. I kind of hate Google for being the Craigslist of email -- a crappy product at $50/yr which sets the maximum price people are willing to pay below the point where anyone can offer a less crappy product. More like $250-$500/yr/mailbox would be a supportable price for real email.
- tommi 14y ago
  Please define "real email" and what justifies it to cost $250-$500/yr/mailbox in your opinion.
  - Terretta 14y ago
    If you're questioning the grandparent's notion that Gmail is "crappy" compared to expensive email products, I'd agree with you. For me, the answer to what justifies a cost for email is deliverability. Of course, having recipients receive your email promptly, and receiving theirs promptly. But more importantly, receiving next to zero spam in my actual inbox, and misclassifying absolutely zero ham in my spam folder. The only solution I've found that accomplishes this at any price is GMail / Google Apps. Not only does it easily keep 12 - 18 year old email addresses that have been on usenet spam free, it happily handled over 3000 spam per second per box while letting through valid emails for sex.com. The owners could actually use their addresses on Google Apps. They hadn't been able to on email systems costing 4 and even 5 figures. Refreshing the spam label while seeing the inbox stay empty, day after day and month after month, was astonishing.
sauteedbiscuits 14y ago
"We have senior contacts at Google who we worked with in order to regain control of the Google Apps accounts" Boy, its nice they get that privilege. I love gmail, but god help me the day I lose access to my account- I can only imagine the support whack-a-mole us mortals would have to go through to get any resolution. Anyone remember the "thomas monopoly" story from a while back where gmail disabled the guy for weeks with no comment and the dude had to contact every person in the world to get someone to look at it? (And then they reinstated his account)
- yaix 14y ago
  My first though too. They got great support, because they have a name. Any normal Gmail user would receive five automated reply emails from Goog, at most. I'd love to move my email somewhere else, but there is not really any alternative.
  - antidoh 14y ago
    Here's a list of alternatives, and discussion. http://www.emaildiscussions.com/showthread.php?t=64118 http://www.emaildiscussions.com/showthread.php?t=64118
- fluidcruft 14y ago
  Google Apps accounts are billed and offer phone support, right? I don't think there's a mechanism to upgrade/migrate free Gmail accounts to Google Apps, though.
- notatoad 14y ago
  Cloudflare isn't special. Any business (or individual) who pays for google apps gets 24/7 email and phone support. I hate this perception that google has terrible customer support. If you want support, stop being a free user.
  - larrys 14y ago
    Google having 24/7 email and phone support for google apps doesn't seem to jive with this statement in the parent: "We have senior contacts at Google who we worked with" Saying "senior contacts" seems to indicate that they have specific high level people that they personally know who were able to escalate beyond what the response would be for a typical user.
deleted 14y ago
[deleted]
druiid 14y ago
So on one side Cloudflare is saying that they just compromised a single account and on the other we have the hacker claiming that all data was compromised... I guess at that point the question becomes not whom is telling the truth, but what any Cloudflare users (myself included) should be doing right about now?
- notatoad 14y ago
  Presumably after the hacker compromised the CEOs email account he had access to any account he wanted, as I interpret the blog post he was only able to use that access to compromise a single account before being locked out. Nobody is necessarily lying.
runn1ng 14y ago
Just an information - the unnamed customer was 4chan.org, and, according to the group ... I think twitter, it was because they don't like My Little Pony fans, that have a board over there. Stupid internet drama indeed.
veverkap 14y ago
Seems to me that there were two flaws evident here. One was Google's reset flow, which they have since (hopefully) corrected. The other issue, that seemed to be glossed over, was that they were BCCing password reset emails to a compromisable account for support/debugging purposes. I understand the desire to be aware of fradulent submissions early, but we, as an industry, should be treating password reset emails (and tokens) like passwords. Do not store them in plain text. I appreciate CloudFlare's transparency on this, but feel they missed an opportunity to address a flaw that probably exists in a large number of other apps.
- FuzzyDunlop 14y ago
  Copying the emails stood out to me as a ridiculously stupid thing to do. It was probably a quick and dirty solution to save coding up a way to monitor resets on the backend.
- MPSimmons 14y ago
  They say that they have disabled that now, so that's something.
wpietri 14y ago
The most interesting part to me is that somebody managed to get phone support for GMail.
- ericabiz 14y ago
  Google Apps for Business (a paid solution) offers phone support: http://www.google.com/apps/intl/en/business/upgrade.html#utm_campaign=upgrade&utm_source=hc-global-upgrade&utm_medium=et http://www.google.com/apps/intl/en/business/upgrade.html#utm... Since they were using their own domain name, that's likely what they were on.
lsh123 14y ago
The funny part - cloudflare still hosts DNS for UGNazi Domain Name: UGNAZI.COM Registrar: ENOM, INC. Whois Server: whois.enom.com Referral URL: http://www.enom.com Name Server: LEE.NS.CLOUDFLARE.COM Name Server: RUTH.NS.CLOUDFLARE.COM Status: clientTransferProhibited Updated Date: 29-may-2012 Creation Date: 22-jan-2012 Expiration Date: 22-jan-2013

118 points by demonfly77 14y ago

this is the tweet where CosmoTheGod states that UGNazi hacked Cloudflare and 4chan https://twitter.com/CosmoTheGod/status/208656791708508160 https://twitter.com/CosmoTheGod/status/208656791708508160

lomegor 14y ago

This is really important news. It was mostly social engineering hacking, but it seems to have been very clever. It's important to notice that it wasn't a flaw in two factor authentication itself, but on the recovery options. I really hate the recovery options. I don't know what they are there. If I forgot my password, my recovery e-mail, I don't want to answer questions to be sure it was me. Who knows how this exactly worked, but it sure seems to be a problem with that logic.

joelhaasnoot 14y ago
So it seems that there was a bug in the two factor authentication affecting "some accounts" during the recovery procedure (see the update to the article). Slightly worrying that the two factor authentication can be bypassed when resetting a password.
- boundlessdreamz 14y ago
  But even after resetting the password, two factor authentication is needed to log in right? How was that bypassed?
  - monopede 14y ago
    The blog author didn't actually have two factor authentication enabled. The headline is wrong.
    - Strom 14y ago
      You are incorrect, quote from the article: all CloudFlare.com accounts use two-factor authentication. We are still working with Google to understand how the hacker was able to reset the password without providing a valid two-factor authentication token.
    - boundlessdreamz 14y ago
      He didn't have it for his personal account. But he did have it for cloudfare.com accounts.

quarterto 14y ago

WHERE IS YOUR GOD NOW, JEFF? [1] [1]: http://www.codinghorror.com/blog/2012/04/make-your-email-hacker-proof.html http://www.codinghorror.com/blog/2012/04/make-your-email-hac...

stephengillie 14y ago
If only that admin had followed this suggestion for his personal account...
stack0v3erfl0w 14y ago
Have you read the article ? the 2 factor authentication wasn't compromised, it was mostly a social engineering attack. Even if it was the 2 factor authentication that was compromised, how would this be Jeff's fault ?
- eastdakota 14y ago
  There was a flaw in the system that Google uses to allow the transfer of control from an account. Two-factor authentication wasn't compromised itself, but the attacker was able to bypass it to access it. That flaw, they tell us, has since been patched on their end.
  - boundlessdreamz 14y ago
    Can you explain this? After the attacker reset the password of cloudfare.com email address what exactly did they do? How exactly did they login?

citricsquid 14y ago

Cloudflare supports hacking group by providing them services -> gets hacked by them. Reminds me very much of what happens in every episode of 24, the terrorists kill their accomplices after they're finished with them, it never works out for anyone involved.

daeken 14y ago
Which "hacking group" was supported by Cloudflare?
- redouane 14y ago
  lulzsec
- citricsquid 14y ago
  Lulzsec was the most prominent, they used Cloudflare to keep their sites online, Cloudflare used it as a marketing tool, here's a blog post about it: http://blog.cloudflare.com/58611873 http://blog.cloudflare.com/58611873
- tedivm 14y ago
  Cloudflare doesn't necessarily "support" hacker groups- although their CEO did a great talk at Defcon last year about keeping Lulsec up (which I consider more of an activist group than a hacker group, personally). However, they do very little to keep their network safe- they have no dedicated abuse team (or, if they do, they're impossible to contact directly), are slow to response to abuse requests (if they respond at all), and even when they do respond take only the most basic steps to filter the issues (which takes literally minutes for their account holders to work around, and they don't actually disable the accounts). Dozens, if not hundreds, of hacker and malware related sites use Cloudflare to mask their origin and essentially use the other shared sites as hostages. If you block one of the malicious sites then you also end up blocking legitimate sites, and since Cloudflare goes out of their way to not take down sites (even those hosting malicious programs- including the drive by exploits that install them) it makes it very difficult to trace the malware back to it's source or actually take it down and stop people from getting infected.

ashconnor 14y ago

The headline of this submission is misleading. tl;dr; Hacker used social engineering to add a secondary recovery email address to the victims Google account. 2-factor wasn't even enabled on the victims personal account but the hacker didn't need a 2-factor code to reset the password on the victims company account. From the article: Google reports that they discovered a "subtle flaw affecting not 2-step verification itself, but the account recovery flow for some accounts. We've now blocked that attack vector to prevent further abuse."

stephengillie 14y ago
The title of the article doesn't even mention 2-factor: Post Mortem: Today's Attack; Apparent Google Apps/Gmail Vulnerability; and How to Protect Yourself Don't commingle business and personal accounts.
Strom 14y ago
It's not misleading. It should not be possible to recover your way past 2-factor auth, which was enabled for the company account.
- deleted 14y ago
  [deleted]
Havoc 14y ago
Thank you. I was skeptical about that headline but didn't feel like reading that wall of text just to make sure its BS.
kinleyd 14y ago
Yes, it qualifies as social engineering rather than cracking. The original title is more appropriate.
- deleted 14y ago
  [deleted]
eli 14y ago

StavrosK 14y ago

If I understand it correctly, the attacker managed to reset the password by adding a secondary, fraudulent email account as a recovery option, and Google just logged them in after the password was reset, not requiring a token from the 2-factor system. That's quite the attack.

demonfly77 14y ago

as stated on Softpedia, UGNazi got access to all Cloudflare customers' IPs and IDs, and they will sell these informations http://news.softpedia.com/news/CloudFlare-Reports-Breach-UGNazi-Takes-Credit-273334.shtml http://news.softpedia.com/news/CloudFlare-Reports-Breach-UGN...

rdl 14y ago

There are a lot of good reasons not to use gmail or google hosted apps (performance, especially, but apparently also security). Two factor auth as google implements it goes partway, but email really is the skeleton key to all accounts. I'm not sure how meaningful Google Authenticator is as two factor auth when you're actually using the iPhone or iPad to access mail much of the time. Google Authenticator protects against untargeted mass attacks, but if you are specifically a target, it's a pretty mediocre two factor solution. I think it's fair to consider administrators of any major SaaS app to be specific targets. Unfortunately, there really isn't a great alternative. The next best thing I've found is to just run your own mail server, which is pretty obnoxious if you have any non-technical users to support. Kerio is the thing which most Mac shops I know use (since people want directory services and the other stuff you can't easily implement just from open source). Yet, there's better search on server on gmail than from iOS to IMAP (i.e. Kerio), so that's a sacrifice. I kind of hate Google for being the Craigslist of email -- a crappy product at $50/yr which sets the maximum price people are willing to pay below the point where anyone can offer a less crappy product. More like $250-$500/yr/mailbox would be a supportable price for real email.

tommi 14y ago
Please define "real email" and what justifies it to cost $250-$500/yr/mailbox in your opinion.
- Terretta 14y ago
  If you're questioning the grandparent's notion that Gmail is "crappy" compared to expensive email products, I'd agree with you. For me, the answer to what justifies a cost for email is deliverability. Of course, having recipients receive your email promptly, and receiving theirs promptly. But more importantly, receiving next to zero spam in my actual inbox, and misclassifying absolutely zero ham in my spam folder. The only solution I've found that accomplishes this at any price is GMail / Google Apps. Not only does it easily keep 12 - 18 year old email addresses that have been on usenet spam free, it happily handled over 3000 spam per second per box while letting through valid emails for sex.com. The owners could actually use their addresses on Google Apps. They hadn't been able to on email systems costing 4 and even 5 figures. Refreshing the spam label while seeing the inbox stay empty, day after day and month after month, was astonishing.

sauteedbiscuits 14y ago

"We have senior contacts at Google who we worked with in order to regain control of the Google Apps accounts" Boy, its nice they get that privilege. I love gmail, but god help me the day I lose access to my account- I can only imagine the support whack-a-mole us mortals would have to go through to get any resolution. Anyone remember the "thomas monopoly" story from a while back where gmail disabled the guy for weeks with no comment and the dude had to contact every person in the world to get someone to look at it? (And then they reinstated his account)

yaix 14y ago
My first though too. They got great support, because they have a name. Any normal Gmail user would receive five automated reply emails from Goog, at most. I'd love to move my email somewhere else, but there is not really any alternative.
- antidoh 14y ago
  Here's a list of alternatives, and discussion. http://www.emaildiscussions.com/showthread.php?t=64118 http://www.emaildiscussions.com/showthread.php?t=64118
fluidcruft 14y ago
Google Apps accounts are billed and offer phone support, right? I don't think there's a mechanism to upgrade/migrate free Gmail accounts to Google Apps, though.
notatoad 14y ago
Cloudflare isn't special. Any business (or individual) who pays for google apps gets 24/7 email and phone support. I hate this perception that google has terrible customer support. If you want support, stop being a free user.
- larrys 14y ago
  Google having 24/7 email and phone support for google apps doesn't seem to jive with this statement in the parent: "We have senior contacts at Google who we worked with" Saying "senior contacts" seems to indicate that they have specific high level people that they personally know who were able to escalate beyond what the response would be for a typical user.

deleted 14y ago

[deleted]

druiid 14y ago

So on one side Cloudflare is saying that they just compromised a single account and on the other we have the hacker claiming that all data was compromised... I guess at that point the question becomes not whom is telling the truth, but what any Cloudflare users (myself included) should be doing right about now?

notatoad 14y ago
Presumably after the hacker compromised the CEOs email account he had access to any account he wanted, as I interpret the blog post he was only able to use that access to compromise a single account before being locked out. Nobody is necessarily lying.

runn1ng 14y ago

Just an information - the unnamed customer was 4chan.org, and, according to the group ... I think twitter, it was because they don't like My Little Pony fans, that have a board over there. Stupid internet drama indeed.

veverkap 14y ago

Seems to me that there were two flaws evident here. One was Google's reset flow, which they have since (hopefully) corrected. The other issue, that seemed to be glossed over, was that they were BCCing password reset emails to a compromisable account for support/debugging purposes. I understand the desire to be aware of fradulent submissions early, but we, as an industry, should be treating password reset emails (and tokens) like passwords. Do not store them in plain text. I appreciate CloudFlare's transparency on this, but feel they missed an opportunity to address a flaw that probably exists in a large number of other apps.

FuzzyDunlop 14y ago
Copying the emails stood out to me as a ridiculously stupid thing to do. It was probably a quick and dirty solution to save coding up a way to monitor resets on the backend.
MPSimmons 14y ago
They say that they have disabled that now, so that's something.

wpietri 14y ago

The most interesting part to me is that somebody managed to get phone support for GMail.

ericabiz 14y ago
Google Apps for Business (a paid solution) offers phone support: http://www.google.com/apps/intl/en/business/upgrade.html#utm_campaign=upgrade&utm_source=hc-global-upgrade&utm_medium=et http://www.google.com/apps/intl/en/business/upgrade.html#utm... Since they were using their own domain name, that's likely what they were on.

lsh123 14y ago

The funny part - cloudflare still hosts DNS for UGNazi Domain Name: UGNAZI.COM Registrar: ENOM, INC. Whois Server: whois.enom.com Referral URL: http://www.enom.com Name Server: LEE.NS.CLOUDFLARE.COM Name Server: RUTH.NS.CLOUDFLARE.COM Status: clientTransferProhibited Updated Date: 29-may-2012 Creation Date: 22-jan-2012 Expiration Date: 22-jan-2013