10 ms·

Quantifying the Impact of “Cloudbleed”

287 points by eastdakota 10y ago

79d697i6fdif 10y ago
In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times. Wow, so just as bad as we thought. We did not find any passwords, credit cards, health records, social security numbers, or customer encryption keys in the sample set. BUT WAIT, THERE'S MORE The sample included thousands of pages and was statistically significant to a confidence level of 99% with a margin of error of 2.5%. Oh, so it could actually be as a high as 2.5% leaking encryption credentials. And if none of the data was found to leak anything sensitive where the fuck is the dataset? I've been around way too long to take a "study" like this at face value without third party verification. I also enjoy the straight up lie at the end: We are continuing to work with third party caches to expunge leaked data and will not let up until every bit has been removed. That sounds great right? Well, its too bad that a lot of 'third parties' are a box sitting on the corporate network edge that hasn't been touched in 5 years. Deleting all of this data from third party caches is not physically possible. In fact it might actually make things worse because it's destroying evidence of which credentials were leaked.
- dahdum 10y ago
  1.2 million page views over 5 months is almost nothing for the amount of traffic going through Cloudflare.
  - 79d697i6fdif 10y ago
    And 1.2 million people dying is nothing compared to the number of people on the earth. IMO a leak this bad should be enough to sink cloudflare. A provider of SSL was randomly spitting out private data onto public websites. OVER A MILLION TIMES. Entire CA's have been shut down for leaking a couple hundred certificates. This has leaked private data over a million times, cloudflare is a joke
    - dahdum 10y ago
      > And 1.2 million people dying is nothing compared to the number of people on the earth. A != B
  - mikeash 10y ago
    It seems to me that the absolute number is what's relevant, not how it compares to the total amount of traffic. That's 1.2 million potential data leaks. That's it "out of a bazillion" doesn't change it.
    - user5994461 10y ago
      Correction: It's 1.2 million definitive leaks. The only question is what is in these leaks exactly.
  - Artemis2 10y ago
    For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot. 0: https://youtu.be/LA-gNoxSLCE?t=2m33s https://youtu.be/LA-gNoxSLCE?t=2m33s
- ProAm 10y ago
  If you are this concerned, change your credentials.
  - 79d697i6fdif 10y ago
    So every website that uses cloudflare should ask their users to change all of their passwords, credit card numbers, and SSN's? This leak is being downplayed by webmasters because it's so incredibly bad that there's no way of handling it. The credentials of practically any internet user could have been leaked. The only "safe" way to handle this is to give everyone in the US new credit cards and SSN's and to reset accounts and security questions for every user on a site with cloudflare
    - ProAm 10y ago
      No but judging by how much you are freaking out in the comments, I was recommending you to change yours. Im not really sure what you want anyone else to do? It seems like you are just screaming at your monitor over something that while a significant bug isn't a huge deal. This is just basic risk management.
    - theGimp 10y ago
      Credit cards and SSNs are regularly compromised. The real problem is that they are used as an authentication mechanism. That's what we should be concerned about. This issue is a drop in the bucket when it comes to the amount of sensitive data leaked.
  - zulln 10y ago
    Password would not help if session cookie is leaked. In many instances you cannot do anything towards that, as many services do not have any "logout all"-feature.
    - gommm 10y ago
      And this is why I changed the key I use for cookies on the application I had that was behind Cloudflare. This triggered all users to be logged out and invalidated any session cookie out there. So, yes, responsible websites can mitigate session cookies being leaked. That said, I am not impressed by Cloudflare's transparency which in this case consists of downplaying things, blaming Google and Taviso and not really taking responsibility.
- bifrost 10y ago
  1.3m pages served with "Extra" data is miniscule considering the number of actual pages served. Ideally you'd have no pages served but when you've got a bugproof technology the world will be your oyster.
- dsl 10y ago
  > We are continuing to work with third party caches One of the caches they worked with was Baidu, which has direct ties to Chinese intelligence. Just because it isn't publicly available doesn't mean people aren't still pouring over it looking for useful data.
  - 79d697i6fdif 10y ago
    Also, a lot of web spiders are not benign. All sorts of bots troll the internet looking specifically for data leaks like publicly visible email addresses and SSN's. I'm sure they're having a field day.
advisedwang 10y ago
Disappointing that their response is "damage control" and downplaying impact.
- AwesomeBean 10y ago
  Why? As long as they come along with facts and data, then that's a good thing.
  - 79d697i6fdif 10y ago
    they only sampled a few thousand leaked responses out of over a million. The margin of error is 2.5% on conclusions because they didn't use enough data, not even close.
geforce 10y ago
Was talking with colleagues about cloudbleed and S3 problems yesterday. I don't feel like many people are actually concerned of the implication of having an internet that isn't an internet anymore, but merely a handful of big companies hosting everyone. Or maybe it's me who don't understand.
- Florin_Andrei 10y ago
  It's almost as if self-reinforcing feedback loops are all over society these days. Being at the top makes it that much easier to stay at the top and crush everything else. Internet companies are just one small slice of this trend.
  - 79d697i6fdif 10y ago
    antitrust is all bark with no bite these days. Most industries having 3-4 "competitors" that don't actually compete is the new norm. I wouldn't be amazed if a link is shown between this situation and the increasing wealth inequality
    - thehardsphere 10y ago
      That's because you're viewing this situation entirely from within your prior beliefs about large corporations, which somehow have convinced you that only 3 or 4 hosting companies exist and everyone must use them. Exactly how do you figure this? There are lots of alternatives to S3: 1. All the cloud stuff, which is the new fangled hotness (3 to 4 companies) 2. Non-cloud hosting providers (hundreds of them, from shared hosting to VPSs and above, and usually cheaper if you don't have high traffic) 3. Hosting your own stuff on your own hardware, which is the cheapest if you really need it, and can be done in a data center just about anywhere in the world with good internet. So, what planet is this that you live on where 3 to 4 companies handle all hosting and we have to use them?
    - cookiecaper 10y ago
      Beyond simple anti-trust, in many cases, we need to reform legal structures so that big players can't use them to block out startups. It's practically impossible to defend yourself well in a lawsuit against any respectably-sized company without a few million lying around, first of all. That affects everything, and big companies use that fact to bully upstarts and other small innovators into shutting down all the time. My own business was effectively shuttered (had to stop selling our primary product) by a C&D from a Fortune 100. It would've been 5-10 years and ~$5 million to see that case all the way through, and under current precedent, it's very likely I would have lost. Aside from that, industries frequently get laws and rules put in place with ostensibly-reasonable rationale, while the actual intent is to make it virtually impossible for disruptive competitors to enter the marketplace. The CFAA is the piece of legislation that primarily enshrines entrenched players in the online space. We also need to reform copyright law and clarify some matters regarding the applicability of EULAs, especially with regard to clickwrap and browsewrap. Once that's done, the flood of innovators that have been held back by big companies dispatching their law firms will finally be able to contribute, and the internet's competitve landscape will truly be back in the hands of the users. It will shift from "Who has my data? I have to go with them" to "I can use any interface I want to access that data", effectively resolving the chicken-and-egg effect that imperils any potential competing social network (not even Google could compete with Facebook on this!).
bifrost 10y ago
I'm glad to see an article that isn't all puppies and kittens, this wasn't a fun thing to deal with and certainly wasn't anticipated. The CloudFlare team did a great job responding to it IMHO. "Of the 1,242,071 requests that triggered the bug, we estimate more than half came from search engine crawlers." This is very important to sort out, most people don't think about security much less the storing of credential or identifying data by search engines, this is a huge part of the incident response. I think that what we should take away from this is that even though the bug existed it was responded to in a reasonable manner.
jlg23 10y ago
TL;DR: "Since this is just a sample, it is not correct to conclude that no passwords, credit cards, health records, social security numbers, or customer encryption keys were ever exposed."
koolba 10y ago
> 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and Downplaying how bad it was that customer cookie's were leaked is what I found particularly egregious in the original response. A session cookie is almost as bad as a password leak. Usually it'll let you everything except change your password (as most sites require your current one as an extra validation).
- chipperyman573 10y ago
  Session cookies can be reset much more easily than asking a user to reset their password. Session cookies also aren't used on multiple sites. It's still a big deal, but leaking a session cookie is much less dangerous than leaking a password.
  - koolba 10y ago
    Lots of sites use session cookies. Many don't automatically expire them or rely on browsers to expire them (ex: signed cookies). Plus rotating the master session key (to force the issue of all users being reset) requires knowing you should do it. By downplaying the issue, CloudFlare is sending the message that customers don't have to do anything.
  - hueving 10y ago
    >Session cookies can be reset much more easily than asking a user to reset their password. This statement is making an assertion about the behavior of vastly different software backends and the skill-set of the people administering the systems. Things that may appear trivial to you may not be for the sysadmin in charge of servers running healthcare applications that have an opaque internal state.
saurik 10y ago
It was my understanding that people were finding numerous examples of leaked data that included Authorization headers, which don't always leak passwords but which still leak session keys. Maybe if espadrine from this linked comment reads this, he could explain more? https://news.ycombinator.com/item?id=13719455 https://news.ycombinator.com/item?id=13719455 FWIW, as an example, Grindr uses CloudFlare: do a Google search for "authorization: grindr3" and you will find a URL which (no longer cached but you can still get snippets) contained an authenticated grindr request, which would be enough to have had temporary access to that person's account. https://costumla.com/wild-west-costumes-for-men.html https://costumla.com/wild-west-costumes-for-men.html "... U Edge Certificate Authority1 0 U San Francisco1 0 U California�1p ��U� N��U � � �f"�0! %��U@ @GET /v3/profiles/[REDACTED] HTTP/1.1 CF-RAY: 33282514b8d957d7 FL-Server: 15f76 Host: grindr.mobi X-Real-IP: [REDACTED] Accept-Encoding: gzip Client-Accept-Encoding: gzip X-Forwarded-Proto: https Connect-Via-Https : on Connect-Via-Port: 443 Connect-Via-IP: 104.16.85.62 Connect-Via-Host: grindr.mobi CF-Visitor: {"scheme":"https"} CF-Host-Origin-IP: [REDACTED] Zone-ID: 22252132 Owner-ID: 2607399 CF-Int-Brand-ID: 100 Zone-Name: grindr.mobi Connection: Keep-Alive X-SSL-Protocol: TLSv1.2 X-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256 X-SSL-Server-Name: grindr.mobi X-SSL-Session-Reused: . SSL-Server-IP: 104.16.85.62 X-SSL-Connection-ID: 15d1b8de6025864d-DFW X-SPDY-Protocol: 3.1 authorization: Grindr3 ... accept: application/json user-agent: grindr3/3.0.13.16790;16790;Free;Android 6.0 CF-Use-OB: 0 Set-Expires-TTL: 14400 CF-Cache-Max-File-Size: 512m Set-SSL-Name: grindr.mobi CF-Cache-Level: byc CF-Unbuffered-Upload: 0 Set-SSL-Client-Cert: 0 Set-Limit-Conn-Cache-Host: 50000 CF-WAN-RG5: 0 CF-Brand-Name: cloudflare CF-Age-Header-Enabled: 0 CF-Respect-Strong-Etag: 0 Set-Proxy-Read-Timeout: 100 Set-Proxy-Send-Timeout: 30 CF-Connecting-IP: [REDACTED] Set-Proxy-Connect-Timeout: 90 Set-Cache-Bypass: 0 Set-SSL-Verify: 0 CF-Force-Miss-TS: 0 Set-Buffering: 0 CF-Pref-OB: 1 Set-Keepalive: 1 CF-Pref-Geoloc: 1 CF-Use-BYC: 0 CF-IPCountry: [REDACTED] CF-IPType ..." edit: I have spent the last fifteen minutes pulling the search snippet (edit: and now an hour; but all on my phone, so this is harder than it should be, and I also am distracted by other stuff). The way you do this is by walking through the parts you can see to get nearby context. (I do this a lot to pull content purged from or inaccessible to or simply updated in Google's cache.) In so doing, while I haven't been able to still have access to the session key (likely too long and unique for a snippet), I have pulled the X-Real-IP address field of this Grindr user and a profile identifier they were checking out (both of which I redacted above, but you could trivially get yourself now using that context). CloudFlare: if you think there isn't private data that was leaked, OR EVEN PRIVATE DATA STILL ACCESSIBLE, you are a bunch of fucking idiots. 1) Clearing the cache isn't sufficient, as for anything built from short plain text words we can pull the snippet. 2) IP addresses and session keys count as "private data". 3) GET requests actually are often sensitive information :/.
rattray 10y ago
tl;dr excerpted: In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times. 1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched; 2) the vast majority of Cloudflare customers had no data leaked; 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and 4) our review is ongoing.
- tyingq 10y ago
  >we now estimate based on our logs the bug was triggered 1,242,071 times Based on looking at 1% of their log entries for 10 days of the total ~150 day period.
deleted 10y ago
[deleted]
ryanlol 10y ago
>In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times. So uh, Cloudflare has logs of all page loads since september at the very least? And I guess with response sizes, since that seems like the most reasonable way for them to come up with such a number. Awesome.
- user5994461 10y ago
  Doubt that they have any log, they have explained before that they only keep 4hours of logs because it's too much volume of data. FYI: Traffic is over 4M requests per second. Guess they only kept a counter of requests. A bit of maths and you get an estimate for the amount of leaks.
tptacek 10y ago
Not everything Cloudflare has written about this breach has been good, but this is solid. In particular, I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug. What remains now is an accounting of how some of the most sensitive C code on the Internet was tested prior to the discovery of this bug (by Cloudflare's own accounting, the underlying error seems to have been present long before it became symptomatic), and what they're doing about that now.
- hueving 10y ago
  >I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug. How would you classify people using all of the crawled data though? It seems sort of hand-wavy to claim nobody deliberately exploited the bug if they used leaked session tokens in crawled data to get access to user accounts.
camel_gopher 10y ago
The wording of the post seems to be tilted towards limiting Cloudflare's potential liability as opposed to objectively quantifying the impact. For example, the author mentions that the 100th largest website handles less than 10B requests per month - which positions near the median of the table of likelihood anticipated leaks. I'm not surprised at this approach as the CEO has a background in law. I think that, combined with his hubris, shaped the tone of this post from an objective post mortem to a goal of minimizing the damage to Cloudflare itself. I met Mr. Prince once at a tech conference. I mentioned that I followed him on Twitter, to which he freaked out a little bit and said "Oh I hate that when I run across people who tell me that". When I heard him tell his name to someone else there, he said his last name was "Prince, like King". I suspect that his attitude is prevalent at Cloudflare, and may have shaped not only the source of this issue, but the response to it as well, since CEO's generally influence the culture.
- tenkabuto 10y ago
  > his attitude I don't quite follow from your description of his reaction. What sort of attitude were you attempting to describe/how'd you perceive his attitude to be?
  - fredsir 10y ago
    I think his attitude, as described, was that he manipulates for his own perceived self importantness and perceived gain. If you mention you follow him on twitter the response is "oh that's so beneath me and I hate it", if you don't know him, he is "prince, like a king, and an important one of that so remember me" kind of attitude. Like a person you wouldn't want to meet in a bar. That is how I understood it anyways.
- tvst 10y ago
  Who would say "Prince, like King" when you can say "Prince, like 'the artist formerly known as'"?
remus 10y ago
It seems to me like the point of this post was to clarify whether the vulnerability was being actively exploited prior to the announcement, and I think it does that well. Just think of how easy it would have been to exploit this bug: buy a load of random domains, host some plausible looking but malformed content on it then setup a few thousand bots to hit those sites and harvest the responses. Keep that plugging away for a few months and you'd easily collect a lot of supposed-to-be-private info. That would have been an order of magnitude worse than having to sift through caches for scraps. Not to say that what's happened is 'good', but it could have been a whole lot worse.
deleted 10y ago
[deleted]
79d697i6fdif 10y ago
Tagging along on the top reply here, but does anyone else notice some serious HN gaming going on in this thread? Accounts commenting that have made 10 comments in 5 years, people defending CF that, after going through their profiles, are clearly linked to CloudFlare or maybe even employees. Top relies suddenly being near the bottom and replaced by posts supporting CloudFlare with statements that don't match what was actually in the blog post. Just seems really really fishy to me
- komali2 10y ago
  I doubt anything more nefarious is happening than a flood of CF employees hitting the thread. Is that gaming?
  - JumpCrisscross 10y ago
    It's, at the very least, disingenuous.
  - SilasX 10y ago
    It's not gaming for a CF employee to offer the own insights, but it would be if they're doing it as part of a coordinated effort. Also, they should be upfront about the affiliation.
- dahdum 10y ago
  I'm defending them because they're an incredible value for the $200/month we pay them, they've handled it responsibly, and the real impact looks vanishingly small.
- uladzislau 10y ago
  HN is very clearly gamed and I see more and more of this, not only in this thread.
  - dang 10y ago
    We take this problem seriously and have years of experience with it. In our experience, "very clearly gamed" nearly always reduces to "seems to me, based on personal biases and incomplete information". When abuse really is going on, that's extremely important, but it is certainly fewer than 10%, probably fewer than 5%, and maybe fewer than 1% of the times we hear this. So we actually have two problems, unfortunately intertwined: (1) abuse and (2) this tendency to imagine it. For this reason, please don't post unsubstantive allegations to HN. If you have evidence, or significant suspicious, email them to hn@ycombinator.com instead.
HalfwayToDice 10y ago
My website gets 75,000,000 total requests per month through CloudFlare, so they estimate 6-11 leaks. I am happy with the way CloudFlare have responded to this.
tyingq 10y ago
These exact looking numbers are based on extrapolating a look at 1% of all requests, for the sites they think were leaking, for a 10 day period. They don't get into the full detail of how they knew which sites to look for. For example, if I created a site with these settings, used it for a week, and then deleted it...is my site counted? What if I changed the settings back to normal? Does it slide under their radar? Did they really look for every site that had those settings at any time, or just any site that had those settings at the specific time they looked? To me, it still feels like there's a window where a bad actor could have created a site that triggers this behavior, and then pumped it with a scraper, over and over.
- caf 10y ago
  Yeah, exactly - this "look at a 10 day period and extrapolate" idea is based on the assumption that the rate of hits to the triggering pages was constant over the whole time, but in the case of looking for a malicious actor that's essentially begging the question. If there was unnatural traffic in there you can't assume its rate would have stayed constant over the whole period.
matt2000 10y ago
But also, we gotta start writing stuff like this parser in safer languages where this class of bug simply can't happen, right? Every time one of these breaches occurs, it's basically the same thing. "Data wasn't what we expected, we read a bunch of extra junk that had nothing to do with the input." It's kind of crazy it's acceptable at all. At this point, these kind of problems are on us as a community that we keep using unsafe tools. Every time we choose one of these languages we are implicitly trading security for performance (a.k.a. money).
- tedivm 10y ago
  If things aren't written in a "safe language" then they should at least be tested. Valgrind should have picked this issue up.
- nikisweeting 10y ago
  The lesson I really hope Cloudflare learns from this is to do thorough fuzz testing before deploying anything (and to write their parsers in a memory-safe language like Rust).
mixedbit 10y ago
The problem is that CloudFlare encourages a fundamentally insecure architecture. CDN should be used to host static files on a separate, cookie-less subdomain. With such architecture consequences of any data leak from a shared CDN infrastructure are limited. With CloudFlare all the requests are passed through the CDN. With dynamic, authenticated requests the benefits of this are dubious. The responses can't be cached anyway, an additional HTTP(S)-level hop is required for each request (and many additional IP-level hops). The drawbacks are now obvious: sensitive, unencrypted data of thousands of different sites resides in the same, not isolated, shared process memory. With such architecture we will always be one C pointer manipulation bug away from another leak.
- problems 10y ago
  The reason CloudFlare encourages this is because it isn't meant for use as a traditional CDN - they're not a CDN at all really. You don't upload content to them. They're meant as a cache and anti-DDoS proxy more than anything else. Look at what happens when anyone gets hit with a large scale DDoS attack these days, the first thing people will tell them, myself included is "CloudFlare". How do you handle a massive spike in load on a tiny, usually empty server? CloudFlare is the go-to tool for this purpose, especially for people who can't afford to pay Akamai, BlackLotus, Amazon or others. And they're impressively cheap for small businesses too. You can't hide from a DDoS attack or save your tiny DigitalOcean box if your web server isn't hidden away behind a service like this. I have nothing against CloudFlare, they've proven themselves extremely trustworthy in many ways and even in handling of past security issues they've been very responsible. But this time that's not the case - the downplaying of the issue is really damaging their reputation in my eyes. I'm a paying customer and I'm quite disappointed to see this.
  - mixedbit 10y ago
    > They're meant as a cache and anti-DDoS proxy more than anything else You can't cache responses to authenticated requests, so in case of DDoS attack against an HTTP end point that requires authentication the best CloudFlare can do to safe the back-end is to drop requests, which makes the attack successful. I really can't see benefits of passing authenticated traffic through CloudFlare.
lima 10y ago
Just consider all the private crawlers whose caches weren't purged. Anything short of invalidating any and all tokens, cookies and sessions is irresponsible and CloudFlare should communicate is as such.
markonen 10y ago
One aspect I haven't seen mentioned yet is that there is a geographic dimension to what ends up in search engine caches; their crawlers end up talking to the nearest Cloudflare point-of-presence so any leaked data is also going to be from requests served in that same general area. For example, my Cloudflare-using site is in the 1B to 10B requests a month bracket, meaning 112–1,118 anticipated leaks per Cloudflare's calculations. It's enough that you'd expect to possibly find some in the search engines, but we haven't. One potential explanation is that our usage is heavily concentrated within a geographic area (the Nordics), and this just isn't where the big search engines are doing their web crawling from.
- eastdakota 10y ago
  Yes, that's correct and not something our analysis has taken into account. Search crawlers are more likely to hit certain PoPs. If your content was less likely to be in one of those PoPs then you're less likely to have leaked data to one of them. We also didn't take into account that the distribution of the ~6,500 sites that triggered the bug weren't even across our infrastructure. We distribute load for any site across some fraction of all the servers in any given PoP. Based on whether you're on a cluster of servers with more or fewer vulnerable sites you'd have been more or less likely to leak data. For the general case the stats are directionally correct and, if anything, conservative (we believe, if anything, overstate the risk). But a particular customer's situation could deviate from the expected probabilities for a number of reasons.
nikisweeting 10y ago
I've linked to this post from the top of the pirate/sites-using-cloudflare README. I'm glad that it's a refreshingly level-headed response that isn't placing the blame on Google or other caches. It's still down-playing the potential impact slightly, but at least it's a more thorough response than some of the HN comments we've been seeing from CF employees. My advice to friends and family currently is to use their "log out all active sessions" buttons, and to reset only crucial passwords. As Techcrunch mentioned in their coverage, many sites will likely pay for identity theft insurance to cover the change of a leak, rather than force password resets and lose their users trust. It's a risk analysis that each site needs to make individually, and it may be the right call to eat the losses of ~10 possibly compromised users instead of forcing millions to reset their passwords.
_kst_ 10y ago
A theoretical question. There seems to be some question about whether any passwords were actually leaked, but in principle some password data could have gotten out. What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)? I have accounts on a number of sites that were affected by Cloudbleed, as indicated by http://www.doesitusecloudflare.com/ http://www.doesitusecloudflare.com/, and I've already changed my passwords on those sites. But the passwords I use are randomly generated and should be practically unguessable given a hashed version. Would I have been at risk if I had left my passwords alone?
- PuffinBlue 10y ago
  I forced user password resets on a forum I run. It is powered by myBB. This is towards the edge of my knowledge circle but... It doesn't appear that any hashing takes place client side[0] with myBB before the password is transmitted to the server, but I may have missed it when rechecking the code. Presumably that would mean in my case users who logged in potentially had their cleartext passwords pass through Cloudflare's network when they submitted their login details. Despite being behind https, when https is terminated at Cloudflare's edge and re-encrypted to my origin server (I use full/strict crypto setting), there was the possibility for decrypted data to be held in memory and then the possibility for it to be injected into the html pages of other sites when the bug was triggered. It would be a very very small probability of actually happening, but not zero. No details from the server should have leaked because only the content of transmitted data that passed over Cloudflare could have been leaked and the comparison of the provided password to the hashed/salted one in the database would have occurred on the server and not passed over the network. So the answer is it would depend on the way the sites you use handled password transmission and comparison. Personally I'd change them, and actually have done so for sites I use. [0] https://community.mybb.com/archive/index.php?thread-88668.html https://community.mybb.com/archive/index.php?thread-88668.ht...
- orthecreedence 10y ago
  > What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)? Depends on the app. The apps I write will do a PBKDF2 w/ 20K rounds on a password before sending to the server, which then hashes that hash again before storing/comparing. I suspect most apps just sent the username/password plaintext over SSL, meaning the plaintext passwords would have been leaked if the app used Cloudflare to terminate SSL. That said, if someone got the PBKDF2ed password hash from the SSL payload, they could just as easily use that to log into an account, but they'd have to at least do a little bit of work to break the hash if they wanted to actually crack the password. In other words, you were absolutely right to change all your passwords.
gwu78 10y ago
If users were discerning based on the inner workings of this organization, they should be switching their nameservers away from Cloudflare. Quantifying the number of domainnames CF has in their zones before and after this reported incident would be interesting. Sadly, I doubt there will be much change.
pron 10y ago
Their messaging is a bit strange. They write: > For the last twelve days we've been reviewing our logs to see if there's any evidence to indicate that a hacker was exploiting the bug before it was patched. We’ve found nothing so far to indicate that was the case. right after they explain: > Cloudbleed is different. It's more akin to learning that a stranger may have listened in on two employees at your company talking over lunch. ... you can't know exactly what the stranger may have heard, including potentially sensitive information about your company. So they say they've found no evidence after explaining that the situation is analogous to a case where no evidence can be found even if information were stolen.
Karupan 10y ago
I'm not going to comment on the methodology of CloudFlare's analysis, but just the fact that they are logging millions of unencrypted HTTP requests (cookies, headers and all) is just scary! Wasn't the point of SSL/TLS that intermediate proxies cannot peek into requests?
- jgrahamc 10y ago
  We're not logging "cookies, headers and all".
  - Karupan 10y ago
    My bad! Just re-read the article and it did say information about requests and responses. Thanks for the clarification.

287 points by eastdakota 10y ago

In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times. Wow, so just as bad as we thought. We did not find any passwords, credit cards, health records, social security numbers, or customer encryption keys in the sample set. BUT WAIT, THERE'S MORE The sample included thousands of pages and was statistically significant to a confidence level of 99% with a margin of error of 2.5%. Oh, so it could actually be as a high as 2.5% leaking encryption credentials. And if none of the data was found to leak anything sensitive where the fuck is the dataset? I've been around way too long to take a "study" like this at face value without third party verification. I also enjoy the straight up lie at the end: We are continuing to work with third party caches to expunge leaked data and will not let up until every bit has been removed. That sounds great right? Well, its too bad that a lot of 'third parties' are a box sitting on the corporate network edge that hasn't been touched in 5 years. Deleting all of this data from third party caches is not physically possible. In fact it might actually make things worse because it's destroying evidence of which credentials were leaked.

dahdum 10y ago
1.2 million page views over 5 months is almost nothing for the amount of traffic going through Cloudflare.
- 79d697i6fdif 10y ago
  And 1.2 million people dying is nothing compared to the number of people on the earth. IMO a leak this bad should be enough to sink cloudflare. A provider of SSL was randomly spitting out private data onto public websites. OVER A MILLION TIMES. Entire CA's have been shut down for leaking a couple hundred certificates. This has leaked private data over a million times, cloudflare is a joke
  - dahdum 10y ago
    > And 1.2 million people dying is nothing compared to the number of people on the earth. A != B
- mikeash 10y ago
  It seems to me that the absolute number is what's relevant, not how it compares to the total amount of traffic. That's 1.2 million potential data leaks. That's it "out of a bazillion" doesn't change it.
  - user5994461 10y ago
    Correction: It's 1.2 million definitive leaks. The only question is what is in these leaks exactly.
- Artemis2 10y ago
  For reference, two years ago Cloudflare was serving more than three million requests per second[0]. I can only imagine this figure has gone up a lot. 0: https://youtu.be/LA-gNoxSLCE?t=2m33s https://youtu.be/LA-gNoxSLCE?t=2m33s
ProAm 10y ago
If you are this concerned, change your credentials.
- 79d697i6fdif 10y ago
  So every website that uses cloudflare should ask their users to change all of their passwords, credit card numbers, and SSN's? This leak is being downplayed by webmasters because it's so incredibly bad that there's no way of handling it. The credentials of practically any internet user could have been leaked. The only "safe" way to handle this is to give everyone in the US new credit cards and SSN's and to reset accounts and security questions for every user on a site with cloudflare
  - ProAm 10y ago
    No but judging by how much you are freaking out in the comments, I was recommending you to change yours. Im not really sure what you want anyone else to do? It seems like you are just screaming at your monitor over something that while a significant bug isn't a huge deal. This is just basic risk management.
  - theGimp 10y ago
    Credit cards and SSNs are regularly compromised. The real problem is that they are used as an authentication mechanism. That's what we should be concerned about. This issue is a drop in the bucket when it comes to the amount of sensitive data leaked.
- zulln 10y ago
  Password would not help if session cookie is leaked. In many instances you cannot do anything towards that, as many services do not have any "logout all"-feature.
  - gommm 10y ago
    And this is why I changed the key I use for cookies on the application I had that was behind Cloudflare. This triggered all users to be logged out and invalidated any session cookie out there. So, yes, responsible websites can mitigate session cookies being leaked. That said, I am not impressed by Cloudflare's transparency which in this case consists of downplaying things, blaming Google and Taviso and not really taking responsibility.
bifrost 10y ago
1.3m pages served with "Extra" data is miniscule considering the number of actual pages served. Ideally you'd have no pages served but when you've got a bugproof technology the world will be your oyster.
dsl 10y ago
> We are continuing to work with third party caches One of the caches they worked with was Baidu, which has direct ties to Chinese intelligence. Just because it isn't publicly available doesn't mean people aren't still pouring over it looking for useful data.
- 79d697i6fdif 10y ago
  Also, a lot of web spiders are not benign. All sorts of bots troll the internet looking specifically for data leaks like publicly visible email addresses and SSN's. I'm sure they're having a field day.

advisedwang 10y ago

Disappointing that their response is "damage control" and downplaying impact.

AwesomeBean 10y ago
Why? As long as they come along with facts and data, then that's a good thing.
- 79d697i6fdif 10y ago
  they only sampled a few thousand leaked responses out of over a million. The margin of error is 2.5% on conclusions because they didn't use enough data, not even close.

geforce 10y ago

Was talking with colleagues about cloudbleed and S3 problems yesterday. I don't feel like many people are actually concerned of the implication of having an internet that isn't an internet anymore, but merely a handful of big companies hosting everyone. Or maybe it's me who don't understand.

Florin_Andrei 10y ago
It's almost as if self-reinforcing feedback loops are all over society these days. Being at the top makes it that much easier to stay at the top and crush everything else. Internet companies are just one small slice of this trend.
- 79d697i6fdif 10y ago
  antitrust is all bark with no bite these days. Most industries having 3-4 "competitors" that don't actually compete is the new norm. I wouldn't be amazed if a link is shown between this situation and the increasing wealth inequality
  - thehardsphere 10y ago
    That's because you're viewing this situation entirely from within your prior beliefs about large corporations, which somehow have convinced you that only 3 or 4 hosting companies exist and everyone must use them. Exactly how do you figure this? There are lots of alternatives to S3: 1. All the cloud stuff, which is the new fangled hotness (3 to 4 companies) 2. Non-cloud hosting providers (hundreds of them, from shared hosting to VPSs and above, and usually cheaper if you don't have high traffic) 3. Hosting your own stuff on your own hardware, which is the cheapest if you really need it, and can be done in a data center just about anywhere in the world with good internet. So, what planet is this that you live on where 3 to 4 companies handle all hosting and we have to use them?
  - cookiecaper 10y ago
    Beyond simple anti-trust, in many cases, we need to reform legal structures so that big players can't use them to block out startups. It's practically impossible to defend yourself well in a lawsuit against any respectably-sized company without a few million lying around, first of all. That affects everything, and big companies use that fact to bully upstarts and other small innovators into shutting down all the time. My own business was effectively shuttered (had to stop selling our primary product) by a C&D from a Fortune 100. It would've been 5-10 years and ~$5 million to see that case all the way through, and under current precedent, it's very likely I would have lost. Aside from that, industries frequently get laws and rules put in place with ostensibly-reasonable rationale, while the actual intent is to make it virtually impossible for disruptive competitors to enter the marketplace. The CFAA is the piece of legislation that primarily enshrines entrenched players in the online space. We also need to reform copyright law and clarify some matters regarding the applicability of EULAs, especially with regard to clickwrap and browsewrap. Once that's done, the flood of innovators that have been held back by big companies dispatching their law firms will finally be able to contribute, and the internet's competitve landscape will truly be back in the hands of the users. It will shift from "Who has my data? I have to go with them" to "I can use any interface I want to access that data", effectively resolving the chicken-and-egg effect that imperils any potential competing social network (not even Google could compete with Facebook on this!).

bifrost 10y ago

I'm glad to see an article that isn't all puppies and kittens, this wasn't a fun thing to deal with and certainly wasn't anticipated. The CloudFlare team did a great job responding to it IMHO. "Of the 1,242,071 requests that triggered the bug, we estimate more than half came from search engine crawlers." This is very important to sort out, most people don't think about security much less the storing of credential or identifying data by search engines, this is a huge part of the incident response. I think that what we should take away from this is that even though the bug existed it was responded to in a reasonable manner.

jlg23 10y ago

TL;DR: "Since this is just a sample, it is not correct to conclude that no passwords, credit cards, health records, social security numbers, or customer encryption keys were ever exposed."

koolba 10y ago

> 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and Downplaying how bad it was that customer cookie's were leaked is what I found particularly egregious in the original response. A session cookie is almost as bad as a password leak. Usually it'll let you everything except change your password (as most sites require your current one as an extra validation).

chipperyman573 10y ago
Session cookies can be reset much more easily than asking a user to reset their password. Session cookies also aren't used on multiple sites. It's still a big deal, but leaking a session cookie is much less dangerous than leaking a password.
- koolba 10y ago
  Lots of sites use session cookies. Many don't automatically expire them or rely on browsers to expire them (ex: signed cookies). Plus rotating the master session key (to force the issue of all users being reset) requires knowing you should do it. By downplaying the issue, CloudFlare is sending the message that customers don't have to do anything.
- hueving 10y ago
  >Session cookies can be reset much more easily than asking a user to reset their password. This statement is making an assertion about the behavior of vastly different software backends and the skill-set of the people administering the systems. Things that may appear trivial to you may not be for the sysadmin in charge of servers running healthcare applications that have an opaque internal state.

saurik 10y ago

It was my understanding that people were finding numerous examples of leaked data that included Authorization headers, which don't always leak passwords but which still leak session keys. Maybe if espadrine from this linked comment reads this, he could explain more? https://news.ycombinator.com/item?id=13719455 https://news.ycombinator.com/item?id=13719455 FWIW, as an example, Grindr uses CloudFlare: do a Google search for "authorization: grindr3" and you will find a URL which (no longer cached but you can still get snippets) contained an authenticated grindr request, which would be enough to have had temporary access to that person's account. https://costumla.com/wild-west-costumes-for-men.html https://costumla.com/wild-west-costumes-for-men.html "... U Edge Certificate Authority1 0 U San Francisco1 0 U California�1p ��U� N��U � � �f"�0! %��U@ @GET /v3/profiles/[REDACTED] HTTP/1.1 CF-RAY: 33282514b8d957d7 FL-Server: 15f76 Host: grindr.mobi X-Real-IP: [REDACTED] Accept-Encoding: gzip Client-Accept-Encoding: gzip X-Forwarded-Proto: https Connect-Via-Https : on Connect-Via-Port: 443 Connect-Via-IP: 104.16.85.62 Connect-Via-Host: grindr.mobi CF-Visitor: {"scheme":"https"} CF-Host-Origin-IP: [REDACTED] Zone-ID: 22252132 Owner-ID: 2607399 CF-Int-Brand-ID: 100 Zone-Name: grindr.mobi Connection: Keep-Alive X-SSL-Protocol: TLSv1.2 X-SSL-Cipher: ECDHE-RSA-AES128-GCM-SHA256 X-SSL-Server-Name: grindr.mobi X-SSL-Session-Reused: . SSL-Server-IP: 104.16.85.62 X-SSL-Connection-ID: 15d1b8de6025864d-DFW X-SPDY-Protocol: 3.1 authorization: Grindr3 ... accept: application/json user-agent: grindr3/3.0.13.16790;16790;Free;Android 6.0 CF-Use-OB: 0 Set-Expires-TTL: 14400 CF-Cache-Max-File-Size: 512m Set-SSL-Name: grindr.mobi CF-Cache-Level: byc CF-Unbuffered-Upload: 0 Set-SSL-Client-Cert: 0 Set-Limit-Conn-Cache-Host: 50000 CF-WAN-RG5: 0 CF-Brand-Name: cloudflare CF-Age-Header-Enabled: 0 CF-Respect-Strong-Etag: 0 Set-Proxy-Read-Timeout: 100 Set-Proxy-Send-Timeout: 30 CF-Connecting-IP: [REDACTED] Set-Proxy-Connect-Timeout: 90 Set-Cache-Bypass: 0 Set-SSL-Verify: 0 CF-Force-Miss-TS: 0 Set-Buffering: 0 CF-Pref-OB: 1 Set-Keepalive: 1 CF-Pref-Geoloc: 1 CF-Use-BYC: 0 CF-IPCountry: [REDACTED] CF-IPType ..." edit: I have spent the last fifteen minutes pulling the search snippet (edit: and now an hour; but all on my phone, so this is harder than it should be, and I also am distracted by other stuff). The way you do this is by walking through the parts you can see to get nearby context. (I do this a lot to pull content purged from or inaccessible to or simply updated in Google's cache.) In so doing, while I haven't been able to still have access to the session key (likely too long and unique for a snippet), I have pulled the X-Real-IP address field of this Grindr user and a profile identifier they were checking out (both of which I redacted above, but you could trivially get yourself now using that context). CloudFlare: if you think there isn't private data that was leaked, OR EVEN PRIVATE DATA STILL ACCESSIBLE, you are a bunch of fucking idiots. 1) Clearing the cache isn't sufficient, as for anything built from short plain text words we can pull the snippet. 2) IP addresses and session keys count as "private data". 3) GET requests actually are often sensitive information :/.

rattray 10y ago

tl;dr excerpted: In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times. 1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched; 2) the vast majority of Cloudflare customers had no data leaked; 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and 4) our review is ongoing.

tyingq 10y ago
>we now estimate based on our logs the bug was triggered 1,242,071 times Based on looking at 1% of their log entries for 10 days of the total ~150 day period.

deleted 10y ago

[deleted]

ryanlol 10y ago

>In total, between 22 September 2016 and 18 February 2017 we now estimate based on our logs the bug was triggered 1,242,071 times. So uh, Cloudflare has logs of all page loads since september at the very least? And I guess with response sizes, since that seems like the most reasonable way for them to come up with such a number. Awesome.

user5994461 10y ago
Doubt that they have any log, they have explained before that they only keep 4hours of logs because it's too much volume of data. FYI: Traffic is over 4M requests per second. Guess they only kept a counter of requests. A bit of maths and you get an estimate for the amount of leaks.

tptacek 10y ago

Not everything Cloudflare has written about this breach has been good, but this is solid. In particular, I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug. What remains now is an accounting of how some of the most sensitive C code on the Internet was tested prior to the discovery of this bug (by Cloudflare's own accounting, the underlying error seems to have been present long before it became symptomatic), and what they're doing about that now.

hueving 10y ago
>I'm satisfied that commercially reasonable efforts have been made to ensure there wasn't deliberate exploitation of the bug. How would you classify people using all of the crawled data though? It seems sort of hand-wavy to claim nobody deliberately exploited the bug if they used leaked session tokens in crawled data to get access to user accounts.

camel_gopher 10y ago

The wording of the post seems to be tilted towards limiting Cloudflare's potential liability as opposed to objectively quantifying the impact. For example, the author mentions that the 100th largest website handles less than 10B requests per month - which positions near the median of the table of likelihood anticipated leaks. I'm not surprised at this approach as the CEO has a background in law. I think that, combined with his hubris, shaped the tone of this post from an objective post mortem to a goal of minimizing the damage to Cloudflare itself. I met Mr. Prince once at a tech conference. I mentioned that I followed him on Twitter, to which he freaked out a little bit and said "Oh I hate that when I run across people who tell me that". When I heard him tell his name to someone else there, he said his last name was "Prince, like King". I suspect that his attitude is prevalent at Cloudflare, and may have shaped not only the source of this issue, but the response to it as well, since CEO's generally influence the culture.

tenkabuto 10y ago
> his attitude I don't quite follow from your description of his reaction. What sort of attitude were you attempting to describe/how'd you perceive his attitude to be?
- fredsir 10y ago
  I think his attitude, as described, was that he manipulates for his own perceived self importantness and perceived gain. If you mention you follow him on twitter the response is "oh that's so beneath me and I hate it", if you don't know him, he is "prince, like a king, and an important one of that so remember me" kind of attitude. Like a person you wouldn't want to meet in a bar. That is how I understood it anyways.
tvst 10y ago
Who would say "Prince, like King" when you can say "Prince, like 'the artist formerly known as'"?

remus 10y ago

It seems to me like the point of this post was to clarify whether the vulnerability was being actively exploited prior to the announcement, and I think it does that well. Just think of how easy it would have been to exploit this bug: buy a load of random domains, host some plausible looking but malformed content on it then setup a few thousand bots to hit those sites and harvest the responses. Keep that plugging away for a few months and you'd easily collect a lot of supposed-to-be-private info. That would have been an order of magnitude worse than having to sift through caches for scraps. Not to say that what's happened is 'good', but it could have been a whole lot worse.

deleted 10y ago

[deleted]

79d697i6fdif 10y ago

Tagging along on the top reply here, but does anyone else notice some serious HN gaming going on in this thread? Accounts commenting that have made 10 comments in 5 years, people defending CF that, after going through their profiles, are clearly linked to CloudFlare or maybe even employees. Top relies suddenly being near the bottom and replaced by posts supporting CloudFlare with statements that don't match what was actually in the blog post. Just seems really really fishy to me

komali2 10y ago
I doubt anything more nefarious is happening than a flood of CF employees hitting the thread. Is that gaming?
- JumpCrisscross 10y ago
  It's, at the very least, disingenuous.
- SilasX 10y ago
  It's not gaming for a CF employee to offer the own insights, but it would be if they're doing it as part of a coordinated effort. Also, they should be upfront about the affiliation.
dahdum 10y ago
I'm defending them because they're an incredible value for the $200/month we pay them, they've handled it responsibly, and the real impact looks vanishingly small.
uladzislau 10y ago
HN is very clearly gamed and I see more and more of this, not only in this thread.
- dang 10y ago
  We take this problem seriously and have years of experience with it. In our experience, "very clearly gamed" nearly always reduces to "seems to me, based on personal biases and incomplete information". When abuse really is going on, that's extremely important, but it is certainly fewer than 10%, probably fewer than 5%, and maybe fewer than 1% of the times we hear this. So we actually have two problems, unfortunately intertwined: (1) abuse and (2) this tendency to imagine it. For this reason, please don't post unsubstantive allegations to HN. If you have evidence, or significant suspicious, email them to hn@ycombinator.com instead.

HalfwayToDice 10y ago

My website gets 75,000,000 total requests per month through CloudFlare, so they estimate 6-11 leaks. I am happy with the way CloudFlare have responded to this.

tyingq 10y ago

These exact looking numbers are based on extrapolating a look at 1% of all requests, for the sites they think were leaking, for a 10 day period. They don't get into the full detail of how they knew which sites to look for. For example, if I created a site with these settings, used it for a week, and then deleted it...is my site counted? What if I changed the settings back to normal? Does it slide under their radar? Did they really look for every site that had those settings at any time, or just any site that had those settings at the specific time they looked? To me, it still feels like there's a window where a bad actor could have created a site that triggers this behavior, and then pumped it with a scraper, over and over.

caf 10y ago
Yeah, exactly - this "look at a 10 day period and extrapolate" idea is based on the assumption that the rate of hits to the triggering pages was constant over the whole time, but in the case of looking for a malicious actor that's essentially begging the question. If there was unnatural traffic in there you can't assume its rate would have stayed constant over the whole period.

matt2000 10y ago

But also, we gotta start writing stuff like this parser in safer languages where this class of bug simply can't happen, right? Every time one of these breaches occurs, it's basically the same thing. "Data wasn't what we expected, we read a bunch of extra junk that had nothing to do with the input." It's kind of crazy it's acceptable at all. At this point, these kind of problems are on us as a community that we keep using unsafe tools. Every time we choose one of these languages we are implicitly trading security for performance (a.k.a. money).

tedivm 10y ago
If things aren't written in a "safe language" then they should at least be tested. Valgrind should have picked this issue up.
nikisweeting 10y ago
The lesson I really hope Cloudflare learns from this is to do thorough fuzz testing before deploying anything (and to write their parsers in a memory-safe language like Rust).

mixedbit 10y ago

The problem is that CloudFlare encourages a fundamentally insecure architecture. CDN should be used to host static files on a separate, cookie-less subdomain. With such architecture consequences of any data leak from a shared CDN infrastructure are limited. With CloudFlare all the requests are passed through the CDN. With dynamic, authenticated requests the benefits of this are dubious. The responses can't be cached anyway, an additional HTTP(S)-level hop is required for each request (and many additional IP-level hops). The drawbacks are now obvious: sensitive, unencrypted data of thousands of different sites resides in the same, not isolated, shared process memory. With such architecture we will always be one C pointer manipulation bug away from another leak.

problems 10y ago
The reason CloudFlare encourages this is because it isn't meant for use as a traditional CDN - they're not a CDN at all really. You don't upload content to them. They're meant as a cache and anti-DDoS proxy more than anything else. Look at what happens when anyone gets hit with a large scale DDoS attack these days, the first thing people will tell them, myself included is "CloudFlare". How do you handle a massive spike in load on a tiny, usually empty server? CloudFlare is the go-to tool for this purpose, especially for people who can't afford to pay Akamai, BlackLotus, Amazon or others. And they're impressively cheap for small businesses too. You can't hide from a DDoS attack or save your tiny DigitalOcean box if your web server isn't hidden away behind a service like this. I have nothing against CloudFlare, they've proven themselves extremely trustworthy in many ways and even in handling of past security issues they've been very responsible. But this time that's not the case - the downplaying of the issue is really damaging their reputation in my eyes. I'm a paying customer and I'm quite disappointed to see this.
- mixedbit 10y ago
  > They're meant as a cache and anti-DDoS proxy more than anything else You can't cache responses to authenticated requests, so in case of DDoS attack against an HTTP end point that requires authentication the best CloudFlare can do to safe the back-end is to drop requests, which makes the attack successful. I really can't see benefits of passing authenticated traffic through CloudFlare.

lima 10y ago

Just consider all the private crawlers whose caches weren't purged. Anything short of invalidating any and all tokens, cookies and sessions is irresponsible and CloudFlare should communicate is as such.

markonen 10y ago

One aspect I haven't seen mentioned yet is that there is a geographic dimension to what ends up in search engine caches; their crawlers end up talking to the nearest Cloudflare point-of-presence so any leaked data is also going to be from requests served in that same general area. For example, my Cloudflare-using site is in the 1B to 10B requests a month bracket, meaning 112–1,118 anticipated leaks per Cloudflare's calculations. It's enough that you'd expect to possibly find some in the search engines, but we haven't. One potential explanation is that our usage is heavily concentrated within a geographic area (the Nordics), and this just isn't where the big search engines are doing their web crawling from.

eastdakota 10y ago
Yes, that's correct and not something our analysis has taken into account. Search crawlers are more likely to hit certain PoPs. If your content was less likely to be in one of those PoPs then you're less likely to have leaked data to one of them. We also didn't take into account that the distribution of the ~6,500 sites that triggered the bug weren't even across our infrastructure. We distribute load for any site across some fraction of all the servers in any given PoP. Based on whether you're on a cluster of servers with more or fewer vulnerable sites you'd have been more or less likely to leak data. For the general case the stats are directionally correct and, if anything, conservative (we believe, if anything, overstate the risk). But a particular customer's situation could deviate from the expected probabilities for a number of reasons.

nikisweeting 10y ago

I've linked to this post from the top of the pirate/sites-using-cloudflare README. I'm glad that it's a refreshingly level-headed response that isn't placing the blame on Google or other caches. It's still down-playing the potential impact slightly, but at least it's a more thorough response than some of the HN comments we've been seeing from CF employees. My advice to friends and family currently is to use their "log out all active sessions" buttons, and to reset only crucial passwords. As Techcrunch mentioned in their coverage, many sites will likely pay for identity theft insurance to cover the change of a leak, rather than force password resets and lose their users trust. It's a risk analysis that each site needs to make individually, and it may be the right call to eat the losses of ~10 possibly compromised users instead of forcing millions to reset their passwords.

_kst_ 10y ago

A theoretical question. There seems to be some question about whether any passwords were actually leaked, but in principle some password data could have gotten out. What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)? I have accounts on a number of sites that were affected by Cloudbleed, as indicated by http://www.doesitusecloudflare.com/ http://www.doesitusecloudflare.com/, and I've already changed my passwords on those sites. But the passwords I use are randomly generated and should be practically unguessable given a hashed version. Would I have been at risk if I had left my passwords alone?

PuffinBlue 10y ago
I forced user password resets on a forum I run. It is powered by myBB. This is towards the edge of my knowledge circle but... It doesn't appear that any hashing takes place client side[0] with myBB before the password is transmitted to the server, but I may have missed it when rechecking the code. Presumably that would mean in my case users who logged in potentially had their cleartext passwords pass through Cloudflare's network when they submitted their login details. Despite being behind https, when https is terminated at Cloudflare's edge and re-encrypted to my origin server (I use full/strict crypto setting), there was the possibility for decrypted data to be held in memory and then the possibility for it to be injected into the html pages of other sites when the bug was triggered. It would be a very very small probability of actually happening, but not zero. No details from the server should have leaked because only the content of transmitted data that passed over Cloudflare could have been leaked and the comparison of the provided password to the hashed/salted one in the database would have occurred on the server and not passed over the network. So the answer is it would depend on the way the sites you use handled password transmission and comparison. Personally I'd change them, and actually have done so for sites I use. [0] https://community.mybb.com/archive/index.php?thread-88668.html https://community.mybb.com/archive/index.php?thread-88668.ht...
orthecreedence 10y ago
> What would actually have been leaked, a cleartext password or the encrypted/hashed password stored on an affected server (or both)? Depends on the app. The apps I write will do a PBKDF2 w/ 20K rounds on a password before sending to the server, which then hashes that hash again before storing/comparing. I suspect most apps just sent the username/password plaintext over SSL, meaning the plaintext passwords would have been leaked if the app used Cloudflare to terminate SSL. That said, if someone got the PBKDF2ed password hash from the SSL payload, they could just as easily use that to log into an account, but they'd have to at least do a little bit of work to break the hash if they wanted to actually crack the password. In other words, you were absolutely right to change all your passwords.

gwu78 10y ago

If users were discerning based on the inner workings of this organization, they should be switching their nameservers away from Cloudflare. Quantifying the number of domainnames CF has in their zones before and after this reported incident would be interesting. Sadly, I doubt there will be much change.

pron 10y ago

Their messaging is a bit strange. They write: > For the last twelve days we've been reviewing our logs to see if there's any evidence to indicate that a hacker was exploiting the bug before it was patched. We’ve found nothing so far to indicate that was the case. right after they explain: > Cloudbleed is different. It's more akin to learning that a stranger may have listened in on two employees at your company talking over lunch. ... you can't know exactly what the stranger may have heard, including potentially sensitive information about your company. So they say they've found no evidence after explaining that the situation is analogous to a case where no evidence can be found even if information were stolen.

Karupan 10y ago

I'm not going to comment on the methodology of CloudFlare's analysis, but just the fact that they are logging millions of unencrypted HTTP requests (cookies, headers and all) is just scary! Wasn't the point of SSL/TLS that intermediate proxies cannot peek into requests?

jgrahamc 10y ago
We're not logging "cookies, headers and all".
- Karupan 10y ago
  My bad! Just re-read the article and it did say information about requests and responses. Thanks for the clarification.