7 ms·

We identified 3,438 unique domains. I'm not sure if those were all sent to Tavis because we were only sending him things that we wanted purged.

tptacek 10y ago
What anomalies would be apparent in your logs if someone malicious had discovered this flaw and used it to generate a large corpus of leaked HTTP content?
- aidos 10y ago
  That's also what I'm interested in. There's a lot of talk about the sites that had the features enabled that allowed the data to escape, but it's the sites that were co-existing with those that were in danger. In terms of the caching, knowing the broken sites tells you where to look in the caches after the fact, but do you have any idea of who's data was leaked? Presumably 2 consecutive requests to the same malformed page could/would leak different data.
  - vmarsy 10y ago
    > Presumably 2 consecutive requests to the same malformed page could/would leak different data. Wouldn't the second request be served from the CDN cache? Since for Cloudfare that particular page is a valid cached page, it would send you that same page on the second request.
    - artursapek 10y ago
      Only if the leaked memory is in the response before the response is cached.
    - aidos 10y ago
      I don't know enough about the layers in the cloudflare system to say. Does it only apply to cached pages? What about https? They would have the ssl termination first and then these errant servers behind that - none of those pages would be cached, right?
    - markonen 10y ago
      Cloudflare doesn't cache HTML pages by default.
- beachstartup 10y ago
  it seems to me you'd have to know at a minimum: 1. every tag pattern that triggers the bug(s) 2. which broken pages with that pattern were requested at an abnormally high frequency or had an unusually short TTL (or some other useful heuristic) 3. on which servers, and at what time, in order to tell 4. who's data lived on the same servers at the same time as those broken pages to even begin to estimate the scope of the leak. and that doesn't even help you find who planted the bad seeds.
rdl 10y ago
3438 domains which someone could have queried, but potentially data from any site which had "recently" passed through Cloudflare would be exposed in response, right? Purging those results helps with search engines, but a hypothetical malicious secret crawler would still potentially have any data from any site.
- taviso 10y ago
  correct
  - rdl 10y ago
    fuck :(
    - deleted 10y ago
      [deleted]
    - genericpseudo 10y ago
      Which means any user who has used any service which uses CloudFlare, right? At least in theory.
      - biafra 10y ago
        How can I find out which services I have accounts with are using cloudflare? Or better have been using cloudflare in recent months? Assume I have a list of domains, where I have accounts.
        teraflop 10y ago
        You can check IP whois records, but it'll be very hard to be 100% sure about any of them. For example, one of the examples from the bug report is Uber, which doesn't use Cloudflare for its home page but apparently does for one of its internal API endpoints.
        eropple 10y ago
tptacek 10y ago
Here's a question your blog post doesn't answer but should, right now: Exactly which search engines and cache providers did you work with to scrub leaked data?
- dsp1234 10y ago
  Also, have you worked with any search engine to notify affected customers. ex: Right now there is in an easily found google cached page with OAuth tokens for very popular fitness wearable's android API endpoints
wilde 10y ago
Are you guys planning to release the list so we can all change our passwords on affected services? Or are you planning on letting those services handle the communication?
- tlrobinson 10y ago
  If I'm understanding correctly, that list would include not only the 3,438 domains with content that triggered the bug, but every Cloudflare customer between 2016-09-22 and 2017-02-18.
  - Xorlev 10y ago
    Can we trust it was only those domains?
    - gog 10y ago
      Not really. If a site is using Cloudflare protection for only some of their subdomains they do not show on this list even if the site itself is in the alexa top 10k sites. And of course all other sites that are not in alexa 10k are not in this list (if they are not on some other lists used, you can see the source of lists in the README of the Github repo).
  - dbmnt 10y ago
    No. Only Cloudflare customers using a subset of features of the SSL proxy service are impacted. Cloudflare has a lot of customers who only use the free DNS service, for example.
    - tptacek 10y ago
      Careful. It appears that any Cloudflare client who was sending HTTP/S traffic through their proxies is affected. A small subset of their customers had the specific problem that triggered the bug, but once triggered, the bug disclosed secrets from all their web customers. You're not exposed if you never sent traffic through their proxies; for instance, if you somehow only used them for DNS.
jameshart 10y ago
What I find remarkable is that the owners of those sites weren't ever aware of this issue. If customers were receiving random chunks of raw nginx memory embedded in pages on my site, I'd probably have heard about it from someone sooner, surely? I guess there is a long tail of pages on the internet whose primary purpose is to be crawled by google and serve as search landing pages - but again, if I had a bug in the HTML in one of my SEO pages that caused googlebot to see it as full of nonsense, I'd see that in my analytics because a page full of uninitialized nginx memory is not going to be an effective pagerank booster.

tptacek 10y ago

What anomalies would be apparent in your logs if someone malicious had discovered this flaw and used it to generate a large corpus of leaked HTTP content?

aidos 10y ago
That's also what I'm interested in. There's a lot of talk about the sites that had the features enabled that allowed the data to escape, but it's the sites that were co-existing with those that were in danger. In terms of the caching, knowing the broken sites tells you where to look in the caches after the fact, but do you have any idea of who's data was leaked? Presumably 2 consecutive requests to the same malformed page could/would leak different data.
- vmarsy 10y ago
  > Presumably 2 consecutive requests to the same malformed page could/would leak different data. Wouldn't the second request be served from the CDN cache? Since for Cloudfare that particular page is a valid cached page, it would send you that same page on the second request.
  - artursapek 10y ago
    Only if the leaked memory is in the response before the response is cached.
  - aidos 10y ago
    I don't know enough about the layers in the cloudflare system to say. Does it only apply to cached pages? What about https? They would have the ssl termination first and then these errant servers behind that - none of those pages would be cached, right?
  - markonen 10y ago
    Cloudflare doesn't cache HTML pages by default.
beachstartup 10y ago
it seems to me you'd have to know at a minimum: 1. every tag pattern that triggers the bug(s) 2. which broken pages with that pattern were requested at an abnormally high frequency or had an unusually short TTL (or some other useful heuristic) 3. on which servers, and at what time, in order to tell 4. who's data lived on the same servers at the same time as those broken pages to even begin to estimate the scope of the leak. and that doesn't even help you find who planted the bad seeds.

rdl 10y ago

3438 domains which someone could have queried, but potentially data from any site which had "recently" passed through Cloudflare would be exposed in response, right? Purging those results helps with search engines, but a hypothetical malicious secret crawler would still potentially have any data from any site.

taviso 10y ago
correct
- rdl 10y ago
  fuck :(
  - deleted 10y ago
    [deleted]
  - genericpseudo 10y ago
    Which means any user who has used any service which uses CloudFlare, right? At least in theory.
    - biafra 10y ago
      How can I find out which services I have accounts with are using cloudflare? Or better have been using cloudflare in recent months? Assume I have a list of domains, where I have accounts.
      - teraflop 10y ago
        You can check IP whois records, but it'll be very hard to be 100% sure about any of them. For example, one of the examples from the bug report is Uber, which doesn't use Cloudflare for its home page but apparently does for one of its internal API endpoints.
      - eropple 10y ago

tptacek 10y ago

Here's a question your blog post doesn't answer but should, right now: Exactly which search engines and cache providers did you work with to scrub leaked data?

dsp1234 10y ago
Also, have you worked with any search engine to notify affected customers. ex: Right now there is in an easily found google cached page with OAuth tokens for very popular fitness wearable's android API endpoints

wilde 10y ago

Are you guys planning to release the list so we can all change our passwords on affected services? Or are you planning on letting those services handle the communication?

tlrobinson 10y ago
If I'm understanding correctly, that list would include not only the 3,438 domains with content that triggered the bug, but every Cloudflare customer between 2016-09-22 and 2017-02-18.
- Xorlev 10y ago
  Can we trust it was only those domains?
  - gog 10y ago
    Not really. If a site is using Cloudflare protection for only some of their subdomains they do not show on this list even if the site itself is in the alexa top 10k sites. And of course all other sites that are not in alexa 10k are not in this list (if they are not on some other lists used, you can see the source of lists in the README of the Github repo).
- dbmnt 10y ago
  No. Only Cloudflare customers using a subset of features of the SSL proxy service are impacted. Cloudflare has a lot of customers who only use the free DNS service, for example.
  - tptacek 10y ago
    Careful. It appears that any Cloudflare client who was sending HTTP/S traffic through their proxies is affected. A small subset of their customers had the specific problem that triggered the bug, but once triggered, the bug disclosed secrets from all their web customers. You're not exposed if you never sent traffic through their proxies; for instance, if you somehow only used them for DNS.

jameshart 10y ago

What I find remarkable is that the owners of those sites weren't ever aware of this issue. If customers were receiving random chunks of raw nginx memory embedded in pages on my site, I'd probably have heard about it from someone sooner, surely? I guess there is a long tail of pages on the internet whose primary purpose is to be crawled by google and serve as search landing pages - but again, if I had a bug in the HTML in one of my SEO pages that caused googlebot to see it as full of nonsense, I'd see that in my analytics because a page full of uninitialized nginx memory is not going to be an effective pagerank booster.