3 ms·

Millions of domains are on Cloudflare. We can't tell how many of them were affected. Either we can search for obvious strings like X-Uber-* and try to scrub th

by kijin 10y ago

Millions of domains are on Cloudflare. We can't tell how many of them were affected. Either we can search for obvious strings like X-Uber-* and try to scrub them one by one, or we can just nuke the caches for all the domains that turned on the problematic features (Scrape Shield, etc.) anytime between last September and last weekend. Cloudflare should supply the full list to all the known search engines including the Internet Archive. Anything less than that is gross negligence. If Cloudflare doesn't want to (or cannot) supply the full list of affected domains, an alternative would be to nuke the caches for all the domains that resolved to a Cloudflare IP [1] anytime between last September and last weekend. I'm pretty sure that Google and Bing can compile this information from their records. They might also be able to tell, even without Cloudflare's cooperation, which of those websites used the problematic features. [1] https://www.cloudflare.com/ips/ https://www.cloudflare.com/ips/

josecastillo 10y ago
Nuking the caches is one thing, but what about services like the Internet Archive whose job it is to hang on to these pages? Pages with leaked data are clearly difficult to identify; removing the leaked data without nuking the document may be impossible, at least in an automated fashion. Are we supposed to erase five months of history from the affected domains? This CloudFlare breach seems to have put a lot of people in a tough spot, but it feels like it's put archivists in an impossible position.
- gech 10y ago
  Good time to shoe in some forced dmca content removal
  - greglindahl 10y ago
    Yeah, abusing the DMCA because it's the only tool you can think of, that's a great idea.
    - kijin 10y ago
      Come to think of it, our society grants search engines the privilege of keeping copies of copyrighted material in exchange for the services they provide. It might not be unreasonable to say that this privilege comes with a certain responsibility to ensure that those copies do not cause excessive harm to others. So although Cloudflare is the one that fucked up, Google et al. also have a responsibility to do whatever they can to protect the public. They should do what they can, with or without Cloudflare's cooperation.
      - d33 10y ago
        I don't believe so. Cloudflare had this responsibility and they messed up. They are the only ones liable.
        greglindahl 10y ago
        Yep, it's a Section 230 for the search engine, probably -- the unfortunate data came from Cloudflare.
        kijin 10y ago
        Cloudflare cannot delete documents from other company's caches. The actual deletion must be performed by Google and Bing, who can then sue Cloudflare for cleanup costs if the latter is unwilling to cooperate. When there's an oil spill, we don't wait for the oil company to come and clean up their own mess. Others clean it up a.s.a.p. and (ideally) then make the oil company pay the fines and damages. CloudBleed is a virtual oil spill. They literally sprayed other people's private data all over the internet.
    - arrty88 10y ago
      Why can't they just search for pages with data after the closing HTML tag
      - ec109685 10y ago
        That isn't uncommon enough. Searching for one of the CF- http headers in the source code, however would work.
- kijin 10y ago
  I agree that nuking entire domains would be bad for the Internet Archive. But I don't think it would be overwhelmingly difficult, nor controversial, to identify and remove the vast majority of "contaminated" documents. This applies to the Internet Archive as well as major search engines. First, we're talking about raw memory pages, not merely malformed HTML. Those memory pages might contain valid HTML, but most of the sensitive information is in the headers, not HTML markup. It won't be very difficult to write a script to identify documents where random headers and POST data have been inserted where they don't belong, or where the markup is so obviously invalid (even compared to similar documents from the same site) that there is a high probability of contamination. Having a full list of contaminated domains would obviously help a lot, because we'll only have to deal with thousands of domains instead of millions. Second, contaminated documents by definition contain information that is NOT what the publisher intended to be crawled, indexed, or archived. So there should be less resistance to removing them. Finally, most of the contaminated domains used features such as Scrape Shield that were intended to deter archival. It's as if the domain had a robots.txt that said "User-agent:* Disallow:/". I'm not sure whether it's even possible for the Internet Archive to archive such domains. If they can, maybe they've been doing it against the publisher's wishes. If they can't, well, there's no problem to begin with.
  - greglindahl 10y ago
    Archives don't delete stuff, nor do they have much capacity to do much computation on their archived data. Whereas if blekko still existed as a search engine, I'd just push code to refuse to show cached pages or snippets containing text that likely means the CloudFlare problem. 15 minutes work, and the underlying data would expire fully in a couple of months. So I would completely disagree with your speculation about what's easy or hard. (Note that I've worked at a search engine and an archive.)
  - singlow 10y ago
    My understanding is that scrape shield protects against certain abusive patterns of robot access, but not all URLs on the domain would ban robots. So archive may have many contaminated pages that scrape shield was enabled for but did not ban all robots from crawling based on various criteria.
renaudg 10y ago
we can just nuke the caches for all the domains that turned on the problematic features (Scrape Shield, etc.) No, IIUC it's MUCH worse than that : a single nginx process would serve many different domains, and you would only need one of them with the problematic features turned on to trigger the bug, and potentially leak data from all other domains served by that same process (whether those other domains had the feature enabled or not) I'm not familiar enough with CF's infrastructure to make an educated guess about the way they shard domains across their nginx fleet (assuming this is what they do), but in order to be able to produce a list of all "tainted" domains that were being served together on the same nginx process, at any point in time since September 2016, well their nginx config generation process had better be perfectly deterministic (and any changes thoroughly logged over time). Also, the bigger the shards, the more neighboring domains are affected by a single "poisoned" one, and the more widespread the problem is. My guess is that if they had had a way to reliably piece together that information, they would have released a list by now.
- bigiain 10y ago
  > My guess is that if they had had a way to reliably piece together that information, they would have released a list by now. Or they've got that list - and it's so vast and scary that legal and/or the board refuse to even acknowledge its existence…

by kijin 10y ago

josecastillo 10y ago

Nuking the caches is one thing, but what about services like the Internet Archive whose job it is to hang on to these pages? Pages with leaked data are clearly difficult to identify; removing the leaked data without nuking the document may be impossible, at least in an automated fashion. Are we supposed to erase five months of history from the affected domains? This CloudFlare breach seems to have put a lot of people in a tough spot, but it feels like it's put archivists in an impossible position.

gech 10y ago
Good time to shoe in some forced dmca content removal
- greglindahl 10y ago
  Yeah, abusing the DMCA because it's the only tool you can think of, that's a great idea.
  - kijin 10y ago
    Come to think of it, our society grants search engines the privilege of keeping copies of copyrighted material in exchange for the services they provide. It might not be unreasonable to say that this privilege comes with a certain responsibility to ensure that those copies do not cause excessive harm to others. So although Cloudflare is the one that fucked up, Google et al. also have a responsibility to do whatever they can to protect the public. They should do what they can, with or without Cloudflare's cooperation.
    - d33 10y ago
      I don't believe so. Cloudflare had this responsibility and they messed up. They are the only ones liable.
      - greglindahl 10y ago
        Yep, it's a Section 230 for the search engine, probably -- the unfortunate data came from Cloudflare.
      - kijin 10y ago
        Cloudflare cannot delete documents from other company's caches. The actual deletion must be performed by Google and Bing, who can then sue Cloudflare for cleanup costs if the latter is unwilling to cooperate. When there's an oil spill, we don't wait for the oil company to come and clean up their own mess. Others clean it up a.s.a.p. and (ideally) then make the oil company pay the fines and damages. CloudBleed is a virtual oil spill. They literally sprayed other people's private data all over the internet.
  - arrty88 10y ago
    Why can't they just search for pages with data after the closing HTML tag
    - ec109685 10y ago
      That isn't uncommon enough. Searching for one of the CF- http headers in the source code, however would work.
kijin 10y ago
I agree that nuking entire domains would be bad for the Internet Archive. But I don't think it would be overwhelmingly difficult, nor controversial, to identify and remove the vast majority of "contaminated" documents. This applies to the Internet Archive as well as major search engines. First, we're talking about raw memory pages, not merely malformed HTML. Those memory pages might contain valid HTML, but most of the sensitive information is in the headers, not HTML markup. It won't be very difficult to write a script to identify documents where random headers and POST data have been inserted where they don't belong, or where the markup is so obviously invalid (even compared to similar documents from the same site) that there is a high probability of contamination. Having a full list of contaminated domains would obviously help a lot, because we'll only have to deal with thousands of domains instead of millions. Second, contaminated documents by definition contain information that is NOT what the publisher intended to be crawled, indexed, or archived. So there should be less resistance to removing them. Finally, most of the contaminated domains used features such as Scrape Shield that were intended to deter archival. It's as if the domain had a robots.txt that said "User-agent:* Disallow:/". I'm not sure whether it's even possible for the Internet Archive to archive such domains. If they can, maybe they've been doing it against the publisher's wishes. If they can't, well, there's no problem to begin with.
- greglindahl 10y ago
  Archives don't delete stuff, nor do they have much capacity to do much computation on their archived data. Whereas if blekko still existed as a search engine, I'd just push code to refuse to show cached pages or snippets containing text that likely means the CloudFlare problem. 15 minutes work, and the underlying data would expire fully in a couple of months. So I would completely disagree with your speculation about what's easy or hard. (Note that I've worked at a search engine and an archive.)
- singlow 10y ago
  My understanding is that scrape shield protects against certain abusive patterns of robot access, but not all URLs on the domain would ban robots. So archive may have many contaminated pages that scrape shield was enabled for but did not ban all robots from crawling based on various criteria.

renaudg 10y ago

we can just nuke the caches for all the domains that turned on the problematic features (Scrape Shield, etc.) No, IIUC it's MUCH worse than that : a single nginx process would serve many different domains, and you would only need one of them with the problematic features turned on to trigger the bug, and potentially leak data from all other domains served by that same process (whether those other domains had the feature enabled or not) I'm not familiar enough with CF's infrastructure to make an educated guess about the way they shard domains across their nginx fleet (assuming this is what they do), but in order to be able to produce a list of all "tainted" domains that were being served together on the same nginx process, at any point in time since September 2016, well their nginx config generation process had better be perfectly deterministic (and any changes thoroughly logged over time). Also, the bigger the shards, the more neighboring domains are affected by a single "poisoned" one, and the more widespread the problem is. My guess is that if they had had a way to reliably piece together that information, they would have released a list by now.

bigiain 10y ago
> My guess is that if they had had a way to reliably piece together that information, they would have released a list by now. Or they've got that list - and it's so vast and scary that legal and/or the board refuse to even acknowledge its existence…