4 ms·

Incident report on memory leak caused by Cloudflare parser bug

115 points by jgrahamc 10y ago

kderbe 10y ago
See also the Project Zero ticket https://bugs.chromium.org/p/project-zero/issues/detail?id=1139 https://bugs.chromium.org/p/project-zero/issues/detail?id=11... and associated Hacker News discussion https://news.ycombinator.com/item?id=13718752 https://news.ycombinator.com/item?id=13718752
npongratz 10y ago
> "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory... The leaked memory has been purged with the help of the search engines." I wonder if the search engines found any interesting patterns of searches performed whose results contained these 770 unique URIs.
- deleted 10y ago
  [deleted]
- ultrahate 10y ago
  I've always wondered if Google did that. Seems obvious enough to just watch people using premade searches and see what they are up to.
papoh 10y ago
mh
jlgaddis 10y ago
@jgrahamc: What are your thoughts about Tavis' remark that Cloudflare's communications (the "draft", post-mortem, etc.) "severely downplays" this issue?
dbaupp 10y ago
Calling this a "memory leak" is rather misleading. The problem is literally leaking the contents of memory, yes, but the term memory leak usually means something else, something that is usually not as problematic as dumping random memory to the internet.
drewlesueur 10y ago
It's interesting that there was another recent Cloudflare bug caused by using an "==" vs "<=" or ">=". It's good to know that programs can get into states that you don't expect. https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ https://blog.cloudflare.com/how-and-why-the-leap-second-affe...
MattBearman 10y ago
I'm a Cloudflare (free) customer, and I've had no email or contact of any kind from Cloudflare. My understanding from reading this and Travis' post is that all Cloudflare customers are affected, is that correct?
- laithshadeed 10y ago
  No. If you have any of those features enabled Email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites as well as you have an invalid HTML markup. Per their blog post it is 0.00003% of the total requests.
  - MattBearman 10y ago
    I thought those were the conditions that caused PII to be included in the HTML response, but the PII could come from any customer as it's a buffer overrun situation. Damn this is confusing
- andreasley 10y ago
  Basically yes. If some of your traffic was flowing through a server on which any customer had the vulnerable features enabled, parts of your traffic may have been dumped.
devy 10y ago
I wonder if they rewrite their unsafe C code with Rust, would this bug still likely to happen?

115 points by jgrahamc 10y ago

See also the Project Zero ticket https://bugs.chromium.org/p/project-zero/issues/detail?id=1139 https://bugs.chromium.org/p/project-zero/issues/detail?id=11... and associated Hacker News discussion https://news.ycombinator.com/item?id=13718752 https://news.ycombinator.com/item?id=13718752

npongratz 10y ago

> "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory... The leaked memory has been purged with the help of the search engines." I wonder if the search engines found any interesting patterns of searches performed whose results contained these 770 unique URIs.

deleted 10y ago
[deleted]
ultrahate 10y ago
I've always wondered if Google did that. Seems obvious enough to just watch people using premade searches and see what they are up to.

papoh 10y ago

jlgaddis 10y ago

@jgrahamc: What are your thoughts about Tavis' remark that Cloudflare's communications (the "draft", post-mortem, etc.) "severely downplays" this issue?

dbaupp 10y ago

Calling this a "memory leak" is rather misleading. The problem is literally leaking the contents of memory, yes, but the term memory leak usually means something else, something that is usually not as problematic as dumping random memory to the internet.

drewlesueur 10y ago

It's interesting that there was another recent Cloudflare bug caused by using an "==" vs "<=" or ">=". It's good to know that programs can get into states that you don't expect. https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ https://blog.cloudflare.com/how-and-why-the-leap-second-affe...

MattBearman 10y ago

I'm a Cloudflare (free) customer, and I've had no email or contact of any kind from Cloudflare. My understanding from reading this and Travis' post is that all Cloudflare customers are affected, is that correct?

laithshadeed 10y ago
No. If you have any of those features enabled Email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites as well as you have an invalid HTML markup. Per their blog post it is 0.00003% of the total requests.
- MattBearman 10y ago
  I thought those were the conditions that caused PII to be included in the HTML response, but the PII could come from any customer as it's a buffer overrun situation. Damn this is confusing
andreasley 10y ago
Basically yes. If some of your traffic was flowing through a server on which any customer had the vulnerable features enabled, parts of your traffic may have been dumped.

devy 10y ago

I wonder if they rewrite their unsafe C code with Rust, would this bug still likely to happen?