4 ms·

It's been pretty entertaining watching taviso's attitude towards CF go from "we trust them" to "dude, you're a tool". I kind of understand what CF is doing her

It's been pretty entertaining watching taviso's attitude towards CF go from "we trust them" to "dude, you're a tool". I kind of understand what CF is doing here: they've screwed up, there's no way for them to clean it up, so all they can do now is deflect attention from the magnitude of their screw up by blaming others for not working fast enough in the hope that their fake paper multibillion dollar valuation doesn't take too big a hit. Still a dick move though. Maybe next time don't use a language without memory safety to parse untrusted input.

erpellan 10y ago
So much this! It's 2017. I get that maybe, _maybe_ at scale you need the fastest code you can get. But you'd damn well better fuzz-test the crap out of it before it goes anywhere near a network connection.
CrLf 10y ago
> Maybe next time don't use a language without memory safety to parse untrusted input. Untrusted input is safely parsed by programs written in languages without memory safety all the time. In fact, most language runtimes with memory safety are implemented in languages _without_ memory safety. What's to criticize here is parsing untrusted input in the same memory space as sensitive information.
- cbr 10y ago
  How would you rewrite websites to optimize them without parsing untrusted input in the same memory space as sensitive information? The thing you're trying to change (the HTML) can have PII or be otherwise sensitive. (I used to work on Google's PageSpeed Service, and if it had had the same bug I think we would have been in the same situation as CF is now.)
  - CrLf 10y ago
    > The thing you're trying to change (the HTML) can have PII or be otherwise sensitive. Sure it can. But leaking PII from the thing you're parsing and leaking PII from any other random request isn't the same thing. I understand the performance implications (and the added effort) of sandboxing the parser, but I'm arguing for it anyway. The mere presence of a man-in-the-middle decrypting HTTPS and pooling cleartext data from many disparate services in a single memory space is already questionable (something for Cloudflare customers - and not Cloudflare itself - to think about) but adding risky features into the mix shouldn't be done without as much isolation as possible. Let's face it: parsers are about the most likely place for this sort of leakage to happen...
    - jefftk 10y ago
      Actually, thinking more, we designed PSS to run in a sandbox specifically because we were parsing untrusted input. But leaking content from one site into responses from other sites would still be possible, because I think we didn't reinitialize the sandbox on every request (way expensive) and each server process handled many sites. Fix that, and then there's still the risk of leaking things between sites via the cache. It's definitely possible to fix this (new sandbox per request, cache is fragmented by something outside of sandbox control) but I'm not sure the service would make sense economically.
  - tedunangst 10y ago
    The obvious solution that occurs to me would be to isolate the "email protection" feature to a second process, thus limiting the scope of the catastrophe to only those sites using the feature and not everyone.
- rocho 10y ago
  But other than that, they didn't even do the most basic testing. I mean, they wrote an HTML parser and didn't test it with malformed input?! It's all over the place on the Internet.
baby 10y ago
> Maybe next time don't use a language without memory safety to parse untrusted input They use Go more than any other companies, your sentence seems a bit odd.
- rybosome 10y ago
  But the code responsible for the bug was generated C. That they use Go in other applications is irrelevant.
  - baby 10y ago
    Any huge codebase will have some C at some point. You use Linux? It's in C. Plus they mentioned that they were trying to replace the vulnerable code before that bug happened.
    - sillysaurus3 10y ago
      The bug happened in September of last year. Since then we can assume they've been leaking 100k-120k of requests per day, unless CloudFlare makes a statement with hard evidence to the contrary. And that assumes no attacker noticed the bug and mined them for private data.

erpellan 10y ago

So much this! It's 2017. I get that maybe, _maybe_ at scale you need the fastest code you can get. But you'd damn well better fuzz-test the crap out of it before it goes anywhere near a network connection.

CrLf 10y ago

> Maybe next time don't use a language without memory safety to parse untrusted input. Untrusted input is safely parsed by programs written in languages without memory safety all the time. In fact, most language runtimes with memory safety are implemented in languages _without_ memory safety. What's to criticize here is parsing untrusted input in the same memory space as sensitive information.

cbr 10y ago
How would you rewrite websites to optimize them without parsing untrusted input in the same memory space as sensitive information? The thing you're trying to change (the HTML) can have PII or be otherwise sensitive. (I used to work on Google's PageSpeed Service, and if it had had the same bug I think we would have been in the same situation as CF is now.)
- CrLf 10y ago
  > The thing you're trying to change (the HTML) can have PII or be otherwise sensitive. Sure it can. But leaking PII from the thing you're parsing and leaking PII from any other random request isn't the same thing. I understand the performance implications (and the added effort) of sandboxing the parser, but I'm arguing for it anyway. The mere presence of a man-in-the-middle decrypting HTTPS and pooling cleartext data from many disparate services in a single memory space is already questionable (something for Cloudflare customers - and not Cloudflare itself - to think about) but adding risky features into the mix shouldn't be done without as much isolation as possible. Let's face it: parsers are about the most likely place for this sort of leakage to happen...
  - jefftk 10y ago
    Actually, thinking more, we designed PSS to run in a sandbox specifically because we were parsing untrusted input. But leaking content from one site into responses from other sites would still be possible, because I think we didn't reinitialize the sandbox on every request (way expensive) and each server process handled many sites. Fix that, and then there's still the risk of leaking things between sites via the cache. It's definitely possible to fix this (new sandbox per request, cache is fragmented by something outside of sandbox control) but I'm not sure the service would make sense economically.
- tedunangst 10y ago
  The obvious solution that occurs to me would be to isolate the "email protection" feature to a second process, thus limiting the scope of the catastrophe to only those sites using the feature and not everyone.
rocho 10y ago
But other than that, they didn't even do the most basic testing. I mean, they wrote an HTML parser and didn't test it with malformed input?! It's all over the place on the Internet.

baby 10y ago

> Maybe next time don't use a language without memory safety to parse untrusted input They use Go more than any other companies, your sentence seems a bit odd.

rybosome 10y ago
But the code responsible for the bug was generated C. That they use Go in other applications is irrelevant.
- baby 10y ago
  Any huge codebase will have some C at some point. You use Linux? It's in C. Plus they mentioned that they were trying to replace the vulnerable code before that bug happened.
  - sillysaurus3 10y ago
    The bug happened in September of last year. Since then we can assume they've been leaking 100k-120k of requests per day, unless CloudFlare makes a statement with hard evidence to the contrary. And that assumes no attacker noticed the bug and mined them for private data.