5 ms·

Changes to Ragel in Response to the CloudFlare Incident

84 points by LaFolle 10y ago

wyldfire 10y ago
For those of us who knew about the incident but not the nitty-gritty details -- "What is Ragel?" > Ragel compiles executable finite state machines from regular languages. Ragel targets C, C++ and ASM. Ragel state machines can not only recognize byte sequences as regular expression machines do, but can also execute code at arbitrary points in the recognition of a regular language. > What kind of task is Ragel good for? > Writing robust protocol implementations.
- tyingq 10y ago
  Cloudflare's postmortem also goes into some detail on what they did wrong with Ragel: http://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/ http://blog.cloudflare.com/incident-report-on-memory-leak-ca...
  - runevault 10y ago
    Believe they originally just stated Ragel had problems and only fixed it to state that they miswrote their stuff after the creator reached out to them.
    - tedivm 10y ago
      Cloudflare is really failing to inspire confidence here. Between them downplaying the issue, refusing to work with the Google engineer who found it, blaming the proxy owners (such as google) for not cleaning up their mess quickly enough, and attempting to dump the blame on groups like Ragel- I just have a hard time believing that Cloudflare will ever handle a security incident properly after this mess.
      - ethbro 10y ago
        You mean by being transparent as opposed to some of the other serious security incidents out there? Could you please clarify exactly how they "downplayed the issue"? The impact is an exercise in statistics. They seem to have compiled what logging / statistics they could, then accurately advised everyone of what they found. They turned around an initial fix on the issue in hours, no? As for web caches, what are they supposed to do, other than plead for their owners to delete them ASAP? As for Ragel, not sure how the initial version of the response read, but the one I read clearly pointed out it was a Cloudflare issue. And given the semantics of the issue, I'd cut them some slack if they misidentified the cause the day after they'd been scrambling to fix this. I'd be interested who you do trust to properly handle a security incident if you're looking for better than this. IBM? Microsoft? Apple? Just try and get a 24-hour fix out of them.
        cbr 10y ago
        At least with "blaming the proxy owners (such as google) for not cleaning up their mess quickly enough" I think tedivim is probably referring to: https://news.ycombinator.com/item?id=13721644 https://news.ycombinator.com/item?id=13721644
        ethbro 10y ago
        I saw that comment when it hit the original thread. I think commenters are conflating Project Zero with Google search cache in the usual version of this sentiment. If I was Cloudflare, and someone had made me aware of this bug, I'd be griping as loudly as possible to get remaining caches cleared ASAP. It's all they really can do. The fact that Bing had unlocated cached documents doesn't impact on Google search being quick or not to clear their own caches. Did Cloudflare cause all this? Yes. Are they working in the interest of their customers to gripe at remaining cachers to refresh their caches? Also yes. Is there something else they could do now that we're here? I can't think of anything more effective.
        tyingq 10y ago
        >not sure how the initial version of the response read, but the one I read clearly pointed out it was a Cloudflare issue Here's a diff of the original and what's there now. https://gist.github.com/anonymous/e8f78f537551465e54a9781d6212a96d https://gist.github.com/anonymous/e8f78f537551465e54a9781d62... To me, it doesn't look like any attempt to change history. They did try to make it more clear that it was their issue, and not Ragel's. But, any initial appearance of that seems accidental.
    - jgrahamc 10y ago
      Incorrect. I originally wrote: To modify the page, we need to read and parse the HTML to find elements that need changing. Since the very early days of Cloudflare, we’ve used a parser written using Ragel. A single .rl file contains an HTML parser used for all the on-the-fly HTML modifications that Cloudflare performs. About a year ago we decided that the Ragel parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain. We first used this new parser for the Automatic HTTP Rewrites feature and have been slowly migrating functionality that uses the old Ragel parser to cf-html. Both cf-html and the old Ragel parser are implemented as NGINX modules compiled into our NGINX builds. These NGINX filter modules parse buffers (blocks of memory) containing HTML responses, make modifications as necessary, and pass the buffers onto the next filter. It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself. The Ragel author contact me on Twitter and I changed the text to (stars around changes): To modify the page, we need to read and parse the HTML to find elements that need changing. Since the very early days of Cloudflare, we’ve used a parser written using Ragel. A single .rl file contains an HTML parser used for all the on-the-fly HTML modifications that Cloudflare performs. About a year ago we decided that the *Ragel-based* parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain. We first used this new parser for the Automatic HTTP Rewrites feature and have been slowly migrating functionality that uses the old Ragel parser to cf-html. Both cf-html and the old Ragel parser are implemented as NGINX modules compiled into our NGINX builds. These NGINX filter modules parse buffers (blocks of memory) containing HTML responses, make modifications as necessary, and pass the buffers onto the next filter. *For the avoidance of doubt: the bug is not in Ragel itself. It is in Cloudflare's use of Ragel. This is our bug and not the fault of Ragel.* It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself. https://twitter.com/jgrahamc/status/835056414062161920 https://twitter.com/jgrahamc/status/835056414062161920
devy 10y ago
Honest question: would Ragel be more robust if it were written in Rust and to generate Rust code?
- pjmlp 10y ago
  Logical errors cannot be avoided regardless of the language, even theorem provers might have incorrect axioms. However memory corruption issues, like in this case (out of bounds corruption), are easily avoidable in any memory safe language.
- aidenn0 10y ago
  The language Ragel is written in is fairly inconsequential. Not because memory errors don't happen in compilers, but because compilers typically don't get malicious inputs. Generating Rust would be useful, but Ragel already has an ML backend IIRC.
  - steveklabnik 10y ago
    There used to be a Rust backend for ragel, but it was well before 1.0 and it bitrotted and then died, as far as I know.
    - erickt 10y ago
      I wrote it [0]. I meant to get it merged upstream, but the author took the repository offline to help build his business, so my fork bitrotted. However, it looks like Rust support may have independently landed upstream [2]? [0]: https://github.com/erickt/ https://github.com/erickt/ [1]: http://www.colm.net/news/2014/10/24/ragel-now-maintained-by-colm-networks.html http://www.colm.net/news/2014/10/24/ragel-now-maintained-by-... [2]: http://www.colm.net/news/2015/09/26/first-dev-version-of-ragel-7-released.html http://www.colm.net/news/2015/09/26/first-dev-version-of-rag...
- JoachimSchipper 10y ago
  Not really. Safe code generating unsafe-code-with-bugs gets you a vulnerable program, so rewriting Ragel in Rust doesn't help. As to generating Rust - Rust isn't magic, and Rust-that-is-as-fast-as-C can't do a lot more work than C. In particular, you don't want to do memory allocation in the fast path of a network server written for performance (note that Rust uses jemalloc, which is a C memory allocation library; Rust memory allocation isn't going to beat C memory allocation.) As such, a realistic Rust program probably needs to re-use network buffers, at which point language-level security doesn't stop you from sending out "old" data that was still in the buffer. [EDIT: but see the reply https://news.ycombinator.com/item?id=13804831 https://news.ycombinator.com/item?id=13804831 below.] A debug build in Rust could allocate-per-request and detect the problem; but allocate-per-request in C is also likely to detect such overflows (indeed, the CloudFlare server did occasionally crash on this bug), and C programmers can also use e.g. VALGRIND_MAKE_MEM_UNDEFINED() for effort comparable to maintaining an alternate allocate-per-request implementation in Rust. The difference for the fast path - where Rust can't do a lot more work, and where you can focus your effort - just isn't very large. (Using a safe language for e.g. low-performance control-plane interfaces can help, however - but at that point, you may well be arguing e.g. Rust vs. Lua.)
deleted 10y ago
[deleted]

84 points by LaFolle 10y ago

For those of us who knew about the incident but not the nitty-gritty details -- "What is Ragel?" > Ragel compiles executable finite state machines from regular languages. Ragel targets C, C++ and ASM. Ragel state machines can not only recognize byte sequences as regular expression machines do, but can also execute code at arbitrary points in the recognition of a regular language. > What kind of task is Ragel good for? > Writing robust protocol implementations.

tyingq 10y ago
Cloudflare's postmortem also goes into some detail on what they did wrong with Ragel: http://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/ http://blog.cloudflare.com/incident-report-on-memory-leak-ca...
- runevault 10y ago
  Believe they originally just stated Ragel had problems and only fixed it to state that they miswrote their stuff after the creator reached out to them.
  - tedivm 10y ago
    Cloudflare is really failing to inspire confidence here. Between them downplaying the issue, refusing to work with the Google engineer who found it, blaming the proxy owners (such as google) for not cleaning up their mess quickly enough, and attempting to dump the blame on groups like Ragel- I just have a hard time believing that Cloudflare will ever handle a security incident properly after this mess.
    - ethbro 10y ago
      You mean by being transparent as opposed to some of the other serious security incidents out there? Could you please clarify exactly how they "downplayed the issue"? The impact is an exercise in statistics. They seem to have compiled what logging / statistics they could, then accurately advised everyone of what they found. They turned around an initial fix on the issue in hours, no? As for web caches, what are they supposed to do, other than plead for their owners to delete them ASAP? As for Ragel, not sure how the initial version of the response read, but the one I read clearly pointed out it was a Cloudflare issue. And given the semantics of the issue, I'd cut them some slack if they misidentified the cause the day after they'd been scrambling to fix this. I'd be interested who you do trust to properly handle a security incident if you're looking for better than this. IBM? Microsoft? Apple? Just try and get a 24-hour fix out of them.
      - cbr 10y ago
        At least with "blaming the proxy owners (such as google) for not cleaning up their mess quickly enough" I think tedivim is probably referring to: https://news.ycombinator.com/item?id=13721644 https://news.ycombinator.com/item?id=13721644
        ethbro 10y ago
        I saw that comment when it hit the original thread. I think commenters are conflating Project Zero with Google search cache in the usual version of this sentiment. If I was Cloudflare, and someone had made me aware of this bug, I'd be griping as loudly as possible to get remaining caches cleared ASAP. It's all they really can do. The fact that Bing had unlocated cached documents doesn't impact on Google search being quick or not to clear their own caches. Did Cloudflare cause all this? Yes. Are they working in the interest of their customers to gripe at remaining cachers to refresh their caches? Also yes. Is there something else they could do now that we're here? I can't think of anything more effective.
      - tyingq 10y ago
        >not sure how the initial version of the response read, but the one I read clearly pointed out it was a Cloudflare issue Here's a diff of the original and what's there now. https://gist.github.com/anonymous/e8f78f537551465e54a9781d6212a96d https://gist.github.com/anonymous/e8f78f537551465e54a9781d62... To me, it doesn't look like any attempt to change history. They did try to make it more clear that it was their issue, and not Ragel's. But, any initial appearance of that seems accidental.
  - jgrahamc 10y ago
    Incorrect. I originally wrote: To modify the page, we need to read and parse the HTML to find elements that need changing. Since the very early days of Cloudflare, we’ve used a parser written using Ragel. A single .rl file contains an HTML parser used for all the on-the-fly HTML modifications that Cloudflare performs. About a year ago we decided that the Ragel parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain. We first used this new parser for the Automatic HTTP Rewrites feature and have been slowly migrating functionality that uses the old Ragel parser to cf-html. Both cf-html and the old Ragel parser are implemented as NGINX modules compiled into our NGINX builds. These NGINX filter modules parse buffers (blocks of memory) containing HTML responses, make modifications as necessary, and pass the buffers onto the next filter. It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself. The Ragel author contact me on Twitter and I changed the text to (stars around changes): To modify the page, we need to read and parse the HTML to find elements that need changing. Since the very early days of Cloudflare, we’ve used a parser written using Ragel. A single .rl file contains an HTML parser used for all the on-the-fly HTML modifications that Cloudflare performs. About a year ago we decided that the *Ragel-based* parser had become too complex to maintain and we started to write a new parser, named cf-html, to replace it. This streaming parser works correctly with HTML5 and is much, much faster and easier to maintain. We first used this new parser for the Automatic HTTP Rewrites feature and have been slowly migrating functionality that uses the old Ragel parser to cf-html. Both cf-html and the old Ragel parser are implemented as NGINX modules compiled into our NGINX builds. These NGINX filter modules parse buffers (blocks of memory) containing HTML responses, make modifications as necessary, and pass the buffers onto the next filter. *For the avoidance of doubt: the bug is not in Ragel itself. It is in Cloudflare's use of Ragel. This is our bug and not the fault of Ragel.* It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself. https://twitter.com/jgrahamc/status/835056414062161920 https://twitter.com/jgrahamc/status/835056414062161920

devy 10y ago

Honest question: would Ragel be more robust if it were written in Rust and to generate Rust code?

pjmlp 10y ago
Logical errors cannot be avoided regardless of the language, even theorem provers might have incorrect axioms. However memory corruption issues, like in this case (out of bounds corruption), are easily avoidable in any memory safe language.
aidenn0 10y ago
The language Ragel is written in is fairly inconsequential. Not because memory errors don't happen in compilers, but because compilers typically don't get malicious inputs. Generating Rust would be useful, but Ragel already has an ML backend IIRC.
- steveklabnik 10y ago
  There used to be a Rust backend for ragel, but it was well before 1.0 and it bitrotted and then died, as far as I know.
  - erickt 10y ago
    I wrote it [0]. I meant to get it merged upstream, but the author took the repository offline to help build his business, so my fork bitrotted. However, it looks like Rust support may have independently landed upstream [2]? [0]: https://github.com/erickt/ https://github.com/erickt/ [1]: http://www.colm.net/news/2014/10/24/ragel-now-maintained-by-colm-networks.html http://www.colm.net/news/2014/10/24/ragel-now-maintained-by-... [2]: http://www.colm.net/news/2015/09/26/first-dev-version-of-ragel-7-released.html http://www.colm.net/news/2015/09/26/first-dev-version-of-rag...
JoachimSchipper 10y ago
Not really. Safe code generating unsafe-code-with-bugs gets you a vulnerable program, so rewriting Ragel in Rust doesn't help. As to generating Rust - Rust isn't magic, and Rust-that-is-as-fast-as-C can't do a lot more work than C. In particular, you don't want to do memory allocation in the fast path of a network server written for performance (note that Rust uses jemalloc, which is a C memory allocation library; Rust memory allocation isn't going to beat C memory allocation.) As such, a realistic Rust program probably needs to re-use network buffers, at which point language-level security doesn't stop you from sending out "old" data that was still in the buffer. [EDIT: but see the reply https://news.ycombinator.com/item?id=13804831 https://news.ycombinator.com/item?id=13804831 below.] A debug build in Rust could allocate-per-request and detect the problem; but allocate-per-request in C is also likely to detect such overflows (indeed, the CloudFlare server did occasionally crash on this bug), and C programmers can also use e.g. VALGRIND_MAKE_MEM_UNDEFINED() for effort comparable to maintaining an alternate allocate-per-request implementation in Rust. The difference for the fast path - where Rust can't do a lot more work, and where you can focus your effort - just isn't very large. (Using a safe language for e.g. low-performance control-plane interfaces can help, however - but at that point, you may well be arguing e.g. Rust vs. Lua.)

deleted 10y ago

[deleted]