4 ms·

When I designed Cloudflare's filters, a.k.a. Firewall Rules... we had an API written in Go, and the edge written in Lua. The risk I foresaw was that building ru

by buro9 3y ago

When I designed Cloudflare's filters, a.k.a. Firewall Rules... we had an API written in Go, and the edge written in Lua. The risk I foresaw was that building rules that act on URLs and perform matching, and testing those, in one language (Go) that were then deployed and executed in another (Lua) would lead to a risk that the difference in the behaviour of the two engines (no matter how much they implemented the same spec) would result in a rule being created that behaved ever so differently elsewhere... for me, that was a huge hole as if it was known it would be leveraged. The first piece of Rust in Cloudflare was introduced for this reason and for this project. We needed a single piece of code to implement the filters that could be executed from Lua, and that Go could call too. It needed to ensure that only a single piece of code was responsible for things like UPPER(), LOWER(), URL_PARSE() precisely to prevent any differences being leveraged.

SerCe 3y ago
Out of interest, was this the project that eventually became wirefilter [1]? [1]: https://github.com/cloudflare/wirefilter https://github.com/cloudflare/wirefilter
- buro9 3y ago
  Yup, that's the one. Uses wireshark display filters syntax to implement the ability to match traffic at any part of the stack. It's still used internally, but clearly they stopped updating the OSS version just after I left. They were updating things to make it fully compatible for everything owasp when I left which meant additions beyond the wireshark definition of display filters. Everything firewall and WAF was moving to the filters and also other traffic matching features too.
  - SerCe 3y ago
    Thank you for releasing it! I adopted wirefilter for a firewall rule testing project, firewalker [1]. But indeed, I wish Cloudflare kept maintaining its OSS version. [1]: https://github.com/SerCeMan/firewalker/ https://github.com/SerCeMan/firewalker/
    - buro9 3y ago
      Unfortunately cloudflare has a poor OSS track record. Either not maintaining the public version, or promising to open things that then don't see the light of day (quicksilver database and replication, and their Rust reverse proxy and nginx/lua replacement - both of which were announced but never released). Most of what is OSS at cloudflare came in from elsewhere (V8) or was needed for collaboration (IETF), rather than started at cloudflare and opened.
notsahil 3y ago
WAF reminded me of the Cloudflare outage on July 2, 2019 due to small regex mistake [1] [1] https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
- buro9 3y ago
  Yup, and the filters described above replaced the pcre regex matching in lua. That was already planned at the time of this incident, but this incident accelerated the pace at which the new system was put in place as well as introduced controls and process for the release of waf rules (regardless of what engine they were applied in)
ReactiveJelly 3y ago
For curious readers, it sounds like the URL equivalent of header smuggling https://en.wikipedia.org/wiki/HTTP_request_smuggling https://en.wikipedia.org/wiki/HTTP_request_smuggling
Analemma_ 3y ago
This is such a cool example of real-world defensive engineering. I think I'll refer back to it when I want to illustrate the concept to somebody.
rcme 3y ago
Why Rust instead of cgo?
- buro9 3y ago
  Lots of reasons that on their own were not enough to decide but together made it compelling. We weren't going to use C (Cloudbleed), cgo didn't give us as much control over the FFI as we wanted and produced slightly harder to read output (maintainability for me, is everything — always write code that the drunk at 2am future version of yourself can understand when you're paged), we didn't necessarily want another garbage collected process despite being good at this (Cloudflare's DNS server is written in Go), we wanted memory safety and wanted to keep things small (Go or Rust stand out), we wanted compile time safety over runtime errors, we believed that Rust macros would help make the filter code more readable / maintainable than rolling our own parser in Go or using YACC (see earlier point about your future self maintaining this at an unGodly hour)... it's not like anything was a determining factor, but the reasons accrued until it felt overwhelming. We certainly were not looking to add another language at the time as being first to do so incurs pushback from the org, adding into all of the build and release chains, and the typical higher bar for proving you know what you're doing. Once done everyone can benefit from having the option of another language and being able to select the right tool for the job, but going first incurs a legitimate cost.

by buro9 3y ago

SerCe 3y ago

Out of interest, was this the project that eventually became wirefilter [1]? [1]: https://github.com/cloudflare/wirefilter https://github.com/cloudflare/wirefilter

buro9 3y ago
Yup, that's the one. Uses wireshark display filters syntax to implement the ability to match traffic at any part of the stack. It's still used internally, but clearly they stopped updating the OSS version just after I left. They were updating things to make it fully compatible for everything owasp when I left which meant additions beyond the wireshark definition of display filters. Everything firewall and WAF was moving to the filters and also other traffic matching features too.
- SerCe 3y ago
  Thank you for releasing it! I adopted wirefilter for a firewall rule testing project, firewalker [1]. But indeed, I wish Cloudflare kept maintaining its OSS version. [1]: https://github.com/SerCeMan/firewalker/ https://github.com/SerCeMan/firewalker/
  - buro9 3y ago
    Unfortunately cloudflare has a poor OSS track record. Either not maintaining the public version, or promising to open things that then don't see the light of day (quicksilver database and replication, and their Rust reverse proxy and nginx/lua replacement - both of which were announced but never released). Most of what is OSS at cloudflare came in from elsewhere (V8) or was needed for collaboration (IETF), rather than started at cloudflare and opened.

notsahil 3y ago

WAF reminded me of the Cloudflare outage on July 2, 2019 due to small regex mistake [1] [1] https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...

buro9 3y ago
Yup, and the filters described above replaced the pcre regex matching in lua. That was already planned at the time of this incident, but this incident accelerated the pace at which the new system was put in place as well as introduced controls and process for the release of waf rules (regardless of what engine they were applied in)

ReactiveJelly 3y ago

For curious readers, it sounds like the URL equivalent of header smuggling https://en.wikipedia.org/wiki/HTTP_request_smuggling https://en.wikipedia.org/wiki/HTTP_request_smuggling

Analemma_ 3y ago

This is such a cool example of real-world defensive engineering. I think I'll refer back to it when I want to illustrate the concept to somebody.

rcme 3y ago

Why Rust instead of cgo?

buro9 3y ago
Lots of reasons that on their own were not enough to decide but together made it compelling. We weren't going to use C (Cloudbleed), cgo didn't give us as much control over the FFI as we wanted and produced slightly harder to read output (maintainability for me, is everything — always write code that the drunk at 2am future version of yourself can understand when you're paged), we didn't necessarily want another garbage collected process despite being good at this (Cloudflare's DNS server is written in Go), we wanted memory safety and wanted to keep things small (Go or Rust stand out), we wanted compile time safety over runtime errors, we believed that Rust macros would help make the filter code more readable / maintainable than rolling our own parser in Go or using YACC (see earlier point about your future self maintaining this at an unGodly hour)... it's not like anything was a determining factor, but the reasons accrued until it felt overwhelming. We certainly were not looking to add another language at the time as being first to do so incurs pushback from the org, adding into all of the build and release chains, and the typical higher bar for proving you know what you're doing. Once done everyone can benefit from having the option of another language and being able to select the right tool for the job, but going first incurs a legitimate cost.