6 ms·

Tell HN: Cloudflare is rewriting my Date response headers

4 points by ezekg 5y ago

Earlier this year I added HTTP response signatures to my business's API, per the HTTP draft spec [0]. These signatures utilize the standard Date response header to help prevent replay attacks among other attack vectors. But to my dismay, Cloudflare rewrites the Date response header at the edge. Meaning, if a particularly unlucky request leaves my servers at 14:01:26.999 with a Date header "Thu, 12 Aug 2021 14:01:26 GMT" and takes ~1ms to process at the edge, the Date header from Cloudflare ends up being "Thu, 12 Aug 2021 14:01:27 GMT". This is problematic when it comes to HTTP response signatures, since the signature is now invalidated. This ends up causing intermittent response signature verification failures. Cloudflare's support previously suggested that I use (and pay for) an unbound Cloudflare Worker to manually set the Date header to the value of some other internal, unmodified X-Date header. I tried this, but that's actually a noop! So not only are they rewriting the Date response header, but there is no way to stop this behavior. Looking at the HTTP spec, this seems in violation of it for a proxy to rewrite the Date after it has already been set [1] [2], but I can't be certain (keyword: "date and time at which the message originated"). Either way, this is unexpected behavior -- I'd expect every header I send in a response to actually make it, unmodified, to the client. But it seems that's not the case with CF. Honestly, I find this to be unacceptable behavior, but perhaps I'm out of line? As you could imagine, this issue has been frustrating and was incredibly hard to debug. I spent days digging into and stepping through Puma and Rails internals, thinking they were somehow overriding my Date header before sending the response. CF support has made it clear that this is a 'will not fix.' Thus, here we are. Perhaps there are a few CF engineers on HN that can help here. [0]: https://tools.ietf.org/id/draft-cavage-http-signatures-08.html [1]: https://datatracker.ietf.org/doc/html/rfc7231#section-7.1.1.2 [2]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18 p.s. anybody have a good Cloudflare alternative for WAF?

nanankcornering 5y ago
It's also the same thing with the "server" header right? Cloudflare will override the server header to their own name. I think this an old thing, not a new feature introduced recently. Well, not an issue for me though. Why not use your own headers?
- ezekg 5y ago
  Thanks for the suggestion. And yes, it likely is an old thing. Using my own X-Date header was an option I tossed around, but unfortunately, this issue was not discovered and fully understood until the feature had been out for a couple months. The issue is so infrequent due to the timing/latency involved that most customers didn't realize there was an issue. It was only after a high volume customer pinged me about the occasional (but frequent with high volume) 'invalid' signatures that the issue started to surface. Ultimately, since I have customers relying on the standard Date header within the signed data, and they're in a variety of environments (some not easily updated), I'm unwilling to introduce a breaking change like this simply to keep CF enabled. I ultimately had to disable CF to resolve the issue, and am looking to fully migrate away from the platform soon. Honestly, it just seems like odd behavior for CF to even do this and I thought I'd give others a heads up. I know startups like Cognito [0] follow that signature spec too, so this could effect other startups as well which implement HTTP response signatures. Especially if it eventually becomes standardized... [0]: https://cognitohq.com/docs/authenticating/ https://cognitohq.com/docs/authenticating/

4 points by ezekg 5y ago

nanankcornering 5y ago

It's also the same thing with the "server" header right? Cloudflare will override the server header to their own name. I think this an old thing, not a new feature introduced recently. Well, not an issue for me though. Why not use your own headers?

ezekg 5y ago
Thanks for the suggestion. And yes, it likely is an old thing. Using my own X-Date header was an option I tossed around, but unfortunately, this issue was not discovered and fully understood until the feature had been out for a couple months. The issue is so infrequent due to the timing/latency involved that most customers didn't realize there was an issue. It was only after a high volume customer pinged me about the occasional (but frequent with high volume) 'invalid' signatures that the issue started to surface. Ultimately, since I have customers relying on the standard Date header within the signed data, and they're in a variety of environments (some not easily updated), I'm unwilling to introduce a breaking change like this simply to keep CF enabled. I ultimately had to disable CF to resolve the issue, and am looking to fully migrate away from the platform soon. Honestly, it just seems like odd behavior for CF to even do this and I thought I'd give others a heads up. I know startups like Cognito [0] follow that signature spec too, so this could effect other startups as well which implement HTTP response signatures. Especially if it eventually becomes standardized... [0]: https://cognitohq.com/docs/authenticating/ https://cognitohq.com/docs/authenticating/