9 ms·

From TFA: XTTP/1.1 500 Error Content-Type: text/html Secret-Header: secret-info Secret info, should not be visible! What the hell backend wou

by dheera 6y ago

From TFA: XTTP/1.1 500 Error Content-Type: text/html Secret-Header: secret-info Secret info, should not be visible! What the hell backend would respond with this? I just tested this with a simple hello world NodeJS backend: const express = require('express'); const app = express(); app.get('/', (req, res) => { res.send('Hello World!') }); app.listen(3000, () => { console.log('Listening'); }); And then tried the example: $ telnet localhost 3000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /? XTTP/1.1 Host: 127.0.0.1 Connection: closeHTTP/1.1 400 Bad Request Connection: close Connection closed by foreign host. Seems like the backend responded with a valid HTTP response even though my request was invalid. Of course that doesn't speak for all backend frameworks people would use, but it would occur to me that a well-designed backend would always speak proper HTTP even if the input isn't proper HTTP, and if the backend receives a bad HTTP request it should immediately send back a 400 (not 500) and never pass it on to the business logic. My brief test above seems to suggest that Express does indeed behave this way. Separately, configure your backends to not spit out secret info in production mode and you should not have to actually worry about this.

throwaway894345 6y ago
I thought one of the value propositions for Nginx is that it's battle-tested so that you don't have to depend so much on your framework for security?
- dheera 6y ago
  If you're using it to serve static files sure, but the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend, which therefore should also be battle-tested. Nginx isn't going to protect you from SQL injection, routes that expose access to sensitive files, and so on. My typical configuration is to rely on Nginx for HTTPS and use an unencrypted HTTP backend listening only on 127.0.0.1 for non-static pages. I rely on the battle-tested encryption logic of Nginx but as far as invalid HTTP requests go that's all on the backend.
  - jlokier 6y ago
    > the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend Not really. That only applies for very simple proxy_pass configurations. NginX is used as more than just a simple proxy. The point of NginX directives like proxy_hide_header and proxy_intercept_errors it that you've configured NginX to do some extra processing. For example, I use proxy_intercept_errors so that backends can use certain error codes to instruct NginX to relay a request on to a different backend, after the first one has made a routing or versioning decision. I also use it in conjunction with X-Retry, X-Accel-Redirect to instruct NginX what to do next; should it replay the original request, or perform a new one given to it by the previous backend in the sequence, and relay state. This is a mechanism for backends to cooperate, and it also helps provide zero-downtime backend upgrades and routing to different hosts, invisible to the client. Those aren't robustness failures by the backends. They aren't lack of battle-testing. The backends are fine; but you still get the wrong result sent to the client in cases where NginX doesn't both to process the responses. The problem is this behaviour of NginX was not documented and there doesn't appear to be a way to tell it to do something else. It can be avoided with code in all the backends, but that code won't come from any specification. It can only be written after reading the NginX C source code to find under what conditions it does this, and have the backends go into a special mode when they detect it, responding with an error differently than they would to other bad request errors. Now I'm wondering if the headers X-Real-IP, X-Forwarded-For and X-Forwarded-Proto are reliably filtered out of the client request before being set or not set by the proxy itself. This is not something a backend can do for itself, it's intrinsic to proxying.
- veesahni 6y ago
  most application servers don't want to be the front for browsers users because they don't handle a large number of 'slow requests' well. nginx will deal with lots of slow clients and only pass to backend once it has the full request. security of an application is much more complex than just throwing nginx infront.
jlokier 6y ago
> What the hell backend would respond with this? The article's tiny Python example using uWSGI does. Some backends do this. Sure, Node doesn't. There are hundreds of backend frameworks widely used behind NginX though. It's not necessarily even a bug: There are legitimate use cases for responding to a not-quite-HTTP request, which is why NginX itself does it. > a well-designed backend would always speak proper HTTP By the same argument, a well-designed HTTP proxy would always speak proper HTTP and reject improper HTTP. Besides, strict filtering on the backend is not enough. Unless someone has audited these, your backend's carefully implemented proper HTTP request filtering might not necessarily be exactly the same as what NginX's filtering logic pattens matches when deciding whether to forward blindly. Sure, the "XTTP" example is obvious, but are you now confident there are no subtler variants of this surprising behaviour? Any mismatch of logic is a bit of wiggle room to sneak a valid HTTP request to your backend whose response is blindly copied back to the client. That failure to process could not only fail to filter or handle backend headers: I wonder if it also fails to add response headers, as well as failing to add request headers the backend depends on. For example X-Forwarded-For, and internal routing headers used to pass NginX variables to the backend. The real problem is that NginX going into "blind forwarding mode" isn't documented, isn't expected behaviour, and there doesn't appear to be any way to turn it off When you know about it, you can operate defensively on the backend by, as you say, being extra careful. But now I'm going to have to read the NginX source code to find out what kind of careful is required. And then check every backend, and block people wanting to add new ones until audited for this issue.
jlokier 6y ago
> Separately, configure your backends to not spit out secret info in production mode and you should not have to actually worry about this It's not really about secrets. That's just the example from the article. (Although, that can happen if the backend is communicating authorisation-to-serve to a cache.) You may be using X-Accel-Redirect: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/ https://www.nginx.com/resources/wiki/start/topics/examples/x... Or any of the other built-in X-Accel response headers that are automatically offered by proxy module: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... It's fairly common to use X-Accel-Redirect to have the backend tell NginX to serve a file. With a malformed request, NginX will serve the header instead of the file, revealing your internal filesystem structure to the client. Often that structure has hashes and versions in the paths. Even without those, it can be quite revealing. That's not a badly designed backend; this feature is useful and intentional. Having that pass through without being processed is a security fail of NginX, and they should at least document it. Better, provide an option to never blindly forward in these cases, or some "if" variable to let the admin configure what they want done. After learning about this, I expect there are plenty of sites out there which will reveal their X-Accel-Redirect paths if you ask them like this, though I can't be bothered to go looking.

by dheera 6y ago

throwaway894345 6y ago

I thought one of the value propositions for Nginx is that it's battle-tested so that you don't have to depend so much on your framework for security?

dheera 6y ago
If you're using it to serve static files sure, but the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend, which therefore should also be battle-tested. Nginx isn't going to protect you from SQL injection, routes that expose access to sensitive files, and so on. My typical configuration is to rely on Nginx for HTTPS and use an unencrypted HTTP backend listening only on 127.0.0.1 for non-static pages. I rely on the battle-tested encryption logic of Nginx but as far as invalid HTTP requests go that's all on the backend.
- jlokier 6y ago
  > the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend Not really. That only applies for very simple proxy_pass configurations. NginX is used as more than just a simple proxy. The point of NginX directives like proxy_hide_header and proxy_intercept_errors it that you've configured NginX to do some extra processing. For example, I use proxy_intercept_errors so that backends can use certain error codes to instruct NginX to relay a request on to a different backend, after the first one has made a routing or versioning decision. I also use it in conjunction with X-Retry, X-Accel-Redirect to instruct NginX what to do next; should it replay the original request, or perform a new one given to it by the previous backend in the sequence, and relay state. This is a mechanism for backends to cooperate, and it also helps provide zero-downtime backend upgrades and routing to different hosts, invisible to the client. Those aren't robustness failures by the backends. They aren't lack of battle-testing. The backends are fine; but you still get the wrong result sent to the client in cases where NginX doesn't both to process the responses. The problem is this behaviour of NginX was not documented and there doesn't appear to be a way to tell it to do something else. It can be avoided with code in all the backends, but that code won't come from any specification. It can only be written after reading the NginX C source code to find under what conditions it does this, and have the backends go into a special mode when they detect it, responding with an error differently than they would to other bad request errors. Now I'm wondering if the headers X-Real-IP, X-Forwarded-For and X-Forwarded-Proto are reliably filtered out of the client request before being set or not set by the proxy itself. This is not something a backend can do for itself, it's intrinsic to proxying.
veesahni 6y ago
most application servers don't want to be the front for browsers users because they don't handle a large number of 'slow requests' well. nginx will deal with lots of slow clients and only pass to backend once it has the full request. security of an application is much more complex than just throwing nginx infront.

jlokier 6y ago

> What the hell backend would respond with this? The article's tiny Python example using uWSGI does. Some backends do this. Sure, Node doesn't. There are hundreds of backend frameworks widely used behind NginX though. It's not necessarily even a bug: There are legitimate use cases for responding to a not-quite-HTTP request, which is why NginX itself does it. > a well-designed backend would always speak proper HTTP By the same argument, a well-designed HTTP proxy would always speak proper HTTP and reject improper HTTP. Besides, strict filtering on the backend is not enough. Unless someone has audited these, your backend's carefully implemented proper HTTP request filtering might not necessarily be exactly the same as what NginX's filtering logic pattens matches when deciding whether to forward blindly. Sure, the "XTTP" example is obvious, but are you now confident there are no subtler variants of this surprising behaviour? Any mismatch of logic is a bit of wiggle room to sneak a valid HTTP request to your backend whose response is blindly copied back to the client. That failure to process could not only fail to filter or handle backend headers: I wonder if it also fails to add response headers, as well as failing to add request headers the backend depends on. For example X-Forwarded-For, and internal routing headers used to pass NginX variables to the backend. The real problem is that NginX going into "blind forwarding mode" isn't documented, isn't expected behaviour, and there doesn't appear to be any way to turn it off When you know about it, you can operate defensively on the backend by, as you say, being extra careful. But now I'm going to have to read the NginX source code to find out what kind of careful is required. And then check every backend, and block people wanting to add new ones until audited for this issue.

jlokier 6y ago

> Separately, configure your backends to not spit out secret info in production mode and you should not have to actually worry about this It's not really about secrets. That's just the example from the article. (Although, that can happen if the backend is communicating authorisation-to-serve to a cache.) You may be using X-Accel-Redirect: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/ https://www.nginx.com/resources/wiki/start/topics/examples/x... Or any of the other built-in X-Accel response headers that are automatically offered by proxy module: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... It's fairly common to use X-Accel-Redirect to have the backend tell NginX to serve a file. With a malformed request, NginX will serve the header instead of the file, revealing your internal filesystem structure to the client. Often that structure has hashes and versions in the paths. Even without those, it can be quite revealing. That's not a badly designed backend; this feature is useful and intentional. Having that pass through without being processed is a security fail of NginX, and they should at least document it. Better, provide an option to never blindly forward in these cases, or some "if" variable to let the admin configure what they want done. After learning about this, I expect there are plenty of sites out there which will reveal their X-Accel-Redirect paths if you ask them like this, though I can't be bothered to go looking.