4 ms·

The one about proxy_pass blindly forwarding syntactically malformed requests and silently failing to process the response is astonishing. It doesn't appear to

The one about proxy_pass blindly forwarding syntactically malformed requests and silently failing to process the response is astonishing. It doesn't appear to be documented. Looking through NginX documentation at http://nginx.org/en/docs/http/ngx_http_proxy_module.html http://nginx.org/en/docs/http/ngx_http_proxy_module.html I don't see anything (e.g. under proxy_hide_header) to say it's sometimes not applied, and there doesn't appear to be any option to prevent this blind forwarding. I would never have expected the backend to receive invalid HTTP from NginX, but more importantly it's not uncommon for backends to send an extra header or two to tell NginX how to serve the response, with NginX removing those headers before serving. How do you even handle this properly? Checking for valid HTTP might not be enough, as you need to exactly match whatever NginX's idea of valid is, rather than matching the HTTP spec.

dheera 6y ago
From TFA: XTTP/1.1 500 Error Content-Type: text/html Secret-Header: secret-info Secret info, should not be visible! What the hell backend would respond with this? I just tested this with a simple hello world NodeJS backend: const express = require('express'); const app = express(); app.get('/', (req, res) => { res.send('Hello World!') }); app.listen(3000, () => { console.log('Listening'); }); And then tried the example: $ telnet localhost 3000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /? XTTP/1.1 Host: 127.0.0.1 Connection: closeHTTP/1.1 400 Bad Request Connection: close Connection closed by foreign host. Seems like the backend responded with a valid HTTP response even though my request was invalid. Of course that doesn't speak for all backend frameworks people would use, but it would occur to me that a well-designed backend would always speak proper HTTP even if the input isn't proper HTTP, and if the backend receives a bad HTTP request it should immediately send back a 400 (not 500) and never pass it on to the business logic. My brief test above seems to suggest that Express does indeed behave this way. Separately, configure your backends to not spit out secret info in production mode and you should not have to actually worry about this.
- throwaway894345 6y ago
  I thought one of the value propositions for Nginx is that it's battle-tested so that you don't have to depend so much on your framework for security?
  - dheera 6y ago
    If you're using it to serve static files sure, but the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend, which therefore should also be battle-tested. Nginx isn't going to protect you from SQL injection, routes that expose access to sensitive files, and so on. My typical configuration is to rely on Nginx for HTTPS and use an unencrypted HTTP backend listening only on 127.0.0.1 for non-static pages. I rely on the battle-tested encryption logic of Nginx but as far as invalid HTTP requests go that's all on the backend.
    - jlokier 6y ago
      > the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend Not really. That only applies for very simple proxy_pass configurations. NginX is used as more than just a simple proxy. The point of NginX directives like proxy_hide_header and proxy_intercept_errors it that you've configured NginX to do some extra processing. For example, I use proxy_intercept_errors so that backends can use certain error codes to instruct NginX to relay a request on to a different backend, after the first one has made a routing or versioning decision. I also use it in conjunction with X-Retry, X-Accel-Redirect to instruct NginX what to do next; should it replay the original request, or perform a new one given to it by the previous backend in the sequence, and relay state. This is a mechanism for backends to cooperate, and it also helps provide zero-downtime backend upgrades and routing to different hosts, invisible to the client. Those aren't robustness failures by the backends. They aren't lack of battle-testing. The backends are fine; but you still get the wrong result sent to the client in cases where NginX doesn't both to process the responses. The problem is this behaviour of NginX was not documented and there doesn't appear to be a way to tell it to do something else. It can be avoided with code in all the backends, but that code won't come from any specification. It can only be written after reading the NginX C source code to find under what conditions it does this, and have the backends go into a special mode when they detect it, responding with an error differently than they would to other bad request errors. Now I'm wondering if the headers X-Real-IP, X-Forwarded-For and X-Forwarded-Proto are reliably filtered out of the client request before being set or not set by the proxy itself. This is not something a backend can do for itself, it's intrinsic to proxying.

dheera 6y ago

From TFA: XTTP/1.1 500 Error Content-Type: text/html Secret-Header: secret-info Secret info, should not be visible! What the hell backend would respond with this? I just tested this with a simple hello world NodeJS backend: const express = require('express'); const app = express(); app.get('/', (req, res) => { res.send('Hello World!') }); app.listen(3000, () => { console.log('Listening'); }); And then tried the example: $ telnet localhost 3000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /? XTTP/1.1 Host: 127.0.0.1 Connection: closeHTTP/1.1 400 Bad Request Connection: close Connection closed by foreign host. Seems like the backend responded with a valid HTTP response even though my request was invalid. Of course that doesn't speak for all backend frameworks people would use, but it would occur to me that a well-designed backend would always speak proper HTTP even if the input isn't proper HTTP, and if the backend receives a bad HTTP request it should immediately send back a 400 (not 500) and never pass it on to the business logic. My brief test above seems to suggest that Express does indeed behave this way. Separately, configure your backends to not spit out secret info in production mode and you should not have to actually worry about this.

throwaway894345 6y ago
I thought one of the value propositions for Nginx is that it's battle-tested so that you don't have to depend so much on your framework for security?
- dheera 6y ago
  If you're using it to serve static files sure, but the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend, which therefore should also be battle-tested. Nginx isn't going to protect you from SQL injection, routes that expose access to sensitive files, and so on. My typical configuration is to rely on Nginx for HTTPS and use an unencrypted HTTP backend listening only on 127.0.0.1 for non-static pages. I rely on the battle-tested encryption logic of Nginx but as far as invalid HTTP requests go that's all on the backend.
  - jlokier 6y ago
    > the moment you use something like proxy_pass you're basically instructing Nginx to hand control to your backend Not really. That only applies for very simple proxy_pass configurations. NginX is used as more than just a simple proxy. The point of NginX directives like proxy_hide_header and proxy_intercept_errors it that you've configured NginX to do some extra processing. For example, I use proxy_intercept_errors so that backends can use certain error codes to instruct NginX to relay a request on to a different backend, after the first one has made a routing or versioning decision. I also use it in conjunction with X-Retry, X-Accel-Redirect to instruct NginX what to do next; should it replay the original request, or perform a new one given to it by the previous backend in the sequence, and relay state. This is a mechanism for backends to cooperate, and it also helps provide zero-downtime backend upgrades and routing to different hosts, invisible to the client. Those aren't robustness failures by the backends. They aren't lack of battle-testing. The backends are fine; but you still get the wrong result sent to the client in cases where NginX doesn't both to process the responses. The problem is this behaviour of NginX was not documented and there doesn't appear to be a way to tell it to do something else. It can be avoided with code in all the backends, but that code won't come from any specification. It can only be written after reading the NginX C source code to find under what conditions it does this, and have the backends go into a special mode when they detect it, responding with an error differently than they would to other bad request errors. Now I'm wondering if the headers X-Real-IP, X-Forwarded-For and X-Forwarded-Proto are reliably filtered out of the client request before being set or not set by the proxy itself. This is not something a backend can do for itself, it's intrinsic to proxying.