4 ms·

I hit this problem too - we had >100k lines of nginx config. My answer was to write a tool to simulate what happened to a request, logging the decisions made ea

I hit this problem too - we had >100k lines of nginx config. My answer was to write a tool to simulate what happened to a request, logging the decisions made each step. It was limited in what config I'd programmed it to understand but did most of what we needed (rewrite rules, access control, proxy_pass ...). I also intended it to be used for regression testing - you could set up a yaml file with requests and specify eg, return this proxied response if the proxied request headers match X. Other efforts I'd seen at automated testing of nginx config relied on testing in a VM, but of necessity used different listen addresses etc. They looked complex to set up and weren't even testing the config as shipped...that's what I was aiming for. Since then we've moved away from that config, and I'm in a different role...keep meaning to see if we can open source that code. However, nginx being such a moving target, with lua config and such, I was never sure if it would find an audience.

patmcguire 9y ago
Docker with docker compose is good for this. Nginx can listen on ports 80 and 443, other containers on the same network can reach it on those ports. I have a setup which takes env vars, so it's not quite the same as shipped, but I can test the whole process of getting a cert from LetsEncrypt, for example.
- bazzargh 9y ago
  (late reply but...) we did actually look at that; someone had a blog post about using docker for testing nginx when we did this (~3yr ago). You hit a number of problems. Our nginx listened on multiple addresses and ports, which weren't equivalent - external https/http arrived on one address, internal traffic from across our clusters arrived on another, with different handling. So, we have to have docker listen on multiple addresses too, or in multiple instances. Then we have the outgoing requests, proxied to various haproxy, varnish instances, themselves on various ports. You can get round this by messing with dnsmasq and routing on your docker host...or avoid that by reorganizing all of the outgoing config so that they talk to a stub server with canned test responses. But, we had thousands of lines of handwritten nginx config as well as tens of thousands of lines of generated config...this was going to be a major effort. And even with all that effort, it didn't give us enough information. You'd try a request on some config and scratch your head to figure out why it didn't end up being processed in the right place. Run it in the simulator though, and you'd find the exact line that had rewritten your request so it didn't reach the line of config you thought was going to kick in... Probably the best approach would have been to patch nginx itself to add the logging we wanted, and have it handle all of those external addresses differently. The challenge there would have been getting the same depth of traceability with nginx doing things like converting rewrite rules to bytecode. But, you could probably get 80% of the way there and avoid bugs from our system not accurately re-implementing their algorithms.

patmcguire 9y ago

Docker with docker compose is good for this. Nginx can listen on ports 80 and 443, other containers on the same network can reach it on those ports. I have a setup which takes env vars, so it's not quite the same as shipped, but I can test the whole process of getting a cert from LetsEncrypt, for example.

bazzargh 9y ago
(late reply but...) we did actually look at that; someone had a blog post about using docker for testing nginx when we did this (~3yr ago). You hit a number of problems. Our nginx listened on multiple addresses and ports, which weren't equivalent - external https/http arrived on one address, internal traffic from across our clusters arrived on another, with different handling. So, we have to have docker listen on multiple addresses too, or in multiple instances. Then we have the outgoing requests, proxied to various haproxy, varnish instances, themselves on various ports. You can get round this by messing with dnsmasq and routing on your docker host...or avoid that by reorganizing all of the outgoing config so that they talk to a stub server with canned test responses. But, we had thousands of lines of handwritten nginx config as well as tens of thousands of lines of generated config...this was going to be a major effort. And even with all that effort, it didn't give us enough information. You'd try a request on some config and scratch your head to figure out why it didn't end up being processed in the right place. Run it in the simulator though, and you'd find the exact line that had rewritten your request so it didn't reach the line of config you thought was going to kick in... Probably the best approach would have been to patch nginx itself to add the logging we wanted, and have it handle all of those external addresses differently. The challenge there would have been getting the same depth of traceability with nginx doing things like converting rewrite rules to bytecode. But, you could probably get 80% of the way there and avoid bugs from our system not accurately re-implementing their algorithms.