7 ms·

Nginx with dynamic upstreams

102 points by Tenzer 11y ago

corobo 11y ago
Thank you for this, it's a decent start for me personally. I was previously using the DNS director in Varnish 3.x to do this which was removed in 4.x. The longer time goes on the more risk it is to be running older software so this has been a great help. Oddly enough I'd never really considered nginx for the job despite using it to reverse proxy elsewhere. Sometimes you just need a poke in the right direction :) Now I just need to figure a way to specify *.internaldomain so that nginx resolves www.example.com.internaldomain - where www.example.com is grabbed from the requested Host: header
- Tenzer 11y ago
  It's good to hear it was of use to somebody :) I know you can use a map in Nginx to do what you ask for, as long as you have a list of domains already: http://nginx.org/en/docs/http/ngx_http_map_module.html#map http://nginx.org/en/docs/http/ngx_http_map_module.html#map. I can only imagine it also being possible to make fully dynamic, I just don't have a clear way of doing it in mind right now.
  - corobo 11y ago
    The sibling comment to this pointed out the $host variable. I imagine something along the line of proxy_pass http://$host.internaldomain; http://$host.internaldomain; or something on that line. I'll have to look into it further :)
  - sprobertson 11y ago
    Lua + Redis is a nice way to do this, I have a small project https://github.com/spro/simon https://github.com/spro/simon that does dynamic routing and load balancing based on Hostname -> IP:Port sets in Redis. Adding a new route is as simple as: sadd backends:hnapi.dev 192.168.0.42:10555
    - Tenzer 11y ago
      Neat! I want to get started on some Lua scripting as well, probably first to just get a bunch of metrics out of Nginx via StatsD. There's access to a whole bunch of numbers via Lua that you otherwise can't get out from the stub_status page.
- tombozi 11y ago
  You can use the $host variable See: http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host http://nginx.org/en/docs/http/ngx_http_core_module.html#var_... And: http://stackoverflow.com/a/15414811 http://stackoverflow.com/a/15414811 Always be careful with user input though.
  - corobo 11y ago
    well damn that's such a simple solution if that variable works in proxy_pass.. One of those "How did I miss that?" solutions Of course rigorous testing before going live, but it gives options. I love options :)
jpmattia 11y ago
Am I wrong for thinking that failure to honor the TTL in the first place is a bug?
- jon-wood 11y ago
  I find it particularly grating that the official way to deal with Nginx ignoring the advertised TTL is to pay $1500 per year, per server, for an Nginx Plus license.
  - MichaelGG 11y ago
    Yes. Some of the nginx plus features are very small, and there's even 3rd party modules that add such features. This particular case seems like an outright bug, with a hack workaround. Would they take a patch to fix it? It makes me think that they must be intentionally omitting features in order to make plus valuable. That seems rather cheesy. Obviously it's their code and right, but I think it shows how hard it is to profit off an open source program.
- lisper 11y ago
  It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never know when the routing rug is going to be yanked out from under you. This makes ELB useless for anything involving a persistent connection. No websockets for you!
  - twic 11y ago
    It's not quite as bad as that. AWS's short TTL means that they can change the IP whenever they like, not that they will. I would imagine that most of the time, long-lived connections to an ELB will be fine. If the ELB does get a new address, then yes, the connection will fail, the client will have to reconnect, and when it does, it will need to do a fresh address lookup. But since the connection is over a network, failure is a possibility regardless of what AWS does, and so clients need to be able to detect failure and reconnect anyway.
bmurphy1976 11y ago
I spent a long time fighting this battle with nginx. I eventually came to the conclusion that while it was technically possible, it just wasn't worth the trouble (and yes, I tried various alternatives such as Tengine and OpenResty). What I settled on was the following: Ngninx for SSL termination in front of an haproxy load balancer. I wrote a simple Python script (92 lines of code) that spits out a new haproxy configuration and gracefully reloads haproxy whenever the DNS changes. That has (after fixing a bug or two in my code) been FAR more reliable than any other solution to this problem we've tried. Bonus: haproxy is a superior load balancer with better runtime metrics and health checking options than nginx.
- culo 11y ago
  did you look at Kong? https://github.com/mashape/kong https://github.com/mashape/kong
- adamonduty 11y ago
  What part of the battle couldn't you solve? Doesn't the author's setting of the resolver directive work for you, or is there something I'm missing? Also, the author doesn't mention it, but you can also set a custom cache timeout with the valid=Xs option. http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver http://nginx.org/en/docs/http/ngx_http_core_module.html#reso...
  - bmurphy1976 11y ago
    It was not that reliable for us. We still had unexplained occurrences of IP caching and it's easy to fat finger the configuration and lose the resolving functionality. Anyway, as I mentioned haproxy has vastly superior load balancing capabilities. Nginx still has a lot of work to do to match haproxy in this area.
- thu 11y ago
  Any reason you don't use HAProxy for SSL termination too?
  - 11y ago
mattdeboard 11y ago
Can you not assign static IPs to ELB instances? This might be a stupid question...ELB is one of the older AWS resources that I've never really touched since Nginx is so powerful & easy to set up.
- cryowaffle 11y ago
  top 3 google responses for "elb static ip" say you can't, the truth may vary
- Tenzer 11y ago
  No, you can't. You can instead create your own EC2 server, give it a static IP (well, "Elastic IP") and then hope your own server doesn't go down. This is the reason why I prefer to use an ELB since I never seen stability issues with it.
Rapzid 11y ago
I've been working with nginx quite a bit recently for use as a reverse proxy in a containerized environment and have become quite disenfranchised with it. For some reason it's the most popular reverse proxy for this sort of stuff, however it's not particularly well suited for it. We have this issue. We also have the lack of active upstream checks and any sort of upstream status reports. Both are hidden behind the nginx Plus paywall which is absurdly expensive in a micro-architecture world. There are various patches, Patches!?, you can apply but the documentation isn't fantastic and this is an extra bit of hassle we don't really deserve.. I think the open internet deserves a better reverse proxy TBH.
- elithrar 11y ago
  Have you looked into HAProxy? Health checks, SSL termination and a reasonable configuration DSL: https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts https://www.digitalocean.com/community/tutorials/an-introduc...
  - Rapzid 11y ago
    I am acutely aware of HAProxy however the rest of the team is, no surprise I'm sure, familiar with nginx. Likely I'll end up doing the usual nginx->haproxy->backend chain again. HAPROXY appears to have an enterprise now(it may have had forever I just have not noticed) but it appears to be much more focused on value-add support and stack validation than hiding simple-to-implement stuff behind a paywall and open-core. This is more of a frustration with nginx which could be so much more for us if: * They were not purposely holding back features so they can be pay-walled * Re-compiling were not necessary to add functionality * More cohesive modification ecosystem I know I said "reverse proxy" which HAProxy is cleary the more superior, but I didn't mean to limit my statement to reverse proxies. nginx does have a lot of other functionality people rely on..
    - Tenzer 11y ago
      Dynamic modules are on the road map, according to this article: http://www.infoworld.com/article/2951849/web-services/nginx-to-hop-on-http-2-train.html http://www.infoworld.com/article/2951849/web-services/nginx-....
Cieplak 11y ago
Here's an implementation that uses nginx's X-accel-* headers to perform dynamic routing: https://github.com/bninja/rump https://github.com/bninja/rump These are what the routing rules look like: https://github.com/bninja/rump/blob/master/test/fixtures/settings/include.conf https://github.com/bninja/rump/blob/master/test/fixtures/set...
fica 11y ago
Kong [1] (nginx + lua) is planning to have this feature for free. This is the issue open for support for dynamic upstream [2]. [1] https://github.com/mashape/kong https://github.com/mashape/kong [2] https://github.com/Mashape/kong/issues/157 https://github.com/Mashape/kong/issues/157
- dekz 11y ago
  Unfortunately kong still has a hard dependency on Cassandra. https://github.com/Mashape/kong/issues/331 https://github.com/Mashape/kong/issues/331
  - sinzone 11y ago
    Before the end of the year Kong will support Postgres.
    - dekz 11y ago
      That is fantastic news!
brryant 11y ago
Another way around the rewrite is to simply proxy_pass with the $uri: https://gist.github.com/sansmischevia/cf425d5ffe09f824cb27 https://gist.github.com/sansmischevia/cf425d5ffe09f824cb27
- Tenzer 11y ago
  Sure, that's also what I cover in the post. The latter part about rewriting is only relevant when you want to have the service behind an ELB to be located at something else than "/".

102 points by Tenzer 11y ago

Thank you for this, it's a decent start for me personally. I was previously using the DNS director in Varnish 3.x to do this which was removed in 4.x. The longer time goes on the more risk it is to be running older software so this has been a great help. Oddly enough I'd never really considered nginx for the job despite using it to reverse proxy elsewhere. Sometimes you just need a poke in the right direction :) Now I just need to figure a way to specify *.internaldomain so that nginx resolves www.example.com.internaldomain - where www.example.com is grabbed from the requested Host: header

Tenzer 11y ago
It's good to hear it was of use to somebody :) I know you can use a map in Nginx to do what you ask for, as long as you have a list of domains already: http://nginx.org/en/docs/http/ngx_http_map_module.html#map http://nginx.org/en/docs/http/ngx_http_map_module.html#map. I can only imagine it also being possible to make fully dynamic, I just don't have a clear way of doing it in mind right now.
- corobo 11y ago
  The sibling comment to this pointed out the $host variable. I imagine something along the line of proxy_pass http://$host.internaldomain; http://$host.internaldomain; or something on that line. I'll have to look into it further :)
- sprobertson 11y ago
  Lua + Redis is a nice way to do this, I have a small project https://github.com/spro/simon https://github.com/spro/simon that does dynamic routing and load balancing based on Hostname -> IP:Port sets in Redis. Adding a new route is as simple as: sadd backends:hnapi.dev 192.168.0.42:10555
  - Tenzer 11y ago
    Neat! I want to get started on some Lua scripting as well, probably first to just get a bunch of metrics out of Nginx via StatsD. There's access to a whole bunch of numbers via Lua that you otherwise can't get out from the stub_status page.
tombozi 11y ago
You can use the $host variable See: http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host http://nginx.org/en/docs/http/ngx_http_core_module.html#var_... And: http://stackoverflow.com/a/15414811 http://stackoverflow.com/a/15414811 Always be careful with user input though.
- corobo 11y ago
  well damn that's such a simple solution if that variable works in proxy_pass.. One of those "How did I miss that?" solutions Of course rigorous testing before going live, but it gives options. I love options :)

jpmattia 11y ago

Am I wrong for thinking that failure to honor the TTL in the first place is a bug?

jon-wood 11y ago
I find it particularly grating that the official way to deal with Nginx ignoring the advertised TTL is to pay $1500 per year, per server, for an Nginx Plus license.
- MichaelGG 11y ago
  Yes. Some of the nginx plus features are very small, and there's even 3rd party modules that add such features. This particular case seems like an outright bug, with a hack workaround. Would they take a patch to fix it? It makes me think that they must be intentionally omitting features in order to make plus valuable. That seems rather cheesy. Obviously it's their code and right, but I think it shows how hard it is to profit off an open source program.
lisper 11y ago
It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually unintentional, but this is a deliberate reinvention of a wheel (IP address caching) that doesn't need to be reinvented. But this is not just nginx's fault. It's also Bad Design on the part of AWS because switching IP addresses this way means you can't keep a TCP socket open to an ELB machine for more than 60 seconds at a time because you never know when the routing rug is going to be yanked out from under you. This makes ELB useless for anything involving a persistent connection. No websockets for you!
- twic 11y ago
  It's not quite as bad as that. AWS's short TTL means that they can change the IP whenever they like, not that they will. I would imagine that most of the time, long-lived connections to an ELB will be fine. If the ELB does get a new address, then yes, the connection will fail, the client will have to reconnect, and when it does, it will need to do a fresh address lookup. But since the connection is over a network, failure is a possibility regardless of what AWS does, and so clients need to be able to detect failure and reconnect anyway.

bmurphy1976 11y ago

I spent a long time fighting this battle with nginx. I eventually came to the conclusion that while it was technically possible, it just wasn't worth the trouble (and yes, I tried various alternatives such as Tengine and OpenResty). What I settled on was the following: Ngninx for SSL termination in front of an haproxy load balancer. I wrote a simple Python script (92 lines of code) that spits out a new haproxy configuration and gracefully reloads haproxy whenever the DNS changes. That has (after fixing a bug or two in my code) been FAR more reliable than any other solution to this problem we've tried. Bonus: haproxy is a superior load balancer with better runtime metrics and health checking options than nginx.

culo 11y ago
did you look at Kong? https://github.com/mashape/kong https://github.com/mashape/kong
adamonduty 11y ago
What part of the battle couldn't you solve? Doesn't the author's setting of the resolver directive work for you, or is there something I'm missing? Also, the author doesn't mention it, but you can also set a custom cache timeout with the valid=Xs option. http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver http://nginx.org/en/docs/http/ngx_http_core_module.html#reso...
- bmurphy1976 11y ago
  It was not that reliable for us. We still had unexplained occurrences of IP caching and it's easy to fat finger the configuration and lose the resolving functionality. Anyway, as I mentioned haproxy has vastly superior load balancing capabilities. Nginx still has a lot of work to do to match haproxy in this area.
thu 11y ago
Any reason you don't use HAProxy for SSL termination too?
- 11y ago

mattdeboard 11y ago

Can you not assign static IPs to ELB instances? This might be a stupid question...ELB is one of the older AWS resources that I've never really touched since Nginx is so powerful & easy to set up.

cryowaffle 11y ago
top 3 google responses for "elb static ip" say you can't, the truth may vary
Tenzer 11y ago
No, you can't. You can instead create your own EC2 server, give it a static IP (well, "Elastic IP") and then hope your own server doesn't go down. This is the reason why I prefer to use an ELB since I never seen stability issues with it.

Rapzid 11y ago

I've been working with nginx quite a bit recently for use as a reverse proxy in a containerized environment and have become quite disenfranchised with it. For some reason it's the most popular reverse proxy for this sort of stuff, however it's not particularly well suited for it. We have this issue. We also have the lack of active upstream checks and any sort of upstream status reports. Both are hidden behind the nginx Plus paywall which is absurdly expensive in a micro-architecture world. There are various patches, Patches!?, you can apply but the documentation isn't fantastic and this is an extra bit of hassle we don't really deserve.. I think the open internet deserves a better reverse proxy TBH.

elithrar 11y ago
Have you looked into HAProxy? Health checks, SSL termination and a reasonable configuration DSL: https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts https://www.digitalocean.com/community/tutorials/an-introduc...
- Rapzid 11y ago
  I am acutely aware of HAProxy however the rest of the team is, no surprise I'm sure, familiar with nginx. Likely I'll end up doing the usual nginx->haproxy->backend chain again. HAPROXY appears to have an enterprise now(it may have had forever I just have not noticed) but it appears to be much more focused on value-add support and stack validation than hiding simple-to-implement stuff behind a paywall and open-core. This is more of a frustration with nginx which could be so much more for us if: * They were not purposely holding back features so they can be pay-walled * Re-compiling were not necessary to add functionality * More cohesive modification ecosystem I know I said "reverse proxy" which HAProxy is cleary the more superior, but I didn't mean to limit my statement to reverse proxies. nginx does have a lot of other functionality people rely on..
  - Tenzer 11y ago
    Dynamic modules are on the road map, according to this article: http://www.infoworld.com/article/2951849/web-services/nginx-to-hop-on-http-2-train.html http://www.infoworld.com/article/2951849/web-services/nginx-....

Cieplak 11y ago

Here's an implementation that uses nginx's X-accel-* headers to perform dynamic routing: https://github.com/bninja/rump https://github.com/bninja/rump These are what the routing rules look like: https://github.com/bninja/rump/blob/master/test/fixtures/settings/include.conf https://github.com/bninja/rump/blob/master/test/fixtures/set...

fica 11y ago

Kong [1] (nginx + lua) is planning to have this feature for free. This is the issue open for support for dynamic upstream [2]. [1] https://github.com/mashape/kong https://github.com/mashape/kong [2] https://github.com/Mashape/kong/issues/157 https://github.com/Mashape/kong/issues/157

dekz 11y ago
Unfortunately kong still has a hard dependency on Cassandra. https://github.com/Mashape/kong/issues/331 https://github.com/Mashape/kong/issues/331
- sinzone 11y ago
  Before the end of the year Kong will support Postgres.
  - dekz 11y ago
    That is fantastic news!

brryant 11y ago

Another way around the rewrite is to simply proxy_pass with the $uri: https://gist.github.com/sansmischevia/cf425d5ffe09f824cb27 https://gist.github.com/sansmischevia/cf425d5ffe09f824cb27

Tenzer 11y ago
Sure, that's also what I cover in the post. The latter part about rewriting is only relevant when you want to have the service behind an ELB to be located at something else than "/".