9 ms·

Taobao's branch of Nginx

121 points by ujeezy 14y ago

mercurial 14y ago
Anyone outside of Taobao using this in production? I'd be really interested in a reliable version of nginx supporting dynamic loading of modules, recompiling nginx everytime you need a custom module quickly loses its charm.
- pyre 14y ago
  > recompiling nginx everytime you need a custom > module quickly loses its charm How often are you adding/removing modules?
  - handelaar 14y ago
    Once, ever, is more than anybody wants. [I'm a huge nginx fan myself but recall that this is coming to the main project soonish.]
    - ck2 14y ago
      Actually, nginx recompiles very quickly from the couple times I've done it (at least compared to apache/php). But I bet for developing modules, loading standalone is a huge help.
      - sbarre 14y ago
        I think this sub-thread may be missing the point that on a production server you may not have installed the whole toolchain required to re-compile your webserver so dynamic module support is actually kind of a bigger deal than you might think.
        adestefan 14y ago
        Where do you compile your modules and how is deploying a loadable module any different than deploying a new nginx binary?
        mercurial 14y ago
        Well, the point is, Apache comes with enough modules that I never had any need to actually go and compile a custom one. And since they can be loaded at runtime, in terms of memory you only pay for what you use. I haven't used nginx often, but the default Debian nginx doesn't come with many things included, I find.
        pyre 14y ago
        Well, the idea solution is that you don't compile things directly on your web server. There's no reason you can't compile it on a workstation and package it for deployment to production. Consider what you would be doing if you had a cluster of web servers. Would you compile manually on each server, or create a package (rpm/deb/tgz/etc) and deploy?
        mercurial 14y ago
        No. But as I pointed out above, Apache comes with 67 modules out-of-the box on Debian, which means you're much less likely to need a custom package, as opposed to the (relative) hassle with nginx. This is why I very much look forward to seeing nginx with reloadable modules.
    - beagle3 14y ago
      nginx compiles quickly enough that a script for "service nginx reload" could recompile and replace the binary, and it will only be a little slower than "service apache reload" will be. It's like a sausage, you can enjoy eating it without seeing how it's made.
      - jlgreco 14y ago
        Can Apache or other httpd's do the same live upgrading trick that Nginx has? Seems like a really wild feature.
        beagle3 14y ago
        I don't think any other popular http server support this. If there's a web server written in Erlang, you can be sure that is available (it's a core Erlang feature). But apache can let you load a new .so without recompiling (or even restarting) the main server, which, for most intents and purposes, gives you equivalent functionality.
- pestaa 14y ago
  Perhaps dynamic loading is useful to speed up tinkering with development and staging, but for production I'll take a system with the least moving parts any day.
- tszming 14y ago
  Compiling nginx is generally very fast (especially if you compare to other server). Also, you can upgrade nginx binary without any downtime, see: http://wiki.nginx.org/CommandLine http://wiki.nginx.org/CommandLine
  - mercurial 14y ago
    If you compare to another sever like Apache which supports dynamic modules, adding a single line in a configuration file (or at worst dropping a .so in a folder, followed by adding a line in a configuration file) is much less work than retrieving gcc and friends and rebuilding a .deb.
    - beagle3 14y ago
      Dropping a new nginx binary and live-migrating to it takes as much time as doing the same on Apahce with a new .so Have you ever actually used nginx? I compile it from source with much-less-than-everything for security reasons (e.g., no proxy, no email, no ssi, no ssl, not any of the other ten modules that come by default), and in the 3 times that I needed a module later, it was a quick ./configure;make;make install;kill -SIG2 (or whatever the signal is to switch to a new version), which was not more painful than I remember from my apache days of 5 years ago. (Has apache changed since then?)
      - mercurial 14y ago
        I count 67 different modules in /etc/apache2/mods-available. Which means you rarely a new .so. On the other hand, Nginx is much more limited in functionalities out-of-the-box. How do you handle deployment of your custom nginx on several boxes?
        beagle3 14y ago
        The same way I handle deployment of everything else. The binary is just one file (/usr/sbin/nginx). Distributing config and restarting processes is the same as with apache. (My recent nginx deployment have 2 machines, so I do it manually, but in the past, I had a homegrown solution that would pull files from version control, and if they changed, asked the binary to restart)
- yaoweibin 14y ago
  Many large Chinese video sites have used Tengine: tudou.com, 56.com, pptv.com etc.
- ra 14y ago
  Yes, we use it in production. However our requirements are for the logging enhancements rather than the dynamic modules. In our experience it's every bit as reliable as mainline nginx.
- Vidart 14y ago
  There are a lot of problems in Tengine implementation: http://mailman.nginx.org/pipermail/nginx/2012-September/035405.html http://mailman.nginx.org/pipermail/nginx/2012-September/0354... http://mailman.nginx.org/pipermail/nginx/2012-September/035416.html http://mailman.nginx.org/pipermail/nginx/2012-September/0354...
mnbbrown 14y ago
There was thread on tengine when it was first made open source: http://news.ycombinator.com/item?id=3645055 http://news.ycombinator.com/item?id=3645055
ck2 14y ago
Hmm nginx cannot natively do piped logging? I like the idea of combined javascript and stylesheets but I'd rather force the combining from the server-side, not a special client-side request. ie. location /style.css { root /css; combine style1.css+style2.css+style3.css; } Ohhh... agentzh wrote the fork? The one who made many popular modules?
- tszming 14y ago
  >> nginx cannot natively do piped logging? "No, nginx will not support logging to a pipe since this is simlpy waste of CPU." - Igor Sysoev (http://markmail.org/message/4bsjfjpg2kgsjhph http://markmail.org/message/4bsjfjpg2kgsjhph)
  - ck2 14y ago
    Interesting, so to rotate nginx logs do you have to restart nginx? I always though piped logging was required to rotate logs without restart. I love the graceful restart and even the replace the executable on the fly nginx has.
    - tszming 14y ago
      You can send a USR1 signal to nginx. http://wiki.nginx.org/LogRotation http://wiki.nginx.org/LogRotation
- yaoweibin 14y ago
  No really, agentzh and chaoslawful developed the nginx_lua module. It's included in Tengine. Tengine is maintained by many other developers (in Chinese): https://github.com/taobao/tengine/wiki/%E7%A4%BE%E5%8C%BA%E5%8C%96%E5%8F%91%E5%B1%95 https://github.com/taobao/tengine/wiki/%E7%A4%BE%E5%8C%BA%E5...
- ck2 14y ago
  Looks like concat is available as it's own module: https://github.com/taobao/nginx-http-concat https://github.com/taobao/nginx-http-concat and another attempt by someone else here: https://github.com/lindsayevans/nginx-static-concat https://github.com/lindsayevans/nginx-static-concat But both of them rely on or accept client-side concat, I'd rather just do it server-side only for security/control.
tzury 14y ago
I think the openresty http://openresty.org/ http://openresty.org/ is the one originated this as the guys who maintain openresty are ex/current Taobao's employees.
yaoweibin 14y ago
Tengine is developed on github. Anyone who is interested in can follow it. URL: https://github.com/taobao/tengine https://github.com/taobao/tengine
happywolf 14y ago
Just curious if Taobao is using this stock branch in production servers. Any proprietary modules loaded?
gadders 14y ago
Slightly off-topic, but are there any other major open-source projects from China? I was wondering whether a) there aren't many (yet) or b) they're mostly publicised in Chinese so off the radar for English speakers.
- yaoweibin 14y ago
  Have you heard of the LVS? (http://www.linuxvirtualserver.org/ http://www.linuxvirtualserver.org/). We are using it for load balance before Tengine. It's steady and high performance, even better than the commercial load balancer.

121 points by ujeezy 14y ago

Anyone outside of Taobao using this in production? I'd be really interested in a reliable version of nginx supporting dynamic loading of modules, recompiling nginx everytime you need a custom module quickly loses its charm.

pyre 14y ago
> recompiling nginx everytime you need a custom > module quickly loses its charm How often are you adding/removing modules?
- handelaar 14y ago
  Once, ever, is more than anybody wants. [I'm a huge nginx fan myself but recall that this is coming to the main project soonish.]
  - ck2 14y ago
    Actually, nginx recompiles very quickly from the couple times I've done it (at least compared to apache/php). But I bet for developing modules, loading standalone is a huge help.
    - sbarre 14y ago
      I think this sub-thread may be missing the point that on a production server you may not have installed the whole toolchain required to re-compile your webserver so dynamic module support is actually kind of a bigger deal than you might think.
      - adestefan 14y ago
        Where do you compile your modules and how is deploying a loadable module any different than deploying a new nginx binary?
        mercurial 14y ago
        Well, the point is, Apache comes with enough modules that I never had any need to actually go and compile a custom one. And since they can be loaded at runtime, in terms of memory you only pay for what you use. I haven't used nginx often, but the default Debian nginx doesn't come with many things included, I find.
      - pyre 14y ago
        Well, the idea solution is that you don't compile things directly on your web server. There's no reason you can't compile it on a workstation and package it for deployment to production. Consider what you would be doing if you had a cluster of web servers. Would you compile manually on each server, or create a package (rpm/deb/tgz/etc) and deploy?
        mercurial 14y ago
        No. But as I pointed out above, Apache comes with 67 modules out-of-the box on Debian, which means you're much less likely to need a custom package, as opposed to the (relative) hassle with nginx. This is why I very much look forward to seeing nginx with reloadable modules.
  - beagle3 14y ago
    nginx compiles quickly enough that a script for "service nginx reload" could recompile and replace the binary, and it will only be a little slower than "service apache reload" will be. It's like a sausage, you can enjoy eating it without seeing how it's made.
    - jlgreco 14y ago
      Can Apache or other httpd's do the same live upgrading trick that Nginx has? Seems like a really wild feature.
      - beagle3 14y ago
        I don't think any other popular http server support this. If there's a web server written in Erlang, you can be sure that is available (it's a core Erlang feature). But apache can let you load a new .so without recompiling (or even restarting) the main server, which, for most intents and purposes, gives you equivalent functionality.
pestaa 14y ago
Perhaps dynamic loading is useful to speed up tinkering with development and staging, but for production I'll take a system with the least moving parts any day.
tszming 14y ago
Compiling nginx is generally very fast (especially if you compare to other server). Also, you can upgrade nginx binary without any downtime, see: http://wiki.nginx.org/CommandLine http://wiki.nginx.org/CommandLine
- mercurial 14y ago
  If you compare to another sever like Apache which supports dynamic modules, adding a single line in a configuration file (or at worst dropping a .so in a folder, followed by adding a line in a configuration file) is much less work than retrieving gcc and friends and rebuilding a .deb.
  - beagle3 14y ago
    Dropping a new nginx binary and live-migrating to it takes as much time as doing the same on Apahce with a new .so Have you ever actually used nginx? I compile it from source with much-less-than-everything for security reasons (e.g., no proxy, no email, no ssi, no ssl, not any of the other ten modules that come by default), and in the 3 times that I needed a module later, it was a quick ./configure;make;make install;kill -SIG2 (or whatever the signal is to switch to a new version), which was not more painful than I remember from my apache days of 5 years ago. (Has apache changed since then?)
    - mercurial 14y ago
      I count 67 different modules in /etc/apache2/mods-available. Which means you rarely a new .so. On the other hand, Nginx is much more limited in functionalities out-of-the-box. How do you handle deployment of your custom nginx on several boxes?
      - beagle3 14y ago
        The same way I handle deployment of everything else. The binary is just one file (/usr/sbin/nginx). Distributing config and restarting processes is the same as with apache. (My recent nginx deployment have 2 machines, so I do it manually, but in the past, I had a homegrown solution that would pull files from version control, and if they changed, asked the binary to restart)
yaoweibin 14y ago
Many large Chinese video sites have used Tengine: tudou.com, 56.com, pptv.com etc.
ra 14y ago
Yes, we use it in production. However our requirements are for the logging enhancements rather than the dynamic modules. In our experience it's every bit as reliable as mainline nginx.
Vidart 14y ago
There are a lot of problems in Tengine implementation: http://mailman.nginx.org/pipermail/nginx/2012-September/035405.html http://mailman.nginx.org/pipermail/nginx/2012-September/0354... http://mailman.nginx.org/pipermail/nginx/2012-September/035416.html http://mailman.nginx.org/pipermail/nginx/2012-September/0354...

mnbbrown 14y ago

There was thread on tengine when it was first made open source: http://news.ycombinator.com/item?id=3645055 http://news.ycombinator.com/item?id=3645055

ck2 14y ago

Hmm nginx cannot natively do piped logging? I like the idea of combined javascript and stylesheets but I'd rather force the combining from the server-side, not a special client-side request. ie. location /style.css { root /css; combine style1.css+style2.css+style3.css; } Ohhh... agentzh wrote the fork? The one who made many popular modules?

tszming 14y ago
>> nginx cannot natively do piped logging? "No, nginx will not support logging to a pipe since this is simlpy waste of CPU." - Igor Sysoev (http://markmail.org/message/4bsjfjpg2kgsjhph http://markmail.org/message/4bsjfjpg2kgsjhph)
- ck2 14y ago
  Interesting, so to rotate nginx logs do you have to restart nginx? I always though piped logging was required to rotate logs without restart. I love the graceful restart and even the replace the executable on the fly nginx has.
  - tszming 14y ago
    You can send a USR1 signal to nginx. http://wiki.nginx.org/LogRotation http://wiki.nginx.org/LogRotation
yaoweibin 14y ago
No really, agentzh and chaoslawful developed the nginx_lua module. It's included in Tengine. Tengine is maintained by many other developers (in Chinese): https://github.com/taobao/tengine/wiki/%E7%A4%BE%E5%8C%BA%E5%8C%96%E5%8F%91%E5%B1%95 https://github.com/taobao/tengine/wiki/%E7%A4%BE%E5%8C%BA%E5...
ck2 14y ago
Looks like concat is available as it's own module: https://github.com/taobao/nginx-http-concat https://github.com/taobao/nginx-http-concat and another attempt by someone else here: https://github.com/lindsayevans/nginx-static-concat https://github.com/lindsayevans/nginx-static-concat But both of them rely on or accept client-side concat, I'd rather just do it server-side only for security/control.

tzury 14y ago

I think the openresty http://openresty.org/ http://openresty.org/ is the one originated this as the guys who maintain openresty are ex/current Taobao's employees.

yaoweibin 14y ago

Tengine is developed on github. Anyone who is interested in can follow it. URL: https://github.com/taobao/tengine https://github.com/taobao/tengine

happywolf 14y ago

Just curious if Taobao is using this stock branch in production servers. Any proprietary modules loaded?

gadders 14y ago

Slightly off-topic, but are there any other major open-source projects from China? I was wondering whether a) there aren't many (yet) or b) they're mostly publicised in Chinese so off the radar for English speakers.

yaoweibin 14y ago
Have you heard of the LVS? (http://www.linuxvirtualserver.org/ http://www.linuxvirtualserver.org/). We are using it for load balance before Tengine. It's steady and high performance, even better than the commercial load balancer.