5 ms·

Apache hacker here. AMA.

powertower 16y ago
What type of setup do you use to analyze, debug, and follow the code? Any tips on getting started with module development?
- brianm 16y ago
  Another apache hacker answering :-) I use emacs and gdb to muck around the code. Apache has a lot of backwards compat warts with regard to which header files define things, unfortunately, but grep (or ack) and ctags will get you around. The best way (by far) to learn about module hackery is with Nick Kew's book, http://www.amazon.com/Apache-Modules-Book-Application-Development/dp/0132409674 http://www.amazon.com/Apache-Modules-Book-Application-Develo...
- bnoordhuis 16y ago
  Another vote for Nick Kew's book. He is a regular poster on the modules-dev mailing list[1], by the way. `httpd -X` and `gdb httpd $(pidof httpd)` are a great help when debugging. [1] http://mail-archives.apache.org/mod_mbox/httpd-modules-dev/ http://mail-archives.apache.org/mod_mbox/httpd-modules-dev/
kristofferR 16y ago
Why is apache so slow compared to Nginx, Cherokee, Lighttpd and many other web server softwares?
- powertower 16y ago
  Mostly due to two reasons: 0) People don't know how to use PHP with it: module or fastcgi; and which MPM to use here. 1) People don't know how to configure the MPM setting to allow Apache full use of server resources (without under-utilizing or over-utilizing). 2) Apache is a full / well-rounded web server application... It's not a specialty server (ex: static content mostly) that can exclude this and that and only provide 1 feature to excel at. Other servers are stripped down compared to Apache. * Not the AMA guy (nor an Apache hacker guy), buy have a little known WAMP distribution (called WampDeveloper, used to be Web.Developer Server Suite) since 2003 with over 250,000 unique ip downloads between 2003-2006 (have stopped counting), and have worked on 1000s of issues for users and clients since the start.
  - calloc 16y ago
    The only reason I still run Apache is because a lot of PHP stuff requires mod_rewrite to create pretty URL's and I haven't had the time to rewrite them using LUA so I can use Lighttpd. But on the same server, using PHP-FPM with FastCGI on both lighttpd and Apache I am serving almost 4x the requests that Apache is serving with less memory overhead. Switching the sites that are currently running on Lighttpd back to Apache as I attempted to do not to long ago to only have to maintain a single server made Apache die a miserable fast death. It just could not keep up, I've had friends look over my config (datacenter techs, help people scale their stuff, porn mostly) and they said it looked fine. Apache was the limiting factor here. I moved from prefork to threaded, that helped a little, but not much. Apache was just using a lot of memory, and overall did not provide the performance I wanted. Switched back to lighttpd, load on the server went down, and the websites were responsive as before.
    - powertower 16y ago
      "made Apache die a miserable fast death. It just could not keep up, I've had friends look over my config (datacenter techs, help people scale their stuff, porn mostly) and they said it looked fine." Apache crashing under load is indicative of seriously bad MPM settings: specifically the # of processes / threads allowed to be used (and some other settings such as KeepAlive timeout, how php is being used, etc). With a properly configured MPM, anything coming over the process / thread limit goes into a backlog queue which is 511 entries long. Anything over this just gets dropped. To iterate, Apache does not crash under load (too many requests), it crashes due to too many processes / threads (an MPM setting) being used and sometimes due to leaking modules such as mod_python/ruby.
- bnoordhuis 16y ago
  Different design goals. Apache is meant to be robust, extensible and portable. * Robust: that's reflected in its internal API that makes it near impossible to leak resources. * Extensible: witness the gazillion modules out there. * Portable: compiles and runs on very exotic or outdated systems. SCO, IRIX, Digital UNIX, VMS, the list goes on. nginx and such were designed from the ground up with performance in mind - and with success - but the trade-off is a lack of portability and an API that is much harder to program to.
  - deleted 16y ago
    [deleted]
  - kujawa 16y ago
    Well gee. It's been a while since I've used Digital UNIX, Irix, SCO, or VMS. Because they've become totally irrelevant. Seems like a waste of resources when there are more pressing things to do than worry about the 5 people who (a) use VMS and (b) demand a bleeding-edge Apache.
    - lambda 16y ago
      No need to be so acerbic. They are providing a valuable service to you, for free. Apache is older, much more ubiquitous, supports more modules, has a more familiar setup and configuration system for many people, runs on more platforms, etc. It has more legacy issues, which are why it's so popular, but also why it moves slower. Nginx is much newer, doesn't have the same legacy issues, and so could afford to focus on speed. There's a reason that Apache is installed on just about every random web host you can find, and has a module for every language or environment you need to deel with, while Nginx is a bit more of a specialty web server, usually used for dedicated sites that can spend the time to tune carefully for the highest performance. They both have their place. It's great that Apache is still innovating moving towards loadable MPMs as well as adding an evented MPM. But it's not a bad thing that they're moving slower than a new server like Nginx; there's room for more than one great free web server in the world.
- getsat 16y ago
  Preforked and threaded blocking IO consumes far more resources than event-driven non-blocking network IO. Process/thread stacks consume a significant amount of memory. This is why nginx ruins Apache at almost anything at scale (especially reverse proxying). Apache falls over while consuming grandiose amounts of memory in the low millions of connections per day while nginx can easily handle 10mm connections per day on a single core machine with 256mb ram. It's ridiculous. On the other hand, for long-lived connections, you're better off using blocking IO since it's more efficient. Most web traffic is not long-lived, though. See Paul's interesting article on NIO vs IO performance: http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-is-not-faster-than.html http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-... http://www.mailinator.com/tymaPaulMultithreaded.pdf http://www.mailinator.com/tymaPaulMultithreaded.pdf And don't forget about Slowloris. If you see your competition using Apache, you really don't need to worry. :) Disclaimer: I have nothing personal against Apache and used it for six years or so. I've just moved on to better software. If it works for you, that's great.
  - rb2k_ 16y ago
    Will those two things included in the new apache change this?: Event MPM: The Event MPM is no longer experimental but is rather fully supported. Asynchronous support: Better support for asynchronous read/write for supporting MPMs and platforms.
    - getsat 16y ago
      Yes, I believe so. I'm really excited that the event MPM is no longer experimental. I had tried it way back in the day, but mod_php could not be safely used with it as mod_php was (is?) not thread-safe. I wasn't aware of FCGI at the time.
  - bad_user 16y ago
    I still use Apache because I trust it; and because the ease with which I can configure a Python server (i.e. mod_wsgi), a Ruby server (passenger) and a Perl server, while being able to make use of the same modules I've been using for years. Also, a properly configured Varnish placed in front of Apache ruins Nginx at almost anything at scale. I've seen it. Most web traffic is not long-lived, though. But most web traffic is blocking. Going NIO requires caching - which is a huge penalty and a PITA; and doing it while not having actual users doesn't make sense.
mthomas 16y ago
Are there any significant barriers to use the event MPM as it stands in the 2.2. Does it fall back to worker when its used under SSL?
- bnoordhuis 16y ago
  No. The main problem the event MPM addresses is HTTP keep-alive. With pre-fork and threaded, every connection takes up a process or thread. This sucks for keep-alive because it can take a while before the client issues the next request. The event MPM puts that idle connection in a kqueue/epoll/etc. pollset and recycles the process/thread for another request. It's a conceptually simple change but it has some profound performance implications.
- brianm 16y ago
  In 2.2, the event mpm doesn't support ssl, but in 2.3/2.4 it does support ssl. In fact, IIRC, if you are using apche as a reverse proxy (mod_proxy) the event mpm should become the standard as it will detach the worker thread both for keepalive on the front end, and when waiting on response from the backend.
- saurik 16y ago
  As a single datapoint, I run a site that handles many billions (this is not rhetorical or an exaggeration) of hits per month using mpm_event: I consider it the only sane/stable way of configuring Apache.
  - patio11 16y ago
    As I found out to my sadness, the only sane configuration is not the one which ships with apt-get apache2. I like many things about Nginx. The fact that it is a production-capable web server out of the box, for example. I know that is probably not a high-priority design goal for Apache for historical reasons, but it seems a very sensible default in 2011.
    - saurik 16y ago
      For the record, "apt-get install apache2-mpm-event" is all you need to switch to mpm_event: I believe it isn't supported on all platforms, and, as a conservative Debian-based distribution using apt-get, I'd prefer "predictable behavior across all of our targets" over "highest performance configuration on any given system". (I also believe there are some corner cases where mpm_event may burn you on 2.2, and both Apache and Debian are going to play it safe there; but, when I looked into this in detail a forever or so ago, I determined that whatever issues they were didn't apply to my setup. As an example: if SSL were an issue, I doubt I would ever use Apache to directly serve SSL anyway, as that's what SSL accelerators are for.) That said, I still wouldn't leave the defaults in place with regards to "number of servers / threads"; although it isn't like nginx doesn't need the same configuration: the default value of worker_{processes,connections} is almost certainly inappropriate for your specific setup. I also use nginx (as a load balancer), and I have those values at 64/10096, up from the Debian-default of 1/1024. The important thing, to me, is what a technology makes possible, not how well it is configured out of the box. Example: it is more damning to me that nginx only does HTTP/1.0 to upstream servers it is proxying for, a reasonably fundamental limitation of the codebase, than any transgression they could make in their default configuration. Seriously: production is not about "out of the box", it never was, and it likely is never going to be. If you are trying to run a production server using "out of the box" defaults you are going to be forever disappointed by the performance and functionality of the offerings. To make a minor modification to a statement I've made before (http://news.ycombinator.com/item?id=2145967 http://news.ycombinator.com/item?id=2145967 is the reference) about database servers: We (as a civilization) simply do not have the science and theory yet to make the practicalities of setting up and maintaining anything at this level of complexity a totally seamless and simple process with well-understood performance characteristics unless you constrain absolutely every single variable.
pwpwp 16y ago
Anything new on Waka?
IgorPartola 16y ago
Will apache support Google's SPDY at some point? Also, how much is having to support all the legacy platforms slowing down development of new features?
- powertower 16y ago
  http://code.google.com/p/mod-spdy/ http://code.google.com/p/mod-spdy/ It already has had support since the start (though non-ASF)... It would be unwise to release a new protocol without releasing a proof-of-concept tie-in for an already existing web server, unless you want the project to be DOA.
lukifer 16y ago
Any plans for native support of WebSockets?

What type of setup do you use to analyze, debug, and follow the code? Any tips on getting started with module development?

brianm 16y ago
Another apache hacker answering :-) I use emacs and gdb to muck around the code. Apache has a lot of backwards compat warts with regard to which header files define things, unfortunately, but grep (or ack) and ctags will get you around. The best way (by far) to learn about module hackery is with Nick Kew's book, http://www.amazon.com/Apache-Modules-Book-Application-Development/dp/0132409674 http://www.amazon.com/Apache-Modules-Book-Application-Develo...
bnoordhuis 16y ago
Another vote for Nick Kew's book. He is a regular poster on the modules-dev mailing list[1], by the way. `httpd -X` and `gdb httpd $(pidof httpd)` are a great help when debugging. [1] http://mail-archives.apache.org/mod_mbox/httpd-modules-dev/ http://mail-archives.apache.org/mod_mbox/httpd-modules-dev/

kristofferR 16y ago

Why is apache so slow compared to Nginx, Cherokee, Lighttpd and many other web server softwares?

powertower 16y ago
Mostly due to two reasons: 0) People don't know how to use PHP with it: module or fastcgi; and which MPM to use here. 1) People don't know how to configure the MPM setting to allow Apache full use of server resources (without under-utilizing or over-utilizing). 2) Apache is a full / well-rounded web server application... It's not a specialty server (ex: static content mostly) that can exclude this and that and only provide 1 feature to excel at. Other servers are stripped down compared to Apache. * Not the AMA guy (nor an Apache hacker guy), buy have a little known WAMP distribution (called WampDeveloper, used to be Web.Developer Server Suite) since 2003 with over 250,000 unique ip downloads between 2003-2006 (have stopped counting), and have worked on 1000s of issues for users and clients since the start.
- calloc 16y ago
  The only reason I still run Apache is because a lot of PHP stuff requires mod_rewrite to create pretty URL's and I haven't had the time to rewrite them using LUA so I can use Lighttpd. But on the same server, using PHP-FPM with FastCGI on both lighttpd and Apache I am serving almost 4x the requests that Apache is serving with less memory overhead. Switching the sites that are currently running on Lighttpd back to Apache as I attempted to do not to long ago to only have to maintain a single server made Apache die a miserable fast death. It just could not keep up, I've had friends look over my config (datacenter techs, help people scale their stuff, porn mostly) and they said it looked fine. Apache was the limiting factor here. I moved from prefork to threaded, that helped a little, but not much. Apache was just using a lot of memory, and overall did not provide the performance I wanted. Switched back to lighttpd, load on the server went down, and the websites were responsive as before.
  - powertower 16y ago
    "made Apache die a miserable fast death. It just could not keep up, I've had friends look over my config (datacenter techs, help people scale their stuff, porn mostly) and they said it looked fine." Apache crashing under load is indicative of seriously bad MPM settings: specifically the # of processes / threads allowed to be used (and some other settings such as KeepAlive timeout, how php is being used, etc). With a properly configured MPM, anything coming over the process / thread limit goes into a backlog queue which is 511 entries long. Anything over this just gets dropped. To iterate, Apache does not crash under load (too many requests), it crashes due to too many processes / threads (an MPM setting) being used and sometimes due to leaking modules such as mod_python/ruby.
bnoordhuis 16y ago
Different design goals. Apache is meant to be robust, extensible and portable. * Robust: that's reflected in its internal API that makes it near impossible to leak resources. * Extensible: witness the gazillion modules out there. * Portable: compiles and runs on very exotic or outdated systems. SCO, IRIX, Digital UNIX, VMS, the list goes on. nginx and such were designed from the ground up with performance in mind - and with success - but the trade-off is a lack of portability and an API that is much harder to program to.
- deleted 16y ago
  [deleted]
- kujawa 16y ago
  Well gee. It's been a while since I've used Digital UNIX, Irix, SCO, or VMS. Because they've become totally irrelevant. Seems like a waste of resources when there are more pressing things to do than worry about the 5 people who (a) use VMS and (b) demand a bleeding-edge Apache.
  - lambda 16y ago
    No need to be so acerbic. They are providing a valuable service to you, for free. Apache is older, much more ubiquitous, supports more modules, has a more familiar setup and configuration system for many people, runs on more platforms, etc. It has more legacy issues, which are why it's so popular, but also why it moves slower. Nginx is much newer, doesn't have the same legacy issues, and so could afford to focus on speed. There's a reason that Apache is installed on just about every random web host you can find, and has a module for every language or environment you need to deel with, while Nginx is a bit more of a specialty web server, usually used for dedicated sites that can spend the time to tune carefully for the highest performance. They both have their place. It's great that Apache is still innovating moving towards loadable MPMs as well as adding an evented MPM. But it's not a bad thing that they're moving slower than a new server like Nginx; there's room for more than one great free web server in the world.
getsat 16y ago
Preforked and threaded blocking IO consumes far more resources than event-driven non-blocking network IO. Process/thread stacks consume a significant amount of memory. This is why nginx ruins Apache at almost anything at scale (especially reverse proxying). Apache falls over while consuming grandiose amounts of memory in the low millions of connections per day while nginx can easily handle 10mm connections per day on a single core machine with 256mb ram. It's ridiculous. On the other hand, for long-lived connections, you're better off using blocking IO since it's more efficient. Most web traffic is not long-lived, though. See Paul's interesting article on NIO vs IO performance: http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-is-not-faster-than.html http://mailinator.blogspot.com/2008/02/kill-myth-please-nio-... http://www.mailinator.com/tymaPaulMultithreaded.pdf http://www.mailinator.com/tymaPaulMultithreaded.pdf And don't forget about Slowloris. If you see your competition using Apache, you really don't need to worry. :) Disclaimer: I have nothing personal against Apache and used it for six years or so. I've just moved on to better software. If it works for you, that's great.
- rb2k_ 16y ago
  Will those two things included in the new apache change this?: Event MPM: The Event MPM is no longer experimental but is rather fully supported. Asynchronous support: Better support for asynchronous read/write for supporting MPMs and platforms.
  - getsat 16y ago
    Yes, I believe so. I'm really excited that the event MPM is no longer experimental. I had tried it way back in the day, but mod_php could not be safely used with it as mod_php was (is?) not thread-safe. I wasn't aware of FCGI at the time.
- bad_user 16y ago
  I still use Apache because I trust it; and because the ease with which I can configure a Python server (i.e. mod_wsgi), a Ruby server (passenger) and a Perl server, while being able to make use of the same modules I've been using for years. Also, a properly configured Varnish placed in front of Apache ruins Nginx at almost anything at scale. I've seen it. Most web traffic is not long-lived, though. But most web traffic is blocking. Going NIO requires caching - which is a huge penalty and a PITA; and doing it while not having actual users doesn't make sense.

mthomas 16y ago

Are there any significant barriers to use the event MPM as it stands in the 2.2. Does it fall back to worker when its used under SSL?

bnoordhuis 16y ago
No. The main problem the event MPM addresses is HTTP keep-alive. With pre-fork and threaded, every connection takes up a process or thread. This sucks for keep-alive because it can take a while before the client issues the next request. The event MPM puts that idle connection in a kqueue/epoll/etc. pollset and recycles the process/thread for another request. It's a conceptually simple change but it has some profound performance implications.
brianm 16y ago
In 2.2, the event mpm doesn't support ssl, but in 2.3/2.4 it does support ssl. In fact, IIRC, if you are using apche as a reverse proxy (mod_proxy) the event mpm should become the standard as it will detach the worker thread both for keepalive on the front end, and when waiting on response from the backend.
saurik 16y ago
As a single datapoint, I run a site that handles many billions (this is not rhetorical or an exaggeration) of hits per month using mpm_event: I consider it the only sane/stable way of configuring Apache.
- patio11 16y ago
  As I found out to my sadness, the only sane configuration is not the one which ships with apt-get apache2. I like many things about Nginx. The fact that it is a production-capable web server out of the box, for example. I know that is probably not a high-priority design goal for Apache for historical reasons, but it seems a very sensible default in 2011.
  - saurik 16y ago
    For the record, "apt-get install apache2-mpm-event" is all you need to switch to mpm_event: I believe it isn't supported on all platforms, and, as a conservative Debian-based distribution using apt-get, I'd prefer "predictable behavior across all of our targets" over "highest performance configuration on any given system". (I also believe there are some corner cases where mpm_event may burn you on 2.2, and both Apache and Debian are going to play it safe there; but, when I looked into this in detail a forever or so ago, I determined that whatever issues they were didn't apply to my setup. As an example: if SSL were an issue, I doubt I would ever use Apache to directly serve SSL anyway, as that's what SSL accelerators are for.) That said, I still wouldn't leave the defaults in place with regards to "number of servers / threads"; although it isn't like nginx doesn't need the same configuration: the default value of worker_{processes,connections} is almost certainly inappropriate for your specific setup. I also use nginx (as a load balancer), and I have those values at 64/10096, up from the Debian-default of 1/1024. The important thing, to me, is what a technology makes possible, not how well it is configured out of the box. Example: it is more damning to me that nginx only does HTTP/1.0 to upstream servers it is proxying for, a reasonably fundamental limitation of the codebase, than any transgression they could make in their default configuration. Seriously: production is not about "out of the box", it never was, and it likely is never going to be. If you are trying to run a production server using "out of the box" defaults you are going to be forever disappointed by the performance and functionality of the offerings. To make a minor modification to a statement I've made before (http://news.ycombinator.com/item?id=2145967 http://news.ycombinator.com/item?id=2145967 is the reference) about database servers: We (as a civilization) simply do not have the science and theory yet to make the practicalities of setting up and maintaining anything at this level of complexity a totally seamless and simple process with well-understood performance characteristics unless you constrain absolutely every single variable.

pwpwp 16y ago

Anything new on Waka?

IgorPartola 16y ago

Will apache support Google's SPDY at some point? Also, how much is having to support all the legacy platforms slowing down development of new features?

powertower 16y ago
http://code.google.com/p/mod-spdy/ http://code.google.com/p/mod-spdy/ It already has had support since the start (though non-ASF)... It would be unwise to release a new protocol without releasing a proof-of-concept tie-in for an already existing web server, unless you want the project to be DOA.

lukifer 16y ago

Any plans for native support of WebSockets?