10 ms·

Wordpress.com serves 70k requests/second using Nginx

131 points by diego 14y ago

ck2 14y ago
Those must be cached pages from dozens of servers on the backend. There is no way in heck it's realtime queries. I can tell when authors are in the WP backend on the server just by looking at the server load because it's crippling. The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem of loading hundreds of files for any kind of page render (even to just authenticate for ajax, etc.) and over a hundred db queries in many cases. A cache-miss in WP is a terrible, terrible thing.
- piers 14y ago
  Don't forget the version of Wordpress that you download is very different from the version that's on Wordpress.com
  - ck2 14y ago
    I know they replace the database class on wp.com to support replication (I think they even released the code once) but I don't think they do extensive changes otherwise unless you've actually read otherwise.
    - robmil 14y ago
      It's called HyperDB, and they did indeed release it: http://codex.wordpress.org/HyperDB http://codex.wordpress.org/HyperDB
  - diego 14y ago
    It's actually not that different, and of course they make as much use of caching as possible. So does my self-hosted wp blog, for that matter. If you are using WP Super Cache correctly, you can deal with significant amounts of traffic. I've seen peaks of hundreds of simultaneous users on my blog, with a server load < 1. The server is a modest aws small instance.
    - ck2 14y ago
      Single author mini-blogs without logged in users are super easy to cache. But multi-author blogs with thousands of logged in users is a nightmare with wp. I suspect 90%+ of wp blogs are in the mini-blog category though.
      - TillE 14y ago
        Why would multiple authors make a difference?
        ck2 14y ago
        Every time an article is created, saved or published, wp uses hundreds of non-cachable queries. Every time an article is published, it causes the cache to be deleted for not only that article but related pages, which means all those pages have to be rendered again. For one author, that can be managed. Many authors, the cache is constantly being defeated.
        diego 14y ago
        What matters is the ratio of reads per write. Blogs with multiple authors have a higher ratio than blogs with single authors.
        batista 14y ago
        >For one author, that can be managed. Many authors, the cache is constantly being defeated. That's irrelevant. It's the pages view count that counts, not how many authors are in the same cms.
- Negitivefrags 14y ago
  Cached or uncached is not relevant to the article. They are talking about using nginx as a load balancer. It just proxies all requests off to the application servers that actually prepare the results. This article is really just about the fact that nginx can efficiently service a very impressive number of connections at once. Those backend servers will be using caches of course though.
- nisa 14y ago
  Wordpress is only reasonable fast if you use APC, memcached and the MySQL query cache. I never liked the caching plugins. If you use nginx just use the fastcgi_cache module and make cookies part of the cache key or omit the caching of pages where the login cookie is set.
  - ck2 14y ago
    APC+mysql cache (not just query but keycache too) is a must, but memcached is pointless unless you have a multi-server website (on a single server APC shared memory is way faster than memcached). All logged out users should be served a completely cached static page that bypasses PHP entirely, it's way faster than even an opcode cache and way less load. WP-Super-Cache can serve static pages to logged out users and makes this rather easy to setup. It's practically a must for wordpress.
    - bad_user 14y ago
      I don't know how the caching plugin in WP works, however the most difficult part of caching is always cache invalidation, which is the reason why Memcache is better than an in-memory cache, even for a single server, because then you have the flexibility of invalidating cache entries from a background process that does not run on that server. It's also great to have the flexibility of adding more servers in case of a huge spike, even if you are running on only one server. Like on Heroku, where you can just increase the number of dynos in realtime. Doing that without a central cache that every server can access can break your MySQL because each new server will have a cold cache.
      - hpatoio 14y ago
        If you want a smart cache invalidation system for NGINX please have a look at this plugin http://bit.ly/nginxmanager http://bit.ly/nginxmanager Pages are cached on NGINX for not logged in user, and when a new content is published all and only the pages that contains that content are deleted from the cached and regenerated at the next request.
    - nisa 14y ago
- barsxl 14y ago
  Many more dynamic pages than you would expect. We do hundreds of thousands of database queries/sec. We do some caching with Batcache - http://wordpress.org/extend/plugins/batcache/ http://wordpress.org/extend/plugins/batcache/ but it's more to handle large spikes in traffic for single pages/blogs. CNN during the US elections, for example.
sowhata 14y ago
And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.
- HyprMusic 14y ago
  According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.
  - cowsaysoink 14y ago
    I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count) From the article it seems more like they have ~100, maybe less, for the load balancers which comes out to 175 requests/second per cpu which is getting a little more reasonable.
    - sowhata 14y ago
      If this is just about load balancing, then they could probably get better performance from haproxy, right? Anyone should be able to exceed 200 req/s with haproxy. In fact, you should be able to get around double that or more. The point is the title is useless. It tells you almost nothing. That blog is a joke.
      - cowsaysoink 14y ago
        Well it is average req/s so peak should be much higher, and they probably can average much more than what they are doing now in case of spikes and other such things. But yeah the title is pretty much useless.
gregcmartin 14y ago
I was one of the core engineers @ layeredtech who managed the servers and HA for wordpress.com when they launched in 2006. If I remember correctly we were using DNS round robin and haproxy -> apache -> mysql all on freebsd systems wow have things come a long ways since then also it's incredible the sustained growth of Wordpress after all this time. good memories... congrats Matt on all your success.
- barsxl 14y ago
  WordPress.com hasn't run on FreeBSD since the TextDrive days which was way before Layered Tech, before me and before WordPress.com was open to the public. We are 100% Debian today, but have used Ubuntu in the past. We never really used HAProxy either. I posted about out load balancer choices back in 2008 - http://barry.wordpress.com/2008/04/28/load-balancer-update/ http://barry.wordpress.com/2008/04/28/load-balancer-update/
jacques_chester 14y ago
I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. And with some fancy footwork in the rewrite rules to support WP Super Cache, it can serve static files and gzipped HTML straight off disk. Unsurprisingly, in this configuration, it runs like the clappers.[1][2] One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later. [1] At least, it used to do this, until I updated everything and it silently stopped working the way it used to (which I only discovered while outlining my current config in comments below). I mean, thanks to memcached and closer servers and running Percona on a separate server it's still pretty fast, but this change bugs me and you can bet I'll be fixing it later today after work. [2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode cache. The query cache because I don't think it buys me enough versus using memcached (just a gut feel, I've got no hard numbers to back me up) and because it often turns into a contention point in read-heavy MySQL instances. Also, by dropping it I can free up more memory for InnoDB. The opcode cache because they tend to be flaky. I've had bad experiences with all three major ones. If you ever pop Wordpress up on a profiled PHP instance you find that its time is spent overwhelmingly on waiting for MySQL, concatenating strings and then streaming it out to the webserver, for which proper on-disk/in-memory caching is the answer. Time spent loading and compiling .php is minimal by comparison, so why put up with the hassle? I don't use varnish because the whole philosophy is to let the OS select what to cache in memory, and ... well I already do that. Plus getting ESI to work on frequently updated objects like "recent comment" widgets is a hairy pain in the arse and I just can't be bothered.
- baconhigh 14y ago
  I'd be interested in seeing your 'fancy footwork' for the rewrite rules :)
  - 14y ago
misiti3780 14y ago
"Ability to reconfigure and upgrade NGINX instances on-the-fly, without dropping user requests." - how do you do this ?
- j2labs 14y ago
  SIGHUP
  - nodesocket 14y ago
    `service nginx reload` will do the trick as well.
    - mfjordvald 14y ago
      Only for reloading the config, not for upgrading the binary.
- mfjordvald 14y ago
  It's explained in detail here: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_... If you are compiling via source then you can also run "make upgrade" after "make install" and it will do it for you.
xxiao 14y ago
can we do similar things at hacknews? it's one of the slowest site I visited daily
mstank 14y ago
Article didn't mention that they also use a CDN (EdgeCast). Helps make the 70k requests/second a lot more bearable.
- barsxl 14y ago
  We serve about 150k req/sec from the CDN. That (obviously) isn't included in the 70k number from the article. The 70k is only what we are serving from the origin.
stickhandle 14y ago
Two takeaways - (1) The article is bumf wrt the Wordpress.com setup. NGINX plays a role, but the thousands of servers for http, db, memcache, load balancing etc is integral to this discussion (and not mentioned). A typical reader couldn't relate if it did. (2) Directed at a more typical reader, use NGINX for hosting your WP.org blogs, company sites etc. Properly configured, your performance to cost ratio will be terrific. Lastly, on a personal note, it pains me when caching plugins are mentioned as the main part of the solution. No need for these folks. Use the NGINX cache instead and never worry again about a buggy/outdated cache plugin (looking at you W3TC) again. Remove that dependency and you will be happier and possibly better looking.
zerop 14y ago
We dropped nginx because of no native support for session affinity. SSL support has been introduced in HAProxy very recently.
- Negitivefrags 14y ago
  Is the ip_hash directive not good enough for what your looking for?
  - zerop 14y ago
    I was not aware of ip_hash. But as per documentation it uses first 3 octets as hashing key, this may not work for me.I want to use nginx on LAN. It will throw all clients to same backend server. http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...
    - Negitivefrags 14y ago
      It should be fairly trivial to patch your copy of nginx to use the entire IP address as the hashing key.
- lazyfunctor 14y ago
  http://wiki.nginx.org/HttpUpstreamRequestHashModule http://wiki.nginx.org/HttpUpstreamRequestHashModule this should work for you
tsahyt 14y ago
This figure is not from a single instance though. When running a single instance of WP I've always found it to be fantastically slow. The greatness in the described stack really lies in Nginx. It's really not WordPress serving 70k requests here, it's some decent load balancing that achieves the effect. The title is somewhat misleading.
- ocharles 14y ago
  Yea, I was trying to find some information on exactly how many servers are responsible for content generation. If this is about 100 machines then yes, 70k/s is pretty damn impressive. If it's 1000 machines, then I'm a little less impressed.
haight_geek 14y ago
1 instance of G-WAN serves 500k requests/second with an typical application using Oracle NoSQL (or Berkeley DB).
haight_geek 14y ago
1 instance of G-WAN serves 500k requests/second with an typical application using Oracle NoSQL (or Berkeley DB).

131 points by diego 14y ago

ck2 14y ago

Those must be cached pages from dozens of servers on the backend. There is no way in heck it's realtime queries. I can tell when authors are in the WP backend on the server just by looking at the server load because it's crippling. The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem of loading hundreds of files for any kind of page render (even to just authenticate for ajax, etc.) and over a hundred db queries in many cases. A cache-miss in WP is a terrible, terrible thing.

piers 14y ago
Don't forget the version of Wordpress that you download is very different from the version that's on Wordpress.com
- ck2 14y ago
  I know they replace the database class on wp.com to support replication (I think they even released the code once) but I don't think they do extensive changes otherwise unless you've actually read otherwise.
  - robmil 14y ago
    It's called HyperDB, and they did indeed release it: http://codex.wordpress.org/HyperDB http://codex.wordpress.org/HyperDB
- diego 14y ago
  It's actually not that different, and of course they make as much use of caching as possible. So does my self-hosted wp blog, for that matter. If you are using WP Super Cache correctly, you can deal with significant amounts of traffic. I've seen peaks of hundreds of simultaneous users on my blog, with a server load < 1. The server is a modest aws small instance.
  - ck2 14y ago
    Single author mini-blogs without logged in users are super easy to cache. But multi-author blogs with thousands of logged in users is a nightmare with wp. I suspect 90%+ of wp blogs are in the mini-blog category though.
    - TillE 14y ago
      Why would multiple authors make a difference?
      - ck2 14y ago
        Every time an article is created, saved or published, wp uses hundreds of non-cachable queries. Every time an article is published, it causes the cache to be deleted for not only that article but related pages, which means all those pages have to be rendered again. For one author, that can be managed. Many authors, the cache is constantly being defeated.
        diego 14y ago
        What matters is the ratio of reads per write. Blogs with multiple authors have a higher ratio than blogs with single authors.
        batista 14y ago
        >For one author, that can be managed. Many authors, the cache is constantly being defeated. That's irrelevant. It's the pages view count that counts, not how many authors are in the same cms.
Negitivefrags 14y ago
Cached or uncached is not relevant to the article. They are talking about using nginx as a load balancer. It just proxies all requests off to the application servers that actually prepare the results. This article is really just about the fact that nginx can efficiently service a very impressive number of connections at once. Those backend servers will be using caches of course though.
nisa 14y ago
Wordpress is only reasonable fast if you use APC, memcached and the MySQL query cache. I never liked the caching plugins. If you use nginx just use the fastcgi_cache module and make cookies part of the cache key or omit the caching of pages where the login cookie is set.
- ck2 14y ago
  APC+mysql cache (not just query but keycache too) is a must, but memcached is pointless unless you have a multi-server website (on a single server APC shared memory is way faster than memcached). All logged out users should be served a completely cached static page that bypasses PHP entirely, it's way faster than even an opcode cache and way less load. WP-Super-Cache can serve static pages to logged out users and makes this rather easy to setup. It's practically a must for wordpress.
  - bad_user 14y ago
    I don't know how the caching plugin in WP works, however the most difficult part of caching is always cache invalidation, which is the reason why Memcache is better than an in-memory cache, even for a single server, because then you have the flexibility of invalidating cache entries from a background process that does not run on that server. It's also great to have the flexibility of adding more servers in case of a huge spike, even if you are running on only one server. Like on Heroku, where you can just increase the number of dynos in realtime. Doing that without a central cache that every server can access can break your MySQL because each new server will have a cold cache.
    - hpatoio 14y ago
      If you want a smart cache invalidation system for NGINX please have a look at this plugin http://bit.ly/nginxmanager http://bit.ly/nginxmanager Pages are cached on NGINX for not logged in user, and when a new content is published all and only the pages that contains that content are deleted from the cached and regenerated at the next request.
  - nisa 14y ago
barsxl 14y ago
Many more dynamic pages than you would expect. We do hundreds of thousands of database queries/sec. We do some caching with Batcache - http://wordpress.org/extend/plugins/batcache/ http://wordpress.org/extend/plugins/batcache/ but it's more to handle large spikes in traffic for single pages/blogs. CNN during the US elections, for example.

sowhata 14y ago

And how many requests/sec per CPU? They have like 300 servers and who knows how many cores. What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

HyprMusic 14y ago
According to the article they have 2 thousand servers. These are Dual Xeon 5620 4 core CPUs with hyper-threading. So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU. That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.
- cowsaysoink 14y ago
  I don't think that is right, it says that automattic has 2000 servers but they also run things like gravitar, akismet, and vaultpress. Which makes me think that using that number is completely wrong. (Also these are just the load balancers not the back end, which could be taking up a large amount of that 2000 server count) From the article it seems more like they have ~100, maybe less, for the load balancers which comes out to 175 requests/second per cpu which is getting a little more reasonable.
  - sowhata 14y ago
    If this is just about load balancing, then they could probably get better performance from haproxy, right? Anyone should be able to exceed 200 req/s with haproxy. In fact, you should be able to get around double that or more. The point is the title is useless. It tells you almost nothing. That blog is a joke.
    - cowsaysoink 14y ago
      Well it is average req/s so peak should be much higher, and they probably can average much more than what they are doing now in case of spikes and other such things. But yeah the title is pretty much useless.

gregcmartin 14y ago

I was one of the core engineers @ layeredtech who managed the servers and HA for wordpress.com when they launched in 2006. If I remember correctly we were using DNS round robin and haproxy -> apache -> mysql all on freebsd systems wow have things come a long ways since then also it's incredible the sustained growth of Wordpress after all this time. good memories... congrats Matt on all your success.

barsxl 14y ago
WordPress.com hasn't run on FreeBSD since the TextDrive days which was way before Layered Tech, before me and before WordPress.com was open to the public. We are 100% Debian today, but have used Ubuntu in the past. We never really used HAProxy either. I posted about out load balancer choices back in 2008 - http://barry.wordpress.com/2008/04/28/load-balancer-update/ http://barry.wordpress.com/2008/04/28/load-balancer-update/

jacques_chester 14y ago

I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more. Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module. My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable. So now I'm on Nginx. And with some fancy footwork in the rewrite rules to support WP Super Cache, it can serve static files and gzipped HTML straight off disk. Unsurprisingly, in this configuration, it runs like the clappers.[1][2] One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later. [1] At least, it used to do this, until I updated everything and it silently stopped working the way it used to (which I only discovered while outlining my current config in comments below). I mean, thanks to memcached and closer servers and running Percona on a separate server it's still pretty fast, but this change bugs me and you can bet I'll be fixing it later today after work. [2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode cache. The query cache because I don't think it buys me enough versus using memcached (just a gut feel, I've got no hard numbers to back me up) and because it often turns into a contention point in read-heavy MySQL instances. Also, by dropping it I can free up more memory for InnoDB. The opcode cache because they tend to be flaky. I've had bad experiences with all three major ones. If you ever pop Wordpress up on a profiled PHP instance you find that its time is spent overwhelmingly on waiting for MySQL, concatenating strings and then streaming it out to the webserver, for which proper on-disk/in-memory caching is the answer. Time spent loading and compiling .php is minimal by comparison, so why put up with the hassle? I don't use varnish because the whole philosophy is to let the OS select what to cache in memory, and ... well I already do that. Plus getting ESI to work on frequently updated objects like "recent comment" widgets is a hairy pain in the arse and I just can't be bothered.

baconhigh 14y ago
I'd be interested in seeing your 'fancy footwork' for the rewrite rules :)
- 14y ago

misiti3780 14y ago

"Ability to reconfigure and upgrade NGINX instances on-the-fly, without dropping user requests." - how do you do this ?

j2labs 14y ago
SIGHUP
- nodesocket 14y ago
  `service nginx reload` will do the trick as well.
  - mfjordvald 14y ago
    Only for reloading the config, not for upgrading the binary.
mfjordvald 14y ago
It's explained in detail here: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_On_The_Fly http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_... If you are compiling via source then you can also run "make upgrade" after "make install" and it will do it for you.

xxiao 14y ago

can we do similar things at hacknews? it's one of the slowest site I visited daily

mstank 14y ago

Article didn't mention that they also use a CDN (EdgeCast). Helps make the 70k requests/second a lot more bearable.

barsxl 14y ago
We serve about 150k req/sec from the CDN. That (obviously) isn't included in the 70k number from the article. The 70k is only what we are serving from the origin.

stickhandle 14y ago

Two takeaways - (1) The article is bumf wrt the Wordpress.com setup. NGINX plays a role, but the thousands of servers for http, db, memcache, load balancing etc is integral to this discussion (and not mentioned). A typical reader couldn't relate if it did. (2) Directed at a more typical reader, use NGINX for hosting your WP.org blogs, company sites etc. Properly configured, your performance to cost ratio will be terrific. Lastly, on a personal note, it pains me when caching plugins are mentioned as the main part of the solution. No need for these folks. Use the NGINX cache instead and never worry again about a buggy/outdated cache plugin (looking at you W3TC) again. Remove that dependency and you will be happier and possibly better looking.

zerop 14y ago

We dropped nginx because of no native support for session affinity. SSL support has been introduced in HAProxy very recently.

Negitivefrags 14y ago
Is the ip_hash directive not good enough for what your looking for?
- zerop 14y ago
  I was not aware of ip_hash. But as per documentation it uses first 3 octets as hashing key, this may not work for me.I want to use nginx on LAN. It will throw all clients to same backend server. http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...
  - Negitivefrags 14y ago
    It should be fairly trivial to patch your copy of nginx to use the entire IP address as the hashing key.
lazyfunctor 14y ago
http://wiki.nginx.org/HttpUpstreamRequestHashModule http://wiki.nginx.org/HttpUpstreamRequestHashModule this should work for you

tsahyt 14y ago

This figure is not from a single instance though. When running a single instance of WP I've always found it to be fantastically slow. The greatness in the described stack really lies in Nginx. It's really not WordPress serving 70k requests here, it's some decent load balancing that achieves the effect. The title is somewhat misleading.

ocharles 14y ago
Yea, I was trying to find some information on exactly how many servers are responsible for content generation. If this is about 100 machines then yes, 70k/s is pretty damn impressive. If it's 1000 machines, then I'm a little less impressed.

haight_geek 14y ago

1 instance of G-WAN serves 500k requests/second with an typical application using Oracle NoSQL (or Berkeley DB).

haight_geek 14y ago

1 instance of G-WAN serves 500k requests/second with an typical application using Oracle NoSQL (or Berkeley DB).