5 ms·

How to hurricane-proof a web server for Harvey

42 points by lainon 9y ago

brianwawok 9y ago
TLDR; Install caching plugins to wordpress, run on a server that is oversized about 100x for steady state load. Which I guess works, but is not that interesting of a technical read? If you were not on wordpress, I bet you could have served that load from a single VM on a cloud provider, with the ability to scale as needed.
- driverdan 9y ago
  Plus it's behind CloudFlare which should absorb a lot of the traffic if headers are configured correctly.
  - jlgaddis 9y ago
    Yes, ~97% or so, as the author noted in TFA.
- lee_ars 9y ago
  > TLDR; Install caching plugins to wordpress... Clarification: the "caching plugins" are just administrative tie-ins to Varnish and Cloudflare, not actual plugins that make WP perform any caching at the WP application layer. The site doesn't use W3 Total Cache or WP Super Cache or any other popular WP cache plugin. (edit - at the wordpress application layer, i mean—talking about that specifically, not layer 7 cache in general)
- CharlesW 9y ago
  Here's the actual TLDR from the article: * Dedicated hardware server—no shared hosting * Gigabit uplink to handle burst traffic * Varnish cache with lots of RAM * Plugin-light WordPress installation * Off-box stats via WordPress/Jetpack * Cloudflare as a CDN * No heavy dependencies (i.e., ad networks)
  - js2 9y ago
    And really "Cloudflare as a CDN" was the most important. It absorbed almost all of the traffic: Between August 22 and August 30, Cloudflare served over five terabytes of cached Space City Weather content via Cloudflare’s CDN. Only about 145GB worth of requests had to be served by the Space City Weather Web server itself. [...] Looking at raw requests instead of bytes, Cloudflare handled about 88 percent of those, too, with only about 12 percent of requests passed back up the chain to the origin server. The origin server ended up seeing peaks of 50 RPS, most of that was absorbed by Varnish. I guess 50 RPS is a lot for WordPress, though I'm surprised given the origin server is a decent box: The server has a quad-core Xeon E3-1230 CPU, 16GB of RAM, and mirrored R1 SSDs.
- DKnoll 9y ago
  Why do you think you can't run WordPress on a VPS? Also scaling hardware is the last resort, there are almost always cheaper optimisations to be made.
otterpro 9y ago
I was hoping to see data center that was barricading itself physically against the flooding with sandbox and preparing with diesel generators and disaster recovery. I think web traffic would've been the last thing on my mind.
DamonHD 9y ago
Good write-up. And it kept him off the streets. Many moons ago when a US Robotics Courier modem on a leased analogue line was still bleeding edge live IP provision in the UK, one of our pages got featured in "Mirsky's Worst of the Web" and our line was just maxed out for days. Divide all the CPU and line speeds by a few orders of magnitude and it all felt much the same getting what would become known as /.-ed... B^>
mi100hael 9y ago
I'd be curious to see whether any additional performance could be eeked out by ditching HAProxy & Varnish. Nginx can terminate TLS and handle caching, which would reduce the overhead of multiple separate processes & connections.
bhhaskin 9y ago
Warning: Autoplaying video ad. No audio, so that is a plus.

42 points by lainon 9y ago

TLDR; Install caching plugins to wordpress, run on a server that is oversized about 100x for steady state load. Which I guess works, but is not that interesting of a technical read? If you were not on wordpress, I bet you could have served that load from a single VM on a cloud provider, with the ability to scale as needed.

driverdan 9y ago
Plus it's behind CloudFlare which should absorb a lot of the traffic if headers are configured correctly.
- jlgaddis 9y ago
  Yes, ~97% or so, as the author noted in TFA.
lee_ars 9y ago
> TLDR; Install caching plugins to wordpress... Clarification: the "caching plugins" are just administrative tie-ins to Varnish and Cloudflare, not actual plugins that make WP perform any caching at the WP application layer. The site doesn't use W3 Total Cache or WP Super Cache or any other popular WP cache plugin. (edit - at the wordpress application layer, i mean—talking about that specifically, not layer 7 cache in general)
CharlesW 9y ago
Here's the actual TLDR from the article: * Dedicated hardware server—no shared hosting * Gigabit uplink to handle burst traffic * Varnish cache with lots of RAM * Plugin-light WordPress installation * Off-box stats via WordPress/Jetpack * Cloudflare as a CDN * No heavy dependencies (i.e., ad networks)
- js2 9y ago
  And really "Cloudflare as a CDN" was the most important. It absorbed almost all of the traffic: Between August 22 and August 30, Cloudflare served over five terabytes of cached Space City Weather content via Cloudflare’s CDN. Only about 145GB worth of requests had to be served by the Space City Weather Web server itself. [...] Looking at raw requests instead of bytes, Cloudflare handled about 88 percent of those, too, with only about 12 percent of requests passed back up the chain to the origin server. The origin server ended up seeing peaks of 50 RPS, most of that was absorbed by Varnish. I guess 50 RPS is a lot for WordPress, though I'm surprised given the origin server is a decent box: The server has a quad-core Xeon E3-1230 CPU, 16GB of RAM, and mirrored R1 SSDs.
DKnoll 9y ago
Why do you think you can't run WordPress on a VPS? Also scaling hardware is the last resort, there are almost always cheaper optimisations to be made.

otterpro 9y ago

I was hoping to see data center that was barricading itself physically against the flooding with sandbox and preparing with diesel generators and disaster recovery. I think web traffic would've been the last thing on my mind.

DamonHD 9y ago

Good write-up. And it kept him off the streets. Many moons ago when a US Robotics Courier modem on a leased analogue line was still bleeding edge live IP provision in the UK, one of our pages got featured in "Mirsky's Worst of the Web" and our line was just maxed out for days. Divide all the CPU and line speeds by a few orders of magnitude and it all felt much the same getting what would become known as /.-ed... B^>

mi100hael 9y ago

I'd be curious to see whether any additional performance could be eeked out by ditching HAProxy & Varnish. Nginx can terminate TLS and handle caching, which would reduce the overhead of multiple separate processes & connections.

bhhaskin 9y ago

Warning: Autoplaying video ad. No audio, so that is a plus.