5 ms·

I abandoned OpenLiteSpeed and went back to good ol' Nginx

83 points by mikece 3y ago

bkallus 3y ago
This makes sense. Nginx is reliable and well-tested, and OLS is neither. See my recent Shmoocon talk (yes yes, self promotion) for an example of a vulnerability that would be caught immediately in Nginx, but persisted in the OLS codebase for >=10 years: https://yewtu.be/watch?v=aKPAX00ft5s&t=2h29m28s https://yewtu.be/watch?v=aKPAX00ft5s&t=2h29m28s
- mmaunder 3y ago
  Really nice work.
whalesalad 3y ago
nginx can be a cache and an ssl termination point so I’m not sure why the author didn’t do this in the first place versus wrapping it in haproxy/varnish.
- godisdad 3y ago
  maybe as a layer of indirection so nginx and varnish could be independent backends? I wasn’t entire sure of this either
- lee_ars 3y ago
  Author here. I settled on the haproxy/varnish/nginx sandwich through an accretive process that started back in 2012 with wanting to speed up nginx by taking advantage of scads of extra RAM on the box I was hosting with at the time (and also being entranced by all the cool header games one can play with Varnish). Fast forward to 2015 as LetsEncrypt started to be a thing, and HTTPS started the mainstream push from a nice-to-have to an ought-to-have feature. I was comfy with Varnish + Nginx and I wanted to keep my cache layer, but Varnish has no SSL termination capabilities. I looked at a few options, including Hitch, but decided on HAProxy because it was more widely used and therefore easier to google answers to questions. Everything worked, so I never re-evaluated until 2022 when I got curious about OLS. As to why Varnish + Nginx versus just Nginx + FastCGI or proxy cache, I was chasing the performance dragon. AIUI, Nginx's fastcgi and proxy caches just store the static-ified versions of dynamic content in flat files, and those files are accessed via the file system, which means they get cached into RAM whenever the file system decides they get cached into RAM. I had the free RAM, and Varnish's RAM-first approach felt like it would be faster. Plus, this was all for personal hosting, and playing with Varnish was fun. (Or at least "fun.")
  - OJFord 3y ago
    Have you considered caddy?
    - lee_ars 3y ago
      No, I'm happy to retreat to the familiar, and nginx is pretty familiar. After having my tail feathers burned with OLS, I doubt I'll switch to anything else for the next several years without an extremely compelling reason.
      - OJFord 3y ago
        Sure I'm not suggesting it now as a follow up, I just wondered if it was something you were aware of and dismissed for some reason over the last several years before opting to try OLS (which I hadn't heard of) instead; especially since you mentioned Let's Encrypt.
      - deleted 3y ago
        [deleted]
  - jauntywundrkind 3y ago
    Thanks for the walkthrough. One particular comment: > "Nginx's fastcgi and proxy caches just store the static-ified versions of dynamic content in flat files, and those files are accessed via the file system, which means they get cached into RAM whenever the file system decides they get cached into RAM. I had the free RAM, and Varnish's RAM-first approach felt like it would be faster." The pagecache should do a pretty good job holding file contents in memory for you! https://tldp.org/LDP/sag/html/buffer-cache.html https://tldp.org/LDP/sag/html/buffer-cache.html I haven't had to tune for this scenario, but I think this might be the one place where you want a high swappiness (and some swap spaces). This will let Linux move unused bits of memory into swap & let even more ram be used for pagecache.
  - rmccue 3y ago
    On the WordPress side, consider using Batcache to generate appropriate Cache-Control headers. While it’ll also cache into your object cache (memcache/Redis), you can disable or ignore that functionality, and then you get a quite good integrated system. We use it with CloudFront’s page caching.
Saris 3y ago
I've tried OLS before, but it's always slower than my nginx setups for general page load times. nginx + cache is a nice simple solution that just works well.
rmccue 3y ago
> I didn’t know too much about OpenLiteSpeed (“OLS” to its friends) other than that it's mentioned a bunch in discussions about WordPress hosting—and since SCW runs WordPress, I started to get interested This was a trend for a bit but I don’t know of any serious hosts off the top of my head who run LiteSpeed these days. Most are running either nginx or good ol Apache (sometimes both!). (We run enterprise scale WP on nginx, and our competitors at this scale basically all do too.)
- high_priest 3y ago
  What about Caddy, me and my friends have moved onto Caddy over a year ago and haven't experienced any issues.
  - rmccue 3y ago
    I’m not aware of anyone running Caddy at any sort of scale for customers. From what I’ve seen, it tends to be only self-hosted people running it. The overall trend in WP hosting has been away from shared and VPS hosting towards managed hosts, who provide a WordPress-specific environment and do things like upgrade WP and plugins for you. As a result, I can’t imagine there’s a huge percentage of sites using Caddy. (That said, even a small percentage will be millions of sites, since the WP install base is massive.)
    - m_sahaf 3y ago
      > I’m not aware of anyone running Caddy at any sort of scale for customers. Well, to name a few... - Stripe (https://twitter.com/caddyserver/status/1559591673511813120 https://twitter.com/caddyserver/status/1559591673511813120) - Mercedes-Benz (https://github.com/caddyserver/caddy/pull/5275#issuecomment-1367478850 https://github.com/caddyserver/caddy/pull/5275#issuecomment-...) - Approximated.app (https://dev.to/carterbryden/how-to-allow-end-user-custom-domains-in-your-app-4hm0 https://dev.to/carterbryden/how-to-allow-end-user-custom-dom...) - FusionAuth (https://fusionauth.io/blog/unlimited-domains-fusionauth https://fusionauth.io/blog/unlimited-domains-fusionauth) The problem is those using Caddy are shy, not that it's not used at all. I know this because I see users removing the `server` header on the Caddy forum all the time, and many of the large users are just shy of their technology stack when it comes to Caddy. Disclaimer: Member of the Caddy team
ComputerGuru 3y ago
WordPress famously used LiteSpeed (and still does) and the only benefit I know of from the time is that LS has a much better PHP SAPI than fcgi with php-fpm. But now we have nginx unit with native php support (no fcgi), which halves the php-fpm base latency and can be run standalone or behind nginx (via an http proxy over tcp or preferably a Unix socket) so the advantages of an LS (or OLS) configuration are probably non-existent.
- pbowyer 3y ago
  "halves the php-fpm base latency" Have you a source for that? I've been having this discussion (that php-fpm is slower than needed) but I haven't found any hard data.
  - ComputerGuru 3y ago
    There are benchmarks readily available if you search “nginx unit php benchmark” (some that show an order of magnitude reduction in latency) without the quotes, but I hesitate to link to any of them as configuring nginx is highly complicated and has so many knobs, particularly with backend upstreams that can be configured in a myriad of ways. My advice would be to try and see.
- rmccue 3y ago
  > WordPress famously used LiteSpeed (and still does) I presume you mean WordPress.com here (WordPress = open source project, WordPress.com = product by co-founder of WordPress; it's confusing). They did run LiteSpeed a long time ago, but haven't in at least a decade; they moved to Nginx. (There's a comment quoted at https://news.ycombinator.com/item?id=6341947 https://news.ycombinator.com/item?id=6341947 from Barry Abrahamson, who's the .com systems guy - sadly, Wired appears to have removed the original. I also checked with colleagues who were there at the time.) Edit: Also, this HN thread might be useful for further info: https://news.ycombinator.com/item?id=4578258 https://news.ycombinator.com/item?id=4578258
  - ComputerGuru 3y ago
    Either time flies or the information I feel I recently read that WP.com was still using LS (which surprised me) was incorrect. I stand corrected.
spudlyo 3y ago
> ... I was profoundly disappointed that the OLS access log didn’t give me useful information and left me flying blind—the LSPHP processes shouldn’t be pegged to the wall while the access and error logs both show absolutely no requests being serviced of any kind. I was profoundly disappointed that the article didn't go on to describe how strace works, in the service of delivering a satisfying answer as to why the CPU was pegged.
- lee_ars 3y ago
  Sadly, I'm too dumb for that.
timthelion 3y ago
I have to say this is very well written. So much long form stuff is just fluff, but this had some nice narative, a beginning a middle AND an end.
- stuff4ben 3y ago
  It's too bad they didn't delve into the technical details of why the processes were pegging the CPU. Would have loved to see that level of Linux debugging narrated.
  - lee_ars 3y ago
    I would love to be able to narrate that level of linux debugging!!
  - timthelion 3y ago
    I can recommend Ellen Ulman, The Bug. She's a treat.
jszymborski 3y ago
Anyone else confuse LiteSpeed for Lighttpd ("lighty") for a moment? https://en.wikipedia.org/wiki/Lighttpd https://en.wikipedia.org/wiki/Lighttpd
- juangacovas 3y ago
  Ummm no, but just to give credit, I'm using lighttpd for production for years now and happy with latest patches and stuff (supports HTTP/2 and several improvements over time), give it a try
skizm 3y ago
I vaguely recall Nginx being taken over (maybe just founders arrested?) by Russian officials at some point? Is Nginx open source enough where the community would notice any shenanigans Russia might do to compromise the integrity of Nginx? No important parts are black box binaries or anything?
- stefanos82 3y ago
  If you feel insecure about NGINX, you can use OpenResty that bundles LuaJIT and help you push its limits even further.
- kayfox 3y ago
  NGINX the company is owned by F5, Inc, a Seattle, Washington company. All of the open source infrastructure for NGINX is maintained by F5 as a result, and work on NGINX is managed by Maxim Konovalov, an F5 employee. F5 moved all the developers they could from Russian offices to offices in the US, Dubai and Poland a while back. Edit: Also, at the time of acquisition, NGINX, Inc was headquartered in San Jose, California.
  - skizm 3y ago
    Neat, thanks for the info!
trog 3y ago
Really interesting write up. As someone that has spent a lot of time debugging these kinds of issues I feel the pain. I still have a lot of Apache instances lying around just on the basis that the stability, simplicity and consistency on offer outweighs the performance benefits of switching even to nginx. That said I am slowly migrating them to nginx as I get more comfort with it. Always interesting to hear about experiments with other stacks!

83 points by mikece 3y ago

This makes sense. Nginx is reliable and well-tested, and OLS is neither. See my recent Shmoocon talk (yes yes, self promotion) for an example of a vulnerability that would be caught immediately in Nginx, but persisted in the OLS codebase for >=10 years: https://yewtu.be/watch?v=aKPAX00ft5s&t=2h29m28s https://yewtu.be/watch?v=aKPAX00ft5s&t=2h29m28s

mmaunder 3y ago
Really nice work.

whalesalad 3y ago

nginx can be a cache and an ssl termination point so I’m not sure why the author didn’t do this in the first place versus wrapping it in haproxy/varnish.

godisdad 3y ago
maybe as a layer of indirection so nginx and varnish could be independent backends? I wasn’t entire sure of this either
lee_ars 3y ago
Author here. I settled on the haproxy/varnish/nginx sandwich through an accretive process that started back in 2012 with wanting to speed up nginx by taking advantage of scads of extra RAM on the box I was hosting with at the time (and also being entranced by all the cool header games one can play with Varnish). Fast forward to 2015 as LetsEncrypt started to be a thing, and HTTPS started the mainstream push from a nice-to-have to an ought-to-have feature. I was comfy with Varnish + Nginx and I wanted to keep my cache layer, but Varnish has no SSL termination capabilities. I looked at a few options, including Hitch, but decided on HAProxy because it was more widely used and therefore easier to google answers to questions. Everything worked, so I never re-evaluated until 2022 when I got curious about OLS. As to why Varnish + Nginx versus just Nginx + FastCGI or proxy cache, I was chasing the performance dragon. AIUI, Nginx's fastcgi and proxy caches just store the static-ified versions of dynamic content in flat files, and those files are accessed via the file system, which means they get cached into RAM whenever the file system decides they get cached into RAM. I had the free RAM, and Varnish's RAM-first approach felt like it would be faster. Plus, this was all for personal hosting, and playing with Varnish was fun. (Or at least "fun.")
- OJFord 3y ago
  Have you considered caddy?
  - lee_ars 3y ago
    No, I'm happy to retreat to the familiar, and nginx is pretty familiar. After having my tail feathers burned with OLS, I doubt I'll switch to anything else for the next several years without an extremely compelling reason.
    - OJFord 3y ago
      Sure I'm not suggesting it now as a follow up, I just wondered if it was something you were aware of and dismissed for some reason over the last several years before opting to try OLS (which I hadn't heard of) instead; especially since you mentioned Let's Encrypt.
    - deleted 3y ago
      [deleted]
- jauntywundrkind 3y ago
  Thanks for the walkthrough. One particular comment: > "Nginx's fastcgi and proxy caches just store the static-ified versions of dynamic content in flat files, and those files are accessed via the file system, which means they get cached into RAM whenever the file system decides they get cached into RAM. I had the free RAM, and Varnish's RAM-first approach felt like it would be faster." The pagecache should do a pretty good job holding file contents in memory for you! https://tldp.org/LDP/sag/html/buffer-cache.html https://tldp.org/LDP/sag/html/buffer-cache.html I haven't had to tune for this scenario, but I think this might be the one place where you want a high swappiness (and some swap spaces). This will let Linux move unused bits of memory into swap & let even more ram be used for pagecache.
- rmccue 3y ago
  On the WordPress side, consider using Batcache to generate appropriate Cache-Control headers. While it’ll also cache into your object cache (memcache/Redis), you can disable or ignore that functionality, and then you get a quite good integrated system. We use it with CloudFront’s page caching.

Saris 3y ago

I've tried OLS before, but it's always slower than my nginx setups for general page load times. nginx + cache is a nice simple solution that just works well.

rmccue 3y ago

> I didn’t know too much about OpenLiteSpeed (“OLS” to its friends) other than that it's mentioned a bunch in discussions about WordPress hosting—and since SCW runs WordPress, I started to get interested This was a trend for a bit but I don’t know of any serious hosts off the top of my head who run LiteSpeed these days. Most are running either nginx or good ol Apache (sometimes both!). (We run enterprise scale WP on nginx, and our competitors at this scale basically all do too.)

high_priest 3y ago
What about Caddy, me and my friends have moved onto Caddy over a year ago and haven't experienced any issues.
- rmccue 3y ago
  I’m not aware of anyone running Caddy at any sort of scale for customers. From what I’ve seen, it tends to be only self-hosted people running it. The overall trend in WP hosting has been away from shared and VPS hosting towards managed hosts, who provide a WordPress-specific environment and do things like upgrade WP and plugins for you. As a result, I can’t imagine there’s a huge percentage of sites using Caddy. (That said, even a small percentage will be millions of sites, since the WP install base is massive.)
  - m_sahaf 3y ago
    > I’m not aware of anyone running Caddy at any sort of scale for customers. Well, to name a few... - Stripe (https://twitter.com/caddyserver/status/1559591673511813120 https://twitter.com/caddyserver/status/1559591673511813120) - Mercedes-Benz (https://github.com/caddyserver/caddy/pull/5275#issuecomment-1367478850 https://github.com/caddyserver/caddy/pull/5275#issuecomment-...) - Approximated.app (https://dev.to/carterbryden/how-to-allow-end-user-custom-domains-in-your-app-4hm0 https://dev.to/carterbryden/how-to-allow-end-user-custom-dom...) - FusionAuth (https://fusionauth.io/blog/unlimited-domains-fusionauth https://fusionauth.io/blog/unlimited-domains-fusionauth) The problem is those using Caddy are shy, not that it's not used at all. I know this because I see users removing the `server` header on the Caddy forum all the time, and many of the large users are just shy of their technology stack when it comes to Caddy. Disclaimer: Member of the Caddy team

ComputerGuru 3y ago

WordPress famously used LiteSpeed (and still does) and the only benefit I know of from the time is that LS has a much better PHP SAPI than fcgi with php-fpm. But now we have nginx unit with native php support (no fcgi), which halves the php-fpm base latency and can be run standalone or behind nginx (via an http proxy over tcp or preferably a Unix socket) so the advantages of an LS (or OLS) configuration are probably non-existent.

pbowyer 3y ago
"halves the php-fpm base latency" Have you a source for that? I've been having this discussion (that php-fpm is slower than needed) but I haven't found any hard data.
- ComputerGuru 3y ago
  There are benchmarks readily available if you search “nginx unit php benchmark” (some that show an order of magnitude reduction in latency) without the quotes, but I hesitate to link to any of them as configuring nginx is highly complicated and has so many knobs, particularly with backend upstreams that can be configured in a myriad of ways. My advice would be to try and see.
rmccue 3y ago
> WordPress famously used LiteSpeed (and still does) I presume you mean WordPress.com here (WordPress = open source project, WordPress.com = product by co-founder of WordPress; it's confusing). They did run LiteSpeed a long time ago, but haven't in at least a decade; they moved to Nginx. (There's a comment quoted at https://news.ycombinator.com/item?id=6341947 https://news.ycombinator.com/item?id=6341947 from Barry Abrahamson, who's the .com systems guy - sadly, Wired appears to have removed the original. I also checked with colleagues who were there at the time.) Edit: Also, this HN thread might be useful for further info: https://news.ycombinator.com/item?id=4578258 https://news.ycombinator.com/item?id=4578258
- ComputerGuru 3y ago
  Either time flies or the information I feel I recently read that WP.com was still using LS (which surprised me) was incorrect. I stand corrected.

spudlyo 3y ago

> ... I was profoundly disappointed that the OLS access log didn’t give me useful information and left me flying blind—the LSPHP processes shouldn’t be pegged to the wall while the access and error logs both show absolutely no requests being serviced of any kind. I was profoundly disappointed that the article didn't go on to describe how strace works, in the service of delivering a satisfying answer as to why the CPU was pegged.

lee_ars 3y ago
Sadly, I'm too dumb for that.

timthelion 3y ago

I have to say this is very well written. So much long form stuff is just fluff, but this had some nice narative, a beginning a middle AND an end.

stuff4ben 3y ago
It's too bad they didn't delve into the technical details of why the processes were pegging the CPU. Would have loved to see that level of Linux debugging narrated.
- lee_ars 3y ago
  I would love to be able to narrate that level of linux debugging!!
- timthelion 3y ago
  I can recommend Ellen Ulman, The Bug. She's a treat.

jszymborski 3y ago

Anyone else confuse LiteSpeed for Lighttpd ("lighty") for a moment? https://en.wikipedia.org/wiki/Lighttpd https://en.wikipedia.org/wiki/Lighttpd

juangacovas 3y ago
Ummm no, but just to give credit, I'm using lighttpd for production for years now and happy with latest patches and stuff (supports HTTP/2 and several improvements over time), give it a try

skizm 3y ago

I vaguely recall Nginx being taken over (maybe just founders arrested?) by Russian officials at some point? Is Nginx open source enough where the community would notice any shenanigans Russia might do to compromise the integrity of Nginx? No important parts are black box binaries or anything?

stefanos82 3y ago
If you feel insecure about NGINX, you can use OpenResty that bundles LuaJIT and help you push its limits even further.
kayfox 3y ago
NGINX the company is owned by F5, Inc, a Seattle, Washington company. All of the open source infrastructure for NGINX is maintained by F5 as a result, and work on NGINX is managed by Maxim Konovalov, an F5 employee. F5 moved all the developers they could from Russian offices to offices in the US, Dubai and Poland a while back. Edit: Also, at the time of acquisition, NGINX, Inc was headquartered in San Jose, California.
- skizm 3y ago
  Neat, thanks for the info!

trog 3y ago

Really interesting write up. As someone that has spent a lot of time debugging these kinds of issues I feel the pain. I still have a lot of Apache instances lying around just on the basis that the stability, simplicity and consistency on offer outweighs the performance benefits of switching even to nginx. That said I am slowly migrating them to nginx as I get more comfort with it. Always interesting to hear about experiments with other stacks!