10 ms·

This page is returning a 500 for me. Perhaps they should use Kubernetes.

by mjlee 5y ago

chrisandchris 5y ago
Probably a classic Hug of Death caused by being #1 on HN.
- intev 5y ago
  I'm actually curious as to how big of an issue this actually is. HN is a relatively niche site. Do we actually have the power to take down a blog of a tech company?
  - deleted 5y ago
    [deleted]
  - codetrotter 5y ago
    Happens regularly. I am surprised you haven’t seen it happen before, given that you’ve been on HN since at least 2011.
    - intev 5y ago
      I moved and live in a different TZ now. Not in the American TZ. That's probably why.
- IceWreck 5y ago
  Even a $5 VPS can easily handle like 10k page views per minute if you use a performant web server i.e. not PHP or Ruby. Or even those with caching.
  - _joel 5y ago
    Caching is the key here, it could still be making database requests which are blocked, unless it's using async.
    - repsilat 5y ago
      DB queries in blogs (especially with this kind of traffic spike) are super correlated, you probably don't even need a cache if you batch identical reads together when the requests arrive within say 10 or 100ms of each other. Not a common technique though. A proof of concept demo: https://github.com/MatthewSteel/carpool https://github.com/MatthewSteel/carpool
  - handrous 5y ago
    > use a performant web server i.e. not PHP or Ruby. PHP in an ordinary deployment behind Apache2 or nginx is very fast. If you're writing a Web service and your top two criteria for the language are "fast" and "scripting language" and PHP doesn't at least make your short-list, you've likely screwed up. It's the one to beat—you pretty much have to go with something compiled to have a good chance of beating it at serving dynamic responses to ordinary Web requests, and even then it's easy to screw up in minor ways and end up slower. Persistent connections, sure, maybe you want something else, but basic request-response stuff? Performance is not a reason to avoid it, unless your other candidate languages are C++ or very well-tuned and carefully written Java or Go, something along those lines. Speed-wise, it is certainly in a whole other category from Ruby or Python (in fact, even "speedy" Nodejs struggles to match it). Now, WordPress is slow because a top design goal is that it's infinitely at-runtime extensible. And lots of PHP sites are slow for the same reason that lots of Ruby, or Node, or Python, or Java, or .Net, or whatever, sites are slow: the developer was very bad at querying databases.
defaultname 5y ago
I know you're being facetious and I chuckled, but it's pretty amazing to see sites fall over under even a moderate load in 2021. A simple Go web server running on a pretty low-end machine can handle an HN-sized torrent with ease, even before sticking a CDN in front of it.
- gtirloni 5y ago
  It's usually a WordPress/Joomla/Magento website that can do 1-5 requests/second at most. EDIT: I'm not saying these softwares can't do more, it's just that usually on most default configurations, people don't bother to use a CDN, optimize the database, use a caching plugin, etc. They just install it, then install another 39 plugins and then ask why everything is slow. It's very common to see WordPress websites failing under load (and I've helped many to optimize their installs so I know it's possible, it's just not what the average WordPress install looks like).
  - briffle 5y ago
    In this case, its appears to be their main web-page as well.
  - amerkhalid 5y ago
    I used to run WordPress site on $5 vps. It could easily handle 100-500 request per second, the trick was to have a caching plugin.
    - tuananh 5y ago
      Agreed. With a very aggressive caching setup, WP can actually handle it pretty well. I used to have a 5mil pageviews per month site running on 5$ instance from DigitalOcean
    - manquer 5y ago
      For 4 years I had Static site generated in Hugo git/CI deployed to CloudFront/S3. Then the team wanted to move and moved to hubspot, it was simply harder to find and train marketing talent who is ready to learn git or even HTML. Far easier to find hubspot/WordPress designer/developers/content writers . Also it becomes easy to blame on the unfamiliar thing your company uses when things don't work for them the way they want it to. I have known teams move to back to very simple default wordpress hosting from more advanced stacks. My point is it harder to get your marketing dept to upgrade tech even if we could do it. I have learnt over the years that solutions that have to be for the people who will use everyday, even if it's poorly managed WordPress. In your example You or your SRE/devops will do all the basic configuration tunning pretty much out of the box. Your IT dept may not be able to at all or unless someone tells them to
arcturus17 5y ago
Yea, or at least something different than what they’re using… You gotta admit, it’s a pretty bad look to come here with a boldly titled systems architecture blog post and have your website crash… A junior web developer nowadays can easily build a system to serve static content that avoids all but the worst hugs of death. If this wasn’t a hug of death then you have service availability issues, regardless…
tifadg1 5y ago
It's ironic they're preaching technical excellence from their high horse, yet their website is returning errors and the status page incorrectly shows everything is working as expected.
tyingq 5y ago
Interesting, as it's Cloudflare fronted, but returns "Cache-Control: max-age=0, private, must-revalidate" in the headers, killing any browser caching.
- zdragnar 5y ago
  Should the browser be caching 500 responses?
  - tyingq 5y ago
    No, this is from one of the intermittent successful responses. You can see it on https://ably.com/support https://ably.com/support also, which seems to be still up. And cloudflare caching of some previous successful page fetch seems to be turned off as well. A cf-cache-status header of "DYNAMIC" means "This resource is not cached by default and there are no explicit settings configured to cache it.".
- tatersolid 5y ago
  WPengine is configured like that, and you can’t change it, even on their highest tiers. Their support doesn’t understand that it defeats the whole purpose of WPengine’s reseller relationship with cloudflare.
oefrha 5y ago
Put your blog on a separate domain blog.company.tld and throw it behind Cloudflare free tier or something. It's pretty amazing how many companies fail to do that, and a moderately viral blog post would bring down their whole damn site when it's the critical moment to convert customers. Having this happen to a post bragging about infrastructure is just adding insult to injury. Edit: As another comment mentioned, this site is actually behind Cloudflare, with cf-cache-status: DYNAMIC (i.e. caching is disabled). I don't know what to say...
- handrous 5y ago
  CF doesn't cache html by default. You have to add a page rule to do it. If your html being dynamically generated is most of your load (it probably is, even if it's not most of your bandwidth—serving plain files with Apache or Nginx or whatever is incredibly cheap, processor-wise) this means CF doesn't help much unless you tell it to.
nickdothutton 5y ago
Or make simpler pages. It’s possible to achieve a modern responsive design for 250kb per page and <25 elements even for someone like me who is far from this part of the stack in interest and ability. If they sold a 256MB LightSail I’d buy it.
- alpaca128 5y ago
  It is possible. If they didn't include well over 1MB of scripts that don't seem to do much but are an order of magnitude fatter than the PNG images on the site. I don't even know what those scripts are supposed to do. This is a static blog post.
rvz 5y ago
For a blog? That is not their main service is it? You don't need Kubernetes for a blog. If you do then something is wrong with your whole backend stack. Perhaps they are the smart ones who want to save money and not burn millions of VC money. The one that REALLY needs to use Kubernetes is actually GitHub. [0] [0] https://news.ycombinator.com/item?id=27366397 https://news.ycombinator.com/item?id=27366397
- cristaloleg 5y ago
  Do you pay for Github? just curious.
dave_sid 5y ago
Best HN comment ever
rob_c 5y ago
I did think the same
taylodl 5y ago
Which raises the question why would I listen to anything else they have to say when they've just demonstrated their inability to scale an even modest amount?
throwawaymanbot 5y ago
Working for me. No 500.

This page is returning a 500 for me. Perhaps they should use Kubernetes.

by mjlee 5y ago

This page is returning a 500 for me. Perhaps they should use Kubernetes.

chrisandchris 5y ago

Probably a classic Hug of Death caused by being #1 on HN.

intev 5y ago
I'm actually curious as to how big of an issue this actually is. HN is a relatively niche site. Do we actually have the power to take down a blog of a tech company?
- deleted 5y ago
  [deleted]
- codetrotter 5y ago
  Happens regularly. I am surprised you haven’t seen it happen before, given that you’ve been on HN since at least 2011.
  - intev 5y ago
    I moved and live in a different TZ now. Not in the American TZ. That's probably why.
IceWreck 5y ago
Even a $5 VPS can easily handle like 10k page views per minute if you use a performant web server i.e. not PHP or Ruby. Or even those with caching.
- _joel 5y ago
  Caching is the key here, it could still be making database requests which are blocked, unless it's using async.
  - repsilat 5y ago
    DB queries in blogs (especially with this kind of traffic spike) are super correlated, you probably don't even need a cache if you batch identical reads together when the requests arrive within say 10 or 100ms of each other. Not a common technique though. A proof of concept demo: https://github.com/MatthewSteel/carpool https://github.com/MatthewSteel/carpool
- handrous 5y ago
  > use a performant web server i.e. not PHP or Ruby. PHP in an ordinary deployment behind Apache2 or nginx is very fast. If you're writing a Web service and your top two criteria for the language are "fast" and "scripting language" and PHP doesn't at least make your short-list, you've likely screwed up. It's the one to beat—you pretty much have to go with something compiled to have a good chance of beating it at serving dynamic responses to ordinary Web requests, and even then it's easy to screw up in minor ways and end up slower. Persistent connections, sure, maybe you want something else, but basic request-response stuff? Performance is not a reason to avoid it, unless your other candidate languages are C++ or very well-tuned and carefully written Java or Go, something along those lines. Speed-wise, it is certainly in a whole other category from Ruby or Python (in fact, even "speedy" Nodejs struggles to match it). Now, WordPress is slow because a top design goal is that it's infinitely at-runtime extensible. And lots of PHP sites are slow for the same reason that lots of Ruby, or Node, or Python, or Java, or .Net, or whatever, sites are slow: the developer was very bad at querying databases.

defaultname 5y ago

I know you're being facetious and I chuckled, but it's pretty amazing to see sites fall over under even a moderate load in 2021. A simple Go web server running on a pretty low-end machine can handle an HN-sized torrent with ease, even before sticking a CDN in front of it.

gtirloni 5y ago
It's usually a WordPress/Joomla/Magento website that can do 1-5 requests/second at most. EDIT: I'm not saying these softwares can't do more, it's just that usually on most default configurations, people don't bother to use a CDN, optimize the database, use a caching plugin, etc. They just install it, then install another 39 plugins and then ask why everything is slow. It's very common to see WordPress websites failing under load (and I've helped many to optimize their installs so I know it's possible, it's just not what the average WordPress install looks like).
- briffle 5y ago
  In this case, its appears to be their main web-page as well.
- amerkhalid 5y ago
  I used to run WordPress site on $5 vps. It could easily handle 100-500 request per second, the trick was to have a caching plugin.
  - tuananh 5y ago
    Agreed. With a very aggressive caching setup, WP can actually handle it pretty well. I used to have a 5mil pageviews per month site running on 5$ instance from DigitalOcean
  - manquer 5y ago
    For 4 years I had Static site generated in Hugo git/CI deployed to CloudFront/S3. Then the team wanted to move and moved to hubspot, it was simply harder to find and train marketing talent who is ready to learn git or even HTML. Far easier to find hubspot/WordPress designer/developers/content writers . Also it becomes easy to blame on the unfamiliar thing your company uses when things don't work for them the way they want it to. I have known teams move to back to very simple default wordpress hosting from more advanced stacks. My point is it harder to get your marketing dept to upgrade tech even if we could do it. I have learnt over the years that solutions that have to be for the people who will use everyday, even if it's poorly managed WordPress. In your example You or your SRE/devops will do all the basic configuration tunning pretty much out of the box. Your IT dept may not be able to at all or unless someone tells them to

arcturus17 5y ago

Yea, or at least something different than what they’re using… You gotta admit, it’s a pretty bad look to come here with a boldly titled systems architecture blog post and have your website crash… A junior web developer nowadays can easily build a system to serve static content that avoids all but the worst hugs of death. If this wasn’t a hug of death then you have service availability issues, regardless…

tifadg1 5y ago

It's ironic they're preaching technical excellence from their high horse, yet their website is returning errors and the status page incorrectly shows everything is working as expected.

tyingq 5y ago

Interesting, as it's Cloudflare fronted, but returns "Cache-Control: max-age=0, private, must-revalidate" in the headers, killing any browser caching.

zdragnar 5y ago
Should the browser be caching 500 responses?
- tyingq 5y ago
  No, this is from one of the intermittent successful responses. You can see it on https://ably.com/support https://ably.com/support also, which seems to be still up. And cloudflare caching of some previous successful page fetch seems to be turned off as well. A cf-cache-status header of "DYNAMIC" means "This resource is not cached by default and there are no explicit settings configured to cache it.".
tatersolid 5y ago
WPengine is configured like that, and you can’t change it, even on their highest tiers. Their support doesn’t understand that it defeats the whole purpose of WPengine’s reseller relationship with cloudflare.

oefrha 5y ago

Put your blog on a separate domain blog.company.tld and throw it behind Cloudflare free tier or something. It's pretty amazing how many companies fail to do that, and a moderately viral blog post would bring down their whole damn site when it's the critical moment to convert customers. Having this happen to a post bragging about infrastructure is just adding insult to injury. Edit: As another comment mentioned, this site is actually behind Cloudflare, with cf-cache-status: DYNAMIC (i.e. caching is disabled). I don't know what to say...

handrous 5y ago
CF doesn't cache html by default. You have to add a page rule to do it. If your html being dynamically generated is most of your load (it probably is, even if it's not most of your bandwidth—serving plain files with Apache or Nginx or whatever is incredibly cheap, processor-wise) this means CF doesn't help much unless you tell it to.

nickdothutton 5y ago

Or make simpler pages. It’s possible to achieve a modern responsive design for 250kb per page and <25 elements even for someone like me who is far from this part of the stack in interest and ability. If they sold a 256MB LightSail I’d buy it.

alpaca128 5y ago
It is possible. If they didn't include well over 1MB of scripts that don't seem to do much but are an order of magnitude fatter than the PNG images on the site. I don't even know what those scripts are supposed to do. This is a static blog post.

rvz 5y ago

For a blog? That is not their main service is it? You don't need Kubernetes for a blog. If you do then something is wrong with your whole backend stack. Perhaps they are the smart ones who want to save money and not burn millions of VC money. The one that REALLY needs to use Kubernetes is actually GitHub. [0] [0] https://news.ycombinator.com/item?id=27366397 https://news.ycombinator.com/item?id=27366397

cristaloleg 5y ago
Do you pay for Github? just curious.

dave_sid 5y ago

Best HN comment ever

rob_c 5y ago

I did think the same

taylodl 5y ago

Which raises the question why would I listen to anything else they have to say when they've just demonstrated their inability to scale an even modest amount?

throwawaymanbot 5y ago

Working for me. No 500.