5 ms·

WordPress VIP sites have been hit by outages

76 points by sidyapa 7y ago

lioeters 7y ago
Here's a plain version without the Oath consent form: https://outline.com/BhZZTZ https://outline.com/BhZZTZ This one might be a better article: https://news.ycombinator.com/item?id=20158002 https://news.ycombinator.com/item?id=20158002
itsaidpens 7y ago
We host our sites on Pantheon - I've loved them. Check them out - pantheon.io
- Ayesh 7y ago
  I know many Pantheon people closely and had seen what they do. It's a good platform, but WordPress.com platform tend to be a lot more mature and reliable, despite this incident.
benbristow 7y ago
Rolling Stone are currently on the default Twenty Seventeen Wordpress theme. https://rollingstone.com/ https://rollingstone.com/ Quite amusing.
- jesseb 7y ago
  Hackaday is borked too. https://hackaday.com/ https://hackaday.com/
JMTQp8lwXL 7y ago
Aren't most of these sites fronted by CDNs that should solve that problem?
- minimaxir 7y ago
  WordPress blogs/themes are not static assets.
  - JMTQp8lwXL 7y ago
    You can still front non-static assets by a CDN. Just update the CDN when the asset changes, or add a query parameter to requests (e.g, ?v=2) when changing things like your favicon.
  - whalesalad 7y ago
    They are often rendered once and served until the underlying content changes or some kind of TTL expires.
  - orf 7y ago
    They kind of are though.
  - nodesocket 7y ago
    Typical Wordpress sites that are built as a pure CMS are absolutely cacheable (not just JS, CSS, images), but the html can be cached of course. The problem occurs with commenting, and dynamic data features.
hbcondo714 7y ago
We have some sites hosted under WordPress.com's Business Plan; luckily it looks like we are not affected by this.
AznHisoka 7y ago
Can't this outage not be fixed? I'm seeing much faster load times!
nkg 7y ago
Bad advertisment
- dang 7y ago
  Maybe so, but could you please stop posting unsubstantive comments to HN?
r721 7y ago
Updates: https://lobby.vip.wordpress.com/2019/06/11/503-errors-and-availability-issues-for-vip-go-sites/ https://lobby.vip.wordpress.com/2019/06/11/503-errors-and-av...
kartan 7y ago
Diversity in nature or technology is important. And it's worrisome that both are disappearing. "The resulting homogeneity and uniformity can offer substantial advantages in both the quantity and quality of crop harvested, but this same genetic homogeneity can also reflect greater susceptibility or pathogens. Thus it appears the more that agricultural selection disturbs the natural balance in favor of variety uniformity over large areas, the more vulnerable such varieties are to losses from epidemics." https://www.nap.edu/read/2116/chapter/5 https://www.nap.edu/read/2116/chapter/5 - https://www.theregister.co.uk/2018/06/01/aws_outage/ https://www.theregister.co.uk/2018/06/01/aws_outage/ - https://www.theverge.com/2019/6/2/18649635/youtube-snapchat-down-outage https://www.theverge.com/2019/6/2/18649635/youtube-snapchat-... - https://www.theregister.co.uk/2019/06/11/wordpress_vip_go_outage/ https://www.theregister.co.uk/2019/06/11/wordpress_vip_go_ou... And if we take the word of Steve Jobs on this one we are for a hard ride: "It turns out the same thing can happen in technology companies that get monopolies, like IBM or Xerox. If you were a product person at IBM or Xerox, so you make a better copier or computer. So what? When you have monopoly market share, the company's not any more successful. So the people that can make the company more successful are sales and marketing people, and they end up running the companies. And the product people get driven out of the decision making forums, and the companies forget what it means to make great products. The product sensibility and the product genius that brought them to that monopolistic position gets rotted out by people running these companies that have no conception of a good product versus a bad product. They have no conception of the craftsmanship that's required to take a good idea and turn it into a good product. And they really have no feeling in their hearts, usually, about wanting to really help the customers." https://www.businessinsider.com/steve-jobs-on-why-innovation-dies-at-tech-monopolies-2014-11?r=US&IR=T https://www.businessinsider.com/steve-jobs-on-why-innovation... With our economy and daily lives so dependent in a few companies, any drop in quality will impact all corners of society.
- bsder 7y ago
  Maximally efficient is minimally robust.
- username444 7y ago
  I'm not sure why you think hosting diversity is disappearing when it comes to WordPress. The past 2 years has seen a huge surge in great companies offering premium managed WordPress hosting services. Liquidweb, cloudways, wpengine, getflywheel, pantheon. Granted, many use google cloud or similar for actual hosting rather than their own metal... But the services built on top have never been better.
jddj 7y ago
Hackaday's case was interesting, they quickly hacked (sorry) a fix together and by mistake ended up piping all comments into the RSS feed. https://hackaday.com/2019/06/11/the-day-hackadays-theme-was-broken/ https://hackaday.com/2019/06/11/the-day-hackadays-theme-was-...
aussieguy1234 7y ago
I used to work on a VIP site, it was their largest customer and we got 600 million monthly pageviews. Scaling WordPress to that level is a challenge, but with multiple levels of network and software caching, it worked. My particular sites wouldn't be down, because of akamai fallback.
- tuananh 7y ago
  i had a site with ~ 10m pageviews monthly that used to run on a $20 DigitalOcean instance. What kind of things you need to do to scale that up to 600m PV monthly?
  - blowski 7y ago
    Depends on whether the majority of views are on static blog posts, as they can be cached and served from nginx. But if you’ve got a lot of dynamic views - comments, admin, personalised - it’s much harder. Still impressive you managed to get that level of performance out of a $20 DO box.
    - toast0 7y ago
      10m page views really doesn't look that big if you bring it down to pageviews/second and there's some reasonable distribution. Assuming a busy day is 1M, and that a busy hour of the busy day has 100,000, but over the hour is evenly distributed, you're looking at under 30 page views per second. Maybe peak is a little more. If this is tuananh's current site, it's Jekyl, so it's all static -- you could probably host that on a 486 (as long as there aren't any big images to clog the nic). If it's something with a little bit of dynamic content, you might need to be careful about how you structure it, but something reasonable should be fine at that level of traffic too, just need to make sure you don't spend a lot of cpu doing useless work (like some frameworks that take 60ms to output hello world)
  - aussieguy1234 7y ago
    WordPress has alot of filters that allow you to take over/modify just about anything including internal functions. We overrode get_posts() to use ElasticSearch instead of MySQL and added lots of caching to alot of things with Memcache. I built a plugin which cached all of the HTML output of every WordPress page and all of the widgets in Memcache. On top of all the server side caching, there was Akamai in front caching every page at the network level.

76 points by sidyapa 7y ago

Here's a plain version without the Oath consent form: https://outline.com/BhZZTZ https://outline.com/BhZZTZ This one might be a better article: https://news.ycombinator.com/item?id=20158002 https://news.ycombinator.com/item?id=20158002

itsaidpens 7y ago

We host our sites on Pantheon - I've loved them. Check them out - pantheon.io

Ayesh 7y ago
I know many Pantheon people closely and had seen what they do. It's a good platform, but WordPress.com platform tend to be a lot more mature and reliable, despite this incident.

benbristow 7y ago

Rolling Stone are currently on the default Twenty Seventeen Wordpress theme. https://rollingstone.com/ https://rollingstone.com/ Quite amusing.

jesseb 7y ago
Hackaday is borked too. https://hackaday.com/ https://hackaday.com/

JMTQp8lwXL 7y ago

Aren't most of these sites fronted by CDNs that should solve that problem?

minimaxir 7y ago
WordPress blogs/themes are not static assets.
- JMTQp8lwXL 7y ago
  You can still front non-static assets by a CDN. Just update the CDN when the asset changes, or add a query parameter to requests (e.g, ?v=2) when changing things like your favicon.
- whalesalad 7y ago
  They are often rendered once and served until the underlying content changes or some kind of TTL expires.
- orf 7y ago
  They kind of are though.
- nodesocket 7y ago
  Typical Wordpress sites that are built as a pure CMS are absolutely cacheable (not just JS, CSS, images), but the html can be cached of course. The problem occurs with commenting, and dynamic data features.

hbcondo714 7y ago

We have some sites hosted under WordPress.com's Business Plan; luckily it looks like we are not affected by this.

AznHisoka 7y ago

Can't this outage not be fixed? I'm seeing much faster load times!

nkg 7y ago

Bad advertisment

dang 7y ago
Maybe so, but could you please stop posting unsubstantive comments to HN?

r721 7y ago

Updates: https://lobby.vip.wordpress.com/2019/06/11/503-errors-and-availability-issues-for-vip-go-sites/ https://lobby.vip.wordpress.com/2019/06/11/503-errors-and-av...

kartan 7y ago

Diversity in nature or technology is important. And it's worrisome that both are disappearing. "The resulting homogeneity and uniformity can offer substantial advantages in both the quantity and quality of crop harvested, but this same genetic homogeneity can also reflect greater susceptibility or pathogens. Thus it appears the more that agricultural selection disturbs the natural balance in favor of variety uniformity over large areas, the more vulnerable such varieties are to losses from epidemics." https://www.nap.edu/read/2116/chapter/5 https://www.nap.edu/read/2116/chapter/5 - https://www.theregister.co.uk/2018/06/01/aws_outage/ https://www.theregister.co.uk/2018/06/01/aws_outage/ - https://www.theverge.com/2019/6/2/18649635/youtube-snapchat-down-outage https://www.theverge.com/2019/6/2/18649635/youtube-snapchat-... - https://www.theregister.co.uk/2019/06/11/wordpress_vip_go_outage/ https://www.theregister.co.uk/2019/06/11/wordpress_vip_go_ou... And if we take the word of Steve Jobs on this one we are for a hard ride: "It turns out the same thing can happen in technology companies that get monopolies, like IBM or Xerox. If you were a product person at IBM or Xerox, so you make a better copier or computer. So what? When you have monopoly market share, the company's not any more successful. So the people that can make the company more successful are sales and marketing people, and they end up running the companies. And the product people get driven out of the decision making forums, and the companies forget what it means to make great products. The product sensibility and the product genius that brought them to that monopolistic position gets rotted out by people running these companies that have no conception of a good product versus a bad product. They have no conception of the craftsmanship that's required to take a good idea and turn it into a good product. And they really have no feeling in their hearts, usually, about wanting to really help the customers." https://www.businessinsider.com/steve-jobs-on-why-innovation-dies-at-tech-monopolies-2014-11?r=US&IR=T https://www.businessinsider.com/steve-jobs-on-why-innovation... With our economy and daily lives so dependent in a few companies, any drop in quality will impact all corners of society.

bsder 7y ago
Maximally efficient is minimally robust.
username444 7y ago
I'm not sure why you think hosting diversity is disappearing when it comes to WordPress. The past 2 years has seen a huge surge in great companies offering premium managed WordPress hosting services. Liquidweb, cloudways, wpengine, getflywheel, pantheon. Granted, many use google cloud or similar for actual hosting rather than their own metal... But the services built on top have never been better.

jddj 7y ago

Hackaday's case was interesting, they quickly hacked (sorry) a fix together and by mistake ended up piping all comments into the RSS feed. https://hackaday.com/2019/06/11/the-day-hackadays-theme-was-broken/ https://hackaday.com/2019/06/11/the-day-hackadays-theme-was-...

aussieguy1234 7y ago

I used to work on a VIP site, it was their largest customer and we got 600 million monthly pageviews. Scaling WordPress to that level is a challenge, but with multiple levels of network and software caching, it worked. My particular sites wouldn't be down, because of akamai fallback.

tuananh 7y ago
i had a site with ~ 10m pageviews monthly that used to run on a $20 DigitalOcean instance. What kind of things you need to do to scale that up to 600m PV monthly?
- blowski 7y ago
  Depends on whether the majority of views are on static blog posts, as they can be cached and served from nginx. But if you’ve got a lot of dynamic views - comments, admin, personalised - it’s much harder. Still impressive you managed to get that level of performance out of a $20 DO box.
  - toast0 7y ago
    10m page views really doesn't look that big if you bring it down to pageviews/second and there's some reasonable distribution. Assuming a busy day is 1M, and that a busy hour of the busy day has 100,000, but over the hour is evenly distributed, you're looking at under 30 page views per second. Maybe peak is a little more. If this is tuananh's current site, it's Jekyl, so it's all static -- you could probably host that on a 486 (as long as there aren't any big images to clog the nic). If it's something with a little bit of dynamic content, you might need to be careful about how you structure it, but something reasonable should be fine at that level of traffic too, just need to make sure you don't spend a lot of cpu doing useless work (like some frameworks that take 60ms to output hello world)
- aussieguy1234 7y ago
  WordPress has alot of filters that allow you to take over/modify just about anything including internal functions. We overrode get_posts() to use ElasticSearch instead of MySQL and added lots of caching to alot of things with Memcache. I built a plugin which cached all of the HTML output of every WordPress page and all of the widgets in Memcache. On top of all the server side caching, there was Akamai in front caching every page at the network level.