4 ms·

Netlify CDN Outage

70 points by ajsharp 7y ago

chrisblackwell 7y ago
I understand the growing pains Netlify must be going through. They have created a great product, and are becoming very popular. I'd love to see the Enterprise version (or any paid version) segregated out from the free sites so we can get more reliable uptime. The build pipeline was down for several hours a few days ago, now a CDN outage makes things much worse, as it's taken down all our sites. I'm sure they will resolve things quickly, in the meantime our clients will start asking about other failover options.
- drewda 7y ago
  According to their overall status page, they already do run two separate CDNs for normal and enterprise customers: https://www.netlifystatus.com/ https://www.netlifystatus.com/
- luhn 7y ago
  Ostensibly they do seem to have that. > Our non-enterprise CDN is failing a small number of page loads, usually with SSL errors; Enterprise CDN is not affected. But looks like there's room for improvement, because the enterprise CDN ended up failing as well. > We are still working to mitigate the effects of these problems and do see some page load failures on the enterprise CDN as well.
gravypod 7y ago
It'd be really cool to see a writeup of what is going on at Netlify.
- nwsm 7y ago
  They are tweeting about it, and I'd expect a post mortem as they have been transparent in the past.
tschellenbach 7y ago
Yup, Stream's blog is down as well. Thought it was WPEngine at first, but turns out it's netlify.
ajsharp 7y ago
netlify.com appears to be affected as well. Curious if this is a software issue or a host issue. I believe netlify rents a lot of its infrastructure from other providers, such as Digital Ocean.
- cfors 7y ago
  At any sort of scale (which I have to imagine netlify is running at), this will be a software issue since hosts are designed to fail. Serving static content is crazy easy for computers, my guess is a bad configuration got pushed out.
- altmind 7y ago
  Here I saw some mentions that it may be related to recent google cloud problems https://news.ycombinator.com/item?id=21418979 https://news.ycombinator.com/item?id=21418979
nwsm 7y ago
My site is back up. (1:13 EST) EDIT: intermittent issues now. Sometimes nothing loads, sometimes everything, sometimes SSL issue, sometimes HTML but no CSS or JS load. (1:22 EST)
losvedir 7y ago
Huh, loaded HN because it seems YouTube videos are down for me. I wonder if it's related.
- nwsm 7y ago
  Another comment claimed that Netlify rents servers from other providers, so this could be the case. FWIW Youtube is fine for me right now.
epoch_100 7y ago
Netlify is describing the outage as being the result of some kind of attack: "We’re currently suffering an attack that’s affecting our Free and Self-Serve CDN. Enterprise is currently unaffected. We are working on mitigating the effects and will update with more information as it becomes available."
volkk 7y ago
insane coincidence but literally the minute i started my first project ever on netlify just to play with it, i deployed it and nothing was loading; i kept googling and thinking it was my fault and then it clicked to check their status page and alas.
- jagannathtech 7y ago
  So it was YOU who took down the network then
twobat 7y ago
Bad publicity is still publicity, right?
swaraj 7y ago
This affects us too. We were able to migrate to zeit now in ~30 mins.
- ngrilly 7y ago
  Will you migrate back to Netlify when Zeit will suffer a similar downtime?
  - swaraj 7y ago
    We're definitely staying on Netlify after this outage is done. We pay for some of their enterprise features, e.g. audit logs. We just need our sites up right now and zeit was a fast, temporary alternative
    - ngrilly 7y ago
      Do your DNS records have a short TTL to enable such a quick switch from one provider to another?
      - swaraj 7y ago
        Fortunately (or non-optimally), our DNS TTLs were 300s (Route 53 default) so our sites were not fully accessible off Zeit for ~5 mins. This wasn't too bad given that our sites were completely inaccessible from Netlify. One day when we're bigger, we'll extend those TTLs
        ngrilly 7y ago
        Short TTLs are bad, but I confess using them in some small scale projects in anticipation of this kind of worst case scenario :)
sergiotapia 7y ago
So I wasn't going crazy haha - just today we migrated our site to Netlify from Heroku and the second I wanted to demo it my boss the COO, it was failing and not loading. It wasn't just netlify though. I also was having trouble loading random websites like aptible, youtube, imgur.
alfredxing 7y ago
Their CDN hostname, `cdn.netlify.com`, resolves to a DigitalOcean node, so it seems like they are rolling their own instead of delegating to a third party.

70 points by ajsharp 7y ago

I understand the growing pains Netlify must be going through. They have created a great product, and are becoming very popular. I'd love to see the Enterprise version (or any paid version) segregated out from the free sites so we can get more reliable uptime. The build pipeline was down for several hours a few days ago, now a CDN outage makes things much worse, as it's taken down all our sites. I'm sure they will resolve things quickly, in the meantime our clients will start asking about other failover options.

drewda 7y ago
According to their overall status page, they already do run two separate CDNs for normal and enterprise customers: https://www.netlifystatus.com/ https://www.netlifystatus.com/
luhn 7y ago
Ostensibly they do seem to have that. > Our non-enterprise CDN is failing a small number of page loads, usually with SSL errors; Enterprise CDN is not affected. But looks like there's room for improvement, because the enterprise CDN ended up failing as well. > We are still working to mitigate the effects of these problems and do see some page load failures on the enterprise CDN as well.

gravypod 7y ago

It'd be really cool to see a writeup of what is going on at Netlify.

nwsm 7y ago
They are tweeting about it, and I'd expect a post mortem as they have been transparent in the past.

tschellenbach 7y ago

Yup, Stream's blog is down as well. Thought it was WPEngine at first, but turns out it's netlify.

ajsharp 7y ago

netlify.com appears to be affected as well. Curious if this is a software issue or a host issue. I believe netlify rents a lot of its infrastructure from other providers, such as Digital Ocean.

cfors 7y ago
At any sort of scale (which I have to imagine netlify is running at), this will be a software issue since hosts are designed to fail. Serving static content is crazy easy for computers, my guess is a bad configuration got pushed out.
altmind 7y ago
Here I saw some mentions that it may be related to recent google cloud problems https://news.ycombinator.com/item?id=21418979 https://news.ycombinator.com/item?id=21418979

nwsm 7y ago

My site is back up. (1:13 EST) EDIT: intermittent issues now. Sometimes nothing loads, sometimes everything, sometimes SSL issue, sometimes HTML but no CSS or JS load. (1:22 EST)

losvedir 7y ago

Huh, loaded HN because it seems YouTube videos are down for me. I wonder if it's related.

nwsm 7y ago
Another comment claimed that Netlify rents servers from other providers, so this could be the case. FWIW Youtube is fine for me right now.

epoch_100 7y ago

Netlify is describing the outage as being the result of some kind of attack: "We’re currently suffering an attack that’s affecting our Free and Self-Serve CDN. Enterprise is currently unaffected. We are working on mitigating the effects and will update with more information as it becomes available."

volkk 7y ago

insane coincidence but literally the minute i started my first project ever on netlify just to play with it, i deployed it and nothing was loading; i kept googling and thinking it was my fault and then it clicked to check their status page and alas.

jagannathtech 7y ago
So it was YOU who took down the network then

twobat 7y ago

Bad publicity is still publicity, right?

swaraj 7y ago

This affects us too. We were able to migrate to zeit now in ~30 mins.

ngrilly 7y ago
Will you migrate back to Netlify when Zeit will suffer a similar downtime?
- swaraj 7y ago
  We're definitely staying on Netlify after this outage is done. We pay for some of their enterprise features, e.g. audit logs. We just need our sites up right now and zeit was a fast, temporary alternative
  - ngrilly 7y ago
    Do your DNS records have a short TTL to enable such a quick switch from one provider to another?
    - swaraj 7y ago
      Fortunately (or non-optimally), our DNS TTLs were 300s (Route 53 default) so our sites were not fully accessible off Zeit for ~5 mins. This wasn't too bad given that our sites were completely inaccessible from Netlify. One day when we're bigger, we'll extend those TTLs
      - ngrilly 7y ago
        Short TTLs are bad, but I confess using them in some small scale projects in anticipation of this kind of worst case scenario :)

sergiotapia 7y ago

So I wasn't going crazy haha - just today we migrated our site to Netlify from Heroku and the second I wanted to demo it my boss the COO, it was failing and not loading. It wasn't just netlify though. I also was having trouble loading random websites like aptible, youtube, imgur.

alfredxing 7y ago

Their CDN hostname, `cdn.netlify.com`, resolves to a DigitalOcean node, so it seems like they are rolling their own instead of delegating to a third party.