7 ms·

Netlify Status – CDN Issues

91 points by yabones 6y ago

jayp 6y ago
They and Vercel both seem to be going thru a lot of growing pains. Quite a few outages over the past year. PS: we use both (for different sites). Probably should consolidate to one.
- parhamn 6y ago
  I always assumed there is an 'underwriter' (like fastly or akamai) for these CDNs. Is that not the case?
  - deleted 6y ago
    [deleted]
  - benatkin 6y ago
    Perhaps they're using CloudFront and providing an end user CDN on top of it. Both are partners with AWS. Both were multicloud, and it seems, aren't anymore.
  - scrose 6y ago
    I know Vercel uses AWS Lambda's behind the scenes to process web requests at least. I'd assume caching is also handled through Cloudfront by default. The place I currently work uses Fastly for caching and Vercel for hosting and it's definitely caused some issues(and much finger pointing on both of their sides) when one of those services makes a breaking change.
- deleted 6y ago
  [deleted]
- WORMS_EAT_WORMS 6y ago
  I’m not sure the future in this space, but I kind just wish Cloudflare and Vercel would join forces. It would make sense to me.
  - jpbow 6y ago
    Cloudflare is already on the way to building out all of the features Vercel has which is exciting. Eventually their Pages product (static hosting) will integrate Workers [1] for serverside APIs. [1] https://blog.cloudflare.com/cloudflare-pages/#oh-and-one-more-thing https://blog.cloudflare.com/cloudflare-pages/#oh-and-one-mor...
    - WORMS_EAT_WORMS 6y ago
      They lack a framework and crystal clear way to start developing apps. Nothing exists that taps their features beyond maybe Flarereact.
bombcar 6y ago
What is Netlify? "An intuitive Git-based workflow and powerful serverless platform to build, deploy, and collaborate on web apps"
- Pfhreak 6y ago
  A quick infrastructure service (from what I understand.) If you are building a JS single page app and want to be able to deploy it and some backing cloud functions without really worrying about CDNs, gateways, etc. Git push, code runs tests, code is deployed, done.
juliansimioni 6y ago
This has been the third major outage for Netlify in the last few weeks. I like the company, they have good people on their team, and their interface and functionality is great (deploy previews are so nice!). But this is probably the last straw, as the static portion of our company's website has been down for 45 minutes now. Fortunately, the beauty of a static site is they're quite easy to host anywhere. We're already on AWS, and it's easy enough to set up CloudFront. It won't be _quite_ as quick to deploy but it will probably rarely if ever break. Guess that's my task for the day :(
- avianlyric 6y ago
  You should checkout Cloudflare Pages. For static stuff it’s a dream to setup, and you get previews out of the box.
  - juliansimioni 6y ago
    The main advantage that AWS will have for us over anything else is that, since AWS already manages our DNS, we are going to be able to offer our visitors the best performance by using geo-specific IP addresses. The static site in question for us lives at the apex record (mywebsite.com), so it's generally not possible for other providers to do this without having them manage our entire DNS infrastructure, which we aren't willing to do. In fact I think this is part of why we've had so many issues with Netlify. It's clear their preferred way to host apex domain sites is to manage the DNS completely.
    - avianlyric 6y ago
      I think cloudflare can do it. They give you the option to set A records for your domain, in your own DNS. Cloudflare runs an AnyCast[0] network and multiple peers, so even through your using static IPs, the traffic will still get routed to Cloudflare nearest PoP, and pages is served by their edge network, so your site will be served from the location nearest to your customer. All without DNS shenanigans. [0] https://www.cloudflare.com/en-gb/learning/cdn/glossary/anycast-network/ https://www.cloudflare.com/en-gb/learning/cdn/glossary/anyca...
tyingq 6y ago
"To make sure you can minimize the impact of our single-homed loadbalancer being down"[1] Interesting. I'm surprised that's how their CDN works. [1] https://answers.netlify.com/t/support-guide-minimizing-impact-of-load-balancer-not-working/34888 https://answers.netlify.com/t/support-guide-minimizing-impac...
danr4 6y ago
Hmmm, lots of recent outages. Has anyone else noticed erratic response times in recent months? My web vitals score sometimes dips heavily because "response time from server" (or whatever it's called). Anyone can recommend another place where I can host? (except Vercel, which has similar results)
- indigodaddy 6y ago
  Firebase?
- jpbow 6y ago
  Cloudflare Pages, begin.com, fly.io, and surge.sh are a few I've come across as alternatives (depending on what features you're after).
colinchartier 6y ago
It must be very stressful to run a production hosting company - DigitalOcean and Cloudflare both went through similar issues in the early days.
- jgrahamc 6y ago
  No comment.
  - colinchartier 6y ago
    Glad you overcame it all in the end - we're happy users, especially given the outstanding uptime for the past few years.
mrzool 6y ago
"We have identified the issue and it is attributed to an upstream provider." The upstream issue is probably at Google: "We are experiencing an issue with L4 load balancers in us-west1-c. Multiple managed services relying on LB and located in this zone might be affected." https://status.cloud.google.com https://status.cloud.google.com This has been going on for at least an hour.
riffic 6y ago
oof, I got bit by this issue this morning. if you're using cloudflare, set your domain's apex (`@`) as a CNAME pointing to the default subdomain (sitename.netlify.app) and use CNAME Flattening. It's the A record pointing to the CDN IP address that's broken.
pscanf 6y ago
Shameless plug, I can suggest StaticDeploy (https://staticdeploy.io/ https://staticdeploy.io/) as an open source, self-hosted alternative to Netlify, which can give you a similar deployment workflow. It's definitely possible to just host directly on S3/CloudFront, but StaticDeploy sets you up quickly with a workflow and a dedicated management interface. Disclaimer: I'm the main developer of the project.
- jgrahamc 6y ago
  Honestly, don't do this. Netlify is having a bad day and it's not fun for them. The great wheel of karma turns around slowly and one day it'll be your turn to have a bad day. Submit StaticDeploy to HN some other day and tell us about it. Sounds cool.
  - bigsparky 6y ago
    Cloudflare CTO telling off the open source project guy, while ignoring all the other comments suggesting cloudflare?
    - jgrahamc 6y ago
      There's a difference between self-promotion and other people making a suggestion.
  - pscanf 6y ago
    Thanks for pointing this out, I admit I did not consider their point of view (the plug was shameless, but in the self-promotion-is-inherently-shameful sense), and I agree it's in bad taste.
charlesworth91 6y ago
This hit me, as someone using A records to point to the Netlify IPs. If you are using GitHub, I found switching over to Github pages quite easy for my static site. I used this guide: https://docs.github.com/en/github/working-with-github-pages/managing-a-custom-domain-for-your-github-pages-site#configuring-an-apex-domain https://docs.github.com/en/github/working-with-github-pages/...
bobfunk 6y ago
Netlify CEO here. I'll try to answer the questions from the thread so far: Some of our customers are affected by an outage of Googles Load Balancer. These customers are not taking advantage of our DNS management, or they are not using a DNS provider that supports CNAME flattening and are using their root domain name for their website (ie, no www prefix). While we don't recommend the setup, we do provide a single IP address to bind an A records for customers that want it. In general we run our edge infrastructure as a large multicloud setup spanning several different network providers, and offer two separate networks, one for free/self-serve customers that will get newer features faster and one for enterprise customers running mission critical projects where we guarantee very high uptime and reliability through formal SLAs. The single IP mentioned above however corresponds to a Google Load Balancer, and they are unfortunately currently having an outage for all load balancers in the relevant region. Read more on https://status.cloud.google.com/ https://status.cloud.google.com/ Again, while we generally don't recommend using the A name setup for anything mission critical, we are currently doing everything we possible can helping enterprise customers that have chosen this setup to change their configuration. Really sorry for all the trouble this are causing for our users, full RCA will be forthcoming.
- martin_bech 6y ago
  Hi Bob, just want to say, I like your service a lot.
  - bobfunk 6y ago
    Thanks! Appreciate the kind words!
    - StavrosK 6y ago
      Seconded, I use it for all my static hosting. Great service.
- WORMS_EAT_WORMS 6y ago
  Are you all having shake ups within the company? I'm not going to deep dive, but I heard some rumors about some higher ups leaving. After the Cloudflare Pages release, I'd be curious of what your future road map looks like and how you all plan to compete and grow. Thanks for all you and your team does. What you have done for front-end development and the community has been nothing but awesome and inspiring.
gregsadetsky 6y ago
Update on the Netlify Status page [0] -- TLDR anyone experiencing this issue should point their apex domain to 75.2.60.5 --- Full announcement: Our team have created a new load balancer instance which is not associated with the upstream provider who is currently experiencing issues. Please update A record values for your site(s) bare domain to 75.2.60.5 to mitigate against this outage. --- Their documentation page [1] now includes the same IP. [0] https://www.netlifystatus.com/ https://www.netlifystatus.com/ [1] https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/#configure-an-apex-domain https://docs.netlify.com/domains-https/custom-domains/config...
- swyx 6y ago
  thanks for the TLDR - i pointed it over and can verify that it fixes the problem. annoying to wait out the caching on mobile devices though, i am not sure how to clear DNS cache on mobile but am not too bothered.
corytheboyd 6y ago
I wish Netlify all the best! In the mean time, I just hopped on to Cloudflare and saw their Pages product is in public beta. Seems to work the same as Netlify for static pages, just tried it out for my personal site and it worked great! I was already using Cloudflare as my CDN and to manage DNS, it's actually really nice to have my entire website configuration live there.
notyourday 6y ago
Whom are they using? In 2021 for static sites behind a CDN being down is... odd. Pretty much all CDNs by now should support equivalent of serve stale.
- swyx 6y ago
  think this is a DNS issue, not CDN. bobfunk noted it is Google, scroll up
  - notyourday 6y ago
    So they implemented their own CDN-ish thing on top of GCP without doing anycast and serve stale and they have a non-trivial number non-mom and pop customers?!
GaltMidas 6y ago
This got me today. Probably could be characterized as the classic case of: Company says a certain use case is unsupported, but tries hard to accommodate users who are stuck with the unsupported use case, so they hack up a decent work around under the hood. Then the technically unsupported use case blows up, so they then have to scramble to support it with a quick work around...for the workaround. The workaround worked. I think at this point it makes me more likely to keep using Netlify. I love the product. And I think I love the support for unsupported un-recommended feature that they supported today. Thanks Netlify Ops!
MaggiD 6y ago
My startup has also suffered from the recent Netlify outages with our main landing page. Last time I already did the research for potential alternatives: - Cloudflare Pages is now available in public beta - even more interesting seemed this offering by PerfOps to put my CDN behind a Load Balancer that can monitor uptime and dynamically shift traffic between multiple CDN sources: https://perfops.net/flexbalancer https://perfops.net/flexbalancer What do you think? - it seems like the multi cloud approach to CDN - but at the same time I'll have a problem if this Load Balancer fails (single point of failure)
- SahAssar 6y ago
  > CDN behind a Load Balancer This sounds crazy to me. Besides the obvious superfluous network/layer hops, complexity and points of failure it would also partition the cache, right? So working against the very thing CDN's optimize for.
- sofixa 6y ago
  There's also Vercel and Firebase ( Hosting ) which are stable ( not in beta as Cloudflare Pages).
hikerclimb 6y ago
Hopefully they are never solved

91 points by yabones 6y ago

jayp 6y ago

They and Vercel both seem to be going thru a lot of growing pains. Quite a few outages over the past year. PS: we use both (for different sites). Probably should consolidate to one.

parhamn 6y ago
I always assumed there is an 'underwriter' (like fastly or akamai) for these CDNs. Is that not the case?
- deleted 6y ago
  [deleted]
- benatkin 6y ago
  Perhaps they're using CloudFront and providing an end user CDN on top of it. Both are partners with AWS. Both were multicloud, and it seems, aren't anymore.
- scrose 6y ago
  I know Vercel uses AWS Lambda's behind the scenes to process web requests at least. I'd assume caching is also handled through Cloudfront by default. The place I currently work uses Fastly for caching and Vercel for hosting and it's definitely caused some issues(and much finger pointing on both of their sides) when one of those services makes a breaking change.
deleted 6y ago
[deleted]
WORMS_EAT_WORMS 6y ago
I’m not sure the future in this space, but I kind just wish Cloudflare and Vercel would join forces. It would make sense to me.
- jpbow 6y ago
  Cloudflare is already on the way to building out all of the features Vercel has which is exciting. Eventually their Pages product (static hosting) will integrate Workers [1] for serverside APIs. [1] https://blog.cloudflare.com/cloudflare-pages/#oh-and-one-more-thing https://blog.cloudflare.com/cloudflare-pages/#oh-and-one-mor...
  - WORMS_EAT_WORMS 6y ago
    They lack a framework and crystal clear way to start developing apps. Nothing exists that taps their features beyond maybe Flarereact.

bombcar 6y ago

What is Netlify? "An intuitive Git-based workflow and powerful serverless platform to build, deploy, and collaborate on web apps"

Pfhreak 6y ago
A quick infrastructure service (from what I understand.) If you are building a JS single page app and want to be able to deploy it and some backing cloud functions without really worrying about CDNs, gateways, etc. Git push, code runs tests, code is deployed, done.

juliansimioni 6y ago

This has been the third major outage for Netlify in the last few weeks. I like the company, they have good people on their team, and their interface and functionality is great (deploy previews are so nice!). But this is probably the last straw, as the static portion of our company's website has been down for 45 minutes now. Fortunately, the beauty of a static site is they're quite easy to host anywhere. We're already on AWS, and it's easy enough to set up CloudFront. It won't be _quite_ as quick to deploy but it will probably rarely if ever break. Guess that's my task for the day :(

avianlyric 6y ago
You should checkout Cloudflare Pages. For static stuff it’s a dream to setup, and you get previews out of the box.
- juliansimioni 6y ago
  The main advantage that AWS will have for us over anything else is that, since AWS already manages our DNS, we are going to be able to offer our visitors the best performance by using geo-specific IP addresses. The static site in question for us lives at the apex record (mywebsite.com), so it's generally not possible for other providers to do this without having them manage our entire DNS infrastructure, which we aren't willing to do. In fact I think this is part of why we've had so many issues with Netlify. It's clear their preferred way to host apex domain sites is to manage the DNS completely.
  - avianlyric 6y ago
    I think cloudflare can do it. They give you the option to set A records for your domain, in your own DNS. Cloudflare runs an AnyCast[0] network and multiple peers, so even through your using static IPs, the traffic will still get routed to Cloudflare nearest PoP, and pages is served by their edge network, so your site will be served from the location nearest to your customer. All without DNS shenanigans. [0] https://www.cloudflare.com/en-gb/learning/cdn/glossary/anycast-network/ https://www.cloudflare.com/en-gb/learning/cdn/glossary/anyca...

tyingq 6y ago

"To make sure you can minimize the impact of our single-homed loadbalancer being down"[1] Interesting. I'm surprised that's how their CDN works. [1] https://answers.netlify.com/t/support-guide-minimizing-impact-of-load-balancer-not-working/34888 https://answers.netlify.com/t/support-guide-minimizing-impac...

danr4 6y ago

Hmmm, lots of recent outages. Has anyone else noticed erratic response times in recent months? My web vitals score sometimes dips heavily because "response time from server" (or whatever it's called). Anyone can recommend another place where I can host? (except Vercel, which has similar results)

indigodaddy 6y ago
Firebase?
jpbow 6y ago
Cloudflare Pages, begin.com, fly.io, and surge.sh are a few I've come across as alternatives (depending on what features you're after).

colinchartier 6y ago

It must be very stressful to run a production hosting company - DigitalOcean and Cloudflare both went through similar issues in the early days.

jgrahamc 6y ago
No comment.
- colinchartier 6y ago
  Glad you overcame it all in the end - we're happy users, especially given the outstanding uptime for the past few years.

mrzool 6y ago

"We have identified the issue and it is attributed to an upstream provider." The upstream issue is probably at Google: "We are experiencing an issue with L4 load balancers in us-west1-c. Multiple managed services relying on LB and located in this zone might be affected." https://status.cloud.google.com https://status.cloud.google.com This has been going on for at least an hour.

riffic 6y ago

oof, I got bit by this issue this morning. if you're using cloudflare, set your domain's apex (`@`) as a CNAME pointing to the default subdomain (sitename.netlify.app) and use CNAME Flattening. It's the A record pointing to the CDN IP address that's broken.

pscanf 6y ago

Shameless plug, I can suggest StaticDeploy (https://staticdeploy.io/ https://staticdeploy.io/) as an open source, self-hosted alternative to Netlify, which can give you a similar deployment workflow. It's definitely possible to just host directly on S3/CloudFront, but StaticDeploy sets you up quickly with a workflow and a dedicated management interface. Disclaimer: I'm the main developer of the project.

jgrahamc 6y ago
Honestly, don't do this. Netlify is having a bad day and it's not fun for them. The great wheel of karma turns around slowly and one day it'll be your turn to have a bad day. Submit StaticDeploy to HN some other day and tell us about it. Sounds cool.
- bigsparky 6y ago
  Cloudflare CTO telling off the open source project guy, while ignoring all the other comments suggesting cloudflare?
  - jgrahamc 6y ago
    There's a difference between self-promotion and other people making a suggestion.
- pscanf 6y ago
  Thanks for pointing this out, I admit I did not consider their point of view (the plug was shameless, but in the self-promotion-is-inherently-shameful sense), and I agree it's in bad taste.

charlesworth91 6y ago

This hit me, as someone using A records to point to the Netlify IPs. If you are using GitHub, I found switching over to Github pages quite easy for my static site. I used this guide: https://docs.github.com/en/github/working-with-github-pages/managing-a-custom-domain-for-your-github-pages-site#configuring-an-apex-domain https://docs.github.com/en/github/working-with-github-pages/...

bobfunk 6y ago

Netlify CEO here. I'll try to answer the questions from the thread so far: Some of our customers are affected by an outage of Googles Load Balancer. These customers are not taking advantage of our DNS management, or they are not using a DNS provider that supports CNAME flattening and are using their root domain name for their website (ie, no www prefix). While we don't recommend the setup, we do provide a single IP address to bind an A records for customers that want it. In general we run our edge infrastructure as a large multicloud setup spanning several different network providers, and offer two separate networks, one for free/self-serve customers that will get newer features faster and one for enterprise customers running mission critical projects where we guarantee very high uptime and reliability through formal SLAs. The single IP mentioned above however corresponds to a Google Load Balancer, and they are unfortunately currently having an outage for all load balancers in the relevant region. Read more on https://status.cloud.google.com/ https://status.cloud.google.com/ Again, while we generally don't recommend using the A name setup for anything mission critical, we are currently doing everything we possible can helping enterprise customers that have chosen this setup to change their configuration. Really sorry for all the trouble this are causing for our users, full RCA will be forthcoming.

martin_bech 6y ago
Hi Bob, just want to say, I like your service a lot.
- bobfunk 6y ago
  Thanks! Appreciate the kind words!
  - StavrosK 6y ago
    Seconded, I use it for all my static hosting. Great service.
WORMS_EAT_WORMS 6y ago
Are you all having shake ups within the company? I'm not going to deep dive, but I heard some rumors about some higher ups leaving. After the Cloudflare Pages release, I'd be curious of what your future road map looks like and how you all plan to compete and grow. Thanks for all you and your team does. What you have done for front-end development and the community has been nothing but awesome and inspiring.

gregsadetsky 6y ago

Update on the Netlify Status page [0] -- TLDR anyone experiencing this issue should point their apex domain to 75.2.60.5 --- Full announcement: Our team have created a new load balancer instance which is not associated with the upstream provider who is currently experiencing issues. Please update A record values for your site(s) bare domain to 75.2.60.5 to mitigate against this outage. --- Their documentation page [1] now includes the same IP. [0] https://www.netlifystatus.com/ https://www.netlifystatus.com/ [1] https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/#configure-an-apex-domain https://docs.netlify.com/domains-https/custom-domains/config...

swyx 6y ago
thanks for the TLDR - i pointed it over and can verify that it fixes the problem. annoying to wait out the caching on mobile devices though, i am not sure how to clear DNS cache on mobile but am not too bothered.

corytheboyd 6y ago

I wish Netlify all the best! In the mean time, I just hopped on to Cloudflare and saw their Pages product is in public beta. Seems to work the same as Netlify for static pages, just tried it out for my personal site and it worked great! I was already using Cloudflare as my CDN and to manage DNS, it's actually really nice to have my entire website configuration live there.

notyourday 6y ago

Whom are they using? In 2021 for static sites behind a CDN being down is... odd. Pretty much all CDNs by now should support equivalent of serve stale.

swyx 6y ago
think this is a DNS issue, not CDN. bobfunk noted it is Google, scroll up
- notyourday 6y ago
  So they implemented their own CDN-ish thing on top of GCP without doing anycast and serve stale and they have a non-trivial number non-mom and pop customers?!

GaltMidas 6y ago

This got me today. Probably could be characterized as the classic case of: Company says a certain use case is unsupported, but tries hard to accommodate users who are stuck with the unsupported use case, so they hack up a decent work around under the hood. Then the technically unsupported use case blows up, so they then have to scramble to support it with a quick work around...for the workaround. The workaround worked. I think at this point it makes me more likely to keep using Netlify. I love the product. And I think I love the support for unsupported un-recommended feature that they supported today. Thanks Netlify Ops!

MaggiD 6y ago

My startup has also suffered from the recent Netlify outages with our main landing page. Last time I already did the research for potential alternatives: - Cloudflare Pages is now available in public beta - even more interesting seemed this offering by PerfOps to put my CDN behind a Load Balancer that can monitor uptime and dynamically shift traffic between multiple CDN sources: https://perfops.net/flexbalancer https://perfops.net/flexbalancer What do you think? - it seems like the multi cloud approach to CDN - but at the same time I'll have a problem if this Load Balancer fails (single point of failure)

SahAssar 6y ago
> CDN behind a Load Balancer This sounds crazy to me. Besides the obvious superfluous network/layer hops, complexity and points of failure it would also partition the cache, right? So working against the very thing CDN's optimize for.
sofixa 6y ago
There's also Vercel and Firebase ( Hosting ) which are stable ( not in beta as Cloudflare Pages).

hikerclimb 6y ago

Hopefully they are never solved