15 ms·
Cloudflare had a partial outage
- 0x262d 4y agoI guess this is probably why I can't log into league of legends atm.
- mikesabbagh 4y agocloudflare has a 100% SLA. This needs revision https://www.cloudflare.com/business-sla/ https://www.cloudflare.com/business-sla/
- ge96 4y agoHaha... I got pinged on my phone a site I manage is down, trying to figure out what's wrong with it, noticing other sites down, realizing it's Cloudflare
- jhgg 4y agoThey're not down worldwide, we're still seeing traffic from some POPs, but it looks like a majority of their POPs are dead. This feels like a bad config push.
- janmo 4y agoGoogle Analytics shows me that I still have quite a lot of traffic, so it is not 100% down. Edit: If I set my VPN to Poland it works.
- rubin55 4y agoJust noticed this morning.. IRCCloud and Discord are affected by this too. Wonder what else.
- sxsde 4y agoSame here! Seems to work with FR proxy tho!
- donkarma 4y agoYeah, using a Mullvad VPN with location set to France lets me access everything
- lkois 4y agoI'm unable to log into Bitwarden Safari extension. That's an alarming detail... Mobile app still works, fortunately
- nfreising 4y agohttps://www.cloudflarestatus.com https://www.cloudflarestatus.com still shows „All Systems Operational“
- mobiuscog 4y agoWas just about to post. Many sites returning 500 in the UK, and cloudflare seems to be the point of failure, including itself.
- giogio 4y agolooks like lot of sites being impacted by this one
- dmonitor 4y agoah fuck
- donkarma 4y agoturns out having a central failure point for the entire web was a bad idea
- noobermin 4y agoSomeone should a website that collates the approx. 90M times this sentiment has been made on this website (a good chunk of me making it) just a reminder how nothing somehow changes on this front the moment it comes back up and people go back to relying on single SAAS's for everything.
- donkarma 4y agoI wonder how long Cloudflare would have to be down for to have a noticeable change
- refulgentis 4y ago14h53m
- saurik 4y agoI remember when some key part of AWS EC2--EBS in us-east-1 maybe?--was down for a few days straight. Honestly, the main thing it taught me was "if you are honest with your customers they will mostly just come back later and buy everything they didn't buy today".
- selcuka 4y agoTo be fair this is not relying on a single SaaS for everything but many people relying on a single SaaS. I mean if you want to use a reverse proxy/CDN, you must rely on someone.
- noobermin 4y agoYes, sorry my rant was more like "everything relies on a single SaaS" rather than the SaaS doing everything, just mixed up the phrasing
- johnxie 4y agoCan confirm, no issues on our infra side. Cloudflare took down the web once again.
- atymic 4y agoYep, same issue. All our services are down, this is very bad. Can't even point directly to the app servers to resolve.
- trinovantes 4y agoIronically isitdownrightnow.com is also down
- zeven7 4y agoThis one seems be due to a hug of death rather than isitdownrightnow.com being behind Cloudflare, probably from too many people checking on all the other sites that are down. I say this because: (1) it eventually loaded for me after I tried a few times and gave it time to load, and (2) its certificate doesn't report to be from Cloudflare but other sites I've checked that are down do
- astrange 4y agodowndetector.com is also down since it's behind Cloudflare.
- Animats 4y agodowndetector.com : Downdetector: ಠ_ಠ Well, this is awkward... 5XX Server Error Web server is returning an unknown error There is an unknown connection issue between Cloudflare and the origin web server. As a result, the web page can not be displayed. Ray ID: 71eac4764c149452 Your IP address: 2600:1700:22e6:2410:2271:5b09:2694:a035 Error reference number: 520 Cloudflare Location: San Jose
- jayhoon 4y agoCF's website is down as well. The CF Status page [0] says everything is working, though. [0] https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
- donkarma 4y agoDo any of these corporate status pages ever work? AWS doesn't and neither does CF. This website is a more useful status page
- prawn 4y agoI don't think I've ever seen a status page report an issue at the time the issue was occurring. Seems a bit pointless.
- hatthew 4y agoIn my experience (with the sites I visit and care about), the status pages are usually pretty accurate and helpful.
- raverbashing 4y agoBecause most of them are manually updated, and not pulling directly from an uptime system (at least for incidents, etc) (Also it doesn't help that uptime monitoring systems are usually stupid and love triggering on false positives)
- jimmydorry 4y agoIronically, the cloudflare.com site is a more reliable indicator. If it doesn't load, then cloudflare is down. Their status page is a joke, likely crippled to reduce legal liability, but at this point it's just an outright misrepresentation.
- Shank 4y ago> Their status page is a joke, likely crippled to reduce legal liability, but at this point it's just an outright misrepresentation. It's just Atlassian Statuspage, which is a manually-updated incident response system. Unlike AWS, Cloudflare actually makes an effort to update it fairly quickly, but it can still be slow-to-update when something is immediately wrong.
- rkwasny 4y agoVery bad start of the day :( Cloudflare.com now up, and websites are coming up, argo tunnels still down
- 56789032433 4y ago
- theowenyoung 4y agoAll sites of mine are down now.
- jadjoubran 4y agoIt's working in Belgium but not in the Netherlands
- csomar 4y agoMaybe they are more decentralized than what we are giving them credit. I'm having different error messages (nginx, dns, 404) on different websites. Not sure if it's a full breakdown of their systems or a coordinated attack.
- dizhn 4y agoMy sites on Hetzner (Germany) are coming up fine as well.
- 56789032433 4y ago
- StanislavPetrov 4y agoSeveral sites I was trying to access all went down at the same time. Came to Hacker News to see what was up - not disappointed! *Including America's Cardroom, perhaps the biggest "offshore" US poker site. I can promise you that there are a lot of people who were playing in tournaments that are very unhappy right now. New York here.
- theowenyoung 4y agohttps://hn.algolia.com https://hn.algolia.com also down now.
- lkois 4y agoCloudflare Warp connects, then prevents anything else from loading. Thought it was my router
- 2b3a51 4y agoYes, I started checking router and wondering if anyone had managed to install some sort of exploit on it as I was getting that 500 nginx page from half a dozen Web sites. In UK, good old bbc.co.uk still working fine...
- murrayb 4y agoExplains why the online training course I was part way through stopped working! Amusing that the quickest diagnosis came from skimming the headlines here :)
- jdshaffer 4y agoI jumped around trying to find out what happened... should have realized HN was a better place to start. wink
- deleted 4y ago[deleted]
- giogio 4y agoAll shopify based stores are down
- imafish 4y agoYep - I have a Shopify app that uses Cloudflare, and was just about to panic when I saw this post. Turns out no one will notice, as all of Shopify is down anyway ¯\_(ツ)_/¯
- mike_d 4y agoThis should hopefully drive home the idea of why HN shouldn't be cheering on Cloudflare's slow takeover of the internet.
- collyw 4y agoOut of interest, what competitors are available? Clodufare protects against DDOS? Does it do other stuff?
- bilekas 4y agoCLoudflare just offer great services. Its straight up fact that even their free model is extremely generous. There is no big conspiracy to 'take over the internet' but when the product is good, the product is good.
- rapnie 4y agoNot a 'big conspiracy'. It's the business model, isn't it? Or isn't CF going for the biggest marketshare and maximizing profits on that, like all the others?
- bilekas 4y agoIt's certainly any business' model to grow as big as possible, but it's a hard business model to implement so competition is hard to find. I just can't blame CF for that imo.
- sph 4y ago
- paradite 4y agoDigitalOcean is down as well.
- rgj 4y agoNo, only their website.
- nonoesp 4y agoI host my apps in DigitalOcean and it seems the main domains are still resolving, (e.g., https://nono.ma https://nono.ma or https://gettingsimple.com https://gettingsimple.com). Subdomains aren't working though (e.g., https://sketch.nono.ma https://sketch.nono.ma). Update: It seems everything is resolving properly now.
- aidos 4y agoDNS looks to be ok, but sites that I have proxied through them aren't working.
- mobiuscog 4y agoThe DNS seems much slower for me, but is working.
- imalerba 4y agoCF status page now showing a wide spread incident. https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj
- skunkworker 4y agoThis is probably the best link for this status instead of the generic cloudflare.com one.
- cloudmonster 4y ago
- nansari 4y agoHmm, I am able to access Cloudflare's own website but sites that are proxied through them give me Nginx's default 500 error page.
- tebbers 4y agoYep, Hubspot is down for me: https://app-eu1.hubspot.com/ https://app-eu1.hubspot.com/
- dymk 4y agoEverything is up for me. Short downtime?
- deleted 4y ago[deleted]
- throwawaylala1 4y agoSingle Point of Failure. LOL.
- devilkin 4y agoAll things work here? Using cf dns, but even cf.com loads. Edit: in belgium
- skunkworker 4y agoI would change the URL to be https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ intead, the cloudflare.com domain looks to be hosted in a different way.
- RobinUS2 4y agois it just DNS?
- 44gg44gg 4y agofirefox DOH uses cloudflare and nextdns. nextdns is down as well
- hgazx 4y agoIs this why 4chan is down?
- prawn 4y ago"Investigating - Cloudflare is investigating wide-spread issues with our services and/or network. Users may experience errors or timeouts reaching Cloudflare’s network or services. We will update this status page to clarify the scope of impact as we continue the investigation. The next update should be expected within 15 minutes."
- mproud 4y agoAm I wrong to use this as an excuse to not use Cloudflare?
- jopsen 4y agoHow about waiting a bit first.. Maybe, things are back faster than you could cover from any trivial issue.
- 0xFACEFEED 4y agoIt's been 30+ minutes. All of my home's Ring cameras have been inaccessible this entire time. It's not that big of a deal for me because I planned for that eventuality, but a lot of people have not. If you run a critical service (like Ring) and your infra is tied to CloudFlare - you're stuck! There is nothing at all you can do. That's freaking scary man. If I was working infra at Ring I'd much prefer to get paged and start fixing the problem. There are very few problems that can't be fixed in 15 minutes if you plan well for failover...
- imafish 4y agoOf course not. If you don't need Cloudflare, don't use it :) For anyone else needing their services, they are a perfectly reliable provider (most of the time). Would make sense to ensure a fail-over, though.
- sph 4y agoApparently yes, people in here are quick to defend our new overlord. "But they're great", they cry, "why should I use anything else?"
- dx034 4y agoI'd love to use another company, but there's no one offering the same for the same price tag. Most of their services are free and they charge very little for the rest. Especially if you have a traffic heavy page with little revenue, Cloudflare is pretty much the only solution for CDN, WAF etc. All the others charge for traffic and cost a fortune.
- niel 4y agoThe Cape Town location does not seem to be impacted in any way. Everything works as expected.
- phire 4y agoOh... this explains why discord is down.
- TeeWEE 4y agolinear.app -> down notion.so -> down Wow, this dependency on cloudflare is wide.
- jgrahamc 4y agoYes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everywhere.
- samcheng 4y agoThanks for being here with timely updates! I knew to come to Hacker News once the alert triggered and a few users started complaining.
- fragmede 4y agoGood thing HN doesn’t use them then!
- lesmond 4y agoEnterprise support has been useless as is the status page. Got more info here
- halotrope 4y agoThank you! This little comment just saved me an hour of investigation. Good luck for getting the system back up asap.
- 4y ago
- pxeger1 4y agoGetting intermittent SERVFAIL from 1.1.1.1 DNS as well
- kc10 4y agoStatuspage seems to be useless, I was just trying to get the status via multiple networks and my mobile network. Ironically other downdetector services are also down.
- trapexit 4y agoSeems to be a lot of stuff flapping right now. I was able to load a client site that was behind Cloudflare, and now not. cloudflare.com was returning Connection Refused, then error 522 cloudflarestatus.com was returning Connection Refused, now can't even resolve the IP My guess would be that a router misconfiguration is being progressively deployed throughout their infrastructure. EDIT: Continues to look like a cascading failure across their network. 1.1.1.1 is now unreachable for me.
- gnaman 4y agoAuth0 login is also not working. Their website is up though
- adamgamble 4y agoSlack seems to be having trouble sending messages as well. Was trying to let my team know i've acked the request and am unable to do so.
- jdshaffer 4y agoDown here in Japan, too. Thought it was my connection for a bit. sigh
- fizer 4y agoHackernews is my new status page. All others are useless.
- pm90 4y agounironically true. first thing i checked.
- Bringoff 4y agoI'd say all others are down
- collaborative 4y agoPlenty of other status posts get flagged and taken down. Guess CF is special
- ab-dm 4y agoYep. Came here when I saw two of my sites go down before I even checked Heroku or Cloudflare
- bpye 4y agoClearly YC should start an Saas web host...
- fizer 4y agoMay be atleast like [Incident HN] to report about incident status.
- whazor 4y agoI first checked DownDetector which is really good for individual services. Except they are now also down...
- i67vw3 4y agoI too went to downdetector to see what was going on and funnily enough is was also down. They also had a funny error message. Edit:- Found it from another user., Downdetector: ಠ_ಠ Well, this is awkward...
- sadhorse 4y ago
- refulgentis 4y agoLol
- cloudmonster 4y ago
- phire 4y agoSo interesting, the actual edge servers work. But the Key/Value store that all cloudflare's configuration data lives on is giving 500 errors
- failTide 4y agoHah. I was seeing a lot of 500 errors across the web and came to HN to see what was up.
- melvyn2 4y agoUp on my phone (AT&T) but down on my main ISP/desktop (AT&T lightspeed). BGP issues too, maybe?
- mwcampbell 4y agoAt times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy (as long as one has backups for disaster recovery, of course). Sure, other things can cause downtime in that kind of infrastructure, but at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong.
- shric 4y agoAre there places to host an independent bare-metal box where the internet provider for that box is more reliable than cloudflare?
- Semaphor 4y agoWe’ve been on Hetzner for several years now. So far the only outages we had were from us moving servers (yeah, we don’t have high availability or load balancing, just a single beefy dedicated server). So, yes?
- jonatron 4y agoLast company I worked for, we had many Hetzner servers. We had many drive failures and CPU fan failures. It's fine if you can deal with a relatively high chance of hardware failure.
- js4ever 4y agoAccording to my monitoring, yes and by a large difference
- gfosco 4y agoAny local data center.
- Santosh83 4y agoPerhaps not, but those who want to avoid Cloudflare for technical or idealogical reasons won't realistically expect identical performance from smaller alternatives. Same as using Linux. People use it knowing fully well it may not support the latest & greatest consumer gadgets like Windows, but unless people use alternatives despite minor downsides, we shouldn't be distressed when we eventually reach a point of global near-monopoly.
- nullcipher 4y agoDNS API failing with 500
- loyalcinnamon 4y agoThat explains why i couldn't access phoronix
- MaxLeiter 4y agoThis aligned with me debugging a separate issue in a program that programmatically uses the npm registry... that wasn't fun.
- ffeiek 4y agoI look forward to a technical report on this outage on their blog.
- bubblethink 4y agoHello 8.8.8.8 my old friend.
- hexage1814 4y agoAgain, why did people decide to centralized like 80% or something of the internet under a single company?
- dcgudeman 4y agoWhy do people ask questions like this? You know the answer. This company offered products or services superior to alternatives so people elected to use them.
- yawnxyz 4y agowow so that's why all of the "is it down for me" sites are down lol
- dangrossman 4y agoI wasted a bunch of time debugging the HTTP 500 errors on my site before I realized everything is 100% OK on my end, and that it's Cloudflare returning the error not my servers.
- RowanH 4y agoDitto - I'm sitting here, wtf I'm not running Nginx on my blog, but I'm getting an Nginx response, hit IP directly....oooh.... right that doesn't make sense it's working fine. Cloudflare can't be down, that's next to, wait, status page (to their credit it's got a status note). HN here we go...
- jgrahamc 4y agoSorry about that.
- RowanH 4y agoAs we'd say, no worries mate! One little blog is probably the least of the s*tstorm you guys just had to deal with :)
- hobofan 4y agoWould it be possible to adjust that 500 page to include an indication that it originates from Cloudflare, for the case that an outage like this happens again in the future?
- jgrahamc 4y agoI was surprised by that also. Will ask.
- liveprogramming 4y ago- Encrypted DNS seems to be having issues (very slow resolution, if any) - Having issues connecting to GitHub (Could be they are using CF, or could be DNS issue - but I'm able to connect fine to Google services) - Twitter loads, but all images fail to resolve - https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ loads very slowly, and no assets (CSS, images, etc) load EDIT from CF :: The issue has been identified and a fix is being implemented. Posted Jun 21, 2022 - 06:57 UTC
- coolandsmartrr 4y agoHere in Japan, Twitter seems to be fully operation, e.g. images load too.
- veb 4y agoGitHub was working fine for me in NZ. So was Twitter. Very interesting outage.
- deadalus 4y agoHad a hard time trusting Cloudflare since they blocked 8chan due to political activism.
- collyw 4y agoOhhh, downvoted, I assume because 8chan are the bad guys.
- woojoo666 4y agoWow lots of websites are affected, including Medium. The perils of centralization strike again. Though ironically, I noticed that the IPFS website uses cloudflare as well. The actual IPFS network is working just fine though, and I'm not aware of IPFS ever having any global outages. Though then again, I'm not aware of any on bittorrent either
- capableweb 4y agoThe concept of "being down" doesn't really apply to protocols. IPFS/BitTorrent never being down is a bit like saying that TCP/HTTP has never been down. Individual servers/client can have connection issues, but obviously won't affect clients not connected to those, and is not because of the protocols themselves.
- woojoo666 4y agoBut the infrastructures those protocols provide (the IPFS network, torrent swarms) can be an alternative to Cloudflare. Which is why I brought it up
- softawre 4y agoNot to state the obvious, but... if a big centralized company built a Cloudflare for IPFS to make it easy for the masses to adopt, that company could go down just as easy as Cloudflare.
- woojoo666 4y agoHow so? Somebody links to a webpage, decentralized resolver converts it to an IPFS hash, which the client queries for any providers of that hash, and retrieves directly from them. No central authority necessary
- jimmygrapes 4y agoLMAO of course when every single thing I tried to use won't load or gives me a useless default 500 nginx error page, I find out why here. Figured it was CloudFlare. Single point of failure, not once.
- eBombzor 4y agoBack up?
- archon810 4y agoYup.
- loganc2342 4y agoI had tinkered with my network settings just before this to troubleshoot an entirely unrelated problem so for a minute there I thought I broke everything lol
- throwaway742 4y agoSame here. I picked a bad time to mess with my DNS config.
- saifulwebid 4y agoGitLab.com is impacted too as they are also behind Cloudflare.
- dnsmichi 4y agoGitLab team member here. Thanks for sharing. Incident: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/62b16aa5d28faf12a3973460 https://status.gitlab.com/pages/incident/5b36dc6502d06804c08... with the latest update: [Monitoring] Services seem to be back to normal, and we continue monitoring. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7288 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7...
- afro88 4y agoEverything working here in Australia
- plasma 4y agoSeems fine in Melbourne but not Sydney
- kondro 4y agoNo it's not.
- dankwizard 4y agoDiscord, Runescape down for me. And some work related systems.. but that's less important.
- ab-dm 4y agoI'm in Melbourne and it all seems ok
- veb 4y agoA lot of my Australian colleagues were saying a lot of things were down, including all of our websites however me being in NZ, I was able to visit them. So I do think Cloudflare actually is a bit more decentralised than we give them credit for really. Just the fact that they messaged here in the HN thread about what was happening, what they knew, and how they were gonna fix it. That's just _awesome_ Kudos to them. I can't wait to see their after report.
- londons_explore 4y agoCloudflare has too many outages. Their core service (DNS and web proxying) should see an outage once every 10 years or less. Much like Google Search (which is a far more complex service). Yet it seems we get an outage more frequently than once a year. In my opinion, that makes the service too unreliable to base my business off - it's not like I can failover to another provider while they're down.
- collyw 4y agoI am guessing this is why substack is down?
- timeimp 4y agoThis is the official Cloudflare incident URL: https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj
- revanx_ 4y agoThis was very educational, all of a sudden I couldn't reach 60% of all websites I normally visit everyday. I guess this is the cost of laziness under the guise of DDOS protection.
- lapsis_beeftech 4y agoYou may also want to consider that this is how the web looks all the time for those of us blocked by Cloudflare.
- mrath 4y agoI almost thought my internet or dns has some problem until I opened HN
- godmode2019 4y agoI was setting up some DNS for a site, when it suddenly stopped working after 30mins of missing with the settings and googling I gave up come in here, and see this. My sites that are just using DNS are working fine, it's only those with the orange cloud, proxy turned on that are broken.
- joebob42 4y agoKind of ironic, there was a big "cloudflare is bad and a central point of failure" article on the front page just a couple days ago. Found it, https://news.ycombinator.com/item?id=31801947 https://news.ycombinator.com/item?id=31801947 edit: Not that I necessarily agree with the article even in light of there being an outage, cloudflare has been pretty good for us. Just thought it was interesting.
- indy 4y agoBrace yourself for a lot more of those kinds of articles in the next few days.
- sph 4y agoI welcome them. Perhaps these outages stop all people rushing to host their single HTML page blog through them.
- indy 4y agounfortunately their tentacles have penetrated deep into the internet: https://news.ycombinator.com/item?id=31820929 https://news.ycombinator.com/item?id=31820929
- FlyingSnake 4y agoWith titles like "Cloudflare considered bad"
- sschueller 4y agoAlso ironic so many are blindly helping create "the great firewall of the USA" because it's easy and cheap.
- hericium 4y ago> "the great firewall of the USA" Not only the US.
- 4y ago
- collyw 4y agoShouldn't have happened in the first place. Should have had something that worked on their own website to indicate the service is down, not needing to come to a somewhat obscure tech forum to find out the details.
- mebutnotme 4y agoTheir status page does show the service down as well as them having identified the issue and working on a fix: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
- synu 4y agoI don't understand why companies simply don't have outages, it seems like it would be a lot less stressful.
- prawn 4y agoThey're talking about getting accurate information from an employee posting on HN rather than on the status page, rather than the outage itself.
- synu 4y agoAh, well in that case they already have that so no need to complain: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
- prawn 4y agoThe comment on HN had more useful information (that the issue was understood and a fix coming) before that status page then updated. I think that's their point. Prior to that, it was some time (in the "all my sites are wrecked" timescale) before the status page had any indication of an outage.
- synu 4y ago
- mburst 4y agoIt's times like these when I'm appreciative of the simplicity of the HN tech stack. Was talking to some people on discord when it went down and then noticed some other websites were down. Came right to HN to see 5 different threads about this. Will be curious to see what the cause of the issue turns out to be
- meatjuice 4y agoThat indicates how much of the Internet depends on cloudflare server.
- daitangio 4y agoLinode seems unreachable too (and related hosted VPS). Is it my problem or is it a general failure on Linode?
- mproud 4y agoI think it has everything to do with CF and not hosting. I have Linode servers without CF, and everything is great.
- pxeger1 4y agohttps://www.cloudflarestatus.com/incidents/xvs51y9qs9dj https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj: Identified The issue has been identified and a fix is being implemented. Posted 5 minutes ago. Jun 21, 2022 - 06:57 UTC
- PresidentObama 4y agoThings like Victorops/Splunk On-call are also down because of this. This means if your alerts are fired through them, you'll peacefully be sleeping through this incident unless your customers wake you up.
- emptyparadise 4y agoWho watches the watchmen?
- impulser_ 4y agoThe worst part is cloud infrastructure companies like DigitalOcean and Linode are both down simply because for some reason they can't build their own infrastructure to not rely on Cloudflare lol.
- dxuh 4y agoI think they rather got overwhelmed by many more requests reaching them that would usually hit Cloudflare. Also, as is widely known, people tend to hammer F5 when something like this happens, additionally increasing the number of requests.
- sirrele 4y agoI believe discord was also affected due to this.. However, I did get messages from my friend in Thailand
- kiwiboy10111 4y ago[dead]
- veb 4y agoYeah the NZ ones just came back online, https://veb.co.nz/ https://veb.co.nz/ was returning a 500 nginx error which confused the shit out of me until I visited HN Seems our DNS was working over time because a lot of the websites that were down for others were working OK for us.
- ripdog 4y agoOdd. My client's aus-hosted, nz-customer site was hit by this.
- easytiger 4y agoOut of interest, do you think for sites like those, cloudflare provides any value?
- rvz 4y agoEverything using Cloudflare is down. Tells you that how much of the internet Cloudflare is part of. Once in a while web2 is going great with Cloudflare. Until when everyone uses it and it goes down.
- deleted 4y ago[deleted]
- ab-dm 4y agoI haven't been impacted in Australia at all, but all of my (probably US based) monitors are going off.
- kar1181 4y agoNot having fun right now, but this brings home just how reliant so much of the internet is on a very few very big service providers.
- caeruleus 4y agoPeep, anyone there? Wow, for me it looked like the world had gone mad. This is a reminder to not only rely on 1.1.1.1 for DNS resolution in PiHole. I host most of my services locally, but ironically could not connect to my own homelab. I use a dedicated domain with DynDNS and did not configure the network and DNS without reliance on external DNS. Surely it's infinitely more likely for me to make a mistake, right?
- archon810 4y agoBack up now as of about 5 min ago.
- deleted 4y ago[deleted]
- new_newbie 4y agoCan someone link me to some information that explains what Cloudflare is besides being a CDN? Like I understand how websites can be served using a CDN and how a lot of the internet depends on that... but I don't see how gaming services like Valorant or cloud providers like AWS or chat room like Discord depend on Cloudflare. Thanks!
- dx034 4y agoTheir WAF is very useful, it makes it very easy to defend against attacks without paying anything. In general, their big plus point is that they offer many services for free, making it easier to onboard. But by now they offer lots of services, although I believe WAF and CDN are probably still the most important to many.
- dx034 4y agoSeems to be back up again, albeit a bit slow.
- someotherperson 4y agoUh, that’s not good. The negatives of centralisation really smack you in the face. I’ll start moving my sites away from Cloudflare soon. Not because it’s bad — in fact it has been amazing, but rather to decentralise.
- 8n4vidtmkvmk 4y agodoesn't help. my site doesn't use CF and I'm still down because digital ocean is down
- hatthew 4y agoLooks like it's mostly good now?
- simonjgreen 4y agoIf only we could come up with a globally distributed set of networks and systems that could be run by millions of entities that don't rely on each other to keep working. Oh no wait...
- renonce 4y agoOur monitoring systems had influxdb endpoints behind cloudflare. Now we not only lost users but also access to data about the impact of outage.
- hi5eyes 4y agomaybe we should start suggesting the use of IPFS/Arweave to host critical infra status pages or similar sites that are occasionally imminently needed
- HelloNurse 4y agoDiscord is back, Kickstarter progressed from 500 to an exotic "Error 1016 Origin DNS error". EDIT: all flushed, Kickstarter works.
- Grollicus 4y agoWhile this sucks, at least we'll have a great writeup to look forward to
- todotask 4y agoI think unpkg were affected by this too.
- fritigern 4y agoAnother reminder that centralized services like Cloudflare are a bad idea.
- phillipseamore 4y agoIt's time to start discussing a fail-open option for us CF users. Most of my sites are using CF for global performance rather than DDoS protection and security. I'd be fine with them changing DNS to point to the origin (or any other user defined IPs) in case of issues (even if it would take hours to return to normal). This is also important for countries with limited connectivity to the Internet, if the PoP in that country looses it's connection back to CF it shuts everything down, so even if the origin is in the next rack over from the PoP, it's un-reachable.
- booi 4y agoWouldn't this expose the origin IPs to attack?
- phillipseamore 4y agoI'm talking about this as an option for users like me, that don't have an attack surface, but need the global performance gains of CF.
- 1123581321 4y agoProbably not many users who need the performance and can handle unexpected failover. There would also be the issue of setting the policy defaults effectively. Most users wouldn’t benefit from this footgun. If you’re serious, you could probably automate this right now with your DNS provider and uptime monitoring.
- phillipseamore 4y agoI'm so serious that I already have failover after 1 hour at the registrar level, but those changes are not immediate and can take up to 24h to roll-in and roll-back due to DNS propagation and caching.
- 1123581321 4y ago
- michelb 4y agoSites are coming back up on my end (NL)
- pqdbr 4y agoI'm having '500 Internal Server Error' (nginx) on a regular tab, and everything working fine on an icognito tab. Go figure.
- NoPie 4y agoI cannot access science.org, quora.com, substack.com at the moment. It shows 500 Internal Server Error. Didn't know why but now it is clear. I guess I just wait until it is fixed.
- masteroftoilet 4y ago
- wheresmycraisin 4y agoThis is such a weird outage, cloudflare sites are down on some of my devices but not others.
- jimmygrapes 4y agoToday a mistake, tomorrow an order from the DOJ. Take heed, Internet.
- hericium 4y agoI really dislike that they are editing their status messages. Entry[1] dated "Jun 21, 2022 - 06:43 UTC" has been edited to include more detail after they posted another entry at 06:57 UTC. There seems to be no indication that the message has been altered. Currently text on the status page may suggest that they identified the problem immediately but it took about 15 minutes. Previously there was a text stating that customers should expect update within 15 minutes. Next message was posted 14 minutes after that but previous message was altered later and nothing indicates this. Cloudflare, not cool. [1] https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj
- e1g 4y agoStrongly agree. Such whitewashing puts all previous incident reports in doubt - can I trust CF summaries of outages, or did they rewrite that history too.
- RobinUS2 4y agoI understand your point, but CloudFlare generally is very transparent, including root cause analysis and their CTO reaching out directly. It could also be a mistake or not so well thought about instead of assuming bad intentions.
- nextaccountic 4y agojgrahamc is in this thread. if he wants to, he may say this was a mistake and swear they won't do this anymore he can also add an [edited] on that entry, among other things
- deleted 4y ago[deleted]
- victor9000 4y agoI'm placing my bets on a config file with emoji characters
- subvarad 4y agoWishing Cloudflare ops teams the best to recover fast from this outage. Meanwhile, we urge customers to check out www.cdnreserve.com , and implement a sound CDN backup strategy (auto-failover) when the primary CDN suffers an outage.
- mike_d 4y agoNever market during another companies outage, offer help instead. Tomorrow it will be you.
- subvarad 4y agoI absolutely agree and very respectfully so. No one is immune to outages. Well said. CDNReserve is designed in a way, that if the outage occurs on one platform it will map the traffic to failover CDN and if the failover/backup CDN suffers an outage, the traffic will be shifted to the primary CDN using CDNReserve. Its built on the premise that the likelihood of two CDNs having outage at the same time is close to ZERO.
- mike_d 4y agoThe likelihood of CDNReserve having an outage on the other hand is 100%. You aren't the first to come up with the idea of a CDN traffic director (I built one), and you'll soon discover customers recognize you are just another single point of failure and not the solution. Best to focus on the things other companies in the space market on, bill optimization, latency optimization, etc.
- subvarad 4y agoAgreed. The likelihood of any platform having an outage is 100%. But the likelihood of two networks having the outage at the same time is close to zero. It's awesome that you built a similar solution in the past. It would be great to jump on a call and learn from your experience if you are open to it.
- plasma 4y agojgrahamc, just some feedback about trying to reach support: 1. I could see my site down, including cloudflare.com with nginx 500 errors, via Sydney AU 2. Logged in to dashboard (via Melbourne AU) that worked; and so was thinking it was an issue with Sydney Cloudflare My experience with Cloudflare has been in the past sometimes servers in some regions have issues and its a transient thing. 3. Status page showed no problems, so I went to "Contact support" and went around in circles (really frustrating) via the "Contact support" link moving me between Community forums, Support ticket, etc. I then see Chat is an option is available with a Business plan, so I upgrade to that, hoping for some real-time support to alert of the Sydney issue. 4. Return to the "Contact support" page after upgrading the plan, but the Chat option still not present on the support screen (and help articles say to return to support page and click "Chat" but it never shows up). 5. Come across https://community.cloudflare.com/t/cloudflare-for-teams-chat-email-support/312310/8 https://community.cloudflare.com/t/cloudflare-for-teams-chat... searching for why I can't see Chat as an option on the support forum saying they're on paid plans with no chat support and its not showing up, so I just give up assuming its broken 6. Open HackerNews and see its at the top. A few moments later the status page reflects the outage. I still can't see the Chat option so I've down-graded the plan again.
- jgrahamc 4y agoThanks. I'll feed that back.
- calyhre 4y agoTheir whole support experience is really not great. Used it a few times those last few years, and I rarely got out satisfied at all. For example, they seems to have what I assume is a separated DB for CF users and CF support users, but with one shared login system. But if you end up updating your email on CF, it's not reflected on their support system and all your tickets are gonna be refused because of the email mismatch, completely disregarding the fact that you just logged in via your CF account. And no way to update it from the support part of course.
- herbst 4y agoMy morning started with crunching logs and not finding any errors and slowly panicking. But we'll, can happen :)
- pmlnr 4y agoSites returning 500 is one thing, people will understand that's an error. Site can't be found because DNS is out is not one that the generic public will start to debug, but instead they'll walk away from the site, sometimes for good. Question: how could be (temporary) DNS errors be made nicer?
- nanankcornering 4y agohugops to cf team. <3
- Linda703 4y ago[dead]
- mavercik1337 4y agoSooner or later they will start to give a d..n about resiliency :)
- deleted 4y ago[deleted]
- qwertox 4y agoThe entire day yesterday performance with Cloudflare was extremely sluggish. Pages which relied on it, even if it's only loading a JS-file from the CDN, would hang for tens of seconds.
- sylware 4y agoyeah, and if cloudflare could make their anti-bot "verification" interoperable with noscript/basic (x)html browsers, and not force those grotesquely and absurdely massive google (blink/geeko) and apple(webkit) web engines, that would be less criminal.
- ncmncm 4y agoStill down, here.
- Drblessing 4y agoCloudflare is great everyone, let's not forget how awesome they are.