19 ms·
Cloudflare was down
- dpcx 6y agoAlso breaking a bunch of deploys because npm and yarn are heavily dependent on Cloudflare, it would seem.
- tomklein 6y agoNPM is down too.
- navinag 6y agoyes
- satysin 6y agoGuess that explains Discord vanishing from the net a few minutes ago.
- heliodor 6y agoIs it me, or has this been happening way too frequently for them lately?
- jeremyjh 6y agoHonestly at their scale once a decade would be too frequent. Too many eggs in this particular basket.
- bithaze 6y agoOnce a decade doesn't seem realistic. At some point you get diminishing returns chasing as many mines as possible.
- twunde 6y agoTo be fair the last major outage they had was 1-2 years ago. That said, when that happened they had two outages in about a month.
- floren 6y ago1.1.1.1 is not resolving anything for me at this time.
- minxomat 6y agoLol, talk about timing. I'm currently working on a TLS library and was pulling my hair out trying to figure out why tests against CF sites suddenly failed. Can't even ask my cohorts on Discord because they are behind CF, too!
- cknoxrun 6y agoNo kidding! We had literally deployed a major page redesign and started watching our analytics drop off on it's way to zero. My heart is racing still. I wouldn't normally be happy for a cloudflare outage but in this case it's better than Google deciding to remove us from their index.
- akuji1993 6y agoThat's unfortunate timing dude. Good for you, it's probably not your mistake :D
- paulmd 6y ago"StackOverflow devs have the most difficult job in the world. After all, when StackOverflow is down, they can't exactly look for help on StackOverflow".
- ryanmccullagh 6y agoDns issues for sure
- aaomidi 6y agoOopsie Daisy half the internet goes down
- parliament32 6y agoEggs and baskets etc etc
- nobleach 6y agoGood DNS practice (at least when I did system admin 10 years ago) was ALWAYS having a secondary at some other location/network. Why do we just put some info in Cloudflare and call it good these days?
- lflux 6y agoFeatures, convience, pricing.
- toast0 6y agoIt's hard to use Cloudflare as a reverse proxy without using them as your delegated name servers (maybe you can use CNAMEs on paid plans?), and fancy dynamic nameservers make it hard to run secondary servers with zone transfers.
- nobleach 6y agoThis is definitely the answer I needed.
- olyjohn 6y agoBecause we don't do tech work anymore. We put in a credit card and click "Buy Now." Also, nobody got fired for using Cloudflare.
- minxomat 6y ago2020 is all in on everything bad that can happen.
- rococode 6y agoDiscord is entirely down right now, both the website and the app itself. Amusingly, a lot of the sites that normally track outages are also down, which made me think it was my internet at first. Downdetector, monitortheinternet, etc. Lots of other big sites that are down: Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, GitLab (502), Fiverr, Upwork, Udemy Edit: 15 min later, looks like things are starting to come back up
- emsy 6y agoSame for the German downtime trackers.
- thepete2 6y agosame here. I tried if hacker news still works and saw this
- belltaco 6y agohttps://status.discord.com/ https://status.discord.com/ is down. Wow.
- Deathmax 6y agoAnd that is why you host your status page on separate infra.
- Lyrex 6y agoWe're dealing with a deeper level problem here. Since a lot of the internet is relying on Cloudflare DNS at some part or another, even many backup solutions fail. Since so much of DNS is centralised in so few services, such outages hit the core infrastructure of the internet.
- cm2187 6y agoA sudden disruption on a large number of services for everybody at once doesn't look like a DNS problem to me, with all the caching in DNS. It would fail progressively and inconsistently.
- r0xsh 6y agoAh Shit, Here We Go Again
- Majora320 6y agoSeems like they managed to break half the internet for everyone.
- gregory90 6y agoCloudflare DNS is down too
- sascha_sl 6y agoDNS seems to be dead, if you have stuff in your cache and the site isn't low-TTL things still kinda work
- icodestuff 6y agoYeah this is absolutely killing us right now.
- karlmcguire 6y ago"All systems operational" What's the point of a status page if it doesn't reflect the real status... It's either the status page goes down with everything else or the status page is wrong. Great. EDIT: Looks like it's accurate now, 20 minutes later.
- Jasper_ 6y agoThe point of the status page is so you can point to it for your five nines SLA and go "look? we were only down for one hour". As soon as the money relies on the metric, the metric will reflect the money.
- mjlawson 6y agoGoodhart's Law[1] in action. [1]https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- zetalemur 6y agoThat's a good one, did not know that. Interesting read.
- post-it 6y agoI've noticed everyone posts links in footnotes, is that just Hacker News etiquette?
- therealdrag0 6y agoYa I think so. Since there isn’t a way to make html links with a short text .
- Jasper_ 6y agoIt's old-school plain-text email etiquette. The original Markdown even used a variant of that syntax for its links definition, known as "reference links". https://daringfireball.net/projects/markdown/syntax#link https://daringfireball.net/projects/markdown/syntax#link
- jpomykala 6y agoFriday
- 1f60c 6y agoThat page still shows "Cloudflare System Status: All Systems Operational" for me, but it's definitely down for me. Along with 1.1.1.1, which is... bad.
- cryptoz 6y agoFor me it says "Minor System Outage" for about 0.1s and then shows "All Systems Operational".
- ehsankia 6y agoEverything works fine for me (Canada), am I missing something or is it over already?
- reactorofr 6y agoStill down here.
- cartoonfoxes 6y agoAlso in Canada. Shit's fucked, yo.
- johnxie 6y agoCan confirm for taskade.com
- JoshGlazebrook 6y agoHaving DNS issues. Had to switch to Google's DNS (8.8.8.8/8.8.4.4) as 1.1.1.1/etc were not resolving anything.
- fpgaminer 6y agoSame. Even then, a bunch of sites are down. Maybe only ones behind Cloudflare? So far I've been trying to hit the various down detector sites and none of them will load. Google, Reddit, Hackernews are all fine.
- drrotmos 6y agoFrom what little I've been able to gather, anything using Cloudflare's DNSes are down.
- jeffbee 6y agoThat only worked until all the records expired from Google's cache.
- en4bz 6y ago1.1.1.1 is dropping ICMP pings while 8.8.8.8 is not but 8.8.8.8 is still returning DNS errors.
- guiambros 6y agoSame. Had to go through the entire troubleshooting process --- is it my internet connection? my DNS resolver? firewall? ISP starting to filter DoT queries somehow? Only last in my mental list was the possibility that Cloudflare would be down. Hope they publish a detailed post-mortem. It's always fun to read (but certainly very painful for those directly involved in writing it).
- microcolonel 6y agoWhat's all this about ”building a better internet”? Wide-reaching general service outages that are invisible to your status page are really not great.
- erichocean 6y agoI wonder if they'll cover why their status page is a steaming pile of garbage in the post-mortem?
- buro9 6y agoFrom what I can see externally this looks like DNS. I wonder if that includes the roots that Cloudflare operate.
- ransom1538 6y agoJesus. Does anyone know anything?
- michael_j_ward 6y agoHaving issues with gitlab myself
- sillysaurusx 6y agoOur TPU management page is also down: https://www.tensorfork.com/tpus https://www.tensorfork.com/tpus Seems cloudflare took out a good chunk of the internet temporarily. Doesn’t HN use cloudflare? Why did it survive? (I haven’t looked for about a year, but I seem to remember HN being proxied behind CF at one point.)
- searchableguy 6y agoIs there a status page for HN?
- saagarjha 6y agoYeah, it's whether news.ycombinator.com loads :P
- sillysaurusx 6y agoEDIT: Yes: https://twitter.com/HNStatus https://twitter.com/HNStatus HN is so reliable that’s it’s almost never needed one. I’m extremely curious how HN survived this; almost positive they used cloudflare at one point. I think the official status page is @hnstatus on Twitter, or something like that.
- toomuchtodo 6y agoThey did use Cloudflare, but also haven’t for some time.
- codegeek 6y agoHN is reliable until AWS Route53 goes down :).
- deleted 6y ago[deleted]
- jimz 6y agoParts of the site that are behind CF like the API are down.
- parliament32 6y agoThe status page linked shows "All Systems Operational" for me. Tested in private browsing and on my mobile. Looks like DNS issues, their nameservers aren't reachable.
- dangwu 6y agoLeague of Legends, Valorant and Discord both down. I took today off to play games...
- FireBeyond 6y agoAnother useless status site. DNS is completely broken. "All systems operational" in nice soothing green. No, not so much.
- whoisjuan 6y agoTheir status page says that everything is operational. So much for a status page when half of the internet breaks down.
- techlaw 6y agoitch.io down isitdownrightnow.com down
- caudamus 6y agoCloudflare's DNS (1.1.1.1) is failing to respond to most/all queries, which I'm observing as the root cause of a bunch of connection issues (name lookup failure). Interestingly the same domains don't show up on google's (8.8.8.8) DNS at all.
- parliament32 6y ago8.8.8.8 is a caching resolver, it still needs to talk to CF's nameservers for authoritative records.
- geerlingguy 6y agoI don't think it's just Cloudflare; I just had a fun 10 minutes seeing servers start flipping on my Server monitoring service[1]. This has only happened once or twice per year, and is usually due to weird global DNS issues. [1] https://servercheck.in/ https://servercheck.in/ (To give an update, I'm seeing from my monitoring systems (about 15 points around the globe) sporadic outages for Microsoft, Apple, Reddit, Bing, Node.js, Twitter, Yahoo, and YouTube. And my own servers (not behind CF at all) are also flipping up and down. It started around 21:14 UTC.)
- cm2187 6y agoa DNS issue wouldn't cripple all of the internet at once, with all the caching.
- RL_Quine 6y agoMost sites set the absolute minimum TTL for every record, for no reason. There’s a lot less caching than you’re thinking.
- cm2187 6y agoNo I see some services failing that have a TTL of 1h.
- qeternity 6y agoEh, what? There are many good reasons to have low TTL DNS...this exact outage being one of them. Update your records to go direct to your servers, and not through Cloudflare and bam you’re back up. Doesn’t work if your TTL is 86400
- unilynx 6y agoDoesn't help as cloudflare wants you to host their name servers with them, so you can't flip any records if the DNS itself is in trouble, like it is now And changing DNS servers often takes many hours (or days, if .net is involved apparently)
- solarkraft 6y agoFinally we see how much we depend on this single company.
- beatrobot 6y agoMore like the Internet is down.
- saagarjha 6y agoWhen you depend on a single company for much of the internet, such things happen :(
- logicalmonster 6y agoGiven that the US is basically in a non-shooting war with China, I wonder if this is something technical or part of some kind of attack. Something that I’d keep in mind.
- searchableguy 6y agoYour username is funny.
- wolfgang42 6y agoThere are enough ways for bits of the Internet to go kablooey on their own that “it’s an attack!” is a pretty big jump to a conclusion. If this turns out to be something other than Cloudflare tripping over a weird bug, my first guess would be that someone fat-fingered a BGP table yet again.
- wolfgang42 6y agoUpdate: Yep, BGP issue, though I was thinking it would be something on the public Internet rather than CF’s backbone.
- jchw 6y agoSomething’s wonky, because it’s not just Cloudflare. One of my personal sites is down that uses nothing but a VPS, and I noticed my Unifi AP disconnect from its controller a little bit ago. Fiber cut? Routing issues?
- parliament32 6y agoIf that VPS is on DO they're down too cause of CF. Or if you set the resolver on your VPS to 1.1.1.1 that's also down.
- jchw 6y agoWhy are digital ocean VPSes down due to a Cloudflare outage? Hoping for a clarifying post mortem...
- unilynx 6y agodigitaloceans VPSes weren't down, but there do seem to be routing issues as TransIP can't reach DigitalOcean AMS3 (but it's all coming back now) Maybe the problem was somewhere on the AMS-IX
- drchiu 6y agoMy Digital Ocean load balancer went down. I think there's probably some internal routing? Would be interested to understand more.
- dpcx 6y agoDO is still up as my machines are still up and accessible.
- rconti 6y agoHuh. My Ubiquiti was reporting WAN link down during this outage. I'm using ATT fiber. I'm wondering if "link down" doesn't mean what I think it means. Now that I check, it says "WAN iface [eth2] transition to state [inactive]". I'm wondering if that means link down or if it's doing service checking. I actually have a WAN2 configured but not plugged in and it was set to "Load Balancing: Failover Only" ... I wonder if all of my 'connection issues' were software assuming my network link is down and switching interfaces to an unavailable one.
- Kye 6y agoThat would explain why Patreon is down. I was going to post a little frog I took a picture of on Lens. Went down just as I opened the app.
- maxioatic 6y agoRIP someone's weekend
- davexunit 6y agoHappy Friday, everyone!
- typingmonkey 6y agoI was trying to fix my router the last 15 minutes :)
- arkitaip 6y agoSame here. Only figured it out because just one of the computers uses Cloudflare dns and the others were fine...
- ghastmaster 6y agoDitto except visa versa. My machine is set to the router which uses cloudflare. Other machines use whatever is default for mac(I try not to touch those). Once I realized they were working and I could access internal network from outside, I started diagnosing DNS. Came here via 8.8.8.8.
- ummonk 6y agoYeah I was waiting for it to fix then tried cell phone and realized that was down too. I assumed it was an issue regional / backbone routing or something. Especially cause status pages which I wouldn't expect to be hosted on AWS (because of the need for status pages to stay up when AWS goes down) seemed to also be down. Didn't realize it could be Cloudflare...
- hugoromano 6y agoI can only see the dashboard down, all my sites with Cloudflare are up.
- megadethz 6y agohn algolia search broken
- redox_ 6y agoyes hn.algolia.com is powered by Cloudflare /o\
- andrewnicolalde 6y agoWas wondering why my DNS wasn’t working...
- deleted 6y ago[deleted]
- bartwe 6y agoDNS is also (partially) down with my ISP (xs4all.nl) it seems
- synack 6y agoI can't change my NS records to point to a different DNS provider because my registrar, Namecheap, also uses Cloudflare. Didn't expect that.
- Aldqueath 6y agothis is great, i already have bad enough internet (rural area with 3 to 6 digits latency and average 4 digits, barely a few kilobytes of speed) and having both google smearing everywhere their recaptchas that are not really friendly toward low speed internet / non chrome users and cloudflare proxying half the internet but lately not really doing a great job at keeping a consistent uptime does not help much at least i am glad hn exists, it is the only thing that loads everywhere
- RL_Quine 6y agoSome POPs are fine.
- deleted 6y ago[deleted]
- alfg 6y agoYep, Cloudflare, DigitalOcean and 1.1.1.1 down for me. I thought it was my internet and was so confused for a bit there.
- drchiu 6y agoJust got a whole bunch of alerts that my services are down. Tried logging into Digital Ocean (who it seems uses Cloudflare) to get it fixed. Could not access their dashboard to reroute things.
- mjayhn 6y agoThank you for the early Friday.
- html5web 6y agoCloudflare status site is also partially down. Some resources are not loading properly.
- zadkey 6y agoI noticed Udemy was down when I wanted to go to the next video I was watching.
- zadkey 6y agoAnd now it's back up.
- solarkraft 6y agoMost pages mentioned here seem functional again.
- Rockjodd 6y agoFor people who reports this is down, which country are you in? Because all the reported sites works flawless from Norway (Europe) :-)
- Filligree 6y agoIreland here, and it's all down.
- britmob 6y agoEast Coast USA here. Any cloudflare site is unreachable, and 1.1.1.1 is giving me massive latency and packet loss :)
- joshstrange 6y agoUS here, everything reported that I've checked is down. My Cloudflare sites are down as well.
- akuji1993 6y agoGermany, everything down.
- looperhacks 6y agoInteresting ... Germany here, too, but I didn't see any down sites
- mercer 6y agoI'm in Holland but everything came back up just now, so maybe you picked the exact right moment to check?
- rvnx 6y agoEstonia, everything working fine
- abafazi 6y agoAustralia, everything worked for me, speaking with my friends in the UK they are saying everything is down
- xenospn 6y agoYup. 1.1.1.1 stopped responding as well.
- Jonnax 6y ago340ms average latency to 1.1.1.1 and 47% packet loss. Many sites are down. But I guess that's the problem with CDNs.
- tomxor 6y agoThought I was going crazy for a second. This affects so many things it's scary, and Cloudflare status page has still not updated. HN got there first.
- nomdep 6y agoSo THAT is why the Internet was acting weirdly!
- alex_young 6y agoYesterday I noticed most of their lava lamps are out (which generate random bits). Perhaps these are a critical component. https://photos.app.goo.gl/g6eR8V2PSY3EVjCLA https://photos.app.goo.gl/g6eR8V2PSY3EVjCLA
- maxk42 6y agoI'm sure you were joking but they actually are: https://blog.cloudflare.com/lavarand-in-production-the-nitty-gritty-technical-details/ https://blog.cloudflare.com/lavarand-in-production-the-nitty...
- leijurv 6y agoSomeone needs to get these lava lamps plugged back in ASAP!
- ATsch 6y agoDespite them actually mixing it into their entropy pools, the lava lamps are still entirely for show. The noise of the camera sensor itself is going to contribute orders of magnitude more entropy than the slow movement of the lamps. It's not completely a fake stunt, but it's certainly headline-optimized.
- soup10 6y agoGood guy cloudflare giving programmers an early weekend.
- neurostimulant 6y agoBut overtime for their own programmers :)
- VectorLock 6y agoA lot of people are saying AWS. I'm having intermittent network connectivity issues intra-AZ, so perhaps they lost a data center or route flapped one.
- LeoPanthera 6y agoI'm having real problems with DNS, is this Cloudflare too? They say "All Systems Operational", so maybe not? Half the damn internet is not currently resolving.
- zubiaur 6y agoYes, same here. Changed DNSs to level 3, all better now.
- ricardo81 6y agoLast I read about 7 million hosts are behind Cloudflare. Maybe around 3% of the web, but who knows if that counts for critical assets etc rather than pages served. Shameful that so much of our decentralised web is so centralised and breakable in one place.
- athesyn 6y agoIronically https://downforeveryoneorjustme.com/ https://downforeveryoneorjustme.com/ is also down. HA
- ddevault 6y agoWill we take this as a much needed lesson about putting all of the internet's eggs into one basket? Probably not.
- stri8ed 6y agoIronic, isitdownrightnow.com is down. All my DigitalOcean instances are down.
- heliodor 6y agoFYI, PagerDuty is not loading! Time to go back to the drawing board, for a lot of us, to re-assess points of failure. Edit: many websites are failing to DNS resolve but the services they provide continue to function fine behind the curtain.
- twunde 6y agoThis is likely your computer's DNS resolver (if you're using 1.1.1.1 you're down. I'd switch to 8.8.8.8 temporarily. We've had pagerduty alerts coming in since the start (a whole bunch of DNS errors from pingdom) and when I click on the slack link, pagerduty works for me
- ed25519FUUU 6y agoIt's unfortunate that both the primary and secondary cloudflare DNS is down. I just switched my secondary to google. This allows my internet to "work" during this time, but adds about 1s latency to resolutions. Presumably that's the time it takes my internal DNS resolver to try the secondary.
- ingenium 6y agoConsidering running your own full resolver like unbound. Then you don't have to rely on a DNS provider like Google or Cloudflare. It's really nice not having the whole internet go down when Google or Cloudflare DNS is down.
- ashleyn 6y agoIt really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at once. I thought for a second that I lost internet in my own house.
- rickyc091 6y agoSame here. Rebooted the router and modem thinking it was me, but my phone was still on wifi then realized it was probably my cloudflare DNS.
- asadlionpk 6y agoThis! I got all sorts of alerts from pingdom and my laptop refused to get online. Pure Panic!
- spiritplumber 6y agoPihole is your friend.
- cls59 6y agoYeah, Pihole made it super easy to cut over to Quad-9 once I figured out what the problem was.
- rickyc091 6y agoLooks like I got another weekend project.
- newhotelowner 6y agoI have a pihole. It didn't help.
- xen2xen1 6y agoYup, reinforces the thought that you never have both DNS servers with the same service.
- jeremyjh 6y ago
- BookmarkSaver 6y agoTwitch.tv channels are like 50/50 right now. Some are ok, some aren't. Basically all Riot Games (League, Valorant, TFT) are down, dunno about LoR.
- jrockway 6y agoI don't use Cloudflare, but I do notice Cloudflare services being down. Right now, I can't get to my own website (hosted on DigitalOcean, not through Cloudflare), but Oh Dear claims it's up. So I suspect that the problem is closer to me than it is to DigitalOcean (or Cloudflare).
- minxomat 6y agoDO might use 1.1.1.1 (or Argon even) for routing between some of their PoPs
- jrockway 6y agoCould be. Things are back now, but I was very surprised that a Cloudflare outage makes it impossible for me to get to my Kubernetes API server. Hidden dependency revealed.
- rgbrenner 6y agoDO uses cloudflare for their DNS... both for digitalocean.com and for their DNS service.
- jrockway 6y agoGood to know! That makes perfect sense based on what I saw during the outage. I had no idea.
- semicolon_storm 6y agoMust be regional or some other factor involved. Various sites others are reporting as offline load for me as does 1.1.1.1.
- exochrono 6y agoMy Pagerduty's been blowing up so I tried to go to their dashboard to pause the notifications for now and pagerduty.com is down XD
- ryanmccullagh 6y agoI thought my issue was with Comcast, then I realized I'm using CF's DNS entries for my home network. I removed those 1.1.1.1 entries and some sites are working. Dang, I'm pretty disappointed in CF. I've never experienced this much an effectful DNS outage.
- julien 6y agoI am probably the only one who cares but even https://downforeveryoneorjustme.com/ https://downforeveryoneorjustme.com/ is down
- bdibs 6y agoIt's up for me.
- dayjah 6y agoFirst place I went to also. Then HN.
- clairegraham 6y agoYep, we are very dependent on Cloudflare :(
- ATsch 6y agoI personally find sites like https://outage.report https://outage.report or https://downdetector.com https://downdetector.com which tally up the number, regions and history of people saying it isn't working for them more conclusive.
- djsumdog 6y agoWeird, it works for me (US)
- Reedx 6y agoNow we need isdownforeveryoneorjustmedown.com
- jlmorton 6y agoMonday Morning RCA: "We pushed out some routine code updates, but this really weird thing happened causing a resource utilization spike on our DNS systems. Because of this other really weird thing, this affected all of our global infrastructure simultaneously. Here's a deep engineering dive into this one weird thing that brought everything down."
- iJohnDoe 6y agoIronically StatusCake is down as well.
- gautamcgoel 6y agoI was trying to play video games but couldn't connect. Amazing how connected the web is now - one big hub goes down and brings the whole house of cards down with it.
- deleted 6y ago[deleted]
- heliodor 6y agoWebGazer.io managed to shoot me an email about my site being down. This in spite of their site being down too.
- th0th 6y agoHi @heliodor, WebGazer founder Gokhan here :) Actually the site is running but not accessible due to the issue. Glad you got the heads up, after a while I had to pause monitoring to prevent side effects.
- rozab 6y agoWe can't keep going on like this. The vulnerability of centralised internet infrastructure is a huge problem for everyone. Somebody, somewhere, really ought to sort it all out
- oliverobscure 6y agoIf it impacts enough wallets, things might change. I'm not holding my breath though.
- parliament32 6y agoWhy not you? Just don't use CF. The more people stay away from CF, the better.
- adsjhdashkj 6y agoI feel like for a lot of sites CF & CDNs are the only way to survive Reddit/HN/etc - do you disagree? I definitely agree in concept with you, but then i think back to how frequently script kiddies took down sites ~10 years ago, or w/e. I feel like what has changes is the massive CDNs in front of so many sites. So while i do want a better solution, i'm not sure what it looks like. Thoughts?
- mongol 6y agoHow does Cloudflare compare to Akamai?
- byteofbits 6y agoIt’s worth mentioning here that 1.1.1.1 is also affected by this outage which initially made me think my internet was gone completely. Changing back to an alternative (such as 8.8.8.8 from google) restored my access to the areas of the internet not using Cloudflare.
- jorgenphi 6y agoThe uptime tool I use (StatusCake) is itself down... Was wondering why I didn't get an alert.
- tomklein 6y agoBack online for me.
- xtracto 6y agoThis is hitting my production environments as well :-(
- decad 6y agoDNS seems to be resolving for me in the UK now
- Hanabishi 6y agoWell, sheit. This is all around the world. Press F.
- formerly_proven 6y agoCentralising on a single host suddenly not a good idea any more?
- awinder 6y agoNextDNS got taken out by this, id been really happy with it up until now. And unfortunately “dns service went down” has a wide enough blast radius at home now that it’s a real pain.
- ricopags 6y agoHow did you verify that? I determined the issue was with Cloudflare's DNS by toggling on NextDNS, which worked and continues to.
- awinder 6y agoMost of my devices went belly up and was trying to figure out what it might be (I run NextDNS on my router), switched off to cell and noticed discord was down too so started thinking about NextDNS. I toggled dns to google and noticed it immediately work.
- mindfreeze 6y agoI was having troubles with overleaf.com
- andrewnicolalde 6y ago1.1.1.1 is back for me now
- rgbrenner 6y agoReminder for firefox users: Firefox uses DNS over HTTPS and the default is cloudflare. If you're having DNS issues, you need to disable it until cloudflare is back up.
- mxschmitt 6y agoSite that use Discord, Linode, Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, Gitlab (502), Fiverr, Upwork, Udemy and many more including 1.1.1.1 dns down. Ref: https://twitter.com/nixcraft/status/1284239374809395200?s=19 https://twitter.com/nixcraft/status/1284239374809395200?s=19
- icey 6y agoSeeing a lot of people mentioning DO, but it has been up for me without any issues (small VPS in SF-2)
- interator7 6y ago2:36 PM PST - status.discord.com is back up.
- britmob 6y agoLooks like it's back. No longer getting issues with 1.1.1.1 and domains are being resolved!
- unilynx 6y agodigitalocean.com DNS (on cloudflare) is now resolving again. looks like several things are coming back now.
- emeraldd 6y agoLooks like Digital Ocean is reporting an issue with their upstream provider: https://status.digitalocean.com/incidents/6wtmldty17g1 https://status.digitalocean.com/incidents/6wtmldty17g1 As big as this is, any chance a major hub/backbone went down?
- usr1106 6y agoIt shows "Minor system outage" when I load the page, but it switches to "All systems operational" immediately. Same behaviour on several attempts.
- iamtheyammer 6y agoAppears to be working for me now ; <<>> DiG 9.10.6 <<>> discordapp.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8092 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;discordapp.com. IN A ;; ANSWER SECTION: discordapp.com. 140 IN A 162.159.135.233 discordapp.com. 140 IN A 162.159.129.233 discordapp.com. 140 IN A 162.159.130.233 discordapp.com. 140 IN A 162.159.134.233 discordapp.com. 140 IN A 162.159.133.233 ;; Query time: 69 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Fri Jul 17 14:37:40 PDT 2020 ;; MSG SIZE rcvd: 137
- devy 6y agoLooks like CF is up!
- MH15 6y agoBack online just now for me in Midwestern US.
- sdenott 6y agoLeague of Legends down too, not sure if related.
- greggyb 6y agoDNS resolution at 1.1.1.1 seems to have gone down and come back up for me in the course of 10-15 minutes.
- bitclaw 6y agoSeems like it's starting to come back.
- clairegraham 6y agoWe were down (downforeveryoneorjustme.com) completely, but back up now (as of a few minutes ago). Our domain wasn't even resolving; we use Cloudflare for frontend and DNS. We had a surge of people checking if Discord was down on our site, then I noticed everything went down shortly after. Discord is still the top check right now. I can't ever remember hitting these kind of traffic numbers before.
- ricardo81 6y agoInteresting data you get in the face of adversity, providing your host resolves!
- wcchandler 6y agoI enjoy your service. Have you ever thought about expanding your offerings? I would love to see a recreation of "Internet Pulse"
- clairegraham 6y agoThanks! Yep, we have a lot of things on the todo. We want to add more user-focused / location-based outage information since our site is still too reliant on simple HTTP checks to report downtime. This is especially a problem with a Discord outage, for example, where the frontend website is not down, but there might be problems with the API, apps, or other components. And I'd like to be able to have our site communicate outages like this Cloudflare one, where more than one site might be affected by a larger provider. Automating that is difficult. This is still a side project, though, so I mostly work on it when I get the urge :)
- wcchandler 6y agoThat would be pretty interesting. Being able to drill down into individual pieces of a stack would be very informative, especially from a parent source. I bet a lot of services would even have that information readily available in their documentation for APIs. How comfortable are you with open source? If you were willing to release your stack on Gitlab/Github, it might be worth your while.
- DangerousPie 6y agoIs this US specific? Everything seems to work fine here in Europe.
- kords 6y agoIt's back now (at least for me).
- jonplackett 6y agoSeems like it’s back right?
- elviswolcott 6y agoThe status page is now showing degraded performance for the Cloudflare API and Recursive DNS.
- rob-olmos 6y agoHopefully with this outage Cloudflare will finally provide non-Enterprise plans a CNAME record, allowing us to quickly bypass Cloudflare.
- amasad 6y agoLooks like it's resolved -- we're coming back up at Repl.it.
- cartoonfoxes 6y agoBack online here.
- fluxsauce 6y agoMy modem also disconnected with signal problems, which was interesting. I'm not sure Cloudflare could have caused that?
- lgats 6y agomaybe your modem uses 1.1.1.1 dns?
- bryan_w 6y agoThose poor Firefox users who enabled DoH
- chris_engel 6y agoSeems to work again for me in germany (Frankfurt)
- mauriciogior 6y agohow can I have cloudflare plus something else as a DNS failover? We are afraid to set a long TTL and have our IP changed for some reason. What do you guys recommend?
- haloblue 6y agoLooks like it's starting to come back in the SE US.
- neurostimulant 6y agoSeem to be localized issue. Cloudflare is up here in my country, but down for many people in US.
- salmaanp 6y agowho deployed on a friday afternoon?
- sbr464 6y agoYep, the DoorDash app is affected currently, my burritos!
- brycewray 6y agoVercel also appears to be dropping out and coming back in intermittently over the last 30 minutes or so. Not aware they're using Cloudflare, although they do mention using AWS.
- xen2xen1 6y agoRemind me to check and see that I have 8.8.8.8 and 1.1.1.1 on my networks, not just one or the other..
- kube-system 6y agoOf course, I read this after I spend an hour debugging some strange DNS issues.
- r1ch 6y agoInterestingly this seemed to only affect resolver service. I use Cloudflare pretty extensively on all my sites, but only in DNS mode (no CDN / proxy). The hosts continued to resolve fine during the outage (following root DNS resolution chain, no recursive resolver involved). I imagine their CDN internally uses their resolver service which explains the outages, and some unrelated 3rd parties who don't use CF on their domain at all still created a hard dependency on CF by using their recursive DNS server.
- randomstring 6y agoWhat luck, I chose today to install a new piece of network gear. I thought I had managed to totally FUBAR my network. DNS was failing, "ping 1.1" (my current goto test "Am I connected to the internet?" as it requires the fewest keystrokes and hits the Cloudflare DNS 1.0.0.1) failed and I just assumed it was my fault. Backed out my changes, and discover in fact, the internet was down.
- BenjiWiebe 6y agoPing 1.1... thanks for that!
- pgrote 6y agoDid anyone else see their ATT internet go down? The DNS issues started and then the Pace 5268AC rebooted. I don't use cloudflare for dns. Does ATT's backend?
- MertsA 6y agoOn the contrary, ATT actually squats on the CloudFlare DNS IP address. IIRC that modem is one of the affected ones where it uses 1.1.1.0/24 internally. You shouldn't even be able to use CloudFlare DNS normally.
- pgrote 6y agoThank you for the response and information. Unhappy coincidence.
- askbill 6y agoFelt like a BGP issue.
- mathattack 6y agoInteresting. They had an outage in the midst of a negotiation I was a part of. Are they less stable than Akamai and the others?
- wenbin 6y agoFriendly reminder (and notes to myself): Don't use Namecheap and Cloudflare at the same time. Namecheap is using cloudflare. So if cloudflare is down, you can't change DNS settings on Namecheap as well!
- adrr 6y agoWhat was interesting and scary is that our monitoring system didn’t notify us. Our email was down because we use cloudflare for DNS and our monitoring provider’s SMS gateway was down. So we didn’t get sms messages.
- Exuma 6y agoAside from this one issue, is switching to 1.1.1.1 a good idea in your guys experience? Right now I just realized I hvae the DNS for my ISP which is probably how they inject bullshit 404 pages full of ads. What is the fastest/best public DNS in your guys experience?
- xur17 6y agoI've been pretty happy with 1.1.1.1 (before now). Might be worth using something like 8.8.8.8 as a backup (Google).
- vulcan01 6y agoYou could also use 9.9.9.9 as your backup, if you're avoiding Google (https://www.quad9.net https://www.quad9.net)
- BenjiWiebe 6y agoFor us, it's cloudflare. Our ISP is connected to KCIX, and cloudflare apparently has 1.1.1.1 servers in Kansas City. No other free DNS provider is as quick, for us. 18ms or so RTT, as opposed to the WISP's internal latency of ~10ms. Central Kansas.
- 77ko 6y agoI've been using https://nextdns.io/ https://nextdns.io/ - works fast and most importantly blocks a bunch of adds (user configurable), so makes browsing on mobile much nicer. Ancedotally the interenet seems faster.
- jgrahamc 6y agoThis was a problem with our backbone network; wasn't caused by an attack. The effect was regional and not global. Naturally, we'll write it all up.
- EE84M3i 6y agoWas it a problem with a provider you use?
- jgrahamc 6y agoLooks like problem with one of our large routers in Atlanta.
- 0x0000000 6y agoLooking forward to the writeup- were these IGP routes?
- nodesocket 6y agoI am in Nashville, and lost 1.1.1.1 and 1.0.0.1 completely. Also one of my sites went down according to statuscake, but not all of them.
- deleted 6y ago[deleted]
- Exuma 6y agoUpdate - This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some potions of the network to not be available. We believe we have addressed the root cause and monitoring systems for stability now.
- arjun27 6y agomore like Cloudflared
- floatingatoll 6y ago> * This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some portions of the network to not be available. We believe we have addressed the root cause and are monitoring systems for stability now.* Jul 17, 22:09 UTC
- tikiman 6y agoI'm surprised so many people still use them. They took my business down (along with half the internet) a few years ago and I learned that they were to large of a point of failure.
- blisseyGo 6y agoA lot of unusual internal traffic seems to be around Thailand (you might have to select "UNUSUAL"): https://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=1&time=18458&view=map https://www.digitalattackmap.com/#anim=1&color=0&country=ALL...
- lgats 6y agobeen pretty large numbers from thailand this month according to that tool
- ranrub 6y agoCedexis gets another lease on life
- chuckdries 6y agolmao it even took down my local stack
- bhaak 6y agothe internet was built to withstand a nuclear war brought down by cloud flares
- easytiger 6y agoBoats. People. Stop putting them in one single boat.
- pier25 6y agoA tweet from Cloudflare's CEO: > Appears that the router in Atlanta announced bad routes (effectively a route leak). Only impacted our backbone. Not all of our PoPs are connected to our backbone, so some would not have seen an issue. Appears to have impacted about 50% of our traffic for a bit over 20 min. https://twitter.com/eastdakota/status/1284259895475236865 https://twitter.com/eastdakota/status/1284259895475236865
- tompccs 6y agoI was paying a bill in a bar in London about an hour ago and they couldn't process any payments. Seems likely related.
- noobermin 6y agoPeople talk about a single point of failure then go on to depend on a single point of failure.
- kebman 6y agoOh, did hacker season just start? * Grabbing popcorn! * So this was an accident, or is it connected to the Twitter hack?
- peterwwillis 6y agoYesterday CloudFlare took down some of our products because they (not us) misconfigured some DNS thing. Kind of funny to see it happen again a day later.
- nodesocket 6y agoThis was especially bad because I use Cloudflare public DNS exclusively at my house and it went down as well. I didn’t even think to check DNS, I just assumed it was AT&T being shiest AT&T. I should probably run a blend of 1.1.1.1 with 8.8.8.8 instead.
- rv-de 6y agoWhat I found particularly interesting is that my MacBook Pro (work laptop) didn't start up anymore properly and I wasn't able to start applications ... sorry, but wtf. Now I hate Apple and their shitty, overpriced products even more.
- sys_64738 6y agoFive nines uptime?