14 ms·

Cloudflare Network Performance Issues

631 points by drcongo 7y ago

buildbuildbuild 7y ago
Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you. https://i.imgur.com/qHBM2JW.png https://i.imgur.com/qHBM2JW.png
- Heliosmaster 7y ago
  Are you sure it wasn't cached? For us it was showing correctly the incident...
  - wepower_ico 7y ago
    Can confirm that we had the same view. Operational while not working.
  - grey-area 7y ago
    All I saw was a vague message at the top, all services were marked green and operational throughout.
  - btown 7y ago
    If cloudflarestatus.com isn't serving text content as Cache-Control: no-cache, that's itself an egregious bug.
    - grenoire 7y ago
      It serves Cache-Control: max-age=0, private, must-revalidate. Same thing as far as I get it, maybe wider support.
    - buildbuildbuild 7y ago
      Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.
      - auscompgeek 7y ago
        Is it? From my end it looks like a plain ec2 instance from statuspage.io.
      - jmngomes 7y ago
        Cloudfront supports Cache-Control: no-cache
        buildbuildbuild 7y ago
        Cloudfront itself can cache the content regardless. (if its headers are showing HIT)
  - acdha 7y ago
    It might have been cached on the server but was not cached on my clients: I did a forced refresh and used a private browser window but it only showed the “almost everything is good” view throughout the duration of the incident. The actual incident page was correct.
- nullify88 7y ago
  We saw an update within 5 minutes of our sites going down. Are you doing caching somewhere?
  - buildbuildbuild 7y ago
    Interesting, I was command+shift+R refreshing and also tried from a VPN in another region. Perhaps our CF-hosted sites returned 502s in my region sooner than yours, causing me to check the status page sooner.
  - seiferteric 7y ago
    Seems unlikely, I don't use cloudflare have never visited the status page before. I noticied several 502 pages this morning and searched for "cloudflare status page" and saw the "all systems operational".
- eeeeeeeeeeeee 7y ago
  It was showing correctly for me the first time I checked it.
- matt_oriordan 7y ago
  Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service https://www.ably.io/blog/honest-status-reporting-aws-service. At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647 https://status.ably.io/incidents/647 We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.
  - MrGilbert 7y ago
    slightly off-topic: Nice and clean, yet detailed enough status page. I like it. :)
    - matt_oriordan 7y ago
      Thanks :)
      - felideon 7y ago
        On the OT subject, the status logo is blurry (at least on a 4k monitor), whereas on the main website it's nice and crisp.
        matt_oriordan 7y ago
        Ok, thanks for the heads up. Will raise a status website issue.
  - joncrane 7y ago
    >How is it that hundreds of developers know there's an issue before AWS do Trust me, they know. They know about problems we never find out about, too.
- kreetx 7y ago
  They should derive whether something is down from how much traffic the status page gets, and have alerts tied to that as well. I think it would be pretty accurate.
- gsich 7y ago
  Why even have a status page if it doesn't reflect reality?
- takeda 7y ago
  It isn't in their best interest to have a reliable status page. They want status page to contain information about failure only after people know about issues through other means. For the same reason cloud providers don't provide rebates on outages unless you ask for it explicitly. I think the only way is to have status page operated by an independent 3rd party, but I don't think there's a viable business model for someone to provide such service. Perhaps there might be even a risk of lawsuits against you.
_yesmeck 7y ago
From https://www.cloudflarestatus.com's https://www.cloudflarestatus.com's response header, it seems it runs on fastly, so it's not down.
lol768 7y ago
Yes, seems to have impacted all CF sites. (UK here) Status was just updated here, but it was showing everything as operational for a while: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
tmlee 7y ago
So it seems... Even https://cloudflare.com/ https://cloudflare.com/ itself is down
- ratsimihah 7y ago
  So cloudflare.com runs on Cloudflare?
  - dexen 7y ago
    Turtles all the way down. But in all frankness, if Cloudflare's own site did not run off of Cloudflare's infrastructure, why would anybody trust them with their websites?
    - beckingz 7y ago
      Ironic isn't it.
  - OskarS 7y ago
    Certainly what I would use if I was building cloudflare.com...
  - kreetx 7y ago
    For the site itself that's totally fine, and the status page is separate as it should be, too.
    - ratsimihah 7y ago
      Fair. Some service recently had their status page hosted on the same server that went down, that was funny.
      - apple4ever 7y ago
        Discord's status page is also down haha
khebbie 7y ago
I see it as well
chromaton 7y ago
dammit
cocoflunchy 7y ago
Wow... our site went down so I went to cloudflare.com -> 502 bad gateway
Raphmedia 7y ago
It is. https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/
yuchi 7y ago
Either downforeveryoneorjustme.com is itself served by Cloudflare too, or has been hugged to death.
- close04 7y ago
  It returns 502 Bad Gateway cloudflare so I assume it's not the HN hug...
- ttctciyf 7y ago
  Also, https://downdetector.co.uk/problems/cloudflare https://downdetector.co.uk/problems/cloudflare is currently detecting the outage in very direct fashion!
- bwb 7y ago
  Yep, we are 100% on CF workers :) Here is a quick image of the peak downtime on downforeveryoneorjustme.com: https://ibb.co/PZ9BMRc https://ibb.co/PZ9BMRc
  - yuchi 7y ago
    Thanks for the fantastic work on the service. And thanks for sharing that stats!
seibelj 7y ago
Single point of failure - we all shouldn’t trust CF alone anymore...
- zzzcpan 7y ago
  Any CDN is a single point of failure and limits your availability to as low as three nines. Although anycast-based CDNs like Cloudflare are much less reliable than DNS-based CDNs, those can do orders of magnitude better.
  - cortesoft 7y ago
    Why do you think anycast based CDNs are worse than DNS based ones?
    - zzzcpan 7y ago
      They rely on a single network infrastructure as opposed to many independent networks with independent edge nodes where isolating faults is rather trivial in comparison.
- deleted 7y ago
  [deleted]
- shard972 7y ago
  but its free
bodono 7y ago
I'm getting '502 Bad Gateway' on all Cloudflare sites here in London too.
jf- 7y ago
Also getting this in the UK. Not completely down, bits and pieces of medium comes through but very slowly and incomplete. I’m also unable to access npm.
tcarn 7y ago
Holy shit. This impacts nearly everything.
auscompgeek 7y ago
https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr (edit: thank you mods for changing the submission URL)
- bytebuster 7y ago
  oh wow. that's a long list.
  - yoran 7y ago
    aka the whole world.
johnstonnorth 7y ago
Yes - seeing lots of our sites offline as of 14:45 GMT.
deleted 7y ago
[deleted]
quicksilver03 7y ago
Several sites behind CloudFlare are returning 502 errors for me as well (France).
deleted 7y ago
[deleted]
perprit 7y ago
omg I was not the only one.
Nas808 7y ago
1.1.1.1 DNS is down, and seeing a lot of 502s on cloudflare sites.
- ttctciyf 7y ago
  I just changed my dns-over-https script to use quad9: # sudo /usr/local/bin/dnsproxy -l 127.0.0.153 -p 53 -u https://cloudflare-dns.com/dns-query -b 1.1.1.1:53 sudo /usr/local/bin/dnsproxy -l 127.0.0.153 -p 53 -u https://9.9.9.9/dns-query The bootstrap 1.1.1.1:53 was working but the DoH dns-query url was not. Hence my visit here :) Edit: Now fixed, it seems. Quick work, if it stays up!
- NKCSS 7y ago
  That's a biggie...
- jakear 7y ago
  The 1.1.1.1 vpn still seems to be working
  - eternalny1 7y ago
    Not here it's not (NE US).
    - wilsonfromdevon 7y ago
      dns is working in London UK.
bytebuster 7y ago
digitalocean.com is down as well (seems to affect cloudflare-gated websites)
chadmckenna 7y ago
Looks like it, one of my sites is cached through Cloudflare, I'm getting a 502 from them. I'm seeing other random sites down as well.
jason_zig 7y ago
So what should the ideal redundancy plan be here? If you can't log into the CDN provider and they are down do you you just have a second one ready (and paid for) and then log into your registrar and be ready to switch to that secondary CDN provider in this scenario? Or is there some sort of load balancing / routing solution between CDN's that I don't know about / understand?
- morpheuskafka 7y ago
  If you use Cloudflare nameservers, you have to change to new nameservers, wait for that to propogate, and then wait for clients cached records TTLs to expire. So it will be a major disruption no matter what you do.
  - apple4ever 7y ago
    What about the API? If that is up, you could script it.
  - manigandham 7y ago
    If you're using them for TLS certs then it's an even bigger problem unless you have them provisioned elsewhere.
    - morpheuskafka 7y ago
      Unless you need EV you can just pull some wildcards from Lets Encrpt (as long as you don't use pubkey pinning). No need to automate as it's just a one off.
  - bowmessage 7y ago
    Can you use your own nameservers that delegate to cloudflare or akami 50% each, then adjust? Is there a service more suited for this than r53?
- deleted 7y ago
  [deleted]
jafingi 7y ago
Noticed that draw.io were down. 1.1.1.1 and all other Cloudflare backed sites as well... I'm off for today.
- auscompgeek 7y ago
  Cleartext DNS 1.1.1.1 is still up though, thankfully.
oliverrowlands 7y ago
What will we do without our favourite "Enterprise MiTM" SaaS provider?
stedaniels 7y ago
Cloudflare DNS 1.1.1.1 down too
- anoldgangstah 7y ago
  Good thing I switched to OpenDNS literally two days ago. edit. not that it matters, I'm just scrolling 9gag.
40four 7y ago
And so it begins......
johnxie 7y ago
Looks like everyone is impacted. We are seeing 502 Bad Gateway across multiple domains and regions on https://taskade.com https://taskade.com
- johnxie 7y ago
  Back up and running!
gadgetoid 7y ago
I just changed a setting in my CloudFlare account... did I break everything?
- darkcha0s 7y ago
  Oi! Change it back!
  - Gigablah 7y ago
    Well, they can’t now :p
    - gadgetoid 7y ago
      Touché. On one hand I think "Maybe I should diversify my infrastructure." And on the other I think "But one of the biggest upsells was convinience." And it's fortunate I don't have a third hand, because I'd be thinking "Oh crap oh crap I just migrated a client website to LightSail + CloudFlare saying how super awesome and robust it would be." But it's okay now because it looks like everything is back up!
- jgrahamc 7y ago
  No
Thaxll 7y ago
After trolling Verizon curious what reasons they will come with for that outage.
- apple4ever 7y ago
  Maybe this is payback by Verizon...
- voidwtf 7y ago
  Care to elaborate how they were "trolling" Verizon?
  - EamonnMR 7y ago
    They came off that way in this blog post: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...
    - acdha 7y ago
      That didn't seem like trolling – just a public call for Verizon to follow internet best practices. Given that most large ISPs treat failures as a PR exercise, that's probably necessary.
      - voidwtf 7y ago
        Agreed. They reached out to Verizon privately, a Tier 1 carrier with expectations and responsibilities as a good netizen, and got no response. They attempted to reach out through Verizon's public forms of communication and got a bullshit irrelevant CS response despite requesting escalation. They then called out Verizon before the community as a whole. They don't have the luxury of waiting for a well prepared letter from some Verizon lawyers. Modern day customer expectations don't allow for it. You may call it trolling, but all I saw was a company asking another company to stop pissing in the public pool.
ratsimihah 7y ago
This article entitled Now Running on Cloudflare seems down https://seankilleen.com/2016/12/now-running-on-cloudflare/ https://seankilleen.com/2016/12/now-running-on-cloudflare/
sprite 7y ago
When you can't update your DNS because your registrar uses CloudFlare also....
- stevekemp2 7y ago
  You could use a dedicated dns provider, such as AWS
lukevp 7y ago
Our website is also down, origin near Dallas.
deleted 7y ago
[deleted]
kortilla 7y ago
Can’t wait to see Verizon’s blog post about this one :)
nnx 7y ago
Damn... getting a bunch of alerts and can't even open Pingdom either... also running on CloudFlare https://my.pingdom.com/newchecks/checks# https://my.pingdom.com/newchecks/checks#
deadalus 7y ago
8chan, 4chan down as well. Damn, this is big. https://cloudflare.com https://cloudflare.com itself is gone.
- chasd00 7y ago
  that's bad news, i like 4chan being up because it keeps them all in one place and they don't start wandering around.
danShumway 7y ago
Don't want to go off topic, but if I want to prevent my website going down because of stuff like this in the future, will having back up DNS entries solve the problem? I know DNS will fall back if it can't reach a service, but would a 502 trigger that?
- rosswilson 7y ago
  If you received a HTTP 502 then DNS must've already resolved. Browsers typically will do a DNS lookup, and then try establishing a TCP connection to one of the returned hosts. Its only if it can't establish a TCP connection to a host will it (sometimes) try another host from the DNS response.
- lol768 7y ago
  No, I don't believe it would.
- SteveNuts 7y ago
  No, you'd need to use an application-aware DNS provider like Route 53 that could detect the failure. Backup DNS entries won't help, what you need is to use multiple DNS providers and add them all to your NS records
  - danShumway 7y ago
    I think this is the correct answer. Since I'm mostly thinking about static sites, I could also have something local that pings the site, and if it goes down, it could update my nameservers to point elsewhere on its own. Probably more trouble than its worth though.
- aclelland 7y ago
  Yeah, you'll want to keep your domain registered through a different registrar and if CF goes down you can update your DNS Name Servers point from CF to something like AWS Route 53. This has a few drawbacks like making sure your Route53 configuration is identical to your CF config, ensuring your origin servers can cope with the additional load if CF caching isn't available and the DNS propagation time required for the Name Servers to update. During the last outage, we were able to get into the CF dashboard and simply disable the proxy which allowed our clients to access our origin servers directly but this time we can't even get into the Dashboard.
jarym 7y ago
Looks like a flare up (sorry my attempt at humour always gets the better of me...).
heavymark 7y ago
Holy crap, cloudflare down, and seemingly all the covers. Major sites such as Digitalocean are all down, and no way to easily disable cloudflare since their site is down.
- hu3 7y ago
  https://www.digitalocean.com/ https://www.digitalocean.com/ frontpage working for me: https://i.imgur.com/8yJRXVI.png https://i.imgur.com/8yJRXVI.png
  - bpicolo 7y ago
    The outage appears to have ended between that post and your post.
    - rc_kas 7y ago
      Yeah, not a long outage, but a big one.
akriukas 7y ago
Same here.
maple3142 7y ago
The outrage this time seems to be more serious than last time. More websites including Cloudflare itself are down.
- sauldcosta 7y ago
  Without question. I've never seen an affected regions list that immense.
max0563 7y ago
Cloudflare is too big.
isatty 7y ago
Pagerduty’s main website seems to be affected.
jakemcgraw 7y ago
was hoping John Graham-Cumming would be in here to give us the behind scenes :(
exdsq 7y ago
Can someone from Cloudflare give us a hint at what's going on?
nodesocket 7y ago
July 2nd, 2019... The day the internet broke.
- Endy 7y ago
  The day the internet broke again.
- triod 7y ago
  Yup, we could have "Days without an accident" counter for the whole internet...
thequailman 7y ago
This seems to be impacting NPM for me.
pmlnr 7y ago
Good. I'm tired of people not learning that trusting a single gateway with 50% of the internet is bad. Yes, I know, free DDOS protection. There has to be another way of doing this, some mesh based DDOS protection or so.
- Avamander 7y ago
  With a lot of these things, if people can't agree to make a standard system to do all of it just as well then there's going to be a big company that does so. We have had this with e-mail, signed exchanges, DDOS protection and will probably have it with many other things unless people pull themselves together at least after these proprietary solutions are created and create better alternatives.
- Waterluvian 7y ago
  I get your point. But I'm not sure I can ever grok the mindset of someone who thinks, "good, that'll teach 'em". Not sure I'd ever want to work with that individual.
  - bovermyer 7y ago
    This is important to recognize. Victim blaming is a highly destructive practice.
    - inimino 7y ago
      Assigning blame for technical business decisions to the people who made those decisions is victim blaming?
      - bovermyer 7y ago
        Targeting the process or behavior that lead to an event is far more effective than targeting the person that triggered it. Do not conflate the two.
  - drunkpotato 7y ago
hokumguru 7y ago
My shopify sites are down this morning, seems to be 502-ing because of this issue.
adamparsons 7y ago
I thought I was going insane, googling it returned nothing, but trusty old hacker news has my back
jshb 7y ago
Tried to go to some usual places where people discuss outages and met with outage. Ouch.
tnolet 7y ago
npm and yarn are not working either...there goes Javascript land...
cube00 7y ago
Not impressed it's serving an error page claiming the underlying host is to blame (this one from discord) Error 502 Bad gateway You - Browser - Working Sydney - Cloudflare - Working storage.googleapis.com - Host - Error What happened? The web server reported a bad gateway error.
minikites 7y ago
Wasn't the internet supposed to be decentralized to avoid this very problem?
- langitbiru 7y ago
  That is the mission of IPFS. However, there is a big gap between their mission and the current capability of their technology. Be the change you want to see in this world. I guess next time I build a personal website, I have to host it in Raspberry Pi at my home. :)
Vaskerville 7y ago
As long as Reddit is not down everything is OK.
maz1b 7y ago
Getting 502s as well.
damniatx 7y ago
oh my god,
deca6cda37d0 7y ago
7,5% of all websites... https://w3techs.com/technologies/details/cn-cloudflare/all/all https://w3techs.com/technologies/details/cn-cloudflare/all/a...
- hanspeter 7y ago
  Add to that all the sites using resources hosted on Cloudflare's CDN.
- snug 7y ago
  9.6% of websites that have one of those CDNs
  - deca6cda37d0 7y ago
    aah i misread so 7,5% of all websites
    - lgats 7y ago
      How about a weighted percentage of the Alexa top 1m?
- colinbartlett 7y ago
  That number matches what I am seeing on StatusGator: Of the 438 status pages we monitor, 52 of them are showing some kind of warn or down notice right now. That's almost 12%. Though some of them might not be because of Cloudflare, the ones I spot checked all do appear related. Medium, DigitalOcean, Shopify, CodeShip, Pingdom, and many more. The impact is staggering.
sb057 7y ago
>We are working to mitigate impact to Internet users in this region. >This incident affects: North America (Ashburn, VA, United States - (IAD), Atlanta, GA, United States - (ATL), Boston, MA, United States - (BOS), Buffalo, NY, United States - (BUF), Calgary, AB, Canada - (YYC), Charlotte, NC, United States - (CLT), Chicago, IL, United States - (ORD), Columbus, OH, United States - (CMH), Dallas, TX, United States - (DFW), Denver, CO, United States - (DEN), Detroit, MI, United States - (DTW), Houston, TX, United States - (IAH), Indianapolis, IN, United States - (IND), Jacksonville, FL, United States - (JAX), Kansas City, MO, United States - (MCI), Las Vegas, NV, United States - (LAS), Los Angeles, CA, United States - (LAX), McAllen, TX, United States - (MFE), Memphis, TN, United States - (MEM), Miami, FL, United States - (MIA), Minneapolis, MN, United States - (MSP), Montgomery, AL, United States - (MGM), Montréal, QC, Canada - (YUL), Nashville, TN, United States - (BNA), Newark, NJ, United States - (EWR), Norfolk, VA, United States - (ORF), Omaha, NE, United States - (OMA), Phoenix, AZ, United States - (PHX), Pittsburgh, PA, United States - (PIT), Portland, OR, United States - (PDX), Queretaro, MX, Mexico - (QRO), Richmond, Virginia - (RIC), Sacramento, CA, United States - (SMF), Salt Lake City, UT, United States - (SLC), San Diego, CA, United States - (SAN), San Jose, CA, United States - (SJC), Saskatoon, SK, Canada - (YXE), Seattle, WA, United States - (SEA), St. Louis, MO, United States - (STL), Tampa, FL, United States - (TPA), Toronto, ON, Canada - (YYZ), Vancouver, BC, Canada - (YVR), Tallahassee, FL, United States - (TLH), Winnipeg, MB, Canada - (YWG)), Middle East (Amman, Jordan - (AMM), Baghdad, Iraq - (BGW), Baku, Azerbaijan - (GYD), Beirut, Lebanon - (BEY), Doha, Qatar - (DOH), Dubai, United Arab Emirates - (DXB), Kuwait City, Kuwait - (KWI), Manama, Bahrain - (BAH), Muscat, Oman - (MCT), Ramallah - (ZDM), Riyadh, Saudi Arabia - (RUH), Tel Aviv, Israel - (TLV)), Asia (Bangkok, Thailand - (BKK), Cebu, Philippines - (CEB), Chengdu, China - (CTU), Chennai, India - (MAA), Colombo, Sri Lanka - (CMB), Dongguan, China - (SZX), Foshan, China - (FUO), Fuzhou, China - (FOC), Guangzhou, China - (CAN), Hangzhou, China - (HGH), Hanoi, Vietnam - (HAN), Hengyang, China - (HNY), Ho Chi Minh City, Vietnam - (SGN), Hong Kong - (HKG), Hyderabad, India - (HYD), Islamabad, Pakistan - (ISB), Jinan, China - (TNA), Karachi, Pakistan - (KHI), Kathmandu, Nepal - (KTM), Kuala Lumpur, Malaysia - (KUL), Lahore, Pakistan - (LHE), Langfang, China - (NAY), Luoyang, China - (LYA), Macau - (MFM), Manila, Philippines - (MNL), Mumbai, India - (BOM), Nanning, China - (NNG), New Delhi, India - (DEL), Osaka, Japan - (KIX), Phnom Penh, Cambodia - (PNH), Qingdao, China - (TAO), Seoul, South Korea - (ICN), Shanghai, China - (SHA), Shenyang, China - (SHE), Shijiazhuang, China - (SJW), Singapore, Singapore - (SIN), Suzhou, China - (SZV), Taipei - (TPE), Tianjin, China - (TSN), Tokyo, Japan - (NRT), Ulaanbaatar, Mongolia - (ULN), Wuhan, China - (WUH), Wuxi, China - (WUX), Xi'an, China - (XIY), Yerevan, Armenia - (EVN), Zhengzhou, China - (CGO), Zuzhou, China - (CSX)), Africa (Cairo, Egypt - (CAI), Casablanca, Morocco - (CMN), Cape Town, South Africa - (CPT), Dar Es Salaam, Tanzania - (DAR), Djibouti City, Djibouti - (JIB), Durban, South Africa - (DUR), Johannesburg, South Africa - (JNB), Lagos, Nigeria - (LOS), Luanda, Angola - (LAD), Maputo, MZ - (MPM), Mombasa, Kenya - (MBA), Port Louis, Mauritius - (MRU), Réunion, France - (RUN), Kigali, Rwanda - (KGL)), Latin America & the Caribbean (Asunción, Paraguay - (ASU), Bogotá, Colombia - (BOG), Buenos Aires, Argentina - (EZE), Curitiba, Brazil - (CWB), Fortaleza, Brazil - (FOR), Lima, Peru - (LIM), Medellín, Colombia - (MDE), Mexico City, Mexico - (MEX), Panama City, Panama - (PTY), Porto Alegre, Brazil - (POA), Quito, Ecuador - (UIO), Rio de Janeiro, Brazil - (GIG), São Paulo, Brazil - (GRU), Santiago, Chile - (SCL), Willemstad, Curaçao - (CUR)), Oceania (Auckland, New Zealand - (AKL), Brisbane, QLD, Australia - (BNE), Melbourne, VIC, Australia - (MEL), Perth, WA, Australia - (PER), Sydney, NSW, Australia - (SYD)), and Europe (Amsterdam, Netherlands - (AMS), Athens, Greece - (ATH), Barcelona, Spain - (BCN), Belgrade, Serbia - (BEG), Berlin, Germany - (TXL), Brussels, Belgium - (BRU), Bucharest, Romania - (OTP), Budapest, Hungary - (BUD), Chișinău, Moldova - (KIV), Copenhagen, Denmark - (CPH), Dublin, Ireland - (DUB), Düsseldorf, Germany - (DUS), Edinburgh, United Kingdom - (EDI), Frankfurt, Germany - (FRA), Geneva, Switzerland - (GVA), Gothenburg, Sweden - (GOT), Hamburg, Germany - (HAM), Helsinki, Finland - (HEL), Istanbul, Turkey - (IST), Kyiv, Ukraine - (KBP), Lisbon, Portugal - (LIS), London, United Kingdom - (LHR), Luxembourg City, Luxembourg - (LUX), Madrid, Spain - (MAD), Manchester, United Kingdom - (MAN), Marseille, France - (MRS), Milan, Italy - (MXP), Moscow, Russia - (DME), Munich, Germany - (MUC), Nicosia, Cyprus - (LCA), Oslo, Norway - (OSL), Paris, France - (CDG), Prague, Czech Republic - (PRG), Reykjavík, Iceland - (KEF), Riga, Latvia - (RIX), Rome, Italy - (FCO), Saint Petersburg, Russia - (LED), Sofia, Bulgaria - (SOF), Stockholm, Sweden - (ARN), Tallinn, Estonia - (TLL), Thessaloniki, Greece - (SKG), Vienna, Austria - (VIE), Vilnius, Lithuania - (VNO), Warsaw, Poland - (WAW), Zagreb, Croatia - (ZAG), Zürich, Switzerland - (ZRH)). Quite a wide region, innit?
dessant 7y ago
A great example for why you shouldn't transfer your domain to Cloudflare Registrar if you're also using their CDN. Those who have transferred their domains cannot change DNS servers to mitigate the outage.
- acdanger 7y ago
  I use namecheap to manage my name servers. It's also down.
- bovermyer 7y ago
  Yeah, I'm regretting that very choice now.
  - matt_oriordan 7y ago
    Problem is not just about using Cloudflare as your DNS registrar really. Even if you have a different registrar, the Cloudflare model is to have the NS (nameserver) records set up to point to Cloudflare, and then they in turn resolve the DNS. You cannot really use Cloudflare without that set up. Changes to nameservers at a registrar level are rarely quick, at least quick enough to mitigate a disaster like this. It's why we've used two completely different domains at Ably (ably.io and ably-realtime.com) for all services we provide. We wrote about a strategy to circumvent this sort of thing a little while back https://www.ably.io/blog/routing-around-single-point-of-failure-dns https://www.ably.io/blog/routing-around-single-point-of-fail.... Given two incidents in a matter of weeks, I think a revisit of that article in light of most businesses who operate on a single domain would be useful :)
- nullify88 7y ago
  You can setup your domains using their CNAME method. You do not have to delegate your entire domain to them. https://support.cloudflare.com/hc/en-us/articles/360020615111-Configuring-a-CNAME-setup https://support.cloudflare.com/hc/en-us/articles/36002061511... Together with a short TTL we were able to recover without relying on their dashboards.
  - 7y ago
obituary_latte 7y ago
This is frustrating - can't access CF to turn off CF to make sites accessible. There should be an emergency admin/dash access to turn off protection for cases like this.
bytebuster 7y ago
And I'm sure that everybody hitting F5 to see if it's back is causing no problem at all, no no. Waiting anxiously for the writeup!
frenchman99 7y ago
Any workarounds or solutions ? I'm an on-call engineer with lots of questions coming in. I'm not sure what I can do apart from moving the domain off Cloudflare, bug DNS propagation would take a few hours and by then Cloudflare might be up again.
- NKCSS 7y ago
  Outages can always happen, when they do with companies like this, at least you'll know that some of the best people out there are working on the issue and that it will be resolved asap. CloudFlare has proven in the past to be a very capable party, I don't think panicking now and try to move everything away is a smart move. Also, a few people have been saying that even if you want to, the site to do so is not reachable, so that would be a challenge as well.
  - deleted 7y ago
    [deleted]
  - frenchman99 7y ago
    Panicking was not the plan. Asking for advice was.
- eeeeeeeeeeeee 7y ago
  My suggestion is wait. I wouldn't even consider flipping my sites over to another DNS unless the outage begins lasting over four hours or so. A lot of top sites use Cloudflare and this sort of outage is extremely rare for them (I can't remember a time when Cloudflare's own site and dashboard were taken offline). Then again, it depends on the priority of your site. But there are tons of top sites on Cloudflare and I bet a lot of those places don't have plans for emergency switching over to another DNS provider / CDN on short notice as it's often a fairly disruptive change, especially now that more frontend logic for a site is implemented alongside the CDN/LB.
BuddhaSource 7y ago
So many crypto sites are down. Coinbase WazirX CoinmarketCap
- cableshaft 7y ago
  Also WorldCoinIndex.
anoldgangstah 7y ago
9gag is having trouble loading the memes.
davexunit 7y ago
Yay centralization!
bensmrs 7y ago
Also fun to notice that you have to agree to their privacy policy to receive updates, which is hosted on their website, which is down
madethemcry 7y ago
The internet is designed to withstand a nuclear attack. It's so sad to see half of the internet going down because of a single system failure.
- chasd00 7y ago
  WAS designed to withstand a nuclear attack. These days, not so much.
- rc_kas 7y ago
  It's very convenient for me to be able to tell me clients their site is down because half the internet is down, and there is nothing I can do about it. A smaller DNS service would not make headlines.
jrwiegand 7y ago
Seems fitting that Cloudflare spoke so aggressively against Verizon[0][1] last week and then this incident happens to them. I will be interested to read the postmortem on this situation. I really like Cloudflare but you should be careful not to jinx yourself with blogs posts like that. [0]: https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/ https://blog.cloudflare.com/the-deep-dive-into-how-verizon-a... [1]: https://web.archive.org/web/20190628223129/https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/ https://web.archive.org/web/20190628223129/https://blog.clou...
ddayday 7y ago
https://www.bbc.co.uk/news/world-europe-48844013 https://www.bbc.co.uk/news/world-europe-48844013
clarkmoody 7y ago
Back for me.
s_dev 7y ago
Even getting this error coming up in Android Studio trying to sync Gradle. Anyone got a good Android NFC tutorial I can read up on with working code examples?
Heliosmaster 7y ago
It seems to be up for me now in AMS.
apple4ever 7y ago
Seems like it is clearing up. Our site is backup. Digital Ocean and Patreon are up as well.
weixiyen 7y ago
It's back up for us now https://sleeper.app https://sleeper.app
oerpli 7y ago
Here in Liechtenstein things are working again.
ddayday 7y ago
on an unrelated note https://www.bbc.co.uk/news/world-europe-48844013 https://www.bbc.co.uk/news/world-europe-48844013
the-dude 7y ago
And it is up again.
deleted 7y ago
[deleted]
anoldgangstah 7y ago
Is 1.1.1.1 affected aswell?
deleted 7y ago
[deleted]
flixic 7y ago
Back online.
rwc 7y ago
Seems to be back to normal now.
deleted 7y ago
[deleted]
jhow15 7y ago
I think we're back up!
AlphaWeaver 7y ago
I'm beginning to see the issues disappearing for me on the East Coast of the US, as of 10:14 EDT.
sauldcosta 7y ago
Some people on Twitter are reporting that it's due to a DDoS. http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=1&time=17948&view=map http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&... seems to indicate Iran is the source?
- po1nter 7y ago
  The date on your link is February 21st.
  - sauldcosta 7y ago
    Right you are, heh. Not sure why it jumped back to that date. False alarm.
- ignaloidas 7y ago
  Wrong date on that page. That's Dec. 2, not Jul. 2
- jgrahamc 7y ago
  It's not.
- allanmacgregor 7y ago
  This seems FUD the dates are wrong and according to that report Iran is the one attacked
mkr-hn 7y ago
Someone needs to turn Cloudflare off and on again.
grey-area 7y ago
Throughout this outage, https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ continued to show green - all services operational, with almost all services marked 'operational' and some vague cryptic message about users in this region being affected: Investigating - Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare. We are working to mitigate impact to Internet users in this region. It seemed very much like a global outage affecting all services. Is this status page not automatically updated with service status, or is it just manually updated by humans? Even if manually updated, surely when posting that status message, the status of all the services should be set to degraded?
- NikolaeVarius 7y ago
  This is not my experience, I have received many updates both through email and updates to the cloudflare status page throughout the incident, except for possibly the first 10 minutes
  - grey-area 7y ago
    You're probably talking about the yellow note at the top of the page (which is still there, with a little more detail now). That was updated and is fine. I'm talking about the service indicators for each service lower down the page, which remained green throughout and appear to be just decoration, not an actual indication of service status, they all said operational throughout the incident (I reloaded a few times). In particular I'm thinking of the Cloudflare Sites and Services section.
nielsole 7y ago
Good reminder for us of why you want to mirror all code dependencies required during autoscaling :/
oxfordmale 7y ago
This is resulting in build failures in our AWS CodePipeline as we can't pull some Docker images from Dockerhub
jimaek 7y ago
jsDelivr was the only free CDN that stayed online during the outage thanks to Multi-CDN.
r1ch 7y ago
The Cloudflare DNS-over-HTTPS resolver was serving up 502 errors as well, though the standard port 53 UDP resolver was working. This event definitely made me regret choosing Cloudflare as my sole DoH server.
- captn3m0 7y ago
  Hear hear Mozilla!
daxorid 7y ago
A possible downstream effect of this: Pingdom appears to be alerting VERY late, at least for us. I'm guessing with 8% of the web affected, their alerting systems aren't prepared for this many simultaneous alerts.
0x123456 7y ago
I’m eagerly waiting to read their blog post: "The Configuration Mistake That Almost Broke the Internet"
yreg 7y ago
The rumor is a DDoS attack http://www.digitalattackmap.com http://www.digitalattackmap.com
eternalny1 7y ago
They have mitigated it ... https://twitter.com/eastdakota/status/1146057946494713858 https://twitter.com/eastdakota/status/1146057946494713858
sebastianconcpt 7y ago
Wonder how can we use CloudFlare and have a fallback plan in place for situations like this. What would be a good architecture for this? So far I've read that would be good to have the registrar out of CloudFlare and use them as CDN only. What else?
profmonocle 7y ago
Once cloudflare.com came back I decided to check out their business SLA, and it's not very encouraging: > For any and each Outage Period during a monthly billing period the Company will provide as a Service Credit an amount calculated as follows: Service Credit = (Outage Period minutes * Affected Customer Ratio) ÷ Scheduled Availability minutes - https://www.cloudflare.com/business-sla/ https://www.cloudflare.com/business-sla/ So assuming an outage affects 100% of your users (this one seems like it did, but that's not clear), they only refund the time the service was offline? According to pingdom this outage lasted ~25 minutes, so that's 25/(31 * 24 * 60) = .056% of our bill, roughly 11 cents. It sounds like you just don't pay for the time the service didn't work, which isn't much of a guarantee, that's just expected (of course you shouldn't pay for services not provided). Most SLAs for critical services have something like under 99.99% uptime you get 10% of your bill back, under 99.5% you get 20% back, under 99% you get 50% back. (*Numbers completely made up to demonstrate the concept.) Am I misreading this? Morning coffee hasn't kicked in yet so maybe I am.
- lukevp 7y ago
  That's the business SLA. The Enterprise one is 100% uptime with a 25x payback so those are the ones they are focusing on keeping up. We were with Verizon before CloudFlare and their SLA was a similar pay back for outage. I think this is pretty typical, what service did you have that had a different setup for the SLA?
  - profmonocle 7y ago
    Amazon EC2 has an SLA like I mentioned: https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
- jeroen 7y ago
  25/(31 * 24 * 60) == 0.056%, not 5.6%.
  - profmonocle 7y ago
    Fixed, thanks. Like I said about the coffee. :)
deleted 7y ago
[deleted]
svirelka 7y ago
Update - Cloudflare has implemented a fix for this issue and is currently monitoring the results. Description: Major outage impacted all Cloudflare services globally. We saw a massive spike in CPU that caused primary and secondary systems to fall over. We shut down the process that was causing the CPU spike. Service restored to normal within ~30 minutes. We’re now investigating the root cause of what happened.
makkesk8 7y ago
Holy hell did everything explode at work, glad they got it resolved pretty quick.
cltsang 7y ago
This is just my personal guess, but it's likely China flexing again after the protest escalation yesterday in Hong Kong. A few weeks ago Telegram was attacked by China [0], and Hong Kong protesters used Telegram to communicate. This time when CloudFlare was down, the most popular local forum among protesters, lihkg.com, was brought down as well. [0]: https://techcrunch.com/2019/06/12/telegram-faces-ddos-attack-in-china-again/ https://techcrunch.com/2019/06/12/telegram-faces-ddos-attack...
TomAnthony 7y ago
They have now released an initial statement [1]: For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again. [1] https://blog.cloudflare.com/cloudflare-outage/ https://blog.cloudflare.com/cloudflare-outage/
- ilkkao 7y ago
  Interesting to hear later if the process was followed. Hard to believe they deploy changes to every production location at once.
- jgrahamc 7y ago
  Sorry, I would have posted this myself but was too busy.
- technonerd 7y ago
  Ahh the "we test in prod" method
  - jgrahamc 7y ago
    Yeah, except that wasn't meant to be the way things work.
- js2 7y ago
  > Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. Sounds like backtracking. If so, I'll bet there's a conversation happening about switching to re2. edit: hmmm, https://github.com/cloudflare/lua-re2 https://github.com/cloudflare/lua-re2 I'm also curious why this rollout wasn't staged.
psim1 7y ago
Title should read, "Cloudflare Outage."
ComputerGuru 7y ago
“Network performance issues” which is to say, completely unavailable for a huge chunk of the world.
elamje 7y ago
It’s better that this happens now than later. I am confident CF will put protections in place to prevent this from happening again, but also put a switch in place to provide an instant fix the next time something like this happens. It does suck to have a service down for a bit, but what CF offers, at the price point is pretty incredible. Good luck to CF, and I wish you the best with coming up with a robust future-proof solution.
elamje 7y ago
Updated initial postmortem: https://blog.cloudflare.com/cloudflare-outage/amp/?__twitter_impression=true https://blog.cloudflare.com/cloudflare-outage/amp/?__twitter...
song 7y ago
What's ironic is that my experience of Cloudflare Sales is that they take advantage of any downtime from their competitor to try and get people to migrate to their services... Beyond scummy
teimer2 7y ago
It's what we get for centralizing this here internet, folks.
Raphmedia 7y ago
"Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw." So, it was a dirty REGEX. I can't even be mad.

631 points by drcongo 7y ago

Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you. https://i.imgur.com/qHBM2JW.png https://i.imgur.com/qHBM2JW.png

Heliosmaster 7y ago
Are you sure it wasn't cached? For us it was showing correctly the incident...
- wepower_ico 7y ago
  Can confirm that we had the same view. Operational while not working.
- grey-area 7y ago
  All I saw was a vague message at the top, all services were marked green and operational throughout.
- btown 7y ago
  If cloudflarestatus.com isn't serving text content as Cache-Control: no-cache, that's itself an egregious bug.
  - grenoire 7y ago
    It serves Cache-Control: max-age=0, private, must-revalidate. Same thing as far as I get it, maybe wider support.
  - buildbuildbuild 7y ago
    Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.
    - auscompgeek 7y ago
      Is it? From my end it looks like a plain ec2 instance from statuspage.io.
    - jmngomes 7y ago
      Cloudfront supports Cache-Control: no-cache
      - buildbuildbuild 7y ago
        Cloudfront itself can cache the content regardless. (if its headers are showing HIT)
- acdha 7y ago
  It might have been cached on the server but was not cached on my clients: I did a forced refresh and used a private browser window but it only showed the “almost everything is good” view throughout the duration of the incident. The actual incident page was correct.
nullify88 7y ago
We saw an update within 5 minutes of our sites going down. Are you doing caching somewhere?
- buildbuildbuild 7y ago
  Interesting, I was command+shift+R refreshing and also tried from a VPN in another region. Perhaps our CF-hosted sites returned 502s in my region sooner than yours, causing me to check the status page sooner.
- seiferteric 7y ago
  Seems unlikely, I don't use cloudflare have never visited the status page before. I noticied several 502 pages this morning and searched for "cloudflare status page" and saw the "all systems operational".
eeeeeeeeeeeee 7y ago
It was showing correctly for me the first time I checked it.
matt_oriordan 7y ago
Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service https://www.ably.io/blog/honest-status-reporting-aws-service. At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647 https://status.ably.io/incidents/647 We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.
- MrGilbert 7y ago
  slightly off-topic: Nice and clean, yet detailed enough status page. I like it. :)
  - matt_oriordan 7y ago
    Thanks :)
    - felideon 7y ago
      On the OT subject, the status logo is blurry (at least on a 4k monitor), whereas on the main website it's nice and crisp.
      - matt_oriordan 7y ago
        Ok, thanks for the heads up. Will raise a status website issue.
- joncrane 7y ago
  >How is it that hundreds of developers know there's an issue before AWS do Trust me, they know. They know about problems we never find out about, too.
kreetx 7y ago
They should derive whether something is down from how much traffic the status page gets, and have alerts tied to that as well. I think it would be pretty accurate.
gsich 7y ago
Why even have a status page if it doesn't reflect reality?
takeda 7y ago
It isn't in their best interest to have a reliable status page. They want status page to contain information about failure only after people know about issues through other means. For the same reason cloud providers don't provide rebates on outages unless you ask for it explicitly. I think the only way is to have status page operated by an independent 3rd party, but I don't think there's a viable business model for someone to provide such service. Perhaps there might be even a risk of lawsuits against you.

_yesmeck 7y ago

From https://www.cloudflarestatus.com's https://www.cloudflarestatus.com's response header, it seems it runs on fastly, so it's not down.

lol768 7y ago

Yes, seems to have impacted all CF sites. (UK here) Status was just updated here, but it was showing everything as operational for a while: https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/

tmlee 7y ago

So it seems... Even https://cloudflare.com/ https://cloudflare.com/ itself is down

ratsimihah 7y ago
So cloudflare.com runs on Cloudflare?
- dexen 7y ago
  Turtles all the way down. But in all frankness, if Cloudflare's own site did not run off of Cloudflare's infrastructure, why would anybody trust them with their websites?
  - beckingz 7y ago
    Ironic isn't it.
- OskarS 7y ago
  Certainly what I would use if I was building cloudflare.com...
- kreetx 7y ago
  For the site itself that's totally fine, and the status page is separate as it should be, too.
  - ratsimihah 7y ago
    Fair. Some service recently had their status page hosted on the same server that went down, that was funny.
    - apple4ever 7y ago
      Discord's status page is also down haha

khebbie 7y ago

I see it as well

chromaton 7y ago

dammit

cocoflunchy 7y ago

Wow... our site went down so I went to cloudflare.com -> 502 bad gateway

Raphmedia 7y ago

It is. https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/

yuchi 7y ago

Either downforeveryoneorjustme.com is itself served by Cloudflare too, or has been hugged to death.

close04 7y ago
It returns 502 Bad Gateway cloudflare so I assume it's not the HN hug...
ttctciyf 7y ago
Also, https://downdetector.co.uk/problems/cloudflare https://downdetector.co.uk/problems/cloudflare is currently detecting the outage in very direct fashion!
bwb 7y ago
Yep, we are 100% on CF workers :) Here is a quick image of the peak downtime on downforeveryoneorjustme.com: https://ibb.co/PZ9BMRc https://ibb.co/PZ9BMRc
- yuchi 7y ago
  Thanks for the fantastic work on the service. And thanks for sharing that stats!

seibelj 7y ago

Single point of failure - we all shouldn’t trust CF alone anymore...

zzzcpan 7y ago
Any CDN is a single point of failure and limits your availability to as low as three nines. Although anycast-based CDNs like Cloudflare are much less reliable than DNS-based CDNs, those can do orders of magnitude better.
- cortesoft 7y ago
  Why do you think anycast based CDNs are worse than DNS based ones?
  - zzzcpan 7y ago
    They rely on a single network infrastructure as opposed to many independent networks with independent edge nodes where isolating faults is rather trivial in comparison.
deleted 7y ago
[deleted]
shard972 7y ago
but its free

bodono 7y ago

I'm getting '502 Bad Gateway' on all Cloudflare sites here in London too.

jf- 7y ago

Also getting this in the UK. Not completely down, bits and pieces of medium comes through but very slowly and incomplete. I’m also unable to access npm.

tcarn 7y ago

Holy shit. This impacts nearly everything.

auscompgeek 7y ago

https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr (edit: thank you mods for changing the submission URL)

bytebuster 7y ago
oh wow. that's a long list.
- yoran 7y ago
  aka the whole world.

johnstonnorth 7y ago

Yes - seeing lots of our sites offline as of 14:45 GMT.

deleted 7y ago

[deleted]

quicksilver03 7y ago

Several sites behind CloudFlare are returning 502 errors for me as well (France).

deleted 7y ago

[deleted]

perprit 7y ago

omg I was not the only one.

Nas808 7y ago

1.1.1.1 DNS is down, and seeing a lot of 502s on cloudflare sites.

ttctciyf 7y ago
I just changed my dns-over-https script to use quad9: # sudo /usr/local/bin/dnsproxy -l 127.0.0.153 -p 53 -u https://cloudflare-dns.com/dns-query -b 1.1.1.1:53 sudo /usr/local/bin/dnsproxy -l 127.0.0.153 -p 53 -u https://9.9.9.9/dns-query The bootstrap 1.1.1.1:53 was working but the DoH dns-query url was not. Hence my visit here :) Edit: Now fixed, it seems. Quick work, if it stays up!
NKCSS 7y ago
That's a biggie...
jakear 7y ago
The 1.1.1.1 vpn still seems to be working
- eternalny1 7y ago
  Not here it's not (NE US).
  - wilsonfromdevon 7y ago
    dns is working in London UK.

bytebuster 7y ago

digitalocean.com is down as well (seems to affect cloudflare-gated websites)

chadmckenna 7y ago

Looks like it, one of my sites is cached through Cloudflare, I'm getting a 502 from them. I'm seeing other random sites down as well.

jason_zig 7y ago

So what should the ideal redundancy plan be here? If you can't log into the CDN provider and they are down do you you just have a second one ready (and paid for) and then log into your registrar and be ready to switch to that secondary CDN provider in this scenario? Or is there some sort of load balancing / routing solution between CDN's that I don't know about / understand?

morpheuskafka 7y ago
If you use Cloudflare nameservers, you have to change to new nameservers, wait for that to propogate, and then wait for clients cached records TTLs to expire. So it will be a major disruption no matter what you do.
- apple4ever 7y ago
  What about the API? If that is up, you could script it.
- manigandham 7y ago
  If you're using them for TLS certs then it's an even bigger problem unless you have them provisioned elsewhere.
  - morpheuskafka 7y ago
    Unless you need EV you can just pull some wildcards from Lets Encrpt (as long as you don't use pubkey pinning). No need to automate as it's just a one off.
- bowmessage 7y ago
  Can you use your own nameservers that delegate to cloudflare or akami 50% each, then adjust? Is there a service more suited for this than r53?
deleted 7y ago
[deleted]

jafingi 7y ago

Noticed that draw.io were down. 1.1.1.1 and all other Cloudflare backed sites as well... I'm off for today.

auscompgeek 7y ago
Cleartext DNS 1.1.1.1 is still up though, thankfully.

oliverrowlands 7y ago

What will we do without our favourite "Enterprise MiTM" SaaS provider?

stedaniels 7y ago

Cloudflare DNS 1.1.1.1 down too

anoldgangstah 7y ago
Good thing I switched to OpenDNS literally two days ago. edit. not that it matters, I'm just scrolling 9gag.

40four 7y ago

And so it begins......

johnxie 7y ago

Looks like everyone is impacted. We are seeing 502 Bad Gateway across multiple domains and regions on https://taskade.com https://taskade.com

johnxie 7y ago
Back up and running!

gadgetoid 7y ago

I just changed a setting in my CloudFlare account... did I break everything?

darkcha0s 7y ago
Oi! Change it back!
- Gigablah 7y ago
  Well, they can’t now :p
  - gadgetoid 7y ago
    Touché. On one hand I think "Maybe I should diversify my infrastructure." And on the other I think "But one of the biggest upsells was convinience." And it's fortunate I don't have a third hand, because I'd be thinking "Oh crap oh crap I just migrated a client website to LightSail + CloudFlare saying how super awesome and robust it would be." But it's okay now because it looks like everything is back up!
jgrahamc 7y ago
No

Thaxll 7y ago

After trolling Verizon curious what reasons they will come with for that outage.

apple4ever 7y ago
Maybe this is payback by Verizon...
voidwtf 7y ago
Care to elaborate how they were "trolling" Verizon?
- EamonnMR 7y ago
  They came off that way in this blog post: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...
  - acdha 7y ago
    That didn't seem like trolling – just a public call for Verizon to follow internet best practices. Given that most large ISPs treat failures as a PR exercise, that's probably necessary.
    - voidwtf 7y ago
      Agreed. They reached out to Verizon privately, a Tier 1 carrier with expectations and responsibilities as a good netizen, and got no response. They attempted to reach out through Verizon's public forms of communication and got a bullshit irrelevant CS response despite requesting escalation. They then called out Verizon before the community as a whole. They don't have the luxury of waiting for a well prepared letter from some Verizon lawyers. Modern day customer expectations don't allow for it. You may call it trolling, but all I saw was a company asking another company to stop pissing in the public pool.

ratsimihah 7y ago

This article entitled Now Running on Cloudflare seems down https://seankilleen.com/2016/12/now-running-on-cloudflare/ https://seankilleen.com/2016/12/now-running-on-cloudflare/

sprite 7y ago

When you can't update your DNS because your registrar uses CloudFlare also....

stevekemp2 7y ago
You could use a dedicated dns provider, such as AWS

lukevp 7y ago

Our website is also down, origin near Dallas.

deleted 7y ago

[deleted]

kortilla 7y ago

Can’t wait to see Verizon’s blog post about this one :)

nnx 7y ago

Damn... getting a bunch of alerts and can't even open Pingdom either... also running on CloudFlare https://my.pingdom.com/newchecks/checks# https://my.pingdom.com/newchecks/checks#

deadalus 7y ago

8chan, 4chan down as well. Damn, this is big. https://cloudflare.com https://cloudflare.com itself is gone.

chasd00 7y ago
that's bad news, i like 4chan being up because it keeps them all in one place and they don't start wandering around.

danShumway 7y ago

Don't want to go off topic, but if I want to prevent my website going down because of stuff like this in the future, will having back up DNS entries solve the problem? I know DNS will fall back if it can't reach a service, but would a 502 trigger that?

rosswilson 7y ago
If you received a HTTP 502 then DNS must've already resolved. Browsers typically will do a DNS lookup, and then try establishing a TCP connection to one of the returned hosts. Its only if it can't establish a TCP connection to a host will it (sometimes) try another host from the DNS response.
lol768 7y ago
No, I don't believe it would.
SteveNuts 7y ago
No, you'd need to use an application-aware DNS provider like Route 53 that could detect the failure. Backup DNS entries won't help, what you need is to use multiple DNS providers and add them all to your NS records
- danShumway 7y ago
  I think this is the correct answer. Since I'm mostly thinking about static sites, I could also have something local that pings the site, and if it goes down, it could update my nameservers to point elsewhere on its own. Probably more trouble than its worth though.
aclelland 7y ago
Yeah, you'll want to keep your domain registered through a different registrar and if CF goes down you can update your DNS Name Servers point from CF to something like AWS Route 53. This has a few drawbacks like making sure your Route53 configuration is identical to your CF config, ensuring your origin servers can cope with the additional load if CF caching isn't available and the DNS propagation time required for the Name Servers to update. During the last outage, we were able to get into the CF dashboard and simply disable the proxy which allowed our clients to access our origin servers directly but this time we can't even get into the Dashboard.

jarym 7y ago

Looks like a flare up (sorry my attempt at humour always gets the better of me...).

heavymark 7y ago

Holy crap, cloudflare down, and seemingly all the covers. Major sites such as Digitalocean are all down, and no way to easily disable cloudflare since their site is down.

hu3 7y ago
https://www.digitalocean.com/ https://www.digitalocean.com/ frontpage working for me: https://i.imgur.com/8yJRXVI.png https://i.imgur.com/8yJRXVI.png
- bpicolo 7y ago
  The outage appears to have ended between that post and your post.
  - rc_kas 7y ago
    Yeah, not a long outage, but a big one.

akriukas 7y ago

Same here.

maple3142 7y ago

The outrage this time seems to be more serious than last time. More websites including Cloudflare itself are down.

sauldcosta 7y ago
Without question. I've never seen an affected regions list that immense.

max0563 7y ago

Cloudflare is too big.

isatty 7y ago

Pagerduty’s main website seems to be affected.

jakemcgraw 7y ago

was hoping John Graham-Cumming would be in here to give us the behind scenes :(

exdsq 7y ago

Can someone from Cloudflare give us a hint at what's going on?

nodesocket 7y ago

July 2nd, 2019... The day the internet broke.

Endy 7y ago
The day the internet broke again.
triod 7y ago
Yup, we could have "Days without an accident" counter for the whole internet...

thequailman 7y ago

This seems to be impacting NPM for me.

pmlnr 7y ago

Good. I'm tired of people not learning that trusting a single gateway with 50% of the internet is bad. Yes, I know, free DDOS protection. There has to be another way of doing this, some mesh based DDOS protection or so.

Avamander 7y ago
With a lot of these things, if people can't agree to make a standard system to do all of it just as well then there's going to be a big company that does so. We have had this with e-mail, signed exchanges, DDOS protection and will probably have it with many other things unless people pull themselves together at least after these proprietary solutions are created and create better alternatives.
Waterluvian 7y ago
I get your point. But I'm not sure I can ever grok the mindset of someone who thinks, "good, that'll teach 'em". Not sure I'd ever want to work with that individual.
- bovermyer 7y ago
  This is important to recognize. Victim blaming is a highly destructive practice.
  - inimino 7y ago
    Assigning blame for technical business decisions to the people who made those decisions is victim blaming?
    - bovermyer 7y ago
      Targeting the process or behavior that lead to an event is far more effective than targeting the person that triggered it. Do not conflate the two.
- drunkpotato 7y ago

hokumguru 7y ago

My shopify sites are down this morning, seems to be 502-ing because of this issue.

adamparsons 7y ago

I thought I was going insane, googling it returned nothing, but trusty old hacker news has my back

jshb 7y ago

Tried to go to some usual places where people discuss outages and met with outage. Ouch.

tnolet 7y ago

npm and yarn are not working either...there goes Javascript land...

cube00 7y ago

Not impressed it's serving an error page claiming the underlying host is to blame (this one from discord) Error 502 Bad gateway You - Browser - Working Sydney - Cloudflare - Working storage.googleapis.com - Host - Error What happened? The web server reported a bad gateway error.

minikites 7y ago

Wasn't the internet supposed to be decentralized to avoid this very problem?

langitbiru 7y ago
That is the mission of IPFS. However, there is a big gap between their mission and the current capability of their technology. Be the change you want to see in this world. I guess next time I build a personal website, I have to host it in Raspberry Pi at my home. :)

Vaskerville 7y ago

As long as Reddit is not down everything is OK.

maz1b 7y ago

Getting 502s as well.

damniatx 7y ago

oh my god,

deca6cda37d0 7y ago

7,5% of all websites... https://w3techs.com/technologies/details/cn-cloudflare/all/all https://w3techs.com/technologies/details/cn-cloudflare/all/a...

hanspeter 7y ago
Add to that all the sites using resources hosted on Cloudflare's CDN.
snug 7y ago
9.6% of websites that have one of those CDNs
- deca6cda37d0 7y ago
  aah i misread so 7,5% of all websites
  - lgats 7y ago
    How about a weighted percentage of the Alexa top 1m?
colinbartlett 7y ago
That number matches what I am seeing on StatusGator: Of the 438 status pages we monitor, 52 of them are showing some kind of warn or down notice right now. That's almost 12%. Though some of them might not be because of Cloudflare, the ones I spot checked all do appear related. Medium, DigitalOcean, Shopify, CodeShip, Pingdom, and many more. The impact is staggering.

sb057 7y ago

>We are working to mitigate impact to Internet users in this region. >This incident affects: North America (Ashburn, VA, United States - (IAD), Atlanta, GA, United States - (ATL), Boston, MA, United States - (BOS), Buffalo, NY, United States - (BUF), Calgary, AB, Canada - (YYC), Charlotte, NC, United States - (CLT), Chicago, IL, United States - (ORD), Columbus, OH, United States - (CMH), Dallas, TX, United States - (DFW), Denver, CO, United States - (DEN), Detroit, MI, United States - (DTW), Houston, TX, United States - (IAH), Indianapolis, IN, United States - (IND), Jacksonville, FL, United States - (JAX), Kansas City, MO, United States - (MCI), Las Vegas, NV, United States - (LAS), Los Angeles, CA, United States - (LAX), McAllen, TX, United States - (MFE), Memphis, TN, United States - (MEM), Miami, FL, United States - (MIA), Minneapolis, MN, United States - (MSP), Montgomery, AL, United States - (MGM), Montréal, QC, Canada - (YUL), Nashville, TN, United States - (BNA), Newark, NJ, United States - (EWR), Norfolk, VA, United States - (ORF), Omaha, NE, United States - (OMA), Phoenix, AZ, United States - (PHX), Pittsburgh, PA, United States - (PIT), Portland, OR, United States - (PDX), Queretaro, MX, Mexico - (QRO), Richmond, Virginia - (RIC), Sacramento, CA, United States - (SMF), Salt Lake City, UT, United States - (SLC), San Diego, CA, United States - (SAN), San Jose, CA, United States - (SJC), Saskatoon, SK, Canada - (YXE), Seattle, WA, United States - (SEA), St. Louis, MO, United States - (STL), Tampa, FL, United States - (TPA), Toronto, ON, Canada - (YYZ), Vancouver, BC, Canada - (YVR), Tallahassee, FL, United States - (TLH), Winnipeg, MB, Canada - (YWG)), Middle East (Amman, Jordan - (AMM), Baghdad, Iraq - (BGW), Baku, Azerbaijan - (GYD), Beirut, Lebanon - (BEY), Doha, Qatar - (DOH), Dubai, United Arab Emirates - (DXB), Kuwait City, Kuwait - (KWI), Manama, Bahrain - (BAH), Muscat, Oman - (MCT), Ramallah - (ZDM), Riyadh, Saudi Arabia - (RUH), Tel Aviv, Israel - (TLV)), Asia (Bangkok, Thailand - (BKK), Cebu, Philippines - (CEB), Chengdu, China - (CTU), Chennai, India - (MAA), Colombo, Sri Lanka - (CMB), Dongguan, China - (SZX), Foshan, China - (FUO), Fuzhou, China - (FOC), Guangzhou, China - (CAN), Hangzhou, China - (HGH), Hanoi, Vietnam - (HAN), Hengyang, China - (HNY), Ho Chi Minh City, Vietnam - (SGN), Hong Kong - (HKG), Hyderabad, India - (HYD), Islamabad, Pakistan - (ISB), Jinan, China - (TNA), Karachi, Pakistan - (KHI), Kathmandu, Nepal - (KTM), Kuala Lumpur, Malaysia - (KUL), Lahore, Pakistan - (LHE), Langfang, China - (NAY), Luoyang, China - (LYA), Macau - (MFM), Manila, Philippines - (MNL), Mumbai, India - (BOM), Nanning, China - (NNG), New Delhi, India - (DEL), Osaka, Japan - (KIX), Phnom Penh, Cambodia - (PNH), Qingdao, China - (TAO), Seoul, South Korea - (ICN), Shanghai, China - (SHA), Shenyang, China - (SHE), Shijiazhuang, China - (SJW), Singapore, Singapore - (SIN), Suzhou, China - (SZV), Taipei - (TPE), Tianjin, China - (TSN), Tokyo, Japan - (NRT), Ulaanbaatar, Mongolia - (ULN), Wuhan, China - (WUH), Wuxi, China - (WUX), Xi'an, China - (XIY), Yerevan, Armenia - (EVN), Zhengzhou, China - (CGO), Zuzhou, China - (CSX)), Africa (Cairo, Egypt - (CAI), Casablanca, Morocco - (CMN), Cape Town, South Africa - (CPT), Dar Es Salaam, Tanzania - (DAR), Djibouti City, Djibouti - (JIB), Durban, South Africa - (DUR), Johannesburg, South Africa - (JNB), Lagos, Nigeria - (LOS), Luanda, Angola - (LAD), Maputo, MZ - (MPM), Mombasa, Kenya - (MBA), Port Louis, Mauritius - (MRU), Réunion, France - (RUN), Kigali, Rwanda - (KGL)), Latin America & the Caribbean (Asunción, Paraguay - (ASU), Bogotá, Colombia - (BOG), Buenos Aires, Argentina - (EZE), Curitiba, Brazil - (CWB), Fortaleza, Brazil - (FOR), Lima, Peru - (LIM), Medellín, Colombia - (MDE), Mexico City, Mexico - (MEX), Panama City, Panama - (PTY), Porto Alegre, Brazil - (POA), Quito, Ecuador - (UIO), Rio de Janeiro, Brazil - (GIG), São Paulo, Brazil - (GRU), Santiago, Chile - (SCL), Willemstad, Curaçao - (CUR)), Oceania (Auckland, New Zealand - (AKL), Brisbane, QLD, Australia - (BNE), Melbourne, VIC, Australia - (MEL), Perth, WA, Australia - (PER), Sydney, NSW, Australia - (SYD)), and Europe (Amsterdam, Netherlands - (AMS), Athens, Greece - (ATH), Barcelona, Spain - (BCN), Belgrade, Serbia - (BEG), Berlin, Germany - (TXL), Brussels, Belgium - (BRU), Bucharest, Romania - (OTP), Budapest, Hungary - (BUD), Chișinău, Moldova - (KIV), Copenhagen, Denmark - (CPH), Dublin, Ireland - (DUB), Düsseldorf, Germany - (DUS), Edinburgh, United Kingdom - (EDI), Frankfurt, Germany - (FRA), Geneva, Switzerland - (GVA), Gothenburg, Sweden - (GOT), Hamburg, Germany - (HAM), Helsinki, Finland - (HEL), Istanbul, Turkey - (IST), Kyiv, Ukraine - (KBP), Lisbon, Portugal - (LIS), London, United Kingdom - (LHR), Luxembourg City, Luxembourg - (LUX), Madrid, Spain - (MAD), Manchester, United Kingdom - (MAN), Marseille, France - (MRS), Milan, Italy - (MXP), Moscow, Russia - (DME), Munich, Germany - (MUC), Nicosia, Cyprus - (LCA), Oslo, Norway - (OSL), Paris, France - (CDG), Prague, Czech Republic - (PRG), Reykjavík, Iceland - (KEF), Riga, Latvia - (RIX), Rome, Italy - (FCO), Saint Petersburg, Russia - (LED), Sofia, Bulgaria - (SOF), Stockholm, Sweden - (ARN), Tallinn, Estonia - (TLL), Thessaloniki, Greece - (SKG), Vienna, Austria - (VIE), Vilnius, Lithuania - (VNO), Warsaw, Poland - (WAW), Zagreb, Croatia - (ZAG), Zürich, Switzerland - (ZRH)). Quite a wide region, innit?

dessant 7y ago

A great example for why you shouldn't transfer your domain to Cloudflare Registrar if you're also using their CDN. Those who have transferred their domains cannot change DNS servers to mitigate the outage.

acdanger 7y ago
I use namecheap to manage my name servers. It's also down.
bovermyer 7y ago
Yeah, I'm regretting that very choice now.
- matt_oriordan 7y ago
  Problem is not just about using Cloudflare as your DNS registrar really. Even if you have a different registrar, the Cloudflare model is to have the NS (nameserver) records set up to point to Cloudflare, and then they in turn resolve the DNS. You cannot really use Cloudflare without that set up. Changes to nameservers at a registrar level are rarely quick, at least quick enough to mitigate a disaster like this. It's why we've used two completely different domains at Ably (ably.io and ably-realtime.com) for all services we provide. We wrote about a strategy to circumvent this sort of thing a little while back https://www.ably.io/blog/routing-around-single-point-of-failure-dns https://www.ably.io/blog/routing-around-single-point-of-fail.... Given two incidents in a matter of weeks, I think a revisit of that article in light of most businesses who operate on a single domain would be useful :)
nullify88 7y ago
You can setup your domains using their CNAME method. You do not have to delegate your entire domain to them. https://support.cloudflare.com/hc/en-us/articles/360020615111-Configuring-a-CNAME-setup https://support.cloudflare.com/hc/en-us/articles/36002061511... Together with a short TTL we were able to recover without relying on their dashboards.
- 7y ago

obituary_latte 7y ago

This is frustrating - can't access CF to turn off CF to make sites accessible. There should be an emergency admin/dash access to turn off protection for cases like this.

bytebuster 7y ago

And I'm sure that everybody hitting F5 to see if it's back is causing no problem at all, no no. Waiting anxiously for the writeup!

frenchman99 7y ago

Any workarounds or solutions ? I'm an on-call engineer with lots of questions coming in. I'm not sure what I can do apart from moving the domain off Cloudflare, bug DNS propagation would take a few hours and by then Cloudflare might be up again.

NKCSS 7y ago
Outages can always happen, when they do with companies like this, at least you'll know that some of the best people out there are working on the issue and that it will be resolved asap. CloudFlare has proven in the past to be a very capable party, I don't think panicking now and try to move everything away is a smart move. Also, a few people have been saying that even if you want to, the site to do so is not reachable, so that would be a challenge as well.
- deleted 7y ago
  [deleted]
- frenchman99 7y ago
  Panicking was not the plan. Asking for advice was.
eeeeeeeeeeeee 7y ago
My suggestion is wait. I wouldn't even consider flipping my sites over to another DNS unless the outage begins lasting over four hours or so. A lot of top sites use Cloudflare and this sort of outage is extremely rare for them (I can't remember a time when Cloudflare's own site and dashboard were taken offline). Then again, it depends on the priority of your site. But there are tons of top sites on Cloudflare and I bet a lot of those places don't have plans for emergency switching over to another DNS provider / CDN on short notice as it's often a fairly disruptive change, especially now that more frontend logic for a site is implemented alongside the CDN/LB.

BuddhaSource 7y ago

So many crypto sites are down. Coinbase WazirX CoinmarketCap

cableshaft 7y ago
Also WorldCoinIndex.

anoldgangstah 7y ago

9gag is having trouble loading the memes.

davexunit 7y ago

Yay centralization!

bensmrs 7y ago

Also fun to notice that you have to agree to their privacy policy to receive updates, which is hosted on their website, which is down

madethemcry 7y ago

The internet is designed to withstand a nuclear attack. It's so sad to see half of the internet going down because of a single system failure.

chasd00 7y ago
WAS designed to withstand a nuclear attack. These days, not so much.
rc_kas 7y ago
It's very convenient for me to be able to tell me clients their site is down because half the internet is down, and there is nothing I can do about it. A smaller DNS service would not make headlines.

jrwiegand 7y ago

Seems fitting that Cloudflare spoke so aggressively against Verizon[0][1] last week and then this incident happens to them. I will be interested to read the postmortem on this situation. I really like Cloudflare but you should be careful not to jinx yourself with blogs posts like that. [0]: https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/ https://blog.cloudflare.com/the-deep-dive-into-how-verizon-a... [1]: https://web.archive.org/web/20190628223129/https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/ https://web.archive.org/web/20190628223129/https://blog.clou...

ddayday 7y ago

https://www.bbc.co.uk/news/world-europe-48844013 https://www.bbc.co.uk/news/world-europe-48844013

clarkmoody 7y ago

Back for me.

s_dev 7y ago

Even getting this error coming up in Android Studio trying to sync Gradle. Anyone got a good Android NFC tutorial I can read up on with working code examples?

Heliosmaster 7y ago

It seems to be up for me now in AMS.

apple4ever 7y ago

Seems like it is clearing up. Our site is backup. Digital Ocean and Patreon are up as well.

weixiyen 7y ago

It's back up for us now https://sleeper.app https://sleeper.app

oerpli 7y ago

Here in Liechtenstein things are working again.

ddayday 7y ago

on an unrelated note https://www.bbc.co.uk/news/world-europe-48844013 https://www.bbc.co.uk/news/world-europe-48844013

the-dude 7y ago

And it is up again.

deleted 7y ago

[deleted]

anoldgangstah 7y ago

Is 1.1.1.1 affected aswell?

deleted 7y ago

[deleted]

flixic 7y ago

Back online.

rwc 7y ago

Seems to be back to normal now.

deleted 7y ago

[deleted]

jhow15 7y ago

I think we're back up!

AlphaWeaver 7y ago

I'm beginning to see the issues disappearing for me on the East Coast of the US, as of 10:14 EDT.

sauldcosta 7y ago

Some people on Twitter are reporting that it's due to a DDoS. http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=1&time=17948&view=map http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&... seems to indicate Iran is the source?

po1nter 7y ago
The date on your link is February 21st.
- sauldcosta 7y ago
  Right you are, heh. Not sure why it jumped back to that date. False alarm.
ignaloidas 7y ago
Wrong date on that page. That's Dec. 2, not Jul. 2
jgrahamc 7y ago
It's not.
allanmacgregor 7y ago
This seems FUD the dates are wrong and according to that report Iran is the one attacked

mkr-hn 7y ago

Someone needs to turn Cloudflare off and on again.

grey-area 7y ago

Throughout this outage, https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ continued to show green - all services operational, with almost all services marked 'operational' and some vague cryptic message about users in this region being affected: Investigating - Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare. We are working to mitigate impact to Internet users in this region. It seemed very much like a global outage affecting all services. Is this status page not automatically updated with service status, or is it just manually updated by humans? Even if manually updated, surely when posting that status message, the status of all the services should be set to degraded?

NikolaeVarius 7y ago
This is not my experience, I have received many updates both through email and updates to the cloudflare status page throughout the incident, except for possibly the first 10 minutes
- grey-area 7y ago
  You're probably talking about the yellow note at the top of the page (which is still there, with a little more detail now). That was updated and is fine. I'm talking about the service indicators for each service lower down the page, which remained green throughout and appear to be just decoration, not an actual indication of service status, they all said operational throughout the incident (I reloaded a few times). In particular I'm thinking of the Cloudflare Sites and Services section.

nielsole 7y ago

Good reminder for us of why you want to mirror all code dependencies required during autoscaling :/

oxfordmale 7y ago

This is resulting in build failures in our AWS CodePipeline as we can't pull some Docker images from Dockerhub

jimaek 7y ago

jsDelivr was the only free CDN that stayed online during the outage thanks to Multi-CDN.

r1ch 7y ago

The Cloudflare DNS-over-HTTPS resolver was serving up 502 errors as well, though the standard port 53 UDP resolver was working. This event definitely made me regret choosing Cloudflare as my sole DoH server.

captn3m0 7y ago
Hear hear Mozilla!

daxorid 7y ago

A possible downstream effect of this: Pingdom appears to be alerting VERY late, at least for us. I'm guessing with 8% of the web affected, their alerting systems aren't prepared for this many simultaneous alerts.

0x123456 7y ago

I’m eagerly waiting to read their blog post: "The Configuration Mistake That Almost Broke the Internet"

yreg 7y ago

The rumor is a DDoS attack http://www.digitalattackmap.com http://www.digitalattackmap.com

eternalny1 7y ago

They have mitigated it ... https://twitter.com/eastdakota/status/1146057946494713858 https://twitter.com/eastdakota/status/1146057946494713858

sebastianconcpt 7y ago

Wonder how can we use CloudFlare and have a fallback plan in place for situations like this. What would be a good architecture for this? So far I've read that would be good to have the registrar out of CloudFlare and use them as CDN only. What else?

profmonocle 7y ago

Once cloudflare.com came back I decided to check out their business SLA, and it's not very encouraging: > For any and each Outage Period during a monthly billing period the Company will provide as a Service Credit an amount calculated as follows: Service Credit = (Outage Period minutes * Affected Customer Ratio) ÷ Scheduled Availability minutes - https://www.cloudflare.com/business-sla/ https://www.cloudflare.com/business-sla/ So assuming an outage affects 100% of your users (this one seems like it did, but that's not clear), they only refund the time the service was offline? According to pingdom this outage lasted ~25 minutes, so that's 25/(31 * 24 * 60) = .056% of our bill, roughly 11 cents. It sounds like you just don't pay for the time the service didn't work, which isn't much of a guarantee, that's just expected (of course you shouldn't pay for services not provided). Most SLAs for critical services have something like under 99.99% uptime you get 10% of your bill back, under 99.5% you get 20% back, under 99% you get 50% back. (*Numbers completely made up to demonstrate the concept.) Am I misreading this? Morning coffee hasn't kicked in yet so maybe I am.

lukevp 7y ago
That's the business SLA. The Enterprise one is 100% uptime with a 25x payback so those are the ones they are focusing on keeping up. We were with Verizon before CloudFlare and their SLA was a similar pay back for outage. I think this is pretty typical, what service did you have that had a different setup for the SLA?
- profmonocle 7y ago
  Amazon EC2 has an SLA like I mentioned: https://aws.amazon.com/compute/sla/ https://aws.amazon.com/compute/sla/
jeroen 7y ago
25/(31 * 24 * 60) == 0.056%, not 5.6%.
- profmonocle 7y ago
  Fixed, thanks. Like I said about the coffee. :)

deleted 7y ago

[deleted]

svirelka 7y ago

Update - Cloudflare has implemented a fix for this issue and is currently monitoring the results. Description: Major outage impacted all Cloudflare services globally. We saw a massive spike in CPU that caused primary and secondary systems to fall over. We shut down the process that was causing the CPU spike. Service restored to normal within ~30 minutes. We’re now investigating the root cause of what happened.

makkesk8 7y ago

Holy hell did everything explode at work, glad they got it resolved pretty quick.

cltsang 7y ago

This is just my personal guess, but it's likely China flexing again after the protest escalation yesterday in Hong Kong. A few weeks ago Telegram was attacked by China [0], and Hong Kong protesters used Telegram to communicate. This time when CloudFlare was down, the most popular local forum among protesters, lihkg.com, was brought down as well. [0]: https://techcrunch.com/2019/06/12/telegram-faces-ddos-attack-in-china-again/ https://techcrunch.com/2019/06/12/telegram-faces-ddos-attack...

TomAnthony 7y ago

They have now released an initial statement [1]: For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again. [1] https://blog.cloudflare.com/cloudflare-outage/ https://blog.cloudflare.com/cloudflare-outage/

ilkkao 7y ago
Interesting to hear later if the process was followed. Hard to believe they deploy changes to every production location at once.
jgrahamc 7y ago
Sorry, I would have posted this myself but was too busy.
technonerd 7y ago
Ahh the "we test in prod" method
- jgrahamc 7y ago
  Yeah, except that wasn't meant to be the way things work.
js2 7y ago
> Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. Sounds like backtracking. If so, I'll bet there's a conversation happening about switching to re2. edit: hmmm, https://github.com/cloudflare/lua-re2 https://github.com/cloudflare/lua-re2 I'm also curious why this rollout wasn't staged.

psim1 7y ago

Title should read, "Cloudflare Outage."

ComputerGuru 7y ago

“Network performance issues” which is to say, completely unavailable for a huge chunk of the world.

elamje 7y ago

It’s better that this happens now than later. I am confident CF will put protections in place to prevent this from happening again, but also put a switch in place to provide an instant fix the next time something like this happens. It does suck to have a service down for a bit, but what CF offers, at the price point is pretty incredible. Good luck to CF, and I wish you the best with coming up with a robust future-proof solution.

elamje 7y ago

Updated initial postmortem: https://blog.cloudflare.com/cloudflare-outage/amp/?__twitter_impression=true https://blog.cloudflare.com/cloudflare-outage/amp/?__twitter...

song 7y ago

What's ironic is that my experience of Cloudflare Sales is that they take advantage of any downtime from their competitor to try and get people to migrate to their services... Beyond scummy

teimer2 7y ago

It's what we get for centralizing this here internet, folks.

Raphmedia 7y ago

"Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw." So, it was a dirty REGEX. I can't even be mad.