8 ms·

Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everyw

Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everywhere.

samcheng 4y ago
Thanks for being here with timely updates! I knew to come to Hacker News once the alert triggered and a few users started complaining.
- fragmede 4y ago
  Good thing HN doesn’t use them then!
lesmond 4y ago
Enterprise support has been useless as is the status page. Got more info here
- dx034 4y ago
  The status page shows about as much information as the post here.
  - lesmond 4y ago
    It didn’t at the time. Their phone line kept cutting us off and then the people there were not too helpful.
  - robbiep 4y ago
    Status page was down for me in Sydney
halotrope 4y ago
Thank you! This little comment just saved me an hour of investigation. Good luck for getting the system back up asap.
golanggeek 4y ago
We are not using cloud flare. But our domain is also not accessible. We are using digital ocean's DNS service for propagating our IP. Does the DigitalOcean's DNS service depend on Cloudflare service?
- tn890 4y ago
  Yes https://www.cloudflare.com/case-studies/digitalocean/ https://www.cloudflare.com/case-studies/digitalocean/
  - ThePowerOfFuet 4y ago
    That link isn't accessible from where I am right now. Alanis Morissette agrees that this is ironic.
    - mproud 4y ago
      Yes, this is actual irony, unlike “rain on your wedding day.”
      - asddubs 4y ago
        but in regards to singing a whole song about things that are not irony being irony: isn't it ironic? don't you think?
        chrisweekly 4y ago
        Yes! IRL Alanis is smart and eloquent, and those that think that song's lyrics are evidence to the contrary are missing the joke.
  - baobob 4y ago
    Moved all my domains to DO specifically to stop donating traffic data to Cloudflare. Absolutely stunned I didn't notice this earlier. It's bullshit all the way down. Are there any companies left offering free DNS usable from Terraform that aren't part of the "Internet Five Eyes"? edit: looks like Linode may be the next best 'not terrible' option
Akuehne 4y ago
Just wanted to also reiterate how thankful I am that you took your time to let us know. It speaks volumes.
- taf2 4y ago
  Agreed- couldn’t figure out what was going on… finally checked here and - ah now I can sleep
  - Abishek_Muthian 4y ago
    Cloudflare going down is one of the things which keeps me awake, My main complaint about Cloudflare is that they are very good at everything they offer that we've become reliant on them for everything.
    - jimmygrapes 4y ago
      "do no evil" springs to mind -- once burned, etc.
      - Abishek_Muthian 4y ago
        Yeah, What's up with the competition to Cloudflare? What's the real barrier for entry? It's not infrastructure anymore, As there is a new PaaS startup every week offering distributed hosting and So why bundling in DNS, DDOS detection+mitigation, cloud workers... with it is so hard?
        subvarad 4y ago
        There are many noteworthy players - Akamai, Fastly etc., and Edge plaoviders like ourselves (Zycada) who complement top CDNs like Akamai, Cloudflare, Fastly.
        Abishek_Muthian 4y ago
        The main difference between Cloudflare and the others mentioned is the price; One can start with CF for a side project for free and continue to use it free till it becomes a viable startup. Others at best offer a limited trial plan, But most are just 'Speak to expert/ Contact us' for pricing which means haggling with a sales rep while we can just build things. Even the paid plans of CF is reasonable when compared with others with better features.
obnauticus 4y ago
Which undersea cable was cut ;)
- noitpmeder 4y ago
  Can't really roll back that change
  - ekimekim 4y ago
    Sure you can, but physically rolling back more cable takes a bit longer :P
janmo 4y ago
Should probably do slower roulouts next time.
bilekas 4y ago
Thanks for the update, just curious if we will get a report on what happened ? In as much detail as can be of course - morbid curiosity mainly. I love the post reports these events usually bring.
- blue_cookeh 4y ago
  Cloudflare are usually pretty good with posting post mortems. https://blog.cloudflare.com/tag/postmortem/ https://blog.cloudflare.com/tag/postmortem/
  - brycewray 4y ago
    https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/ https://blog.cloudflare.com/cloudflare-outage-on-june-21-202...
rcshubhadeep 4y ago
Thanks for letting us know
llausa 4y ago
hats off to you sir - would not want to be in your shoes right now but thanks for the updates
- jgrahamc 4y ago
  I don't have shoes on.
  - tomglynch 4y ago
    In tumultuous times (fix wasn't implemented at this point), the Cloudflare CTO still has time for some wit. Love it!
  - stavros 4y ago
    That's the point, two people can't fit in them.
b3lvedere 4y ago
Thank you for your very fast support!
keyle 4y ago
It's back wooo! And I'm saying this for the last time: no one type google into google!
deleted 4y ago
[deleted]
MarkyMark210 4y ago
DR much ?
- jgrahamc 4y ago
  I don't know what that means.
  - mike_d 4y ago
    DR means "disaster recovery," it is a formal plan used to respond to and mitigate potential risks to the business. Things like having a communications plan for an incident, or a backup office outside of your main office natural disaster zone.
    - jgrahamc 4y ago
      Ah. Just one more reason I hate acronyms. They obscure what the person is trying to say.
dangrossman 4y ago
All my sites behind Cloudflare had come back up, and have now gone back to serving 500 errors. The Cloudflare Dashboard is also no longer fully loading.
- jgrahamc 4y ago
  Where are you located?
  - dangrossman 4y ago
    Raleigh, NC, USA Sites are gradually reappearing as I type this. Some of my sites, and doordash.com, were returning 500 errors again just a minute ago. They just came back up, followed by the CF dashboard loading again.
  - hakanderyal 4y ago
    I'm from Turkey, and I also have been seeing intermittent errors for the last 10 minutes. Seems ok now.
mikesabbagh 4y ago
CF SRE team need to rethink their published SLA of 100%. This is not reasonable. https://www.cloudflare.com/business-sla/ https://www.cloudflare.com/business-sla/
- deleted 4y ago
  [deleted]
- subvarad 4y ago
  I completely can understand your emotion. But even the top CDNs can have outages of some form or the other. If site uptime is important, check out https://www.cdnreserve.com/ https://www.cdnreserve.com/ - it's built on the design principle that the likelihood of two separate platforms having an outage at the same time is close to zero.
- arccy 4y ago
  that just means they're willing to pay for the marketing number, not that they will actually achieve it
- stn_za 4y ago
  Yep, I'd promise 99.95 at a stretch, never 100%. They are not being honest with themselves here
  - latch 4y ago
    I'm not sure you and your parent understand what an SLA means. It's an agreement that, when broken, incurs a penalty. They aren't saying they guarantee 100% uptime. They're saying they'll pay you for any downtime. It's literally the 3rd paragraph: > 1.2 Penalties. If the Service fails to meet the above service level, the Customer will receive a credit equal to the result of the Service Credit calculation in Section 6 of this SLA. (Most people I know consider them meaningless marketing BS that's really just meant to trick people or satisfy some make-work checkbox)
    - greyface- 4y ago

Yes, not worldwide but a lot of places. Problem with our backbone. We know what. Rollbacks etc. happening. Bring it back up in chunks. Should be back up everywhere.

samcheng 4y ago

Thanks for being here with timely updates! I knew to come to Hacker News once the alert triggered and a few users started complaining.

fragmede 4y ago
Good thing HN doesn’t use them then!

lesmond 4y ago

Enterprise support has been useless as is the status page. Got more info here

dx034 4y ago
The status page shows about as much information as the post here.
- lesmond 4y ago
  It didn’t at the time. Their phone line kept cutting us off and then the people there were not too helpful.
- robbiep 4y ago
  Status page was down for me in Sydney

halotrope 4y ago

Thank you! This little comment just saved me an hour of investigation. Good luck for getting the system back up asap.

golanggeek 4y ago

We are not using cloud flare. But our domain is also not accessible. We are using digital ocean's DNS service for propagating our IP. Does the DigitalOcean's DNS service depend on Cloudflare service?

tn890 4y ago
Yes https://www.cloudflare.com/case-studies/digitalocean/ https://www.cloudflare.com/case-studies/digitalocean/
- ThePowerOfFuet 4y ago
  That link isn't accessible from where I am right now. Alanis Morissette agrees that this is ironic.
  - mproud 4y ago
    Yes, this is actual irony, unlike “rain on your wedding day.”
    - asddubs 4y ago
      but in regards to singing a whole song about things that are not irony being irony: isn't it ironic? don't you think?
      - chrisweekly 4y ago
        Yes! IRL Alanis is smart and eloquent, and those that think that song's lyrics are evidence to the contrary are missing the joke.
- baobob 4y ago
  Moved all my domains to DO specifically to stop donating traffic data to Cloudflare. Absolutely stunned I didn't notice this earlier. It's bullshit all the way down. Are there any companies left offering free DNS usable from Terraform that aren't part of the "Internet Five Eyes"? edit: looks like Linode may be the next best 'not terrible' option

Akuehne 4y ago

Just wanted to also reiterate how thankful I am that you took your time to let us know. It speaks volumes.

taf2 4y ago
Agreed- couldn’t figure out what was going on… finally checked here and - ah now I can sleep
- Abishek_Muthian 4y ago
  Cloudflare going down is one of the things which keeps me awake, My main complaint about Cloudflare is that they are very good at everything they offer that we've become reliant on them for everything.
  - jimmygrapes 4y ago
    "do no evil" springs to mind -- once burned, etc.
    - Abishek_Muthian 4y ago
      Yeah, What's up with the competition to Cloudflare? What's the real barrier for entry? It's not infrastructure anymore, As there is a new PaaS startup every week offering distributed hosting and So why bundling in DNS, DDOS detection+mitigation, cloud workers... with it is so hard?
      - subvarad 4y ago
        There are many noteworthy players - Akamai, Fastly etc., and Edge plaoviders like ourselves (Zycada) who complement top CDNs like Akamai, Cloudflare, Fastly.
        Abishek_Muthian 4y ago
        The main difference between Cloudflare and the others mentioned is the price; One can start with CF for a side project for free and continue to use it free till it becomes a viable startup. Others at best offer a limited trial plan, But most are just 'Speak to expert/ Contact us' for pricing which means haggling with a sales rep while we can just build things. Even the paid plans of CF is reasonable when compared with others with better features.

obnauticus 4y ago

Which undersea cable was cut ;)

noitpmeder 4y ago
Can't really roll back that change
- ekimekim 4y ago
  Sure you can, but physically rolling back more cable takes a bit longer :P

janmo 4y ago

Should probably do slower roulouts next time.

bilekas 4y ago

Thanks for the update, just curious if we will get a report on what happened ? In as much detail as can be of course - morbid curiosity mainly. I love the post reports these events usually bring.

blue_cookeh 4y ago
Cloudflare are usually pretty good with posting post mortems. https://blog.cloudflare.com/tag/postmortem/ https://blog.cloudflare.com/tag/postmortem/
- brycewray 4y ago
  https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/ https://blog.cloudflare.com/cloudflare-outage-on-june-21-202...

rcshubhadeep 4y ago

Thanks for letting us know

llausa 4y ago

hats off to you sir - would not want to be in your shoes right now but thanks for the updates

jgrahamc 4y ago
I don't have shoes on.
- tomglynch 4y ago
  In tumultuous times (fix wasn't implemented at this point), the Cloudflare CTO still has time for some wit. Love it!
- stavros 4y ago
  That's the point, two people can't fit in them.

b3lvedere 4y ago

Thank you for your very fast support!

keyle 4y ago

It's back wooo! And I'm saying this for the last time: no one type google into google!

deleted 4y ago

[deleted]

MarkyMark210 4y ago

DR much ?

jgrahamc 4y ago
I don't know what that means.
- mike_d 4y ago
  DR means "disaster recovery," it is a formal plan used to respond to and mitigate potential risks to the business. Things like having a communications plan for an incident, or a backup office outside of your main office natural disaster zone.
  - jgrahamc 4y ago
    Ah. Just one more reason I hate acronyms. They obscure what the person is trying to say.

dangrossman 4y ago

All my sites behind Cloudflare had come back up, and have now gone back to serving 500 errors. The Cloudflare Dashboard is also no longer fully loading.

jgrahamc 4y ago
Where are you located?
- dangrossman 4y ago
  Raleigh, NC, USA Sites are gradually reappearing as I type this. Some of my sites, and doordash.com, were returning 500 errors again just a minute ago. They just came back up, followed by the CF dashboard loading again.
- hakanderyal 4y ago
  I'm from Turkey, and I also have been seeing intermittent errors for the last 10 minutes. Seems ok now.

mikesabbagh 4y ago

CF SRE team need to rethink their published SLA of 100%. This is not reasonable. https://www.cloudflare.com/business-sla/ https://www.cloudflare.com/business-sla/

deleted 4y ago
[deleted]
subvarad 4y ago
I completely can understand your emotion. But even the top CDNs can have outages of some form or the other. If site uptime is important, check out https://www.cdnreserve.com/ https://www.cdnreserve.com/ - it's built on the design principle that the likelihood of two separate platforms having an outage at the same time is close to zero.
arccy 4y ago
that just means they're willing to pay for the marketing number, not that they will actually achieve it
stn_za 4y ago
Yep, I'd promise 99.95 at a stretch, never 100%. They are not being honest with themselves here
- latch 4y ago
  I'm not sure you and your parent understand what an SLA means. It's an agreement that, when broken, incurs a penalty. They aren't saying they guarantee 100% uptime. They're saying they'll pay you for any downtime. It's literally the 3rd paragraph: > 1.2 Penalties. If the Service fails to meet the above service level, the Customer will receive a credit equal to the result of the Service Credit calculation in Section 6 of this SLA. (Most people I know consider them meaningless marketing BS that's really just meant to trick people or satisfy some make-work checkbox)
  - greyface- 4y ago