4 ms·

Cloudflare API, dashboard, tunnels down

140 points by supermatti 3y ago

supermatti 3y ago
Tunnels work until you restart them
- sorenbs 3y ago
  Thats great to hear. Would be scary if they experienced a full outage.
- whalesalad 3y ago
  I hope you didn’t find that out the hard way.
  - supermatti 3y ago
    the hard way is the only way
    - baetylus 3y ago
      this man has found out
- SSLy 3y ago
  This feels like data and control planes being at least somewhat separated.
  - piperswe 3y ago
    As a general rule, Cloudflare hot paths (CDN, etc.) tend to still work during an API outage. Communication between the API servers and the hot paths are mostly async, and (usually) work fine independently.
ChrisArchitect 3y ago
Seeing improvements on this now as of 5 mins ago
AtNightWeCode 3y ago
[flagged]
- deleted 3y ago
  [deleted]
- strangescript 3y ago
  lol what
  - hallway_monitor 3y ago
    Right? This whole thread is more reddit than HN
- teaearlgraycold 3y ago
  If you want to find out who uses Cloudflare, can I interest you in what's known as the domain name service?
  - AtNightWeCode 3y ago
    You and all the others did not get it did you. The size of the outcry corresponds to the user base. If you fked something up that was really used at scale this thread would have exploded.
  - AtNightWeCode 3y ago
    We sell CF with every customer. Not the point...
- deleted 3y ago
  [deleted]
- AtNightWeCode 3y ago
  [flagged]
Johnny555 3y ago
On the positive side, the postmortem should make for interesting reading.
- vaughnegut 3y ago
  This has become my knee jerk reaction to these kinds of outages. It's always interesting when the more indepth engineering-focused post-mortem comes out
- SheinhardtWigCo 3y ago
  It’s Cloudflare, so you know it’ll be good.
  - rudasn 3y ago
    And you know its going to be DNS related caused by a bad config push :p
jxf 3y ago
Confirming that I'm seeing improvements on this. Tunnels weren't working for me before but they are now.
dijit 3y ago
If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production. People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard. I guess the argument is that outsourcing is easier, but then you are very much out of control. “the internet” in its practical entirety going down becomes more possible every passing day.
- ta1243 3y ago
  Operating at scale is hard. However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.
  - baq 3y ago
    In practice, cloudflare or AWS down means half the internet doesn't work at all. The half that works - routing around problematic networks - isn't very helpful when the problematic networks are the useful ones.
    - ta1243 3y ago
      Depends what you want to get to. HN doesn't use any centralised services. If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.
      - djbusby 3y ago
        Isn't HN centralized on like two servers in CA?
cust 3y ago
If Cloudflare goes down, Internet goes down.
- deleted 3y ago
  [deleted]
- unshavedyak 3y ago
  I’m sure you’re not. Enough people do care up upvote, though. Of course that’s just the people who care, not those who don’t.
  - deleted 3y ago
    [deleted]
- corobo 3y ago
  Confusing reply wording due to comment edit. Was: "I can't be the only person who literally doesn't give a F?"
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
mrcino 3y ago
Brace yourself, DDoS is coming...
RockRobotRock 3y ago
working for me @ 10:23 AM in Oregon
Raed667 3y ago
I started leaning the Shopify CLI and syntax and was surprised to know that nothing works without Cloudflare tunnels. There is an option to use ngrok instead, but I have never seen such a setup for "local" development.
- paulddraper 3y ago
  I used ngrok for Shopify dev; I didn't know there was such a thing as CF tunnels.
  - Raed667 3y ago
    I don't know when this became the default, but if you don't define --tunnel-url it will use CF
bern4444 3y ago
Downtime happens - no one is happy about it but it comes with the job and downtime will happen whether you use cloud or run on prem. Cloudflare at least accurately reflects the incident on their status page. Often dashboards will show everything green despite the service being down. In these moments communication is critical and Cloudflare does it well especially as we all know when it comes to the post mortem. They are very transparent about these issues and incidents.
- adrr 3y ago
  Has Akamai had a global outage? They are the largest player in the market and i don't recall one global outage from them.
  - bern4444 3y ago
    Google is a friend :) Looks like there were 2 outages[0], one in June and one in July of 2021. [0] https://www.reuters.com/technology/websites-airlines-banks-tech-companies-down-widespread-outage-2021-07-22/ https://www.reuters.com/technology/websites-airlines-banks-t...
    - Alupis 3y ago
      These would be the only ones big enough and long-lasting enough to make the news for non-techies. Your regular, run-of-the-mill outages aren't reported in Reuters... but are likely much more common than you'd believe.
- NicoJuicy 3y ago
  True. It's also nuts that this downtime according to creation of the post -> resolved in comments, was only 27 minutes... I've literally been on standby to get an answer what happened with a huge cloud provider for 3 weeks now and today they got back with static assets that weren't reachable ( ico's ???) They don't even know what happened since it's not logged in their logging, since it's using middleware and because of the sudden weekend downtime, they middleware wasn't triggered... After a detailed description that we found that it was related to a app service plan, they came back with 404's on static assets... Only now it got escalated.. Fuck the cloud, outside of cloudflare. The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare . Then their was a infra update and it also didn't get logged... Because of that we thought it was the wrong party... 18 hours down, redeployment fixed it, nothing logged on their end. Luckily one service wasn't redeployed ( barely used), so we can prove there was an issue. Ho boy... I'm wondering how many more weeks it will take...
hipadev23 3y ago
Relax, they just had to install a new beam splitter for VIGIL
- cassonmars 3y ago
  VIGIL?
  - deleted 3y ago
    [deleted]
belter 3y ago
Is it just my impression or the Cloudflare Down seems to be a frequent post? https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=cloudflare+down
senderista 3y ago
At least they're smart enough not to host their own status page...
rwaksmunski 3y ago
Status page that actually shows the current status, such a rare phenomenon.

140 points by supermatti 3y ago

supermatti 3y ago

Tunnels work until you restart them

sorenbs 3y ago
Thats great to hear. Would be scary if they experienced a full outage.
whalesalad 3y ago
I hope you didn’t find that out the hard way.
- supermatti 3y ago
  the hard way is the only way
  - baetylus 3y ago
    this man has found out
SSLy 3y ago
This feels like data and control planes being at least somewhat separated.
- piperswe 3y ago
  As a general rule, Cloudflare hot paths (CDN, etc.) tend to still work during an API outage. Communication between the API servers and the hot paths are mostly async, and (usually) work fine independently.

ChrisArchitect 3y ago

Seeing improvements on this now as of 5 mins ago

AtNightWeCode 3y ago

[flagged]

deleted 3y ago
[deleted]
strangescript 3y ago
lol what
- hallway_monitor 3y ago
  Right? This whole thread is more reddit than HN
teaearlgraycold 3y ago
If you want to find out who uses Cloudflare, can I interest you in what's known as the domain name service?
- AtNightWeCode 3y ago
  You and all the others did not get it did you. The size of the outcry corresponds to the user base. If you fked something up that was really used at scale this thread would have exploded.
- AtNightWeCode 3y ago
  We sell CF with every customer. Not the point...
deleted 3y ago
[deleted]
AtNightWeCode 3y ago
[flagged]

Johnny555 3y ago

On the positive side, the postmortem should make for interesting reading.

vaughnegut 3y ago
This has become my knee jerk reaction to these kinds of outages. It's always interesting when the more indepth engineering-focused post-mortem comes out
SheinhardtWigCo 3y ago
It’s Cloudflare, so you know it’ll be good.
- rudasn 3y ago
  And you know its going to be DNS related caused by a bad config push :p

jxf 3y ago

Confirming that I'm seeing improvements on this. Tunnels weren't working for me before but they are now.

dijit 3y ago

If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production. People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard. I guess the argument is that outsourcing is easier, but then you are very much out of control. “the internet” in its practical entirety going down becomes more possible every passing day.

ta1243 3y ago
Operating at scale is hard. However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.
- baq 3y ago
  In practice, cloudflare or AWS down means half the internet doesn't work at all. The half that works - routing around problematic networks - isn't very helpful when the problematic networks are the useful ones.
  - ta1243 3y ago
    Depends what you want to get to. HN doesn't use any centralised services. If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.
    - djbusby 3y ago
      Isn't HN centralized on like two servers in CA?

cust 3y ago

If Cloudflare goes down, Internet goes down.

deleted 3y ago
[deleted]
unshavedyak 3y ago
I’m sure you’re not. Enough people do care up upvote, though. Of course that’s just the people who care, not those who don’t.
- deleted 3y ago
  [deleted]
corobo 3y ago
Confusing reply wording due to comment edit. Was: "I can't be the only person who literally doesn't give a F?"
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]

mrcino 3y ago

Brace yourself, DDoS is coming...

RockRobotRock 3y ago

working for me @ 10:23 AM in Oregon

Raed667 3y ago

I started leaning the Shopify CLI and syntax and was surprised to know that nothing works without Cloudflare tunnels. There is an option to use ngrok instead, but I have never seen such a setup for "local" development.

paulddraper 3y ago
I used ngrok for Shopify dev; I didn't know there was such a thing as CF tunnels.
- Raed667 3y ago
  I don't know when this became the default, but if you don't define --tunnel-url it will use CF

bern4444 3y ago

Downtime happens - no one is happy about it but it comes with the job and downtime will happen whether you use cloud or run on prem. Cloudflare at least accurately reflects the incident on their status page. Often dashboards will show everything green despite the service being down. In these moments communication is critical and Cloudflare does it well especially as we all know when it comes to the post mortem. They are very transparent about these issues and incidents.

adrr 3y ago
Has Akamai had a global outage? They are the largest player in the market and i don't recall one global outage from them.
- bern4444 3y ago
  Google is a friend :) Looks like there were 2 outages[0], one in June and one in July of 2021. [0] https://www.reuters.com/technology/websites-airlines-banks-tech-companies-down-widespread-outage-2021-07-22/ https://www.reuters.com/technology/websites-airlines-banks-t...
  - Alupis 3y ago
    These would be the only ones big enough and long-lasting enough to make the news for non-techies. Your regular, run-of-the-mill outages aren't reported in Reuters... but are likely much more common than you'd believe.
NicoJuicy 3y ago
True. It's also nuts that this downtime according to creation of the post -> resolved in comments, was only 27 minutes... I've literally been on standby to get an answer what happened with a huge cloud provider for 3 weeks now and today they got back with static assets that weren't reachable ( ico's ???) They don't even know what happened since it's not logged in their logging, since it's using middleware and because of the sudden weekend downtime, they middleware wasn't triggered... After a detailed description that we found that it was related to a app service plan, they came back with 404's on static assets... Only now it got escalated.. Fuck the cloud, outside of cloudflare. The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare . Then their was a infra update and it also didn't get logged... Because of that we thought it was the wrong party... 18 hours down, redeployment fixed it, nothing logged on their end. Luckily one service wasn't redeployed ( barely used), so we can prove there was an issue. Ho boy... I'm wondering how many more weeks it will take...

hipadev23 3y ago

Relax, they just had to install a new beam splitter for VIGIL

cassonmars 3y ago
VIGIL?
- deleted 3y ago
  [deleted]

belter 3y ago

Is it just my impression or the Cloudflare Down seems to be a frequent post? https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=cloudflare+down

senderista 3y ago

At least they're smart enough not to host their own status page...

rwaksmunski 3y ago

Status page that actually shows the current status, such a rare phenomenon.