14 ms·

Cloudflare API Down

286 points by supermatti 3y ago

supermatti 3y ago
for example cloudflare tunnels will not work if restarted, our production is running with 1 single tunnel now.
- ddorian43 3y ago
  Whats the reason for using tunnels and not just ip addresses?
  - wongarsu 3y ago
    You don't have to expose any ports to the internet, preventing people from finding and directly attacking your origin servers.
    - bigjoes 3y ago
      Only downside seems to be the Performance of tunnels in Containers. I use them for my personal Website, did a bit of Loadtesting and was able to get significantly more RPS without the CF Tunnel. Might be something on my end tho, not sure.
      - esdf 3y ago
        That's interesting. Cloudflare tunnels do a few things that I expected to to make it perform better in general: obviously TLS termination on CF's side where they likely have faster hardware doing that (at least faster than many customers), then the keep-alive sockets for tunnel<->CF, and I think they use UDP/QUIC for the tunnel<->CF connection[0] which I figure could remove some latency. [0]: `lsof -i | grep cloudfl` shows me 4 UDP connections & 1 TCP
  - 542458 3y ago
    Makes firewall/ACL administration much simpler for one. Also makes it easier to hide and/or rotate origin IPs.
- hk1337 3y ago
  I hate that Shopify app has cloudflare tunnels ingrained by default. You can use other tunnels, i.e. ngrok, but it's a lot more manual with the setup.
allyant 3y ago
Cannot seem to access the web UI also.
khaledh 3y ago
We use Cloudflare WARP (which is down) to access GitHub. Time to slack off.
- capableweb 3y ago
  If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...
  - AdamN 3y ago
    Something distributed??
  - bluejekyll 3y ago
    I wonder, have there been any case studies on phishing/trojan type attacks by code (git changes) via email?
  - khaledh 3y ago
    Most of the time it's not about the code. Project management, issue tracking and prioritization, discussions, PR reviews, searching across the organization repos, etc.
    - Terretta 3y ago
      There are (all too rare) tools that back those objects with git as well. And there's always fossil ... https://fossil-scm.org/home/doc/trunk/www/index.wiki https://fossil-scm.org/home/doc/trunk/www/index.wiki But it's not git. :-(
  - afavour 3y ago
    Repeat after me: GitHub is more than just Git. GitHub is more than just Git.
    - capableweb 3y ago
      Repeat after me: Git is more than just files. Git is more than just files.
deleted 3y ago
[deleted]
lionkor 3y ago
someone will have their coffee break cut short at cloudflare, oops!
- francisofascii 3y ago
  and extend other's coffee break, since their site is down and can't work.
- blackoil 3y ago
  Many more will get an extended coffee break.
  - amelius 3y ago
    Some will get an eternal coffee break.
    - NicoJuicy 3y ago
      Tbh. Cloudflare is transparant with these kind of cases, they don't do "scape goating". When everyone in tech was firing, they didn't. When an employee made a mistake, Cloudflare refused to name the person and said it was a mistake of the organisation ( was looking for the article, but couldn't find it). During the last years ( lot's of companies fired lots employees because of rising interest, overhiring during Covid, ... ). Contrary: The only time they fired employees in a larger quantity, was in Q2 2023 for the sales department, regarding underperformers ( performing 1/5th of the average). This was described here based on the earnings call: https://softwarestackinvesting.com/cloudflare-net-q2-2023-earnings-review/ https://softwarestackinvesting.com/cloudflare-net-q2-2023-ea... and it's about 100 employees ( on > 3300), where they tried to re-hire others ( who got fired elsewhere because of the raised interests of the FED).
junon 3y ago
Two outages in the last week now. EDIT: To the dead comment response, this has nothing to do with programming language choice.
- cinnor 3y ago
  [flagged]
deleted 3y ago
[deleted]
rewq4321 3y ago
Not just the API - I'm now suddenly having DNS troubles with one of my domains. I think maybe only domains/sites that use their "configuration rules", or some specific feature like that.
- _lvbh 3y ago
  Happened to me yesterday
greyface- 3y ago
Flexential PDX02 reportedly lost power around the same time this started.
- Rygian 3y ago
  Which would you guess, cause or effect?
- jabart 3y ago
  Yikes, I'm curious how that happened. Our Flexential has 5 generators and a basement full of battery backups along with isolated zones.
  - greyface- 3y ago
    https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149... > Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet).
    - jabart 3y ago
      Our local test those like once a month. Looks like our account manager will get a quick check in after this settles to make sure we won't have this issue.
      - acdha 3y ago
        Testing those is SOP but it’s easy to do wrong: I’ve heard of multiple times where the tests had worked but the system failed in a true emergency because of some other factor. The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generator because the main fuel tank pump flooded (AWS was completely unaffected). You just don’t tempt the DR fates like that…
pc_edwin 3y ago
This is not looking good. I really hope they don't move away from dogfooding because of this. Also vercel deployments (and edge functions in general) aren't working. Their status page says: "We identified the root cause as an issue with one of our upstream providers and working with them towards mitigation." I wonder if edge functions use workers under the hood lol. LINK: https://www.vercel-status.com/ https://www.vercel-status.com/
- jadbox 3y ago
  I'm pretty sure edge functions are just workers with an little extra code for metrics and stuff.
- flutas 3y ago
  Updated on the CF side: > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. Someone else in the thread shared this thread [0] which is about Flexential PDX02 losing power at the same time. > Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. Sounds like the entire DC went down and their failover didn't handle it as gracefully as it should have. [0]: https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149...
- brycewray 3y ago
  > I wonder if edge functions use workers under the hood lol. Yep: https://news.ycombinator.com/item?id=29003514 https://news.ycombinator.com/item?id=29003514
- pc_edwin 3y ago
  Digital Ocean too: https://status.digitalocean.com/incidents/bw3r3j9b5ph5 https://status.digitalocean.com/incidents/bw3r3j9b5ph5 I don't know if I should be proud or scared that a lot of these products are just wrappers around Cloudflare offerings. In this case I don't think its a wrapper, DOs App Platform is a bit too sophisticated to be on workers. I'm guessing a bunch of key workloads using workers.
kklisura 3y ago
On the day of their earnings call...
tppiotrowski 3y ago
Bad timing: quarterly earnings call is this afternoon.
- londons_explore 3y ago
  Perhaps someone was rushing to release something that was going to be announced on the earnings call...
  - tppiotrowski 3y ago
    That was yesterday. They announced pricing changes for Cloudflare workers. I think it won't affect me but heavy workloads will have to pay more. They used to charge a flat fee but now are moving to $/CPU time. Overall I still love Cloudflare and run all my backend stuff there. It just feels simpler and cleaner than AWS but it's slowly starting to get cluttered as more features are released.
    - zozskuh 3y ago
      "Cloudflare, Inc. Class A Common Stock is expected* to report earnings on 11/02/2023 after market close." https://www.nasdaq.com/market-activity/stocks/net/earnings https://www.nasdaq.com/market-activity/stocks/net/earnings
    - NicoJuicy 3y ago
      ? They announced that on 28/09 fyi. It's not new and they didn't change a flat fee before. They still have the 5 € / month fee. https://blog.cloudflare.com/workers-pricing-scale-to-zero/ https://blog.cloudflare.com/workers-pricing-scale-to-zero/
      - tppiotrowski 3y ago
        You're right. The email yesterday was that the voluntary opt in period to the new pricing was starting. Everyone will be switched over on March 1st.
corobo 3y ago
Did someone re-commit this bug from the other day? https://blog.cloudflare.com/cloudflare-incident-on-october-30-2023/ https://blog.cloudflare.com/cloudflare-incident-on-october-3... E: nah, looks like a power outage in their brain DC. See other comments for details :)
- noahjk 3y ago
  Every time CF goes down, we get rewarded with an interesting article. At this point I’ve been pavloved into enjoying outages!
  - corobo 3y ago
    Hah, right! I do love an in-depth postmortem
surprisetalk 3y ago
https://www.cloudflarestatus.com https://www.cloudflarestatus.com
cinnor 3y ago
[flagged]
- bluetidepro 3y ago
  Satire?
  - cryptonector 3y ago
    Clearly.
- gzer0 3y ago
  https://www.reuters.com/technology/microsoft-teams-down-thousands-users-india-downdetector-2023-01-25/ https://www.reuters.com/technology/microsoft-teams-down-thou... https://www.reuters.com/technology/amazon-says-multiple-cloud-services-down-users-2023-06-13/ https://www.reuters.com/technology/amazon-says-multiple-clou... https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...
ryandvm 3y ago
I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.
- ebbp 3y ago
  In principle I agree with this, but do feel this is said more readily about Cloudflare than other companies it could said about - such as Amazon (via AWS), Google and Microsoft. Perhaps my own mental model is wrong, but I see them as a credible challenger to those very oligopolistic companies, and wish there were more Cloudflares.
  - acejam 3y ago
    To each their own, but I think this is said more frequently about Cloudflare because they are often playing the middleman, via their CDN service. In comparison, AWS and others are the actual origin.
  - msmith 3y ago
    I feel the same way. What about Akamai, Fastly, or Okta? Maybe Cloudflare gets more attention because their low end plans are accessible to anyone.
    - risyachka 3y ago
      Its not just low end plans. Their pricing is basically the only one that feels fair. They don't charge you for bandwidth, unlike others that try to make on it as much as possible, while at the same time having other services also priced significantly higher.
      - rvnx 3y ago
        + (Global) Cloudflare Workers are amazing compared to Google Cloud Functions or other services that are regional, expensive and slow to start.
      - 3y ago
hk1337 3y ago
Cloudflare has been down a lot this week. Technically, what I am thinking of was Hubspot was down but apparently it was related to Cloudflare issues.
- lawgimenez 3y ago
  Supabase also went down I guess a few days ago because of Cloudflare.
- willio58 3y ago
  Kinsta pages went down because of cloudflare taking our company’s marketing site with it for a good 5-10 minutes.
buro9 3y ago
When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PDX > ship updates / data to PoPs. If you take out PDX, then as so much runs on fresh data, it starts getting stale. I doubt everything has changed since then, so this is unlikely just "API down" and more likely that a lot of things are now in a degraded state as they're running on stale information (no update from PDX)... this includes things like load balancing, the tiered caching (Argo Smart Routing), Warp / Zero Trust, etc. Even if it were only "API down", then bear in mind that a lot of automation customers have will block attacks by calling the API... "API down" is a hell of a window of opportunity for attackers. Note that just before I'd left they'd been investing in standing up AMS (I think) but had never successfully tested a significant failover, and the majority of services that needed fresh state did not know how to do this. PS: :scream: most of the observability was also based in PDX, so hugs to all the teams and SREs currently running blind.
- flutas 3y ago
  Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. [0]: https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149...
  - supriyo-biswas 3y ago
    The more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.
nicky0 3y ago
Looks like it's taken out Digitalocean App Platform deployments too: https://status.digitalocean.com/incidents/bw3r3j9b5ph5 https://status.digitalocean.com/incidents/bw3r3j9b5ph5
8ig8 3y ago
DreamHost having issues at PDX since 4:54AM PDT... https://www.dreamhoststatus.com https://www.dreamhoststatus.com
spicykraken 3y ago
> Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. Oh, no! The lava lamps are off, there's not enough entropy! https://www.cloudflare.com/learning/ssl/lava-lamp-encryption/ https://www.cloudflare.com/learning/ssl/lava-lamp-encryption...
t3rabytes 3y ago
Same PDX outage has been affecting Workday's WD5 dc, too.
valgaze 3y ago
Edge— “These products are impacted at the control plane / core level, meaning that only the changes to the existing configuration are affected, but the product is functioning at the edge …”
j16sdiz 3y ago
The most impressive thing is: The status page actually works
- josh_carterPDX 3y ago
  Isn't it ironic that of all the services failing the status page never seems to go down? Of course it's likely hosted somewhere else, but I always chuckled and thought the same thing when there was a major outage somewhere.
  - xp84 3y ago
    I think that's less of a coincidence than due to the advice many people give that you separate your status page from the infrastructure it's reporting on in every single way possible. One example of the approach is the separate top-level domain usually used. If I were setting a status page up, I would ensure it was with a different cloud provider, and ideally a separate side of the country or separate continent from my principal DC.
- lillecarl 3y ago
  "Powered by Atlassian Statuspage"
- latraveler 3y ago
  why don't they just make the plane out of the status page?
deleted 3y ago
[deleted]
alams 3y ago
Looks like Salesforce and Cloudflare sharing the same datacenter. Salesforce had the same issue today and they mentioned similar power failure as the root cause. It is a known fact that salesforce is running its "Hyperforce" in AWS. Does that mean Cloudflare also using AWS? Does anyone faced any issue on AWS resources today?
- nacholibrev 3y ago
  Judging from the AWS status page it seems that all services are up and running without any issues. https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
- NicoJuicy 3y ago
  Cloudflare, as far is I know, doesn't use any AWS products.
pyuser583 3y ago
And their quarterly earnings reports are due today. Ouch.
dogweather 3y ago
Cloudflare really embarrassed me: I had attempted to do an ad hoc demo for my client in a morning meeting. Whoops.
- ulti552 3y ago
  oof sorry to hear that
dogweather 3y ago
It's 60 degrees in PDX; the power outage doesn't seem weather related.
JohnMakin 3y ago
"Sure, why not use cloudflare provider rather than simply hardcode values into our IAC, what could go wrong?" Hands completely tied.. crazy how long this has been ongoing.
sackfield 3y ago
I've been trying to bring up a Github Enterprise instance and this is giving me a lot of grief (wanted to proxy it through Cloudflare DNS). It's been going for quite a while... I'm really quite shocked at the length of recovery time needed. Currently can't view analytics to find out why my instance isn't working.
dmix 3y ago
It's now been 12 hours and they are still having issues. I'm taking an online video course in music production and they host their videos on cloudflare stream and none of them work. https://www.cloudflare.com/products/cloudflare-stream/ https://www.cloudflare.com/products/cloudflare-stream/ The single point of failure of half the internet continues to rear it's head.
- sanat 3y ago
  It's now 24 hours and Stream is still not up.
- alberth 3y ago
  > "These issues do not affect the serving of cached files via the Cloudflare CDN or other security features at the Cloudflare Edge." https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.cloudflarestatus.com/incidents/hm7491k53ppg I thought this only affected the Dashboard (and functionality inside the dashboard). Not anything else. Is that not the case, since you're saying cloud flare video streaming isn't working?
lbriner 3y ago
I think most of us can forgive pretty much any mistake but for a service company, there are two golden rules: 1) Know that you have a problem quickly. We reported an issue yesterday and it took them 3 hours to acknowledge that there was a problem, causing us to waste hours trying to make sure nothing on our system was causing it 2) Always have a quick way to roll back changes or mitigate a problem. I think they were having some power supply issue but it is still a business risk that should have been solvable quickly. There are certainly not the only ones but when you keep blogging about your company's technical prowess, this makes you look stupid.
deleted 3y ago
[deleted]
datavirtue 3y ago
I managed to log in last night but there wasn't much use to doing so. Once I was in I could barely run a domain search in the registration app. Errors were being thrown on every dashboard.
nodesocket 3y ago
Meanwhile CloudFlare stock is up 10% today. lol
willtemperley 3y ago
And yet their share price is up over 5% today https://uk.finance.yahoo.com/quote/NET?.tsrc=applewf&guccounter=1 https://uk.finance.yahoo.com/quote/NET?.tsrc=applewf&guccoun...
- kmlx 3y ago
  they reported strong earnings yesterday: https://finance.yahoo.com/news/cloudflare-inc-nyse-net-q3-144726557.html https://finance.yahoo.com/news/cloudflare-inc-nyse-net-q3-14...
zackmorris 3y ago
IMHO edge servers are a waste of time that expose conceptual flaws in the internet's design. We should have had free public proxy servers that relayed any publicly-accessible (public domain) static files automagically like Coral CDN, which is difficult to even find information about anymore: https://en-academic.com/dic.nsf/enwiki/495553 https://en-academic.com/dic.nsf/enwiki/495553 And that caching should have happened in the ISP's backbone or user's devices so that stuff like municipal wifi "just works". Why are we all downloading the same Netflix stream over and over instead of using a protocol more like BitTorrent with a hash tree or content addressable memory using Subresource Integrity (SRI) for stuff like scripts and fonts instead of compiling it all into huge app.js files? Ridiculous. Short of that, basically all servers should be using something like Varnish cache, which drops in with little configuration, leaving the main lag time the distance between countries: http://varnish-cache.org/intro/index.html#intro http://varnish-cache.org/intro/index.html#intro One Node.js instance can serve like a million users per month. And it only takes 65 ms for light to get anywhere in the world (12,000 miles divided by 186,000 miles per second). Maybe 200 ms if the speed of light is 3 times slower in wires and fiber optic lines. Yet tons of cloud apps run slower than 1990s cgi-bin servers. Imagine how we could improve the internet's core technologies with even the tiniest fraction of a fraction of the tech sector's profits and any leisure time at all. This is perhaps the greatest disappointment of my life, watching all of the innovations that I thought would happen come about in a private/corporate fashion which weakens our freedom and privacy.
- gtirloni 3y ago
  I think your point boils down to "vendor's edge cache are too far away and should be at the city level", right? If so, major cities already have local Netflix caches in many ISPs, for example. Same is done for Google, Facebook and others. It takes peering contracts, colocation, etc, but it exists extensively. Maybe you're arguing for government-sponsored caching infrastructure?
  - zackmorris 3y ago
    Well, sort of. I feel that something like Interplanetary File System (IPFS) should have been the original web. Or at least, after BitTorrent arrived, we should have all moved to content-addressable memories because they automatically handle caching and scaling. That would have required solving circles of trust with potential improvements to SSL certificates so that they don't require a central authority, like a distributed version of letsencrypt.org. It also would have needed improvements to DNS for making domain-data pairs instead of domain-IP. And at the very least, IPv6 should have improved UDP so that it "just works" everywhere and doesn't have to deal with NAT hole punching. That stuff represents the real work of rolling out a true P2P internet. Since we did none of that, we cemented the centralized web we have today, full of walled gardens funding the copyright nanny state. So that people are so aghast at seeing the real internet running on stuff like TikTok that their first instinct is to ban it. To me, it feels like we're living under a bizarro version of the internet, so far removed from its academic/declarative/data-driven roots that it's almost unrecognizable. Javascript is just distributed async desktop programming now, reminiscent of the C++ days of the late 1980s, complete with all of the nondeterminism that we worked so hard to get away from. Paid services like Cloudflare and 5G are a symptom of that inside-the-box thinking. If we had real distributed P2P through our routers and cell phones, we probably wouldn't need ISPs today, because we'd be accustomed to gigabit download speeds and would only perceive long distance as slow and worth paying for. In which case, maybe the government should pay to maintain the backbone and perhaps satellite internet like Starlink. Which would be a good thing IMHO, because at least there would be oversight and we wouldn't have to worry about billionaire narcissists making the satellites highly reflective and visible to the naked eye, like carving a McDonalds sign on the moon.

Cloudflare API Down

286 points by supermatti 3y ago

supermatti 3y ago

for example cloudflare tunnels will not work if restarted, our production is running with 1 single tunnel now.

ddorian43 3y ago
Whats the reason for using tunnels and not just ip addresses?
- wongarsu 3y ago
  You don't have to expose any ports to the internet, preventing people from finding and directly attacking your origin servers.
  - bigjoes 3y ago
    Only downside seems to be the Performance of tunnels in Containers. I use them for my personal Website, did a bit of Loadtesting and was able to get significantly more RPS without the CF Tunnel. Might be something on my end tho, not sure.
    - esdf 3y ago
      That's interesting. Cloudflare tunnels do a few things that I expected to to make it perform better in general: obviously TLS termination on CF's side where they likely have faster hardware doing that (at least faster than many customers), then the keep-alive sockets for tunnel<->CF, and I think they use UDP/QUIC for the tunnel<->CF connection[0] which I figure could remove some latency. [0]: `lsof -i | grep cloudfl` shows me 4 UDP connections & 1 TCP
- 542458 3y ago
  Makes firewall/ACL administration much simpler for one. Also makes it easier to hide and/or rotate origin IPs.
hk1337 3y ago
I hate that Shopify app has cloudflare tunnels ingrained by default. You can use other tunnels, i.e. ngrok, but it's a lot more manual with the setup.

allyant 3y ago

Cannot seem to access the web UI also.

khaledh 3y ago

We use Cloudflare WARP (which is down) to access GitHub. Time to slack off.

capableweb 3y ago
If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...
- AdamN 3y ago
  Something distributed??
- bluejekyll 3y ago
  I wonder, have there been any case studies on phishing/trojan type attacks by code (git changes) via email?
- khaledh 3y ago
  Most of the time it's not about the code. Project management, issue tracking and prioritization, discussions, PR reviews, searching across the organization repos, etc.
  - Terretta 3y ago
    There are (all too rare) tools that back those objects with git as well. And there's always fossil ... https://fossil-scm.org/home/doc/trunk/www/index.wiki https://fossil-scm.org/home/doc/trunk/www/index.wiki But it's not git. :-(
- afavour 3y ago
  Repeat after me: GitHub is more than just Git. GitHub is more than just Git.
  - capableweb 3y ago
    Repeat after me: Git is more than just files. Git is more than just files.

deleted 3y ago

[deleted]

lionkor 3y ago

someone will have their coffee break cut short at cloudflare, oops!

francisofascii 3y ago
and extend other's coffee break, since their site is down and can't work.
blackoil 3y ago
Many more will get an extended coffee break.
- amelius 3y ago
  Some will get an eternal coffee break.
  - NicoJuicy 3y ago
    Tbh. Cloudflare is transparant with these kind of cases, they don't do "scape goating". When everyone in tech was firing, they didn't. When an employee made a mistake, Cloudflare refused to name the person and said it was a mistake of the organisation ( was looking for the article, but couldn't find it). During the last years ( lot's of companies fired lots employees because of rising interest, overhiring during Covid, ... ). Contrary: The only time they fired employees in a larger quantity, was in Q2 2023 for the sales department, regarding underperformers ( performing 1/5th of the average). This was described here based on the earnings call: https://softwarestackinvesting.com/cloudflare-net-q2-2023-earnings-review/ https://softwarestackinvesting.com/cloudflare-net-q2-2023-ea... and it's about 100 employees ( on > 3300), where they tried to re-hire others ( who got fired elsewhere because of the raised interests of the FED).

junon 3y ago

Two outages in the last week now. EDIT: To the dead comment response, this has nothing to do with programming language choice.

cinnor 3y ago
[flagged]

deleted 3y ago

[deleted]

rewq4321 3y ago

Not just the API - I'm now suddenly having DNS troubles with one of my domains. I think maybe only domains/sites that use their "configuration rules", or some specific feature like that.

_lvbh 3y ago
Happened to me yesterday

greyface- 3y ago

Flexential PDX02 reportedly lost power around the same time this started.

Rygian 3y ago
Which would you guess, cause or effect?
jabart 3y ago
Yikes, I'm curious how that happened. Our Flexential has 5 generators and a basement full of battery backups along with isolated zones.
- greyface- 3y ago
  https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149... > Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet).
  - jabart 3y ago
    Our local test those like once a month. Looks like our account manager will get a quick check in after this settles to make sure we won't have this issue.
    - acdha 3y ago
      Testing those is SOP but it’s easy to do wrong: I’ve heard of multiple times where the tests had worked but the system failed in a true emergency because of some other factor. The best one was probably the time Internap made a cheap shot at AWS about outages before hurricane Sandy hit, only to have their NYC data center go offline once their generators burned through the small amount of fuel they had near the generator because the main fuel tank pump flooded (AWS was completely unaffected). You just don’t tempt the DR fates like that…

pc_edwin 3y ago

This is not looking good. I really hope they don't move away from dogfooding because of this. Also vercel deployments (and edge functions in general) aren't working. Their status page says: "We identified the root cause as an issue with one of our upstream providers and working with them towards mitigation." I wonder if edge functions use workers under the hood lol. LINK: https://www.vercel-status.com/ https://www.vercel-status.com/

jadbox 3y ago
I'm pretty sure edge functions are just workers with an little extra code for metrics and stuff.
flutas 3y ago
Updated on the CF side: > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. Someone else in the thread shared this thread [0] which is about Flexential PDX02 losing power at the same time. > Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. Sounds like the entire DC went down and their failover didn't handle it as gracefully as it should have. [0]: https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149...
brycewray 3y ago
> I wonder if edge functions use workers under the hood lol. Yep: https://news.ycombinator.com/item?id=29003514 https://news.ycombinator.com/item?id=29003514
pc_edwin 3y ago
Digital Ocean too: https://status.digitalocean.com/incidents/bw3r3j9b5ph5 https://status.digitalocean.com/incidents/bw3r3j9b5ph5 I don't know if I should be proud or scared that a lot of these products are just wrappers around Cloudflare offerings. In this case I don't think its a wrapper, DOs App Platform is a bit too sophisticated to be on workers. I'm guessing a bunch of key workloads using workers.

kklisura 3y ago

On the day of their earnings call...

tppiotrowski 3y ago

Bad timing: quarterly earnings call is this afternoon.

londons_explore 3y ago
Perhaps someone was rushing to release something that was going to be announced on the earnings call...
- tppiotrowski 3y ago
  That was yesterday. They announced pricing changes for Cloudflare workers. I think it won't affect me but heavy workloads will have to pay more. They used to charge a flat fee but now are moving to $/CPU time. Overall I still love Cloudflare and run all my backend stuff there. It just feels simpler and cleaner than AWS but it's slowly starting to get cluttered as more features are released.
  - zozskuh 3y ago
    "Cloudflare, Inc. Class A Common Stock is expected* to report earnings on 11/02/2023 after market close." https://www.nasdaq.com/market-activity/stocks/net/earnings https://www.nasdaq.com/market-activity/stocks/net/earnings
  - NicoJuicy 3y ago
    ? They announced that on 28/09 fyi. It's not new and they didn't change a flat fee before. They still have the 5 € / month fee. https://blog.cloudflare.com/workers-pricing-scale-to-zero/ https://blog.cloudflare.com/workers-pricing-scale-to-zero/
    - tppiotrowski 3y ago
      You're right. The email yesterday was that the voluntary opt in period to the new pricing was starting. Everyone will be switched over on March 1st.

corobo 3y ago

Did someone re-commit this bug from the other day? https://blog.cloudflare.com/cloudflare-incident-on-october-30-2023/ https://blog.cloudflare.com/cloudflare-incident-on-october-3... E: nah, looks like a power outage in their brain DC. See other comments for details :)

noahjk 3y ago
Every time CF goes down, we get rewarded with an interesting article. At this point I’ve been pavloved into enjoying outages!
- corobo 3y ago
  Hah, right! I do love an in-depth postmortem

surprisetalk 3y ago

https://www.cloudflarestatus.com https://www.cloudflarestatus.com

cinnor 3y ago

[flagged]

bluetidepro 3y ago
Satire?
- cryptonector 3y ago
  Clearly.
gzer0 3y ago
https://www.reuters.com/technology/microsoft-teams-down-thousands-users-india-downdetector-2023-01-25/ https://www.reuters.com/technology/microsoft-teams-down-thou... https://www.reuters.com/technology/amazon-says-multiple-cloud-services-down-users-2023-06-13/ https://www.reuters.com/technology/amazon-says-multiple-clou... https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...

ryandvm 3y ago

I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.

ebbp 3y ago
In principle I agree with this, but do feel this is said more readily about Cloudflare than other companies it could said about - such as Amazon (via AWS), Google and Microsoft. Perhaps my own mental model is wrong, but I see them as a credible challenger to those very oligopolistic companies, and wish there were more Cloudflares.
- acejam 3y ago
  To each their own, but I think this is said more frequently about Cloudflare because they are often playing the middleman, via their CDN service. In comparison, AWS and others are the actual origin.
- msmith 3y ago
  I feel the same way. What about Akamai, Fastly, or Okta? Maybe Cloudflare gets more attention because their low end plans are accessible to anyone.
  - risyachka 3y ago
    Its not just low end plans. Their pricing is basically the only one that feels fair. They don't charge you for bandwidth, unlike others that try to make on it as much as possible, while at the same time having other services also priced significantly higher.
    - rvnx 3y ago
      + (Global) Cloudflare Workers are amazing compared to Google Cloud Functions or other services that are regional, expensive and slow to start.
    - 3y ago

hk1337 3y ago

Cloudflare has been down a lot this week. Technically, what I am thinking of was Hubspot was down but apparently it was related to Cloudflare issues.

lawgimenez 3y ago
Supabase also went down I guess a few days ago because of Cloudflare.
willio58 3y ago
Kinsta pages went down because of cloudflare taking our company’s marketing site with it for a good 5-10 minutes.

buro9 3y ago

When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PDX > ship updates / data to PoPs. If you take out PDX, then as so much runs on fresh data, it starts getting stale. I doubt everything has changed since then, so this is unlikely just "API down" and more likely that a lot of things are now in a degraded state as they're running on stale information (no update from PDX)... this includes things like load balancing, the tiered caching (Argo Smart Routing), Warp / Zero Trust, etc. Even if it were only "API down", then bear in mind that a lot of automation customers have will block attacks by calling the API... "API down" is a hell of a window of opportunity for attackers. Note that just before I'd left they'd been investing in standing up AMS (I think) but had never successfully tested a significant failover, and the majority of services that needed fresh state did not know how to do this. PS: :scream: most of the observability was also based in PDX, so hugs to all the teams and SREs currently running blind.

flutas 3y ago
Someone else posted about PDX02 going down entirely[0], so sounds like this is the root cause, especially with the latest status update. > Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. > [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site. [0]: https://puck.nether.net/pipermail/outages/2023-November/014915.html https://puck.nether.net/pipermail/outages/2023-November/0149...
- supriyo-biswas 3y ago
  The more concerning issue here is that their control plane is based out of a single datacenter. A multi-datacenter setup, which, based on their stack, could just be jobs running on top of a distributed key-value store (and for the uninitiated, this is effectively what Kubernetes is) could greatly alleviate such concerns.

nicky0 3y ago

Looks like it's taken out Digitalocean App Platform deployments too: https://status.digitalocean.com/incidents/bw3r3j9b5ph5 https://status.digitalocean.com/incidents/bw3r3j9b5ph5

8ig8 3y ago

DreamHost having issues at PDX since 4:54AM PDT... https://www.dreamhoststatus.com https://www.dreamhoststatus.com

spicykraken 3y ago

> Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services. Oh, no! The lava lamps are off, there's not enough entropy! https://www.cloudflare.com/learning/ssl/lava-lamp-encryption/ https://www.cloudflare.com/learning/ssl/lava-lamp-encryption...

t3rabytes 3y ago

Same PDX outage has been affecting Workday's WD5 dc, too.

valgaze 3y ago

Edge— “These products are impacted at the control plane / core level, meaning that only the changes to the existing configuration are affected, but the product is functioning at the edge …”

j16sdiz 3y ago

The most impressive thing is: The status page actually works

josh_carterPDX 3y ago
Isn't it ironic that of all the services failing the status page never seems to go down? Of course it's likely hosted somewhere else, but I always chuckled and thought the same thing when there was a major outage somewhere.
- xp84 3y ago
  I think that's less of a coincidence than due to the advice many people give that you separate your status page from the infrastructure it's reporting on in every single way possible. One example of the approach is the separate top-level domain usually used. If I were setting a status page up, I would ensure it was with a different cloud provider, and ideally a separate side of the country or separate continent from my principal DC.
lillecarl 3y ago
"Powered by Atlassian Statuspage"
latraveler 3y ago
why don't they just make the plane out of the status page?

deleted 3y ago

[deleted]

alams 3y ago

Looks like Salesforce and Cloudflare sharing the same datacenter. Salesforce had the same issue today and they mentioned similar power failure as the root cause. It is a known fact that salesforce is running its "Hyperforce" in AWS. Does that mean Cloudflare also using AWS? Does anyone faced any issue on AWS resources today?

nacholibrev 3y ago
Judging from the AWS status page it seems that all services are up and running without any issues. https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
NicoJuicy 3y ago
Cloudflare, as far is I know, doesn't use any AWS products.

pyuser583 3y ago

And their quarterly earnings reports are due today. Ouch.

dogweather 3y ago

Cloudflare really embarrassed me: I had attempted to do an ad hoc demo for my client in a morning meeting. Whoops.

ulti552 3y ago
oof sorry to hear that

dogweather 3y ago

It's 60 degrees in PDX; the power outage doesn't seem weather related.

JohnMakin 3y ago

"Sure, why not use cloudflare provider rather than simply hardcode values into our IAC, what could go wrong?" Hands completely tied.. crazy how long this has been ongoing.

sackfield 3y ago

I've been trying to bring up a Github Enterprise instance and this is giving me a lot of grief (wanted to proxy it through Cloudflare DNS). It's been going for quite a while... I'm really quite shocked at the length of recovery time needed. Currently can't view analytics to find out why my instance isn't working.

dmix 3y ago

It's now been 12 hours and they are still having issues. I'm taking an online video course in music production and they host their videos on cloudflare stream and none of them work. https://www.cloudflare.com/products/cloudflare-stream/ https://www.cloudflare.com/products/cloudflare-stream/ The single point of failure of half the internet continues to rear it's head.

sanat 3y ago
It's now 24 hours and Stream is still not up.
alberth 3y ago
> "These issues do not affect the serving of cached files via the Cloudflare CDN or other security features at the Cloudflare Edge." https://www.cloudflarestatus.com/incidents/hm7491k53ppg https://www.cloudflarestatus.com/incidents/hm7491k53ppg I thought this only affected the Dashboard (and functionality inside the dashboard). Not anything else. Is that not the case, since you're saying cloud flare video streaming isn't working?

lbriner 3y ago

I think most of us can forgive pretty much any mistake but for a service company, there are two golden rules: 1) Know that you have a problem quickly. We reported an issue yesterday and it took them 3 hours to acknowledge that there was a problem, causing us to waste hours trying to make sure nothing on our system was causing it 2) Always have a quick way to roll back changes or mitigate a problem. I think they were having some power supply issue but it is still a business risk that should have been solvable quickly. There are certainly not the only ones but when you keep blogging about your company's technical prowess, this makes you look stupid.

deleted 3y ago

[deleted]

datavirtue 3y ago

I managed to log in last night but there wasn't much use to doing so. Once I was in I could barely run a domain search in the registration app. Errors were being thrown on every dashboard.

nodesocket 3y ago

Meanwhile CloudFlare stock is up 10% today. lol

willtemperley 3y ago

And yet their share price is up over 5% today https://uk.finance.yahoo.com/quote/NET?.tsrc=applewf&guccounter=1 https://uk.finance.yahoo.com/quote/NET?.tsrc=applewf&guccoun...

kmlx 3y ago
they reported strong earnings yesterday: https://finance.yahoo.com/news/cloudflare-inc-nyse-net-q3-144726557.html https://finance.yahoo.com/news/cloudflare-inc-nyse-net-q3-14...

zackmorris 3y ago

IMHO edge servers are a waste of time that expose conceptual flaws in the internet's design. We should have had free public proxy servers that relayed any publicly-accessible (public domain) static files automagically like Coral CDN, which is difficult to even find information about anymore: https://en-academic.com/dic.nsf/enwiki/495553 https://en-academic.com/dic.nsf/enwiki/495553 And that caching should have happened in the ISP's backbone or user's devices so that stuff like municipal wifi "just works". Why are we all downloading the same Netflix stream over and over instead of using a protocol more like BitTorrent with a hash tree or content addressable memory using Subresource Integrity (SRI) for stuff like scripts and fonts instead of compiling it all into huge app.js files? Ridiculous. Short of that, basically all servers should be using something like Varnish cache, which drops in with little configuration, leaving the main lag time the distance between countries: http://varnish-cache.org/intro/index.html#intro http://varnish-cache.org/intro/index.html#intro One Node.js instance can serve like a million users per month. And it only takes 65 ms for light to get anywhere in the world (12,000 miles divided by 186,000 miles per second). Maybe 200 ms if the speed of light is 3 times slower in wires and fiber optic lines. Yet tons of cloud apps run slower than 1990s cgi-bin servers. Imagine how we could improve the internet's core technologies with even the tiniest fraction of a fraction of the tech sector's profits and any leisure time at all. This is perhaps the greatest disappointment of my life, watching all of the innovations that I thought would happen come about in a private/corporate fashion which weakens our freedom and privacy.

gtirloni 3y ago
I think your point boils down to "vendor's edge cache are too far away and should be at the city level", right? If so, major cities already have local Netflix caches in many ISPs, for example. Same is done for Google, Facebook and others. It takes peering contracts, colocation, etc, but it exists extensively. Maybe you're arguing for government-sponsored caching infrastructure?
- zackmorris 3y ago
  Well, sort of. I feel that something like Interplanetary File System (IPFS) should have been the original web. Or at least, after BitTorrent arrived, we should have all moved to content-addressable memories because they automatically handle caching and scaling. That would have required solving circles of trust with potential improvements to SSL certificates so that they don't require a central authority, like a distributed version of letsencrypt.org. It also would have needed improvements to DNS for making domain-data pairs instead of domain-IP. And at the very least, IPv6 should have improved UDP so that it "just works" everywhere and doesn't have to deal with NAT hole punching. That stuff represents the real work of rolling out a true P2P internet. Since we did none of that, we cemented the centralized web we have today, full of walled gardens funding the copyright nanny state. So that people are so aghast at seeing the real internet running on stuff like TikTok that their first instinct is to ban it. To me, it feels like we're living under a bizarro version of the internet, so far removed from its academic/declarative/data-driven roots that it's almost unrecognizable. Javascript is just distributed async desktop programming now, reminiscent of the C++ days of the late 1980s, complete with all of the nondeterminism that we worked so hard to get away from. Paid services like Cloudflare and 5G are a symptom of that inside-the-box thinking. If we had real distributed P2P through our routers and cell phones, we probably wouldn't need ISPs today, because we'd be accustomed to gigabit download speeds and would only perceive long distance as slow and worth paying for. In which case, maybe the government should pay to maintain the backbone and perhaps satellite internet like Starlink. Which would be a good thing IMHO, because at least there would be oversight and we wouldn't have to worry about billionaire narcissists making the satellites highly reflective and visible to the naked eye, like carving a McDonalds sign on the moon.