10 ms·

Cloudflare Dashboard and API Service Issues

110 points by easywiththe 7y ago

verroq 7y ago
By degraded performance it means down completely.
- haik90 7y ago
  yes, completely down. I've trying for 10 minutes or so, it just loading (test from various location using VPN)
- mmm_grayons 7y ago
  Yeah, dashboard still appears to be completely down.
deleted 7y ago
[deleted]
js4ever 7y ago
It's triggering the regular cloudflare error when trying to access their dashboard. Error 522: If you're the owner of this website: Contact your hosting provider letting them know your web server is not completing requests. An Error 522 means that the request was able to connect to your web server, but that the request didn't finish. The most likely cause is that something on your server is hogging resources. I can't wait to see the postmortem, I wonder if it's a DDOS, network/hardware issue or a deployment error
- pul 7y ago
  They've added the cause on their status page now: > [...] a disruption that occurred during a maintenance. https://www.cloudflarestatus.com/incidents/g7nd3k80rxdb https://www.cloudflarestatus.com/incidents/g7nd3k80rxdb
bithavoc 7y ago
The pushed a change to fix the DNS propagation 45 minutes ago[0]. Edge servers continue to proxy but no new records are being served. [0]https://www.cloudflarestatus.com/incidents/57shkf1841kh https://www.cloudflarestatus.com/incidents/57shkf1841kh
- gazelleeatslion 7y ago
  Noticed this yesterday around 5pm EST. Edit: Also noticed that when generating API keys, the dropdown wouldn't list all my accounts for setting permissions. Just assumed it was all related or something. Either way, overall super insanely reliable product/service and could not live without.
arn 7y ago
Can't clear our cache via api or manually, so our cached HTML pages are stuck until the natural expire happens -- which are set somewhat long. Not great for a blog / news site. for example, if we publish a story, our front page won't reflect it.
partiallypro 7y ago
While they are fixing this, could they please roll out a feature to allow me to assign users to only specific domains? Biggest complaint about Cloudflare, heck even GoDaddy lets you do that at no cost.
- nikisweeting 7y ago
  https://www.cloudflare.com/en-ca/products/cloudflare-access/ https://www.cloudflare.com/en-ca/products/cloudflare-access/ https://developers.cloudflare.com/access/setting-up-access/ https://developers.cloudflare.com/access/setting-up-access/
  - partiallypro 7y ago
    This is not what I'm talking about, I'm talking about the ability to edit a zone.
- judge2020 7y ago
  You can, but only once you upgrade to Enterprise - the delegated dashboard per-zone access functionality is part of their business model. See: https://www.sec.gov/Archives/edgar/data/1477333/000119312519222176/g735023g16s02.jpg https://www.sec.gov/Archives/edgar/data/1477333/000119312519... (RBAC)
  - partiallypro 7y ago
    I said at no cost. Other registrars/DNS providers provide this for free.
robinhouston 7y ago
This has broken publishing in our app, which purges the file from the Cloudflare cache when something is republished. We’re ignoring errors from the Cloudflare API, but that isn’t enough in this case because it isn’t returning an error – it’s just hanging till the request times out. We’re pushing an emergency config change to skip the cache invalidation, which will stop it timing out but means republished projects won’t update (because the old version will still be cached). Godspeed to the Cloudflare engineers who are presumably scrambling to fix this!
- darkerside 7y ago
  Do you not have some kind of task runner you can offload this to? That seems like it would be of general benefit.
  - robinhouston 7y ago
    The trouble with returning to the user before the cache has been purged is that they think it hasn’t worked, because they still see the old cached version.
    - darkerside 7y ago
      I'm probably missing something but it sounds like cache invalidation happens on republishing, not on user request for content.
      - robinhouston 7y ago
        You’re absolutely right. But user requests for content are made directly to Cloudflare, not through the app, and it’s common for users to click directly through to the published content after republishing.
        darkerside 7y ago
        Ah, that makes sense. You could store latest task status per content in your database and have a JavaScript component that polls and shows a message if cache invalidation is incomplete. But for how often Cloudflare goes down, it's probably not worth it.
mattashii 7y ago
> "Cloudflare is continuing to investigate a network connectivity issue in the data center which serves API and dashboard functions." This implies that CF hosts its API and dashboard all in one DC, which I find an _interesting_ observation. One would expect a company like CF to host its critical infrastructure in a redundant fashion.
- tyingq 7y ago
  It's certainly not ideal. But it's not unusual to spend a lot more time on making the runtime very redundant, but less time/money on dashboards and configuration change underpinnings. Doesn't work well in this case since it kills invalidating cache items for customers. The comments seem to imply that having a redundant way to refresh the page cache, even if it were global/domain versus page, would be an okay backup for many.
  - mattashii 7y ago
    I agree that first priority would be data integrety (which would be the runtime). But a large part of the CF experience of a CF customer would be the availability of their management APIs/dashboards, and that would be another part to optimize for. I'm really suprised that they hosted all those non-vital but still quite critical services in just one DC, or somehow had one DC as a single point of failure. Network issues happen "regular enough" to want to protect against that, or at least have mitigations available.
    - bostik 7y ago
      To be fair, you have these latent single points of failures even in the most resilient distributed systems. Such as S3. The bucket names are globally unique, which means that their source of truth is in a single location. (Virginia, IIRC.) Now... a small thought exercise. If I wanted to take down a Cloudflare datacenter and I had access to a few suitably careless remote hands, I'd take out the power supplies to the core routers, and while the external network is out of commission, power down the racks where they have their PXE servers. That should keep anything, within the DC, from being unable to recover on its own.
nikisweeting 7y ago
All our sites using Argo Tunnels go down whenever the API goes down, but the dashboard still claims Argo Tunnels are unaffected :/ At least we'll get a good blog post out of it in a few days.
- jbergstroem 7y ago
  Yes, same here -- which equals site down. Nothing on status page yet. > Apr 15 17:01:37 mysite cloudflared[12412]: time="2020-04-15T17:01:37Z" level=info msg="Connected to SJC" connectionID=0 > Apr 15 17:03:09 mysite cloudflared[12412]: time="2020-04-15T17:03:09Z" level=error msg="Register tunnel error from server side" connectionID=0 error="Server error: Reached maximum retry 5: dial tcp 198.41.248.96:9100: connect: connection timed out" > Apr 15 17:03:09 mysite cloudflared[12412]: time="2020-04-15T17:03:09Z" level=info msg="Retrying in 8s seconds" connectionID=0
  - nikisweeting 7y ago
    Here's our logs in case it helps anyone: > hera ra | time="2020-04-15T17:08:09Z" level=error msg="Unable to dial edge" error="DialContext error: dial tcp 198.41.200.233:7844: i/o timeout" > hera | time="2020-04-15T17:08:09Z" level=info msg="Retrying in 1s seconds" hera | time="2020-04-15T17:08:09Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 172.18.0.2:36016->198.41.192.227:7844: i/o timeout" > hera | time="2020-04-15T17:08:09Z" level=info msg="Retrying in 1s seconds" hera | time="2020-04-15T17:08:11Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 172.18.0.2:44626->198.41.192.7:7844: i/o timeout" > hera | time="2020-04-15T17:08:11Z" level=info msg="Retrying in 4s seconds" > hera | time="2020-04-15T17:08:13Z" level=info msg="Connected to EWR" > hera | time="2020-04-15T17:08:13Z" level=error msg="Register tunnel error from server side" connectionID=0 error="Server error: Reached maximum retry 5: dial tcp 198.41.248.96:9100: connect: connection timed out" > hera | time="2020-04-15T17:08:13Z" level=warning msg="Tunnel disconnected due to error" error="Server error: Reached maximum retry 5: dial tcp 198.41.248.96:9100: connect: connection timed out" > hera | time="2020-04-15T17:08:19Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 172.18.0.2:51974->198.41.192.107:7844: i/o timeout"
gramakri 7y ago
All DNS APIs are failing :(
- zymhan 7y ago
  As are our CircleCI piplines that call out to Cloudflare. Time to get some fresh air!
rkwasny 7y ago
Update from CEO https://twitter.com/eastdakota/status/1250479501726760960 https://twitter.com/eastdakota/status/1250479501726760960
- zymhan 7y ago
  How on earth can rented "remote hands" at a datacenter take down the ENTIRE Cloudflare management layer?
  - MichaelApproved 7y ago
    and for over 3 hours? What could they do accidentally that would cause an outage for this long? The tweet says they're failing over to their backup facility. I would've expected that fail over to happen much faster. Seems like they have two issues going on. First, the remote hands could take down the datacenter. Second, their fail over is taking this long to come online. I also wonder how much Covid impacted the process, if at all. I'm looking forward to the details after they get things back online. Good luck CF engineers!
    - foobarbazetc 7y ago
      Probably nuked the main DB.
    - eb0la 7y ago
      Three hours for a physical problem like this is very little time. Once you discover your equipment is not there, you still have to put it on its place, power up, reconnect al cabling, and check networking. From this kind of mess to a fully functional infrastructure you need at least 12h-16h to function minimally. Probably takes 2-3 days to have that node working as before. When I was on call, I always jocked with my colleages the worst incident you could have in a datatenter was someone swapping two cables.
  - mcpherrinm 7y ago
    I don't know how big Cloudflare's management layer is, but I'm assuming it's relatively small (maybe dozens of racks at most), hosted in some sort of datacenter (Equinix, Digital Realty, Coresite, etc) that provides the remote hands. Maybe some piece of core network equipment. A small colo may have a single pair of routers, switches, or firewalls at its edge. If one had failed for some reason, and the remote hands removed the wrong one, it is possible you could knock the entire colo offline. There's a bunch of other possible components: Storage platforms, power, maybe something like an HSM storing secrets, or even just a key database server. Their failover to their backup facility may be impaired by the fact that well, their management plane is down. They probably rely on their own services. Avoiding chicken-and-egg issues can require careful ahead-of-time planning.
capableweb 7y ago
Wonder if this is somehow related to the "The Devastating Decline of a Brilliant Young Coder" article that was published today https://www.wired.com/story/lee-holloway-devastating-decline-brilliant-young-coder/ https://www.wired.com/story/lee-holloway-devastating-decline... (discussed here: https://news.ycombinator.com/item?id=22878136 https://news.ycombinator.com/item?id=22878136) One can wonder what kind of access Lee might still have in Cloudflare infrastructure. CEO says "Not attack related" but weird coincidence it happen on the very same day.
- jasongill 7y ago
  Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity"
- searchableguy 7y ago
  I am worried about you.
redm 7y ago
I wish cloudflarestatus.com (powered by StatusPage) offered a subscription (like https://status.box.com/ https://status.box.com/, also on StatusPage) so you could get a pro-active notice about outages. I had to debug customer issues to find out that this was down. Even if they don't want to offer this to the general public (free customers), they should have another notice mechanism for enterprise customers.
- pmccarren 7y ago
  StatusPage has native Subscriber Notifications [0]. Cloudflare must not have it setup on their end. [0] https://status.io/features https://status.io/features
  - redm 7y ago
    I know, that's my point. My guess is they have not enabled it because of the cost.
  - burlesona 7y ago
    Wrong URL, Statuspage is at https://statuspage.io https://statuspage.io :)
- dijital 7y ago
  You could hook something up to their Status Page's RSS feed. We have a #provider-updates Slack channel for things like this. https://www.cloudflarestatus.com/history.atom https://www.cloudflarestatus.com/history.atom
  - htilford 7y ago
    https://www.cloudflarestatus.com/history.rss https://www.cloudflarestatus.com/history.rss and https://www.cloudflarestatus.com/history.json https://www.cloudflarestatus.com/history.json work as well.
- htilford 7y ago
  It's pull rather than push, but you can use the public API to get notified of incidents: https://www.cloudflarestatus.com/api/v2 https://www.cloudflarestatus.com/api/v2
jgrahamc 7y ago
Coming back now.
iampims 7y ago
Postmortem thread: https://news.ycombinator.com/item?id=22884833 https://news.ycombinator.com/item?id=22884833

110 points by easywiththe 7y ago

verroq 7y ago

By degraded performance it means down completely.

haik90 7y ago
yes, completely down. I've trying for 10 minutes or so, it just loading (test from various location using VPN)
mmm_grayons 7y ago
Yeah, dashboard still appears to be completely down.

deleted 7y ago

[deleted]

js4ever 7y ago

It's triggering the regular cloudflare error when trying to access their dashboard. Error 522: If you're the owner of this website: Contact your hosting provider letting them know your web server is not completing requests. An Error 522 means that the request was able to connect to your web server, but that the request didn't finish. The most likely cause is that something on your server is hogging resources. I can't wait to see the postmortem, I wonder if it's a DDOS, network/hardware issue or a deployment error

pul 7y ago
They've added the cause on their status page now: > [...] a disruption that occurred during a maintenance. https://www.cloudflarestatus.com/incidents/g7nd3k80rxdb https://www.cloudflarestatus.com/incidents/g7nd3k80rxdb

bithavoc 7y ago

The pushed a change to fix the DNS propagation 45 minutes ago[0]. Edge servers continue to proxy but no new records are being served. [0]https://www.cloudflarestatus.com/incidents/57shkf1841kh https://www.cloudflarestatus.com/incidents/57shkf1841kh

gazelleeatslion 7y ago
Noticed this yesterday around 5pm EST. Edit: Also noticed that when generating API keys, the dropdown wouldn't list all my accounts for setting permissions. Just assumed it was all related or something. Either way, overall super insanely reliable product/service and could not live without.

arn 7y ago

Can't clear our cache via api or manually, so our cached HTML pages are stuck until the natural expire happens -- which are set somewhat long. Not great for a blog / news site. for example, if we publish a story, our front page won't reflect it.

partiallypro 7y ago

While they are fixing this, could they please roll out a feature to allow me to assign users to only specific domains? Biggest complaint about Cloudflare, heck even GoDaddy lets you do that at no cost.

nikisweeting 7y ago
https://www.cloudflare.com/en-ca/products/cloudflare-access/ https://www.cloudflare.com/en-ca/products/cloudflare-access/ https://developers.cloudflare.com/access/setting-up-access/ https://developers.cloudflare.com/access/setting-up-access/
- partiallypro 7y ago
  This is not what I'm talking about, I'm talking about the ability to edit a zone.
judge2020 7y ago
You can, but only once you upgrade to Enterprise - the delegated dashboard per-zone access functionality is part of their business model. See: https://www.sec.gov/Archives/edgar/data/1477333/000119312519222176/g735023g16s02.jpg https://www.sec.gov/Archives/edgar/data/1477333/000119312519... (RBAC)
- partiallypro 7y ago
  I said at no cost. Other registrars/DNS providers provide this for free.

robinhouston 7y ago

This has broken publishing in our app, which purges the file from the Cloudflare cache when something is republished. We’re ignoring errors from the Cloudflare API, but that isn’t enough in this case because it isn’t returning an error – it’s just hanging till the request times out. We’re pushing an emergency config change to skip the cache invalidation, which will stop it timing out but means republished projects won’t update (because the old version will still be cached). Godspeed to the Cloudflare engineers who are presumably scrambling to fix this!

darkerside 7y ago
Do you not have some kind of task runner you can offload this to? That seems like it would be of general benefit.
- robinhouston 7y ago
  The trouble with returning to the user before the cache has been purged is that they think it hasn’t worked, because they still see the old cached version.
  - darkerside 7y ago
    I'm probably missing something but it sounds like cache invalidation happens on republishing, not on user request for content.
    - robinhouston 7y ago
      You’re absolutely right. But user requests for content are made directly to Cloudflare, not through the app, and it’s common for users to click directly through to the published content after republishing.
      - darkerside 7y ago
        Ah, that makes sense. You could store latest task status per content in your database and have a JavaScript component that polls and shows a message if cache invalidation is incomplete. But for how often Cloudflare goes down, it's probably not worth it.

mattashii 7y ago

> "Cloudflare is continuing to investigate a network connectivity issue in the data center which serves API and dashboard functions." This implies that CF hosts its API and dashboard all in one DC, which I find an _interesting_ observation. One would expect a company like CF to host its critical infrastructure in a redundant fashion.

tyingq 7y ago
It's certainly not ideal. But it's not unusual to spend a lot more time on making the runtime very redundant, but less time/money on dashboards and configuration change underpinnings. Doesn't work well in this case since it kills invalidating cache items for customers. The comments seem to imply that having a redundant way to refresh the page cache, even if it were global/domain versus page, would be an okay backup for many.
- mattashii 7y ago
  I agree that first priority would be data integrety (which would be the runtime). But a large part of the CF experience of a CF customer would be the availability of their management APIs/dashboards, and that would be another part to optimize for. I'm really suprised that they hosted all those non-vital but still quite critical services in just one DC, or somehow had one DC as a single point of failure. Network issues happen "regular enough" to want to protect against that, or at least have mitigations available.
  - bostik 7y ago
    To be fair, you have these latent single points of failures even in the most resilient distributed systems. Such as S3. The bucket names are globally unique, which means that their source of truth is in a single location. (Virginia, IIRC.) Now... a small thought exercise. If I wanted to take down a Cloudflare datacenter and I had access to a few suitably careless remote hands, I'd take out the power supplies to the core routers, and while the external network is out of commission, power down the racks where they have their PXE servers. That should keep anything, within the DC, from being unable to recover on its own.

nikisweeting 7y ago

All our sites using Argo Tunnels go down whenever the API goes down, but the dashboard still claims Argo Tunnels are unaffected :/ At least we'll get a good blog post out of it in a few days.

jbergstroem 7y ago
Yes, same here -- which equals site down. Nothing on status page yet. > Apr 15 17:01:37 mysite cloudflared[12412]: time="2020-04-15T17:01:37Z" level=info msg="Connected to SJC" connectionID=0 > Apr 15 17:03:09 mysite cloudflared[12412]: time="2020-04-15T17:03:09Z" level=error msg="Register tunnel error from server side" connectionID=0 error="Server error: Reached maximum retry 5: dial tcp 198.41.248.96:9100: connect: connection timed out" > Apr 15 17:03:09 mysite cloudflared[12412]: time="2020-04-15T17:03:09Z" level=info msg="Retrying in 8s seconds" connectionID=0
- nikisweeting 7y ago
  Here's our logs in case it helps anyone: > hera ra | time="2020-04-15T17:08:09Z" level=error msg="Unable to dial edge" error="DialContext error: dial tcp 198.41.200.233:7844: i/o timeout" > hera | time="2020-04-15T17:08:09Z" level=info msg="Retrying in 1s seconds" hera | time="2020-04-15T17:08:09Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 172.18.0.2:36016->198.41.192.227:7844: i/o timeout" > hera | time="2020-04-15T17:08:09Z" level=info msg="Retrying in 1s seconds" hera | time="2020-04-15T17:08:11Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 172.18.0.2:44626->198.41.192.7:7844: i/o timeout" > hera | time="2020-04-15T17:08:11Z" level=info msg="Retrying in 4s seconds" > hera | time="2020-04-15T17:08:13Z" level=info msg="Connected to EWR" > hera | time="2020-04-15T17:08:13Z" level=error msg="Register tunnel error from server side" connectionID=0 error="Server error: Reached maximum retry 5: dial tcp 198.41.248.96:9100: connect: connection timed out" > hera | time="2020-04-15T17:08:13Z" level=warning msg="Tunnel disconnected due to error" error="Server error: Reached maximum retry 5: dial tcp 198.41.248.96:9100: connect: connection timed out" > hera | time="2020-04-15T17:08:19Z" level=error msg="Unable to dial edge" error="Handshake with edge error: read tcp 172.18.0.2:51974->198.41.192.107:7844: i/o timeout"

gramakri 7y ago

All DNS APIs are failing :(

zymhan 7y ago
As are our CircleCI piplines that call out to Cloudflare. Time to get some fresh air!

rkwasny 7y ago

Update from CEO https://twitter.com/eastdakota/status/1250479501726760960 https://twitter.com/eastdakota/status/1250479501726760960

zymhan 7y ago
How on earth can rented "remote hands" at a datacenter take down the ENTIRE Cloudflare management layer?
- MichaelApproved 7y ago
  and for over 3 hours? What could they do accidentally that would cause an outage for this long? The tweet says they're failing over to their backup facility. I would've expected that fail over to happen much faster. Seems like they have two issues going on. First, the remote hands could take down the datacenter. Second, their fail over is taking this long to come online. I also wonder how much Covid impacted the process, if at all. I'm looking forward to the details after they get things back online. Good luck CF engineers!
  - foobarbazetc 7y ago
    Probably nuked the main DB.
  - eb0la 7y ago
    Three hours for a physical problem like this is very little time. Once you discover your equipment is not there, you still have to put it on its place, power up, reconnect al cabling, and check networking. From this kind of mess to a fully functional infrastructure you need at least 12h-16h to function minimally. Probably takes 2-3 days to have that node working as before. When I was on call, I always jocked with my colleages the worst incident you could have in a datatenter was someone swapping two cables.
- mcpherrinm 7y ago
  I don't know how big Cloudflare's management layer is, but I'm assuming it's relatively small (maybe dozens of racks at most), hosted in some sort of datacenter (Equinix, Digital Realty, Coresite, etc) that provides the remote hands. Maybe some piece of core network equipment. A small colo may have a single pair of routers, switches, or firewalls at its edge. If one had failed for some reason, and the remote hands removed the wrong one, it is possible you could knock the entire colo offline. There's a bunch of other possible components: Storage platforms, power, maybe something like an HSM storing secrets, or even just a key database server. Their failover to their backup facility may be impaired by the fact that well, their management plane is down. They probably rely on their own services. Avoiding chicken-and-egg issues can require careful ahead-of-time planning.

capableweb 7y ago

Wonder if this is somehow related to the "The Devastating Decline of a Brilliant Young Coder" article that was published today https://www.wired.com/story/lee-holloway-devastating-decline-brilliant-young-coder/ https://www.wired.com/story/lee-holloway-devastating-decline... (discussed here: https://news.ycombinator.com/item?id=22878136 https://news.ycombinator.com/item?id=22878136) One can wonder what kind of access Lee might still have in Cloudflare infrastructure. CEO says "Not attack related" but weird coincidence it happen on the very same day.

jasongill 7y ago
Hanlon's Razor: "Never attribute to malice that which is adequately explained by stupidity"
searchableguy 7y ago
I am worried about you.

redm 7y ago

I wish cloudflarestatus.com (powered by StatusPage) offered a subscription (like https://status.box.com/ https://status.box.com/, also on StatusPage) so you could get a pro-active notice about outages. I had to debug customer issues to find out that this was down. Even if they don't want to offer this to the general public (free customers), they should have another notice mechanism for enterprise customers.

pmccarren 7y ago
StatusPage has native Subscriber Notifications [0]. Cloudflare must not have it setup on their end. [0] https://status.io/features https://status.io/features
- redm 7y ago
  I know, that's my point. My guess is they have not enabled it because of the cost.
- burlesona 7y ago
  Wrong URL, Statuspage is at https://statuspage.io https://statuspage.io :)
dijital 7y ago
You could hook something up to their Status Page's RSS feed. We have a #provider-updates Slack channel for things like this. https://www.cloudflarestatus.com/history.atom https://www.cloudflarestatus.com/history.atom
- htilford 7y ago
  https://www.cloudflarestatus.com/history.rss https://www.cloudflarestatus.com/history.rss and https://www.cloudflarestatus.com/history.json https://www.cloudflarestatus.com/history.json work as well.
htilford 7y ago
It's pull rather than push, but you can use the public API to get notified of incidents: https://www.cloudflarestatus.com/api/v2 https://www.cloudflarestatus.com/api/v2

jgrahamc 7y ago

Coming back now.

iampims 7y ago

Postmortem thread: https://news.ycombinator.com/item?id=22884833 https://news.ycombinator.com/item?id=22884833