6 ms·

Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to upd

Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you. https://i.imgur.com/qHBM2JW.png https://i.imgur.com/qHBM2JW.png

Heliosmaster 7y ago
Are you sure it wasn't cached? For us it was showing correctly the incident...
- wepower_ico 7y ago
  Can confirm that we had the same view. Operational while not working.
- grey-area 7y ago
  All I saw was a vague message at the top, all services were marked green and operational throughout.
- btown 7y ago
  If cloudflarestatus.com isn't serving text content as Cache-Control: no-cache, that's itself an egregious bug.
  - grenoire 7y ago
    It serves Cache-Control: max-age=0, private, must-revalidate. Same thing as far as I get it, maybe wider support.
  - buildbuildbuild 7y ago
    Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.
    - auscompgeek 7y ago
      Is it? From my end it looks like a plain ec2 instance from statuspage.io.
    - jmngomes 7y ago
      Cloudfront supports Cache-Control: no-cache
      - buildbuildbuild 7y ago
        Cloudfront itself can cache the content regardless. (if its headers are showing HIT)
- acdha 7y ago
  It might have been cached on the server but was not cached on my clients: I did a forced refresh and used a private browser window but it only showed the “almost everything is good” view throughout the duration of the incident. The actual incident page was correct.
nullify88 7y ago
We saw an update within 5 minutes of our sites going down. Are you doing caching somewhere?
- buildbuildbuild 7y ago
  Interesting, I was command+shift+R refreshing and also tried from a VPN in another region. Perhaps our CF-hosted sites returned 502s in my region sooner than yours, causing me to check the status page sooner.
- seiferteric 7y ago
  Seems unlikely, I don't use cloudflare have never visited the status page before. I noticied several 502 pages this morning and searched for "cloudflare status page" and saw the "all systems operational".
eeeeeeeeeeeee 7y ago
It was showing correctly for me the first time I checked it.
matt_oriordan 7y ago
Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service https://www.ably.io/blog/honest-status-reporting-aws-service. At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647 https://status.ably.io/incidents/647 We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.
- MrGilbert 7y ago
  slightly off-topic: Nice and clean, yet detailed enough status page. I like it. :)
  - matt_oriordan 7y ago
    Thanks :)
    - felideon 7y ago
      On the OT subject, the status logo is blurry (at least on a 4k monitor), whereas on the main website it's nice and crisp.
      - matt_oriordan 7y ago
        Ok, thanks for the heads up. Will raise a status website issue.
- joncrane 7y ago
  >How is it that hundreds of developers know there's an issue before AWS do Trust me, they know. They know about problems we never find out about, too.
kreetx 7y ago
They should derive whether something is down from how much traffic the status page gets, and have alerts tied to that as well. I think it would be pretty accurate.
gsich 7y ago
Why even have a status page if it doesn't reflect reality?
takeda 7y ago
It isn't in their best interest to have a reliable status page. They want status page to contain information about failure only after people know about issues through other means. For the same reason cloud providers don't provide rebates on outages unless you ask for it explicitly. I think the only way is to have status page operated by an independent 3rd party, but I don't think there's a viable business model for someone to provide such service. Perhaps there might be even a risk of lawsuits against you.

Heliosmaster 7y ago

Are you sure it wasn't cached? For us it was showing correctly the incident...

wepower_ico 7y ago
Can confirm that we had the same view. Operational while not working.
grey-area 7y ago
All I saw was a vague message at the top, all services were marked green and operational throughout.
btown 7y ago
If cloudflarestatus.com isn't serving text content as Cache-Control: no-cache, that's itself an egregious bug.
- grenoire 7y ago
  It serves Cache-Control: max-age=0, private, must-revalidate. Same thing as far as I get it, maybe wider support.
- buildbuildbuild 7y ago
  Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.
  - auscompgeek 7y ago
    Is it? From my end it looks like a plain ec2 instance from statuspage.io.
  - jmngomes 7y ago
    Cloudfront supports Cache-Control: no-cache
    - buildbuildbuild 7y ago
      Cloudfront itself can cache the content regardless. (if its headers are showing HIT)
acdha 7y ago
It might have been cached on the server but was not cached on my clients: I did a forced refresh and used a private browser window but it only showed the “almost everything is good” view throughout the duration of the incident. The actual incident page was correct.

nullify88 7y ago

We saw an update within 5 minutes of our sites going down. Are you doing caching somewhere?

buildbuildbuild 7y ago
Interesting, I was command+shift+R refreshing and also tried from a VPN in another region. Perhaps our CF-hosted sites returned 502s in my region sooner than yours, causing me to check the status page sooner.
seiferteric 7y ago
Seems unlikely, I don't use cloudflare have never visited the status page before. I noticied several 502 pages this morning and searched for "cloudflare status page" and saw the "all systems operational".

eeeeeeeeeeeee 7y ago

It was showing correctly for me the first time I checked it.

matt_oriordan 7y ago

Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service https://www.ably.io/blog/honest-status-reporting-aws-service. At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647 https://status.ably.io/incidents/647 We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.

MrGilbert 7y ago
slightly off-topic: Nice and clean, yet detailed enough status page. I like it. :)
- matt_oriordan 7y ago
  Thanks :)
  - felideon 7y ago
    On the OT subject, the status logo is blurry (at least on a 4k monitor), whereas on the main website it's nice and crisp.
    - matt_oriordan 7y ago
      Ok, thanks for the heads up. Will raise a status website issue.
joncrane 7y ago
>How is it that hundreds of developers know there's an issue before AWS do Trust me, they know. They know about problems we never find out about, too.

kreetx 7y ago

They should derive whether something is down from how much traffic the status page gets, and have alerts tied to that as well. I think it would be pretty accurate.

gsich 7y ago

Why even have a status page if it doesn't reflect reality?

takeda 7y ago

It isn't in their best interest to have a reliable status page. They want status page to contain information about failure only after people know about issues through other means. For the same reason cloud providers don't provide rebates on outages unless you ask for it explicitly. I think the only way is to have status page operated by an independent 3rd party, but I don't think there's a viable business model for someone to provide such service. Perhaps there might be even a risk of lawsuits against you.