4 ms·

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Absolute lack of faith in cloudflare rn. Th

> I find that pretty embarrassing for a company like Cloudflare, which powers such relevant parts of the internet. Absolute lack of faith in cloudflare rn. This is amateur hour stuff. It's especially egregious that these are new services that were rolled out without HA.

NicoJuicy 3y ago
? Tbh. As far as I can see, their data plane worked at the edge. Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush. Their control plane was bad though. But since most products worked, that's more redundancy than most products. The proposed solution is simple: - GA requires to be in the high availability cluster - test entire DC outages
- throwaway6920 3y ago
  > Tbh. As far as I can see, their data plane worked at the edge. Arguable, it's best to think of the edge as a buffering point in addition to processing. Aggregation has to happen somewhere, and that's where shit hit the fan.
  - NicoJuicy 3y ago
    ? That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement. Cloudflare's data lives in the edge and is constantly moving. The only thing not living in the edge ( as was noticed), is stream, logpush and new image resize requests ( existing ones worked fine) from the data plane
    - throwaway6920 3y ago
      >That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement. You're being loose in your usage of 'data'. No one is talking about cached copies of an upstream, but you probably are. Read the post mortem a bit more closely. They explicitly state that the control plane(s) source of truth lives in core, and that logs aggregate back to core for analytics and service ingestion. Think through the implications on that one.
      - e1g 3y ago
        That’s my interpretation as well. There is one central brain, and “the edge” is like the nervous system that collects signals, sends it to the brain, and is _eventually consistent_ with instructions/config generated by the brain.
- yowai 3y ago
  From what I was reading on the status page & customers here on HN, WARP + Zero Trust were also majorly affected, which would be quite impactful for a company using these products for their internal authentication. It's not just streams, image upload & Logpush.
  - NicoJuicy 3y ago
    Those customers were impacted until the DC was back up ( 1-2 hours?) On the config plane. The data plane ( which I mentioned) had no issues. It's literally in the title what was affected: "Post Mortem on Cloudflare Control Plane and Analytics Outage" Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that. Source: I watched it all happen in the cloudflare discord channel. If you know anyone that is claiming to be affected on the data plane for the services you mentioned, that would be an interesting one. Note: I remember emails were also more affected though.
    - yowai 3y ago
      > Those customers were impacted until the DC was back up ( 1-2 hours?) On the config plane. Which was still like ~12+ hours, if we check the status page. >Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that. What good is a status page that's lying to you? Especially since CF manually updates it, anyway? >Source: I watched it all happen in the cloudflare discord channel. Wow, as a business customer I definitely like watching some Discord channel for status updates.
      - NicoJuicy 3y ago
        ? This wasn't about status updates going to discord only. There is literally a discussion section on the discord, named: #general-discussions Not everything was clear in the discord too ( eg. The healthchecks were discussed there), that's not something you want to copy-paste in the status updates... Priority for cloudflare seemed to get everything back up. And what they thought was down, was always mentioned in the status updates.
- jorams 3y ago
  The combination of "newer products" and then having "our Stream service" as the only named service in the post-mortem is very odd, since Stream is hardly a "newer product". It was launched in 2017 and went GA in 2018[2]. If after 5 years it still didn't have a disaster recovery procedure I find it hard to believe they even considered it. [1]: https://blog.cloudflare.com/introducing-cloudflare-stream/ https://blog.cloudflare.com/introducing-cloudflare-stream/ [2]: https://www.cloudflare.com/press-releases/2018/cloudflare-stream-now-generally-available-to-democratize-video-streaming/ https://www.cloudflare.com/press-releases/2018/cloudflare-st...

NicoJuicy 3y ago

? Tbh. As far as I can see, their data plane worked at the edge. Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush. Their control plane was bad though. But since most products worked, that's more redundancy than most products. The proposed solution is simple: - GA requires to be in the high availability cluster - test entire DC outages

throwaway6920 3y ago
> Tbh. As far as I can see, their data plane worked at the edge. Arguable, it's best to think of the edge as a buffering point in addition to processing. Aggregation has to happen somewhere, and that's where shit hit the fan.
- NicoJuicy 3y ago
  ? That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement. Cloudflare's data lives in the edge and is constantly moving. The only thing not living in the edge ( as was noticed), is stream, logpush and new image resize requests ( existing ones worked fine) from the data plane
  - throwaway6920 3y ago
    >That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement. You're being loose in your usage of 'data'. No one is talking about cached copies of an upstream, but you probably are. Read the post mortem a bit more closely. They explicitly state that the control plane(s) source of truth lives in core, and that logs aggregate back to core for analytics and service ingestion. Think through the implications on that one.
    - e1g 3y ago
      That’s my interpretation as well. There is one central brain, and “the edge” is like the nervous system that collects signals, sends it to the brain, and is _eventually consistent_ with instructions/config generated by the brain.
yowai 3y ago
From what I was reading on the status page & customers here on HN, WARP + Zero Trust were also majorly affected, which would be quite impactful for a company using these products for their internal authentication. It's not just streams, image upload & Logpush.
- NicoJuicy 3y ago
  Those customers were impacted until the DC was back up ( 1-2 hours?) On the config plane. The data plane ( which I mentioned) had no issues. It's literally in the title what was affected: "Post Mortem on Cloudflare Control Plane and Analytics Outage" Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that. Source: I watched it all happen in the cloudflare discord channel. If you know anyone that is claiming to be affected on the data plane for the services you mentioned, that would be an interesting one. Note: I remember emails were also more affected though.
  - yowai 3y ago
    > Those customers were impacted until the DC was back up ( 1-2 hours?) On the config plane. Which was still like ~12+ hours, if we check the status page. >Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that. What good is a status page that's lying to you? Especially since CF manually updates it, anyway? >Source: I watched it all happen in the cloudflare discord channel. Wow, as a business customer I definitely like watching some Discord channel for status updates.
    - NicoJuicy 3y ago
      ? This wasn't about status updates going to discord only. There is literally a discussion section on the discord, named: #general-discussions Not everything was clear in the discord too ( eg. The healthchecks were discussed there), that's not something you want to copy-paste in the status updates... Priority for cloudflare seemed to get everything back up. And what they thought was down, was always mentioned in the status updates.
jorams 3y ago
The combination of "newer products" and then having "our Stream service" as the only named service in the post-mortem is very odd, since Stream is hardly a "newer product". It was launched in 2017 and went GA in 2018[2]. If after 5 years it still didn't have a disaster recovery procedure I find it hard to believe they even considered it. [1]: https://blog.cloudflare.com/introducing-cloudflare-stream/ https://blog.cloudflare.com/introducing-cloudflare-stream/ [2]: https://www.cloudflare.com/press-releases/2018/cloudflare-stream-now-generally-available-to-democratize-video-streaming/ https://www.cloudflare.com/press-releases/2018/cloudflare-st...