3 ms·

Flexential is near us and other major markets. I know our local one just did their monthly UPS battery and generator test. Benefit to Flex is that they have int

by jabart 3y ago

Flexential is near us and other major markets. I know our local one just did their monthly UPS battery and generator test. Benefit to Flex is that they have interconnects to their nearby DCs and use Megaport for connecting to Cloud providers. They have a lot of various options and for us their rack pricing was pretty good.

yellow_lead 3y ago
This is the vendor that cloudflare blames for their latest outage, right? Not saying they are to blame, but it seems like there were electrical issues in the affected data center. Could be something for op to look out for.
- kosolam 3y ago
  If the price is good, having a fallback or a few is much safer than reliance on a single dc. So much so that it matters less if one of dcs burns or whatever. Cloudflare issues are theirs - they can’t blame the dc.
  - auspiv 3y ago
    You mean Cloudflare can't at all blame Flexential for: 1) not informing CF of unplanned maintenance on electrical 2) not informing CF that they had failed over to generator due to a ground fault on their PGE transformer 3) not informing CF that ALL their generators failed and were on backup batteries 4) not informing CF that power had completely failed within the DC 5) notifying CF that anything at all had happened for 48 minutes (which was after all the UPS batteries had died) Yes, this is a one sided analysis. I haven't seen the Flexential one (is there one?). https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/ https://blog.cloudflare.com/post-mortem-on-cloudflare-contro...
    - greyface- 3y ago
      It's buck-passing; the responsibility to keep Cloudflare services available to users lies with Cloudflare, not Flexential. A well-designed distributed system is able to withstand an unplanned DC failure without the user noticing. They can certainly note that Flexential's failure was a causal factor in their outage, but if they do, they're also admitting to inadequate engineering practices.
      - kosolam 3y ago
        Yep. Basically what I meant as well. Thanks for clarifying it even more.
    - crote 3y ago
      Flexential can absolutely be blamed on all of that, and if I were their customer I'd be massively pissed off too. However, CF massively failed at the whole "entire DC goes down" mitigation thing. This time it was Flexential screwing up power, but it could've also been a fire, a flood, a tornado, an earthquake... Once you get to CFs scale, you simply cannot hide behind your vendors anymore. CF has gotten big enough that we can fully expect them to just deal with it.
- jabart 3y ago
  It's why I mentioned our maintenance notifications. I know our DC has lost utility power and not been an issue. Those types of outages you need a plan for at any DC. I'm sure if someone took a dump truck to our DC it would shutdown in a different way. Our plan is using our AWS direct connect to roll everything over to AWS.

by jabart 3y ago

yellow_lead 3y ago

This is the vendor that cloudflare blames for their latest outage, right? Not saying they are to blame, but it seems like there were electrical issues in the affected data center. Could be something for op to look out for.

kosolam 3y ago
If the price is good, having a fallback or a few is much safer than reliance on a single dc. So much so that it matters less if one of dcs burns or whatever. Cloudflare issues are theirs - they can’t blame the dc.
- auspiv 3y ago
  You mean Cloudflare can't at all blame Flexential for: 1) not informing CF of unplanned maintenance on electrical 2) not informing CF that they had failed over to generator due to a ground fault on their PGE transformer 3) not informing CF that ALL their generators failed and were on backup batteries 4) not informing CF that power had completely failed within the DC 5) notifying CF that anything at all had happened for 48 minutes (which was after all the UPS batteries had died) Yes, this is a one sided analysis. I haven't seen the Flexential one (is there one?). https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/ https://blog.cloudflare.com/post-mortem-on-cloudflare-contro...
  - greyface- 3y ago
    It's buck-passing; the responsibility to keep Cloudflare services available to users lies with Cloudflare, not Flexential. A well-designed distributed system is able to withstand an unplanned DC failure without the user noticing. They can certainly note that Flexential's failure was a causal factor in their outage, but if they do, they're also admitting to inadequate engineering practices.
    - kosolam 3y ago
      Yep. Basically what I meant as well. Thanks for clarifying it even more.
  - crote 3y ago
    Flexential can absolutely be blamed on all of that, and if I were their customer I'd be massively pissed off too. However, CF massively failed at the whole "entire DC goes down" mitigation thing. This time it was Flexential screwing up power, but it could've also been a fire, a flood, a tornado, an earthquake... Once you get to CFs scale, you simply cannot hide behind your vendors anymore. CF has gotten big enough that we can fully expect them to just deal with it.
jabart 3y ago
It's why I mentioned our maintenance notifications. I know our DC has lost utility power and not been an issue. Those types of outages you need a plan for at any DC. I'm sure if someone took a dump truck to our DC it would shutdown in a different way. Our plan is using our AWS direct connect to roll everything over to AWS.