4 ms·

It is unfair to blame Cloudflare (or AWS, or Azure, or GitHub) for what’s happening, and I say that as one of the biggest “yellers at the cloud” on here. Ultim

by Nextgrid 11mo ago

It is unfair to blame Cloudflare (or AWS, or Azure, or GitHub) for what’s happening, and I say that as one of the biggest “yellers at the cloud” on here. Ultimately end-users don’t have a relationship with any of those companies. They have relationships with businesses that chose to rely on them. Cloudflare, etc publish SLAs and compensation schedules in case those SLAs are missed. Businesses chose to accept those SLAs and take on that risk. If Cloudflare/etc signed a contract promising a certain SLA (with penalties) and then chose to not pay out those penalties then there would be reasons to ask questions, but nothing suggests they’re not holding up their side of the deal - you will absolutely get compensated (in the form of a refund on your bill) in case of an outage. The issue is that businesses accept this deal and then scream when it goes wrong, yet are unwilling to pay for a solution that does not fail in this way. Those solutions exist - you absolutely can build systems that are reliable and/or fail in a predictable and testable manner; it’s simply more expensive and requires more skill than just slapping a few SaaSes and CNCF projects together. But it is possible - look at the uptime of card networks, stock exchanges, or airplane avionics. It’s just more expensive and the truth is that businesses don’t want to pay for it (and neither are their end-customers - they will bitch about outages, but will immediately run the other way if you ask them to pony up for a more reliable system - and the ones that don’t, already run such a system and were unaffected by the recent outages).

psim1 11mo ago
> It is unfair to blame Cloudflare (or AWS, or Azure, or GitHub) for what’s happening > Ultimately end-users don’t have a relationship with any of those companies. They have relationships with businesses that chose to rely on them Could you not say this about any supplier relationship? No, in this case, we all know the root of the outage is CloudFlare, so it absolutely makes sense to blame CloudFlare, and not their customers.
- Nextgrid 11mo ago
  Devil’s advocate: I operate the equivalent of an online lemonade stand, some shitty service at a cheap price offered with little guarantees (“if I fuck up I’ll refund you the price of your ‘lemonade’”) for hobbyists to use to host their blog and Visa decides to use it in their critical path. Then this “lemonade stand” goes down. Do you think it’s fair to blame me? I never chose to be part of Visa’s authorization loop, and after all is done I did indeed refund them the price of their “lemonade”. It’s Visa’s fault they introduced a single point of failure with inadequate compensation schedules in their critical path.
  - stronglikedan 11mo ago
    > Do you think it’s fair to blame me? Absolutely, yes. Where's your backup plan for when Visa doesn't behave as you expect? It's okay to not have one, but it's also your fault for not having one, and that is the sole reason that the lemonade stand went down.
    - Nextgrid 11mo ago
      > Where's your backup plan for when Visa doesn't behave as you expect? I don’t have (nor have to have) such a plan, I offer X service with Y guarantees paying out Z dollars if I don’t hold up my part of the bargain. In this hypothetical situation if Visa signs up I assumed they wanted to host their marketing website or some low-hanging fruit, it’s not my job to check what they’re using it for (in fact it would be preferable for me not to check, as I’d be seeing unencrypted card numbers and PII otherwise).
      - mschulkind 11mo ago
        The person above who replied to you thinks you're talking about a proverbial lemonade stand taking payments via Visa. That's the misunderstanding. That aside, I think the example is good. It's a bit like priority inversion in scheduling. With no agreement from the lemonade seller they've suddenly changed greatly in terms of their criticality to some value creation chain.
- stronglikedan 11mo ago
  If I'm paying a company that chose Cloudflare, and my SLA with that company entitles me to some sort of compensation for outages, then I expect that company to compensate me regardless of whose fault it is, and regardless of whether they were compensated by Cloudflare. I can know that the cause of the outage is Cloudflare, but also know that the company that I'm paying should have had a backup plan and not be solely reliable on one vendor. In other words, I care about who I pay, not who they decide to use.
- wongarsu 11mo ago
  Don't we say that about all supplier relationships? If my Samsung washing machine stops working I blame Samsung. Even when it turns out that it was a broken drive belt I don't blame the manufacturer of the drive belt, or whoever produced the rubber that went into the drive belt, or whoever made the machine involved in the production of this batch of rubber. Samsung choose to put the drive belt in my washing machine, that's where the buck stops. They are free to litigate the matter internally, but I only care about Samsung selling me a washing machine that's now broken Same with cloudflare. If you run your site on cloudflare you are responsible for any downtime caused to your site by cloudflare What we can blame cloudflare for is having so many customers that a cloudflare outage has outsized impact compared to the more uncorrelated outages we would have if sites were distributed among many smaller providers. But that's not quite the same as blaming any individual site being down on cloudflare
  - raincole 11mo ago
    > Don't we say that about all supplier relationships? No always. If the farm sells packs of poisoned bacon to the supermarket, we blame the farm. It's more about if the website/supermarket can reasonably do the QA.
mschuster91 11mo ago
> look at the uptime of card networks, stock exchanges, or airplane avionics. In fact, I'd say... airplane avionics are not what you should be looking at. Boeing's 787? Reboot every 51 days or risk the pilots getting wrong airspeed indicators! No, I'm not joking [1], and it's not the first time either [2], and it's not just Boeing [3]. [1] https://www.theregister.com/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/2020/04/02/boeing_787_power_cycl... [2] https://www.theregister.com/2015/05/01/787_software_bug_can_shut_down_planes_generators/ https://www.theregister.com/2015/05/01/787_software_bug_can_... [3] https://www.theregister.com/2019/07/25/a350_power_cycle_software_bug_149_hours/ https://www.theregister.com/2019/07/25/a350_power_cycle_soft...
- Nextgrid 11mo ago
  > Reboot every 51 days or risk the pilots getting wrong airspeed indicators If this is documented then fair enough - airlines don’t have to buy airplanes that need rebooting every 51 days, they can vote with their wallets and Boeing is welcome to fix it. If not documented, I hope regulators enforced penalties high enough to force Boeing to get their stuff together. Either way, the uptime of avionics (and redundancies - including the unreliable airspeed checklists) are much higher than anything conventional software “engineering” has been putting out the past decade.
  - mschuster91 11mo ago
    Personally, I'm of the opinion that either of these three defects should not ever have been possible in the first place, that should have been prevented by formal verification. If some dingus widget provider has a downtime, no one dies. If an airplane loses power mid flight due to a software error... that's not guaranteed any more.
    - cadamsdotcom 11mo ago
      The requirement to reboot is documented, and it’s an order of magnitude away from being a regular problem - ie. 51 days instead of every 51 minutes. This puts it in the same maintenance bucket as parts that need regular replacement. To use the “part needing regular replacement” analogy - the bug could be fixed for an extra $Y. But everything in engineering is a trade off. Choosing not to fix the bug might bring costs down to something competitive with the market. Plus, like any part that could be made more reliable for extra cost, there might be no benefit due to service schedules. Say you could design a part that is 60% worn at service time vs a cheaper part that’s 90% worn - it they will both need replacing at the same time, you’d go for the lower cost design. Eg. if some service task requires resetting the avionics every 40 days, that’s less than 51 days so there may as well be no bug.

psim1 11mo ago

> It is unfair to blame Cloudflare (or AWS, or Azure, or GitHub) for what’s happening > Ultimately end-users don’t have a relationship with any of those companies. They have relationships with businesses that chose to rely on them Could you not say this about any supplier relationship? No, in this case, we all know the root of the outage is CloudFlare, so it absolutely makes sense to blame CloudFlare, and not their customers.

Nextgrid 11mo ago
Devil’s advocate: I operate the equivalent of an online lemonade stand, some shitty service at a cheap price offered with little guarantees (“if I fuck up I’ll refund you the price of your ‘lemonade’”) for hobbyists to use to host their blog and Visa decides to use it in their critical path. Then this “lemonade stand” goes down. Do you think it’s fair to blame me? I never chose to be part of Visa’s authorization loop, and after all is done I did indeed refund them the price of their “lemonade”. It’s Visa’s fault they introduced a single point of failure with inadequate compensation schedules in their critical path.
- stronglikedan 11mo ago
  > Do you think it’s fair to blame me? Absolutely, yes. Where's your backup plan for when Visa doesn't behave as you expect? It's okay to not have one, but it's also your fault for not having one, and that is the sole reason that the lemonade stand went down.
  - Nextgrid 11mo ago
    > Where's your backup plan for when Visa doesn't behave as you expect? I don’t have (nor have to have) such a plan, I offer X service with Y guarantees paying out Z dollars if I don’t hold up my part of the bargain. In this hypothetical situation if Visa signs up I assumed they wanted to host their marketing website or some low-hanging fruit, it’s not my job to check what they’re using it for (in fact it would be preferable for me not to check, as I’d be seeing unencrypted card numbers and PII otherwise).
    - mschulkind 11mo ago
      The person above who replied to you thinks you're talking about a proverbial lemonade stand taking payments via Visa. That's the misunderstanding. That aside, I think the example is good. It's a bit like priority inversion in scheduling. With no agreement from the lemonade seller they've suddenly changed greatly in terms of their criticality to some value creation chain.
stronglikedan 11mo ago
If I'm paying a company that chose Cloudflare, and my SLA with that company entitles me to some sort of compensation for outages, then I expect that company to compensate me regardless of whose fault it is, and regardless of whether they were compensated by Cloudflare. I can know that the cause of the outage is Cloudflare, but also know that the company that I'm paying should have had a backup plan and not be solely reliable on one vendor. In other words, I care about who I pay, not who they decide to use.
wongarsu 11mo ago
Don't we say that about all supplier relationships? If my Samsung washing machine stops working I blame Samsung. Even when it turns out that it was a broken drive belt I don't blame the manufacturer of the drive belt, or whoever produced the rubber that went into the drive belt, or whoever made the machine involved in the production of this batch of rubber. Samsung choose to put the drive belt in my washing machine, that's where the buck stops. They are free to litigate the matter internally, but I only care about Samsung selling me a washing machine that's now broken Same with cloudflare. If you run your site on cloudflare you are responsible for any downtime caused to your site by cloudflare What we can blame cloudflare for is having so many customers that a cloudflare outage has outsized impact compared to the more uncorrelated outages we would have if sites were distributed among many smaller providers. But that's not quite the same as blaming any individual site being down on cloudflare
- raincole 11mo ago
  > Don't we say that about all supplier relationships? No always. If the farm sells packs of poisoned bacon to the supermarket, we blame the farm. It's more about if the website/supermarket can reasonably do the QA.

mschuster91 11mo ago

> look at the uptime of card networks, stock exchanges, or airplane avionics. In fact, I'd say... airplane avionics are not what you should be looking at. Boeing's 787? Reboot every 51 days or risk the pilots getting wrong airspeed indicators! No, I'm not joking [1], and it's not the first time either [2], and it's not just Boeing [3]. [1] https://www.theregister.com/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ https://www.theregister.com/2020/04/02/boeing_787_power_cycl... [2] https://www.theregister.com/2015/05/01/787_software_bug_can_shut_down_planes_generators/ https://www.theregister.com/2015/05/01/787_software_bug_can_... [3] https://www.theregister.com/2019/07/25/a350_power_cycle_software_bug_149_hours/ https://www.theregister.com/2019/07/25/a350_power_cycle_soft...

Nextgrid 11mo ago
> Reboot every 51 days or risk the pilots getting wrong airspeed indicators If this is documented then fair enough - airlines don’t have to buy airplanes that need rebooting every 51 days, they can vote with their wallets and Boeing is welcome to fix it. If not documented, I hope regulators enforced penalties high enough to force Boeing to get their stuff together. Either way, the uptime of avionics (and redundancies - including the unreliable airspeed checklists) are much higher than anything conventional software “engineering” has been putting out the past decade.
- mschuster91 11mo ago
  Personally, I'm of the opinion that either of these three defects should not ever have been possible in the first place, that should have been prevented by formal verification. If some dingus widget provider has a downtime, no one dies. If an airplane loses power mid flight due to a software error... that's not guaranteed any more.
  - cadamsdotcom 11mo ago
    The requirement to reboot is documented, and it’s an order of magnitude away from being a regular problem - ie. 51 days instead of every 51 minutes. This puts it in the same maintenance bucket as parts that need regular replacement. To use the “part needing regular replacement” analogy - the bug could be fixed for an extra $Y. But everything in engineering is a trade off. Choosing not to fix the bug might bring costs down to something competitive with the market. Plus, like any part that could be made more reliable for extra cost, there might be no benefit due to service schedules. Say you could design a part that is 60% worn at service time vs a cheaper part that’s 90% worn - it they will both need replacing at the same time, you’d go for the lower cost design. Eg. if some service task requires resetting the avionics every 40 days, that’s less than 51 days so there may as well be no bug.