5 ms·

I can't say for your employeer, but at AWS this is not our intent. Our job is to help customers build better architectures, and if you read the framework you wi

I can't say for your employeer, but at AWS this is not our intent. Our job is to help customers build better architectures, and if you read the framework you will see its agnostic of vendor. You can't approach reliability with the "the sky is falling down, so theres no point" approach. You are actually going to have think about component failure, blast radius and what happens afterwards a failure. rather than spreading FUD, why not help customers by showing them the better math?

klodolph 7y ago
> I can't say for your employeer, but at AWS this is not our intent. Lost the antecedent, here. What is not your intent? Granted my understanding of cloud economics is somewhat murky. My general impression is that the big selling point of cloud is the move from capex to opex, and the second selling point is that you don’t need the same level of operations expertise to run cloud compared to on-prem. My hot take is that for high reliability you still need tons of operations expertise, and in these scenarios, solutions like (partial) on-prem and multi-provider become much more favorable. > rather than spreading FUD, why not help customers by showing them the better math? I feel like I’m being accused of spreading FUD, and I want to know why? All I am really trying to say here is that you can’t just do simple arithmetic on published #nines and end up with something that approximates the truth for your service in a useful way. Depending on the operation of your business this approximation may be acceptable or it may not be. Just to recap, the bad math is to come up with some threshold for acceptable performance in all components, model each as an independent Bernoulli variable, and then plug them into some boolean formula. This is the math published in the guide here, and you can do it with arithmetic on #nines. The reason why this is bad is because this leads you towards a shallow understanding of your system and creates pressure to inflate estimates of availability beyond actual availability, sometimes much so. Unfortunately, if you are really interested in calculating availability you need to come up with a model for your particular system. This is a complex subject that involves coming up with a model which strikes some balance between accuracy, simplicity (so it can be understood and used to inform strategy), usefulness (to customers / downstream), and supportability (to engineers working on the service). I can't tell you how to do that, my best guess is to hire someone who knows enough statistics to be dangerous and lock them in a room for three months with a terminal and access to your metrics. (As a side note, the tools for this kind of analysis are much better for systems like electronic circuits. For example, a part might be labeled as "1% tolerance" but when you are running simulations you use a probability distribution.) When you actually do this, you often find out that your system is much less available (reliable, durable) than it was designed to be. It’s common that your cloud provider could stay within SLO and your service would still be “down” as far as your definition went. So then you have the engineering problem of figuring out how to improve things, which may involve multi-region, multi-cloud, on-prem, redesigning parts of your system, changing utilization targets, etc. The model helps because it can reveal key insights like “if you improve latency in this part of the system, it improves availability in this other part of the system”, so you can decide where to spend engineering resources. I can guarantee you that the folks who are in charge of, say, EC2 are not just doing math with #nines of the systems underlying EC2. They are measuring and modeling. If I really wanted to figure out how to calculate availability of my systems, I would want to read case studies of real-world systems.
- FigmentEngine 7y ago
  on the intent question, you said: > Yep. Amazon (GCP, Azure) are making bank off the idea that you can just pay opex to run in multiple regions and lay off your SREs, but if you really wanted high availability (and not just some service credits when things go wrong) you would spend the engineering effort to go multi-provider. That is not AWS's intent. And its FUD to suggest that somehow cloud vendors want to make money out of reliability. For example, in AWS, a webapp with a single Region approach using multiple AZs will give you higher availablility, but similar costs to a single AZ approach.
  - reallydude 7y ago
    > the idea that you can just pay opex to run in multiple regions and lay off your SREs, > That is not AWS's intent. Yet that's what all their produced literature describes. > its FUD to suggest that somehow cloud vendors want to make money out of reliability. It's literally a selling point. Not sure how mentioning it is FUD. It's FUD (and rightly so) to point out that your cross-platform solution is likely less production-ready and reliable than AWS (in total) as a single point of failure.
    - jasonlotito 7y ago
      > Yet that's what all their produced literature describes. No, it's not. I've read it. The Well Architected stuff is actually really good about not being AWS only. These are generally good principals to keep in mind regardless of which provider you go with. And that's how it's written, and that's how AWS teaches it in person (I've actually been through their Well Architected course). Yes, they use AWS tooling to teach it, but nothing about it requires AWS. So, no. You are completely wrong. > It's literally a selling point. Not sure how mentioning it is FUD. Because the pricing AWS has doesn't necessarily increase cost with increase reliability. Suggesting that you need to spend more money to increase reliability is FUD. > It's FUD (and rightly so) to point out that your cross-platform solution is likely less production-ready and reliable than AWS (in total) as a single point of failure. That's not what was pointed out. Rather, it was pointed out that you can get high reliability without resorting to multi-cloud and higher costs. For you to continue suggesting otherwise, you first have to start off by explaining why you think high availability can't be obtained using a single cloud provider.

klodolph 7y ago

> I can't say for your employeer, but at AWS this is not our intent. Lost the antecedent, here. What is not your intent? Granted my understanding of cloud economics is somewhat murky. My general impression is that the big selling point of cloud is the move from capex to opex, and the second selling point is that you don’t need the same level of operations expertise to run cloud compared to on-prem. My hot take is that for high reliability you still need tons of operations expertise, and in these scenarios, solutions like (partial) on-prem and multi-provider become much more favorable. > rather than spreading FUD, why not help customers by showing them the better math? I feel like I’m being accused of spreading FUD, and I want to know why? All I am really trying to say here is that you can’t just do simple arithmetic on published #nines and end up with something that approximates the truth for your service in a useful way. Depending on the operation of your business this approximation may be acceptable or it may not be. Just to recap, the bad math is to come up with some threshold for acceptable performance in all components, model each as an independent Bernoulli variable, and then plug them into some boolean formula. This is the math published in the guide here, and you can do it with arithmetic on #nines. The reason why this is bad is because this leads you towards a shallow understanding of your system and creates pressure to inflate estimates of availability beyond actual availability, sometimes much so. Unfortunately, if you are really interested in calculating availability you need to come up with a model for your particular system. This is a complex subject that involves coming up with a model which strikes some balance between accuracy, simplicity (so it can be understood and used to inform strategy), usefulness (to customers / downstream), and supportability (to engineers working on the service). I can't tell you how to do that, my best guess is to hire someone who knows enough statistics to be dangerous and lock them in a room for three months with a terminal and access to your metrics. (As a side note, the tools for this kind of analysis are much better for systems like electronic circuits. For example, a part might be labeled as "1% tolerance" but when you are running simulations you use a probability distribution.) When you actually do this, you often find out that your system is much less available (reliable, durable) than it was designed to be. It’s common that your cloud provider could stay within SLO and your service would still be “down” as far as your definition went. So then you have the engineering problem of figuring out how to improve things, which may involve multi-region, multi-cloud, on-prem, redesigning parts of your system, changing utilization targets, etc. The model helps because it can reveal key insights like “if you improve latency in this part of the system, it improves availability in this other part of the system”, so you can decide where to spend engineering resources. I can guarantee you that the folks who are in charge of, say, EC2 are not just doing math with #nines of the systems underlying EC2. They are measuring and modeling. If I really wanted to figure out how to calculate availability of my systems, I would want to read case studies of real-world systems.

FigmentEngine 7y ago
on the intent question, you said: > Yep. Amazon (GCP, Azure) are making bank off the idea that you can just pay opex to run in multiple regions and lay off your SREs, but if you really wanted high availability (and not just some service credits when things go wrong) you would spend the engineering effort to go multi-provider. That is not AWS's intent. And its FUD to suggest that somehow cloud vendors want to make money out of reliability. For example, in AWS, a webapp with a single Region approach using multiple AZs will give you higher availablility, but similar costs to a single AZ approach.
- reallydude 7y ago
  > the idea that you can just pay opex to run in multiple regions and lay off your SREs, > That is not AWS's intent. Yet that's what all their produced literature describes. > its FUD to suggest that somehow cloud vendors want to make money out of reliability. It's literally a selling point. Not sure how mentioning it is FUD. It's FUD (and rightly so) to point out that your cross-platform solution is likely less production-ready and reliable than AWS (in total) as a single point of failure.
  - jasonlotito 7y ago
    > Yet that's what all their produced literature describes. No, it's not. I've read it. The Well Architected stuff is actually really good about not being AWS only. These are generally good principals to keep in mind regardless of which provider you go with. And that's how it's written, and that's how AWS teaches it in person (I've actually been through their Well Architected course). Yes, they use AWS tooling to teach it, but nothing about it requires AWS. So, no. You are completely wrong. > It's literally a selling point. Not sure how mentioning it is FUD. Because the pricing AWS has doesn't necessarily increase cost with increase reliability. Suggesting that you need to spend more money to increase reliability is FUD. > It's FUD (and rightly so) to point out that your cross-platform solution is likely less production-ready and reliable than AWS (in total) as a single point of failure. That's not what was pointed out. Rather, it was pointed out that you can get high reliability without resorting to multi-cloud and higher costs. For you to continue suggesting otherwise, you first have to start off by explaining why you think high availability can't be obtained using a single cloud provider.