5 ms·

I would be curious to hear what any current or former AWS employees think the internal consequences become for this. From the outside, it feels like they are g

by tyingq 5y ago

I would be curious to hear what any current or former AWS employees think the internal consequences become for this. From the outside, it feels like they are going to have to do something different to get some customer confidence back. Some sort of "mea culpa" with an explanation of what they are going to change.

manojlds 5y ago
Jeff Bezos comes back /s
oonis 5y ago
It’ll be a lot of finger pointing and teams covering their own ass
- tyingq 5y ago
  That seems to be the status quo. I'm curious if the current trend is enough to break that tradition.
- sgt 5y ago
  Or maybe breaking stuff is celebrated? For every type of downtime there will be lessons learned.
  - cconstantine 5y ago
    Every place I'v worked has celebrated fixing things that have broken, and forces engineers to fight for prevention. It's not exactly celebrating things breaking, but it sure looks a lot like it.
  - Arainach 5y ago
    That's only true if lessons are learned. If an outage happens and you do nothing to fix it (or worse, do something entirely too specific without fixing the real problem and thinking you've done it) the exact same problem will happen again and you'll get another opportunity to not learn the lesson.
awsthrow8264 5y ago
Can't speak for other teams, but the dynamics in mine are pretty great. We don't overwork and leadership puts explicit emphasis on work/life quality. From what I've heard in other teams (including some that own the big services you're familiar with), it can be a shitshow. Oncalls get paged 10*N times a week (typically 24/7 2-week rotation with two active oncalls, but depends), teams are leaking talent, desperately trying to hire good engineers while keeping the lights on.
gregjor 5y ago
Apologies are not actionable. Service providers promise some number of 9s percent uptime, not zero problems. It should be obvious that perfection is not achievable, and that at scale small problems can get magnified. Customer confidence is only an issue if there are alternative providers that don’t have problems. That isn’t the case with cloud hosting.
- tyingq 5y ago
  I'm unclear on what you're responding to. I wasn't saying an apology was the right remedy, nor did I suggest they should have zero issues. Customer confidence is an issue because they've had 4 notable issues in a very short timeframe. Edit: I think you're reading something into my comments that's more than what I've said. I am curious what the AWS response to 4 major incidents in a month's time will be. That's it. At least in my circles, it is an issue for customers. I respect that it appears not to be in your circles. Ahh. By "response", I did not mean the incident summary. I meant the overarching company response, if any. Like policies around change control, capacity additions, and so on.
  - gregjor 5y ago
    The issues affected only a fraction of customers. Long threads on HN speculating about the causes and armchair quarterbacking have little to do with customer confidence. All of the big cloud providers have problems, so unless there's a service with similar offerings and reach with demonstrably better reliability customer confidence is not going to matter -- people will use the least bad of the available choices. I have multiple customers hosting on AWS, and a couple on GCP, and one on Azure. Azure by far has the most problems in my limited experience. I have servers on AWS (US East Ohio and US West Oregon) with 600+ days of uptime. None of my customers except the one on Azure have ever contacted me about an outage, in two or three years. At the same time I can read long threads like this on HN full of doom and gloom and predictions about the decline of AWS. Which customers are we talking about? I know I'm not the customer for AWS -- I don't pay for hosting. Having moved more than a dozen companies from self-hosting and co-located hosting to AWS (or GCP, Azure if that's what they want) I can say that they are 100% happier. Relatively infrequent outages that AWS has the resources and incentive to jump all over and fix are preferable to trying to get me or some other pissed off engineer to drive in and try to figure things out. If quality, reliability, and frequent outages drove customers away then Tesla would have withered up years ago. There are more factors in play here than what a small number of committed tech geeks (like me) think about how AWS could be doing things better.

by tyingq 5y ago

manojlds 5y ago

Jeff Bezos comes back /s

oonis 5y ago

It’ll be a lot of finger pointing and teams covering their own ass

tyingq 5y ago
That seems to be the status quo. I'm curious if the current trend is enough to break that tradition.
sgt 5y ago
Or maybe breaking stuff is celebrated? For every type of downtime there will be lessons learned.
- cconstantine 5y ago
  Every place I'v worked has celebrated fixing things that have broken, and forces engineers to fight for prevention. It's not exactly celebrating things breaking, but it sure looks a lot like it.
- Arainach 5y ago
  That's only true if lessons are learned. If an outage happens and you do nothing to fix it (or worse, do something entirely too specific without fixing the real problem and thinking you've done it) the exact same problem will happen again and you'll get another opportunity to not learn the lesson.

awsthrow8264 5y ago

Can't speak for other teams, but the dynamics in mine are pretty great. We don't overwork and leadership puts explicit emphasis on work/life quality. From what I've heard in other teams (including some that own the big services you're familiar with), it can be a shitshow. Oncalls get paged 10*N times a week (typically 24/7 2-week rotation with two active oncalls, but depends), teams are leaking talent, desperately trying to hire good engineers while keeping the lights on.

gregjor 5y ago

Apologies are not actionable. Service providers promise some number of 9s percent uptime, not zero problems. It should be obvious that perfection is not achievable, and that at scale small problems can get magnified. Customer confidence is only an issue if there are alternative providers that don’t have problems. That isn’t the case with cloud hosting.

tyingq 5y ago
I'm unclear on what you're responding to. I wasn't saying an apology was the right remedy, nor did I suggest they should have zero issues. Customer confidence is an issue because they've had 4 notable issues in a very short timeframe. Edit: I think you're reading something into my comments that's more than what I've said. I am curious what the AWS response to 4 major incidents in a month's time will be. That's it. At least in my circles, it is an issue for customers. I respect that it appears not to be in your circles. Ahh. By "response", I did not mean the incident summary. I meant the overarching company response, if any. Like policies around change control, capacity additions, and so on.
- gregjor 5y ago
  The issues affected only a fraction of customers. Long threads on HN speculating about the causes and armchair quarterbacking have little to do with customer confidence. All of the big cloud providers have problems, so unless there's a service with similar offerings and reach with demonstrably better reliability customer confidence is not going to matter -- people will use the least bad of the available choices. I have multiple customers hosting on AWS, and a couple on GCP, and one on Azure. Azure by far has the most problems in my limited experience. I have servers on AWS (US East Ohio and US West Oregon) with 600+ days of uptime. None of my customers except the one on Azure have ever contacted me about an outage, in two or three years. At the same time I can read long threads like this on HN full of doom and gloom and predictions about the decline of AWS. Which customers are we talking about? I know I'm not the customer for AWS -- I don't pay for hosting. Having moved more than a dozen companies from self-hosting and co-located hosting to AWS (or GCP, Azure if that's what they want) I can say that they are 100% happier. Relatively infrequent outages that AWS has the resources and incentive to jump all over and fix are preferable to trying to get me or some other pissed off engineer to drive in and try to figure things out. If quality, reliability, and frequent outages drove customers away then Tesla would have withered up years ago. There are more factors in play here than what a small number of committed tech geeks (like me) think about how AWS could be doing things better.