3 ms·

I hate to say this, but I genuinely think customers are going to find AWS become more unreliable as time goes on. It seems to me that you can have high levels

I hate to say this, but I genuinely think customers are going to find AWS become more unreliable as time goes on. It seems to me that you can have high levels of reliability, a great working culture and high levels of staffing, but if you mess with any one of these elements you will find the wheels start falling off the train.

arrakeenrevived 3y ago
I think this is true of any company. As things get larger, red tape increases, and the mess of spaghetti code and complexity/inter-dependencies of systems means it gets harder and harder for any one person or team to fully understand the environment and avoid/recover from issues. I actually think that the "pizza team" model in Amazon can help mitigate this, as each individual team operates as its own entity that is discouraged from relying on other teams for uptime... but this is only effective to a point.
- chris_wot 3y ago
  As the article points out, this is a very expensive way of operating and requires very high headcount. They don’t want mass layoffs because of the cost of doing so, thus they employ the unethical tactic of making life miserable for the ones they would have tried to lay off. What it also means is that they can use younger and cheaper workers. They can also outsource their work. Thus my concerns about reliability. It seems like a house of cards. I truly wonder for how long they can keep this up. Given our reliance on cloud computing and their prominent position within it, it may become a very rocky ride at some point for businesses who have transitioned their operational infrastructure to AWS. I think a lot of people treat AWS as an infinite resource that will never run out and will be completely reliable forever. I think, sadly, many organisations may get a shock some time into the near future.
  - pylua 3y ago
    Here is maybe an uncomfortable truth for software engineers: it’s more efficient from a cost perspective to just something out there that is good enough, even if it makes no logical sense and is a mess. Maintainability and readability are best solved through brute force hiring instead of better practices.
    - chris_wot 3y ago
      In what world does maintainability and reliability increase through brute force hiring of low skilled and experienced staff?
      - pylua 3y ago
        The world where good enough / more cpu resources and just works is easier to get to than design the system in a logical way.
        chris_wot 3y ago
        In other words, it doesn’t exist.
brunooliv 3y ago
It’s already happening… it’s a total mess of weird microservices that make no sense to reason about and the hidden costs on things like free tier etc… not good
- seadan83 3y ago
  I think I may disagree about the causes. Amazon A/B tests just about everything, the implication being hidden costs would have been found in A/B testing to net increase revenue. That is just profit motive and business ethics coming into play. The scattering of weird microservices I think is the same reason the Amazon detail & home page never get a full ground-up update. It's hard to do large scale data experiments to demonstrate the benefit, and it's too many teams with too much at stake (too many cooks)
  - jocaal 3y ago
    > Amazon A/B tests just about everything Are they testing for the UI's that confuse customers the most? \s
    - seadan83 3y ago
      In a way actually! The things that are hard to A/B test get neglected. If you need A/B testing (ie: data) to make any decision, then places where data is purely qualitative - there will be no data to be had and therefore no decision can be made. The Expedia example really comes to mind too of micro-optimizations and everyone working towards their own org's goal can all make sense individually, but really fail when taken as a whole: https://www.qualtrics.com/blog/upstream-thinking-saved-expedia-millions/ https://www.qualtrics.com/blog/upstream-thinking-saved-exped... Expedia had an example of this: - the sales team wanted sales - the phone support team wanted to turn over calls quickly - If there were a way to measure confusion, it would be improved and Can I A/B test whether there will be a 10% increase in sales if I lower prices on Ec2 instances by 5% - YES! Can I A/B test that the presence of 30 service options is the tipping point of spending 1 hour to get something done vs wanting to hire an "AWS"
  - seadan83 3y ago
ActorNightly 3y ago
Do you really think that engineers must commit code on a weekly basis to prevent infrastructure from crashing down because of some sort of "decay" that happens in the software naturally?

arrakeenrevived 3y ago

I think this is true of any company. As things get larger, red tape increases, and the mess of spaghetti code and complexity/inter-dependencies of systems means it gets harder and harder for any one person or team to fully understand the environment and avoid/recover from issues. I actually think that the "pizza team" model in Amazon can help mitigate this, as each individual team operates as its own entity that is discouraged from relying on other teams for uptime... but this is only effective to a point.

chris_wot 3y ago
As the article points out, this is a very expensive way of operating and requires very high headcount. They don’t want mass layoffs because of the cost of doing so, thus they employ the unethical tactic of making life miserable for the ones they would have tried to lay off. What it also means is that they can use younger and cheaper workers. They can also outsource their work. Thus my concerns about reliability. It seems like a house of cards. I truly wonder for how long they can keep this up. Given our reliance on cloud computing and their prominent position within it, it may become a very rocky ride at some point for businesses who have transitioned their operational infrastructure to AWS. I think a lot of people treat AWS as an infinite resource that will never run out and will be completely reliable forever. I think, sadly, many organisations may get a shock some time into the near future.
- pylua 3y ago
  Here is maybe an uncomfortable truth for software engineers: it’s more efficient from a cost perspective to just something out there that is good enough, even if it makes no logical sense and is a mess. Maintainability and readability are best solved through brute force hiring instead of better practices.
  - chris_wot 3y ago
    In what world does maintainability and reliability increase through brute force hiring of low skilled and experienced staff?
    - pylua 3y ago
      The world where good enough / more cpu resources and just works is easier to get to than design the system in a logical way.
      - chris_wot 3y ago
        In other words, it doesn’t exist.

brunooliv 3y ago

It’s already happening… it’s a total mess of weird microservices that make no sense to reason about and the hidden costs on things like free tier etc… not good

seadan83 3y ago
I think I may disagree about the causes. Amazon A/B tests just about everything, the implication being hidden costs would have been found in A/B testing to net increase revenue. That is just profit motive and business ethics coming into play. The scattering of weird microservices I think is the same reason the Amazon detail & home page never get a full ground-up update. It's hard to do large scale data experiments to demonstrate the benefit, and it's too many teams with too much at stake (too many cooks)
- jocaal 3y ago
  > Amazon A/B tests just about everything Are they testing for the UI's that confuse customers the most? \s
  - seadan83 3y ago
    In a way actually! The things that are hard to A/B test get neglected. If you need A/B testing (ie: data) to make any decision, then places where data is purely qualitative - there will be no data to be had and therefore no decision can be made. The Expedia example really comes to mind too of micro-optimizations and everyone working towards their own org's goal can all make sense individually, but really fail when taken as a whole: https://www.qualtrics.com/blog/upstream-thinking-saved-expedia-millions/ https://www.qualtrics.com/blog/upstream-thinking-saved-exped... Expedia had an example of this: - the sales team wanted sales - the phone support team wanted to turn over calls quickly - If there were a way to measure confusion, it would be improved and Can I A/B test whether there will be a 10% increase in sales if I lower prices on Ec2 instances by 5% - YES! Can I A/B test that the presence of 30 service options is the tipping point of spending 1 hour to get something done vs wanting to hire an "AWS"
- seadan83 3y ago

ActorNightly 3y ago

Do you really think that engineers must commit code on a weekly basis to prevent infrastructure from crashing down because of some sort of "decay" that happens in the software naturally?