3 ms·

It’s already happening… it’s a total mess of weird microservices that make no sense to reason about and the hidden costs on things like free tier etc… not good

seadan83 3y ago
I think I may disagree about the causes. Amazon A/B tests just about everything, the implication being hidden costs would have been found in A/B testing to net increase revenue. That is just profit motive and business ethics coming into play. The scattering of weird microservices I think is the same reason the Amazon detail & home page never get a full ground-up update. It's hard to do large scale data experiments to demonstrate the benefit, and it's too many teams with too much at stake (too many cooks)
- jocaal 3y ago
  > Amazon A/B tests just about everything Are they testing for the UI's that confuse customers the most? \s
  - seadan83 3y ago
    In a way actually! The things that are hard to A/B test get neglected. If you need A/B testing (ie: data) to make any decision, then places where data is purely qualitative - there will be no data to be had and therefore no decision can be made. The Expedia example really comes to mind too of micro-optimizations and everyone working towards their own org's goal can all make sense individually, but really fail when taken as a whole: https://www.qualtrics.com/blog/upstream-thinking-saved-expedia-millions/ https://www.qualtrics.com/blog/upstream-thinking-saved-exped... Expedia had an example of this: - the sales team wanted sales - the phone support team wanted to turn over calls quickly - If there were a way to measure confusion, it would be improved and Can I A/B test whether there will be a 10% increase in sales if I lower prices on Ec2 instances by 5% - YES! Can I A/B test that the presence of 30 service options is the tipping point of spending 1 hour to get something done vs wanting to hire an "AWS"
- seadan83 3y ago
  My fault for a milquetoast response. I disagree. My thoughts are informed here by insider experience at Amazon. I believe I observed these problems over a decade ago (hidden fees, high attrition, poor culture, scattered & non-consolidated service offerings). EG: There were a number of places that could have had a better customer experience, but A/B testing showed there was more money to be made than customers lost. Second example, seeing what was required to get 2 to 4 teams to cooperate was impressive.. The detail and home page touches the bread and butter for entire orgs, thousands of people, ramifications for dozens of teams and then dozens of backend teams behind them. I believe it was an Amazon VP that I was talking who pointed out that the homepage and detail page had not changed very much over time at all (lots of evolution, but never revolution, never fully redone, never will be fully redone). This was pertinent because we were in an org that ran offshoot sites, similar in a lot of ways, but less hands in the pot & we did have liberty to do a full rewrite of our detail pages & homepages. In the conversation, the VP of that org was highlighting that flexibility and went into small detail why that was not the case for the big mother-ship retail pages.
- simonw 3y ago
  A/B testing is an Amazon thing, but is it also an AWS thing? I thought AWS engineering culture was pretty different.
  - seadan83 3y ago
    Here is a page from AWS executives with insights on how your organization can also be data-driven too! https://aws.amazon.com/executive-insights/content/how-do-you-become-a-data-driven-organization/ https://aws.amazon.com/executive-insights/content/how-do-you... AWS does have differences for sure - but the "everything is only decided based on numbers unless you are Jeff Bezos" is universally true. Being that rigorous about data-driven-decisions is really quite powerful. The S-team are 1000% of this mind-set. I'm paraphrasing, the saying at Amazon that they tell new recruits and pride themselves over is that there are only three correct answers: "(1) Yes, and here is the data why. (2) No, and here is the data why. (3) I don't know, and I'll have the data shortly"
    - simonw 3y ago
      I see A/B testing as just one small component of data-driven decision making. You can make still great decisions based on your data without doing the thing where you ship the same feature twice and see which one works better.
      - seadan83 3y ago
        A priori, data-driven decision making is very powerful. One nice thing about working with former Amazon engineers, is they do bring that mindset. No premature optimization, no belief in facts without data. I think when you get burned a few many times to realize you believe things that are wrong and data shows you to be wrong, you tend to think more about why you know things and what data you have. It's powerful. By way of clarification, I rarely observed the same feature being shipped twice. It's generally more the 'A' test was the existing functionality and 'B' was whatever change you wanted to make. The A/B testing platform at Amazon received a lot of investment - it's powerful and so it's used for a lot more than mere A/B testing. It's also a feature toggle platform, lots of things are shipped under that framework so they can be "turned on & off". The metrics collection was powerful and highly integrated. Even if there is no actual comparison in parallel, the metrics that are generated were worth a lot (and often needed, as everyone's goals are all data based, so you want data to show you are hitting those goals. Can't just say we launched 5 features, and can't say we are just still here, it's data or nothing at Amazon) Most importantly, the "great decisions" is interesting. The power of gathering data very quickly and often is you see when decisions were not great. Amazon was excellent at this, things that made more money were given support, things that lost money were quickly terminated and re-organized. The data provides light on the decisions, it's virtually a BizDev super-power to be able to pivot away from losers so quickly like that.

seadan83 3y ago

I think I may disagree about the causes. Amazon A/B tests just about everything, the implication being hidden costs would have been found in A/B testing to net increase revenue. That is just profit motive and business ethics coming into play. The scattering of weird microservices I think is the same reason the Amazon detail & home page never get a full ground-up update. It's hard to do large scale data experiments to demonstrate the benefit, and it's too many teams with too much at stake (too many cooks)

jocaal 3y ago
> Amazon A/B tests just about everything Are they testing for the UI's that confuse customers the most? \s
- seadan83 3y ago
  In a way actually! The things that are hard to A/B test get neglected. If you need A/B testing (ie: data) to make any decision, then places where data is purely qualitative - there will be no data to be had and therefore no decision can be made. The Expedia example really comes to mind too of micro-optimizations and everyone working towards their own org's goal can all make sense individually, but really fail when taken as a whole: https://www.qualtrics.com/blog/upstream-thinking-saved-expedia-millions/ https://www.qualtrics.com/blog/upstream-thinking-saved-exped... Expedia had an example of this: - the sales team wanted sales - the phone support team wanted to turn over calls quickly - If there were a way to measure confusion, it would be improved and Can I A/B test whether there will be a 10% increase in sales if I lower prices on Ec2 instances by 5% - YES! Can I A/B test that the presence of 30 service options is the tipping point of spending 1 hour to get something done vs wanting to hire an "AWS"
seadan83 3y ago
My fault for a milquetoast response. I disagree. My thoughts are informed here by insider experience at Amazon. I believe I observed these problems over a decade ago (hidden fees, high attrition, poor culture, scattered & non-consolidated service offerings). EG: There were a number of places that could have had a better customer experience, but A/B testing showed there was more money to be made than customers lost. Second example, seeing what was required to get 2 to 4 teams to cooperate was impressive.. The detail and home page touches the bread and butter for entire orgs, thousands of people, ramifications for dozens of teams and then dozens of backend teams behind them. I believe it was an Amazon VP that I was talking who pointed out that the homepage and detail page had not changed very much over time at all (lots of evolution, but never revolution, never fully redone, never will be fully redone). This was pertinent because we were in an org that ran offshoot sites, similar in a lot of ways, but less hands in the pot & we did have liberty to do a full rewrite of our detail pages & homepages. In the conversation, the VP of that org was highlighting that flexibility and went into small detail why that was not the case for the big mother-ship retail pages.
simonw 3y ago
A/B testing is an Amazon thing, but is it also an AWS thing? I thought AWS engineering culture was pretty different.
- seadan83 3y ago
  Here is a page from AWS executives with insights on how your organization can also be data-driven too! https://aws.amazon.com/executive-insights/content/how-do-you-become-a-data-driven-organization/ https://aws.amazon.com/executive-insights/content/how-do-you... AWS does have differences for sure - but the "everything is only decided based on numbers unless you are Jeff Bezos" is universally true. Being that rigorous about data-driven-decisions is really quite powerful. The S-team are 1000% of this mind-set. I'm paraphrasing, the saying at Amazon that they tell new recruits and pride themselves over is that there are only three correct answers: "(1) Yes, and here is the data why. (2) No, and here is the data why. (3) I don't know, and I'll have the data shortly"
  - simonw 3y ago
    I see A/B testing as just one small component of data-driven decision making. You can make still great decisions based on your data without doing the thing where you ship the same feature twice and see which one works better.
    - seadan83 3y ago
      A priori, data-driven decision making is very powerful. One nice thing about working with former Amazon engineers, is they do bring that mindset. No premature optimization, no belief in facts without data. I think when you get burned a few many times to realize you believe things that are wrong and data shows you to be wrong, you tend to think more about why you know things and what data you have. It's powerful. By way of clarification, I rarely observed the same feature being shipped twice. It's generally more the 'A' test was the existing functionality and 'B' was whatever change you wanted to make. The A/B testing platform at Amazon received a lot of investment - it's powerful and so it's used for a lot more than mere A/B testing. It's also a feature toggle platform, lots of things are shipped under that framework so they can be "turned on & off". The metrics collection was powerful and highly integrated. Even if there is no actual comparison in parallel, the metrics that are generated were worth a lot (and often needed, as everyone's goals are all data based, so you want data to show you are hitting those goals. Can't just say we launched 5 features, and can't say we are just still here, it's data or nothing at Amazon) Most importantly, the "great decisions" is interesting. The power of gathering data very quickly and often is you see when decisions were not great. Amazon was excellent at this, things that made more money were given support, things that lost money were quickly terminated and re-organized. The data provides light on the decisions, it's virtually a BizDev super-power to be able to pivot away from losers so quickly like that.