3 ms·

Better customer experiences often times lead to increased metrics. They are not totally different things.

by ssharp 4y ago

wongarsu 4y ago
Maybe if you are very aware of the fact that your goal and metric aren't totally aligned, but that often gets lost. As a result A/B testing for longer website visits can make websites that make it more obvious that the information you want is there, but also make the path to actually get it longer. A/B testing for engagement might promote divisive behavior and fights. A/B testing for read rate or clicks might lead to trust loss. I think a lot of lessons from AI safety apply surprisingly well to A/B testing, mainly around how hard it is to align your actual goals with the metrics you use for optimization, and how disastrous the consequences can be. It doesn't have to go wrong, but it's incredibly hard to ensure it goes right, especially if it's the only feedback you have.
- noirbot 4y ago
  But that's just optimizing for bad metrics. At this point, anyone who thinks "engagement" and "time spent on page" are customer-positive metrics is in a different mental space than you and I. There's a lot of ineffable things that make up good customer experience that would be hard-to-impossible to A/B test, but it doesn't mean that A/B testing is "unsafe" just because it could be used to optimize for bad things any more than any other telemetry or metrics gathering could be bad because you could optimize for evil things. And at the same time, bad management and product leadership can optimize and develop towards bad goals with plenty of tools that aren't A/B testing. It seems to miss the point to blame/stigmatize a specific tool because it's been used poorly by a few bad actors in a public way.
  - rout39574 4y ago
    I think the point is that most metrics are "bad metrics" for this purpose, as suggested by Goodheart's law. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Further, I imagine that the obvious "known bad" metrics are not selected only by "A few bad apples". I think it's likely they are selected by the mass of business actors looking for current quarter results.
    - noirbot 4y ago
      For sure. I don't think there's general "overall" metrics that you want to be testing against every single time on every change outside of basic performance metrics for loading or rendering in real-world environments. I wasn't at all trying to say that only a few places are optimizing for bad things, but as you see all over this thread, there's a number of companies that immediately come to people's minds as bad actors when it comes to A/B testing - Google, Meta, Microsoft. There's plenty of other companies that are more ethical about it, or use it as part of rolling out general changes and collecting feedback. I know half of the time I log into the AWS Console it has some sort of "Hey, we're testing out a new upcoming UX for this page. Click here if you want to go back to the old one", which seems like a decent way for them to get feedback on the new designs while not drastically disrupting things.
  - throwaway290 4y ago
    Bad management can certainly ruin things without A/B testing. It doesn't excuse A/B testing simply being a poor tool among all you have access to. Talking to users and stakeholders, for example, provides infinitely more input. (Edit: yeah in many cases measuring what users do, directly watching or via analytics, is also useful.)
    - noirbot 4y ago
      Definitely - I'm not trying to say A/B testing is amazing, just that a lot of the comments have a strong "if you do A/B testing you're evil and are out to manipulate people" bent to them, which I think is too far in the other direction. Talking to people is great, but getting a representative sample is hard, and often people are bad at both understanding what they want, expressing it, or even being accurate about how they use things. I know when I was working closer to the UX side of the business before, I was constantly surprised by both what users would say they want AND by how users actually used the products. In my mind, A/B testing is good as a sort of "final pass" to serve as broad, semi-random validation that the change you're looking to make does actually do the thing that it's intended to do. It's not great for early on when you don't really know what to measure or look for, or if the change is remotely reasonable, but it can help check for if your focus group/user panel happened to be weirdly skewed in their usage/desires.
- ssharp 4y ago
  I've spent a lot of my career doing A/B testing, including doing that role exclusively for a number of years. I specialize in ecommerce, so maybe I have too narrow of a view here, but in that vast majority of cases, I am optimizing for revenue per visitor, which is a function of conversion rate and average order value. There are sometimes leading indicators like engagement, but in ecommerce, you're afforded the luxury of basing things on revenue or even bottom line. I really don't like the positioning of ALL A/B testing as unethical behavior where you're hostilely trying to take advantage of a user. It's quite the opposite. There are a lot of extremely poor user experiences out there and a quality testing program can help improve user experiences, remove risk from making sweeping changes, and help you learn more about your audience and market. The vast majority of the successful testing I've done is done around trying to HELP users navigate the site and product catalog, understand the product, and purchase the product. Attention spans are fleeting with online shopping and even the smallest points of confusion or friction can turn shoppers off. Additionally, often times I'll read into test results after a month or so to see if there were any issues with orders that might indicate purchases from disinterested people or misaligned expectations.

Better customer experiences often times lead to increased metrics. They are not totally different things.

by ssharp 4y ago

Better customer experiences often times lead to increased metrics. They are not totally different things.

Maybe if you are very aware of the fact that your goal and metric aren't totally aligned, but that often gets lost. As a result A/B testing for longer website visits can make websites that make it more obvious that the information you want is there, but also make the path to actually get it longer. A/B testing for engagement might promote divisive behavior and fights. A/B testing for read rate or clicks might lead to trust loss. I think a lot of lessons from AI safety apply surprisingly well to A/B testing, mainly around how hard it is to align your actual goals with the metrics you use for optimization, and how disastrous the consequences can be. It doesn't have to go wrong, but it's incredibly hard to ensure it goes right, especially if it's the only feedback you have.

noirbot 4y ago
But that's just optimizing for bad metrics. At this point, anyone who thinks "engagement" and "time spent on page" are customer-positive metrics is in a different mental space than you and I. There's a lot of ineffable things that make up good customer experience that would be hard-to-impossible to A/B test, but it doesn't mean that A/B testing is "unsafe" just because it could be used to optimize for bad things any more than any other telemetry or metrics gathering could be bad because you could optimize for evil things. And at the same time, bad management and product leadership can optimize and develop towards bad goals with plenty of tools that aren't A/B testing. It seems to miss the point to blame/stigmatize a specific tool because it's been used poorly by a few bad actors in a public way.
- rout39574 4y ago
  I think the point is that most metrics are "bad metrics" for this purpose, as suggested by Goodheart's law. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Further, I imagine that the obvious "known bad" metrics are not selected only by "A few bad apples". I think it's likely they are selected by the mass of business actors looking for current quarter results.
  - noirbot 4y ago
    For sure. I don't think there's general "overall" metrics that you want to be testing against every single time on every change outside of basic performance metrics for loading or rendering in real-world environments. I wasn't at all trying to say that only a few places are optimizing for bad things, but as you see all over this thread, there's a number of companies that immediately come to people's minds as bad actors when it comes to A/B testing - Google, Meta, Microsoft. There's plenty of other companies that are more ethical about it, or use it as part of rolling out general changes and collecting feedback. I know half of the time I log into the AWS Console it has some sort of "Hey, we're testing out a new upcoming UX for this page. Click here if you want to go back to the old one", which seems like a decent way for them to get feedback on the new designs while not drastically disrupting things.
- throwaway290 4y ago
  Bad management can certainly ruin things without A/B testing. It doesn't excuse A/B testing simply being a poor tool among all you have access to. Talking to users and stakeholders, for example, provides infinitely more input. (Edit: yeah in many cases measuring what users do, directly watching or via analytics, is also useful.)
  - noirbot 4y ago
    Definitely - I'm not trying to say A/B testing is amazing, just that a lot of the comments have a strong "if you do A/B testing you're evil and are out to manipulate people" bent to them, which I think is too far in the other direction. Talking to people is great, but getting a representative sample is hard, and often people are bad at both understanding what they want, expressing it, or even being accurate about how they use things. I know when I was working closer to the UX side of the business before, I was constantly surprised by both what users would say they want AND by how users actually used the products. In my mind, A/B testing is good as a sort of "final pass" to serve as broad, semi-random validation that the change you're looking to make does actually do the thing that it's intended to do. It's not great for early on when you don't really know what to measure or look for, or if the change is remotely reasonable, but it can help check for if your focus group/user panel happened to be weirdly skewed in their usage/desires.
ssharp 4y ago
I've spent a lot of my career doing A/B testing, including doing that role exclusively for a number of years. I specialize in ecommerce, so maybe I have too narrow of a view here, but in that vast majority of cases, I am optimizing for revenue per visitor, which is a function of conversion rate and average order value. There are sometimes leading indicators like engagement, but in ecommerce, you're afforded the luxury of basing things on revenue or even bottom line. I really don't like the positioning of ALL A/B testing as unethical behavior where you're hostilely trying to take advantage of a user. It's quite the opposite. There are a lot of extremely poor user experiences out there and a quality testing program can help improve user experiences, remove risk from making sweeping changes, and help you learn more about your audience and market. The vast majority of the successful testing I've done is done around trying to HELP users navigate the site and product catalog, understand the product, and purchase the product. Attention spans are fleeting with online shopping and even the smallest points of confusion or friction can turn shoppers off. Additionally, often times I'll read into test results after a month or so to see if there were any issues with orders that might indicate purchases from disinterested people or misaligned expectations.