4 ms·

I'm sorry, you could make this case about some kinds of telemetry, but specifically not A/B testing. Speaking from work experience: A/B testing doesn't look int

I'm sorry, you could make this case about some kinds of telemetry, but specifically not A/B testing. Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent. These metrics rarely align with a better experience for the user (outside of like, prettier buttons and stuff), and instead tend to result in less-informative, less-agentic software (information and choice often distract from conversions!)

andrewmutz 4y ago
That's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form. It's a great tool, and its impact all depends on how you use it.
- brundolf 4y ago
  I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing
  - andrewmutz 4y ago
    Data entry
  - scott_w 4y ago
    You can measure the time between two key actions that operate as a proxy for task completion.
- Retric 4y ago
  There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with. The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.
  - judge2020 4y ago
    The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.
  - dado3212 4y ago
    As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.
- g_p 4y ago
  This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives. If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome. The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.
  - MikePlacid 4y ago
    > The challenge is when you throw in an ad-based revenue model The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects: - less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society; - streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”. That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?). I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.
    - sokoloff 4y ago
      The number of people willing to use Facebook today vastly exceeds the number willing to pay $25/yr to use Facebook. I bet by at least 4 orders of magnitude and likely 5 or 6.
costcofries 4y ago
This is complete BS. I run hundreds of a/b tests each quarter and I specifically refuse to run the types of experiments you allude to. My a/b testing is all about helping users achieve the things (the outcomes) that they want to achieve by using our product in the first place. If we can help them do that, with more ease, then we are creating a better experience. Perhaps you should just agree that, "not all a/b testing is the same".
- fragmede 4y ago
  How is that BS? Other companies don't have you there to say no to them and are definitely running the kind of experiments you're too good for.
  - costcofries 4y ago
    Did you even read my whole comment? It's BS because he/she/they blanketed it without taking any nuance which i tried to do with my comment + an example! Quote - "Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent"

andrewmutz 4y ago

That's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form. It's a great tool, and its impact all depends on how you use it.

brundolf 4y ago
I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing
- andrewmutz 4y ago
  Data entry
- scott_w 4y ago
  You can measure the time between two key actions that operate as a proxy for task completion.
Retric 4y ago
There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with. The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.
- judge2020 4y ago
  The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.
- dado3212 4y ago
  As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.
g_p 4y ago
This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives. If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome. The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.
- MikePlacid 4y ago
  > The challenge is when you throw in an ad-based revenue model The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects: - less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society; - streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”. That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?). I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.
  - sokoloff 4y ago
    The number of people willing to use Facebook today vastly exceeds the number willing to pay $25/yr to use Facebook. I bet by at least 4 orders of magnitude and likely 5 or 6.

costcofries 4y ago

This is complete BS. I run hundreds of a/b tests each quarter and I specifically refuse to run the types of experiments you allude to. My a/b testing is all about helping users achieve the things (the outcomes) that they want to achieve by using our product in the first place. If we can help them do that, with more ease, then we are creating a better experience. Perhaps you should just agree that, "not all a/b testing is the same".

fragmede 4y ago
How is that BS? Other companies don't have you there to say no to them and are definitely running the kind of experiments you're too good for.
- costcofries 4y ago
  Did you even read my whole comment? It's BS because he/she/they blanketed it without taking any nuance which i tried to do with my comment + an example! Quote - "Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent"