4 ms·

That's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form. It's a great tool, and

That's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form. It's a great tool, and its impact all depends on how you use it.

brundolf 4y ago
I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing
- andrewmutz 4y ago
  Data entry
- scott_w 4y ago
  You can measure the time between two key actions that operate as a proxy for task completion.
Retric 4y ago
There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with. The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.
- judge2020 4y ago
  The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.
- dado3212 4y ago
  As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.
g_p 4y ago
This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives. If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome. The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.
- MikePlacid 4y ago
  > The challenge is when you throw in an ad-based revenue model The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects: - less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society; - streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”. That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?). I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.
  - sokoloff 4y ago
    The number of people willing to use Facebook today vastly exceeds the number willing to pay $25/yr to use Facebook. I bet by at least 4 orders of magnitude and likely 5 or 6.

brundolf 4y ago

I'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing

andrewmutz 4y ago
Data entry
scott_w 4y ago
You can measure the time between two key actions that operate as a proxy for task completion.

Retric 4y ago

There are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with. The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.

judge2020 4y ago
The fear with direct user research is that, unless you have a team and budget for getting enough of a sample, one-on-ones might not only be unhelpful but actively harmful if you implement something that solves that customers' problem but otherwise gets in the way for other customers.
dado3212 4y ago
As someone who’s run literally hundreds of A/B tests, many of them on the backs of UX research with users in the field, people have no idea what they want. The anecdata is a place to investigate, but never the end of the journey.

g_p 4y ago

This is an interesting example, and perhaps pushes part of the blame and dislike for A/B testing onto tech companies' incentives. If you're building a tool to make life easier for the user, something that gives them a better experience is your optimal outcome. This seems like a scenario where A/B can produce a good outcome. The challenge is when you throw in an ad-based revenue model, and the A/B testing is then optimized for the opposite (eyeball-hours, linear metres scrolled per session, ad spots passed, ads clicked) - engagement-based business models end up (I'd argue) A/B optimizing for the opposite of what their users want, to get them to spend longer doing a task they could have done quicker.

MikePlacid 4y ago
> The challenge is when you throw in an ad-based revenue model The funny thing is - the ad-based revenue model is not the only possible variant. Last time I’ve checked Facebook’s profits per user were $7 per quarter, that is $28 a year. At the same time I am paying LiveJournal $25 a year for the ad-free version. Just taking my money looks like a much better model in many respects: - less overhead: a lot of people doing these studies how to force me to look at something I do not want to look at will be free to do something more useful to the society; - streamlined relationship between me and my publisher: in this model there is no advertiser who can say “I do not like these texts, no revenue for you”. That’s why I prefer to pay for some Substack authors, like Matt Taibbi and Glen Greenwald, than to try to fish their texts for free amid some sea of “clever” advertising (hey AI testers, I bought this thing already, what’s the point of forcing it on me again and again?). I kinda wish that Brave model (my money distributed between sites I visited) got more traction. It looks much more healthy.
- sokoloff 4y ago
  The number of people willing to use Facebook today vastly exceeds the number willing to pay $25/yr to use Facebook. I bet by at least 4 orders of magnitude and likely 5 or 6.