4 ms·

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlin

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even really visit anymore. Yes, I'm guity myself of clicking on those articles when I do visit, but at a certain point I've found that Ars doesn't have the news I'm after, so I turn elsewhere and now Ars has one less viewer.

smueller1234 8y ago
I think what you say is practically spot on. Yet I'd like to add that I don't think that testing frameworks (at this point it would be misleading to call them strictly A/B testing) HAVE to only reflect short term gains. It's hard to come up with (proxy) metrics that hold that kind of short term optimization in check. Just to pick one fairly obvious example for e-commerce: you can track returns/customer service contacts as a health metric or even explicitly assign a value to them to fold them into the primary metric you're optimizing. These kinds of safeguards typically need longer recording periods, so it's potentially a lot of data science and engineering effort to build tools that can handle the long term data collection and analysis. But it's not impossible. It's rather something people love to pretend is not a problem.
vintagedave 8y ago
> I've seen this repeatedly with ArsTechnica Can you expand? I am an Ars reader, and I too find it frustrating compared to what it used to be. I wish there were more in-depth technical articles; I find it too light on details, written for a non-tech audience. I really, really wish it had solid technical content, since it's what got me reading it. But I don't characterise it as click-bait (it seems clear) nor as especially political, except in so far as politics intersects technology and climate. In both those it seems balanced or erring towards freedom.
- kolpa 8y ago
  Politics and a Climate and Freedom are interesting, but not what makes Ars Technica interesting.
swied 8y ago
So true about the danger of focusing on short term gains. I think that it is a problem of picking the wrong things to optimize and then sampling on the wrong entities. Conversion rate is often the most commonly used target, but, it is short sighted. In the big picture a company is really just trying to maximize profit. Note that you can't use traditional A/B testing tools on a value like total profit. You need an average if you are going to run an A/B test. For conversion rates your average is conversions per user. In this case you sample on the denominator -- the users. What do you sample on if you are testing total profit per quarter? This is the ultimate conundrum for websites who use A/B testing. They don't understand how they might actually get more new/returning users by simply improving the user experience. Instead, they run marketing campaigns to drive growth and then squeeze every dollar out of the customers when they arrive. This is not a good long term strategy. I have reviewed hundreds of A/B tests over the years. The only ones that I think succeeded were the ones that focused on improving the users experience.
- s-shellfish 8y ago
  User experience analysis is always going to require a human mind to synthesize the material. The problem is that automation eventually gets noticed by users, thus automation implies to a user 'the programmers/whoever believes my mind and preferences can be automated'. That's always going to throw a wrench into automation. So yes, it's hard work that requires real thinking with a clear focus on the users and how the users feel about the software. Software needs to care about users if it is dependent on users for it's business model!
mockingbirdy 8y ago
Winning the game but losing the meta-game.
tehlike 8y ago
Not entirely true. You can setup your experiment to be user keyed (as it should be) and can run it for a long time, or can run the hold back for a long time Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.
- kolpa 8y ago
  Long-term A/B testing has major problems in the modern world where users can communicate with each other (both onsite and out of band) and can use multiple accounts / devices / browsers
  - tehlike 8y ago
    communicating user is less of a concern, cross device _is_ a real concern. In some cases it's minimized with user logins, though.
citrablue 8y ago
What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27 http://dilbert.com/strip/2014-10-27
- swied 8y ago
  A/B testing is a great tool for incrementally improving upon a product design given the correct business strategy. Just don't let the limitations of A/B testing drive how you choose your business strategy. The strategy is ultimately about maximizing total profit, and not necessarily short term profit per user.
  - citrablue 8y ago
    How do you create a discipline around making good decisions for business strategy? I've seen "highest paid person" win, I've seen "Most vocal" win, I've seen "5 versus 1" win. The testing advocates I've worked with have an attitude of "I don't know what will work - that's why we're testing". I have not seem much of that attitude from most other individuals/teams involved in decision making -- they have preferred to say "this will work", and get angry when I say, "compared to what? how do you know?" edit - fwiw, I totally agree with understanding the limitations of testing. You have to know what it can be good for. My argument is that it's actually better for long term strategy than most other decision making processes I have experienced, which usually boils down to "gut based".
- s-shellfish 8y ago
  Listen to people about their complaints and preferences as intellectual equals.
  - citrablue 8y ago
    Sure, but not everyone's opinion is equally valid. I've worked with a number of design teams who confuse their opinion (an educated opinion, which I respect) as a fact. Facts beat opinions -- if a short-term optimization will be a long term loser, then prove it. Unfortunately, it's not really possible to prove/disprove an opinion on what a design element will do. Additionally, optimization testing allows you to try out different options. Going with someone's (educated) gut feel is a worse decision than exploring the option space. Ideally, you'd test everything. In the real world, resource constraints apply, so at issue is how to establish an internal discipline where facts trump opinions, and opinions can be tested then either discarded or confirmed.
Mirioron 8y ago
The last point you mentioned is why I stopped visiting Ars. I can even live with the articles themselves being slanted, but the comments section seemed even more heavily slanted.
mygo 8y ago
This has less to do with the use of A/B testing than it does the experimental design. What are they testing for in their metrics? Is it just click rates? Well that’s a patently losing strategy in the long term. What if whether or not you came back over time is what was being measured? Would that break some cardinal rule that all A/B tests must be done within a short time period? Researchers has been doing A/B testing for centuries. Till now they’ve just called it having a control variable and an independent variable. Experimental design is a big deal, and something that is taught to every graduate student researcher, but I don’t see the tech blogs giving experimental design enough focus — at least not in their click-bait headlines. Sometimes people don’t design good experiments and then blame the methods.
- RA_Fisher 8y ago
  Bingo! "A/B testing" aka completely randomized design: https://www.itl.nist.gov/div898/handbook/pri/section3/pri331.htm https://www.itl.nist.gov/div898/handbook/pri/section3/pri331... Mountains can be moved with randomized control trials.
Jommi 8y ago
That's so true! Too many companies just optimize for short term instead of long term. Yet, on the other hand it's completely natural: Not every founder wants to stay at their company for ever. Not all companies have the objective of staying alive. It's much more exciting and thrilling to chase that hockey stick growth and IPO/buyout.

smueller1234 8y ago

I think what you say is practically spot on. Yet I'd like to add that I don't think that testing frameworks (at this point it would be misleading to call them strictly A/B testing) HAVE to only reflect short term gains. It's hard to come up with (proxy) metrics that hold that kind of short term optimization in check. Just to pick one fairly obvious example for e-commerce: you can track returns/customer service contacts as a health metric or even explicitly assign a value to them to fold them into the primary metric you're optimizing. These kinds of safeguards typically need longer recording periods, so it's potentially a lot of data science and engineering effort to build tools that can handle the long term data collection and analysis. But it's not impossible. It's rather something people love to pretend is not a problem.

vintagedave 8y ago

> I've seen this repeatedly with ArsTechnica Can you expand? I am an Ars reader, and I too find it frustrating compared to what it used to be. I wish there were more in-depth technical articles; I find it too light on details, written for a non-tech audience. I really, really wish it had solid technical content, since it's what got me reading it. But I don't characterise it as click-bait (it seems clear) nor as especially political, except in so far as politics intersects technology and climate. In both those it seems balanced or erring towards freedom.

kolpa 8y ago
Politics and a Climate and Freedom are interesting, but not what makes Ars Technica interesting.

swied 8y ago

So true about the danger of focusing on short term gains. I think that it is a problem of picking the wrong things to optimize and then sampling on the wrong entities. Conversion rate is often the most commonly used target, but, it is short sighted. In the big picture a company is really just trying to maximize profit. Note that you can't use traditional A/B testing tools on a value like total profit. You need an average if you are going to run an A/B test. For conversion rates your average is conversions per user. In this case you sample on the denominator -- the users. What do you sample on if you are testing total profit per quarter? This is the ultimate conundrum for websites who use A/B testing. They don't understand how they might actually get more new/returning users by simply improving the user experience. Instead, they run marketing campaigns to drive growth and then squeeze every dollar out of the customers when they arrive. This is not a good long term strategy. I have reviewed hundreds of A/B tests over the years. The only ones that I think succeeded were the ones that focused on improving the users experience.

s-shellfish 8y ago
User experience analysis is always going to require a human mind to synthesize the material. The problem is that automation eventually gets noticed by users, thus automation implies to a user 'the programmers/whoever believes my mind and preferences can be automated'. That's always going to throw a wrench into automation. So yes, it's hard work that requires real thinking with a clear focus on the users and how the users feel about the software. Software needs to care about users if it is dependent on users for it's business model!

mockingbirdy 8y ago

Winning the game but losing the meta-game.

tehlike 8y ago

Not entirely true. You can setup your experiment to be user keyed (as it should be) and can run it for a long time, or can run the hold back for a long time Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.

kolpa 8y ago
Long-term A/B testing has major problems in the modern world where users can communicate with each other (both onsite and out of band) and can use multiple accounts / devices / browsers
- tehlike 8y ago
  communicating user is less of a concern, cross device _is_ a real concern. In some cases it's minimized with user logins, though.

citrablue 8y ago

What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27 http://dilbert.com/strip/2014-10-27

swied 8y ago
A/B testing is a great tool for incrementally improving upon a product design given the correct business strategy. Just don't let the limitations of A/B testing drive how you choose your business strategy. The strategy is ultimately about maximizing total profit, and not necessarily short term profit per user.
- citrablue 8y ago
  How do you create a discipline around making good decisions for business strategy? I've seen "highest paid person" win, I've seen "Most vocal" win, I've seen "5 versus 1" win. The testing advocates I've worked with have an attitude of "I don't know what will work - that's why we're testing". I have not seem much of that attitude from most other individuals/teams involved in decision making -- they have preferred to say "this will work", and get angry when I say, "compared to what? how do you know?" edit - fwiw, I totally agree with understanding the limitations of testing. You have to know what it can be good for. My argument is that it's actually better for long term strategy than most other decision making processes I have experienced, which usually boils down to "gut based".
s-shellfish 8y ago
Listen to people about their complaints and preferences as intellectual equals.
- citrablue 8y ago
  Sure, but not everyone's opinion is equally valid. I've worked with a number of design teams who confuse their opinion (an educated opinion, which I respect) as a fact. Facts beat opinions -- if a short-term optimization will be a long term loser, then prove it. Unfortunately, it's not really possible to prove/disprove an opinion on what a design element will do. Additionally, optimization testing allows you to try out different options. Going with someone's (educated) gut feel is a worse decision than exploring the option space. Ideally, you'd test everything. In the real world, resource constraints apply, so at issue is how to establish an internal discipline where facts trump opinions, and opinions can be tested then either discarded or confirmed.

Mirioron 8y ago

The last point you mentioned is why I stopped visiting Ars. I can even live with the articles themselves being slanted, but the comments section seemed even more heavily slanted.

mygo 8y ago

This has less to do with the use of A/B testing than it does the experimental design. What are they testing for in their metrics? Is it just click rates? Well that’s a patently losing strategy in the long term. What if whether or not you came back over time is what was being measured? Would that break some cardinal rule that all A/B tests must be done within a short time period? Researchers has been doing A/B testing for centuries. Till now they’ve just called it having a control variable and an independent variable. Experimental design is a big deal, and something that is taught to every graduate student researcher, but I don’t see the tech blogs giving experimental design enough focus — at least not in their click-bait headlines. Sometimes people don’t design good experiments and then blame the methods.

RA_Fisher 8y ago
Bingo! "A/B testing" aka completely randomized design: https://www.itl.nist.gov/div898/handbook/pri/section3/pri331.htm https://www.itl.nist.gov/div898/handbook/pri/section3/pri331... Mountains can be moved with randomized control trials.

Jommi 8y ago

That's so true! Too many companies just optimize for short term instead of long term. Yet, on the other hand it's completely natural: Not every founder wants to stay at their company for ever. Not all companies have the objective of staying alive. It's much more exciting and thrilling to chase that hockey stick growth and IPO/buyout.