5 ms·
And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less
by londons_explore 4y ago
And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less productive.
A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience.
Multiply that by the number of people in the world and the number of products people use, and A/B testing is really up there as possibly one of the most beneficial ideas ever.
I really don't understand those who claim it should be banned - I see no way that testing two different versions of a website with people who desire to use that website can bring sufficient harm to outweigh those massive benefits.
- fragmede 4y agoThe specific test that did it for me, is that Facebook ran this experiment where they logged users out and then wouldn't let them log them back in despite the correct password, just to toy with them to see how long/hard they would keep trying to login, in order to see how addicted they were to Facebook. I was in the "B" group, and felt so humiliated at how many times I tried to reset my password to get into Facebook.
- andromeduck 4y agoI had this happen to me I think, and another case where half my links/screenshots would randomly get censored. Not just like random blogs/tweets/forums but even to super mainstream stuff like wikipedia/cnn/bbc.
- rjbwork 4y agoWow. That is insanely user hostile and borderline gaslighting/psychological torture. That is truly one of the most insane experiments I've ever hard of someone running.
- anthonypasq 4y agoi agree, Zuck should be put on trial for violating the Geneva convention because some people couldnt log into facebook.
- anonomousename 4y agoIs there any evidence of this? Especially for people that use a password manager, this seems incredibly stupid on Facebooks side.
- strulovich 4y agoNever heard of this one. Any articles about it?
- warkdarrior 4y agoThe GP forgot their password, couldn't get back in, and made up a story to mask their shame.
- fragmede 4y agoIt was a footnote in the wake of the main psychological experiments facebook ran on its users back in 2014*, which is overshadowing my searches for this particular detail. * https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/?sh=65846fca197c https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook...
- barrenko 4y agoOthers have done it after as well. Hell, it may not have originated at FB.
- kuramitropolis 4y agoI still get that sometimes - I think someone left it on for all the Tor nodes in Germany...
- techsplooge 4y ago
- ulchar 4y agoCould you give an example of the insights you gained by running this experiment?
- techsplooge 4y agoWe can look at the different demographics of people who tried to log in less than others, and try determine why they aren't as hooked as others, and work to rectify by improving their experience!
- rrradical 4y agoCertainly an interesting contribution to the world.
- brundolf 4y agoI'm sorry, you could make this case about some kinds of telemetry, but specifically not A/B testing. Speaking from work experience: A/B testing doesn't look into the nuances of usability or productivity, it looks at easy-to-quantify metrics like conversion rates and money spent. These metrics rarely align with a better experience for the user (outside of like, prettier buttons and stuff), and instead tend to result in less-informative, less-agentic software (information and choice often distract from conversions!)
- andrewmutz 4y agoThat's not true. I've done A/B tests in business software on how long it takes the user to get their job done on a data-intensive form. It's a great tool, and its impact all depends on how you use it.
- brundolf 4y agoI'm having a difficult time imagining a situation where people's actual productivity using a piece of software can be so easily measured. I'm sure it happens, but I think it's safe to say this is the exception to the rule when it comes to A/B testing
- andrewmutz 4y agoData entry
- scott_w 4y agoYou can measure the time between two key actions that operate as a proxy for task completion.
- Retric 4y agoThere are much better methodologies for speeding up worker productivity than A/B testing. A/B testing is designed to extract information from people you can’t do more complicated tests such as eye tracking or motion studies with. The major issue with A/B testing in the workplace is it causes confusion and slows people down when you change things. Which makes these tests really expensive even if they are seemingly easy to preform. So, I would call it useful but flawed.
- TeMPOraL 4y agoIt's a matter of degree and kind. There's also a neat sleight of hand here. Your inventor of the wheel surely tested multiple variants to optimize for the utility of his invention to the user. The A/B testing that's problematic is about optimizing taking advantage of the user. That doesn't lead to better experience, but the opposite. This is what's increasingly popular, and this is what people complain about or want to see banned. Related: attention economy is predicated on bad user experience, because it makes money from friction.
- r00fus 4y ago> And without A/B testing, every product you use would be worse. The primary goal of A/B testing is to see what's more profitable. If that happens to result in better UI that's a side effect. In fact, it could result in less usability (relevant to this conversation, it probably resulted in the frustrating "algorithm-based" timeline at FB/Twitter/etc).
- londons_explore 4y agoThe hexagonal wheel probably wasn't very profitable either.
- milosmns 4y ago> > The primary goal of A/B testing is to see what's more profitable. Well, I think I'll provide a disagreeing opinion. :) I assume this opinion probably comes from your past experiences, and I believe it is true in many cases. Since I'm not American and have never worked in an American corporate environment, I can't say what is true over there... but my experience in EU and Canada with A/A/B, A/B/C and typical A/B testing (as well as building such testing tools for others) was not like that. For example, when building tutorials for users, profitability is far from being the primary objective. Same goes for building documentation, programming languages, open-source software, internal tooling and other such things. Of course, I get that in the end, profitability is the primary goal of the company (with some exceptions). But I maintain that not all A/B tests have profitability as their primary goal, which makes the previous statement an incorrect generalization IMO.
- blarghyblarg 4y agoah yes, the good old "Let's A/B test our capability to influence emotional states using the news feeds." Totally the same as A/B testing button placement. Totally.
- vippy 4y ago10000% this.
- xzlzx 4y agoA/B testing crosses a fundamental line when people are the product.
- SahAssar 4y ago> A/B testing isn't a new thing - I'm sure the inventor of the wheel experimented with different shapes, and the buyer of the hexagonal wheel probably didn't have the best user experience. Consider yourself that inventor, would you A/B test hexagonal vs round?
- andsoitis 4y agoI could have hypotheses around wheel dimensions (e.g. width, diameter), materials, etc. that are absolutely great targets for A/B testing.
- SahAssar 4y agoThat sounds like a clear no, you already know the answer to the hypothesized A/B test. As for the other aspects they sound like great targets for testing within different use-cases, but I'm not sure why that'd be an A/B test as we think of them now.
- NaturalPhallacy 4y ago>And without A/B testing, every product you use would be worse. Worse for whom? I feel like a lot of the A/B testing results in more revenue, a more addictive app, and less user satisfaction, because they're not testing for anything beneficial to the user, because at least with FB, you're not the customer, their advertisers are the customer.
- sciclaw 4y agoAgreed. A/B testing helps you meet a desired goal. The desired goal is where ethical questions come in. For example, I have used A/B testing to see find ways to help users get a task done with fewer clicks, saving them time.
- ouid 4y agoThis is so wrong in its conclusion, that its hard to know where to start. First, we should be clear that we are talking about involuntary, undisclosed A/B testing. I have not experienced a product become better for the user as a result of involuntary A/B testing in my entire adult life. Producers and consumer have both an adversarial relationship and a mutually beneficial relationship, and the distinction between these two is essentially the split between voluntary A/B tests and involuntary ones. In the adversarial component, the producer is trying to figure out how to extract more money from the consumer, without improving the product. Alternatively, (and equivalently), how to make the product cheaper, but also worse, in a way that yhe customer doesnt notice (with their wallet). A proactive version of the "market for lemons". For instance, if you A/B test your cancellation process to minimize the number of people who cancel their subscriptions, you will almost certainly do something that makes you some additional money, and is also unambiguously evil. Any A/B testing that is mutual benefit to consumers and producers can be done with consent, by volunteers. And the miniscule amount of scientific rigor you would lose by doing so is not worth the tremendous sacrifice we have seen in quality of consumables in the past 2 decades (probably longer, but i do not have the personal experience to go longer) You might be compelled to describe involuntary A/B testing as a strategy for maximizing evil subject to the constraint that it be legal, but it often dips its toes into seeing what is illegal but still profitable, and is capable of fundamentally undermining our legal system and even our political system. The technology has grown more powerful. The addition of computers that can optimize essentially arbitrary objective functions has serious existential implications for humanity. A blanket ban on the practice, incurring the total dissolution of any corporate entity found guilty of the practice of involuntary A/B testing, would be a start.
- dlkf 4y ago> I have not experienced a product become better for the user as a result of involuntary A/B testing in my entire adult life. If you did, how would you know?
- ouid 4y agowell, for starters, there would have to have been a product that improved at all. Those are already rare enough that I can enumerate them, and in each of those instances involuntary A/B testing can be ruled out for other reasons. When craigslist added the map that shows you where all of the people are offering the thing you are interested in. That was a very good change, but thats pretty far from how craigslist operates. When dominos stopped serving hot glue on cardboard, its pretty easy to see how that didnt come about by furtive A/B testing. They were pretty confident people would like the new pizza more than the old pizza. So they told them about it. Boy did that work for dominos. That actually speaks more generally to my point. If you're making a change that you think people will like, you tell them about it, because even if it turns out that they dont like it more, the fact that they thought they would and you did it generates quite a lot of good will for them.
- AnIdiotOnTheNet 4y ago> And without A/B testing, every product you use would be worse. Not only would it be less profitable, but it would also be harder to use, less useful, and less productive. Do we live in the same universe? As far as I can tell, software keeps trending worse. Usability is terrible, options and settings keep getting moved around and hidden, software is less responsive than it used to be...
- londons_explore 4y agoYou are free to keep using MS-DOS for your computing needs... But the majority of people choose to use a modern computer, presumably because they find it overall more useful than their old MS DOS computer and software. Sure - there are gripes, but they must be outweighed by something pretty big for 99.99% of people to choose a new computer over a 30 year old one.
- epolanski 4y ago> And without A/B testing, every product you use would be worse Imagine thinking that seriously.
- dlkf 4y agoIt’s a completely ubiquitous practice. So either it does generally help, or every software company ever doesn’t know what they are doing. To me the latter view is the one that’s hard to take seriously.
- joehendrey 4y agoUX is orthogonal to the thing most web-based software companies optimise for. Unless users are paying for the service, what the company cares about is user engagement, not user experience. It's not that every software company ever doesn't know what they're doing, it's just that what they're doing is _at best_ tangentially related to improving UX. It is clearly not in the user's best interest that they feel compelled to check Facebook frequently throughout the day, or spend hours scrolling through their feed. You can spin that as the company just improving the experience so that people want to keep using it, but fundamentally it is intentional psychological manipulation. Within a university, research with human subjects is required to pass an ethical review before it is allowed to proceed. Given the scale and impact of the research conducted by Facebook on its users, it is entirely reasonable to hold them to the same standard.
- dlkf 4y ago> what they're doing is _at best_ tangentially related to improving UX . . . You can spin that as the company just improving the experience so that people want to keep using it, but fundamentally it is intentional psychological manipulation. So we agree that ab testing is good for optimizing toward a numerical objective. You then seem to think that either: A) There are simply no numerical objectives that correlate with good ux or B) Every software company ever is optimizing toward perverse incentives by which they take more money from their users while making their products worse It’s probably B that you believe, and this is such a myopic and paternalistic view. There are a couple cases where it’s a problem, eg cancellation flows. But this problem is orthogonal to AB testing (try cancelling your newspaper subscription in 1994). AB testing is mostly just trying to improve the rate at which people sign up or buy something, and in this case, your objection hinges on the hidden premise that people are idiots. > Within a university, research with human subjects is required to pass an ethical review before it is allowed to proceed. Given the scale and impact of the research conducted by Facebook on its users, it is entirely reasonable to hold them to the same standard. One man’s ponens is another’s tollens. See: https://slatestarcodex.com/2017/08/29/my-irb-nightmare/ https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
- bumby 4y agoThere's a reason Tristan Harris called upon SV to avoid "A/B testing ourselves into the 'gradient descent of mankind'". My qualms are not so much with the method as the morals that guide it. It's agnostic but when operationalized in a faulty moral framework can definitely lead to bad results.