12 ms·
"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much
by costcofries 4y ago
"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. "
Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive more value from the product. Bad A/B testing tries to make things more discoverable, where discoverability is often just noise and distractions. Good A/B testing ensures that the money machine, as you put it, pays its dues to users by making the product experience delightful.
- loriverkutya 4y agoI never seen an A/B test where the goal was not to maximise profit for the company.
- dotecoffee 4y agoHave you ever seen "you've already bought this" on Amazon? Highly tested, reduced sales. Overruled by judgement. Stayed on the site. AB test results don't mean you leave your judgement behind.
- iudqnolq 4y agoMaybe. I find that very useful when I'm trying to buy the same thing again. It acts as a trust marker. I do wish they kept better historic purchase data. I assume it's deleted absurdly quickly for compliance?
- account42 4y agoHuh, Amazon keeps years of order history for me and you can't even remove it, just "hide" orders which will then still be available just with an extra step. And I'm in the EU so no idea where else they would have more compliance problems with that.
- costcofries 4y agoThat's probably intentional and a sign that it was a well designed A/B test :)
- koheripbal 4y agoThis comment is way too broad and cynical. It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.
- ThalesX 4y agoWhat would the other advantages of A|B testing be for a corporation instead of widening the funnel, increasting lifetime revenue or other such bottom-line focused goals?
- throwaway98797 4y agoyou can genuinely find out what message makes more sense. explaining what your product does is hard when you’re the product expert. A/b testing ad copy helps folks parse what your product does. If your product is actually good then having more people understand what it is, is also good.
- dvtrn 4y agoAnd all of that amounts to what if no one decides to cough up money to use your product?
- notriddle 4y agoAssuming you aren’t running a charity, increasing the bottom line is always your goal. The difference between a company that burns user goodwill in the process, and one that doesn’t, is the difference between short-term and long-term thinking. The downside with hired testers is that they are unlikely to be a statistically representative sample of your target market. I don’t think this is actually a problem for most startups or new products being launched by major players, since a lot of UX issues early in product development will be obvious to anyone who isn’t the original developers, and the number of actual users you have won’t be big enough to serve as a statistically representative sample of your target market anyway.
- pessimizer 4y ago
- mattnewton 4y agoOver what time horizon? You should optimize short term metrics (for feasibility) that are (as best you can approximate) the cause of long term profits. The profits part is what makes it the business, using metrics for decisions are what makes it an A/B test. I think where people run into trouble is when they are optimizing only for this quarter’s revenue directly because long term value is too hard to measure.
- bentcorner 4y agoWhat would an A/B test where the goal was not to maximize profit even look like? The very act of creating an A/B test is because the worker wants to improve something in search of higher profits.
- adrr 4y agoIsn't that any goal of an organization? Everything ladders up to the success of the company. Especially if you're a corporation where you have fiduciary responsibility to the shareholders. AB testing is beyond UI. It could be AB testing algorithms for recommendations or search results. When i worked for a CPG company, we would AB test the sample sizes in the boxes. To answer the question what is right amount of product for a sample. We would test shipping to speed, was it worth the upgrade to expedited shipping in terms of LTV to justify the extra cost.
- mcrad 4y agoOh yeah? I've never seen a marketing department with a decent understanding of microeconomics. Most in fact are trying to maximize budget ie. expense.
- drewcoo 4y ago"You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong." How many tech startup patterns fit that? That's a sign that either the pattern does not generalize well or it's snake oil.
- costcofries 4y agoI think of it as validating and improving imperfect user assumptions.
- thatguy0900 4y agoThis reads exactly like parody corporatespeak jargon.
- Lutger 4y agoI don't think so. A/B testing is just not a cure-all and alternative for good design, that is all. I mean, if the pattern is 'don't bother thinking too much about ux, just A/B test everything in PROD to death', then yeah, you are right. However, there are many products which allow users to enter into a beta or test group, where they are the willing subjects to their experiments (in exchange for the latest shiny new stuff). This has the aspect of consent, leaving the 'stable ring' free of such variability. The fact that google and many startups are not using such consent and offering stability, doesn't mean it can't be done or isn't done.
- 4y ago
- DiggyJohnson 4y agoI’m not sure this addresses the core criticism of using your users as a testbed.
- koheripbal 4y agoDo people honestly think that Apple doesn't do A/B testing?
- teakettle42 4y agoApple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.
- mrkstu 4y agoYep- it does this in betas, but not in production apps.
- selfhoster11 4y agoIt's almost like they use a dedicated focus/beta testing group or something, instead of making all their users join in.
- daniel_reetz 4y agoI can't think of an Apple engineer who would make such a definitive statement on behalf of the company, in public. I also can't think of anyone working there who would have definitive knowledge across all apps and software, because disclosure would prevent such knowledge. Apple introduced A/B testing to Testflight in 2017 and more A/B testing to the App store this year. Source: I worked at Apple, but not in software.
- deleted 4y ago[deleted]
- izacus 4y agoGreat, but what exactly does that mean for this conversation and how users percieve Apple products?
- svnpenn 4y ago> Your perspective is extremely short-sighted no, yours is. if some company wants to do some testing, they SHOULD PAY users for that. A/B testing is just exploiting users to get free testing.
- NyxWulf 4y agoA/B testing is a tool, and as Deming said, the aim defines the system. In your definition of Good, you are defining the aim. I've been in a similar situation, where I created a relatively sophisticated A/* testing and control system. My idea of good use of the system ended up being very different from how the team employing the system thought about it. I believe that is part of the point of the post, that unintended, and even unimagined side effects plague even the best of ideas.
- jcelerier 4y ago> Your perspective is extremely short-sighted. it's the perspective of the normal users. every time i'm using a website and it does not behave exactly the same than for other people or I notice some AB testing, in my head it goes "who the fuck these people think they are?". The computing experience must be consistent and repeatable. If I wanted something that can change depending on the current position of the stars I'd ask another human, not a computer. A/B testing can result in this type of behaviour but that's just poor A/B testing.
- manmal 4y agoUX feedback is usually retrieved via user interviews. I personally have never heard from a product person „Let’s A/B test whether this is delightful“. And I think that’s because delightfulness or satisfaction is impossible to quantify in A/B tests. You only get to measure things like engagement, signup rates, retention etc. - cold hard taps on the screen, and no more. And I must say that I‘m glad that, right now, apps can’t just scan my face (or cortisol levels, or pheromones or…) for emotional clues while I read their pesky push notifications that want to coax me back into their daily active user base.
- mgraczyk 4y agoThis is not really true. You can pop up surveys in the app or on the site. They can be super non-intrusive. For small effects and large user bases, you simply cannot get useful information from UX studies.
- saagarjha 4y agoWell, then, the reason why people hate A/B testing is because everyone does it wrong. People can write fast software in Python too but the reason it’s known as being a slow language is because it invariably gets used differently.
- civilized 4y agoYou're missing GP's point. This is their experience with A/B testing as practiced. It's not about whether A/B testing could theoretically be great.
- 8note 4y agoA/B testing is always going to be distracting to your users though. Sometimes they'll be in A, and sometimes in B. The button moves or disappears or appears. Your user does not get an experience they can rely on