10 ms·

Annoying A/B testing mistakes

292 points by Twixes 3y ago

londons_explore 3y ago
I want an A/B test framework that automatically optimizes the size of the groups to maximize revenue. At first, it would pick say a 50/50 split. Then as data rolls in that shows group A is more likely to convert, shift more users over to group A. Keep a few users on B to keep gathering data. Eventually, when enough data has come in, it might turn out that flow A doesn't work at all for users in France - so the ideal would be for most users in France to end up in group B, whereas the rest of the world is in group A. I want the framework to do all this behind the scenes - and preferably with statistical rigorousness. And then to tell me which groups have diminished to near zero (allowing me to remove the associated code).
- iLoveOncall 3y ago
  That sounds like you don't want A/B testing at all.
  - londons_explore 3y ago
    Indeed - I really want A/B testing combined with conversion optimization.
  - hotstickyballs 3y ago
    Exactly. That just sounds like a Bayesian update
    - NavinF 3y ago
      It's https://en.wikipedia.org/wiki/Multi-armed_bandit https://en.wikipedia.org/wiki/Multi-armed_bandit
  - bobsmooth 3y ago
    Why can't users just tell me what works!
- jdwyah 3y ago
  Get yourself a multi arm bandit and some Thompson sampling https://engineering.ezcater.com/multi-armed-bandit-experimentation https://engineering.ezcater.com/multi-armed-bandit-experimen...
- quadrature 3y ago
  Curious about your use case. Is the idea that you wa t to optimize the conversion and then you would remove the experiment code with the winning variant ?. Or would you prefer to keep the code in and have it continuously optimize variants ?.
  - joseda-hg 3y ago
    There might be a particular situation where B might be more effective than A, and therefore should be kept, if only for that specific situation There might be a cutoff point, where maintaining B would cost more than it's worth, but that's a parameter you will have to determine for each test
  - londons_explore 3y ago
    I'd expect to be running tens of experiments at any one time. Some of those experiments might be variations in wording or colorschemes - others might be entirely different signup flows. I'd let the experiment framework decide (ie. optimize) who gets shown what. Over time, the maintenance burden of tens of experiments (and every possible user being in any combination of experiments) would exceed the benefits, so then I'd want to end some experiments, keeping just whatever variant performs best. And I'd be making new experiments with new ideas.
- withinboredom 3y ago
  Sadly, there is an issue with the Novelty Effect[1]. If you push traffic to the current winner, it probably won't validate that its the actual winner. So you may trade more conversions now, for a higher churn than you can tolerate later. For example, you run two campaigns: 1. Get my widgets, one year only 19.99! 2. Get my widgets, first year only 19.99! The first one may win, but they all cancel at the second year because they thought it was only for one year. They all leave reviews complaining that you scammed them. So, I would venture that this idea is a bad one, but sounds good on paper. [1]: https://medium.com/geekculture/the-novelty-effect-an-important-factor-to-be-aware-of-when-running-a-b-tests-c080856130d3 https://medium.com/geekculture/the-novelty-effect-an-importa... PS. A/B tests don't just provide you with evidence that one solution might be better than the other, they also provide some protection in that a number of participants will get the status-quo.
  - travisjungroth 3y ago
    > So, I would venture that this idea is a bad one, but sounds good on paper. It's a great idea, it's just vulnerable to non-stationary effects (novelty effect, seasonality, etc). But it's actually no worse than fixed time horizon testing for your example if you run the test less than a year. You A/B test that copy for a month, push everyone to A, and you're still not going to realize it's actually worse.
    - withinboredom 3y ago
      Yeah. If churn is part of the experiment, then even after you stop the a/b test for treatment, you may have to wait at least a year before you have the final results.
- deleted 3y ago
  [deleted]
- PheonixPharts 3y ago
  As others have mentioned, you're referring to Thompson sampling and plenty of testing providers offer this (and if you have any DS on staff, they'll be more than happy to implement it). My experience is that there's a good reason why this hasn't taken off: the returns for this degree of optimization are far lower than you think. I once worked with a very eager, but junior DS who thought that we should build out a massive internal framework for doing this. He didn't quite understand the math behind it, so I build him a demo to understand the basics. What we realized in running the demo under various conditions is that the total return on adding this complexity to optimization was negligible at the scale we were operating at and required much more complexity than our current set up. This pattern repeats in a lot of DS related optimization in my experience. The difference between a close guess and perfectly optimal is often surprisingly little. Many DS teams perform optimizations on business processes that yield a lower improvement in revenue than the salary of the DS that built it.
  - brookst 3y ago
    Small nit: it’s a bad idea if NPV of future returns is less than the cost. If someone making $100k/yr can produce one $50k/yr optimization that NPV’s out to $260k, it’s worth it. I suspect you meant that, just a battle I have at work a lot with people who only look at single-year returns.
  - tomfutur 3y ago
    Besides complexity, a price you pay with multi-armed bandits is that you learn less about the non-optimal options (because as your confidence grows that an option is not the best, you run fewer samples through it). It turns out the people running these experiments are often not satisfied to learn "A is better than B." They want to know "A is 7% better than B," but a MAB system will only run enough B samples to make the first statement.
throwaway084t95 3y ago
That's not Simpson's Paradox. Simpson's Paradox is when the aggregate winner is different from the winner in each element of a partition, not just some of them
- hammock 3y ago
  What it is is confounding
- robertlacok 3y ago
  Exactly. On that topic – what do you do when you observe that in your test results? What's the right way to interpret the data?
  - ssharp 3y ago
    With random bucketing happening at the global level for any test, the proper thing to do is to take any segments that show interesting (and hopefully statistically significant) results that differ from the global results and test those segments individually so the random bucketing happens at that segment level. There are two issues at play here -- one is that the sample sizes for the segments may not be high enough, the other is that the more segments you look at , the greater the probability for finding a false positive.
  - throwaway084t95 3y ago
    Let's consider an example that would be a case of Simpson's Paradox. Suppose you are A/B testing two different landing pages, and you want to know which will make more people become habitual users. You partition on whether the user adds at least one friend in their first 5 minutes on the platform. It might be that landing page A makes people who add a friend in the first 5 minutes more likely to become habitual users, and it also makes people who don't add a friend in the first 5 minutes more likely to become habitual users. But page A makes people less likely to add a friend in the first 5 minutes, and people who add a friend in the first 5 minutes are overwhelmingly more likely to become habitual users than people who don't. So, in this case at least, it seems like the aggregate statistics are most relevant, but the fact that page A is bad mainly because it makes people less likely to add a friend in the first 5 minutes is also very interesting; maybe there is some way of combining A and B to get the good qualities of each and avoid the bad qualities of both
rmetzler 3y ago
If I read the first mistake correctly, then getFeatureFlag() has the side-effect to count how often it was called and uses this to calculate the outcome of the experiment? Wow. I don't know what to say....
- pjm331 3y ago
  This is indeed the case. Have run into a few surprising things like this when implementing posthog experiments recently
- sometimes_all 3y ago
  Yeah I felt that way too. Initially I thought I wasn't sure what I was missing, since the only difference is that the order of the checks is switched, and the function will still return the same true/false in both cases. Then I thought about side effects and it felt icky.
- willsmith72 3y ago
  Yep gross...
- alsiola 3y ago
  Writing an article about developer mistakes is easier than redesigning your rubbish API though.
- xp84 3y ago
  That's how every one of these tools works, that’s the whole point of using them: you only call them when you’re going to actually show the variation to the user. If you’re running a test that modifies the homepage only, you shouldn’t be calling that decision method in, say, your global navigation code that you show everyone. Or, for instance, if your test only affects how the header looks for subscribers, you have to put an outer if statement “if subscriber“ before the “if test variation.“ How else would it correctly know exactly who saw the test?
- dyeje 3y ago
  When you call the feature flag, it’s going to put the user into one of the groups. The article is saying you don’t want to add irrelevant users (in the example, ones that had already done the action they were testing) because it’s going to skew your results.
wasmitnetzen 3y ago
Posthog is on developerdans "Ads & Tracking" blocklist[1], if you're wondering why this doesn't load. [1]: https://github.com/lightswitch05/hosts/blob/master/docs/lists/ads-and-tracking-extended.txt https://github.com/lightswitch05/hosts/blob/master/docs/list...
- RobotToaster 3y ago
  Just noticed that myself, It's also in the Adguard DNS list.
masswerk 3y ago
Ad 7) > Good hypothesis: User research showed that users are unsure of how to proceed to the checkout page. Changing the button's color will lead to more users noticing it (…) Mind that you have to prove first that this preposition is actually true. Your user research is probably exploratory, qualitative data based on a small sample. At this point, it's rather an assumption. You have to transform and test this (by quantitative means) for validity and significance. Only then you can proceed to the button-hypothesis. Otherwise, you are still testing multiple things at once, based on an unclear hypothesis, while merely assuming that part of this hypothesis is actually valid.
- arrrg 3y ago
  In practice you often cannot test that in a quantitative way. Especially since it’s about a state of mind. However, you should not dismiss qualitative results out of hand. If you do usability testing of the checkout flow with five participants and three actually verbalize the hypothesis during checkout (“hm, I’m not sure how to get to the next step here”, “I don’t see the button to continue”, after searching for 30s: “ah, there it is!” – after all of which a good moderater would also ask follow up questions to better understand why they think it was hard for them to find the way to the next step and what their expectations were) then that‘s plenty of evidence for the first part of the hypothesis, allowing you to move on to testing the second part. It would be madness to quantitatively verify the first part. A total waste of resources. To be honest: with (hypothetical) evidence as clear as that from user research I would probably skip the A/B testing and go straight to implementing a solution if the problem is obvious enough and there are best practice examples. Only if designers are unsure about whether their proposed solution to the problem actually works would I consider testing that. Also: quantitative studies are not the savior you want them to be. Especially if it’s about details in the perception of users … and that’s coming from me, a user researcher who loves to do quantitative product evaluation and isn’t even all that firm in all qualitative methods.
  - masswerk 3y ago
    You really have to be able to build samples based on the first part of the hypothesis: you should test 4 groups for a crosstab. (Also, homogeneity may be an issue.) Transitioning from qualitative to quantitative methods is really the tricky part in social research. Mind that 3/5 doesn't meet the criteria of a binary test. In statistical terms, you do know nothing, this is still random. Moreover, even if metrics are suggesting that some users are spending considerable time, you still don't know why: it's still an assumption based on a negligible sample. So, the first question should be really, how do I operationalize the variable "user is disoriented", and, what does this exactly mean. (Otherwise, you're in for spurious correlation of all sorts. I.e. you still don't know why some users display disorientations and others not. Instead of addressing the underlying issue, you rather fix this by an obtrusive button design, which may have negative impact on the other group.)
mabbo 3y ago
> The solution is to use an A/B test running time calculator to determine if you have the required statistical power to run your experiment and for how long you should run your experiment. Wouldn't it be better to have an A/B testing system that just counts how many users have been in each assignment group and end when you have the required statistical power? Time just seems like a stand in for "that should be enough", when in reality you might have a change in how many users get exposed that differs from your expectations.
- aliceryhl 3y ago
  Running the experiment until you have a specific pre-determined number of observations is okay. However, the deceptively similar scheme of running it until the results are statistical significant is not okay!
  - mreezie 3y ago
    If you want statistical significance of 1/20 and you check 20 times... you are likely to find it.
alberth 3y ago
Enough traffic. Isn’t the biggest problem with A/B testing that very few web sites even have enough traffic to properly measure statistical differences. Essentially making A/B testing for 99.9% of websites useless.
- Retric 3y ago
  A/B testing works fine even at a hundred users per day. More visitors means you can run more tests and notice smaller differences, but that’s also a lot of work which smaller sites don’t really justify.
- xp84 3y ago
  I have worked for some pretty arrogant, business types who fancy themselves “data driven“ but actually they knew nothing about statistics. What that actually meant was they forced us to run AB tests for every change, and when the tests nearly always showed no particular statistical significance, they would accept the insignificant results if it supported their agenda, or if the insignificant results were against their desired outcome, they would run the test longer until it happend to flop the other way. The whole thing was such a joke. You definitely need some very smart math people to do this in a way that isn't pointless.
- hanezz 3y ago
  This. Ron Kohavi 1) has some excellent resources on this 2). There is a lot of noise in data, that is very often misattributed to 'findings' in the context of A/B testing. Replication of A/B tests should be much more common in the CRO industry, it can lead to surprising yet sobering insights into real effects. 1) https://experimentguide.com/ https://experimentguide.com/ 2) https://bit.ly/ABTestingIntuitionBusters https://bit.ly/ABTestingIntuitionBusters
2OEH8eoCRo0 3y ago
Every engineer? Electrical engineers? Kernel developers? Defense workers? I hesitate to write this (because I don't want to be negative) but I get a sense that most software "engineers" have a very narrow view of the industry at large. Or this forum leans a particular way. I haven't A/B tested in my last three roles. Two of them were defense jobs, my current job deals with the Linux kernel.
- o1y32 3y ago
  Was going to say the same thing. Lots of articles have clickbait titles, but this one is especially bad. Even among software engineers, only a small percentage will ever do any A/B testing, not to mention that often "scientists" or other roles are in charge of designing, running and analyzing A/B test experiments.
  - altano 3y ago
    A very large percentage of product engineers at some of the biggest tech companies, eg Meta, regularly run their own experiments.
- chefandy 3y ago
  I used to get knots in my hair about these distinctions, but in retrospect, I was just being pedantic. It's a headline— not a synopsis or formal tagging system. Context makes it perfectly clear to most in a web-focused software industry crowd which "engineers" might be doing a/b testing. Also, my last three jobs haven't included a lot of stuff I read about here; why should that affect the headline?
- deleted 3y ago
  [deleted]
- jldugger 3y ago
  > Two of them were defense jobs, my current job deals with the Linux kernel. I don't work on the kernel, but one of the most professionally useful talks about the Linux kernel was an engineer talking about how to use statistical tests on perf related changes with small effects[1]. It's not an _online_ A/B technique but sometimes you pay attention to how other fields approach things in order to learn how to improve your own field. [1]: https://lca2021.linux.org.au/schedule/presentation/31/ https://lca2021.linux.org.au/schedule/presentation/31/
withinboredom 3y ago
I built an internal a/b testing platform with a team of 3-5 over the years. It needed to handle extreme load (hundreds of millions of participants in some cases). Our team also had a sister team responsible for teaching/educating teams about how to do proper a/b testing -- they also reviewed implementations/results on-demand. Most of the a/b tests they reviewed (note the survivorship bias here, they were reviewed because they were surprising results) were incorrectly implemented and had to be redone. Most companies I worked at before or since did NOT have a team like this, and blindly trusted the results without hunting for biases, incorrect implementations, bugs, or other issues.
- srveale 3y ago
  Do you know if there were common mistakes for the incorrect implementations? Were they simple mistakes or more because someone misunderstood a nuance of stats?
  - withinboredom 3y ago
    I don't remember much specifics, but IIRC, most of the implementation related ones were due to an anti-pattern from the older a/b testing framework. Basically, the client would try and determine if the user was eligible to be in the A/B test (instead of relying on the framework), then in an API handler, get the user's assignment. This would mean the UI would think the user wasn't in the A/B test at all, while the API would see the user as in the A/B test. In this case, the user would be experiencing the 'control' while the framework thought they were experiencing something else. That was a big one for awhile, and it would skew results. Hmmm, another common one was doing geographic experiments when part of the experiment couldn't be geofenced for technological reasons. Or forgetting that a user could leave a geofence and removing access the feature after they'd already been given access to it. Almost all cases boiled down to showing the user one thing while thinking we were showing them something else.
    - srveale 3y ago
      I wonder if that falls under mistake #4 from the article, or if there's another category of mistake: "Actually test what you think you're testing." Seems simple but with a big project I could see that being the hardest part.
2rsf 3y ago
Another challenge, related more to implementation than theory, is having too many experiments running in parallel. As a company grows there will be multiple experiments running in parallel executed by different teams. The underlying assumption is that they are independent, but it is not necessarily true or at least not entirely correct. For example a graphics change on the main page together with a change in the login logic. Obviously this can be solved by communication, for example documenting running experiments, but like many other aspects in AB testing there is a lot of guesswork and gut feeling involved.
- cantSpellSober 3y ago
  A better solve is E2E or unit tests to make sure A/B segments aren't conflicting. At the enterprise level there's simply too many teams testing too much to keep track of it in, say, a spreadsheet.
kimukasetsu 3y ago
The biggest mistake engineers make is determining sample sizes. It is not trivial to determine the sample size for a trial without prior knowledge of effect sizes. Instead of waiting for a fixed sample size, I would recommend using a sequential testing framework: set a stopping condition and perform a test for each new batch of sample units. This is called optional stopping and it is not possible using a classic t-test, since Type I and II errors are only valid at a determined sample size. However, other tests make it possible: see safe anytime-valid statistics [1, 2] or, simply, bayesian testing [3, 4]. [1] https://arxiv.org/abs/2210.01948 https://arxiv.org/abs/2210.01948 [2] https://arxiv.org/abs/2011.03567 https://arxiv.org/abs/2011.03567 [3] https://pubmed.ncbi.nlm.nih.gov/24659049/ https://pubmed.ncbi.nlm.nih.gov/24659049/ [4] http://doingbayesiandataanalysis.blogspot.com/2013/11/optional-stopping-in-data-collection-p.html?m=1 http://doingbayesiandataanalysis.blogspot.com/2013/11/option...
- travisjungroth 3y ago
  People often don’t determine sample sizes at all! And doing power calculations without an idea of effect size isn’t just hard but impossible. It’s one of the inputs to the formula. But at least it’s fast so you can sort of guess and check. Anytime valid inference helps with this situation, but it doesn’t solve it. If you’re trying to detect a small effect, it would be nicer to figure out you need a million samples up front versus learning that because your test with 1,000 samples a day took three years. Still, anytime is way better than fixed IMO. Fixed almost never really exists. Every A/B testing platform I’ve seen allows peeking. I work with the author of the second paper you listed. The math looks advanced, but it’s very easy to implement.
- hackernewds 3y ago
  The biggest mistake is engineers owning experimentation. They should be owned by data scientists. Realize though that is a luxury, but I also see this trend in blue chip companies
  - pbae 3y ago
    Did a data scientist write this? You don't need to be a member of a priesthood to run experiments. You just need to know what you're doing.
alsiola 3y ago
On point 7 ((Testing an unclear hypothesis), while agreeing with the overall point, I strongly disagree with the examples. > Bad Hypothesis: Changing the color of the "Proceed to checkout" button will increase purchases. This is succinct, clear, and is very clear what the variable/measure will be. > Good hypothesis: User research showed that users are unsure of how to proceed to the checkout page. Changing the button's color will lead to more users noticing it and thus more people will proceed to the checkout page. This will then lead to more purchases. > User research showed that users are unsure of how to proceed to the checkout page. Not a hypothesis, but a problem statement. Cut the fluff. > Changing the button's color will lead to more users noticing it and thus more people will proceed to the checkout page. This is now two hypotheses. > This will then lead to more purchases. Sorry I meant three hypotheses.
- kevinwang 3y ago
  It is surely helpful to have a "mechanism of action" so that you're not just blindly AB testing and falling victim to coincidences like in https://xkcd.com/882/ https://xkcd.com/882/ . Not sure if people do this, but with a mechanism of action in place you can state a prior belief and turn your AB testing results into actual posteriors instead of frequentist metrics like p-values which are kind of useless.
  - datastoat 3y ago
    That xkcd comic highlights the problem with observational (as opposed to controlled) studies. TFA is about A/B testing, i.e. controlled studies. It’s the fact that you (the investigator) is controlling the treatment assignment that allows you to draw causal conclusions. What you happen to believe about the mechanism of action doesn’t matter, at least as far as the outcome of this particular experiment is concerned. Of course, your conjectured mechanism of action is likely to matter for what you decide to investigate next. Also, frequentism / Bayesianism is orthogonal to causal / correlational interpretations.
    - carlmr 3y ago
      I think what kevinwang is getting at, is that if you A/B test with a static version A and enough versions of B, at some point you will get statistically significant results if you repeat it often enough. Having a control doesn't mean you can't fall victim to this.
methou 3y ago
Probably off-topic, but how do opt out from most of A/B testings?
deleted 3y ago
[deleted]
dbroockman 3y ago
Another one: don’t program your own AB testing framework! Every time I’ve seen engineers try to build this on their own, it fails an AA test (where both versions are the same so there should be no difference). Common reasons are overly complicated randomization schemes (keep it simple!) and differences in load times between test and control.
- giraffe_lady 3y ago
  I don't keep that up with it but it seems like the ecosystem has kind of collapsed the last few years though? Like you have optimizely and its competitors that are fully focused on huge enterprise with "call us" pricing right out the gate. VWO has a clunky & aged tech stack that was already causing problems when I used it a couple years ago and seems unchanged since then. If you're a medium-small business I see why you'd be tempted to roll your own. Trustworthy options under $15k/year are not apparent.
- HWR_14 3y ago
  Shouldn't AA tests fail a certain percentage of the time? Typically, 5% of the time?
franze 3y ago
plus, mind the Honeymoon Effect something new performs better cause its new if you have a platform with lots pf returning users this one will hit you again and again. so even if you have a winner after the test and make the change permanent, revisit it 2 months later and see if you are now really better of. all changes of a/b tests in sum has a high chance to just get an average platform in the sum of all changes.
mtlmtlmtlmtl 3y ago
Surprised no one said this yet, so I'll bite the bullet. I don't think A/B testing is a good idea at all for the long term. Seems like a recipe for having your software slowly evolved into a giant heap of dark patterns. When a metric becomes a target, it ceases to be a good metric.
- cantSpellSober 3y ago
  Good multivariate testing and (statistically significant) data doesn't do that. It shows lots of ways to improve your UX, and if your guesses at improving UX actually work. Example from TFA: > more people signed up using Google and Github, overall sign-ups didn't increase, and nor did activation Less friction on login for the user, 0 gains in conversions, they shipped it anyway. That's not a dark pattern. If you're intentionally trying to make dark patterns it will help with that too I guess; the same way a hammer can build a house, or tear it down, depending on use.
  - mtlmtlmtlmtl 3y ago
    I often see this argument, and although I can happily accept the examples given in defence as making sense, I never see an argument that this multivariate approach solves the problem in general and doesn't merely ameliorate some of the worst cases(I suppose I'm open to the idea that it could at least get it from "worse than the disease" to "actually useful in moderation"). Fundamentally, if you pick some number of metrics, you're always leaving some number of possible metrics "dark", right? Is there some objective method of deciding which metrics should be chosen, and which shouldn't?
    - cantSpellSober 3y ago
      "user trust" is a good one, abeit hard to measure Rolled out some tests to streamline cancelling subscriptions in response to user feedback, with Marketing's begrudging approval. Short term, predictably, we saw an increase in cancellations, then a decrease and eventual levelling out. Long term we continued to see an increase in subscriptions after rollout, and focused on more important questions like "how do we provide a good product that a user doesn't want to cancel?"
Sohcahtoa82 3y ago
The one mistake I assume happens too much is trying to measure "engagement". Imagine a website is testing a redesign, and they want to decide if people like it by measuring how long they spend on the site to see if it's more "engaging". But the new site makes information harder to find, so they spend more time on the site browsing and trying to find what they're looking for. Management goes, "Oh, users are delighted with the new site! Look how much time they spend on it!" not realizing how frustrated the users are.
- ravenstine 3y ago
  Engagement is my favorite form of metrics pseudoscience. A classic example is when engagement actually goes up, not because the design change is better, but because it frustrates and confuses the user, causing them to click around more and remain on the site longer. Without a focus group, there's really no way to determine whether the users are actually "delighted". EDIT: For some reason it didn't compute with me that you already referred to the same example. I've seen that exact scenario play out in real life, though.
  - Sohcahtoa82 3y ago
    I bet the reddit redesign used a similar faulty measurement of engagement. "People spent more time scrolling the feed, people must enjoy it!" No, the feed takes up more space, so now I can only fit 1 or 2 items on my screen at once, rather than 10, so I have to scroll more to see more content.
    - ravenstine 3y ago
      That would not surprise me in the least! In fact, that's exactly what happened at a company I used to work for (that shall remain nameless). At the behest of the design team, we implemented a complete redesign of our site which included changing the home page so that at most only two media items could be on-screen at a time, and the ads which used to be simple banners now were woven between the feed of items. I remember sitting in a meeting where we had A/B tested this new homepage, and witnessing some data analyst guy giving a presentation which included how "engagement in the B-group was increased by N-percent!!!" The directors of web content were awestruck by this despite no context or explanation as to why supposed "engagement" was higher with the new design. The test wasn't even carried out for a long duration of time. For all anyone knew, users were confused and spent more time clicking around because they were looking for something they were accustomed to in the original design. And no, it did not matter that I brought up my reasons for skepticism; anything that made a number increase made it into the final design. Then, we actually had focus groups, long after the point at which we should have been consulting them, and the feedback we received was overwhelmingly lukewarm or negative. Much of it vindicated my concerns the entire time; users didn't actually like scrolling. Then again, I guess if they're viewing more ads, then who cares what the user thinks?? Never have I felt more like I was living in a Dilbert comic than that time.
jedberg 3y ago
The biggest mistake engineers make about A/B testing is not recognizing local maxima. Your test may be super successful, but there may be an even better solution that's significantly different than what you've arrived at. It's important to not only A/B test minor changes, but occasionally throw in some major changes to see if it moves the same metric, possibly even more than your existing success.
jldugger 3y ago
> 6. Not accounting for seasonality Doesn't the online nature of an A/B automatically account for this?
realjohng 3y ago
Thanks for posting this. It’s to the point and easy to understand. And much needed- most companies seem to do testing without teaching the intricacies involved.
donretag 3y ago
If anyone from posthog is reading this, please fix your RSS feed. The link actually points back to the blog homepage.
- corywatilo 3y ago
  Will take a look, thanks for the heads up!
time4tea 3y ago
Annoying illegal cookie consent banner?
pil0u 3y ago
In the second table, shouldn't the mobile control conversion rate be 3.33%?
iudqnolq 3y ago
Point one seems to be an API naming issue. I would not anticipate getFeatureFlag to increment a hit counter. Seems like it should be called something like participateInFlagTest or whatever. Or maybe it should take a (key, arbitraryId) instead of just (key), use the hash of the id to determine if the flag is set, and idempotently register a hit for the id.
drpixie 3y ago
> Relying too much on A/B tests for decision-making Need I say more? Or just keep tweaking your website until it becomes a mindless, grey, sludge.

292 points by Twixes 3y ago

I want an A/B test framework that automatically optimizes the size of the groups to maximize revenue. At first, it would pick say a 50/50 split. Then as data rolls in that shows group A is more likely to convert, shift more users over to group A. Keep a few users on B to keep gathering data. Eventually, when enough data has come in, it might turn out that flow A doesn't work at all for users in France - so the ideal would be for most users in France to end up in group B, whereas the rest of the world is in group A. I want the framework to do all this behind the scenes - and preferably with statistical rigorousness. And then to tell me which groups have diminished to near zero (allowing me to remove the associated code).

iLoveOncall 3y ago
That sounds like you don't want A/B testing at all.
- londons_explore 3y ago
  Indeed - I really want A/B testing combined with conversion optimization.
- hotstickyballs 3y ago
  Exactly. That just sounds like a Bayesian update
  - NavinF 3y ago
    It's https://en.wikipedia.org/wiki/Multi-armed_bandit https://en.wikipedia.org/wiki/Multi-armed_bandit
- bobsmooth 3y ago
  Why can't users just tell me what works!
jdwyah 3y ago
Get yourself a multi arm bandit and some Thompson sampling https://engineering.ezcater.com/multi-armed-bandit-experimentation https://engineering.ezcater.com/multi-armed-bandit-experimen...
quadrature 3y ago
Curious about your use case. Is the idea that you wa t to optimize the conversion and then you would remove the experiment code with the winning variant ?. Or would you prefer to keep the code in and have it continuously optimize variants ?.
- joseda-hg 3y ago
  There might be a particular situation where B might be more effective than A, and therefore should be kept, if only for that specific situation There might be a cutoff point, where maintaining B would cost more than it's worth, but that's a parameter you will have to determine for each test
- londons_explore 3y ago
  I'd expect to be running tens of experiments at any one time. Some of those experiments might be variations in wording or colorschemes - others might be entirely different signup flows. I'd let the experiment framework decide (ie. optimize) who gets shown what. Over time, the maintenance burden of tens of experiments (and every possible user being in any combination of experiments) would exceed the benefits, so then I'd want to end some experiments, keeping just whatever variant performs best. And I'd be making new experiments with new ideas.
withinboredom 3y ago
Sadly, there is an issue with the Novelty Effect[1]. If you push traffic to the current winner, it probably won't validate that its the actual winner. So you may trade more conversions now, for a higher churn than you can tolerate later. For example, you run two campaigns: 1. Get my widgets, one year only 19.99! 2. Get my widgets, first year only 19.99! The first one may win, but they all cancel at the second year because they thought it was only for one year. They all leave reviews complaining that you scammed them. So, I would venture that this idea is a bad one, but sounds good on paper. [1]: https://medium.com/geekculture/the-novelty-effect-an-important-factor-to-be-aware-of-when-running-a-b-tests-c080856130d3 https://medium.com/geekculture/the-novelty-effect-an-importa... PS. A/B tests don't just provide you with evidence that one solution might be better than the other, they also provide some protection in that a number of participants will get the status-quo.
- travisjungroth 3y ago
  > So, I would venture that this idea is a bad one, but sounds good on paper. It's a great idea, it's just vulnerable to non-stationary effects (novelty effect, seasonality, etc). But it's actually no worse than fixed time horizon testing for your example if you run the test less than a year. You A/B test that copy for a month, push everyone to A, and you're still not going to realize it's actually worse.
  - withinboredom 3y ago
    Yeah. If churn is part of the experiment, then even after you stop the a/b test for treatment, you may have to wait at least a year before you have the final results.
deleted 3y ago
[deleted]
PheonixPharts 3y ago
As others have mentioned, you're referring to Thompson sampling and plenty of testing providers offer this (and if you have any DS on staff, they'll be more than happy to implement it). My experience is that there's a good reason why this hasn't taken off: the returns for this degree of optimization are far lower than you think. I once worked with a very eager, but junior DS who thought that we should build out a massive internal framework for doing this. He didn't quite understand the math behind it, so I build him a demo to understand the basics. What we realized in running the demo under various conditions is that the total return on adding this complexity to optimization was negligible at the scale we were operating at and required much more complexity than our current set up. This pattern repeats in a lot of DS related optimization in my experience. The difference between a close guess and perfectly optimal is often surprisingly little. Many DS teams perform optimizations on business processes that yield a lower improvement in revenue than the salary of the DS that built it.
- brookst 3y ago
  Small nit: it’s a bad idea if NPV of future returns is less than the cost. If someone making $100k/yr can produce one $50k/yr optimization that NPV’s out to $260k, it’s worth it. I suspect you meant that, just a battle I have at work a lot with people who only look at single-year returns.
- tomfutur 3y ago
  Besides complexity, a price you pay with multi-armed bandits is that you learn less about the non-optimal options (because as your confidence grows that an option is not the best, you run fewer samples through it). It turns out the people running these experiments are often not satisfied to learn "A is better than B." They want to know "A is 7% better than B," but a MAB system will only run enough B samples to make the first statement.

throwaway084t95 3y ago

That's not Simpson's Paradox. Simpson's Paradox is when the aggregate winner is different from the winner in each element of a partition, not just some of them

hammock 3y ago
What it is is confounding
robertlacok 3y ago
Exactly. On that topic – what do you do when you observe that in your test results? What's the right way to interpret the data?
- ssharp 3y ago
  With random bucketing happening at the global level for any test, the proper thing to do is to take any segments that show interesting (and hopefully statistically significant) results that differ from the global results and test those segments individually so the random bucketing happens at that segment level. There are two issues at play here -- one is that the sample sizes for the segments may not be high enough, the other is that the more segments you look at , the greater the probability for finding a false positive.
- throwaway084t95 3y ago
  Let's consider an example that would be a case of Simpson's Paradox. Suppose you are A/B testing two different landing pages, and you want to know which will make more people become habitual users. You partition on whether the user adds at least one friend in their first 5 minutes on the platform. It might be that landing page A makes people who add a friend in the first 5 minutes more likely to become habitual users, and it also makes people who don't add a friend in the first 5 minutes more likely to become habitual users. But page A makes people less likely to add a friend in the first 5 minutes, and people who add a friend in the first 5 minutes are overwhelmingly more likely to become habitual users than people who don't. So, in this case at least, it seems like the aggregate statistics are most relevant, but the fact that page A is bad mainly because it makes people less likely to add a friend in the first 5 minutes is also very interesting; maybe there is some way of combining A and B to get the good qualities of each and avoid the bad qualities of both

rmetzler 3y ago

If I read the first mistake correctly, then getFeatureFlag() has the side-effect to count how often it was called and uses this to calculate the outcome of the experiment? Wow. I don't know what to say....

pjm331 3y ago
This is indeed the case. Have run into a few surprising things like this when implementing posthog experiments recently
sometimes_all 3y ago
Yeah I felt that way too. Initially I thought I wasn't sure what I was missing, since the only difference is that the order of the checks is switched, and the function will still return the same true/false in both cases. Then I thought about side effects and it felt icky.
willsmith72 3y ago
Yep gross...
alsiola 3y ago
Writing an article about developer mistakes is easier than redesigning your rubbish API though.
xp84 3y ago
That's how every one of these tools works, that’s the whole point of using them: you only call them when you’re going to actually show the variation to the user. If you’re running a test that modifies the homepage only, you shouldn’t be calling that decision method in, say, your global navigation code that you show everyone. Or, for instance, if your test only affects how the header looks for subscribers, you have to put an outer if statement “if subscriber“ before the “if test variation.“ How else would it correctly know exactly who saw the test?
dyeje 3y ago
When you call the feature flag, it’s going to put the user into one of the groups. The article is saying you don’t want to add irrelevant users (in the example, ones that had already done the action they were testing) because it’s going to skew your results.

wasmitnetzen 3y ago

Posthog is on developerdans "Ads & Tracking" blocklist[1], if you're wondering why this doesn't load. [1]: https://github.com/lightswitch05/hosts/blob/master/docs/lists/ads-and-tracking-extended.txt https://github.com/lightswitch05/hosts/blob/master/docs/list...

RobotToaster 3y ago
Just noticed that myself, It's also in the Adguard DNS list.

masswerk 3y ago

Ad 7) > Good hypothesis: User research showed that users are unsure of how to proceed to the checkout page. Changing the button's color will lead to more users noticing it (…) Mind that you have to prove first that this preposition is actually true. Your user research is probably exploratory, qualitative data based on a small sample. At this point, it's rather an assumption. You have to transform and test this (by quantitative means) for validity and significance. Only then you can proceed to the button-hypothesis. Otherwise, you are still testing multiple things at once, based on an unclear hypothesis, while merely assuming that part of this hypothesis is actually valid.

arrrg 3y ago
In practice you often cannot test that in a quantitative way. Especially since it’s about a state of mind. However, you should not dismiss qualitative results out of hand. If you do usability testing of the checkout flow with five participants and three actually verbalize the hypothesis during checkout (“hm, I’m not sure how to get to the next step here”, “I don’t see the button to continue”, after searching for 30s: “ah, there it is!” – after all of which a good moderater would also ask follow up questions to better understand why they think it was hard for them to find the way to the next step and what their expectations were) then that‘s plenty of evidence for the first part of the hypothesis, allowing you to move on to testing the second part. It would be madness to quantitatively verify the first part. A total waste of resources. To be honest: with (hypothetical) evidence as clear as that from user research I would probably skip the A/B testing and go straight to implementing a solution if the problem is obvious enough and there are best practice examples. Only if designers are unsure about whether their proposed solution to the problem actually works would I consider testing that. Also: quantitative studies are not the savior you want them to be. Especially if it’s about details in the perception of users … and that’s coming from me, a user researcher who loves to do quantitative product evaluation and isn’t even all that firm in all qualitative methods.
- masswerk 3y ago
  You really have to be able to build samples based on the first part of the hypothesis: you should test 4 groups for a crosstab. (Also, homogeneity may be an issue.) Transitioning from qualitative to quantitative methods is really the tricky part in social research. Mind that 3/5 doesn't meet the criteria of a binary test. In statistical terms, you do know nothing, this is still random. Moreover, even if metrics are suggesting that some users are spending considerable time, you still don't know why: it's still an assumption based on a negligible sample. So, the first question should be really, how do I operationalize the variable "user is disoriented", and, what does this exactly mean. (Otherwise, you're in for spurious correlation of all sorts. I.e. you still don't know why some users display disorientations and others not. Instead of addressing the underlying issue, you rather fix this by an obtrusive button design, which may have negative impact on the other group.)

mabbo 3y ago

> The solution is to use an A/B test running time calculator to determine if you have the required statistical power to run your experiment and for how long you should run your experiment. Wouldn't it be better to have an A/B testing system that just counts how many users have been in each assignment group and end when you have the required statistical power? Time just seems like a stand in for "that should be enough", when in reality you might have a change in how many users get exposed that differs from your expectations.

aliceryhl 3y ago
Running the experiment until you have a specific pre-determined number of observations is okay. However, the deceptively similar scheme of running it until the results are statistical significant is not okay!
- mreezie 3y ago
  If you want statistical significance of 1/20 and you check 20 times... you are likely to find it.

alberth 3y ago

Enough traffic. Isn’t the biggest problem with A/B testing that very few web sites even have enough traffic to properly measure statistical differences. Essentially making A/B testing for 99.9% of websites useless.

Retric 3y ago
A/B testing works fine even at a hundred users per day. More visitors means you can run more tests and notice smaller differences, but that’s also a lot of work which smaller sites don’t really justify.
xp84 3y ago
I have worked for some pretty arrogant, business types who fancy themselves “data driven“ but actually they knew nothing about statistics. What that actually meant was they forced us to run AB tests for every change, and when the tests nearly always showed no particular statistical significance, they would accept the insignificant results if it supported their agenda, or if the insignificant results were against their desired outcome, they would run the test longer until it happend to flop the other way. The whole thing was such a joke. You definitely need some very smart math people to do this in a way that isn't pointless.
hanezz 3y ago
This. Ron Kohavi 1) has some excellent resources on this 2). There is a lot of noise in data, that is very often misattributed to 'findings' in the context of A/B testing. Replication of A/B tests should be much more common in the CRO industry, it can lead to surprising yet sobering insights into real effects. 1) https://experimentguide.com/ https://experimentguide.com/ 2) https://bit.ly/ABTestingIntuitionBusters https://bit.ly/ABTestingIntuitionBusters

2OEH8eoCRo0 3y ago

Every engineer? Electrical engineers? Kernel developers? Defense workers? I hesitate to write this (because I don't want to be negative) but I get a sense that most software "engineers" have a very narrow view of the industry at large. Or this forum leans a particular way. I haven't A/B tested in my last three roles. Two of them were defense jobs, my current job deals with the Linux kernel.

o1y32 3y ago
Was going to say the same thing. Lots of articles have clickbait titles, but this one is especially bad. Even among software engineers, only a small percentage will ever do any A/B testing, not to mention that often "scientists" or other roles are in charge of designing, running and analyzing A/B test experiments.
- altano 3y ago
  A very large percentage of product engineers at some of the biggest tech companies, eg Meta, regularly run their own experiments.
chefandy 3y ago
I used to get knots in my hair about these distinctions, but in retrospect, I was just being pedantic. It's a headline— not a synopsis or formal tagging system. Context makes it perfectly clear to most in a web-focused software industry crowd which "engineers" might be doing a/b testing. Also, my last three jobs haven't included a lot of stuff I read about here; why should that affect the headline?
deleted 3y ago
[deleted]
jldugger 3y ago
> Two of them were defense jobs, my current job deals with the Linux kernel. I don't work on the kernel, but one of the most professionally useful talks about the Linux kernel was an engineer talking about how to use statistical tests on perf related changes with small effects[1]. It's not an _online_ A/B technique but sometimes you pay attention to how other fields approach things in order to learn how to improve your own field. [1]: https://lca2021.linux.org.au/schedule/presentation/31/ https://lca2021.linux.org.au/schedule/presentation/31/

withinboredom 3y ago

I built an internal a/b testing platform with a team of 3-5 over the years. It needed to handle extreme load (hundreds of millions of participants in some cases). Our team also had a sister team responsible for teaching/educating teams about how to do proper a/b testing -- they also reviewed implementations/results on-demand. Most of the a/b tests they reviewed (note the survivorship bias here, they were reviewed because they were surprising results) were incorrectly implemented and had to be redone. Most companies I worked at before or since did NOT have a team like this, and blindly trusted the results without hunting for biases, incorrect implementations, bugs, or other issues.

srveale 3y ago
Do you know if there were common mistakes for the incorrect implementations? Were they simple mistakes or more because someone misunderstood a nuance of stats?
- withinboredom 3y ago
  I don't remember much specifics, but IIRC, most of the implementation related ones were due to an anti-pattern from the older a/b testing framework. Basically, the client would try and determine if the user was eligible to be in the A/B test (instead of relying on the framework), then in an API handler, get the user's assignment. This would mean the UI would think the user wasn't in the A/B test at all, while the API would see the user as in the A/B test. In this case, the user would be experiencing the 'control' while the framework thought they were experiencing something else. That was a big one for awhile, and it would skew results. Hmmm, another common one was doing geographic experiments when part of the experiment couldn't be geofenced for technological reasons. Or forgetting that a user could leave a geofence and removing access the feature after they'd already been given access to it. Almost all cases boiled down to showing the user one thing while thinking we were showing them something else.
  - srveale 3y ago
    I wonder if that falls under mistake #4 from the article, or if there's another category of mistake: "Actually test what you think you're testing." Seems simple but with a big project I could see that being the hardest part.

2rsf 3y ago

Another challenge, related more to implementation than theory, is having too many experiments running in parallel. As a company grows there will be multiple experiments running in parallel executed by different teams. The underlying assumption is that they are independent, but it is not necessarily true or at least not entirely correct. For example a graphics change on the main page together with a change in the login logic. Obviously this can be solved by communication, for example documenting running experiments, but like many other aspects in AB testing there is a lot of guesswork and gut feeling involved.

cantSpellSober 3y ago
A better solve is E2E or unit tests to make sure A/B segments aren't conflicting. At the enterprise level there's simply too many teams testing too much to keep track of it in, say, a spreadsheet.

kimukasetsu 3y ago

The biggest mistake engineers make is determining sample sizes. It is not trivial to determine the sample size for a trial without prior knowledge of effect sizes. Instead of waiting for a fixed sample size, I would recommend using a sequential testing framework: set a stopping condition and perform a test for each new batch of sample units. This is called optional stopping and it is not possible using a classic t-test, since Type I and II errors are only valid at a determined sample size. However, other tests make it possible: see safe anytime-valid statistics [1, 2] or, simply, bayesian testing [3, 4]. [1] https://arxiv.org/abs/2210.01948 https://arxiv.org/abs/2210.01948 [2] https://arxiv.org/abs/2011.03567 https://arxiv.org/abs/2011.03567 [3] https://pubmed.ncbi.nlm.nih.gov/24659049/ https://pubmed.ncbi.nlm.nih.gov/24659049/ [4] http://doingbayesiandataanalysis.blogspot.com/2013/11/optional-stopping-in-data-collection-p.html?m=1 http://doingbayesiandataanalysis.blogspot.com/2013/11/option...

travisjungroth 3y ago
People often don’t determine sample sizes at all! And doing power calculations without an idea of effect size isn’t just hard but impossible. It’s one of the inputs to the formula. But at least it’s fast so you can sort of guess and check. Anytime valid inference helps with this situation, but it doesn’t solve it. If you’re trying to detect a small effect, it would be nicer to figure out you need a million samples up front versus learning that because your test with 1,000 samples a day took three years. Still, anytime is way better than fixed IMO. Fixed almost never really exists. Every A/B testing platform I’ve seen allows peeking. I work with the author of the second paper you listed. The math looks advanced, but it’s very easy to implement.
hackernewds 3y ago
The biggest mistake is engineers owning experimentation. They should be owned by data scientists. Realize though that is a luxury, but I also see this trend in blue chip companies
- pbae 3y ago
  Did a data scientist write this? You don't need to be a member of a priesthood to run experiments. You just need to know what you're doing.

alsiola 3y ago

On point 7 ((Testing an unclear hypothesis), while agreeing with the overall point, I strongly disagree with the examples. > Bad Hypothesis: Changing the color of the "Proceed to checkout" button will increase purchases. This is succinct, clear, and is very clear what the variable/measure will be. > Good hypothesis: User research showed that users are unsure of how to proceed to the checkout page. Changing the button's color will lead to more users noticing it and thus more people will proceed to the checkout page. This will then lead to more purchases. > User research showed that users are unsure of how to proceed to the checkout page. Not a hypothesis, but a problem statement. Cut the fluff. > Changing the button's color will lead to more users noticing it and thus more people will proceed to the checkout page. This is now two hypotheses. > This will then lead to more purchases. Sorry I meant three hypotheses.

kevinwang 3y ago
It is surely helpful to have a "mechanism of action" so that you're not just blindly AB testing and falling victim to coincidences like in https://xkcd.com/882/ https://xkcd.com/882/ . Not sure if people do this, but with a mechanism of action in place you can state a prior belief and turn your AB testing results into actual posteriors instead of frequentist metrics like p-values which are kind of useless.
- datastoat 3y ago
  That xkcd comic highlights the problem with observational (as opposed to controlled) studies. TFA is about A/B testing, i.e. controlled studies. It’s the fact that you (the investigator) is controlling the treatment assignment that allows you to draw causal conclusions. What you happen to believe about the mechanism of action doesn’t matter, at least as far as the outcome of this particular experiment is concerned. Of course, your conjectured mechanism of action is likely to matter for what you decide to investigate next. Also, frequentism / Bayesianism is orthogonal to causal / correlational interpretations.
  - carlmr 3y ago
    I think what kevinwang is getting at, is that if you A/B test with a static version A and enough versions of B, at some point you will get statistically significant results if you repeat it often enough. Having a control doesn't mean you can't fall victim to this.

methou 3y ago

Probably off-topic, but how do opt out from most of A/B testings?

deleted 3y ago

[deleted]

dbroockman 3y ago

Another one: don’t program your own AB testing framework! Every time I’ve seen engineers try to build this on their own, it fails an AA test (where both versions are the same so there should be no difference). Common reasons are overly complicated randomization schemes (keep it simple!) and differences in load times between test and control.

giraffe_lady 3y ago
I don't keep that up with it but it seems like the ecosystem has kind of collapsed the last few years though? Like you have optimizely and its competitors that are fully focused on huge enterprise with "call us" pricing right out the gate. VWO has a clunky & aged tech stack that was already causing problems when I used it a couple years ago and seems unchanged since then. If you're a medium-small business I see why you'd be tempted to roll your own. Trustworthy options under $15k/year are not apparent.
HWR_14 3y ago
Shouldn't AA tests fail a certain percentage of the time? Typically, 5% of the time?

franze 3y ago

plus, mind the Honeymoon Effect something new performs better cause its new if you have a platform with lots pf returning users this one will hit you again and again. so even if you have a winner after the test and make the change permanent, revisit it 2 months later and see if you are now really better of. all changes of a/b tests in sum has a high chance to just get an average platform in the sum of all changes.

mtlmtlmtlmtl 3y ago

Surprised no one said this yet, so I'll bite the bullet. I don't think A/B testing is a good idea at all for the long term. Seems like a recipe for having your software slowly evolved into a giant heap of dark patterns. When a metric becomes a target, it ceases to be a good metric.

cantSpellSober 3y ago
Good multivariate testing and (statistically significant) data doesn't do that. It shows lots of ways to improve your UX, and if your guesses at improving UX actually work. Example from TFA: > more people signed up using Google and Github, overall sign-ups didn't increase, and nor did activation Less friction on login for the user, 0 gains in conversions, they shipped it anyway. That's not a dark pattern. If you're intentionally trying to make dark patterns it will help with that too I guess; the same way a hammer can build a house, or tear it down, depending on use.
- mtlmtlmtlmtl 3y ago
  I often see this argument, and although I can happily accept the examples given in defence as making sense, I never see an argument that this multivariate approach solves the problem in general and doesn't merely ameliorate some of the worst cases(I suppose I'm open to the idea that it could at least get it from "worse than the disease" to "actually useful in moderation"). Fundamentally, if you pick some number of metrics, you're always leaving some number of possible metrics "dark", right? Is there some objective method of deciding which metrics should be chosen, and which shouldn't?
  - cantSpellSober 3y ago
    "user trust" is a good one, abeit hard to measure Rolled out some tests to streamline cancelling subscriptions in response to user feedback, with Marketing's begrudging approval. Short term, predictably, we saw an increase in cancellations, then a decrease and eventual levelling out. Long term we continued to see an increase in subscriptions after rollout, and focused on more important questions like "how do we provide a good product that a user doesn't want to cancel?"

Sohcahtoa82 3y ago

The one mistake I assume happens too much is trying to measure "engagement". Imagine a website is testing a redesign, and they want to decide if people like it by measuring how long they spend on the site to see if it's more "engaging". But the new site makes information harder to find, so they spend more time on the site browsing and trying to find what they're looking for. Management goes, "Oh, users are delighted with the new site! Look how much time they spend on it!" not realizing how frustrated the users are.

ravenstine 3y ago
Engagement is my favorite form of metrics pseudoscience. A classic example is when engagement actually goes up, not because the design change is better, but because it frustrates and confuses the user, causing them to click around more and remain on the site longer. Without a focus group, there's really no way to determine whether the users are actually "delighted". EDIT: For some reason it didn't compute with me that you already referred to the same example. I've seen that exact scenario play out in real life, though.
- Sohcahtoa82 3y ago
  I bet the reddit redesign used a similar faulty measurement of engagement. "People spent more time scrolling the feed, people must enjoy it!" No, the feed takes up more space, so now I can only fit 1 or 2 items on my screen at once, rather than 10, so I have to scroll more to see more content.
  - ravenstine 3y ago
    That would not surprise me in the least! In fact, that's exactly what happened at a company I used to work for (that shall remain nameless). At the behest of the design team, we implemented a complete redesign of our site which included changing the home page so that at most only two media items could be on-screen at a time, and the ads which used to be simple banners now were woven between the feed of items. I remember sitting in a meeting where we had A/B tested this new homepage, and witnessing some data analyst guy giving a presentation which included how "engagement in the B-group was increased by N-percent!!!" The directors of web content were awestruck by this despite no context or explanation as to why supposed "engagement" was higher with the new design. The test wasn't even carried out for a long duration of time. For all anyone knew, users were confused and spent more time clicking around because they were looking for something they were accustomed to in the original design. And no, it did not matter that I brought up my reasons for skepticism; anything that made a number increase made it into the final design. Then, we actually had focus groups, long after the point at which we should have been consulting them, and the feedback we received was overwhelmingly lukewarm or negative. Much of it vindicated my concerns the entire time; users didn't actually like scrolling. Then again, I guess if they're viewing more ads, then who cares what the user thinks?? Never have I felt more like I was living in a Dilbert comic than that time.

jedberg 3y ago

The biggest mistake engineers make about A/B testing is not recognizing local maxima. Your test may be super successful, but there may be an even better solution that's significantly different than what you've arrived at. It's important to not only A/B test minor changes, but occasionally throw in some major changes to see if it moves the same metric, possibly even more than your existing success.

jldugger 3y ago

> 6. Not accounting for seasonality Doesn't the online nature of an A/B automatically account for this?

realjohng 3y ago

Thanks for posting this. It’s to the point and easy to understand. And much needed- most companies seem to do testing without teaching the intricacies involved.

donretag 3y ago

If anyone from posthog is reading this, please fix your RSS feed. The link actually points back to the blog homepage.

corywatilo 3y ago
Will take a look, thanks for the heads up!

time4tea 3y ago

Annoying illegal cookie consent banner?

pil0u 3y ago

In the second table, shouldn't the mobile control conversion rate be 3.33%?

iudqnolq 3y ago

Point one seems to be an API naming issue. I would not anticipate getFeatureFlag to increment a hit counter. Seems like it should be called something like participateInFlagTest or whatever. Or maybe it should take a (key, arbitraryId) instead of just (key), use the hash of the id to determine if the flag is set, and idempotently register a hit for the id.

drpixie 3y ago

> Relying too much on A/B tests for decision-making Need I say more? Or just keep tweaking your website until it becomes a mindless, grey, sludge.