9 ms·

P-Hacking in Startups

306 points by thaisstein 1y ago

irq-1 1y ago
1−(1−0.05)^9=64 (small mistake; should be ^20)
derektank 1y ago
I don't have any first hand experience with customer facing startups, SaaS or otherwise. How common is rigorous testing in the first place?
- dayjah 1y ago
  As you scale it improves. More often at a small scale you ask users and they’ll give you invaluable information. As you scale you abstract folks into buckets. At about 1million MAU I’ve found A/B testing and p-value starts to make sense.
bcyn 1y ago
Great read, thanks! Could you dive a little deeper into example 2 & pre-registration? Conceptually I understand how the probability of false positives increases with the number of variants. But how does a simple act such as "pre-registration" change anything? It's not as if observing another metric that already existed changes anything about what you experimented with.
- PollardsRho 1y ago
  If you have many metrics that could possibly be construed as "this was what we were trying to improve", that's many different possibilities for random variation to give you a false positive. If you're explicit at the start of an experiment that you're considering only a single metric a success, it turns any other results you get into "hmm, this is an interesting pattern that merits further exploration" and not "this is a significant result that confirms whatever I thought at the beginning." It's basically a variation on the multiple comparisons, but sneakier: it's easy to spend an hour going through data and, over that time, test dozens of different hypotheses. At that point, whatever p-value you'd compute for a single comparison isn't relevant, because after that many comparisons you'd expect at least one to have uncorrected p = 0.05 by random chance.
- noodletheworld 1y ago
  There are many resources that will explain this rigorously if you search for the term “p-hacking”. The TLDR as I understand it is: All data has patterns. If you look hard enough, you will find something. How do you tell the difference between random variance and an actual pattern? It’s simple and rigorously correct to only search the data for a single metric; other methods, eg. Bonferroni correction (divide p by k) exist, but are controversial (1). Basically, are you a statistician? If not, sticking to the best practices in experimentation means your results are going to be meaningful. If you see a pattern in another metric, run another experiment. [1] - https://pmc.ncbi.nlm.nih.gov/articles/PMC1112991/ https://pmc.ncbi.nlm.nih.gov/articles/PMC1112991/
andy99 1y ago
> Imagine you're a product manager trying to optimize your website’s dashboard. Your goal is to increase user signups. This would be Series B or later right? I don't really feel like it's a core startup behavior.
simonw 1y ago
On the one hand, this is a very nicely presented explanation of how to run statistically significant A/B style tests. It's worth emphasizing though that if your startup hasn't achieved product market fit yet this kind of thing is a huge waste of time! Build features, see if people use them.
- noodletheworld 1y ago
  “This kind of thing” being running AB tests at all. There’s no reason to run AB / MVT tests at all if you’re not doing them properly.
  - killerstorm 1y ago
    Proper way to make decisions is Bayesian. Take into account every bit of evidence, all the time. Statistical hypothesis testing is a simplification for people who don't understand Bayesian approach.
- cdavid 1y ago
  A/B testing does not have to involve micro optimization. If done well, it can reduce the risk / cost of trying things. For example, you can A/B test something before investing a full prod development, etc. When pushing for some ML-based improvements (e.g. new ranking algo), you also want to use it. This is why the cover of the reference A/B test book for product dev has a hippo: A/B test is helpful against just following the HIghest Paid Person Opinion. The practice is ofc more complicated, but that's more organizational/politics.
  - simonw 1y ago
    In my own career I've only ever seen it increase the cost of development. The vast majority of A/B test results I've seen showed no significant win in one direction or the other, in which case why did we just add six weeks of delay and twice the development work to the feature? Usually it was because the Highest Paid Person insisted on an A/B test because they weren't confident enough to move on without that safety blanket. There are other, much cheaper things you can do to de-risk a new feature. Build a quick prototype and run a usability test with 2-3 participants - you get more information for a fraction of the time and cost of an A/B test.
shoo 1y ago
related book: Trustworthy Online Controlled Experiments https://experimentguide.com/ https://experimentguide.com/
Jemaclus 1y ago
> This isn't academic nit-picking. It's how medical research works when lives are on the line. Your startup's growth deserves the same rigor. But does it, really? A lot of companies sell... well, let's say "not important" stuff. Most companies don't cost peoples' lives when you get it wrong. If you A/B test user signups for a startup that sells widgets, people aren't living or dying based on the results. The consequences of getting it wrong are... you sell fewer widgets? While I understand the overall point of the post -- and agree with it! -- I do take issue with this particular point. A lot of companies are, arguably, _too rigorous_ when it comes to testing. At my last company, we spent 6 weeks waiting for stat sig. But within 48 hours, we had a positive signal. Conversion was up! Not statistically significant, but trending in the direction we wanted. But to "maintain rigor," we waited 6 weeks before turning it... and the final numbers were virtually the same as the 48 hour numbers. Note: I'm not advocating stopping tests as soon as something shows trending in the right direction. The third scenario on the post points this out as a flaw! I do like their proposal for "peeking" and subsequent testing. But, really, let's just be realistic about what level of "rigor" is required to make decisions. We aren't shooting rockets into space. We're shipping software. We can change things if we get them wrong. It's okay. The world won't end. IMO, the right framing here is: your startup deserves to be as rigorous as is necessary to achieve its goals. If its goals are "stat sig on every test," then sure, treat it like someone might die if you're wrong. (I would argue that you have the wrong goals, in this case, but I digress...) But if your goals are "do no harm, see if we're heading in the right vector, and trust that you can pivot if it turns out you got a false positive," then you kind of explicitly don't need to treat it with the same rigor as a medical test.
- epgui 1y ago
  It does, if you assume you care about the validity of the results or about making changes that improve your outcomes. The degree of care can be different in less critical contexts, but then you shouldn’t lie to yourself about how much you care.
  - renjimen 1y ago
    But there’s an opportunity cost that needs to be factored in when waiting for a stronger signal.
cckolon 1y ago
Example 01 is basically the “green jellybeans cause acne” problem https://xkcd.com/882/ https://xkcd.com/882/
kylecazar 1y ago
I like the points and I'll probably link to this. I'll add one from my experience as a PM dealing with very "testy" peers in early stage startups: don't do any of this if you don't have {enough} users -- rely on intuition and focus on the core product.
- physix 1y ago
  I was waiting for that comment to appear. If your core product isn't any good, A/B testing seems like rearranging the deck chairs on the Titanic.
ryan-duve 1y ago
Good news: no p-value threshold needs to be passed to switch from one UI layout to another. As long as they all cost the same amount of money to host/maintain/whatever, the point estimate is sufficient. The reason is, at the end of the day, some layout has to be shown, and if each option had an equal number of visitors during the test, you can safely pick the one with the most signups. When choosing one of several A/B test options, a hypothesis test is not needed to validate the choice.
- ec109685 1y ago
  Yes, but assuming it was enhancing something already there, it was all pointless work.
ec109685 1y ago
Keep in mind that Frequent A/B tests burn statistical “credit.” Any time you ship a winner at p = 0.05 you’ve spent 5 % of your false-positive budget. Do that five times in a quarter and the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %. There are several approaches you can take to reduce that source of error: Quarterly alpha ledger Decide how much total risk you want this quarter (say 10 %). Divide the remaining α by the number of experiments left and make that the threshold for the next launch. Forces the “is this button-color test worth 3 % of our credibility?” conversation. More info: “Sequential Testing in Practice: Why Peeking Is a Problem and How to Fix It” (https://medium.com/@aisagescribe/sequential-testing-in-practice-why-peeking-is-a-problem-and-how-to-fix-it-59d7024344f6 https://medium.com/@aisagescribe/sequential-testing-in-pract...). Benjamini–Hochberg (BH) for metric sprawl Once you watch a dozen KPIs, Bonferroni buries real lifts. BH ranks all the p-values at the end, then sets the cut so that, say, only 5 % of declared winners are false positives. You keep power, and you can run the same BH step on the primary metric from every experiment each quarter to catch lucky launches. More info: “Controlling False Discoveries: A Guide to BH Correction in Experimentation” (https://www.statsig.com/perspectives/controlling-false-discoveries-guide https://www.statsig.com/perspectives/controlling-false-disco...). Bayesian shrinkage + 5 % “ghost” control for big fleets FAANG-scale labs run hundreds of tests and care about 0.1 % lifts. They pool everything in a simple hierarchical model; noisy effects get pulled toward the global mean, so only sturdy gains stay above water. Before launch, they sanity-check against a small slice of traffic that never saw any test. Cuts winner’s-curse inflation by ~30 %. Clear explainer: “How We Avoid A/B Testing Errors with Shrinkage” (https://eng.wealthfront.com/2015/10/29/how-we-avoid-ab-testing-errors-with-shrinkage/ https://eng.wealthfront.com/2015/10/29/how-we-avoid-ab-testi...) and (https://www.statsig.com/perspectives/informed-bayesian-ab-testing https://www.statsig.com/perspectives/informed-bayesian-ab-te...) <10 tests a quarter: alpha ledger or yolo; dozens of tests and KPIs: BH; hundreds of live tests: shrinkage + ghost control.
- akoboldfrying 1y ago
  > the chance at least one is noise is 1 – 0.95⁵ ≈ 23 % Yes, but that's not really the big deal that you're making it out to be, since it's (usually) not an all-or-nothing thing. Usually, the wins are additive. The chance of each winner being genuine is still 95% (assuming no p-hacking), and so the expected number of wins out of those 5 will be be 0.95 * 5 = 4.75 wins (by linearity of expectation), which is a solid win rate.
tmoertel 1y ago
When reading this article, be aware that there are some percent signs missing, and their absence might cause confusion. For example: > After 9 peeks, the probability that at least one p-value dips below 0.05 is: 1 − (1 − 0.05)^9 = 37. There should be a percent sign after that 37. (Probabilities cannot be greater than one.)
blobbers 1y ago
1 - (1-0.95)^9 = 64 Did they generate this blog post with AI? That math be hallucinating. Don’t need a calculator to see that.
- blobbers 1y ago
  I’m so confused by the math in this article. It’s also not 37. I can’t be the only person scratching their head.
  - Retric 1y ago
    Probably mangled the expression. (0.95)^9 = 63%, 1 - (0.95)^9 = 37% Hmm, (1 - (1-0.95))^9 also = 63%. No idea why 64, closest I can see is 1-(0.95)^20 or 1-(1-0.05)^20 = 64%.
    - blobbers 1y ago
      Yeah, I thought he was talking about 1 out of 20 features, but that's kind of why I was wondering if AI had written it. Sometimes it'll have mis-aligned figures etc.
- larfus 1y ago
  Read a few more posts and it shouts GPT occasionally. Plus the author's (as I like to call them still) role is listed as 'Content Engineer' which isn't inspiring either. Too bad, the topics sounded interesting.
vzaliva 1y ago
I would look in the directon of SPRT: https://en.wikipedia.org/wiki/Sequential_probability_ratio_test https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...
esafak 1y ago
If you like reading blogs, I suggest Spotify's: https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions https://engineering.atspotify.com/2023/03/choosing-sequentia...
akoboldfrying 1y ago
Yes! (Correct) pre-registration is everything. ("Correct" meaning: There's no point "pre-registering" if you fail to account for the number of tests you'll do -- but hopefully the fact that you have thought to pre-register at all is a strong indication that you should be performing such corrections.) That said, I agree with the other poster here about how important this really is for startups. It's critical to know if the drug really improves lung function; it's probably not critical to know whether the accent colour on your landing page should be mauve or aqua blue.
hlieberman 1y ago
Wouldn't the correct tool here be a multi-armed bandit optimization, like an epsilon-greedy algorithm?
BrenBarn 1y ago
Aside from the p-values, I don't understand the reasoning behind whatever "experiment" is being used for the A/B testing. What test is being done whose result is interpreted as "A is winner"? The discussion is about those being separate comparisons, and yeah, okay, but what are they comparisons of? Each group in isolation vs. all the others? If (as the article says) the hypothesis is "layout influences signup behavior" then it seems more reasonable to do a chi-squared test on a contingency table of layout vs. signed-up-or-didn't, which would give you one p-value for "is there anything here at all". And then, if there isn't. . . it means you can just ship whatever you want! The real root cause of p-hacking is glossed over in the article: "Nobody likes arriving empty-handed to leadership meetings." This is the corporate equivalent of "no one will publish a null result", and is just as harmful here. The statistical techniques described are fine, but there's not necessarily a reason to fortify your stats against multiple comparisons rather than just accepting a null result. And you can, because of the other thing I kept thinking when reading this: you have to ship something. There isn't really a "control" condition if you're talking about building a website from scratch. So whether the result is null doesn't really matter. It's not like comparing different medicines or fertilizers or something where if none of them work you just do nothing; there is no "do nothing" option in this situation. So why not just take a simple effect measurement (e.g., proportion who signed up) and pick the layout that performs best? If that result is statistically significant, great, it means you picked the best one, and if it's not, it just means it doesn't matter which one you pick, so the one you picked is still fine. (And if you have an existing design and you're trying to see if a new one will be better, the null result just means "there's no reason to switch", which means the existing design is also fine.)
dooglius 1y ago
If the goal is for the company to maximize profit from having the best page, this is an instance of a very well-studied problem https://en.m.wikipedia.org/wiki/Multi-armed_bandit?useskin=vector https://en.m.wikipedia.org/wiki/Multi-armed_bandit?useskin=v... and one can do much better than statistical significance testing. (If the goal is to validate scientific theories, or there are other extenuating factors, things may be different.)
kdamica 1y ago
Hard disagree with this. Unlike medical experiments, the cost of being wrong startup experiments is very low: you thought there was a small effect and there was none. It’s usually just a matter of pushing one variant vs another and moving on. There are certainly scenarios where more rigor is appropriate, but usually those come from trying to figure out why you’re seeing a certain effect and how that should affect your overall company strategy. My advice for startups is to run lots of experiments, do bad statistics, and know that you’re going to have some false positives so that you don’t take every result as gospel.
- bravesoul2 1y ago
  The danger I think is less the numbers but what are you measuring makes sense. E.g. sure your A beats B in click through rate. But if the person then thinks fuck I was duped and closes the browser then that's no good.
- aaron695 1y ago
  [dead]
- lemmsjid 1y ago
  Huh, I agree with your last sentence but think the author did a good job of explaining that the cost of layout experimentation in a startup can grow over time if the results of the experiments are overstated. The first question for a startup should always be: is this work worth doing in the first place? Tinkering with layout can be a tempting but fruitless rabbit hole. Even if it doesn’t tie up resources it can lead to a false sense of progress and get product thinking stuck in local maximae.
NoahZuniga 1y ago
Why don't the p values in the first figure sum to 1?
- begemotz 1y ago
  Because that is not how p-values work. Each 'row' is a separate hypothesis test against some null hypothesis. And frankly the conclusion that "B is winner" because it has the lowest p-value also demonstrates the author's limited understanding. P-values tell you very little, and generally nothing about what you really want to know. In this context, you would be better off looking at effect size estimates of key metrics that would signal "better" - not the p-values.
mo_42 1y ago
> Back to the dashboard experiment: after you applied the Bonferroni correction you got... nothing. I guess you got something: Users are not sensitive to these changes, or that any effect is too small to detect with your current sample size/test setup. In a startup scenario, I'd quickly move on and possibly ship all developed options if good enough. Also, running A/B tests might not be the most appropriate method in such a scenario. What about user-centric UX research methods?
psychoslave 1y ago
P-value is something new for me, so the post is starting with prerequisites in mind that I miss. Tough I can go search by myself, would anyone have some online resources I can follow and test myself against to recommend, please?
- the-mitr 1y ago
  Here you go https://pmc.ncbi.nlm.nih.gov/articles/PMC5187603/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5187603/
  - psychoslave 1y ago
    Thanks, seems relevant, I would appreciate also resources on epistomological level, when and where it was devised and any historical context of development. Thanks again
    - pkhuong 1y ago
      https://en.wikipedia.org/wiki/P-value#History https://en.wikipedia.org/wiki/P-value#History
cornel_io 1y ago
Even though this post says exactly the thing that most Proper Analysts will say, and write long LinkedIn posts about where other Proper Analysts congratulate them on standing up for Proper Analysis in the face of Evil And Stupid Business Dummies who just want to make bad decisions based on too little data, it's wrong. The Jedi Bell Curve meme is in full effect on this topic, and I say this as someone who took years to get over the midwit hump and correct my mistaken beliefs. The business reality is, you aren't Google. You can't collect a hundred million data points for each experiment that you run so that you can reliably pick out 0.1% effects. Most experiments will have a much shorter window than any analyst wants them to, and will have far too few users, with no option to let them run longer. You still have to make a damned decision, now, and move on to the next feature (which will also be tested in a heavily underpowered manner). Posts like this say that you should be really, REALLY careful about this, and apply Bonferonni corrections and make sure you're not "peeking" (or if you do peek, apply corrections that are even more conservative), preregister, etc. All the math is fine, sure. But if you take this very seriously and are in the situation that most startups are in where the data is extremely thin and you need to move extremely fast, the end result is that you should reject almost every experiment (and if you're leaning on tests, every feature). That's the "correct" decision, academically, because most features lie in the sub 5% impact range on almost any metric you care about, and with a small number of users you'll never have enough power to pick out effects that small (typically you'd want maybe 100k, depending on the metric you're looking at, and YOU probably have a fraction of that many users). But obviously the right move is not to just never change the product because you can't prove that the changes are good - that's effectively applying a very strong prior in favor of the control group, and that's problematic. Nor should you just roll out whatever crap your product people throw at the wall: while there is a slight bias in most experiments in favor of the variant, it's very slight, so your feature designers are probably building harmful stuff about half the time. You should apply some filter to make sure they're helping the product and not just doing a random walk through design space. The best simple strategy in a real world where most effect sizes are small and you never have the option to gather more data really is to do the dumb thing: run experiments for as long as you can, pick whichever variant seems like it's winning, rinse and repeat. Yes, you're going to be picking the wrong variant way more often than your analysts would prefer, but that's way better than never changing the product or holding out for the very few hugely impactful changes that you are properly powered for. On average, over the long run, blindly picking the bigger number will stack small changes, and while a lot of those will turn out to be negative, your testing will bias somewhat in favor of positive ones and add up over time. And this strategy will provably beat one that does Proper Statistics and demands 95% confidence or whatever equivalent Bayesian criteria you use, because it leaves room to accept the small improvements that make up the vast majority of feature space. There's an equivalent and perhaps simpler way to justify this, which is to throw out the group labels: if we didn't know which one was the control and we had to pick which option was better, then quite obviously, regardless of how much data we have, we just pick the one that shows better results in the sample we have. Including if there's just a single user in each group! In an early product, this is TOTALLY REASONABLE, because your current product sucks, and you have no reason to think that the way it is should not be messed with. Late lifecycle products probably have some Chesterton's fence stuff going on, so maybe there's more of an argument to privilege the control, but those types of products should have enough users to run properly powered tests.
light_hue_1 1y ago
Don't do any of this. It's very outdated advice. And you're going to get it wrong anyway. These threshold adjustment methods were invented before we had access to reasonable computers. There's a far simpler method that covers every case: permutation tests. https://bookdown.org/ybrandvain/Applied-Biostats/perm1.html https://bookdown.org/ybrandvain/Applied-Biostats/perm1.html You shuffle the data. Say you want to know if viewing time is affected by color. Literally randomly shuffle viewing time and color. Then, look at that distribution. Is the data that you observed significant? As long as you shuffle everything related to your experiment and you don't double dip into the data you're going to get things right. This also has the big advantage that it doesn't overcorrect like traditional methods which apply such strict corrections that eventually it's impossible to get significant results. This post hasn't even begun scratching the surface on what can go wrong with traditional tests. Just don't use them. This has nothing to do with speed or rigor. Permutation tests are much simpler to run and faster to analyze. Sadly we keep teaching crappy statistics to our students.
- syntacticsalt 1y ago
  Permutation tests don't account for family-wise error rate effects, so I'm curious why you would say that "it doesn't overcorrect like traditional methods". I'm also curious why you say those "cover every case", because permutation tests tend to be underpowered, and also tend to be cumbersome when it comes to constructing confidence intervals of statistics, compared to something like the bootstrap. Don't get me wrong -- I like permutation tests, especially for their versatility, but as one tool out of a bunch of methods.
kookamamie 1y ago
…or, you could have a product that does not hinge on some micro-optimization of a website layout.
- roncesvalles 1y ago
  Exactly. This is what micro-optimization looks like on the product side.
Palmik 1y ago
This isn't just a startup thing. This is common also at FAANG. Not only are expriments commonly multi-arm, you also repeat your experiment (usually after making some changes) if the previous experiment failed / did not pass the launch criteria. This is further complicated by the fact that lauch criteria is usually not well defined ahead of time. Unless it's a complete slam dunk, you won't know until your launch meeting whether the experiment will be approved for launch or not. It's mostly vibe based, determined based on tens or hundreds of "relevant" metric movements, often decided on the whim of the stakeholder sitting at the lauch meeting.
- setgree 1y ago
  You're describing conditioning analyses on data. Gelman and Loken (2013) put it like this: > The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of fishing or examining multiple p values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre specified and, as a result, were contingent on data.
- netcan 1y ago
  Is this terrible? The idea is not do do science. The idea is to loosely systematize and conceptualize innovation. To generate options and create a failure tolerant system. I'm sure improvements could be made... but this isn't about being a valid or invalid expirement.
  - zeroCalories 1y ago
    It seems bad to me because you're giving yourself the illusion of effect.
  - godelski 1y ago
    > The idea is not do do science. The idea is to loosely systematize and conceptualize innovation. Why are you acting like these are completely different frameworks? You have the same goals
welpo 1y ago
On the third point (peeking at p-values), I created an A/A test simulator that compares peeking vs not peeking in terms of false positive rate: https://stop-early-stopping.osc.garden/ https://stop-early-stopping.osc.garden/
freehorse 1y ago
I do not understand what the first tests are supposed to do. The author says the: > Your hypothesis is: layout influences signup behavior. I would expect that then the null hypothesis is that *layout does not influence signup behavior*. I would think that then an ANOVA (or an equivalent linear model) to be what tests this hypothesis, where you test the 4 layouts (or the 4 new layouts plus a control?) in one factor. If you get a significant p-value (no multiple tests required) you go on with post-hoc tests to look into comparisons between the different layouts (for 4 layouts, it should be 6 tests). But then you can use ways to control for multiple comparisons that are not as strict as just dividing your threshold by the number of comparisons, eg with Tukey's test. But here I assume there is a control (as in some for users are still presented the old layout?) and each layout is compared to that control? If I would see that distribution of p-values I would just intuitively think that the experiment is underpowered. P-values from null tests are supposed to be distributed uniformly between 0 and 1, while these cluster around 0.05. It rather seems like a situation that it is hard to make inferences from because of issues in designing the experiment itself. For example, I would rather have fewer layouts, driven by some expert design knowledge, rather than a lot of randomish layouts. The first increases statistical power, because the fewer tests you investigate, the less you have to adjust your p-values. But also, the fewer layouts you have, the more users you have per group (as the test is between groups) which also increases statistical power. The article is not wrong overall about how to control p-values etc, but I think that this knowledge is important not just to "do the right analysis" but, even more importantly, understand the limitations of an experimental design and structure it in a way that it may succeed in telling you something. To this end, g*power [0] is a useful tool that eg can let one calculate sample size in advance based on predicted effect size and power required. [0] https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psy...
- majormajor 1y ago
  The fraction of A/B tests I've seen personally that mentioned ANOVA at all is very small. Or thought that critically about experiment design. Understanding of p values is also generally poor; prob/stat education in engineering and business degrees seems to be the least-covered-or-respected type of math. Even at places that want to ruthlessly prioritize velocity over rigor I think it would be better to at least switch things up and worry more about effect size than p-value. Don't bother waiting to see if marginal effects are "significant" statistically if they aren't significant from the POV of "we need to do things that can 10x our revenue since we're a young startup."
calrain 1y ago
This feels like a billion dollar company problem, not a startup problem
- gspetr 1y ago
  These aren't mutually exclusive, unless you have an extremely narrow definition of a startup: https://techcrunch.com/2025/02/20/38-startups-have-become-unicorns-so-far-in-2024-heres-the-full-list/ https://techcrunch.com/2025/02/20/38-startups-have-become-un...
  - calrain 1y ago
    You're right, but a startup has priorities a billion dollar company doesn't. And a billion dollar company can spend more time on min/max than a startup that is trying to get to PMF.
    - gspetr 1y ago
      A billion dollar company isn't invincible, you can still be zeroed out by an unfavorable market. https://en.wikipedia.org/wiki/Exodus_Communications https://en.wikipedia.org/wiki/Exodus_Communications In 2025 dollars their peak would have been $59.5B. To wit, this is roughly what Doordash is worth today, and that's a top-5 YC company. It died 9 months later.
vmesel 1y ago
Parabéns pelo conteúdo Thais!
snowstormsun 1y ago
The problem probably more often just is that the product/vision itself is not as good as it is sold to shareholders. No PM will like to push back on this, so they're stuck with things like optimizing the landing page because "clearly the business vision is flawless, it must be that users don't understand it correctly".
kgwgk 1y ago
> Users are randomly assigned to one of the four layouts and you track their activity. Your hypothesis is: layout influences signup behavior. > You plan ship the winner if the p-value for one of the layout choices falls below the conventional threshold of 0.05. Tests P-value B is winner 0.041 A is winner 0.051 D is winner 0.064 C is winner 0.063 What kind of comparison between the results for the four options makes each of them a likely winner? They all rank very well in whatever metric is being used! Or maybe the are they being compared with a fifth - much worse - alternative.
- vjerancrnjak 1y ago
  This kind of ranking is also not correct. You have to compare the outcomes , not their p values. Ranking by p values is just silly, just like ranking by avg metric is silly. Startups in general have to make decisions with high signal, thinking that 100 improvements of 1% p=0.05 will actually compound in an environment with so much noise is delusion. I’d say doing this kind of silliness in a startup is just ceremonial, helpful long term if people feel they are doing a good job optimizing a compounding metric, even though it never materializes.
meindnoch 1y ago
I've never seen AB-testing micro-optimisation affect the bottom line of a software company by more than 3%.
kgwgk 1y ago
> Setting a p-value threshold of 0.05 is equivalent to saying: "I’m willing to accept a 5% chance of shipping something that only looked good by chance." No, it means "I’m willing to ship something that if it was not better than the alternative it would have had only a 5% chance of looking as good as it did.”
- phaedrus441 1y ago
  This! I see this all the time in medicine.
  - __MatrixMan__ 1y ago
    I'm under the impression that there was a paper on "ego depletion" has sort of become a poster child for p-value hacking. It has sort of poisoned the topic. Upon learning of such a hack one should rationally revert to a null hypothesis, not revert to the assumption that the hacked conclusion was false. As a climber I see ego depletion happen all the time. You find a crumbly hold, or get harassed by an insect, or whatever else it is, and you conclude that the next move is the crux. Then other people climb it and nobody agrees with you--that move was one of the easier ones. Anecdata, of course, I just wish we could learn from the bad science and then be washed clean, rather than be haunted by it.
- wavemode 1y ago
  Can you elaborate on the difference between your statement and the author's?
  - likecarter 1y ago
    Author: 5% chance it could be same or worse Parent: 5% chance it could be same
    - esafak 1y ago
      @wavemode: In other words, the probability of it being the exactly the same is typically (for continuous random variables) zero, so we consider the tail probability; that of it being the same or more extreme. edit: Will the down voter please explain yourself? p-values are tail probabilities, and points have zero measure in continuous random variables.
aDyslecticCrow 1y ago
Whenever working with this kind of probabilities, i always throw in a python rand() as a comparison. It sanity checks the calculation of threshold with very low rist of miscalculation. Of course calculating the threshold properly needs to be done aswell... but a rand() is so quick and simple to add as a napkin check.
aDyslecticCrow 1y ago
Whenever working with this kind of probabilities, i always throw in a python rand() as a comparison. It sanity checks the calculation of threshold with very low rist of miscalculation. Of course calculating the threshold properly needs to be done aswell... but a rand() is so quick and simple to add as a napkin check, and would catch how silly the first analysis is.
begemotz 1y ago
> Your hypothesis is: layout influences signup behavior. This might be your hypothesis but this isn't the hypothesis that the p-value is related to. >Setting a p-value threshold of 0.05 is equivalent to saying: "I’m willing to accept a 5% chance of shipping something that only looked good by chance." P-values don't provide any information about "chance occurrence" but rather they test the probability of observing a particular outcome assuming a particular state of the world (i.e. the null hypothesis). > Bonferonni Besides being a very aggressive (i.e. conservative) correction, I would imagine that in industry, just as in science, the motivations to observe results will mean it wont be used. There are other way more reasonable corrections. > Avoid digging through metrics post hoc The reasonable solution is not to ignore the results that you found but to interpret them appropriately. Maybe the result actually does signal improved intention - alternatively maybe it is noise. Treat it as exploratory data. If that improved retention was real, is this important? Important enough to appropriately retest for?
b0a04gl 1y ago
[dead]
zug_zug 1y ago
Yes this is real, and even happens at larger companies. The surface issue is that when somebody has an incentive to self-measure their success then they have an incentive to overestimate (I increased retention by 14% by changing the shade of the "About Us" button!). Which means the root-cause issue is managers who create environments where self-reporting improvements without any rigor or any contrary perspective. Ultimately they are the ones foot-gunning themselves (by letting their team focus on false vanity metrics).
stared 1y ago
I think we should declare a moratorium on the use of p-values. If you don't understand what a p-value is, you shouldn't use it. If you do understand p-values, you're probably already moving away from them. The Bayesian approach makes much more sense. I highly recommend David MacKay's "Information Theory, Inference and Learning Algorithms," as well as "Bayesian Methods for Hackers": https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers https://github.com/CamDavidsonPilon/Probabilistic-Programmin.... At the same time, startups aren't science experiments. Our goal is not necessarily to prove conclusively whether something is statistically "better". Rather, our goal is to solve real-world problems. Suppose we run an A/B test, and its result indicates B is better according to whatever statistical test we've used. In this scenario, we will likely select B—frankly, regardless of whether B is truly better or merely indistinguishable from A. However, what truly matters in practice are the metrics we choose to measure. Picking the wrong metric can lead to incorrect conclusions. For example, suppose our data shows that users spend, on average, two more seconds on our site with the new design (with p < 0.001 or whatever). That might be a positive result—or it could simply mean the new design causes slower loading or more confusion, frustrating users instead of benefiting them.
chickenzzzzu 1y ago
Just ship useful stuff. Stop with all this overanalysis.
rossdavidh 1y ago
Nice stuff for explaining what p-hacking is, but the big difference between medical "does this drug work" tests and software "does this layout improve signups" tests, is that in the latter, the downsides are much less. Most pharmaceuticals have undesirable side-effects, and they always cost money to produce as well as develop; a layout change (unless it has some huge code complexity, which it typically won't) has very little. The cost of developing it has already been paid. If it is more likely to help than not, the right decision is to ship it. This makes the importance of not p-hacking far less for software, than in the pharmaceutical case. You may have some kind of ongoing cost (code complexity or server load or whatever), but in the example given, I would not expect that, and it wasn't addressed in the article. The cost of a false positive is an important factor. If you get ten probably-helpful, or even ten 50%-likely-to-be-helpful changes out, instead of 2 or 3 definitely-helpful changes, this is a net benefit.

306 points by thaisstein 1y ago

1−(1−0.05)^9=64 (small mistake; should be ^20)

I don't have any first hand experience with customer facing startups, SaaS or otherwise. How common is rigorous testing in the first place?

dayjah 1y ago
As you scale it improves. More often at a small scale you ask users and they’ll give you invaluable information. As you scale you abstract folks into buckets. At about 1million MAU I’ve found A/B testing and p-value starts to make sense.

bcyn 1y ago

Great read, thanks! Could you dive a little deeper into example 2 & pre-registration? Conceptually I understand how the probability of false positives increases with the number of variants. But how does a simple act such as "pre-registration" change anything? It's not as if observing another metric that already existed changes anything about what you experimented with.

PollardsRho 1y ago
If you have many metrics that could possibly be construed as "this was what we were trying to improve", that's many different possibilities for random variation to give you a false positive. If you're explicit at the start of an experiment that you're considering only a single metric a success, it turns any other results you get into "hmm, this is an interesting pattern that merits further exploration" and not "this is a significant result that confirms whatever I thought at the beginning." It's basically a variation on the multiple comparisons, but sneakier: it's easy to spend an hour going through data and, over that time, test dozens of different hypotheses. At that point, whatever p-value you'd compute for a single comparison isn't relevant, because after that many comparisons you'd expect at least one to have uncorrected p = 0.05 by random chance.
noodletheworld 1y ago
There are many resources that will explain this rigorously if you search for the term “p-hacking”. The TLDR as I understand it is: All data has patterns. If you look hard enough, you will find something. How do you tell the difference between random variance and an actual pattern? It’s simple and rigorously correct to only search the data for a single metric; other methods, eg. Bonferroni correction (divide p by k) exist, but are controversial (1). Basically, are you a statistician? If not, sticking to the best practices in experimentation means your results are going to be meaningful. If you see a pattern in another metric, run another experiment. [1] - https://pmc.ncbi.nlm.nih.gov/articles/PMC1112991/ https://pmc.ncbi.nlm.nih.gov/articles/PMC1112991/

andy99 1y ago

> Imagine you're a product manager trying to optimize your website’s dashboard. Your goal is to increase user signups. This would be Series B or later right? I don't really feel like it's a core startup behavior.

simonw 1y ago

On the one hand, this is a very nicely presented explanation of how to run statistically significant A/B style tests. It's worth emphasizing though that if your startup hasn't achieved product market fit yet this kind of thing is a huge waste of time! Build features, see if people use them.

noodletheworld 1y ago
“This kind of thing” being running AB tests at all. There’s no reason to run AB / MVT tests at all if you’re not doing them properly.
- killerstorm 1y ago
  Proper way to make decisions is Bayesian. Take into account every bit of evidence, all the time. Statistical hypothesis testing is a simplification for people who don't understand Bayesian approach.
cdavid 1y ago
A/B testing does not have to involve micro optimization. If done well, it can reduce the risk / cost of trying things. For example, you can A/B test something before investing a full prod development, etc. When pushing for some ML-based improvements (e.g. new ranking algo), you also want to use it. This is why the cover of the reference A/B test book for product dev has a hippo: A/B test is helpful against just following the HIghest Paid Person Opinion. The practice is ofc more complicated, but that's more organizational/politics.
- simonw 1y ago
  In my own career I've only ever seen it increase the cost of development. The vast majority of A/B test results I've seen showed no significant win in one direction or the other, in which case why did we just add six weeks of delay and twice the development work to the feature? Usually it was because the Highest Paid Person insisted on an A/B test because they weren't confident enough to move on without that safety blanket. There are other, much cheaper things you can do to de-risk a new feature. Build a quick prototype and run a usability test with 2-3 participants - you get more information for a fraction of the time and cost of an A/B test.

shoo 1y ago

related book: Trustworthy Online Controlled Experiments https://experimentguide.com/ https://experimentguide.com/

Jemaclus 1y ago

> This isn't academic nit-picking. It's how medical research works when lives are on the line. Your startup's growth deserves the same rigor. But does it, really? A lot of companies sell... well, let's say "not important" stuff. Most companies don't cost peoples' lives when you get it wrong. If you A/B test user signups for a startup that sells widgets, people aren't living or dying based on the results. The consequences of getting it wrong are... you sell fewer widgets? While I understand the overall point of the post -- and agree with it! -- I do take issue with this particular point. A lot of companies are, arguably, _too rigorous_ when it comes to testing. At my last company, we spent 6 weeks waiting for stat sig. But within 48 hours, we had a positive signal. Conversion was up! Not statistically significant, but trending in the direction we wanted. But to "maintain rigor," we waited 6 weeks before turning it... and the final numbers were virtually the same as the 48 hour numbers. Note: I'm not advocating stopping tests as soon as something shows trending in the right direction. The third scenario on the post points this out as a flaw! I do like their proposal for "peeking" and subsequent testing. But, really, let's just be realistic about what level of "rigor" is required to make decisions. We aren't shooting rockets into space. We're shipping software. We can change things if we get them wrong. It's okay. The world won't end. IMO, the right framing here is: your startup deserves to be as rigorous as is necessary to achieve its goals. If its goals are "stat sig on every test," then sure, treat it like someone might die if you're wrong. (I would argue that you have the wrong goals, in this case, but I digress...) But if your goals are "do no harm, see if we're heading in the right vector, and trust that you can pivot if it turns out you got a false positive," then you kind of explicitly don't need to treat it with the same rigor as a medical test.

epgui 1y ago
It does, if you assume you care about the validity of the results or about making changes that improve your outcomes. The degree of care can be different in less critical contexts, but then you shouldn’t lie to yourself about how much you care.
- renjimen 1y ago
  But there’s an opportunity cost that needs to be factored in when waiting for a stronger signal.

cckolon 1y ago

Example 01 is basically the “green jellybeans cause acne” problem https://xkcd.com/882/ https://xkcd.com/882/

kylecazar 1y ago

I like the points and I'll probably link to this. I'll add one from my experience as a PM dealing with very "testy" peers in early stage startups: don't do any of this if you don't have {enough} users -- rely on intuition and focus on the core product.

physix 1y ago
I was waiting for that comment to appear. If your core product isn't any good, A/B testing seems like rearranging the deck chairs on the Titanic.

ryan-duve 1y ago

Good news: no p-value threshold needs to be passed to switch from one UI layout to another. As long as they all cost the same amount of money to host/maintain/whatever, the point estimate is sufficient. The reason is, at the end of the day, some layout has to be shown, and if each option had an equal number of visitors during the test, you can safely pick the one with the most signups. When choosing one of several A/B test options, a hypothesis test is not needed to validate the choice.

ec109685 1y ago
Yes, but assuming it was enhancing something already there, it was all pointless work.

ec109685 1y ago

Keep in mind that Frequent A/B tests burn statistical “credit.” Any time you ship a winner at p = 0.05 you’ve spent 5 % of your false-positive budget. Do that five times in a quarter and the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %. There are several approaches you can take to reduce that source of error: Quarterly alpha ledger Decide how much total risk you want this quarter (say 10 %). Divide the remaining α by the number of experiments left and make that the threshold for the next launch. Forces the “is this button-color test worth 3 % of our credibility?” conversation. More info: “Sequential Testing in Practice: Why Peeking Is a Problem and How to Fix It” (https://medium.com/@aisagescribe/sequential-testing-in-practice-why-peeking-is-a-problem-and-how-to-fix-it-59d7024344f6 https://medium.com/@aisagescribe/sequential-testing-in-pract...). Benjamini–Hochberg (BH) for metric sprawl Once you watch a dozen KPIs, Bonferroni buries real lifts. BH ranks all the p-values at the end, then sets the cut so that, say, only 5 % of declared winners are false positives. You keep power, and you can run the same BH step on the primary metric from every experiment each quarter to catch lucky launches. More info: “Controlling False Discoveries: A Guide to BH Correction in Experimentation” (https://www.statsig.com/perspectives/controlling-false-discoveries-guide https://www.statsig.com/perspectives/controlling-false-disco...). Bayesian shrinkage + 5 % “ghost” control for big fleets FAANG-scale labs run hundreds of tests and care about 0.1 % lifts. They pool everything in a simple hierarchical model; noisy effects get pulled toward the global mean, so only sturdy gains stay above water. Before launch, they sanity-check against a small slice of traffic that never saw any test. Cuts winner’s-curse inflation by ~30 %. Clear explainer: “How We Avoid A/B Testing Errors with Shrinkage” (https://eng.wealthfront.com/2015/10/29/how-we-avoid-ab-testing-errors-with-shrinkage/ https://eng.wealthfront.com/2015/10/29/how-we-avoid-ab-testi...) and (https://www.statsig.com/perspectives/informed-bayesian-ab-testing https://www.statsig.com/perspectives/informed-bayesian-ab-te...) <10 tests a quarter: alpha ledger or yolo; dozens of tests and KPIs: BH; hundreds of live tests: shrinkage + ghost control.

akoboldfrying 1y ago
> the chance at least one is noise is 1 – 0.95⁵ ≈ 23 % Yes, but that's not really the big deal that you're making it out to be, since it's (usually) not an all-or-nothing thing. Usually, the wins are additive. The chance of each winner being genuine is still 95% (assuming no p-hacking), and so the expected number of wins out of those 5 will be be 0.95 * 5 = 4.75 wins (by linearity of expectation), which is a solid win rate.

tmoertel 1y ago

When reading this article, be aware that there are some percent signs missing, and their absence might cause confusion. For example: > After 9 peeks, the probability that at least one p-value dips below 0.05 is: 1 − (1 − 0.05)^9 = 37. There should be a percent sign after that 37. (Probabilities cannot be greater than one.)

blobbers 1y ago

1 - (1-0.95)^9 = 64 Did they generate this blog post with AI? That math be hallucinating. Don’t need a calculator to see that.

blobbers 1y ago
I’m so confused by the math in this article. It’s also not 37. I can’t be the only person scratching their head.
- Retric 1y ago
  Probably mangled the expression. (0.95)^9 = 63%, 1 - (0.95)^9 = 37% Hmm, (1 - (1-0.95))^9 also = 63%. No idea why 64, closest I can see is 1-(0.95)^20 or 1-(1-0.05)^20 = 64%.
  - blobbers 1y ago
    Yeah, I thought he was talking about 1 out of 20 features, but that's kind of why I was wondering if AI had written it. Sometimes it'll have mis-aligned figures etc.
larfus 1y ago
Read a few more posts and it shouts GPT occasionally. Plus the author's (as I like to call them still) role is listed as 'Content Engineer' which isn't inspiring either. Too bad, the topics sounded interesting.

vzaliva 1y ago

I would look in the directon of SPRT: https://en.wikipedia.org/wiki/Sequential_probability_ratio_test https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...

esafak 1y ago

If you like reading blogs, I suggest Spotify's: https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions https://engineering.atspotify.com/2023/03/choosing-sequentia...

akoboldfrying 1y ago

Yes! (Correct) pre-registration is everything. ("Correct" meaning: There's no point "pre-registering" if you fail to account for the number of tests you'll do -- but hopefully the fact that you have thought to pre-register at all is a strong indication that you should be performing such corrections.) That said, I agree with the other poster here about how important this really is for startups. It's critical to know if the drug really improves lung function; it's probably not critical to know whether the accent colour on your landing page should be mauve or aqua blue.

hlieberman 1y ago

Wouldn't the correct tool here be a multi-armed bandit optimization, like an epsilon-greedy algorithm?

BrenBarn 1y ago

Aside from the p-values, I don't understand the reasoning behind whatever "experiment" is being used for the A/B testing. What test is being done whose result is interpreted as "A is winner"? The discussion is about those being separate comparisons, and yeah, okay, but what are they comparisons of? Each group in isolation vs. all the others? If (as the article says) the hypothesis is "layout influences signup behavior" then it seems more reasonable to do a chi-squared test on a contingency table of layout vs. signed-up-or-didn't, which would give you one p-value for "is there anything here at all". And then, if there isn't. . . it means you can just ship whatever you want! The real root cause of p-hacking is glossed over in the article: "Nobody likes arriving empty-handed to leadership meetings." This is the corporate equivalent of "no one will publish a null result", and is just as harmful here. The statistical techniques described are fine, but there's not necessarily a reason to fortify your stats against multiple comparisons rather than just accepting a null result. And you can, because of the other thing I kept thinking when reading this: you have to ship something. There isn't really a "control" condition if you're talking about building a website from scratch. So whether the result is null doesn't really matter. It's not like comparing different medicines or fertilizers or something where if none of them work you just do nothing; there is no "do nothing" option in this situation. So why not just take a simple effect measurement (e.g., proportion who signed up) and pick the layout that performs best? If that result is statistically significant, great, it means you picked the best one, and if it's not, it just means it doesn't matter which one you pick, so the one you picked is still fine. (And if you have an existing design and you're trying to see if a new one will be better, the null result just means "there's no reason to switch", which means the existing design is also fine.)

dooglius 1y ago

If the goal is for the company to maximize profit from having the best page, this is an instance of a very well-studied problem https://en.m.wikipedia.org/wiki/Multi-armed_bandit?useskin=vector https://en.m.wikipedia.org/wiki/Multi-armed_bandit?useskin=v... and one can do much better than statistical significance testing. (If the goal is to validate scientific theories, or there are other extenuating factors, things may be different.)

kdamica 1y ago

Hard disagree with this. Unlike medical experiments, the cost of being wrong startup experiments is very low: you thought there was a small effect and there was none. It’s usually just a matter of pushing one variant vs another and moving on. There are certainly scenarios where more rigor is appropriate, but usually those come from trying to figure out why you’re seeing a certain effect and how that should affect your overall company strategy. My advice for startups is to run lots of experiments, do bad statistics, and know that you’re going to have some false positives so that you don’t take every result as gospel.

bravesoul2 1y ago
The danger I think is less the numbers but what are you measuring makes sense. E.g. sure your A beats B in click through rate. But if the person then thinks fuck I was duped and closes the browser then that's no good.
aaron695 1y ago
[dead]
lemmsjid 1y ago
Huh, I agree with your last sentence but think the author did a good job of explaining that the cost of layout experimentation in a startup can grow over time if the results of the experiments are overstated. The first question for a startup should always be: is this work worth doing in the first place? Tinkering with layout can be a tempting but fruitless rabbit hole. Even if it doesn’t tie up resources it can lead to a false sense of progress and get product thinking stuck in local maximae.

NoahZuniga 1y ago

Why don't the p values in the first figure sum to 1?

begemotz 1y ago
Because that is not how p-values work. Each 'row' is a separate hypothesis test against some null hypothesis. And frankly the conclusion that "B is winner" because it has the lowest p-value also demonstrates the author's limited understanding. P-values tell you very little, and generally nothing about what you really want to know. In this context, you would be better off looking at effect size estimates of key metrics that would signal "better" - not the p-values.

mo_42 1y ago

> Back to the dashboard experiment: after you applied the Bonferroni correction you got... nothing. I guess you got something: Users are not sensitive to these changes, or that any effect is too small to detect with your current sample size/test setup. In a startup scenario, I'd quickly move on and possibly ship all developed options if good enough. Also, running A/B tests might not be the most appropriate method in such a scenario. What about user-centric UX research methods?

psychoslave 1y ago

P-value is something new for me, so the post is starting with prerequisites in mind that I miss. Tough I can go search by myself, would anyone have some online resources I can follow and test myself against to recommend, please?

the-mitr 1y ago
Here you go https://pmc.ncbi.nlm.nih.gov/articles/PMC5187603/ https://pmc.ncbi.nlm.nih.gov/articles/PMC5187603/
- psychoslave 1y ago
  Thanks, seems relevant, I would appreciate also resources on epistomological level, when and where it was devised and any historical context of development. Thanks again
  - pkhuong 1y ago
    https://en.wikipedia.org/wiki/P-value#History https://en.wikipedia.org/wiki/P-value#History

cornel_io 1y ago

Even though this post says exactly the thing that most Proper Analysts will say, and write long LinkedIn posts about where other Proper Analysts congratulate them on standing up for Proper Analysis in the face of Evil And Stupid Business Dummies who just want to make bad decisions based on too little data, it's wrong. The Jedi Bell Curve meme is in full effect on this topic, and I say this as someone who took years to get over the midwit hump and correct my mistaken beliefs. The business reality is, you aren't Google. You can't collect a hundred million data points for each experiment that you run so that you can reliably pick out 0.1% effects. Most experiments will have a much shorter window than any analyst wants them to, and will have far too few users, with no option to let them run longer. You still have to make a damned decision, now, and move on to the next feature (which will also be tested in a heavily underpowered manner). Posts like this say that you should be really, REALLY careful about this, and apply Bonferonni corrections and make sure you're not "peeking" (or if you do peek, apply corrections that are even more conservative), preregister, etc. All the math is fine, sure. But if you take this very seriously and are in the situation that most startups are in where the data is extremely thin and you need to move extremely fast, the end result is that you should reject almost every experiment (and if you're leaning on tests, every feature). That's the "correct" decision, academically, because most features lie in the sub 5% impact range on almost any metric you care about, and with a small number of users you'll never have enough power to pick out effects that small (typically you'd want maybe 100k, depending on the metric you're looking at, and YOU probably have a fraction of that many users). But obviously the right move is not to just never change the product because you can't prove that the changes are good - that's effectively applying a very strong prior in favor of the control group, and that's problematic. Nor should you just roll out whatever crap your product people throw at the wall: while there is a slight bias in most experiments in favor of the variant, it's very slight, so your feature designers are probably building harmful stuff about half the time. You should apply some filter to make sure they're helping the product and not just doing a random walk through design space. The best simple strategy in a real world where most effect sizes are small and you never have the option to gather more data really is to do the dumb thing: run experiments for as long as you can, pick whichever variant seems like it's winning, rinse and repeat. Yes, you're going to be picking the wrong variant way more often than your analysts would prefer, but that's way better than never changing the product or holding out for the very few hugely impactful changes that you are properly powered for. On average, over the long run, blindly picking the bigger number will stack small changes, and while a lot of those will turn out to be negative, your testing will bias somewhat in favor of positive ones and add up over time. And this strategy will provably beat one that does Proper Statistics and demands 95% confidence or whatever equivalent Bayesian criteria you use, because it leaves room to accept the small improvements that make up the vast majority of feature space. There's an equivalent and perhaps simpler way to justify this, which is to throw out the group labels: if we didn't know which one was the control and we had to pick which option was better, then quite obviously, regardless of how much data we have, we just pick the one that shows better results in the sample we have. Including if there's just a single user in each group! In an early product, this is TOTALLY REASONABLE, because your current product sucks, and you have no reason to think that the way it is should not be messed with. Late lifecycle products probably have some Chesterton's fence stuff going on, so maybe there's more of an argument to privilege the control, but those types of products should have enough users to run properly powered tests.

light_hue_1 1y ago

Don't do any of this. It's very outdated advice. And you're going to get it wrong anyway. These threshold adjustment methods were invented before we had access to reasonable computers. There's a far simpler method that covers every case: permutation tests. https://bookdown.org/ybrandvain/Applied-Biostats/perm1.html https://bookdown.org/ybrandvain/Applied-Biostats/perm1.html You shuffle the data. Say you want to know if viewing time is affected by color. Literally randomly shuffle viewing time and color. Then, look at that distribution. Is the data that you observed significant? As long as you shuffle everything related to your experiment and you don't double dip into the data you're going to get things right. This also has the big advantage that it doesn't overcorrect like traditional methods which apply such strict corrections that eventually it's impossible to get significant results. This post hasn't even begun scratching the surface on what can go wrong with traditional tests. Just don't use them. This has nothing to do with speed or rigor. Permutation tests are much simpler to run and faster to analyze. Sadly we keep teaching crappy statistics to our students.

syntacticsalt 1y ago
Permutation tests don't account for family-wise error rate effects, so I'm curious why you would say that "it doesn't overcorrect like traditional methods". I'm also curious why you say those "cover every case", because permutation tests tend to be underpowered, and also tend to be cumbersome when it comes to constructing confidence intervals of statistics, compared to something like the bootstrap. Don't get me wrong -- I like permutation tests, especially for their versatility, but as one tool out of a bunch of methods.

kookamamie 1y ago

…or, you could have a product that does not hinge on some micro-optimization of a website layout.

roncesvalles 1y ago
Exactly. This is what micro-optimization looks like on the product side.

Palmik 1y ago

This isn't just a startup thing. This is common also at FAANG. Not only are expriments commonly multi-arm, you also repeat your experiment (usually after making some changes) if the previous experiment failed / did not pass the launch criteria. This is further complicated by the fact that lauch criteria is usually not well defined ahead of time. Unless it's a complete slam dunk, you won't know until your launch meeting whether the experiment will be approved for launch or not. It's mostly vibe based, determined based on tens or hundreds of "relevant" metric movements, often decided on the whim of the stakeholder sitting at the lauch meeting.

setgree 1y ago
You're describing conditioning analyses on data. Gelman and Loken (2013) put it like this: > The problem is there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of fishing or examining multiple p values. We discuss in the context of several examples of published papers where data-analysis decisions were theoretically-motivated based on previous literature, but where the details of data selection and analysis were not pre specified and, as a result, were contingent on data.
netcan 1y ago
Is this terrible? The idea is not do do science. The idea is to loosely systematize and conceptualize innovation. To generate options and create a failure tolerant system. I'm sure improvements could be made... but this isn't about being a valid or invalid expirement.
- zeroCalories 1y ago
  It seems bad to me because you're giving yourself the illusion of effect.
- godelski 1y ago
  > The idea is not do do science. The idea is to loosely systematize and conceptualize innovation. Why are you acting like these are completely different frameworks? You have the same goals

welpo 1y ago

On the third point (peeking at p-values), I created an A/A test simulator that compares peeking vs not peeking in terms of false positive rate: https://stop-early-stopping.osc.garden/ https://stop-early-stopping.osc.garden/

freehorse 1y ago

I do not understand what the first tests are supposed to do. The author says the: > Your hypothesis is: layout influences signup behavior. I would expect that then the null hypothesis is that *layout does not influence signup behavior*. I would think that then an ANOVA (or an equivalent linear model) to be what tests this hypothesis, where you test the 4 layouts (or the 4 new layouts plus a control?) in one factor. If you get a significant p-value (no multiple tests required) you go on with post-hoc tests to look into comparisons between the different layouts (for 4 layouts, it should be 6 tests). But then you can use ways to control for multiple comparisons that are not as strict as just dividing your threshold by the number of comparisons, eg with Tukey's test. But here I assume there is a control (as in some for users are still presented the old layout?) and each layout is compared to that control? If I would see that distribution of p-values I would just intuitively think that the experiment is underpowered. P-values from null tests are supposed to be distributed uniformly between 0 and 1, while these cluster around 0.05. It rather seems like a situation that it is hard to make inferences from because of issues in designing the experiment itself. For example, I would rather have fewer layouts, driven by some expert design knowledge, rather than a lot of randomish layouts. The first increases statistical power, because the fewer tests you investigate, the less you have to adjust your p-values. But also, the fewer layouts you have, the more users you have per group (as the test is between groups) which also increases statistical power. The article is not wrong overall about how to control p-values etc, but I think that this knowledge is important not just to "do the right analysis" but, even more importantly, understand the limitations of an experimental design and structure it in a way that it may succeed in telling you something. To this end, g*power [0] is a useful tool that eg can let one calculate sample size in advance based on predicted effect size and power required. [0] https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psy...

majormajor 1y ago
The fraction of A/B tests I've seen personally that mentioned ANOVA at all is very small. Or thought that critically about experiment design. Understanding of p values is also generally poor; prob/stat education in engineering and business degrees seems to be the least-covered-or-respected type of math. Even at places that want to ruthlessly prioritize velocity over rigor I think it would be better to at least switch things up and worry more about effect size than p-value. Don't bother waiting to see if marginal effects are "significant" statistically if they aren't significant from the POV of "we need to do things that can 10x our revenue since we're a young startup."

calrain 1y ago

This feels like a billion dollar company problem, not a startup problem

gspetr 1y ago
These aren't mutually exclusive, unless you have an extremely narrow definition of a startup: https://techcrunch.com/2025/02/20/38-startups-have-become-unicorns-so-far-in-2024-heres-the-full-list/ https://techcrunch.com/2025/02/20/38-startups-have-become-un...
- calrain 1y ago
  You're right, but a startup has priorities a billion dollar company doesn't. And a billion dollar company can spend more time on min/max than a startup that is trying to get to PMF.
  - gspetr 1y ago
    A billion dollar company isn't invincible, you can still be zeroed out by an unfavorable market. https://en.wikipedia.org/wiki/Exodus_Communications https://en.wikipedia.org/wiki/Exodus_Communications In 2025 dollars their peak would have been $59.5B. To wit, this is roughly what Doordash is worth today, and that's a top-5 YC company. It died 9 months later.

vmesel 1y ago

Parabéns pelo conteúdo Thais!

snowstormsun 1y ago

The problem probably more often just is that the product/vision itself is not as good as it is sold to shareholders. No PM will like to push back on this, so they're stuck with things like optimizing the landing page because "clearly the business vision is flawless, it must be that users don't understand it correctly".

kgwgk 1y ago

> Users are randomly assigned to one of the four layouts and you track their activity. Your hypothesis is: layout influences signup behavior. > You plan ship the winner if the p-value for one of the layout choices falls below the conventional threshold of 0.05. Tests P-value B is winner 0.041 A is winner 0.051 D is winner 0.064 C is winner 0.063 What kind of comparison between the results for the four options makes each of them a likely winner? They all rank very well in whatever metric is being used! Or maybe the are they being compared with a fifth - much worse - alternative.

vjerancrnjak 1y ago
This kind of ranking is also not correct. You have to compare the outcomes , not their p values. Ranking by p values is just silly, just like ranking by avg metric is silly. Startups in general have to make decisions with high signal, thinking that 100 improvements of 1% p=0.05 will actually compound in an environment with so much noise is delusion. I’d say doing this kind of silliness in a startup is just ceremonial, helpful long term if people feel they are doing a good job optimizing a compounding metric, even though it never materializes.

meindnoch 1y ago

I've never seen AB-testing micro-optimisation affect the bottom line of a software company by more than 3%.

kgwgk 1y ago

> Setting a p-value threshold of 0.05 is equivalent to saying: "I’m willing to accept a 5% chance of shipping something that only looked good by chance." No, it means "I’m willing to ship something that if it was not better than the alternative it would have had only a 5% chance of looking as good as it did.”

phaedrus441 1y ago
This! I see this all the time in medicine.
- __MatrixMan__ 1y ago
  I'm under the impression that there was a paper on "ego depletion" has sort of become a poster child for p-value hacking. It has sort of poisoned the topic. Upon learning of such a hack one should rationally revert to a null hypothesis, not revert to the assumption that the hacked conclusion was false. As a climber I see ego depletion happen all the time. You find a crumbly hold, or get harassed by an insect, or whatever else it is, and you conclude that the next move is the crux. Then other people climb it and nobody agrees with you--that move was one of the easier ones. Anecdata, of course, I just wish we could learn from the bad science and then be washed clean, rather than be haunted by it.
wavemode 1y ago
Can you elaborate on the difference between your statement and the author's?
- likecarter 1y ago
  Author: 5% chance it could be same or worse Parent: 5% chance it could be same
  - esafak 1y ago
    @wavemode: In other words, the probability of it being the exactly the same is typically (for continuous random variables) zero, so we consider the tail probability; that of it being the same or more extreme. edit: Will the down voter please explain yourself? p-values are tail probabilities, and points have zero measure in continuous random variables.

aDyslecticCrow 1y ago

Whenever working with this kind of probabilities, i always throw in a python rand() as a comparison. It sanity checks the calculation of threshold with very low rist of miscalculation. Of course calculating the threshold properly needs to be done aswell... but a rand() is so quick and simple to add as a napkin check.

aDyslecticCrow 1y ago

begemotz 1y ago

> Your hypothesis is: layout influences signup behavior. This might be your hypothesis but this isn't the hypothesis that the p-value is related to. >Setting a p-value threshold of 0.05 is equivalent to saying: "I’m willing to accept a 5% chance of shipping something that only looked good by chance." P-values don't provide any information about "chance occurrence" but rather they test the probability of observing a particular outcome assuming a particular state of the world (i.e. the null hypothesis). > Bonferonni Besides being a very aggressive (i.e. conservative) correction, I would imagine that in industry, just as in science, the motivations to observe results will mean it wont be used. There are other way more reasonable corrections. > Avoid digging through metrics post hoc The reasonable solution is not to ignore the results that you found but to interpret them appropriately. Maybe the result actually does signal improved intention - alternatively maybe it is noise. Treat it as exploratory data. If that improved retention was real, is this important? Important enough to appropriately retest for?

b0a04gl 1y ago

[dead]

zug_zug 1y ago

Yes this is real, and even happens at larger companies. The surface issue is that when somebody has an incentive to self-measure their success then they have an incentive to overestimate (I increased retention by 14% by changing the shade of the "About Us" button!). Which means the root-cause issue is managers who create environments where self-reporting improvements without any rigor or any contrary perspective. Ultimately they are the ones foot-gunning themselves (by letting their team focus on false vanity metrics).

stared 1y ago

I think we should declare a moratorium on the use of p-values. If you don't understand what a p-value is, you shouldn't use it. If you do understand p-values, you're probably already moving away from them. The Bayesian approach makes much more sense. I highly recommend David MacKay's "Information Theory, Inference and Learning Algorithms," as well as "Bayesian Methods for Hackers": https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers https://github.com/CamDavidsonPilon/Probabilistic-Programmin.... At the same time, startups aren't science experiments. Our goal is not necessarily to prove conclusively whether something is statistically "better". Rather, our goal is to solve real-world problems. Suppose we run an A/B test, and its result indicates B is better according to whatever statistical test we've used. In this scenario, we will likely select B—frankly, regardless of whether B is truly better or merely indistinguishable from A. However, what truly matters in practice are the metrics we choose to measure. Picking the wrong metric can lead to incorrect conclusions. For example, suppose our data shows that users spend, on average, two more seconds on our site with the new design (with p < 0.001 or whatever). That might be a positive result—or it could simply mean the new design causes slower loading or more confusion, frustrating users instead of benefiting them.

chickenzzzzu 1y ago

Just ship useful stuff. Stop with all this overanalysis.

rossdavidh 1y ago

Nice stuff for explaining what p-hacking is, but the big difference between medical "does this drug work" tests and software "does this layout improve signups" tests, is that in the latter, the downsides are much less. Most pharmaceuticals have undesirable side-effects, and they always cost money to produce as well as develop; a layout change (unless it has some huge code complexity, which it typically won't) has very little. The cost of developing it has already been paid. If it is more likely to help than not, the right decision is to ship it. This makes the importance of not p-hacking far less for software, than in the pharmaceutical case. You may have some kind of ongoing cost (code complexity or server load or whatever), but in the example given, I would not expect that, and it wasn't addressed in the article. The cost of a false positive is an important factor. If you get ten probably-helpful, or even ten 50%-likely-to-be-helpful changes out, instead of 2 or 3 definitely-helpful changes, this is a net benefit.