1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes ha

1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption. Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect). More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation. 2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.