The cited paper above, and even the influential 2018 Facebook paper by Gordon et al that tries to lobby for randomized controlled trials both disagree with your

The cited paper above, and even the influential 2018 Facebook paper by Gordon et al that tries to lobby for randomized controlled trials both disagree with your claim. The observed incremental lift on digital conversion outcomes (particularly account signups and purchases from an online shopping cart) is (median) somewhere around 8% - 17% based on close to 500 different very large field experiments. The distribution is skewed so that the ~30% standard deviation represents mostly outcomes higher and only somewhere between 9% and 22% showed negative lift (which, as the Facebook paper points out, can also be related to the difficulty of observational statistical methods used for post facto causal inference). 8% lift is quite significant. There’s no open secret of “small impact” at all. https://www.kellogg.northwestern.edu/faculty/gordon_b/files/fb_comparison.pdf https://www.kellogg.northwestern.edu/faculty/gordon_b/files/...