I am strongly of the opinion that even if you got a non-significant result doing your A/B test, assuming we're testing something of no cost to change (like a we

I am strongly of the opinion that even if you got a non-significant result doing your A/B test, assuming we're testing something of no cost to change (like a web design that has already been deployed for the sake of the test), your point estimate for the effect is the "best data you have" and you should act on it. To provide an example, you run a test and find Page 1 outperformed Page 2 by a factor of 1.1, but that was non-significant at your desired significance level (say, p=0.05 and power = 0.8). You should deploy page 1 instead of page 2, assuming there are no other costs associated with deploying page 1 instead of page 2, because your BEST GUESS RIGHT NOW is that page 1 is 1.1x better than page 2.