I've seen that culture too. I don't want to throw my colleagues under the bus but the other teams at my company that have done A/B testing to compare design cho

I've seen that culture too. I don't want to throw my colleagues under the bus but the other teams at my company that have done A/B testing to compare design choices don't perform proper controls or pre-register hypothesis. Eg. when they want to divide the two groups, they often divide them based on some classification (gender, age, geography etc.) rather than dividing them up completely randomly. In my team we try to be methodical. I'm just a lowly engineer but one of my team mates is a statistician and another is a PhD student. We know we need to pre-register the hypothesis and at what significance we're testing for first, divide the groups randomly and run the experiment for a set time period. We've gotten a lot more negative results and been proven wrong in our guesses than the other teams. For some reason I take pride in that.