I've also observed the problem where analysts or product managers cherry pick the data that validates their assumptions. A "funny" example I ran into was produ

I've also observed the problem where analysts or product managers cherry pick the data that validates their assumptions. A "funny" example I ran into was product wanted to ship a notification that "test better" (i.e., had a higher interaction rate) in A/B testing. Some developers pointed out that the notification was likely seeing higher click-throughs because it was incredibly vague and users clicked through to figure out what it was about, and this would likely result in user frustration. Well, this insight was ignored and the notification change was shipped. Our notification opt-out rate spiked a few weeks later.