3 ms·

I'm not sure what argument you're referring to? I quoted two claims from the parent comment and refuted those based off of industry experience. > I have seriou

I'm not sure what argument you're referring to? I quoted two claims from the parent comment and refuted those based off of industry experience. > I have serious doubts that a company can run and do well designed statistical experiments when academic experts are plagued by p-hacking and other foot guns. By "company" are you referring to individual advertisers or digital advertising platforms? Why do you doubt that either are incapable of running well-designed experiments? It's in both parties best interest to spend ad dollars as efficiently as possible. While it's certainly possible that every one of the 100 million+ ad experiments ran over the past decade have been fatally flawed, it's highly unlikely. I guess the alternative scenario is that digital advertising is one giant hoax being kept secret by the millions of employees who work in the industry, but given how easily tech news leaks, this also seems extremely improbable.

User23 6y ago
Another option is that the experiment design was wrong for every single one of those experiments. Sounds unlikely right? Running the same broken experiment any number of times won't change that it's broken. So if these "100 million+" ad experiments are more or less copies of each other they could all be wrong. How likely is that? I don't know, but I do know that at a big Silicon Valley tech company that you've definitely heard of I was involved in an exercise where we were told to create an experiment. The type A take charge person in our group did what they do, which in this case was talking over and shutting down the Los Alamos trained particle physicist who was trying to explain to her why her "experiment" could never show anything.