3 ms·

That's great, thanks for your experience, but can't comment on the argument provided? Isn't it possible that these experiments you talk about are fatally flawe

• Spivak 6y ago
  That’s kinda a weird take when there’s every incentive to be the the person/manager/director who saves tens to hundreds of millions of dollars in ad spend. Sure marketing people want to protect their jobs but the odds of “advertising is entirely worthless” being this closely guarded secret kept by millions of people in the industry or being a collective delusion is pretty darn low. The reward for defectors is just too high.
• skronch 6y ago
  IMO large social media platforms are actually much more capable of running these experiments than academics. (Disclaimer: I work on ads at one of these companies). Platforms can accurately determine who engaged with an ad (basic logging on their sites), they have infrastructure to create statistically balanced ad experiments, and can also accurately determine whether a conversion happened (either through a conversion pixel or through data brokers). Running these tests on behalf of advertising clients, or for internal research is fairly standard. If we couldn't prove statistically that our ads were working, I would have left a long time ago.
  • tfehring 6y ago
    I think the difference from academic studies that fail to replicate is less about capability and more about the fact that experimentation in adtech is fundamentally straightforward and occurs under far more idealized conditions - those you mentioned, plus massive sample sizes and easy iteration - than most academic research. Of course, this just reinforces your point that experimentation in adtech is largely not subject to the same issues that have fueled the replication crisis in academia. (I'm a data scientist but not in adtech.)
• crakhamster01 6y ago
  I'm not sure what argument you're referring to? I quoted two claims from the parent comment and refuted those based off of industry experience. > I have serious doubts that a company can run and do well designed statistical experiments when academic experts are plagued by p-hacking and other foot guns. By "company" are you referring to individual advertisers or digital advertising platforms? Why do you doubt that either are incapable of running well-designed experiments? It's in both parties best interest to spend ad dollars as efficiently as possible. While it's certainly possible that every one of the 100 million+ ad experiments ran over the past decade have been fatally flawed, it's highly unlikely. I guess the alternative scenario is that digital advertising is one giant hoax being kept secret by the millions of employees who work in the industry, but given how easily tech news leaks, this also seems extremely improbable.
  • User23 6y ago
    Another option is that the experiment design was wrong for every single one of those experiments. Sounds unlikely right? Running the same broken experiment any number of times won't change that it's broken. So if these "100 million+" ad experiments are more or less copies of each other they could all be wrong. How likely is that? I don't know, but I do know that at a big Silicon Valley tech company that you've definitely heard of I was involved in an exercise where we were told to create an experiment. The type A take charge person in our group did what they do, which in this case was talking over and shutting down the Los Alamos trained particle physicist who was trying to explain to her why her "experiment" could never show anything.