5 ms·

> No one can actually prove it has any ROI at all. No one is willing to run the experiments necessary. This is entirely false. I work in adtech, and almost all

> No one can actually prove it has any ROI at all. No one is willing to run the experiments necessary. This is entirely false. I work in adtech, and almost all companies run experiments in order to optimize their ad spend (everything from Ad A vs Ad B, to Ad vs No Ad, to Channel A vs Channel B, and more). This isn't to say that advertising always produces ROI. Quite often, experiments will be run that will show a certain strategy isn't performing well, and the company will adjust accordingly. It's incredibly naïve to think that companies are flushing half a trillion dollars a year down the toilet on advertising without any attempt to validate their investments.

voxl 6y ago
That's great, thanks for your experience, but can't comment on the argument provided? Isn't it possible that these experiments you talk about are fatally flawed? I have serious doubts that a company can run and do well designed statistical experiments when academic experts are plagued by p-hacking and other foot guns.
- Spivak 6y ago
  That’s kinda a weird take when there’s every incentive to be the the person/manager/director who saves tens to hundreds of millions of dollars in ad spend. Sure marketing people want to protect their jobs but the odds of “advertising is entirely worthless” being this closely guarded secret kept by millions of people in the industry or being a collective delusion is pretty darn low. The reward for defectors is just too high.
- skronch 6y ago
  IMO large social media platforms are actually much more capable of running these experiments than academics. (Disclaimer: I work on ads at one of these companies). Platforms can accurately determine who engaged with an ad (basic logging on their sites), they have infrastructure to create statistically balanced ad experiments, and can also accurately determine whether a conversion happened (either through a conversion pixel or through data brokers). Running these tests on behalf of advertising clients, or for internal research is fairly standard. If we couldn't prove statistically that our ads were working, I would have left a long time ago.
  - tfehring 6y ago
    I think the difference from academic studies that fail to replicate is less about capability and more about the fact that experimentation in adtech is fundamentally straightforward and occurs under far more idealized conditions - those you mentioned, plus massive sample sizes and easy iteration - than most academic research. Of course, this just reinforces your point that experimentation in adtech is largely not subject to the same issues that have fueled the replication crisis in academia. (I'm a data scientist but not in adtech.)
- crakhamster01 6y ago
  I'm not sure what argument you're referring to? I quoted two claims from the parent comment and refuted those based off of industry experience. > I have serious doubts that a company can run and do well designed statistical experiments when academic experts are plagued by p-hacking and other foot guns. By "company" are you referring to individual advertisers or digital advertising platforms? Why do you doubt that either are incapable of running well-designed experiments? It's in both parties best interest to spend ad dollars as efficiently as possible. While it's certainly possible that every one of the 100 million+ ad experiments ran over the past decade have been fatally flawed, it's highly unlikely. I guess the alternative scenario is that digital advertising is one giant hoax being kept secret by the millions of employees who work in the industry, but given how easily tech news leaks, this also seems extremely improbable.
  - User23 6y ago
    Another option is that the experiment design was wrong for every single one of those experiments. Sounds unlikely right? Running the same broken experiment any number of times won't change that it's broken. So if these "100 million+" ad experiments are more or less copies of each other they could all be wrong. How likely is that? I don't know, but I do know that at a big Silicon Valley tech company that you've definitely heard of I was involved in an exercise where we were told to create an experiment. The type A take charge person in our group did what they do, which in this case was talking over and shutting down the Los Alamos trained particle physicist who was trying to explain to her why her "experiment" could never show anything.
baron_harkonnen 6y ago
> without any attempt to validate their investments. I've also worked in adtech, for quite a long time on all sides, and my experience is that far more companies do research to justify their ad spend not validate it. I have managed plenty of A/B tests in my day, each one claiming to show some improvement, 10%, 15% etc (some, as you said, showing no improvement). Even though these tests are often run "correctly", essentially nobody goes back and asks "wait, we had a 10% increase multiple times in the past year, but is our <metric> really showing the cumulative improvement we expected?" The greatest trick in the industry is that because VC are pouring money into everything, everywhere, every metric appears to grow. At every startup, everywhere, pre-pandemic, numbers were going up because people were pouring money into the system. I've worked at companies who I know for a fact their adtech product cannot and does not work, yet their business continues to explode because in recent months nearly all ad spend has been on digital ads. I've talked to companies whose entire function is bidding optimization who literally do not understand how to optimize bidding given the information you have. Absolutely people are running "experiments" but the function of the "experiments" is to justify that the ad team is worth having, and then that the VP of marketing is doing their job and then that the CEO has hired some super smart people, and then that the VCs might have really found a unicorn. Everyone sees what they want and no one really wants to ask the question "wait, does this really work? are these tests really able to capture the complexity of the environment?" And if you are one of those ornery people that insists on probing into the details and seeing if any of this is working, you will eventually get fired. Ad tech is largely a scam, but a huge number of rich and smart people benefit from the illusion that it is not, so we continue to see experiments showing that everything works as expected.
- crakhamster01 6y ago
  I agree that there a lot of (smaller) players in the ad tech space who are scamming companies out of money. And especially if you've been in this industry for a while, this practice was much more prevalent a decade ago. I don't think your view is representative of how most F500 companies, or many of the new DTC brands, invest on major ad tech platforms though (FB/G/Snap/etc.). Experienced marketers try to measure metrics as close to core business KPIs as possible, and comparing ads based off of simple A/B tests is increasingly becoming a technique of the past. Measuring the incrementality of ad campaigns through long-term holdout groups gives companies a much more accurate read of their investment, while bringing higher statistical rigor as well. So rather than looking for cumulative increases on <metric> after advertising for a year, you could instead point to group A (who saw no ads) and see that group B (who saw ads) drove a 1.5x higher <metric>. I'm not sure how you can say advertising is a scam when there are clear examples of popular brands that most likely wouldn't exist today without digital ads? Take Allbirds as an example. There are dozens of consumer shoe brands that are already in physical stores and have higher brand recognition; how can you argue that advertising didn't help them cut through the noise and grow their bottom line?
ogre_codes 6y ago
> I work in adtech, and almost all companies run experiments in order to optimize their ad spend ... It's incredibly naïve to think that companies are flushing half a trillion dollars a year down the toilet on advertising without any attempt to validate their investments. We have a very large and conspicuous example here that suggests that Uber hasn't been doing this. Running experiments like you suggest is difficult and to do them right requires some fairly specialized knowledge to do correctly. I'd doubt more than a small percentage of companies have the expertise to effectively run any kind of advertising experiment that returns useful data. Most businesses lack that knowledge and rely on metrics provided by the people selling the advertising—who are likely to provide self-serving numbers. While I'm sure some of the bigger F500 companies do a good job validating their ad spend, I most companies don't even know how. Certainly the majority of small businesses have no idea how effective their advertising spend is aside from very crude word-of-mouth feedback. I suspect this creeps way up into the Fortune 500 as well.
- crakhamster01 6y ago
  > We have a very large and conspicuous example here that suggests that Uber hasn't been doing this. I'm not sure this article is the smoking gun that the author makes it out to be. By 2017 Uber had reached saturation in what could reasonably be considered their addressable market. From my perspective, turning off app install campaigns and not seeing any dip in acquisition validates this position. Meanwhile, their other major app, UberEats, is a business that exists in a highly competitive market, and they continue to invest in app marketing in order to grow that division. If the takeaway from this article was that Uber discovered app marketing was useless, I doubt you'd see them make that same mistake again. > While I'm sure some of the bigger F500 companies do a good job validating their ad spend, most companies don't even know how. If anything, I think bigger companies are more prone to overspending/spending inefficiently. While they do experiment to try and optimize strategy, there is some business inertia that gives them the flexibly to move a little slower, since one poor marketing decision will not sink the business overnight. Conversely, small businesses don't have the resources to investment as much in marketing. This leads to poorly run ad campaigns, but rather than continuing to invest in a bad strategy, they typically just kill the effort altogether. Most small businesses don't need complex experiments to understand how marketing effects their bottom line - poor ROI is a lot more apparent when you're low on funds. That's also why it's in an ad tech platform's best interest to make efficient advertising as accessible as possible.

voxl 6y ago

That's great, thanks for your experience, but can't comment on the argument provided? Isn't it possible that these experiments you talk about are fatally flawed? I have serious doubts that a company can run and do well designed statistical experiments when academic experts are plagued by p-hacking and other foot guns.

Spivak 6y ago
That’s kinda a weird take when there’s every incentive to be the the person/manager/director who saves tens to hundreds of millions of dollars in ad spend. Sure marketing people want to protect their jobs but the odds of “advertising is entirely worthless” being this closely guarded secret kept by millions of people in the industry or being a collective delusion is pretty darn low. The reward for defectors is just too high.
skronch 6y ago
IMO large social media platforms are actually much more capable of running these experiments than academics. (Disclaimer: I work on ads at one of these companies). Platforms can accurately determine who engaged with an ad (basic logging on their sites), they have infrastructure to create statistically balanced ad experiments, and can also accurately determine whether a conversion happened (either through a conversion pixel or through data brokers). Running these tests on behalf of advertising clients, or for internal research is fairly standard. If we couldn't prove statistically that our ads were working, I would have left a long time ago.
- tfehring 6y ago
  I think the difference from academic studies that fail to replicate is less about capability and more about the fact that experimentation in adtech is fundamentally straightforward and occurs under far more idealized conditions - those you mentioned, plus massive sample sizes and easy iteration - than most academic research. Of course, this just reinforces your point that experimentation in adtech is largely not subject to the same issues that have fueled the replication crisis in academia. (I'm a data scientist but not in adtech.)
crakhamster01 6y ago
I'm not sure what argument you're referring to? I quoted two claims from the parent comment and refuted those based off of industry experience. > I have serious doubts that a company can run and do well designed statistical experiments when academic experts are plagued by p-hacking and other foot guns. By "company" are you referring to individual advertisers or digital advertising platforms? Why do you doubt that either are incapable of running well-designed experiments? It's in both parties best interest to spend ad dollars as efficiently as possible. While it's certainly possible that every one of the 100 million+ ad experiments ran over the past decade have been fatally flawed, it's highly unlikely. I guess the alternative scenario is that digital advertising is one giant hoax being kept secret by the millions of employees who work in the industry, but given how easily tech news leaks, this also seems extremely improbable.
- User23 6y ago
  Another option is that the experiment design was wrong for every single one of those experiments. Sounds unlikely right? Running the same broken experiment any number of times won't change that it's broken. So if these "100 million+" ad experiments are more or less copies of each other they could all be wrong. How likely is that? I don't know, but I do know that at a big Silicon Valley tech company that you've definitely heard of I was involved in an exercise where we were told to create an experiment. The type A take charge person in our group did what they do, which in this case was talking over and shutting down the Los Alamos trained particle physicist who was trying to explain to her why her "experiment" could never show anything.

baron_harkonnen 6y ago

> without any attempt to validate their investments. I've also worked in adtech, for quite a long time on all sides, and my experience is that far more companies do research to justify their ad spend not validate it. I have managed plenty of A/B tests in my day, each one claiming to show some improvement, 10%, 15% etc (some, as you said, showing no improvement). Even though these tests are often run "correctly", essentially nobody goes back and asks "wait, we had a 10% increase multiple times in the past year, but is our <metric> really showing the cumulative improvement we expected?" The greatest trick in the industry is that because VC are pouring money into everything, everywhere, every metric appears to grow. At every startup, everywhere, pre-pandemic, numbers were going up because people were pouring money into the system. I've worked at companies who I know for a fact their adtech product cannot and does not work, yet their business continues to explode because in recent months nearly all ad spend has been on digital ads. I've talked to companies whose entire function is bidding optimization who literally do not understand how to optimize bidding given the information you have. Absolutely people are running "experiments" but the function of the "experiments" is to justify that the ad team is worth having, and then that the VP of marketing is doing their job and then that the CEO has hired some super smart people, and then that the VCs might have really found a unicorn. Everyone sees what they want and no one really wants to ask the question "wait, does this really work? are these tests really able to capture the complexity of the environment?" And if you are one of those ornery people that insists on probing into the details and seeing if any of this is working, you will eventually get fired. Ad tech is largely a scam, but a huge number of rich and smart people benefit from the illusion that it is not, so we continue to see experiments showing that everything works as expected.

crakhamster01 6y ago
I agree that there a lot of (smaller) players in the ad tech space who are scamming companies out of money. And especially if you've been in this industry for a while, this practice was much more prevalent a decade ago. I don't think your view is representative of how most F500 companies, or many of the new DTC brands, invest on major ad tech platforms though (FB/G/Snap/etc.). Experienced marketers try to measure metrics as close to core business KPIs as possible, and comparing ads based off of simple A/B tests is increasingly becoming a technique of the past. Measuring the incrementality of ad campaigns through long-term holdout groups gives companies a much more accurate read of their investment, while bringing higher statistical rigor as well. So rather than looking for cumulative increases on <metric> after advertising for a year, you could instead point to group A (who saw no ads) and see that group B (who saw ads) drove a 1.5x higher <metric>. I'm not sure how you can say advertising is a scam when there are clear examples of popular brands that most likely wouldn't exist today without digital ads? Take Allbirds as an example. There are dozens of consumer shoe brands that are already in physical stores and have higher brand recognition; how can you argue that advertising didn't help them cut through the noise and grow their bottom line?

ogre_codes 6y ago

> I work in adtech, and almost all companies run experiments in order to optimize their ad spend ... It's incredibly naïve to think that companies are flushing half a trillion dollars a year down the toilet on advertising without any attempt to validate their investments. We have a very large and conspicuous example here that suggests that Uber hasn't been doing this. Running experiments like you suggest is difficult and to do them right requires some fairly specialized knowledge to do correctly. I'd doubt more than a small percentage of companies have the expertise to effectively run any kind of advertising experiment that returns useful data. Most businesses lack that knowledge and rely on metrics provided by the people selling the advertising—who are likely to provide self-serving numbers. While I'm sure some of the bigger F500 companies do a good job validating their ad spend, I most companies don't even know how. Certainly the majority of small businesses have no idea how effective their advertising spend is aside from very crude word-of-mouth feedback. I suspect this creeps way up into the Fortune 500 as well.

crakhamster01 6y ago
> We have a very large and conspicuous example here that suggests that Uber hasn't been doing this. I'm not sure this article is the smoking gun that the author makes it out to be. By 2017 Uber had reached saturation in what could reasonably be considered their addressable market. From my perspective, turning off app install campaigns and not seeing any dip in acquisition validates this position. Meanwhile, their other major app, UberEats, is a business that exists in a highly competitive market, and they continue to invest in app marketing in order to grow that division. If the takeaway from this article was that Uber discovered app marketing was useless, I doubt you'd see them make that same mistake again. > While I'm sure some of the bigger F500 companies do a good job validating their ad spend, most companies don't even know how. If anything, I think bigger companies are more prone to overspending/spending inefficiently. While they do experiment to try and optimize strategy, there is some business inertia that gives them the flexibly to move a little slower, since one poor marketing decision will not sink the business overnight. Conversely, small businesses don't have the resources to investment as much in marketing. This leads to poorly run ad campaigns, but rather than continuing to invest in a bad strategy, they typically just kill the effort altogether. Most small businesses don't need complex experiments to understand how marketing effects their bottom line - poor ROI is a lot more apparent when you're low on funds. That's also why it's in an ad tech platform's best interest to make efficient advertising as accessible as possible.