8 ms·
> No one is willing to run the experiments necessary The people that would have the power to run this experiment have their entire careers depending on things
by Nextgrid 6y ago
> No one is willing to run the experiments necessary
The people that would have the power to run this experiment have their entire careers depending on things staying as-is. Running the experiment carries a significant risk of exposing that the advertising operations they're responsible for provide much less ROI than they pretend it does.
The unwillingness of anyone to run such an experiment is already an answer. Why wouldn't someone jump at an opportunity to prove the thing/service they provide actually works, unless they were unsure about it themselves?
- deleted 6y ago[deleted]
- garciasn 6y agoI work in the field. Incrementality testing is a big part of marketing measurement at any reputable agency. Any claims to the contrary are FUD. That said, companies like P&G, Airbnb, and Uber, which are oft-cited as examples of digital not being worth it, fail to understand their own brand recognition and organic power, built through prior marketing efforts, as key to their current standing. Sure, TODAY, it doesn’t have the impact they’d like it to have but the investments PRIOR were key to ensuring their success.
- quacked 6y agoQuit and do something that isn't fundamentally detrimental to everyone's quality of life. http://jacek.zlydach.pl/blog/2019-07-31-ads-as-cancer.html http://jacek.zlydach.pl/blog/2019-07-31-ads-as-cancer.html
- dasil003 6y agoBrand is a lot more than marketing spend, and not all brands are equal. Google and Facebook have companies that depend primarily on performance marketing spend over a barrel.
- andrewjl 6y ago> as key to their current standing. It seems like it would be important to measure this standing and turn down spend once it is reached. I can see how there's a lack of incentive to help large advertising spenders understand this.
- TeMPOraL 6y agoHow do you determine a reputable agency though? I worked next desk to people running a small ad agency. Because we shared office (and I found them an intern), I got a very good look at how they're working. What I've seen can be summarized as: people who have zero clue or interest in statistics writing "reports", with "graphs" they don't even understand beyond "pointing up = good", proving positive ROI to customers - who also have zero clue or interest in understanding the numbers in the report, and not enough visibility into the whole funnel to independently check attribution. Both the agency and the clients were engaging in a shared and completely unjustified fiction of positive advertising return - and as long as both sides were happy, the money kept flowing. I've been long since suspecting a lot of advertising on-line looks like that. Every now and then, I see evidence in favor. Like that good ol' Optimizely debacle, where it turned out Optimizely was structurally optimized to help people make invalid A/B tests, that erred on the side of concluding the interventions were working[0]. And sure, big brands with some superstar ad teams probably do this right. But I think there's enough slack in most businesses that advertising spend can get quite far detached from actual ROI without anyone noticing (and with plenty of people happily riding the gravy train). -- [0] - https://news.ycombinator.com/item?id=10872359 https://news.ycombinator.com/item?id=10872359
- johnrgrace 6y agoHave you ever engaged with the large advertizing holding companies? They have teams of HIGHLY acomplished data scientists, boutique vendors for specific niches etc. Even if you don't pay for it there will be some people skilled at statistics looking at reports. If you pay for it, there will be data scientists doing analysis. The large ad firms deliver a very good product but it usually isn't cheap.
- TeMPOraL 6y agoI haven't, and I don't doubt that's true (at least in many cases; I've learned not to underestimate the chances of big companies ending up running scams either). I've personally dealt mostly with the other end - the individuals and small agencies - and what I saw revealed total lack of necessary competences for the reports to be corresponding to reality. Perhaps it's understandable - after all, people who have the required knowledge likely end up working for big advertisers, or in entirely different fields. But small business owners don't pick these big advertising companies either.
- baron_harkonnen 6y ago> Incrementality testing is a big part of marketing measurement at any reputable agency In a huge number of cases these tests are run by people that don't have the statistical background to property run and understand these tests, in the remaining cases there is almost never follow through to ensure that the results of the test have persisted after the experiment. People will say "oh this test shows 10% improvement, and this one does too! and this one! and this one!" But then you should see a nearly 50% improvement just because of ad spend and you almost never do. Nobody ever check that all the numbers add up, they just want the numbers that someone reported to feel like they are sound enough to hold up to scrutiny but scrutiny is never applied.
- adwww 6y agoI used to work in this industry. A small tech team investigated fraud on our platform and developed a system that was pretty robust at detecting and potentially shutting it down. But literally nobody was interested - even the people advertising don't want to know. The people spending money are typically networks, media buyers, ad agencies, etc, far removed from the actual brand. There are so many parties who want a slice of the brand's cash that they are all long past caring about whether the ad is being viewed by a human or not.
- walshemj 6y agoAgency or client side? I assure you the big brands CFO's take a huge amount of interest in what is spent on advertising.
- syshum 6y agoIn my experience the Sales and Marketing teams of any large company have the loosest restrictions on how they account for their spending. Other dept;s like IT have to justify every penny, but Sales and Marketing not sooo much
- nkozyra 6y agoI think there's a very loose belief that sales is how money is made, while they people that build and test products are pure expense. I've seen it not just in the phenomenon you describe, but also in layoffs and restructurings.
- tsjq 6y agoYes. Cost centre and profit centre.
- tsjq 6y agoYes. Sales and marketing : "profit centers" . Ever increasing budgets. Engineering , R&D, Manufacturing, Tech, Testing : "cost centers". Ever tightening budgets. MBA, Beancouting 101
- johnrgrace 6y agoThe experments are run. Why don't you look at the work that comes out of the Advertizing research foundation.
- x0x0 6y agoUber was just incredibly incompetent to not audit their ad spend at all. I worked at an ad company. It was an absolutely standard metric to eg geo-fence ads out of a state or two for 3 months to demonstrate the impact of ads. This isn't easily externally visible, but tests like this are standard practice. Particularly in the app install space, which is sketchy as hell once you stop buying from the top handful of vendors, buyers should be auditing by a couple million in annual spend. To get to $150m without looking hard at big chunks of their spend is just plain arrogance and/or incompetence.
- abhgh 6y agoI partly work in this space, and I can confirm that geo-fencing over a time period is the probably the most transparent and convincing (to the client) way to do this. Essentially you mark out local regions (as small as zip codes or as large as states, or something else in this spectrum), and ensure that your advertisements only show within certain "fences". And then you compare old and new numbers within those fences. The technical aspect aside, there are a lot of "soft" factors that help: regular communication of easily consumable numbers/graphs/metrics to the client, calling out inconsistencies, etc. In other words, this is far from a fire-and-forget exercise; this can be an intense monitoring and client-engagement exercise. The regular client engagement sometimes helps you in pinpointing cause-effect in the observations too. For ex company X might decide to advertise less of product Y in a certain state Z because of new laws there. But the team you're interacting with might not be aware of this change, or might not be cognizant of its potential impact on an advt. campaign. Regular dialogue helps here since you might observe a change in sales trends, and bring it up in a meeting - and the client team might be able to then rationalize the change. This is healthy for both parties.
- justapassenger 6y agoIt’s like saying that no one who works on reliability of the systems is willing to run experiment to throw 100% errors for a month, because running experiment like that may show that reliability of the systems doesn’t matter. And then using data from a one system that went down and nothing happened as a proof that systems reliability doesn’t matter at all, and it’s huge scam by engineers.
- throwaway201103 6y ago> Why wouldn't someone jump at an opportunity to prove the thing/service they provide actually works, unless they were unsure about it themselves? Because advertising in some form certainly works. If you can determine that approach "A" that everybody is doing is actually a waste of money but approach "B" is effective, then you can develop services around approach "B" and market them based on these findings.
- Nextgrid 6y agoNobody is arguing that advertising doesn't work at all, but my argument (which I think a lot of other people agree with) is that the effectiveness of online advertising (and maybe advertising in general) is overblown.
- masonic 6y agoAdvertising that does not actually reach the claimed consumer, by definition, cannot possibly work.
- seppin 6y agoThis is no industry with as much snake oil BS as adtech. It also happens to be the sole form of revenue all of the largest tech firms enjoy. No wonder there hasn't been a real audit. Also, all those marketing execs buying all the ads would be out of a job too, so they aren't speak up either.
- kevinmchugh 6y ago> It also happens to be the sole form of revenue all of the largest tech firms enjoy I don't think this is true unless you qualify it a bit. Apple, Amazon, Microsoft, don't make most of their money on ads. Even Google has other revenue streams.
- seppin 6y ago> Even Google has other revenue streams. How long do they survive in their current form relying solely on them?
- kevinmchugh 6y agoDays, probably, but that's not the point
- seppin 6y agoIt's exactly my point. Outside of Apple, all the biggest tech firms are 100% reliant on a form of income that is, at best, partially fraudulent. More realistically, almost entirely so.
- hertzrat 6y ago> The people that would have the power to run this experiment have their entire careers depending on things staying as-is. Any large company could invest in some experimentation, whether their marketing directors want it or not. It makes sense that at least a few do but just don’t publish the results
- kumarvvr 6y agoYep. Like the old adage goes, You cant make a man understand something, if his job depends on him not understanding it.
- ogre_codes 6y ago> The unwillingness of anyone to run such an experiment is already an answer. Why wouldn't someone jump at an opportunity to prove the thing/service they provide actually works, unless they were unsure about it themselves? Or they have run the experiment and the results haven't lines up with their own personal biases so they were discarded.
- jedberg 6y agoExactly. That’s what the episode says too. The only people who could run the experiments hinge their entire career and industry on the results.
- nicbou 6y ago"It is difficult to get a man to understand something, when his salary depends upon his not understanding it!"
- denton-scratch 6y ago"If you believed something different you wouldn't be sitting where you're sitting." ~Chomsky to Andrew Marr.
- Kyro38 6y ago~ Upton Sinclair
- fortran77 6y agoAnd, unlike most Internet Quotes, this one is apparently real! https://quoteinvestigator.com/2017/11/30/salary/ https://quoteinvestigator.com/2017/11/30/salary/
- dllthomas 6y ago> Why wouldn't someone jump at an opportunity to prove the thing/service they provide actually works, unless they were unsure about it themselves? Because everyone is already acting like they know it works, so the only way that experiment can change things is in a way that's bad for the person in question. In that situation, they should (from a local, selfish perspective) be resisting even if they're awfully sure it does work (and perhaps even if they're right!). Given that, I don't think the behavior has already given us an answer.
- deleted 6y ago[deleted]
- xapata 6y agoThe analysis is often performed by a different team, thus no conflict of interest.
- Paradigma11 6y agoAnd if the team doesn't get the desired result there will be a different team next time.
- xapata 6y agoDepends which team has the power. Sometimes it's the team auditing the results that's more respected.
- disgruntledphd2 6y agoFacebook's ad research team have run a lot of experiments. The goal was to demonstrate an impact of in-store sales from internet advertising. They did the first studies in about 2008-10, and have continued running these studies ever since. They even built a tool so that advertisers can run these studies, and get a sense for the incremental impact of their ad. Google have a similar (less full-featured) system. Really, it's the rest of the ecosystem that has much of the fraud, and I think that a lot of people in the industry are aware of this.