5 ms·

Facebook knew for years ad reach estimates were based on ‘wrong data’

180 points by hashingroll 6y ago

squibbles 6y ago
I have seen many people fail to realize (and sometimes to understand at all) that taking measurements of large numbers of people is always fraught with errors of all sorts. No matter how hard you try, it is like the universe always pushes back to make sure the data are imperfect. Sometimes those imperfections are acceptably small, and other times the imperfections are uncomfortably large. In my opinion, anyone involved in such large-scale data collection and analysis should acknowledge the inevitability of error and provide disclaimers about potential sources of error.
- fab1an 6y ago
  Yes, but if the error consistently results in you getting paid more than you should have, it's not really an error, is it? No, we would in fact call that fraud.
  - hari_seldon_ 6y ago
    +1, the issue here is not that errors can occur but instead how a company responds. It’s tough to be sympathetic to leaders in a company that were aware of the errors and then took no action for years to remedy it, particularly as the error resulted in overcharging customers.
- Analemma_ 6y ago
  Funny how these mistakes never seem to undercount ad impression numbers though.
mtnGoat 6y ago
Facebook did something shady? I know most of the stuff coming out these days is in the past, but I really hope they are turning the corner and starting to act like they poses a spine and want anything but more dollars. Otherwise I see a dark future and a lot of litigation in their future.
- NicoJuicy 6y ago
  Well, there's something going on in Australia currently. I don't think they turned a page yet.
bryanrasmussen 6y ago
It is difficult to get Mark Zuckerberg to understand something is wrong if it is making him the right amount of money.
- bluefirebrand 6y ago
  I'm not gonna sit here and defend the Zuck, but your statement is true of many, maybe even most, people. Once enough money has entered the picture, people become very flexible.
  - bryanrasmussen 6y ago
    I did say it mainly because I found the image funny, but I agree, I doubt I would be significantly more moral, but I do find it funny.
- meepmorp 6y ago
  “It is difficult to get a man to understand something, when his salary depends on his not understanding it,” as Upton Sinclair put it.
  - bryanrasmussen 6y ago
    that's what I was riffing off of.
- uniqueid 6y ago
  I'm speculating here but Facebook strikes me as being different than most companies: given an equally viable choice between an ethical and an unethical solution to a problem, Facebook seems to prefer to choose the latter. I think it makes the execs feel ruthless and clever.
jtbayly 6y ago
We experienced these blatantly false promises from FB, so we stopped buying ads from them. Hoping to get a few tenths of a penny on the dollar back from them from this lawsuit. Hoping it hurts them a bit more than a few tenths of a penny on the dollar, though.
cambalache 6y ago
Class action lawsuit coming in 3,2..
cryptica 6y ago
Facebook is not an advertising company. They're a parasitic revenue laundering company. They feed off the cashflow of soon-to-be-bankrupt startups as part of giant global VC-fueled pyramid money laundering scheme. Founders get paid to take huge loans on their startup's books and run their startups straight into the ground while laundering all their capital to beef up Facebook's revenue for the benefit of big VCs and their crony capitalist friends who own Facebook shares. The shell company's employees and regular tax payers (who lose value via monetary inflation caused by these dubious bank loans) take the fall while all the key participants of the scheme get off scot free. Rince and repeat; real definition of 'serial entrepreneur'.
- ppf 6y ago
  One day, books will be written about the predatory and degrading tactics used by Silicon Valley companies of all types. It is ironic that in an area of the country that prides itself on its own "progressiveness", people are more than happy to grind employees of speculative startup companies into the ground in a effort to raze the competition with free products, while living off endless VC investments. Google, as it exists now, is the natural outcome of this process.
  - foobiekr 6y ago
    This process actually does deserve a well-researched book.
throw_away892 6y ago
I remember not too long ago Uber cancelling previous arrangements with Facebook advertising over a similar fraud.
- nindalf 6y ago
  You remember incorrectly. https://which-50.com/how-uber-learned-it-had-a-huge-ad-fraud-problem/ https://which-50.com/how-uber-learned-it-had-a-huge-ad-fraud...
  - throw_away892 6y ago
    Ah yes. Thanks for the memory. So they effectively cut their ad spend on Facebook by 2/3rds (an experiment) and saw virtually no difference between the two budgets. An obvious indication of reporting fraud.
raiyu 6y ago
Why large companies fail - they put revenue ahead of customers. How long a company can get away with that depends on their moats, competitive landscape, and what alternatives exist. In the case of FB, obviously there's a huge moat and a semi monopoly. There is some competition from Twitter and Snapchat, but really FB acquiring Instagram doubled their market cap, because that is the only other social platform that has truly reached their scale, and also has an effective, if not more effective, ad platform. This is also pretty typical even at smaller companies. Different people have different "masters". The product manager is rightfully trying to build the best product for their customers - the advertisers. But the upper management has bonuses tied to revenue usually through stock compensation, so their interest is in their own pockets. The product manager who correctly points out a real issue, is then seen as a negative, rather than someone doing a fantastic job.
zwaps 6y ago
Even three years ago, I have seen academic work (presentations) on Facebooks effectiveness of advertising - based on facebooks own data - and the conclusion was pretty dire. The only reason (yes I asked) Facebook continued to allow these studies was because Google did even worse. I wonder if all this stuff ever got published. Never heard anything in the press, so I am guessing Facebook eventually pulled the plug. I wonder if I saved the drafts tho :>
- octostone 6y ago
  Please share if you have!
  - zwaps 6y ago
    Here is one from originally 2016 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3033144 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3033144 Apparently - I did a quick search - conversion rate at facebook is said to be on average up to 9% (in marketing materials) or around 4-5%. In this study, the authors show conversion rate consistently sub 1%, which is however not the point of the article. Rather, they also show that the lift in conversion (so the actual impact of the ad) as calculated by observational measures usually over-estimates the actual lift (sometimes significantly). Observational measures were those used by advertisers on facebook - maybe even now? The more snarky comments from the presentation are missing from the draft, of course.
    - deleted 6y ago
      [deleted]
- staplung 6y ago
  Probably not exactly the presentation you mention but Freakonomics explored the issue of advertising effectiveness in a 2 part series. Part 2 is about online ads. The picture it paints is pretty bad. https://freakonomics.com/podcast/advertising-part-2/ https://freakonomics.com/podcast/advertising-part-2/ One thing I was hoping they'd explore was whether any of the recent companies that participated in the fb ad boycott realized afterward that the advertising dollars were not really working for them.
Avshalom 6y ago
Same shit that caused "pivot to video" and killed so many things. https://twitter.com/adamconover/status/1183209875859333120 https://twitter.com/adamconover/status/1183209875859333120
- LiquidSky 6y ago
  It's infuriating how Facebook faced basically no actual consequences for that. Their lies decimated entire industries. I had many writer friends who lost their jobs and were in extremely desperate situations because their employers were convinced to pivot to video based on those lies.
  - ehnto 6y ago
    I wasn't in the drivers seat, but I don't understand how CH could have continued with native video on FB if they had the numbers that actually mattered, ad revenue from their own site. Like even if their engagement was through the roof, it should have been clear their ad revenue was tanking, so why didn't they revert?
    - Tarsul 6y ago
      sunk cost fallacy maybe? Maybe they didn't really have the numbers either or didn't trust their own (real) numbers. Or their total revenue was up and they didn't quite get the causalities right... Anyway, it happened.
tryonenow 6y ago
>Facebook has come clean on a slight error in how it presented video view time on its platform. A mismatch in how average video view time is calculated and how it is defined — resulting in that number being reportedly inflated by a trivial 60 to 80 percent, for a short period of two years. I don't understand - a "slight" error and 60-80% is "trivial" for a "short" period of two years? Is it just me or is this language excessively apologetic? Or is it meant to be ironic?
- kevincox 6y ago
  Sounds very sarcastic to me.
kgog 6y ago
At this point, it's hard to make a case for why Facebook is NOT a criminal enterprise.
minikites 6y ago
I'm sure the free market will punish Facebook, so we don't need to worry about this.
john_moscow 6y ago
From my own personal experience with online advertisement, it pays off in one specific case. If a user is right now comparing multiple solutions to a specific problem, showing them an ad for your solution could pay off. E.g. if someone searches for plumbers near their home, an ad of your plumbing company will have a decent CTR and may bring in business. Bombarding someone looking for random entertainment after a hard day at work with ads of luxury cars would be quite futile. The advertisement companies know this very well, but they also know that the specific, well-aimed ads make a small fraction of their revenue. So they have a very direct interest in using the success metrics from the specific ads to sell their platform to a much wider range of customers, for whom it will never pay off. There's also considerable FUD[0] involved in the decision process. Sure, you don't see your ads paying off directly, but there must be that legendary brand awareness building up in the background. What if you decide to pull the plug on ads, and in a year your company's sales drop by 20%? Do you want to risk your entire career over a choice of saving your employer's money that isn't your own money anyway? Also a huge chunk of advertisers are clueless people that decided to try ads for the first time. They don't know what to expect in terms of payoff, they don't know to make specific problem-oriented ads (and whether their case is even a problem-oriented case). So they spend some ad dollars and move on. But due to the scale of the market, it still adds up to a healthy profit for the platform. [0] https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt
z3rgl1ng 6y ago
Absolutely no scruples to be found at FB. I’d never take a check from them.
snambi 6y ago
Facebook ads are scam. FB Ads don't reach anyone.

180 points by hashingroll 6y ago

I have seen many people fail to realize (and sometimes to understand at all) that taking measurements of large numbers of people is always fraught with errors of all sorts. No matter how hard you try, it is like the universe always pushes back to make sure the data are imperfect. Sometimes those imperfections are acceptably small, and other times the imperfections are uncomfortably large. In my opinion, anyone involved in such large-scale data collection and analysis should acknowledge the inevitability of error and provide disclaimers about potential sources of error.

fab1an 6y ago
Yes, but if the error consistently results in you getting paid more than you should have, it's not really an error, is it? No, we would in fact call that fraud.
- hari_seldon_ 6y ago
  +1, the issue here is not that errors can occur but instead how a company responds. It’s tough to be sympathetic to leaders in a company that were aware of the errors and then took no action for years to remedy it, particularly as the error resulted in overcharging customers.
Analemma_ 6y ago
Funny how these mistakes never seem to undercount ad impression numbers though.

mtnGoat 6y ago

Facebook did something shady? I know most of the stuff coming out these days is in the past, but I really hope they are turning the corner and starting to act like they poses a spine and want anything but more dollars. Otherwise I see a dark future and a lot of litigation in their future.

NicoJuicy 6y ago
Well, there's something going on in Australia currently. I don't think they turned a page yet.

bryanrasmussen 6y ago

It is difficult to get Mark Zuckerberg to understand something is wrong if it is making him the right amount of money.

bluefirebrand 6y ago
I'm not gonna sit here and defend the Zuck, but your statement is true of many, maybe even most, people. Once enough money has entered the picture, people become very flexible.
- bryanrasmussen 6y ago
  I did say it mainly because I found the image funny, but I agree, I doubt I would be significantly more moral, but I do find it funny.
meepmorp 6y ago
“It is difficult to get a man to understand something, when his salary depends on his not understanding it,” as Upton Sinclair put it.
- bryanrasmussen 6y ago
  that's what I was riffing off of.
uniqueid 6y ago
I'm speculating here but Facebook strikes me as being different than most companies: given an equally viable choice between an ethical and an unethical solution to a problem, Facebook seems to prefer to choose the latter. I think it makes the execs feel ruthless and clever.

jtbayly 6y ago

We experienced these blatantly false promises from FB, so we stopped buying ads from them. Hoping to get a few tenths of a penny on the dollar back from them from this lawsuit. Hoping it hurts them a bit more than a few tenths of a penny on the dollar, though.

cambalache 6y ago

Class action lawsuit coming in 3,2..

cryptica 6y ago

Facebook is not an advertising company. They're a parasitic revenue laundering company. They feed off the cashflow of soon-to-be-bankrupt startups as part of giant global VC-fueled pyramid money laundering scheme. Founders get paid to take huge loans on their startup's books and run their startups straight into the ground while laundering all their capital to beef up Facebook's revenue for the benefit of big VCs and their crony capitalist friends who own Facebook shares. The shell company's employees and regular tax payers (who lose value via monetary inflation caused by these dubious bank loans) take the fall while all the key participants of the scheme get off scot free. Rince and repeat; real definition of 'serial entrepreneur'.

ppf 6y ago
One day, books will be written about the predatory and degrading tactics used by Silicon Valley companies of all types. It is ironic that in an area of the country that prides itself on its own "progressiveness", people are more than happy to grind employees of speculative startup companies into the ground in a effort to raze the competition with free products, while living off endless VC investments. Google, as it exists now, is the natural outcome of this process.
- foobiekr 6y ago
  This process actually does deserve a well-researched book.

throw_away892 6y ago

I remember not too long ago Uber cancelling previous arrangements with Facebook advertising over a similar fraud.

nindalf 6y ago
You remember incorrectly. https://which-50.com/how-uber-learned-it-had-a-huge-ad-fraud-problem/ https://which-50.com/how-uber-learned-it-had-a-huge-ad-fraud...
- throw_away892 6y ago
  Ah yes. Thanks for the memory. So they effectively cut their ad spend on Facebook by 2/3rds (an experiment) and saw virtually no difference between the two budgets. An obvious indication of reporting fraud.

raiyu 6y ago

Why large companies fail - they put revenue ahead of customers. How long a company can get away with that depends on their moats, competitive landscape, and what alternatives exist. In the case of FB, obviously there's a huge moat and a semi monopoly. There is some competition from Twitter and Snapchat, but really FB acquiring Instagram doubled their market cap, because that is the only other social platform that has truly reached their scale, and also has an effective, if not more effective, ad platform. This is also pretty typical even at smaller companies. Different people have different "masters". The product manager is rightfully trying to build the best product for their customers - the advertisers. But the upper management has bonuses tied to revenue usually through stock compensation, so their interest is in their own pockets. The product manager who correctly points out a real issue, is then seen as a negative, rather than someone doing a fantastic job.

zwaps 6y ago

Even three years ago, I have seen academic work (presentations) on Facebooks effectiveness of advertising - based on facebooks own data - and the conclusion was pretty dire. The only reason (yes I asked) Facebook continued to allow these studies was because Google did even worse. I wonder if all this stuff ever got published. Never heard anything in the press, so I am guessing Facebook eventually pulled the plug. I wonder if I saved the drafts tho :>

octostone 6y ago
Please share if you have!
- zwaps 6y ago
  Here is one from originally 2016 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3033144 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3033144 Apparently - I did a quick search - conversion rate at facebook is said to be on average up to 9% (in marketing materials) or around 4-5%. In this study, the authors show conversion rate consistently sub 1%, which is however not the point of the article. Rather, they also show that the lift in conversion (so the actual impact of the ad) as calculated by observational measures usually over-estimates the actual lift (sometimes significantly). Observational measures were those used by advertisers on facebook - maybe even now? The more snarky comments from the presentation are missing from the draft, of course.
  - deleted 6y ago
    [deleted]
staplung 6y ago
Probably not exactly the presentation you mention but Freakonomics explored the issue of advertising effectiveness in a 2 part series. Part 2 is about online ads. The picture it paints is pretty bad. https://freakonomics.com/podcast/advertising-part-2/ https://freakonomics.com/podcast/advertising-part-2/ One thing I was hoping they'd explore was whether any of the recent companies that participated in the fb ad boycott realized afterward that the advertising dollars were not really working for them.

Avshalom 6y ago

Same shit that caused "pivot to video" and killed so many things. https://twitter.com/adamconover/status/1183209875859333120 https://twitter.com/adamconover/status/1183209875859333120

LiquidSky 6y ago
It's infuriating how Facebook faced basically no actual consequences for that. Their lies decimated entire industries. I had many writer friends who lost their jobs and were in extremely desperate situations because their employers were convinced to pivot to video based on those lies.
- ehnto 6y ago
  I wasn't in the drivers seat, but I don't understand how CH could have continued with native video on FB if they had the numbers that actually mattered, ad revenue from their own site. Like even if their engagement was through the roof, it should have been clear their ad revenue was tanking, so why didn't they revert?
  - Tarsul 6y ago
    sunk cost fallacy maybe? Maybe they didn't really have the numbers either or didn't trust their own (real) numbers. Or their total revenue was up and they didn't quite get the causalities right... Anyway, it happened.

tryonenow 6y ago

>Facebook has come clean on a slight error in how it presented video view time on its platform. A mismatch in how average video view time is calculated and how it is defined — resulting in that number being reportedly inflated by a trivial 60 to 80 percent, for a short period of two years. I don't understand - a "slight" error and 60-80% is "trivial" for a "short" period of two years? Is it just me or is this language excessively apologetic? Or is it meant to be ironic?

kevincox 6y ago
Sounds very sarcastic to me.

kgog 6y ago

At this point, it's hard to make a case for why Facebook is NOT a criminal enterprise.

minikites 6y ago

I'm sure the free market will punish Facebook, so we don't need to worry about this.

john_moscow 6y ago

From my own personal experience with online advertisement, it pays off in one specific case. If a user is right now comparing multiple solutions to a specific problem, showing them an ad for your solution could pay off. E.g. if someone searches for plumbers near their home, an ad of your plumbing company will have a decent CTR and may bring in business. Bombarding someone looking for random entertainment after a hard day at work with ads of luxury cars would be quite futile. The advertisement companies know this very well, but they also know that the specific, well-aimed ads make a small fraction of their revenue. So they have a very direct interest in using the success metrics from the specific ads to sell their platform to a much wider range of customers, for whom it will never pay off. There's also considerable FUD[0] involved in the decision process. Sure, you don't see your ads paying off directly, but there must be that legendary brand awareness building up in the background. What if you decide to pull the plug on ads, and in a year your company's sales drop by 20%? Do you want to risk your entire career over a choice of saving your employer's money that isn't your own money anyway? Also a huge chunk of advertisers are clueless people that decided to try ads for the first time. They don't know what to expect in terms of payoff, they don't know to make specific problem-oriented ads (and whether their case is even a problem-oriented case). So they spend some ad dollars and move on. But due to the scale of the market, it still adds up to a healthy profit for the platform. [0] https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt

z3rgl1ng 6y ago

Absolutely no scruples to be found at FB. I’d never take a check from them.

snambi 6y ago

Facebook ads are scam. FB Ads don't reach anyone.