8 ms·

This shows the limitations of ML: facebook has incredible amounts of preference & behaviour data on its users; and can't even meet incredibly generic categories

• sorokod 6y ago
  Well, a combination of objective limitations and FB's ability to execute.
  • sidlls 6y ago
    But mostly objective limitations. The most advanced minds executing the most robust, cutting edge ML aren't likely to do very much better. And most of the metrics by which they might suggest otherwise end up being flawed themselves due to the bias introduced by the feedback loops involved. When your predicate influences the next version's antecedent, it's easy to show a big ROI, lift, or whatever other jargon you might care to use.
• yarcob 6y ago
  I think the problem is deeper than that. Every company wants to advertise to high earning college educated people, and nobody wants to advertise to unemployed anti-vaxers. But presumably Facebook has a lot more of the latter. So they have to get a bit "creative" with their targeting if they want to sell their inventory...
• emcq 6y ago
  This article is quoting employees conversations from 2016. It wasn't until late 2016 that Google published research on wide and deep recommender systems. I'd be interested to see if people still think behavioral data is poor and ML has not provided advances. That said some demographics are broadly correct from behavioral tracking (e.g. a child who watches baby shark on repeat), but good luck telling the difference between a rich child and a less rich child.
• ma2rten 6y ago
  I can't speak for other people, but the reason that I think we are in an AI boom has nothing to do with online advertising. It is that computers can now recognize images, translate language, have conversations, generate articles, make realistic looking pictures, play the game of go, solve protein folding, generate realistic text to speech, recognize voices. All these things were not possible 5 years ago. Every year there is significant progress. I believe that in a couple of years we will see many more applications. This will be as much of a paradigm shift as the invention of the personal computer, the internet and smart phones.
  • nopriorarrests 6y ago
    >recognize voices. All these things were not possible 5 years ago. FTR: https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking Dragon Systems released NaturallySpeaking 1.0 as their first continuous dictation product in 1997. <...> As of 2012 LG Smart TVs include voice recognition feature powered by the same speech engine as Dragon NaturallySpeaking.
    • fossuser 6y ago
      Yeah I played with dragon in 97 and it was awful - it didn’t work at all, completely unusable. Today voice transcription is a solved problem and while their engine might be the same in name - I’d be surprised if the approach isn’t totally different than what they were doing in 97, either that or the LG tv voice transcription probably doesn’t work as well as everyone else’s. The deep learning revolution and the applications we’ve seen since 2015 are a major step forward and something truly different. People pretending otherwise are just acting cynical in some attempt to project intelligence or seem wise, it doesn’t work.
      • nopriorarrests 6y ago
        Of course it was awful. It was 1997. But you can't claim that something "wasn't possible 5 years" ago, if 7 years ago said feature was included in inexpensive consumer product (LG TV). I'm not acting cynical, but it's tiresome for me to see people who claim that 20-30 years ago we all were living in a caves and catching bugs with wooden sticks, and now boom, ML! Regarding "something truly different", well, my personal computing / mobile experience not changed that much from 2015. Honestly speaking, progress from 1995 to 2000 felt much more impressive and 'truly different'. I mean, think of it, during this timeframe we went from DOOM via V.34 modems to amazon.com and ordering pizza online.
• stickfigure 6y ago
  I guess there's two ways of looking at this: * 40% of targeted viewers didn't match "high-earner, college educated", the system is crap! or * 60% of targeted viewers did match "high-earner, college educated", that's amazing! I dunno... do you have a better way of reaching those targeted viewers? Do you have a better way of measuring results than abstract page views? Depending on the context, 60% could be very worth it.
  • deleted 6y ago
    [deleted]
  • aeternum 6y ago
    Yes, humans are also notoriously bad at Bayesian probability. This may actually be extremely impressive if high earners make up something like less than 1% of general page views. In that case, non-targeted advertising gives you a <1% chance of reaching your audience per impression. FB is much better.
  • indigochill 6y ago
    >... do you have a better way of reaching those targeted viewers? I'd be curious to see how blogs do at targeting certain demographics. Presumably if you had a blog aimed at high-earning college educated people you could get a pretty high ratio of (target demographic/all viewers). But I assume advertisers generally trust metrics more than intuition so you need something like Facebook or Google's ad network to identify someone as belonging to a certain demographic before the advertiser will believe it.
    • nopriorarrests 6y ago
      >Presumably if you had a blog aimed at high-earning college educated people you could get a pretty high ratio of (target demographic/all viewers). Correct. The problem of course is scale. How much money such blog can accept from advertisers before it will get absurdly expensive to buy ads from such blog? FB gives you targeting and scale, and you need both to run meaningful advertising campaign.
  • jariel 6y ago
• ashtonkem 6y ago
  According to FB metrics, FB AI magic works great. Then again, according to FB metrics video engaged better than text ... and it turns out they were lying. I wouldn’t put it past them to goose the numbers a bit for their core product.
  • hertzrat 6y ago
    The “videos engage better than text” idea has been dominant for a few years now. It’s based on a Facebook statement?
    • ashtonkem 6y ago
      Facebook pushed that argument hard, and there is an ongoing lawsuit over whether they misled their customers.
      • hertzrat 6y ago
        What’s the evidence they misled customers? Internal documents?
        • nopriorarrests 6y ago
          They confessed. https://www.nytimes.com/2016/09/24/business/media/facebook-apologizes-for-overstating-video-metrics.html https://www.nytimes.com/2016/09/24/business/media/facebook-a...
  • antupis 6y ago
    This or put more generally ads don't work as FB says they do. It is much easier to blame faulty ml than say that people dont click ads.
• drzoltar 6y ago
  I’m a bit confused why we are assuming the low performance is the fault ML. I would think instead that this low performance is deliberate because it locally optimizes Facebooks KPIs. My understanding is that Facebook has a lot of data on the elasticity of ad buyers. If Facebook were to have even more precise algorithms, then they would likely charge even more money for impressions and clicks. Presumably buyers may be reticent to pay even higher prices. Alternatively more precise targeting would highly bias the money people spend toward larger companies that can afford better keywords.
  • mjburgess 6y ago
    If you look at the communication, it seems that the Ad people didn't have a clear idea how well-targeted anything was. It's easy to see how the statistical problems compound. FB does not have exact earning levels, so you have to infer that from, say, likes. Let's say you can build a model salary(likeBMW, likeTravel, ...) = %likeBMW + %likeTravel +... This gives you an estimate which is (70-80)% accurate, so you predict >£250k/yr 70pc of the time. In c. 25% you mispredict. Now it seems to me that this 25% is going to compound across several categories: when you say "College AND HighEarner AND ..." you get more than 50% of your target group not matching this exact criteria (all you need to fail is one condition to fail to match this conjunction). And according to FB comms, it looks like >50% didn't match client's chosen criteria. I think this is the right analysis of the issue. ML systems of this kind are very bad at making targeted judgements. It's really little more accurate than guessing the mean of something (eg., salary) for your group. All ML has to do, for FB/Google/etc. is improve targeting a few percent to have a significant value proposition. However, the propaganda has it that ML systems can "target" you, etc. Only in the way a nuclear bomb "targets" a house.
    • drzoltar 6y ago
      Could it be a misunderstanding of the outcome, or perhaps bias in Facebooks algorithms? The best predictor of your target is: your target. In this case it’s predicting who will buy your product. Using a proxy such as income and education is a good start but surely cannot be the best approach because it artificially limits the search space. The best target is some ML-determined combination of features. My guess is that for niche items such as expensive watches and cars, Facebook’s targeting may be too liberal with lower income earners. But I think this the result of being very good at targeting the general audience for medium-low priced goods and services.
• visarga 6y ago
  It's easy to target but what FB wants to do is to sell junk traffic as if it were highly lucrative. It prefers to sell more rather than do proper targeting. The traffic swap was an old black-hat trick used by ad network affiliates even before 2000 (they used to push pop-under traffic as SEO traffic, for example). Just cheating. It's not right to judge ML by the quality of recommender systems. First of all, recommenders are just a small corner of ML, and on this scale are only developed at few companies. Second, I am not 100% sure they maximize what you want to see, instead they maximize what will make them more money. That's why it sucks.
• twelve40 6y ago
  Instead of limitations of ML, could be simply a matter of not having enough inventory with the desired parameters to sell to everyone who wants it. Which doesn't make it any less sleazy if true, of course.
• deleted 6y ago
  [deleted]
• marcinzm 6y ago
  AI doesn't need to be perfect to beat other approaches hands down and make hundreds of billions for companies using it. Having worked in the field, I'll take the 40+% revenue increases I got with AI in A/B tests any day of the week.
• selimthegrim 6y ago
  Someone better name their next paper after Hans Holbein
• cbsmith 6y ago
  There's a confusion here about what is really going on here. In truth, advertisers care more about volume that quality, and so the industry has a bias in that direction. It's entirely possible to have much higher precision, but that precision turns out to be not what advertisers are looking for. For example, take "college educated high-earner". Well, there are a bunch of such types who don't spend or making purchasing decisions like the rest of that group, and a bunch of people outside that group who behave exactly like them. How did the advertiser figure out that they want that group? Did they really figure out that if you aren't meeting those criteria their ad is a waste of time? Of course not. They're just guessing, and they don't intend for their guess to constrain their audience.
• 1vuio0pswjnm7 6y ago
  Is the hype working? (Sales.)
• Invictus0 6y ago
  I disagree that this shows anything about ML at all. The particular example here--high-earner, college educated-- should be trivial to deduce without machine learning. Just to start, many users add their college directly to their profile. How is it possible that they can't even get that right?
• Rastonbury 6y ago
  I think what's scary is how much a blackbox it is. Some people rave at how the FB algo has improved to the point where they set the broadest ad audience possible. 2 years ago segmentation and testing was the name of the game. Now you apparently can just let it loose