18 ms·

ML is not that good at predicting consumers' choices

251 points by macleginn 4y ago

ape4 4y ago
You just bought a washing machine... could I interest you in a washing machine?
- im3w1l 4y ago
  GPT can solve this! I prompted it with "Sarah bought a washing machine and a ". It completed "dryer.". Another "If you buy a hammer you might also want to buy " -> "a nail". Ill forgive the singular. Just to be clear those are not cherry picked - they were my first two attempts.
  - ape4 4y ago
    Putting those together... I actually bought a pair of anti hammer arrestors for the washing machine ;)
  - thaumasiotes 4y ago
    > GPT can solve this! I prompted it with "Sarah bought a washing machine and a ". It completed "dryer.". The most natural interpretation there is that Sarah bought a washing machine and a dryer simultaneously, not that, after buying a washing machine the month prior, she was finally ready to buy a dryer.
  - Mordisquitos 4y ago
    Fair enough regarding the washer/dryer, but I'm unimpressed by the hammer/nail example. If anything, I might want to buy "some nails" rather than "a nail", but I would argue that's not a brilliant guess either—if I'm buying a hammer I most likely already own the item that I want to hit with it. Some more interesting suggestions could have been for example "some pliers", "a set of screwdrivers", "a pair of work gloves", "safety goggles", etc.
- mdp2021 4y ago
  While the chief absurdity is very clear (also mocked by Spitting Image - J.B. on a date: "You loved that steak? Good, I'll order another one!"), I am afraid that the intended idea may be that your memory about the ads of what you just bought will last as much as said goods. Utter nightmare (unnatural obsolescence, systemic perversity, pollution...) but. I have met R'n'D who admitted the goal was just to have something new to have people want to replace the old, on unsubstantial grounds.
- armchairhacker 4y ago
  I think the reason this happens is that when you start looking for washing machines, you start getting ads for them. Then when you buy nobody tells the ad companies that you just bought a washing machine so they still send you ads because they think you’re still looking. Even if you just went straight to the model site and clicked “buy”.
  - thaumasiotes 4y ago
    We know that's not the reason; Amazon is infamous for advertising washing machines to people who have just bought a washing machine from Amazon.
    - jsjohnst 4y ago
      Amazon can’t suggest a “product you might like” if their life depended on it. They have 19 years of purchase history for me spanning thousands of product categories. Want to know a category I’ve never looked at or bought from Amazon, women's handbags. What’s near the top of the suggestions right now on the homepage for me? You guessed it!
      - thaumasiotes 4y ago
        > Amazon can’t suggest a “product you might like” if their life depended on it. Actually, there is one method they use to make suggestions that works very well. If you have a wishlist, sometimes they'll recommend items from there.
      - oriolid 4y ago
        The weirdest thing is that at one point, I think it's now almost 10 years, they had somewhat OK recommendations. It's not like "you bought programming books and Battlestar Galactica DVDs, so you might be interested in Neal Stephenson's novels" is really advanced but it's still miles ahead of the "you just bought a washing machine, so you might be interested in washing machines" level.
        4y ago
- bolasanibk 4y ago
  I cannot remember the reference now, but the reasoning I read was a person who just bought an item x might: 1. return the item if they are not satisfied with it and get a replacement Or 2. buy another one as a gift if they really like it. Both of these result in a higher fraction of conversions in this kind of targeting vs other targeting criteria.
  - astrea 4y ago
    I’m also more inclined to believe that the ad networks simply don’t have a way of knowing a sale was made.
- deleted 4y ago
  [deleted]
- wrycoder 4y ago
  I buy a package of underwear. All I see for next three weeks on my browser is close ups of men’s briefs. It’s embarrassing, when associates glance at my screen.
  - speedgoose 4y ago
    I agree it can be embarrassing to not use an ad blocker.
  - jstx1 4y ago
    Kind of your fault - you're buying your underwear on a work device and you don't use an adblocker.
    - wrycoder 4y ago
      1. Not a work device. 2. We are discussing how ads are selected for presentation.
- jsjohnst 4y ago
  While this annoys the hell out of me too, there is a semi-logical explanation that makes sense in many cases: The ad targeting knows you looked at washing machines, but doesn’t know you purchased.
  - kristopolous 4y ago
    Which is an extremely trivial check to add - if you got assigned that ticket, you'd probably point it at like 2 or so hours. However, they've been like this for over a decade so it's likely there intentionally. here's one way that could be possible: There could be some popular third party service that's integrated on many e-commerce sites that sells this information and doesn't actually give a damn if you bought the refrigerator or not. They're selling you, not the refrigerator. That's the problem with data brokers, it's mostly low quality data.
    - jsjohnst 4y ago
      > Which is an extremely trivial check to add - if you got assigned that ticket, you'd probably point it at like 2 or so hours. Yep, totally a 2hr task for an engineer who works on homedepot.com to “check” that you bought a fridge from lowes.com after you first price shopped the other site. Also a two hour task for a Google engineer to know you bought one in person at Best Buy after researching online first. Yes, there are basic cases (buying from same merchant as who’s suggesting) that should be handled, but let’s not foolishly pretend that’s the average case, let alone majority / all.
      - cerved 4y ago
        not to mention cross-device browsing
      - dahfizz 4y ago
        Amazon consistently shows me ads for products I bought on Amazon. Don't bother talking about edge cases when the simple case doesn't work at all.
- Asooka 4y ago
  A slightly more ridiculous one I experienced recently was when I searched where to buy tool X. Started getting ads for why I need tool X and why it's the best tool ever. I already want one, I'm looking where to buy, not trying to learn what it's for!
- saurik 4y ago
  College Humor--may it rest in peace :( :( :(--did a great bit about this a few years ago. https://www.youtube.com/watch?v=KbKdKcGJ4tM https://www.youtube.com/watch?v=KbKdKcGJ4tM It is absolutely ridiculous how useless targeted ads actually are... and seeing a real person try to sell you this stuff really underscores the insanity.
  - maxbond 4y ago
    College Humor is alive and well! At least if you enjoy Dungeons and Dragons and silly game shows.
- BonoboIO 4y ago
  Sometimes you want a company like Facebook or google deliver ads to your potential costumer, but you don’t want, that the ad company knows how much product you sold. You know your ROI, but your ad company doesn‘t. I heard multiple stories from amazon, that hey still let the ad campaign or targeting running even if you bought the product for random times so external companies could not get insight in your businesses. If you got the washing machine you will not click to buy another. If you pay per click, it makes no difference how long you let the campaign running if it is targeted. I work in the ad industry.
  - BonoboIO 4y ago
- Gibbon1 4y ago
  I merely looked at pressure cookers and got ads for full figured bra's and support garments for six months.
jrm4 4y ago
I'm not surprised at this result, mostly because of the inaccurate noise that the business of "marketing," (i.e. specifically marketing people selling their not-very-effective services) generates.
cj 4y ago
While not exactly aligned to the research, I've been surprised how poor Nest Thermostat's learning feature is. The main selling point for Nest is having a "learning thermostat". Perhaps my schedule is just not predictable enough, but the auto-generated temperature schedules it generates after its "learning" period is not even close to what I would manually set up on a normal thermostat. Maybe I'm just an "edge case" or part of the "long tail"
- actusual 4y ago
  Nah, you're not. I just gave up on mine and have a schedule. I also turned off "pre-cooling" because it would just kick on at like 6pm to "cool" the house for bedtime. I also bought several temperature sensors to use, which are fun. At night I have the thermostat use the sensor in my bedroom, then goes back to the main thermostat during the day.
  - foobarian 4y ago
    See the next logical step is to outfit the output vents with servo-controlled actuators so you can fine-tune where the air is going!
    - actusual 4y ago
      Additionally: - Use a python script to switch heating from my heat pump to my gas furnace when energy prices invert. - Have live monitoring of my HVAC pressure health so that my servo controlled actuators don't bork my system.
- PaulHoule 4y ago
  When people hear that FAANG is involved in something an "Emperor's Clothes" effect kicks in and people stop making the usual assumption that "if it doesn't work for me it probably doesn't work for other people"
- bell-cot 4y ago
  Or, maybe they invested far more cash and care in marketing that feature than in programming that feature...
hourago 4y ago
> Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise The worst part of big data is the data itself. Used to be common will be shared on Facebook webs about "what is your political compass". There results were used to create political profiles of users and targeted propaganda. You don't need ML to predict the data that there user already has given.
humanistbot 4y ago
"It's tough to make predictions, especially about the future." ― Yogi Berra
- alberto_ol 4y ago
  probably not by yogi berra https://quoteinvestigator.com/2013/10/20/no-predict/ https://quoteinvestigator.com/2013/10/20/no-predict/
annoyingnoob 4y ago
Maybe humans have free will after all.
- ugjka 4y ago
  random will perhaps
  - Spivak 4y ago
    It's funny you say random because if consumer choice was actually random with some known distribution it would be extremely predictable, no ML needed.
    - nequo 4y ago
      Known distribution doesn't mean extremely predictable. For example, if your water consumption is log-Cauchy, I will have a very hard time predicting it because the variance is infinite.
oxfordmale 4y ago
It is the same on Netflix. I have phases where I watch a certain genre for a few weeks and then move on. For example after a few Scandi crime series it is time for something else. However, at the same time my daughter loves Animé and pretty only watch that. It is really hard for an ML algorithm to grab these nuances.
- bertil 4y ago
  Netflix makes a far more obvious sin: not having “who is watching” as boolean choices. If I am watching with my partner, I want both of our accounts to mark that series as viewed. And I really want Netflix to tell me what I’m watching with her so that I don’t continue watching it without her because I will be single if that happens (again).
  - oxfordmale 4y ago
    It would be a great revenue stream for Netflix. Are you sure you want to watch this without your partner ? Yes ? We recommend the following service for finding temporary accommodation on short notice
- golemiprague 4y ago
rvz 4y ago
So this machine learning and deep learning hype has shown that it is a gimmick isn't it? After years of surveilling, collecting and training on user data it still doesn't work or gets attacked very easily over spoilt pixels and many other attacks? What a complete waste of time, money and CO2 being burned up in the data centers.
- jacquesm 4y ago
  No, it just isn't a silver bullet for every problem under the sun. But quite a few record holders on various problems are ML solutions and that is unlikely to change for the foreseeable future. It's just that as soon as you start out on every problem with 'ML will solve this!' that you're going to end up with a bunch of crap. The right tool for the problem wins every time.
- Enginerrrd 4y ago
  I don't know.... I think back on google search back in the ~2014 era. It was good. Like scary good. Like I'd type "B" and it would suggest "Btu to Joules conversion" and things like that. Actually it was better than that... it would anticipate things I hadn't even searched for before with very very little prompting. It seemed to adapt to context whether I was at work, on my phone, at home, etc. The results were exactly what I was looking for. Then it got taken over by ads and SEO and corrupting influences and it's just not that good anymore. IMO, the problem with DL isn't the tech. It's the way its being used. The reality is: For 99% of things advertised to me, I don't want to buy the goddamn product, and no amount of advertising will make me want to buy it. It's gotten to the point where if I see an ad for a product I think I'm more likely to buy a competitor whose ad I haven't seen because I assume the competitor is investing more in the product than the marketing. And everyone seems to have forgotten about hybrid approaches of ML and human beings that, IMO, are really good. But alas, "they don't scale". But at the same time, it's really interesting. For as much data as facebook should have about me, their ad rec's really suck and always have. (Perhaps it's because my only ad clicks ever are accidental ones?) I'm kind of astounded at how poor that result is. That said, I'm always very impressed by spotify's recommender system. I think it's one of the best on the net. Another thing I find interesting is that non-vote-based social media feed systems all really suck. Once they ditched chronological ordering it stopped appealing to me, and I don't know exactly why that is. Evidently I'm on some tail of the curve they don't care about.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
Plough_Jogger 4y ago
This review omits techniques from reinforcement learning (especially bandits) that have been used successfully in industry for years now.
- bertil 4y ago
  I think that the main issue is less the technique (although… yes, please use RL if you can) and more the lack of data. Browsing gives very little insight: dwell-time is a poor proxy for interest, and mixes horrid ideas that are so bad they are worth sharing with friends and confusing photos where you need to squint to figure out if it’s what you are looking for. Both e-commerce and social media are really not good at gathering express feedback for what people want and valuing that expressly. Please, let me tell you that I did spend time looking at this thread about the latest reality TV scandal but I don‘t want to hear about it ever again! Please, let me tag options as “maybe” or let me tell you what you’d need to change for me to buy that shirt. Public, performative Likes and Favourite lists that are instantly reactivation spam-fodder… Come on, you know better. I used to work for a big e-commerce site (the leading site for 18-25 y.o. females). We had millions of references (really) and it was a problem. The search team had layers upon layers of ranking algos, incredible papers at conference… but still, low impact on conversion. It was more than anything else that we could do, but nowhere as transformative as it could be. Instead, I suggested copying the Tinder interaction in a companion app: * left, never see that item again; * right, add it to a long list of stuff you might want to revisit. We probably would have to separate that from the Favourite list to avoid clutter, but maybe not, to make that selection worthwhile. The learning you could get from that dataset, even with a basic RL algo to queue suggestions… People thought it was “too much” which I’m still bitter about.
- jeffreyrogers 4y ago
  How are bandits used in consumer choice problems? Bandits solve almost the inverse problem: which choice to offer/take when it's uncertain which is best, but the problem under consideration in the blog post is about predicting which choice a consumer will pick, a standard marketing problem.
teruakohatu 4y ago
> Currently, we are still far from a point where machines are able to abstract high-level concepts from data or engage in reasoning and reflection Of course when an AI does that, we then say its just doing statistics, not reasoning. Until you have built a recommendation engine from scratch, it is hard to appreciate the complexity. I don't mean the complexity of the code or algorithm (ALS and Spark are straightforward enough) but the contextual problem. Models end up being large collections of models in a complex hierarchy, with hyperparams to tune higher level concepts such as "surprise" or business targets such as "revenue", "engagement" etc. TikTok have nailed this, as has Spotify.
- Barrin92 4y ago
  >Of course when an AI does that, we then say its just doing statistics, not reasoning. no, AI simply doesn't do that. Even Demis Hassabis of Deepmind fame in a recent interview pointed this out. Machine learning is great on averaging out a large amount of data, which is often useful, but it doesn't generate true novelty in any human sense. AI can play Go, it can't invent Go. In the same way today's recommender systems are great at averaging out my last 50 shopping items or spotify playlist but they can't take a real guess at what truly new thing I'd like based on a genuine understanding of say, my personality. Which is reflected in the quality of recommendations which is mostly "the thing you just bought/watched", which is ironically often incredibly uninteresting.
  - teruakohatu 4y ago
    > but it doesn't generate true novelty in any human sense. Few humans do either, even great artists create from, or react to, the art they experienced in their life. At the end of the day this is a rather metaphysical question. A Go model is trained to play Go, not create new games. > on a genuine understanding of say, my personality That is exactly what they are trying to do. The purpose of many recommendation systems is to uncover the latent variables and categories that you might like. They are not just averaging what you listened to. It is feasible (but very unwise) to predict certain users like to listen to happy upbeat tracks on Fridays, and sad songs sung in Romance languages on Mondays. Each time you skip a track or relisten to a track (implicit feedback) or like a track (explicit feedback) you are giving it information, which might be more than just the interaction (the timestamp, the IP address location, how quick you were to skip the track etc.). On the other hand, the purpose of those systems is simply to optimize what generates the most short term income, in which case, you get 'boring'/simple shopping recommendations that on average make more profit. Systems have tried to predict model the human experience, but in the past there have been epic fails so nobody is trying to explicitly do that anymore. The classic example would be stores deciding teens were pregnant and inadvertently alerting their parents. Nobody wants to see ads for Amazon's in-house toilet paper brand just as they are about to get up to go to the toilet.
tomcam 4y ago
I can personally vouch that Amazon, Twitter, and YouTube all do horrible horrible jobs predicting my taste. And they have got worse over the years, not better
- hourago 4y ago
  That may make sense of you are not the average consumer. Optimizing for the most common case makes sense. I see that with Google search prediction, it's good but many times it predicts very sensible words for general use but not in the topic that I'm interested.
- jltsiren 4y ago
  My favorite experience with Amazon: I had just preordered novel 9 of The Expanse, and I got an email recommending something else from the same authors: novel 8 of the Expanse. A more sensible recommendation engine might have assumed that someone who preorders part n+1 of a series may already have part n. Not to mention that Amazon should have known that I already had novel 8 on my Kindle. I guess generating personalized recommendations at scale is still too expensive. We just get recommendations based on what other customers with vaguely similar tastes were interested in.
  - kmnc 4y ago
    Funny thing is, your very comment is an indirect praise of the very thing they were advertising to you, and here it is being read by thousands of people. Are we so sure absurdly terrible ads don't actually beat out actually good well tailored ones? Looking through the history of radio ads, television ads, it seems like the best ads are always the stupidest. "Head on, apply directly to the forehead!" isn't so far off from "You bought a washing machine? Buy another!". The reality is, advertising optimizes to target stupid people because stupid people spend money. It is easier to trick a moron then sell a smart man something they actually want.
  - alpaca128 4y ago
    > Not to mention that Amazon should have known that I already had novel 8 on my Kindle. Amazon doesn't seem to understand many things surrounding the Kindle. For example, it calculates the progress reading through a book by the last page I looked at. That means if I finished a book and jumped to the introduction it'll now be convinced I only read 1% of the book. This is so dumb, and I don't know why they even do it that way - the Kindle hardware should easily be capable of precisely keeping track of what pages I looked at.
aaronax 4y ago
ML is there to maximize business income--nothing else. If ML was benefiting me, it would know that 90% of the time I fire up Hulu I plan to watch the next episode of what I was watching last time. And it would make that a one click action. Instead I have to scroll past promotional garbage...every single time. Assholes.
- buscoquadnary 4y ago
  Honestly a lot of this ML to me seems eerily similar to how in older times people would use sheep entrails or crow droppings to try and predict the future. I mean basically that is what ML is, trying to predict the future, the difference is they called it magic, we call it math, but both seem to have about the same outcome, or understandability.
  - treesprite82 4y ago
    > I mean basically that is what ML is, trying to predict the future If being so reductive, that's also the scientific method. Form a model on some existing data, with the goal of it being predictive on new unseen data. Key is in favoring the more predictive models. > they called it magic, we call it math, but both seem to have about the same outcome Find me some sheep entrails that can do this: https://imagen.research.google/ https://imagen.research.google/
    - ninkendo 4y ago
      One of the oft overlooked, yet critically important aspects of the scientific method is the hypothesis. You don’t design an experiment having absolutely no idea what to expect. You have an educated guess in mind (the hypothesis), and you design the experiment in such a way that says “this result will rule out my hypothesis, while this other result might confirm it.” Just trying two things at random and picking the one that makes some arbitrary metric go up, is not the scientific method. It’s gradient descent.
      - treesprite82 4y ago
mgraczyk 4y ago
Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. It reminds me a lot of other populist folk-science belief, like vaccine hesitancy. Despite overwhelming data to the contrary, a huge portion of the US population believes that they are somehow better off contracting COVID-19 naturally versus getting the vaccine. I think when effect sizes per individual are small and only build up across large populations, people tend to believe whatever aligns best with their identity.
- mushufasa 4y ago
  I think you may be conflating the topics and goals of adjacent exercises; predicting consumer behavior is not the same thing as optimizing a conversion pipeline.
  - mgraczyk 4y ago
    The examples they give in section two are directly relevant to optimizing conversion pipelines. They pretty clearly intend to be describing something relevant the e-commerce user experience.
- nojito 4y ago
  >Always interesting to see outsiders writing papers about this I don't think you know who andrew gelman is. Additionally, that's not the conclusion derived from this study.
  - mgraczyk 4y ago
    The actual conclusion of the study is so absurd that it's not worth engaging with seriously. That is, to maximally understand, and therefore predict, consumer preferences is likely to require information outside of data on choices and behavior, but also on what it is like to be human. I was responding to the interpretation from the blog post, which is more reasonable.
duxup 4y ago
Is there much that is good about predicting this stuff? I find Amazon loves to tell me to buy ... the thing they know I just bought and you don't need more than one of ... I hardly ever get ads or offers for things I want. How do you mess that up?
- alephxyz 4y ago
  Google seems like they target by age, gender and income rather than by interests. Sometimes it's convinced I'm a yuppie and keeps showing me luxury cars, personal care/beauty products and high end electronics (when I have zero interest in any of those products). Ironically I find the "dumb" ads on cable tv news to be a lot more effective since they have to target by interests.
shaburn 4y ago
gwbas1c 4y ago
> for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard Translation: Computers can't read minds. A bigger generalization is that, whenever a software feature becomes essentially mind reading; someone's either feeding a hype engine or letting their imagination run away. The best things to do in that case is to pop the bubble if you can, or walk away. I will often clearly state, "Computers can't read minds. You're making a lot of assumptions that will most likely prove false."
malkia 4y ago
Some years ago, I worked on a team "Ads Human Eval" - we had raters hired to do A/B testing for ads. These evaluated questionaires carefuly crafted by our linguists, and then analyzed by the statisticians providing feedback to the (internal) group that wanted to know more about. So the best experience was this internal event that we had, where the raters would say that certain Ad would not fare well (long term), while the initial metrics (automated) were showing the opposite (short temr). So then we'll gather into this event, and people would "debug" these and try to find where the differences are coming through. Then we had to help another group, where ML failed miserably detecting ads that should've not been shown on specific media, and raters came to help giving the correct answers. The one thing that I've learned is that humans are not going to be replaced any time soon by AI, and I've been telling my folks, friends or anyone (new-born luddities) - that automation is not going to fully replace us. We'll still be needed as teachers, evaluators, fixers, tweakers/hackers - e.g. someone saying - this is right, and this is not, this needs adjustment, etc. (to the machine, ai, etc.). Maybe machines are going to take over us one day, but until then, I'm not worried... (I've also understood I knew nothing about staticics, and how valuable linguists are when comes to forming clear, concise and non-confusing (no double meaning) questions)
- Melatonic 4y ago
  I dont think most people are arguing that machines will replace everyone anytime soon - it is that they will replace a huge portion of people. If one person can do the job of 10,000 by being the tweaker / approver of an advanced AI that is still 9,999 jobs eliminated. That might be hyperbole (you still probably will need people to support that system)
  - malkia 4y ago
    I agree, but it's true that some jobs should not simply exists. To this day, if I go to our airport in Sofia (Bulgaria), and my baggage is over the limit of 20 or was it 25kg I have to go to another place, pay for it and come back (why? bureaucracy - not only I have to do it, but I'm slowing anyone waiting for me to this - it's like 25-50 meters one place to the other) Unlike Frakfurt, Munich or Heathrow airport where I can pay that fine right there. Some jobs simply should not exist.
macNchz 4y ago
I've shared this before on HN, but it never fails to make me laugh when I think about it: >Several years ago a conversation about a similar topic prompted me to look at the ad targeting data Facebook had on me. At the time I'd had a Facebook account for 12 years with lots of posts, group memberships and ~500 friends. Their cutting edge data collection and complex ad targeting algorithms had identified my "Hobbies and activities" as: "Mosquito", "Hobby", "Leaf" and "Species": https://imgur.com/nWCWn63 https://imgur.com/nWCWn63. Whatever that means.
sarahlwalks 4y ago
As far as I'm concerned, the question is how ML/AI stacks up against the competition -- humans. I don't know, but I'd bet the answer is that ML is much better. Let's say at least 20 percent better, but I imagine it's much higher than that. Second, this is only saying that right now, ML's performance is "not that good." It says nothing about future technical advances. If you look at the track record of ML in the past three decades, it's amazing, and if that performance is repeated in the next three decades, who even knows what things might look like. (Machine sentience? Maybe.)
IAmWorried 4y ago
It seems to me like the "generation" use case of ML is much more promising than the "prediction" or "classification" use case. It's tough to predict things in general because our universe is fundamentally uncertain. How is some computer going to predict that some mugger sees a target at some random spot and decides to mug them? But the progress in text to image and text generation really blows my mind.
wheelerof4te 4y ago
ML is not that good at predicting.
quickthrower2 4y ago
Once the ML can understand Breakthrough Advertising, it might have a chance.
abotsis 4y ago
My Instagram ad conversations say otherwise.
linsomniac 4y ago
That's probably because people themselves are bad at predicting their own choices. Depending on mood, the day, even your hydration level, you can feel vastly different about something. I believe this was a conclusion of the Netflix Prize contest, getting much more accurate at predictions was hard because people would not reliably rate a movie the same.
metajunkies 4y ago
Google has fallen off the radar for me. I'm not one to waste time on someone who, if you ask me, is just another advertising company. It's the same old wine in new bottle. They pretend to predict correctly. Their false positives are true negatives.
Bolkan 4y ago
Ooh, this customer purchased a laptop. Lets fill up his recommendations with laptop ads.
rocket_surgeron 4y ago
As far as I can tell every single penny of the hundreds of millions (billions?) of dollars that have been spent on ML recommendation engines has been wasted and the team of highly-compensated developers who spent years burning through cash building Hadoop clusters, buying GPUs, building dashboards and churning out thousands of tables and charts for execs to ponder over just to build something that recommends I rebuy something I already bought could have been replaced with a 24-year old communications major who also understands the products being sold and the people buying them.
IncRnd 4y ago
ML may not be good at predicting consumers' choices, but multiple regression is. Multiple regression has been used for a long time for this. The side benefit is that it is mathematically repeatable while also being tweakable with variable inclusion or exclusion. https://scholar.google.com/scholar?q=%22multiple+regression%22+consumer+choice https://scholar.google.com/scholar?q=%22multiple+regression%...
FredPret 4y ago
I’ve only ever seen good ads on Facebook pre-iOS privacy update. Every other online ad has been a dumpster fire, from gross images, things aimed at people decades older or younger, or large items I just bought and aren’t buying again in years.
cerved 4y ago
> We suggest marketers focus less on trying to predict consumer choices with great accuracy and more on how the information environment affects the choice of their products. Similar to the thesis of How Brands Grow by Byron Sharp
throw__away7391 4y ago
Somehow I have the opposite experience of everyone else on this site? I frequently get targeted ads for exactly the item I'm interested in, e.g. a pair a shoes that actually look very similar to a pair I was considering but from a different brand. My YouTube recommendations bring me a ton of new videos on topics I'm interested in. I do a tiny amount of curation, but overall rarely get objectively bad videos recommended. Amazon recommends consumable products for me at approximately the rate I consume them. When my pet died and I stopped purchasing treats, they seem to have figured that out and stopped recommending them. The "you just bought a car, would you like to buy another car?" does happen from time to time, but not all that often to be honest.
- 2c2c2c 4y ago
  My guess is targeted advertising works great up until the point you share your devices with other people. parents and their kids sharing accounts, laptops, tablets leads to a bunch of junk input and this makes up the majority of population
  - zinekeller 4y ago
    Personal anecdote but my personal experience is that targeted advertising seems to be predicting my past me. I can't describe it exactly, but it seems to never realise that I'm done with (something), especially with once-in-a-moon purchases (like laptops).
muzani 4y ago
I buy a Marvel Unlimited subscription. The entire internet: "I heard you like comics. Would a Marvel Unlimited subscription interest you?"
deleted 4y ago
[deleted]
dooopy 4y ago
It probably just doesn't have the right data yet.
nsonha 4y ago
but it good for luring them into those choices so virtually same thing
csense 4y ago
I have a sneaking suspicion that, for many business problems, ML does no better than linear regression on cleaned-up data.
smcin 4y ago
The title is oversimplified clickbaity, and their actual finding is [from abstract], and my reaction below: Our main conclusion is that for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard. In fact, in many cases, prediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante. Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise—so much so that it is often a waste of effort. My initial reaction on skimming it is "savvy consumers are becoming increasingly desensitized to a sea of Facebook ads, Amazon fake reviews and rigged star ratings, undisclosed compensated influencers", aka the "unprecedented information environment" as the author describes things. I wouldn't call an Amazon review or FB ad/influencer post/affiliate link "information" as distinct to "this laptop has a rated battery life of 18h" or "this toaster comes in the following 5 color choices"; it's simply an influencing attempt; whether those contain any information (/misinformation), and whether users trust that they contain accurate information, seems to be something the study doesn't want to look into. Really the authors seem to be giving a very-judgment-free pass to anything calling itself "information". Consider "unprecedented information environment" could also characterize the 2016 and 2020 US elections, 2016 UK Brexit referendum and 2022 Philippines election: "it is important to first understand how consumers make choices, particularly in the current information environment in which they have access to an unprecedented amount of information at the time they are making decisions." [obviously third-party political advertising is far less trustworthy than consumer advertising, but still]. What if the authors merely succeeded in proving that the rise in targeted influencing attempts has rendered the public more wary of targeted influencing attempts?
Havoc 4y ago
They'd also do well to add some common sense filters. Amazon must have 5+ years of regular purchase history on my. You'd think thye could deduce by now that I am not in the market for pet and baby stuff. Sometimes I'd settle for some vanilla common sense. Watch Episode 1 of something on Prime. You'd think next time it would suggest Episode 2, right? It does...its thumbnail #25 in the grid & below the fold.

251 points by macleginn 4y ago

ape4 4y ago

You just bought a washing machine... could I interest you in a washing machine?

im3w1l 4y ago
GPT can solve this! I prompted it with "Sarah bought a washing machine and a ". It completed "dryer.". Another "If you buy a hammer you might also want to buy " -> "a nail". Ill forgive the singular. Just to be clear those are not cherry picked - they were my first two attempts.
- ape4 4y ago
  Putting those together... I actually bought a pair of anti hammer arrestors for the washing machine ;)
- thaumasiotes 4y ago
  > GPT can solve this! I prompted it with "Sarah bought a washing machine and a ". It completed "dryer.". The most natural interpretation there is that Sarah bought a washing machine and a dryer simultaneously, not that, after buying a washing machine the month prior, she was finally ready to buy a dryer.
- Mordisquitos 4y ago
  Fair enough regarding the washer/dryer, but I'm unimpressed by the hammer/nail example. If anything, I might want to buy "some nails" rather than "a nail", but I would argue that's not a brilliant guess either—if I'm buying a hammer I most likely already own the item that I want to hit with it. Some more interesting suggestions could have been for example "some pliers", "a set of screwdrivers", "a pair of work gloves", "safety goggles", etc.
mdp2021 4y ago
While the chief absurdity is very clear (also mocked by Spitting Image - J.B. on a date: "You loved that steak? Good, I'll order another one!"), I am afraid that the intended idea may be that your memory about the ads of what you just bought will last as much as said goods. Utter nightmare (unnatural obsolescence, systemic perversity, pollution...) but. I have met R'n'D who admitted the goal was just to have something new to have people want to replace the old, on unsubstantial grounds.
armchairhacker 4y ago
I think the reason this happens is that when you start looking for washing machines, you start getting ads for them. Then when you buy nobody tells the ad companies that you just bought a washing machine so they still send you ads because they think you’re still looking. Even if you just went straight to the model site and clicked “buy”.
- thaumasiotes 4y ago
  We know that's not the reason; Amazon is infamous for advertising washing machines to people who have just bought a washing machine from Amazon.
  - jsjohnst 4y ago
    Amazon can’t suggest a “product you might like” if their life depended on it. They have 19 years of purchase history for me spanning thousands of product categories. Want to know a category I’ve never looked at or bought from Amazon, women's handbags. What’s near the top of the suggestions right now on the homepage for me? You guessed it!
    - thaumasiotes 4y ago
      > Amazon can’t suggest a “product you might like” if their life depended on it. Actually, there is one method they use to make suggestions that works very well. If you have a wishlist, sometimes they'll recommend items from there.
    - oriolid 4y ago
      The weirdest thing is that at one point, I think it's now almost 10 years, they had somewhat OK recommendations. It's not like "you bought programming books and Battlestar Galactica DVDs, so you might be interested in Neal Stephenson's novels" is really advanced but it's still miles ahead of the "you just bought a washing machine, so you might be interested in washing machines" level.
      - 4y ago
bolasanibk 4y ago
I cannot remember the reference now, but the reasoning I read was a person who just bought an item x might: 1. return the item if they are not satisfied with it and get a replacement Or 2. buy another one as a gift if they really like it. Both of these result in a higher fraction of conversions in this kind of targeting vs other targeting criteria.
- astrea 4y ago
  I’m also more inclined to believe that the ad networks simply don’t have a way of knowing a sale was made.
deleted 4y ago
[deleted]
wrycoder 4y ago
I buy a package of underwear. All I see for next three weeks on my browser is close ups of men’s briefs. It’s embarrassing, when associates glance at my screen.
- speedgoose 4y ago
  I agree it can be embarrassing to not use an ad blocker.
- jstx1 4y ago
  Kind of your fault - you're buying your underwear on a work device and you don't use an adblocker.
  - wrycoder 4y ago
    1. Not a work device. 2. We are discussing how ads are selected for presentation.
jsjohnst 4y ago
While this annoys the hell out of me too, there is a semi-logical explanation that makes sense in many cases: The ad targeting knows you looked at washing machines, but doesn’t know you purchased.
- kristopolous 4y ago
  Which is an extremely trivial check to add - if you got assigned that ticket, you'd probably point it at like 2 or so hours. However, they've been like this for over a decade so it's likely there intentionally. here's one way that could be possible: There could be some popular third party service that's integrated on many e-commerce sites that sells this information and doesn't actually give a damn if you bought the refrigerator or not. They're selling you, not the refrigerator. That's the problem with data brokers, it's mostly low quality data.
  - jsjohnst 4y ago
    > Which is an extremely trivial check to add - if you got assigned that ticket, you'd probably point it at like 2 or so hours. Yep, totally a 2hr task for an engineer who works on homedepot.com to “check” that you bought a fridge from lowes.com after you first price shopped the other site. Also a two hour task for a Google engineer to know you bought one in person at Best Buy after researching online first. Yes, there are basic cases (buying from same merchant as who’s suggesting) that should be handled, but let’s not foolishly pretend that’s the average case, let alone majority / all.
    - cerved 4y ago
      not to mention cross-device browsing
    - dahfizz 4y ago
      Amazon consistently shows me ads for products I bought on Amazon. Don't bother talking about edge cases when the simple case doesn't work at all.
Asooka 4y ago
A slightly more ridiculous one I experienced recently was when I searched where to buy tool X. Started getting ads for why I need tool X and why it's the best tool ever. I already want one, I'm looking where to buy, not trying to learn what it's for!
saurik 4y ago
College Humor--may it rest in peace :( :( :(--did a great bit about this a few years ago. https://www.youtube.com/watch?v=KbKdKcGJ4tM https://www.youtube.com/watch?v=KbKdKcGJ4tM It is absolutely ridiculous how useless targeted ads actually are... and seeing a real person try to sell you this stuff really underscores the insanity.
- maxbond 4y ago
  College Humor is alive and well! At least if you enjoy Dungeons and Dragons and silly game shows.
BonoboIO 4y ago
Sometimes you want a company like Facebook or google deliver ads to your potential costumer, but you don’t want, that the ad company knows how much product you sold. You know your ROI, but your ad company doesn‘t. I heard multiple stories from amazon, that hey still let the ad campaign or targeting running even if you bought the product for random times so external companies could not get insight in your businesses. If you got the washing machine you will not click to buy another. If you pay per click, it makes no difference how long you let the campaign running if it is targeted. I work in the ad industry.
- BonoboIO 4y ago
Gibbon1 4y ago
I merely looked at pressure cookers and got ads for full figured bra's and support garments for six months.

jrm4 4y ago

I'm not surprised at this result, mostly because of the inaccurate noise that the business of "marketing," (i.e. specifically marketing people selling their not-very-effective services) generates.

cj 4y ago

While not exactly aligned to the research, I've been surprised how poor Nest Thermostat's learning feature is. The main selling point for Nest is having a "learning thermostat". Perhaps my schedule is just not predictable enough, but the auto-generated temperature schedules it generates after its "learning" period is not even close to what I would manually set up on a normal thermostat. Maybe I'm just an "edge case" or part of the "long tail"

actusual 4y ago
Nah, you're not. I just gave up on mine and have a schedule. I also turned off "pre-cooling" because it would just kick on at like 6pm to "cool" the house for bedtime. I also bought several temperature sensors to use, which are fun. At night I have the thermostat use the sensor in my bedroom, then goes back to the main thermostat during the day.
- foobarian 4y ago
  See the next logical step is to outfit the output vents with servo-controlled actuators so you can fine-tune where the air is going!
  - actusual 4y ago
    Additionally: - Use a python script to switch heating from my heat pump to my gas furnace when energy prices invert. - Have live monitoring of my HVAC pressure health so that my servo controlled actuators don't bork my system.
PaulHoule 4y ago
When people hear that FAANG is involved in something an "Emperor's Clothes" effect kicks in and people stop making the usual assumption that "if it doesn't work for me it probably doesn't work for other people"
bell-cot 4y ago
Or, maybe they invested far more cash and care in marketing that feature than in programming that feature...

hourago 4y ago

> Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise The worst part of big data is the data itself. Used to be common will be shared on Facebook webs about "what is your political compass". There results were used to create political profiles of users and targeted propaganda. You don't need ML to predict the data that there user already has given.

humanistbot 4y ago

"It's tough to make predictions, especially about the future." ― Yogi Berra

alberto_ol 4y ago
probably not by yogi berra https://quoteinvestigator.com/2013/10/20/no-predict/ https://quoteinvestigator.com/2013/10/20/no-predict/

annoyingnoob 4y ago

Maybe humans have free will after all.

ugjka 4y ago
random will perhaps
- Spivak 4y ago
  It's funny you say random because if consumer choice was actually random with some known distribution it would be extremely predictable, no ML needed.
  - nequo 4y ago
    Known distribution doesn't mean extremely predictable. For example, if your water consumption is log-Cauchy, I will have a very hard time predicting it because the variance is infinite.

oxfordmale 4y ago

It is the same on Netflix. I have phases where I watch a certain genre for a few weeks and then move on. For example after a few Scandi crime series it is time for something else. However, at the same time my daughter loves Animé and pretty only watch that. It is really hard for an ML algorithm to grab these nuances.

bertil 4y ago
Netflix makes a far more obvious sin: not having “who is watching” as boolean choices. If I am watching with my partner, I want both of our accounts to mark that series as viewed. And I really want Netflix to tell me what I’m watching with her so that I don’t continue watching it without her because I will be single if that happens (again).
- oxfordmale 4y ago
  It would be a great revenue stream for Netflix. Are you sure you want to watch this without your partner ? Yes ? We recommend the following service for finding temporary accommodation on short notice
golemiprague 4y ago

rvz 4y ago

So this machine learning and deep learning hype has shown that it is a gimmick isn't it? After years of surveilling, collecting and training on user data it still doesn't work or gets attacked very easily over spoilt pixels and many other attacks? What a complete waste of time, money and CO2 being burned up in the data centers.

jacquesm 4y ago
No, it just isn't a silver bullet for every problem under the sun. But quite a few record holders on various problems are ML solutions and that is unlikely to change for the foreseeable future. It's just that as soon as you start out on every problem with 'ML will solve this!' that you're going to end up with a bunch of crap. The right tool for the problem wins every time.
Enginerrrd 4y ago
I don't know.... I think back on google search back in the ~2014 era. It was good. Like scary good. Like I'd type "B" and it would suggest "Btu to Joules conversion" and things like that. Actually it was better than that... it would anticipate things I hadn't even searched for before with very very little prompting. It seemed to adapt to context whether I was at work, on my phone, at home, etc. The results were exactly what I was looking for. Then it got taken over by ads and SEO and corrupting influences and it's just not that good anymore. IMO, the problem with DL isn't the tech. It's the way its being used. The reality is: For 99% of things advertised to me, I don't want to buy the goddamn product, and no amount of advertising will make me want to buy it. It's gotten to the point where if I see an ad for a product I think I'm more likely to buy a competitor whose ad I haven't seen because I assume the competitor is investing more in the product than the marketing. And everyone seems to have forgotten about hybrid approaches of ML and human beings that, IMO, are really good. But alas, "they don't scale". But at the same time, it's really interesting. For as much data as facebook should have about me, their ad rec's really suck and always have. (Perhaps it's because my only ad clicks ever are accidental ones?) I'm kind of astounded at how poor that result is. That said, I'm always very impressed by spotify's recommender system. I think it's one of the best on the net. Another thing I find interesting is that non-vote-based social media feed systems all really suck. Once they ditched chronological ordering it stopped appealing to me, and I don't know exactly why that is. Evidently I'm on some tail of the curve they don't care about.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

Plough_Jogger 4y ago

This review omits techniques from reinforcement learning (especially bandits) that have been used successfully in industry for years now.

bertil 4y ago
I think that the main issue is less the technique (although… yes, please use RL if you can) and more the lack of data. Browsing gives very little insight: dwell-time is a poor proxy for interest, and mixes horrid ideas that are so bad they are worth sharing with friends and confusing photos where you need to squint to figure out if it’s what you are looking for. Both e-commerce and social media are really not good at gathering express feedback for what people want and valuing that expressly. Please, let me tell you that I did spend time looking at this thread about the latest reality TV scandal but I don‘t want to hear about it ever again! Please, let me tag options as “maybe” or let me tell you what you’d need to change for me to buy that shirt. Public, performative Likes and Favourite lists that are instantly reactivation spam-fodder… Come on, you know better. I used to work for a big e-commerce site (the leading site for 18-25 y.o. females). We had millions of references (really) and it was a problem. The search team had layers upon layers of ranking algos, incredible papers at conference… but still, low impact on conversion. It was more than anything else that we could do, but nowhere as transformative as it could be. Instead, I suggested copying the Tinder interaction in a companion app: * left, never see that item again; * right, add it to a long list of stuff you might want to revisit. We probably would have to separate that from the Favourite list to avoid clutter, but maybe not, to make that selection worthwhile. The learning you could get from that dataset, even with a basic RL algo to queue suggestions… People thought it was “too much” which I’m still bitter about.
jeffreyrogers 4y ago
How are bandits used in consumer choice problems? Bandits solve almost the inverse problem: which choice to offer/take when it's uncertain which is best, but the problem under consideration in the blog post is about predicting which choice a consumer will pick, a standard marketing problem.

teruakohatu 4y ago

> Currently, we are still far from a point where machines are able to abstract high-level concepts from data or engage in reasoning and reflection Of course when an AI does that, we then say its just doing statistics, not reasoning. Until you have built a recommendation engine from scratch, it is hard to appreciate the complexity. I don't mean the complexity of the code or algorithm (ALS and Spark are straightforward enough) but the contextual problem. Models end up being large collections of models in a complex hierarchy, with hyperparams to tune higher level concepts such as "surprise" or business targets such as "revenue", "engagement" etc. TikTok have nailed this, as has Spotify.

Barrin92 4y ago
>Of course when an AI does that, we then say its just doing statistics, not reasoning. no, AI simply doesn't do that. Even Demis Hassabis of Deepmind fame in a recent interview pointed this out. Machine learning is great on averaging out a large amount of data, which is often useful, but it doesn't generate true novelty in any human sense. AI can play Go, it can't invent Go. In the same way today's recommender systems are great at averaging out my last 50 shopping items or spotify playlist but they can't take a real guess at what truly new thing I'd like based on a genuine understanding of say, my personality. Which is reflected in the quality of recommendations which is mostly "the thing you just bought/watched", which is ironically often incredibly uninteresting.
- teruakohatu 4y ago
  > but it doesn't generate true novelty in any human sense. Few humans do either, even great artists create from, or react to, the art they experienced in their life. At the end of the day this is a rather metaphysical question. A Go model is trained to play Go, not create new games. > on a genuine understanding of say, my personality That is exactly what they are trying to do. The purpose of many recommendation systems is to uncover the latent variables and categories that you might like. They are not just averaging what you listened to. It is feasible (but very unwise) to predict certain users like to listen to happy upbeat tracks on Fridays, and sad songs sung in Romance languages on Mondays. Each time you skip a track or relisten to a track (implicit feedback) or like a track (explicit feedback) you are giving it information, which might be more than just the interaction (the timestamp, the IP address location, how quick you were to skip the track etc.). On the other hand, the purpose of those systems is simply to optimize what generates the most short term income, in which case, you get 'boring'/simple shopping recommendations that on average make more profit. Systems have tried to predict model the human experience, but in the past there have been epic fails so nobody is trying to explicitly do that anymore. The classic example would be stores deciding teens were pregnant and inadvertently alerting their parents. Nobody wants to see ads for Amazon's in-house toilet paper brand just as they are about to get up to go to the toilet.

tomcam 4y ago

I can personally vouch that Amazon, Twitter, and YouTube all do horrible horrible jobs predicting my taste. And they have got worse over the years, not better

hourago 4y ago
That may make sense of you are not the average consumer. Optimizing for the most common case makes sense. I see that with Google search prediction, it's good but many times it predicts very sensible words for general use but not in the topic that I'm interested.
jltsiren 4y ago
My favorite experience with Amazon: I had just preordered novel 9 of The Expanse, and I got an email recommending something else from the same authors: novel 8 of the Expanse. A more sensible recommendation engine might have assumed that someone who preorders part n+1 of a series may already have part n. Not to mention that Amazon should have known that I already had novel 8 on my Kindle. I guess generating personalized recommendations at scale is still too expensive. We just get recommendations based on what other customers with vaguely similar tastes were interested in.
- kmnc 4y ago
  Funny thing is, your very comment is an indirect praise of the very thing they were advertising to you, and here it is being read by thousands of people. Are we so sure absurdly terrible ads don't actually beat out actually good well tailored ones? Looking through the history of radio ads, television ads, it seems like the best ads are always the stupidest. "Head on, apply directly to the forehead!" isn't so far off from "You bought a washing machine? Buy another!". The reality is, advertising optimizes to target stupid people because stupid people spend money. It is easier to trick a moron then sell a smart man something they actually want.
- alpaca128 4y ago
  > Not to mention that Amazon should have known that I already had novel 8 on my Kindle. Amazon doesn't seem to understand many things surrounding the Kindle. For example, it calculates the progress reading through a book by the last page I looked at. That means if I finished a book and jumped to the introduction it'll now be convinced I only read 1% of the book. This is so dumb, and I don't know why they even do it that way - the Kindle hardware should easily be capable of precisely keeping track of what pages I looked at.

aaronax 4y ago

ML is there to maximize business income--nothing else. If ML was benefiting me, it would know that 90% of the time I fire up Hulu I plan to watch the next episode of what I was watching last time. And it would make that a one click action. Instead I have to scroll past promotional garbage...every single time. Assholes.

buscoquadnary 4y ago
Honestly a lot of this ML to me seems eerily similar to how in older times people would use sheep entrails or crow droppings to try and predict the future. I mean basically that is what ML is, trying to predict the future, the difference is they called it magic, we call it math, but both seem to have about the same outcome, or understandability.
- treesprite82 4y ago
  > I mean basically that is what ML is, trying to predict the future If being so reductive, that's also the scientific method. Form a model on some existing data, with the goal of it being predictive on new unseen data. Key is in favoring the more predictive models. > they called it magic, we call it math, but both seem to have about the same outcome Find me some sheep entrails that can do this: https://imagen.research.google/ https://imagen.research.google/
  - ninkendo 4y ago
    One of the oft overlooked, yet critically important aspects of the scientific method is the hypothesis. You don’t design an experiment having absolutely no idea what to expect. You have an educated guess in mind (the hypothesis), and you design the experiment in such a way that says “this result will rule out my hypothesis, while this other result might confirm it.” Just trying two things at random and picking the one that makes some arbitrary metric go up, is not the scientific method. It’s gradient descent.
    - treesprite82 4y ago

mgraczyk 4y ago

Always interesting to see outsiders writing papers about this, using anecdote and unrelated data (mostly political and real world purchase data in this case) to argue that ML doesn't make useful predictions. Meanwhile I look at randomized controlled trial data showing millions of dollars in revenue uplift directly attributable to ML vs non-ML backed conversion pipelines, offsetting the cost of doing the ML by >10x. It reminds me a lot of other populist folk-science belief, like vaccine hesitancy. Despite overwhelming data to the contrary, a huge portion of the US population believes that they are somehow better off contracting COVID-19 naturally versus getting the vaccine. I think when effect sizes per individual are small and only build up across large populations, people tend to believe whatever aligns best with their identity.

mushufasa 4y ago
I think you may be conflating the topics and goals of adjacent exercises; predicting consumer behavior is not the same thing as optimizing a conversion pipeline.
- mgraczyk 4y ago
  The examples they give in section two are directly relevant to optimizing conversion pipelines. They pretty clearly intend to be describing something relevant the e-commerce user experience.
nojito 4y ago
>Always interesting to see outsiders writing papers about this I don't think you know who andrew gelman is. Additionally, that's not the conclusion derived from this study.
- mgraczyk 4y ago
  The actual conclusion of the study is so absurd that it's not worth engaging with seriously. That is, to maximally understand, and therefore predict, consumer preferences is likely to require information outside of data on choices and behavior, but also on what it is like to be human. I was responding to the interpretation from the blog post, which is more reasonable.

duxup 4y ago

Is there much that is good about predicting this stuff? I find Amazon loves to tell me to buy ... the thing they know I just bought and you don't need more than one of ... I hardly ever get ads or offers for things I want. How do you mess that up?

alephxyz 4y ago
Google seems like they target by age, gender and income rather than by interests. Sometimes it's convinced I'm a yuppie and keeps showing me luxury cars, personal care/beauty products and high end electronics (when I have zero interest in any of those products). Ironically I find the "dumb" ads on cable tv news to be a lot more effective since they have to target by interests.

shaburn 4y ago

gwbas1c 4y ago

> for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard Translation: Computers can't read minds. A bigger generalization is that, whenever a software feature becomes essentially mind reading; someone's either feeding a hype engine or letting their imagination run away. The best things to do in that case is to pop the bubble if you can, or walk away. I will often clearly state, "Computers can't read minds. You're making a lot of assumptions that will most likely prove false."

malkia 4y ago

Some years ago, I worked on a team "Ads Human Eval" - we had raters hired to do A/B testing for ads. These evaluated questionaires carefuly crafted by our linguists, and then analyzed by the statisticians providing feedback to the (internal) group that wanted to know more about. So the best experience was this internal event that we had, where the raters would say that certain Ad would not fare well (long term), while the initial metrics (automated) were showing the opposite (short temr). So then we'll gather into this event, and people would "debug" these and try to find where the differences are coming through. Then we had to help another group, where ML failed miserably detecting ads that should've not been shown on specific media, and raters came to help giving the correct answers. The one thing that I've learned is that humans are not going to be replaced any time soon by AI, and I've been telling my folks, friends or anyone (new-born luddities) - that automation is not going to fully replace us. We'll still be needed as teachers, evaluators, fixers, tweakers/hackers - e.g. someone saying - this is right, and this is not, this needs adjustment, etc. (to the machine, ai, etc.). Maybe machines are going to take over us one day, but until then, I'm not worried... (I've also understood I knew nothing about staticics, and how valuable linguists are when comes to forming clear, concise and non-confusing (no double meaning) questions)

Melatonic 4y ago
I dont think most people are arguing that machines will replace everyone anytime soon - it is that they will replace a huge portion of people. If one person can do the job of 10,000 by being the tweaker / approver of an advanced AI that is still 9,999 jobs eliminated. That might be hyperbole (you still probably will need people to support that system)
- malkia 4y ago
  I agree, but it's true that some jobs should not simply exists. To this day, if I go to our airport in Sofia (Bulgaria), and my baggage is over the limit of 20 or was it 25kg I have to go to another place, pay for it and come back (why? bureaucracy - not only I have to do it, but I'm slowing anyone waiting for me to this - it's like 25-50 meters one place to the other) Unlike Frakfurt, Munich or Heathrow airport where I can pay that fine right there. Some jobs simply should not exist.

macNchz 4y ago

I've shared this before on HN, but it never fails to make me laugh when I think about it: >Several years ago a conversation about a similar topic prompted me to look at the ad targeting data Facebook had on me. At the time I'd had a Facebook account for 12 years with lots of posts, group memberships and ~500 friends. Their cutting edge data collection and complex ad targeting algorithms had identified my "Hobbies and activities" as: "Mosquito", "Hobby", "Leaf" and "Species": https://imgur.com/nWCWn63 https://imgur.com/nWCWn63. Whatever that means.

sarahlwalks 4y ago

As far as I'm concerned, the question is how ML/AI stacks up against the competition -- humans. I don't know, but I'd bet the answer is that ML is much better. Let's say at least 20 percent better, but I imagine it's much higher than that. Second, this is only saying that right now, ML's performance is "not that good." It says nothing about future technical advances. If you look at the track record of ML in the past three decades, it's amazing, and if that performance is repeated in the next three decades, who even knows what things might look like. (Machine sentience? Maybe.)

IAmWorried 4y ago

It seems to me like the "generation" use case of ML is much more promising than the "prediction" or "classification" use case. It's tough to predict things in general because our universe is fundamentally uncertain. How is some computer going to predict that some mugger sees a target at some random spot and decides to mug them? But the progress in text to image and text generation really blows my mind.

wheelerof4te 4y ago

ML is not that good at predicting.

quickthrower2 4y ago

Once the ML can understand Breakthrough Advertising, it might have a chance.

abotsis 4y ago

My Instagram ad conversations say otherwise.

linsomniac 4y ago

That's probably because people themselves are bad at predicting their own choices. Depending on mood, the day, even your hydration level, you can feel vastly different about something. I believe this was a conclusion of the Netflix Prize contest, getting much more accurate at predictions was hard because people would not reliably rate a movie the same.

metajunkies 4y ago

Google has fallen off the radar for me. I'm not one to waste time on someone who, if you ask me, is just another advertising company. It's the same old wine in new bottle. They pretend to predict correctly. Their false positives are true negatives.

Bolkan 4y ago

Ooh, this customer purchased a laptop. Lets fill up his recommendations with laptop ads.

rocket_surgeron 4y ago

As far as I can tell every single penny of the hundreds of millions (billions?) of dollars that have been spent on ML recommendation engines has been wasted and the team of highly-compensated developers who spent years burning through cash building Hadoop clusters, buying GPUs, building dashboards and churning out thousands of tables and charts for execs to ponder over just to build something that recommends I rebuy something I already bought could have been replaced with a 24-year old communications major who also understands the products being sold and the people buying them.

IncRnd 4y ago

ML may not be good at predicting consumers' choices, but multiple regression is. Multiple regression has been used for a long time for this. The side benefit is that it is mathematically repeatable while also being tweakable with variable inclusion or exclusion. https://scholar.google.com/scholar?q=%22multiple+regression%22+consumer+choice https://scholar.google.com/scholar?q=%22multiple+regression%...

FredPret 4y ago

I’ve only ever seen good ads on Facebook pre-iOS privacy update. Every other online ad has been a dumpster fire, from gross images, things aimed at people decades older or younger, or large items I just bought and aren’t buying again in years.

cerved 4y ago

> We suggest marketers focus less on trying to predict consumer choices with great accuracy and more on how the information environment affects the choice of their products. Similar to the thesis of How Brands Grow by Byron Sharp

throw__away7391 4y ago

Somehow I have the opposite experience of everyone else on this site? I frequently get targeted ads for exactly the item I'm interested in, e.g. a pair a shoes that actually look very similar to a pair I was considering but from a different brand. My YouTube recommendations bring me a ton of new videos on topics I'm interested in. I do a tiny amount of curation, but overall rarely get objectively bad videos recommended. Amazon recommends consumable products for me at approximately the rate I consume them. When my pet died and I stopped purchasing treats, they seem to have figured that out and stopped recommending them. The "you just bought a car, would you like to buy another car?" does happen from time to time, but not all that often to be honest.

2c2c2c 4y ago
My guess is targeted advertising works great up until the point you share your devices with other people. parents and their kids sharing accounts, laptops, tablets leads to a bunch of junk input and this makes up the majority of population
- zinekeller 4y ago
  Personal anecdote but my personal experience is that targeted advertising seems to be predicting my past me. I can't describe it exactly, but it seems to never realise that I'm done with (something), especially with once-in-a-moon purchases (like laptops).

muzani 4y ago

I buy a Marvel Unlimited subscription. The entire internet: "I heard you like comics. Would a Marvel Unlimited subscription interest you?"

deleted 4y ago

[deleted]

dooopy 4y ago

It probably just doesn't have the right data yet.

nsonha 4y ago

but it good for luring them into those choices so virtually same thing

csense 4y ago

I have a sneaking suspicion that, for many business problems, ML does no better than linear regression on cleaned-up data.

smcin 4y ago

The title is oversimplified clickbaity, and their actual finding is [from abstract], and my reaction below: Our main conclusion is that for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard. In fact, in many cases, prediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante. Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise—so much so that it is often a waste of effort. My initial reaction on skimming it is "savvy consumers are becoming increasingly desensitized to a sea of Facebook ads, Amazon fake reviews and rigged star ratings, undisclosed compensated influencers", aka the "unprecedented information environment" as the author describes things. I wouldn't call an Amazon review or FB ad/influencer post/affiliate link "information" as distinct to "this laptop has a rated battery life of 18h" or "this toaster comes in the following 5 color choices"; it's simply an influencing attempt; whether those contain any information (/misinformation), and whether users trust that they contain accurate information, seems to be something the study doesn't want to look into. Really the authors seem to be giving a very-judgment-free pass to anything calling itself "information". Consider "unprecedented information environment" could also characterize the 2016 and 2020 US elections, 2016 UK Brexit referendum and 2022 Philippines election: "it is important to first understand how consumers make choices, particularly in the current information environment in which they have access to an unprecedented amount of information at the time they are making decisions." [obviously third-party political advertising is far less trustworthy than consumer advertising, but still]. What if the authors merely succeeded in proving that the rise in targeted influencing attempts has rendered the public more wary of targeted influencing attempts?

Havoc 4y ago

They'd also do well to add some common sense filters. Amazon must have 5+ years of regular purchase history on my. You'd think thye could deduce by now that I am not in the market for pet and baby stuff. Sometimes I'd settle for some vanilla common sense. Watch Episode 1 of something on Prime. You'd think next time it would suggest Episode 2, right? It does...its thumbnail #25 in the grid & below the fold.