12 ms·

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They w

I don't have time to read the entire paper but I would like to share an anecdote. I worked at a company with a well staffed/funded machine learning team. They were in charge of recommendation systems - think along the lines of youtube up next videos. My team wanted better recommendations (really, less editorial intensive) so the ML team spent weeks crafting 12 or more variants of their recommendation system for our content. We then ran that system for months in an A/B testing system that judged user behaviour by several KPI. The result was all variants performed equally well within statistically insignificant bounds. The best performing variant happened to be random. Talking to other groups that had gone through the exact same process our results were pretty typical. These guys were all very intelligent and the code and systems they had implemented were pretty impressive. I'm guessing the system they built would have cost a few million dollars if built from scratch. We did use this "AI/ML" in our marketing so maybe it was payed for by increased sales through use of buzz words. But my experience was that in most limited use cases the technology was ineffective.

bulldoa 7y ago
Isn't sparse recommendation for videos kind of solved in netflix prize, where the winner uses SVD to extract signature characteristic and recommend videos base on that?
- wtvanhest 7y ago
  Probably not. I say this because for many months, i would visit netflix and not want to watch anything. Eventually I cancelled my subscription after many years. I think I'd rather have a random collection of titles than a recommended list for me.
- howlin 7y ago
  There are a lot of ways of formalizing the problem of recommendation. Perhaps the variant of the problem used by Netflix is "solved", but it's kind of an odd one. Basically, they built a system to answer questions of the following form: "Given that user X watched media Y, what rating would they give it?" They trained and tested on media that users have already rated. Some of the ratings are masked and thus need to be "predicted" for the test. The issue is that the Netflix dataset has a baked-in assumption that a recommender system should show media that a user is likely to have ranked highly. It may be more important to show the user media they wouldn't have found (and thus ranked) at all. Or perhaps a user will be more engaged with something controversial rather than generically acceptable. Who knows?
- jfarmer 7y ago
  Just because the medium is the same doesn't mean the customer wants the same types of recommendations in two different contexts.
- dahart 7y ago
  I feel like if the Netflix Prize results had truly solved the problem, then they’d still be using them. It seems like the video recommendations aren’t as good as they were ten years ago during the prize competition, and they’re no longer based on what I might like but rather what Netflix wants me to watch.
nerdponx 7y ago
That's surprising to hear. Comparing model performance to a randomized baseline model is a "must-have" on my team before we feel comfortable presenting to management.
- ACow_Adonis 7y ago
  An old team I advised for a while also compared model performance to a randomised baseline model. What they didn't seem to get however was that a randomised baseline model would beat a randomised baseline model on a naive comparison 50% of the time, so their understanding of randomness/statistical significance/performance metrics was way off. So while they believed they were also testing their models before presenting to management, none of them were implementing their comparison/measurements properly, and huge parts of their work were just p- hacking and pulling random high performing results out of the tails of the many models they built and compared. So while it's good your team makes comparison to baselines (it's alarming how many don't even do that), my experience also suggests a huge number who think they're comparing to reasonable baselines and using metrics to measure their performance aren't actually doing so properly.
  - bulldoa 7y ago
    I am confused, if a new model beats randomly selected randomised model 100% of time for each experiment why does it matter if randomised model beats other randomised models? Are they only comparing against the subset of worst randomised models?
    - soVeryTired 7y ago
      I think he's saying something like the following: 1/ the team implemeted a naive baseline 2/ they implemeted a more sophisticated model that depended on some parameter p 3/ for 100 different values of p, they examined its performance, and picked the model with the best performance Now they're not quite subject to the multiple comparisons problem there, since the models with different values of p aren't independent from one another. But they're not not suffering from it either. It mostly depends on the model. But it's a very easy mistake to make. I'd say many many academic papers make the same mistake.
cycrutchfield 7y ago
If the team can’t even beat random, then I think that says more about your team (or perhaps your features) than about ML as a whole.
- DenisM 7y ago
  Today ML can solve some problems. In the future it might solve some problems with advances in the field. Yet other problems will likely remain unsolved, such as the stock market, or the weather, or predicting /dev/rand "Up Next" problem can easily fall into any of the three buckets.
  - cycrutchfield 7y ago
    YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems".
    - deleted 7y ago
      [deleted]
    - simion314 7y ago
      >YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems". IMO YT AI is the opposite of intelligent , it still recommends things I disliked. for some reason this basic rule of not showing something that I explicitly disliked was to hard for it to learn, I am wondering if it is truly an AI behind it or just statistics
      - dhairya 7y ago
        It's intentional. Controversy is a strong signal for the youtube algorithm.
        simion314 7y ago
        I don't think so, the videos I was referring were music videos. I engaged with it in a way but hitting dislike on a music genre I don't want to listen(I normally don't dislike things because is not my genre) in the hope the algorithm will learn but it was even worse, it did not learned that I disliked artist X and genre Y it continue to play the exact video that I disliked, The bad algorithm will force the unhappy user to use manually created playlists leaving less people to engage with the algorithm and probably have the algorithm getting worse in time as more users will avoid it And even more interesting trying to google "how to make youtube not show X" it is a complete fail, it will just show you youtube video results.
nilkn 7y ago
I wonder if this could be a case of mismatch between what the recommendations system was designed to do and what the business actually needed it to do. Your team evaluated the models based on live KPIs in an A/B testing environment, but did the recommendations team develop the system specifically with those KPIs in mind? Did they ever have access to adequate information to truly solve the problem your team needed solved? And was the same result observed for other uses of their recommendation systems?
- reggieband 7y ago
  > did the recommendations team develop the system specifically with those KPIs in mind? Yes they did - in fact they had input on defining them and helped in tracking them. > Did they ever have access to adequate information to truly solve the problem your team needed solved? They believed so. Their team was also responsible for our company data warehousing so they knew even better than me what data was available. Basically any piece of data that could be available they had access to. > And was the same result observed for other uses of their recommendation systems? I did not have first-hand access to the results of their use in other recommendation contexts. As I mentioned in my original post I only had second-hand accounts from other teams that went the same route. They reported similar results to me.
  - mobjack 7y ago
    Some ideas seem to attract smart people like moths to a flame. It seems like everyone who joins my company to shake things up follows the same path of wanting personalized content to acquire new customers. But in reality we just don't have enough data points on people before they become customers to segment people that way. Even if we could, being able to accurately Every time I see people go through the motions of attempting to implement this until they eventually give up. This idea looks like an obvious win and big companies have done them before with success, but is extremely hard to impossible to pull off for our small company.
mezentius 7y ago
>>The best performing variant happened to be random. Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions. In certain domains, I think, it's quite difficult to beat the perceived performance of an AI system that merely makes committed random decisions (i.e. carried out over time) within a set of reasonable choices. If we don’t understand what an agent is doing, we often assume that there is some devious and subtle purpose behind its actions.
- nradov 7y ago
  Humans are great at finding "patterns" in random noise.
  - JimmyRuska 7y ago
    Sometimes being unpredictable makes for a good strategy
  - zamadatix 7y ago
    We are very good at finding patterns where there aren't any but it's important to remember that random is actually the best answer a lot of the times. Maybe it's as simple as random plays being the hardest to predict and that's the best until you can get an AI trained on the meta of how to plan for what the current player is thinking you'll do.
- keyle 7y ago
  It's quite common in games for AI to pick a random decision. Simply put, a good AI is a character/NPC that appears to have a mind of its own, and its own life. Nothing beats random at explaining someone's behaviour based on a personal history you don't know. If AI responded/acted based on a predefined set of patterns that could be recognizable, the player would automatically feel it (pattern matching) and makes the NPC far less interesting.
JoeAltmaier 7y ago
The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.
- stefco_ 7y ago
  Using a markov chain with your stochastic matrix set to I...
  - thanatropism 7y ago
    Using “a I”
- nicholas73 7y ago
  To be fair San Jose weather is heavily biased towards sameness.
  - JoeAltmaier 7y ago
    Yes, but why did all the 'smart' people using models and training networks get a worse result? A condemnation of modelling, to be sure.
- deleted 7y ago
  [deleted]
- mark-r 7y ago
  I have a friend who lived in San Jose who just wrote the forecast on his whiteboard and left it there, because it never changed. It was funny, because he came from Minnesota where the weather is never the same two days in a row.
  - dbcurtis 7y ago
    Ha! I came here to say something similar. Here in San Jose, the weather tomorrow will be the same as today, most of the time. My joke when I lived in Minnesota was: "If you don't like the weather, just wait 10 minutes. It will be different." Gotta say though, I don't miss my slow blower even just a wee little bit.
jfarmer 7y ago
A relevant Twitter thread: https://twitter.com/NeuroStats/status/1192679554306887681 https://twitter.com/NeuroStats/status/1192679554306887681 At the risk of projecting, this has the hallmark of bad experimental design. The best experiments are designed to determine which of many theories better account for what we observe. (When I write "you" or "your" below, I don't mean YOU specifically, but anyone designing the kind of experiment you describe.) One model of gravity says the postition/time curve of a ball dropped from a height should look like X. Another model of gravty says it should look like Y. You drop many balls, plot their position/time, and see which of the two models' curves match what you observe. The goal isn't to get the curve; the goal is to decide which model is a better picture of our universe. If the plotted curve looks kinda-sorta like X but NOTHING like Y, you've at least learned that Y is not a good model. What models/theories of customer behavior were your experiments designed to distinguish between? My guess is "none" because someone thinking about the problem scientifically would start with a single experiment whose results are maximally dispositive and go from there. They wouldn't spend a bunch of time up-front designing 12 distinct experiments. So it wasn't really an experiment in the scientific sense, but rather a kind of random optimization exercise: do 12 somewhat-less-than-random things and see which, if any, improve the metrics we care about. Random observations aren't bad, but you'd do them when you're trying to build a model, not when you're trying to determine to what extent a model corresponds with reality. For example, are there any dimensions along which the 12 variants ARE distinguishable from one another? That might point the way to learning something interesting and actionable about your customers. Did the team treat the random algorithm as the control? Well, if you believe some of your customers are engaged by novelty then maybe random is maximally novel (or at least equivalently novel), and so it's not really a control. What about negative experiments, i.e., recommendations your current model would predict have a NEGATIVE impact on your KPIs? If those experiments DON't produce a negative impact then you've learned that some combination of the following is the case: 1. The current customer model is inaccurate 2. The model is accurate but the KPIs don't measure what you believe they do (test validity) 3. The KPIs measure what you believe they do but the instrumentation is broken Some examples of NEGATIVE experiments: What if you always recommend a video that consists of nothing but 90 minutes of static? What if you always recommend the video a user just watched? What if you recommend the Nth prior video a user watched, creating a recommendation cycle? Imagine if THOSE experiments didn't impact the KPIs, either. In that universe, you'd expect the outcome you observed with your 12 ML experiments. In fact, after observering 12 distinct ML models give indistingiushable results, I'd be seriously wondering if my analytics infrastructure was broken and/or whether KPIs measured what we thought they did.
adenverd 7y ago
I worked at a larger services marketplace, helping data scientists get their models into production as A/B experiments. We had an interesting and related challenge in our search ranking algorithms: we wanted to rank order results by the predicted lifetime value of establishing a relationship between searcher and each potential service provider. In our case, a 1% increase in LTV from one of these experiments would be...big. Really big. Improving performance of these ranking models was notoriously difficult. 50% of the experiments we'd run would show no statistically significant change, or would even decrease performance. Another 40% or so would improve one funnel KPI, but decrease another, leading to no net improvement in $$. Only 10% or so of experiments would actually show a marginal improvement to cohort LTV. I'm not sure how much of this is actually "there's very little marginal value to be gained here" versus lack of rigor and a cohesive approach to modeling. The data scientists were very good at what they do, but ownership of models frequently changed hands, and documentation and reporting about what experiments had previously been tried was almost non-existent. All that to say, productizing ML/AI is very time- and resource-intensive, and it's not always clear why something did/didn't work. It also requires a lot of supporting infrastructure and a data platform that most startups would balk at the cost of.
- JimmyRuska 7y ago
  If you have historical data to validate against, you can set a leader board on models run against older data, and always leave part of the data out and unavailable for test https://gluebenchmark.com/leaderboard/ https://gluebenchmark.com/leaderboard/ This encourages a simple first version and incremental complexity, rather than starting very complex 6 months in, and never having an easy baseline to compare to. A simple baseline can spawn off several creative methods of improvement to research. The other case is that the models should be run against simple cases that are easy to understand and easy to confirm. This way there's always a human QA component available to make sure results are sensible.
codingslave 7y ago
I have had a similar experience, but I really do think it points more to the team than it does to the efficacy of machine learning. I was on a team of extremely intelligent people, but they were very academic, with minimal practical coding skills, refusals to use version control etc. By academic, I mean graduates from the number one university in their respective fields, top publications, etc. They produced very little actual value. Great theoretical ideas, in depth understanding of different methods of optimization, etc. The team I was on before that one was a bunch of scrappy engineers from Poland, India and the USA with no graduate degrees, but 20 years coding and distributed systems experience each. The difference in problem solving ability, the speed at which they moved, broke down problems, tried out different methods, was staggering. I think ML is suffering from a prestige problem, and many companies are suffering for it. The wrong people are being hired and promoted, with business leaders calling the shots on who runs machine learning projects without fully understanding who can actually deliver.
btilly 7y ago
This reminds me of a job interview I was on. I was asked about how I would use AI/machine learning for their problem space. Since they seemed to be smart and level-headed, I answered honestly, "Pick something unimportant, use a machine learning algorithm just to get familiar with the tools, ignore the result unless it happens to work, then put machine learning in your marketing materials. But keep track of it, and if it is useful to you in 5 years, be sure to use it for real." They said, "That's about what we concluded except that we didn't get around to actually doing that pilot project yet." I got the job. :-)
- mc3 7y ago
  That sounds odd, like they don't really need machine learning (unless it is to snare investors?).
  - gpm 7y ago
    Snaring customers too. I swear, people are obsessed with "machine learning" even when the domain really isn't suited for it.
    - bigred100 7y ago
      I sometimes wonder if management and engineers don’t have more in common than acknowledged. Publications such as Harvard Business Review have huge coverage of things like managing AI, and being able to say you managed an “AI project” might mean something.
    - xmprt 7y ago
      I wonder if people are becoming more savvy and anti-ML after all the issues people have with Facebook and Google collecting a lot more data than people are comfortable with.
      - SuoDuanDao 7y ago
        I suspect what people dislike is big data. Anecdotally, I'd love to hear about a success where some lone genius codes a better tool using publicly available datasets. I don't like the idea of BigCorp using their larger servers to dominate the space.
deleted 7y ago
[deleted]
madrox 7y ago
I had a similar experience at one point. A team put energy into building a recommendation system, and were able to demonstrate that the "Recommended for you" content performed better than all other content editorial. After getting challenged a bit, though, turns anything performs better when put under the header "Recommended for you."
- mike_hock 7y ago
  > put energy put energy And they say English doesn't have reduplication!
- dgellow 7y ago
  Well, that’s a good lesson to learn, just put everything under “recommended for you” header! :)
- jshen 7y ago
  You aren’t talking about dcom are you?
deleted 7y ago
[deleted]
noodlenotes 7y ago
Recommender systems are notoriously hard because it's difficult to do better than just recommending the most popular content. You can recommend more personalized content at the expense of KPIs like click-through rate.
stefan_ 7y ago
Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought.
- propogandist 7y ago
  This can happen when a retargeting campaign doesn't have a 'burn pixel' or conversion event trigger. It's a common oversight, which can cause a re-targeting program to kick-off unnecessarily (or cause ads that have followed you around to become obvious)
  - idoh 7y ago
    The AI innovation must be that they can figure out which marketers are likely to forget to have a burn pixel because those marketers drive more revenue.
  - scottlocklin 7y ago
    You'd think with all the "AI" out there, they could match a sales DB entry with the CRM DB entry, but, in fact, they basically can't.
  - hnick 7y ago
    And yet I haven't seen one ad that has a 'not interested' feature similar to YouTube. I mean, if I could stop seeing washing machines or whatever, I'd probably click it.
    - propogandist 7y ago
      do you clear your cookies and html5 storage? That should wipe any personalization that's happening, but the ads will become very generic then. You can also block ads (Ublock/Umatrix)
      - hnick 7y ago
ArtWomb 7y ago
Personal recommendation systems all have tradeoffs. It's just the nature of curation as an intangible endeavor. You can love "Scarface", "Heat" and "LA Confidential" but still find "Casino" boring ;) More on such tradeoffs in a recent case study from DeepMind on Google Play Store app recommendations. Even they acknowledge the same techniques that surface 30% cost efficiencies in data center cooling, may not be completely applicable to "taste" https://deepmind.com/blog/article/Advanced-machine-learning-helps-Play-Store-users-discover-personalised-apps https://deepmind.com/blog/article/Advanced-machine-learning-...
vishwajeetv 7y ago
Interestingly, Steam's recommendation algorithm to show "similar to a game" by their 'learning machines' works very well. I have found really good games via that, and none of them did show up in the regular recommendations/carousel on top.
- lawlessone 7y ago
  pretty sure it just uses the tags players give it, strategy, sci-fi , openworld etc.
mikorym 7y ago
IMH (and biased) O, a lot of great coders are implimentors, or let's say applied computers scientists. That is great for building incredible open source software and a lot of other things that I would not be able to do given a 1000 years. However (again IMHBO) a specific ML and any other specific application of stastistics or mathematics becomes really tricky once your use case is explicitly defined. You then need intimate and deep knowledge of the tools that you are using (e.g.: Should I even use NN? Should I even use genetic algorithms? Should I even use x?) but ML for most people is shorthand for NN and its variants or maybe shorthand for something else specific rather than in principle. A well aimed shot at PCA [1] can often solve your problem. Or at least, tell you what the problem looks like. This is just an example, but IMHBO people waste their time learning ML and not learning mathematics and statistics. IMHBO I still think that self-driving cars can be solved by defining a list of 1000 or so rules, by hand, by humans, and by consensus. The computer vision part is the ML part. [1] https://en.wikipedia.org/wiki/Principal_component_analysis https://en.wikipedia.org/wiki/Principal_component_analysis
- jvanderbot 7y ago
  I could not agree more about self driving cars. They will disrupt and cause us to actually look at our terrible transportation infastructure, not learn to survive in it.
herval 7y ago
Doesn’t it depend on the size and variety of your dataset? I can see “random” performing well in a set of <1000 videos, all on similar subject spaces (eg “memes”, or “python”), but recommending relevant stuff gets much harder as the amount of content grows...
swiley 7y ago
I feel like A/B testing isn’t a great way to determine correctness either. IMO even in interface designing you should be arguing from first principles rather than relying on telemetry and other empirical data.
temp43t453 7y ago
excuse me for butting into dead conversation, but your team used ml to inapplicable case. (at least they aсquired experiense). up next recommendations wont work without advanced image recognition and topics gathering - basically titles/tags for most videos are garbage and clickbait, and most of youtube work by watching buzzed videos: some well known (by a big amout of watchers) "influencers" push video of some topic (thing/brand) then it get traction from other content creators - they produce videos about it and watchers tend to stick to buzzed topics. it's like news about news. if your team used ml to recommend up next on your own video hosting your result simply means your videos are equally not on topic Or non-interesting for your service auditory; or they are garbage.

bulldoa 7y ago

Isn't sparse recommendation for videos kind of solved in netflix prize, where the winner uses SVD to extract signature characteristic and recommend videos base on that?

wtvanhest 7y ago
Probably not. I say this because for many months, i would visit netflix and not want to watch anything. Eventually I cancelled my subscription after many years. I think I'd rather have a random collection of titles than a recommended list for me.
howlin 7y ago
There are a lot of ways of formalizing the problem of recommendation. Perhaps the variant of the problem used by Netflix is "solved", but it's kind of an odd one. Basically, they built a system to answer questions of the following form: "Given that user X watched media Y, what rating would they give it?" They trained and tested on media that users have already rated. Some of the ratings are masked and thus need to be "predicted" for the test. The issue is that the Netflix dataset has a baked-in assumption that a recommender system should show media that a user is likely to have ranked highly. It may be more important to show the user media they wouldn't have found (and thus ranked) at all. Or perhaps a user will be more engaged with something controversial rather than generically acceptable. Who knows?
jfarmer 7y ago
Just because the medium is the same doesn't mean the customer wants the same types of recommendations in two different contexts.
dahart 7y ago
I feel like if the Netflix Prize results had truly solved the problem, then they’d still be using them. It seems like the video recommendations aren’t as good as they were ten years ago during the prize competition, and they’re no longer based on what I might like but rather what Netflix wants me to watch.

nerdponx 7y ago

That's surprising to hear. Comparing model performance to a randomized baseline model is a "must-have" on my team before we feel comfortable presenting to management.

ACow_Adonis 7y ago
An old team I advised for a while also compared model performance to a randomised baseline model. What they didn't seem to get however was that a randomised baseline model would beat a randomised baseline model on a naive comparison 50% of the time, so their understanding of randomness/statistical significance/performance metrics was way off. So while they believed they were also testing their models before presenting to management, none of them were implementing their comparison/measurements properly, and huge parts of their work were just p- hacking and pulling random high performing results out of the tails of the many models they built and compared. So while it's good your team makes comparison to baselines (it's alarming how many don't even do that), my experience also suggests a huge number who think they're comparing to reasonable baselines and using metrics to measure their performance aren't actually doing so properly.
- bulldoa 7y ago
  I am confused, if a new model beats randomly selected randomised model 100% of time for each experiment why does it matter if randomised model beats other randomised models? Are they only comparing against the subset of worst randomised models?
  - soVeryTired 7y ago
    I think he's saying something like the following: 1/ the team implemeted a naive baseline 2/ they implemeted a more sophisticated model that depended on some parameter p 3/ for 100 different values of p, they examined its performance, and picked the model with the best performance Now they're not quite subject to the multiple comparisons problem there, since the models with different values of p aren't independent from one another. But they're not not suffering from it either. It mostly depends on the model. But it's a very easy mistake to make. I'd say many many academic papers make the same mistake.

cycrutchfield 7y ago

If the team can’t even beat random, then I think that says more about your team (or perhaps your features) than about ML as a whole.

DenisM 7y ago
Today ML can solve some problems. In the future it might solve some problems with advances in the field. Yet other problems will likely remain unsolved, such as the stock market, or the weather, or predicting /dev/rand "Up Next" problem can easily fall into any of the three buckets.
- cycrutchfield 7y ago
  YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems".
  - deleted 7y ago
    [deleted]
  - simion314 7y ago
    >YouTube's "Up Next" recommendations do (significantly) better than random, therefore "Today ML can solve some problems". IMO YT AI is the opposite of intelligent , it still recommends things I disliked. for some reason this basic rule of not showing something that I explicitly disliked was to hard for it to learn, I am wondering if it is truly an AI behind it or just statistics
    - dhairya 7y ago
      It's intentional. Controversy is a strong signal for the youtube algorithm.
      - simion314 7y ago
        I don't think so, the videos I was referring were music videos. I engaged with it in a way but hitting dislike on a music genre I don't want to listen(I normally don't dislike things because is not my genre) in the hope the algorithm will learn but it was even worse, it did not learned that I disliked artist X and genre Y it continue to play the exact video that I disliked, The bad algorithm will force the unhappy user to use manually created playlists leaving less people to engage with the algorithm and probably have the algorithm getting worse in time as more users will avoid it And even more interesting trying to google "how to make youtube not show X" it is a complete fail, it will just show you youtube video results.

nilkn 7y ago

I wonder if this could be a case of mismatch between what the recommendations system was designed to do and what the business actually needed it to do. Your team evaluated the models based on live KPIs in an A/B testing environment, but did the recommendations team develop the system specifically with those KPIs in mind? Did they ever have access to adequate information to truly solve the problem your team needed solved? And was the same result observed for other uses of their recommendation systems?

reggieband 7y ago
> did the recommendations team develop the system specifically with those KPIs in mind? Yes they did - in fact they had input on defining them and helped in tracking them. > Did they ever have access to adequate information to truly solve the problem your team needed solved? They believed so. Their team was also responsible for our company data warehousing so they knew even better than me what data was available. Basically any piece of data that could be available they had access to. > And was the same result observed for other uses of their recommendation systems? I did not have first-hand access to the results of their use in other recommendation contexts. As I mentioned in my original post I only had second-hand accounts from other teams that went the same route. They reported similar results to me.
- mobjack 7y ago
  Some ideas seem to attract smart people like moths to a flame. It seems like everyone who joins my company to shake things up follows the same path of wanting personalized content to acquire new customers. But in reality we just don't have enough data points on people before they become customers to segment people that way. Even if we could, being able to accurately Every time I see people go through the motions of attempting to implement this until they eventually give up. This idea looks like an obvious win and big companies have done them before with success, but is extremely hard to impossible to pull off for our small company.

mezentius 7y ago

>>The best performing variant happened to be random. Some years ago I heard an anecdote from a developer who had worked on a video game about American football. The gist of it was that they had tested various sophisticated systems for an AI opponent to choose a possible offensive/defensive play, but the one that the players often considered the most "intelligent" was the one that simply made random decisions. In certain domains, I think, it's quite difficult to beat the perceived performance of an AI system that merely makes committed random decisions (i.e. carried out over time) within a set of reasonable choices. If we don’t understand what an agent is doing, we often assume that there is some devious and subtle purpose behind its actions.

nradov 7y ago
Humans are great at finding "patterns" in random noise.
- JimmyRuska 7y ago
  Sometimes being unpredictable makes for a good strategy
- zamadatix 7y ago
  We are very good at finding patterns where there aren't any but it's important to remember that random is actually the best answer a lot of the times. Maybe it's as simple as random plays being the hardest to predict and that's the best until you can get an AI trained on the meta of how to plan for what the current player is thinking you'll do.
keyle 7y ago
It's quite common in games for AI to pick a random decision. Simply put, a good AI is a character/NPC that appears to have a mind of its own, and its own life. Nothing beats random at explaining someone's behaviour based on a personal history you don't know. If AI responded/acted based on a predefined set of patterns that could be recognizable, the player would automatically feel it (pattern matching) and makes the NPC far less interesting.

JoeAltmaier 7y ago

The San Jose Mercury News had a weather-forecasting contest. It was won one year by a 12-year-old, who's algorithm was "The weather tomorrow will be the same as the weather today". A kind of AI I guess.

stefco_ 7y ago
Using a markov chain with your stochastic matrix set to I...
- thanatropism 7y ago
  Using “a I”
nicholas73 7y ago
To be fair San Jose weather is heavily biased towards sameness.
- JoeAltmaier 7y ago
  Yes, but why did all the 'smart' people using models and training networks get a worse result? A condemnation of modelling, to be sure.
deleted 7y ago
[deleted]
mark-r 7y ago
I have a friend who lived in San Jose who just wrote the forecast on his whiteboard and left it there, because it never changed. It was funny, because he came from Minnesota where the weather is never the same two days in a row.
- dbcurtis 7y ago
  Ha! I came here to say something similar. Here in San Jose, the weather tomorrow will be the same as today, most of the time. My joke when I lived in Minnesota was: "If you don't like the weather, just wait 10 minutes. It will be different." Gotta say though, I don't miss my slow blower even just a wee little bit.

jfarmer 7y ago

A relevant Twitter thread: https://twitter.com/NeuroStats/status/1192679554306887681 https://twitter.com/NeuroStats/status/1192679554306887681 At the risk of projecting, this has the hallmark of bad experimental design. The best experiments are designed to determine which of many theories better account for what we observe. (When I write "you" or "your" below, I don't mean YOU specifically, but anyone designing the kind of experiment you describe.) One model of gravity says the postition/time curve of a ball dropped from a height should look like X. Another model of gravty says it should look like Y. You drop many balls, plot their position/time, and see which of the two models' curves match what you observe. The goal isn't to get the curve; the goal is to decide which model is a better picture of our universe. If the plotted curve looks kinda-sorta like X but NOTHING like Y, you've at least learned that Y is not a good model. What models/theories of customer behavior were your experiments designed to distinguish between? My guess is "none" because someone thinking about the problem scientifically would start with a single experiment whose results are maximally dispositive and go from there. They wouldn't spend a bunch of time up-front designing 12 distinct experiments. So it wasn't really an experiment in the scientific sense, but rather a kind of random optimization exercise: do 12 somewhat-less-than-random things and see which, if any, improve the metrics we care about. Random observations aren't bad, but you'd do them when you're trying to build a model, not when you're trying to determine to what extent a model corresponds with reality. For example, are there any dimensions along which the 12 variants ARE distinguishable from one another? That might point the way to learning something interesting and actionable about your customers. Did the team treat the random algorithm as the control? Well, if you believe some of your customers are engaged by novelty then maybe random is maximally novel (or at least equivalently novel), and so it's not really a control. What about negative experiments, i.e., recommendations your current model would predict have a NEGATIVE impact on your KPIs? If those experiments DON't produce a negative impact then you've learned that some combination of the following is the case: 1. The current customer model is inaccurate 2. The model is accurate but the KPIs don't measure what you believe they do (test validity) 3. The KPIs measure what you believe they do but the instrumentation is broken Some examples of NEGATIVE experiments: What if you always recommend a video that consists of nothing but 90 minutes of static? What if you always recommend the video a user just watched? What if you recommend the Nth prior video a user watched, creating a recommendation cycle? Imagine if THOSE experiments didn't impact the KPIs, either. In that universe, you'd expect the outcome you observed with your 12 ML experiments. In fact, after observering 12 distinct ML models give indistingiushable results, I'd be seriously wondering if my analytics infrastructure was broken and/or whether KPIs measured what we thought they did.

adenverd 7y ago

I worked at a larger services marketplace, helping data scientists get their models into production as A/B experiments. We had an interesting and related challenge in our search ranking algorithms: we wanted to rank order results by the predicted lifetime value of establishing a relationship between searcher and each potential service provider. In our case, a 1% increase in LTV from one of these experiments would be...big. Really big. Improving performance of these ranking models was notoriously difficult. 50% of the experiments we'd run would show no statistically significant change, or would even decrease performance. Another 40% or so would improve one funnel KPI, but decrease another, leading to no net improvement in $$. Only 10% or so of experiments would actually show a marginal improvement to cohort LTV. I'm not sure how much of this is actually "there's very little marginal value to be gained here" versus lack of rigor and a cohesive approach to modeling. The data scientists were very good at what they do, but ownership of models frequently changed hands, and documentation and reporting about what experiments had previously been tried was almost non-existent. All that to say, productizing ML/AI is very time- and resource-intensive, and it's not always clear why something did/didn't work. It also requires a lot of supporting infrastructure and a data platform that most startups would balk at the cost of.

JimmyRuska 7y ago
If you have historical data to validate against, you can set a leader board on models run against older data, and always leave part of the data out and unavailable for test https://gluebenchmark.com/leaderboard/ https://gluebenchmark.com/leaderboard/ This encourages a simple first version and incremental complexity, rather than starting very complex 6 months in, and never having an easy baseline to compare to. A simple baseline can spawn off several creative methods of improvement to research. The other case is that the models should be run against simple cases that are easy to understand and easy to confirm. This way there's always a human QA component available to make sure results are sensible.

codingslave 7y ago

I have had a similar experience, but I really do think it points more to the team than it does to the efficacy of machine learning. I was on a team of extremely intelligent people, but they were very academic, with minimal practical coding skills, refusals to use version control etc. By academic, I mean graduates from the number one university in their respective fields, top publications, etc. They produced very little actual value. Great theoretical ideas, in depth understanding of different methods of optimization, etc. The team I was on before that one was a bunch of scrappy engineers from Poland, India and the USA with no graduate degrees, but 20 years coding and distributed systems experience each. The difference in problem solving ability, the speed at which they moved, broke down problems, tried out different methods, was staggering. I think ML is suffering from a prestige problem, and many companies are suffering for it. The wrong people are being hired and promoted, with business leaders calling the shots on who runs machine learning projects without fully understanding who can actually deliver.

btilly 7y ago

This reminds me of a job interview I was on. I was asked about how I would use AI/machine learning for their problem space. Since they seemed to be smart and level-headed, I answered honestly, "Pick something unimportant, use a machine learning algorithm just to get familiar with the tools, ignore the result unless it happens to work, then put machine learning in your marketing materials. But keep track of it, and if it is useful to you in 5 years, be sure to use it for real." They said, "That's about what we concluded except that we didn't get around to actually doing that pilot project yet." I got the job. :-)

mc3 7y ago
That sounds odd, like they don't really need machine learning (unless it is to snare investors?).
- gpm 7y ago
  Snaring customers too. I swear, people are obsessed with "machine learning" even when the domain really isn't suited for it.
  - bigred100 7y ago
    I sometimes wonder if management and engineers don’t have more in common than acknowledged. Publications such as Harvard Business Review have huge coverage of things like managing AI, and being able to say you managed an “AI project” might mean something.
  - xmprt 7y ago
    I wonder if people are becoming more savvy and anti-ML after all the issues people have with Facebook and Google collecting a lot more data than people are comfortable with.
    - SuoDuanDao 7y ago
      I suspect what people dislike is big data. Anecdotally, I'd love to hear about a success where some lone genius codes a better tool using publicly available datasets. I don't like the idea of BigCorp using their larger servers to dominate the space.

deleted 7y ago

[deleted]

madrox 7y ago

I had a similar experience at one point. A team put energy into building a recommendation system, and were able to demonstrate that the "Recommended for you" content performed better than all other content editorial. After getting challenged a bit, though, turns anything performs better when put under the header "Recommended for you."

mike_hock 7y ago
> put energy put energy And they say English doesn't have reduplication!
dgellow 7y ago
Well, that’s a good lesson to learn, just put everything under “recommended for you” header! :)
jshen 7y ago
You aren’t talking about dcom are you?

deleted 7y ago

[deleted]

noodlenotes 7y ago

Recommender systems are notoriously hard because it's difficult to do better than just recommending the most popular content. You can recommend more personalized content at the expense of KPIs like click-through rate.

stefan_ 7y ago

Since we are sharing anecdotes, I can report it's been 20 years of buying stuff on the internet and the combined billions of ad tracking research dollars spent by Amazon and Google have not yet come up with a better algorithm than to bombard me with ads for the exact same thing I just bought.

propogandist 7y ago
This can happen when a retargeting campaign doesn't have a 'burn pixel' or conversion event trigger. It's a common oversight, which can cause a re-targeting program to kick-off unnecessarily (or cause ads that have followed you around to become obvious)
- idoh 7y ago
  The AI innovation must be that they can figure out which marketers are likely to forget to have a burn pixel because those marketers drive more revenue.
- scottlocklin 7y ago
  You'd think with all the "AI" out there, they could match a sales DB entry with the CRM DB entry, but, in fact, they basically can't.
- hnick 7y ago
  And yet I haven't seen one ad that has a 'not interested' feature similar to YouTube. I mean, if I could stop seeing washing machines or whatever, I'd probably click it.
  - propogandist 7y ago
    do you clear your cookies and html5 storage? That should wipe any personalization that's happening, but the ads will become very generic then. You can also block ads (Ublock/Umatrix)
    - hnick 7y ago

ArtWomb 7y ago

Personal recommendation systems all have tradeoffs. It's just the nature of curation as an intangible endeavor. You can love "Scarface", "Heat" and "LA Confidential" but still find "Casino" boring ;) More on such tradeoffs in a recent case study from DeepMind on Google Play Store app recommendations. Even they acknowledge the same techniques that surface 30% cost efficiencies in data center cooling, may not be completely applicable to "taste" https://deepmind.com/blog/article/Advanced-machine-learning-helps-Play-Store-users-discover-personalised-apps https://deepmind.com/blog/article/Advanced-machine-learning-...

vishwajeetv 7y ago

Interestingly, Steam's recommendation algorithm to show "similar to a game" by their 'learning machines' works very well. I have found really good games via that, and none of them did show up in the regular recommendations/carousel on top.

lawlessone 7y ago
pretty sure it just uses the tags players give it, strategy, sci-fi , openworld etc.

mikorym 7y ago

IMH (and biased) O, a lot of great coders are implimentors, or let's say applied computers scientists. That is great for building incredible open source software and a lot of other things that I would not be able to do given a 1000 years. However (again IMHBO) a specific ML and any other specific application of stastistics or mathematics becomes really tricky once your use case is explicitly defined. You then need intimate and deep knowledge of the tools that you are using (e.g.: Should I even use NN? Should I even use genetic algorithms? Should I even use x?) but ML for most people is shorthand for NN and its variants or maybe shorthand for something else specific rather than in principle. A well aimed shot at PCA [1] can often solve your problem. Or at least, tell you what the problem looks like. This is just an example, but IMHBO people waste their time learning ML and not learning mathematics and statistics. IMHBO I still think that self-driving cars can be solved by defining a list of 1000 or so rules, by hand, by humans, and by consensus. The computer vision part is the ML part. [1] https://en.wikipedia.org/wiki/Principal_component_analysis https://en.wikipedia.org/wiki/Principal_component_analysis

jvanderbot 7y ago
I could not agree more about self driving cars. They will disrupt and cause us to actually look at our terrible transportation infastructure, not learn to survive in it.

herval 7y ago

Doesn’t it depend on the size and variety of your dataset? I can see “random” performing well in a set of <1000 videos, all on similar subject spaces (eg “memes”, or “python”), but recommending relevant stuff gets much harder as the amount of content grows...

swiley 7y ago

I feel like A/B testing isn’t a great way to determine correctness either. IMO even in interface designing you should be arguing from first principles rather than relying on telemetry and other empirical data.

temp43t453 7y ago

excuse me for butting into dead conversation, but your team used ml to inapplicable case. (at least they aсquired experiense). up next recommendations wont work without advanced image recognition and topics gathering - basically titles/tags for most videos are garbage and clickbait, and most of youtube work by watching buzzed videos: some well known (by a big amout of watchers) "influencers" push video of some topic (thing/brand) then it get traction from other content creators - they produce videos about it and watchers tend to stick to buzzed topics. it's like news about news. if your team used ml to recommend up next on your own video hosting your result simply means your videos are equally not on topic Or non-interesting for your service auditory; or they are garbage.