4 ms·

Why couldn't it be ported to production?

textminer 14y ago
If I recall, BellKor made many, many models based on Gradient Boosted Decision Trees, Restricted Boltzmann Machines, and kNN. They tried many different feature subsets, added temporal weighting, and tried many reduced-dimensionality representations (SVD, NMF). They then stacked them all together into one final ensemble whose RSME beat everyone else's on a hidden validation set. In a production environment, this is probably an insane amount of transformation, feature extraction, and classification for marginally little gains in precision (as defined here). But I'm only a year or two in to building production-environment classifiers, and nothing at Netflix's scale (though not tiny either-- it is a problem if I can't do feature extraction and high-precision/-recall* classification within a few milliseconds). * - mid 90s, for a hard NLP/social graph problem.
- willis77 14y ago
  It's funny to me that people are so quick to poo-poo the complicated modeling done for the Netflix prize. When did production-worthiness become the only important thing? It's like saying Watson was useless because it can't play a concurrent game of Jeopardy with thousands of people on the web. Just like in research, it turns out that relaxing real-world constraints on a problem is often a great way to make progress. I would not have to search long or far to provide much worse uses of million-dollar grants/projects/big-data-software.
  - textminer 14y ago
    Don't mean to poo-poo. I love scrambling for models and methods. In the latest problem I worked on, I did the same thing. Implementing papers, trying to avail myself of semi-supervised or reduced-dimension representations, tweaking models and features every which way... it's illuminating work, and good things come out of it. But, in the end, companies like Netflix or where I work are immediately looking for cheap ways to make X happen easier, better, and cheaper. But then hopefully the smart papers go on a shelf or are easily Googleable, and the rest of us get to learn from their efforts. I can fit long-term and short-term goals in my brain, too, mister.
rm999 14y ago
Largely what textminer said. In fact they didn't use anything from the grand-prize winner because it would require too much engineering effort and because their concentration on the problem shifted. They did however use the two best methods from the "progress prize", so I wouldn't call the contest a failure. >If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then. From http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html http://techblog.netflix.com/2012/04/netflix-recommendations-...
__mtb__ 14y ago
I am a 'Kaggler', I have done well (prize winner) in some past competitions and I could not agree more. In the last week of a competition, you go into kitchen sink mode trying anything and everything to squeeze the last few points out of your models. The objective of the competition from the competitors standpoint is to win, not create a solution that could be deployed into the sponsor's production environment. On going maintenance of the model is not a consideration. As far as the netflix competition goes though, the final solutions did help publicize the potential of ensembling and RBMs - I am grateful for that. My personal approach to the competitions is to use them as an opportunity to try out new modeling techniques (i.e. RMBs, deep beliefs etc...) on real world data. It has the added bonus of potentially paying off with prize money (as long as you don't get screwed[1]). I would like to hear from past Kaggle sponsors and see what they have done with the winning models. [1] http://www.gequest.com/c/flight/forums/t/4284/acknowledging-two-more-great-competitors http://www.gequest.com/c/flight/forums/t/4284/acknowledging-...

If I recall, BellKor made many, many models based on Gradient Boosted Decision Trees, Restricted Boltzmann Machines, and kNN. They tried many different feature subsets, added temporal weighting, and tried many reduced-dimensionality representations (SVD, NMF). They then stacked them all together into one final ensemble whose RSME beat everyone else's on a hidden validation set. In a production environment, this is probably an insane amount of transformation, feature extraction, and classification for marginally little gains in precision (as defined here). But I'm only a year or two in to building production-environment classifiers, and nothing at Netflix's scale (though not tiny either-- it is a problem if I can't do feature extraction and high-precision/-recall* classification within a few milliseconds). * - mid 90s, for a hard NLP/social graph problem.

willis77 14y ago
It's funny to me that people are so quick to poo-poo the complicated modeling done for the Netflix prize. When did production-worthiness become the only important thing? It's like saying Watson was useless because it can't play a concurrent game of Jeopardy with thousands of people on the web. Just like in research, it turns out that relaxing real-world constraints on a problem is often a great way to make progress. I would not have to search long or far to provide much worse uses of million-dollar grants/projects/big-data-software.
- textminer 14y ago
  Don't mean to poo-poo. I love scrambling for models and methods. In the latest problem I worked on, I did the same thing. Implementing papers, trying to avail myself of semi-supervised or reduced-dimension representations, tweaking models and features every which way... it's illuminating work, and good things come out of it. But, in the end, companies like Netflix or where I work are immediately looking for cheap ways to make X happen easier, better, and cheaper. But then hopefully the smart papers go on a shelf or are easily Googleable, and the rest of us get to learn from their efforts. I can fit long-term and short-term goals in my brain, too, mister.

rm999 14y ago

Largely what textminer said. In fact they didn't use anything from the grand-prize winner because it would require too much engineering effort and because their concentration on the problem shifted. They did however use the two best methods from the "progress prize", so I wouldn't call the contest a failure. >If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then. From http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html http://techblog.netflix.com/2012/04/netflix-recommendations-...

__mtb__ 14y ago

I am a 'Kaggler', I have done well (prize winner) in some past competitions and I could not agree more. In the last week of a competition, you go into kitchen sink mode trying anything and everything to squeeze the last few points out of your models. The objective of the competition from the competitors standpoint is to win, not create a solution that could be deployed into the sponsor's production environment. On going maintenance of the model is not a consideration. As far as the netflix competition goes though, the final solutions did help publicize the potential of ensembling and RBMs - I am grateful for that. My personal approach to the competitions is to use them as an opportunity to try out new modeling techniques (i.e. RMBs, deep beliefs etc...) on real world data. It has the added bonus of potentially paying off with prize money (as long as you don't get screwed[1]). I would like to hear from past Kaggle sponsors and see what they have done with the winning models. [1] http://www.gequest.com/c/flight/forums/t/4284/acknowledging-two-more-great-competitors http://www.gequest.com/c/flight/forums/t/4284/acknowledging-...