4 ms·

What a simple idea that will go a long way. There are great many algorithms/models out there that are seemingly great and do wonders on toy data, but in the rea

What a simple idea that will go a long way. There are great many algorithms/models out there that are seemingly great and do wonders on toy data, but in the real world fail not only because the data isn't comprehensive (it isn't sampled well), but also because people misuse existing models and tweak them enough to work for that data and those tweaks make it on acclaimed journals/conferences. These kinds of efforts will help tremendously to filter out all the noise because now, you'll have to show results for real-world data to argue that your algorithms can generalize and/or that your tweaks actually matter. Besides Amazon, Google has taught us that more data is better and sometimes less complex algorithms that scale but give you only a marginal hit on accuracy are preferred; ie, cost functions matter, and that's something you don't get practice with in the academic world. When you think about data sets to boost performance, think outside the directly dependent ones. There is so much data out there now and it's not being used. This I think will change soon.

cdavid 16y ago
That's indeed a good idea. It does not solve one real issue with recommendation, though: the objective measure is whether people buy more stuff, which means running experiments not only on real data but in the real business environment.
- izendejas 16y ago
  Right, the cost functions could be defined in terms profit maximization (thus, the cost being lost sales, eg). Either way, these experiments are very difficult to control because you can't fully validate models by predicting sales a posteriori with holdouts once you have the data as you don't know if some algorithm might influence certain sales not reflected in such real-life data. You could try a/b testing, but I'm not sure the Amazons are willing to run someone's models very readily. Either way, this is a good thing.

cdavid 16y ago

That's indeed a good idea. It does not solve one real issue with recommendation, though: the objective measure is whether people buy more stuff, which means running experiments not only on real data but in the real business environment.

izendejas 16y ago
Right, the cost functions could be defined in terms profit maximization (thus, the cost being lost sales, eg). Either way, these experiments are very difficult to control because you can't fully validate models by predicting sales a posteriori with holdouts once you have the data as you don't know if some algorithm might influence certain sales not reflected in such real-life data. You could try a/b testing, but I'm not sure the Amazons are willing to run someone's models very readily. Either way, this is a good thing.