3 ms·
The best algorithms in the world won't make up for a lack of data. Few companies are in a position to make good recommendation engines. I also don't think math
by bad_user 16y ago
The best algorithms in the world won't make up for a lack of data. Few companies are in a position to make good recommendation engines.
I also don't think math is to be blamed -- human psychology plays an important role -- i.e. I may be interested in software-dev stuff or in startups stuff, but not every hour of every day. Basically I'm inclined to read such stuff only in the morning, while drinking my coffee (the perfect time of day for big plans).
But in the afternoon I prefer reading articles about my OTHER (lighter) interests / hobbies, like photography. And I also prefer getting stuff solved, like finding an electrician to fix the poor wiring in my house or searching for gifts for my wife/child.
For any person on this earth, immediate interest in something is relative, based on time of day, time of year, mood, current needs, current problems, current hobbies, etc...
A good recommendation engine is practically unfeasible -- because if it cannot anticipate your future desires, both short-term and long-term, then it's useless; because people "search" for stuff as soon as they realize they have a new need/desire, and then it is too late to make a recommendation (at this time, the user's history is only useful to "understand" / parse the search query).
Recommendations need to happen before the user searches for stuff.
For example -- I searched for books on drawing / colors on Amazon, and bought 2. Amazon should have anticipated that I may like getting into photography, as they do know a lot about me, but it didn't, even though Amazon's recommendations are some of the best. Now that I already bought stuff, I don't care about special offers on the same kind of stuff that I already bought.
Also, people with "hackish" attitudes have done pretty amazing stuff btw, contributions to AI included -- software isn't just math, it's also biology / chemistry / physics / psychology / philosophy, amongst others.
- assiotis 16y agoThere is no reason why time of the day, temperature outside, whether Obama made a joke or not, etc can't all be features for a classifier to learn. I love the idea of an algorithm anticipating my next move and I'd argue that for the vast majority of people you can find repeating patterns that are almost always relevant. These patterns will not be simple or obvious and as you currently pointed out, the lack of data is what is holding most innovation behind. I feel that companies that own the data like Amazon will have a lot to gain if they do a Netflix-style competition to improve their recommendations engine.
- crasshopper 16y agoassiotis, have you tried giving a bazillion data points to an SVM? One needs a sensible kernel as well.
- jplewicke 16y agoI agree completely if we're talking about a recommendation engine that's supposed to always be able to find something of immediate interest for you. However, that problem is at least as hard as building a general AI, since not even my friends can always think of something I would find immediately interesting. If you relax the scope of what you want a recommendation engine to do, they can do an awful lot. Directed Edge is absolutely killing it here -- http://directededge.com/ http://directededge.com/ . By providing a white-label API so that developers can specify the data on which to base recommendations and by using some superior algorithms, they're able to come up with impressive results for recommendations like "People who might like to buy this might also like to buy this", or "These users might be your friends", or "You should read this other article."
- crasshopper 16y agobad_user, how is what you mentioned beyond mathematical modeling? Something else you may have neglected in this response: the software doesn't have to just make a recommendation. It can present you an array of options which you filter. A dynamical interface.