8 ms·

Directed Edge is pretty cool. So far, though, I think we've been hearing a lot about their graph database, and not much about how they're actually going to do

Directed Edge is pretty cool. So far, though, I think we've been hearing a lot about their graph database, and not much about how they're actually going to do recommendations. Personally, I think the latter is the harder problem.

wheels 17y ago
To be honest, that's because I'm comfortable dissecting our graph-database publicly, but less so for our ranking algorithms. There's been far more work put into the recommendations engine than the graph DB and the beginnings of the engine predate the DB. The original prototypes worked with our Store class, which is the DB abstraction that the engine sees, and were purely in-memory and then there were a number of things that I swapped in to replace the storage layer, eventually settling on our own DB. That said, I do have one big blog post half-written on "the problems of sparse data" that I'll eventually roll out.
- caffeine 17y ago
  That would be very interesting (to me, at least). In particular, I'd be curious to know how you test your inferences. I guess you could run it on the NetFlix/GitHub prizes ... but in general, I'm finding that getting reliable data on which to validate algos is one of the biggest challenges. In any case, I'd be interested to hear about the math, since there are about 50 different ways of doing this stuff. Even just vague stuff like "We might use a boltzmann machine". There's some interesting work done on optimal stimulus selection (MacKay has a paper on it, and there's one in a neuroscience setting by Paninski). The idea is that you generate data for which the response will give you maximum information. So you figure out which edges would be most valuable to your algo's predictions (using information theory) and then make recommendations based on hypotheses about those edges, which are later confirmed/denied by user behavior. This gives you an optimal learning loop.

wheels 17y ago

To be honest, that's because I'm comfortable dissecting our graph-database publicly, but less so for our ranking algorithms. There's been far more work put into the recommendations engine than the graph DB and the beginnings of the engine predate the DB. The original prototypes worked with our Store class, which is the DB abstraction that the engine sees, and were purely in-memory and then there were a number of things that I swapped in to replace the storage layer, eventually settling on our own DB. That said, I do have one big blog post half-written on "the problems of sparse data" that I'll eventually roll out.

caffeine 17y ago
That would be very interesting (to me, at least). In particular, I'd be curious to know how you test your inferences. I guess you could run it on the NetFlix/GitHub prizes ... but in general, I'm finding that getting reliable data on which to validate algos is one of the biggest challenges. In any case, I'd be interested to hear about the math, since there are about 50 different ways of doing this stuff. Even just vague stuff like "We might use a boltzmann machine". There's some interesting work done on optimal stimulus selection (MacKay has a paper on it, and there's one in a neuroscience setting by Paninski). The idea is that you generate data for which the response will give you maximum information. So you figure out which edges would be most valuable to your algo's predictions (using information theory) and then make recommendations based on hypotheses about those edges, which are later confirmed/denied by user behavior. This gives you an optimal learning loop.