4 ms·

Maybe I’m misunderstanding something fundamental, but I’m struggling to see the reason for this work. They have a (presumably) supervised model that estimates t

Maybe I’m misunderstanding something fundamental, but I’m struggling to see the reason for this work. They have a (presumably) supervised model that estimates the likelihood that a user will like a song. Then they train an RL model using the supervised model as an oracle. They note that the optimal result for the RL model is just making the same prediction as the supervised model. What are we doing here folks? Training models to match other models? Why? That’s good enough to get into KDD?

pizza 3y ago
Premises: - the offline model data has a non-sequential world model - the offline model is a great starting point but slow to react to realtime changes in user preferences - the in-streaming of new data points makes the reward prediction inherently sequential - hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data) If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky. If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency. It's a good achievement.

pizza 3y ago

Premises: - the offline model data has a non-sequential world model - the offline model is a great starting point but slow to react to realtime changes in user preferences - the in-streaming of new data points makes the reward prediction inherently sequential - hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data) If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky. If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency. It's a good achievement.