Premises: - the offline model data has a non-sequential world model - the offline model is a great starting point but slow to react to realtime changes in use

Premises: - the offline model data has a non-sequential world model - the offline model is a great starting point but slow to react to realtime changes in user preferences - the in-streaming of new data points makes the reward prediction inherently sequential - hence you could use a cheap fast online model for fast reward prediction updates, that derives from a good starting point The big question is: what do you do to adapt to just one new data point? You're trying to learn theta' := train(old data ++ new data) ≈ train(old data) ++ dTrain(new data) ≈ theta_offline ++ dTheta(new data) If you change the offline model: well it's a big base model and there are always M users with N new data points coming in. Rerouting all training into the same common model theta, dynamically and live, in response to vast amounts of incoming data, while balancing batching-latency overheads or catastrophic forgetting or reversion to the mean preference, would be tricky. If you change the online model: the delta per-user can be cheap to track individually with essentially increment-only/append-only per-user data structures. You can get away with approximating user_theta' ≈ offline_theta + delta(train(new data)) ≈ offline_theta + online_train(new_data). This can be easily parallelized and personalized and zero latency. It's a good achievement.