3 ms·

>>Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed

>>Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed custom linear prediction engine, favouring models with fewer parameters, batching requests, and pre-computation and/or caching. Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bring...or how you can really cache a prediction request

Falcorian 7y ago
Here is an overly simple example: Pre-compute the recommended hotels for my top users every night. Now when that user comes back, they see a slightly stale recommendation, but it's lightning fast. You can also pre-compute and cache some of the inputs cheer model, like maybe a vector representation of the description of a hotel.
ovi256 7y ago
Batching just uses hardware capabilities efficiently. Either vector instructions in CPUs or batch capabilities in GPUs. For the same hardware load, you can process several samples instead of just one. Pre-computation means running your model on samples in advance, before the model result is needed, so it's ready to use instantly when needed. Caching works probably because there are model results that are reused again and again, so it makes sense to cache them. For example, there are deep models that process room pictures, room and customer characteristics. Only customer characteristics change between customers, so it makes sense to cache the features output by the deep CNN that processes the room pictures. Once you start doing prediction at scale, there are lots of these optimizations to pick up.
chudi 7y ago
In my company we use this approach, instead of infer online, we just run all our models overnight and save the results on a database that we serve trough an API, that gives you constant time on the latency, its a shotgun approach as much of the recommendations aren't served (specially if are user facing and not item facing ones) but works really well. All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.

Falcorian 7y ago

Here is an overly simple example: Pre-compute the recommended hotels for my top users every night. Now when that user comes back, they see a slightly stale recommendation, but it's lightning fast. You can also pre-compute and cache some of the inputs cheer model, like maybe a vector representation of the description of a hotel.

ovi256 7y ago

Batching just uses hardware capabilities efficiently. Either vector instructions in CPUs or batch capabilities in GPUs. For the same hardware load, you can process several samples instead of just one. Pre-computation means running your model on samples in advance, before the model result is needed, so it's ready to use instantly when needed. Caching works probably because there are model results that are reused again and again, so it makes sense to cache them. For example, there are deep models that process room pictures, room and customer characteristics. Only customer characteristics change between customers, so it makes sense to cache the features output by the deep CNN that processes the room pictures. Once you start doing prediction at scale, there are lots of these optimizations to pick up.

chudi 7y ago

In my company we use this approach, instead of infer online, we just run all our models overnight and save the results on a database that we serve trough an API, that gives you constant time on the latency, its a shotgun approach as much of the recommendations aren't served (specially if are user facing and not item facing ones) but works really well. All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.