3 ms·
This depends a lot on what you're doing. If you are ranking 1M qps in a recommender system, then training cost will be tiny compared to inference.
by mgraczyk 4y ago
This depends a lot on what you're doing. If you are ranking 1M qps in a recommender system, then training cost will be tiny compared to inference.
- lajamerr 4y agoI wonder if there's room for model caching. Sacrifice some personalization for near similar results so you aren't hitting the model so often.
- mgraczyk 4y agoYeah we did lots of things like this at Instagram. Can be very brittle and dangerous though to share any caching amongst multiple users. If you work at Facebook you can search for some SEVs related to this lol