5 ms·
What about the A/B testing? What do you use for A/B strategy. How many predictions are being served by the model per second?
by superkitty 7y ago
What about the A/B testing? What do you use for A/B strategy. How many predictions are being served by the model per second?
- FridgeSeal 7y agoFor most of the stuff we’ve deployed, we’re not yet operating at a scale/level of interest where A/B rearing is worth it. Additionally, the purposes we’re using most of these models for don’t really necessitate A/B testing. When we do need A/B testing, we’ll probably use something like Seldon. As for predictions/second, not very much at the moment: 1 per 30 seconds maybe? It’s not deployed into a Kubernetes cluster because of scaling requirements, it’s because that’s where all our other services greet deployed till, and it’s more beneficial (ops and cost wise) to also deploy into there than it is to bother with having a separate workflow for deploying to lambda’s or SageMaker.
- streetcat1 7y agoSo how do you know if a new version of a model is better than the existing serving version?
- FridgeSeal 7y agoAs currently the only person doing data science things for the team, I’ll test to make sure changes I make to model/feature engineering/etc result in a better model. We’re not constantly, constantly retraining our models, because our incoming data and behaves the same. We’ve had the same model in prod for 4 months now; we don’t have any pressing issues with its predictions, and looking through the logs of what the input was the the output, it’s still performing as expected, so we’ll probably leave it longer.
- streetcat1 7y agoI see, so how do you measure the difference between the incoming data and your training data? Also, it looks like you have a very low volume of predictions?