2 ms·
I wouldn't put the model under fastapi or any similar framework, i would serve it from a different process to also allow me to serve multiple versions of the mo
by higeorge13 4y ago
I wouldn't put the model under fastapi or any similar framework, i would serve it from a different process to also allow me to serve multiple versions of the model as well (similar to tf serving). But eventually we have an API call to some web framework calling this different process and requiring a response with the model recommendations to be returned in a few milliseconds, how is a message queue appropriate for such a real-time use case, could you elaborate?
- kroolik 4y agoWould you describe getting response from the model within few milliseconds cpu or memory intensive? What I assumed as to the characteristics of inference is a process that takes multiple seconds to minutes. In a system that requires a response to the customer within a few milliseconds through web api, how do you ensure the performance in Python? I'm genuinely interested as that sounds outside of what stock CPython can do except trivial logic. Another aspect that's completely ignored are the requirement - you might need the response time in milliseconds, couple of times per day. Others might need to serve hundreds of requests. You also need to consider the budget available.
- hetspookjee 4y agoJust to add some ballparks: I have a yolov5 model that I run in PyTorch on a 3080 and the inference time is roughly 20ms. And inference with a 200estimator random forest is a few ms on a cpu. I can imagine that in some scenarios it’s not enough though.
- kroolik 4y agoIndeed it might not be enough, and there is no one size that fits them all ;) For inference times of 50ms I wouldn't bother complicating the setup that much. Again, it all depends on your requirement and trade offs available.