4 ms·
Calling the ML model should be done via Django Channels or a similar background queue and not directly within the view (request handler) as if the inputs/infere
by ampdepolymerase 6y ago
Calling the ML model should be done via Django Channels or a similar background queue and not directly within the view (request handler) as if the inputs/inference is too big, it will block and cause issues for load handling.
- tecoholic 6y agoI agree. The ML Algorithms and models can run into huge sizes. I have one with half a gig of serialised model, when running takes up close to 3.5 Gigs. This approach of directly using it in the views would pose scaling issues.
- pplonski86 6y agoIt really depends. For sure there are models which should be processed in the background. It is a very basic tutorial that shows how simple ML models can be used as REST API. In the tutorial I was using Random Forest trained on UCI Adult Income data set. The final model was small, and computing predictions was fast. I would love to add more advanced part of the tutorial with background processing for large models.
- lmeyerov 6y agoInference should be fast, so an async view (which we're excited finally happened!) should be fine. It's similar to waiting on a db. Read consistency means using similar memory. Likewise, you want it to be warm, not reloaded between calls. In our case, we do on-the-fly GPU calls, with weird spiking behavior, so been looking for a good serverless GPU solution for this kind of stuff.