4 ms·
I would do it in python using one of the fast, modern ASGI servers like uvicorn. Zero downtime model updates can be done using a redis cache to persist models.
by sandGorgon 7y ago
I would do it in python using one of the fast, modern ASGI servers like uvicorn.
Zero downtime model updates can be done using a redis cache to persist models.
In any case, that's a solved problem using haproxy and kubernetes.
Not sure why go has these advantages
- hnaccy 7y agoIf you do model inference in web server process it will be compute bound and lock up the web server, is there a preferred/clean way to req/rec or similar pass the jobs to second process and allow web server process to non-blocking wait for response?
- woeirua 7y agoYes, use a dedicated job server/process like Celery.