4 ms·
Euh, What ? Your model serving code (likely a python installation and a few libs to answer HTTP requests, like FastAPI, Pydantic, etc...) lives in your docker
by Fiahil 5y ago
Euh, What ?
Your model serving code (likely a python installation and a few libs to answer HTTP requests, like FastAPI, Pydantic, etc...) lives in your docker image, and their installation are documented in the Dockerfile. This is likely ~10-20 lines and about 500MB for a "standard" python installation.
Then, on container startup, your model-serving-app collects the latest model from an external artifact repository. This step usually belong in an init container if you use Kubernetes.
Finally, the app starts and begin serving requests with its in-memory model.
The size of the running container scales linearly with the model size, and the image size stays "small" (for a python installation). You also gain fast auto-update mechanism where your app just need a container restart to fetch the latest model.
This works on laptops and raspberry pis.
- ellisv 5y ago> Then, on container startup, your model-serving-app collects the latest model from an external artifact repository. This step usually belong in an init container if you use Kubernetes. I agree but there are some reasons why folks choose to bake the model into the container image rather than fetching. E.g. time waiting for the model to transfer, artifact repository may be down, etc.
- Fiahil 5y agoIt's mostly because it's easier. No additional token necessary, No config, No Service discovery. For small models (less than 100MB), it's generally not an issue to embed them directly in the image. However, if you're getting GB-size images with less than half of it related to the serving code and its library, then it's time to upgrade.
- jerpint 5y agoYes, this is a generic workflow, but when you start adding in e.g. torch[vision, audio], scipy, or any other "useful" processing library, things get out of hand pretty quickly and you need to start being more intelligent about what is really necessary to have inside the container. So for a model that is even a few mb, you have a surrounding container of a couple of GBs
- Fiahil 5y agoWhy would you need _any_ processing library in your runtime image ? You're not training anything, your model is set in stone.
- jerpint 5y agoYou need to run a forward pass somehow through a model, so at the very least you need something to define your operations + load your weights. You also need whatever libraries for any kind of signal processing you might do before to your inputs