4 ms·
Pytorch has multiple production paths for deployment models without the python interpreter already, like torchscript and libtorch, or more annoying paths like o
by fooblaster 3y ago
Pytorch has multiple production paths for deployment models without the python interpreter already, like torchscript and libtorch, or more annoying paths like onnx export and onnx runtime. So there is no need to have rust to solve this problem.
- malcolmgreaves 3y agoThose are great technologies for making the model inference independent of Python. However, there's always a good deal of pre and post processing, or other business logic, that needs to go around model inference. This sort of thing needs to be done in a general programming language. It's in these situations where Python is often used (because the supporting model code is often written by the same person, and that code is probably going to be Python because your model training and eval code is most likely going to be Python too). This is where having a non-Python PL, such as Rust, can make a huge difference for both ease of production deployment/maintenance as well as having truely efficient production inference.
- fooblaster 3y agoI've found that there is usually very little code to be shared within training and inference when most of the pre and post processing is pulled into the model itself with custom operations. Relatively few people are aware that you can write training code in C++ today with torch to share a common language with inference and training, but very few take advantage of this in practice.