3 ms·
Those are great technologies for making the model inference independent of Python. However, there's always a good deal of pre and post processing, or other busi
by malcolmgreaves 3y ago
Those are great technologies for making the model inference independent of Python. However, there's always a good deal of pre and post processing, or other business logic, that needs to go around model inference. This sort of thing needs to be done in a general programming language. It's in these situations where Python is often used (because the supporting model code is often written by the same person, and that code is probably going to be Python because your model training and eval code is most likely going to be Python too).
This is where having a non-Python PL, such as Rust, can make a huge difference for both ease of production deployment/maintenance as well as having truely efficient production inference.
- fooblaster 3y agoI've found that there is usually very little code to be shared within training and inference when most of the pre and post processing is pulled into the model itself with custom operations. Relatively few people are aware that you can write training code in C++ today with torch to share a common language with inference and training, but very few take advantage of this in practice.