4 ms·
Starting with a trained PyTorch model, it builds optimized C++ binaries for running inference (not training) on NVidia and AMD GPUs. Various optimizations ment
by skirmish 3y ago
Starting with a trained PyTorch model, it builds optimized C++ binaries for running inference (not training) on NVidia and AMD GPUs. Various optimizations mentioned a lot, so presumably models run faster than with just running them via regular PyTorch.
- stevenwliao 3y agoHow much faster is it?
- pumanoir 3y agoDepends on the model and GPU. Here is an example of almost 2x on a 3060 for StableDiffusion: https://www.youtube.com/watch?v=_6BsUijOWoM https://www.youtube.com/watch?v=_6BsUijOWoM
- prsutherland 3y agoI'm curious why that is called "rendering" rather than "compiling". Is the code boiler plate and just a change in the NN's representation?