3 ms·
It looks like executorch is for edge devices (phones / IoT / etc). I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems li
by mlmandude 3y ago
It looks like executorch is for edge devices (phones / IoT / etc).
I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/executorch/tree/main/backends https://github.com/pytorch/executorch/tree/main/backends , but seems like it might be possible. Is this something you are thinking about?
- iseeyuan 3y agoYes ExecuTorch is currently targeted at Edge devices. The runtime is written in C++ with 50KB binary size (without kernels) and should run in most of platforms. You are right that we have not integrated to Nvidia backend yet. Have you tried torch.compile() in PyTorch 2.0? It would do the Nvidia optimization for you without Torchscript. If you have specific binary size or edge specific request, feel free to file issues in https://github.com/pytorch/executorch/issues https://github.com/pytorch/executorch/issues
- fooblaster 3y agotorch.compile only works with python from what I understand. Many people need a native way to run GPU models, but don't want the bloat of full libtorch.