3 ms·

Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)

whobair 3y ago
hey - this might be a rather specific use case, but our applications need to run "real-time" and allocation-free - are there any guarantees on that from your side? assuming fixed shapes, pre-allocating everything for the forward pass should probably be possible in theory, but i guess that wasn't really a relevant factor in the design of it all
mlmandude 3y ago
It looks like executorch is for edge devices (phones / IoT / etc). I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/executorch/tree/main/backends https://github.com/pytorch/executorch/tree/main/backends , but seems like it might be possible. Is this something you are thinking about?
- iseeyuan 3y ago
  Yes ExecuTorch is currently targeted at Edge devices. The runtime is written in C++ with 50KB binary size (without kernels) and should run in most of platforms. You are right that we have not integrated to Nvidia backend yet. Have you tried torch.compile() in PyTorch 2.0? It would do the Nvidia optimization for you without Torchscript. If you have specific binary size or edge specific request, feel free to file issues in https://github.com/pytorch/executorch/issues https://github.com/pytorch/executorch/issues
  - fooblaster 3y ago
    torch.compile only works with python from what I understand. Many people need a native way to run GPU models, but don't want the bloat of full libtorch.
nothrowaways 3y ago
Nice tool with ugly name
- BudaDude 3y ago
  It's one of those names that sound better said aloud than spelled out. Excited to see where this goes though!
modeless 3y ago
Looks cool! Does the Vulkan backend work on PC? MLC-LLM proves it can work well and it would be cool to have a cross platform, minimal runtime, GPU-agnostic backend for PC too, not just mobile.
- ssjia 3y ago
  The Vulkan backend does work on PC, as the only requirement is that Vulkan drivers are present. However, it was developed with mobile use-cases in mind and we haven't validated/optimized performance for PC. As an aside, the Vulkan backend is tied to TorchScript at the moment, so it is not yet compatible with ExecuTorch. However, we are also planning to introduce a Vulkan delegate for ExecuTorch which will enable GPU delegation through ExecuTorch.
p3zz1 3y ago
[dead]
suyash 3y ago
Is it possible to execute a light weight language model, perhaps this https://github.com/facebookresearch/llama https://github.com/facebookresearch/llama using ExecuTorch to run on smartphone in real time for a chatbot app ? Please share some guidance.
alex_hirner 3y ago
We like Rust for its build tooling and safe concurrency primitives. Thus we are eyeballing with candle [1]. OTOH, writing platform specific backends is a huge undertaking. Do you think backends such as MPS may become a shared effort? [1] https://github.com/huggingface/candle https://github.com/huggingface/candle [2] https://github.com/huggingface/candle/issues/313 https://github.com/huggingface/candle/issues/313

hey - this might be a rather specific use case, but our applications need to run "real-time" and allocation-free - are there any guarantees on that from your side? assuming fixed shapes, pre-allocating everything for the forward pass should probably be possible in theory, but i guess that wasn't really a relevant factor in the design of it all

mlmandude 3y ago

It looks like executorch is for edge devices (phones / IoT / etc). I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/executorch/tree/main/backends https://github.com/pytorch/executorch/tree/main/backends , but seems like it might be possible. Is this something you are thinking about?

iseeyuan 3y ago
Yes ExecuTorch is currently targeted at Edge devices. The runtime is written in C++ with 50KB binary size (without kernels) and should run in most of platforms. You are right that we have not integrated to Nvidia backend yet. Have you tried torch.compile() in PyTorch 2.0? It would do the Nvidia optimization for you without Torchscript. If you have specific binary size or edge specific request, feel free to file issues in https://github.com/pytorch/executorch/issues https://github.com/pytorch/executorch/issues
- fooblaster 3y ago
  torch.compile only works with python from what I understand. Many people need a native way to run GPU models, but don't want the bloat of full libtorch.

nothrowaways 3y ago

Nice tool with ugly name

BudaDude 3y ago
It's one of those names that sound better said aloud than spelled out. Excited to see where this goes though!

modeless 3y ago

Looks cool! Does the Vulkan backend work on PC? MLC-LLM proves it can work well and it would be cool to have a cross platform, minimal runtime, GPU-agnostic backend for PC too, not just mobile.

ssjia 3y ago
The Vulkan backend does work on PC, as the only requirement is that Vulkan drivers are present. However, it was developed with mobile use-cases in mind and we haven't validated/optimized performance for PC. As an aside, the Vulkan backend is tied to TorchScript at the moment, so it is not yet compatible with ExecuTorch. However, we are also planning to introduce a Vulkan delegate for ExecuTorch which will enable GPU delegation through ExecuTorch.

p3zz1 3y ago

[dead]

suyash 3y ago

Is it possible to execute a light weight language model, perhaps this https://github.com/facebookresearch/llama https://github.com/facebookresearch/llama using ExecuTorch to run on smartphone in real time for a chatbot app ? Please share some guidance.

alex_hirner 3y ago

We like Rust for its build tooling and safe concurrency primitives. Thus we are eyeballing with candle [1]. OTOH, writing platform specific backends is a huge undertaking. Do you think backends such as MPS may become a shared effort? [1] https://github.com/huggingface/candle https://github.com/huggingface/candle [2] https://github.com/huggingface/candle/issues/313 https://github.com/huggingface/candle/issues/313