6 ms·

ExecuTorch: Run PyTorch programs on mobile and edge devices

94 points by resolve-ev 3y ago

vladimirzaytsev 3y ago
This is very cool.
ge96 3y ago
interference or inference?
- cs702 3y ago
  Obviously the poster meant to write inference. Maybe autocorrect messed up the headline? It wouldn't be the first time autocorrect has messed up a headline on HN. EDIT: dang fixed the headline. Thanks dang!
- dang 3y ago
  We've changed the title now to use language from the article itself. (submitted title was "ExecuTorch: Enabling On-Device interference for embedded devices") Edit: I switched "anywhere" to "mobile and edge devices" per https://pytorch.org/blog/pytorch-edge/ https://pytorch.org/blog/pytorch-edge/. One does not get away with saying things like "anywhere" in HN titles...
flakiness 3y ago
There is also PyTorch mobile [1]. I'm curious how these two relate. [Edit] And here is the answer [2] > PyTorch Mobile uses TorchScript to allow PyTorch models to run on devices with limited resources. ExecuTorch has a significantly smaller memory size and a dynamic memory footprint resulting in superior performance and portability compared to PyTorch Mobile. Also, ExecuTorch does not rely on TorchScript, and instead leverages PyTorch 2 compiler and export functionality for on-device execution of PyTorch models. [1] https://pytorch.org/mobile/home/ https://pytorch.org/mobile/home/ [2] https://pytorch.org/executorch/stable/intro-overview.html#how-is-executorch-different-from-pytorch-mobile-lite-interpreter https://pytorch.org/executorch/stable/intro-overview.html#ho...
- phh 3y ago
  Yeah it's a bit weird to see the difference. > Also, ExecuTorch does not rely on TorchScript, and instead leverages PyTorch 2 compiler and export functionality for on-device execution of PyTorch models. I think this is the major change: AFAIU you could consider ExecuTorch as ""just a rewrite"" of PyTorch mobile using PyTorch 2 compiler. BUT this is should still be pretty major. It was pretty hard to retarget models to be compatible with TorchScript. But most recent models will want to target PyTorch 2 compiler for the performance boost. Which means they'll automatically get ExecuTorch support. As someone who got burned several times in trying to port models to TorchScript and run them on Androids, I'm happy they did that.
liuliu 3y ago
Very cool. A few notes: 1. MPS backend uses MPSGraph exclusively, might hit some performance ceilings limited by MPSGraph. s4nnc moved more and more ops from MPSGraph to Metal directly to have better control on both allocation and some erratic behaviors from MPSGraph. 2. CoreML backend uses coremltools, thus it carries all the baggage of that: requiring to generate CoreML model AOT because Python dependency, have no control over memory planning, weight quantization scheme, or where to put the weights. Dynamic shape might further making memory planning worse as static shape is the main use-case of CoreML so far and far better tested. I would love to see an updated port of coremltools either in Swift or C++ (ONNX's coremltools implementation is in v3 I believe, and coremltools moved to v7 spec?).
ai-joe 3y ago
ONNX Runtime has supported this for years - https://onnxruntime.ai/blogs/pytorch-on-the-edge https://onnxruntime.ai/blogs/pytorch-on-the-edge
- iseeyuan 3y ago
  One difference here is that there is no conversion for ExecuTorch. In general conversions introduce inconsistency and are difficult to maintain, so ExecuTorch eliminates that step.
shoumikhin 3y ago
Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)
- whobair 3y ago
  hey - this might be a rather specific use case, but our applications need to run "real-time" and allocation-free - are there any guarantees on that from your side? assuming fixed shapes, pre-allocating everything for the forward pass should probably be possible in theory, but i guess that wasn't really a relevant factor in the design of it all
- mlmandude 3y ago
  It looks like executorch is for edge devices (phones / IoT / etc). I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/executorch/tree/main/backends https://github.com/pytorch/executorch/tree/main/backends , but seems like it might be possible. Is this something you are thinking about?
  - iseeyuan 3y ago
    Yes ExecuTorch is currently targeted at Edge devices. The runtime is written in C++ with 50KB binary size (without kernels) and should run in most of platforms. You are right that we have not integrated to Nvidia backend yet. Have you tried torch.compile() in PyTorch 2.0? It would do the Nvidia optimization for you without Torchscript. If you have specific binary size or edge specific request, feel free to file issues in https://github.com/pytorch/executorch/issues https://github.com/pytorch/executorch/issues
    - fooblaster 3y ago
      torch.compile only works with python from what I understand. Many people need a native way to run GPU models, but don't want the bloat of full libtorch.
brrrrrm 3y ago
Any plan for web support? :)
- iseeyuan 3y ago
  Could you elaborate? What kind of web support you are looking for?
- LarsDu88 3y ago
  There's wasi-nn for that I think. Whatever the original question means
LarsDu88 3y ago
Saw this at pytorch con. Very cool! Was used to build the hand tracking that shipped in quest 3 apparently
pjmlp 3y ago
The Android example is still too low level and full of CLI stuff, it would be easier for adoption, if it was more focused on Android Studio and building AARs.

94 points by resolve-ev 3y ago

vladimirzaytsev 3y ago

This is very cool.

ge96 3y ago

interference or inference?

cs702 3y ago
Obviously the poster meant to write inference. Maybe autocorrect messed up the headline? It wouldn't be the first time autocorrect has messed up a headline on HN. EDIT: dang fixed the headline. Thanks dang!
dang 3y ago
We've changed the title now to use language from the article itself. (submitted title was "ExecuTorch: Enabling On-Device interference for embedded devices") Edit: I switched "anywhere" to "mobile and edge devices" per https://pytorch.org/blog/pytorch-edge/ https://pytorch.org/blog/pytorch-edge/. One does not get away with saying things like "anywhere" in HN titles...

flakiness 3y ago

There is also PyTorch mobile [1]. I'm curious how these two relate. [Edit] And here is the answer [2] > PyTorch Mobile uses TorchScript to allow PyTorch models to run on devices with limited resources. ExecuTorch has a significantly smaller memory size and a dynamic memory footprint resulting in superior performance and portability compared to PyTorch Mobile. Also, ExecuTorch does not rely on TorchScript, and instead leverages PyTorch 2 compiler and export functionality for on-device execution of PyTorch models. [1] https://pytorch.org/mobile/home/ https://pytorch.org/mobile/home/ [2] https://pytorch.org/executorch/stable/intro-overview.html#how-is-executorch-different-from-pytorch-mobile-lite-interpreter https://pytorch.org/executorch/stable/intro-overview.html#ho...

phh 3y ago
Yeah it's a bit weird to see the difference. > Also, ExecuTorch does not rely on TorchScript, and instead leverages PyTorch 2 compiler and export functionality for on-device execution of PyTorch models. I think this is the major change: AFAIU you could consider ExecuTorch as ""just a rewrite"" of PyTorch mobile using PyTorch 2 compiler. BUT this is should still be pretty major. It was pretty hard to retarget models to be compatible with TorchScript. But most recent models will want to target PyTorch 2 compiler for the performance boost. Which means they'll automatically get ExecuTorch support. As someone who got burned several times in trying to port models to TorchScript and run them on Androids, I'm happy they did that.

liuliu 3y ago

Very cool. A few notes: 1. MPS backend uses MPSGraph exclusively, might hit some performance ceilings limited by MPSGraph. s4nnc moved more and more ops from MPSGraph to Metal directly to have better control on both allocation and some erratic behaviors from MPSGraph. 2. CoreML backend uses coremltools, thus it carries all the baggage of that: requiring to generate CoreML model AOT because Python dependency, have no control over memory planning, weight quantization scheme, or where to put the weights. Dynamic shape might further making memory planning worse as static shape is the main use-case of CoreML so far and far better tested. I would love to see an updated port of coremltools either in Swift or C++ (ONNX's coremltools implementation is in v3 I believe, and coremltools moved to v7 spec?).

ai-joe 3y ago

ONNX Runtime has supported this for years - https://onnxruntime.ai/blogs/pytorch-on-the-edge https://onnxruntime.ai/blogs/pytorch-on-the-edge

iseeyuan 3y ago
One difference here is that there is no conversion for ExecuTorch. In general conversions introduce inconsistency and are difficult to maintain, so ExecuTorch eliminates that step.

shoumikhin 3y ago

Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)

whobair 3y ago
hey - this might be a rather specific use case, but our applications need to run "real-time" and allocation-free - are there any guarantees on that from your side? assuming fixed shapes, pre-allocating everything for the forward pass should probably be possible in theory, but i guess that wasn't really a relevant factor in the design of it all
mlmandude 3y ago
It looks like executorch is for edge devices (phones / IoT / etc). I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/executorch/tree/main/backends https://github.com/pytorch/executorch/tree/main/backends , but seems like it might be possible. Is this something you are thinking about?
- iseeyuan 3y ago
  Yes ExecuTorch is currently targeted at Edge devices. The runtime is written in C++ with 50KB binary size (without kernels) and should run in most of platforms. You are right that we have not integrated to Nvidia backend yet. Have you tried torch.compile() in PyTorch 2.0? It would do the Nvidia optimization for you without Torchscript. If you have specific binary size or edge specific request, feel free to file issues in https://github.com/pytorch/executorch/issues https://github.com/pytorch/executorch/issues
  - fooblaster 3y ago
    torch.compile only works with python from what I understand. Many people need a native way to run GPU models, but don't want the bloat of full libtorch.

brrrrrm 3y ago

Any plan for web support? :)

iseeyuan 3y ago
Could you elaborate? What kind of web support you are looking for?
LarsDu88 3y ago
There's wasi-nn for that I think. Whatever the original question means

LarsDu88 3y ago

Saw this at pytorch con. Very cool! Was used to build the hand tracking that shipped in quest 3 apparently

pjmlp 3y ago

The Android example is still too low level and full of CLI stuff, it would be easier for adoption, if it was more focused on Android Studio and building AARs.