4 ms·
I've found the process of porting custom ML models to iOS extremely difficult. AFAIK the only way to leverage Apple Neural Engine (and get the best performance
by babl-yc 3y ago
I've found the process of porting custom ML models to iOS extremely difficult.
AFAIK the only way to leverage Apple Neural Engine (and get the best performance) is to use CoreML. The only documented way to use CoreML is via coremltools, which takes a trace of a PyTorch model and attempts to translate it into a protobuf graph understood by CoreML.
This process often fails and requires model changes, or worse "succeeds" but gives you the wrong output when you run the model. Additionally, you have to play detective to figure out why some operations run on the CPU, or GPU instead of ANE.
It's exciting to see more tools like this for working with tensor-like objects, but I really wish Apple would make porting custom models in a high performance manner easier.
- m00x 3y agoI'm not sure why there aren't more companies supporting ONNX. It's so nice to use if it's supported by the platform/model.
- lifthrasiir 3y agoNot every model can be easily converted to ONNX though, especially with PyTorch.
- mromanuk 3y agoI agree with you, I tried ONNX, more or less the same problem as converting to CoreML, plus issues of iOS compatibility.
- woodson 3y agoThat’s true, one has to design the model for the deployment target. Especially avoiding in-place tensor operations and python control flow helps for tracing.
- sgu999 3y agoI don't understand why Apple isn't trying to integrate better with the standard tools for that field. I guess it makes sense to lock in app devs, but ML eng.? That said I've had good success with onnxruntime recently [0]. [0] https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html https://onnxruntime.ai/docs/execution-providers/CoreML-Execu...
- domschl 3y agoThe project probably at least partially serves as documentation for other platforms to integrate Silicon acceleration. It basically demonstrates how to use macOS Accelerate and Metal MPS (metal performance shaders) using C++ for Machine Learning and training optimization. Thus other platforms can simply take this backend-code and integrate it. (Pytorch basically did that already with Apple's help).
- docfort 3y agoIt doesn’t use MPS, so at least that part is more interesting than the usual approach.
- yobanate 3y agoNailed it. I think more than partially. What happens in this repo will spread to the other major frameworks and over time, clever ideas that spawn on other projects will be reimplemented with Apple's adjustments back into the repo. It's a brilliant and efficient way to interact with the community, that can likely be measured in more sales of their hardware over time.
- beeboobaa 3y agoIt's apple. If they can't lock you in and milk you it's not worth doing.
- mromanuk 3y agoI've found that converting a custom PyTorch model to CoreML can be quite complex. I had to modify certain data types to facilitate the conversion process, which was not straightforward. The process becomes particularly frustrating when the model appears to convert successfully, but then fails to produce any output or loses layers entirely. Additionally, the debug information provided by the conversion tool isn't very helpful, adding to the challenge. As an iOS developer with no prior experience in Python, I found myself in a unique position. I needed to build a custom model for one of my keyboard apps to handle tasks like spellchecking, grammar correction, next-word prediction, and autocompletion. This necessity pushed me to learn Python and PyTorch. After mastering these, I then had to convert my knowledge back to Swift and CoreML. Ideally, I would have preferred to build my model directly in Swift & CoreML, but the current tools and resources for this approach are limited. This limitation is particularly evident in terms of the ease of use and flexibility that Python and PyTorch offer.