7 ms·
TensorFlow on Mobile: TensorFlow Lite
- Quenty 9y agoI’m surprised it doesn’t take advantage of the iPhone’s “neural engine” if it’s available.
- RubenSandwich 9y agoApple uses a different model format[0], so tensorflow models need to be converted first before they can be used. So Apple would need to add support for this tensorflow lite/mobile model format to work in Core ML. [0]https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml https://developer.apple.com/documentation/coreml/converting_...
- J0-nas 9y agoAnother format is onnx [0], where Apple and Goodle don't seem to participate. I don't know the politics behind it but there should be a common format for all libraries/platforms. [0] https://onnx.ai/ https://onnx.ai/
- stochastic_monk 9y agoonnx seems to be an initiative to try to allow Microsoft's and Facebook's AI platforms to compete with Tensorflow. Considering Tensorflow is more a grab at developer mindshare than an ideal platform [for example, its performance lags by a factor of two behind MXNet and Torch], I think it's a smart plan.
- J0-nas 9y agoI don't think the last sentence is fair to trensorflow. Torch has been around for ~15 years compared to the 3 of TF. You'd expect TF to catch up in terms of performance in the future.
- stochastic_monk 9y agoI think it's more than fair. I'm certain that more developer-hours have been spent on TensorFlow than on Torch. They very well may catch up, but that doesn't mean that hype and the Google name aren't the reason it's gotten popular.
- pj_mukh 9y agoQuestion: So does this mean, Tensorflow Lite will have worse performance on iOS than a natively implemented CoreML model of the same design would?
- RubenSandwich 9y agoNot necessarily. They are just data formats that encode the different weights and layers of the neurel net. (That is an oversimplification, but enough to give you a basic grasp of what is happening under the covers.) Apple is obviously investing a lot into CoreML, so expect some speed ups just due to effort but I would not expect factors of magnituite difference in performance. (This might be wrong as Tensorflow Lite is still quite immature, but as it matures I believe it will be comparable in performance.)
- jacksmith21006 9y agoHow about support for the PVC in the new Pixel phones?
- halflings 9y agoAnother commenter noted that is supported; they provide a tool to convert the tflite model to Core ML format: https://developers.googleblog.com/2017/12/announcing-core-ml-support.html https://developers.googleblog.com/2017/12/announcing-core-ml...
- JepZ 9y agoWould be super cool if they would even port their stuff to Web Assembly. Now we have some Android and some iOS solution, but webasm would be truly platform independent.
- jahewson 9y agoWithout SIMD I’m guessing the performance wouldn’t be great.
- deepsun 9y agoDoesn't JIT automatically SIMDifies some operations on arrays? Although I agree it would be better to have a control over that, instead of hoping JIT compiler would do the right thing.
- dmitrygr 9y agoGeneric automatic vectorization remains very much an unsolved problem. Outside of very specific cases, even modern compilers very much suck at it. That's why most vectorizable libraries have hand-coded SSE/NEON assembly blocks. I would not expect a JIT, which has to be fast, to have time to auto-vectorize anything at all, much less to do it well.
- arnioxux 9y agoI think there is someone working on a webgl version: https://tenso.rs/ https://tenso.rs/
- zitterbewegung 9y agoSome people are wondering if Tensorflow Lite would support CoreML / iPhone's neural engine. Tensorflow is not supported with coremltools (but I suppose Google could contribute a patch for this). I'm not sure about the CoreML libraries on the phone but from my understanding it may work. Keras is supported out of the box with the Tensorflow backend though. See https://medium.com/@JMangia/super-simple-end-to-end-test-of-keras-tensorflow-and-coreml-f247ab73fb42 https://medium.com/@JMangia/super-simple-end-to-end-test-of-... and https://github.com/apple/coremltools https://github.com/apple/coremltools
- rambossa 9y agonot supported (or is this the patch you mention)? https://developers.googleblog.com/2017/12/announcing-core-ml-support.html https://developers.googleblog.com/2017/12/announcing-core-ml...
- n8henrie 9y agoI’m looking forward to some tflite on Raspberry Pi tutorials.
- bengalister 9y agoAn issue has been created on github: https://github.com/tensorflow/tensorflow/issues/14589 https://github.com/tensorflow/tensorflow/issues/14589 It seems that it will be supported but there's no timeline.
- lovelearning 9y agoIt looks like this issue has been resolved to a reasonable extent. I just tried building TFLite for a lark on a Raspberry Pi 2 and succeeded without much hassle and any of that bazel nonsense. It's a standalone component - they provide a Makefile and I just had to run make. The example label_image inference app took a bit more work - I had to write my own Makefile - but wasn't difficult. I'm able to run inference on images on RPi2 with both floating and quantized MobileNets tflite models. Will try videos now. For anybody else interested in trying out TFLite on your desktop or Raspberry Pi or any SBC, I've put up instructions and Makefile at https://pastebin.com/LDEGyG46 https://pastebin.com/LDEGyG46
- alex_hirner 9y agoThanks for providing those instructions. What kind of performance do you get for both MobileNet versions?
- lovelearning 9y agoI did not do any performance profiling yesterday. I am currently interested in deploying object detection models for video streams, and plan to do detailed profiling of those when ready. That said, the label_image classification example does provide some timing information. On the 500x600px test image that comes with it, non-quantized classifier for 10 iterations took ~900 ms on a RPi2 while quantized took ~300 ms. The confidence values were much lower in the quantized version, but I didn't spot any major misclassification.
- jacksmith21006 9y agoBe curious if Google will add support for the PVC in the Pixel 2 phones?
- sandGorgon 9y agoThe training export from Google CloudML -> Tensorflow Lite is still not clear. Since the freeze_graph scripts use Bazel, it is hard to pull this off on the cloud
- reubenmorais 9y agoYou can call freeze_graph directly from Python code, it's in tensorflow.python.tools.freeze_graph. https://github.com/tensorflow/tensorflow/blob/ba032db13627945e7cc772dbd3d85d257aef3ab9/tensorflow/python/tools/freeze_graph.py#L209 https://github.com/tensorflow/tensorflow/blob/ba032db1362794...
- makapuf 9y agoJust out of Fosdem and was attending a very interesting talk on tensorflow running on microcontrollers (uTensor project) not dissimilar to this.