11 ms·
Accelerating TensorFlow Performance on Mac
- cs_bot 6y agoWill I be able to train models (PyTorch, Keras, TF) and do python dev on the M1 MBP?
- olliej 6y agoWait, so are they using the GPUs or the "neural engine"? because the gpu approach should also work on any other machine right?
- spookyboomba 6y agoAs the charts compare Intel based Macs vs the new M1 chip, they would be comparing CPU performance.
- loukrazy 6y agoThe charts are comparing “accelerated” performance versus what appears to be a cpu only baseline. It is not clear which M1 hardware beyond the CPU is used
- TYPE_FASTER 6y agoLooks like Tensor Flow now supports ML Compute, which in turn uses the appropriate hardware. In the case of the Mac Pro, I'm guessing ML Compute is using the GPU. In the case of the Intel MacBook Pro 13", which as far as I can tell from Apple's site can't be purchased with a discrete GPU, that will be either the Intel Iris GPU, or the CPU. In the case of the M1 MacBook Pro 13", I'm assuming ML Compute prioritizes the Neural Engine over the GPU (and CPU), but don't know if there are use cases where the GPU would be preferable.
- pram 6y agoIt looks like the class only has cpu and gpu types so, as has been the case, figuring out when the 'neural engine' is invoked is remains opaque: https://developer.apple.com/documentation/mlcompute/mlcdevice https://developer.apple.com/documentation/mlcompute/mlcdevic...
- roughly 6y agoThe Apple ML Research group announcement has more details: https://machinelearning.apple.com/updates/ml-compute-training-on-mac https://machinelearning.apple.com/updates/ml-compute-trainin... Key line: Until now, TensorFlow has only utilized the CPU for training on Mac. The new tensorflow_macos fork of TensorFlow 2.4 leverages ML Compute to enable machine learning libraries to take full advantage of not only the CPU, but also the GPU in both M1- and Intel-powered Macs for dramatically faster training performance. So, looks like it's faster on both Intel & M1, but the M1 MBP has a much faster GPU than the Intel MBP
- microtherion 6y agoBut note that the ANE and the GPU are different facilities.
- roughly 6y agoCorrect - I’d guess this is Not using the neural engine. I don’t know enough about that hardware to hazard a guess about how easy it would be to get that part of the chip involved.
- syntaxing 6y agoI wish they would compare to a 1080Ti or something along those lines. Nevertheless, this is pretty neat minus the RAM implications because realistically, I'm guessing you can use 5GB of RAM at most for training. I always wanted a computer where I can test the model for a couple batches (that doesn't cost $1200+) and then push to the cloud to train for scalability.
- tpetry 6y agoSeems i am not the only one who wants to know whether the M1 neural engine is performant enough for prototyping :) It will be interesting how long the other deep learning frameworks will need to support the M1. Pytorch has not yet achieved comparable performance on a TPU compared to tensorflow.
- syntaxing 6y agoIt would be pretty awesome to write an app, train the network, and deploy it using the same computer! I sometimes try to use google Colab to train some fun stuff (like style transfer) but sometimes it gives me a headache (grateful it is free though!!) It's hard to predict if the session will just stop and persistent data is a pain. Having an affordable (ironic using this word with Apple) for ML prototyping would be amazing.
- FPGAhacker 6y agoYou know, they don’t mention the neural engine. I know very little of ML but maybe the neural engine isn’t helpful on the training side?
- meowphius 6y agoThe API of the neural engine is closed for some reason.
- therealmarv 6y agoreally? So WHO can use it then when not normal developers? Only Apple?
- tpetry 6y agoThe data compared to a normal priced gpu for a desktop would be interesting. To get a feeling whether the m1 would be interesting for prototyping.
- lisnake 6y agoThe way I'm reading this is that M1 in $700 Mac Mini is only two times slower than Mac Pro, which is what, 4-5 thousands USD?
- syntaxing 6y agoHmm I'm reading this different, the first graph is seconds/batch. So the lower the number should be better. The M1 Mini should perform 4X faster than the Mac Pro Intel in terms of ML training.
- samtheprogram 6y agoI believe you're misreading GP's comment and/or the associated graphs. The graph with 4x performance increase is comparing a MacBook Pro (Intel) with a MacBook Pro (M1). GP is comparing the MacBook Pro (M1) results in said graph with the Mac Pro (Intel) result in the following graph, where the Mac Pro is roughly 2x.
- ed25519FUUU 6y agoHow is the actual acceleration happening here? Is it done via optimizations with the GPU or the usage of the Neural Engine, the Unified Memory or some combination?
- kube-system 6y ago> Until now, TensorFlow has only utilized the CPU for training on Mac. The new tensorflow_macos fork of TensorFlow 2.4 leverages ML Compute to enable machine learning libraries to take full advantage of not only the CPU, but also the GPU in both M1- and Intel-powered Macs for dramatically faster training performance. It sounds like the performance gains are because they are now using GPU and CPU instead of just CPU. https://machinelearning.apple.com/updates/ml-compute-training-on-mac https://machinelearning.apple.com/updates/ml-compute-trainin...
- subzerofun 6y agoFrom: https://github.com/apple/tensorflow_macos#device-selection-optional https://github.com/apple/tensorflow_macos#device-selection-o... "There is an optional mlcompute.set_mlc_device(device_name=’any') API for ML Compute device selection. The default value for device_name is 'any’, which means ML Compute will select the best available device on your system, including multiple GPUs on multi-GPU configurations." Hope this is not a stupid question: Does this TF version just work with M1 GPU-Cores / Intels iGPU or also with AMD and supported Nvidia GPUs?
- gok 6y agoIt uses Metal, so the AMD GPUs like the ones in the shipping Mac Pro work.
- singhrac 6y agoJust to clarify - when they say using the GPU on Intel-based Macs, they mean the integrated GPU, not the discrete AMD GPU that my 16in MBP comes with?
- devindotcom 6y agoThe tests are on 13-inch MBPs (Intel and M1) so I would assume not.
- deeeeplearning 6y agoDoes anyone doing serious commercial grade ML actually train models on their macbook?
- fxtentacle 6y agoSurely not. I'm working on a PC with 64GB of main RAM and 12GB of dedicated GPU RAM and I already have to do tricks to be able to squeeze state of the art models into my limited memory. For comparison, NVIDIA just upgraded from 40GB to 80GB.
- vimy 6y agoUnified memory on Apple silicon means the gpu/neural engine has as much ram as the mac. A future mbp with 64 gb ram will be able to fit a lot of models. A whole ML laptop for the price of one gpu.
- fxtentacle 6y agoAgree, if Apple ever goes back to offering 64+ GB RAM Notebooks, they will become a viable choice again.
- simonh 6y agoContrary to the way you and a lot of people have been spinning things recently, Apple has not discontinued the 16" Macbook Pro, and you can buy an Apple laptop today in all their form factors with just as much memory as you could a few weeks ago. The M1 machines are only replacing models that only went up to 16 GB already.
- tantalor 6y agoI thought ML models were tiny, small enough to run on mobile devices.
- nl 6y agoMost ML models can be shrunk at the cost of some accuracy, but some models are extremely large.
- rubatuga 6y agoWhy would I use tensorflow when pytorch exists?
- fxtentacle 6y agoBecause you're Google? Or you work with them or their partners? Or because you like their approach? Also, in my experience an optimized XLA export is usually faster than Pytorch.
- marcyb5st 6y agoDisclaimer, I work for Google, but not on Tensorflow. I agree that in TF 1.x the ergonomics are/were bad, but things improved considerably in TF 2.x. In the case you don't like that either, the tf.keras package offer another more friendlier option that integrates well with the other tf packages (e.g. tf.data, tf.estimators, ...). Finally, I believe the decision also depends on your use case: If you are just experimenting, maybe PyTorch gets to your results faster. However, I also I think that being able to use TFX (https://www.tensorflow.org/tfx https://www.tensorflow.org/tfx) seamlessly saves you a lot of time when you need to put your models in production.
- alibrarydweller 6y agoAs someone who has been trying to set up a simple use case of TFX for six months I respectfully disagree :)
- therealmarv 6y agoWhat is better? Macbook with Apple's M1 or (old) Macbook with Intel CPU and external AMD GPU?
- jonplackett 6y agoDepends if you’re working in a coffee shop
- cocoa19 6y agoCall me old, but I can't understand how people can sit in the most uncomfortable chairs for hours, tilting my head and neck down towards my laptop. A workday at Starbucks would cause me head and shoulder pain for days.
- neurostimulant 6y agoIn my case, I only do that when I'm getting fed up of working on the same place everyday. Working in a coffee shop is not ideal, but the environment is different enough to reset my mood. Obviously I'm not going to work in a coffee shop every day though, and I haven't work on a coffee shop this year due to covid.
- antipaul 6y agoAgain, M1 chip shows impressive performance gains. First graph shows, what, 3-4x improvement over Intel?
- jeroenhd 6y agoDoes it? It shows the accelerated performance vs Metal performance on a generic GPU vs performance on a GPU with dedicated ML units on it. It's not a performance improvement "over Intel", it's a performance improvement "over hardware without dedicated ML acceleration in the new version". There's a similar massive performance gain with the Mac Pro example, showing that the new version being tested is able to leverage GPU power much more effectively. Comparing the dedicated ML hardware to the generic just-decode-video-and-animate-windows-at-decent-speeds Intel Iris doesn't really prove any performance gain. There's no apples to apples comparison to be made here. If Apple and Nvidia would finally get over themselves, you might be able to make a fair comparison between the M1 and Intel + CUDA. The Intel "pro" laptop they're comparing against only comes with a 1.7GHz Intel CPU and only Intel's mediocre integrated graphics, so I don't see why you would use that version for anything professional related to ML anyway; at best you'd use it to test your pipelines. All you can conclude from this is that the new M1 chips perform better at TensorFlow than the Intel chips + GPUs in the previous models of Macbook after Apple made a Mac-optimised version that better leverages GPU power.
- vimy 6y ago"On my models currently in test, it’s like using a 1080 or 1080 TI." https://twitter.com/spurpura/status/1329168059647488000 https://twitter.com/spurpura/status/1329168059647488000 That's amazing.
- qd6pwu4 6y agoIs it training or inference? Inference only is amazing enough, and I don't expect M1 to beat 1080 for training deep neural networks.
- why_only_15 6y agoThe benchmarks here are for training it says.
- jonplackett 6y agoQuestion - ML models are hard to debug because it’s all happening on the GPU. If you have shared memory with the CPU could that make it easier to see what’s going on inside your model?
- BobbyJo 6y agoIs that why they are hard to debug? I thought they were hard to debug because deep learning models consist of lots of black-box-ish components. I've worked with GPU software before, and it is a pain to debug, but you generally don't have to care about that level when using things like tensorflow or pytorch.
- skwb 6y agoNot unless you're writing the backend CUDA code for TF honestly. Most of the time you're accessing your data/models/structure through the TF API so you really aren't doing too much that directly interfaces with the GPU. To put it another way, when you're developing ML models (99.9% of the time), it's not like people are writing if statements in CUDA.
- debbiedowner 6y agoIt's cool, but is it 16bit perhaps? Would be fair to compare to tensorbook of similar cost. Desktop wise, you can buy a 1080 for <400 on eBay and the latest gen nvidia are obviously much faster for still less cost than the mac pro.
- sandGorgon 6y agoThis is funny - it clearly points to Intel & AMD as the culprits to why we didn't have a CUDA alternative for this long. Not Nvidia as long reviled. If Apple can work with...Google to get their framework changed for the M1, then there is absolutely no excuse for Intel/AMD. They had a decade to fix this. They deserve their fate.
- rfoo 6y agoSurely this is funny: too many Apple fanboys get on the hype train. AMD had been working with Google to get their framework changed for AMD's GPU for more than two years [1], and all their work are upstreamed. Oh, and AMD's ROCm/OpenCL support is really for general computing, i.e. CUDA alternative, unlike the ML Compute here. ML Compute is something Apple created specifically for running neural networks, nothing more, and roughly equivalent to TensorRT / Android NN if you want to compare with other platforms. And it was here because the wall of Apple's walled garden is too high that nobody other than them can effectively optimize NN inference/training on their chip. [1] https://pypi.org/project/tensorflow-rocm/#history https://pypi.org/project/tensorflow-rocm/#history
- raverbashing 6y agoCool Are they getting competitive results?
- pjmlp 6y agoHow it can be a CUDA alternative when their driver support is still a joke and not available in all OSes?
- sandGorgon 6y agoFYI - i have never used a macbook in my life. I only use XPS with Fedora. So this frustration is personal. I have been party to get AMD/Intel's CUDA alternative out the door on some of the ML libraries - which one is it now ? OpenCL...SYCL ...ROCm...PlaidML ? I cant remember. All this time I was pissed at nvidia - surely they were playing subversive politics to kill all of this. With so many initiatives, surely AMD/Intel had their heart in the right place. Apple and Google are cutthroat rivals. And they worked together for a just-released chip to get fully working acceleration support. Here's where it gets sadder for me - Tensorflow has included a GPU accelerated version of Numpy ( https://twitter.com/fchollet/status/1292893864986984448?lang=en https://twitter.com/fchollet/status/1292893864986984448?lang...) . Numpy itself is only accelerated using BLAS/LAPACK which cant leverage GPU all that well. https://www.tensorflow.org/api_docs/python/tf/experimental/numpy https://www.tensorflow.org/api_docs/python/tf/experimental/n... At this point, it is basically a sealed deal - if you're even remotely dabbling in data science, you better be working on a Mac. Which makes me hate my XPS all the more :(
- sooheon 6y agoCould this mean Jax support is coming via XLA?
- freewizard 6y agoSounds like even though it runs on all Macs with macOS 11.x, AMD GPU on Intel Macs are not optimized: https://github.com/apple/tensorflow_macos/issues/7 https://github.com/apple/tensorflow_macos/issues/7