3 ms·

I can't see how something that tiny can compete in any meaningful way with a giant nVidia type card for training. I'd imagine it's more for running models that

I can't see how something that tiny can compete in any meaningful way with a giant nVidia type card for training. I'd imagine it's more for running models that have been trained already, like all the stuff they mentioned with Final Cut.

skohan 6y ago
Yeah I would imagine it's intended for similar use-cases as they use for iOS - for instance image/voice/video processing using ML models, and maybe for playing around with training, but it's not going to compete with a discreet GPU for heavy-duty training tasks
ojosilva 6y ago
For an 18-hour battery life computer (Macbook Air) that now doesn't even have a fan, it's for a complete different market segment from where nvidia cards dwell.
SoSoRoCoCo 6y ago
Not all NN models are behometh BERTs, U-Nets or ResNets. Person detection, keyword spotting, anomaly detection... there are lots of smaller neural nets that can be accelerated by a wide range of hardware.
mmm_grayons 6y ago
Can anyone who knows about machine learning hardware comment on how much faster dedicated hardware is as opposed to, say, a vulkan compute shader?
- qayxc 6y ago
  That depends entirely on the hardware of both the ML accelerator and the GPU in question, as well as model architecture, -data and -size. Unfortunately Apple was very vague when they described the method that yielded the claimed "9x faster ML" performance. They compared the results using an "Action Classification Model" (size? data types? dataset- and batch size?) between an 8-core i7 and their M1 SoC. It isn't clear whether they're referring to training or inference and if it took place on the CPU or the SoC's iGPU and no GPU was mentioned anywhere either. So until an independent 3rd party review is available, your question cannot be answered. 9x with dedicated hardware over a thermally- and power constrained CPU is no surprise, though. Even the notoriously weak previous generation Intel SoCs could deliver up to 7.73x improvement when using the iGPU [1] with certain models. As you can see in the source, some models don't even benefit from GPU acceleration (at least as far as Intel's previous gen SoCs are concerned). In the end, Apple's hardware isn't magic (even if they will say otherwise;) and more power will translate into higher performance so their SoC will be inferior to high-power GPUs running compute shaders. [1] https://software.intel.com/content/www/us/en/develop/articles/accelerate-deep-learning-inference-with-integrated-intel-processor-graphics-rev-2-0.html https://software.intel.com/content/www/us/en/develop/article...
- darsnack 6y ago
  On the NVidia A100, the standard FP32 performance is 20 TFLOPs, but if you use the tensor cores and all the ML features available then it peaks out at 300+ TFLOPs. Not exactly your question, but a simple reference point. Now the accelerator in the M1 is only 11 TFLOPs. So it’s definitely not trying to compete as an accelerator for training.
vbezhenar 6y ago
Isn't it better to rent a cloud with as many GPUs as necessary for a time needed to train the model? I don't know state of things in ML.
- nicdc 6y ago
  Curious to hear responses to this too..
- nelsondev 6y ago
  For productionizing/training massive models, yes. But in the development phase, when you are testing on a smaller corpus of data, to make sure your code works, the on-laptop dedicated chip could expedite the development process.
- PeterisP 6y ago
  I agree with the parent poster that it's probably more about inference, not training. If ML developers can assume that consumer machines (at least "proper consumer machines, like those made by Apple") will have support to do small-scale ML calculations efficiently, then that enables including various ML-based thingies in random consumer apps.
- FridgeSeal 6y ago
  Not necessarily. It can be surprisingly cost-effective to invest a few $k in a hefty machine(s) with some high-end GPU's to train with due to the exceedingly hefty price of cloud GPU compute. The money invested up-front in the machine(s) pays itself off in (approximately) a couple of months. The "neural" chips in these machines are for accelerating inference. I.e. you already have a trained model, you quantise and shrink it, export it to ONNX or whatever Apple's CoreML requires, ship it to the client, and then it runs extra-fast, with relatively small power draw on the client machine due to the dedicated/specialised hardware.
- imtringued 6y ago
  Cloud GPU instances are very expensive. If you get consumer GPUs not only do you save money, you can sell them afterwards for 50% of the purchasing price.
xiphias2 6y ago
Sure, inferencing doesn't need floating point instructions, so NVIDIA will stay the only real solution for desktop/laptop based model training for a long time.
BSVogler 6y ago
It is just not meant to be a training device so comparing with data Center or developer GPUs is useless. Faster inference for end-users is what is mentioned by Apple and the only use case where this hardware makes sense.

skohan 6y ago

Yeah I would imagine it's intended for similar use-cases as they use for iOS - for instance image/voice/video processing using ML models, and maybe for playing around with training, but it's not going to compete with a discreet GPU for heavy-duty training tasks

ojosilva 6y ago

For an 18-hour battery life computer (Macbook Air) that now doesn't even have a fan, it's for a complete different market segment from where nvidia cards dwell.

SoSoRoCoCo 6y ago

Not all NN models are behometh BERTs, U-Nets or ResNets. Person detection, keyword spotting, anomaly detection... there are lots of smaller neural nets that can be accelerated by a wide range of hardware.

mmm_grayons 6y ago

Can anyone who knows about machine learning hardware comment on how much faster dedicated hardware is as opposed to, say, a vulkan compute shader?

qayxc 6y ago
That depends entirely on the hardware of both the ML accelerator and the GPU in question, as well as model architecture, -data and -size. Unfortunately Apple was very vague when they described the method that yielded the claimed "9x faster ML" performance. They compared the results using an "Action Classification Model" (size? data types? dataset- and batch size?) between an 8-core i7 and their M1 SoC. It isn't clear whether they're referring to training or inference and if it took place on the CPU or the SoC's iGPU and no GPU was mentioned anywhere either. So until an independent 3rd party review is available, your question cannot be answered. 9x with dedicated hardware over a thermally- and power constrained CPU is no surprise, though. Even the notoriously weak previous generation Intel SoCs could deliver up to 7.73x improvement when using the iGPU [1] with certain models. As you can see in the source, some models don't even benefit from GPU acceleration (at least as far as Intel's previous gen SoCs are concerned). In the end, Apple's hardware isn't magic (even if they will say otherwise;) and more power will translate into higher performance so their SoC will be inferior to high-power GPUs running compute shaders. [1] https://software.intel.com/content/www/us/en/develop/articles/accelerate-deep-learning-inference-with-integrated-intel-processor-graphics-rev-2-0.html https://software.intel.com/content/www/us/en/develop/article...
darsnack 6y ago
On the NVidia A100, the standard FP32 performance is 20 TFLOPs, but if you use the tensor cores and all the ML features available then it peaks out at 300+ TFLOPs. Not exactly your question, but a simple reference point. Now the accelerator in the M1 is only 11 TFLOPs. So it’s definitely not trying to compete as an accelerator for training.

vbezhenar 6y ago

Isn't it better to rent a cloud with as many GPUs as necessary for a time needed to train the model? I don't know state of things in ML.

nicdc 6y ago
Curious to hear responses to this too..
nelsondev 6y ago
For productionizing/training massive models, yes. But in the development phase, when you are testing on a smaller corpus of data, to make sure your code works, the on-laptop dedicated chip could expedite the development process.
PeterisP 6y ago
I agree with the parent poster that it's probably more about inference, not training. If ML developers can assume that consumer machines (at least "proper consumer machines, like those made by Apple") will have support to do small-scale ML calculations efficiently, then that enables including various ML-based thingies in random consumer apps.
FridgeSeal 6y ago
Not necessarily. It can be surprisingly cost-effective to invest a few $k in a hefty machine(s) with some high-end GPU's to train with due to the exceedingly hefty price of cloud GPU compute. The money invested up-front in the machine(s) pays itself off in (approximately) a couple of months. The "neural" chips in these machines are for accelerating inference. I.e. you already have a trained model, you quantise and shrink it, export it to ONNX or whatever Apple's CoreML requires, ship it to the client, and then it runs extra-fast, with relatively small power draw on the client machine due to the dedicated/specialised hardware.
imtringued 6y ago
Cloud GPU instances are very expensive. If you get consumer GPUs not only do you save money, you can sell them afterwards for 50% of the purchasing price.

xiphias2 6y ago

Sure, inferencing doesn't need floating point instructions, so NVIDIA will stay the only real solution for desktop/laptop based model training for a long time.

BSVogler 6y ago

It is just not meant to be a training device so comparing with data Center or developer GPUs is useless. Faster inference for end-users is what is mentioned by Apple and the only use case where this hardware makes sense.