4 ms·

I’m not sure why they added this feature. All Apple SoCs have far more energy efficient compute than the CPU. This would only make sense for really tiny models

by ein0p 2y ago

I’m not sure why they added this feature. All Apple SoCs have far more energy efficient compute than the CPU. This would only make sense for really tiny models which need extremely quick forward pass. For such models the overhead of a GPU or Neural Engine kernel launch would be quite noticeable. But for those the old NEON was already OK, and if not, there also is a dedicated matrix unit there called AMX. Seems kinda random to me.

adrian_b 2y ago
This replaces AMX, it is its successor. The older Apple CPUs implemented a custom form of AMX that was not standardized by Arm. Presumably as a result of cooperation with Apple, the Arm ISA now includes a set of instructions with the same purpose like the original Apple AMX. The newer Apple CPUs have been updated to use the standard Arm ISA, instead of their older proprietary ISA. In the Apple CPUs, the former AMX and the current SME provide a much higher throughput than the CPU cores, even if lower than the GPU, and a much lower latency than the GPU, even if higher than the CPU cores. AMX/SME is implemented as a separate accelerator, distinct from the CPU cores, because this saves power and area in comparison with implementing such instructions in each CPU core. The Apple CPUs do not attempt to compete in high-performance computing applications, so the extra throughput provided by a separate shared matrix operation accelerator is good enough for them.
- saagarjha 2y ago
  This has both actually.
  - adrian_b 2y ago
    That must be for preserving the compatibility with the older software versions. The same happens in x86, where there are hundreds of obsolete instructions, which have been replaced by better instructions, but which are still supported to allow the execution of old programs. Both the new Arm SME instructions and the old Apple AMX instructions are executed by the same hardware matrix operation accelerator. Previously Arm has extended the Aarch64 ISA with the SVE instructions, in order to support the Fujitsu supercomputer. Then they were not satisfied with the original SVE and they have extended it into SVE2. I suppose that something similar has happened with SME. Apple must have negotiated with Arm the inclusion of matrix and vector operations implemented by a separate shared accelerator. The result was SME, which differs from the original AMX either because Apple has thought some improvements based on the experience with the first instruction set or because Arm has desired some changes from the Apple proposal.
Archit3ch 2y ago
Dedicated FP64 is great for real-time audio processing. Like an included DSP chip.
- phkahler 2y ago
  Isn't FP32 sufficient for audio processing? Even though we have 24bit DACs and ADCs these days I feel like 16bit was really good enough. FP32 with 24bit mantissa should avoid rounding errors at the 16bit level right?
  - Archit3ch 2y ago
    It depends on the application. 16bit is enough for representation. 24bit is enough for recording (some leeway because recording levels won't be ideal). FP32 for processing with simple effects (e.g. mixer, some EQs). If that's enough for your needs, you can SIMD/GPU to your heart's content. FP64 for high Q filters, phasors, LU decompositions.
  - rerdavies 2y ago
    ##### Is 16 bits good enough? Amp models need all the precision they can get. The effective range of output signals is greatly compressed because the signal is soft-clipped by the amplifier's non-linear response. Real guitar amplifiers have significant levels of noise in their output signals; digital guitar simulations of guitar amplifiers are typically even more sensitive to noise in their input signals. Currently, probably the easiest way to tell the difference between recordings of real guitar amps and neural model simulations of guitar amps: how they respond to signal noise in their inputs. A really good ADC may have a 24-bit representation, but it will only have an 18 to 20 bit signal to noise ratio. Cheap audio adapters (pretty much all the audio adapters costing less than $100) will happily deliver you an input signal in 24-bit (or even 32-bit) representation, but will have less than 16 bits of signal above the noise floor. For example, I have an M-AUDIO Fast Track usb audio adapter ($50) that provides 24-bit input but only has 12 bits of signal above the noise floor, even with levels meticulously set. Guitar amp models sound horrible when using this device. But when I use my MOTU-M2 (~$200) which probably provides a full 20 bits of signal above the noise floor, the same models sound faaabulous! Those extra bits of SN/R are precious. An amp simulation of an input signal on a cheap ADC sounds noticeably "fizzier" than an amp simulation of the same input signal on an ADC with 19 actual significant bits of actual signal above the noise floor. So 16 bits is not good enough. And 24 bits does make a difference (even if it's only 19 bits of actual difference) ##### Would FP64 be better? Currently, Machine Learning models of guitar amps use FP32, because they are extremely compute-intensive when running in realtime (and extremely compute intensive when training the model in realtime). Would FP64 calculations improve the quality of amp simulations? That would depend on how much precision gets lost while performing ML simulation. Probably a fair bit of precision does gets lost, between the massive matrix multiplies that are involved, and the calculation of non-linear activation functions (typically atan functions in current ML guitar models). Roughly, I think the answer goes like this. We have an input signal with 19 bits of precision. And the 19th bit seems to make a difference. FP32 provides 24 bits of precision -- 5 extra bits of precision -- to avoid rounding errors while calculation massive matrix multiplies, and at least two rounds of atan activation functions (some of which are in a feedback loop). Are those five extra bits of guard precision being consumed during processing? Heck yes! I'm almost certain that the quality of amp models would improve if the models were trained in FP64, and am reasonably certain that quality would improve if realtime calculations were performed in FP64 as well. But on a Raspberry Pi (and probably on a x64 device as well), neural models cannot be run with FP64 precision in realtime. An ML-based amp model consumes bout 45% of available CPU bandwidth running with FP32 precision. Running with FP64 precious would add least quadruple that. As a point of interest, matrix multiplies running on a Raspberry Pi 4 Arm Cortex A72 are almost completely limited by memory bandwidth to L2 cache and main memory. And that performance is (mostly) constrained by the tile size used in the matrix multiples, which (when using A72 neon registers) is constrained by the number of neon registers available. I believe that performance would roughly increase linearly as a function of available tile size. Whether it's linear or not depends a bit on how well matrix units deal with Nx1 matrices (vectors). Although the to perform NxM matrix multiples dominates, a significant amount of execution time also gets spent doing Nx1 and/or vector processing. Whether the corresponding performance boost is good enough to allow realtime audio processing at FP64.... the only way to find out would be to do it. * Results based on extensive optimization and profiling of Toob ML and TooB Neural Amp Modeler guitar effects hosted by [PiPedal](https://rerdavies.github.io/pipedal/ https://rerdavies.github.io/pipedal/)
GeekyBear 2y ago
Matrix multiplication is very commonly used in science and engineering, not just machine learning. The neural engine is optimized for machine learning use cases. This standardized successor to AMX is more general purpose than the neural engine and has much improved matrix multiplication performance vs NEON. As a bonus, since this is no longer just an experimental implementation of a matrix unit, you get documented access to the new ARM standardized low level instruction set.
brigade 2y ago
The neural engine by design cannot handle all possible kernels, and the GPU is significantly slower for integer math, and cannot do fp64. Then for the iPhone SoCs with 4 or 5 core GPUs, the GPU is a bit slower for fp16 and fp32 too.
lxgr 2y ago
> if not, there also is a dedicated matrix unit there called AMX This seems to be the successor to AMX.

by ein0p 2y ago

adrian_b 2y ago

This replaces AMX, it is its successor. The older Apple CPUs implemented a custom form of AMX that was not standardized by Arm. Presumably as a result of cooperation with Apple, the Arm ISA now includes a set of instructions with the same purpose like the original Apple AMX. The newer Apple CPUs have been updated to use the standard Arm ISA, instead of their older proprietary ISA. In the Apple CPUs, the former AMX and the current SME provide a much higher throughput than the CPU cores, even if lower than the GPU, and a much lower latency than the GPU, even if higher than the CPU cores. AMX/SME is implemented as a separate accelerator, distinct from the CPU cores, because this saves power and area in comparison with implementing such instructions in each CPU core. The Apple CPUs do not attempt to compete in high-performance computing applications, so the extra throughput provided by a separate shared matrix operation accelerator is good enough for them.

saagarjha 2y ago
This has both actually.
- adrian_b 2y ago
  That must be for preserving the compatibility with the older software versions. The same happens in x86, where there are hundreds of obsolete instructions, which have been replaced by better instructions, but which are still supported to allow the execution of old programs. Both the new Arm SME instructions and the old Apple AMX instructions are executed by the same hardware matrix operation accelerator. Previously Arm has extended the Aarch64 ISA with the SVE instructions, in order to support the Fujitsu supercomputer. Then they were not satisfied with the original SVE and they have extended it into SVE2. I suppose that something similar has happened with SME. Apple must have negotiated with Arm the inclusion of matrix and vector operations implemented by a separate shared accelerator. The result was SME, which differs from the original AMX either because Apple has thought some improvements based on the experience with the first instruction set or because Arm has desired some changes from the Apple proposal.

Archit3ch 2y ago

Dedicated FP64 is great for real-time audio processing. Like an included DSP chip.

phkahler 2y ago
Isn't FP32 sufficient for audio processing? Even though we have 24bit DACs and ADCs these days I feel like 16bit was really good enough. FP32 with 24bit mantissa should avoid rounding errors at the 16bit level right?
- Archit3ch 2y ago
  It depends on the application. 16bit is enough for representation. 24bit is enough for recording (some leeway because recording levels won't be ideal). FP32 for processing with simple effects (e.g. mixer, some EQs). If that's enough for your needs, you can SIMD/GPU to your heart's content. FP64 for high Q filters, phasors, LU decompositions.
- rerdavies 2y ago
  ##### Is 16 bits good enough? Amp models need all the precision they can get. The effective range of output signals is greatly compressed because the signal is soft-clipped by the amplifier's non-linear response. Real guitar amplifiers have significant levels of noise in their output signals; digital guitar simulations of guitar amplifiers are typically even more sensitive to noise in their input signals. Currently, probably the easiest way to tell the difference between recordings of real guitar amps and neural model simulations of guitar amps: how they respond to signal noise in their inputs. A really good ADC may have a 24-bit representation, but it will only have an 18 to 20 bit signal to noise ratio. Cheap audio adapters (pretty much all the audio adapters costing less than $100) will happily deliver you an input signal in 24-bit (or even 32-bit) representation, but will have less than 16 bits of signal above the noise floor. For example, I have an M-AUDIO Fast Track usb audio adapter ($50) that provides 24-bit input but only has 12 bits of signal above the noise floor, even with levels meticulously set. Guitar amp models sound horrible when using this device. But when I use my MOTU-M2 (~$200) which probably provides a full 20 bits of signal above the noise floor, the same models sound faaabulous! Those extra bits of SN/R are precious. An amp simulation of an input signal on a cheap ADC sounds noticeably "fizzier" than an amp simulation of the same input signal on an ADC with 19 actual significant bits of actual signal above the noise floor. So 16 bits is not good enough. And 24 bits does make a difference (even if it's only 19 bits of actual difference) ##### Would FP64 be better? Currently, Machine Learning models of guitar amps use FP32, because they are extremely compute-intensive when running in realtime (and extremely compute intensive when training the model in realtime). Would FP64 calculations improve the quality of amp simulations? That would depend on how much precision gets lost while performing ML simulation. Probably a fair bit of precision does gets lost, between the massive matrix multiplies that are involved, and the calculation of non-linear activation functions (typically atan functions in current ML guitar models). Roughly, I think the answer goes like this. We have an input signal with 19 bits of precision. And the 19th bit seems to make a difference. FP32 provides 24 bits of precision -- 5 extra bits of precision -- to avoid rounding errors while calculation massive matrix multiplies, and at least two rounds of atan activation functions (some of which are in a feedback loop). Are those five extra bits of guard precision being consumed during processing? Heck yes! I'm almost certain that the quality of amp models would improve if the models were trained in FP64, and am reasonably certain that quality would improve if realtime calculations were performed in FP64 as well. But on a Raspberry Pi (and probably on a x64 device as well), neural models cannot be run with FP64 precision in realtime. An ML-based amp model consumes bout 45% of available CPU bandwidth running with FP32 precision. Running with FP64 precious would add least quadruple that. As a point of interest, matrix multiplies running on a Raspberry Pi 4 Arm Cortex A72 are almost completely limited by memory bandwidth to L2 cache and main memory. And that performance is (mostly) constrained by the tile size used in the matrix multiples, which (when using A72 neon registers) is constrained by the number of neon registers available. I believe that performance would roughly increase linearly as a function of available tile size. Whether it's linear or not depends a bit on how well matrix units deal with Nx1 matrices (vectors). Although the to perform NxM matrix multiples dominates, a significant amount of execution time also gets spent doing Nx1 and/or vector processing. Whether the corresponding performance boost is good enough to allow realtime audio processing at FP64.... the only way to find out would be to do it. * Results based on extensive optimization and profiling of Toob ML and TooB Neural Amp Modeler guitar effects hosted by [PiPedal](https://rerdavies.github.io/pipedal/ https://rerdavies.github.io/pipedal/)

GeekyBear 2y ago

Matrix multiplication is very commonly used in science and engineering, not just machine learning. The neural engine is optimized for machine learning use cases. This standardized successor to AMX is more general purpose than the neural engine and has much improved matrix multiplication performance vs NEON. As a bonus, since this is no longer just an experimental implementation of a matrix unit, you get documented access to the new ARM standardized low level instruction set.

brigade 2y ago

The neural engine by design cannot handle all possible kernels, and the GPU is significantly slower for integer math, and cannot do fp64. Then for the iPhone SoCs with 4 or 5 core GPUs, the GPU is a bit slower for fp16 and fp32 too.

lxgr 2y ago

> if not, there also is a dedicated matrix unit there called AMX This seems to be the successor to AMX.