Although both LDM inference and encoder-decoder and decoder-only LLM inference are both fundamentally autoregressive in nature, LLM inference is memory bound an

Although both LDM inference and encoder-decoder and decoder-only LLM inference are both fundamentally autoregressive in nature, LLM inference is memory bound and LDM inference is compute bound. In that light, it makes sense that the difference between a 4090 and M1 Pro isn't as pronounced as one would expect at first approximation. Also, as you hint whisper.cpp certainly isn't one of the fastest implementations of whisper inference out there. Perhaps a comparison between a pure PyTorch version running on the 4090 with an MLX version of Whisper running on the M1 Pro would be fairer. Or better yet, run the whisper encoder on ANE with CoreML and have the decoder running with Metal and Accelerate (which uses Apple's undocumented AMX ISA) using MLX, since MLX currently does not use the ANE. IIRC, whisper.cpp has a similar optimization on Apple hardware, where it optionally runs the encoder using CoreML and the decoder using Metal.