In Stable Diffusion land, onnx performance was not that great compared to ML compilers and some other implementations (at least when I tried it). Also, llama.c

In Stable Diffusion land, onnx performance was not that great compared to ML compilers and some other implementations (at least when I tried it). Also, llama.cpp is excellent at splitting the workload between CPU and accelerators since its so CPU focused. You can run 13B or 33B in a 6GB GPU and still get some acceleration. Also, as said above, quantization. That is make or break. There is no reason to run a 7B model at fp16 when you can run 13B or 30B in the same memory pool at 2-5 bits.