The UIs/containers people use for 4090s specifically have some kind of performance issue with Stable Diffusion, last I checked. It runs no faster than a 3090.

The UIs/containers people use for 4090s specifically have some kind of performance issue with Stable Diffusion, last I checked. It runs no faster than a 3090. I think this can be solved by running RTX 4000 cards with CUDA 12.1 and pytorch nightly cu121... But thats a very uncommon setup. Also, if we are talking cloud deployments, cheap CPU instances with 32GB-64GB of RAM are a somewhat sane way to test LLMs, thanks to llama.cpp. A GPU will give you better throughput, of course. But CPU support for Falcon and MPT specifically are in flux, see this post for instance: https://huggingface.co/TheBloke/mpt-30B-instruct-GGML https://huggingface.co/TheBloke/mpt-30B-instruct-GGML