If you want to set up an AI server for your own use, it's exceedingly easy to install LM Studio and hit the "serve an API" button. Testing performance this way

If you want to set up an AI server for your own use, it's exceedingly easy to install LM Studio and hit the "serve an API" button. Testing performance this way, I got about 0.5-1.5 tokens per second with an 8GB 4bit quantized model on an old DL360 rack-mount server with 192GB RAM and 2 E5-2670 CPUs. I got about 20-50 tokens per second on my laptop with a mobile RTX 4080.