4 ms·
Can anyone suggest what's the cheapest hardware to run this model locally with a reasonable performance?
by thw_9a83c 1y ago
Can anyone suggest what's the cheapest hardware to run this model locally with a reasonable performance?
- daemonologist 1y agoSince there's no quantized version available at the moment, you'll need ~20 GB of memory for the weights plus some extra for the KV cache. CPU with 32 GB RAM will be the cheapest and still reasonably fast given the relatively small number of activated parameters.
- thw_9a83c 1y agoThank you! I don't even know what a "quantized version" is, but I was expecting answers about NVIDIA graphics cards and their memory. My computer has 24GB of memory, but I'll go for 64GB to run this locally on a new computer.