3 ms·
You still need a GPU. I'm nothing even close to a knowledgeable expert on this, but I've dabbled enough to have an idea how to answer this. The special thing
by geuis 2y ago
You still need a GPU.
I'm nothing even close to a knowledgeable expert on this, but I've dabbled enough to have an idea how to answer this.
The special thing about why GPU's are used to train models is that their architecture is directly applicable to parallelize the kind of vector math that goes on under the hood when adjusting weights between nodes in all the layers of a neural network.
The weights files (GGUF, etc) are just data describing how the networks need to be built up to be functional. Think compressed ZIP file versus uncompressed text document.
You can run a lot of models on just a cpu, buts its gonna be slooooooow. For example, I've been tweaking on running a Mixtral8x7b model on my 2019 Intel Macbook Pro with llama.cpp. It works, but it may be running at 1-2 tokens per second at max, and that's even with the limited GPU offloading I figured out how to do.
- hnfong 2y agoCPU inference is “OK” on quantized models smaller than 10B Gives at least a couple tokens per sec on fast modern CPUs. So it really depends on how fast the speed needs to be and the expected loads, compute cost and budget etc etc
- dheera 2y agoDepends on the application. If you can process "offline" for an hour and then cache the results, CPU inference is fine. GPUs are expensive.
- idontpost 2y ago> If you can process "offline" for an hour and then cache the results, CPU inference is fine. > GPUs are expensive. Depends on the GPU. I've found T4 GPUs to be cheaper than CPU compute on AWS when testing throughput per $ of spend.
- geuis 2y agoVery true. This is the approach I've been taking. My project isn't human-interactive and I'm fine with it taking a couple hours for each run to finish. The downside is that it makes development and testing super slow without the speedup you get from having local GPU power.
- deleted 2y ago[deleted]