4 ms·
Any sense of speed? My assumption is that shuttling the weights in/out of the GPU is slow. Does GPU load + processing beat an entirely CPU solution? Doubly so i
by 0cf8612b2e1e 2y ago
Any sense of speed? My assumption is that shuttling the weights in/out of the GPU is slow. Does GPU load + processing beat an entirely CPU solution? Doubly so if it is a huge model where the model cannot sit fully in RAM?
- p1esk 2y agoDepends on your CPU. I once tried 70b llama on 256 thread Epyc, it was around 1/10 of A100 (80GB) speed.
- logicallee 2y agohow much disk space did it use?
- p1esk 2y agoI didn’t check, but iirc it was an fp16 model checkpoint which we converted to int8 for inference, so I assume 140GB?
- logicallee 2y agoThanks, I finished downloading it (which took many hours) onto an external hard drive (by adding a HF_HOME environmental variable for where to store that cache). Its size was 262 GB.