3 ms·
Depends on your CPU. I once tried 70b llama on 256 thread Epyc, it was around 1/10 of A100 (80GB) speed.
by p1esk 2y ago
Depends on your CPU. I once tried 70b llama on 256 thread Epyc, it was around 1/10 of A100 (80GB) speed.
- logicallee 2y agohow much disk space did it use?
- p1esk 2y agoI didn’t check, but iirc it was an fp16 model checkpoint which we converted to int8 for inference, so I assume 140GB?
- logicallee 2y agoThanks, I finished downloading it (which took many hours) onto an external hard drive (by adding a HF_HOME environmental variable for where to store that cache). Its size was 262 GB.