3 ms·
What a misleading article. I thought they'd done some breakthrough in resource efficiency. This is just the old and slow method tools like Ollama used.
by Gloomily3819 2y ago
What a misleading article. I thought they'd done some breakthrough in resource efficiency. This is just the old and slow method tools like Ollama used.
- gavmor 2y agoWhat method is that? Layer offloading?
- Hugsun 2y agoYes, it's either that, or CPU inference. The article doesn't say. It doesn't mention quantization either.
- logicallee 2y agoDo you know how much disk space this takes total? When I ran it, it downloaded nearly 30 gigabytes of models and seemed to be on track to download 28 more 5 gigabyte chunks (for a total of 150 gigabytes of disk space or maybe more). What is the total size before it finishes?
- lostmsu 2y ago70B parameters * 2 bytes each (fp16 or bf16) = 140GB I wish models sizes were published in bytes.
- logicallee 2y agoThanks, I finished downloading it (which took many hours) onto an external hard drive (by adding a HF_HOME environmental variable for where to store that cache). Its size was 262 GB.