3 ms·
> llama.cpp which runs a 13 billion parameter model on a 6GB GPU I think that's a typo there too, the 13B model needs like 10G of memory for 4 bits, it's the
by moffkalast 3y ago
> llama.cpp which runs a 13 billion parameter model on a 6GB GPU
I think that's a typo there too, the 13B model needs like 10G of memory for 4 bits, it's the 7B one that fits into 6G. Well unless you do the split thing with some layers on the CPU I guess.
- DANmode 3y agohttps://news.ycombinator.com/item?id=35937505 https://news.ycombinator.com/item?id=35937505
- moffkalast 3y agoYeah that's the split layer mode I mentioned. With 6G one can do about 18 layers, which is less than half of the 40 total for 13B.