3 ms·

It's not a new technique at this point, but it still blows my mind that you can quantize a model trained at fp16 down to 4 bits, and still have coherent inferen

It's not a new technique at this point, but it still blows my mind that you can quantize a model trained at fp16 down to 4 bits, and still have coherent inference results. > While running, the model uses about 4GB of RAM and Activity Monitor shows it using 748% CPU - which makes sense since I told it to use 8 CPU cores. > I imagine it's possible to run a larger model such as 13B on this hardware, but I've not figured out how to do that yet. Naively it seems like you could repeat the same process outlined in the article but with "13B" in place of "7B". What's the catch?

summarity 4y ago
No catch, just works. 30B works fine on an M1 Max with 64GB of RAM, had to go for the M1 Ultra at 128GB for 65B.
- detrites 4y ago
  What's the tokens/s on those?
  - minxomat 4y ago
    With 16 threads, about 140ms per token for 30B, 300ms per token for 65B I should also mention that 65B should be able to run on 64GB systems. Total system memory consumption on M1 Ultra is about 67GB when running nothing else.
- bojangleslover 4y ago
  I'm not able to run 13B and from his wiki: > Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models.
  - minxomat 4y ago
    This has been fixed almost 2 days ago now. It’s literally mentioned at the top of the repo.
  - simonw 4y ago
    This commit landed 7 hours ago (since I wrote my TIL): https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6eb56678fdee4c09219ddb85b640 https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e...
- xiphias2 4y ago
  You have both at home / work?
  - minxomat 4y ago
    A laptop and a desktop (Mac Studio)
- cjbprime 4y ago
  I was wondering if Apple Silicon would be uniquely suited for high-GPU-RAM tasks because it shares memory across the system. But I guess in this case it's a CPU model, so that's unrelated. Is that right? Do you think you could run these models on GPU instead?
raihansaputra 4y ago
I think it's about the quantization process and loading the model. I needed just above 64GB of RAM + Swap to quantize the 35B model to int4. Not to mention much slower inference time.
- montebicyclelo 4y ago
  Hmm, wonder if the whole model is being loaded into memory for the quantisation, and whether that's necessary. (Also, shame the models can't be distributed in 4bit, (due to the license).)
  - raihansaputra 4y ago
    Yeah it's basically wholly loaded to memory for quantisation. Some optimization should be possible, and yes the quantized models can't be distributed sadly.
eternalban 4y ago
My intuition for this is that a high dimensional vector with 16bits per dimension reduced to 4bits would be ok if these are effectively low cardinality dimensions and even with 4 bits things can be distinctly mapped into the vector space. It should break down on high cardinality dimensions (say things which are only distinguished in a few significant dimensions & rest being effectively zerovalue). In otherwords, given a vector of say 12288 dimensions (GPT), a 4-bit dimension, if vectors were uniformly distributed in the embedding space, is a choice space of 16^12288. That's in 4 bits. The 16-bit space is huge. I think serious errors will crop up only if we're looking at items that cluster in a small subset 'd' of those 12288 dimensions. So at some small d, 16^d will result in vector collisions for certain type or category of inputs.
aledalgrande 4y ago
What's the best read on quantization of a model? Thx
akira2501 4y ago
> and still have coherent inference results. With language being as expressive as it is, I'm not sure why people consider highly subjective measure of "coherence" to be a selling point? You can generate random text and get semi-coherent sentences a surprising amount of the time. Why not accuracy?
- zamadatix 4y ago
  You probably want more than one measure for a meaningful conclusion. I'm not sure how you can generate random text and get semi-coherent sentences though. Even bending that to allow markov chains and you get clearly incoherent thoughts. On the flip side it's possible to program in perfect clarity and accuracy but it's hard to do that without making the output feel like you're talking with a spreadsheet instead of having a conversation.

summarity 4y ago

No catch, just works. 30B works fine on an M1 Max with 64GB of RAM, had to go for the M1 Ultra at 128GB for 65B.

detrites 4y ago
What's the tokens/s on those?
- minxomat 4y ago
  With 16 threads, about 140ms per token for 30B, 300ms per token for 65B I should also mention that 65B should be able to run on 64GB systems. Total system memory consumption on M1 Ultra is about 67GB when running nothing else.
bojangleslover 4y ago
I'm not able to run 13B and from his wiki: > Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models.
- minxomat 4y ago
  This has been fixed almost 2 days ago now. It’s literally mentioned at the top of the repo.
- simonw 4y ago
  This commit landed 7 hours ago (since I wrote my TIL): https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6eb56678fdee4c09219ddb85b640 https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e...
xiphias2 4y ago
You have both at home / work?
- minxomat 4y ago
  A laptop and a desktop (Mac Studio)
cjbprime 4y ago
I was wondering if Apple Silicon would be uniquely suited for high-GPU-RAM tasks because it shares memory across the system. But I guess in this case it's a CPU model, so that's unrelated. Is that right? Do you think you could run these models on GPU instead?

raihansaputra 4y ago

I think it's about the quantization process and loading the model. I needed just above 64GB of RAM + Swap to quantize the 35B model to int4. Not to mention much slower inference time.

montebicyclelo 4y ago
Hmm, wonder if the whole model is being loaded into memory for the quantisation, and whether that's necessary. (Also, shame the models can't be distributed in 4bit, (due to the license).)
- raihansaputra 4y ago
  Yeah it's basically wholly loaded to memory for quantisation. Some optimization should be possible, and yes the quantized models can't be distributed sadly.

eternalban 4y ago

My intuition for this is that a high dimensional vector with 16bits per dimension reduced to 4bits would be ok if these are effectively low cardinality dimensions and even with 4 bits things can be distinctly mapped into the vector space. It should break down on high cardinality dimensions (say things which are only distinguished in a few significant dimensions & rest being effectively zerovalue). In otherwords, given a vector of say 12288 dimensions (GPT), a 4-bit dimension, if vectors were uniformly distributed in the embedding space, is a choice space of 16^12288. That's in 4 bits. The 16-bit space is huge. I think serious errors will crop up only if we're looking at items that cluster in a small subset 'd' of those 12288 dimensions. So at some small d, 16^d will result in vector collisions for certain type or category of inputs.

aledalgrande 4y ago

What's the best read on quantization of a model? Thx

akira2501 4y ago

> and still have coherent inference results. With language being as expressive as it is, I'm not sure why people consider highly subjective measure of "coherence" to be a selling point? You can generate random text and get semi-coherent sentences a surprising amount of the time. Why not accuracy?

zamadatix 4y ago
You probably want more than one measure for a meaningful conclusion. I'm not sure how you can generate random text and get semi-coherent sentences though. Even bending that to allow markov chains and you get clearly incoherent thoughts. On the flip side it's possible to program in perfect clarity and accuracy but it's hard to do that without making the output feel like you're talking with a spreadsheet instead of having a conversation.