13 ms·

I hope popularity of large models like this one drives more work on CPU inference of quantized models. It is extremely disappointing one can't run 4 or even 8 b

I hope popularity of large models like this one drives more work on CPU inference of quantized models. It is extremely disappointing one can't run 4 or even 8 bit quantized models on a cpu. Inference I did with fp32 on a last gen AVX2 CPU show me it is definitely usable if you're willing to wait a bit longer for each token (I got about 1token per 2s on a ryzen 3700x, 32GB ram, with falcon-7B-instruct and this is with about 1gb of ram in the swap). I don't quite understand why people aren't working on cpu quantization. Allegedly openvino supports _some_ cpu quantization, but certainly not 4 bit. Bitsandbytes is gpu only. Why? Is there any technical reasons? I recently checked and for a price of a 24gb rtx3090 I can get a really nice cpu (ryzen 9 5950x) and max it with 128gb of ram. I'd love to be able to use it for int8 or 4 bit inference...

huac 3y ago
because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications.
- mark_l_watson 3y ago
  Or, inference that can fluidly use available CPUs, GPUs, and neural units on M1/M2/M3.
- wmf 3y ago
  Not if a 96 GB GPU costs $30K?
  - huac 3y ago
    The OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference. (to be fair, you can do better than the OP's performance, but we _are_ talking several magnitudes of performance, not to mention energy efficiency, which matters at scale)
    - qeternity 3y ago
      > Equivalent inference on a 4090 ($1600) is up to 400 tok/s What? Who's doing 400 t/s @ 7b on a 4090?
      - huac 3y ago
        I am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;). batch size 1 is more like 200 tok/s, still a very large improvement. the reason why this is still an interesting comparison is that it's comparing consumer hardware vs scaled datacenter setups. datacenter setups are much more efficient in large part because they are designed to process many inputs at once, fully utilizing the hardware. there are advantages to scale in inference, for the gpu-rich.
logicchains 3y ago
>It is extremely disappointing one can't run 4 or even 8 bit quantized models on a cpu What do you mean? Llama.cpp can do 8 and 4 bit quantisation on CPU, and even supports Falcon 40B.
- Roark66 3y ago
  Yes on arm. Sorry I meant the cpu (x86_64) not a cpu. :-D
  - brucethemoose2 3y ago
    Its quite fast on x86. llama.cpp looks really good on Mac ARM CPUs because: - they have tons of memory bandwidth - they have a really good proprietary acceleration library (accelerate) But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.
    - summarity 3y ago
      Ampere has native acceleration (Ampere Optimised Frameworks) for AI workloads
      - brucethemoose2 3y ago
        I dont think llama.cpp integrates this though. Accelerate is all but depreciated because of the Metal backend anyway.
    - api 3y ago
      These things are quite memory bandwidth bound, so if you are getting any kind of machine to run them go for the absolute fastest RAM you can.
    - deleted 3y ago
      [deleted]
  - gliptic 3y ago
    I told you a while ago that llama.cpp supports 4/8-bit (and other) quantization just fine (and fast) on x86_64.
brucethemoose2 3y ago
https://github.com/ggerganov/ggml https://github.com/ggerganov/ggml TinyGrad is also targeting CPU inference, and IIRC it works ok in Apache TVM. One note is that prompt ingestion is extremely slow on CPU compared to GPU. So short prompts are fine (and tokens can be streamed once the prompt is ingested), but long prompts feel extremely sluggish. Another is that CPUs with more than 128-bit DDR5 memory busses are very expensive, and CPU token generation is basically RAM bandwidth bound.
appplication 3y ago
I would be willing to bet Apple will be one to invest in CPU-based work in this space
snailmailman 3y ago
I’ve been using llama.cpp to try out some large-ish models on my cpu for a while now. I’m on an amd 3600, with 32gb ram. I can run 30B 4bit and 5bit models. It’s definitely “usable” but smaller models definitely run faster. I don’t have a good graphics card, but it does work on cpu. With more ram I think the 65B and larger models are runnable as well

huac 3y ago

because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications.

mark_l_watson 3y ago
Or, inference that can fluidly use available CPUs, GPUs, and neural units on M1/M2/M3.
wmf 3y ago
Not if a 96 GB GPU costs $30K?
- huac 3y ago
  The OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference. (to be fair, you can do better than the OP's performance, but we _are_ talking several magnitudes of performance, not to mention energy efficiency, which matters at scale)
  - qeternity 3y ago
    > Equivalent inference on a 4090 ($1600) is up to 400 tok/s What? Who's doing 400 t/s @ 7b on a 4090?
    - huac 3y ago
      I am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;). batch size 1 is more like 200 tok/s, still a very large improvement. the reason why this is still an interesting comparison is that it's comparing consumer hardware vs scaled datacenter setups. datacenter setups are much more efficient in large part because they are designed to process many inputs at once, fully utilizing the hardware. there are advantages to scale in inference, for the gpu-rich.

logicchains 3y ago

>It is extremely disappointing one can't run 4 or even 8 bit quantized models on a cpu What do you mean? Llama.cpp can do 8 and 4 bit quantisation on CPU, and even supports Falcon 40B.

Roark66 3y ago
Yes on arm. Sorry I meant the cpu (x86_64) not a cpu. :-D
- brucethemoose2 3y ago
  Its quite fast on x86. llama.cpp looks really good on Mac ARM CPUs because: - they have tons of memory bandwidth - they have a really good proprietary acceleration library (accelerate) But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.
  - summarity 3y ago
    Ampere has native acceleration (Ampere Optimised Frameworks) for AI workloads
    - brucethemoose2 3y ago
      I dont think llama.cpp integrates this though. Accelerate is all but depreciated because of the Metal backend anyway.
  - api 3y ago
    These things are quite memory bandwidth bound, so if you are getting any kind of machine to run them go for the absolute fastest RAM you can.
  - deleted 3y ago
    [deleted]
- gliptic 3y ago
  I told you a while ago that llama.cpp supports 4/8-bit (and other) quantization just fine (and fast) on x86_64.

brucethemoose2 3y ago

https://github.com/ggerganov/ggml https://github.com/ggerganov/ggml TinyGrad is also targeting CPU inference, and IIRC it works ok in Apache TVM. One note is that prompt ingestion is extremely slow on CPU compared to GPU. So short prompts are fine (and tokens can be streamed once the prompt is ingested), but long prompts feel extremely sluggish. Another is that CPUs with more than 128-bit DDR5 memory busses are very expensive, and CPU token generation is basically RAM bandwidth bound.

appplication 3y ago

I would be willing to bet Apple will be one to invest in CPU-based work in this space

snailmailman 3y ago

I’ve been using llama.cpp to try out some large-ish models on my cpu for a while now. I’m on an amd 3600, with 32gb ram. I can run 30B 4bit and 5bit models. It’s definitely “usable” but smaller models definitely run faster. I don’t have a good graphics card, but it does work on cpu. With more ram I think the 65B and larger models are runnable as well