20 ms·

I run CodeLlama70B in my home lab using two RTX 4090s and the MLC AI framework. It provides Muti-GPU support and is extremely fast on consumer grade hardware. I

I run CodeLlama70B in my home lab using two RTX 4090s and the MLC AI framework. It provides Muti-GPU support and is extremely fast on consumer grade hardware. I'm seeing about 30 tokens/sec. https://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Inference-on-Multiple-NVDIA-AMD-GPUs https://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infer...

yumraj 3y ago
How much is the power consumption?
- KingOfCoders 3y ago
  ~1kW I guess.
  - MuffinFlavored 3y ago
    1kW under load for 30 tokens/sec and each request is about 100-1000 tokens and each response is about 100-1000 tokens (so a response takes about 3-33s)
- lhl 3y ago
  I have a similar system. When idle, nvtop reports about 11-12W/4090. When doing bs=1 inference, the default PL for my cards is 450W each, although I've found that I can keep 97% of prefill and 99%+ of inference performance lowering the power limit to 400W. With system overhead, basically I'd guess 50W when idle, 900W when actively working.
  - MuffinFlavored 3y ago
    > 900W when actively working. I'm trying to think of my typical ChatGPT conversations and the flow of them. I probably ask it anywhere between 150 and 1,000 tokens, and then I wait for it to respond to me about 150 to 3,000 tokens At 30 tokens a second, responses can be "900W" for about 5s to 100s Then as a human you have to sit there and read what it says (system doesn't need to generate tokens then as you interpret them). I'm trying to think what that usage graph looks like/what it translates roughly to in terms of full usage for an hour so you can estimate daily usage in kilowatt hour to quantify how intensive peak 900w load really is in real world use...
refulgentis 3y ago
I'm kinda stunned at that speed. I wonder if quantized models would affect that, i.e. GGUFs via llama.cpp. My M2 MBP would beat that, but, M2s aren't that great, like, my 3090s faster.
2099miles 3y ago
I run llama chat 70b on a p3 8x large (4 Tesla) and it runs at like 1-5 tokens per sec. And I’m running the model with only 4 bit precision. Are you doing anything else?

yumraj 3y ago

How much is the power consumption?

KingOfCoders 3y ago
~1kW I guess.
- MuffinFlavored 3y ago
  1kW under load for 30 tokens/sec and each request is about 100-1000 tokens and each response is about 100-1000 tokens (so a response takes about 3-33s)
lhl 3y ago
I have a similar system. When idle, nvtop reports about 11-12W/4090. When doing bs=1 inference, the default PL for my cards is 450W each, although I've found that I can keep 97% of prefill and 99%+ of inference performance lowering the power limit to 400W. With system overhead, basically I'd guess 50W when idle, 900W when actively working.
- MuffinFlavored 3y ago
  > 900W when actively working. I'm trying to think of my typical ChatGPT conversations and the flow of them. I probably ask it anywhere between 150 and 1,000 tokens, and then I wait for it to respond to me about 150 to 3,000 tokens At 30 tokens a second, responses can be "900W" for about 5s to 100s Then as a human you have to sit there and read what it says (system doesn't need to generate tokens then as you interpret them). I'm trying to think what that usage graph looks like/what it translates roughly to in terms of full usage for an hour so you can estimate daily usage in kilowatt hour to quantify how intensive peak 900w load really is in real world use...

refulgentis 3y ago

I'm kinda stunned at that speed. I wonder if quantized models would affect that, i.e. GGUFs via llama.cpp. My M2 MBP would beat that, but, M2s aren't that great, like, my 3090s faster.

2099miles 3y ago

I run llama chat 70b on a p3 8x large (4 Tesla) and it runs at like 1-5 tokens per sec. And I’m running the model with only 4 bit precision. Are you doing anything else?