3 ms·

> No, it can't run at all. https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg EDIT: This was ran on a 1080ti + 5900x.

by XCSme 3y ago

> No, it can't run at all. https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg EDIT: This was ran on a 1080ti + 5900x. Initial generation takes around 10-30seconds (like it has to upload the model to GPU), but then it starts answering immediately, at around 3 words per second.

wokwokwok 3y ago
Did you check your GPU utilization? Typically when it runs that way it runs on the CPU, not the GPU. Are you sure you're actually offloading any work to the GPU? At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all. The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#additional-options https://github.com/ggerganov/llama.cpp/blob/master/examples/... It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.
- XCSme 3y ago
  I have to check, something does indeed seem weird, especially with the PC freezing like that. Maybe it runs on the CPU. > quantized version Yes, it is 4bit quantized, but still has 24.6GB
spxneo 3y ago
this is some new flex to debate online: copying and pasting the other sides argument and waiting for your local LLM to explain why they are wrong. how much is your hardware at today's value? what are the specs? that is impressive even though its 3 words per second. if you want to bump it up to 30, do you then 10x your current hardware cost?
- XCSme 3y ago
  That question was just an example (Lorem ipsum), it was easy to copy paste to demo the local LLM, I didn't intend to provide more context to the discussion. I ordered a 2nd 3090, which has 24GB VRAM. Funny how it was $2.6k 3 years ago and now is $600. You can probuild a decent AI local machine for around $1000.
  - spxneo 3y ago
    https://howmuch.one/product/average-nvidia-geforce-rtx-3090-24gb/price-history https://howmuch.one/product/average-nvidia-geforce-rtx-3090-... you are right there is a huge drop in price
    - XCSme 3y ago
      New it's hard to find, but the 2nd hand market is filled with them.
  - taneq 3y ago
    Where are you seeing 24GB 3090s for $600?
    - XCSme 3y ago
      2nd hand market
llm_trw 3y ago
Congratulations on using CPU inference.

by XCSme 3y ago

wokwokwok 3y ago

Did you check your GPU utilization? Typically when it runs that way it runs on the CPU, not the GPU. Are you sure you're actually offloading any work to the GPU? At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all. The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#additional-options https://github.com/ggerganov/llama.cpp/blob/master/examples/... It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.

XCSme 3y ago
I have to check, something does indeed seem weird, especially with the PC freezing like that. Maybe it runs on the CPU. > quantized version Yes, it is 4bit quantized, but still has 24.6GB

spxneo 3y ago

this is some new flex to debate online: copying and pasting the other sides argument and waiting for your local LLM to explain why they are wrong. how much is your hardware at today's value? what are the specs? that is impressive even though its 3 words per second. if you want to bump it up to 30, do you then 10x your current hardware cost?

XCSme 3y ago
That question was just an example (Lorem ipsum), it was easy to copy paste to demo the local LLM, I didn't intend to provide more context to the discussion. I ordered a 2nd 3090, which has 24GB VRAM. Funny how it was $2.6k 3 years ago and now is $600. You can probuild a decent AI local machine for around $1000.
- spxneo 3y ago
  https://howmuch.one/product/average-nvidia-geforce-rtx-3090-24gb/price-history https://howmuch.one/product/average-nvidia-geforce-rtx-3090-... you are right there is a huge drop in price
  - XCSme 3y ago
    New it's hard to find, but the 2nd hand market is filled with them.
- taneq 3y ago
  Where are you seeing 24GB 3090s for $600?
  - XCSme 3y ago
    2nd hand market

llm_trw 3y ago

Congratulations on using CPU inference.