3 ms·
I was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".
by npteljes 3y ago
I was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".
- amilios 3y agoIs this with the 4-bit quantization? The only issue with it is that inference is incredibly slow with it on right now, but that should be fixed up in the next few weeks I think.
- netsec_burn 3y agoI'm running it on Serge with 51 GB of RAM. I don't think it requires any GPU memory (llama.cpp runs on the CPU), I have a RTX 2060 in the system I'm running it on. Correct me if I'm wrong.
- forgingahead 3y agoHow many tokens per second are you getting with your setup?
- netsec_burn 3y agoAbout .5 tokens per second if I had to guess.