3 ms·
... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost o
by segmondy 3y ago
... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.
- redox99 3y agoHow? 24GB VRAM can only handle 30B 4bit. Are you offloading some layers to the CPU? That kills performance AFAIK.
- danpalmer 3y agoGPT-3.5 is up to 175B parameters, GPT-4 (which is what OP is asking for) has been speculated as having 1T parameters, although that seems a little high to me. It's easy to run a much worse model on much worse hardware, but there's a reason why it's only companies with huge datacenter investments running the top models.
- ukuina 3y agoThe 1T number is wild hype. It's likely GPT4 is actually a mixture-of-experts with smaller models under the hood.
- humanistbot 3y ago30B isn't big anymore. GPT-4 is rumored to have 1T parameters.
- jbellis 3y agoDon't 4090 and 3090 have the same amount of vram?