4 ms·
> At the moment, WebGPT averages ~300ms per token on GPT-2 124M running on a 2020 M1 Mac with Chrome Canary. How do ChatGPT on GPT-3.5 / GPT-4 compare?
by MuffinFlavored 4y ago
> At the moment, WebGPT averages ~300ms per token on GPT-2 124M running on a 2020 M1 Mac with Chrome Canary.
How do ChatGPT on GPT-3.5 / GPT-4 compare?
- zamadatix 4y agoI don't have exact numbers but trying all 3 out: on a local 4090 this seems mildly slower than GPT-4 and ridiculously slower than GPT 3.5 (both via cloud GPUs of course). That said all are within the realm of usability in terms of speed though GPT 3.5 is really in a whole different class of being able to be used nearly interactively without delay.
- lxe 4y agoInteresting, I'm getting 100ms/token on plain old wasm with 4 threads via ggml, using a 1.7B quantized cerebras model.
- brucethemoose2 4y agoI can do 60 tokens in 24s on my 2020 ASUS G14, Thorium 111 + Windows 10. Nvidia-smi says my RTX 2060 is 24% loaded and has ~1.1G eaten up. Thats slower than Vicuna 7B (aka LLaMa 7000M, GPT 3ish? model) on linux on the same machine, where I get about 3.5 tokens/sec and 97% usage. So... yeah, performance is not so great yet.