3 ms·
I looked into this a bit last week. The OPT-30b model seems close to GPT-3 in capabilities, and running on 4x 24GB NVIDIA A10G GPUs, can generate 90 tokens per
by generalizations 4y ago
I looked into this a bit last week. The OPT-30b model seems close to GPT-3 in capabilities, and running on 4x 24GB NVIDIA A10G GPUs, can generate 90 tokens per second. I wonder how fast that model could run on 4x 24GB NVIDIA P40 GPUs - which currently cost about $110 on ebay. It'd be a lot slower, but even if it's 1/90th the speed, that's still 1 token per second.
- danielbln 4y ago1 token/s is way too slow for dialog, especially since a token isn't even a word, but often part of a word. 1 t/s might be sufficient for asynchronous processing, but if you want a chatGPT like therapy dialog, then that's not good enough.
- generalizations 4y agoDo we know what the speed difference actually is? I'm not sure what benchmarks would measure that. My best plan so far is to just run a smaller model on one of the GPUs and see how long it takes.