4 ms·
This is one of the reasons I'm looking forward to being able to run a language model with similar capabilities to GPT-3 entirely on hardware I control - like I
by simonw 4y ago
This is one of the reasons I'm looking forward to being able to run a language model with similar capabilities to GPT-3 entirely on hardware I control - like I can today with Stable Diffusion.
My current impression is that this is a few years away, due to the intense compute requirements for that scale of model. But I think it will happen eventually.
- generalizations 4y agoI looked into this a bit last week. The OPT-30b model seems close to GPT-3 in capabilities, and running on 4x 24GB NVIDIA A10G GPUs, can generate 90 tokens per second. I wonder how fast that model could run on 4x 24GB NVIDIA P40 GPUs - which currently cost about $110 on ebay. It'd be a lot slower, but even if it's 1/90th the speed, that's still 1 token per second.
- danielbln 4y ago1 token/s is way too slow for dialog, especially since a token isn't even a word, but often part of a word. 1 t/s might be sufficient for asynchronous processing, but if you want a chatGPT like therapy dialog, then that's not good enough.
- generalizations 4y agoDo we know what the speed difference actually is? I'm not sure what benchmarks would measure that. My best plan so far is to just run a smaller model on one of the GPUs and see how long it takes.
- theptip 4y agoAgreed. It's promising that Apple in particular are working on ways to run models like Stable Diffusion locally on mobile device hardware. I think they probably have the best chance of making that happen, given their vertical integration and hardware-centric / non-ad-based economic model. One speculative possiblity I haven't seen discussed much -- up until now, we've not seen large-scale work on hardware implementation of these NN models, since the frontier of capabilities has been advancing so fast. However, perhaps we're now approacing the first point where a baked-in-silicon (or at least part-baked) model might be economically viable. Especially if someone can figure out a way to efficiently fine-tune such a model (something like an overlay filesystem), or perhaps something like a FPGA where the underlying logical structure can be reprogrammed.