2 ms·
None of the models out in the open really have GPT-3-ish capabilities, not even llama-65b. That aside, https://github.com/ggerganov/llama.cpp https://github.co
by int_19h 4y ago
None of the models out in the open really have GPT-3-ish capabilities, not even llama-65b.
That aside, https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable amount of system RAM for larger models - 20 Gb for llama-30b, 40 Gb for llama-65b. Swap is an option here, but performance with it is abysmal.
These things are huge, and I don't think there is any way around that. You can play tricks with quantization, and larger models seem to be more tolerant of it, but even if, as some claim, 65b could be quantized to 2-bit while retaining decent performance, that would still need 20 Gb of RAM (CPU or GPU) to load - never mind additional requirements for actual inference.
The breakthrough here is more likely to come from new hardware that is highly optimized for those LLMs - basically just matmul for various bitnesses all the way down to 2-bit plus as much fast memory as can be packed into it for a given price point.
- ineedasername 4y agoWhat about something that's maybe more along the lines of a GPT-2.5?
- int_19h 4y agoOf the small models, Vicuna and WizardLM seem to be the best of what's available right now. Quantized 7B models are just under 4 Gb of RAM, so they should fit on most GPUs out there; and even if not, they're "fast enough" on the CPU. But if you have enough RAM to run vicuna-13b (12 Gb should do it quite nicely), I'd start there.