2 ms·
I can run GPT-3.5 level models on my laptop (albeit a chunky high end one). It is indeed only a matter of time.
by api 3y ago
I can run GPT-3.5 level models on my laptop (albeit a chunky high end one). It is indeed only a matter of time.
- bugglebeetle 3y agoI’ve unfortunately not found an open source model that can perform at the this level for anything beyond madlibs-style text generation. Most of my prompts are a combination of sequential tasks and one-shot prompting, typically between 1K-2K tokens and everything I’ve tried falls down at this. Granted, I haven’t gone beyond 30B models, but the gulf still seems quite wide (and even wider compared to GPT-4). Really hope it does close in the next year!
- api 3y agoI admit I have not pushed them that far, though I am also using 40B and 65B models which may perform better than a 30B model. Nothing open approaches GPT-4 yet, which I have heard is an overlay of multiple 220B models. The only way you will see that on even high end consumer hardware is advances in compression and pruning, but those are happening. There’s a lot of research in those areas. Of course then cloud models could use the same techniques to get even bigger, if the model and training will keep scaling. Edit: as AI matures you may also see lots of custom ASICs hit the market. I could see those getting cheap enough to run many hundred B models locally. The acceleration of an ASIC designed for a specific type of model over CPU and GPU could be dramatic, reducing the amount of silicon you need.
- smoldesu 3y agoIn my experience, Wizard-Vicuna output at 13b will match or exceed the prose quality of GPT3.5 in smaller generation tasks. I've been using it for a Discord bot to great success.