4 ms·
I feel like this is the greatest demand for LLMs at the moment too. It's hard to believe we're only 8 months into this industry, so I imagine we'll start seein
by minsc_and_boo 3y ago
I feel like this is the greatest demand for LLMs at the moment too.
It's hard to believe we're only 8 months into this industry, so I imagine we'll start seeing smaller footprints soon.
- simbolit 3y ago8 months from what point? Gpt3 is 36 months old. Dalle-e is 28 months old. Even StableDiffusion is like 11 months old.
- minsc_and_boo 3y agoFair, I should have said 8 months since the market exploded.
- brucethemoose2 3y agoWe already do. MLC-LLM and Llama.cpp have Vulkan/OpenCL/Metal 3 bit implementations. That can run llama 7B (or maybe even 13b?) in 8GB. TBH devices just need more ram for coherent output though. Llama 13b and 33b are so much "smarter" and more coherent than 7B with 3 bit quant.
- TeddyDD 3y ago13b q5 llama report „total VRAM used: 8321 MB” so 3bit will most likely fit into 8GB.