3 ms·
As the hype is dying down it's becoming a little bit clearer that AI isn't like blockchain and might be actually useful (for non generative purposes at least)
by loyalcinnamon 11mo ago
As the hype is dying down it's becoming a little bit clearer that AI isn't like blockchain and might be actually useful (for non generative purposes at least)
I'm curious what counts as a midsize model; 4B, 8B, or something larger/smaller?
What models would you recommend? I have 12GB of vram so anything larger than 8B might be really slow, but i am not sure
- DSingularity 11mo agoIt can depend on your use case. Are you editing a large code base and will thus make lots of completion requests with large contexts?
- riskable 11mo agoMy take: Large: Requires >128GB VRAM Medium: 32-128GB VRAM Small: 16GB VRAM Micro: Runs on a microcontroller or GPUs with just 4GB of VRAM There's really nothing worthwhile for general use cases that runs in under 16GB (from my testing) except a grammar-checking model that I can't remember the name of at the moment. gpt-oss:20b runs on 16GB of VRAM and it's actually quite good (for coding, at least)! Especially with Python. Prediction: The day that your average gaming PC comes with 128GB of VRAM is the day developers will stop bothering with cloud-based AI services. gpt-oss:120b is nearly as good as gpt5 and we're still at the beginning of the AI revolution.