3 ms·
Depends on what you consider vaguely useful, but any modern GPU which can fit the weights at a reasonable quantization (so, 16+ GB, maybe 12 in a pinch) will pr
by daemonologist 2y ago
Depends on what you consider vaguely useful, but any modern GPU which can fit the weights at a reasonable quantization (so, 16+ GB, maybe 12 in a pinch) will probably be tolerable. Bare minimum is probably ~3060 12GB.
- apothegm 2y agoThanks! So this could actually be worth running on a mid-specced recentish Mac laptop?
- daemonologist 2y agoMacs are a bit odd because you're using system memory (so, more but slower) and Apple's APIs, but it's definitely worth a try. I'd guess >= M2 Pro + >= 24 GB RAM would do okay. Also maybe try an MLX version? (e.g. https://huggingface.co/justinmeans/DeepCoder-14B-Preview-mlx-4Bit https://huggingface.co/justinmeans/DeepCoder-14B-Preview-mlx...)
- apothegm 2y agoSweet. Thank you so much!