3 ms·
There is definitely a demand for a 30B model (aka a model that will comfortably fit on 24GB GPUs (or 32GB of system RAM) and squeeze into 16GB).
by brucethemoose2 3y ago
There is definitely a demand for a 30B model (aka a model that will comfortably fit on 24GB GPUs (or 32GB of system RAM) and squeeze into 16GB).
- nullc 3y agoLlama.cpp inference on fast cpus is perfectly usable for 70B parameter models.