4 ms·
it's not entirely their fault, they rely on xformers, and that library is gpu only. other models will happily run on cpu only mode, depending on your environme
by avereveard 3y ago
it's not entirely their fault, they rely on xformers, and that library is gpu only.
other models will happily run on cpu only mode, depending on your environment there are super easy ways to get started, and 32 core should be ok for a llama2 13b and bearable with some patient for running 33b models. for reference I'm willingly running 13b llama2 on cpu only mode so I can leave the gpu to diffusers, and it's just enough to be generating at a comfortable reading speed.