3 ms·
If you want to run them fast, a 12GB GPU (e.g 3060) for the 13B and a 24GB GPU for the 34B (e.g 3090). Otherwise llama.cpp CPU inference would work on most mach
by redox99 3y ago
If you want to run them fast, a 12GB GPU (e.g 3060) for the 13B and a 24GB GPU for the 34B (e.g 3090). Otherwise llama.cpp CPU inference would work on most machines.