3 ms·
If you have enough system RAM, you could make it into an AI server. This is more GPU VRAM bound more than anything else, but software like llama.cpp will let yo
by selfhoster11 3y ago
If you have enough system RAM, you could make it into an AI server. This is more GPU VRAM bound more than anything else, but software like llama.cpp will let you run mixed inference across a GPU and your CPU. Check out /r/LocalLlama on Reddit if you would like to learn more.
Of course image generation models are also an option and generally need less VRAM.