2 ms·
Jeff - check out the distributed-llama project...you should be able to distribute over entire cluster
by Havoc 1y ago
Jeff - check out the distributed-llama project...you should be able to distribute over entire cluster
- burnte 1y agoHe mentioned that in the video.
- yjftsjthsd-h 1y agohttps://github.com/b4rtaz/distributed-llama https://github.com/b4rtaz/distributed-llama ?
- geerlingguy 1y agoI've been testing Exo (seems dead), llama.cpp RPC (has a lot of performance limitations) and distributed-llama (faster but has some Vulkan quirks and only works with a few models). See my AI cluster automation setup here: https://github.com/geerlingguy/beowulf-ai-cluster https://github.com/geerlingguy/beowulf-ai-cluster I was building that through the course of making this video, because it's insane how much manual labor people put into building home AI clusters :D