4 ms·

llama.cpp supports splitting work across multiple nodes on a network already it essentially just copies a chunk of the model to each one, works well for situat

llama.cpp supports splitting work across multiple nodes on a network already it essentially just copies a chunk of the model to each one, works well for situations where each machine has limited vram

ay 2y ago
Any pointers to RTFM / llama repo for that ? I could not find anything on a cursory look. Thanks in advance !
- Palomides 2y ago
  https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc https://github.com/ggerganov/llama.cpp/tree/master/examples/... you run that on the remote nodes