3 ms·
A PC with a small GPU coordinating and then llama.cpp using the llama RPC stuff (over RDMA to reduce latency) talking to the two nodes. I dunno, maybe he'll do
by cmrdporcupine 2mo ago
A PC with a small GPU coordinating and then llama.cpp using the llama RPC stuff (over RDMA to reduce latency) talking to the two nodes.
I dunno, maybe he'll do a write-up someday.
- runekaagaard 2mo agoOK thanks! Would love to read :)