4 ms·
> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN b
by alchemist1e9 2y ago
> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though.
Could be a big deal if it allows cluster of smaller GPUs to compete with a single large VRAM GPU.
Unfortunately I’m a few months of date - which is an eternity in LLM inference techniques - so I’m not sure what current state of distributed inference looks like.
- Palomides 2y agollama.cpp supports splitting work across multiple nodes on a network already it essentially just copies a chunk of the model to each one, works well for situations where each machine has limited vram
- vessenes 2y agoYeah, I think these methods could be baked into llama.cpp or some other higher up the toolchain python library or what have you. They shard out each layer (ish?) to the edges, and recombine that layer inference at the master node, while the outside edges load up their next bit if they need to; I would guess the devil is in the details for all the possible tensor types and architectures (for instance, how shall we implement skip layers?).