Yeah, I think these methods could be baked into llama.cpp or some other higher up the toolchain python library or what have you. They shard out each layer (ish?

Yeah, I think these methods could be baked into llama.cpp or some other higher up the toolchain python library or what have you. They shard out each layer (ish?) to the edges, and recombine that layer inference at the master node, while the outside edges load up their next bit if they need to; I would guess the devil is in the details for all the possible tensor types and architectures (for instance, how shall we implement skip layers?).