3 ms·

Thanks! And sorry for not yet merging many of those. The problem is, I'm dealing with tensor parallelism for the CUDA and Metal-RDMA fork right now, so was not

• zozbot234 3mo ago
  I don't want to sound overly dismissive, but for quite a few practical cases pipeline parallelism with micro-batches will be a likely win over tensor parallelism. Of course this inherently comes with a requirement to take batched inference seriously for local use, at least in special cases where this doesn't put too much of a requirement on memory capacity. By comparison, tensor parallelism is probably good wrt. making memory- and KV-cache hungry models like GLM 5.2 and perhaps Kimi genuinely viable in a non-trivial local inference scenario.
• kamranjon 3mo ago
  Oh that’s great - I’m generally more interested in single machine use cases but the RDMA stuff is super interesting. I do wonder if you are planning to eventually delegate some of the merging responsibilities? I see some other projects follow a similar pattern when they grow to a certain size.