2 ms·
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even
by nullc 2mo ago
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
- ericd 2mo agoGood point. It's much more of an issue when running dense models with tensor parallelism. In that case, I'd look for an MoE model instead.