4 ms·
I would like to know how fast this is compared to a model shared on 3 A100s with 32 GBs of VRAM each. What I am also interested in, is why model sharding has t
by dauertewigkeit 3y ago
I would like to know how fast this is compared to a model shared on 3 A100s with 32 GBs of VRAM each.
What I am also interested in, is why model sharding has to be done manually. It seems like, one should be able to write a framework that will take your forward step and distribute the amount of layers on the available GPUs, automatically. But I haven't come across such a framework yet.
- ttul 3y agoJax has auto-sharding: https://jax.readthedocs.io/en/latest/notebooks/Distributed_arrays_and_automatic_parallelization.html https://jax.readthedocs.io/en/latest/notebooks/Distributed_a...
- simon_acca 3y ago> take your forward step and distribute the amount of layers on the available GPUs, automatically This is part of the value proposition of Mojo, Chris Lattner’s latest project. The compiler infrastructure is still in its infancy, but looks promising: https://www.modular.com/mojo https://www.modular.com/mojo
- josephg 3y agoProbably much faster. PCIe is the limiting factor in a 3x A100 setup. Having all that memory directly accessible by the GPU will make a massive difference. Even the massive 480 gigs of cpu memory in this architecture can be accessed many times faster than pcie allows.
- touisteur 3y agoHopefully if you have 3x A100 you have bridged them with NVLink and that's your bottleneck. If you want more bandwith you'll need a NVSwitch. Or a H100 GPU (Lovelace didn't get NVLink, not even the A40-little-NVLink-port - or PCIe5 for that matter) with next gen NVLink.
- yeahwhatever10 3y agoDeepSpeed?