4 ms·
100 GPU rig would not have 512 GB in one address space, accessible by 16 GPUs. Each GPU can directly address the memory on any of the 16 GPUs.
by twtw 9y ago
100 GPU rig would not have 512 GB in one address space, accessible by 16 GPUs. Each GPU can directly address the memory on any of the 16 GPUs.
- gravypod 9y agoFor the 300k you'll save just hire a full time dev to write you a distributed implementation of your solver.
- slizard 9y agoHahha, good one. Just that by the time that one dev implements everything, you'll be overtaken by the competition -- at least that's what everyone's fear is, and it's partly warranted.
- sounds 9y agoMany problems are bandwidth-limited. More cores may indeed actually hurt performance as contention for the limiting resource makes it increasingly hard for any core to run efficiently. https://superuser.com/questions/2489/cpu-cores-the-more-the-better https://superuser.com/questions/2489/cpu-cores-the-more-the-...
- metaobject 9y agoYou can't spend your way out of the latency issues associated with such a distributed architecture, though. That's one of things NVLink helps with, actually.
- throwaway84742 9y ago100 GPU rig of 1080ti’s would have 1.1TB VRAM overall though. And most jobs don’t scale well beyond 4 GPUs anyway.
- llukas 9y agoThat is problem DGX-2 interconnect solves.
- throwaway84742 9y agoNot for currently available deep learning frameworks.
- llukas 9y agoYou're right but only because DGX-2 is not shipping yet. ;P
- shaklee3 9y agoNot only that, but the switch has the exact same throughput as the actual memory of the V100 until you get to fairly large block sizes. It was shown in a presentation.