3 ms·
With interconnect being the biggest limitation these days I don’t think this would work.
by WanderPanda 3y ago
With interconnect being the biggest limitation these days I don’t think this would work.
- xnx 3y agoI'm not familiar with all the varied uses of GPUs but it seems like image generation could feasibly be distributed: large upfront download of models, then small inputs of text and settings, and small output of resulting images.
- WanderPanda 3y agoFor inference I agree! But training requires centralized gradient steps
- vajrabum 3y agoIf you're in a data center and running large training jobs then RDMA over Nvidia Mellanox Infiniband cards over high speed ethernet (like 100GB) are used to ship coefficients around without having that transfer bottleneck in the CPU.
- xxpor 3y ago100 gig, that's considered cute nowadays. https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances-powered-by-nvidia-h100-tensor-core-gpus-for-accelerating-generative-ai-and-hpc-applications/ https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances... 3.2 terabits.