3 ms·

For inference I agree! But training requires centralized gradient steps

• vajrabum 3y ago
  If you're in a data center and running large training jobs then RDMA over Nvidia Mellanox Infiniband cards over high speed ethernet (like 100GB) are used to ship coefficients around without having that transfer bottleneck in the CPU.
  • xxpor 3y ago
    100 gig, that's considered cute nowadays. https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances-powered-by-nvidia-h100-tensor-core-gpus-for-accelerating-generative-ai-and-hpc-applications/ https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances... 3.2 terabits.
    • dekhn 3y ago
      I don't think that machine has a single nic with that bandwidth- I'd guess it's 8 400Gbps cards or something similar.
      • asciimike 3y ago
        Correct, it's 8x 400G cards, one per GPU.