3 ms·
The experimental setup involves using a cluster of 56 machines connected by an InfiniBand FDR network That bit might have something to do with it.
by HopenHeyHi 3y ago
The experimental setup involves using a cluster of 56 machines connected by an InfiniBand FDR network
That bit might have something to do with it.
- justinclift 3y agoHeh Heh Heh A bulk purchase of ~60 FDR IB cards, cabling, and network switches to support them sounds pretty expensive. That being said, IB FDR gear is "older tech" now so the cards and switches can commonly be found reasonably cheaply on Ebay. The switches tend to be bloody loud though, so they're not something you'd want nearby if you can help it.
- necubi 3y agoAmazon will rent you infiband-class machines like the C6in.metal (https://instances.vantage.sh/aws/ec2/c6in.metal https://instances.vantage.sh/aws/ec2/c6in.metal) with 200Gb/s of bandwidth. With EFA (https://aws.amazon.com/hpc/efa/ https://aws.amazon.com/hpc/efa/) you can use HPC features like rDMA.