4 ms·

I'm not sure it has to be IB. Alibaba (I think) use RoCE, which can have fairly respectable latency. Which ones do have IB apart from Azure, Oracle, and speci

by gnufx 6y ago

I'm not sure it has to be IB. Alibaba (I think) use RoCE, which can have fairly respectable latency. Which ones do have IB apart from Azure, Oracle, and specialist HPC vendors? But yes, when I last knew, EFA was still an order of magnitude worse latency than you expect for an HPC fabric, around the same as I got on 1G Ethernet a decade ago. Amazon assured us even the original networking was entirely suitable for HPC, and it took some work to get an admission that didn't include typical quantum chemistry, for instance. It actually claps out after three nodes on that sort of thing, though I didn't expect it to scale even that far. In contrast, Azure published figures for cp2k scaling (to >1k cores if I recall correctly).

stonogo 6y ago
Azure and Oracle were who I had in mind. I think I recall Google's sales team promising it Real Soon Now (i.e. as soon as a federal agency demanded it). I share your experience regarding public cloud's misunderstanding of HPC applications. It's great for the bioinformatics crowd and such but once you need to run nwchem or cp2k you need an interconnect that works. Now I'm disappointed to see that even if SDR turns out to be a good solution, it's so far AWS-only and deliberately opaque so they can screw with it without telling the customer. All part of the bad pattern we're stuck in, where vendors big enough to offer computing at scale don't understand supercomputing, and supercomputing experts get bought and blended into the background without ever scaling up to competitiveness. Shame HP Cloud got canned before HPE bought Cray.