16 ms·
Supercomputing on Nitro in AWS Cloud
- WoahNoun 6y agoThe Monday Night Infrastructure keynote from re:Invent was all about the changes required to support true super-computing use cases inside a placement group. The nitro virtualization layer is a huge part of it (the other being the networking between hosts within a placement group). https://www.youtube.com/watch?v=GPUWATKe15E https://www.youtube.com/watch?v=GPUWATKe15E
- ZeroCool2u 6y ago"Instead of preserving packets order, SRD sends the packets over as many network paths as possible, while avoiding overloaded paths." From my naive perspective, this sounds like a great way to end up overloading all the paths that aren't currently overloaded. I suppose maybe this works out when you have global information about the network, unlike the regular internet though?
- convolvatron 6y agoexcellent point. even if you do, i think this is a pretty substantial measurement and control theory problem.
- fanf2 6y agoAIUI the sender manipulates bits in the packets that the switches use for ECMP (equal cost multipath) to shard the traffic across the available paths. The protocol is designed for very fast feedback about path latency so it can avoid building up queues in switches; if any path seems congested then the sender stops using its ECMP value. There isn’t any duplication of packets (unless they are lost) nor does it rely on global information (because that is far too slow). (Based on this discussion https://twitter.com/_msw_/status/1297223835519815681?s=21 https://twitter.com/_msw_/status/1297223835519815681?s=21 as well as the paper)
- stonogo 6y agoThis seems like Amazon trying to make excuses for being the only cloud provider who thinks they can pursue HPC workloads without Infiniband. I can't help but notice that this paper consists of nothing but network bandwidth testing, and their latency testing was bulk-only. Also, IEEE Micro is a weird place to publish real supercomputing work. Does Amazon have a plan for tightly-coupled large-scale software on AWS? Is there an exascale team at AWS? edit: I found a thread on Twitter[1] where it is explained to us that the protocol in question will remain proprietary because nobody is smart enough to keep up with Amazon engineering, and Amazon engineers' convenience is more important than standards. This is extremely disappointing -- it turns the entire fabric into a black box and slams the door on meaningful repeatability for large-scale modeling efforts. It's so frustrating to see all this effort fenced off behind 90s Business Dude mindsets. https://twitter.com/_msw_/status/1297231949895831553 https://twitter.com/_msw_/status/1297231949895831553
- wmf 6y agoOTOH, why should we want an Infiniband monoculture (which is also a Mellanox/Nvidia monopoly)? Why not let AWS try their weird stuff and let it fail on its merits?
- stonogo 6y agoBreaking the infiniband monoculture is why I am sad they're sitting on this protocol. I would like to deploy it to my cluster in my datacenter for testing. I cannot, and that sucks. Even if SRD is the greatest thing ever, it doesn't do me any good to replace an infiniband monoculture with an AWS monopoly. I can't even use public-cloud computing for about two thirds of my workload, so this is a non-starter.
- wmf 6y agoEven if the protocol was documented you couldn't use it without hardware which AWS won't sell you.
- 6y ago