4 ms·

This seems like Amazon trying to make excuses for being the only cloud provider who thinks they can pursue HPC workloads without Infiniband. I can't help but n

This seems like Amazon trying to make excuses for being the only cloud provider who thinks they can pursue HPC workloads without Infiniband. I can't help but notice that this paper consists of nothing but network bandwidth testing, and their latency testing was bulk-only. Also, IEEE Micro is a weird place to publish real supercomputing work. Does Amazon have a plan for tightly-coupled large-scale software on AWS? Is there an exascale team at AWS? edit: I found a thread on Twitter[1] where it is explained to us that the protocol in question will remain proprietary because nobody is smart enough to keep up with Amazon engineering, and Amazon engineers' convenience is more important than standards. This is extremely disappointing -- it turns the entire fabric into a black box and slams the door on meaningful repeatability for large-scale modeling efforts. It's so frustrating to see all this effort fenced off behind 90s Business Dude mindsets. https://twitter.com/_msw_/status/1297231949895831553 https://twitter.com/_msw_/status/1297231949895831553

wmf 6y ago
OTOH, why should we want an Infiniband monoculture (which is also a Mellanox/Nvidia monopoly)? Why not let AWS try their weird stuff and let it fail on its merits?
- stonogo 6y ago
  Breaking the infiniband monoculture is why I am sad they're sitting on this protocol. I would like to deploy it to my cluster in my datacenter for testing. I cannot, and that sucks. Even if SRD is the greatest thing ever, it doesn't do me any good to replace an infiniband monoculture with an AWS monopoly. I can't even use public-cloud computing for about two thirds of my workload, so this is a non-starter.
  - wmf 6y ago
    Even if the protocol was documented you couldn't use it without hardware which AWS won't sell you.
    - stonogo 6y ago
      We'll never know. I have access to network hardware that is extremely configurable via FPGA. "What hardware does this protocol require for successful deployment" is just one of the many questions that are currently only answerable via Amazon trademarks.
      - wmf 6y ago
        I have a feeling that anyone competent enough to implement SRD in an FPGA could also invent their own similar protocol.
        stonogo 6y ago
        That may be true, but I'm not funded to develop similar protocols. I am funded to evaluate them, which, again, I cannot do because of how they're handling this work.
- gnufx 6y ago
  I agree on monoculture, the point is it does appear to fail (and the Cisco usNIC offering looked better). There are alternative HPC fabrics in new hardware: OPA (although being canned by Intel), Shasta (Cray), BXI (Bull), and Tofu (Fujitsu), at least, and surely AWS could build their own if they wanted. I don't know whether Oracle still have their own IB, which might be in their cloud if you want to get involved with Oracle. Edit: Fix Cisco fabric name, add OPA.
gnufx 6y ago
I'm not sure it has to be IB. Alibaba (I think) use RoCE, which can have fairly respectable latency. Which ones do have IB apart from Azure, Oracle, and specialist HPC vendors? But yes, when I last knew, EFA was still an order of magnitude worse latency than you expect for an HPC fabric, around the same as I got on 1G Ethernet a decade ago. Amazon assured us even the original networking was entirely suitable for HPC, and it took some work to get an admission that didn't include typical quantum chemistry, for instance. It actually claps out after three nodes on that sort of thing, though I didn't expect it to scale even that far. In contrast, Azure published figures for cp2k scaling (to >1k cores if I recall correctly).
- stonogo 6y ago
  Azure and Oracle were who I had in mind. I think I recall Google's sales team promising it Real Soon Now (i.e. as soon as a federal agency demanded it). I share your experience regarding public cloud's misunderstanding of HPC applications. It's great for the bioinformatics crowd and such but once you need to run nwchem or cp2k you need an interconnect that works. Now I'm disappointed to see that even if SDR turns out to be a good solution, it's so far AWS-only and deliberately opaque so they can screw with it without telling the customer. All part of the bad pattern we're stuck in, where vendors big enough to offer computing at scale don't understand supercomputing, and supercomputing experts get bought and blended into the background without ever scaling up to competitiveness. Shame HP Cloud got canned before HPE bought Cray.
gnufx 6y ago
This must have been downvoted by people who don't understand HPC, as usual here. While, typically, latency is the most important metric for an HPC fabric, it's not the only one that may be more important than bandwidth (message rate, contention, offloading...). It's also important to have detailed monitoring, which I guess you won't get from any such provider. Being told, even of the old networking, that it's all terribly clever and we shouldn't ask questions isn't a good way sales pitch.
_msw_ 6y ago
I don't think it has anything to do with being smarter than anyone else. We are still learning, and tuning, based on customer feedback and our own continuous testing. Cloud infrastructure evolves rapidly, and that can present challenges in repeatability when that goal is not taken into consideration with how changes are deployed. This is something we're listing to customers on as well, and trying to find a good balance between delivering predictable and repeatable large scale experiments with making continuous improvements in performance and capabilities. One way to ensure repeatability is to build a static cluster that does not materially improve in performance or capabilities (from the underlying infrastructure perspective) during its installed life. I'll go back and review the Twittwr thread to see if I could have said things better. There is no firm position on if the underlying SRD transport algorithms should be publically standardized or not. Practically speaking, I think it would be premature to do so at this moment. But that does not mean it is out of the question for the future, nor does it mean that more information about the implement cannot be shared with customers and the broader HPC community. Generally speaking, customer feedback is an important part of helping drive the overall direction and prioritization.
- stonogo 6y ago
  "Releasing this would be premature" is always the line that comes out of private research, with the inevitable result that by the time the company decides release is no longer premature, fundamental flaws are enshrined at so deep a level as to cause the technology to be abandoned. Protocols developed in a small world tend to suffer from overfit to the business case that's funding them. Technology developed collaboratively with a wider research community is generally more successful in the long run. This is especially true in HPC, where the dominance of Infiniband comes in part from the availability of OpenSM (i.e., you don't need a UFM license to use the fabric, and you can implement your own routing topology using available tooling). If the intent behind SDR is as a stunt showpiece to sell Nitro instances, then I'm barking up the wrong tree. If the hope of the developers is that it makes a meaningful difference in the HPC world, then the time to open up is yesterday. I find it deeply confusing that Amazon seems to realize this enough to present a paper to IEEE for peer review but not enough to actually participate in open science. I hope your organization reconsiders this position.

wmf 6y ago

OTOH, why should we want an Infiniband monoculture (which is also a Mellanox/Nvidia monopoly)? Why not let AWS try their weird stuff and let it fail on its merits?

stonogo 6y ago
Breaking the infiniband monoculture is why I am sad they're sitting on this protocol. I would like to deploy it to my cluster in my datacenter for testing. I cannot, and that sucks. Even if SRD is the greatest thing ever, it doesn't do me any good to replace an infiniband monoculture with an AWS monopoly. I can't even use public-cloud computing for about two thirds of my workload, so this is a non-starter.
- wmf 6y ago
  Even if the protocol was documented you couldn't use it without hardware which AWS won't sell you.
  - stonogo 6y ago
    We'll never know. I have access to network hardware that is extremely configurable via FPGA. "What hardware does this protocol require for successful deployment" is just one of the many questions that are currently only answerable via Amazon trademarks.
    - wmf 6y ago
      I have a feeling that anyone competent enough to implement SRD in an FPGA could also invent their own similar protocol.
      - stonogo 6y ago
        That may be true, but I'm not funded to develop similar protocols. I am funded to evaluate them, which, again, I cannot do because of how they're handling this work.
gnufx 6y ago
I agree on monoculture, the point is it does appear to fail (and the Cisco usNIC offering looked better). There are alternative HPC fabrics in new hardware: OPA (although being canned by Intel), Shasta (Cray), BXI (Bull), and Tofu (Fujitsu), at least, and surely AWS could build their own if they wanted. I don't know whether Oracle still have their own IB, which might be in their cloud if you want to get involved with Oracle. Edit: Fix Cisco fabric name, add OPA.

gnufx 6y ago

I'm not sure it has to be IB. Alibaba (I think) use RoCE, which can have fairly respectable latency. Which ones do have IB apart from Azure, Oracle, and specialist HPC vendors? But yes, when I last knew, EFA was still an order of magnitude worse latency than you expect for an HPC fabric, around the same as I got on 1G Ethernet a decade ago. Amazon assured us even the original networking was entirely suitable for HPC, and it took some work to get an admission that didn't include typical quantum chemistry, for instance. It actually claps out after three nodes on that sort of thing, though I didn't expect it to scale even that far. In contrast, Azure published figures for cp2k scaling (to >1k cores if I recall correctly).

stonogo 6y ago
Azure and Oracle were who I had in mind. I think I recall Google's sales team promising it Real Soon Now (i.e. as soon as a federal agency demanded it). I share your experience regarding public cloud's misunderstanding of HPC applications. It's great for the bioinformatics crowd and such but once you need to run nwchem or cp2k you need an interconnect that works. Now I'm disappointed to see that even if SDR turns out to be a good solution, it's so far AWS-only and deliberately opaque so they can screw with it without telling the customer. All part of the bad pattern we're stuck in, where vendors big enough to offer computing at scale don't understand supercomputing, and supercomputing experts get bought and blended into the background without ever scaling up to competitiveness. Shame HP Cloud got canned before HPE bought Cray.

gnufx 6y ago

This must have been downvoted by people who don't understand HPC, as usual here. While, typically, latency is the most important metric for an HPC fabric, it's not the only one that may be more important than bandwidth (message rate, contention, offloading...). It's also important to have detailed monitoring, which I guess you won't get from any such provider. Being told, even of the old networking, that it's all terribly clever and we shouldn't ask questions isn't a good way sales pitch.

_msw_ 6y ago

I don't think it has anything to do with being smarter than anyone else. We are still learning, and tuning, based on customer feedback and our own continuous testing. Cloud infrastructure evolves rapidly, and that can present challenges in repeatability when that goal is not taken into consideration with how changes are deployed. This is something we're listing to customers on as well, and trying to find a good balance between delivering predictable and repeatable large scale experiments with making continuous improvements in performance and capabilities. One way to ensure repeatability is to build a static cluster that does not materially improve in performance or capabilities (from the underlying infrastructure perspective) during its installed life. I'll go back and review the Twittwr thread to see if I could have said things better. There is no firm position on if the underlying SRD transport algorithms should be publically standardized or not. Practically speaking, I think it would be premature to do so at this moment. But that does not mean it is out of the question for the future, nor does it mean that more information about the implement cannot be shared with customers and the broader HPC community. Generally speaking, customer feedback is an important part of helping drive the overall direction and prioritization.

stonogo 6y ago
"Releasing this would be premature" is always the line that comes out of private research, with the inevitable result that by the time the company decides release is no longer premature, fundamental flaws are enshrined at so deep a level as to cause the technology to be abandoned. Protocols developed in a small world tend to suffer from overfit to the business case that's funding them. Technology developed collaboratively with a wider research community is generally more successful in the long run. This is especially true in HPC, where the dominance of Infiniband comes in part from the availability of OpenSM (i.e., you don't need a UFM license to use the fabric, and you can implement your own routing topology using available tooling). If the intent behind SDR is as a stunt showpiece to sell Nitro instances, then I'm barking up the wrong tree. If the hope of the developers is that it makes a meaningful difference in the HPC world, then the time to open up is yesterday. I find it deeply confusing that Amazon seems to realize this enough to present a paper to IEEE for peer review but not enough to actually participate in open science. I hope your organization reconsiders this position.