3 ms·

I don't think it has anything to do with being smarter than anyone else. We are still learning, and tuning, based on customer feedback and our own continuous te

by _msw_ 6y ago

I don't think it has anything to do with being smarter than anyone else. We are still learning, and tuning, based on customer feedback and our own continuous testing. Cloud infrastructure evolves rapidly, and that can present challenges in repeatability when that goal is not taken into consideration with how changes are deployed. This is something we're listing to customers on as well, and trying to find a good balance between delivering predictable and repeatable large scale experiments with making continuous improvements in performance and capabilities. One way to ensure repeatability is to build a static cluster that does not materially improve in performance or capabilities (from the underlying infrastructure perspective) during its installed life. I'll go back and review the Twittwr thread to see if I could have said things better. There is no firm position on if the underlying SRD transport algorithms should be publically standardized or not. Practically speaking, I think it would be premature to do so at this moment. But that does not mean it is out of the question for the future, nor does it mean that more information about the implement cannot be shared with customers and the broader HPC community. Generally speaking, customer feedback is an important part of helping drive the overall direction and prioritization.

stonogo 6y ago
"Releasing this would be premature" is always the line that comes out of private research, with the inevitable result that by the time the company decides release is no longer premature, fundamental flaws are enshrined at so deep a level as to cause the technology to be abandoned. Protocols developed in a small world tend to suffer from overfit to the business case that's funding them. Technology developed collaboratively with a wider research community is generally more successful in the long run. This is especially true in HPC, where the dominance of Infiniband comes in part from the availability of OpenSM (i.e., you don't need a UFM license to use the fabric, and you can implement your own routing topology using available tooling). If the intent behind SDR is as a stunt showpiece to sell Nitro instances, then I'm barking up the wrong tree. If the hope of the developers is that it makes a meaningful difference in the HPC world, then the time to open up is yesterday. I find it deeply confusing that Amazon seems to realize this enough to present a paper to IEEE for peer review but not enough to actually participate in open science. I hope your organization reconsiders this position.

by _msw_ 6y ago

stonogo 6y ago

"Releasing this would be premature" is always the line that comes out of private research, with the inevitable result that by the time the company decides release is no longer premature, fundamental flaws are enshrined at so deep a level as to cause the technology to be abandoned. Protocols developed in a small world tend to suffer from overfit to the business case that's funding them. Technology developed collaboratively with a wider research community is generally more successful in the long run. This is especially true in HPC, where the dominance of Infiniband comes in part from the availability of OpenSM (i.e., you don't need a UFM license to use the fabric, and you can implement your own routing topology using available tooling). If the intent behind SDR is as a stunt showpiece to sell Nitro instances, then I'm barking up the wrong tree. If the hope of the developers is that it makes a meaningful difference in the HPC world, then the time to open up is yesterday. I find it deeply confusing that Amazon seems to realize this enough to present a paper to IEEE for peer review but not enough to actually participate in open science. I hope your organization reconsiders this position.