I do think there’s a paradigm shift coming. it’s a combination of the ongoing shift away from latency- to throughput oriented design with the capabilities shown

I do think there’s a paradigm shift coming. it’s a combination of the ongoing shift away from latency- to throughput oriented design with the capabilities shown in new interlinks, especially nvlink/nvswitch. This allows DGX-2 to already cover a fair amount of what would otherwise have to be programmed for midsized clusters - if it can be made to scale one more order of magnitude (i.e. ~ 10 DGX) I think there’s not much left that wouldn’t fit there but would fit something like Titan. There’s not that much so embarassingly parallel that the communication overhead doesn’t constrain it, and if doesn’t, you again don’t care much about data locality as it becomes trivial (e.g. compute intensive map function).