Right, so parallelization is usually for special cases, and you're absolutely right that some people will still want parallelization. What I meant is that most

Right, so parallelization is usually for special cases, and you're absolutely right that some people will still want parallelization. What I meant is that most labs probably will be fine with the speed we offer. But there are some cases in which you will still want to parallelize, primarily for AI supercomputers. For applications such as supercomputing, you need to move to something that does a lot of work per iteration but makes much more progress per iteration. The traditional way to do this is with big batch sizes, but this can lead to "sharp" local minima (I'm aware of the recent papers purporting to achieve good generalization with large mini batch sizes and another achieving good generalization with sharp minima, but my own experiments ad their reported ones leave me unconvinced) which don't lead to good accuracy. As a result, we've had to develop our own way to do that, separate from the interconnect scheme. As far as a little bit more about the details of the interconnect scheme for HPC/supercomputing (please note that it's DL specific, although there are some general purpose aspects), I'd be pretty reluctant naturally ;), but I'd be willing to go more in detail in private. Tl;dr: What I meant is that it's good enough for most labs, but we can still parallelize better for HPC/supercomputing applications, we just don't recommend it for most users.