The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN la

The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN layers has much more parameters than CNN layers, and all-reduce for gradients on these FCN layers are not economical (BTW, that's why Transformers do mixed model / data parallelism as well, or do large batches, they have a lot parameters and want to do these all-reduce as late as possible (or avoid at all, with model parallelism)). Otherwise the peer comment answered your question really well!