In general the trend is toward more complex architectures, which increasingly blurs the line between modular nets and sparsely connected parts of a single model

In general the trend is toward more complex architectures, which increasingly blurs the line between modular nets and sparsely connected parts of a single model. IMO it's only meaningful to talk about modularity if the modules can be separately trained, or at least have their own loss functions. But of course its always useful to train or finetune the entire system end-to-end. A semi-recent success story is Tacotron 2, a speech synthesizer that has one model to convert text to a spectrogram, and a second model to convert that spectrogram into pcm (waveform) data. Also, Francois Chollet, the author of the popular deep learning library Keras, has some interesting thoughts on the future of ML along these lines that I find compelling: https://blog.keras.io/the-future-of-deep-learning.html https://blog.keras.io/the-future-of-deep-learning.html