Yes, that's increasingly the case, but there's no fundamental reason for nets to be trained end-to-end as a single entity. Going back a few years it used to be

Yes, that's increasingly the case, but there's no fundamental reason for nets to be trained end-to-end as a single entity. Going back a few years it used to be quite common for people to use fixed word embeddings such as word2vec rather than learning them, and for image classification to take an ImageNet-pretrained general purpose model, then freeze the lower convolutional feature-detector layers and only train a new model "head" for more specialized use. End-to-end learnt embeddings are going to be more optimal though, and in the context of these massive models the computational cost of training them is a drop in the bucket!