I think the implication here is that it is learned during pre-training. You'll usually do some sort of pre-training (think running a model through imagenet) tha

I think the implication here is that it is learned during pre-training. You'll usually do some sort of pre-training (think running a model through imagenet) that boils the image down into the vector embeddings, and then you have the model reconstruct the original image using only the vectors. This is basically a reduced order modeling approach. After you have an encoder that is able to turn images into vector embeddings that contain enough information to reconstrust the original image, you can then use that pre-trained network and apply it to another task with similar images. e.g. you pre-trained on imagenet reconstruction, and then you adjust the task to detect whether there is a dog riding a bicycle in the image through regular supervised training. This approach has proven to be better than just training a network from scratch (or a generic pre-trained network) to detect bicycle dogs.