1. > Self-supervised learning allows a neural network to figure out for itself what matters. 2. > Such “supervised” training requires data laboriously labeled

1. > Self-supervised learning allows a neural network to figure out for itself what matters. 2. > Such “supervised” training requires data laboriously labeled by humans, and the neural networks often take shortcuts, [...]. For example, a neural network might use the presence of grass to recognize a photo of a cow, because cows are typically photographed in fields. I don't really agree with that distinction. In that regard self-supervised and supervised are strictly equivalent. For 1., this is true of classic supervised model too, e.g. with an "old school" inception model trained on labeled Imagenet data, the bottom layers have learned more fundamental concepts that are generally useful such as edge detections, colours, texture and so on, that where never labeled or specifically required by the task. That's why a classic pattern is to freeze those layers when retraining the model for a new task. For 2., the exact same things happen in self-supervised learning, and in any ML task for that matter: gradient descent will always try to find the shortest path (after all, that's why it works). One common self-supervised learning task is to take a patch from an image and then ask the model to figure out which image that patch came from among a large set of images (contrastive loss). If you are not careful with how the images are transformed, the model can rely on effective but ultimately useless techniques to achieve this, a typical example being detecting artifacts in the picture linked to the type of camera and lens that was used to take the picture, allowing massively filter the potential images candidates to consider in the target batch. After all, self-supervised learning is just about procedurally generating the labels for a task instead of depending on humans to do so, but it is still supervised. Besides being able to leverage massive amount of data that would require the entire population of earth working for amazon mechanical turk to label, the real difference is that with SSL we can easily try many different tasks and find ones which lend themselves better (e.g. because they have the right level of difficulty) to teach the network generalizable concepts that will yield better performances on downstream tasks. But this is strictly the result of having a much shorter iteration/experiment loop, not of some qualitatively superior learning caused by the fact that the label was not human-made.