Two main things you can do: 1) Transfer learning -- start with self supervised model and either fine tune parameters or freeze parameters + add another layer t

Two main things you can do: 1) Transfer learning -- start with self supervised model and either fine tune parameters or freeze parameters + add another layer to train your task (with way fewer params/necessary labels since you already learned about the input distribution) 2) Nearest neighbor/clustering -- no need to label all classes, simply fetch similar examples (eg find semantically similar sentences in a corpus).