I'm not an expert, but I feel that: Unsupervised techniques work really well for language modelling. There is also weakly supervised and distant-supervision,

I'm not an expert, but I feel that: Unsupervised techniques work really well for language modelling. There is also weakly supervised and distant-supervision, where the labels are "noisy" or not exactly what you want. You're right in that strong supervision, where you basically trust your class label, works really well, because it's probably the easiest case. Combining unsupervised (e.g. pre-trained language models) with a very small set of strongly labeled data, or a larger set of weakly labeled data, seems to work pretty well too.