2 ms·
Why Do Researchers Care About Small Language Models?
- mdp2021 2y agoSmall article with little depth, but it mentions a few core points. One of them is related to a statement that got me a puzzling lynching a few days a go: after a member rejoiced after seeing that AMD's new LLMs are based on open material, I asked why exactly, noting that «A great part of the important material for knowledge and mandatory intellectual exercise is not "open" (possibly "accessible", yet under copy rights)». Now this article mentions how > “The reason [Small Language Models] get so good with such small models and such little data is that they use high-quality data instead of the messy stuff” [Zico] Kolter[, Carnegie Mellon Uni] said This should be valid not just for "distillation", but also for the original dataset (collect our intellectual output as opposed to vast random output...). And, while it may be true that "quantity can bring emergency", this phenomenon should be studied quantitatively and qualitatively, so we can calibrate the resources to get what we want.