By leaning on inductive biases, we've historically been able to get pretty far with relatively small models and litte data. The trouble is, applying inductive

By leaning on inductive biases, we've historically been able to get pretty far with relatively small models and litte data. The trouble is, applying inductive biases requires that you make assumptions about the problem, which requires that you understand the problem. The stronger the inductive bias, the stronger the assumptions. Creating fantasticly large models allows us to eschew the inductive bias in favour of model capacity. That capacity would lead to useless, fragile models which overfit without mountains of data. It's great to explore the maxima of either approach so we can define some kind of loose Pareto frontier of model size and performance. I do think there's tonnes of room to benefit from the pissing contest the tech giants are currently engaged in and learn from the massive models what can be done for smaller models. The RAG pipeline is kind of an approach to add some inductive bias to such large models. Currently people tend to use the same giant causal LLMs for RAG as they do for non-RAG Chat purposes, but I'm certain you can get much smaller models performing very well for RAG. In fact, I'm sure plenty of folks are doing it already considering the number of eyeballs on LLM and RAG atm.